Removing backend workflow for writing plugin data to s3

[manasaV3]

Description

Relates to #1192

• Removes backend workflow to fetch and write data to s3.
• Removes other components that are no longer being used.

Notes

• We should validate that the plugin s3 files are no longer being used before merging this change.

[aganders3]

LGTM - I always love to see this kind of +/-!

I notice you removed datadog.py - perhaps this also means you can remove datadog-lambda from requirements.txt. Maybe even remove the whole Dockerfile? It has CMD ["datadog_lambda.handler.handler"] but I'm not sure if it's also used for something else.

Oh I guess also regarding the overall intent - I guess this means the new workflow writing to Dynamo is confirmed working? Is that documented somewhere?

[manasaV3]

I notice you removed datadog.py - perhaps this also means you can remove datadog-lambda from requirements.txt. Maybe even remove the whole Dockerfile? It has CMD ["datadog_lambda.handler.handler"] but I'm not sure if it's also used for something else.

This is a great call out. 😁 As there are some data that the datadog captures such as latency trace for the request, it is beneficial to retain the data-dog component in the Dockerfile for now, this, in turn, requires retaining the datadog-lambda requirement.

Oh I guess also regarding the overall intent - I guess this means the new workflow writing to Dynamo is confirmed working? Is that documented somewhere?

We validated the data being written to Dynamo using feature flags. The front end has been rendering the data from Dynamo for a few weeks. In addition, I also ran some scripts to ascertain the data parity between s3 and dynamo data. 😄

[aganders3]

Nice, thanks! I am a little familiar with DataDog but was not familiar with the lambda wrapper. I see what's happening here now. 😄

[github-actions]

This PR has been included in release: v23.08.1, see the release notes.