chore(eap): write script to send scrubbed data into a gcs bucket #6698
Conversation
| file_name = f"scrubbed_spans_data_{current_time}.csv.gz" | ||
|
|
||
| query = f""" | ||
| INSERT INTO FUNCTION gcs('https://storage.googleapis.com/{self.gcp_bucket_name}/{file_name}', |
There was a problem hiding this comment.
Dump that into a directory with the date of the dump.
You can probably make that one variable to be called later in the log.
|
|
||
| def execute(self, logger: JobLogger) -> None: | ||
| cluster = get_cluster(StorageSetKey.EVENTS_ANALYTICS_PLATFORM) | ||
| connection = cluster.get_query_connection(ClickhouseClientSettings.QUERY) |
There was a problem hiding this comment.
Let's do one query that does an INSERT INTO <gcs_bucket> (SELECT ... FROM eap_spans_2_local) instead.
And make sure it's running asynchronous so this script can stop with success.
There was a problem hiding this comment.
Let's do one query that does an INSERT INTO <gcs_bucket> (SELECT ... FROM eap_spans_2_local) instead.
How is this different from what I already have?
And make sure it's running asynchronous so this script can stop with success.
I don't think snuba supports running async queries, but I think we can add support for it with https://github.com/mymarilyn/aioch
No description provided.