Store LVK alerts in BigQuery #232
Conversation
| # create bigquery dataset and table | ||
| bq --location="${region}" mk --dataset "${bq_dataset}" | ||
|
|
||
| cd templates || exit |
There was a problem hiding this comment.
should have an exit code (and ideally, helpful message) to debug if an issue occurs.
| bq --location="${region}" mk --dataset "${bq_dataset}" | ||
|
|
||
| cd templates || exit | ||
| bq mk --table "${PROJECT_ID}:${bq_dataset}.${alerts_table}" "bq_${survey}_${alerts_table}_schema.json" |
There was a problem hiding this comment.
The better way to do this is to use a subshell. Then you're not changing the implicit global state.
cd templates
bq mk --table ....
-> (cd templates && bq mk --table ....) || exit 5
There was a problem hiding this comment.
Thank you. We should make this replacement in a lot of places (#240).
There was a problem hiding this comment.
It would be good to have unit tests on this, but IIRC it's not a capability yinz have atm.
There was a problem hiding this comment.
@hernandezc1 Add a comment at the top explaining where the information in this file came from and include a url. We would like to be able to point at a machine-readable schema definition (like an avro schema file) provided by the survey. But I think we haven't found one for LVK, is that right?
If you had to pull this together from info on their website, you probably had to make more decisions here than we usually do. @tregle pointed out good things to consider when creating a schema (thanks! It encouraged me to give this file a better look as well). Some of them we probably want to implement, but some of them we probably shouldn't because we aren't the data producers. (And a comment at the top can help make this clear.) For example, we don't want to change a field name, but we could add units to a description if it would make things more clear and we can find/point to them in LVK's docs. But you're the one with the most background here so you'll have to decide specifics.
One other question, How did you decide on the mode and type values here? If we have an avro schema, we can use it to create the bigquery table, and then download this json version for future use. That's the easiest way to make this schema match (as much as possible) the alerts' schema, which is really what it needs to do. (And we should have a bit of code in here that will do that. #239) If you haven't exhausted all the avenues yet, look/ask again to see if you can get your hands on a real schema definition from LVK. Avro would be great, but json or anything else they're using on the backend would be helpful (and a schema version?).
This PR creates the necessary files and subdirectories needed to store LIGO/Virgo/KAGRA (LVK) alert information in BigQuery.
Here is a more in-depth explanation of how things work.
setup_broker dir:
The
setup_broker/lvkdirectory contains several bash scripts that are used to deploy an instance of our broker that is specific to LVK. In this PR, I've updated the "main" deployment script (setup_broker.sh) so that it defines and creates (or deletes) the necessary BigQuery and Pub/Sub resources to store LVK alert information in BigQuery.The idea here is that LVK alerts that our consumer publishes to a Pub/Sub topic (
lvk-alerts) will trigger our Cloud Function (lvk-storeBigQuery), and the Cloud Function will insert the alert data into a BigQuery table and publish a message to a new Pub/Sub topic (lvk-BigQuery). The message published tolvk-BigQuerycontains the original alert information as well as the alert type and the name of the table to which the alert information was stored to as message attributes.In
setup_broker/lvkyou'll also see that I've added atemplatedirectory, which contains the schema of the BigQuery table.cloud_functions dir
The
cloud_functions/lvkdirectory contains four new files: aREADME, arequirements.txtfile, the cloud function's deployment script (deploy.sh) and a Python script (main.py) used to store the alert information in BigQuery and publish a message to the topiclvk-BigQuery.main.pynow uses thepittgoogle-client.