Add an owner.txt to the ingestion bucket #67
Conversation
| parameters: | ||
| BucketName: !stack_output_external recover-dev-ingestion-bucket::Bucket | ||
| SynapseIds: "3461799,3455604" # RecoverETL and synapse-service-sysbio-recover-data-storage-01 | ||
| OwnerTxtKeyPrefix: {{ stack_group_config.namespace }} |
There was a problem hiding this comment.
Was there a particular reason we are removing this for the prod version and having the owner.txt being written to the root of the bucket? i think the prod config for the input bucket has this
There was a problem hiding this comment.
We don't really control the organization of data in the production ingestion bucket. So far data has been written to two different folders at the bucket root (adults\/v1 and pediatric\/v1), so unless we move the production data elsewhere, digital health will need to have access to the bucket root and update their scripts to copy data to that "elsewhere" folder. Having everything at the bucket root doesn't really bother me, because there is only one set of production data. We might send that data through different namespaces (e.g., main and staging), but there isn't a separate set of production data for main and another for staging.
Synapse already has read/write access to the ingestion bucket, so the only thing to do to enable the use of the bucket as an external storage location is to write an owner.txt.
In develop, owner.txt is written to
main.In prod, owner.txt is written to the bucket root. This is where production data (e.g.,
adults\/v1andpediatric\/v1) already live.