Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WORKFLOWS-70] Create public buckets for iGenomes reference files #74

Merged
merged 2 commits into from
Nov 10, 2021

Conversation

BrunoGrandePhD
Copy link
Contributor

I was hoping to re-use the s3-bucket-v2.j2 template from aws-infra, but I didn't want to have to deal with the S3objects macro, especially because it's not relevant to our use case. I opted for adapting the template for our purposes.

I'm still figuring out how to copy the files. Unfortunately, interacting with the nf-core bucket requires using --no-sign-request, which then prevents the requests to our bucket to work since they're unauthenticated (see example command below). I'm currently staging files locally on a Service Catalog instance. I included a script that I will run once the bucket is deployed to production.

aws s3 --no-sign-request sync "s3://source-bucket/" "s3://target-bucket/"

@BrunoGrandePhD BrunoGrandePhD requested a review from a team as a code owner November 4, 2021 23:42
@BrunoGrandePhD
Copy link
Contributor Author

I can confirm that the region restrictions works as expected:

# From a Service Catalog instance in us-east-1
$ aws s3 cp s3://sage-igenomes-dev/igenomes/Homo_sapiens/Ensembl/GRCh37/Sequence/Chromosomes/MT.fa - | head -n 1
>MT

# From my personal computer in Seattle
$ aws s3 cp s3://sage-igenomes-dev/igenomes/Homo_sapiens/Ensembl/GRCh37/Sequence/Chromosomes/MT.fa - | head -n 1
download failed: s3://sage-igenomes-dev/igenomes/Homo_sapiens/Ensembl/GRCh37/Sequence/Chromosomes/MT.fa to - An error occurred (403) when calling the HeadObject operation: Forbidden

@BrunoGrandePhD BrunoGrandePhD changed the title Create public buckets for iGenomes reference files [WORKFLOWS-70] Create public buckets for iGenomes reference files Nov 4, 2021
Copy link
Collaborator

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@BrunoGrandePhD BrunoGrandePhD merged commit 6f1ea98 into main Nov 10, 2021
@BrunoGrandePhD BrunoGrandePhD deleted the bgrande/workflows-70/igenomes-bucket branch November 10, 2021 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants