Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support alternative S3 endpoints #597

Open
jdidion opened this issue Aug 31, 2022 · 1 comment
Open

Support alternative S3 endpoints #597

jdidion opened this issue Aug 31, 2022 · 1 comment

Comments

@jdidion
Copy link

jdidion commented Aug 31, 2022

The aws cli supports alternative S3 endpoints with the --endpoint-url option. It would be great if miniwdl could expose some method of setting an alternative endpoint. I would be happy to implement this and submit a PR, but I want to do it in the right way. Specifically, I want to do it in a way that will also work with the Amazon Genomics CLI. @mlin I'd appreciate suggestions.

** Eventually this can be done much more cleanly via environment variables once aws/aws-sdk#230 is merged.

@mlin
Copy link
Collaborator

mlin commented Sep 1, 2022

@jdidion Thanks, for downloads this would be a straightforward modification to the downloader code here.

  1. The way it asks which docker image to use for AWS CLI provides an example of how to get the endpoint URL from configuration.
  2. Note there's a separate, similar Directory downloader below the File downloader, which we'd presumably want to modify in the same way.
  3. S3 downloads in AGC go through this codepath but uploads run separately -- not sure if you need one or the other or both.
  4. AGC uses a pinned docker image with miniwdl and miniwdl-aws baked in, so when it would be live in AGC would depend on when their devs have time/motivation to pull in a new version and release it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants