-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow reading datasets from private s3 bucket #74
Conversation
313c547
to
f6f504d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation looks good, but I think that we can get rid of the old urllib
approach and not require the user to pass the bucket name including the s3://
prefix on it.
a4ee416
to
5df83bf
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks almost ready, but there is a situation that should be fixed to allow system-wide credentials to be used.
726bb4e
to
f7c2700
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one last code style change (line break after if/else block) and this is ready to go :-)
f7c2700
to
4724aee
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
Add functionality to download datasets from a private s3 bucket.
boto3
needs to be used instead of urllib.✅ Verified that e2e flow works when pulling data from a public s3 bucket
✅ Verified that e2e flow works when pulling data from a private s3 bucket