Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use COS/S3 presign-url for download URLs of bulkdata export results #882

Closed
albertwang-ibm opened this issue Apr 1, 2020 · 2 comments
Closed

Use COS/S3 presign-url for download URLs of bulkdata export results #882

albertwang-ibm opened this issue Apr 1, 2020 · 2 comments
Assignees
@prb112
Labels
bulk-data enhancement New feature or request
Milestone
Sprint 2021-03

Comments

@albertwang-ibm
Copy link
Contributor

albertwang-ibm commented Apr 1, 2020 •
edited by prb112
Loading

Summary
Add an option to use a temporary url using COS/S3 presign-url for download URLs of bulkdata export results

Details
(1) The download url will be something like following which is much longer than current download url and it's in different format from bulkexport spec.

?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIAJQ6UAEQOACU54C3A%2F20180927%2Feu-west-1%2Fs3%2Faws4_request
&X-Amz-Date=20180927T100139Z
&X-Amz-Expires=900
&X-Amz-Signature=f6fa35129753e7626c850a531379436a555447bfbd597c19e3177ae3d2c48387
&X-Amz-SignedHeaders=host

(2) cleanup or expiration the generated export results themselves is out of scope of this issue.

https://cloud.ibm.com/docs/services/cloud-object-storage/hmac?topic=cloud-object-storage-presign-url#presign-url-get-java
@albertwang-ibm albertwang-ibm added the enhancement New feature or request label Apr 1, 2020
JohnTimm added a commit that referenced this issue Apr 14, 2020
@JohnTimm
Issue #882 - Initial implementation of ServerRegistryResourceProvider
c5fa4e9 
Signed-off-by: John T.E. Timm <johntimm@us.ibm.com>
@prb112 prb112 added this to the Sprint 2021-03 milestone Feb 27, 2021
@prb112 prb112 self-assigned this Feb 27, 2021
@prb112 prb112 linked a pull request Feb 27, 2021 that will close this issue
Improve ImportExport with JavaBatch #1999
Merged
prb112 added a commit that referenced this issue Feb 28, 2021
@prb112
Improve ImportExport with JavaBatch
35bf619 
- Fix output URL so that it is an actual URL

- Implements Use COS/S3 presign-url for download URLs of bulkdata export
results #882

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
@prb112 prb112 removed a link to a pull request Mar 9, 2021
Improve ImportExport with JavaBatch #1999
Merged
This was referenced Mar 11, 2021
Closed
Closed
@lmsurpre
Copy link
Member

I set presigned to true, configured an ibm-cos storageProvider with hmac auth, and set public to true. Then I performed export and confirmed that:

  1. I could download the resulting export by using the URL with the X-Amz query parameters
  2. I could not download the resulting export without passing the X-Amz query parameters

I will try to circle back and confirm that these download URLs are no longer accessible after 24 hours (the current validity period), and I've opened #2053 for making this configurable in the future.

@lmsurpre
Copy link
Member

Confirmed that the pre-signed URLs generated on Friday are no longer accessible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prb112 prb112

Labels
bulk-data enhancement New feature or request
Projects
None yet
Milestone
Sprint 2021-03
Development

No branches or pull requests
3 participants
@prb112 @lmsurpre @albertwang-ibm