Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly handle unsigned chunked uploads #1785

Merged
merged 14 commits into from
Apr 28, 2024

Conversation

afranken
Copy link
Member

@afranken afranken commented Apr 12, 2024

Description

  • Prepare ITs for different test files
    • Many hard-coded paths, sizes etc make it hard to test with files other than the default "sampleFile.txt" which is 36bytes in size. Using even slightly larger payloads leads to errors during uploads.
  • Correctly handle unsigned chunked uploads
    • Known issue: when using one of the Async SDK clients, uploads sometimes fail when the file size is >16KB. Not yet sure why. Uploads <16KB work just fine.

Related Issue

#1662

Tasks

  • I have signed the CLA.
  • I have written tests and verified that they fail without my change.

@afranken afranken added the bug label Apr 12, 2024
@afranken afranken self-assigned this Apr 12, 2024
@afranken afranken force-pushed the 1662-fix-multipart-upload-problems branch from d62fb0e to 1274eb6 Compare April 12, 2024 15:25
Many hard-coded paths, sizes etc make it hard to test with files other
than the default "sampleFile.txt" which is 36bytes in size.
Using even slightly larger payloads leads to errors during uploads.

#1662
@afranken afranken force-pushed the 1662-fix-multipart-upload-problems branch 3 times, most recently from 3c3ce24 to ee58208 Compare April 12, 2024 16:15
Some http clients (like the AWS SDKs) do not cope well if we return
errors from APIs before consuming the incoming stream.
Make sure we always consume all bytes before processing the streams.

#1662
@afranken afranken force-pushed the 1662-fix-multipart-upload-problems branch from ee58208 to 8fb575b Compare April 12, 2024 16:16
afranken and others added 7 commits April 12, 2024 18:35
This just uses AWS SDKv2 utility classes for doing the heavy lifting.
Use more Kotlin constructs
Make tests more readable
Make tests more stable for exchanging the default input file
Make tests more stable for exchanging the default input file
Write incoming streams directly to a temp file, removing chunk
information, optionally read checksum information.

Fixes #1662
@afranken afranken force-pushed the 1662-fix-multipart-upload-problems branch from af0fa6a to 9f41990 Compare April 24, 2024 20:42
@afranken afranken changed the title Fix Multipart upload issues Correctly handle unsigned chunked uploads Apr 26, 2024
This updates Jetty to 12.0.8 where they changed the UriCompliance
yet again. We need to use "UNSAFE" now.
jetty/jetty.project#11495
@afranken afranken force-pushed the 1662-fix-multipart-upload-problems branch from 1d07b75 to 3468740 Compare April 28, 2024 13:22
@afranken afranken merged commit 42abe2a into main Apr 28, 2024
6 checks passed
@afranken afranken deleted the 1662-fix-multipart-upload-problems branch April 28, 2024 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant