Fix S3HttpHandler chunked-encoding handling #72378
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DaveCTurner
added
>test
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
The `S3HttpHandler` reads the contents of the uploaded blob, but if the upload used chunked encoding then the reader would skip one or more `\r\n` sequences if they appeared at the start of a chunk. This commit reworks the reader to be stricter about its interpretation of chunks, and removes some indirection via streams since we can work pretty much entirely on the underlying `BytesReference` instead. Closes elastic#72358
DaveCTurner
force-pushed
the
2021-04-28-fix-s3-fixture-chunk-handling
branch
from
16d96ca to
845e03e
Compare
original-brownbear
approved these changes
Apr 28, 2021
There was a problem hiding this comment.
I think #72365 is the simpler option, but this one LGTM as well.
If we want to go with this one, can we add https://github.com/elastic/elasticsearch/pull/72365/files#diff-47fd5429e7187cd1b173e53b38a5896c3810c6b89890b84e89334536288c6b84R124 to it to get more (and easier to debug) coverage on this though? :)
|
Ok I shamelessly pinched some of your ideas from #72365. |
tlrx
approved these changes
Apr 28, 2021
DaveCTurner
added a commit
that referenced
this pull request
Apr 28, 2021
The `S3HttpHandler` reads the contents of the uploaded blob, but if the upload used chunked encoding then the reader would skip one or more `\r\n` sequences if they appeared at the start of a chunk. This commit reworks the reader to be stricter about its interpretation of chunks, and removes some indirection via streams since we can work pretty much entirely on the underlying `BytesReference` instead. Closes #72358
DaveCTurner
added a commit
that referenced
this pull request
Apr 28, 2021
The `S3HttpHandler` reads the contents of the uploaded blob, but if the upload used chunked encoding then the reader would skip one or more `\r\n` sequences if they appeared at the start of a chunk. This commit reworks the reader to be stricter about its interpretation of chunks, and removes some indirection via streams since we can work pretty much entirely on the underlying `BytesReference` instead. Closes #72358
|
Thanks both 🚀 |
DaveCTurner
added a commit
that referenced
this pull request
Apr 28, 2021
The `S3HttpHandler` reads the contents of the uploaded blob, but if the upload used chunked encoding then the reader would skip one or more `\r\n` sequences if they appeared at the start of a chunk. This commit reworks the reader to be stricter about its interpretation of chunks, and removes some indirection via streams since we can work pretty much entirely on the underlying `BytesReference` instead. Closes #72358
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
S3HttpHandlerreads the contents of the uploaded blob, but if theupload used chunked encoding then the reader would skip one or more
\r\nsequences if they appeared at the start of a chunk.This commit reworks the reader to be stricter about its interpretation
of chunks, and removes some indirection via streams since we can work
pretty much entirely on the underlying
BytesReferenceinstead.Closes #72358