-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimizations in s3 async upload flow and guards against S3 async SDK… #11327
Optimizations in s3 async upload flow and guards against S3 async SDK… #11327
Conversation
Can we split this PR up? Seems like there are a couple of mostly unrelated changes in this PR (namely the guard stream access after callback seems unrelated to the large upload changes).
Since this is intended as a temporary fix, can we do it without introducing a user-facing repository setting? Similar question related to the upload retry setting...is this something we want to be configurable in the long term? |
@andrross Settings should be present due to added functionalities and these are dynamic repository settings which users can control. Tomorrow when we replace any of it then settings will just be ineffective. We won't run into any backward compatibility issue. We can call this out in CHANGELOG for informational purposes when these are replaced. |
26350ff
to
94cde48
Compare
❌ Gradle check result for 94cde48: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
94cde48
to
cb924a1
Compare
❕ Gradle check result for cb924a1: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #11327 +/- ##
============================================
+ Coverage 71.25% 71.38% +0.13%
- Complexity 58936 58981 +45
============================================
Files 4890 4890
Lines 277447 277497 +50
Branches 40313 40317 +4
============================================
+ Hits 197696 198101 +405
+ Misses 63338 62914 -424
- Partials 16413 16482 +69 ☔ View full report in Codecov by Sentry. |
cb924a1
to
29d11f6
Compare
29d11f6
to
7103f3d
Compare
❕ Gradle check result for 29d11f6: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
7103f3d
to
9463c0a
Compare
❌ Gradle check result for 9463c0a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: vikasvb90 <vikasvb@amazon.com>
9463c0a
to
a1ebc61
Compare
❕ Gradle check result for a1ebc61: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
The backport to
To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11327-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 77a4daf3a13866ab1887dc4174f63ff1fb9912fb
# Push it to GitHub
git push --set-upstream origin backport/backport-11327-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x Then, create a pull request where the |
Signed-off-by: vikasvb90 <vikasvb@amazon.com> (cherry picked from commit 77a4daf) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
(cherry picked from commit 77a4daf) Signed-off-by: vikasvb90 <vikasvb@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: vikasvb90 <vikasvb@amazon.com>
Signed-off-by: vikasvb90 <vikasvb@amazon.com>
Signed-off-by: vikasvb90 <vikasvb@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Description
This PR optimizes S3 async upload flow and add guards against unexpected S3 SDK errors.
For points 1 and 2, a new PR is forked out : #11334
Check List
New functionality has been documented.New functionality has javadoc addedCommit changes are listed out in CHANGELOG.md file (See: Changelog)Public documentation issue/PR createdBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.