Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Add S3 async upload utilities and models #7217

Merged

Conversation

raghuvanshraj
Copy link
Contributor

@raghuvanshraj raghuvanshraj commented Apr 18, 2023

Credits: @vikasvb90 and @itiyamas for the design and core implementations of the feature.

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 1b6db1f to e29d3b6 Compare April 24, 2023 18:14
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from e29d3b6 to ea5db84 Compare May 4, 2023 06:53
@github-actions
Copy link
Contributor

github-actions bot commented May 4, 2023

Gradle Check (Jenkins) Run Completed with:

Copy link
Collaborator

@gbbafna gbbafna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still continuing with the review. these are quite involved changes and would need a comprehensive test plan/coverage as well .

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from ea5db84 to 530d43e Compare May 24, 2023 15:53
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 530d43e to 4e5d98a Compare May 25, 2023 03:49
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj changed the title Add S3 async upload utilities and models [Remote Store] Add S3 async upload utilities and models May 25, 2023
@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 4e5d98a to 6c12571 Compare May 25, 2023 08:04
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=tasks.list/10_basic/tasks_list test}

@codecov
Copy link

codecov bot commented May 25, 2023

Codecov Report

Merging #7217 (7ebc460) into main (7dddb31) will decrease coverage by 0.05%.
The diff coverage is 77.19%.

@@             Coverage Diff              @@
##               main    #7217      +/-   ##
============================================
- Coverage     70.95%   70.90%   -0.05%     
- Complexity    57056    57098      +42     
============================================
  Files          4759     4768       +9     
  Lines        269691   270169     +478     
  Branches      39454    39495      +41     
============================================
+ Hits         191359   191564     +205     
- Misses        62184    62412     +228     
- Partials      16148    16193      +45     
Impacted Files Coverage Δ
...a/org/opensearch/repositories/s3/SocketAccess.java 100.00% <ø> (ø)
...sitories/s3/async/AsyncTransferEventLoopGroup.java 46.15% <46.15%> (ø)
...org/opensearch/repositories/s3/S3AsyncService.java 69.49% <69.49%> (ø)
...ch/repositories/s3/async/AsyncTransferManager.java 73.79% <73.79%> (ø)
...search/repositories/s3/AmazonAsyncS3Reference.java 83.33% <83.33%> (ø)
...earch/repositories/s3/async/AsyncPartsHandler.java 86.53% <86.53%> (ø)
.../repositories/s3/async/AsyncExecutorContainer.java 87.50% <87.50%> (ø)
.../repositories/s3/AmazonAsyncS3WithCredentials.java 100.00% <100.00%> (ø)
...g/opensearch/repositories/s3/S3ClientSettings.java 80.63% <100.00%> (+3.52%) ⬆️
...pensearch/repositories/s3/async/UploadRequest.java 100.00% <100.00%> (ø)
... and 1 more

... and 476 files with indirect coverage changes

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 6c12571 to a9488f3 Compare May 29, 2023 01:58
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: a9488f3
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from a9488f3 to 432171d Compare May 29, 2023 12:56
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL: null
  • CommitID: 432171d
    Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
    Is the failure a flaky test unrelated to your change?

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 432171d to 6aad504 Compare June 5, 2023 08:15
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from 6aad504 to 6f4b0b9 Compare June 5, 2023 10:54
@github-actions
Copy link
Contributor

github-actions bot commented Jun 5, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@raghuvanshraj raghuvanshraj force-pushed the multi-part-upload-plugin-pr-2 branch from f06995b to 630f250 Compare June 15, 2023 09:29
Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

@gbbafna gbbafna merged commit c40323b into opensearch-project:main Jul 12, 2023
@raghuvanshraj raghuvanshraj added the backport 2.x Backport to 2.x branch label Jul 12, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 12, 2023
Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
(cherry picked from commit c40323b)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
dzane17 pushed a commit to dzane17/OpenSearch that referenced this pull request Jul 12, 2023
…roject#7217)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
gbbafna pushed a commit that referenced this pull request Jul 13, 2023
(cherry picked from commit c40323b)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
buddharajusahil pushed a commit to buddharajusahil/OpenSearch that referenced this pull request Jul 18, 2023
…roject#7217)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
Signed-off-by: sahil buddharaju <sahilbud@amazon.com>
baba-devv pushed a commit to baba-devv/OpenSearch that referenced this pull request Jul 29, 2023
…roject#7217)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…roject#7217)

Signed-off-by: Raghuvansh Raj <raghraaj@amazon.com>
Co-authored-by: Vikas Bansal <vikasvb@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants