Skip to content

Skip metadata upload when no new segments during refresh #19198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
HyunSangHan wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Skip metadata upload when no new segments during refresh #19198

HyunSangHan wants to merge 6 commits into from

Conversation

@HyunSangHan
Copy link
Contributor

@HyunSangHan HyunSangHan commented Sep 1, 2025

Description

Optimizes remote store refresh performance by avoiding unnecessary metadata uploads when no new segment files are uploaded. The metadata upload is now conditional on having new segments to upload.

Related Issues

Resolves #17530

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers Storage:Remote labels Sep 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2025

❌ Gradle check result for 0dc8255: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-trigger-bot
Copy link
Contributor

opensearch-trigger-bot bot commented Oct 1, 2025

This PR is stalled because it has been open for 30 days with no activity.

@opensearch-trigger-bot opensearch-trigger-bot bot added the stalled Issues that have stalled label Oct 1, 2025
@HyunSangHan HyunSangHan closed this Oct 8, 2025
@github-project-automation github-project-automation bot moved this from ✅ Done to 🏗 In progress in Storage Project Board Oct 8, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 8, 2025

❌ Gradle check result for 0dc8255: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@opensearch-trigger-bot opensearch-trigger-bot bot removed the stalled Issues that have stalled label Oct 8, 2025
sandeshkr419
sandeshkr419 reviewed Oct 8, 2025
View reviewed changes
Copy link
Member

@sandeshkr419 sandeshkr419 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks reasonable. I have a minor comment.

Also, is there a way we can add a unit test case for this well?

@sandeshkr419
Copy link
Member

sandeshkr419 commented Oct 9, 2025

Can you please also add a changelog for this change.
Refer: https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#Changelog

@HyunSangHan
Skip metadata upload when no new segments during refresh
4fe60e1 
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
@HyunSangHan
Refactor metadata upload optimization
996a79e 
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
@HyunSangHan
Add changelog
a23c55c 
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
@HyunSangHan
Add test cases: testSkipsMetadataUploadWhenNoNewSegments, testUploads…
fcae85d 
…WhenNewSegmentsPresent

Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
@HyunSangHan
Fix affected test cases
1f94f4f 
Signed-off-by: Hyunsang Han <gustkd3@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2025

❌ Gradle check result for 1f94f4f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sandeshkr419
sandeshkr419 approved these changes Oct 21, 2025
View reviewed changes
Copy link
Member

@sandeshkr419 sandeshkr419 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's rebase and get this merged!
Thanks @HyunSangHan for working on this!

@github-project-automation github-project-automation bot moved this from 🏗 In progress to 👀 In review in Storage Project Board Oct 21, 2025
sandeshkr419

This comment was marked as duplicate.

Sign in to view

@sandeshkr419 sandeshkr419 mentioned this pull request Oct 30, 2025
Closed
3 tasks
@sandeshkr419
Merge branch 'main' into issue/17530
6b48ac9 
Signed-off-by: Sandesh Kumar <sandeshkr419@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2025

❌ Gradle check result for 6b48ac9: TIMEOUT

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in Storage Project Board Nov 5, 2025
@sandeshkr419 sandeshkr419 reopened this Nov 5, 2025
@github-project-automation github-project-automation bot moved this from ✅ Done to 🏗 In progress in Storage Project Board Nov 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 5, 2025

❌ Gradle check result for 6b48ac9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2025

❌ Gradle check result for 6b48ac9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2025

❌ Gradle check result for 6b48ac9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419
Copy link
Member

sandeshkr419 commented Nov 15, 2025

@HyunSangHan Did you get a chance into looking into test failures?

This was referenced Dec 8, 2025
Draft
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sandeshkr419 sandeshkr419 sandeshkr419 approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@cwperks cwperks Awaiting requested review from cwperks cwperks is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah

Assignees

No one assigned

Labels

enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers Storage:Remote

Projects

Storage Project Board
Status: 🏗 In progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Skip metadata upload if no new segment files to upload during refresh for remote store clusters

2 participants

@HyunSangHan @sandeshkr419