Handle translog upload during primary relocation for remote-backed indexes #5804

ashking94 · 2023-01-10T20:25:14Z

Signed-off-by: Ashish Singh ssashish@amazon.com

Description

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-01-10T21:56:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9225/
CommitID: 1f7afe8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-01-11T07:40:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/9285/
CommitID: 2fd4055
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Ashish Singh <ssashish@amazon.com>

github-actions · 2023-01-17T19:16:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/9490/
CommitID: 58344ca

gbbafna

LGTM. Thanks @ashking94 .

ashking94 · 2023-01-18T05:04:09Z

Backport failed due to #5618 not being backported. Backporting this first and then seeing if it can be backported with tag again.

opensearch-trigger-bot · 2023-01-18T07:25:26Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5804-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 44aabe9bb57e33716fbafcc81d29c5385a55f34f
# Push it to GitHub
git push --set-upstream origin backport/backport-5804-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5804-to-2.x.

ashking94 · 2023-01-18T07:33:00Z

Failing now due to #5344 not being in 2.x.

Bukhtawar · 2023-01-19T06:01:13Z

server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java

    private boolean upload(Long primaryTerm, Long generation) throws IOException {
-        logger.trace("uploading translog for {} {} ", primaryTerm, generation);
+        // During primary relocation (primary-primary peer recovery), both the old and the new primary have engine
+        // created with the RemoteFsTranslog. Both primaries are equipped to upload the translogs. The primary mode check
+        // below ensures that the real primary only is uploading. Before the primary mode is set as true for the new
+        // primary, the engine is reset to InternalEngine which also initialises the RemoteFsTranslog which in turns
+        // downloads all the translogs from remote store and does a flush before the relocation finishes.
+        if (primaryModeSupplier.getAsBoolean() == false) {
+            logger.trace("skipped uploading translog for {} {}", primaryTerm, generation);


Can we think about a No-op version of RemoteFsTranslog that switches atomically on tracker updates, rather than passing around primary mode here.

Evaluated the approach you suggested. This is a bit tricky as currently the translog is supplied through translog factory and then the Translog implementation gets initialised only when the engine is created or reset. For the new primary during relocation, the engine is reset before the primary mode is changed from false into true and the primary mode change on older primary (during relocation) does not reset or recreates the engine. Although this is possible, this seems to be make this simple check to a bit complicated. Currently, changes in primaryMode has not listeners or observers. Will try this again after clearing of some high ROI items for remote-backed store items. Thanks for bringing this up.

…r remote-backed indexes (opensearch-project#5804) * Upload translog only if primaryMode is true Signed-off-by: Ashish Singh <ssashish@amazon.com>

opensearch-trigger-bot · 2023-01-20T18:10:46Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-5804-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 44aabe9bb57e33716fbafcc81d29c5385a55f34f
# Push it to GitHub
git push --set-upstream origin backport/backport-5804-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5804-to-2.x.

…r remote-backed indexes (#5804) * Upload translog only if primaryMode is true Signed-off-by: Ashish Singh <ssashish@amazon.com> (cherry picked from commit 44aabe9) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

ashking94 · 2023-01-30T05:56:12Z

Backport is failing with java compilation failure. I will try to raise backport manually.

…r remote-backed indexes (opensearch-project#5804) * Upload translog only if primaryMode is true Signed-off-by: Ashish Singh <ssashish@amazon.com>

…r remote-backed indexes (#5804) (#6064) * Upload translog only if primaryMode is true Signed-off-by: Ashish Singh <ssashish@amazon.com>

ashking94 added skip-changelog distributed framework Storage:Durability Issues and PRs related to the durability framework v2.6.0 'Issues and PRs related to version v2.6.0' labels Jan 10, 2023

ashking94 force-pushed the 5795 branch from 1f7afe8 to 2fd4055 Compare January 11, 2023 07:15

ashking94 force-pushed the 5795 branch from 2fd4055 to 2ab8a4e Compare January 12, 2023 15:52

ashking94 marked this pull request as ready for review January 12, 2023 16:00

ashking94 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners January 12, 2023 16:00

Add UTs

58344ca

Signed-off-by: Ashish Singh <ssashish@amazon.com>

gbbafna approved these changes Jan 18, 2023

View reviewed changes

sachinpkale approved these changes Jan 18, 2023

View reviewed changes

gbbafna merged commit 44aabe9 into opensearch-project:main Jan 18, 2023

gbbafna added the backport 2.x Backport to 2.x branch label Jan 18, 2023

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jan 18, 2023

gbbafna added v2.6.0 'Issues and PRs related to version v2.6.0' and removed v2.6.0 'Issues and PRs related to version v2.6.0' labels Jan 18, 2023

opensearch-project deleted a comment from opensearch-trigger-bot bot Jan 18, 2023

Bukhtawar reviewed Jan 19, 2023

View reviewed changes

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jan 20, 2023

ashking94 added backport 2.x Backport to 2.x branch and removed backport 2.x Backport to 2.x branch labels Jan 30, 2023

opensearch-trigger-bot bot mentioned this pull request Jan 30, 2023

[Backport 2.x] Handle translog upload during primary relocation for remote-backed indexes #6062

Closed

ashking94 mentioned this pull request Jan 30, 2023

[Backport 2.x] Handle translog upload during primary relocation for remote-backed indexes (#5804) #6064

Merged

6 tasks

gbbafna pushed a commit that referenced this pull request Jan 30, 2023

[Remote Translog] Handle translog upload during primary relocation fo…

13bfdd6

…r remote-backed indexes (#5804) (#6064) * Upload translog only if primaryMode is true Signed-off-by: Ashish Singh <ssashish@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle translog upload during primary relocation for remote-backed indexes #5804

Handle translog upload during primary relocation for remote-backed indexes #5804

ashking94 commented Jan 10, 2023 •

edited

Loading

github-actions bot commented Jan 10, 2023

github-actions bot commented Jan 11, 2023

github-actions bot commented Jan 17, 2023

gbbafna left a comment

ashking94 commented Jan 18, 2023

opensearch-trigger-bot bot commented Jan 18, 2023

ashking94 commented Jan 18, 2023

Bukhtawar Jan 19, 2023

ashking94 Jan 20, 2023

opensearch-trigger-bot bot commented Jan 20, 2023

ashking94 commented Jan 30, 2023

Handle translog upload during primary relocation for remote-backed indexes #5804

Handle translog upload during primary relocation for remote-backed indexes #5804

Conversation

ashking94 commented Jan 10, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Jan 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 11, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 17, 2023

Gradle Check (Jenkins) Run Completed with:

gbbafna left a comment

Choose a reason for hiding this comment

ashking94 commented Jan 18, 2023

opensearch-trigger-bot bot commented Jan 18, 2023

ashking94 commented Jan 18, 2023

Bukhtawar Jan 19, 2023

Choose a reason for hiding this comment

ashking94 Jan 20, 2023

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 20, 2023

ashking94 commented Jan 30, 2023

ashking94 commented Jan 10, 2023 •

edited

Loading