Adds support to download Remote Cluster State marker file #2

linuxpi · 2023-08-10T10:21:22Z

Description

refactor listByPrefixSortedOrder method to provide flexibility to use it without listener. Most of the usages today don't need the listener based async impl. With current refactoring, we can use it both ways -- sync and async.
Adds utilities to download latest/any Remote Cluster State marker file

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-10T10:22:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: a499ad0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-10T10:24:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 77cac81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-10T12:13:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: e8226d4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-14T19:40:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 1680f3e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: bansvaru <bansvaru@amazon.com>

…tResetAndAbort is flaky (opensearch-project#9206) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…-project#9082) This is the first round refactor of the foundation Task classes from the server module to core and common library. This is split into separate commits due to control the surface area impacted by the change. This commit refactors the following classes: * o.o.action.ActionFuture -> :libs:opensearch-common * o.o.action.ActionListener -> :libs:opensearch-core * o.o.action.ActionResponse -> :libs:opensearch-core * o.o.action.NotifyOnceListener -> :libs:opensearch-core * o.o.common.compress.Compressor -> :libs:opensearch-core * o.o.node.ReportingService -> o.o.core.service in :libs:opensearch-core * o.o.tasks.TaskCancelledExceptoin -> :libs:opensearch-core * o.o.tasks.TaskId -> :libs:opensearch-core * o.o.common.xcontent.ObjectParserHelper -> :libs:opensearch-core The following task resource tracking classes are refactored into a new resourcetracker namespace under the o.o.core.tasks packed in :libs:opensearch-core * o.o.tasks.ResourceStats * o.o.tasks.ResourceStatsType * o.o.tasks.ResourceUsageMetric * o.o.tasks.TaskResourceStats * o.o.tasks.TaskResourceUsage * o.o.tasks.TaskThreadUsage * o.o.tasks.ThreadResourceInfo These refactors are part of the effort to split foundation classes into core libraries in support of modularity for serverless and cloud native implementations. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

…nsearch-project#9221) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

…/repository-azure (opensearch-project#9231) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) Signed-off-by: bansvaru <bansvaru@amazon.com> --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add more shard level tests. Signed-off-by: Marc Handalian <handalm@amazon.com> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <handalm@amazon.com> * Self review. Signed-off-by: Marc Handalian <handalm@amazon.com> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <handalm@amazon.com> * PR feedback. Signed-off-by: Marc Handalian <handalm@amazon.com> --------- Signed-off-by: Marc Handalian <handalm@amazon.com>

…9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. Signed-off-by: bansvaru <bansvaru@amazon.com> --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…earch-project#9245) Signed-off-by: bansvaru <bansvaru@amazon.com>

…search-project#9252) * Force upgrade the vulnerable dependencies of hadoop-minicluster Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add changelog entry Signed-off-by: Thomas Farr <tsfarr@amazon.com> --------- Signed-off-by: Thomas Farr <tsfarr@amazon.com>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

…itory-gcs (opensearch-project#9230) * Bump com.google.code.gson:gson in /plugins/repository-gcs Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.9.0 to 2.10.1. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10.1) Signed-off-by: bansvaru <bansvaru@amazon.com> --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…t#9228) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.2.0 to 1.8.0. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v1.2.0...v1.8.0) Signed-off-by: bansvaru <bansvaru@amazon.com> --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…-project#9244) Signed-off-by: Marc Handalian <handalm@amazon.com>

…est (opensearch-project#9079) * Unmute testIndexingWithSegRep rolling upgrade test Signed-off-by: Suraj Singh <surajrider@gmail.com> * Assert on row count before processing the result Signed-off-by: Suraj Singh <surajrider@gmail.com> * Filter rows with 0 doc count Signed-off-by: Suraj Singh <surajrider@gmail.com> * Retry assertHitCount Signed-off-by: Suraj Singh <surajrider@gmail.com> * Handle exception Signed-off-by: Suraj Singh <surajrider@gmail.com> * Add comment Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>

…r meaning and relations between them (opensearch-project#9223) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Peter Nied <petern@amazon.com> Co-authored-by: Peter Nied <petern@amazon.com>

Signed-off-by: Jay Deng <jayd0104@gmail.com>

…RestoreService Signed-off-by: bansvaru <bansvaru@amazon.com>

…toreService Signed-off-by: bansvaru <bansvaru@amazon.com>

Signed-off-by: bansvaru <bansvaru@amazon.com>

…atin checks Signed-off-by: bansvaru <bansvaru@amazon.com>

Signed-off-by: bansvaru <bansvaru@amazon.com>

github-actions · 2023-08-16T10:23:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: c1cb834
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

soosinha and others added 3 commits August 8, 2023 07:48

Upload all index metadata to remote

327dc4c

Merge branch 'opensearch-project:main' into remote_state

dbe33d9

Refactor changes

495ec89

linuxpi changed the base branch from main to remote_state August 10, 2023 10:21

soosinha added 3 commits August 11, 2023 14:19

Remove redundant methods

2e53375

incremental metadata upload

9806f28

Change serialize logic

2386402

linuxpi force-pushed the remote-state-marker-download branch from e8226d4 to 1680f3e Compare August 14, 2023 19:39

soosinha and others added 18 commits August 15, 2023 12:15

Update serialization logic for ClusterMetadataMarker

3873b77

Add support to download Remote Cluster State marker file

1caca7a

Signed-off-by: bansvaru <bansvaru@amazon.com>

Remove redudentent methods

5723f54

Signed-off-by: bansvaru <bansvaru@amazon.com>

Add support to download IndexMetadata from Remote Cluster State

116f3bd

Signed-off-by: bansvaru <bansvaru@amazon.com>

[BUG] org.opensearch.client.RestClientSingleHostIntegTests.testReques…

501343b

…tResetAndAbort is flaky (opensearch-project#9206) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Add release notes for 1.3.12 (opensearch-project#9214)

d626bc9

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

[Remote Store] Fix couple of Remote Store flaky test and use bulk api…

402b817

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Automatically update GitHub Actions dependencies with dependabot (ope…

188fbd3

…nsearch-project#9221) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Fix test SegmentReplicationIndexShardTests.testPrimaryRestart. (opens…

efb2b88

…earch-project#9245) Signed-off-by: bansvaru <bansvaru@amazon.com>

[Remote Store] Introduce RemoteStoreRestoreService to handle restore …

39e31e9

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <bansvaru@amazon.com>

Fix flaky test SegmentReplicationIT.testDeleteOperations. (opensearch…

46cd627

…-project#9244) Signed-off-by: Marc Handalian <handalm@amazon.com>

dreamer-89 and others added 17 commits August 16, 2023 14:59

docCountError must be 0 for shard level reduce (opensearch-project#9246)

25dc45a

Signed-off-by: Jay Deng <jayd0104@gmail.com>

Add support to remote restore IndexMetadata and introduce RemoteStore…

2b70379

…RestoreService Signed-off-by: bansvaru <bansvaru@amazon.com>

Reuse RestoreCompletionResponse from RestoreService in RemoteStoreRes…

88a213c

…toreService Signed-off-by: bansvaru <bansvaru@amazon.com>

Minor Refactoring and Spotless fixes

1873f78

Signed-off-by: bansvaru <bansvaru@amazon.com>

fail restore for same index name and uuid. also harden existing valid…

226bbb1

…atin checks Signed-off-by: bansvaru <bansvaru@amazon.com>

add first integ test and minor fixes to logic

86ea74e

Signed-off-by: bansvaru <bansvaru@amazon.com>

add docstring for RemoteStoreRestoreService

2870b89

Signed-off-by: bansvaru <bansvaru@amazon.com>

Refactor integ test code

ca61f61

Signed-off-by: bansvaru <bansvaru@amazon.com>

Remove shortcut changes

9bf7636

Signed-off-by: bansvaru <bansvaru@amazon.com>

Final Refactor

92270a1

Signed-off-by: bansvaru <bansvaru@amazon.com>

Add clusterUUID param support in Remote Restore API

a9ad9bd

Signed-off-by: bansvaru <bansvaru@amazon.com>

rebase with latest changes on main

1359b8e

Signed-off-by: bansvaru <bansvaru@amazon.com>

refactor integ tests and incorporate PR comments

0c2db92

Signed-off-by: bansvaru <bansvaru@amazon.com>

remove stale integ test override

35a4f62

Signed-off-by: bansvaru <bansvaru@amazon.com>

integrate restore changes with state upload and download

c1cb834

Signed-off-by: bansvaru <bansvaru@amazon.com>

linuxpi force-pushed the remote-state-marker-download branch from 1680f3e to c1cb834 Compare August 16, 2023 10:22

soosinha force-pushed the remote_state branch 5 times, most recently from 50d6db3 to a4f1137 Compare August 21, 2023 08:50

soosinha force-pushed the remote_state branch 6 times, most recently from 2506421 to 4824bb5 Compare August 30, 2023 06:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support to download Remote Cluster State marker file #2

Adds support to download Remote Cluster State marker file #2

linuxpi commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 14, 2023

github-actions bot commented Aug 16, 2023

Adds support to download Remote Cluster State marker file #2

Are you sure you want to change the base?

Adds support to download Remote Cluster State marker file #2

Conversation

linuxpi commented Aug 10, 2023

Description

Related Issues

Check List

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with: