Use remote recovery when topic is configured to use remote data #22908
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
f461d52 to
837455f
Compare
bashtanov
changed the title
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
837455f to
1321799
Compare
|
new failures in https://buildkite.com/redpanda/redpanda/builds/53044#01915b88-9afa-4011-9e70-1a139c563e0f: new failures in https://buildkite.com/redpanda/redpanda/builds/53548#01918f2f-9bd9-49ec-b693-5be4f3e5348e: new failures in https://buildkite.com/redpanda/redpanda/builds/54942#01921e82-34f4-4454-997c-574449687a2a: |
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
1321799 to
12c75b9
Compare
bharathv
reviewed
Aug 19, 2024
bashtanov
reviewed
Aug 19, 2024
| admin_client = admin_client or self._admin | ||
| if tolerate_stopped_nodes: | ||
| started_node_ids = {self.node_id(n) for n in self.started_nodes()} |
There was a problem hiding this comment.
TBH I find having a variable declared/undeclared depending on the condition somewhat error-prone and harder to read too. What do you think of something like the following?
if tolerate_stopped_nodes:
started_node_ids = {self.node_id(n) for n in self.started_nodes()}
node_check_predicate = lambda n: n in started_node_ids
else:
node_check_predicate = lambda n: True
...
ready = all([n['config_version'] >= config_version for n in status if node_check_predicate(n)])
12c75b9 to
010e107
Compare
bashtanov
previously approved these changes
Aug 22, 2024
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
010e107 to
2cadadd
Compare
mmaslankaprv
force-pushed
the
recovery-ts
branch
2 times, most recently
from
c766d9d to
fef7f59
Compare
Signed-off-by: Michał Maślanka <michal@redpanda.com>
Signed-off-by: Michał Maślanka <michal@redpanda.com>
Added log entry emitted when partition instance is being created. The entry will allow us to quickly identify partition configuration. Signed-off-by: Michał Maślanka <michal@redpanda.com>
When topic is configured to use remote data the data can be used when force recovering partitions that lost majority. In this case instead of creating an empty partition replica instance we customize arguments passed into the `partition_manger::manage` method to enable remote recovery of replica data. Signed-off-by: Michał Maślanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
fef7f59 to
b964da2
Compare
|
/ci-repeat 1 |
|
unrelated test failure: https://redpandadata.atlassian.net/issues/CORE-7002 |
bashtanov
previously approved these changes
Aug 30, 2024
Added replicating some data and waiting for then to be uploaded to the cloud when executing node wise recovery. This way a test is able to verify if cloud storage data are used when force re-configuring partitions with lost majority. Signed-off-by: Michał Maślanka <michal@redpanda.com>
mmaslankaprv
force-pushed
the
recovery-ts
branch
from
b964da2 to
de93ab9
Compare
bashtanov
approved these changes
Sep 2, 2024
|
/ci-repeat 1 |
2 similar comments
|
/ci-repeat 1 |
|
/ci-repeat 1 |
bharathv
approved these changes
Oct 4, 2024
| // topic being cloud enabled implies existence of overrides | ||
| ntp_config.get_overrides().recovery_enabled | ||
| = storage::topic_recovery_enabled::yes; | ||
| rtp.emplace(*initial_rev, cfg->partition_count); |
There was a problem hiding this comment.
I'm not too familiar with initial_version perhaps @ztlpn can take another look.
|
/ci-repeat 1 |
Retry command for Build#56133please wait until all jobs are finished before running the slash command |
|
known ci failure: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When topic is configured to use remote data the data can be used when
force recovering partitions that lost majority. In this case instead of
creating an empty partition replica instance we customize arguments
passed into the
partition_manger::managemethod to enable remoterecovery of replica data.
Backports Required
Release Notes
Improvements