Migrations infra, part 3: shard-local worker logic #20840
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bashtanov
force-pushed
the
migrations-infra-abw
branch
from
d7879c2 to
5f4cccb
Compare
|
/dt |
With reconciliation loop, raft0 leadership updates and migration updates running concurrently and stuffed with yield points, we need to spread them apart to avoid race conditions.
In addition to NTP and sought migration state, worker may need some information from the migration definition. Adding data types for pieces of this information related to individual partitions.
… state To pull topic-specific details from migration definition later
… info Build info packs to be dispatched to workers on shards when scheduling partition work.
Spawn, retry and gather results from partition operations on shards.
Since we process RPC replies asynchronously, the reply map may be written into while we read from it. Move it to process.
bashtanov
force-pushed
the
migrations-infra-abw
branch
from
5f4cccb to
79ec19a
Compare
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/51512#0190b631-6c64-440b-abcf-74c0835f90cd |
|
test failure: #21376 |
| // this call must only tinker with `it` within the current seastar task, | ||
| // it may be invalidated later! | ||
| ssx::spawn_with_gate(_gate, [this, it]() { | ||
| return do_work(it).then([ntp = it->first, |
There was a problem hiding this comment.
i think here we should use then_wrapped to handle exception thrown from do_work, wdyt ?
There was a problem hiding this comment.
Yeah, thanks, we need to make sure we don't lose it. I'd rather wrap everything in do_work into a try-catch so it never throws, since it returns an error code. Do you think it'd be okay?
|
overal lgtm, one comment on exception handling |
mmaslankaprv
approved these changes
Jul 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backend to gather from migration definition and to provide to worker information necessary to perform per-partition work.
Worker to spawn, retry and gather results from partition operations.
Backports Required
Release Notes