[mysql] Snapshot split the chunks asynchronously (ververica#915) #931
Conversation
| executor.submit( | ||
| () -> { | ||
| while (remainingTables.size() > 0) { | ||
| TableId nextTable = remainingTables.pollFirst(); |
There was a problem hiding this comment.
How we make sure the table splits generation has finished if failover happened ?
There was a problem hiding this comment.
Thank you for your comments. I have revised the code again.
fuyun2024
force-pushed
the
asynchronously-snapshot-spliting
branch
from
2058b7c to
73e8520
Compare
fuyun2024
force-pushed
the
asynchronously-snapshot-spliting
branch
from
c375bdf to
00b97c6
Compare
There was a problem hiding this comment.
Thanks @fuyun2024 for the contribution, could you also add some unit tests to cover the change?
| @@ -180,27 +191,60 @@ private void captureNewlyAddedTables() { | |||
| } | |||
| } | |||
|
|
|||
| private void asynchronouslySplitIfNeed() { | |||
There was a problem hiding this comment.
| private void asynchronouslySplitIfNeed() { | |
| private void startaAsynchronouslySplit() { |
| if (!remainingTables.isEmpty()) { | ||
| if (executor == null) { | ||
| ThreadFactory threadFactory = | ||
| new ThreadFactoryBuilder().setNameFormat("snapshot-split").build(); |
There was a problem hiding this comment.
| new ThreadFactoryBuilder().setNameFormat("snapshot-split").build(); | |
| new ThreadFactoryBuilder().setNameFormat("snapshot-splitting").build(); |
| } | ||
| return getNext(); | ||
| } else { | ||
| return Optional.empty(); |
There was a problem hiding this comment.
Could we release the executor resource after splitting finished ?
| } catch (InterruptedException e) { | ||
| e.printStackTrace(); | ||
| } |
There was a problem hiding this comment.
please throw FlinkRuntimeException with a detail error message.
| return getNext(); | ||
| } else { | ||
| return Optional.empty(); | ||
| } else if (!remainingTables.isEmpty()) { |
There was a problem hiding this comment.
There's a race condition that remainingTables is empty but the asynchronous thread just added the snapshot splits of last table to remainingSplits, and then the last table will be missed.
fuyun2024
force-pushed
the
asynchronously-snapshot-spliting
branch
2 times, most recently
from
f8d9e5b to
0af1600
Compare
fuyun2024
force-pushed
the
asynchronously-snapshot-spliting
branch
from
ec6c681 to
c8836ba
Compare
No description provided.