[cdc-connector][cdc-base] Add SNAPSHOT mode for Incremental CDC Source

[loserwang1024]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Current, Incremental CDC Source connector is always an unbound stream, no matter in any mode (INITIAL,EARLIEST_OFFSET, LATEST_OFFSET, SPECIFIC_OFFSETS，TIMESTAMP).

Sometimes, users just want to replication bounded data then recycle the Flink resource. Moreover, some connectors can also eliminate impact on database after finishing job, for example, remove slot for Postgresql connector.

Solution

Add SNAPSHOT mode in Incremental CDC Source, only read log until reaching the max high_watermarks.

the total process:

1. Split into multiple chunks(snapshot splits）and read them in same way as initial mode.
2. Read streaming split until the max high_watmark.
3. Stop the job(with NoMoreSplitsEvent)

Then we have the consistency snapshot at the point the max high_watmark.

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[gong]

@loserwang1024 I think SNAPSHOT_ONLY should only include snapshot split and don't care backfill.

[loserwang1024]

@loserwang1024 I think SNAPSHOT_ONLY should only include snapshot split and don't care backfill.

@gong, But If we only include snapshot split , cannot get a consistency snapshot at any point, unless any operation is not allowed during snapshot splits read. For example, we have three snapshot splits. During the first split and third split, some update operations are applied. When read third snapshot split, it can read these operations to it, but the first splits cannot.
Make an extreme assumption： after the first split finished, we stop the job, then restart it from the savepoint in tomorrow. When the job finished, we get partial data in first split which is today's version , while get partial data in third split which is tomorrow's version.

Ideally, we want to get a consistency snapshot of database at the point of job start. But the framework cannot do it unless any operation is not allowed during snapshot splits read. So we can only get a consistency snapshot of database at the max high_watermark. When the job finished, we return data in some certain period of database.

If any user want what you said, he can only set the split to (-∞,+∞), then set backfill.skip = true. So he can get the consistency snapshot of database( guarantee by only once JDBC lookup with MVCC of database)

[gong]

@loserwang1024 I think SNAPSHOT_ONLY should only include snapshot split and don't care backfill.

@gong, But If we only include snapshot split , cannot get a consistency snapshot at any point, unless any operation is not allowed during snapshot splits read. For example, we have three snapshot splits. During the first split and third split, some update operations are applied. When read third snapshot split, it can read these operations to it, but the first splits cannot. Make an extreme assumption： after the first split finished, we stop the job, then restart it from the savepoint in tomorrow. When the job finished, we get partial data in first split which is today's version , while get partial data in third split which is tomorrow's version.

Ideally, we want to get a consistency snapshot of database at the point of job start. But the framework cannot do it unless any operation is not allowed during snapshot splits read. So we can only get a consistency snapshot of database at the max high_watermark. When the job finished, we return data in some certain period of database.

If any user want what you said, he can only set the split to (-∞,+∞), then set backfill.skip = true. So he can get the consistency snapshot of database( guarantee by only once JDBC lookup with MVCC of database)

@loserwang1024 I think SNAPSHOT_ONLY usage scenario only for once data synchronization. User should use other scan mode if database data change when data synchronization.

[loserwang1024]

I think SNAPSHOT_ONLY usage scenario only for once data synchronization.

I agree with you. This is why need this issue.Now the job also will stop at the max high_watermarks.

if database data change when data synchronization.

But each snapshot split only read the image chunk data at the time of its execution, so each chunk will read different time versions of database. I want all chunks to reach the same version of database data.

[gong]

I think SNAPSHOT_ONLY usage scenario only for once data synchronization.

I agree with you. This is why need this issue.Now the job also will stop at the max high_watermarks.

if database data change when data synchronization.

But each snapshot split only read the image chunk data at the time of its execution, so each chunk will read different time versions of database. I want all chunks to reach the same version of database data.

I understand your idea.

[whhe]

I think it would be better if the SNAPSHOT_ONLY mode can be combined with SPECIFIC_OFFSETS/TIMESTAMP mode to ensure a overall exactly-once data capturing, so a consistency snapshot makes sense to me.

[loserwang1024]

I think it would be better if the SNAPSHOT_ONLY mode can be combined with SPECIFIC_OFFSETS/TIMESTAMP mode to ensure a overall exactly-once data capturing, so a consistency snapshot makes sense to me.

@whhe , Yes, I total agree with you. Maybe we can provide this ability later with an end-option(compare with startup-option). Not only snapshot can use it, but also can use in earliest or latest or SPECIFIC_OFFSETS mode .Though it will be not easy because now we can not guarantee the execution time of snapshot phase, so can not guarantee SPECIFIC_OFFSETS/TIMESTAMP is after snapshot phase.

But in this issue, I focus on SNAPSHOT_ONLY, and the ability can also be reuse later.