Tracking Issues and Feature for Correctness and Stability

[maxshuang]

Milestone

• 2021-12-31

  • Add Ts Tracking log for Error Troubleshooting
  • Need a better Ts managerment scheme a design draft
  • Online DDL test-infra cases
  • cdc met nil pointer panic in owner scheduler #2980
  • Add more log for IO wait in owner to detect stuck #3884

• 2022-01-15

  • Run transaction test cases
  • Need test 100~200 changefeeds scenarios

• 2022-01-30

Correctness

• ResolvedTs/CheckpointTs Management

  • Add Ts Tracking log for Error Troubleshooting [P0]
    • Need Better Ts tracking log in TiCDC for Error TroubleShooting #3841 @maxshuang
  • Need a better Ts managerment scheme [P1] first step, a design draft
    • Need a Better Ts managerment scheme for TiCDC #3842

• Scheduler

  • Sink double write [P1]
    • [TODO] add issue link
  • Table lose. [P1]
    • [TiCDC Bug Report] lost a table when rename the table in the upstream #3607 to be confirmed
  • [TODO] Add Chaos test cases for Scheduler [P2]

• Data Parsing [P1]

  • Avro: missing tests for column types #3644 [P2]
  • Need more unit test for different data types in mounter #3843
  • https://github.com/pingcap/ticdc/issues/2302

• DDL & DML [P0]

  • [TODO] Online DDL test-infra cases @maxshuang
  • [TODO] Run transaction test cases https://github.com/pingcap/automated-tests/tree/master/ticases/transaction [P1]
  • Default value may cause more data inconsistency #4048. @maxshuang

• Data verification scheme [P2]

  • [TODO] a module level data verification scheme
  • [TODO] break down data verification logic into different modules
  • [TODO] demo the verification solution

Stability

• Failover

• Scheduler

  • HA scenarios [P2]
    • Capture/KvNode network isolation
      • Need a feature to deal with network isolation between capture and tikv node #3616
      • TiCDC take long time (may be a day) to recover from TiKV cluster failover #4516
  • Load balance [P2]
    • table scheduler is changefeed level, which could lead to imbalance in some scenarios #3654
  • Owner abnormal scenarios [P0]
    • TiCDC owner gets stuck when PD leader shutdowns and transfers to new node #3615
    • cdc met nil pointer panic in owner scheduler #2980 @CharlesCheung96
    • Add more log for IO wait in owner to detect stuck #3884 @CharlesCheung96
    • TiCDC stability issue when PD cluster is not stable #4251
    • Owner exits unexpectedly when PD leader is killed -9 #4248
    • A scenario that TiCDC could have more than one owner #4051

• Changefeeds Management [P1]

  • need test 100~200 changefeeds scenarios

• Upgrade/Downgrade Compatibility [P1]

  • Changefeed state compatibility
    • Upgrade cluster from 4.0.14 to 5.3.0，the stopped changefeed automatically becomes to run #3473
    • some changefeeds become [CDC:ErrSnapshotLostByGC] when upgrade from 5.1.0 to 5.1.1-hotfix #4115