TiCDC keeps warning on removed changefeeds

[lichunzhu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   Set up TiCDC, start a changefeed, and then remove it forcely.

2. What did you expect to see?
   Prometheus alerts won't be triggered any more.

3. What did you see instead?
   cdc_checkpoint_high_delay
   cdc_processor_checkpoint_tso_no_change_for_1m
   cdc_resolvedts_high_delay
   tikv_cdc_min_resolved_ts_no_change_for_1m
   still alerts.

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • TiCDC version (execute cdc version):

     (paste TiCDC version here)
     

     v5.0.1,v4.0.12

[lichunzhu]

Although prometheus metrics is deleted, we should also adjust alert rules.

[lichunzhu]

more details: this changefeed is already deleted forcely.

[amyangfei]

I have checked in a test environment,

• where I created two changedfeeds, naming bypass-cf-1 and bypass-cf-2

• after running some time, I removed the changefeed bypass-cf-2

• It is to observe that metrics of bypass-cf-2 stop updating, as follows

• The given alert doesn't fire in prometheus, for example, the expr for alert item changes(ticdc_processor_checkpoint_ts[1m]) is expr:changes(ticdc_processor_checkpoint_ts[1m]) < 1, we can check the expression in prometheus and only metric from bypass-cf-1 is available

To summarize, the given bug can't be reproduced, and maybe there exists an active changefeed that is not replicating normally in the user's scenario.

[amyangfei]

Closed by #2177 and #23133

[3pointer]

tikv should handle tikv_cdc_min_resolved_ts_no_change_for_1m alert.