owner: fix memory accumulated when owner consume etcd update slow (#1225)

[ti-srebot]

cherry-pick #1225 to release-4.0

---

What problem does this PR solve?

Fix for a memory accumulated issue in owner

https://github.com/pingcap/ticdc/blob/eed57c94491fd140f0a8a0528acc574bc542bb0b/cdc/owner.go#L1024-L1039

The output channel consume speed (which relays on how often L1035 can be run) may be much slower than the etcd key update frequency, which will lead to data accumulated in etcd watch client

If changefeed number is equal or larger than 2, the bug is 100% triggered.

What is changed and how it works?

use notifier to work as the event trigger

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below) (Have tested in an internal test cluster)
  • setup a ticdc cluster, create one or more changefeeds
  • running cdc for 10mins, get memory profile of cdc owner by http://<ip>:<port>/debug/pprof/heap
  • find the memory usage of etcd.clientv3, should be less than 5MB

before fix:

after:

Release note

• Fix a bug that the cdc owner might consume too much memory in the etcd watch client

[ti-srebot]

/run-all-tests

[amyangfei]

/merge

[ti-srebot]

/run-all-tests