Improve processor management and error handling

[hicqu]

Is your feature request related to a problem?

• If sink error occurs, it's better to let puller and redo keep running;
• If puller error occurs, it's better to let sink and redo keep running;
  • although generally puller doesn't emit errors, seems the only error can be "etcd context canceled"
• If redo error occurs, it's better to let sink keep running;
• If etcd session is broken, processor.Close should be called correctly, or some stale context will be left. For example, SortEngine instances.
• Many sub components of Processor use errCh directly. When pushing an error into the channel, some different styles are used:
  • with a default branch in select; so the error can be dropped silenctly;
  • with a case ctx.Done() branch in select, which means the processor has been closed correctly.
  • Personally I prefer the second style, but code needs to be simplified.

Describe the feature you'd like

I want to

• Seperate Processor lifetime from etcd session. Processor can only be closed explicitly instead of canceling the etcd session's context;
• Ensure Processor.Close will always be called correctly when destroying a Processor instance;
• Unify and simplify error handling in all sub components of Processor;
• Isolate error handling of sink, puller and redo, so that one component failure won't break whole system.

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

No response

[nongfushanquan]

/label affects-7.1

[nongfushanquan]

/remove-type feature
/type enhancement

[nongfushanquan]

/area ticdc

[nongfushanquan]

/label affects-7.1

[nongfushanquan]

/remove-label affects-6.5

[nongfushanquan]

/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.