log(ticdc): output the zap internal errors to stderr

[zhaoxinyu]

What problem does this PR solve?

#3362

What is changed and how it works?

Use a new field ErrorOutputPath in log.Config to output the internal errors of zap (eg. no space left on device). See pingcap/log#24 for more detail.

Previously, the log lib used in cdc outputs the log internal errors to the same log file designated in log.FileLogConfig.Filename. So if the underlying disk is full, those errors are silently discarded, which makes users hard to do troubleshooting.

Check List

Tests

• Unit test
• Integration test

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• hi-rustin
• overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov-commenter]

Codecov Report

Merging #3778 (1178513) into master (3873d39) will decrease coverage by 1.8645%.
The diff coverage is 66.4799%.

Flag	Coverage Δ
cdc	58.7083% <66.4799%> (+0.4718%)	⬆️
dm	52.2460% <ø> (-3.7886%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #3778        +/-   ##
================================================
- Coverage   57.0741%   55.2096%   -1.8646%     
================================================
  Files           478        486         +8     
  Lines         56551      60033      +3482     
================================================
+ Hits          32276      33144       +868     
- Misses        20978      23527      +2549     
- Partials       3297       3362        +65     

[overvenus]

Does it output all error-level log to "stderr"? If so, I think it's a bit verbose.

[zhaoxinyu]

Does it output all error-level log to "stderr"? If so, I think it's a bit verbose.

@overvenus No, all messages logged by cdc is written to log file as before. This PR only affects the output of internal errors of zap. For example, the following message will be sent to stderr:

2021-12-09 10:42:49.452647412 +0800 CST m=+0.299023123 write error: write /home/xinyuzhao/tidb-deploy/cdc-9300/log/cdc.log: no space left on device

Reference: https://github.com/uber-go/zap/blob/master/zapcore/entry.go#L222-L225

[Rustin170506]

Do we have a way to test it, please?

[zhaoxinyu]

Do we have a way to test it, please?

I will take a look at how zap lib test for this scenario.

[Rustin170506]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 9b45545

[zhaoxinyu]

/merge

[ti-chi-bot]

@zhaoxinyu: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[zhaoxinyu]

/assign @hi-rustin

[zhaoxinyu]

/run-dm-integration-test

[Rustin170506]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 78b170a