owner: fix a bug which lead to replication stopped and no error report (#1814)

[ti-chi-bot]

This is an automated cherry-pick of #1814

Bug phenomenon

The resolved TS and checkpoint TS of a change feed is stopped, but no error reported

How to confirm the bug

We can check the owner log, and find logs like this:

[2021/04/25 21:39:48.899 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.899 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.989 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]
[2021/04/25 21:39:48.989 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:48.989 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.049 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]
[2021/04/25 21:39:49.049 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":1103,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.049 +08:00] [WARN] [changefeed.go:561] ["the target capture is not exist, sent the table to orphanTables"] [job="{\"From\":\"b51b0e86-373c-49a2-a7c4-42c98fde4e5e\",\"To\":\"5ec78795-88ea-43b3-803b-229a3ededc3b\",\"TableID\":694,\"TableReplicaInfo\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"Status\":1}"]
[2021/04/25 21:39:49.111 +08:00] [INFO] [changefeed.go:374] ["dispatch table success"] [capture-id=1e8e5271-9340-4b08-b0b4-a00dd456eda6] [status="{\"tables\":{\"1103\":{\"start-ts\":424504976445538305,\"mark-table-id\":0},\"312\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"452\":{\"start-ts\":424504976340680705,\"mark-table-id\":0},\"694\":{\"start-ts\":424504976445538305,\"mark-table-id\":0}},\"operation\":null,\"admin-job-type\":0}"]

Trigger conditions

A capture is offline when some table is moving by the owner.

Versions have this bug

[4.0.0, 4.0.13], 5.0.0-rc, [5.0.0, 5.0.1]

Bug mechanism

when the owner moves a table, the owner tries to remove the table from source capture and add the table to target capture.
if the target capture is offline at the same time, the owner should add this table to orphan tables and dispatch this table in the next tick.

but the owner forget to remove the invalid move table job, so the owner will add the table to orphan table every tick, leads to the other logic does not work properly

Check List

Tests

• Unit test

Release note

• fix a bug about resolved ts stopped when move a table.

[ti-chi-bot]

@ti-chi-bot: This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick, it must first be approved (/lgtm + /merge) by the collaborators.

AFTER it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• amyangfei
• liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[amyangfei]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: ac38e9d

[amyangfei]

/run-integration-tests

[ti-chi-bot]

@ti-chi-bot: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.