feat(tiflow): support downloading dependencies from a specified branch

[purelind]

support downloading dependencies from a specified branch in PR titles.

Related to this PR: pingcap/tiflow#11572

[ti-chi-bot]

I have already done a preliminary review for you, and I hope to help you do a better job.

Based on the pull request title and description, the key change is adding support for downloading dependencies from a specified branch. This is achieved by computing the branch from the pull request title and passing it as an environment variable to the script.

Potential problems:

• It's not clear how the component.computeBranchFromPR function works and whether it's reliable for all cases. It would be good to have some documentation or comments explaining how it works.
• The script is quite long and could benefit from some refactoring and better error handling.
• There are no unit tests or integration tests added with this pull request.

Fixing suggestions:

• Add some comments or documentation explaining how the component.computeBranchFromPR function works and what are its limitations.
• Refactor the script to make it more readable and maintainable. For example, extract repeated code into functions, use variables instead of repeating long commands multiple times, and add error handling.
• Consider adding some unit tests or integration tests to ensure that the new functionality works as expected.

Overall, the changes in the pull request seem reasonable but it would be good to address the potential problems and suggestions before merging it.

[purelind]

/hold

[purelind]

/retest

[ti-chi-bot]

I have already done a preliminary review for you, and I hope to help you do a better job.

Key changes:

1. The author added the ability to compute the branch for each component (tidb, pd, tikv, tiflash) based on the pull request title in the Groovy scripts. This feature allows for the download of dependencies from specific branches instead of the master branch.
2. This is done by adding a computeBranchFromPR function which determines the branch to use for each component. The resulting branch is then exported as an environment variable which is used in the download script.
3. The changes have been applied to multiple integration test scripts including pull_cdc_integration_kafka_test.groovy, pull_cdc_integration_pulsar_test.groovy, pull_cdc_integration_storage_test.groovy, pull_cdc_integration_test.groovy, pull_dm_compatibility_test.groovy, and pull_dm_integration_test.groovy.

Potential problems:

1. If the computeBranchFromPR function does not exist or is not working correctly, the new feature will not work as expected.
2. If the specified branch does not exist or the dependencies are not available on the specified branch, the download will fail.

Fixing suggestions:

1. Ensure that the computeBranchFromPR function exists and works correctly. This function should take into consideration all possible pull request title formats and should return the correct branch.
2. Add error handling to catch and handle the case when the specified branch does not exist or the dependencies are not available on that branch. This could include falling back to the master branch or providing a clear error message to the user.
3. It would be helpful to add some tests to ensure that the new feature works as expected.
4. The author should also update the PR description to explain how to use the new feature, including how to specify the branch in the PR title.

[purelind]

replay test link

• cdc kafka https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_cdc_integration_kafka_test/detail/pull_cdc_integration_kafka_test/2715/pipeline
• cdc mysql https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_cdc_integration_test/detail/pull_cdc_integration_test/2638/pipeline
• cdc pulsar https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_cdc_integration_pulsar_test/detail/pull_cdc_integration_pulsar_test/2316/pipeline
• engine
  https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_engine_integration_test/detail/pull_engine_integration_test/2423/pipeline
• dm integrationn https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_dm_integration_test/detail/pull_dm_integration_test/2803/pipeline
• dm compatibility
  https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftiflow%2Fpull_dm_compatibility_test/detail/pull_dm_compatibility_test/2654/pipeline

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wuhuizuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pipelines/OWNERS [wuhuizuo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-09-06 10:23:46.511573924 +0000 UTC m=+6296.251997855: ☑️ agreed by wuhuizuo.
• 2024-09-09 02:52:09.892946552 +0000 UTC m=+238399.633370491: ✖️🔁 reset by purelind.

[ti-chi-bot]

New changes are detected. LGTM label has been removed.

[purelind]

/unhold

[purelind]

/hold