always save checkpoint to the warmup folder

[YuJuncen]

What problem does this PR solve?

Fixed #5506.

What is changed and how does it work?

The checkpoint file is stored at the work directory where the warmup binary runs. And in most cases, it is / and won't be persisted to a PV.
This PR changes the warmup_step script, make it always set checkpoint to the directory to be warmed up(Which is probably a real EBS disk).
(Also notice that, it is impossible to pass a glob pattern to warmup_step anymore.)

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

Fixed a bug that may cause the warmup checkpoint is unavailable.

[BornChanger]

/retest

[codecov-commenter]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (1fcdf52) 61.32% compared to head (a83fced) 61.32%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5507      +/-   ##
==========================================
- Coverage   61.32%   61.32%   -0.01%     
==========================================
  Files         234      234              
  Lines       29985    29996      +11     
==========================================
+ Hits        18388    18394       +6     
- Misses       9758     9760       +2     
- Partials     1839     1842       +3     

Flag	Coverage Δ
unittest	61.32% <80.00%> (-0.01%)	⬇️

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BornChanger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BornChanger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-01-16 14:45:10.58439663 +0000 UTC m=+280752.148694335: ☑️ agreed by BornChanger.
• 2024-01-16 14:47:01.952144365 +0000 UTC m=+280863.516442069: ✖️🔁 reset by ti-chi-bot[bot].

[ti-chi-bot]

New changes are detected. LGTM label has been removed.

[BornChanger]

/cherry-pick release-1.5

[ti-chi-bot]

@BornChanger: new pull request created to branch release-1.5: #5522.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.