tracker(dm): add task name to unistore folder name #5341

lance6716 · 2022-05-06T06:12:27Z

Signed-off-by: lance6716 lance6716@gmail.com

What problem does this PR solve?

Issue Number: ref #5334

What is changed and how it works?

sometimes unistore folder is abnormally big, we add task name to the folder name to know which task it belongs.

Also, clean that folder for unit tests by Close()

Check List

Tests

Unit test

Code changes

Side effects

Related changes

Release note

 `None`.

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot · 2022-05-06T06:12:28Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

D3Hunter
lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov-commenter · 2022-05-06T06:26:36Z

Codecov Report

Merging #5341 (f95886d) into master (687e248) will decrease coverage by 0.0316%.
The diff coverage is 53.7070%.

Flag	Coverage Δ
cdc	`60.9393% <53.7070%> (+0.3407%)`	⬆️
dm	`52.2553% <ø> (-0.2143%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5341        +/-   ##
================================================
- Coverage   56.1240%   56.0923%   -0.0317%     
================================================
  Files           522        527         +5     
  Lines         65325      69193      +3868     
================================================
+ Hits          36663      38812      +2149     
- Misses        25094      26666      +1572     
- Partials       3568       3715       +147

lichunzhu

Rest LGTM

lichunzhu · 2022-05-06T08:29:24Z

dm/pkg/schema/tracker.go

@@ -149,7 +149,7 @@ func NewTracker(ctx context.Context, task string, sessionCfg map[string]string,
 		}
 	}

-	storePath, err = ioutil.TempDir("./", "schema-tracker")
+	storePath, err = ioutil.TempDir("./", task+"-schema-tracker")


What about using a sub-directory?

BTW, I think we should add a data-dir configuration for dm-worker. We have relay-dir,dump-dir, and schema-tracker file directories now, but all the default values of them are the dm-worker binary's dir. For TiUP data-dir and deploy-dir are separated which looks a bit weird.

Maybe we can do this in the next PR.

Will do it later, if we can't find the root cause we will change the implementation, so that configuration will be deprecated soon.

lichunzhu

LGTM

lance6716 · 2022-05-10T03:03:53Z

/merge

ti-chi-bot · 2022-05-10T03:03:56Z

This pull request has been accepted and is ready to merge.

Commit hash: f95886d

lance6716 · 2022-05-10T05:33:25Z

/run-dm-integration-test

lance6716 · 2022-05-10T05:34:16Z

/hold

failed to create schema tracker" RawCause:"mkdirtemp ab?c/b%c\303\213d-schema-tracker: pattern contains path separator\

Signed-off-by: lance6716 <lance6716@gmail.com>

lance6716 · 2022-05-10T07:16:36Z

/unhold
/run-dm-integration-test

lance6716 · 2022-05-10T07:48:47Z

PTAL for b3afb8c

@D3Hunter @lichunzhu

lichunzhu · 2022-05-10T08:09:48Z

PTAL for b3afb8c

@D3Hunter @lichunzhu

Will this cause different tasks to have the same schema-tracker file name? What about using dumpling's template https://github.com/pingcap/tidb/blob/master/dumpling/export/prepare.go#L68

lance6716 · 2022-05-10T08:12:25Z

PTAL for b3afb8c
@D3Hunter @lichunzhu

Will this cause different tasks to have the same schema-tracker file name? What about using dumpling's template https://github.com/pingcap/tidb/blob/master/dumpling/export/prepare.go#L68

ioutil.TempDir will append different suffix for them, so we can only roughly tell them apart.

https://github.com/pingcap/tidb/blob/master/dumpling/export/prepare.go#L68 it's too complicated, I can't understand what it's doing 😢 Can you give an example how to use it for a string input?

lichunzhu · 2022-05-10T08:24:45Z

ioutil.TempDir will append different suffix for them, so we can only roughly tell them apart.

https://github.com/pingcap/tidb/blob/master/dumpling/export/prepare.go#L68 it's too complicated, I can't understand what it's doing 😢 Can you give an example how to use it for a string input?

pingcap/dumpling#131 (comment)
It can escape the special characters in the object names.
e.g.: table-partition/TICASE-2024 will be replaced by table-partition%2FTICASE-2024.
It seems that we don't need this name for debugging, so I think the current implementation is okay.

Signed-off-by: lance6716 <lance6716@gmail.com>

lance6716 · 2022-05-10T08:32:59Z

531cbbd

@lichunzhu PTAL

lichunzhu

LGTM

lance6716 · 2022-05-10T08:49:35Z

/merge

ti-chi-bot · 2022-05-10T08:49:37Z

This pull request has been accepted and is ready to merge.

Commit hash: 531cbbd

ti-chi-bot · 2022-05-10T08:49:52Z

@lance6716: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

D3Hunter · 2022-05-10T09:09:09Z

dm/pkg/schema/tracker.go

+	'>': {}, ',': {}, '.': {}, '?': {}, '/': {}, ' ': {},
+}
+
+func newTmpFolderForTracker(task string) (string, error) {


why not use url.PathEscape

D3Hunter · 2022-05-10T09:10:01Z

/hold

Signed-off-by: lance6716 <lance6716@gmail.com>

D3Hunter · 2022-05-10T09:17:48Z

/merge

ti-chi-bot · 2022-05-10T09:17:50Z

This pull request has been accepted and is ready to merge.

Commit hash: e5715e0

lance6716 · 2022-05-10T09:18:58Z

/unhold

D3Hunter · 2022-05-10T09:33:35Z

/run-verify

lance6716 · 2022-05-10T10:43:32Z

/run-verify

ref #5334

tracker(dm): add task name to unistore folder name

f95886d

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label May 6, 2022

lance6716 added the area/dm Issues or PRs related to DM. label May 6, 2022

ti-chi-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 6, 2022

lichunzhu reviewed May 6, 2022

View reviewed changes

D3Hunter approved these changes May 10, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 10, 2022

lichunzhu approved these changes May 10, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 10, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022

ti-chi-bot added 2 commits May 10, 2022 11:04

Merge branch 'master' into tracker-folder-name

8635a20

Merge branch 'master' into tracker-folder-name

0956052

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 10, 2022

fix CI

b3afb8c

Signed-off-by: lance6716 <lance6716@gmail.com>

ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed status/can-merge Indicates a PR has been approved by a committer. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 10, 2022

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 10, 2022

address comment

531cbbd

Signed-off-by: lance6716 <lance6716@gmail.com>

lichunzhu approved these changes May 10, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022

Merge branch 'master' into tracker-folder-name

e038bcc

D3Hunter reviewed May 10, 2022

View reviewed changes

ti-chi-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed status/can-merge Indicates a PR has been approved by a committer. labels May 10, 2022

remove the wheel

e5715e0

Signed-off-by: lance6716 <lance6716@gmail.com>

lance6716 force-pushed the tracker-folder-name branch from b116a46 to e5715e0 Compare May 10, 2022 09:16

ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 10, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 10, 2022

ti-chi-bot merged commit 57c19c8 into pingcap:master May 10, 2022

lance6716 added a commit that referenced this pull request May 12, 2022

tracker(dm): add task name to unistore folder name (#5341)

1493dc1

ref #5334

lance6716 deleted the tracker-folder-name branch October 13, 2022 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tracker(dm): add task name to unistore folder name #5341

tracker(dm): add task name to unistore folder name #5341

lance6716 commented May 6, 2022

ti-chi-bot commented May 6, 2022 •

edited

Loading

codecov-commenter commented May 6, 2022 •

edited

Loading

lichunzhu left a comment

lichunzhu May 6, 2022

lance6716 May 6, 2022

lichunzhu left a comment

lance6716 commented May 10, 2022

ti-chi-bot commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lichunzhu commented May 10, 2022

lance6716 commented May 10, 2022 •

edited

Loading

lichunzhu commented May 10, 2022

lance6716 commented May 10, 2022

lichunzhu left a comment

lance6716 commented May 10, 2022

ti-chi-bot commented May 10, 2022

ti-chi-bot commented May 10, 2022

D3Hunter May 10, 2022

D3Hunter commented May 10, 2022

D3Hunter commented May 10, 2022

ti-chi-bot commented May 10, 2022

lance6716 commented May 10, 2022

D3Hunter commented May 10, 2022

lance6716 commented May 10, 2022

tracker(dm): add task name to unistore folder name #5341

tracker(dm): add task name to unistore folder name #5341

Conversation

lance6716 commented May 6, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented May 6, 2022 • edited Loading

codecov-commenter commented May 6, 2022 • edited Loading

Codecov Report

lichunzhu left a comment

Choose a reason for hiding this comment

lichunzhu May 6, 2022

Choose a reason for hiding this comment

lance6716 May 6, 2022

Choose a reason for hiding this comment

lichunzhu left a comment

Choose a reason for hiding this comment

lance6716 commented May 10, 2022

ti-chi-bot commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lance6716 commented May 10, 2022

lichunzhu commented May 10, 2022

lance6716 commented May 10, 2022 • edited Loading

lichunzhu commented May 10, 2022

lance6716 commented May 10, 2022

lichunzhu left a comment

Choose a reason for hiding this comment

lance6716 commented May 10, 2022

ti-chi-bot commented May 10, 2022

ti-chi-bot commented May 10, 2022

D3Hunter May 10, 2022

Choose a reason for hiding this comment

D3Hunter commented May 10, 2022

D3Hunter commented May 10, 2022

ti-chi-bot commented May 10, 2022

lance6716 commented May 10, 2022

D3Hunter commented May 10, 2022

lance6716 commented May 10, 2022

ti-chi-bot commented May 6, 2022 •

edited

Loading

codecov-commenter commented May 6, 2022 •

edited

Loading

lance6716 commented May 10, 2022 •

edited

Loading