Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dumpling: update dumpling document on how to reduce memory usage #5860

Merged
merged 3 commits into from
Apr 6, 2021
Merged

dumpling: update dumpling document on how to reduce memory usage #5860

merged 3 commits into from
Apr 6, 2021

Conversation

lichunzhu
Copy link
Contributor

First-time contributors' checklist

What is changed, added or deleted? (Required)

Update dumpling document on how to reduce memory usage. Make sure users can get this important infomation.

Which TiDB version(s) do your changes apply to? (Required)

  • master (the latest development version)
  • v5.0 (TiDB 5.0 versions)
  • v4.0 (TiDB 4.0 versions)
  • v3.1 (TiDB 3.1 versions)
  • v3.0 (TiDB 3.0 versions)
  • v2.1 (TiDB 2.1 versions)

What is the related PR or file link(s)?

  • This PR is translated from:
  • Other reference link(s):

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@ti-chi-bot ti-chi-bot added missing-translation-status This PR does not have translation status info. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 26, 2021
@TomShawn
Copy link
Contributor

/translation doing

@ti-chi-bot ti-chi-bot added translation/doing This PR’s assignee is translating this PR. and removed missing-translation-status This PR does not have translation status info. labels Mar 26, 2021
@TomShawn
Copy link
Contributor

/label needs-cherry-pick-4.0
/label needs-cherry-pick-5.0
/status PTAL
/size small
/assign TomShawn
/label require-LGT1

@ti-chi-bot ti-chi-bot added status/PTAL This PR is ready for reviewing. size/small Changes of a small size. needs-cherry-pick-4.0 require-LGT1 Indicates that the PR requires an LGTM. labels Mar 26, 2021
@TomShawn
Copy link
Contributor

@lichunzhu Please involve a technical review~

@TomShawn
Copy link
Contributor

/cc 3pointer overvenus

overvenus
overvenus reviewed Mar 29, 2021
View reviewed changes
dumpling-overview.md
@@ -60,10 +60,18 @@ dumpling \
--filetype sql \
--threads 32 \
-o /tmp/test \
-r 200000 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's too large and may cause TiDB OOM.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we suggest TiDB uses 1GB memory, and the default threads is 4, then avg_row_length is
1GB / 4 / 200000 = 1342. It's more than 1KB. In most conditions avg_row_length is less than this.

dumpling-overview.md
@@ -60,10 +60,18 @@ dumpling \
--filetype sql \
--threads 32 \
-o /tmp/test \
-r 200000 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also add -r to other examples, e.g., ### 导出到 csv 文件

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in 0c0706b

@ti-chi-bot ti-chi-bot removed the size/small Changes of a small size. label Mar 29, 2021
@overvenus
Copy link
Member

/lgtm

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 29, 2021
3pointer
3pointer reviewed Mar 29, 2021
View reviewed changes
dumpling-overview.md Outdated

> **注意：**
>
> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-r -F 不能同时生效，也描述下吧

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现在是可以同时生效的

@3pointer
Copy link
Contributor

/lgtm

@overvenus
Copy link
Member

@TomShawn PTAL, thanks!

TomShawn
TomShawn reviewed Apr 6, 2021
View reviewed changes
dumpling-overview.md Outdated Show resolved Hide resolved
dumpling-overview.md Outdated Show resolved Hide resolved
dumpling-overview.md Outdated Show resolved Hide resolved
dumpling-overview.md Outdated

> **注意：**
>
> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。
> 如果导出的单表大小超过 10 GB，**强烈建议**指定 `-r` 和 `-F` 参数。

dumpling-overview.md Outdated Show resolved Hide resolved
dumpling-overview.md Outdated Show resolved Hide resolved
@lichunzhu @TomShawn
Apply suggestions from code review
6df2aa6 
Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>
@TomShawn
Copy link
Contributor

TomShawn commented Apr 6, 2021

/lgtm

@TomShawn
Copy link
Contributor

TomShawn commented Apr 6, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 6df2aa6

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 6, 2021
@ti-chi-bot ti-chi-bot merged commit b8655e3 into pingcap:master Apr 6, 2021
ti-srebot pushed a commit to ti-srebot/docs-cn that referenced this pull request Apr 6, 2021
@lichunzhu @ti-srebot
cherry pick pingcap#5860 to release-4.0
70901a9 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Apr 6, 2021
Merged
11 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #5962

ti-srebot pushed a commit to ti-srebot/docs-cn that referenced this pull request Apr 6, 2021
@lichunzhu @ti-srebot
cherry pick pingcap#5860 to release-5.0
e87095d 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Apr 6, 2021
Merged
11 tasks
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #5963

@lichunzhu lichunzhu deleted the updateDumplingDoc branch April 6, 2021 09:59
ti-chi-bot pushed a commit that referenced this pull request Apr 6, 2021
@ti-srebot
dumpling: update dumpling document on how to reduce memory usage (#5860…
52ef4b9 
…) (#5963)
ti-chi-bot pushed a commit that referenced this pull request Apr 6, 2021
@ti-srebot
dumpling: update dumpling document on how to reduce memory usage (#5860…
3c21cc7 
…) (#5962)
@TomShawn TomShawn mentioned this pull request Apr 13, 2021
Merged
10 tasks
@TomShawn
Copy link
Contributor

/translation done
/remove-translation doing

@ti-chi-bot ti-chi-bot added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR’s assignee is translating this PR. labels Apr 13, 2021
This was referenced Apr 19, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TomShawn TomShawn TomShawn left review comments

@3pointer 3pointer 3pointer left review comments

@overvenus overvenus overvenus left review comments

Assignees

@TomShawn TomShawn

Labels
require-LGT1 Indicates that the PR requires an LGTM. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. status/PTAL This PR is ready for reviewing. translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@lichunzhu @TomShawn @overvenus @ti-chi-bot @3pointer @ti-srebot