Resolve DDL and insert inot select Possible space inflation issues(#366) #1493
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on issues(stoneatom#366) Cause of the problem: 1. For multiple versions, Tianmu needs to first copy an original pack when performing DML operations, Modify the copied package and use append write or overwrite write after modification (If there is invalid space in the DATA file that can be written to the current pack, use overwrite write, otherwise use append write) to write to the file, After the latest package is written to a file, the latest version chain will point to the address that was last written. There is a problem with the current (TianmuAttr:: LoadData) logic. Every time you call (TianmuAttr:: LoadData), Will write data to disk, If there are multiple rows written in a transaction, there will be multiple copies of data, "Because the current transaction has not been committed, the space for previous repeated writes has not been released, so the logic of overwriting writes will not be reached.", "I only follow the logic of additional writing, which is the fundamental reason for the skyrocketing space.". If you encounter a particularly large multiline write transaction, it will lead to a space explosion. Moreover, disk IO is performed once per load line, which can also lead to degraded insert performance. Solution: To optimize the logic of (TianmuAttr:: LoadData), it is necessary to determine whether the data in the pack is full before saving changes, Is whether to reach 65536 lines, and if so, write again, If it cannot be reached, it is necessary to write again in the commit phase.
…ving direct insert performance.
|
This pull request's title should follow requirements next. @konghaiya please check it 👇. Valid format: Valid types:
|
14 tasks
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## stonedb-5.7-dev #1493 +/- ##
====================================================
+ Coverage 0 42.95% +42.95%
====================================================
Files 0 1838 +1838
Lines 0 397379 +397379
====================================================
+ Hits 0 170712 +170712
- Misses 0 226667 +226667
... and 1835 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
|
This pull request's title should follow requirements next. @konghaiya please check it 👇. Valid format: Valid types:
|
1 similar comment
|
This pull request's title should follow requirements next. @konghaiya please check it 👇. Valid format: Valid types:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cause of the problem:
Modify the copied package and use append write or overwrite write after modification
(If there is invalid space in the DATA file that can be written to the current pack, use overwrite write, otherwise use append write) to write to the file,
After the latest package is written to a file, the latest version chain will point to the address that was last written.
There is a problem with the current (TianmuAttr:: LoadData) logic. Every time you call (TianmuAttr:: LoadData),
Will write data to disk,
If there are multiple rows written in a transaction, there will be multiple copies of data,
"Because the current transaction has not been committed, the space for previous repeated writes has not been released, so the logic of overwriting writes will not be reached.",
"I only follow the logic of additional writing, which is the fundamental reason for the skyrocketing space.".
If you encounter a particularly large multiline write transaction, it will lead to a space explosion.
Moreover, disk IO is performed once per load line, which can also lead to degraded insert performance.
Solution:
To optimize the logic of (TianmuAttr:: LoadData), it is necessary to determine whether the data in the pack is full before saving changes,
Is whether to reach 65536 lines, and if so, write again,
If it cannot be reached, it is necessary to write again in the commit phase.
Summary about this PR
Issue Number: close #366
Issue Number: close #1079
Issue Number: close #689
Tests Check List
Changelog
Documentation