Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge small files in dmfiles #7771

Merged
merged 24 commits into from
Jul 24, 2023
Merged

Merge small files in dmfiles #7771

merged 24 commits into from
Jul 24, 2023

Conversation

hongyunyan
Copy link
Contributor

@hongyunyan hongyunyan commented Jul 10, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: close #7595

Problem Summary:

What is changed and how it works?

  1. we introduce the new dmfile format::V3. In which format, we will merge all the tiny files in dmfile(including index / mark / null.data, every file whose size is less than dt_small_file_size_threshold) to x.merged. When the data is quite tiny, the file in dmf_x may like the picutre shows
    image

Besides the meta format is keep the same as #6826, and the V3 is completely compatible with the format in S3 now.

  1. we provide a dttool to migrate the dmfile in v3 to v2.
  2. we do a dtworkload on both v2 and v3, find that there is no noticeable difference in performance.
  3. we will first still make STORAGE_FORMAT_V4 as default, and in next sprint, will start to use STORAGE_FORMAT_V5 as default. Besides, we pass all the tests in STORAGE_FORMAT_V5.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

cut down file numbers in STORAGE_FORMAT_V5

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 10, 2023
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 10, 2023
@ti-chi-bot ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 10, 2023
@ti-chi-bot ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 14, 2023
@hongyunyan
Copy link
Contributor Author

/run-all-tests

@hongyunyan
Copy link
Contributor Author

/run-all-tests

@hongyunyan
Copy link
Contributor Author

/run-all-tests

@hongyunyan
Copy link
Contributor Author

/run-all-tests

@hongyunyan
Copy link
Contributor Author

/run-unit-test

@hongyunyan
Copy link
Contributor Author

/run-all-tests

@hongyunyan
Copy link
Contributor Author

/run-unit-test

1 similar comment
@hongyunyan
Copy link
Contributor Author

/run-unit-test

@ti-chi-bot ti-chi-bot bot requested a review from JinheLin July 17, 2023 09:13
@hongyunyan
Copy link
Contributor Author

/cc @lidezhu

@ti-chi-bot ti-chi-bot bot requested a review from lidezhu July 18, 2023 02:33
@hongyunyan
Copy link
Contributor Author

/run-all-tests

flowbehappy
flowbehappy reviewed Jul 20, 2023
View reviewed changes
dbms/src/Server/Server.cpp Outdated
@@ -957,8 +957,8 @@ int Server::main(const std::vector<std::string> & /*args*/)
{
// If the user does not explicitly set format_version in the config file but
// enables S3, then we set up a proper format version to support S3.
setStorageFormat(5);
LOG_INFO(log, "Using format_version={} (infer by S3 is enabled).", STORAGE_FORMAT_V5.identifier);
setStorageFormat(100);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

STORAGE_FORMAT_V100.identifier

lidezhu
lidezhu approved these changes Jul 24, 2023
View reviewed changes
Copy link
Contributor

@lidezhu lidezhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 24, 2023
@ti-chi-bot ti-chi-bot bot added the lgtm label Jul 24, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 24, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JinheLin, lidezhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 24, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 24, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-07-24 02:01:07.481121544 +0000 UTC m=+147010.075652524: ☑️ agreed by lidezhu.
  • 2023-07-24 03:56:42.888384306 +0000 UTC m=+153945.482915291: ☑️ agreed by JinheLin.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 24, 2023

@hongyunyan: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@hongyunyan
Copy link
Contributor Author

/run-unit-test

@ti-chi-bot ti-chi-bot bot merged commit 69bc04f into pingcap:master Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@flowbehappy flowbehappy flowbehappy left review comments

@lidezhu lidezhu lidezhu approved these changes

@JinheLin JinheLin JinheLin approved these changes

@breezewish breezewish Awaiting requested review from breezewish

Assignees
No one assigned
Labels
approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Reduce the number of files to avoid inodes exhausting
4 participants
@hongyunyan @flowbehappy @JinheLin @lidezhu