Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve startup time of Distributed engine. #25663

Merged
merged 2 commits into from
Jun 27, 2021

Conversation

azat
Copy link
Collaborator

@azat azat commented Jun 24, 2021

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Improve startup time of Distributed engine.

Detailed description / Documentation draft:

  • create directory monitors in parallel (this also includes rmdir in
    case of directory is empty, since even if the directory is empty it
    may take some time to remove it, due to waiting for journal or if the
    directory is large, i.e. it had lots of files before, since remember
    ext4 does not truncate the directory size on each unlink 1)
  • initialize increment in parallel too (since it does readdir())

- create directory monitors in parallel (this also includes rmdir in
  case of directory is empty, since even if the directory is empty it
  may take some time to remove it, due to waiting for journal or if the
  directory is large, i.e. it had lots of files before, since remember
  ext4 does not truncate the directory size on each unlink [1])
- initialize increment in parallel too (since it does readdir())

  [1]: https://lore.kernel.org/linux-ext4/930A5754-5CE6-4567-8CF0-62447C97825C@dilger.ca/
@robot-clickhouse robot-clickhouse added the pr-improvement Pull request with some product improvements label Jun 24, 2021
@alexey-milovidov alexey-milovidov self-assigned this Jun 27, 2021
@alexey-milovidov alexey-milovidov merged commit 1b644b9 into ClickHouse:master Jun 27, 2021
@azat azat deleted the dist-startup branch June 27, 2021 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-improvement Pull request with some product improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants