Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Optimize write time cost when periodic full compaction #907

Merged
merged 37 commits into from
May 25, 2023
Merged

fix: Optimize write time cost when periodic full compaction #907

merged 37 commits into from
May 25, 2023

Conversation

MichaelLeeHZ
Copy link
Contributor

Which issue does this PR close?

Closes #

Rationale for this change

Currently cost time of writing data will increase during periodic full compaction, resulting in write requests failing.

What changes are included in this PR?

  • make aliyun params keep_alive configurable
  • use io_runtime to execute object store, not compact_runtime
  • use max_ongoing_task to reduce traffic peak of compaction tasks
  • remove compaction after flush

Are there any user-facing changes?

How does this change test

@MichaelLeeHZ MichaelLeeHZ marked this pull request as ready for review May 23, 2023 07:58
chunshao90
chunshao90 reviewed May 23, 2023
View reviewed changes
analytic_engine/src/instance/flush_compaction.rs Show resolved Hide resolved
analytic_engine/src/instance/flush_compaction.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
chunshao90
chunshao90 reviewed May 24, 2023
View reviewed changes
analytic_engine/src/setup.rs Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
jiacai2050
jiacai2050 reviewed May 24, 2023
View reviewed changes
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
analytic_engine/src/compaction/scheduler.rs Outdated Show resolved Hide resolved
components/object_store/src/metrics.rs Outdated Show resolved Hide resolved
jiacai2050
jiacai2050 approved these changes May 24, 2023
View reviewed changes
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MichaelLeeHZ MichaelLeeHZ merged commit b2b21e1 into apache:main May 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Rachelint Rachelint Rachelint left review comments

@chunshao90 chunshao90 chunshao90 left review comments

@jiacai2050 jiacai2050 jiacai2050 approved these changes

@ShiKaiWi ShiKaiWi Awaiting requested review from ShiKaiWi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MichaelLeeHZ @jiacai2050 @chunshao90 @Rachelint