Optimization Partitioner #241
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chia7712
reviewed
Feb 18, 2022
app/src/main/java/org/astraea/partitioner/nodeLoadMetric/NodeLoadClient.java
Show resolved
Hide resolved
app/src/main/java/org/astraea/partitioner/smoothPartitioner/SmoothWeightPartitioner.java
Outdated
Show resolved
Hide resolved
|
我繼續優化，減少了一些臨時變量，讓GC可以減輕一些工作。 但還存在一些問題 #244 ，我會開新的pr處理。 |
請問這句是什麼意思？ |
就是astraea partitioner 在10KiB的record.size大小下，传输速率能达到与kafka同样的速度（原先只有500M/s，第一次修改后大概有850MB/S,这次修改让他能够到达1000MB/s） |
所以是在叢集從0開始跑，沒有刻意製造不均勻的負載的狀態下？ |
製造與不製造都能做到相同速度。 |
chia7712
reviewed
Feb 21, 2022
app/src/main/java/org/astraea/partitioner/smoothPartitioner/SmoothWeightPartitioner.java
Show resolved
Hide resolved
chia7712
reviewed
Feb 21, 2022
chia7712
reviewed
Feb 21, 2022
app/src/main/java/org/astraea/partitioner/smoothPartitioner/SmoothWeightPartitioner.java
Outdated
Show resolved
Hide resolved
chia7712
reviewed
Feb 21, 2022
app/src/main/java/org/astraea/partitioner/smoothPartitioner/SmoothWeightPartitioner.java
Outdated
Show resolved
Hide resolved
chia7712
previously approved these changes
Feb 21, 2022
app/src/main/java/org/astraea/partitioner/nodeLoadMetric/PartitionerUtils.java
Show resolved
Hide resolved
app/src/main/java/org/astraea/partitioner/nodeLoadMetric/PartitionerUtils.java
Outdated
Show resolved
Hide resolved
app/src/main/java/org/astraea/partitioner/nodeLoadMetric/PartitionerUtils.java
Outdated
Show resolved
Hide resolved
chinghongfang
approved these changes
Feb 22, 2022
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
優化Partitioner效能，現在在10KiB的record.size下也能保持一定的效能。
右邊爲改善過後的astraea partitioner的數據，原先的數據結果大概爲30min avg=500MB/s
主要的改善點爲減少不必要的該方法
brokersWeight調用。JMC告訴我該方法佔了整個程式12%的cpu使用量。而如果metrics不變這個方法的結果就不會改變。metrics最快也是1s更新一次，所以改爲超過1s再進行一次計算。