Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance tool non throttle #1378

Merged
merged 5 commits into from
Dec 31, 2022

Conversation

harryteng9527
Copy link
Collaborator

@harryteng9527 harryteng9527 commented Dec 31, 2022

此 PR 為 Performance tool 新增一個判斷式,用來判斷有沒有開啟 throttle

突然想到當初沒有測到沒加入 throttle 功能的打資料效果,發現會影響到 performance tool producers 的吞吐量,如下圖實作throttle前實作throttle後的差異

實作throttle前的 Revision number: 3123a22
throttle_bug_1
測試指令:

./start_app.sh performance --bootstrap.servers 192.168.103.171:9092,192.168.103.172:9092,192.168.103.173:9092 --topics a1,a2,a10 --producers 5 --consumers 0 --run.until 3m

經過確認後發現是修改 partition selector 的問題,需要加上一個判斷式來判斷有無 throttle


下圖為此 PR 修正後的測試 throttle 功能的數據
throttle_fix_1

  1. 沒有設置 throttle 的指令
ACCOUNT=harryteng9527 VERSION=fix-non-throttle ./start_app.sh performance --bootstrap.servers 192.168.103.171:9092,192.168.103.172:9092 --topics a1,a2,a10 --producers 5 --consumers 0 --run.until 3m
  1. 設置 throttle 的指令
ACCOUNT=harryteng9527 VERSION=fix-non-throttle ./start_app.sh performance --bootstrap.servers 192.168.103.171:9092,192.168.103.172:9092 --topics a1,a2,a10 --producers 5 --consumers 0 --run.until 3m --throttle a1-0:5MB/s,a2-0:10MB/s,a2-1:20MB/s

下次會在更細心的檢查,不好意思


不過在修改此 PR 時,發現新增完 throttle 功能後,throughput 會下降,如上圖 新增 throttle 前未 throttle

造成吞吐量下降的原因應該是 queue 的影響,我把目前 producers 共享一個 queue 的作法改成每個 producer 使用一個 queue 和 把目前 size 為 1000 的 queue 加大到 3000 後,發現 throughput 上升了兩倍,如下圖吞吐量數據

image

@harryteng9527
Copy link
Collaborator Author

@chia7712 再麻煩學長看一下會不會影響到 release 的時程,謝謝

@chia7712
Copy link
Contributor

造成吞吐量下降的原因應該是 queue 的影響,我把目前 producers 共享一個 queue 的作法改成每個 producer 使用一個 queue 和 把目前 size 為 1000 的 queue 加大到 3000 後,發現 throughput 上升了兩倍,如下圖吞吐量數據

請問這個有包含在此PR中嗎

@harryteng9527
Copy link
Collaborator Author

請問這個有包含在此PR中嗎

沒有,要現在放上去嗎

@chia7712
Copy link
Contributor

沒有,要現在放上去嗎

1 similar comment
@chia7712
Copy link
Contributor

沒有,要現在放上去嗎

@chia7712
Copy link
Contributor

沒有,要現在放上去嗎

是的,跟效能有關的話麻煩先上,要包含在 0.1.0

chia7712
chia7712 previously approved these changes Dec 31, 2022
Copy link
Contributor

@chia7712 chia7712 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harryteng9527
Copy link
Collaborator Author

已新增 TODO,再麻煩學長 review ,謝謝

@chia7712
Copy link
Contributor

已新增 TODO,再麻煩學長 review ,謝謝

記得開議題

@chia7712 chia7712 merged commit 5ed4636 into opensource4you:main Dec 31, 2022
@harryteng9527 harryteng9527 deleted the fix-non-throttle branch April 17, 2023 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants