Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: sysbench randoms point perf fluctuate significantly on daily regression #20993

Open
1 task done
aressu1985 opened this issue Dec 27, 2024 · 3 comments
Open
1 task done
Assignees
Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone

Comments

@aressu1985
Copy link
Contributor

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

2.0-dev

Commit ID

a060ad1

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
3*PROXY: 3C 7G
- OS type:
- Others:

Actual Behavior

[tps = 38483.32]
job link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/12468455215/job/34813223925

profile:
image

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%226VN%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7B%20pod%20%3D%20%27nightly-regression-dis-tp-cn-9qvrv%27%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22process_cpu:cpu:nanoseconds:cpu:nanoseconds%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221734990420000%22,%22to%22:%221734990779000%22%7D%7D%7D&schemaVersion=1&orgId=1

[tps = 57880]
job link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/12455141774

profile:
image

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%226VN%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7B%20pod%20%3D%20%27nightly-regression-dis-tp-cn-lsnkv%27%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22process_cpu:cpu:nanoseconds:cpu:nanoseconds%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221734731460000%22,%22to%22:%221734731819000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

NA

Additional information

No response

@aressu1985 aressu1985 added kind/bug Something isn't working needs-triage severity/s0 Extreme impact: Cause the application to break down and seriously affect the use labels Dec 27, 2024
@aressu1985 aressu1985 added this to the 2.0.2 milestone Dec 27, 2024
@badboynt1
Copy link
Contributor

tps变低的时候, CollectOffsetsByPrefixInFactory这个函数,以及gc的开销明显变大了。 这是最主要的两块
@XuPeng-SH

@badboynt1 badboynt1 assigned XuPeng-SH and unassigned badboynt1 Dec 27, 2024
@XuPeng-SH XuPeng-SH assigned gouhongshen and unassigned XuPeng-SH Dec 27, 2024
@gouhongshen
Copy link
Contributor

这个函数的实现看起来没啥问题,可能是调用的频次或者 prefix in 带的数据变多了,要进一步排查。

@aressu1985 aressu1985 modified the milestones: 2.0.2, 2.1.0 Jan 6, 2025
@gouhongshen
Copy link
Contributor

no progress

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Development

No branches or pull requests

5 participants