Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: compare benchmark between PR and master #1244

Merged
merged 3 commits into from
Apr 25, 2023

Conversation

Desiki-high
Copy link
Member

@Desiki-high Desiki-high commented Apr 22, 2023

We should compare the benchmark between PR and master. Ref: #1221.

1. Refator the benchmark summary.

To add more features in benchmark summary, bash is difficult to dev. So we use python to refactor.

2. Add arg mode for the benchmark_summary.py.

Adaptate different benchmark summmarys for push, schedule and pull_request triggered events. We only need to compare the benchmark in pull request event.

3. Add image-size in benchmark.

Calculate the sum of each layers(nydus blob) size in manifest.

4. Add the benchmark-compare jobs in smoke test.

  • two build jobs in master(nydusify and nydusd).
  • five benchmark jobs with master build jobs(we don't need to repeat benchmark-oci).
  • one benchmark summary job for compare the result.

benchmark summary on pull_request

bench-result pull-elapsed(s) create-elapsed(s) run-elapsed(s) total-elapsed(s) image-size(MB) read-amount(MB) read-count
oci 30.920854 0.391352 1.362971 32.675177 207.08 - -
nydus-all-prefetch-master 0.691130 2.257612 21.528317 24.477059 211.01 212.97 154
nydus-all-prefetch 0.920582 2.620666 22.260284 25.801532 211.01 211.24 151
zran-all-prefetch-master 1.100418 2.788972 22.447882 26.337272 9.89 216.66 161
zran-all-prefetch 0.891196 3.327982 21.676711 25.895889 9.89 218.4 179
nydus-no-prefetch-master 1.141136 0.817566 11.113056 13.071758 211.01 72.38 129
nydus-no-prefetch 0.685296 0.834542 9.747356 11.267194 211.01 73.23 128
zran-no-prefetch-master 0.888910 1.892924 9.543010 12.324844 9.89 74.05 181
zran-no-prefetch 0.905690 1.959104 10.028516 12.893310 9.89 74.93 182
nydus-filelist-prefetch-master 0.911920 2.200072 8.759456 11.871448 211.03 81.47 164
nydus-filelist-prefetch 0.689176 1.567108 8.298095 10.554379 211.03 80.92 162

benchmark summary on push or schedule

bench-result pull-elapsed(s) create-elapsed(s) run-elapsed(s) total-elapsed(s) image-size(MB) read-amount(MB) read-count
oci 28.269394 0.216570 0.981491 29.467455 207.08 - -
nydus-all-prefetch 0.885440 2.740374 21.686351 25.312165 211.01 211.27 152
zran-all-prefetch 0.692972 2.905208 21.927755 25.525935 9.89 216.9 173
nydus-no-prefetch 0.684420 0.793996 9.642554 11.120970 211.01 72.37 128
zran-no-prefetch 0.922228 1.938336 9.941942 12.802506 9.89 73.66 181
nydus-filelist-prefetch 0.684902 1.977024 8.149241 10.811167 211.03 80.39 165

@Desiki-high Desiki-high requested review from a team as code owners April 22, 2023 15:12
@Desiki-high Desiki-high requested review from bergwolf, liubogithub and hsiangkao and removed request for a team April 22, 2023 15:12
@codecov
Copy link

codecov bot commented Apr 22, 2023

Codecov Report

Merging #1244 (b12a3d9) into master (a78db8b) will decrease coverage by 0.03%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1244      +/-   ##
==========================================
- Coverage   45.03%   45.01%   -0.03%     
==========================================
  Files         126      126              
  Lines       37382    37382              
  Branches    37382    37382              
==========================================
- Hits        16836    16827       -9     
- Misses      19659    19668       +9     
  Partials      887      887              

see 4 files with indirect coverage changes

@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 2 times, most recently from 2067d1a to 37e69db Compare April 22, 2023 16:34
@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 3 times, most recently from 0ad1e4c to 2440bd0 Compare April 23, 2023 08:31
@Desiki-high Desiki-high added the enhancement New feature or request label Apr 23, 2023
@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 3 times, most recently from 063bd21 to d6c9233 Compare April 23, 2023 11:29
1. refactor.
2. add the arg mode to adapt two benchmark summary modes.

Signed-off-by: Desiki-high <ding_yadong@foxmail.com>
@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 12 times, most recently from 1ecd822 to 8b6f47a Compare April 24, 2023 10:37
@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 3 times, most recently from b0ac69d to f1aac18 Compare April 24, 2023 11:01
compare the benchmark result between PR and master when smoke test triggered by pull request

Signed-off-by: Desiki-high <ding_yadong@foxmail.com>
@Desiki-high Desiki-high force-pushed the benchmark-pr-master branch 4 times, most recently from 5c80376 to 657023c Compare April 24, 2023 11:48
Signed-off-by: Desiki-high <ding_yadong@foxmail.com>
@Desiki-high Desiki-high changed the title [WIP]feat: compare benchmark between PR and master feat: compare benchmark between PR and master Apr 24, 2023
@anolis-bot
Copy link
Collaborator

@Desiki-high , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/69835

@anolis-bot
Copy link
Collaborator

@Desiki-high , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@anolis-bot
Copy link
Collaborator

@Desiki-high , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/69851

@anolis-bot
Copy link
Collaborator

@Desiki-high , The CI test is completed, please check result:

Test CaseTest Result
build rust golang image✅ SUCCESS
compile nydusd✅ SUCCESS
compile ctr remote✅ SUCCESS
compile nydus snapshotter✅ SUCCESS
run container with rafs✅ SUCCESS
run container with zran✅ SUCCESS
run container with rafs and compile linux✅ SUCCESS

Congratulations, your test job passed!

@Desiki-high Desiki-high requested a review from imeoer April 25, 2023 02:58
Copy link
Collaborator

@imeoer imeoer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imeoer imeoer merged commit 7edea8a into dragonflyoss:master Apr 25, 2023
@Desiki-high Desiki-high deleted the benchmark-pr-master branch April 25, 2023 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
anolis_test_pass enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: Compare the current benchmark tests with the master branch's.
3 participants