Skip to content

SSI smoke tests should only run as part of nightly #8914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
randomanderson merged 1 commit into from
Jun 3, 2025
Merged

SSI smoke tests should only run as part of nightly #8914

randomanderson merged 1 commit into from
Jun 3, 2025

Conversation

@randomanderson
Copy link
Contributor

@randomanderson randomanderson commented Jun 3, 2025

What Does This Do

SSI smoke tests are extremely flaky and should only run as part of the nightly build, for now. This was missed as part of #8900

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@randomanderson randomanderson requested a review from a team as a code owner June 3, 2025 16:03
@randomanderson randomanderson requested review from erikayasuda and removed request for a team June 3, 2025 16:03
@github-actions
Copy link
Contributor

github-actions bot commented Jun 3, 2025

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@pr-commenter
Copy link

pr-commenter bot commented Jun 3, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master landerson/ssi-nightly-only
git_commit_date 1746789389 1748966393
git_commit_sha ad6d5fe a293b4c
release_version 1.50.0-SNAPSHOT~ad6d5fef42 1.50.0-SNAPSHOT~a293b4cb79
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1748968923 1748968923
ci_job_id 965826307 965826307
ci_pipeline_id 66830315 66830315
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-rdwhfy8--project-304-concurrent-0-wgk3kup7 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-rdwhfy8--project-304-concurrent-0-wgk3kup7 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 4 performance improvements and 2 performance regressions! Performance is the same for 52 metrics, 13 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:AppSec better
[-8.072ms; -3.173ms] or [-14.282%; -5.615%]		 50.892ms 56.515ms
scenario:startup:petclinic:profiling:ProfilingAgent better
[-7.814ms; -4.589ms] or [-7.189%; -4.221%]		 102.502ms 108.704ms
scenario:startup:petclinic:profiling:GlobalTracer better
[-21.834ms; -17.742ms] or [-5.754%; -4.676%]		 359.676ms 379.464ms
scenario:startup:petclinic:profiling:AppSec worse
[+5.586ms; +8.794ms] or [+10.303%; +16.221%]		 61.403ms 54.213ms
scenario:startup:petclinic:profiling:Profiling better
[-7.817ms; -4.590ms] or [-7.189%; -4.221%]		 102.526ms 108.729ms
scenario:startup:petclinic:tracing:AppSec worse
[+1.107ms; +5.750ms] or [+2.041%; +10.598%]		 57.683ms 54.254ms
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1026388
Total [baseline] (8.513 s) : 0, 8512746
Agent [candidate] (1.022 s) : 0, 1021844
Total [candidate] (8.56 s) : 0, 8559695
section iast
Agent [baseline] (1.144 s) : 0, 1144469
Total [baseline] (9.146 s) : 0, 9146103
Agent [candidate] (1.148 s) : 0, 1148348
Total [candidate] (9.173 s) : 0, 9173488
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.145 s) : 0, 1145437
Total [baseline] (9.173 s) : 0, 9173068
Agent [candidate] (1.156 s) : 0, 1155757
Total [candidate] (9.179 s) : 0, 9179398
section iast_TELEMETRY_OFF
Agent [baseline] (1.147 s) : 0, 1147293
Total [baseline] (9.274 s) : 0, 9273938
Agent [candidate] (1.146 s) : 0, 1146486
Total [candidate] (9.241 s) : 0, 9241475
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.026 s -
Agent iast 1.144 s 118.082 ms (11.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.145 s 119.049 ms (11.6%)
Agent iast_TELEMETRY_OFF 1.147 s 120.905 ms (11.8%)
Total tracing 8.513 s -
Total iast 9.146 s 633.357 ms (7.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.173 s 660.323 ms (7.8%)
Total iast_TELEMETRY_OFF 9.274 s 761.192 ms (8.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent iast 1.148 s 126.504 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.156 s 133.913 ms (13.1%)
Agent iast_TELEMETRY_OFF 1.146 s 124.641 ms (12.2%)
Total tracing 8.56 s -
Total iast 9.173 s 613.794 ms (7.2%)
Total iast_HARDCODED_SECRET_DISABLED 9.179 s 619.703 ms (7.2%)
Total iast_TELEMETRY_OFF 9.241 s 681.78 ms (8.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (685.865 ms) : 0, 685865
BytebuddyAgent [candidate] (683.052 ms) : 0, 683052
GlobalTracer [baseline] (240.145 ms) : 0, 240145
GlobalTracer [candidate] (239.963 ms) : 0, 239963
AppSec [baseline] (54.831 ms) : 0, 54831
AppSec [candidate] (57.076 ms) : 0, 57076
Debugger [baseline] (12.089 ms) : 0, 12089
Debugger [candidate] (6.178 ms) : 0, 6178
Remote Config [baseline] (709.794 µs) : 0, 710
Remote Config [candidate] (732.803 µs) : 0, 733
Telemetry [baseline] (9.097 ms) : 0, 9097
Telemetry [candidate] (11.397 ms) : 0, 11397
section iast
BytebuddyAgent [baseline] (798.929 ms) : 0, 798929
BytebuddyAgent [candidate] (801.178 ms) : 0, 801178
GlobalTracer [baseline] (229.037 ms) : 0, 229037
GlobalTracer [candidate] (230.45 ms) : 0, 230450
IAST [baseline] (29.159 ms) : 0, 29159
IAST [candidate] (28.322 ms) : 0, 28322
AppSec [baseline] (49.606 ms) : 0, 49606
AppSec [candidate] (50.535 ms) : 0, 50535
Debugger [baseline] (5.957 ms) : 0, 5957
Debugger [candidate] (5.908 ms) : 0, 5908
Remote Config [baseline] (589.566 µs) : 0, 590
Remote Config [candidate] (604.345 µs) : 0, 604
Telemetry [baseline] (7.793 ms) : 0, 7793
Telemetry [candidate] (7.879 ms) : 0, 7879
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (799.24 ms) : 0, 799240
BytebuddyAgent [candidate] (806.267 ms) : 0, 806267
GlobalTracer [baseline] (229.31 ms) : 0, 229310
GlobalTracer [candidate] (231.615 ms) : 0, 231615
IAST [baseline] (29.136 ms) : 0, 29136
IAST [candidate] (27.94 ms) : 0, 27940
AppSec [baseline] (49.772 ms) : 0, 49772
AppSec [candidate] (51.777 ms) : 0, 51777
Debugger [baseline] (5.986 ms) : 0, 5986
Debugger [candidate] (5.992 ms) : 0, 5992
Remote Config [baseline] (586.376 µs) : 0, 586
Remote Config [candidate] (597.115 µs) : 0, 597
Telemetry [baseline] (7.882 ms) : 0, 7882
Telemetry [candidate] (7.993 ms) : 0, 7993
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (800.546 ms) : 0, 800546
BytebuddyAgent [candidate] (798.949 ms) : 0, 798949
GlobalTracer [baseline] (229.657 ms) : 0, 229657
GlobalTracer [candidate] (230.511 ms) : 0, 230511
IAST [baseline] (22.635 ms) : 0, 22635
IAST [candidate] (28.105 ms) : 0, 28105
AppSec [baseline] (56.515 ms) : 0, 56515
AppSec [candidate] (50.892 ms) : 0, 50892
Debugger [baseline] (5.968 ms) : 0, 5968
Debugger [candidate] (5.999 ms) : 0, 5999
Remote Config [baseline] (621.016 µs) : 0, 621
Remote Config [candidate] (600.516 µs) : 0, 601
Telemetry [baseline] (7.836 ms) : 0, 7836
Telemetry [candidate] (7.924 ms) : 0, 7924
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1017505
Total [baseline] (11.108 s) : 0, 11108203
Agent [candidate] (1.024 s) : 0, 1024325
Total [candidate] (11.094 s) : 0, 11094409
section appsec
Agent [baseline] (1.164 s) : 0, 1164057
Total [baseline] (11.296 s) : 0, 11296441
Agent [candidate] (1.168 s) : 0, 1168087
Total [candidate] (11.261 s) : 0, 11261125
section iast
Agent [baseline] (1.153 s) : 0, 1152903
Total [baseline] (11.284 s) : 0, 11284339
Agent [candidate] (1.148 s) : 0, 1148112
Total [candidate] (11.354 s) : 0, 11354051
section profiling
Agent [baseline] (1.281 s) : 0, 1281252
Total [baseline] (11.473 s) : 0, 11473197
Agent [candidate] (1.263 s) : 0, 1263304
Total [candidate] (11.442 s) : 0, 11441544
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent appsec 1.164 s 146.551 ms (14.4%)
Agent iast 1.153 s 135.398 ms (13.3%)
Agent profiling 1.281 s 263.746 ms (25.9%)
Total tracing 11.108 s -
Total appsec 11.296 s 188.238 ms (1.7%)
Total iast 11.284 s 176.136 ms (1.6%)
Total profiling 11.473 s 364.994 ms (3.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.024 s -
Agent appsec 1.168 s 143.761 ms (14.0%)
Agent iast 1.148 s 123.787 ms (12.1%)
Agent profiling 1.263 s 238.979 ms (23.3%)
Total tracing 11.094 s -
Total appsec 11.261 s 166.716 ms (1.5%)
Total iast 11.354 s 259.642 ms (2.3%)
Total profiling 11.442 s 347.135 ms (3.1%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (679.584 ms) : 0, 679584
BytebuddyAgent [candidate] (684.729 ms) : 0, 684729
GlobalTracer [baseline] (238.789 ms) : 0, 238789
GlobalTracer [candidate] (239.779 ms) : 0, 239779
AppSec [baseline] (54.254 ms) : 0, 54254
AppSec [candidate] (57.683 ms) : 0, 57683
Debugger [baseline] (9.872 ms) : 0, 9872
Debugger [candidate] (6.937 ms) : 0, 6937
Remote Config [baseline] (691.553 µs) : 0, 692
Remote Config [candidate] (724.965 µs) : 0, 725
Telemetry [baseline] (10.784 ms) : 0, 10784
Telemetry [candidate] (10.829 ms) : 0, 10829
section appsec
BytebuddyAgent [baseline] (702.482 ms) : 0, 702482
BytebuddyAgent [candidate] (704.011 ms) : 0, 704011
GlobalTracer [baseline] (237.081 ms) : 0, 237081
GlobalTracer [candidate] (238.657 ms) : 0, 238657
AppSec [baseline] (175.535 ms) : 0, 175535
AppSec [candidate] (176.774 ms) : 0, 176774
Debugger [baseline] (6.263 ms) : 0, 6263
Debugger [candidate] (6.029 ms) : 0, 6029
Remote Config [baseline] (630.821 µs) : 0, 631
Remote Config [candidate] (640.588 µs) : 0, 641
Telemetry [baseline] (7.35 ms) : 0, 7350
Telemetry [candidate] (7.417 ms) : 0, 7417
IAST [baseline] (22.132 ms) : 0, 22132
IAST [candidate] (21.824 ms) : 0, 21824
section iast
BytebuddyAgent [baseline] (805.179 ms) : 0, 805179
BytebuddyAgent [candidate] (800.367 ms) : 0, 800367
GlobalTracer [baseline] (230.575 ms) : 0, 230575
GlobalTracer [candidate] (229.787 ms) : 0, 229787
AppSec [baseline] (48.209 ms) : 0, 48209
AppSec [candidate] (52.023 ms) : 0, 52023
Debugger [baseline] (5.921 ms) : 0, 5921
Debugger [candidate] (5.972 ms) : 0, 5972
Remote Config [baseline] (616.342 µs) : 0, 616
Remote Config [candidate] (589.382 µs) : 0, 589
Telemetry [baseline] (7.859 ms) : 0, 7859
Telemetry [candidate] (7.949 ms) : 0, 7949
IAST [baseline] (30.206 ms) : 0, 30206
IAST [candidate] (27.91 ms) : 0, 27910
section profiling
ProfilingAgent [baseline] (108.704 ms) : 0, 108704
ProfilingAgent [candidate] (102.502 ms) : 0, 102502
BytebuddyAgent [baseline] (673.479 ms) : 0, 673479
BytebuddyAgent [candidate] (674.298 ms) : 0, 674298
GlobalTracer [baseline] (379.464 ms) : 0, 379464
GlobalTracer [candidate] (359.676 ms) : 0, 359676
AppSec [baseline] (54.213 ms) : 0, 54213
AppSec [candidate] (61.403 ms) : 0, 61403
Debugger [baseline] (6.093 ms) : 0, 6093
Debugger [candidate] (6.01 ms) : 0, 6010
Remote Config [baseline] (666.751 µs) : 0, 667
Remote Config [candidate] (649.189 µs) : 0, 649
Telemetry [baseline] (8.046 ms) : 0, 8046
Telemetry [candidate] (8.024 ms) : 0, 8024
Profiling [baseline] (108.729 ms) : 0, 108729
Profiling [candidate] (102.526 ms) : 0, 102526
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-03T16:11:33 2025-06-03T16:29:03
git_branch master landerson/ssi-nightly-only
git_commit_date 1746789389 1748966393
git_commit_sha ad6d5fe a293b4c
release_version 1.50.0-SNAPSHOT~ad6d5fef42 1.50.0-SNAPSHOT~a293b4cb79
start_time 2025-06-03T16:11:19 2025-06-03T16:28:48
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1748968757 1748968757
ci_job_id 965826308 965826308
ci_pipeline_id 66830315 66830315
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-bap2zmvn-project-304-concurrent-0-oo6ldxi2 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-bap2zmvn-project-304-concurrent-0-oo6ldxi2 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant iast iast

Summary

Found 0 performance improvements and 9 performance regressions! Performance is the same for 6 metrics, 15 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:appsec unstable
[+36.558ms; +38.006ms] or [+435.429%; +452.679%]		 worse
[-497.167op/s; -465.054op/s] or [-84.518%; -79.059%]		 45.677ms 107.125op/s 8.396ms 588.235op/s
scenario:load:petclinic:appsec_no_iast unstable
[+44.935ms; +46.959ms] or [+508.200%; +531.090%]		 worse
[-487.561op/s; -454.347op/s] or [-87.152%; -81.215%]		 54.789ms 88.486op/s 8.842ms 559.441op/s
scenario:load:petclinic:code_origins unstable
[+59.883ms; +62.428ms] or [+796.011%; +829.830%]		 worse
[-603.437op/s; -571.610op/s] or [-92.024%; -87.170%]		 68.678ms 68.214op/s 7.523ms 655.738op/s
scenario:load:petclinic:no_agent worse
[+614.323µs; +663.070µs] or [+8.016%; +8.653%]		 worse
[-70.899op/s; -34.238op/s] or [-10.989%; -5.307%]		 8.302ms 592.593op/s 7.663ms 645.161op/s
scenario:load:petclinic:profiling worse
[+3.873ms; +4.046ms] or [+47.855%; +49.989%]		 worse
[-230.027op/s; -172.267op/s] or [-37.667%; -28.209%]		 12.052ms 409.540op/s 8.093ms 610.687op/s
scenario:load:petclinic:tracing worse
[+780.780µs; +837.506µs] or [+10.605%; +11.376%]		 worse
[-88.052op/s; -42.274op/s] or [-13.208%; -6.341%]		 8.171ms 601.504op/s 7.362ms 666.667op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42
    dateFormat X
    axisFormat %s
section baseline
no_agent (378.549 µs) : 359, 398
.   : milestone, 379,
iast (518.717 µs) : 497, 540
.   : milestone, 519,
iast_FULL (746.189 µs) : 724, 768
.   : milestone, 746,
iast_GLOBAL (559.172 µs) : 537, 581
.   : milestone, 559,
iast_HARDCODED_SECRET_DISABLED (531.168 µs) : 508, 554
.   : milestone, 531,
iast_INACTIVE (486.899 µs) : 464, 509
.   : milestone, 487,
iast_TELEMETRY_OFF (519.453 µs) : 496, 543
.   : milestone, 519,
tracing (459.711 µs) : 437, 482
.   : milestone, 460,
section candidate
no_agent (383.771 µs) : 363, 404
.   : milestone, 384,
iast (523.217 µs) : 501, 545
.   : milestone, 523,
iast_FULL (744.7 µs) : 721, 768
.   : milestone, 745,
iast_GLOBAL (566.159 µs) : 543, 590
.   : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (517.744 µs) : 496, 539
.   : milestone, 518,
iast_INACTIVE (466.337 µs) : 444, 489
.   : milestone, 466,
iast_TELEMETRY_OFF (515.884 µs) : 493, 539
.   : milestone, 516,
tracing (471.66 µs) : 449, 494
.   : milestone, 472,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.549 µs [358.947 µs, 398.15 µs] -
iast 518.717 µs [497.098 µs, 540.335 µs] 140.168 µs (37.0%)
iast_FULL 746.189 µs [724.274 µs, 768.104 µs] 367.64 µs (97.1%)
iast_GLOBAL 559.172 µs [537.486 µs, 580.857 µs] 180.623 µs (47.7%)
iast_HARDCODED_SECRET_DISABLED 531.168 µs [508.115 µs, 554.221 µs] 152.619 µs (40.3%)
iast_INACTIVE 486.899 µs [464.322 µs, 509.477 µs] 108.351 µs (28.6%)
iast_TELEMETRY_OFF 519.453 µs [496.299 µs, 542.607 µs] 140.905 µs (37.2%)
tracing 459.711 µs [437.431 µs, 481.991 µs] 81.163 µs (21.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 383.771 µs [363.31 µs, 404.232 µs] -
iast 523.217 µs [501.355 µs, 545.08 µs] 139.446 µs (36.3%)
iast_FULL 744.7 µs [720.975 µs, 768.425 µs] 360.929 µs (94.0%)
iast_GLOBAL 566.159 µs [542.799 µs, 589.519 µs] 182.388 µs (47.5%)
iast_HARDCODED_SECRET_DISABLED 517.744 µs [496.207 µs, 539.28 µs] 133.973 µs (34.9%)
iast_INACTIVE 466.337 µs [444.037 µs, 488.637 µs] 82.566 µs (21.5%)
iast_TELEMETRY_OFF 515.884 µs [492.694 µs, 539.074 µs] 132.113 µs (34.4%)
tracing 471.66 µs [449.425 µs, 493.895 µs] 87.889 µs (22.9%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42
    dateFormat X
    axisFormat %s
section baseline
no_agent (7.663 ms) : 7641, 7686
.   : milestone, 7663,
appsec (8.396 ms) : 8370, 8422
.   : milestone, 8396,
appsec_no_iast (8.842 ms) : 8811, 8873
.   : milestone, 8842,
code_origins (7.523 ms) : 7493, 7552
.   : milestone, 7523,
iast (260.791 µs) : 237, 285
.   : milestone, 261,
profiling (8.093 ms) : 8064, 8122
.   : milestone, 8093,
tracing (7.362 ms) : 7336, 7389
.   : milestone, 7362,
section candidate
no_agent (8.302 ms) : 8279, 8325
.   : milestone, 8302,
appsec (45.677 ms) : 44726, 46629
.   : milestone, 45677,
appsec_no_iast (54.789 ms) : 53459, 56118
.   : milestone, 54789,
code_origins (68.678 ms) : 67007, 70350
.   : milestone, 68678,
iast (61.383 ms) : 60277, 62489
.   : milestone, 61383,
profiling (12.052 ms) : 11942, 12162
.   : milestone, 12052,
tracing (8.171 ms) : 8145, 8198
.   : milestone, 8171,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 7.663 ms [7.641 ms, 7.686 ms] -
appsec 8.396 ms [8.37 ms, 8.422 ms] 732.497 µs (9.6%)
appsec_no_iast 8.842 ms [8.811 ms, 8.873 ms] 1.179 ms (15.4%)
code_origins 7.523 ms [7.493 ms, 7.552 ms] -140.327 µs (-1.8%)
iast 260.791 µs [236.707 µs, 284.875 µs] -7.402 ms (-96.6%)
profiling 8.093 ms [8.064 ms, 8.122 ms] 429.611 µs (5.6%)
tracing 7.362 ms [7.336 ms, 7.389 ms] -300.979 µs (-3.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 8.302 ms [8.279 ms, 8.325 ms] -
appsec 45.677 ms [44.726 ms, 46.629 ms] 37.375 ms (450.2%)
appsec_no_iast 54.789 ms [53.459 ms, 56.118 ms] 46.487 ms (559.9%)
code_origins 68.678 ms [67.007 ms, 70.35 ms] 60.376 ms (727.3%)
iast 61.383 ms [60.277 ms, 62.489 ms] 53.081 ms (639.4%)
profiling 12.052 ms [11.942 ms, 12.162 ms] 3.75 ms (45.2%)
tracing 8.171 ms [8.145 ms, 8.198 ms] -130.533 µs (-1.6%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master landerson/ssi-nightly-only
git_commit_date 1746789389 1748966393
git_commit_sha ad6d5fe a293b4c
release_version 1.50.0-SNAPSHOT~ad6d5fef42 1.50.0-SNAPSHOT~a293b4cb79
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1748968466 1748968466
ci_job_id 965826309 965826309
ci_pipeline_id 66830315 66830315
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-eqiwq57k-project-304-concurrent-0-0y9j00xt 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-eqiwq57k-project-304-concurrent-0-0y9j00xt 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (2.395 ms) : 2347, 2443
.   : milestone, 2395,
iast (2.177 ms) : 2116, 2238
.   : milestone, 2177,
iast_GLOBAL (2.214 ms) : 2153, 2275
.   : milestone, 2214,
profiling (2.042 ms) : 1992, 2091
.   : milestone, 2042,
tracing (1.994 ms) : 1948, 2041
.   : milestone, 1994,
section candidate
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (2.389 ms) : 2341, 2437
.   : milestone, 2389,
iast (2.187 ms) : 2126, 2248
.   : milestone, 2187,
iast_GLOBAL (2.217 ms) : 2156, 2278
.   : milestone, 2217,
profiling (2.027 ms) : 1978, 2076
.   : milestone, 2027,
tracing (1.991 ms) : 1944, 2037
.   : milestone, 1991,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.479 ms [1.468 ms, 1.491 ms] -
appsec 2.395 ms [2.347 ms, 2.443 ms] 915.217 µs (61.9%)
iast 2.177 ms [2.116 ms, 2.238 ms] 697.475 µs (47.1%)
iast_GLOBAL 2.214 ms [2.153 ms, 2.275 ms] 734.6 µs (49.7%)
profiling 2.042 ms [1.992 ms, 2.091 ms] 562.416 µs (38.0%)
tracing 1.994 ms [1.948 ms, 2.041 ms] 515.009 µs (34.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.479 ms [1.468 ms, 1.491 ms] -
appsec 2.389 ms [2.341 ms, 2.437 ms] 909.454 µs (61.5%)
iast 2.187 ms [2.126 ms, 2.248 ms] 707.413 µs (47.8%)
iast_GLOBAL 2.217 ms [2.156 ms, 2.278 ms] 737.756 µs (49.9%)
profiling 2.027 ms [1.978 ms, 2.076 ms] 548.019 µs (37.0%)
tracing 1.991 ms [1.944 ms, 2.037 ms] 511.448 µs (34.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~a293b4cb79, baseline=1.50.0-SNAPSHOT~ad6d5fef42
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.765 s) : 14765000, 14765000
.   : milestone, 14765000,
appsec (14.957 s) : 14957000, 14957000
.   : milestone, 14957000,
iast (18.505 s) : 18505000, 18505000
.   : milestone, 18505000,
iast_GLOBAL (18.172 s) : 18172000, 18172000
.   : milestone, 18172000,
profiling (15.313 s) : 15313000, 15313000
.   : milestone, 15313000,
tracing (14.742 s) : 14742000, 14742000
.   : milestone, 14742000,
section candidate
no_agent (14.982 s) : 14982000, 14982000
.   : milestone, 14982000,
appsec (14.772 s) : 14772000, 14772000
.   : milestone, 14772000,
iast (18.362 s) : 18362000, 18362000
.   : milestone, 18362000,
iast_GLOBAL (18.126 s) : 18126000, 18126000
.   : milestone, 18126000,
profiling (15.14 s) : 15140000, 15140000
.   : milestone, 15140000,
tracing (14.984 s) : 14984000, 14984000
.   : milestone, 14984000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.765 s [14.765 s, 14.765 s] -
appsec 14.957 s [14.957 s, 14.957 s] 192.0 ms (1.3%)
iast 18.505 s [18.505 s, 18.505 s] 3.74 s (25.3%)
iast_GLOBAL 18.172 s [18.172 s, 18.172 s] 3.407 s (23.1%)
profiling 15.313 s [15.313 s, 15.313 s] 548.0 ms (3.7%)
tracing 14.742 s [14.742 s, 14.742 s] -23.0 ms (-0.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.982 s [14.982 s, 14.982 s] -
appsec 14.772 s [14.772 s, 14.772 s] -210.0 ms (-1.4%)
iast 18.362 s [18.362 s, 18.362 s] 3.38 s (22.6%)
iast_GLOBAL 18.126 s [18.126 s, 18.126 s] 3.144 s (21.0%)
profiling 15.14 s [15.14 s, 15.14 s] 158.0 ms (1.1%)
tracing 14.984 s [14.984 s, 14.984 s] 2.0 ms (0.0%)

@randomanderson randomanderson added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 3, 2025
@randomanderson randomanderson enabled auto-merge (squash) June 3, 2025 17:03
@randomanderson randomanderson merged commit c06ee54 into master Jun 3, 2025
509 of 511 checks passed
@randomanderson randomanderson deleted the landerson/ssi-nightly-only branch June 3, 2025 19:57
@github-actions github-actions bot added this to the 1.50.0 milestone Jun 3, 2025
bric3 added a commit that referenced this pull request Jun 4, 2025
@bric3
Revert "SSI smoke tests should only run as part of nightly (#8914)"
4733dd3 
This reverts commit c06ee54.
@bric3 bric3 mentioned this pull request Jun 4, 2025
Merged
PerfectSlayer pushed a commit that referenced this pull request Jun 5, 2025
@bric3
Revert "SSI smoke tests should only run as part of nightly (#8914)" (#…
1205c9a 
…8924)

This reverts commit c06ee54.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erikayasuda erikayasuda erikayasuda approved these changes

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.50.0

Development

Successfully merging this pull request may close these issues.

3 participants

@randomanderson @erikayasuda