Remove muzzle from CircleCI #8703
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
randomanderson
added
tag: no release notes
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 56 metrics, 15 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.027 s) : 0, 1026873
Total [baseline] (10.509 s) : 0, 10508623
Agent [candidate] (1.022 s) : 0, 1021744
Total [candidate] (10.534 s) : 0, 10534101
section appsec
Agent [baseline] (1.167 s) : 0, 1167269
Total [baseline] (10.815 s) : 0, 10814876
Agent [candidate] (1.161 s) : 0, 1161219
Total [candidate] (10.715 s) : 0, 10714618
section iast
Agent [baseline] (1.156 s) : 0, 1156329
Total [baseline] (10.875 s) : 0, 10874877
Agent [candidate] (1.157 s) : 0, 1156501
Total [candidate] (10.908 s) : 0, 10907541
section profiling
Agent [baseline] (1.281 s) : 0, 1281301
Total [baseline] (10.857 s) : 0, 10857315
Agent [candidate] (1.299 s) : 0, 1299291
Total [candidate] (10.869 s) : 0, 10868849
gantt
title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.791 ms) : 0, 688791
BytebuddyAgent [candidate] (682.86 ms) : 0, 682860
GlobalTracer [baseline] (241.782 ms) : 0, 241782
GlobalTracer [candidate] (239.932 ms) : 0, 239932
AppSec [baseline] (55.033 ms) : 0, 55033
AppSec [candidate] (55.318 ms) : 0, 55318
Debugger [baseline] (7.74 ms) : 0, 7740
Debugger [candidate] (10.434 ms) : 0, 10434
Remote Config [baseline] (705.669 µs) : 0, 706
Remote Config [candidate] (677.076 µs) : 0, 677
Telemetry [baseline] (9.194 ms) : 0, 9194
Telemetry [candidate] (9.114 ms) : 0, 9114
section appsec
BytebuddyAgent [baseline] (705.134 ms) : 0, 705134
BytebuddyAgent [candidate] (700.748 ms) : 0, 700748
GlobalTracer [baseline] (237.461 ms) : 0, 237461
GlobalTracer [candidate] (236.486 ms) : 0, 236486
AppSec [baseline] (175.75 ms) : 0, 175750
AppSec [candidate] (175.618 ms) : 0, 175618
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (5.972 ms) : 0, 5972
Remote Config [baseline] (643.955 µs) : 0, 644
Remote Config [candidate] (623.999 µs) : 0, 624
Telemetry [baseline] (7.84 ms) : 0, 7840
Telemetry [candidate] (7.453 ms) : 0, 7453
IAST [baseline] (21.684 ms) : 0, 21684
IAST [candidate] (21.708 ms) : 0, 21708
section iast
BytebuddyAgent [baseline] (807.993 ms) : 0, 807993
BytebuddyAgent [candidate] (808.503 ms) : 0, 808503
GlobalTracer [baseline] (231.389 ms) : 0, 231389
GlobalTracer [candidate] (231.733 ms) : 0, 231733
AppSec [baseline] (51.925 ms) : 0, 51925
AppSec [candidate] (50.86 ms) : 0, 50860
Debugger [baseline] (5.951 ms) : 0, 5951
Debugger [candidate] (5.868 ms) : 0, 5868
Remote Config [baseline] (597.752 µs) : 0, 598
Remote Config [candidate] (587.39 µs) : 0, 587
Telemetry [baseline] (7.868 ms) : 0, 7868
Telemetry [candidate] (7.941 ms) : 0, 7941
IAST [baseline] (25.398 ms) : 0, 25398
IAST [candidate] (26.697 ms) : 0, 26697
section profiling
BytebuddyAgent [baseline] (673.443 ms) : 0, 673443
BytebuddyAgent [candidate] (683.62 ms) : 0, 683620
GlobalTracer [baseline] (378.377 ms) : 0, 378377
GlobalTracer [candidate] (383.617 ms) : 0, 383617
AppSec [baseline] (54.783 ms) : 0, 54783
AppSec [candidate] (54.584 ms) : 0, 54584
Debugger [baseline] (6.142 ms) : 0, 6142
Debugger [candidate] (6.214 ms) : 0, 6214
Remote Config [baseline] (655.501 µs) : 0, 656
Remote Config [candidate] (725.455 µs) : 0, 725
Telemetry [baseline] (8.128 ms) : 0, 8128
Telemetry [candidate] (8.238 ms) : 0, 8238
ProfilingAgent [baseline] (109.271 ms) : 0, 109271
ProfilingAgent [candidate] (111.437 ms) : 0, 111437
Profiling [baseline] (109.297 ms) : 0, 109297
Profiling [candidate] (111.462 ms) : 0, 111462
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044599
Total [baseline] (8.732 s) : 0, 8731574
Agent [candidate] (1.022 s) : 0, 1021836
Total [candidate] (8.648 s) : 0, 8648324
section iast
Agent [baseline] (1.156 s) : 0, 1155965
Total [baseline] (9.264 s) : 0, 9264452
Agent [candidate] (1.147 s) : 0, 1147454
Total [candidate] (9.251 s) : 0, 9250785
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1149828
Total [baseline] (9.238 s) : 0, 9237635
Agent [candidate] (1.152 s) : 0, 1152123
Total [candidate] (9.257 s) : 0, 9256904
section iast_TELEMETRY_OFF
Agent [baseline] (1.145 s) : 0, 1145448
Total [baseline] (9.32 s) : 0, 9320267
Agent [candidate] (1.16 s) : 0, 1160469
Total [candidate] (9.267 s) : 0, 9266576
gantt
title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (701.042 ms) : 0, 701042
BytebuddyAgent [candidate] (682.16 ms) : 0, 682160
GlobalTracer [baseline] (244.115 ms) : 0, 244115
GlobalTracer [candidate] (239.786 ms) : 0, 239786
AppSec [baseline] (55.578 ms) : 0, 55578
AppSec [candidate] (55.158 ms) : 0, 55158
Debugger [baseline] (9.936 ms) : 0, 9936
Debugger [candidate] (10.776 ms) : 0, 10776
Remote Config [baseline] (708.155 µs) : 0, 708
Remote Config [candidate] (685.892 µs) : 0, 686
Telemetry [baseline] (9.214 ms) : 0, 9214
Telemetry [candidate] (9.81 ms) : 0, 9810
section iast
BytebuddyAgent [baseline] (807.373 ms) : 0, 807373
BytebuddyAgent [candidate] (801.148 ms) : 0, 801148
GlobalTracer [baseline] (231.363 ms) : 0, 231363
GlobalTracer [candidate] (230.023 ms) : 0, 230023
AppSec [baseline] (50.653 ms) : 0, 50653
AppSec [candidate] (51.142 ms) : 0, 51142
Debugger [baseline] (5.962 ms) : 0, 5962
Debugger [candidate] (5.937 ms) : 0, 5937
Remote Config [baseline] (599.505 µs) : 0, 600
Remote Config [candidate] (586.466 µs) : 0, 586
Telemetry [baseline] (7.941 ms) : 0, 7941
Telemetry [candidate] (7.889 ms) : 0, 7889
IAST [baseline] (27.754 ms) : 0, 27754
IAST [candidate] (27.325 ms) : 0, 27325
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.359 ms) : 0, 802359
BytebuddyAgent [candidate] (803.73 ms) : 0, 803730
GlobalTracer [baseline] (230.699 ms) : 0, 230699
GlobalTracer [candidate] (231.174 ms) : 0, 231174
AppSec [baseline] (49.726 ms) : 0, 49726
AppSec [candidate] (48.982 ms) : 0, 48982
Debugger [baseline] (5.94 ms) : 0, 5940
Debugger [candidate] (5.924 ms) : 0, 5924
Remote Config [baseline] (580.318 µs) : 0, 580
Remote Config [candidate] (587.295 µs) : 0, 587
Telemetry [baseline] (7.951 ms) : 0, 7951
Telemetry [candidate] (7.924 ms) : 0, 7924
IAST [baseline] (28.381 ms) : 0, 28381
IAST [candidate] (30.256 ms) : 0, 30256
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.595 ms) : 0, 798595
BytebuddyAgent [candidate] (809.369 ms) : 0, 809369
GlobalTracer [baseline] (230.392 ms) : 0, 230392
GlobalTracer [candidate] (233.321 ms) : 0, 233321
AppSec [baseline] (55.318 ms) : 0, 55318
AppSec [candidate] (55.873 ms) : 0, 55873
Debugger [baseline] (5.962 ms) : 0, 5962
Debugger [candidate] (6.153 ms) : 0, 6153
Remote Config [baseline] (591.224 µs) : 0, 591
Remote Config [candidate] (628.569 µs) : 0, 629
Telemetry [baseline] (7.907 ms) : 0, 7907
Telemetry [candidate] (7.908 ms) : 0, 7908
IAST [baseline] (23.156 ms) : 0, 23156
IAST [candidate] (23.531 ms) : 0, 23531
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (1.342 ms) : 1322, 1362
. : milestone, 1342,
appsec (1.721 ms) : 1698, 1745
. : milestone, 1721,
appsec_no_iast (1.694 ms) : 1670, 1717
. : milestone, 1694,
code_origins (1.66 ms) : 1633, 1687
. : milestone, 1660,
iast (1.509 ms) : 1485, 1533
. : milestone, 1509,
profiling (1.511 ms) : 1488, 1533
. : milestone, 1511,
tracing (1.508 ms) : 1483, 1533
. : milestone, 1508,
section candidate
no_agent (1.351 ms) : 1332, 1370
. : milestone, 1351,
appsec (1.736 ms) : 1713, 1759
. : milestone, 1736,
appsec_no_iast (1.721 ms) : 1699, 1744
. : milestone, 1721,
code_origins (1.677 ms) : 1650, 1704
. : milestone, 1677,
iast (1.512 ms) : 1488, 1536
. : milestone, 1512,
profiling (1.502 ms) : 1479, 1526
. : milestone, 1502,
tracing (1.478 ms) : 1454, 1502
. : milestone, 1478,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (380.448 µs) : 361, 400
. : milestone, 380,
iast (520.947 µs) : 499, 542
. : milestone, 521,
iast_FULL (731.209 µs) : 709, 753
. : milestone, 731,
iast_GLOBAL (558.383 µs) : 537, 580
. : milestone, 558,
iast_HARDCODED_SECRET_DISABLED (525.431 µs) : 503, 548
. : milestone, 525,
iast_INACTIVE (460.434 µs) : 438, 483
. : milestone, 460,
iast_TELEMETRY_OFF (518.258 µs) : 495, 542
. : milestone, 518,
tracing (458.135 µs) : 436, 480
. : milestone, 458,
section candidate
no_agent (386.926 µs) : 367, 407
. : milestone, 387,
iast (513.323 µs) : 492, 535
. : milestone, 513,
iast_FULL (729.843 µs) : 708, 752
. : milestone, 730,
iast_GLOBAL (565.785 µs) : 544, 587
. : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (514.932 µs) : 493, 537
. : milestone, 515,
iast_INACTIVE (462.758 µs) : 441, 485
. : milestone, 463,
iast_TELEMETRY_OFF (504.573 µs) : 481, 528
. : milestone, 505,
tracing (459.549 µs) : 437, 482
. : milestone, 460,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1460, 1483
. : milestone, 1471,
appsec (2.409 ms) : 2360, 2459
. : milestone, 2409,
iast (2.187 ms) : 2125, 2249
. : milestone, 2187,
iast_GLOBAL (2.222 ms) : 2160, 2285
. : milestone, 2222,
profiling (2.044 ms) : 1993, 2096
. : milestone, 2044,
tracing (2.01 ms) : 1962, 2059
. : milestone, 2010,
section candidate
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (2.4 ms) : 2350, 2449
. : milestone, 2400,
iast (2.176 ms) : 2114, 2238
. : milestone, 2176,
iast_GLOBAL (2.224 ms) : 2161, 2286
. : milestone, 2224,
profiling (2.03 ms) : 1980, 2080
. : milestone, 2030,
tracing (1.999 ms) : 1951, 2047
. : milestone, 1999,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~5957d2a141, baseline=1.50.0-SNAPSHOT~d5d53cd553
dateFormat X
axisFormat %s
section baseline
no_agent (15.449 s) : 15449000, 15449000
. : milestone, 15449000,
appsec (15.189 s) : 15189000, 15189000
. : milestone, 15189000,
iast (18.55 s) : 18550000, 18550000
. : milestone, 18550000,
iast_GLOBAL (18.299 s) : 18299000, 18299000
. : milestone, 18299000,
profiling (15.057 s) : 15057000, 15057000
. : milestone, 15057000,
tracing (15.071 s) : 15071000, 15071000
. : milestone, 15071000,
section candidate
no_agent (14.873 s) : 14873000, 14873000
. : milestone, 14873000,
appsec (14.997 s) : 14997000, 14997000
. : milestone, 14997000,
iast (18.378 s) : 18378000, 18378000
. : milestone, 18378000,
iast_GLOBAL (18.155 s) : 18155000, 18155000
. : milestone, 18155000,
profiling (15.217 s) : 15217000, 15217000
. : milestone, 15217000,
tracing (14.967 s) : 14967000, 14967000
. : milestone, 14967000,
|
smola
reviewed
Apr 21, 2025
Member
smola
left a comment
smola
left a comment
There was a problem hiding this comment.
We should add a required job to the GitLab pipeline and set it to required for merge in the GitHub settings.
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
Would muzzle runs consistently in GitLab without a pro Nexus? (just checking)
Contributor
Author
|
@PerfectSlayer Pro Nexus is more of a nice to have than a necessity. |
Contributor
Author
|
@smola the |
Contributor
|
So is muzzle running from GitLab a required GitHub PR check already? |
Contributor
Author
|
@PerfectSlayer I don't have the admin role to change required checks. The |
sarahchen6
approved these changes
May 5, 2025
Contributor
sarahchen6
left a comment
sarahchen6
left a comment
There was a problem hiding this comment.
Perhaps someone with an admin role should set muzzle Gitlab jobs (or the required Gitlab job) as required for PRs in the meantime, but otherwise LGTM!
Contributor
|
I don’t think you can do it on the UI. We will need to code some GH API call to update the PR checks requirements. |
Contributor
|
Same remarks as @PerfectSlayer |
Contributor
Author
|
@PerfectSlayer @bric3 Turns out, muzzle checks weren't using the proxy. During the rate limit incident last week, muzzle failures in Gitlab rose to ~10%. (Fix Here) The stability will only increase when the proxy is used. |
Contributor
We just migrated #8771 so you would be able to add them from there 🙌 And I will let @bric3 and @sarahchen6 approve, migrate, and ensure removal the CI pipelines to prevent being myself a blocker of the migration :) |
Contributor
Also, the gatekeeper check currently excludes Gitlab muzzle checks due to low success rate. I'm thinking we should probably wait to merge this PR until after the muzzle checks stabilize, they are removed from the excluded list in the gatekeeper check, and the gatekeeper check is required. 🤔 |
Contributor
Author
|
@sarahchen6 I removed muzzle from the all-green excluded list and think this PR is ready to merge Looking at the past month for muzzle only,
This is with the Maven Central incident, updates to the internal proxy, and actual muzzle failures because of the release of new libraries. Muzzle is very stable in Gitlab (less failures than CircleCI and only one-half of one-percent flakiness at worse). Comparatively, |
Contributor
|
My only concern would be that the gatekeeping job is not yet required, so there would be nothing requiring muzzle tests to pass before merging a PR. Otherwise, the numbers look good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Removes muzzle jobs from CircleCI
Motivation
Eventually, all jobs will be migrated from CircleCI to GitLab. Muzzle jobs in Gitlab have comparable failure rates of 1-3% with a flakyness of 0%. The flakyness indicates that these failures were actual failures and not issues with the CI system.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]