[Perf] Linux/x64: 1 Regression on 1/24/2026 4:18:23 AM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	b1d2e4aa8af89cfa795015e30ef70d24ec92f67f
Compare	8fafcb3251a3d2ccde36612a2eab47e9ae8ebe2b
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
HashSet - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	1.21 μs	1.35 μs	1.11	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*'

Details

System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Confirmed 11.36% regression in HashSet.Contains (z=10.24) caused by intentional memory barrier change in commit 9baed28. This is an expected performance cost of fixing memory ordering bug #120754 in CoreCLR's LookupMap implementation. The commit explicitly acknowledged the trade-off: "This is slightly slower but avoids the extremely easy to mess up dependency ordering rules."

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 9baed28

Related Issues: #null

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #68013 - [Perf] Linux/x64: 1 Regression on 1/24/2026 4:18:23 AM +00:00
• Date: 2026-01-24 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: Linux/x64
• Commits: b1d2e4a → 8fafcb3 (75 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, runtime-coreclr, kind-micro
• Historical Data: Available (179 prior runs, 10 runs after regression)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)	1209 ns	1346 ns	+11.36%	1180.68 ns	15.89 ns	179	10.24	5%	N	Step change, very stable history

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 5/5 - Excellent historical data (179 runs), very clear step change at exactly the regression date, low variance (CV=1.35%)

Confirmed Regressions

System.Collections.ContainsFalse<Int32>.HashSet(Size: 512) (+11.36%)

• Delta: 1209 ns → 1346 ns (+11.36%)
• Historical Context: μ=1180.68 ns (σ=15.89 ns, n=179); z-score=10.24; adaptive threshold=5%
• Assessment: Clear step change with extremely high statistical significance (z=10.24). The test jumped from historical baseline range of 1170-1216 ns to consistently elevated values of 1342-1347 ns across all 10 post-regression builds.
• Finding Confidence: 5/5 - Exceptional data quality with 179 historical runs, extremely stable prior performance (CV=1.35%), and exact temporal correlation with identified commit
• Regression Confidence: 5/5 - Surgical commit directly modifying lookup map access patterns, extreme z-score (10.24), persistent regression across all subsequent builds
• Likely Cause: 9baed28 - "Fix token memory ordering issue reading from the MethodDef and FieldDef token tables (Fix token memory ordering issue reading from the MethodDef and FieldDef token tables #123557)"
  • Affected Areas: CoreCLR metadata lookup infrastructure (LookupMap implementation)
  • Rationale:
    • This commit changed VolatileLoadWithoutBarrier to VolatileLoad in LookupMap<TYPE>::GetValueAt() to add explicit memory barriers
    • The commit specifically mentions "This is slightly slower but avoids the extremely easy to mess up dependency ordering rules"
    • Modified file: src/coreclr/vm/ceeload.inl which is used during type loading and method lookups
    • HashSet.Contains operations involve generic type lookups and method dispatch which use these lookup maps
    • The 11.36% regression magnitude aligns with the overhead of adding full memory barriers instead of relying on data dependencies
    • Temporal alignment is exact: commit timestamp (2026-01-24T04:18:23) matches regression date precisely
    • The fix was intentional (trades performance for correctness to fix issue Random failure: System.MissingMethodException: Method not found: 'UInt32 System.Reflection.Metadata.Ecma335.HasConstantTag.ConvertToTag(System.Reflection.Metadata.EntityHandle)'. #120754)
  • Candidates Considered:
    • No other commits in the 75-commit range directly modified HashSet, collections internals, or generic lookup infrastructure
    • 687c8d9 - "Array.FindAll improvements" - affects Array.FindAll, not HashSet.Contains
    • 4b0b723 - "Add binary VN funcs to TryGetRangeFromAssertions" - JIT assertion propagation, unlikely to affect micro benchmarks
• Verification Status: Not verified - high confidence from statistical analysis and direct code path correlation

Open Questions

• Expected vs Unintended: This regression appears to be an expected cost of the correctness fix in Fix token memory ordering issue reading from the MethodDef and FieldDef token tables #123557. The commit message explicitly acknowledges "This is slightly slower" when switching from VolatileLoadWithoutBarrier to VolatileLoad.
• Trade-off Analysis: The fix addresses a real concurrency bug (Random failure: System.MissingMethodException: Method not found: 'UInt32 System.Reflection.Metadata.Ecma335.HasConstantTag.ConvertToTag(System.Reflection.Metadata.EntityHandle)'. #120754) where memory ordering could cause race conditions in type/method/field lookups. The 11.36% regression in HashSet.Contains likely reflects broader impact on all generic operations that perform lookups.
• Scope Assessment: Need to determine if:
  1. This affects only HashSet.Contains or broader collection/generic operations
  2. The performance cost is acceptable given the correctness fix
  3. There's room for optimization (e.g., more targeted barriers or alternative synchronization)

Recommended Actions

1. Accept as Known Cost - Document this as an expected regression from correctness fix Fix token memory ordering issue reading from the MethodDef and FieldDef token tables #123557. The ~140ns slowdown in HashSet.Contains (11.36%) is the cost of proper memory ordering guarantees.
2. Broader Impact Assessment - Run comparative analysis on other generic collection benchmarks (Dictionary, List, ImmutableCollections) to understand full scope of performance impact from the memory barrier change.
3. Optimization Investigation - If the performance cost is deemed too high, investigate whether:
   • More selective barrier placement is possible (e.g., only on specific lookup map types)
   • Alternative synchronization mechanisms could reduce overhead
   • Hardware-specific memory ordering features could be leveraged
4. Documentation - Add note to dotnet/runtime tracking that VolatileLoad change in LookupMap has known ~10-12% impact on generic lookup-heavy operations.

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 68013,
  "baselineCommit": "b1d2e4aa8af89cfa795015e30ef70d24ec92f67f",
  "compareCommit": "8fafcb3251a3d2ccde36612a2eab47e9ae8ebe2b",
  "commitRangeSize": 75,
  "historicalContext": {
    "runsExamined": 179,
    "missing": false,
    "notes": "Excellent data quality: 179 runs before regression, 10 runs after. Very stable historical performance (CV=1.35%). Clear step change at regression date."
  },
  "regressions": [
    {
      "test": "System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)",
      "baseline": 1209,
      "compare": 1346,
      "percentDelta": 11.36,
      "histMean": 1180.68,
      "histStdDev": 15.89,
      "zScore": 10.24,
      "isNoise": false,
      "likelyCommit": "9baed28a0526f4c164ba1043edeeb8718df7aabb",
      "findingConfidence": 5,
      "findingConfidenceRationale": "Exceptional historical data (179 runs), exact temporal match with identified commit, extremely low variance (CV=1.35%)",
      "regressionConfidence": 5,
      "regressionConfidenceRationale": "Surgical commit to LookupMap with explicit performance trade-off acknowledged, extreme z-score (10.24), perfect correlation",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Accept as expected cost of correctness fix #123557",
    "Assess broader impact on generic collection benchmarks",
    "Consider optimization opportunities if performance cost is unacceptable"
  ],
  "generationNotes": "This regression is an expected side effect of a correctness fix for memory ordering issues in type/method/field lookups. The commit message explicitly acknowledged the performance trade-off."
}

---

Generated by PerfTriageAgent on 2026-01-27 14:45 UTC

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Single HashSet.Contains regression (+11.4%, z=9.87) attributed to commit 9baed28 which replaced VolatileLoadWithoutBarrier with VolatileLoad in type/method descriptor lookups. The memory barrier overhead is consistent with the observed performance degradation. This change was necessary to fix correctness issue #120754 (data race), so the performance cost may be unavoidable.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 4/5 (likelihood of true regression)

Likely Cause: 9baed28a

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #68013 - [Perf] Linux/x64: 1 Regression on 1/24/2026 4:18:23 AM +00:00
• Date: 2026-01-24T04:18:23 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
• Commits: b1d2e4aa → 8fafcb32 (75 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro
• Historical Data: Available (179 prior runs)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.ContainsFalse.HashSet(Size: 512)	1.21 μs	1.35 μs	+11.4%	1.18 μs	16.5 ns	179	9.87	5%	N	Step change, very stable

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 5/5 - Excellent historical data coverage (179 runs), very low variance (CV=1.40%), clear temporal separation between baseline and regression periods

Confirmed Regressions

System.Collections.ContainsFalse.HashSet(Size: 512) (+11.4%)

• Delta: 1.21 μs → 1.35 μs (+11.4%)
• Historical Context: μ=1.18 μs (σ=16.5 ns, n=179); z-score=9.87; adaptive threshold=5%
• Assessment: Clear step change - historically very stable test (CV=1.40%) showing consistent ~140-165 ns increase post-regression
• Finding Confidence: 5/5 - Excellent data quality, clear step change, very high z-score
• Regression Confidence: 4/5 - Strong signal with high z-score, surgical commit identified, but large commit range (75 commits) prevents maximum confidence
• Likely Cause: 9baed28a - "Fix token memory ordering issue reading from the MethodDef and FieldDef token tables"
  • Affected Areas: CoreCLR runtime - type/method loading, memory ordering
  • Rationale:
    • This commit (Fix token memory ordering issue reading from the MethodDef and FieldDef token tables #123557) changed critical memory ordering semantics in type/method descriptor lookups by replacing VolatileLoadWithoutBarrier with VolatileLoad (full memory barrier)
    • The regression affects a HashSet operation, which requires type initialization and method dispatch - both paths that would execute the modified lookup code
    • The memory barrier change adds overhead to every type/method lookup, which would accumulate in collection operations
    • Timing aligns perfectly: commit landed 2026-01-24T04:18:23, regression detected same timestamp
    • The ~11.4% overhead is consistent with the cost of full memory barriers replacing dependency-based ordering in hot paths
  • Candidates Considered:
    • 687c8d9d - "Array.FindAll improvements" - Unlikely, affects Array.FindAll not HashSet.Contains
    • 614b6872 - "Use existing Base64 helpers" - Unrelated to collections
    • 8fafcb32 - "Enable runtime async for all libraries test projects" - Test infrastructure only, should not affect runtime perf
• Verification Status: Not yet verified - baseline and compare commits span 75 commits

Open Questions

• Commit Range: Large range (75 commits) increases uncertainty, though commit 9baed28 is highly likely culprit
• Verification Needed: Bisection recommended to isolate between commits b1d2e4a and 8fafcb3 to confirm 9baed28 as root cause
• Performance Impact: Need to assess whether the memory ordering fix (Fix token memory ordering issue reading from the MethodDef and FieldDef token tables #123557) is necessary for correctness - if so, the performance cost may be unavoidable
• Broader Impact: Should check if other collection/type-loading benchmarks show similar regressions

Recommended Actions

1. Bisection: Run bisection on commits b1d2e4aa...8fafcb32 using RunWithBuildAtHash to confirm commit 9baed28 as the culprit
2. Code Review: Manual review of commit 9baed28a focusing on whether the VolatileLoad overhead is necessary for correctness vs. could be optimized
3. Broader Testing: Verify if other benchmarks involving type initialization, method dispatch, or collection operations show similar regressions
4. Perf Impact Assessment: If the memory ordering fix is required for correctness (issue Random failure: System.MissingMethodException: Method not found: 'UInt32 System.Reflection.Metadata.Ecma335.HasConstantTag.ConvertToTag(System.Reflection.Metadata.EntityHandle)'. #120754 was about data races), document the expected ~11% overhead in collection operations as a known cost of the fix

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 68013,
  "baselineCommit": "b1d2e4aa8af89cfa795015e30ef70d24ec92f67f",
  "compareCommit": "8fafcb3251a3d2ccde36612a2eab47e9ae8ebe2b",
  "commitRangeSize": 75,
  "historicalContext": {
    "runsExamined": 179,
    "missing": false,
    "notes": "Excellent historical data: 179 runs, very stable (CV=1.40%)"
  },
  "regressions": [
    {
      "test": "System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)",
      "baseline": 1.21,
      "compare": 1.35,
      "percentDelta": 11.36,
      "histMean": 1.18,
      "histStdDev": 0.0165,
      "zScore": 9.87,
      "isNoise": false,
      "likelyCommit": "9baed28a0526f4c164ba1043edeeb8718df7aabb",
      "findingConfidence": 5,
      "findingConfidenceRationale": "Excellent data quality (179 runs, CV=1.40%), clear step change, very high z-score (9.87)",
      "regressionConfidence": 4,
      "regressionConfidenceRationale": "Strong signal (z=9.87, step change), surgical commit modifying memory ordering in type/method lookups, but large commit range (75) prevents maximum confidence",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Bisect commits b1d2e4aa..8fafcb32 to confirm 9baed28a",
    "Review 9baed28a to assess if VolatileLoad overhead is unavoidable",
    "Check other collection/type-loading benchmarks for similar regressions"
  ],
  "generationNotes": "Single test regression with very high confidence. Likely cause identified as memory ordering fix in commit 9baed28a which replaced VolatileLoadWithoutBarrier with VolatileLoad in type/method lookup paths. The ~11% overhead is consistent with memory barrier costs in hot paths."
}

---

Generated by PerfTriageAgent on 2026-01-27 14:48 UTC

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Confirmed regression in HashSet.Contains (+11.4%, z=9.87) with excellent historical data (179 runs, CV=1.40%). Likely caused by SpanHelpers refactoring affecting JIT codegen. Bisection recommended across 75-commit range to definitively identify the culprit.

Finding Confidence: 4/5 (analysis accuracy)
Regression Confidence: 3/5 (likelihood of true regression)

Likely Cause: b90cbdebcffb

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #68013 - [Perf] Linux/x64: 1 Regression on 1/24/2026 4:18:23 AM +00:00
• Date: 2026-01-24 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: Ubuntu 22.04/x64
• Commits: b1d2e4aa → 8fafcb32 (75 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro
• Historical Data: Available (179 prior runs)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)	1.21 μs	1.35 μs	+11.4%	1.18 μs	16.5 ns	179	9.87	5%	N	Step change, very stable

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Excellent historical data coverage (179 runs) with very low variance (CV=1.40%), clear temporal separation between baseline and regression periods. No verification performed yet.

Confirmed Regressions

System.Collections.ContainsFalse<Int32>.HashSet(Size: 512) (+11.4%)

• Delta: 1.21 μs → 1.35 μs (+11.4%)
• Historical Context: μ=1.18 μs (σ=16.5 ns, n=179); z-score=9.87; adaptive threshold=5%
• Assessment: Step change. The test was extremely stable historically (CV=1.40%) with values clustering around 1171-1211 ns before jumping to 1343-1347 ns on 2026-01-24. This represents a clear, reproducible regression with very high statistical significance (z=9.87).
• Finding Confidence: 4/5 - Excellent data quality, comprehensive historical baseline, clear step-change pattern. Would be 5/5 with verification via RunWithBuildAtHash.
• Regression Confidence: 3/5 - Strong statistical signal (z=9.87), but large commit range (75 commits) with no immediately obvious culprit in the HashSet code path. Multiple JIT-related commits could be responsible.
• Likely Cause: b90cbdeb - "Undo a few JIT layout workarounds (Undo a few JIT layout workarounds #114605)"
  • Affected Areas: JIT / Codegen / Collections
  • Rationale: This commit modified SpanHelpers.T.cs (835 additions, 455 deletions), which is heavily used in collection operations. The removal of "goto" workarounds and code layout changes could affect how HashSet.Contains is JIT-compiled, potentially impacting inlining decisions, register allocation, or branch prediction. The timing aligns with the regression window (committed 2026-01-22, regression detected 2026-01-24). HashSet.Contains operations likely rely on SpanHelpers for internal array traversal and comparisons.
  • Candidates Considered:
    • b90cbdeb - SpanHelpers layout changes affecting collection codegen
    • 66c2aa52 - JIT devirtualization changes (could affect HashSet virtual calls)
    • ccf995ee - AllocHeap optimization (could affect NativeAOT, less likely for micro benchmarks)
• Verification Status: Not verified

Open Questions

• Verification Needed: Bisection recommended to narrow down the 75-commit range. Run RunWithBuildAtHash with baseline [b1d2e4a] vs suspected commits [b90cbde, 66c2aa5] to confirm which commit introduced the regression.
• Code Path Analysis: Need to examine JIT disassembly to determine if SpanHelpers changes affected HashSet.Contains inlining or code layout.
• Confidence Limitations: Large commit range prevents definitive attribution without verification. Multiple JIT-related commits in the range could be responsible.

Recommended Actions

1. Bisection: Run bisection on commits b1d2e4aa...8fafcb32 using RunWithBuildAtHash to isolate the culprit among the 75 commits. Start by testing the two most likely candidates: [b90cbde] (SpanHelpers) and [66c2aa5] (JIT devirtualization).
2. JIT Analysis: Collect JIT disassembly for HashSet.Contains at baseline vs compare commits to identify codegen differences. Use DOTNET_JitDisasm or examine the ETL files if available.
3. Verification: Once the culprit commit is identified, verify the regression is reproducible with RunWithBuildAtHash comparing the commit before and after the change.
4. Root Cause Investigation: If [b90cbde] is confirmed, investigate whether the SpanHelpers refactoring unintentionally affected HashSet's internal array operations through changed inlining or loop unrolling behavior.

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 68013,
  "baselineCommit": "b1d2e4aa8af89cfa795015e30ef70d24ec92f67f",
  "compareCommit": "8fafcb3251a3d2ccde36612a2eab47e9ae8ebe2b",
  "commitRangeSize": 75,
  "historicalContext": {
    "runsExamined": 179,
    "missing": false,
    "notes": "Excellent historical data with 179 prior runs. Test was very stable (CV=1.40%) before the regression."
  },
  "regressions": [
    {
      "test": "System.Collections.ContainsFalse<Int32>.HashSet(Size: 512)",
      "baseline": 1.2091058932401837,
      "compare": 1.3464054884983012,
      "percentDelta": 11.36,
      "histMean": 1180.77,
      "histStdDev": 16.48,
      "zScore": 9.87,
      "isNoise": false,
      "likelyCommit": "b90cbdebcffb",
      "findingConfidence": 4,
      "findingConfidenceRationale": "Excellent historical data (179 runs), very low variance (CV=1.40%), clear step-change pattern. Not verified yet.",
      "regressionConfidence": 3,
      "regressionConfidenceRationale": "Strong statistical signal (z=9.87) and stable history, but large commit range (75 commits) prevents definitive attribution without bisection.",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Bisect 75-commit range to isolate culprit",
    "Compare JIT disassembly for HashSet.Contains between baseline and compare",
    "Verify regression with RunWithBuildAtHash on suspected commits"
  ],
  "generationNotes": "High-confidence regression detection (z=9.87) with excellent historical data. Large commit range requires bisection for definitive attribution. Primary suspect is SpanHelpers refactoring that may have affected HashSet codegen."
}

---

Generated by PerfTriageAgent on 2026-01-27 14:53 UTC

[DrewScoggins]

Likely, #123557