Fix WASM boot config ContentRoot to use IntermediateOutputPath#124125
Fix WASM boot config ContentRoot to use IntermediateOutputPath#124125lewing wants to merge 4 commits intodotnet:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the Blazor WASM SDK targets to define the build boot config static web asset from its actual on-disk generation location (obj/) to prevent stale-asset Identity selection during incremental builds (which can lead to incorrect SRI integrity values at runtime).
Changes:
- Change
DefineStaticWebAssetsContentRootfor the build boot config asset from$(OutDir)wwwrootto$(IntermediateOutputPath).
Canceled AzDO jobs (typically from timeouts) still have pipeline artifacts containing binlogs. The SendToHelix.binlog contains Helix job IDs that can be queried directly to recover actual test results. Discovered while investigating PR #124125 where a 3-hour timeout caused a WasmBuildTests job to be canceled, but all 226 Helix work items had actually passed.
Canceled AzDO jobs (typically from timeouts) still have pipeline artifacts containing binlogs. The SendToHelix.binlog contains Helix job IDs that can be queried directly to recover actual test results. Discovered while investigating PR #124125 where a 3-hour timeout caused a WasmBuildTests job to be canceled, but all 226 Helix work items had actually passed.
Canceled WasmBuildTests actually passed ✅The browser-wasm windows Release WasmBuildTests job in build 1284169 was canceled after the 3-hour timeout, but all Helix work items completed successfully. Recovery steps:
The failure was purely the AzDO job wrapper timing out while waiting to collect results, not an actual test failure. |
Canceled AzDO jobs (typically from timeouts) still have pipeline artifacts containing binlogs. The \SendToHelix.binlog\ contains Helix job IDs that can be queried directly to recover actual test results. ## What changed - Added **Recovering Results from Canceled Jobs** section to helix skill SKILL.md - Documents the workflow: download artifacts → load binlog → extract Helix job IDs → query Helix API - Includes concrete example from PR #124125 (226 work items all passed despite 3h timeout cancellation) - Added tip about binlog MCP server for structured binlog analysis ## Context Discovered while investigating #124125 where \�rowser-wasm windows Release WasmBuildTests\ was canceled after 3 hours. The script reported no failures (correct — no *failed* jobs) but also couldn't surface that the tests actually passed. With the documented recovery workflow, the Helix results were fully recoverable.
| CopyToOutputDirectory="PreserveNewest" | ||
| CopyToPublishDirectory="Never" | ||
| ContentRoot="$(OutDir)wwwroot" | ||
| ContentRoot="$(IntermediateOutputPath)" |
There was a problem hiding this comment.
I believe this needs to happen in other places like the definition of the .wasm files as assets (after webcil conversion).
I believe the important bit here is that ContentRoot + RelativePath must exist for all assets
There was a problem hiding this comment.
Done. Added a commit that sets per-item ContentRoot metadata on each _WebCilAssetsCandidates item to %(RootDir)%(Directory) (its own directory). This ensures candidateFullPath.StartsWith(normalizedContentRoot) is always true in DefineStaticWebAssets.ComputeCandidateIdentity, so Identity equals the physical file path for all candidates — webcil-converted files in obj/webcil/, runtime pack files in packs/, PDBs from obj/, etc.
The task-level ContentRoot parameter is kept as a fallback but the per-item metadata takes precedence via ComputePropertyValue.
|
Bad base commit? |
b649bd9 to
dc5c351
Compare
yup, and then github stopped responding |
## Summary Work around [dotnet/dnceng#6072](dotnet/dnceng#6072) in the CI analysis skill by using the Helix \ListFiles\ endpoint instead of file URIs from the \Details\ endpoint. ## Problem The Helix work item Details API (\GET .../workitems/{id}\) has two bugs in its \Files[].Uri\ values: 1. **Subdirectory flattening**: Files uploaded from subdirectories (e.g. \xharness-output/logs/test/build.binlog\) get URIs with only the base filename (\.../files/build.binlog\), so multiple distinct files with the same base name collide 2. **Unicode rejection**: The \/files/{name}\ endpoint validates against an ASCII-only regex, rejecting filenames containing unicode characters (e.g. CJK test directory names like \鿀蜒枛遫䡫煉\) The previous workaround reconstructed permalink URIs from the \FileName\ field, but this still routed through the broken \/files/\ endpoint which rejects unicode. ## Fix Use the separate \ListFiles\ endpoint (\GET .../workitems/{id}/files\) which returns direct blob storage URIs. These URIs: - Preserve full subdirectory paths - Have properly percent-encoded unicode - Don't route through the permalink/regex validation layer Verified against PR #124125 CI artifacts: 53 files with unicode paths all return HTTP 200 via ListFiles blob URIs, vs broken/colliding URIs from the Details endpoint.
Change the boot config's DefineStaticWebAssets ContentRoot from wwwroot to so the asset Identity points to the actual file location on disk rather than a stale copy in the output wwwroot folder. This fixes SRI integrity failures during incremental Blazor WASM builds where the compressed boot config used a stale fingerprint from the wwwroot copy instead of the fresh file in obj/. Fixes aspnetcore#65271
Replace the fragile FileName-based scanning of all StaticWebAsset items with direct references to _WasmBuildBootConfigStaticWebAsset and _WasmPublishBootConfigStaticWebAsset. These items are already produced by the DefineStaticWebAssets task in _GenerateBuildWasmBootJson and _AddPublishWasmBootJsonToStaticWebAssets respectively. The previous approach relied on FileName containing the fingerprint (e.g. dotnet.FINGERPRINT.js), which is an implementation detail of how DefineStaticWebAssets computes Identity. Using the pipeline's own output items is correct regardless of ContentRoot or Identity format.
WebCil candidates come from multiple source directories (obj/webcil/ for converted DLLs, runtime pack for native files, etc.) but were all defined with ContentRoot pointing to OutputPath/wwwroot. Use IntermediateOutputPath as ContentRoot instead. Files outside obj/ (e.g., runtime pack native files) get synthesized Identities with CopyCandidates to materialize the fingerprinted copies. Files already under obj/ (webcil-converted DLLs) use their physical path directly. This avoids the stale wwwroot Identity problem that caused SRI integrity failures during incremental builds, while maintaining correct publish behavior through CopyCandidate materialization.
dc5c351 to
2e74845
Compare
…on guides (#124240) ## ci-analysis: structured output, MCP integration, and deep investigation guides ### Changes to `Get-CIStatus.ps1` (+216/-132) - **Add `[CI_ANALYSIS_SUMMARY]` JSON block** — structured summary emitted at end of script with all key facts (builds, failed jobs, known issues, PR correlation, recommendation hint) - **Replace 47-line if/elseif recommendation chain** with a single `recommendationHint` field in JSON (one of: `BUILD_SUCCESSFUL`, `KNOWN_ISSUES_DETECTED`, `LIKELY_PR_RELATED`, `POSSIBLY_TRANSIENT`, `REVIEW_REQUIRED`, `MERGE_CONFLICTS`, `NO_BUILDS`) - **Add `failedJobDetails` to JSON** — per-job `errorCategory` (test-failure, build-error, test-timeout, crash, tests-passed-reporter-failed, unclassified), `errorSnippet`, and `helixWorkItems` - **Add `failedJobDetailsTruncated`** — boolean flag indicating when `-MaxJobs` cap means `failedJobDetails` is incomplete vs `failedJobNames` - **Add top-level `knownIssues`** from Build Analysis (not per-job — Build Analysis reports at the PR level, not per-job) - **Add timeout pattern** to `Format-TestFailure` — catches `Timed Out (timeout` that was previously invisible - **Show log tail in PR mode** when no failure pattern matches (Helix Job mode already did this) - **Add accumulation variables** for cross-build aggregation (`totalFailedJobs`, `totalLocalFailures`, `lastBuildJobSummary`) - **Fix early-continue scoping bug** — job summary computation was at end of build loop, after 3 `continue` paths that skipped it - **Fix empty array falsy check** — `if ($listFiles)` → proper count check - **Fix `mergeable_state` trimming** — `gh api --jq` output trimmed to prevent whitespace comparison failures - **Remove interpretive prose** — "These failures are likely PR-related" moved from script to agent reasoning - **Fix empty catch** — merge state error now logged via `Write-Verbose` ### Changes to `SKILL.md` (+97/-144, net reduction) - **Add Step 0: Gather Context** — PR type classification table (code, flow, backport, merge, dependency update) - **Add Step 3: Verify before claiming** — systematic checklist - **Add build progression analysis** (Step 2, item 4) — comparing pass/fail across PR builds to narrow down which commit introduced a failure - **Add prior-build mismatch detection** (Step 2, item 6) — ask user when they reference jobs not in current results - **Document `failedJobDetails`** — per-failure error categories in Interpreting Results - **Add Build Analysis check status enforcement** — red check means unaccounted failures exist, never claim "all known" when it's red - **Add timeout recovery workflow** — explicit guidance for verifying timed-out builds have passing Helix results via `hlx_status` - **Add crash/canceled job recovery procedure** — step-by-step using `hlx_batch_status`, `hlx_files`, `hlx_download_url` to recover results from crashed Helix work items - **Fix MCP tool references** — use canonical short-form tool names consistently - **Condense anti-patterns** — tighter, more targeted, near relevant steps - **Net token reduction** — despite adding new content, SKILL.md shrank from ~4.6K to ~3.5K tokens ### New reference files - `references/azure-cli.md` — Azure CLI deep investigation guide - `references/binlog-comparison.md` — binlog comparison workflow - `references/delegation-patterns.md` — subagent delegation patterns (5 patterns including parallel artifact extraction and canceled job recovery) - `references/build-progression-analysis.md` — commit-to-build correlation using `triggerInfo.pr.sourceSha`, SQL-based progression tracking, MCP-first with AzDO MCP tools as primary ### Updated reference files - `references/manual-investigation.md` — fix nonexistent `msbuild-mcp analyze` tool refs, use real `mcp-binlog-tool-*` tools ### Design principles - **Data/reasoning boundary**: Script emits structured JSON facts → agent synthesizes recommendations. No more canned prose from the script. - **MCP-first**: AzDO MCP tools (`get_builds`, `get_build_log_by_id`) and Helix MCP tools (`hlx_status`, `hlx_logs`) positioned as primary, CLI/script as fallback. - **Token budget**: Orchestrating SKILL.md kept within 2K-4K token budget by extracting depth to `references/`. - **SQL for structured investigations**: Build progression tracking uses SQL tables to persist SHAs across context, enabling queries for pass→fail transitions and target branch movement. ### Testing - Multi-model subagent testing (Sonnet 4 + GPT-5 + Opus 4.5) — two review rounds with findings addressed - Live MCP integration test confirmed `hlx_status`, `hlx_logs`, `get_builds`, `get_build_log_by_id` all work - Real-world validation against PRs #123245, #123883, #124125, #124232
Replace task-level ContentRoot= with per-item
ContentRoot=%(RootDir)%(Directory) on build-time WebCil candidates, and
add per-item ContentRoot on publish-time candidates.
Per-item ContentRoot ensures DefineStaticWebAssets resolves each asset's
Identity to its actual file path on disk, preventing synthesized paths
for runtime pack files that live outside IntermediateOutputPath. At
publish time, promoted assets carry AssetKind=Build which means
contentRoot is not nulled (IsPublish('Build')=false), so the per-item
ContentRoot is used by ComputeCandidateIdentity to produce Identity=
real FullPath instead of a fingerprinted filename that doesn't exist.
| RemoveMetadata="Integrity;Fingerprint" /> | ||
| <!-- Set per-item ContentRoot so each asset's Identity matches its actual file on disk --> | ||
| <_NewWebCilPublishStaticWebAssetsCandidatesNoMetadata Update="@(_NewWebCilPublishStaticWebAssetsCandidatesNoMetadata)" ContentRoot="%(RootDir)%(Directory)" /> | ||
| <_PromotedWasmPublishStaticWebAssets Update="@(_PromotedWasmPublishStaticWebAssets)" ContentRoot="%(RootDir)%(Directory)" /> |
There was a problem hiding this comment.
_PromotedWasmPublishStaticWebAssets are produced by ComputeWasmPublishAssets, which explicitly sets ContentRoot to $(PublishDir)wwwroot for publish assets. Overwriting ContentRoot here to %(RootDir)%(Directory) will redirect publish asset identities/content roots back to their build-time paths (often under $(OutputPath)), which risks incorrect publish manifests/copy behavior and can break incremental publish. Consider not overriding ContentRoot for promoted publish assets (or only doing so conditionally when it is empty).
| <_PromotedWasmPublishStaticWebAssets Update="@(_PromotedWasmPublishStaticWebAssets)" ContentRoot="%(RootDir)%(Directory)" /> | |
| <_PromotedWasmPublishStaticWebAssets Update="@(_PromotedWasmPublishStaticWebAssets)" Condition="'%(ContentRoot)' == ''" ContentRoot="%(RootDir)%(Directory)" /> |
|
@javiercn I've been doing deep binlog analysis of the CI failures (builds 4-6) and want to make sure I'm heading in the right direction. What the latest commit does: Sets per-item Why the previous approaches failed:
My concern: You said "ContentRoot + RelativePath must exist for all assets." Per-item ContentRoot fixes Identity (Copy source) but doesn't satisfy that invariant for runtime pack files — because RelativePath includes The Options I see:
Which direction would you prefer? |
Summary
Fix a .NET 11 regression causing SRI integrity failures during incremental Blazor WASM builds. Changes in
Microsoft.NET.Sdk.WebAssembly.Browser.targets:DefineStaticWebAssetsContentRootfrom$(OutDir)wwwrootto$(IntermediateOutputPath)%(FileName)%(Extension)-based scanning with direct references to the boot config output itemsContentRoot="%(RootDir)%(Directory)"on_WebCilAssetsCandidatesso each asset's Identity resolves to its actual file path on diskContentRoot="%(RootDir)%(Directory)"on both_NewWebCilPublishStaticWebAssetsCandidatesNoMetadataand_PromotedWasmPublishStaticWebAssets— the same fix applied to publish candidatesRegression
This is a regression in .NET 11 (works in 10.0). It was introduced by dotnet/sdk#52283, which fixed an esproj compression bug by flipping the order in
AssetToCompress.TryFindInputFilePathto preferRelatedAsset(Identity) overRelatedAssetOriginalItemSpec. That fix was correct for esproj, but exposed a latent issue in the WASM SDK targets: the boot config and webcil assets' Identity pointed to awwwrootcopy rather than the actual source files.Before sdk#52283,
OriginalItemSpechappened to point to the real file and was checked first, masking the wrongContentRoot. After the flip,RelatedAsset(Identity) is checked first, and its stalewwwrootpath is used — producing incorrect SRI hashes on incremental builds.Reported in aspnetcore#65271.
Problem
The WASM boot config file (e.g.
dotnet.boot.js) is generated at$(IntermediateOutputPath)(theobj/folder), but its static web asset was defined withContentRoot="$(OutDir)wwwroot". This causedDefineStaticWebAssetsto compute an Identity pointing to thewwwrootcopy rather than the actual file inobj/.The same issue applied to WebCil asset candidates — files from
obj/webcil/, the runtime pack, and other directories were all defined withContentRoot="$(OutputPath)wwwroot", producing synthetic Identities underwwwroot/that could become stale during incremental builds.Fix
1. Boot config ContentRoot
Change
ContentRootto$(IntermediateOutputPath)so the asset Identity matches the real file location on disk. TheCopyToOutputDirectory="PreserveNewest"attribute still ensures the file is copied towwwrootfor serving.This follows Javier's suggestion in dotnet/sdk#52847 to "stop defining these assets with an item spec in the wwwroot folder and just define them in their original location on disk".
2. Preload matching simplification
The
_AddWasmPreloadBuildPropertiesand_AddWasmPreloadPublishPropertiestargets previously scanned all@(StaticWebAsset)items by%(FileName)%(Extension)to find the boot config asset. This relied on the Identity path containing the fingerprint in the filename, which is an implementation detail of howDefineStaticWebAssetscomputes Identity based onContentRoot.The fix replaces the scanning with direct references to
@(_WasmBuildBootConfigStaticWebAsset)and@(_WasmPublishBootConfigStaticWebAsset)— the output items already produced byDefineStaticWebAssets. This is both correct and simpler.3. WebCil per-item ContentRoot (build-time)
Instead of using a single task-level
ContentRootparameter (which forces all candidates through the same ContentRoot path, causing synthesized Identity for files outside that directory), set per-itemContentRoot="%(RootDir)%(Directory)"on each_WebCilAssetsCandidatesitem. This means:obj/webcil/→ ContentRoot = their parent dir →FullPath.StartsWith(ContentRoot)= true → Identity = real FullPath ✅dotnet.native.js, ICU.dat) → ContentRoot = their parent dir in the runtime pack →FullPath.StartsWith(ContentRoot)= true → Identity = real FullPath ✅No synthesized paths, no CopyCandidate entries — every asset's Identity is its actual file on disk.
4. WebCil per-item ContentRoot (publish-time)
The publish
DefineStaticWebAssetscall inProcessPublishFilesForWasmpreviously had no ContentRoot at all — neither task-level nor per-item. This caused publish candidates (especially promoted build assets with fingerprint placeholders in their RelativePath) to have their fingerprinted filename baked into the item spec as Identity, producing paths likedotnet.native.7z98fd2ohl.wasmthat don't exist on disk → MSB3030 "Could not copy file".The fix adds per-item
ContentRoot="%(RootDir)%(Directory)"on both:_NewWebCilPublishStaticWebAssetsCandidatesNoMetadata(freshly WebCil-converted publish files)_PromotedWasmPublishStaticWebAssets(build assets promoted to publish)Why this works: Promoted assets carry
AssetKind=Buildfrom the build-timeDefineStaticWebAssets. InDefineStaticWebAssets.csline 252:IsPublish("Build") = false, so contentRoot is NOT nulled for publish. The per-item ContentRoot = each file's parent directory →candidateFullPath.StartsWith(contentRoot)= true →computed=false→ Identity = real FullPath on disk.What's not changed
ContentRoot="$(PublishDir)wwwroot"): Publish builds are clean and don't have the incremental staleness problem.Build Progression
Fixes dotnet/aspnetcore#65271