The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
985590	dotnet/aspire	Build	Log
985565	dotnet/aspire	Build	Log
985535	dotnet/runtime	Build product	Log	dotnet/runtime#112937
2666961	dotnet-aspnetcore	Generate delegation SAS Token for dotnetbuilds/internal (azureLinux30Net8BuildAmd64)	Log	#48632
985455	dotnet/aspire	Run helix tests	Log
985436	dotnet/aspire	Run helix tests	Log
985405	dotnet/aspire	Run helix tests	Log
984989	dotnet/runtime	Send to Helix	Log
2666787	dotnet-roslyn	Test	Log
2666772	dotnet-dotnet-monitor	Build	Log
2666743	dotnet-dotnet-docker	🛡 Guardian: AntiMalware Scanner (Source)	Log
2666746	dotnet-performance	Send job to Helix (Unix)	Log
2666740	dotnet-roslyn	Test	Log
984806	dotnet/sdk	Prepare BuildLogs staging directory	Log	dotnet/sdk#47338
2666733	dotnet-dotnet	Validate Signing	Log
2666720	dotnet-roslyn	Test	Log
2666709	dotnet-roslyn	Test	Log
2666585	dotnet-sdk	Run AoT Tests	Log	#48610
984993	dotnet/roslyn	Run Integration Tests	Log
2666693	dotnet-roslyn	Test	Log
2666676	dotnet-runtime	Send job to Helix (Unix)	Log
2666667	dotnet-roslyn	Test	Log
2666663	dotnet-roslyn	Test	Log
2666632	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@dev/loganbussell/release-staging to s/dotnet-dotnet-docker	Log
2666641	dotnet-roslyn	Test	Log
2666637	dotnet-roslyn	Test	Log
2666605	dotnet-roslyn	Test	Log
984852	dotnet/roslyn	Test	Log	dotnet/roslyn#77659
2666583	dotnet-dotnet-monitor	Build	Log
2666584	dotnet-sdk	Run AoT Tests	Log	#48626
2666426	dotnet-dotnet	SDL Binary Analysis Checks (1ES PT)	Log
2666514	dotnet-sdk	Build	Log	#48610
2666541	dotnet-sdk	Run AoT Tests in Helix	Log	#48563
2666540	dotnet-sdk	Run AoT Tests in Helix	Log	#48564
2666542	dotnet-sdk	Run Tests in Helix	Log	#48565
2666536	dotnet-roslyn	Test	Log
2666429	dotnet-dotnet	SDL Binary Analysis Checks (1ES PT)	Log
2666401	dotnet-dotnet	Build	Log
984385	dotnet/runtime	Send to Helix	Log
983276	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
2666263	dotnet-runtime	Send job to Helix (Unix)	Log
2666044	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
2665946	dotnet-dotnet	Join Verticals	Log
2666055	dotnet-runtime	Send job to Helix (Unix)	Log
983910	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113633
983513	dotnet/runtime	Send tests to Helix (Unix)	Log
983892	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113632
983872	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113575
983861	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#112704
983810	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113242
2665813	dotnet-dotnet	Build	Log
2665919	dotnet-performance	Send job to Helix (Unix)	Log
2665853	dotnet-runtime	Send job to Helix (Unix)	Log
2665841	dotnet-dotnet-docker	Test Images	Log
983680	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#77133
2665831	dotnet-roslyn	Test	Log
2665829	dotnet-roslyn	Test	Log
2665757	dotnet-dotnet	Run Tests	Log
2665756	dotnet-dotnet	Run Tests	Log
983595	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#77644
983584	maestro-auth-test/build-result-analysis-test	Run Tests (Windows)	Log	maestro-auth-test/build-result-analysis-test#2039
2665778	dotnet-roslyn	Test	Log
2665771	dotnet-roslyn	Test	Log
2665770	dotnet-roslyn	Test	Log
2665582	dotnet-dotnet	🔒 Publish Artifacts (On Failure)	Log
2665583	dotnet-dotnet	🔒 Publish Artifacts (On Failure)	Log
2665732	dotnet-sdk	Run Tests in Helix	Log	#48565
2665652	dotnet-dotnet-buildtools-prereqs-docker	Build Images (Authenticated)	Log
2665602	dotnet-runtime	Send job to Helix (Unix)	Log
2665568	dotnet-arcade-validation	PowerShell@2	Log
983068	dotnet/xharness	Run tests in Helix	Log
982956	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113483
982984	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1380
982915	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113481
982810	dotnet/runtime	Send to Helix	Log
982817	dotnet/runtime	Send to Helix	Log
982807	dotnet/runtime	Send tests to Helix (Unix)	Log
982758	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113313
2665360	dotnet-runtime	Send job to Helix (Unix)	Log
2665089	dotnet-runtime	Send job to Helix (Unix)	Log
2664982	dotnet-wcf	OneLocBuild	Log
2664914	dotnet-runtime	Send job to Helix (Unix)	Log
982174	dotnet/runtime	AOT compile CoreCLR tests	Log
2664810	dotnet-runtime	Component Detection	Log
2664853	dotnet-sdk	Run AoT Tests in Helix	Log	#48508
982138	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#77619
2664800	dotnet-runtime	Send job to Helix (Unix)	Log
982086	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113564
982070	dotnet/runtime	Send to Helix	Log
982065	dotnet/runtime	Send to Helix	Log
982060	dotnet/runtime	Send to Helix	Log
982067	dotnet/runtime	Send to Helix	Log
982034	dotnet/runtime	AOT compile CoreCLR tests	Log
2664522	dotnet-sdk	Run AoT Tests in Helix	Log	#48510
2664521	dotnet-sdk	Run AoT Tests in Helix	Log	#48508
2664523	dotnet-sdk	Run AoT Tests in Helix	Log	#48509
981872	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113576
981835	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#111004
2664376	dotnet-runtime	Send job to Helix (Unix)	Log
981795	dotnet/runtime	Send tests to Helix (Windows)	Log
Displaying 100 of 1122 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
42	308	1053

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)