restarted. Azure DevOps can't recover from restarts. #3879

lewing · 2024-08-26T19:12:06Z

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
877992	dotnet/runtime	browser-wasm linux Release LibraryTests
877722	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	Log	dotnet/runtime#109363
877737	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#110114
877182	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109996
876967	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109493
876526	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110079
876314	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#100729
876035	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109891
875979	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109298
875455	dotnet/runtime	browser-wasm linux Release LibraryTests
875443	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109801
875365	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109996
875155	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109302
875223	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110031
875118	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109891
874957	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
874925	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110001
874871	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109302
874592	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109638
874462	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109232
874361	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110001
871580	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	Log	dotnet/runtime#109895
873836	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107230
873809	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109832
873800	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109763
873781	dotnet/runtime	browser-wasm linux Release LibraryTests
873528	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109137
873428	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107118
873272	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109957
873245	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations		dotnet/runtime#109900
873220	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#105303
873212	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109932
871972	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#108178
873041	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109232
873030	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
872677	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109932
872671	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109137
872662	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109936
872656	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109861
872620	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109901
872372	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109928
871779	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109901
871762	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109900
871592	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109861
871520	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109673
871513	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109889
871482	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109540
871361	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109861
871165	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109450
871091	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109873
871026	dotnet/runtime	browser-wasm linux Release LibraryTests
870972	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109808
870954	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109866
870865	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109638
870841	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109852
870807	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109861
870755	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109832
870529	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109838
870441	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109137
870419	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109849
870390	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109814
869504	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109806
869434	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109766
870278	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
870187	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109832
870136	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104637
870124	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109840
870111	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109838
870074	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
869959	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#107118
869886	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109540
869849	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109447
869833	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109320
869790	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109819
869781	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109814
869750	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109818
869660	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109638
869149	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109363
869615	dotnet/runtime	browser-wasm linux Release LibraryTests
869464	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109638
869412	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109731
869235	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109801
869174	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109766
869162	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109799
868813	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#109363
868738	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109776
868709	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109772
868643	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109769
859527	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#108731
868616	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109731
868592	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109765
868540	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading		dotnet/runtime#99596
868525	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109766
868500	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109765
868432	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109731
868381	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109756
868368	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109758
868311	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#104955
867811	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109684
868061	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#109744
Displaying 100 of 226 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	41	226

The text was updated successfully, but these errors were encountered:

am11 · 2024-10-07T05:54:23Z

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

am11 · 2024-10-07T12:57:20Z

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

steveisok · 2024-10-07T23:32:59Z

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

ericstj · 2024-10-10T17:47:41Z

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

snakex64 · 2024-10-30T15:07:32Z

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?

lewing added First Responder Known Build Error labels Aug 26, 2024

guihkx mentioned this issue Aug 26, 2024

"Error: 'NoneType' object is not subscriptable" on Channel.from_directory DevilXD/TwitchDropsMiner#547

Closed

jkotas mentioned this issue Aug 26, 2024

Fix alloc-dealloc mismatch dotnet/runtime#106957

Merged

github-staff deleted a comment from masooddahmedd Sep 10, 2024

Maoni0 mentioned this issue Sep 16, 2024

instrumentation (mostly) for DATAS dotnet/runtime#107853

Merged

lewing mentioned this issue Sep 17, 2024

Fix RegexOptions.Compiled|IgnoreCase perf when dynamic code isn't supported dotnet/runtime#107874

Merged

CarnaViire mentioned this issue Sep 18, 2024

WebSocket Feedback Follow-up dotnet/runtime#107662

Merged

am11 mentioned this issue Oct 7, 2024

Disable linux-x64 dev-innerloop dotnet/runtime#108581

Merged

am11 mentioned this issue Oct 7, 2024

Move more helpers from DomainAssembly to Assembly dotnet/runtime#108568

Merged

This was referenced Oct 7, 2024

[main] Update dependencies from 7 repositories dotnet/runtime#108385

Merged

[main] Update dependencies from dotnet/arcade dotnet/runtime#108323

Merged

[main] Update dependencies from dotnet/source-build-reference-packages dotnet/runtime#108593

Merged

am11 mentioned this issue Oct 8, 2024

[RISC-V] Fix some minor errors dotnet/runtime#107179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

snakex64 commented Oct 30, 2024 •

edited

Loading

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

Comments

lewing commented Aug 26, 2024 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Known issue core information

Release Note Category

Release Note Description

Additional information about the issue reported

Known issue validation

Report

Summary

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

snakex64 commented Oct 30, 2024 • edited Loading

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading

snakex64 commented Oct 30, 2024 •

edited

Loading