[Megathread] Infrastructure Issues

[nagilson]

We have a wide range of infrastructure issues, and it has gotten to the point that we often have to rerun CI on PRs several times before they succeed due to transient failures. At this point it has become a significant pain-point (evident here #28317 (comment)) for the team and it's likely worth having the domestic-cat rotation individual look into one of these on rotation. I encourage all of you to add by editing this post any issues you encounter. They are separated into things that are actionable for our team or other contributors who flow their code into our repo, and things which are actionable for dnc-eng.

Closed issues

#29008 #29006 #28802 #28801 #27766 #27658 #29289

Open issues:
Global property flow race conditions: https://github.com/dotnet/sdk/pull/29168/checks?check_run_id=9644380039

Issue:

error]src\Layout\redist\targets\sdks\sdks.csproj(0,0): error NU1301: (NETCORE_ENGINEERING_TELEMETRY=Build) Failed to retrieve information about 'Microsoft.NET.Sdk.WindowsDesktop' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/a65e5cb4-26c0-410f-9457-06db3c5254be/nuget/v3/flat2/microsoft.net.sdk.windowsdesktop/index.json'.

Child Issue: N/A

Example Run PR + Pipeline: https://github.com/dotnet/sdk/runs/8712332231

Estimated Impact: 1

Issue:

[error]ShellJSInternalError: ENOSPC: no space left on device, write
Child Issue: N/A
Example Run PR + Pipeline: [Pipelines - Run 20220921.69 logs (azure.com)](https://dev.azure.com/dnceng-public/public/_build/results?buildId=25573&view=logs&j=92885c4a-db2e-5086-f9ba-51524576e2ac&t=437333b1-f621-5ac8-6b31-945800dcd511&l=51)?
Estimated Impact: 4

Issue:

'D:\a\1\s\artifacts\bin\Release\Sdks\Microsoft.NET.Sdk.Razor\targets\Microsoft.NET.Sdk.Razor.StaticWebAssets.Pack.CrossTargeting.targets' because it is being used by another process. [TargetFramework=net472]

Child Issue: Does not yet exist.

Example Run PR + Pipeline: #28327, https://github.com/dotnet/sdk/runs/8756003552

Estimated Impact: 4

Issue:

Child Issue: Does not yet exist.
[Example Run PR + Pipeline: |
](https://dev.azure.com/dnceng-public/public/_build/results?buildId=272072&view=ms.vss-test-web.build-test-results-tab&runId=5373286&resultId=100222&paneView=debug)

Expected command to exit with 0 but it did not.\r\nFile Name: D:\a\_work\1\s\artifacts\bin\redist\Release\dotnet\dotnet.exe\r\nArguments: new console --no-restore --force --debug:custom-hive D:\a\_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SharedHomeDirectory\20230512164846363\r\nExit Code: 100\r\nStdOut:\r\n\r\nStdErr:\r\nTemplate "Console App" could not be created.\r\nFailed to create template.\r\nDetails: Error while processing file /content/ConsoleApplication-CSharp/Company.ConsoleApplication1.csproj\r\nThe process cannot access the file 'D:\a\_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\CanOverwriteFilesWithForce\20230512165119230\20230512165119230.csproj' because it is being used by another process.\r\n\r\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#100\r\n

Issue:

Child Issue: Does not yet exist.
Example Run PR + Pipeline: https://dev.azure.com/dnceng-public/public/_build/results?buildId=276879&view=results SkipsLocalizationOnInstantiate_WhenInvalidFormat SkipsLocalizationOnInstantiate_WhenLocalizationValidationFails Estimated Impact: unknown

Issue:

Child Issue: Does not yet exist.
Example Run PR + Pipeline: https://dev.azure.com/dnceng-public/public/_build/results?buildId=298136&view=ms.vss-test-web.build-test-results-tab&runId=6027854&resultId=102677&paneView=debug

Issue:

Child Issue: https://github.com//issues/33956 Example Run PR + Pipeline: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=336802 Estimated Impact: ?

Get "http://localhost:5010/v2/": read tcp 127.0.0.1:40054->127.0.0.1:5010: read: connection reset by peer

Issue:

Child Issue: CannotDisplayUnknownPackageDetails failing with unexpected error Example Run PR + Pipeline: https://dev.azure.com/dnceng/internal/_build/results?buildId=2249124&view=ms.vss-test-web.build-test-results-tab&runId=51948431&resultId=100087&paneView=debug
Estimated Impact: failing every internal PR

##issue:
#36089

DNC-Eng Related Issues:

Fixed:

• 429 Errors: Tracking Nuget 429s arcade#10885
• 503 Errors: 503's arcade#10943
  Open:

Issue Template

Issue:

Child Issue: Does not yet exist.
Example Run PR + Pipeline: |
Estimated Impact: [0-5]

[Forgind]

Issue with Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewLocaleTests.SkipsLocalizationOnInstantiate_WhenInvalidFormat:

The command output did not contain expected result: name_de-DE:äÄßöÖüÜ\r\n\r\nFile Name: D:\a\_work\1\s\artifacts\bin\redist\Release\dotnet\dotnet.exe\r\nArguments: new -i D:\a\_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885 --debug:custom-hive D:\a\_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\Home\20230512221848884\r\nExit Code: 0\r\nStdOut:\r\nWarnung: Die Verwendung von \"dotnet new --install\" ist veraltet. Verwenden Sie stattdessen \"dotnet new install\".\r\nF├╝hren Sie Folgendes aus, um weitere Informationen zu erhalten: \r\n dotnet new install -h\r\n\r\nDie folgenden Vorlagenpakete werden installiert:\r\n D:\a\_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885\r\n\r\nErfolg: D:\a\work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885 installierte die folgenden Vorlagen:\r\nVorlagenname Kurzname Sprache Tags \r\n------------------ ----------------------------------- ------- ----------\r\nname_de-DE:├ñ├ä├ƒ├╢├û├╝├£ TestAssets.TemplateWithLocalization [C#] Test Asset\r\nStdErr:\r\n\r\n
at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at Microsoft.NET.TestFramework.Assertions.CommandResultAssertions.HaveStdOutContaining(String pattern) in //src/Tests/Microsoft.NET.TestFramework/Assertions/CommandResultAssertions.cs:line 61
at Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewLocaleTests.SkipsLocalizationOnInstantiate_WhenInvalidFormat() in D:\a_work\1\s\src\Tests\dotnet-new.Tests\DotnetNewLocaleTests.cs:line 149
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Output:

D:\a_work\1\s\artifacts\bin\redist\Release\dotnet\dotnet.exe new -i D:\a_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885 --debug:custom-hive D:\a_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\Home\20230512221848884
Warnung: Die Verwendung von "dotnet new --install" ist veraltet. Verwenden Sie stattdessen "dotnet new install".
F├╝hren Sie Folgendes aus, um weitere Informationen zu erhalten:
dotnet new install -h

Die folgenden Vorlagenpakete werden installiert:
D:\a_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885

Erfolg: D:\a_work\1\s\artifacts\tmp\Release\dotnet-new.IntegrationTests\SkipsLocalizationOnInstantiate_WhenInvalidFormat\20230512221848885 installierte die folgenden Vorlagen:
Vorlagenname Kurzname Sprache Tags

---

name_de-DE:äÄßöÖüÜ TestAssets.TemplateWithLocalization [C#] Test Asset

[Forgind]

@nagilson, I don't know German, but it looks like the problem here ^ is that the output was encoded improperly...any flakiness in your change?

[baronfel]

Issue: Transient container builds tests

Cause: The container tests on Ubuntu push to a local container registry to mimic a real remote registry. This local container registry is run in Docker on the ubuntu host, and sometimes it does not start up as expected.

Presentation: Generally something about a connection problem to localhost:5010/v2. Something like:

Expected command to pass but it did not.
File Name: docker
Arguments: push localhost:5010/dotnet/runtime:6.0
Exit Code: 1
StdOut:
The push refers to repository [localhost:5010/dotnet/runtime]
StdErr:
Get "http://localhost:5010/v2/": EOF

Mitigation: Restart the tests, these issues usually are very transient and only impact the Ubuntu leg

[nagilson]

@nagilson, I don't know German, but it looks like the problem here ^ is that the output was encoded improperly...any flakiness in your change?

It's certainly possible and did initially impact tests (when it was put into the SDK.) It looks like its not MSBuild output so I don't think it's that change.

I believe template engine has some of its own code for encoding and it was slightly flaky even before my changes. If it was my change, the garbled characters shouldn't occur because it forced the encoding to be UTF-8 which shows no garbled characters regardless of the language. So this is the behavior we would expect to see if the newer change was not running. LMK if you want to investigate together

[Forgind]

the garbled characters shouldn't occur because it forced the encoding to be UTF-8 which shows no garbled characters

This is true-ish. UTF-8 can represent all characters regardless of language but doesn't necessarily represent characters properly, so if something expects œÂi☼↑, it'll generate some bits equivalent to that, and there's some (possibly identical) set of bits equivalent to that in UTF-8, but it's not necessarily the same bits. In this case, the characters aren't just random ? but real characters, just not the right characters, so I could easily imagine it being UTF-8 misinterpreting something that should be something else.

[nagilson]

Hm, I wouldn't expect that to happen either, but it's possible. The best information to have would be to store the output of running "chcp" before and after the test to see what encoding is being used when it happens next time.

[marcpopMSFT]

another instance of the flakiness that @Forgind recorded. https://dev.azure.com/dnceng-public/public/_build/results?buildId=276879&view=ms.vss-test-web.build-test-results-tab&runId=5490452&resultId=100005&paneView=debug

Reminder that we should update the description above rather than having to search replies.

[nagilson]

This is no longer being used and we have better approaches with bots now