outerloop legs failing to detect dotnet

[carlossanlop]

Original issue was opened in dotnet/arcade: dotnet/arcade#11185

Many outerloop legs fail with this message when attempting to run unit tests:

./RunTests.sh: line 168: /tmp/helix/working/A6B808E3/p/dotnet: No such file or directory

The investigation found that this may have been caused by a runtime change that happened between these two runs (from 8/2 to 8/3):

pr/public/dotnet/runtime/refs/pull/73300/merge
pr/public/dotnet/runtime/refs/pull/73020/merge

Kusto Query

Jobs
| where QueueName == "osx.1200.amd64.open"
| extend Propz = parse_json(Properties)
| where Propz["DefinitionName"] == "runtime-libraries-coreclr outerloop"
| where Propz["System.PhaseName"] == "libraries_build_OSX_x64_Debug"
| order by Finished desc 

The main suspect PR is one that seeked to remove the host packages conditionally and was merged at a time between the above two PRs likely merge commits: https://github.com/dotnet/runtime/pull/73095/files

We need to confirm this issue caused the regression and get it fixed.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Original issue was opened in dotnet/arcade: dotnet/arcade#11185

Many outerloop legs fail with this message when attempting to run unit tests:

./RunTests.sh: line 168: /tmp/helix/working/A6B808E3/p/dotnet: No such file or directory

The investigation found that this may have been caused by a runtime change that happened between these two runs (from 8/2 to 8/3):

pr/public/dotnet/runtime/refs/pull/73300/merge
pr/public/dotnet/runtime/refs/pull/73020/merge

Kusto Query

Jobs
| where QueueName == "osx.1200.amd64.open"
| extend Propz = parse_json(Properties)
| where Propz["DefinitionName"] == "runtime-libraries-coreclr outerloop"
| where Propz["System.PhaseName"] == "libraries_build_OSX_x64_Debug"
| order by Finished desc 

The main suspect PR is one that seeked to remove the host packages conditionally and was merged at a time between the above two PRs likely merge commits: https://github.com/dotnet/runtime/pull/73095/files

We need to confirm this issue caused the regression and get it fixed.

Author:	carlossanlop
Assignees:	-
Labels:	area-Infrastructure
Milestone:	-

[elinor-fung]

#73095 made it so that the host.native subset needs to be built for the libraries testhost to be complete. It was added to the default clr/mono subsets, but it looks like the build isn't relying on those.

This is the portion of the change that handled it in PR/innerloop test runs:

runtime/eng/pipelines/libraries/run-test-job.yml

Lines 91 to 93 in cb6e5e7

	- ${{ if ne(parameters.liveRuntimeBuildConfig, '') }}:
	- script: $(_buildScript)
	-subset host.native+libs.pretest

It seems outerloop doesn't go through that. My guess would be that host.native is not being built, so host components (like dotnet) are missing from testhost. Whichever part of the outerloop pipeline is building libs.pretest probably needs to include host.native first.

[hoyosjs]

There was a PR I had a while back: #74468. Viktor said he wanted to do this differently, probably merging outer and inner loop as you point out.

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Original issue was opened in dotnet/arcade: dotnet/arcade#11185

Many outerloop legs fail with this message when attempting to run unit tests:

./RunTests.sh: line 168: /tmp/helix/working/A6B808E3/p/dotnet: No such file or directory

The investigation found that this may have been caused by a runtime change that happened between these two runs (from 8/2 to 8/3):

pr/public/dotnet/runtime/refs/pull/73300/merge
pr/public/dotnet/runtime/refs/pull/73020/merge

Kusto Query

Jobs
| where QueueName == "osx.1200.amd64.open"
| extend Propz = parse_json(Properties)
| where Propz["DefinitionName"] == "runtime-libraries-coreclr outerloop"
| where Propz["System.PhaseName"] == "libraries_build_OSX_x64_Debug"
| order by Finished desc 

The main suspect PR is one that seeked to remove the host packages conditionally and was merged at a time between the above two PRs likely merge commits: https://github.com/dotnet/runtime/pull/73095/files

We need to confirm this issue caused the regression and get it fixed.

Author:	carlossanlop
Assignees:	-
Labels:	area-Infrastructure-libraries, area-Infrastructure, untriaged
Milestone:	-

[danmoseley]

fwiw the badssl readme says "badssl.com is meant for manual testing of security UI in web clients."

[antonfirsov]

Because of this, outerloop is completely unreliable now. Note that only about 12% of the cases are passing. This is very unfortunate eg. for the networking libs, where a significant portion of the tests live in outerloop, and we prefer to run them on every PR touching relevant functionality.

[ViktorHofer]

I disabled the outerloop pipelines months ago and they weren't supposed to be running at all (because they are fundamentally broken). Presumably they were unintentionally re-enabled when the migration to dnceng-public.visualstudio.com happened.

I also just noticed that the outerloop mono pipeline doesn't exist anymore, it was probably lost during the migration.

I'm attempting to fix the broken outerloop pipeline in #79652 but I can't give any guarantees as most of the related code was written by someone else.