Switch to Azure Linux queues #112024

richlander · 2025-01-31T00:45:01Z

Switch to Azure Linux 3 queues for an initial set of pipelines as a test.

Replay of Switch to Azure Linux VM #108967
The Azure Linux queues have since been updated

dotnet-policy-service · 2025-01-31T01:26:05Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

richlander · 2025-01-31T05:03:15Z

PTAL @vcsjones. Looks like there is a mismatch with Azure Linux being the VM host for some reason.

vcsjones · 2025-01-31T14:47:05Z

@richlander will take a look today. It's going to take me a bit to get my AZL test / dev environment working again, so answers might not come until Monday.

richlander · 2025-01-31T16:14:58Z

We should probably wait until this issue is resolved: dotnet/dnceng#4892

vcsjones · 2025-01-31T16:23:40Z

@richlander Looks like we found another edge case in SCOSSL. I filed issue #112036, and opened PR #112037 to disable the tests for now.

there is a mismatch with Azure Linux being the VM host for some reason.

I suspect the VM has a slightly newer version of Azure Linux than the containers. I would expect the containers to start failing if/when they are re-created that have the latest symcrypt-openssl and openssl-devel packages.

richlander · 2025-01-31T17:10:57Z

It appears that the newest OpenSSL is installed in our images. I don't think the VM has another source of packages. Are we missing packages in our helix images, perhaps?

$ docker run --rm -it mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64 bash -c "tdnf list --installed | grep ssl"
openssl.x86_64                               3.3.2-1.azl3              @System
openssl-libs.x86_64                          3.3.2-1.azl3              @System

These seem to be the total list available.

$ docker run --rm -it mcr.microsoft.com/azurelinux/base/core:3.0 bash -c "tdnf list | grep ssl"
openssl.x86_64                               3.3.2-1.azl3              @System
openssl-libs.x86_64                          3.3.2-1.azl3              @System
apr-util-openssl.x86_64                      1.6.3-2.azl3         azurelinux-official-base
mod_ssl.x86_64                               2.4.61-1.azl3        azurelinux-official-base
mod_ssl.x86_64                               2.4.62-1.azl3        azurelinux-official-base
openssl.x86_64                               3.3.0-2.azl3         azurelinux-official-base
openssl.x86_64                               3.3.2-1.azl3         azurelinux-official-base
openssl-devel.x86_64                         3.3.0-2.azl3         azurelinux-official-base
openssl-devel.x86_64                         3.3.2-1.azl3         azurelinux-official-base
openssl-libs.x86_64                          3.3.0-2.azl3         azurelinux-official-base
openssl-libs.x86_64                          3.3.2-1.azl3         azurelinux-official-base
openssl-perl.x86_64                          3.3.0-2.azl3         azurelinux-official-base
openssl-perl.x86_64                          3.3.2-1.azl3         azurelinux-official-base
openssl-pkcs11.x86_64                        0.4.12-1.azl3        azurelinux-official-base
openssl-static.x86_64                        3.3.0-2.azl3         azurelinux-official-base
openssl-static.x86_64

akoeplinger · 2025-01-31T17:32:23Z

eng/pipelines/coreclr/templates/helix-queues-setup.yml

@@ -98,9 +98,9 @@ jobs:
    # Linux x64
    - ${{ if eq(parameters.platform, 'linux_x64') }}:
      - ${{ if eq(variables['System.TeamProject'], 'public') }}:
-        - Ubuntu.2204.Amd64.Open
+        - azurelinux.3.amd64.open


should we add an Ubuntu container run so we don't lose coverage for that platform?

This is a great question, in two different ways.

We have Ubuntu coverage with Arm64. I think that's sufficient:

runtime/eng/pipelines/coreclr/templates/helix-queues-setup.yml

Lines 73 to 75 in 572533b

- (Ubuntu.2004.Arm64.Open)Ubuntu.2204.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-20.04-helix-arm64v8

- ${{ if eq(variables['System.TeamProject'], 'internal') }}:

- (Ubuntu.2004.Arm64)Ubuntu.2204.ArmArch@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-20.04-helix-arm64v8

.

However, I'm not sure why we use this raw VM pattern at all. We only do that because we can (matching arch). I think I should convert x64 to look the same as the other architectures (use containers). That would be cleaner and easier for people looking at this scheme for the first time. I know it confused me. In theory, we could then skip putting all extra dependencies on the Azure Linux VM.

Thoughts?

The main benefit of the raw VM pattern is that it removes one layer of complexity (containers), e.g. there are small differences between running in the container or not like cgroup constraints etc. I'm also not sure whether it might have infrastructure benefits i.e. workitems start faster when they don't need to fetch the container image before running.

Question is whether that's enough reason to keep the existing setup.

vcsjones · 2025-01-31T18:03:32Z

It appears that the newest OpenSSL is installed in our images.

Do the tests pass in our container images then? I think AZL3 recently took an update to OpenSSL 3.3 and that's when the functionality started not working.

richlander · 2025-01-31T18:08:56Z

CI is a bit broken right now. We should wait to see what results look like after it is back.

dougbu · 2025-01-31T20:20:29Z

We should probably wait until this issue is resolved: dotnet/dnceng#4892

resolution is at least temporarily going to be a rollout revert unfortunately. that'll break the azurelinux.3.amd64* queues again 😦

richlander · 2025-02-01T00:03:12Z

Closing until the queue we need is active again.

richlander · 2025-03-12T00:22:55Z

New VM images were deployed last week.

vcsjones · 2025-03-12T02:52:03Z

Yayyyyy cryptography got a clean run.

richlander · 2025-03-12T02:56:04Z

We are doing much better this time!

richlander · 2025-03-12T03:15:15Z

PTAL @hoyosjs -- looks like a diagnostic failure

vcsjones · 2025-03-12T03:24:19Z

Build analysis tagged it as a known failure (#110643) so likely is not AZL specific.

richlander · 2025-03-12T04:02:13Z

/ba-g "Known issue: #110643"

richlander · 2025-03-12T04:02:35Z

Sounds good. Just need an approval now.

richlander · 2025-03-12T15:46:41Z

@vcsjones do you expect that backporting this PR to release branches would run cleanly or there are changes that'd we'd need to backport? This is not a request for anything (other than information). My instinct is to make this a going forward change, but I wanted to better understand feasibility, particularly if I get asked.

vcsjones · 2025-03-12T16:18:20Z

would run cleanly or there are changes that'd we'd need to backport?

Maybe. I did have to disable one test to get a green run. #112037.

However, that change is not necessary when the fix from SCOSSL gets picked up in the images, which is over at microsoft/SymCrypt-OpenSSL#105

So, either

If the images include that SCOSSL fix, then I would expect you get a green run.
If the images do not include that SCOSSL fix, then back porting Disable Shake Read tests on AzureLinux #112037 would fix it.

Generally I would recommend waiting for 1, because 2 is a temporary work around, which would mean back porting a temporary work around, and eventually reverting it.

richlander · 2025-03-12T17:01:39Z

I'm not a fan of taking on extra work in release, so your guidance sound great.

Switch to Azure Linux queues

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

3e4f9cb

dotnet-issue-labeler bot added the needs-area-label label Jan 31, 2025

dotnet-policy-service bot assigned richlander Jan 31, 2025

richlander added area-Infrastructure and removed needs-area-label labels Jan 31, 2025

This was referenced Jan 31, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

EgorBo mentioned this pull request Feb 2, 2025

[Linux-arm64] ModuleNotFoundError: No module named 'requests' dotnet/dnceng#4892

Closed

6 tasks

akoeplinger reviewed Jan 31, 2025

View reviewed changes

richlander closed this Feb 1, 2025

github-actions bot locked and limited conversation to collaborators Mar 6, 2025

richlander reopened this Mar 12, 2025

Merge branch 'main' into azure-linux-vm

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

1a7b046

dotnet unlocked this conversation Mar 12, 2025

build-analysis bot mentioned this pull request Mar 12, 2025

System.Diagnostics.Process.Tests failure on net10.0-linux-Debug-arm64-Mono_Minijit_Debug #110643

Open

akoeplinger approved these changes Mar 12, 2025

View reviewed changes

richlander merged commit 82c916f into dotnet:main Mar 12, 2025
88 checks passed

richlander deleted the azure-linux-vm branch March 12, 2025 14:45

This was referenced Mar 12, 2025

Change build pipeline to use Azure Linux VM #113439

Open

Implement pinning for container images consumed in dotnet/runtime #113455

Open

jakobbotsch mentioned this pull request Mar 13, 2025

Test failure: JIT/Regression/CLR-x86-JIT/dev10/b400791/_b400971b400971/_b400971b400971.cmd #113457

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to Azure Linux queues #112024

Switch to Azure Linux queues #112024

richlander commented Jan 31, 2025 •

edited

Loading

dotnet-policy-service bot commented Jan 31, 2025

richlander commented Jan 31, 2025

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025 •

edited

Loading

akoeplinger Jan 31, 2025 •

edited

Loading

richlander Jan 31, 2025

akoeplinger Feb 2, 2025

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025

dougbu commented Jan 31, 2025

richlander commented Feb 1, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

	- (Ubuntu.2004.Arm64.Open)Ubuntu.2204.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-20.04-helix-arm64v8
	- ${{ if eq(variables['System.TeamProject'], 'internal') }}:
	- (Ubuntu.2004.Arm64)Ubuntu.2204.ArmArch@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-20.04-helix-arm64v8

Switch to Azure Linux queues #112024

Switch to Azure Linux queues #112024

Conversation

richlander commented Jan 31, 2025 • edited Loading

dotnet-policy-service bot commented Jan 31, 2025

richlander commented Jan 31, 2025

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025 • edited Loading

akoeplinger Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

richlander Jan 31, 2025

Choose a reason for hiding this comment

akoeplinger Feb 2, 2025

Choose a reason for hiding this comment

vcsjones commented Jan 31, 2025

richlander commented Jan 31, 2025

dougbu commented Jan 31, 2025

richlander commented Feb 1, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Mar 12, 2025

vcsjones commented Mar 12, 2025

richlander commented Mar 12, 2025

richlander commented Jan 31, 2025 •

edited

Loading

richlander commented Jan 31, 2025 •

edited

Loading

akoeplinger Jan 31, 2025 •

edited

Loading