Skip to content

Conversation

@agocke
Copy link
Member

@agocke agocke commented Dec 10, 2025

(cherry picked from commit eaafd7c) (cherry picked from commit 1905046)

Copilot AI review requested due to automatic review settings December 10, 2025 23:56
@github-actions github-actions bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Dec 10, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the default macOS build images from macOS 13 to macOS 15 (with Xcode 16) for public builds, and adds test exclusions for Swift interop stress tests that are incompatible with the new environment.

  • Updates public macOS build pool to use 'macos-15' image
  • Adds documentation comment explaining that build platforms should always use the latest available version
  • Excludes Swift ABI stress tests for mono minijit x64 configuration due to known issues on the new platform

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
eng/pipelines/common/xplat-setup.yml Updates public macOS pool from 'macos-13' to 'macos-15' and adds policy comment about using latest platform versions
src/tests/issues.targets Adds test exclusions for Swift interop stress tests (SwiftRetAbiStress and SwiftCallbackAbiStress) for mono minijit x64 configuration, referencing issue #121983

@agocke agocke changed the title Move default build images to macos 15 (and xcode 16) (#120589) [release/8.0] Move default build images to macos 15 (and xcode 16) (#120589) Dec 11, 2025
@agocke agocke added the Servicing-approved Approved for servicing release label Dec 11, 2025
@agocke
Copy link
Member Author

agocke commented Dec 11, 2025

@steveisok it looks like mono build is failing. Any insight?

@steveisok
Copy link
Member

@steveisok it looks like mono build is failing. Any insight?

Taking a look.

@steveisok steveisok requested review from lewing and removed request for lambdageek December 11, 2025 20:26
lambdageek
lambdageek previously approved these changes Dec 11, 2025
Copy link
Member

@lambdageek lambdageek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is cherrypicking from c48aa81

I don't remember the subtleties anymore, but this seems right

@lambdageek lambdageek dismissed their stale review December 11, 2025 22:16

not quite there yet

@lambdageek
Copy link
Member

lambdageek commented Dec 11, 2025

@steveisok I think you also need c48aa81#diff-b2942ad0f35c702862be561fa03e56f885e6b362b4b5522a21e23cc4d9e444ba the aot-compiler.c changes

@lambdageek
Copy link
Member

lambdageek commented Dec 11, 2025

On main this line

MONO_EMIT_NEW_LCOMPARE_IMM (cfg, ins->sreg1, (-(int)2147483648));

has INT_MIN

MONO_EMIT_NEW_LCOMPARE_IMM (cfg, ins->sreg1, INT_MIN);

fixed here:

c85492f

@steveisok
Copy link
Member

fixed here:

c85492f

Thanks!

@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

@akoeplinger
Copy link
Member

The crashes we see on Build Libraries Test Run release coreclr osx x64 Debug look related, we're seeing them on #122422 as well

@agocke
Copy link
Member Author

agocke commented Dec 12, 2025

Yeah, it looks like Roslyn is somehow entering infinite recursion. @AaronRobinsonMSFT @davidwrighton any immediate ideas based on the stack traces? The only change here is updating the mac build baseline.

@akoeplinger
Copy link
Member

akoeplinger commented Dec 16, 2025

@agocke given it is just one job should we disable the affected test projects and file an issue to unblock the PR(s)?

@agocke
Copy link
Member Author

agocke commented Dec 16, 2025

@akoeplinger It's not one job, it's dozens of tests in 5 different test projects.

@akoeplinger
Copy link
Member

akoeplinger commented Dec 16, 2025

ah sorrry yeah, I just looked at the GitHub status and it doesn't show the full list of failing workitems there

it's still one AzDO job though, so we could also disable the whole job...

@agocke
Copy link
Member Author

agocke commented Jan 6, 2026

@davidwrighton @AaronRobinsonMSFT bumping after the break

@agocke
Copy link
Member Author

agocke commented Jan 7, 2026

Looking at a different log file, I think this is potentially an issue with EH. @jkotas @janvorli does this stack trace look familiar?

Stack overflow.
Repeat 21 times:
--------------------------------
   at System.Reflection.RuntimeModule.get_RuntimeType()
   at System.RuntimeType+RuntimeTypeCache..ctor(System.RuntimeType)
   at System.RuntimeType.InitializeCache()
   at System.RuntimeType.get_Cache()
   at System.RuntimeType.GetCachedName(System.TypeNameKind)
   at System.RuntimeType.ToString()
   at System.Runtime.CompilerServices.DefaultInterpolatedStringHandler.AppendFormatted[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon)
   at System.Runtime.InteropServices.ExternalException.ToString()
--------------------------------
   at Xunit.Sdk.AssertEqualityComparer`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Collections.IEqualityComparer)

It looks like it may only repro on Mac x64

@agocke
Copy link
Member Author

agocke commented Jan 7, 2026

Note that the managed stack before the exception looks different in a lot of cases, e.g.

Stack overflow.
Repeat 20 times:
--------------------------------
   at System.Reflection.RuntimeModule.get_RuntimeType()
   at System.RuntimeType+RuntimeTypeCache..ctor(System.RuntimeType)
   at System.RuntimeType.InitializeCache()
   at System.RuntimeType.get_Cache()
   at System.RuntimeType.GetCachedName(System.TypeNameKind)
   at System.RuntimeType.ToString()
   at System.Runtime.CompilerServices.DefaultInterpolatedStringHandler.AppendFormatted[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon)
   at System.Runtime.InteropServices.ExternalException.ToString()
--------------------------------
   at System.Xml.Linq.XAttribute..ctor(System.Xml.Linq.XName, System.Object)
   at Xunit.DelegatingXmlCreationSink.CreateTestResultElement(Xunit.Abstractions.ITestResultMessage, System.String)

@agocke
Copy link
Member Author

agocke commented Jan 7, 2026

From David: this smells like some sort of corruption, not debuggable from a trace. We'll need a dump and probably access to one of the Helix machines + a live repro.

@janvorli
Copy link
Member

janvorli commented Jan 7, 2026

@janvorli does this stack trace look familiar?

@agocke no, it does not. Can you please share a link to the log file from which you've copied that trace? I can try to repro it locally.

@janvorli
Copy link
Member

janvorli commented Jan 9, 2026

I can repro the issue locally executing the bits from the failed run pulled by runfo. And yet it works perfectly fine if I build it locally using exactly the same XCode tools version, repeating exactly the command lines that CI uses to build the stuff on my arm64 mac running the whole build under rosetta so that it matches build on x64 mac.
So it seems to indicate something is wrong with the build machines.
It is hard to reason about the problem using the build from the CI as the libcoreclr.dylib is missing symbols and there is no libcoreclr.dylib.dwarf file.

@janvorli
Copy link
Member

janvorli commented Jan 9, 2026

Update: I can repro it with local build now, I haven't realized libraries tests on macOS use release build of the runtime.

agocke and others added 7 commits January 9, 2026 15:52
@steveisok
Copy link
Member

Looks like there is a consistent crash in System.Net.Http.Functional.Tests

@steveisok
Copy link
Member

I believe we are running into #97966 as a result of the bump.

@dotnet/ncl can you please confirm? I'd like to move this forward as 8.0 servicing is blocked until we can get this PR in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-Infrastructure Servicing-approved Approved for servicing release

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

6 participants