[main] Update dependencies from 9 repositories #84635

…otnet-optimization build 20230406.7 optimization.linux-arm64.MIBC.Runtime , optimization.linux-x64.MIBC.Runtime , optimization.windows_nt-arm64.MIBC.Runtime , optimization.windows_nt-x64.MIBC.Runtime , optimization.windows_nt-x86.MIBC.Runtime , optimization.PGO.CoreCLR From Version 1.0.0-prerelease.23175.4 -> To Version 1.0.0-prerelease.23206.7

…ild 20230412.6 Microsoft.DotNet.HotReload.Utils.Generator.BuildTool From Version 1.1.0-alpha.0.23179.3 -> To Version 8.0.0-alpha.0.23212.6

…otnet-optimization build 20230411.4 optimization.linux-arm64.MIBC.Runtime , optimization.linux-x64.MIBC.Runtime , optimization.windows_nt-arm64.MIBC.Runtime , optimization.windows_nt-x64.MIBC.Runtime , optimization.windows_nt-x86.MIBC.Runtime , optimization.PGO.CoreCLR From Version 1.0.0-prerelease.23175.4 -> To Version 1.0.0-prerelease.23211.4

Microsoft.NETCore.Runtime.ICU.Transport From Version 8.0.0-preview.4.23203.1 -> To Version 8.0.0-preview.4.23212.1

…30412.1 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 1.0.0-prerelease.23178.2 -> To Version 1.0.0-prerelease.23212.1

…otnet-optimization build 20230412.3 optimization.linux-arm64.MIBC.Runtime , optimization.linux-x64.MIBC.Runtime , optimization.windows_nt-arm64.MIBC.Runtime , optimization.windows_nt-x64.MIBC.Runtime , optimization.windows_nt-x86.MIBC.Runtime , optimization.PGO.CoreCLR From Version 1.0.0-prerelease.23175.4 -> To Version 1.0.0-prerelease.23212.3

pavelsavara · 2023-04-14T06:13:36Z

[21:14:22] fail: Error: ERR24: Unexpected error: System.ArgumentOutOfRangeException: The added or subtracted value results in an un-representable DateTime. (Parameter 'months')
    at li (http://127.0.0.1:49238/dotnet.js:3:142848)
    at m.javaScriptExports.call_entry_point (http://127.0.0.1:49238/dotnet.js:3:203696)
    at Object.Rn [as runMain] (http://127.0.0.1:49238/dotnet.js:3:98952)
    at run (http://127.0.0.1:49238/test-main.js:379:50)

pavelsavara · 2023-04-14T08:04:58Z

Process terminated. Encountered infinite recursion while looking up resource 'Word_At' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.
   at System.Environment.FailFast(System.String)
   at System.SR.InternalGetResourceString(System.String)
   at System.SR.GetResourceString(System.String)
   at System.Diagnostics.StackTrace.ToString(TraceFormat, System.Text.StringBuilder)
   at System.Diagnostics.StackTrace.ToString(TraceFormat)
   at System.Diagnostics.DebugProvider.Fail(System.String, System.String)
   at System.Diagnostics.Debug.Fail(System.String, System.String)

steveisok · 2023-04-14T14:30:41Z

/azp run runtime-ioslike

azure-pipelines · 2023-04-14T14:30:52Z

Azure Pipelines successfully started running 1 pipeline(s).

lewing · 2023-04-14T15:48:21Z

@steveisok should we split the icu flow out and let the rest complete?

lewing · 2023-04-21T19:25:06Z

should we roll back the dotnet-optimization update here to unblock the other flow?

AndyAyersMS · 2023-04-21T19:50:16Z

Taking a look now...

Tests seem to be calling FailFast from managed code, still tracking down why.

Looks like we hit an assert in one of these two methods:

runtime/src/libraries/System.Private.CoreLib/src/System/SpanHelpers.Packed.cs

Lines 787 to 807 in 281c462

    
           [MethodImpl(MethodImplOptions.AggressiveInlining)] 
        
           private static Vector256<byte> PackSources(Vector256<short> source0, Vector256<short> source1) 
        
           { 
        
               Debug.Assert(Avx2.IsSupported); 
        
               // Pack two vectors of characters into bytes. While the type is Vector256<short>, these are really UInt16 characters. 
        
               // X86: Downcast every character using saturation. 
        
               // - Values <= 32767 result in min(value, 255). 
        
               // - Values  > 32767 result in 0. Because of this we can't accept needles that contain 0. 
        
               return Avx2.PackUnsignedSaturate(source0, source1).AsByte(); 
        
           } 
        
           [MethodImpl(MethodImplOptions.AggressiveInlining)] 
        
           private static Vector128<byte> PackSources(Vector128<short> source0, Vector128<short> source1) 
        
           { 
        
               Debug.Assert(Sse2.IsSupported); 
        
               // Pack two vectors of characters into bytes. While the type is Vector128<short>, these are really UInt16 characters. 
        
               // X86: Downcast every character using saturation. 
        
               // - Values <= 32767 result in min(value, 255). 
        
               // - Values  > 32767 result in 0. Because of this we can't accept needles that contain 0. 
        
               return Sse2.PackUnsignedSaturate(source0, source1).AsByte(); 
        
           }

and are unable to handle an assert happening quite this early (the code gets reinvoked as part of assert handling).

So possibly we're calling a version we should not be calling? Will keep digging, but progress is slow; WinDBG keeps disconnecting unexpectedly.

… 20230421.2 runtime.linux-arm64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.linux-arm64.Microsoft.NETCore.Runtime.ObjWriter , runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.ObjWriter , runtime.linux-musl-x64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.linux-musl-x64.Microsoft.NETCore.Runtime.ObjWriter , runtime.linux-x64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter , runtime.osx-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.osx-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.osx-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.osx-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-arm64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.win-arm64.Microsoft.NETCore.Runtime.ObjWriter , runtime.win-x64.Microsoft.NETCore.Runtime.JIT.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.ObjWriter From Version 14.0.0-alpha.1.23179.1 -> To Version 14.0.0-alpha.1.23221.2

AndyAyersMS · 2023-04-21T21:04:31Z

So possibly we're calling a version we should not be calling? Will keep digging, but progress is slow; WinDBG keeps disconnecting unexpectedly.

FWIW I can repro running on a checked build, with no DOTNET env vars set. Same setup is fine with a latest main build. So has nothing to do directly with PGO or OSR. (using the Runtime_77968 test case, which is very simple).

Just a guess at this point -- I see the number of methods being jitted drop considerably, so perhaps:

the new pgo data causes more methods to be prejitted
(perhaps) some of these run afoul of the somewhat subtle HW dependence issues in prejitting (IIRC methods must check their own HW dependences explicitly and can't rely on checks done by callers, and it seems like the new PackedSpan stuff referenced above may not do this). @davidwrighton do you recall where we wrote up these rules?
but if so, seemingly this would cause widespread failures since these methods are active at app startup -- so I can't yet explain why it just seems to impact a handful of tests like it does.

davidwrighton · 2023-04-21T21:15:15Z

Yes, those functions appear to be violating the special rules for System.Private.CoreLib. The rule for writing code in there is that each and every function MUST behave identically whether or not the particular intrinsics are enabled. This means that you can't safely write a function which assumes that Avx2 is enabled in System.Private.CoreLib. See the special rules in https://github.com/dotnet/runtime/blob/main/docs/design/coreclr/botr/vectors-and-intrinsics.md

davidwrighton · 2023-04-21T21:16:07Z

@AndyAyersMS The problem is that failing to follow the rules will only cause problems if tiering occurs in a problematic order.

tannergooding · 2023-04-21T21:34:13Z

@davidwrighton, the guidance needs to be updated a bit, as for crossgen 2: https://github.com/dotnet/runtime/blob/main/docs/design/coreclr/botr/vectors-and-intrinsics.md#code-review-rules-for-use-of-platform-intrinsics-1

Any use of a platform intrinsic in the codebase SHOULD be wrapped with a call to the associated IsSupported property. This wrapping may be done within the same function that uses the hardware intrinsic, but this is not required as long as the programmer can control all entrypoints to a function that uses the hardware intrinsic.

Edit Nevermind, I missed the subsequent:

Since System.Private.CoreLib.dll is known to be code reviewed with the code review rules as written above for crossgen1 with System.Private.CoreLib.dll, it is possible to relax rule "Code which attempts to use instruction sets outside of the optimistic set will generate code that will not be used on machines with support for the instruction set." What this will do is allow the generation of non-optimal code for these situations, but through the magic of code review, the generated logic will still work correctly.

davidwrighton · 2023-04-21T21:37:21Z

@tannergooding Note that that comment applies to usage of intrinsics OUTSIDE of System.Private.CoreLib. System.Private.CoreLib also needs to follow the rules in https://github.com/dotnet/runtime/blob/main/docs/design/coreclr/botr/vectors-and-intrinsics.md#crossgen1-model-of-hardware-intrinsic-usage

davidwrighton · 2023-04-21T21:38:38Z

@tannergooding Yes, the wording is unfortunately confusing. I'd happily approve a PR which removed comments about crossgen1 and simply talked about special rules for System.Private.CoreLib.

tannergooding · 2023-04-21T21:42:47Z

Note that that comment applies to usage of intrinsics OUTSIDE of System.Private.CoreLib. System.Private.CoreLib also needs to follow the rules in

It would be beneficial if the doc explicitly reiterated that corelib needs to follow the rules of crossgen1. Right now its somewhat inferred, but its very easy to miss or misinterpret things.

That being said, I wonder in general if the rules for corelib are the right thing here overall. We have the need and desire to factor out things into helpers that are only for one platform. This overall helps with maintainability and often JIT throughput/perf. Having these different rules for corelib makes it a lot easier to accidentally break things and much harder to write "clean code".

lewing · 2023-04-21T22:16:19Z

So wrt to this pr, it sounds like the correct thing to do is remove the dotnet-optimization flow from it to unblock the other flow? If that is the case @sbomer already has a separate pr with the optimization flow that is triggering the existing problem I'll also mark the dotnet-optimization darc subscription as non-batchable until the underlying issue is worked out.

AndyAyersMS · 2023-04-21T22:42:18Z

So wrt to this pr, it sounds like the correct thing to do is remove the dotnet-optimization flow from it to unblock the other flow? If that is the case @sbomer already has a separate pr with the optimization flow that is triggering the existing problem I'll also mark the dotnet-optimization darc subscription as non-batchable until the underlying issue is worked out.

Yes, let's do that. We can sort those issues out separately.

tannergooding · 2023-04-21T22:43:34Z

We can sort those issues out separately.

👍, I imagine we have several of these that have snuck in.

AndyAyersMS · 2023-04-21T22:48:26Z

We can sort those issues out separately.

👍, I imagine we have several of these that have snuck in.

@BruceForstall and were wondering if there could be some special test mode that might uncover these issues more readily, especially if there is no simple way to check for them directly. Say we have a crossgen2 of SPC that crossgens everything it can, and then some runtime mode that randomly decides to use prejitted code or not -- running that we'd get a much more varied mixtures of prejitted and jitted code.

lewing · 2023-04-23T19:21:33Z

I've disabled dontet-optimization batching and the new flow is #85193

davidwrighton · 2023-04-24T18:46:05Z

@AndyAyersMS, @BruceForstall, @tannergooding Sigh. Yes the rules in System.Private.CoreLib are a problem. Unfortunately, we derive significant startup performance wins from these special rules, but I'll spend some time this week thinking about if we can make writing code .... sensible again in S.P.C When we only had a small amount of intrinsics usage this was reasonable, but we've been increasing their usage, and testing for this is impractical.

davidwrighton · 2023-04-24T21:04:40Z

Ok, it looks like we can build an analyzer based off of the logic of the PlatformCompatibilityAnalyzer, to find all the problematic code, and we could build a pattern with a slightly modified variant of the BypassReadyToRun attribute that would allow a bail-out for helper methods so that we can follow good engineering practices. Given the impact on the build today, I'm taking this as my current high priority item.

sbomer · 2023-04-24T21:58:27Z

For a quick solution, we may not even need to write a new analyzer - it looks like you can make the existing platform analyzer do roughly what we want with a few tweaks: https://gist.github.com/sbomer/4bfb41750259d60810aeb0b92bcc0279.

Unblocks dotnet-optimization flow and #84711. Includes a dotnet-optimization update to confirm this it avoids the issue discussed in #84635. This fixes that issue by preventing R2R for any Avx2 helpers which assume they are called with Avx2 enabled.

dotnet-issue-labeler bot added the area-codeflow for labeling automated codeflow label Apr 11, 2023

Update dependencies from https://github.com/dotnet/hotreload-utils bu…

4b11a50

…ild 20230412.6 Microsoft.DotNet.HotReload.Utils.Generator.BuildTool From Version 1.1.0-alpha.0.23179.3 -> To Version 8.0.0-alpha.0.23212.6

dotnet-maestro bot changed the title ~~[main] Update dependencies from dnceng/internal/dotnet-optimization~~ [main] Update dependencies from dnceng/internal/dotnet-optimization dotnet/hotreload-utils Apr 12, 2023

dotnet-maestro bot changed the title ~~[main] Update dependencies from dnceng/internal/dotnet-optimization dotnet/hotreload-utils~~ [main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization Apr 12, 2023

Update dependencies from https://github.com/dotnet/icu build 20230412.1

100e52b

Microsoft.NETCore.Runtime.ICU.Transport From Version 8.0.0-preview.4.23203.1 -> To Version 8.0.0-preview.4.23212.1

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization~~ [main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu Apr 12, 2023

Use -Wno-deprecated-declarations on wasi and wasm.

9478e9e

steveisok requested a review from marek-safar as a code owner April 12, 2023 20:05

Steve Pfister and others added 3 commits April 12, 2023 17:05

Add HAVE_UCOL_CLONE check to mono for ICU

eec167a

configure.cmake not working for wasm, bake it into mono cmake

bb1a31e

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu~~ [main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness Apr 13, 2023

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness~~ [main] Update dependencies from dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness dotnet/hotreload-utils Apr 13, 2023

dotnet-maestro bot changed the title ~~[main] Update dependencies from dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness dotnet/hotreload-utils~~ [main] Update dependencies from dotnet/icu dotnet/xharness dotnet/hotreload-utils dnceng/internal/dotnet-optimization Apr 13, 2023

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/icu dotnet/xharness dotnet/hotreload-utils dnceng/internal/dotnet-optimization~~ [main] Update dependencies from dotnet/xharness dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu Apr 13, 2023

Bump icudt symbol

48b201d

steveisok requested review from lewing, pavelsavara and kg as code owners April 13, 2023 19:58

build-analysis bot mentioned this pull request Apr 13, 2023

Tracking issue for CI build timeouts #76454

Closed

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/xharness dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu~~ [main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness Apr 14, 2023

dotnet-maestro bot changed the title ~~[main] Update dependencies from dotnet/hotreload-utils dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness~~ [main] Update dependencies from dnceng/internal/dotnet-optimization dotnet/icu dotnet/xharness dotnet/hotreload-utils Apr 14, 2023

danmoseley mentioned this pull request Apr 21, 2023

Introduction to vectorization with Vector128 and Vector256 #84115

Merged

Roll back dotnet-optimization updates

933e8f1

lewing approved these changes Apr 22, 2023

View reviewed changes

stephentoub merged commit a3e0d47 into main Apr 22, 2023

stephentoub deleted the darc-main-df7736a2-e675-492c-ba93-267f741558dc branch April 22, 2023 12:18

lewing mentioned this pull request Apr 23, 2023

[main] Update dependencies from dnceng/internal/dotnet-optimization #85193

Closed

runfoapp bot mentioned this pull request Apr 24, 2023

Infra improvements for Helix #68176

Closed

sbomer mentioned this pull request Apr 25, 2023

Fix Avx2 helpers #85275

Merged

ghost locked as resolved and limited conversation to collaborators May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[main] Update dependencies from 9 repositories #84635

[main] Update dependencies from 9 repositories #84635

dotnet-maestro bot commented Apr 11, 2023 •

edited

Loading

pavelsavara commented Apr 14, 2023

pavelsavara commented Apr 14, 2023

steveisok commented Apr 14, 2023

azure-pipelines bot commented Apr 14, 2023

lewing commented Apr 14, 2023

lewing commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023 •

edited

Loading

AndyAyersMS commented Apr 21, 2023 •

edited

Loading

davidwrighton commented Apr 21, 2023

davidwrighton commented Apr 21, 2023

tannergooding commented Apr 21, 2023 •

edited

Loading

davidwrighton commented Apr 21, 2023

davidwrighton commented Apr 21, 2023

tannergooding commented Apr 21, 2023

lewing commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023

tannergooding commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023

lewing commented Apr 23, 2023 •

edited

Loading

davidwrighton commented Apr 24, 2023

davidwrighton commented Apr 24, 2023

sbomer commented Apr 24, 2023

[main] Update dependencies from 9 repositories #84635

[main] Update dependencies from 9 repositories #84635

Conversation

dotnet-maestro bot commented Apr 11, 2023 • edited Loading

From https://dev.azure.com/dnceng/internal/_git/dotnet-optimization

From https://github.com/dotnet/hotreload-utils

From https://github.com/dotnet/xharness

From https://github.com/dotnet/emsdk

From https://github.com/dotnet/runtime

From https://github.com/dotnet/llvm-project

From https://github.com/dotnet/runtime-assets

From https://github.com/dotnet/cecil

pavelsavara commented Apr 14, 2023

pavelsavara commented Apr 14, 2023

steveisok commented Apr 14, 2023

azure-pipelines bot commented Apr 14, 2023

lewing commented Apr 14, 2023

lewing commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023 • edited Loading

AndyAyersMS commented Apr 21, 2023 • edited Loading

davidwrighton commented Apr 21, 2023

davidwrighton commented Apr 21, 2023

tannergooding commented Apr 21, 2023 • edited Loading

davidwrighton commented Apr 21, 2023

davidwrighton commented Apr 21, 2023

tannergooding commented Apr 21, 2023

lewing commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023

tannergooding commented Apr 21, 2023

AndyAyersMS commented Apr 21, 2023

lewing commented Apr 23, 2023 • edited Loading

davidwrighton commented Apr 24, 2023

davidwrighton commented Apr 24, 2023

sbomer commented Apr 24, 2023

dotnet-maestro bot commented Apr 11, 2023 •

edited

Loading

AndyAyersMS commented Apr 21, 2023 •

edited

Loading

AndyAyersMS commented Apr 21, 2023 •

edited

Loading

tannergooding commented Apr 21, 2023 •

edited

Loading

lewing commented Apr 23, 2023 •

edited

Loading