Make config binding gen incremental #89587

layomia · 2023-07-27T18:13:42Z

Fixes #83534. #89226 is a related change wrt. perf.

This PR aims to solve the incremental generation perf issue in the config binding generator, so that diffs in input compilations that don't change prior effective inputs the generator (or the resulting output) do not cause regeneration.

It also moves Roslyn helpers, generator helpers, and the TypeRef identity token to common locations to be shared by the JSON and config binding generators.

ghost · 2023-07-27T18:13:58Z

Tagging subscribers to this area: @dotnet/area-extensions-configuration
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #83534.

Author:	layomia
Assignees:	layomia
Labels:	`area-Extensions-Configuration`
Milestone:	8.0.0

src/libraries/Common/src/SourceGenerators/TypeRef.cs

...libraries/System.Text.RegularExpressions/gen/System.Text.RegularExpressions.Generator.csproj

layomia · 2023-08-09T00:45:34Z

There are a few remaining TODOs, but this PR is ready for review.

Validate the perf benefit from being incremental (i.e. improved IDE performance like Filter list of candidate invocations for parsing in binding generator #89226 (comment))
In upcoming commits (no more rebasing)
- add incremental tests like what we have in System.Text.Json
- Iron out the diagnostics issues I'm facing in this change (might peel off to a follow-up PR)
  - The parser isn't registering all the diagnostics we had prior to this change (see here, and here)
  - Pragma source suppressions for SYSLIB1103 & SYSLIB1104 are not working (see here)

...tion.Binder/tests/SourceGenerationTests/Baselines/ConfigurationBinder/GetValue.generated.txt

src/libraries/Common/src/SourceGenerators/DiagnosticInfo.cs

src/libraries/Common/src/SourceGenerators/ImmutableEquatableArray.cs

src/libraries/Common/src/SourceGenerators/TypeRef.cs

...aries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.Emitter.cs

src/libraries/Microsoft.Extensions.Configuration.Binder/gen/ConfigurationBindingGenerator.cs

layomia · 2023-09-27T05:42:16Z

8.0 work left

Reinstate pragma suppressions for diagnostics (Address diagnostic issues in runtime incremental source generators #92509)

9.0

Investigate how to not Collect list of candidate invocations (Make config binding gen incremental #89587 (comment))
Investigate further optimizations for invocation filtering (Config binder gen: further tighten filtering constraint for input binding method invocations #90687)
Address PR feedback on ImmutableEquatableArray impl
Use set semantics for incremental model (Create an internal sorted equatable collection type for incremental source generators #89318)
Sort object properties optimally in emitted config binding gen logic #92594
Misc stuff in Binder gen incremental final #92654

layomia · 2023-09-27T05:54:26Z

...iguration.Binder/tests/SourceGenerationTests/Baselines/ConfigurationBinder/Get.generated.txt

@@ -95,7 +95,15 @@ namespace Microsoft.Extensions.Configuration.Binder.SourceGeneration
        public static void BindCore(IConfiguration configuration, ref int[] instance, bool defaultValueIfNotFound, BinderOptions? binderOptions)
        {
            var temp2 = new List<int>();
-            BindCore(configuration, ref temp2, defaultValueIfNotFound: false, binderOptions);
+
+            foreach (IConfigurationSection section in configuration.GetChildren())


Logic for arrays now inlined. No more BindCore to bind to temp List<T>.

Thank you for calling out the reasoning behind the baseline diffs.

layomia · 2023-09-27T05:58:03Z

...ensions.Configuration.Binder/tests/SourceGenerationTests/Baselines/Collections.generated.txt

@@ -37,7 +37,7 @@ namespace Microsoft.Extensions.Configuration.Binder.SourceGeneration
        #endregion IConfiguration extensions.

        #region Core binding extensions.
-        private readonly static Lazy<HashSet<string>> s_configKeys_ProgramMyClassWithCustomCollections = new(() => new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "CustomDictionary", "CustomList", "IReadOnlyList", "IReadOnlyDictionary" });
+        private readonly static Lazy<HashSet<string>> s_configKeys_ProgramMyClassWithCustomCollections = new(() => new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "CustomDictionary", "CustomList", "ICustomDictionary", "ICustomCollection", "IReadOnlyList", "UnsupportedIReadOnlyDictionaryUnsupported", "IReadOnlyDictionary" });


Used to guard against unknown keys. Correctly include these props even their types are not supported.

layomia · 2023-09-27T05:58:44Z

...ensions.Configuration.Binder/tests/SourceGenerationTests/Baselines/Collections.generated.txt

@@ -85,28 +85,6 @@ namespace Microsoft.Extensions.Configuration.Binder.SourceGeneration
            }
        }

-        public static void BindCore(IConfiguration configuration, ref List<int> instance, bool defaultValueIfNotFound, BinderOptions? binderOptions)


Like with arrays above, proxy binding methods are no longer used for any types. Here & below.

layomia · 2023-09-27T06:00:48Z

...soft.Extensions.Configuration.Binder/tests/SourceGenerationTests/GeneratorTests.Baselines.cs

@@ -141,7 +142,7 @@ public static void Main()

                            public class MyClass
                            {
-                                public string MyString { get; set; }
+                                public string? MyString { get; set; }


Testing removal of EraseCompileTimeMetadata - #89587 (comment).

Addressed Jared's feedback & have sign-off from Roslyn team.

layomia · 2023-09-27T06:08:37Z

/backport to release/8.0

github-actions · 2023-09-27T06:08:50Z

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6322080805

layomia · 2023-09-27T06:09:18Z

Will cherry-pick any needed follow ups into 8.0 backport PR.

github-actions · 2023-09-27T06:10:52Z

@layomia backporting to release/8.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: Make config binding gen incremental
Applying: Iterate on implementation
error: sha1 information is lacking or useless (src/libraries/Microsoft.Extensions.Configuration.Binder/gen/Microsoft.Extensions.Configuration.Binder.SourceGeneration.csproj).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Iterate on implementation
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

github-actions · 2023-09-27T06:10:53Z

@layomia an error occurred while backporting to release/8.0, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.

ericstj · 2023-09-27T14:07:39Z

Maybe we are missing b960f64 from release/8.0?

* Make config binding gen incremental * Iterate on implementation * Add incremental tests & driver * Make incremental tests pass and revert functional regression * Address failing tests * Make tests pass * Suppress diagnostic * Address feedback on diag info creation * Refactor member access expr parsing to indicate assumptions * Address feedback & do misc clean up * Adjust model to minimize baseline diff / misc clean up

ericstj · 2023-09-27T14:10:33Z

Yep, that was it. I was able to cleanly cherry-pick using this:

git checkout release/8.0
git pull upstream release/8.0
git checkout -b backport/pr-89587-to-release/8.0
git cherry-pick b960f64749df76cc237358a0bb44511754489b0c
git cherry-pick 8e760396b1be0e9094634a3e77741bf2408f1d74

https://github.com/dotnet/runtime/tree/backport/pr-89587-to-release/8.0
If it looks good, please open a PR.

* Make config binding gen incremental * Iterate on implementation * Add incremental tests & driver * Make incremental tests pass and revert functional regression * Address failing tests * Make tests pass * Suppress diagnostic * Address feedback on diag info creation * Refactor member access expr parsing to indicate assumptions * Address feedback & do misc clean up * Adjust model to minimize baseline diff / misc clean up

layomia · 2023-09-27T17:36:33Z

Backport PR has been opened: #92730.

* Make config binding gen incremental * Iterate on implementation * Add incremental tests & driver * Make incremental tests pass and revert functional regression * Address failing tests * Make tests pass * Suppress diagnostic * Address feedback on diag info creation * Refactor member access expr parsing to indicate assumptions * Address feedback & do misc clean up * Adjust model to minimize baseline diff / misc clean up Co-authored-by: Carlos Sánchez López <1175054+carlossanlop@users.noreply.github.com>

* Minor code cleanup in TensorPrimitives tests (#92575) * Normalize some test naming * Alphabetize tests * Improve mistmatched length tests with all positions of the shorter tensor * Alphabetize methods in TensorPrimitives.cs * Update android-bionic.md (#92632) * Move TargetsCurrent to net9 and add net8 workload (#91480) * Move TargetsCurrent to net9 and add net8 workload * Fix version references * Update src/mono/nuget/Microsoft.NET.Workload.Mono.Toolchain.Current.Manifest/WorkloadManifest.targets.in * [wasm] build net8 workload * Update emsdk * Update current template to reference net9 * Bump 8.0 version used for workloads * Fix version for latest emscripten packages * fix typo in 8.0 version used for the workload * disamiguate templates * WBT: explicitly use net8.0 projects for template projects * Update emsdk dependency to get the workload fix * fix * Add some addtional workarounds for net8 * Remove extra character * Fix test * More wasi fixes * Add net8 wasi-wasm runtime pack reference * Add wasi-experimental-net8 workload * [wasi] Fix use of workload * [wasm] WBT: Fix test * wasi: Allow wasi-wasm runtimepacks even when targeting net8 * fix test --------- Co-authored-by: Ankit Jain <radical@gmail.com> * Improve nullability check for generic `.ctor` parameters (#92514) * implement absent generic ctor param check * fix code style * Improve nullability check for generic parameters in ctor `NullabilityInfoContext.CheckParameterMetadataType` didn't have code paths for parameters in constructors, leading to wrong nullability results. The PR adds a code path for constructor parameters. Fix #92487 * add tests on nullability of ctors and methods with generic parameters * fix test issues with AOT trimming * [PERF] Add hybrid globalization testing runs (#89825) Add blazor hybrid globalization runs. This includes updating the Blazor and iOS test names to take into account hybridGlobalization and setting up a standard for scenario run configs going forward, at least for now. By having the hybridglobaliztion in both the runconfig and the name when different from the default, the names will only update for non-default settings auto-updating PowerBI while the runconfigs will be available whenever necessary. * JitDump improvements and other minor cleanup (#92510) * JitDump improvements and other cleanups * More comment cleanups * Be consistent in capitalization of `GenTree` * JIT: Remove CallArgABIInformation::IsStruct (#92635) Since we store signature types now this bit is no longer necessary. * [wasi] fixed the order of WASI_AFTER_RUNTIME_LOADED_CALLS (#92552) * Add DebuggerDisplay to Meter and Instruments (#91496) * Add net8 wasi workload tests (#92653) * Add net8 wasi workload tests * Update eng/testing/tests.wasi.targets Co-authored-by: Ankit Jain <radical@gmail.com> * [wasm] CI: trigger WBT on changes to eng/testing/tests.{browser,wasm,wasi}.targets * Update eng/testing/tests.wasi.targets * Alias the net8 runtime pack correctly --------- Co-authored-by: Ankit Jain <radical@gmail.com> * Update the Windows ARM64 unwinder (#92604) * Update the Windows ARM64 unwinder This change updates the Windows ARM64 unwinder to match the current state in Windows. It contains a fix for a bug that is needed as a basis for a .NET issue fix. * Reflect PR feedback * [mono][llvm] Remove support for llvm versions before 14.x. (#88346) * Vectorize TensorPrimitives.Min/Max{Magnitude} (#92618) * Vectorize TensorPrimitives.Min/Max{Magnitude} * Use AdvSimd.Max/Min * Rename some parameters/locals for consistency * Improve HorizontalAggregate * Move a few helpers * Avoid scalar path for returning found NaN * [main] Update dependencies from dotnet/runtime dotnet/source-build-reference-packages dotnet/emsdk dotnet/hotreload-utils dotnet/sdk (#92584) [main] Update dependencies from dotnet/runtime dotnet/source-build-reference-packages dotnet/emsdk dotnet/hotreload-utils dotnet/sdk - Coherency Updates: - runtime.linux-arm64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-x64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-arm64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-x64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-arm64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-x64.Microsoft.NETCore.Runtime.ObjWriter: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-arm64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-x64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-x64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-arm64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-x64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-arm64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-x64.Microsoft.NETCore.Runtime.JIT.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.linux-musl-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - runtime.osx-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools: from 16.0.5-alpha.1.23452.1 to 16.0.5-alpha.1.23472.1 (parent: Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport) - Merge branch 'main' into darc-main-be922536-a638-4652-9241-ddc0996cfe5a * Avoiding trying to load the native library Microsoft.DiaSymReader.Native.<arch>.dll on Linux (#92492) * 92278: add DFEATURE_ISYM_READER definition only for windows --------- Co-authored-by: Andrey.Kudashkin <Andrey.Kudashkin@russianpost.ru> Co-authored-by: Jan Kotas <jkotas@microsoft.com> * Convert SpinWait to QCall (#92675) * define bool as Interop.BOOL to prevent upper bytes setting native bool (#92679) * Remove all PGO assets except for the runtime PGO archive. (#92668) * Stop setting separate properties for BUNDLE_PROBE, HOSTPOLICY_EMBEDDED, PINVOKE_OVERRIDE (#92448) * Make config binding gen incremental (#89587) * Make config binding gen incremental * Iterate on implementation * Add incremental tests & driver * Make incremental tests pass and revert functional regression * Address failing tests * Make tests pass * Suppress diagnostic * Address feedback on diag info creation * Refactor member access expr parsing to indicate assumptions * Address feedback & do misc clean up * Adjust model to minimize baseline diff / misc clean up * Extend preinitialization interpreter (#92470) Things that I added: * Support for `typeof(T) == typeof(Bar)` (this will be useful later, we'll eventually be able to also freeze these). * Support static interface method calls * Constrained method calls on valuetypes * More `ReadOnlySpan` construction patterns, `.Length` * More indirect load/store support Contributes to #78681. To full resolve this, we need to fix up things so we can answer `Sse2.IsSupported`. * Make it possible to preinitialize HW intrinsic IsSupported (#92666) * Move the IL rewriting for HW intrinsics `IsSuported` calls to `ILProvider` from `RyuJitCompilation` * Also rewrite constant true/false * [mono][aot] Type load checks do not fail at compile time but produce a runtime exception (#91261) * Enable tests. * When AOTing, type checks do not fail compilation but create a runtime exception. * Cleaned up type load error cleaning. TypeLoadException icall now has a message with type name. * Removed another instance of indiscriminate exception clearing. * Fixed build warning. * Using class const instead of string const. Reverted some compile to runtime errors that were not necessary for the unit tests. * White space. * Fixed build warning. * Trying to fix weird AOT errors, fixed type load throw function. * Fixed build error. * Special handling for classes that are NULL. * Providing for a null klass when generating exception. * Removed flow control directive from macro. * Fixed stack corruption. * Attempt to push the correct type onto the stack. * Fixing uninitialized ins. * Fixing ro_type. * Initializing ins. * Complex cases with type load failures replace method body with a throw. * Cleaning up superfluous code changes. * Restored sizeof cosntant on failed types. * [mono] Implement Vector128.Shuffle () for llvm on x64. (#92656) * JIT: Handle remainder accesses more precisely in physical promotion liveness (#92651) The liveness pass in physical promotion will currently handle any struct LCL_FLD access of a physically promoted struct as accessing the remainder. However, if the LCL_FLD only touches promoted fields then the remainder is not actually used. There was a TODO around this which this PR fixes as I stumbled upon a case this would improve. * Add tests for `UnsafeAccessor` on fields on generic types (#92657) * Add tests for field access on generics The tests are currently disabled. * Fix ILC to compile UnsafeAccessorsTests UnsafeAccessorsTests passes on NAOT. * Switch to etw python script (#92508) * initial work with hacks to switch to EtwProvider python script * move to generated scripts * Fixes for some link issues * fix link issue * adding private etw callback to enable GC events * Fix x86 build break * fixing Linux build break * fixing gcpriv.h * making minimal typedefs * FB * Fix for posix break * Fix Excessive Encoding in Test Logs (#92286) * Removed special encoding that was rendering the test logs near impossible to read properly. * Adjusted the offending test to print the invalid character's hex code instead, and fixed it alongside its sibling test because they didn't handle all correct/incorrect cases properly. * Added special handling for illegal XML characters in the test results' XML logs. * Simplified the sanitizing algorithm to one pass, as per Dan's feedback. * Fix LLVMAOT Mono runtime variant official build to produce correctly named runtime packs (#92712) In https://github.com/dotnet/runtime/commit/75ee623b8f0350a4b4be86fa71745a74beb059d1 the condition in `src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.props` got changed from checking `MonoBundleLLVMOptimizer` to `MonoAOTEnableLLVM` but we weren't setting that property in runtime-official.yml so both jobs produced runtime packs with the same suffix, resulting in the artifact uploads randomly overwriting each other. * Change order of loads in LowerMemcmp (#92704) * Fix arm64 fragment unwinding (#92678) A bug in the Windows arm64 unwinder that existed a long time ago has caused problems with unwinding in functions split in multiple fragments in case the location in the function was in a secondary fragment. At that time, it was not discovered that it was a bug in the unwinder and it got "fixed" in the runtime by always using the first fragment unwind info. However, now it turned out that was actually incorrect in some cases. Checking the current state of the Windows unwinder revealed that a bug was fixed there that was causing the problem we were seeing. Effectively ignoring all the shadow prolog unwind info in the secondary fragments. This change reverts the old fix after the unwinder was updated. * Add illink analyzer support for field/property initializers (#92600) The dataflow analyzer was exiting early for cases where the owning symbol was not an `IMethodSymbol`. This meant we weren't running dataflow analysis for field and property initializers. This fixes it by allowing through cases where the owning symbol is not an `IMethodSymbol`, and adding testcases to validate that we don't hit asserts in the code paths that only light up for methods. * Don't build libraries native packages in the PGO leg (#92729) * Change how test assemblies opt-in to LibraryImportGenerator usage (#92661) * Move more sprintf usages to snprintf (#92674) * [wasm] GetChromeVersions: Fix fetching v8 version given a chrome version (#92667) * [wasm] GetChromeVersions: Fix fetching v8 version given a chrome version * Address feedback from Ilona Tomkowicz * Updated XML documentation for `IConfigurationProvider.GetReloadToken`. (#92720) * Avoid membarrier on lower Android versions (#92686) Hopefully fixes #92196. I don't actually have an ARM64 device with an old Android version so I can't testing it actually fixes the problem, but it's a plausible fix. * Revert "Remove Latin1CharSearchValues (#91884)" (#92726) * Revert "Remove Latin1CharSearchValues (#91884)" This reverts commit 4a09c82215399c27f52277a8db7178270410c693. * Keep the projitems formatting * [main] Update dependencies from dotnet/roslyn-analyzers (#92639) Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Sven Boemer <sbomer@gmail.com> * [wasm] Fix Wasm.Build.Tests on `main` (#92741) * CI: Don't include ref pack when the tests need workload .. as it already includes it. https://github.com/dotnet/runtime/issues/92732 broke this as a side-effect which caused the `microsoft.netcore.app.ref` directory to not be created. Fixes https://github.com/dotnet/runtime/issues/92732 . * [wasm] WBT: Update skiasharp reference `blz_deploy_on_build_Debug_True_npl3f0nk_qee.csproj : error NU1903: Package 'SkiaSharp' 2.88.4-preview.76 has a known high severity vulnerability` * Make PCP tests conditional by algorithm The ConditionalFact tests for a functional TPM using P-256. Tests assumed that if the TPM supported P-256, then P-384 and RSA is supported as well. This is not always the case - some TPMs implement 256 without support for 384. This changes the TPM conditional facts to be per-algorithm. * Small refactor to BuildElement to address NRT changes (#92742) * [mono] Enable SIMD intrinsics on winx64. (#92673) * [mono] Enable SIMD intrinsics on winx64. * Re-enable decompose on SIMD intrinsics on Windows. --------- Co-authored-by: lateralusX <lateralusx.github@gmail.com> * JIT: Promote size-wise improvements in physical promotion (#92717) I hit the following case: ``` Evaluating access byref @000 Single write-back cost: 3 Write backs: 0 Read backs: 0 Estimated cycle improvement: 0 cycles per invocation Estimated size improvement: 2 bytes Disqualifying replacement ``` These cases happen when the blocks that have candidates for promotion in them have bbWeight equal to 0. If we estimate a size improvement without a cycle improvement it still makes sense to promote a replacement. More generally, a large size improvement can make up for a small cycle regression, so add a heuristic similar to the existing one for this. I've set it to be quite conservative: we require 100 bytes of size improvement before we allow 1 cycle of regression. This is enough to handle the common case where the cycle improvement is 0 due to the bbWeight = 0. * Update TensorPrimitives aggregations to vectorize handling of remaining elements (#92672) * Update TensorPrimitives.CosineSimilarity to vectorize handling of remaining elements * Vectorize remainder handling for Aggregate helpers * JIT: Unify and clean up unspilling (#91663) * [wasm][debugger] Support passing identifiers to methods (#92758) * Basic fix. * More tests. * Move tests to more suitable place. * Pause earlier. * [llvm] Avoid zero extending non-negative constant array indexes, its not needed and it prevents abcrem from working. (#92760) * Fix link in ILLink.Tasks README.md (#92769) * [main] Update dependencies from dotnet/xharness dotnet/cecil dotnet/sdk (#92700) * Update dependencies from https://github.com/dotnet/xharness build 20230927.1 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 8.0.0-prerelease.23471.1 -> To Version 8.0.0-prerelease.23477.1 * Update dependencies from https://github.com/dotnet/cecil build 20230926.1 Microsoft.DotNet.Cecil From Version 0.11.4-alpha.23468.2 -> To Version 0.11.4-alpha.23476.1 * Update dependencies from https://github.com/dotnet/sdk build 20230927.2 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23476.1 -> To Version 9.0.100-alpha.1.23477.2 * Update dependencies from https://github.com/dotnet/xharness build 20230927.1 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 8.0.0-prerelease.23471.1 -> To Version 8.0.0-prerelease.23477.1 * Update dependencies from https://github.com/dotnet/cecil build 20230926.1 Microsoft.DotNet.Cecil From Version 0.11.4-alpha.23468.2 -> To Version 0.11.4-alpha.23476.1 * Update dependencies from https://github.com/dotnet/sdk build 20230927.63 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23476.1 -> To Version 9.0.100-alpha.1.23477.63 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Update dependencies from https://dev.azure.com/dnceng/internal/_git/dotnet-optimization build 20230927.3 (#92761) optimization.linux-arm64.MIBC.Runtime , optimization.linux-x64.MIBC.Runtime , optimization.windows_nt-arm64.MIBC.Runtime , optimization.windows_nt-x64.MIBC.Runtime , optimization.windows_nt-x86.MIBC.Runtime , optimization.PGO.CoreCLR From Version 1.0.0-prerelease.23471.3 -> To Version 1.0.0-prerelease.23477.3 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Update JIT format job to pass new `--cross` argument on Linux (#92751) * Update JIT format job to pass new `--cross` argument on Linux This argument is passed through to the jit-format tool and is used when invoking build-runtime.sh on Linux when in a cross-build scenario, such as CI builds in Mariner. * Fix copy/paste error * Removes redundant code from `JsonCamelCaseNamingPolicy.cs` (#92738) * Remove Unwrap flag from UniqueComInterfaceMarshaller (#92599) The Unwrap flag only has effect when UniqueInstance is not set. To avoid confusion from anyone referencing this code, we should remove it here. NativeAOT needed to move the Unwrap code to inside the !UniqueInstance block to match behavior of CoreCLR. This should only be noticeable when using ComWrappers to wrap an unwrap the same object in the same NativeAOT instance. In-Proc COM with different servers and clients won't hit this behavior. * Account port number already included within server string (#92748) * Account port number already included within server string * Refactor the test * Apply feedback * Delete misc unnecessary code (#92764) * [main] Update dependencies from dotnet/installer (#92703) * Update dependencies from https://github.com/dotnet/installer build 20230927.3 Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23474.1 -> To Version 9.0.100-alpha.1.23477.3 * [wasm] WBT: Update skiasharp reference `blz_deploy_on_build_Debug_True_npl3f0nk_qee.csproj : error NU1903: Package 'SkiaSharp' 2.88.4-preview.76 has a known high severity vulnerability` * Update dependencies from https://github.com/dotnet/installer build 20230927.26 Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23474.1 -> To Version 9.0.100-alpha.1.23477.26 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Ankit Jain <radical@gmail.com> * Add more static preinitialization support (#92739) Resolves #78681 (or "there's nothing else we'd be willing to do for it"). This is the rest of what I implemented trying to get `SearchValues.Create(someVeryLongString)` to preinitialize. It doesn't actually enable more `SearchValues` scenarios because I eventually hit codepaths that would require us to implement hardware intrinsics support in the interpreter. The `SearchValues` scenarios that we do support were implemented in #92470 and #92666. But since I already implemented this, here it is; maybe it will be useful for something else: * Support for modelling `Span<X>`, including creating it from stackalloc * Support for math/comparisons with native int * `Unsafe.Add` * Flesh out TensorPrimitives XML docs (#92749) * Flesh out TensorPrimitives XML docs * Address PR feedback - Remove use of FusedMultiplyAdd from all but CosineSimilarity - Remove comments about platform/OS-specific behavior from Add/AddMultiply/Subtract/Multiply/MultiplyAdd/Divide/Negate - Loosen comments about NaN and which exact one is returned * Address PR feedback * Converge Representations between NativeAOT and CoreCLR (#91821) * Update RyuJit overview (#92789) * Correctly set sendTrustList flag when saving credentials to cache (#92731) * JIT: Make effect handling in lowering less conservative (#92710) The interference checking in lowering bases some of its checks on GenTree::gtFlags. This is conservative since it includes effect flags of operands. For LIR this does not really make sense and ends up being conservative. This PR replaces the relevant uses of gtFlags with a new GenTree::OperEffects() that computes the relevant effect flags for the node, excluding operands. We already know how to recompute effect flags other than GTF_GLOB_REF and GTF_ORDER_SIDEEFF. This PR adds functions for these as well (the GTF_GLOB_REF version GenTree::OperRequiresGlobRefFlag is courtesy of @SingleAccretion). For GTF_ORDER_SIDEEFF we add a GenTree::OperSupportsOrderingSideEffect which captures explicitly (and conservatively) the current cases where we are setting the flag, and only allows these cases to support the flag. Setting the flag for other cases may result in the flag being removed or ignored. There is a new `GenTree::SetHasOrderingSideEffect` to add the flag which also asserts that it is only added for trees that are supported. Fix #92699 * [wasm] Supress policheck warning in blazor-sample (#92711) * [wasm] Supress policheck warning in blazor-sample Replace the offending part in the layout name. I think the suit-spade is false positive, I used just sp in place of spade to silence it. * Feedback * [mono] Cleanup unused runtime functions (#91681) - Removes unused functions - Removes cmake configure checks for functions/headers that are no longer needed - Renames HAVE_UWP_WINAPI_SUPPORT to HAVE_APP_WINAPI_SUPPORT - Move MSVC warning disables into cmake so it is more visible Co-authored-by: Johan Lorensson <lateralusx.github@gmail.com> * Do not nop-out SSA definitions in block morphing (#92786) SSA definitions cannot be deleted. * Implement StoreVector64x2 and StoreVector128x2 for Arm64 (#92109) * Implement StoreVector128x2 for Arm64 * Remove redundant implmentations * Implement StoreVector64x2 for Arm64 * Remove StoreVector64x2 implementation for Arm64 This reverts commit 49ef72e3a3eaa58d3b3338dc5d6d80a7ca0b50b5. * Fix instruction type for the StoreVector128x2 intrinsic * Review comments: * Arrange APIs alphabetically * Add StoreVector64x2 * fix the invalid instructions * Add test cases * Update src/coreclr/jit/hwintrinsicarm64.cpp Co-authored-by: Bruce Forstall <brucefo@microsoft.com> --------- Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com> Co-authored-by: Bruce Forstall <brucefo@microsoft.com> * Vectorize TensorPrimitives.ConvertToHalf (#92715) * Enable TensorPrimitives to perform in-place operations (#92820) Some operations would produce incorrect results if the same span was passed as both an input and an output. When vectorization was employed but the span's length wasn't a perfect multiple of a vector, we'd do the standard trick of performing one last operation on the last vector's worth of data; however, that relies on the operation being idempotent, and if a previous operation has overwritten input with a new value due to the same memory being used for input and output, some operations won't be idempotent. This fixes that by masking off the already processed elements. It adds tests to validate in-place use works, and it updates the docs to carve out this valid overlapping. * JIT: Optimize SequenceEqual to use ccmp on ARM64 (#92810) In the original PR we could not get this this working due to some conservative interference. This now does the right thing with #92710 merged. Also change LowerCallMemcmp/LowerCallMemmove to return next node to lower just to align it a bit more with other functions. * [main] Update dependencies from dnceng/internal/dotnet-optimization (#92813) Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Move variable scope tracking code (#92800) Move it out of codegencommon.cpp and into scopeinfo.cpp. This is both to centralize the code but also to reduce the size of the very large codegencommon.cpp. * Allow `key#value` for superpmi JIT option specification (#92803) superpmi.py will pass this through from the `-jitoption` / `-base_jit_option` / `-diff_jit_option` to superpmi.exe `-jitoption` and `-jit2option`. Currently, the format is `key=value`. I wrap invocation of superpmi.py with Windows batch file scripting, which has an annoying problem of "eating" the equals size `=`. This works around that problem. I can't think of any case where `#` is needed in a key or value, hence that choice as an additional option. * Use a different crossgen2 when running crossgen2 during our build than the crossgen2 that we are shipping (#92677) * Fix Common.Tests.GetPrettyName_CannotRead_ReturnsNull test for root user (#92695) * fix Common.Tests.GetPrettyName_CannotRead_ReturnsNull test for root user * remove direct call to libc in Common.Tests.GetPrettyName_CannotRead_ReturnsNull * Update src/libraries/Common/tests/Tests/Interop/OSReleaseTests.cs * split Common.Tests.OSReleaseTests.GetPrettyName_CannotRead_ReturnsNull into two test cases * replace ifs with ConditionalFact in Common.Tests.OSReleaseTests class --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> * Update dependencies from https://github.com/dotnet/installer build 20230928.5 (#92817) Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23477.26 -> To Version 9.0.100-alpha.1.23478.5 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Update dependencies from https://github.com/dotnet/roslyn-analyzers build 20230928.1 (#92814) Microsoft.CodeAnalysis.Analyzers , Microsoft.CodeAnalysis.NetAnalyzers From Version 3.11.0-beta1.23475.2 -> To Version 3.11.0-beta1.23478.1 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Add stress mode for arm/arm64 function fragment splitting (#92802) Set JitSplitFunctionSize to either 4 or 200 under new STRESS_UNWIND mode. * Always emit jump from hot finally block to cold target (#92797) On some platforms, CodeGen::genCallFinally() will remove the jump between a finally block in a call-finally pair to its jump target if the target is its immediate successor in the block list (in other words, we just fall through). However, if we are doing hot/cold splitting, it is possible for the finally block to be the last hot block, and its target the first cold block. Thus, if the two are contiguous in the block list but in separate regions, we must always emit a jump. * [tests][iOS] Fix artifacts path (#92783) * Vectorize TensorPrimitives.ConvertToSingle (#92779) * Vectorize TensorPrimitives.ConvertToSingle * Address PR feedback * Remove CompilationProvider dependency from the source generator. (#92833) * Apply sequence equality comparison to the final Regex incremental value. (#92835) * Apply sequence equality comparison to the final Regex incremental value. * Avoid using SequenceEqual * Throw exception in TensorPrimitives for unsupported span overlaps (#92838) * Allow multiple post-build steps and allow templated pre and post-build steps (#92375) * Include info about system call errors in some exceptions from operating on named mutexes (#92603) * Include info about system call errors in some exceptions from operating on named mutexes - Added new PAL APIs for creating and opening mutexes that take a string buffer for system call error info. These are called with a stack-allocated buffer and upon error the system call errors are appended to the exception message. - When there is a system call failure that leads to the PAL API failing, some info is appended to the error string, including the system call, relevant arguments, return value, and `errno` - `chmod` on OSX seemingly can be interrupted by signals, fixed to retry. Also fixed a couple other small things. Fixes https://github.com/dotnet/runtime/issues/89090 * Remove fgUpdateFlowGraph from optOptimizeFlow (#92839) * Split off patched code into separate .S file and disable subsections-via-symbols for it (#92555) * [amd64/arm64] Split off patched code into separate .S file and disable subsections-via-symbols for it * [amd64/arm64] Split off patched code into separate .asm file [arm64] Move JIT_UpdateWriteBarrierState out of the patched region to match implementation in .S file * Remove NO_SUBSECTIONS_VIA_SYMBOLS * JIT: fix self-conflicting HFA arg prolog handling for arm64 (#92355) Fix prolog handling in the case where the in-body destination register for an HFA overlaps with one of the HFA argument registers. For instance the HFA is passed in `s0-s3` and needs to end up in `v3`. This requires special handling because the dependence analysis done in `genFnPrologCalleeRegArgs` only tracks entire registers, not parts of registers. Fixes #83167 * Update targetingpacks.targets (#88991) * Update targetingpacks.targets The .NET 8 Preview 6 SDK has the features required to simplify the targetingpacks.targets logic. * Update targetingpacks.targets * Update targetingpacks.targets * Update known items * Update targetingpacks.targets * Update targetingpacks.targets * [wasm] Use specific version of v8 for tests (#91633) * [wasm] Add support for installing V8 * [wasm] Use provisioned v8 for library tests * [wasm] WBT: Use provisioned v8 * [wasm] enable use of provisioned v8 for library tests * [wasm] add MSBUILD_ARGS for build-runtime-tests make target * update docs * Don't install v8 for runtime tests * [wasm] CI: trigger library test jobs when chrome version changes * Disable provisioning v8 when building runtime tests * address review feedback * [wasm] Disable installing v8 for runtime tests * Address review feedback * fix stamping for v8 * Automated bump of chrome version (#92854) * Preinitialize pop/switch/Type.IsValueType (#92841) These showed up in ASP.NET Stage1 and were low hanging enough. * Use UnsafeAccessor in JSHostImplementation instead of reflection (#92755) * [main] Update dependencies from dotnet/emsdk dotnet/sdk (#92815) [main] Update dependencies from dotnet/emsdk dotnet/sdk * [main] Update dependencies from dotnet/installer (#92848) * Update dependencies from https://github.com/dotnet/installer build 20230929.5 Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23478.5 -> To Version 9.0.100-alpha.1.23479.5 * Update dependencies from https://github.com/dotnet/installer build 20230929.5 Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23478.5 -> To Version 9.0.100-alpha.1.23479.5 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Convert all remaining tests in the Loader subtree to the merged model (#92407) * Call fgRenumberBlocks in optIfConversion (#92821) * [llvm] Fix spilling of valuetypes to the stack if they are passed by addr. (#92658) The previous code would spill the valuetype when it was passed by addr, and another bblock could try to read from the unitialized stack location. * Check DotNetFinalVersionKind when setting WorkloadVersionSuffix (#91792) * [RISC-V] regArg dependcies unrolling in genFnPrologCalleeRegArgs() (#91904) * [RISC-V] Fix target type unsignedness detection in genFloatToIntCast() (#92694) * [RISC-V] Fix target type unsignedness detection in genFloatToIntCast() treeNode->gtFlags & GTF_UNSIGNED means unsignedness of the source type. Use varTypeIsUnsigned instead which checks for VTF_UNS on target type classification. Fixes TryConvertToSaturatingUInt64Test and TryConvertToTruncatingUInt64Test from System.Runtime.Numerics.ComplexTests_GenericMath. * Fix compilation without FEATURE_TIERED_COMPILATION * [browser] Remove duplicated marshaling of return value for JSExport (#92403) * Remove duplicated marshaling of return value for JSExport * Move unmarshal and return value marshal into try block * Update intellisense.targets (#92868) * Update intellisense.targets * Update System.Text.Json.csproj * Ensure the adapter name 100% matching when parsing proc/net/dev (#92187) * Ensure the adapter name 100% matching when parsing proc/net/dev * Update src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/StringParsingHelpers.Statistics.cs Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> * Move the stackalloc out of the loop * Update src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/StringParsingHelpers.Statistics.cs Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> --------- Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> * [mono] Only emit pshufb when ssse3 is enabled. (#92842) Fixes https://github.com/dotnet/runtime/issues/92827. * [browser][nodejs] keep runtime alive for JSExport calls (#92871) * [wasm] Disable MetricsSupport feature by default (#92696) This should improve the startup perf and size. * Fix build of IJW test after VS upgrade (#92878) The latest build of VS carries a C/C++ compiler which produces warning C5271: ``` src\native\corehost\test\ijw\ijw.cpp(6): warning C5271: consider replacing #using <System.Console.dll> with command line argument /FU "F:\dotnet\runtime2\.dotnet\packs\Microsoft.NETCore.App.Ref\8.0.0-rc.1.23414.4\ref\net8.0\System.Console.dll" src\native\corehost\test\ijw\ijw.cpp(7): warning C5271: consider replacing #using <System.Runtime.Loader.dll> with command line argument /FU "F:\dotnet\runtime2\.dotnet\packs\Microsoft.NETCore.App.Ref\8.0.0-rc.1.23414.4\ref\net8.0\System.Runtime.Loader.dll" ``` This breaks the build on Windows. For now I'm disabling the warning as the real fix is more complex (we would need to calculate the path to the required assemblies in CMake somehow). * Add linux-arm64 workload definitions (#92892) * [wasm] Set InstallV8ForTests=true only for windows/linux (#92896) .. on CI, or in a container (like codespaces). Without this it would be `true` on macOS by default, and then fail with: `error : V8 provisioning only supported on Linux, and windows.` * [wasm] Perf pipeline - fix blazor_scenarios run for hybrid globalization (#92898) * [wasm] Perf pipeline - fix blazor_scenarios run for hybrid globalization Fails with: ``` Traceback (most recent call last): File "/mnt/vss/_work/1/s/Payload/performance/scripts/ci_setup.py", line 487, in <module> __main(sys.argv[1:]) File "/mnt/vss/_work/1/s/Payload/performance/scripts/ci_setup.py", line 483, in __main main(CiSetupArgs(**vars(args))) File "/mnt/vss/_work/1/s/Payload/performance/scripts/ci_setup.py", line 411, in main dotnet_version = dotnet.get_dotnet_version(target_framework_moniker, args.cli) if args.dotnet_versions == [] else args.dotnet_versions[0] File "/mnt/vss/_work/1/s/Payload/performance/scripts/dotnet.py", line 581, in get_dotnet_version raise RuntimeError( RuntimeError: Unable to determine the .NET SDK used for net8.0 ``` This is because the definition didn't copy over the `--dotnet-versions 8.0.0` workaround needed for now. And this runs only once a week, so it was discovered on Oct 2(Monday) even though it was merged on Sep 26(Friday). * [wasm] Perf: run the hybrid-globalization job on runtime-wasm-perf also, for validation * Update owner list (#92900) * [mono][android] Add Android linux-arm64 workload definitions (#92899) * Add linux-arm64 workload definitions * Add linux-arm64 for android workloads * [main] Update dependencies from dotnet/roslyn (#92578) * Update dependencies from https://github.com/dotnet/roslyn build 20230924.2 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23474.2 * Update dependencies from https://github.com/dotnet/roslyn build 20230924.3 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23474.3 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.1 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.1 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.2 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.2 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.3 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.3 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.4 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.4 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.5 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.5 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.6 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.6 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.7 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.7 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.8 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.8 * Update dependencies from https://github.com/dotnet/roslyn build 20230925.10 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23475.10 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.3 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.3 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.6 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.6 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.13 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.13 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.14 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.14 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.15 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.15 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.21 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.21 * Update dependencies from https://github.com/dotnet/roslyn build 20230926.22 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23476.22 * Update dependencies from https://github.com/dotnet/roslyn build 20230927.1 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23477.1 * Update dependencies from https://github.com/dotnet/roslyn build 20230927.4 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23477.4 * Update dependencies from https://github.com/dotnet/roslyn build 20230928.3 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23478.3 * Update dependencies from https://github.com/dotnet/roslyn build 20230928.4 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23478.4 * Update dependencies from https://github.com/dotnet/roslyn build 20231001.1 Microsoft.CodeAnalysis , Microsoft.CodeAnalysis.CSharp , Microsoft.Net.Compilers.Toolset From Version 4.8.0-3.23474.1 -> To Version 4.8.0-3.23501.1 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Fix deadlock in EventPipeEventDispatcher (#92806) * Remove 'tracing' from Native AOT PR runs (#92825) Tracing heavily increases the time it takes to run Native AOT tests. We already run these tests in the outerloop and the probability that any given PR will break these tests is low. Outerloop coverage should be good enough right now. * Clean up the crossgen2_publish project and local/live packs references (#92826) * Inline some of the options for the new crossgen2_publish project. * Resolve TODOs in targetingpacks.targets * Crossgen1 is long gone. Don't try to discover it in our override targets. * Move ReadyToRun.targets infra into the shared repo infrastructure and have projects automatically opt-in to it instead of the LKG crossgen2 when they are targeting the live build. * Remove extraneous property set (the same value is calculated automatically already) * Remove outdated comment. * Fix NativeAOT and installer legs * Condition turning off pack downloads based on opt-in to local pack usage. * Use the LKG host instead of the 7.0 host as the fallback for NativeExports. * Apply suggestions from code review Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> * Remove AdditionalProperties as they aren't needed (global properties on the command line are already transitive) * PR feedback * Hook into the targets pipeline to avoid overridding targets for R2Ring projects that reference the live framework packs. Move the "target override" logic back to where we build the runtime pack as that's the only place where we need crossgen2 and can't reference the runtime pack (as we're building it). * Don't set CoreCLRArtifactsPath manually. * PR feedback --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> * fix typo (#92893) * SPMI: Disable CodeQL in superpmi-collect pipeline (#92872) This weekend's runs hit a bunch of timeouts due to auto-injected CodeQL. * SPMI: Simplify and improve reporting of context information (#92824) Currently we have multiple separate mechanisms to report information back from superpmi.exe: 1. -baseMetricsSummary/metricsSummary, which outputs a .csv file with aggregated statics for all contexts from the perspective of the base JIT 2. -diffMetricsSummary, which is the corresponding for the diff JIT when diffing 3. -diffsInfo, which during diffing will output a .csv with individual rows for every context that had diffs in it This PR replaces these three mechanisms with a -details argument. When passed, superpmi.exe will write a .csv file to the specified path that contains a row for every context. The arg is supported in both replay and diff mode but creates .csv files with slightly different formats for these. For replays the header output is: ``` Context,Context size,Result,MinOpts,Size,Instructions ``` For diffs the output is: ``` Context,Context size,Base result,Diff result,MinOpts,Has diff,Base size,Diff size,Base instructions,Diff instructions ``` superpmi.py is changed to utilize this new output instead, which involves computing some of the same details we were getting from the metrics summaries before. Prerequisite for #85755 * [mono][jit] Arm64 SIMD regs are now zeroed with movi instead of eor (#92882) * SIMD regs are now zeroed with movi instead of eor. * Simplified vector length selection. * JIT: Merge consecutive stores (#92852) Co-authored-by: Egor <egorbo@Egors-MacBook-Pro.local> Co-authored-by: Jakob Botsch Nielsen <Jakob.botsch.nielsen@gmail.com> * Improve throughput / allocations of JsonNode.GetPath (#92284) * Improve throughput / allocations of JsonNode.GetPath The current implementation is creating a `List<string>` and appending each segment to it, which in most of the cases is allocating a `string`. Then it iterates through that list in reverse order appending to a newly-created `StringBuilder`, which it then `ToString`s. In this change, it instead just uses `ValueStringBuilder`, appending to it as it goes. In doing so, it does reverse the order of enumeration. Previously each node would effectively do: ```C# void GetPath() { AddNode(this); parent?.GetPath(); } ``` and now it's doing: ```C# void GetPath() { parent?.GetPath(); AddNode(this); } ``` While C# doesn't emit tail calls, with optimizations enabled, it's feasible the JIT might emit the recursive call as a jmp rather than a call, in which case it would avoid possible stack dives. However, that's not guaranteed, and doesn't happen today in tier 0 and other unoptimized code. On top of that, to get such a deep nesting in a JsonNode, you need to either go out of your way to create one manually using the JsonNode/Object/Array/Value constructors, or you need to use JsonSerializer.Deserializer, overriding its default MaxDepth, and in the case of a really deep input, it's also recursive and will stack overflow in smaller situations. I have a different version of this change that keeps the same ordering, passing around a span and a length separately, and prepending to the end of the span, but it results in more complicated code, so I'd prefer this variation that just uses ValueStringBuilder unless we have real concerns. * Address PR feedback * Update dependencies from https://github.com/dotnet/source-build-externals build 20231002.3 (#92936) Microsoft.SourceBuild.Intermediate.source-build-externals From Version 9.0.0-alpha.1.23475.2 -> To Version 9.0.0-alpha.1.23502.3 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * [wasm] Use intended ports when running DevServer (#92913) * [wasm] Ignore empty `$ASPNETCORE_URLS` * [wasm] DevServer: honor urls specified in the options * [wasm] CI: Don't trigger non-wbt jobs on wasm-app-host changes * CI: don't trigger wasm runtime tests on wasm-app-host changes * This vectorizes TensorPrimitives.Log2 (#92897) * Add a way to support operations that can't be vectorized on netstandard * Updating TensorPrimitives.Log2 to be vectorized on .NET Core * Update src/libraries/System.Numerics.Tensors/src/System/Numerics/Tensors/TensorPrimitives.netstandard.cs Co-authored-by: Stephen Toub <stoub@microsoft.com> * Ensure we do an arithmetic right shift in the Log2 vectorization * Ensure the code can compile on .NET 7 * Ensure that edge cases are properly handled and don't resolve to `x` * Ensure that Log2 special results are explicitly handled. --------- Co-authored-by: Stephen Toub <stoub@microsoft.com> * Add analyzer support for inline array access (#92736) This allows analysis of inline array access operations, by treating them similarly to array access. However, like ILLink/ILCompiler it doesn't understand inline array creation, so doesn't track them as arrays. The result is that values read out of an inline array are unknown, so this produces dataflow warnings when such a value is passed to a location with dataflow requirements, matching the ILLink/ILCompiler behavior. Using `InlineArray` required referencing a more recent of the .NET 8 reference assemblies. Fixes https://github.com/dotnet/runtime/issues/88684 * [main] Update dependencies from dotnet/installer (#92933) Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23479.5 -> To Version 9.0.100-alpha.1.23502.7 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> * Add testing for #92539. (#92926) * Add testing for #92539. * Remove netfx test skips. * Fix illink task lock during live build (#92928) * Fix illink task lock during live build Fixes https://github.com/dotnet/runtime/discussions/92126 * Update illink.targets * Throw NotSupportedException when applying JsonObjectHandling.Populate on types with parameterized constructors. (#92937) * CI: Don't trigger runtime pipelines on perf pipeline only changes (#92903) * CI: Don't trigger runtime pipelines on perf pipeline only changes * address review feedback from @ cincuranet * Condition the use of NetCoreAppPrevious TFM (#92941) * Condition the use of NetCoreAppPrevious TFM NuGet doesn't support duplicate TFMs in the TargetFrameworks string. Condition the use of NetCoreAppPrevious TFMs until NuGet supports that (which is planned afaik). * Fix ODBC project TFM * Don't publish crossgen2 as NativeAOT when doing a cross-os build. (#92948) * Update PGO to use the correct post-build steps model (#92958) * Adding Log2 tests covering some special values (#92946) * Expose an internal ISimdVector interface and being using it to deduplicate some SIMD code (#90764) * Adding an internal ISimdVector`2 interface * Move LastIndexOfValueType to use ISimdVector`2 * Fix a couple minor whitespace nits and remove an unnecessary local * [wasm] Disable `TensorPrimitivesTests.ConvertToHalf_SpecialValues` (#92953) Failing test: `System.Numerics.Tensors.Tests.TensorPrimitivesTests.ConvertToHalf_SpecialValues` Issue: https://github.com/dotnet/runtime/issues/92885 * JIT: Expand unaligned address recognition for ARM32 (#92938) The JIT has some backwards compatibility for accessing unaligned float fields on ARM32. With physical promotion, we can end up with some new patterns that we didn't handle. Expand the pattern matching to handle a constant address unaligned address. Fix #92382 * runtime-wasm-perf: add triggers for PRs (#92799) * CI: runtime-wasm-perf: add triggers for running on PRs This is useful to prevent perf pipeline from breaking when changes are made in `dotnet/runtime`. * CI: Add run-scenarios-job.yml to list of perf pipeline specific files * [wasm] wasmbrowser - change the default webserver port to 0, to randomly select a port (#92952) …mly select a port * Adding a vectorized implementation of TensorPrimitives.Log (#92960) * Adding a vectorized implementation of TensorPrimitives.Log * Make sure to hit Ctrl+S --------- Co-authored-by: Stephen Toub <stoub@microsoft.com> Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com> Co-authored-by: Larry Ewing <lewing@microsoft.com> Co-authored-by: Ankit Jain <radical@gmail.com> Co-authored-by: karakasa <karakasa@users.noreply.github.com> Co-authored-by: Parker Bibus <parkerbibus@microsoft.com> Co-authored-by: Bruce Forstall <brucefo@microsoft.com> Co-authored-by: Jakob Botsch Nielsen <Jakob.botsch.nielsen@gmail.com> Co-authored-by: Filip W <filip@strathweb.com> Co-authored-by: Badre BSAILA <54767641+pedrobsaila@users.noreply.github.com> Co-authored-by: Jan Vorlicek <janvorli@microsoft.com> Co-authored-by: Zoltan Varga <vargaz@gmail.com> Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Andrey Kudashkin <neyromant@mail.ru> Co-authored-by: Andrey.Kudashkin <Andrey.Kudashkin@russianpost.ru> Co-authored-by: Jan Kotas <jkotas@microsoft.com> Co-authored-by: yowl <scott.waye@hubse.com> Co-authored-by: Jeremy Koritzinsky <jekoritz@microsoft.com> Co-authored-by: Elinor Fung <elfung@microsoft.com> Co-authored-by: Layomi Akinrinade <laakinri@microsoft.com> Co-authored-by: Jan Dupej <109523496+jandupej@users.noreply.github.com> Co-authored-by: Aaron Robinson <arobins@microsoft.com> Co-authored-by: Lakshan Fernando <lakshanf@hotmail.com> Co-authored-by: Ivan Diaz Sanchez <ivdiazsa@microsoft.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> Co-authored-by: Egor Bogatov <egorbo@gmail.com> Co-authored-by: Sven Boemer <sbomer@gmail.com> Co-authored-by: Hazel <contact@tacosontitan.com> Co-authored-by: Miha Zupan <mihazupan.zupan1@gmail.com> Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Kevin Jones <kevin@vcsjones.com> Co-authored-by: Levi Broderick <GrabYourPitchforks@users.noreply.github.com> Co-authored-by: lateralusX <lateralusx.github@gmail.com> Co-authored-by: Ilona Tomkowicz <32700855+ilonatommy@users.noreply.github.com> Co-authored-by: Tarcisio <60912483+tcortega@users.noreply.github.com> Co-authored-by: Jackson Schuster <36744439+jtschuster@users.noreply.github.com> Co-authored-by: Buyaa Namnan <bunamnan@microsoft.com> Co-authored-by: Andrew Au <andrewau@microsoft.com> Co-authored-by: SingleAccretion <62474226+SingleAccretion@users.noreply.github.com> Co-authored-by: Radek Zikmund <32671551+rzikm@users.noreply.github.com> Co-authored-by: Radek Doulik <radek.doulik@gmail.com> Co-authored-by: SwapnilGaikwad <swapnil.gaikwad@arm.com> Co-authored-by: Kunal Pathak <Kunal.Pathak@microsoft.com> Co-authored-by: Tymoteusz Wenerski <tymoteusz.wenerski@gmail.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Aman Khalid <amankhalid@microsoft.com> Co-authored-by: Mitchell Hwang <16830051+mdh1418@users.noreply.github.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: Koundinya Veluri <kouvel@users.noreply.github.com> Co-authored-by: Filip Navara <filip.navara@gmail.com> Co-authored-by: Andy Ayers <andya@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Tomáš Rylek <trylek@microsoft.com> Co-authored-by: Djuradj Kurepa <91743470+dkurepa@users.noreply.github.com> Co-authored-by: t-mustafin <66252296+t-mustafin@users.noreply.github.com> Co-authored-by: Tomasz Sowiński <tomeksowi@gmail.com> Co-authored-by: Marek Fišera <mara@neptuo.com> Co-authored-by: skyoxZ <skyoxZ@qq.com> Co-authored-by: Pavel Savara <pavel.savara@gmail.com> Co-authored-by: Vitek Karas <10670590+vitek-karas@users.noreply.github.com> Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: David Mason <davmason@microsoft.com> Co-authored-by: Andy Gocke <angocke@microsoft.com> Co-authored-by: Milos Kotlar <kotlarmilos@gmail.com> Co-authored-by: Egor <egorbo@Egors-MacBook-Pro.local> Co-authored-by: Tanner Gooding <tagoo@outlook.com>

layomia added the area-Extensions-Configuration label Jul 27, 2023

layomia added this to the 8.0.0 milestone Jul 27, 2023

layomia requested review from eiriktsarpalis and tarekgh July 27, 2023 18:13

layomia self-assigned this Jul 27, 2023

eiriktsarpalis reviewed Jul 27, 2023

View reviewed changes

src/libraries/Common/src/SourceGenerators/TypeRef.cs Outdated Show resolved Hide resolved

eiriktsarpalis reviewed Jul 27, 2023

View reviewed changes

src/libraries/Common/src/SourceGenerators/TypeRef.cs Outdated Show resolved Hide resolved

layomia force-pushed the binder-gen-perf-parsing branch 3 times, most recently from 589ad73 to cb459ca Compare August 6, 2023 23:34

build-analysis bot mentioned this pull request Aug 7, 2023

Intermittent build failure in AfterSourceBuild: "Could not write state file" #76488

Open

layomia force-pushed the binder-gen-perf-parsing branch 3 times, most recently from ca663eb to 40a64d2 Compare August 8, 2023 06:29

ericstj reviewed Aug 8, 2023

View reviewed changes

...libraries/System.Text.RegularExpressions/gen/System.Text.RegularExpressions.Generator.csproj Outdated Show resolved Hide resolved

layomia force-pushed the binder-gen-perf-parsing branch 3 times, most recently from 142fc78 to d90b278 Compare August 9, 2023 00:33

layomia requested review from eiriktsarpalis and eerhardt August 9, 2023 00:45

layomia marked this pull request as ready for review August 9, 2023 00:46

layomia requested a review from jeffhandley as a code owner August 9, 2023 00:46

eerhardt reviewed Aug 9, 2023

View reviewed changes

...tion.Binder/tests/SourceGenerationTests/Baselines/ConfigurationBinder/GetValue.generated.txt Outdated Show resolved Hide resolved