Move portable RID graph into runtime and clean-up #92211
Conversation
dotnet-issue-labeler
bot
added
the
needs-area-label
ViktorHofer
force-pushed
the
MicrosoftNETCorePlatformsRID
branch
from
b421a72
to
51e0d4e
Compare
1. Move portable RID graph into runtime 2. Allow updates to both the non-portable and portable RID graphs under source build. 3. Clean-up project and remove hacks
ViktorHofer
force-pushed
the
MicrosoftNETCorePlatformsRID
branch
from
51e0d4e
to
2f5eb8f
Compare
| - `OmitRIDs`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be referenced nor defined. | ||
| - `OmitRIDDefinitions`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be defined by this RuntimeGroup, but will be referenced: useful in case some other RuntimeGroup (or runtime.json template) defines them. | ||
| - `OmitRIDReferences`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will be defined but not referenced by this RuntimeGroup. | ||
| The RID graphs are frozen and shouldn't be updated anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID. |
There was a problem hiding this comment.
| The RID graphs are frozen and shouldn't be updated anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID. | |
| The RID graphs should be only updated with new base OSes. The RID graphs shouldn't be updated with new OS flavor- and version-specific RIDs anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID. |
There was a problem hiding this comment.
Makes sense, thanks. Do we somewhere already capture that we don't need to update the RID graph for building an unknown Unix distribution as part of source build? cc @tmds
There was a problem hiding this comment.
The RID graphs shouldn't be updated with new OS flavor- and version-specific RIDs anymore.
More specifically: the build automatically takes care of adding the non-portable rid for the OS during source-build.
There was a problem hiding this comment.
The one difficulty that I don't think many are aware of is that the libraries TFM infrastructure reads from the RID graph (portable by default) to calculate the compatibility mapping when referencing other projects. That's what powers TFMs like net8.0-unix.
As the Microsoft.NETCore.Platforms package builds too late, we can't leverage the source built updated RID graph. Therefore we still require manual additions to the RID graph just for the sake of being able to target RIDs in our libraries.
This problem would go away if libraries would use OS runtime detection instead of at build time. Looking at the haiku PR, there aren't just a handful of these libraries. Collapsing build time platforms is general goodness as it reduces the build graph (makes evaluation, restore, build, pack, ... faster).
There was a problem hiding this comment.
As the Microsoft.NETCore.Platforms package builds too late, we can't leverage the source built updated RID graph. Therefore we still require manual additions to the RID graph just for the sake of being able to target RIDs in our libraries.
At least for .NET 8, it doesn't seem necessary for the non-portable rid that gets source-built to be in the graph.
This problem would go away if libraries would use OS runtime detection instead of at build time. Looking at the haiku PR, there aren't just a handful of these libraries. Collapsing build time platforms is general goodness as it reduces the build graph (makes evaluation, restore, build, pack, ... faster).
Yea, we shouldn't create rid specific libs when they are not needed. I imagine most of them exists for rid specific assets, so they are needed.
Haiku is different from adding the non-portable rid like we're doing with source-build.
haiku a direct base of unix. It's more similar to a portable rid than to a non-portable rid.
Maybe some changes are possible to reduce the verbosity/repetition.
There was a problem hiding this comment.
Yea, we shouldn't create rid specific libs when they are not needed. I imagine most of them exists for rid specific assets, so they are needed.
System.Data.Odbc is a prime example of a library that targets way too many platforms. I would imagine that all the Unix derived platforms could be collapsed into a single build by using runtime checks (which would then get optimized away when the linker is used in the consuming app).
There was a problem hiding this comment.
System.Data.Odbc is a prime example of a library that targets way too many platforms. I would imagine that all the Unix derived platforms could be collapsed into a single build by using runtime checks (which would then get optimized away when the linker is used in the consuming app).
I took a quick look. It seems the split is mostly for finding the platform specific name of the Odbc32 library. Perhaps we can do something with NativeLibrary instead. @ViktorHofer may be you can create an issue to investigate it further.
There was a problem hiding this comment.
Exactly. ODBC is tracked via #53900 but there are probably also inbox libraries that would benefit from a build platform simplification.
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
|
@dsplaisted @agocke @elinor-fung can you please review the changes? Is that we want? @tmds do we actually want to update both RID graphs for source build or would the portable one be enough? |
I'm not sure about that anymore as |
|
It's true that we will need to keep the non-portable RID graph as it was in .NET 8 around, as it needs to be included in the I must admit I don't know where SDK gets the RID graph from when it generates a self-contained app. Wherever that is, we probably need to keep it there for now. |
This doesn't cause something to get included/excluded. It's a choice between the legacy graph and the new portable graph. If you need to set this, then something is broken when using the new portable graph. Do you know what opting into the legacy graph is supposed to fix? |
The SDK gets it from runtime via the Microsoft.NETCore.Platforms non-shipping transport package (the project file that is changed in this PR). |
Even .NET 8+ apps can opt into the "old" behavior where the host reads the RID graph from |
It sounded like we are considering removing that switch in .NET 9 or future. I assume that the need for keeping the non portable RID graph alive would then go away? |
That's a good point. I must admit don't know if we already made a plan to deprecate the old RID graph. @elinor-fung would know best. |
#90000 was created for removing the legacy rid graph. |
|
Looks like I'm out of the loop then. If we want to remove the RID graph from the framework I would expect test failures in the host tests (that was the reason I added the change to the framework build). |
We don't have a specific release set for removing the switch. I think .NET 9 is a bit too soon though - my thought had been having at least a release of warning that the switch would go away. But yes, once the switch is removed, the non-portable graph could go way. |
|
If the legacy graph is maintained for .NET 9, then we should to update it as part of source-build too, to maintain the legacy behavior. |
Agreed. Perfect, that's already the current state of the PR. Just waiting for some reviews. |
Co-authored-by: Andy Gocke <angocke@microsoft.com>
ViktorHofer
added
area-Infrastructure-libraries
and removed
needs-area-label
|
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries Issue DetailsSee discussion in #86391 for more details
cc @elinor-fung @akoeplinger @jkotas @am11
|
|
@dsplaisted can you please follow-up on deleting the non portable RID graph from the sdk repo (probably including the task) and consuming it from the Microsoft.NETCore.Platforms transport package? edit tracked by dotnet/sdk#35750 |
commit a5b75b8 Author: Jakob Botsch Nielsen <Jakob.botsch.nielsen@gmail.com> Date: Wed Sep 20 22:04:58 2023 +0200 JIT: Fix invalid containment of vector broadcasts (dotnet#92333) The containment checks for vector broadcasts were missing a size check, meaning that a uint broadcast could contain a ubyte/ushort indirection. That would lead to out-of-bounds reads. Fix dotnet#83387 commit 614d864 Author: Stephen Toub <stoub@microsoft.com> Date: Wed Sep 20 15:56:37 2023 -0400 Use Utf8JsonWriterCache in JsonNode.To{Json}String (dotnet#92358) commit c0b5150 Author: Andy Gocke <angocke@microsoft.com> Date: Wed Sep 20 12:46:37 2023 -0700 Bring back CopyOutputSymbolsToPublishDirectory (dotnet#92315) I accidentally removed this property from AOT compilation when adding support for Mac dsym bundles. This change re-enables support for suppressing debugging symbols in the output. Fixes dotnet#92188 commit b4be77b Author: Kunal Pathak <Kunal.Pathak@microsoft.com> Date: Wed Sep 20 10:17:22 2023 -0700 Update the assert for BlendVariable (dotnet#92183) * Update the assert for BlendVariable * Add test cases * Add Sse41.IsSupported check commit e235aef Author: Miha Zupan <mihazupan.zupan1@gmail.com> Date: Wed Sep 20 17:45:31 2023 +0200 Set severity of rule CA1870 to warning (dotnet#92135) * Set severity of rule CA1870 to warning * Replace one more usage in nativeaot corelib * Set severity for tests as well * pragma disable the rule in nativeaot's reflection impl commit 901f780 Author: Ilona Tomkowicz <32700855+ilonatommy@users.noreply.github.com> Date: Wed Sep 20 17:45:01 2023 +0200 [wasm][debugger] Add tests for indexing by object schema (dotnet#92268) * Indexing with object: works. * Update expected line numbers. commit d6ff465 Author: Johan Lorensson <lateralusx.github@gmail.com> Date: Wed Sep 20 17:24:39 2023 +0200 Add missing case for constrained gsharedvt call. (dotnet#92338) dotnet@1b788f4 added a new value to our MonoRgctxInfoType enum type, but appears that all cases where not full adjusted. Running System.Buffers tests in full AOT hits the assert in info_equal about missing case, https://github.com/dotnet/runtime/blob/0dc5903679606b072adac70a268cdb77d1147b3e/src/mono/mono/mini/mini-generic-sharing.c#L2908. This commit adds the new enum value and align handling similar to other cases added by that commit. commit 36ab905 Author: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com> Date: Wed Sep 20 09:51:37 2023 -0500 Update dependencies from https://github.com/dotnet/installer build 20230919.3 (dotnet#92339) Microsoft.Dotnet.Sdk.Internal From Version 9.0.100-alpha.1.23464.17 -> To Version 9.0.100-alpha.1.23469.3 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> commit 32c3355 Author: Stephen Toub <stoub@microsoft.com> Date: Wed Sep 20 09:42:27 2023 -0400 Fix downlevel build break in TensorPrimitives (dotnet#92269) * Fix downlevel build break in TensorPrimitives * Make net6.0 Tensors use ns2.0 implementation * Add net6.0 and net7.0 to Tensors temporarily since those are shipping in 8.0 branch * Only build net6.0 and net7.0 Tensors when not in source-build --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com> commit e8c3052 Author: Matt Thalman <mthalman@microsoft.com> Date: Wed Sep 20 07:45:55 2023 -0500 Update Newtonsoft.Json from 13.0.1 to 13.0.3 (dotnet#92298) commit b4912a7 Author: Zoltan Varga <vargaz@gmail.com> Date: Wed Sep 20 08:12:52 2023 -0400 [wasi] Fix llvm target triple. (dotnet#92256) commit 0dc5903 Author: Artur Zgodziński <bivaro@gmail.com> Date: Wed Sep 20 11:45:46 2023 +0100 Fix trimming of DebuggerDisplay with Name (dotnet#92191) The `Name` and `Type` property of the `DebuggerDisplay` attribute accepts the same format string as its `Value` property, but does not prevent trimming members it references. Thanks to this fix, members referenced by any of these two properties are not trimmed and can be displayed by a debugger. commit 521e1e6 Author: Marie Píchová <11718369+ManickaP@users.noreply.github.com> Date: Wed Sep 20 12:28:18 2023 +0200 [QUIC] Throw ODE if connection/listener is disposed (dotnet#92215) * AcceptConnection/StreamAsync now throw ODE in case the listener/connection was stopped by DisposeAsync. * Fix exception type and make behavior stable for disposal commit d411f50 Author: Stephen Toub <stoub@microsoft.com> Date: Wed Sep 20 06:24:58 2023 -0400 Avoid unnecessary array allocation in JsonHelpers.Utf8GetString on netstandard (dotnet#92304) commit 5883b72 Author: Tarek Mahmoud Sayed <tarekms@microsoft.com> Date: Tue Sep 19 19:52:38 2023 -0700 Fix options Validation with objects have indexers (dotnet#92309) commit fcf7b11 Author: Sven Boemer <sbomer@gmail.com> Date: Tue Sep 19 17:51:32 2023 -0700 Prevent restoring illink for native-binplace.proj (dotnet#92289) Fixes dotnet#92194. The reference to illink from `native-binplace.proj`, built as a reference of `build-native.proj`, was hitting a nuget bug with static graph restore. The bug seems to be specific to something about the project file (maybe the language-specific targets, since `native-binplace.proj` imports the `Microsoft.NET.Sdk`, but doesn't have a `csproj` extension). Fixed by explicitly marking this as not a source project, which will prevent the import of illink.targets. commit b049f42 Author: Egor Bogatov <egorbo@gmail.com> Date: Wed Sep 20 01:39:30 2023 +0200 Fix optSwitchConvert (dotnet#92249) Co-authored-by: Egor <egorbo@Egors-MacBook-Pro.local> commit 41a8e39 Author: Tanner Gooding <tagoo@outlook.com> Date: Tue Sep 19 15:09:19 2023 -0700 Ensure VN handles both forms of the xarch shift instructions for SIMD (dotnet#91601) commit 3b9b4fd Author: Viktor Hofer <viktor.hofer@microsoft.com> Date: Tue Sep 19 23:29:29 2023 +0200 Move portable RID graph into runtime and clean-up (dotnet#92211) * Move portable RID graph into runtime and clean-up 1. Move portable RID graph into runtime 2. Allow updates to both the non-portable and portable RID graphs under source build. 3. Clean-up project and remove hacks * Update README and delete test * Fix RID graph update when the key already exists * Update src/libraries/Microsoft.NETCore.Platforms/readme.md Co-authored-by: Jan Kotas <jkotas@microsoft.com> * Update src/libraries/Microsoft.NETCore.Platforms/readme.md Co-authored-by: Andy Gocke <angocke@microsoft.com> --------- Co-authored-by: Jan Kotas <jkotas@microsoft.com> Co-authored-by: Andy Gocke <angocke@microsoft.com> commit 1185d19 Author: Tanner Gooding <tagoo@outlook.com> Date: Tue Sep 19 13:41:15 2023 -0700 Don't generate AddMask as it requires more explicit consideration of semantics (dotnet#92282) commit a7cafec Author: Carlos Sánchez López <1175054+carlossanlop@users.noreply.github.com> Date: Tue Sep 19 12:14:24 2023 -0700 [main] Bump Microsoft.Private.IntelliSense package version (dotnet#92255) commit 094801e Author: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com> Date: Tue Sep 19 12:13:49 2023 -0700 [main] Update dependencies from dotnet/runtime dotnet/emsdk dotnet/hotreload-utils dotnet/cecil dotnet/sdk dotnet/source-build-reference-packages (dotnet#92175) * Update dependencies from https://github.com/dotnet/emsdk build 20230915.3 Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport From Version 9.0.0-alpha.1.23457.3 -> To Version 9.0.0-alpha.1.23465.3 * Update dependencies from https://github.com/dotnet/sdk build 20230915.37 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23465.4 -> To Version 9.0.100-alpha.1.23465.37 * Update dependencies from https://github.com/dotnet/emsdk build 20230915.3 Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport From Version 9.0.0-alpha.1.23457.3 -> To Version 9.0.0-alpha.1.23465.3 * Update dependencies from https://github.com/dotnet/sdk build 20230916.1 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23465.4 -> To Version 9.0.100-alpha.1.23466.1 * Update dependencies from https://github.com/dotnet/emsdk build 20230915.3 Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport From Version 9.0.0-alpha.1.23457.3 -> To Version 9.0.0-alpha.1.23465.3 * Update dependencies from https://github.com/dotnet/sdk build 20230918.4 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23465.4 -> To Version 9.0.100-alpha.1.23468.4 * Update dependencies from https://github.com/dotnet/runtime build 20230916.6 Microsoft.DotNet.ILCompiler , Microsoft.NET.ILLink.Tasks , Microsoft.NET.Sdk.IL , Microsoft.NETCore.App.Runtime.win-x64 , Microsoft.NETCore.ILAsm , runtime.native.System.IO.Ports , System.Text.Json From Version 9.0.0-alpha.1.23460.2 -> To Version 9.0.0-alpha.1.23466.6 * Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20230915.1 Microsoft.SourceBuild.Intermediate.source-build-reference-packages From Version 8.0.0-alpha.1.23457.1 -> To Version 9.0.0-alpha.1.23465.1 * Update dependencies from https://github.com/dotnet/emsdk build 20230915.3 Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport From Version 9.0.0-alpha.1.23457.3 -> To Version 9.0.0-alpha.1.23465.3 * Update dependencies from https://github.com/dotnet/hotreload-utils build 20230918.2 Microsoft.DotNet.HotReload.Utils.Generator.BuildTool From Version 8.0.0-alpha.0.23461.1 -> To Version 8.0.0-alpha.0.23468.2 * Update dependencies from https://github.com/dotnet/cecil build 20230918.2 Microsoft.DotNet.Cecil From Version 0.11.4-alpha.23461.1 -> To Version 0.11.4-alpha.23468.2 * Update dependencies from https://github.com/dotnet/sdk build 20230918.31 Microsoft.DotNet.ApiCompat.Task From Version 9.0.100-alpha.1.23465.4 -> To Version 9.0.100-alpha.1.23468.31 * Update dependencies from https://github.com/dotnet/source-build-reference-packages build 20230918.3 Microsoft.SourceBuild.Intermediate.source-build-reference-packages From Version 8.0.0-alpha.1.23457.1 -> To Version 9.0.0-alpha.1.23468.3 --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> commit 9bd0e0d Author: Jeremy Koritzinsky <jekoritz@microsoft.com> Date: Tue Sep 19 12:11:38 2023 -0700 Remove "Is supported on this TFM" logic from marshalling generators and instead handle it during factory construction (dotnet#91768) Co-authored-by: Jackson Schuster <36744439+jtschuster@users.noreply.github.com> commit 17eff3b Author: Andy Ayers <andya@microsoft.com> Date: Tue Sep 19 11:37:48 2023 -0700 JIT: generalize assert to handle SIMD64 (dotnet#92235) Fixes dotnet#91799. commit 67dbbeb Author: Andy Ayers <andya@microsoft.com> Date: Tue Sep 19 11:30:38 2023 -0700 JIT: add missing xarch RMW case (dotnet#92252) Handle the case where we're indirectly updating a local with a value that is not a constant. Fixes dotnet#92218.
ghost
locked as resolved and limited conversation to collaborators
See discussion in #86391 for more details
cc @elinor-fung @akoeplinger @jkotas @am11