Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move portable RID graph into runtime and clean-up #92211

Merged
merged 6 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -471,4 +471,11 @@
<CustomBeforeNoTargets>$(RepositoryEngineeringDir)NoTargetsSdk.BeforeTargets.targets</CustomBeforeNoTargets>
<CustomAfterTraversalTargets>$(RepositoryEngineeringDir)TraversalSdk.AfterTargets.targets</CustomAfterTraversalTargets>
</PropertyGroup>

<PropertyGroup>
<!-- Keep in sync with outputs defined in Microsoft.NETCore.Platforms.csproj. -->
<BundledRuntimeIdentifierGraphFile>$([MSBuild]::NormalizePath('$(ArtifactsBinDir)', 'Microsoft.NETCore.Platforms', 'runtime.json'))</BundledRuntimeIdentifierGraphFile>
<BundledRuntimeIdentifierGraphFile Condition="!Exists('$(BundledRuntimeIdentifierGraphFile)')">$([MSBuild]::NormalizePath('$(LibrariesProjectRoot)', 'Microsoft.NETCore.Platforms', 'src', 'runtime.json'))</BundledRuntimeIdentifierGraphFile>
</PropertyGroup>

</Project>
1 change: 1 addition & 0 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -216,4 +216,5 @@
<InstallersRelativePath>Runtime/$(SharedFrameworkNugetVersion)/</InstallersRelativePath>
</PropertyGroup>
</Target>

</Project>
14 changes: 0 additions & 14 deletions eng/AvoidRestoreCycleOnSelfReference.targets

This file was deleted.

1 change: 0 additions & 1 deletion src/libraries/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,6 @@
<Import Project="$(RepositoryEngineeringDir)slngen.targets" Condition="'$(IsSlnGen)' == 'true'" />

<Import Project="$(RepositoryEngineeringDir)illink.targets" Condition="'$(IsSourceProject)' == 'true' or '$(ExplicitlyImportCustomILLinkTargets)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)AvoidRestoreCycleOnSelfReference.targets" Condition="'$(AvoidRestoreCycleOnSelfReference)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)nativeSanitizers.targets" />

<ItemGroup Condition="'$(UseTargetFrameworkPackage)' != 'false'">
Expand Down
Original file line number Diff line number Diff line change
@@ -1,63 +1,24 @@
Microsoft Visual Studio Solution File, Format Version 12.00
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TestUtilities", "..\Common\tests\TestUtilities\TestUtilities.csproj", "{39BCA125-321F-490F-AD4E-28DCB4406969}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.NETCore.Platforms", "src\Microsoft.NETCore.Platforms.csproj", "{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.NETCore.Platforms.Tests", "tests\Microsoft.NETCore.Platforms.Tests.csproj", "{0C60F372-5C73-4BFA-9B91-5659C88F9750}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ComInterfaceGenerator", "..\System.Runtime.InteropServices\gen\ComInterfaceGenerator\ComInterfaceGenerator.csproj", "{89E78703-8B4C-4911-A4E6-794E7DC4D581}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibraryImportGenerator", "..\System.Runtime.InteropServices\gen\LibraryImportGenerator\LibraryImportGenerator.csproj", "{45449066-3A31-43E5-B705-20D667080A23}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microsoft.Interop.SourceGeneration", "..\System.Runtime.InteropServices\gen\Microsoft.Interop.SourceGeneration\Microsoft.Interop.SourceGeneration.csproj", "{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{F96FBD24-3BB3-4D02-9884-4D90F94DD3C0}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "gen", "gen", "{E8010E1D-FDAF-481D-AA34-3B115B667E4B}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{39BCA125-321F-490F-AD4E-28DCB4406969}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Debug|Any CPU.Build.0 = Debug|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Release|Any CPU.ActiveCfg = Release|Any CPU
{39BCA125-321F-490F-AD4E-28DCB4406969}.Release|Any CPU.Build.0 = Release|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11}.Release|Any CPU.Build.0 = Release|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0C60F372-5C73-4BFA-9B91-5659C88F9750}.Release|Any CPU.Build.0 = Release|Any CPU
{89E78703-8B4C-4911-A4E6-794E7DC4D581}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{89E78703-8B4C-4911-A4E6-794E7DC4D581}.Debug|Any CPU.Build.0 = Debug|Any CPU
{89E78703-8B4C-4911-A4E6-794E7DC4D581}.Release|Any CPU.ActiveCfg = Release|Any CPU
{89E78703-8B4C-4911-A4E6-794E7DC4D581}.Release|Any CPU.Build.0 = Release|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Debug|Any CPU.Build.0 = Debug|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Release|Any CPU.ActiveCfg = Release|Any CPU
{45449066-3A31-43E5-B705-20D667080A23}.Release|Any CPU.Build.0 = Release|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{39BCA125-321F-490F-AD4E-28DCB4406969} = {E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}
{0C60F372-5C73-4BFA-9B91-5659C88F9750} = {E4827496-6F39-4CA0-8F4A-ACDE9DFEBE5C}
{BFFF96CC-06AA-4291-9F93-3E77F23DBB11} = {F96FBD24-3BB3-4D02-9884-4D90F94DD3C0}
{89E78703-8B4C-4911-A4E6-794E7DC4D581} = {E8010E1D-FDAF-481D-AA34-3B115B667E4B}
{45449066-3A31-43E5-B705-20D667080A23} = {E8010E1D-FDAF-481D-AA34-3B115B667E4B}
{2E3568B1-EC27-4F02-BC0E-71DD3FD7735B} = {E8010E1D-FDAF-481D-AA34-3B115B667E4B}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {E946A528-C3E7-48EC-AD6D-AE84ED2B11AC}
Expand Down
80 changes: 2 additions & 78 deletions src/libraries/Microsoft.NETCore.Platforms/readme.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Runtime IDs
The package `Microsoft.NETCore.Platforms` defines the runtime identifiers (RIDs) used by .NET packages to represent runtime-specific assets in NuGet packages.
The `Microsoft.NETCore.Platforms` transport package contains the portable and non-portable runtime identifier graph files for redistribution in the dotnet/sdk repository.

## What is a RID?
A RID is an opaque string that identifies a platform. RIDs have relationships to other RIDs by "importing" the other RID. In that way a RID is a directed graph of compatible RIDs.
Expand Down Expand Up @@ -76,80 +76,4 @@ runtimes/win/lib/netstandard1.0/foo.dll
When resolving for netstandard1.5/win7-x64 will select `lib/netstandard1.5/foo.dll` for the compile asset and `runtimes/win/lib/netstandard1.0/foo.dll` for the runtime asset.

## Adding new RIDs

### Why do I need to add a new RID?
NuGet's extensibility mechanism for platform-specific assets requires a RID be defined for any platform that needs assets specific to that platform. Unlike TFMs, which have a known relationship in NuGet (eg net4.5 is compatible with net4.0), RIDs are opaque strings which NuGet knows nothing about. The definition and relationship of RIDs comes solely from the `runtime.json` files within the root of the packages referenced by the project.
As such, whenever we want to put a new RID in a project.json in order to get assets specific for that RID we have to define the rid in some package. Typically that package is `Microsoft.NETCore.Platforms` if the RID is "official". If you'd like to prototype you can put the RID in any other package and so long as that package is referenced you can use that RID.

### Do I really need to add a new RID?
If you're prototyping on a platform that is compatible with an existing platform then you can reuse the RID for that existing platform. New RIDs are only needed when an asset needs to be different on a particular platform.

`Microsoft.NETCore.Platforms` attempts to define all RIDs that packages may need, and as such will define RIDs for platforms that we don't actually cross compile for. This is to support higher-level packages, 3rd party packages, that may need to cross-compile for that RID.

### Adding a new OS
Add a new `RuntimeGroup` item in `runtimeGroups.props`.

For example:
```xml
<RuntimeGroup Include="myLinuxDistro">
<Parent>linux</Parent>
<Architectures>x86;x64;arm</Architectures>
<Versions>42.0;43.0</Versions>
</RuntimeGroup>
```

This will create a new RID for `myLinuxDistro` where `myLinuxDistro` should be the string used for the `ID=` value in the `/etc/os-release` file.

Whenever modifying the `runtimeGroups.props` make sure to pack the project via the `dotnet pack` command and inspect if the generated package contains the desired changes.

RuntimeGroup items have the following format:
- `Identity`: the base string for the RID, without version architecture, or qualifiers.
- `Parent`: the base string for the parent of this RID. This RID will be imported by the baseRID, architecture-specific, and qualifier-specific RIDs (with the latter two appending appropriate architecture and qualifiers).
- `Versions`: A list of strings delimited by semi-colons that represent the versions for this RID.
- `TreatVersionsAsCompatible`: Default is true. When true, version-specific RIDs will import the previous version-specific RID in the Versions list, with the first version importing the version-less RID. When false all version-specific RIDs will import the version-less RID (bypassing previous version-specific RIDs)
- `OmitVersionDelimiter`: Default is false. When true no characters will separate the base RID and version (EG: win7). When false a '.' will separate the base RID and version (EG: osx.10.12).
- `ApplyVersionsToParent`: Default is false. When true, version-specific RIDs will import version-specific Parent RIDs similar to is done for architecture and qualifier (see Parent above).
- `Architectures`: A list of strings delimited by semi-colons that represent the architectures for this RID.
- `AdditionalQualifiers`: A list of strings delimited by semi-colons that represent the additional qualifiers for this RID. Additional qualifers do not stack, each only applies to the qualifier-less RIDs (so as not to cause combinatorial exponential growth of RIDs).

### Adding a new version to an existing OS
Find the existing `RuntimeGroup` in `runtimeGroups.props` and add the version to the list of `Versions`, separated by a semi-colon.

If the version you are adding needs to be treated as not-compatible with previous versions and the `RuntimeGroup` has not set `TreatVersionsAsCompatible`=`false` then you may create a new `RuntimeGroup` to represent the new compatibility band.

### Checking your work
After making a change to `runtimeGroups.props` you can examine the resulting changes in `runtime.json` and `runtime.compatibility.json`.

`runtime.json` is the graph representation of the RIDs and is what ships in the package.

`runtime.compatibility.json` is a flattened version of the graph that shows the RID precedence for each RID in the graph.

### Version compatibility
Version compatibility is represented through imports. If a platform is considered compatible with another version of the same platform, or a specific version of another platform, then it can import that platform. This permits packages to reuse assets that were built for the imported platform on the compatible platform. Compatibility here is a bit nebulous because inevitably different platforms will have observable differences that can cause compatibility problems. For the purposes of RIDs we'll try to represent compatibility as versions of a platform that are explicitly advertised as being compatible with a previous version and/or another platform and don't have any known broad breaking changes. It is usually better to opt to treat platforms as compatible since that enables the scenario of building an asset for a particular version and using that in future versions, otherwise you force people to cross-compile for all future versions the moment they target a specific version.

## Appendix : details of RID graph generation

### Naming convention
We use the following convention in all newly-defined RIDs. Some RIDs (win7-x64, win8-x64) predate this convention and don't follow it, but all new RIDs should follow it.
`[os name].[version]-[architecture]-[additional qualifiers]`, for example `osx.10.10-x64` or `ubuntu.14.04-x64`
- `[os name]` can contain any characters other than `.`
- `[version]` can contain any characters other than `-`. Typically a numeric version like 14.04 or 10.0.
- `[architecture]` can contain any characters other than `-`. Typically: `x86`, `x64`, `arm`, `arm64`
- `[additional qualifiers]` can be things like `aot`. Used to further differentiate different platforms.

For all of these we strive to make them something that can be uniquely discoverable at runtime, so that a RID may be computed from an executing application. As such these properties should be derivable from `/etc/os-release` or similar platform APIs / data.

### Import convention
Imports should be used when the added RID is considered compatible with an existing RID.

1. Architecture-specific RIDs should first import the architecture-less RID. EG: `osx.10.11-x64` should first import `osx.10.11`.
2. Architecture-specific RIDs that are compatible with a previous version RID for the same OS should then import the previous version, architecture specific RID. EG: `osx.10.11-x64` should then import `osx.10.10-x64`. If there is no earlier compatible/supported version, then a versionless RID should be imported. EG: `osx.10.10-x64` should import `osx-x64`.
3. Architecture-less RIDs that are compatible with a previous version RID for the same OS should then import the previous version, architecture neutral RID. EG: `osx.10.11` should import `osx.10.10`. If there is no earlier compatible/supported version, then a versionless RID should be imported. EG: `osx.10.10` should import `osx`.
4. Version-less RIDs should import an OS category. EG: `osx-x64` should import `unix-x64`, `osx` should import `unix`.

### Advanced RuntimeGroup metadata
The following options can be used under special circumstances but break the normal precedence rules we try to establish by generating the RID graph from common logic. These options make it possible to create a RID fallback chain that doesn't match the rest of the RIDs and therefore is hard for developers/package authors to reason about. Only use these options for cases where you know what you are doing and have carefully reviewed the resulting RID fallbacks using the CompatibliltyMap.

- `OmitRIDs`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be referenced nor defined.
- `OmitRIDDefinitions`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will not be defined by this RuntimeGroup, but will be referenced: useful in case some other RuntimeGroup (or runtime.json template) defines them.
- `OmitRIDReferences`: A list of strings delimited by semi-colons that represent RIDs calculated from this RuntimeGroup that should be omitted from the RuntimeGraph. These RIDs will be defined but not referenced by this RuntimeGroup.
The RID graphs are frozen and shouldn't be updated anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The RID graphs are frozen and shouldn't be updated anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID.
The RID graphs should be only updated with new base OSes. The RID graphs shouldn't be updated with new OS flavor- and version-specific RIDs anymore. Build from source automatically adds the non-portable distro RID encoded via the `OutputRID` property into the RID graph which allows build tools to target that RID.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@ViktorHofer ViktorHofer Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks. Do we somewhere already capture that we don't need to update the RID graph for building an unknown Unix distribution as part of source build? cc @tmds

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The RID graphs shouldn't be updated with new OS flavor- and version-specific RIDs anymore.

More specifically: the build automatically takes care of adding the non-portable rid for the OS during source-build.

Copy link
Member Author

@ViktorHofer ViktorHofer Sep 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one difficulty that I don't think many are aware of is that the libraries TFM infrastructure reads from the RID graph (portable by default) to calculate the compatibility mapping when referencing other projects. That's what powers TFMs like net8.0-unix.

As the Microsoft.NETCore.Platforms package builds too late, we can't leverage the source built updated RID graph. Therefore we still require manual additions to the RID graph just for the sake of being able to target RIDs in our libraries.

This problem would go away if libraries would use OS runtime detection instead of at build time. Looking at the haiku PR, there aren't just a handful of these libraries. Collapsing build time platforms is general goodness as it reduces the build graph (makes evaluation, restore, build, pack, ... faster).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the Microsoft.NETCore.Platforms package builds too late, we can't leverage the source built updated RID graph. Therefore we still require manual additions to the RID graph just for the sake of being able to target RIDs in our libraries.

At least for .NET 8, it doesn't seem necessary for the non-portable rid that gets source-built to be in the graph.

This problem would go away if libraries would use OS runtime detection instead of at build time. Looking at the haiku PR, there aren't just a handful of these libraries. Collapsing build time platforms is general goodness as it reduces the build graph (makes evaluation, restore, build, pack, ... faster).

Yea, we shouldn't create rid specific libs when they are not needed. I imagine most of them exists for rid specific assets, so they are needed.

Haiku is different from adding the non-portable rid like we're doing with source-build.
haiku a direct base of unix. It's more similar to a portable rid than to a non-portable rid.

Maybe some changes are possible to reduce the verbosity/repetition.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, we shouldn't create rid specific libs when they are not needed. I imagine most of them exists for rid specific assets, so they are needed.

System.Data.Odbc is a prime example of a library that targets way too many platforms. I would imagine that all the Unix derived platforms could be collapsed into a single build by using runtime checks (which would then get optimized away when the linker is used in the consuming app).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

System.Data.Odbc is a prime example of a library that targets way too many platforms. I would imagine that all the Unix derived platforms could be collapsed into a single build by using runtime checks (which would then get optimized away when the linker is used in the consuming app).

I took a quick look. It seems the split is mostly for finding the platform specific name of the Odbc32 library. Perhaps we can do something with NativeLibrary instead. @ViktorHofer may be you can create an issue to investigate it further.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. ODBC is tracked via #53900 but there are probably also inbox libraries that would benefit from a build platform simplification.

Loading
Loading