source-build on unknown linux distros

[tmds]

For source-build we want to be able to build .NET using source-build on a system where the rid is not known to .NET.

We want this build to succeed, and the rid to be known to .NET.

For the build to succeed, we need to build using another rid which is known and compatible. I think this may be possible already (to some extent) by using the DOTNET_RUNTIME_ID envvar.

For the rid to be known to .NET, it needs to know its place in the graph.

Example pseudo build commands:

Build on Fedora 35 (rid derived from os-release is fedora.35):

./build.sh --build-rid linux --parent-rids fedora,linux

Build on RHEL 8 (rid derived from os-release is rhel.8):

./build.sh --build-rid linux --parent-rids linux

Build on CentOS 8 (rid derived from os-release is centos.8):

./build.sh --build-rid linux --parent-rids rhel.8,linux

Relates to NuGet/Home#5862, dotnet/source-build#297.

cc @ericstj @omajid @crummel

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

Thanks for filing this issue, @tmds ! I think this captures our issues with RID well: we just need a way for a random Linux distro to get an RID automatically, and assign some parents RIDs (mostly linux, but maybe musl too).

cc @dagood @MichaelSimons @dleeapho

[ghost]

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

For source-build we want to be able to build .NET using source-build on a system where the rid is not known to .NET.

We want this build to succeed, and the rid to be known to .NET.

For the build to succeed, we need to build using another rid which is known and compatible. I think this may be possible already (to some extent) by using the DOTNET_RUNTIME_ID envvar.

For the rid to be known to .NET, it needs to know its place in the graph.

Example pseudo build commands:

Build on Fedora 35 (rid derived from os-release is fedora.35):

./build.sh --build-rid linux --parent-rids fedora,linux

Build on RHEL 8 (rid derived from os-release is rhel.8):

./build.sh --build-rid linux --parent-rids linux

Build on CentOS 8 (rid derived from os-release is centos.8):

./build.sh --build-rid linux --parent-rids rhel.8,linux

Relates to NuGet/Home#5862, dotnet/source-build#297.

cc @ericstj @omajid @crummel

Author:	tmds
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[ericstj]

For the build to succeed, we need to build using another rid which is known and compatible. I think this may be possible already (to some extent) by using the DOTNET_RUNTIME_ID envvar.

It would be good to understand if this is sufficient or if you would also need to modify the shared-framework's deps file (where the RID fallbacks live) in the tool shared framework. Doing the latter is possible but grows the complexity of this.

runtime/src/libraries/Microsoft.NETCore.Platforms/pkg/runtimeGroups.props

Lines 248 to 258 in 09a1256

	<Target Name="GenerateRuntimeJson" BeforeTargets="CreatePackage">
	<!-- Generates a Runtime graph using RuntimeGroups and diffs it with the graph described by runtime.json and runtime.compatibility.json
	Specifying UpdateRuntimeFiles=true skips the diff and updates those files.
	The graph can be visualized using the generated dmgl -->
	<MakeDir Directories="$(PackageReportDir)" />
	<GenerateRuntimeGraph RuntimeGroups="@(RuntimeGroupWithQualifiers)"
	RuntimeJson="runtime.json"
	CompatibilityMap="runtime.compatibility.json"
	RuntimeDirectedGraph="$(PackageReportDir)$(Id)$(NuspecSuffix)-runtime.json.dgml"
	UpdateRuntimeFiles="$(UpdateRuntimeFiles)" />
	</Target>

can be used to generate, today it writes to the repo location but that could be overridden. We'd need to expose the right set of options to extend the graph. Then when building make sure that the generated runtime.json was used everywhere instead of the checked in one.

[ericstj]

Also, once you produce such a shared framework, are you OK with nuget restore not knowing about? NuGet still gets the runtimegraph from the Micrsoft.NETCore.Platforms package which the user's package graph brings down. This is what is used for resolving and publishing self-contained apps.

[omajid]

Also, once you produce such a shared framework, are you OK with nuget restore not knowing about?

Is there a fallback here? To something like linux-x64? Or is it the case that if nuget doesn't know about the RID, restore wont work at all for RID-specific assets?

This is what is used for resolving and publishing self-contained apps.

Does that mean self-contained apps wont build/work? What about normal Framework-Dependent apps?

[ericstj]

I may be mistaken. It looks like the SDK did the work to pull out the runtime.json, bundle it in the SDK and provide it to NuGet. Thanks @dsplaisted, @nkolev92
NuGet/NuGet.Client@04166a6
dotnet/sdk@2723589

Assuming we flow the built content of Microsoft.NETCore.Platfroms from sourcebuild up through the SDK/installer then it looks like this should work. NuGet will "merge" the contents of the SDK's runtimegraph with whatever is defined in packages. So even if someone added the public shipped platforms package it would still have your source-build defined RIDs.

[tmds]

It would be good to understand if this is sufficient or if you would also need to modify the shared-framework's deps file (where the RID fallbacks live) in the tool shared framework. Doing the latter is possible but grows the complexity of this.

If it doesn't work already, probably we can pick up DOTNET_RUNTIME_ID in a few more places and avoid the complexity.

[ericstj]

@tmds can you better specify the inputs of what you'd like to see happen?

Describe what needs to be specified, vs what is calculated from the system, and how that is used by the build system. Also talk about how things interact with existing properties / parameters to the build

Interesting considerations:

• Is cross-targeting relevant, or perhaps required? Both by arch and distro. IOW will you always be building on the system that exposes the RID you're trying to add or are you sometimes building on a different RID?
• Does this also need to impact the configuration system in dotnet/runtime libraries for selecting which configurations to build?

[tmds]

I don't know how this would interact with the existing properties/parameters, and configurations. We need to figure that out.

When we use source-build there are two steps:

• First https://github.com/dotnet/source-build is used on a system to generate a tarball. The rid may be not known to .NET so we want to be able to specify a compatible rid (SourceBuildRid).
• Then that tarball is used on another system (TargetRid, e.g. fedora.32-x64) to generate a .NET installer for that system.

We would like the tarball to be usable on a range of target distros. So, the tarball would target something common (BuildRid, like linux-x64).

TargetRid can be a unknown rid to .NET. As part of the build, .NET needs to learn how that rid connects to known rids (TargetRidGraph, e.g. fedora.32 -> fedora -> linux).

For example. We run source-build on linux-x64 to generate a tarball that is usable on linux-arm64, that is used on fedora.32-arm64, with rid graph fedora.32 -> fedora -> linux, gives us these values:

SourceBuildRid=linux-x64
BuildRid=linux-arm64
TargetRid=fedora.32-arm64
TargetRidGrah=fedora,linux

Describe what needs to be specified, vs what is calculated from the system

TargetRid is calculated when building .NET. The other values can be specified.

Is cross-targeting relevant, or perhaps required?

We don't have a need to generate cross-target tarballs. So SourceBuildRid = BuildRid for our use-cases.
We do want the source-build tarball to be usable on multiple platforms (TargetRid is compatible with BuildRid).

@ericstj is it more clear?

cc @MichaelSimons @dleeapho @omajid

[ericstj]

We would like the tarball to be usable on a range of target distros. So, the tarball would target something common

So this means that the tarball cannot have a single RID burned in, it should either have many or we should delay the RID generation until the second phase. I'd prefer the latter since it supports any hypothetical RID we already have a way to "burn in" RIDs: commit to source.

Do you need these distros to "know" about the others. For instance do you (or your customers) need to build a self-contained application that targets fedora.32-arm64 on a some.other.source.built.distro.42-x64. If these all need to "know" about the others then that would require the ability to add many RIDs, rather than just one. It feels like that's not so much of a scenario, if someone wants a RID to be globally known the expectation should be to add that to source-control.

Can you think about this one:

Does this also need to impact the configuration system in dotnet/runtime libraries for selecting which configurations to build?

Are any of new RIDs drastically new, or can you imagine them being covered by this logic:

runtime/Directory.Build.props

Lines 12 to 21 in a4c765d

	<PropertyGroup Label="CalculateTargetOS">
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('OSX'))">OSX</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('FREEBSD'))">FreeBSD</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('NETBSD'))">NetBSD</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('ILLUMOS'))">illumos</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('SOLARIS'))">Solaris</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSUnixLike())">Linux</TargetOS>
	<TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('WINDOWS'))">windows</TargetOS>
	<TargetsMobile Condition="'$(TargetOS)' == 'iOS' or '$(TargetOS)' == 'MacCatalyst' or '$(TargetOS)' == 'tvOS' or '$(TargetOS)' == 'Android' or '$(TargetOS)' == 'Browser'">true</TargetsMobile>
	</PropertyGroup>

[omajid]

Does this also need to impact the configuration system in dotnet/runtime libraries for selecting which configurations to build?

Can you clarify what you mean by "configurations" here? Is it Release/Debug? Whether to use bundled libraries? Or dlopen() vs linking dynamically against system libraries?

My understanding is that source-build provides separate general ways to control this, separate from the exact RID computation/graph-patching.

For example, there's generic logic to use System libraries:

https://github.com/dotnet/source-build/blob/b5b6eaec5c49507361cfe35604559962b93322cf/Directory.Build.props#L30

And can even set up more specific build tweaks at build-time:

https://github.com/dotnet/source-build/blob/b5b6eaec5c49507361cfe35604559962b93322cf/repos/runtime.common.props#L20-L22

IMO, that seems to work fine. We could implicitly infer that an RID of linux-x64 means PortableBuild == true, but that doesn't seem too important right now.

[omajid]

Both by arch and distro. IOW will you always be building on the system that exposes the RID you're trying to add or are you sometimes building on a different RID?

The general workflow for a Linux distribution like Fedora (also true for RHEL) - aside from bootstrapping - is that we always/only build on the exact same RID as the target one, both in terms of distro name and architecture.

There's one additional wrench worth considering, though: what if the system's RID changes. This happens during OS development: a distribution may identify itself as "fedora.35-x64", then it would get branched off and identify itself as "fedora.36-x64" without any other changes in libraries, packages or tools. If we can build .NET just before the RID change, we want to be able to take the SDK we just built and use it to build something just after the RID change.

This scenario currently works, but we are adding to the RID graph via source-build, AFAIK.

[ericstj]

Can you clarify what you mean by "configurations" here?

Sorry, I should avoid the old jargon. I'm talking about how we resolve against the TargetFramework strings (which were previously encoding in Configuration values). We have a separate smaller mapping of compatible TargetOS values that could need to change for new OSes: https://github.com/dotnet/runtime/blob/main/src/libraries/OSGroups.json

From what I see in this discussion I'm in agreement that we can leave this stuff alone and assume it will just work for the case we're examining.

If we can build .NET just before the RID change, we want to be able to take the SDK we just built and use it to build something just after the RID change.

This scenario currently works, but we are adding to the RID graph via source-build, AFAIK.

I think it mostly works, but if you change the RID after you build .NET then I wouldn't expect the RID graph to "know" about that RID. Do you have an example of this, can you point me to the bits that were built on the previous RID then used on the new one? Examine the Microsoft.NETCore.App.deps.json in the shared framework and the RuntimeIdentifierGraph.json in the SDK. Do these understand your new RID? If not then some things won't work (self-contained app targeting new RID, framework dependent app selecting rid-specific assets at runtime).

TargetRid=fedora.32-arm64
TargetRidGrah=fedora,linux

I'm not sure we want to allow ad-hoc parenting rules. All this information already exists in https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.NETCore.Platforms/pkg/runtimeGroups.props
We can just find the closest lower version and use it's architectures/import behavior when generating new graph entries.
I think if we force folks to create entirely new group definitions then we would need them to specify everything that might appear in those runtimeGroup items, which can be rather error prone. It would also rule out the additions over version compatible entries.

I'll put up a PR in arcade that I think can facilitate the type of additions I describe above and will share it here.

[tmds]

it should either have many or we should delay the RID generation until the second phase. I'd prefer the latter since it supports any hypothetical RID we already have a way to "burn in" RIDs: commit to source.

I agree, RID generation should be in the second phase.

Do you need these distros to "know" about the others. For instance do you (or your customers) need to build a self-contained application that targets fedora.32-arm64 on a some.other.source.built.distro.42-x64. If these all need to "know" about the others then that would require the ability to add many RIDs, rather than just one.

I also see no strong need for this.

This scenario currently works, but we are adding to the RID graph via source-build, AFAIK.

We need to do it in the second phase.

if you change the RID after you build .NET then I wouldn't expect the RID graph to "know" about that RID.

I don't expect us to get in that situation. When I upgrade from Fedora 32 to 33, that will change my system rid, but I should also receive updated dotnet packages that are built for Fedora 33 (and therefore use the fedora.33 rid).

I'm not sure we want to allow ad-hoc parenting rules.

I'm fine if we can specify a single parent. Distros can be added in the upstream dotnet/runtime repo.
So this would be ParentRid=fedora (since fedora is a known rid).

[omajid]

Do you have an example of this, can you point me to the bits that were built on the previous RID then used on the new one?

This package was built for Fedora 34: https://koji.fedoraproject.org/koji/buildinfo?buildID=1696922

The RID changed and trying to build the next .NET SDK patch release using that build failed: https://koji.fedoraproject.org/koji/buildinfo?buildID=1708626

But it worked with a hack here: https://koji.fedoraproject.org/koji/buildinfo?buildID=1711461. We didn't patch the SDK we were using to build, just the one we are building.

We know something built with the fedora.34-x64 is being used as the build SDK because the build log (root.log) tells us that this dependency was installed before building the Fedora 35 build:

DEBUG util.py:446:   dotnet-sdk-5.0                          x86_64  5.0.102-2.fc34                         build   49 M

Here's a dockerfile that shows the 2 SDKs and their graphs:

FROM fedora:33

RUN dnf install --setopt tsflags=nodocs --refresh -y \
    fedora-packager \
    tar \
    wget \
    which && \
    dnf clean all -y

RUN mkdir f34-build && \
    cd f34-build && \
    koji download-build -a x86_64 dotnet5.0-5.0.102-2.fc34 && \
    for r in *; do rpm2cpio "$r" | cpio -id; done

RUN mkdir f35-build && \
    cd f35-build && \
    koji download-build -a x86_64 dotnet5.0-5.0.103-2.fc35 && \
    for r in *; do rpm2cpio "$r" | cpio -id; done

CMD find \( -iname 'Microsoft.NETCore.App.deps.json' -or -iname 'RuntimeIdentifierGraph.json' \) -exec echo {} \; -exec cat {} \; 2>/dev/null

I don't expect us to get in that situation. When I upgrade from Fedora 32 to 33, that will change my system rid, but I should also receive updated dotnet packages that are built for Fedora 33 (and therefore use the fedora.33 rid).

Unfortunately, as packagers of source-build, we encounter this once every Fedora (or any other distro, I suspect) release. We build a package, at some point after our build, Fedora's development branch changes it's version (eg, 34 -> 35). We then have to use the last build to build the next one. Users shouldn't get to see it - unless they are running the development branch - but we, packagers, will.

[tmds]

Unfortunately, as packagers of source-build, we encounter this once every Fedora (or any other distro, I suspect) release. We build a package, at some point after our build, Fedora's development branch changes it's version (eg, 34 -> 35). We then have to use the last build to build the next one. Users shouldn't get to see it - unless they are running the development branch - but we, packagers, will.

With what is proposed, both the first phase and second phase should work when the platform rid is not yet known. So I think this will work.

[tmds]

@ericstj are you, or is someone else looking at this?

[ericstj]

Yeah, I'm actually preparing a draft PR right now. Should be available in a hour or so.

[ericstj]

FYI: draft PR #50157
Arcade change to map RID additions into the RID graph honoring rules defined in existing RuntimeGroups: dotnet/arcade#7141

Please review user experience here to make sure it meets your needs.

[omajid]

If I understand the PRs correctly, they infer the current OS's RID, guess the parent RID, and add the current RID to the RID graph (if missing).

Going back to the initial examples:

Build on Fedora 35 (rid derived from os-release is fedora.35):

./build.sh --build-rid linux --parent-rids fedora,linux

The parent-rids part is inferred correctly, and the build-rid is controlled via --portableBuild?

Build on RHEL 8 (rid derived from os-release is rhel.8):

./build.sh --build-rid linux --parent-rids linux

Same?

Build on CentOS 8 (rid derived from os-release is centos.8):

./build.sh --build-rid linux --parent-rids rhel.8,linux

This wouldn't work, would it? It will say centos.8 inherits from centos (which inherits from linux), right? (I am okay with that, just want to make sure I understand this correctly).

[ericstj]

they infer the current OS's RID

Infer was probably bad naming on my part: we infer the place in the graph but the RID is specified. There are already mechanisms today for determining what RID is being built. It can be specified or inferred. We leverage these rather than introducing anything new.

I've also added a property called InferRuntimeIdentifiers which can be use to add an arbitrary set of RIDs, where they belong, in the graph.

guess the parent RID

There is no guessing at parent, this follows the rules which are described in the checked in RuntimeGroups. So rather than let the caller specify a different hierarchy they add the new RID to the existing hierarchy where it belongs. As much as I can I'm trying to make it behave "as if" the change was checked into RuntimeGroups.props.

The above cases you describe should all work correctly. In those cases it looks like you're trying to build the product as if it is a linux-<arch> build but then also add RIDs to the graph for some new version of a known distro. In those cases you would need to specify the InferRuntimeIdentifiers (which I think is behaving like the --parent-rids command). So InferRuntimeIdentifiers=centos.10 would add centos.10 RIDs to the graph as if the change was made here, inheriting from rhel:

runtime/src/libraries/Microsoft.NETCore.Platforms/pkg/runtimeGroups.props

Lines 51 to 57 in 366cd23

	<RuntimeGroup Include="centos">
	<Parent>rhel</Parent>
	<Architectures>x64;arm64</Architectures>
	<Versions>8;9</Versions>
	<ApplyVersionsToParent>true</ApplyVersionsToParent>
	<TreatVersionsAsCompatible>false</TreatVersionsAsCompatible>
	</RuntimeGroup>

I think I'll rename InferRuntimeIdentifiers to AddRuntimeIdentifiers. I'll also refine some of the tests in arcade to better illustrate how this is working. I

[tmds]

build but then also add RIDs to the graph for some new version of a known distro. In those cases you would need to specify the InferRuntimeIdentifiers (which I think is behaving like the --parent-rids command)

--parent-rids was meant to specify the known parent. The GenerateRuntimeGraph implementation is actually deferring this from the input provided. For our use-cases the deferring works fine since the distros we care about are already known.
For general source-build, you may want to support an unknown distro (which, for example, should inherit from linux-musl).

The InferRuntimeIdentifiers/AddRuntimeIdentifiers allows to specify rids to add. For our use-cases, where we build on the same OS that we target, we will want this value to default to what dotnet considers the rid for the build platform (derived from /etc/os-release on Linux).

Independent of the GenerateRuntimeGraph, we also want to be able to override what .NET considers the rid of the build platform. This may be done by using/extending DOTNET_RUNTIME_ID. When the default from the previous paragraph is determined we want to get the actual platform value, and not the overridden value.

[ericstj]

For general source-build, you may want to support an unknown distro (which, for example, should inherit from linux-musl).

Who's going to compute the correct parent? The end user who's running the source build? Might you need to add more than one? Supporting this is pretty trivial: I added a sample here that assumes linux, we can come up with scheme to encode arbitrary parents either as item metadata or some format in the string or a separate parameter, but I want to make sure we understand the end scenario. At some point we'll need to say that we don't need to support more complex additions and those require a change in source: IMHO that point should be driven by our current customer scenarios for source build.

where we build on the same OS that we target, we will want this value to default to what dotnet considers the rid for the build platform

I see, I was making the default be the RID of the Packages we build I can change that to the computed RID (though the OutputRID is also derived from the computed RID). I think it's very important to be crisp about the purpose of these properties: we have a number of different RID values in the system already and adding new ones is going to cause confusion and misuse. Today the "computed" RID is retrieved here:

runtime/Directory.Build.props

Line 103 in 6496624

<_parseDistroRid Condition="'$(_parseDistroRid)' == '' and '$(MSBuildRuntimeType)' == 'core'">$([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier)</_parseDistroRid>

Is this the value you'd prefer to be the default?

Independent of the GenerateRuntimeGraph,

I believe this should work correctly.

runtime/src/libraries/System.Runtime.InteropServices.RuntimeInformation/tests/RuntimeIdentifierTests.cs

Lines 24 to 30 in 6496624

	RemoteInvokeOptions options = new RemoteInvokeOptions();
	options.StartInfo.EnvironmentVariables.Add("DOTNET_RUNTIME_ID", "overridenFromEnv-rid");
	RemoteExecutor.Invoke(() =>
	{
	Assert.Equal("overridenFromEnv-rid", RuntimeInformation.RuntimeIdentifier);
	}, options).Dispose();

[tmds]

Who's going to compute the correct parent? The end user who's running the source build? Might you need to add more than one?

The source-build user could specify it. It would be a single value.

At some point we'll need to say that we don't need to support more complex additions and those require a change in source: IMHO that point should be driven by our current customer scenarios for source build.

Yes, this is a scenario we don't need for the distros we target.

Is this the value you'd prefer to be the default?

It's not clear for me what the differences are between these rids. When we build the source-build tarball, then the output packages would be for the rid of /etc/os-release.

Now when we run source-build to create the tarball, we'd like to be able to create a tarball that works across a range of linux-x64 distros. I'm not sure if packages are built for the tarball at this stage. If they are, I assume this means the output packages would be for the linux-x64 rid.

[tmds]

@ericstj does the rid logic make sense?

we'd like to be able to create a tarball that works across a range of linux-x64 distros

As future steps, this tarball could be the output of source-build, produced by Microsoft.

The tarball can be compiled with a pre-built SDK from Microsoft. Or with an SDK that was already source-build compiled for a previous patch version, or for a previous (compatible) version of the distro.

cc @crummel @dseefeld @MichaelSimons @omajid

[ericstj]

The source-build user could specify it. It would be a single value.

This makes sense, we can easily add the ability to allow the source-build user to specify a new parent. I'll add that to the current changes. I'm in the propcess of moving this over from dotnet/arcade to dotnet/runtime to make it easier to maintain. Next steps on my side are to finish this port and add it to the draft PR in dotnet runtime and move out of draft.

Is this the value you'd prefer to be the default?

It's not clear for me what the differences are between these rids. When we build the source-build tarball, then the output packages would be for the rid of /etc/os-release.

I was pointing to the existing property reads from $([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier) which should derive it's value from /etc/os-release with an override of DOTNET_RUNTIME_ID.

Here's what I'm imagining the interface with the user. During a source build:

• Automatically include $([System.Runtime.InteropServices.RuntimeInformation]::RuntimeIdentifier) in RID graph
• Automatically include PackageRID|OutputRID in RID graph
• Both of above will be added to AdditionalRIDs property
• When considering any unknown RID family, choose the parent based on single user-specified parent RID, lets call it AdditionalRIDsParent

[tmds]

This sounds good to me. When we say build, we need to think about two different builds:

• When source-build produces a tarball.
• When the source-build tarball is used with an SDK to produce a source-built SDK.

For our use-cases. The first build is for linux-x64/linux-arm64. The second build is for the RID of /etc/os-release.