Implement RID for SunOS-derived operating systems #37016

am11 · 2020-05-26T15:46:17Z

This PR implements:

distinction between Illumos and Oracle Solaris, which are two forks of OpenSolaris since 2010.
RID for Illumos and Solaris:
- Solaris and Illumos derived from Unix.
- OmniOS, OpenIndiana and SmartOS derived from Illumos.

A note for OpenIndiana: struct utsname members currently do not provide OS release information like other distros, so it is left version-less (similar to Gentoo). Additionally, we cannot rely on /etc/release/version etc. which do not contain consistent results across the zones.

Contributes to: #34944.

ghost · 2020-05-26T15:46:23Z

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

am11 · 2020-05-26T15:46:52Z

cc @janvorli, @jclulow

am11 · 2020-05-26T15:49:44Z

@janvorli, @wfurt, I have also fixed a FreeBSD case in pal.unix.cpp, where pos was unused in string::append. Please let me know if you agree with it.

jkotas · 2020-05-26T16:27:01Z

eng/Configurations.props

@@ -52,7 +52,8 @@
    <TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('OSX'))">OSX</TargetOS>
    <TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('FREEBSD'))">FreeBSD</TargetOS>
    <TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('NETBSD'))">NetBSD</TargetOS>
-    <TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('SUNOS'))">SunOS</TargetOS>
+    <TargetOS Condition="'$(TargetOS)' == '' and $([MSBuild]::IsOSPlatform('ILLUMOS'))">Illumos</TargetOS>


What was wrong with using SUNOS as the name for all the flavors?

Illumos and Oracle Solaris are two forks of OpenSolaris which were created in 2010.

Despite of many similaries, binaries compiled on an Illumos distro are not compatible with Oracle Solaris and vice versa. They differ by:

libc and its version; to begin with, dlopen/dlsym specially look for ILLUMOS xyz when we try to execute Illumos binary on Solaris.

supported syscalls; made by the libc itself (before the user code even begins).

Like Darwin is to macOS, iOS, tvOS etc. some utilities and tools such as uname, python and cmake still use the legacy SunOS name on both platforms for compatibility reasons. However, on Illumos uname -o returns illumos while Oracle Solaris does not support -o option.

binaries compiled on an Illumos distro are not compatible with Oracle Solaris and vice versa

The same is true e.g. between Ubuntu and Alpine. We do not have different TargetOS for these two.

True. On the other hand, their history is also similar to how OS X and FreeBSD used to share same base, but now they are diverged. Although there are some similarities in API usage, we consider them two different platforms on TargetOS level.

If I am not mistaken modern binaries from Solaris 11+ don't run at all on illumos and the reverse is also true. Some older binaries from the Solaris 10 days might still work on both though.

Treating both the same has caused a lot of ugly hacks in Makefiles over the years, every few years there is also talk from dropping SunOS entirely from uname. But as of now that would break too many software that treat both the same.

For new languages (.e.g rust, go, ...) we are treating illumos seperate. Although IIRC for go illumos is a child of sunos (but I can't quickly find the page I was thinking of to confirm this)

am11 · 2020-05-27T18:09:31Z

@jkotas, @janvorli, I have resolved merge conflicts and rebased with running master. Please take another look when you get a chance.

The idea here is to keep Illumos and Solaris as separate platforms in MSBuild scripts and dotnet code from the start; by avoiding SunOS concept (which was superseded by Solaris in mid 90's). We can change it if it is deemed suitable, and I do not feel strongly. Please let me know. This is just something I was recommended by community members to treat two platforms separate.

This PR serves as a prerequisite to the next step; cross-compilation via Ubuntu. I am trying to make progress and upstream changes in meaningful chunks sequentially.

jkotas · 2020-05-27T18:25:39Z

@ericstj - RID graph changes

src/libraries/pkg/Microsoft.NETCore.Platforms/runtimeGroups.props

ericstj · 2020-05-27T19:13:20Z

src/libraries/pkg/Microsoft.NETCore.Platforms/runtimeGroups.props

+      <Versions>11</Versions>
+    </RuntimeGroup>
+
+    <RuntimeGroup Include="illumos">


Will this be a something we leverage to build portable binaries (similar to linux-x64)? If not, should you account for such a RID and should solaris import it? cc @janvorli

Tested with dotnet binary: when compiled on an illumos based distro, SmartOS, and copied artifacts/bin to other illumos distros: OpenIndiana and OmniOS it worked; but not on Oracle Solaris 11x.

Oracle Solaris from that perspective is a separate platform:

different syscalls supported by illumos and Solaris kernels.

they both have their own libc implementations; POSIX compliant but different, so the library loader (ld) looks for different versions and fails if we try to run binary compiled on one, ld fails on the other platform. If we satisfy it somehow (local copy of libc and set LD_LIBRARY_PATH to point to its directory), that fails when local libc makes unsupported syscall to kernel.

Both platforms have portable binaries for their own distros/versions, but they do not mix from derivation perspective.

illumos and Oracle Solaris were forked from OpenSolaris in 2010, since then they have diverged.
This is similar to, FreeBSD and OS X used to share same base once, but lately they have diverged.

See my comment here #37016 (comment), for new stuff they are almost certainly no longer binary compatible.

Edit: but all illumos distros (SmartOS, OmniOS, OpenIndiana, ...) should be though.

Different platform API could hypothetically be shimmed. I'll defer to @janvorli on the feasibility of doing something like portable-linux here and if we need to think about that from the start. I suppose it could always be added later.

I don't think it makes sense to create a portable thing here - we don't have portability between MUSL and GLIBC based distros for similar reasons.

I am referring to illumos vs solaris.

Also note that illumos to solaris is not only like glibc to musl difference (which is two libcs for same Linux kernel) but they are also freebsd to macOS different (which is two different kernels and their corresponding libcs).

src/libraries/OSGroups.json

eng/Configurations.props

eng/build.sh

eng/native/init-distro-rid.sh

ericstj · 2020-05-27T20:52:20Z

src/installer/corehost/cli/hostmisc/pal.unix.cpp

+    return ridOS;
+}
+#elif defined(__sun)
+pal::string_t pal::get_current_os_rid_platform()


If I remember correctly, there was a time where we were trying very hard to avoid having special case mapping in the RID calculation. It means folks will need a new copy of the RID algorithm in order to get the right result on these platforms. It will also mean the old algorithm will stop returning the truth on these platforms. @eerhardt

The way the new 5.0 API RuntimeInformation.RuntimeIdentifier is implemented is it just returns the RID from the host. So as long as any users are using this new API, they will get the correct RID.

If users are using the deprecated Microsoft.DotNet.PlatformAbstractions library, they won't get the correct RID on these systems, and will need to update to the new API.

cc @vitek-karas @swaroop-sridhar - for the host changes.

@eerhardt, does the deprecated PlatformAbstractions library provide certain feature which RuntimeIndentifier does not? I found that dotnet/sdk still using it in the following three files:

sdk$ git grep -li PlatformAbstractions src/Cli/Microsoft.DotNet.Cli.Utils/WindowsRegistryEnvironmentPathEditor.cs src/Layout/redist/targets/GenerateLayout.targets src/Tasks/Microsoft.NET.Build.Tasks/ProjectContext.cs

Generally, I think it would be a better model, if we don't have to update platforms in SDK repo separately each time a new platform is introduced in runtime repo. For example, roslyn, aspnet, even msbuild and many other repos don't know/care how many OS platforms are supported by runtime, they just work everywhere (MSIL ftw). At least SDK should not block the new platforms by default (especially if that hardcoded platform list is just used for telemetry). :)

I found that dotnet/sdk still using it in the following three files:

In the two .cs files, it is only being used by a using statement. This was a left-over of old code, and can be removed. The reason the namespace still exists is because another library (Microsoft.Extensions.DependencyModel) has an Obsolete type in that namespace.

For src/Layout/redist/targets/GenerateLayout.targets, this was actually telling the build system to not include the PlatformAbstractions assembly coming from vstest. vstest doesn't reference that library anymore, so it can be removed as well.

I've removed these with dotnet/sdk#11857.

Generally, I think it would be a better model, if we don't have to update platforms in SDK repo separately each time a new platform is introduced in runtime repo.

At least SDK should not block the new platforms by default (especially if that hardcoded platform list is just used for telemetry). :)

Can you open an issue in dotnet/sdk for this? Note that it isn't just telemetry - but also the SDK prints OS platform information in dotnet --info.

am11 · 2020-05-28T14:44:28Z

Mono Product Build Linux arm64 debug is unrelated (git clone failed on that machine).

wfurt

LGTM. (leaving RID parts to the experts)

ericstj

I'm OK with mechanical side of the RID changes. I'd like explicit signoff from @eerhardt and @janvorli as well.

eerhardt

From a RID stand-point this looks good to me.

am11 · 2020-06-09T07:55:02Z

Resolved a merge conflict.
@janvorli, @jkotas, could you please take a look?

janvorli

LGTM, thank you!

am11 · 2020-06-11T13:06:06Z

All checks have passed
Good to merge? :)

ericstj · 2020-06-11T15:27:33Z

Thank you for contribution @am11

Dotnet-GitSync-Bot added the area-Infrastructure label May 26, 2020

jkotas reviewed May 26, 2020

View reviewed changes

Implement RID for SunOS-derived operating systems

e006cc7

jkotas requested a review from ericstj May 27, 2020 18:26

ericstj requested a review from wfurt May 27, 2020 18:59

ericstj reviewed May 27, 2020

View reviewed changes

am11 commented May 27, 2020

View reviewed changes

src/libraries/OSGroups.json Outdated Show resolved Hide resolved

sjorge reviewed May 27, 2020

View reviewed changes

eng/Configurations.props Outdated Show resolved Hide resolved

sjorge reviewed May 27, 2020

View reviewed changes

eng/build.sh Outdated Show resolved Hide resolved

sjorge reviewed May 27, 2020

View reviewed changes

eng/build.sh Outdated Show resolved Hide resolved

sjorge reviewed May 27, 2020

View reviewed changes

eng/native/init-distro-rid.sh Outdated Show resolved Hide resolved

ericstj reviewed May 27, 2020

View reviewed changes

jaredpar mentioned this pull request May 27, 2020

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed

Use official capitalization in all strings

8e5ae6e

wfurt approved these changes May 29, 2020

View reviewed changes

Compile get rid functions separately

ded38ca

ericstj approved these changes Jun 2, 2020

View reviewed changes

eerhardt approved these changes Jun 3, 2020

View reviewed changes

Merge branch 'master' into feature/sunos/rid

d01653f

janvorli approved these changes Jun 9, 2020

View reviewed changes

swaroop-sridhar approved these changes Jun 9, 2020

View reviewed changes

janvorli merged commit c783b1f into dotnet:master Jun 11, 2020

am11 deleted the feature/sunos/rid branch June 11, 2020 13:44

am11 mentioned this pull request Jun 22, 2020

Tracking issue for illumos and Solaris x86-64 port work #34944

Open

19 tasks

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement RID for SunOS-derived operating systems #37016

Implement RID for SunOS-derived operating systems #37016

am11 commented May 26, 2020

ghost commented May 26, 2020

am11 commented May 26, 2020

am11 commented May 26, 2020

jkotas May 26, 2020

am11 May 26, 2020

jkotas May 26, 2020

am11 May 26, 2020

sjorge May 27, 2020

am11 commented May 27, 2020

jkotas commented May 27, 2020

ericstj May 27, 2020

am11 May 27, 2020 •

edited

Loading

sjorge May 27, 2020 •

edited

Loading

ericstj May 27, 2020

janvorli May 27, 2020

janvorli May 27, 2020

am11 May 29, 2020

ericstj May 27, 2020

eerhardt May 27, 2020

eerhardt May 27, 2020

am11 Jun 2, 2020 •

edited

Loading

eerhardt Jun 2, 2020

am11 commented May 28, 2020

wfurt left a comment

ericstj left a comment

eerhardt left a comment

am11 commented Jun 9, 2020

janvorli left a comment

am11 commented Jun 11, 2020

ericstj commented Jun 11, 2020

Implement RID for SunOS-derived operating systems #37016

Implement RID for SunOS-derived operating systems #37016

Conversation

am11 commented May 26, 2020

ghost commented May 26, 2020

am11 commented May 26, 2020

am11 commented May 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

am11 commented May 27, 2020

jkotas commented May 27, 2020

Choose a reason for hiding this comment

am11 May 27, 2020 • edited Loading

Choose a reason for hiding this comment

sjorge May 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

am11 Jun 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

am11 commented May 28, 2020

wfurt left a comment

Choose a reason for hiding this comment

ericstj left a comment

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

am11 commented Jun 9, 2020

janvorli left a comment

Choose a reason for hiding this comment

am11 commented Jun 11, 2020

ericstj commented Jun 11, 2020

am11 May 27, 2020 •

edited

Loading

sjorge May 27, 2020 •

edited

Loading

am11 Jun 2, 2020 •

edited

Loading