Use lib prefix in all native library names #33236

am11 · 2020-03-05T14:57:47Z

Also consolidated related MSBuild properties in
/eng/native/naming.props file.

stephentoub · 2020-03-05T15:04:19Z

IIRC, the DllImport loader will probe for the specified name + suffix (e.g. ".so") before it probes for the "lib" prefix + name + suffix. Given that, we might want to also update https://github.com/dotnet/runtime/blob/master/src/libraries/Common/src/Interop/Unix/Interop.Libraries.cs to include the "lib" prefix in each name, to skip one of the probing steps.

jkotas · 2020-03-05T15:07:21Z

cc @tarekgh

am11 · 2020-03-05T15:29:26Z

Would the cost of RuntimeInformation.IsOSPlatform(OSPlatform.OSX) check (#738; for dylib vs. so) overweigh in comparison to the probing logic; if they both are evaluated in static context?

stephentoub · 2020-03-05T15:30:53Z

Would the cost of RuntimeInformation.IsOSPlatform(OSPlatform.OSX) check (#738; for dylib vs. so) overweigh in comparison to the probing logic

Probing requires disk access. So, no :) But I'm also not sure why IsOSPlatform is relevant to this issue? The runtime is built for the target platform/OS; the ".dylib" vs ".so" is hardcoded.

danmoseley · 2020-03-05T18:11:13Z

cc @tarekgh since he was on the issue

danmoseley · 2020-03-05T18:12:03Z

src/libraries/Common/src/Interop/Unix/Interop.Libraries.cs

- internal const string CryptoNative = "System.Security.Cryptography.Native.OpenSsl";
- internal const string CompressionNative = "System.IO.Compression.Native";
- internal const string IOPortsNative = "System.IO.Ports.Native";
+ internal const string SystemNative = "libSystem.Native";


would lib.System.Native (and so forth) be better? more readable than libSystem.Native perhaps

no, this is not the Linux convention. usually `lib' is prefixed without dots.

plus adding the . might make it such that the loader won't find the library automatically as for anybody PInvoking to System.Native, I believe the runtime will try either System.Native or preappending lib, but it won't find it if it also needs to pre-append a dot.

Makes sense

tarekgh · 2020-03-05T18:12:33Z

Thanks @danmosemsft for tagging me. @jkotas already tagged me before too.

joperezr · 2020-03-05T18:52:21Z

I'll take a look at the failures on the builds to see what is wrong

am11 · 2020-03-05T19:04:35Z

@joperezr, thank you. I am investigating macOS build failure locally, if you could point out issue with Windows failure, that would help a lot. :)

joperezr · 2020-03-05T19:14:03Z

~~I think all of the issues are the same in all builds, which~~ EDIT: The issue in the OSX build is during the installers build when trying to crossgen the shared framework. The error that the crossgen compiler is throwing looks like:

Unable to load Jit Compiler: libclrjit.dylib, hr=0x8007007e
Error: compilation failed for "/Users/runner/runners/2.165.0/work/1/s/artifacts/bin/pkg/netcoreapp5.0/runtime/OSX-Debug-x64/Microsoft.CSharp.dll" (0x8007007e)

The command we are sending to crossgen seem fine to me though:

/Users/runner/runners/2.165.0/work/1/s/artifacts/transport/coreclr/sharedFramework/crossgen @/Users/runner/runners/2.165.0/work/1/s/artifacts/obj/netcoreapp/Debug/netcoreapp5.0/osx-x64/crossgen/Microsoft.CSharp.rsp

And the .rsp contents are:

-nologo
-readytorun
-in /Users/runner/runners/2.165.0/work/1/s/artifacts/bin/pkg/netcoreapp5.0/runtime/OSX-Debug-x64/Microsoft.CSharp.dll
-out /Users/runner/runners/2.165.0/work/1/s/artifacts/bin/osx-x64.Debug/crossgen/netcoreapp/runtimes/osx-x64/lib/netcoreapp5.0/Microsoft.CSharp.dll
-platform_assemblies_paths /Users/runner/runners/2.165.0/work/1/s/artifacts/bin/pkg/netcoreapp5.0/runtime/OSX-Debug-x64/:/Users/runner/runners/2.165.0/work/1/s/artifacts/transport/coreclr/sharedFramework/:/Users/runner/runners/2.165.0/work/1/s/artifacts/transport/coreclr/:/Users/runner/runners/2.165.0/work/1/s/artifacts/bin/pkg/netcoreapp5.0/runtime/OSX-Debug-x64/
-JITPath /Users/runner/runners/2.165.0/work/1/s/artifacts/transport/coreclr/sharedFramework/libclrjit.so

joperezr · 2020-03-05T19:27:22Z

src/libraries/Native/Unix/System.Security.Cryptography.Native/CMakeLists.txt

-set_target_properties(System.Security.Cryptography.Native.OpenSsl PROPERTIES PREFIX "")
-
-# Disable the "lib" prefix and override default name
-set_target_properties(System.Security.Cryptography.Native.OpenSsl-Static PROPERTIES PREFIX "")


I'm not too familiar with this so probably @bartonjs should take a look, but looks like in here you are also changing the override of the default name

and override default name

this part of the comment refers to the line below. There are two variants of this library produced by this script: shared and static. We have prefix override for each which this delta is removing.

Yes this change is correct. We could however preserve the override default name part of the comment for the next line which sets the OUTPUT_NAME on the static lib.

joperezr · 2020-03-05T19:27:51Z

src/libraries/Native/Unix/System.Security.Cryptography.Native.Apple/CMakeLists.txt

-set_target_properties(System.Security.Cryptography.Native.Apple PROPERTIES PREFIX "")
-
-# Disable the "lib" prefix and override default name
-set_target_properties(System.Security.Cryptography.Native.Apple-Static PROPERTIES PREFIX "")


Same comment here, this is not only adding lib as a prefix, it is removing our override

am11 · 2020-03-05T20:40:42Z

-JITPath /Users/runner/runners/2.165.0/work/1/s/artifacts/transport/coreclr/sharedFramework/libclrjit.so

this is the issue on macOS, @joperezr. crossgen's response file is picking up .so (linux extension) instead of macOS' .dylib extension, the dylib file does exist in that directory:

$ otool -L ./artifacts/bin/coreclr/OSX.x64.Debug/sharedFramework/libclrjit.dylib

./artifacts/bin/coreclr/OSX.x64.Debug/sharedFramework/libclrjit.dylib:
	@rpath/libclrjit.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1069.11.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59306.41.2)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)

Interestingly, the error message doesn't show the actual filename that was requested (and failed) to load: Unable to load Jit Compiler: libclrjit.dylib, hr=0x8007007e

bartonjs · 2020-03-05T23:46:04Z

src/libraries/Common/src/Interop/Unix/Interop.Libraries.cs

+ internal const string SystemNative = "libSystem.Native";
+ internal const string GlobalizationNative = "libSystem.Globalization.Native";
+ internal const string NetSecurityNative = "libSystem.Net.Security.Native";
+ internal const string CryptoNative = "libSystem.Security.Cryptography.Native.OpenSsl";


This change, overall, makes me sad. We intentionally stripped off the lib prefix as part of "these aren't general purpose libraries, don't look here". But if we need to, we need to, I guess.

The alternative may be to statically link these libraries into the runtime for Android where it is causing problems. We are going to have this option for single file flavor of CoreCLR anyway, so doing it for Android as well (or even everywhere) should not be a big deal.

CC @akoeplinger and @marek-safar to be aware of the suggestion.

statically link these libraries into the runtime

Xamarin.Android does not support this mode. I think this would be nice but we won't be in a position to deliver this for .net5. This looks more like .net6 enhancement because we would need to do native linking during publish phase (this depends on Android NDK, etc).

/cc @jonpryor

For CoreCLR, the plan is to statically link the libraries as part of the dotnet/runtime build (see #32823). We are not planning to do native linking during publish phase.

That won't work for us. The thinking right now is to copy the dynamic libs which are used (during publish) and extend that to full static linking for Android as we plan to do for iOS (and wasm) during publish

Just curious - why it does not work for you?

Primary because of size. If an app uses no crypto we should not link libSystem.Net.Security.Native and you can apply that to all native libs where System.IO.Compression.Native is probably the biggest one.

The moment someone brings in any managed HTTP implementation, client or server, it's going to end up requiring both of those. Presumably then the size concern would only be if they were using a platform-specific implementation (e.g. AndroidHttpHandler) where the platform provided all of that support?

Also consolidated related MSBuild properties in `/eng/native/naming.props` file.

akoeplinger · 2020-03-06T11:50:03Z

src/coreclr/src/.nuget/Directory.Build.props

@@ -12,6 +12,9 @@
 <!-- build the transport package which includes product and symbols in addition to standard packages -->
 <CreatePackedPackage Condition="'$(CreatePackedPackage)' == ''">true</CreatePackedPackage>

+ <!-- indicates that lib prefix is not added to library names for Unix-like operating systems -->
+ <SkipLibraryPrefixFromUnix>true</SkipLibraryPrefixFromUnix>


what's the reason for this property? why would we want to skip the prefix?

In case of coreclr, the prefixes were not added in the original version. Apparently, there is a job on CI that crosscompile clrjit.so (Linux version) on Windows. That one was failing, so I restored it by skipping the prefix here. It will require a bit more investigation and it can be avoided (at least in theory) 😅

src/libraries/Common/src/Interop/Unix/Interop.Libraries.cs

tarekgh · 2020-03-06T17:16:36Z

all reviewers, do you have any objection proceeding with this PR? any more feedback?

For the issue linking the native library statically, we can track it as a different work item.

tarekgh · 2020-03-06T17:18:34Z

CC @janvorli

akoeplinger

LGTM

joperezr

LGTM too

tarekgh · 2020-03-06T23:03:18Z

I have created the issue #33311 to track the static linking of the native libraries and I am going to merge this one.

tarekgh · 2020-03-07T00:33:11Z

Thanks @am11 for your help with this issue.

Dotnet-GitSync-Bot added the area-Infrastructure-libraries label Mar 5, 2020

danmoseley reviewed Mar 5, 2020

View reviewed changes

joperezr reviewed Mar 5, 2020

View reviewed changes

bartonjs reviewed Mar 5, 2020

View reviewed changes

Use lib prefix in all native library names

ae23c6b

Also consolidated related MSBuild properties in `/eng/native/naming.props` file.

am11 marked this pull request as ready for review March 6, 2020 06:15

am11 requested review from akoeplinger, EgorBo, marek-safar, steveisok and vargaz as code owners March 6, 2020 06:15

akoeplinger reviewed Mar 6, 2020

View reviewed changes

akoeplinger approved these changes Mar 6, 2020

View reviewed changes

joperezr approved these changes Mar 6, 2020

View reviewed changes

tarekgh mentioned this pull request Mar 6, 2020

Static link the native libraries to the runtime #33311

Closed

tarekgh merged commit 3d6f8a2 into dotnet:master Mar 7, 2020

am11 deleted the feature/libraries-native-libprfix branch March 7, 2020 00:58

EgorBo mentioned this pull request Mar 18, 2020

Update SDK in global.json to 5.0.100-preview.3.20168.11 #33735

Merged

danmoseley mentioned this pull request Apr 29, 2020

Fixing break on OOB System.IO.Ports package where native dependency was no longer produced #35591

Merged

jkotas mentioned this pull request Jul 21, 2020

Use lib prefix for native library names for consistency #39717

Merged

am11 mentioned this pull request Dec 7, 2020

renaming clrcompression --> System.IO.Compression.Native #45713

Merged

ghost locked as resolved and limited conversation to collaborators Dec 10, 2020

Use lib prefix in all native library names #33236

Use lib prefix in all native library names #33236

Conversation

am11 commented Mar 5, 2020 • edited by jkotas Loading

stephentoub commented Mar 5, 2020

jkotas commented Mar 5, 2020

am11 commented Mar 5, 2020

stephentoub commented Mar 5, 2020 • edited Loading

danmoseley commented Mar 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarekgh commented Mar 5, 2020

joperezr commented Mar 5, 2020

am11 commented Mar 5, 2020

joperezr commented Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

am11 commented Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

jkotas Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marek-safar Mar 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tarekgh commented Mar 6, 2020

tarekgh commented Mar 6, 2020

akoeplinger left a comment

Choose a reason for hiding this comment

joperezr left a comment

Choose a reason for hiding this comment

tarekgh commented Mar 6, 2020

tarekgh commented Mar 7, 2020

am11 commented Mar 5, 2020 •

edited by jkotas

Loading

stephentoub commented Mar 5, 2020 •

edited

Loading

joperezr commented Mar 5, 2020 •

edited

Loading

am11 commented Mar 5, 2020 •

edited

Loading

jkotas Mar 5, 2020 •

edited

Loading

marek-safar Mar 6, 2020 •

edited

Loading