Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving and unifying handling of native Libraries code in embedded/singlefile scenarios #41299

Closed
VSadov opened this issue Aug 24, 2020 · 3 comments
Assignees
Milestone

Comments

@VSadov
Copy link
Member

VSadov commented Aug 24, 2020

Improving handling of native Libraries code in embedded/singlefile scenarios

For reasons that vary .Net libraries rely on nontrivial amount of native code. The native code is factored into a few well-known native dynamic-link libraries (.dll, .so, etc). As an example we have 4 libraries like this on Linux – "libSystem.Native", "libSystem.IO.Compression.Native", "libSystem.Net.Security.Native", "libSystem.Security.Cryptography.Native.OpenSsl". The number of libraries varies depending on platform.

This is a very simple model which has its own advantages. In particular there is no need for coreclr to have any special knowledge about these libraries.

Where this becomes inconvenient and fragile is when we want to package the native libraries differently - for example as a part of a single file host. To handle such requirement in the current implementation we produce static libraries for the native code and link that into coreclr/host. Since the native dlls are not longer around in this configuration, coreclr intercepts loading of these libraries and redirects to the containing host binary.

There is a number of issues:

  • Since coreclr and libraries are built separately, there is a possibility of a mismatch that could make them incompatible. (i.e. Debug vs. Release, different versions of libc, new/free operator mismatch ...)
  • Intercepting and redirecting the load happens in PAL layer. This is a problem when the same must be done on Windows.
  • Accessing the native methods via regular PInvokes requires that the native host re-exports the methods. That easily runs into platform or linker dependent behaviors regarding re-exporting of library functions from final executables.

Proposal:

  • Build native libraries as a part of CoreClr build.
    The source files should still be located under Libraries partition, so that they can be shared in source form with Mono, for example.

  • The “embedded” case builds native libraries as a part of coreclr binary, thus bypassing the explicit lib files.

  • The "embedded" case recognizes well-known libraries as a special case of QCall and dispatches calls to the embedded implementations.
    Example: “libSystem.Native” would normally be a name of the native library to load, but in “embedded” case, the DLLEXPORT methods of libSystem.Native would be registered in the runtime similarly to QCall and “libSystem.Native” would be recognized as another QCall variety.
    QCall mechanism here allows us to “shim” the location of native methods while not changing the managed code that calls them.

  • The QCall registration table for the native calls will be automatically generated by scanning managed libraries.

Some unresolved details:

  • If managed callers are built after coreclr, we would need to reconcile how to do the QCall generation before coreclr builds.
  • Should the generation of QCall entries be a part of the build (producing a temporary source file) or a dev-time step (the file is explicitly generated and stored in Git).
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Aug 24, 2020
@Dotnet-GitSync-Bot
Copy link
Collaborator

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@trylek trylek added this to the 6.0.0 milestone Aug 31, 2020
@trylek
Copy link
Member

trylek commented Aug 31, 2020

Assigning to .NET 6 milestone as this is a design proposal, not a bug fix, and I believe it's too late in the .NET 5 release cycle to introduce new feature work.

@trylek trylek removed the untriaged New issue has not been triaged by the area owner label Aug 31, 2020
@VSadov VSadov self-assigned this Oct 21, 2020
@VSadov
Copy link
Member Author

VSadov commented Dec 18, 2020

This was implemented in #44505

@VSadov VSadov closed this as completed Dec 18, 2020
@ghost ghost locked as resolved and limited conversation to collaborators Jan 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants