dart compile exe support static linking of C code

[dcharkes]

dart compile exe enables compiling Dart source code to a standalone executable.

It would be nice if we could link in static libraries in this process so that we can use DynamicLibrary.executable() to look up the symbols.

Current workaround: Use shared libraries, use DynamicLibrary.open() with a relative path, and ship a folder with the dart compile exe executable and the shared libraries (*.so / *.dll / *.dylib).

Technical work that needs to happen for this:

• Make dart compile exe use a proper format instead of appending the precompiled programing as a blob to the dart precompiled runtime by using a a proper linker on each target platform.
• Add options to pass in linker arguments.

(cc @mkustermann, @cskau-g I don't think we had an issue tracking this yet.)

[dvc94ch]

was just trying to figure out how to do this, this would be great, yes.

[xros]

Yes. That's what we need. Compile C/C++ libs together into a standalone executable.

I come up with an idea that. In the future we can have a function called StaticLibrary.open().
And after that, everytime when we try to compile it, dart compiler can add the C/C++ libs to the end of the main binary. When the binary is executed, it would look for the place of the C/C++ lib where was attached to itself. It would have some affect on loading time, and RAM consumption. But it's easier to deliver apps. Just make it optional.

This feature can be only for dart compile exe, not for aot-snapshot, jit-snapshot, kernel.

[dcharkes]

Make dart compile exe use a proper format instead of appending the precompiled program as a blob to the dart precompiled runtime by using a a proper linker on each target platform.

Work underway for MacOS: https://dart-review.googlesource.com/c/sdk/+/228080

cc @sstrickl

[dvc94ch]

Also please make the linker configurable, cross linking with lld works on all platforms these days and would be nice to support in addition/instead of the system linker.

[dcharkes]

We would like to address this using the static bindings with @FfiNatives:

• [vm/ffi] Explore using FfiNative for static linking #49418

[laoshaw]

Rust, Golang and even Python can be built into one single exe without external lib dependencies, i.e. static exe. Dart please add this, something like dart compile static-exe

[maks]

@laoshaw dart compile already produces a single, executable without any dependencies except the usual C runtime ones:

$ dart compile exe main.dart 
$ ldd main.exe 
        linux-vdso.so.1 (0x00007fff4a3fc000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63d4122000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63d411d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63d4036000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63d3e0e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f63d47ea000)

This issue is specifically about being able to do so when also link in static libraries into that executable when using Dart FFI.

[laoshaw]

@laoshaw dart compile already produces a single, executable without any dependencies except the usual C runtime ones:

$ dart compile exe main.dart 
$ ldd main.exe 
        linux-vdso.so.1 (0x00007fff4a3fc000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63d4122000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63d411d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63d4036000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63d3e0e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f63d47ea000)

This issue is specifically about being able to do so when also link in static libraries into that executable when using Dart FFI.

Yes, I would like ldd main.exe outputs not a dynamic executable instead of listing all those C libraries, that makes depolyments as simple as golang, is this possible?

[dcharkes]

This issue tracks statically linking in C libraries that package developers provide.

Statically linking in the C runtime dependencies of the dart aot runtime would be a different project. @laoshaw do you have any issues with deployment depending on the C runtime libraries that are available? These libraries are always pre-installed on Linux. And bundling them would significantly increase the binary size of your final program.

[mkustermann]

Yes, I would like ldd main.exe outputs not a dynamic executable instead of listing all those C libraries, that makes depolyments as simple as golang, is this possible?

Doing this may already be a showstopper as it may break a number of things, see e.g. discussion here. I also recall that such statically linked binaries will have trouble opening dynamic libraries, which is something dart:ffi provides and we cannot break.

[laoshaw]

musl-gcc can link dart:ffi statically?

still, in cases without dart::ffi, it would be nice to support static linking like how Go provides.

that Golang static binary made deployment really easy, and will help to expand Dart's usage as well.

[rusty-snake]

https://dart.dev/tools/dart-compile:

The exe subcommand produces a standalone executable for Windows, macOS, or Linux. A standalone executable is native machine code that's compiled from the specified Dart file and its dependencies, plus a small Dart runtime that handles type checking and garbage collection.

[…]

Known limitations
The exe subcommand has some known limitations:

• No cross-compilation support
• No support for dart:mirrors and dart:developer

A standalone executable that depended on a glibc with correct version being present at runtime is not what I expect when I hear "standalone executable".

Given that this will not be "fixed" any time soon (no activity for 1.5 years) this should be clarified / added to known limitations.

[chances]

@rusty-snake Much to my chagrin, there are tons of "standalone" applications in the wild that depend on a specific C standard library being installed to function. Regardless, this issue is discussing statically linking against user-supplied C libraries at compile-time.

@dcharkes Consider renaming this issue to reduce confusion.