☂️ [vm/ffi] Build hooks & Code assets

[dcharkes]

Build hooks and code assets support in Dart standalone is available in preview on the main channel. (In Dart 3.2 and later it was available behind an experimental flag: dart --enable-experiment=native-assets run [path/to/script.dart].)

Current ongoing work can be tracked in: Native Assets

This is a high level "feature" issue that we can refer to from various places.

The current way of bundling native code with Dart packages is far from ideal, we will introduce "Native Assets" to Dart.

Problem

As of right now, one can do one (or more) of the following:

1. ship sources in a Dart standalone package and let users run a build by calling dart run <your-package>/setup.dart see webcrypto, or
2. make your bin/setup.dart download the native library, or
3. ship sources a flutter plugins with flutter create --template=plugin_ffi --platform=..., or
4. ship a script that downloads the native library during flutter build see realm_flutter.

This leads to pain for both package developers (options 1-4) and package users (option 1-2). Moreover, there is currently no way to make a package with native code that works both for flutter and Dart standalone.

High level solution

We introduce the concept of "Native Assets" to the Dart eco system.

A native asset is a native library (compiled from native source code).

Three components of the "Native Assets" solution in Dart:

1. A standardized way to build or download native assets. This will be done by a CLI.
2. A transparent way to find native symbols in these native assets in Dart.
3. Integration with all the dart and flutter tooling to link the above two together.

Try now

Native assets support in Dart standalone is available in Dart 3.2 behind an experimental flag : dart --enable-experiment=native-assets run [path/to/script.dart].

Native assets support in Flutter is in development. Progress can be tracked in flutter/flutter#129757.

An example on how to use native assets can be found here.

Checklist

• Basic Dart support: [vm/ffi] Native asset resolution by the Dart VM for FfiNatives #49803
• Flutter support: ☂️ Build hooks & Code assets flutter/flutter#129757
• Refine core support:
  • [vm/ffi] Make FfiNatives more concise #50097
  • Allow access to global variables in ffi static binding #50551
  • Allow getting address of native symbols when using static ffi binding #50552
  • [ffi] tree-shaking does not remove dependency .so library from the build #52970
  • [vm/ffi] Explore using FfiNative for static linking #49418
  • dart compile exe support static linking of C code #47718
  • [vm/ffi] FfiNative return types treatment inconsistent with asFunction #49518
  • [native_assets][resource_identifiers] Deferred loading and assets #55809
  • [native_assets] Support dart pub global run #56044
  • [native_assets] Hot restart native libs #55850
• Graduate out from experiment flag (--enable-experiment=native-assets)

[dcharkes]

Related issues:

• [vm/ffi] Native asset resolution by the Dart VM for FfiNatives #49803
• [vm/ffi] Make FfiNatives more concise #50097
• Allow access to global variables in ffi static binding #50551
• Allow getting address of native symbols when using static ffi binding #50552
• dart compile exe support static linking of C code #47718
• [vm/ffi] Explore using FfiNative for static linking #49418
• [vm/ffi] FfiNative return types treatment inconsistent with asFunction #49518
• [ffi] tree-shaking does not remove dependency .so library from the build #52970

Related repositories:

• https://github.com/dart-lang/native

[GregoryConrad]

Hi, as a current user of option number 4 above, this sounds really enticing. Right now I am using CMake for Linux/Windows/Android and Ruby (with inline bash) for macOS/iOS to download & extract binaries and it is a significant pain point. A few questions:

1. Would this allow for static linking, dynamic linking, and dynamic loading? Or only a subset of these? It also might be nice to allow users to choose what to do per platform (e.g., static link on iOS/macOS and dynamic link/load on other platforms).
2. Do you envision this being available as a part of Dart 3.0? My guess is no (at least for the initial release), but wanted to ask regardless.
3. It will not always be possible to build native assets on the client-side unfortunately (e.g. Rust, Go, etc.). Will we be able to upload native assets directly to pub.dev, or at least reference the native assets from there (through some artifact repo or GitHub releases for example)?
4. How will web/wasm be handled? It would be good to have some integration to ease development/release, but not too too much, so things like wasmer-js can be supported to use WASM with WASI on web.

[dcharkes]

Hi @GregoryConrad, nice to meet you!

Hi, as a current user of option number 4 above, this sounds really enticing. Right now I am using CMake for Linux/Windows/Android and Ruby (with inline bash) for macOS/iOS to download & extract binaries and it is a significant pain point.

Thanks for bearing with us in the mean time! 🙃

A few questions:

1. Would this allow for static linking, dynamic linking, and dynamic loading? Or only a subset of these? It also might be nice to allow users to choose what to do per platform (e.g., static link on iOS/macOS and dynamic link/load on other platforms).

We will start with dynamic loading: @FfiNative doing dlopen under the hood with the dynamic libraries. In JIT mode (dart ..., flutter test) we will always use dynamic loading.

In AOT (dart compile ..., flutter build) we can think about invoking the native linker and static linking the Dart AOT snapshot with the static libraries, this is tracked in:

• [vm/ffi] Explore using FfiNative for static linking #49418

I have not explored using the native linker to do dynamic loading. I'm not sure if that will be of any benefit in any Dart or Flutter usage scenario. @GregoryConrad do you have a scenario in mind where you would prefer using dynamic loading?

2. Do you envision this being available as a part of Dart 3.0? My guess is no (at least for the initial release), but wanted to ask regardless.

I cannot comment on that. We'd want to ship the feature when it's ready, not earlier.

3. It will not always be possible to build native assets on the client-side unfortunately (e.g. Rust, Go, etc.). Will we be able to upload native assets directly to pub.dev, or at least reference the native assets from there (through some artifact repo or GitHub releases for example)?

We're in discussions about that, but likely yes. We have two options, and might want to support both:

1. The package author precompiles for all targets and uploads the binaries to pub, the package users then download the binaries needed for the architectures they're targetting. (Requirements here is that "A standardized way to build or download native assets. This will be done by a CLI." must have the assets in a standardized location on disk or that the CLI takes an output folder so that pub and the CLI can cooperate on getting binaries in the right place.) Uploading to pub is tracked in:

• Native assets on pub pub#3693

2. A package author can have a build_dependency on package:rust_compiler which is a dart package that downloads a rust compiler. (Requirements here are that we should probably have some kind of "This will install X GB of tools on your computer. Continue: Y/N" in our CLI.)

4. How will web/wasm be handled? It would be good to have some integration to ease development/release, but not too too much, so things like wasmer-js can be supported to use WASM with WASI on web.

Best case scenario we could have a unified API for interacting with the C library on native platforms and a to WASM compiled library on the web:

• [ffi/js-interop/wasm] Unified API for using dart:ffi and JS Interop with WASM #46690

Further discussion can be on that issue.

[GregoryConrad]

I have not explored using the native linker to do dynamic loading. I'm not sure if that will be of any benefit in any Dart or Flutter usage scenario. @GregoryConrad do you have a scenario in mind where you would prefer using dynamic loading?

I don't actually have a case for using dynamic linking over dynamic loading, or vice versa. When I said:

It also might be nice to allow users to choose what to do per platform

I was more concerned with making sure we would be able to choose between static linking and dynamic linking/loading per platform. If someone else has a use case, however, I would be happy to hear it.

2. A package author can have a build_dependency on package:rust_compiler which is a dart package that downloads a rust compiler. (Requirements here are that we should probably have some kind of "This will install X GB of tools on your computer. Continue: Y/N" in our CLI.)

Although I like the idea here a lot, I foresee some issues with this approach. Especially in my project, some custom build steps and quite a few other package dependencies (besides just rustup) are needed to handle cross compilation, especially to android (due to linking with the NDK). I currently have these all done in CI and I think I would just prefer to ship the binaries in a native assets bundle directly. Perhaps having a rust_compiler dependency would work great for many projects, but for some projects with more complex dependencies/needs, this may not be as feasible. Also, if Flutter supports cross-compilation between desktop platforms in the future, this could also be problematic.

And thanks for the rest of your input! Much appreciated.