__extenddftf2 is found to be missing at runtime when Rust's cargo is driving the build

[adamnovak]

I'm trying to build rusqlite for Rust, using wasi-sdk 20 as my CC_wasm32_wasi environment variable value.

This appears to work, but at runtime in wasmtime I can't load the resulting binaries, because __extenddftf2 is missing.

wasmtime target/wasm32-wasi/release/query.wasm --sample "_gbwt_ref" --contig x --interval 1..10 --context 100 --distinct x.gbz.db
Error: failed to run main module `target/wasm32-wasi/release/query.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/release/query.wasm"
    1: unknown import: `env::__extenddftf2` has not been defined

I noticed that symbol name in wasi-sdk-20.0/share/wasi-sysroot/share/wasm32-wasi/undefined-symbols.txt. Isn't it supposed to be defined by the SDK, if the compiler doesn't throw an error when code tries to use it?

Is this builtin somehow out of scope for the SDK? Am I supposed to be using another library to provide it? Do I need to somehow convince rusqlite or SQLite itself that it isn't actually available for its C code?

[adamnovak]

It looks like this symbol is asked for if the C code being compiled tries to use long double. Setting LIBSQLITE3_FLAGS="-DLONGDOUBLE_TYPE=double" in my rusqlite build worked around the problem for me specifically. But it would be nice to have implementations for the long double functions introduced by the SDK's compiler.

Maybe the SDK's compiler just shouldn't provide a distinct long double if the backing functions aren't written yet? I think long double == double is an allowed C compiler setup, but long double -> program doesn't link probably isn't.

[sbc100]

__extenddftf2 should be defined in compiler-rt (llvm-project/compiler-rt/lib/builtins/extenddftf2.c). Do you know where you compiler-rt library is coming from? Can you check it for that symbol?

The version of compiler-rt that ships with wasm-sdk (https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-20/libclang_rt.builtins-wasm32-wasi-20.0.tar.gz) does seem to define it:

$ llvm-nm lib/wasi/libclang_rt.builtins-wasm32.a |  grep __extenddftf2
00000001 T __extenddftf2

[adamnovak]

Hmmm. It does look like the symbol is in extenddftf2.c.obj inside the archive, if I do ./wasi-sdk-20.0/bin/llvm-nm wasi-sdk-20.0/lib/clang/16/lib/wasi/libclang_rt.builtins-wasm32.a.

Is it possible that the linking step for my C code is missing -lclang_rt.builtins-wasm32 somehow? Or does ./wasi-sdk-20.0/bin/wasm-ld or whatever the linker entry point is know to use that library automatically? Or maybe cargo isn't calling the SDK's linker and is using its own linker, since I haven't set any variable to point it at the SDK's linker?

I'm not having problems with missing pieces of normal libc, which seems to be defined elsewhere and linked in some other way.

[sbc100]

wasm-ld doesn't add any libraries automatically. Its up the driver (normally clang, or emcc) to add those.

I'd be very surprised if compiler-rt was not being included by your the driver since there are many symbols in compiler-rt that are pretty much essential to all programs. Can you get your driver to print the wasm-ld command that it is using internally? Some kind of -v command maybe?

[adamnovak]

The post at https://petermalmgren.com/wasm-components-sqlite-vfs/ shows a more complicated setup for attacking Rust to the SDK than I am using, and suggests using CARGO_TARGET_WASM32_WASI_LINKER to override Rust's linker with the SDK one.

But I tried that:

CC_wasm32_wasi="$(pwd)/wasi-sdk-20.0/bin/clang" CARGO_TARGET_WASM32_WASI_LINKER="$(pwd)/wasi-sdk-20.0/bin/wasm-ld" LIBSQLITE3_FLAGS="-DSQLITE_THREADSAFE=0" cargo build --release --target=wasm32-wasi

And even though it's definitely calling the SDK's wasm-ld to do the linking in that case, I still end up with a final .wasm file that wants to import __extenddftf2 from somewhere.

If I don't use CARGO_TARGET_WASM32_WASI_LINKER it looks like it links with rust-lld instead.

According to RUSTC_LOG="rustc_codegen_ssa::back::link=info" CC_wasm32_wasi="$(pwd)/wasi-sdk-20.0/bin/clang" LIBSQLITE3_FLAGS="-DSQLITE_THREADSAFE=0" cargo build --release --target=wasm32-wasi my link command for the final binary is:

INFO rustc_codegen_ssa::back::link preparing Executable to "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.wasm"
INFO rustc_codegen_ssa::back::link LC_ALL="C" PATH="/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/self-contained:/opt/homebrew/opt/coreutils/libexec/gnubin:/Users/anovak/workspace/nvm/versions/node/v16.20.2/bin:/Users/anovak/workspace/amazon-genomics-cli/bin/local:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/Users/anovak/workspace/goenv/shims:/Users/anovak/workspace/goenv/bin:/Users/anovak/Library/Python/3.9/bin:/Users/anovak/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/usr/local/go/bin:/Users/anovak/.cargo/bin:/Users/anovak/Library/Python/3.8/bin" VSLANG="1033" "rust-lld" "-flavor" "wasm" "--rsp-quoting=posix" "--export" "__main_void" "-z" "stack-size=1048576" "--stack-first" "--allow-undefined" "--fatal-warnings" "--no-demangle" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/self-contained/crt1-command.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.00.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.01.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.02.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.03.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.04.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.05.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.06.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.07.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.08.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.query.f0675a9840201e39-cgu.09.rcgu.o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.4sp02d1fcpzp4xf2.rcgu.o" "-L" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps" "-L" "/Users/anovak/workspace/gbz-base/target/release/deps" "-L" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/build/libsqlite3-sys-c907a90c041b9f01/out" "-L" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libgetopts-a8ca6758bff208db.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libunicode_width-f7e20684d9d722a5.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libgbz_base.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libgbwt-3adea79020d3afd1.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libsimple_sds-2501a837742dbb3a.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/librusqlite-1221bf86d2e0fab9.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libbitflags-e884fe3e0c11dc17.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libsmallvec-862d6a58a4173eb4.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libfallible_streaming_iterator-5afedc3702fff3e4.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libfallible_iterator-ca3b1d2ee21a166e.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libhashlink-ce27c5bda55b1fdc.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libhashbrown-79ec296a4b1f42e5.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libahash-16d4597bab12dba5.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libonce_cell-3660462554ca687b.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libcfg_if-1c9b596c6322903c.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/libzerocopy-a402780b144d72d6.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/liballocator_api2-f2fb80c21c77e34e.rlib" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/liblibsqlite3_sys-6934f7ea2a87c421.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libstd-f299d69fcb5800cb.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libpanic_abort-8f6bd66673a464e1.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libwasi-37434f3ca08ef38e.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/librustc_demangle-5d93914955cad719.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libstd_detect-6c4e4338a0cfb733.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libhashbrown-9721e435132a6604.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/librustc_std_workspace_alloc-4e2ae9cbdcdb96fa.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libminiz_oxide-8d3c953d484b96a6.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libadler-372b805e7b59d097.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libunwind-c33053a00d807d75.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libcfg_if-5274c215bf94b2f4.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/liblibc-7e9a271479cd6119.rlib" "-l" "c" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/liballoc-1702bda0ec03aba7.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/librustc_std_workspace_core-84f0153a6bee220e.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libcore-5eea29fb480b41eb.rlib" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libcompiler_builtins-a0294efb55e060eb.rlib" "-L" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib" "-L" "/Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/self-contained" "-o" "/Users/anovak/workspace/gbz-base/target/wasm32-wasi/release/deps/query-a76dc5067563751a.wasm" "--gc-sections" "-O3"
INFO rustc_codegen_ssa::back::link linker stderr:

INFO rustc_codegen_ssa::back::link linker stdout:

This is apparently calling "rust-lld" "-flavor" "wasm". Notably, there's a /Users/anovak/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/wasm32-wasi/lib/libcompiler_builtins-a0294efb55e060eb.rlib in there, which helpfully defines a bunch of other soft float routines like __floatsisf or __extendsfdf2, but not __extenddftf2.

So I think I am ending up using a compiler builtins library provided by the Rust toolchain somehow, and not the one that C code built with the SDK needs to link against.

[sbc100]

Maybe wasm32-wasi/lib/libcompiler_builtins-a0294efb55e060eb.rlib is just out of date?

Also, shouldn't the linker report __extenddftf2 as undefined at build time or is rust using the --allow-undefined linker flag to just ignore any undefined symbols at link time?

[sbc100]

Yes I see --allow-undefined in the link flags. That seems unfortunate since it delays these types of errors until runtime.

[sbc100]

So this seems like a rust issue and not a wasi-sdk issue. Maybe open an issue there?

[adamnovak]

I went ahead and reported this against the Rust compiler builtins library, but I think they are probably going to say it is out of scope for them. If you look at https://github.com/rust-lang/compiler-builtins#unimplemented-functions it lists extenddftf2.c, presumably the home of __extenddftf2, as explicitly unimplemented, because Rust has no such thing as an f128 type and thus Rust code never needs to call it.

[adamnovak]

They also suggest a way to get a Rust project to use the C compiler's versions of the intrinsics, but it seems to be impossible to do on any stable release of the Rust compiler ever, which presumably means people can't genuinely use it? rust-lang/compiler-builtins#562

[sbc100]

In the short term you might also want to just go with the LIBSQLITE3_FLAGS="-DLONGDOUBLE_TYPE=double" solution since using 128bit long double is expensive in terms of both code size and performance (in both cases due to the emulation functions).

[adamnovak]

I think that is the workaround I'm going to have to take for now. But I'm worried that whatever the next problem is to come out of using the "wrong" intrinsics won't be so easy to bypass.

[sbc100]

I agree it seems wrong that rust doesn't include these functions in its compiler-rt in this case. However I does seem like a rust toolchain issue, not a wasi-sdk one, so perhaps we could close this in favor of an issue in a rust repo?

[sbc100]

CC @alexcrichton who knows rust stuff

[alexcrichton]

Thanks for the cc, I can try to help out a bit here! Agreed with all of the diagnosis and commentary so far, so I'll offer some thoughts on where this symbol might live.

As y'all have found the Rust version of compiler-builtins lives here and is a libcompiler-rt.a lookalike. This is more-or-less a Rust port of compiler-rt from LLVM, and its compilation is tailored for each target (the build.rs is relatively complicated) and is a mixture of Rust code and C code depending on the target (some all C, some all Rust, some a mix).

Historically the compiler-builtins crate was intended to be Rust-specific meaning that it contains intrinsics that LLVM might emit calls to for rustc-generated code. This means that historically support for f128 would not be included because Rust code would never use it. The theory being that if C code needed it then authors of crates could specify -l/path/to/libcompiler-rt.a for their system compiler.

Now that being said lots of time has passed in the interim since I worked on things here and so perspectives may have shifted and/or things changed in the meantime. In that sense I'd say that historically intrinsics like this were the responsibility of Rust crates using C, but nowadays it may be shifted elsewhere. I also think that the theory of using the system-provided compiler-rt is probably a disproportionately larger task than might otherwise be required implementing intrinsics like this.

Given all that, I'd recommend one of two routes:

• First confirm upstream whether it's ok to include this symbol or not (e.g. your issue). If perspectives have shifted then it'd be easiest to "just" include this when compiling for wasm.
• Alternatively one option could be to add a Cargo feature to the compiler-builtins crate to enable these intrinsics. That way they'd be off-by-default for users of Rust and would only be used for those using the crate in the ecosystem. I'll note though that as you've seen this requires nightly Rust since the crate doesn't compile on stable.

If using Rust stable is a requirement then I'd recommend updating the sqlite build script to pass -lclang_rt.builtins-wasm32 or similar. Another option would be to distribute that binary with the necessary objects with the sqlite3 crate.

[adamnovak]

If using Rust stable is a requirement then I'd recommend updating the sqlite build script to pass -lclang_rt.builtins-wasm32 or similar. Another option would be to distribute that binary with the necessary objects with the sqlite3 crate.

So this would involve doing something to the rusqlite crate so that it exposes to dependent projects that they need to link against the compiler-provided libcompiler-rt if they link against rusqlite? And then another per-platform environment variable like CC_wasm32_wasi to provide the platform-specific libcompiler-rt path down to rusqlite, so it can then send it back up to dependent crates? (Since I don't think the WASI SDK cc implementation has a way to ask for the path from the compiler binary?)

Because I think the intrinsics don't get linked in until the very end; I don't think rusqlite's build artifacts are meant to have the intrinsics already linked in and inlined.

[sbc100]

Regarding getting the name of libcompiler-rt you can use clang -print-libgcc-file-name (verified that this works with wasi-sdk).

[alexcrichton]

That would be a localized change to rusqlite, yes, and with @sbc100's suggestion you'd execute that and then print out cargo:rustc-link-search and cargo:rustc-link-lib. It shouldn't require any extra configuration on behalf of users other than already setting CC_wasm32_wasi.