LLVM bindings have become incorrect in places.

[eddyb]

Because of x86/x86_64 specifics, certain LLVM functions that used to return unsigned int and now return unsigned long long continue to be usable, ending up in truncated return values.

pub fn LLVMABISizeOfType(TD: TargetDataRef, Ty: TypeRef) -> c_uint;

unsigned long long LLVMABISizeOfType(LLVMTargetDataRef TD, LLVMTypeRef Ty);

This is just one example of such incorrectness arising over time, there could be more.
Would be nice if we could use bindgen every LLVM upgrade to verify our bindings (or just generate new ones).

[ahmedcharles]

Just curious, what's bindgen? Are there docs on how to use it?

[eddyb]

@ahmedcharles I've updated the description with a link to bindgen.

[steveklabnik]

I spoke about this with @eddyb on IRC today. This is especially pernicious, because we could be accidentally getting UB in corner cases.

A tactic to close this bug:

1. Generate headers from LLVM with bindgen
2. Run diff
3. see if any of it is a particular issue.

Humans looking at it will probably be needed, given that bindgen isn't perfect.

[tamird]

I took a look at this, starting with the rather naive:

#!/usr/bin/env sh

set -eu

LLVM_CONFIG_PATH=$(find build -name llvm-config -type f | head -n1)
CXX_FLAGS=$($LLVM_CONFIG_PATH --cxxflags)

echo CXX_FLAGS: $CXX_FLAGS

RUST_BACKTRACE=1 bindgen $@ src/rustllvm/rustllvm.h -o src/librustc_llvm/ffi.rs -- $CXX_FLAGS

which produces the output:

$ ./bindgen.sh
CXX_FLAGS: -I/Users/tamird/src/rust/build/x86_64-apple-darwin/llvm/include -ffunction-sections -fdata-sections -fPIC -m64 -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -Werror=date-time -std=c++11 -O3 -DNDEBUG -fno-exceptions -fno-rtti -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
error: invalid argument '-std=c++11' not allowed with 'C/ObjC', err: true
thread 'main' panicked at 'Unable to generate bindings: ()', src/libcore/result.rs:906:4
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: std::panicking::try::do_call
  11: __rust_maybe_catch_panic
  12: bindgen::main
  13: __rust_maybe_catch_panic
  14: std::rt::lang_start

[tamird]

Using --cflags rather than --cxxflags produces c++ parsing failures:

$ ./bindgen.sh
C_FLAGS: -I/Users/tamird/src/rust/build/x86_64-apple-darwin/llvm/include -ffunction-sections -fdata-sections -fPIC -m64 -stdlib=libc++ -fPIC -Wall -W -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wdelete-non-virtual-dtor -Wstring-conversion -Werror=date-time -O3 -DNDEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:263:9: error: unknown type name '__char16_t'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:264:9: error: unknown type name '__char32_t'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:403:1: error: unknown type name 'namespace'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:403:14: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:598:1: error: unknown type name 'template'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:598:10: error: expected identifier or '('
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:599:1: error: unknown type name 'template'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:599:10: error: expected identifier or '('
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstddef:44:1: error: unknown type name 'namespace'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstddef:44:1: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstring:67:1: error: unknown type name 'namespace'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstring:67:1: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:135:38: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:136:30: error: variable has incomplete type 'void'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:136:38: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:137:30: error: variable has incomplete type 'void'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:137:38: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:144:38: error: expected ';' after top level declarator
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:149:38: error: expected ';' after top level declarator
fatal error: too many errors emitted, stopping now [-ferror-limit=]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:263:9: error: unknown type name '__char16_t', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:264:9: error: unknown type name '__char32_t', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:403:1: error: unknown type name 'namespace', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:403:14: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:598:1: error: unknown type name 'template', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:598:10: error: expected identifier or '(', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:599:1: error: unknown type name 'template', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:599:10: error: expected identifier or '(', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstddef:44:1: error: unknown type name 'namespace', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstddef:44:1: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstring:67:1: error: unknown type name 'namespace', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cstring:67:1: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:135:38: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:136:30: error: variable has incomplete type 'void', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:136:38: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:137:30: error: variable has incomplete type 'void', err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:137:38: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:144:38: error: expected ';' after top level declarator, err: true
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/new:149:38: error: expected ';' after top level declarator, err: true
fatal error: too many errors emitted, stopping now [-ferror-limit=], err: true
thread 'main' panicked at 'Unable to generate bindings: ()', src/libcore/result.rs:906:4
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: std::panicking::try::do_call
  11: __rust_maybe_catch_panic
  12: bindgen::main
  13: __rust_maybe_catch_panic
  14: std::rt::lang_start

[tamird]

Ah, renaming rustllvm.h to rustllvm.hpp gets this further, but we hit rust-lang/rust-bindgen#492:

./bindgen.sh 2>bindgen.stderr

gist containing bindgen.sh & bindgen.stderr

[FreezyLemon]

If someone's interested in this (like me, originally), I'll save you some time:

The above PR #46353 was rejected due to A) bindgen needing an unwanted dependency and B) the team preferring a validation of the bindings over re-generating the bindings automatically via bindgen. (This is understandable, as the hand-written bindings are way nicer than bindgens output)

ctest was proposed as a crate that allows testing the existing bindings. However, after some research, ctest is pretty much abandoned and even the fork ctest2 suffers from outdated dependencies, which results in the crate practically being unable to handle rustc_codegen_llvm which is the crate that would need to be parsed. I tried.

The way I see it, the validation can't be reliably automated at the moment unless ctest2 gets a major update/rewrite or an alternative pops up (maybe with syn-based parsing).

Edit: Actually, it might be possible. For example. libz-sys uses ctest2. I guess the code in libz-sys is simple enough to be parsed by the outdated syntex crate. Technically, the bindings could be moved to a separate crate (e.g. rustc_llvm_bindings) and if the code is simple/"old" enough, syntex and ctest should be able to handle it.

[cuviper]

I think the bindings should probably move to rustc_llvm, since that's the one that actually prepares the link library, like a typical *-sys crate. That's also where we define our C++ shims, then we would have all LLVM ffi in one place.

[FreezyLemon]

Oh, I didn't know a crate like that already existed. I'll take a look, maybe it's possible to get ctest to work with it.

[FreezyLemon]

ctest still fails at parsing even after moving ffi.rs to rustc_llvm. It might be possible by refactoring the code, but it's not trivial. It's honestly not something I'm willing to spend more time on at the moment. It might make more sense to "just wait" and hope ctest gets an update