Parsing header files for constants and structs

[simonbyrne]

One issue BinaryProvider doesn't address is how to extract header-defined constants and structs to create corresponding Julia objects. This is a problem when package ABIs can vary depending on compile-time flags, e.g. SuiteSparse (https://github.com/JuliaLang/julia/issues/34725).

One trick used in the past was to generate a small C file that would print out the constants, but these won't run in the cross-compilation environment (JuliaPackaging/Yggdrasil#505).

One potential solution is to allow calling Clang.jl as part of the build process (and have it set up to link to the correct headers, etc)?

[simonbyrne]

cc: @giordano

[giordano]

One potential solution is to allow calling Clang.jl as part of the build process (and have it set up to link to the correct headers, etc)?

Does Clang.jl only read the header files? If so, how does it handle different platforms with different macro predefined?

[simonbyrne]

Does Clang.jl only read the header files? If so, how does it handle different platforms with different macro predefined?

It requires that you provide all the system headers (which I presume the binarbuilder image provides): https://juliainterop.github.io/Clang.jl/stable/#C-bindings-generator-1

That said, I'm not sure if it can handle things like long varying between platforms.

[simonbyrne]

That said, I'm not sure if it can handle things like long varying between platforms.

Actually, that might not be an issue, since it probably wouldn't be evaluated at the Clang.jl.

[staticfloat]

It.... is indeed possible that we could integrate Clang.jl to do this. It's something that will have to go onto the wishlist for now; Clang.jl isn't that easy to use, last I tried. It's not quite as intelligent as clang itself; you need to do a lot of manual feeding of headers and directories and whatnot to get something useful. I think it's a good goal, but I would also want to see if we can get something working that uses, for instance, gcc or clang itself, as I'm sure that will be much less fragile. If all we need are compile-time constant values, we can build a tool that prints things at compile-time using $CC. Example from this SO question:

template<int s> struct Wow;
struct foo {
    int a,b;
};
Wow<sizeof(foo)> wow;

$ g++ -c test.cpp
test.cpp:5: error: aggregate ‘Wow<8> wow’ has incomplete type and cannot be defined

So we can build a tool that does something like says "give me the header files and the expression, I will shove the expression within the incomplete template, parse the error message, and give you back the result". Not the most elegant solution, but it should work in most simple cases. :P

[Gnimuc]

So we can build a tool that does something like says "give me the header files and the expression, I will shove the expression within the incomplete template, parse the error message, and give you back the result".

LibTooling is a great way for building such tools.

[simonbyrne]

Is there a way we could at least expose the necessary hooks so that we could try out something like @Gnimuc's example here JuliaPackaging/Yggdrasil#505 (comment)?

[Gnimuc]

I made an example here for those who would like to explore the way that calling Clang.jl as part of the build/prebuid process.

[fingolfin]

Isn't this kinda what https://github.com/analytech-solutions/CBinding.jl does?

[Gnimuc]

You can consider Clang.jl as an AOT version of CBinding.jl.

I made an example here for those who would like to explore the way that calling Clang.jl as part of the build/prebuid process.

This ends up being a multi-platform support module: https://github.com/JuliaInterop/Clang.jl/blob/master/src/shards/JLLEnvs.jl for setting up cross-compile environments locally without BB.

The OP has been fixed in this way: https://github.com/JuliaLang/SuiteSparse.jl/tree/master/lib