Rework the package -- Take 1

[Gnimuc]

Well, I ended up reworking the whole package.

Features added by this PR:

• Add a Generators module for wrapper generation. The context now maintains a DAG of the original AST of the C code. This DAG can be used for CTU analysis. The old wrap methods are split into different passes, this makes it easier to add features to the generator, e.g. printing useful C comments(fix Function and other signatures in comments #195).
• Expose a generic way for users who would like to add translation rules for custom definitions. (fix Minor issues when wrapping libav #81)
• Forward declarations in C code can now be correctly handled by the generator(fix Some header files entirely skipped #240), mutual-referenced structs are de-referenced at the pointer field types.
• All tag-types that have a special layout(__attribute__/bitfield/nested anonymous) are generated as a memory-pool-like struct. The size is queried from libclang, so it guarantees that issues like Potential wrong alignments and sizes of structs with C unions #238 won't happen(assume the code is system-independent). (fix Potential wrong alignments and sizes of structs with C unions #238, fix no method value(::CLPackedAttr) #224)
• High-level API is simplified. With this PR, the new API function create_context only accepts 3 arguments(headers, compiler arguments, options) instead of 8 arguments. Additional options are loaded from a .toml file. By default, the generator prints a complete module to a single file.
• The package is now self-contained.

In short, the goal is to make the wrapper generator easier to extend for new features.

As mentioned above, this PR is breaking. But I guess it's not that hard to upgrade the old scripts. I'm going to manually submit upgrading PRs to the downstream repos, which can also serve as a validation of the correctness of this PR.

TODO list:

• Add back basic macro support
• Generate nested anonymous structs(fix Unnamed struct members are skipped  #233)
• Generate getproperty/setproperty for structs with special layout
• Support vararg functions with the new @ccall macro(callback to support possible vararg definitions? #17, WARNING: Skipping VarArg Function CLCursor (Clang.cindex.FunctionDecl)  #146)
• Register LibClang.jl as a standalone package

Other bugfixes: fix #280, fix #279, fix #136, fix #126

[ihnorton]

I'm trying this out, and it looks amazing ❤️

The only thing I've run in to so far is handling of {u}int{32,64}_t arguments -- they seem to be translated as bare uint32_t identifier, which doesn't exist (adding const uint32_t = UInt32 etc. fixed it). I'm happy to open an issue if you prefer.

edit: specifically, looks like that only happens for return types, function args work fine.

[Gnimuc]

hi @ihnorton, could you share the C code that triggers the issue? I simplified the rule for translating function return types, but this change may be wrong for some corner cases.

[ihnorton]

The signatures are here. The driver is exactly as you posted here but changed the names and lib path.

Looking closer, I see what is going on (overlooked the warning comment 🙃 ): these are all classified as CXType_FunctionNoProto. In the current master I believe this situation is handled here:

Clang.jl/src/type.jl

Lines 208 to 222 in 01bf3e8

	"""
	resolve_type(t::CLType) -> CLType
	This function attempts to work around some limitations of the current libclang API.
	"""
	function resolve_type(t::CLType)::CLType
	if kind(t) == CXType_Unexposed
	# try to resolve Unexposed type to cursor definition.
	cursor = typedecl(t)
	if !isnull(cursor) && kind(cursor) != CXCursor_NoDeclFound
	return type(cursor)
	end
	end
	# otherwise, this will either be a builtin or unexposed client needs to sort out.
	return t
	end

(I had tried the same header a while back with an earlier master, and the return type was translated correctly as UInt64 etc.)

[Gnimuc]

Thanks for the hint. Should now be fixed by df52021.

[ihnorton]

Works perfectly now, thanks!

[vrzh]

This rework effort is looking great!

Is it on purpose that in test/test.toml, there is no "prologue" stuff included? otherwise Ctime_t will not be defined in the output:

prologue_file_path = "../gen/prologue.jl"

Also, I have been unlucky that the C library I am trying to generate bindings for has a function which has a parameter named Ptr, which conflicts with the Julia Ptr. Generating the bindings succeeds, but when trying them I was getting:

ERROR: LoadError: could not evaluate ccall argument type (it might depend on a local variable)

I got around it by:

import Clang.Generators.RESERVED_WORDS
# ...
push!(RESERVED_WORDS, "Ptr")

Does this warrant enough of a use-case to expose a function/macro to add other words which may conflict? Similar to @add_def which helps to add extra definitions?

[Gnimuc]

Is it on purpose that in test/test.toml, there is no "prologue" stuff included? otherwise Ctime_t will not be defined in the output:

I'm mainly running https://github.com/Gnimuc/GeneratorScripts for meaningful testing of the generated files, so here only the generator's public API is tested for now.

Does this warrant enough of a use-case to expose a function/macro to add other words which may conflict? Similar to @add_def which helps to add extra definitions?

@vrzh, thanks for the feedback! Without adding new features, a better fix would be writing a rewriter for that specific function like this one.

If you'd like to have this feature in Clang.jl, I'd suggest that we just add a new entry in the toml file like:

[codegen]
function_argument_conflict_symbols = ["Ptr"]  # should be empty by default

and prefix the argument names with "_" according to this list in the codegen.jl.

[Gnimuc]

@vrzh, OK, I just implemented that in the latest commit, could you give it a try?

[vrzh]

@vrzh, OK, I just implemented that in the latest commit, could you give it a try?

Doesn't seem to work for me. I am new to Julia so most likely my fault. I tried the following in a clean Julia environment.

test.jl:

using Pkg
Pkg.add(PackageSpec(name="Clang", rev="4fb6690"))

using Clang.Generators
using Clang.LibClang.Clang_jll

headers = [ "test.h" ]
options = load_options(joinpath(@__DIR__, "test.toml"))
args = [ "-I." ]

# import Clang.Generators.RESERVED_WORDS
# push!(RESERVED_WORDS, "Ptr")

ctx = create_context(headers, args, options)

build!(ctx)

test.toml:

[general]
library_name = "libtest"

[codegen]
function_argument_conflict_symbols = ["Ptr"] 

test.h:

void f(int * x, int Ptr);

Output:

function f(x, Ptr)
    ccall((:f, libtest), Cvoid, (Ptr{Cint}, Cint), x, Ptr)
end

I agree with you that a rewriter is the best way to go (as you could also have C function names, C struct names, etc. which also clash), but an entry in the toml file as you suggest is the easiest for new users.

[Gnimuc]

@vrzh Oops! Should be fixed now.

I agree with you that a rewriter is the best way to go (as you could also have C function names, C struct names, etc. which also clash), but an entry in the toml file as you suggest is the easiest for new users.

As C doesn't have any name mangling stuff, a common C library usually has its own manually mangled prefixes, so generally speaking, function/struct name collisions rarely happen in practice.

[vrzh]

@vrzh Oops! Should be fixed now.

Perfect! Works great, thanks a lot.

[Gnimuc]

This PR is too long, so let's merge this to master.