Document how to use meson to build a cffi cextension

[mattip]

setuptools is not the only python build backend. Projects in the scientific python stack have moved to meson. It would be nice to provide an example of how to best use meson to build a cffi c-extension module.

[mattip]

I think this would help @minrk in ported pyzmq to meson.

[inklesspen]

I believe the big issue with this is that cffi relies on setuptools in some cases; specifically, it uses the distutils Distribution and Extension classes to compile extension modules, and as of Python 3.12 these have been removed from the stdlib and are only available in setuptools.

However, distutils is only actually needed for the step which automatically compiles the generated c source. Unfortunately, at the moment doing without this requires digging a little deeper into the internals of cffi. (I suspect it could be made simpler to access, but I don't know what constraints the cffi devs are working under here.)

from cffi import FFI

ffibuilder = FFI()

ffibuilder.cdef("int abs(int);")

ffibuilder.set_source("_simple", "#include <stdlib.h>")

if __name__ == "__main__":
    from cffi.recompiler import Recompiler
    # This code is based on FFI.emit_c_code and the functions it calls.
    # kwds are options which would be passed to the distutils Extension class
    # see https://setuptools.pypa.io/en/stable/userguide/ext_modules.html#setuptools.Extension
    # this may or may not be relevant when using meson
    module_name, source, source_extension, kwds = ffibuilder._assigned_source
    recompiler = Recompiler(ffibuilder, module_name)
    recompiler.collect_type_table()
    recompiler.collect_step_tables()
    filename = module_name + source_extension
    with open(filename, "w") as f:
        recompiler.write_source_to_f(f, source)

This Python script can be run with just cffi as a dependency. It generates the C extension source and writes it to a file; it should be obvious how to make it write to a different location. I suspect you can make this work with meson's Generating sources support.

[arigo]

@inklesspen This might be an oversight. Would it help if we move the line 1546 of recompile.py into the if call_c_compiler: branch a few lines below?

diff --git a/src/cffi/recompiler.py b/src/cffi/recompiler.py
index 4167bc05..19522470 100644
--- a/src/cffi/recompiler.py
+++ b/src/cffi/recompiler.py
@@ -1543,10 +1543,10 @@ def recompile(ffi, module_name, preamble, tmpdir='.', call_c_compiler=True,
             else:
                 target = '*'
         #
-        ext = ffiplatform.get_extension(ext_c_file, module_name, **kwds)
         updated = make_c_source(ffi, module_name, preamble, c_file,
                                 verbose=compiler_verbose)
         if call_c_compiler:
+            ext = ffiplatform.get_extension(ext_c_file, module_name, **kwds)
             patchlist = []
             cwd = os.getcwd()
             try:

EDIT: right, it needs more care because sometimes it also returns ext. The point is, this ext is not used when emit_c_code() is called, so my question is: is that the only line causing problem in your case? We can refactor the code a little bit to avoid calling ffiplatform.get_extension() in this situation.

[inklesspen]

The issue there is the else branch on if call_c_compiler; it returns the ext instance, for use in FFI.distutils_extension. Essentially you have three use cases for recompile: generate c source, generate c source and compile it, generate c source and distutils metadata. And the boolean logic doesn't work as well with that, as you've noticed.

My ideal scenario would be a version of emit_c_code which accepts a file-like object instead of a filename, and which can output c source to that file-like object (without using ext, and therefore without using distutils). But I suspect I will also want access to the contents of the _assigned_source tuple, and I would prefer not to use underscore-prefixed properties… (Specifically, I think module_name and kwds may be useful to a consuming build tool.)

[arigo]

OK, that makes sense. I'm going to check what makes sense (and still works generally instead of just in one case---there are a lot of other cases in recompile()...). But feel free to give a patch or open a pull request, too.

[inklesspen]

@mattip I put together https://github.com/inklesspen/meson-python-cffi-example which shows one possible way to do this.

[inklesspen]

Since the PRs are not getting merged, I decided to make a tool to help solve the issue: https://pypi.org/project/cffi-buildtool/

[arigo]

I "force-merged" #81. Sorry, I'm really not used to programming-via-clicking-links-on-github. (We moved to github because nowadays there is no real alternative. I kinda hate git's approach about many things but there is no choice anymore.)

[inklesspen]

Thanks. There's one bit of this that still strikes me as being excessively convoluted:

I call ffi.emit_c_code('somefile.c'). This in turn passes that filename into recompile(...), which passes it into make_c_source(...), and then into _make_c_or_py_source(...)

_make_c_or_py_source creates a Recompiler instance, calls a few setup methods, then creates an io.StringIO instance, has the Recompiler send output to it, gets the value, then opens the output file based on the passed-in filename and writes into it.

It would be neat if I could pass a file-like object into emit_c_code instead of a filename, and then this would get passed down the chain and used instead of the StringIO instance. Otherwise, if the calling code wants to get the C source as a string, it has to use the tempfile module or something similar.

[arigo]

Sure, that makes sense to me. You can propose a PR or else I'll get to it sometime in the future.