Incorporating Cythonized shared library files in the build workflow

[erickpeirson]

Moving this over from Slack

---

Hey folks, I have a use case in Python that I suspect isn't terribly special, and I wonder if folks could point me in the right direction.

I have a codebase with a large library, and a handful of applications that use the library. The final distribution target is a container image, and I'm hoping to build the application itself as a Pex. When we ship the final build, the modules in the library are cythonized to shared library files. We do this today via setuptools build_ext, and implement our own cache so that we only rebuild modules that have changed. I can define a distribution target for the library in Pants that triggers the build_ext routine, but in this case our cache doesn't work. And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload.

It seems like what I want is to be able to define a target for each eventual shared library file, with dependencies looking something like:

pex -> application code -> .so file -> .py source

Before I dive too deep into the plugin framework, any pointers or feedback? If I were to go the route of writing a plugin, any words of advice for how to think about this?

[erickpeirson]

@alexey-tereshenkov-oxb replied:

---

welcome! 🙂

Let's brainstorm!

I can define a distribution target for the library in Pants that triggers the build_ext routine, but in this case our cache doesn't work.

Right, so you are thinking of python_distribution target either with an existing setup.py or a generated one. You say the cache doesn't work because when building a Python wheel, Pants will re-compile all your sources (using the setup.py tooling), am I understanding it right?

And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload.

So the PEX files are not related to the python_distribution target (your Python wheel), are they? The pex_binary target would only understand the Python sources, whereas you want to tell what .so files to stuff into the .pex file from the build directory produced when Python wheel was generated?

[erickpeirson]

You say the cache doesn't work because when building a Python wheel, Pants will re-compile all your sources (using the setup.py tooling), am I understanding it right?

Jup, precisely.

The pex_binary target would only understand the Python sources, whereas you want to tell what .so files to stuff into the .pex file from the build directory produced when Python wheel was generated?

Yes, just about. I should clarify that I actually don't need to preserve the setuptools-based wheel-building, that's just how we do it today. Ideally, each .so could be produced separately as part of the build process.

[benjyw]

I believe someone has written a cython plugin for internal use, but I don't recall who. I just see it mentioned as an example here: https://www.pantsbuild.org/docs/how-does-pants-work#a-powerful-plugin-system

[erickpeirson]

Haha, I noticed that -- quite a tease!

[benjyw]

Ah, here is more context: #14752

[benjyw]

So cc @sureshjoshi and @houtenjack

[benjyw]

@erickpeirson I know little about Cython, but are you just using it to speed up otherwise-pure-python code in .py files? Or are there also .c source files involved?

[erickpeirson]

Hey @benjyw , thanks for jumping in. We're just speeding up pure Python, so there should not be any other dependencies to track (I hope)

[benjyw]

I'll chew on this a little before throwing down opinions...

[erickpeirson]

I'm still getting my head around the Pants architecture, so this may be a total red herring, but it feels like there is a similarity here to Protobuf code generation: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/codegen/protobuf/python/rules.py

[benjyw]

Hmm, I think we'd want the behavior to be controlled per target, in the BUILD file. Otherwise attempting to use cython on just a tiny corner of the codebase will end up using it everywhere.

[benjyw]

Hi, just checking in on this. Possibly codegen is in fact the way to go here, the main tricky design bit is the target type modeling. Because it has to support operating (e.g., running tests) on the uncythonized Python, presumably.

Is the cythonizing just for building dists? Would you run tests on the uncompiled python?

[erickpeirson]

Is the cythonizing just for building dists? Would you run tests on the uncompiled python?

We would want to run tests on both, actually. Running tests against uncompiled Python is nice for calculating line coverage. But we find that there are cases in which the Cythonized version behaves slightly differently, so testing the behavior with the SOs is also important.

[benjyw]

Makes sense. So would you want ./pants test :: to run tests on both every time, or would it be more like ./pants test :: and ./pants --no-python-cython test :: (or something like that)?

[erickpeirson]

Ideally the latter:

./pants test :: and ./pants --no-python-cython test ::

[tachikoma-li]

Hi @benjyw, thanks for all the work in Pants. It's a nice tool to use!
Just wondering if there is any progress on this feature.

We have a similar requirement in our project, to generate .so from .pyx using Cython. I managed to write a simple plugin following the add-codegen tutorial. The plugin seems to work when running pants export-codegen ::, which exports the expected .so to /dist/codegen. However, while running with pytest or building the package, the codegen doesn't seem to get called. I have already used the enable_codegen flag within the plugin and added the dependencies of the cython files (.pyx) in the corresponding BUILD file. Running pants dependencies shows the test depends on the .pyx file. However, the codegen is not used. I noticed in your above comment

the main tricky design bit is the target type modeling. Because it has to support operating (e.g., running tests) on the uncythonized Python, presumably.

I guess the difference between the Cython plugin and the Protobuf example is that we are generateing .so instead of Python source files. Is there anything else needed for this to work in tests/building?

Edit: seems one needs to define the type for .so to inherit from PythonSourceField, rather than SingleSourceField. That way, it would be included when Pants tries the get the dependencies for the test. Now pytest can work. But there is still some issue with building the package. From the debug info, I can see the .so file is generated in one sandbox but is not included in the final wheel file. I guess this might be due to some misconfiguration somewhere.