pypa/build on Linux - extension / binary modules in purelib

[tcuongd]

Hello! I'm building a package on Linux with extension modules, using setuptools + wheel, and pypa/build to run the build command. I've added a custom distribution class to force platform-specific wheels:

class BinaryDistribution(Distribution):
    """Distribution which always forces a binary package with platform name"""
    def has_ext_modules(self):
        return True

    def is_pure(self):
        return False

setup(..., distclass=BinaryDistribution)

This successfully generates a Linux-specific wheel. Now I want to "repair" the wheel in order to publish it to PyPi. I'm using pypa/auditwheel for this, but it gives this error:

RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:
  	prophet_model.bin
  	diagnose
  	print
  	stanc
  	stansummary
  	diagnose.o
  	print.o
  	stansummary.o
  	bernoulli
  	main.o
The wheel has to be platlib compliant in order to be repaired by auditwheel.

I had initially thought it could be an issue with auditwheel (posted a question here: pypa/auditwheel#333 (comment)) but I noticed that auditwheel is simple checking the metadata of the wheel here. I'm wondering if there's actually an issue in the wheel creation that's causing the extension modules to land in purelib. Would really appreciate any tips / where to start looking to solve this!

[henryiii]

Linux with extension modules

If you have extension modules, setuptools will always make this platform specific. I'm guessing you are trying to add them manually instead of using setuptools? If that's the case, I'd generally override the extension build mechanism rather than distclass, such as in https://github.com/pybind/python_example/blob/master/setup.py

[FFY00]

This shouldn't be an issue with pypa/build as we simply provision the environment and invoke the backend, we take no part in the wheel building itself.

Could you provide more details on what you are doing, and perhaps a reproducible too? It seems to you are missing patching in someplace.

[FFY00]

I would transfer this to the setuptools repo, but I can't. @jaraco could you? Thanks 😊

[henryiii]

Should this go to packaging-problems, perhaps?

[FFY00]

Well, I guess it is close enough, but this really is about customizing the setuptools build.

[henryiii]

I don't think this is a bug in setuptools, but rather a question on how to do something slightly tricky in packaging (and it touches several projects, auditwheel + setuptools at least).

[FFY00]

I guess, but the root issue really is setuptools, auditwheel is just complaining about a bad wheel. Feel free to move it to packaging-problems, it is not that clear where this belongs.

[henryiii]

Let's start there. I think we should first assume it's a user issue, and if it turns out to be a setuptools bug, it can move there. Same thing with issues about build, most of them start in packaging-problems and never need to move, as they are user issues with setup.py/setup.cfg. Once in a while, they are bugs in build, and then we move them here.

[layday]

The problem is that you're declaring the root to be "platlib" but the stuff you're packaging setuptools/wheel believes to be pure, so it places them in xyz.data/purelib in the wheel. Then when auditwheel inspects the wheel it finds them there and errors out. I assume this is for facebook/prophet - you don't need a custom distclass here - you just need to subclass build_ext instead of build_py.

[layday]

Looks like you got it sorted so I'll go ahead and close this one.