Ambiguous packaging configuration

[gmacon]

Description

I'm using capa in a system that uses Nix to define the environment. That means that we rebuild the capa package from source within our environment. Building the source unmodified results in a broken installation because the sub-packages are missing. When I build outside of Nix, I get a working build, but with warnings like this one for every sub-package:

!!

        ********************************************************************************
        ############################
        # Package would be ignored #
        ############################
        Python recognizes 'capa.features' as an importable package[^1],
        but it is absent from setuptools' `packages` configuration.

        This leads to an ambiguous overall configuration. If you want to distribute this
        package, please make sure that 'capa.features' is explicitly added
        to the `packages` configuration field.

        Alternatively, you can also rely on setuptools' discovery methods
        (for example by using `find_namespace_packages(...)`/`find_namespace:`
        instead of `find_packages(...)`/`find:`).

        You can read more about "package discovery" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

        If you don't want 'capa.features' to be distributed and are
        already explicitly excluding 'capa.features' via
        `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
        you can try to use `exclude_package_data`, or `include-package-data=False` in
        combination with a more fine grained `package-data` configuration.

        You can read more about "package data files" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


        [^1]: For Python, any directory (with suitable naming) can be imported,
              even if it does not contain any `.py` files.
              On the other hand, currently there is no concept of package data
              directory, all directories are treated like packages.
        ********************************************************************************

!!

For the time being, I'm carrying a patch that adds all of the sub-packages explicitly to the tool.setuptools.packages list, but I think this should be addressed here as well. Y'all may have your own opinions about what the best way to resolve this is, I used the explicit list because it was fast.

Steps to Reproduce

git clone https://github.com/mandiant/capa
cd capa
python3 -m venv venv
. venv/bin/activate
pip install build
pyproject-build

Expected behavior:

The package is built without warnings.

Actual behavior:

Warnings like the one above appear for every sub-package of capa.

Versions

I got this exact output on a checkout of v6.1.0-781-gfbe0440361b8 with Python 3.9.6.

Additional Information

[mr-tz]

I can confirm the warnings. Do you want to submit a PR with your fixes?

[gmacon]

I opened a PR, but it's complaining that I'm not covered by a CLA. I might be able to get one signed if you give me a few months, but I really don't feel like fighting the bureaucracy for a change of this size.

[williballenthin]

@gmacon can you share with us where you learned to configure the package discovery correctly, so we can also learn and author a patch that does the same (without copying your work)?

i totally understand the CLA concerns in this case and appreciate your patience working with us. thank you!

[gmacon]

I read the page linked in the warning: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

If you want to talk to lawyers, though, I suspect this change is so small that there isn't anything copyrightable in it. There's also the possibility that what I did is in some sense unique; i.e. your change will be identical to mine even if you don't look at my change first.

[mr-tz]

I've read the guide and a few other pages and came to the same solution.
While doing this I've thought about moving the data files (in assets and sigs) under capa (which could be renamed to src) so we can include them in the builds. What do you think of that?

[gmacon]

You wouldn't rename capa to src, you would move capa to be under src (i.e. mkdir src; git mv capa src/capa), but that sounds fine to me otherwise.