[BUG] attr: not working in version field of setup.cfg

[solsword]

setuptools version

62.1.0

Python version

3.10.2

OS

macOS

Additional environment information

I get the same error in each of:

• Python 3.7.9 with setuptools 47.1.0
• Python 3.9.10 with setuptools 62.0.0
• Python 3.10.2 with setuptools 62.1.0

My Python versions 3.9 and 3.10 were installed using homebrew, and 3.7.9 comes with Thonny.

Description

I'm trying to build a basic package and maintain a version number in one place, using declarative setup.cfg and building with python -m build. I'm following option #1 from this guide:

https://packaging.python.org/en/latest/guides/single-sourcing-package-version/

Which also corresponds with the example provided at the top of this page:

https://setuptools.pypa.io/en/latest/userguide/declarative_config.html

I've looked at issues like:

• [BUG] Using attr for version specification with namespace packages #2919 (with namespace packages)
• [BUG] __version__ must be imported first to be used as attr #3176 (with other imports involved)

But I'm not using a namespace package and I'm not using any imports (see file contents below). Essentially, even if the package JUST has version: attr:mypackage.__version__, the build fails. Interestingly, it's able to create the egg info directory, and picks up the package version correctly there, but fails with a ModuleNotFoundError at a later step. The full log is included at the end of this report.

I can confirm that with at least the 3.9.10 Python and setuptools 62.0.0, this issue does NOT occur if the package is provided as a single file rather than as a directory with an __init__.py file. Perhaps the bug is as simple as a code path in the AST parsing branch (which we seem to be triggering) that isn't checking for that and assumes mypackage.__version__ could only be found in mypackage.py? I have tried using attr:mypackage.__init__.__version__ but unsurprisingly that didn't work. I even tried putting the version number in a different file mypackage/_verison.py and using attr:mypackage._version.__version__ but that didn't work either.

This seems similar to #2605, but I'm not getting the first error from that bug involving NoneType, and from the info in that bug, it seems like it might be related to #2919 if they're using namespace packages, which I'm not.

Note: If it turns out I'm doing something I shouldn't be, it might be worth considering a docs update in one of the places I pointed to, since I feel like I'm following the documentation to the letter here.

Files used to reproduce (I also included the creation of these files in my reproduction steps):

pyproject.toml:

[build-system]
requires = [ "setuptools", "wheel" ]
build-backend = "setuptools.build_meta"

setup.cfg:

[metadata]
name = pkg
version = attr: pkg.__version__
description = Test for version attr: bug
long_description = N/A
url = https://www.example.com
author = Peter Mawhorter
author_email = pmawhort@wellesley.edu
license = N/A
classifiers =
    Programming Language :: Python :: 3 :: Only

[options]
python_requires = >=3.7
provides = pkg
py_modules = pkg

pkg.__init__.py (just this one line, nothing else):

__version__ = "0.1.1"

Expected behavior

Creating a minimal package using attr: my_project.__version__ in seutp.cfg and building using python -m build should work without errors, even if the package is a directory with an __init__.py where the version variable is defined, especially if that __init__.py doesn't have any other imports in it.

How to Reproduce

These commands create the pyproject.toml, setup.cfg and pkg/__init__.py files described above and run python -m build:

#!/bin/sh
echo '[build-system]\nrequires = [ "setuptools", "wheel" ]\nbuild-backend = "setuptools.build_meta"' > pyproject.toml
echo '[metadata]\nname = pkg\nversion = attr: pkg.__version__\ndescription = Test for version attr: bug\nlong_description = N/A\nurl = https://www.example.com\nauthor = Peter Mawhorter\nauthor_email = pmawhort@wellesley.edu\nlicense = N/A\nclassifiers =\n    Programming Language :: Python :: 3 :: Only\n\n[options] python_requires = >=3.7\nprovides = pkg\npy_modules = pkg' > setup.cfg
mkdir pkg
echo '__version__ = "0.1.1"' > pkg/__init__.py
python -m build

Output

* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools, wheel)
* Getting dependencies for sdist...
running egg_info
creating pkg.egg-info
writing pkg.egg-info/PKG-INFO
writing dependency_links to pkg.egg-info/dependency_links.txt
writing top-level names to pkg.egg-info/top_level.txt
writing manifest file 'pkg.egg-info/SOURCES.txt'
file pkg.py (for module pkg) not found
reading manifest file 'pkg.egg-info/SOURCES.txt'
writing manifest file 'pkg.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing pkg.egg-info/PKG-INFO
writing dependency_links to pkg.egg-info/dependency_links.txt
writing top-level names to pkg.egg-info/top_level.txt
file pkg.py (for module pkg) not found
reading manifest file 'pkg.egg-info/SOURCES.txt'
writing manifest file 'pkg.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
creating pkg-0.1.1
creating pkg-0.1.1/pkg.egg-info
copying files to pkg-0.1.1...
copying pyproject.toml -> pkg-0.1.1
copying setup.cfg -> pkg-0.1.1
copying pkg.egg-info/PKG-INFO -> pkg-0.1.1/pkg.egg-info
copying pkg.egg-info/SOURCES.txt -> pkg-0.1.1/pkg.egg-info
copying pkg.egg-info/dependency_links.txt -> pkg-0.1.1/pkg.egg-info
copying pkg.egg-info/top_level.txt -> pkg-0.1.1/pkg.egg-info
Writing pkg-0.1.1/setup.cfg
Creating tar archive
removing 'pkg-0.1.1' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools, wheel)
* Getting dependencies for wheel...
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
    return self._get_build_requires(
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
    self.run_setup()
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 1, in <module>
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 122, in setup
    dist.parse_config_files()
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/dist.py", line 850, in parse_config_files
    setupcfg.parse_configuration(
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 167, in parse_configuration
    meta.parse()
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 446, in parse
    section_parser_method(section_options)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 417, in parse_section
    self[name] = value
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 238, in __setitem__
    value = parser(value)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 552, in _parse_version
    return expand.version(self._parse_attr(value, self.package_dir, self.root_dir))
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/setupcfg.py", line 372, in _parse_attr
    return expand.read_attr(attr_desc, package_dir, root_dir)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/expand.py", line 188, in read_attr
    spec = _find_spec(module_name, path)
  File "/private/var/folders/sb/njhjnvlx3wz0f_8l_szm1m000000gp/T/build-env-8agqk1_z/lib/python3.10/site-packages/setuptools/config/expand.py", line 203, in _find_spec
    raise ModuleNotFoundError(module_name)
ModuleNotFoundError: pkg

ERROR Backend subproccess exited when trying to invoke get_requires_for_build_wheel

[abravalheri]

Hi @solsword, thank you very much for reporting this.

I think the main problem is the following:

  [options]
  python_requires = >=3.7
  provides = pkg
- py_modules = pkg
+ packages = pkg

Note: If it turns out I'm doing something I shouldn't be, it might be worth considering a docs update in one of the places I pointed to, since I feel like I'm following the documentation to the letter here.

What would be your suggestion?
In the https://setuptools.pypa.io/en/latest/userguide/declarative_config.html page all the examples use the packages keyword.
The expressions packages and modules have different meanings given by language itself.

[solsword]

Thanks for the quick response! I actually just figured this out myself as well, and indeed, that was the issue.

Something that would have helped me in this case is some kind of error or warning message when py_modules and/or py_packages is used incorrectly!

The ultimate cause of the ModuleNotFound is that the build package when trying to build a wheel from the sdist fails, because since I screwed up the py_modules vs. packages option, the sdist didn't have the package in it (of course, it doesn't have pkg.py either, because that file doesn't exist). I think that setuptools should complain if asked to include a module that doesn't exist (or vice versa, a package that doesn't exist). But maybe there's a use-case for that that I'm overlooking?

It's actually good that the build crashes in a way because it indicates that the source distribution is missing something important! But googling the specific error message that I got led me down the wrong rabbit-hole.

Sorry for the mostly-spurious bug here! I'm not closing because I think some kind of error or warning about the mis-use of py_modules vs. pacakges would be helpful. This error came about because unsurprisingly I forget a lot of config details between starting new projects, and I copied over a config from a single-file project into my pacakge-based project.

Note that if you try to set packages = nope naming a non-existent package, then the build system gives a nice error: "error: package directory 'nope' does not exist". But if you do the same thing with a module that doesn't exit, setting py_modules = nope it "builds" an empty sdist without that module, but without any kind of error or warning.

Looking into the code, in at least one place (line 278 of _distutils/command/build_py.py) the code says:

        if not self.check_module(module, module_file):
            continue

so there is code to check that a module exists, but if it doesn't, we ignore that fact (the parallel code for packages prints the error I just mentioned). Maybe that line could print a warning at least? The comment implies that there might be a .pyc file which could have been the motivation for ignoring missing .py files?

[abravalheri]

so there is code to check that a module exists, but if it doesn't, we ignore that fact (the parallel code for packages prints the error I just mentioned). Maybe that line could print a warning at least? The comment implies that there might be a .pyc file which could have been the motivation for ignoring missing .py files?

That makes some sense to me... With packages that is very clear right? Even if you have a __init__.pyc file instead of a .py, the package is characterized by the existence of a folder, right?

I don't know exactly how to proceed here, but it seems that this issue involves changes in https://github.com/pypa/distutils... would you like to give it a go and propose a PR there?

[solsword]

Hmmm. That repo says that it's a shadow of the builtin distutils which are scheduled for removal, presumably in favor of setuptools... So I feel like even if they were receptive to adding code to generate a warning, and even if that got ported to Python core, it would eventually be phased out and replaced by the code in this package? But maybe I'm misunderstanding the status of this repo, that one, and the Python built-in code itself. Their README does say (about synchronizing with setuptools):

Simply merge the changes directly into setuptools' repo.

So if the version of build_py.py which is going to be used once distutils is removed from the python stdlib is the one in _distutils inside of setuptools, this seems like the place to make the change?

[abravalheri]

There is always a chance the maintainers in distutils might decide that the change is not worthy. But since it is a purely informational change that does not affect the way the package works, they might accept it.

The workflow setuptools use is to constantly merge the changes from distutils into the setuptools._distutils folder... So if we change things directly into setuptools they might be overwritten after a while...

[solsword]

Okay that makes perfect sense to me. I'll file a request over there and reference this; I'll mark this as closed now since further work belongs there.