listing git files failed on git repo

[jaraco]

While troubleshooting pypa/distutils#224, I've stumbled onto another issue.

I'm running pyproject-build . on a number of projects, and in the output, I see the following warning for both sdist and wheel generation:

ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any

This is happening across multiple projects I develop.

 draft @ git clone gh://jaraco/pip-run
Cloning into 'pip-run'...
remote: Enumerating objects: 3447, done.
remote: Counting objects: 100% (819/819), done.
remote: Compressing objects: 100% (318/318), done.
remote: Total 3447 (delta 483), reused 788 (delta 467), pack-reused 2628
Receiving objects: 100% (3447/3447), 688.86 KiB | 6.75 MiB/s, done.
Resolving deltas: 100% (2058/2058), done.
 draft @ cd pip-run
 pip-run main @ pyproject-build -V
build 1.0.3 (/Users/jaraco/.local/pipx/venvs/build/lib/python3.12/site-packages/build)
 pip-run main @ pyproject-build .
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=56, setuptools_scm[toml]>=3.4.1)
* Getting build dependencies for sdist...
running egg_info
creating pip_run.egg-info
writing pip_run.egg-info/PKG-INFO
writing dependency_links to pip_run.egg-info/dependency_links.txt
writing entry points to pip_run.egg-info/entry_points.txt
writing requirements to pip_run.egg-info/requires.txt
writing top-level names to pip_run.egg-info/top_level.txt
writing manifest file 'pip_run.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pip_run.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing pip_run.egg-info/PKG-INFO
writing dependency_links to pip_run.egg-info/dependency_links.txt
writing entry points to pip_run.egg-info/entry_points.txt
writing requirements to pip_run.egg-info/requires.txt
writing top-level names to pip_run.egg-info/top_level.txt
adding license file 'LICENSE'
writing manifest file 'pip_run.egg-info/SOURCES.txt'
running check
creating pip-run-12.4.1.dev26+gbf42021
creating pip-run-12.4.1.dev26+gbf42021/.github
creating pip-run-12.4.1.dev26+gbf42021/.github/workflows
creating pip-run-12.4.1.dev26+gbf42021/docs
creating pip-run-12.4.1.dev26+gbf42021/examples
creating pip-run-12.4.1.dev26+gbf42021/pip_run
creating pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
creating pip-run-12.4.1.dev26+gbf42021/pip_run/retention
creating pip-run-12.4.1.dev26+gbf42021/tests
copying files to pip-run-12.4.1.dev26+gbf42021...
copying .coveragerc -> pip-run-12.4.1.dev26+gbf42021
copying .editorconfig -> pip-run-12.4.1.dev26+gbf42021
copying .pre-commit-config.yaml -> pip-run-12.4.1.dev26+gbf42021
copying .readthedocs.yaml -> pip-run-12.4.1.dev26+gbf42021
copying LICENSE -> pip-run-12.4.1.dev26+gbf42021
copying NEWS.rst -> pip-run-12.4.1.dev26+gbf42021
copying README.rst -> pip-run-12.4.1.dev26+gbf42021
copying conftest.py -> pip-run-12.4.1.dev26+gbf42021
copying mypy.ini -> pip-run-12.4.1.dev26+gbf42021
copying pip-run.py -> pip-run-12.4.1.dev26+gbf42021
copying pyproject.toml -> pip-run-12.4.1.dev26+gbf42021
copying pytest.ini -> pip-run-12.4.1.dev26+gbf42021
copying ruff.toml -> pip-run-12.4.1.dev26+gbf42021
copying setup.cfg -> pip-run-12.4.1.dev26+gbf42021
copying towncrier.toml -> pip-run-12.4.1.dev26+gbf42021
copying tox.ini -> pip-run-12.4.1.dev26+gbf42021
copying .github/FUNDING.yml -> pip-run-12.4.1.dev26+gbf42021/.github
copying .github/dependabot.yml -> pip-run-12.4.1.dev26+gbf42021/.github
copying .github/workflows/main.yml -> pip-run-12.4.1.dev26+gbf42021/.github/workflows
copying docs/conf.py -> pip-run-12.4.1.dev26+gbf42021/docs
copying docs/cowsay.svg -> pip-run-12.4.1.dev26+gbf42021/docs
copying docs/history.rst -> pip-run-12.4.1.dev26+gbf42021/docs
copying docs/index.rst -> pip-run-12.4.1.dev26+gbf42021/docs
copying examples/plotter.ipynb -> pip-run-12.4.1.dev26+gbf42021/examples
copying examples/pydragon -> pip-run-12.4.1.dev26+gbf42021/examples
copying examples/pydragon.py -> pip-run-12.4.1.dev26+gbf42021/examples
copying examples/test-mongodb-covered-query.py -> pip-run-12.4.1.dev26+gbf42021/examples
copying pip_run/__init__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/__main__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/_py38compat.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/commands.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/deps.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/launch.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/read-deps.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/scripts.py -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run/usage.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run
copying pip_run.egg-info/PKG-INFO -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run.egg-info/SOURCES.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run.egg-info/dependency_links.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run.egg-info/entry_points.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run.egg-info/requires.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run.egg-info/top_level.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
copying pip_run/retention/__init__.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention
copying pip_run/retention/destroy.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention
copying pip_run/retention/persist.py -> pip-run-12.4.1.dev26+gbf42021/pip_run/retention
copying tests/test_commands.py -> pip-run-12.4.1.dev26+gbf42021/tests
copying tests/test_deps.py -> pip-run-12.4.1.dev26+gbf42021/tests
copying tests/test_launch.py -> pip-run-12.4.1.dev26+gbf42021/tests
copying tests/test_scripts.py -> pip-run-12.4.1.dev26+gbf42021/tests
copying pip_run.egg-info/SOURCES.txt -> pip-run-12.4.1.dev26+gbf42021/pip_run.egg-info
Writing pip-run-12.4.1.dev26+gbf42021/setup.cfg
Creating tar archive
removing 'pip-run-12.4.1.dev26+gbf42021' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools>=56, setuptools_scm[toml]>=3.4.1)
* Getting build dependencies for wheel...
running egg_info
writing pip_run.egg-info/PKG-INFO
writing dependency_links to pip_run.egg-info/dependency_links.txt
writing entry points to pip_run.egg-info/entry_points.txt
writing requirements to pip_run.egg-info/requires.txt
writing top-level names to pip_run.egg-info/top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'pip_run.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pip_run.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib
copying pip-run.py -> build/lib
creating build/lib/pip_run
copying pip_run/deps.py -> build/lib/pip_run
copying pip_run/read-deps.py -> build/lib/pip_run
copying pip_run/__init__.py -> build/lib/pip_run
copying pip_run/_py38compat.py -> build/lib/pip_run
copying pip_run/launch.py -> build/lib/pip_run
copying pip_run/scripts.py -> build/lib/pip_run
copying pip_run/commands.py -> build/lib/pip_run
copying pip_run/__main__.py -> build/lib/pip_run
creating build/lib/pip_run/retention
copying pip_run/retention/destroy.py -> build/lib/pip_run/retention
copying pip_run/retention/persist.py -> build/lib/pip_run/retention
copying pip_run/retention/__init__.py -> build/lib/pip_run/retention
running egg_info
writing pip_run.egg-info/PKG-INFO
writing dependency_links to pip_run.egg-info/dependency_links.txt
writing entry points to pip_run.egg-info/entry_points.txt
writing requirements to pip_run.egg-info/requires.txt
writing top-level names to pip_run.egg-info/top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'pip_run.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pip_run.egg-info/SOURCES.txt'
copying pip_run/usage.txt -> build/lib/pip_run
installing to build/bdist.macosx-14-arm64/wheel
running install
running install_lib
creating build/bdist.macosx-14-arm64
creating build/bdist.macosx-14-arm64/wheel
creating build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/deps.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/read-deps.py -> build/bdist.macosx-14-arm64/wheel/pip_run
creating build/bdist.macosx-14-arm64/wheel/pip_run/retention
copying build/lib/pip_run/retention/destroy.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention
copying build/lib/pip_run/retention/persist.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention
copying build/lib/pip_run/retention/__init__.py -> build/bdist.macosx-14-arm64/wheel/pip_run/retention
copying build/lib/pip_run/__init__.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/_py38compat.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/launch.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/scripts.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/commands.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/__main__.py -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip_run/usage.txt -> build/bdist.macosx-14-arm64/wheel/pip_run
copying build/lib/pip-run.py -> build/bdist.macosx-14-arm64/wheel
running install_egg_info
Copying pip_run.egg-info to build/bdist.macosx-14-arm64/wheel/pip_run-12.4.1.dev26+gbf42021-py3.12.egg-info
running install_scripts
creating build/bdist.macosx-14-arm64/wheel/pip_run-12.4.1.dev26+gbf42021.dist-info/WHEEL
creating '/Users/jaraco/draft/pip-run/dist/.tmp-fg6udeqq/pip_run-12.4.1.dev26+gbf42021-py3-none-any.whl' and adding 'build/bdist.macosx-14-arm64/wheel' to it
adding 'pip-run.py'
adding 'pip_run/__init__.py'
adding 'pip_run/__main__.py'
adding 'pip_run/_py38compat.py'
adding 'pip_run/commands.py'
adding 'pip_run/deps.py'
adding 'pip_run/launch.py'
adding 'pip_run/read-deps.py'
adding 'pip_run/scripts.py'
adding 'pip_run/usage.txt'
adding 'pip_run/retention/__init__.py'
adding 'pip_run/retention/destroy.py'
adding 'pip_run/retention/persist.py'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/LICENSE'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/METADATA'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/WHEEL'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/entry_points.txt'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/top_level.txt'
adding 'pip_run-12.4.1.dev26+gbf42021.dist-info/RECORD'
removing build/bdist.macosx-14-arm64/wheel
Successfully built pip-run-12.4.1.dev26+gbf42021.tar.gz and pip_run-12.4.1.dev26+gbf42021-py3-none-any.whl

If I install the build requirements locally and run py -m build --no-isolation . and put a breakpoint in the setuptools_scm._file_finders.git:_git_toplevel, I see that when the error occurs, the git command is being run against a copy of the source that doesn't include the git directory. Something about the build ecosystem is causing the .git directory that I clearly cloned not to be included in the build. I tried running with some older versions of build going back to 0.9, but that didn't seem to make a difference.

So far, I've only replicated this on my own system. I'd test in an isolated environment, but I don't have ready access to Docker at the moment. I'll test again after restoring Docker.

[jaraco]

I've replicated the issue in a Linux Docker container as well, so the issue isn't peculiar to my environment.

Curiously, even though I'm seeing these errors when running builds, I'm finding that files are being included that would seem to depend on git metadata, such as Lorem ipsum.txt in jaraco.text.

Oh. I think I see what's happening now. build is building the sdist first, which does rely on the file finders, but then builds the wheel from the sdist, which relies on the manifest in the sdist. I'll have to revisit the original project (distutils) to see where things are going wrong there (why nothing is being discovered).

[jaraco]

Aha. So I've clarified my confusion. The errors are expected to be emitted. They just need not to appear during the sdist phase. The reason there were issues with the distutils build was not because of scm file finders, but because of an exclude directive. The error from setuptools_scm was a red herring.

[bulletmark]

@jaraco , sorry but your last two comments are a little too technical for me to understand. I get this same error every time I do a build even though the build seems successful. How can I get rid of that error message?

[jaraco]

Because of the way setuptools is engineered, it will necessarily involve setuptools_scm at both phases of the build, during the sdist and then again at the wheel. It's okay to see the error message during the wheel build because it no longer needs the SCM metadata because that's been embedded in the sdist during the sdist phase. In short, I don't think there is currently a way for Setuptools or setuptools_scm to know that it's doing a "wheel from sdist" and thus to disable features like "search for files using git metadata". Therefore, I don't think there's a way to get rid of the error message. Perhaps there's something that could be done, but it would require coordination between build and the backends (setuptools) and their plugins (setuptools_scm).

I'll move this issue to packaging-problems for consideration.

[bulletmark]

Thanks for your reply.

The message reports a big fat ERROR, not even a warning so I assumed I must have a mistake somewhere and have spent ages trying different fixes which did not change anything. It is disconcerting to see that ERROR reported so this clearly must be considered a bug.

[ssbarnea]

Is there anything we can to do avoid this message or the only option is to attempt to switch to another packaging alternative? Clearly the linked packaging issue seems to claim is a setuptools issue and not generic packaging one.

To my surprise i seen two that updated their scm to 8 to get rid of if, but that does not seem to be the case.

[RonnyPfannschmidt]

The error only ought to happen when git is detected but breaks horribly

We might be able to narrow it down better if the file finder hooks received more than a path

Unfortunately they don't so there's situations with false positives outweigh ing the rwst atm

[RonnyPfannschmidt]

I just took a look again, theres a bug in finding the git top level that's in need of a fix