Help finding where cibuildwheel is modifying git managed files

[djhoese]

I'm using cibuildwheel with the Vispy project (https://github.com/vispy/vispy/). In previous releases we had hardcoded the version number in our setup.py but have since migrated to using setuptools-scm (https://pypi.org/project/setuptools-scm/) to automatically generate the version number from the previous tag. So if the current git state matches a git tag, then that git tag is used as the version number. If there are local uncommited changes or we are commits ahead of the last tag then the version becomes something like 0.6.3.dev0+abcdef.d20191104 where abcdef is the current commit hash.

I'm working on our first release since this migration and am running in to an issue where cibuildwheel seems to be modifying one of vispy's git-managed files just before building a wheel. This causes the version for the package to become like the longer version example above and then fails to upload to PyPI since it is an invalid version number.

I'm trying to track down what cibuildwheel might be modifying but am not finding anything obvious yet. Putting a git status in my BEFORE_BUILD shows that we are in a detached head state but no git-managed files are modified (the output directory is created). Any other ideas?

[djhoese]

I should note this happens in all environments (OSX, Windows, Linux).

[YannickJadoul]

@djhoese Do you have a run of the CI? Or is there some simple way we can try this ourselves?

What about running with CIBW_TEST_COMMAND="git status"?

[djhoese]

Sorry to make all this noise. I walked through some things last night and I think this is not cibuildwheel's fault and was just hoping for a simple answer (ex. "oh yeah cibuildwheel modifies your setup.py").

For anyone else running in to something like this, the Vispy project has a jupyter widget that it builds as part of the setup.py process. It uses npm to do this (as is standard with building a custom jupyter widgets) and results in some new javascript files that get bundled with the sdist/wheel distribution of the Vispy package. The problem is (not completely confirmed yet), that npm will update the git-committed package-lock.json file with the newest versions of javascript dependencies. This causes the local git repository directory to become "dirty" which causes setuptools_scm to produce a different version number. Since the package-lock.json file could change depending on npm version or new javascript dependency releases, this isn't easy to make consistent across platforms.

My next attempt at a solution will be to add something to the setup.py an extra environment variable option to reset the git repository after the javascript widget has been built. I will also be talking with the jupyter widget people to see if I'm doing anything non-standard (like committing this package-lock.json file).

[djhoese]

Last update: turns out it was the package-lock.json being updated inside setup.py. Untracking the file for my application was the simplest work around.

[YannickJadoul]

Ah, good you found it. And even better to know cibuildwheel is not to blame ;-)
Thanks for the nice writeup and explanation; if others run into this, it's clear why, now