Skip to content

Commit

Permalink
pex generation: remove use of lock file and wheels
Browse files Browse the repository at this point in the history
The way we generate pex files broke when we added more dependencies (we
use numpy for graph generation). The choice was to build from source,
without wheels; but for that we need to be able to re-build numpy, and
this can be time-consuming.

If we remove the --no-wheel option, we go back to the exact versions
pinned in the generated requirements/base.txt; this follows the initial
intent of having reproducible builds for the pex. But this does not work
because the current pex implementation does not support locking versions
like this, while also having a local package (here hwbench itself) in a
directory (and not a wheel which has a hash). We get this error:

 ERROR: Can't verify hashes for these file:// requirements because they point to directories:

See this pex issues and the other linked ones:
pex-tool/pex#429 ; currently the pex project
is building its own lock file format to better fit its needs, but that
would not help us here.

So this change relaxes the constraints a bit: no more lock file. The pex
builds won't be reproducible, and hopefully it won't be an issue since
the artifacts will be archived. And no more wheels: this should make the
pexes generation significantly faster.

Also upgrade the dependency resolver to be the same one we use with
pip-compile (backtracking), so that we'd get a similar result if pex and
requirements/base.txt are generated at the same time.

Change-Id: If6d677838f86880776a125c1f5676cbf1b08ddbb
  • Loading branch information
anisse committed May 21, 2024
1 parent fb12867 commit 0efec37
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ commands =
# Creates a source archive in sdist/.
{envpython} setup.py sdist --dist-dir=sdist --format=gztar
# Puts binary archives in dist/.
{envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='--disable-cache --no-wheel -r requirements/base.txt' --bdist-all
{envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='--disable-cache -r requirements/base.in --resolver-version pip-2020-resolver' --bdist-all

# Linter environment
[testenv:lint]
Expand Down

0 comments on commit 0efec37

Please sign in to comment.