Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

problem with requirements.txt generated by pipenv lock -r #429

Closed
xsteadfastx opened this issue Nov 4, 2017 · 4 comments
Closed

problem with requirements.txt generated by pipenv lock -r #429

xsteadfastx opened this issue Nov 4, 2017 · 4 comments
Assignees
@jsirois
Labels
enhancement

Comments

@xsteadfastx
Copy link

xsteadfastx commented Nov 4, 2017 •
edited
Loading

i try to get my little python tool deployable better with pex. and pex looks really really promising.

i use pipenv for developing and it has a option to generate requirements.txt files... so i tried a

pipenv run pex -r <(pipenv lock -r)

and got this

Traceback (most recent call last):
  File "/home/mpreuss/.local/share/virtualenvs/DoTheBackup-mzDsHcdU/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/mpreuss/.local/share/virtualenvs/DoTheBackup-mzDsHcdU/lib/python3.6/site-packages/pex/bin/pex.py", line 620, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/mpreuss/.local/share/virtualenvs/DoTheBackup-mzDsHcdU/lib/python3.6/site-packages/pex/bin/pex.py", line 548, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/home/mpreuss/.local/share/virtualenvs/DoTheBackup-mzDsHcdU/lib/python3.6/site-packages/pex/bin/pex.py", line 548, in <listcomp>
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/home/mpreuss/.local/share/virtualenvs/DoTheBackup-mzDsHcdU/lib/python3.6/site-packages/pex/resolvable.py", line 71, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: chardet==3.0.4 --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691  --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae

the generated requirements.txt looks like this:

certifi==2017.7.27.1 --hash=sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704  --hash=sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5
chardet==3.0.4 --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691  --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae
click==6.7 --hash=sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d  --hash=sha256:f15516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b
idna==2.6 --hash=sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4  --hash=sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f
pendulum==1.3.1 --hash=sha256:80955d968c441d2d381d2b073dd445a6b027f791b36268442cdb624f85667b1c  --hash=sha256:9d5216a268278eac44e3e07ad823634d90add7bb93d9e460fd489f55f14f702f  --hash=sha256:f31cb65920b5754b603696a1bd4f922ee5be13500afb2bfa832fbf951c8348fc  --hash=sha256:94add846ed1568ff43252c48b90760d7cfdc2a0da2167efb092942d50aa048e5  --hash=sha256:930b0765492f063bc10007039f865dd0acf3d91a70b350165a4e30fdba40cc93  --hash=sha256:6bfea234e03db8a0ad06f6ebbae4d43de7825ff1a7b63e67602e9bc5f26225eb  --hash=sha256:37cf099f2fabb62d5a069c91a29a2cb731fff5213abc682fd6364090e68a51d4  --hash=sha256:f24e475e41cdbd43834e14652830cc0192d006ef0e8633b13c0e126d9a391d2f
python-dateutil==2.6.1 --hash=sha256:95511bae634d69bc7329ba55e646499a842bc4ec342ad54a8cdb65645a0aad3c  --hash=sha256:891c38b2a02f5bb1be3e4793866c8df49c7d19baabf9c1bad62547e0b4866aca
pytz==2017.3 --hash=sha256:80af0f3008046b9975242012a985f04c5df1f01eed4ec1633d56cc47a75a6a48  --hash=sha256:feb2365914948b8620347784b6b6da356f31c9d03560259070b2f30cff3d469d  --hash=sha256:59707844a9825589878236ff2f4e0dc9958511b7ffaae94dc615da07d4a68d33  --hash=sha256:d0ef5ef55ed3d37854320d4926b04a4cb42a2e88f71da9ddfdacfde8e364f027  --hash=sha256:c41c62827ce9cafacd6f2f7018e4f83a6f1986e87bfd000b8cfbd4ab5da95f1a  --hash=sha256:8cc90340159b5d7ced6f2ba77694d946fc975b09f1a51d93f3ce3bb399396f94  --hash=sha256:dd2e4ca6ce3785c8dd342d1853dd9052b19290d5bf66060846e5dc6b8d6667f7  --hash=sha256:699d18a2a56f19ee5698ab1123bbcc1d269d061996aeb1eda6d89248d3542b82  --hash=sha256:fae4cffc040921b8a2d60c6cf0b5d662c1190fe54d718271db4eb17d44a185b7
pytzdata==2017.3.1 --hash=sha256:cd5b72400a7378b3b45eef5929cbe97ed44c3368685c35c477e316ebaa7e1809  --hash=sha256:e87376f2ee7cb89af5ddea5ed07ce3e98a55f891d07ae87d8c49e99f069423f2
pyyaml==3.12 --hash=sha256:3262c96a1ca437e7e4763e2843746588a965426550f3797a79fca9c6199c431f  --hash=sha256:16b20e970597e051997d90dc2cddc713a2876c47e3d92d59ee198700c5427736  --hash=sha256:e863072cdf4c72eebf179342c94e6989c67185842d9997960b3e69290b2fa269  --hash=sha256:bc6bced57f826ca7cb5125a10b23fd0f2fff3b7c4701d64c439a300ce665fff8  --hash=sha256:c01b880ec30b5a6e6aa67b09a2fe3fb30473008c85cd6a67359a1b15ed6d83a4  --hash=sha256:827dc04b8fa7d07c44de11fabbc888e627fa8293b695e0f99cb544fdfa1bf0d1  --hash=sha256:592766c6303207a20efc445587778322d7f73b161bd994f227adaa341ba212ab  --hash=sha256:5f84523c076ad14ff5e6c037fe1c89a7f73a3e04cf0377cb4d017014976433f3  --hash=sha256:0c507b7f74b3d2dd4d1322ec8a94794927305ab4cebbe89cc47fe5e81541e6e8  --hash=sha256:b4c423ab23291d3945ac61346feeb9a0dc4184999ede5e7c43e1ffb975130ae6  --hash=sha256:ca233c64c6e40eaa6c66ef97058cdc80e8d0157a443655baa1b2966e812807ca  --hash=sha256:4474f8ea030b5127225b8894d626bb66c01cda098d47a2b0d3429b6700af9fd8  --hash=sha256:326420cbb492172dec84b0f65c80942de6cedb5233c413dd824483989c000608  --hash=sha256:5ac82e411044fb129bae5cfbeb3ba626acb2af31a8d17d175004b70862a741a7
requests==2.18.4 --hash=sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b  --hash=sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e
six==1.11.0 --hash=sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb  --hash=sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9
typing==3.6.2 --hash=sha256:349b1f9c109c84b53ac79ac1d822eaa68fc91d63b321bd9392df15098f746f53  --hash=sha256:63a8255fe7c6269916baa440eb9b6a67139b0b97a01af632e7bd2842e1e02f15  --hash=sha256:d514bd84b284dd3e844f0305ac07511f097e325171f6cc4a20878d11ad771849
tzlocal==1.4 --hash=sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059
urllib3==1.22 --hash=sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b  --hash=sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f

i have a feeling that it has problems with parsing the hashes in the line strings. but thats just a idea.
@kwlzn
Copy link
Contributor

kwlzn commented Nov 22, 2017

yeah, support for --hash isn't plumbed to pex yet. a PR implementing this would be welcome.

@itdaniher itdaniher mentioned this issue Dec 22, 2017
Closed
@staticaland
Copy link

staticaland commented Dec 23, 2017 •
edited
Loading

Yeah, hashing is a very useful feature. I use this workaround:

pip download -r <(pipenv lock -r) --dest ./wheelhouse
pex . -r <(pipenv lock -r | cut -d' ' -f1) -f ./wheelhouse --no-index -c somescript.py -o somescript.pex

Would appreciate a PR for sure.

@kevinkjt2000
Copy link

kevinkjt2000 commented May 7, 2018 •
edited
Loading

pipenv lock -r seems to default to not outputting hashes now.

@jsirois jsirois mentioned this issue Oct 16, 2019
Closed
anisse added a commit to anisse/hwbench that referenced this issue May 21, 2024
@anisse
pex generation: remove use of lock file and wheels
0efec37 
The way we generate pex files broke when we added more dependencies (we
use numpy for graph generation). The choice was to build from source,
without wheels; but for that we need to be able to re-build numpy, and
this can be time-consuming.

If we remove the --no-wheel option, we go back to the exact versions
pinned in the generated requirements/base.txt; this follows the initial
intent of having reproducible builds for the pex. But this does not work
because the current pex implementation does not support locking versions
like this, while also having a local package (here hwbench itself) in a
directory (and not a wheel which has a hash). We get this error:

 ERROR: Can't verify hashes for these file:// requirements because they point to directories:

See this pex issues and the other linked ones:
pex-tool/pex#429 ; currently the pex project
is building its own lock file format to better fit its needs, but that
would not help us here.

So this change relaxes the constraints a bit: no more lock file. The pex
builds won't be reproducible, and hopefully it won't be an issue since
the artifacts will be archived. And no more wheels: this should make the
pexes generation significantly faster.

Also upgrade the dependency resolver to be the same one we use with
pip-compile (backtracking), so that we'd get a similar result if pex and
requirements/base.txt are generated at the same time.

Change-Id: If6d677838f86880776a125c1f5676cbf1b08ddbb
@anisse anisse mentioned this issue May 23, 2024
Closed
@jsirois jsirois self-assigned this May 24, 2024
@jsirois
Copy link
Member

jsirois commented May 24, 2024

#2412 mentioned this issue which I failed to close several years ago. Pex has supported requirements files with hashes since at least Pex 2.1.22 (via #1114). Using the example repo from #2412, for example:

# ~/dev/anisse/pex-minimal-repros on  main [?] via  v3.10.12
:; make requirements.txt
python3 -m venv .env
./.env/bin/pip install --upgrade pip-tools pip pex
Collecting pip-tools
  Using cached pip_tools-7.4.1-py3-none-any.whl (61 kB)
Requirement already satisfied: pip in ./.env/lib/python3.10/site-packages (22.0.2)
Collecting pip
  Using cached pip-24.0-py3-none-any.whl (2.1 MB)
Collecting pex
  Using cached pex-2.3.1-py2.py3-none-any.whl (3.4 MB)
Requirement already satisfied: setuptools in ./.env/lib/python3.10/site-packages (from pip-tools) (59.6.0)
Collecting pyproject-hooks
  Using cached pyproject_hooks-1.1.0-py3-none-any.whl (9.2 kB)
Collecting tomli
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting click>=8
  Using cached click-8.1.7-py3-none-any.whl (97 kB)
Collecting build>=1.0.0
  Using cached build-1.2.1-py3-none-any.whl (21 kB)
Collecting wheel
  Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Collecting packaging>=19.1
  Using cached packaging-24.0-py3-none-any.whl (53 kB)
Installing collected packages: wheel, tomli, pyproject-hooks, pip, pex, packaging, click, build, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Uninstalling pip-22.0.2:
      Successfully uninstalled pip-22.0.2
Successfully installed build-1.2.1 click-8.1.7 packaging-24.0 pex-2.3.1 pip-24.0 pip-tools-7.4.1 pyproject-hooks-1.1.0 tomli-2.0.1 wheel-0.43.0
./.env/bin/pip-compile --output-file=requirements.txt --generate-hashes requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --generate-hashes --output-file=requirements.txt requirements.in
#
numpy==1.26.4 \
    --hash=sha256:03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b \
    --hash=sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818 \
    --hash=sha256:1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20 \
    --hash=sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0 \
    --hash=sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010 \
    --hash=sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a \
    --hash=sha256:3373d5d70a5fe74a2c1bb6d2cfd9609ecf686d47a2d7b1d37a8f3b6bf6003aea \
    --hash=sha256:47711010ad8555514b434df65f7d7b076bb8261df1ca9bb78f53d3b2db02e95c \
    --hash=sha256:4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71 \
    --hash=sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110 \
    --hash=sha256:52b8b60467cd7dd1e9ed082188b4e6bb35aa5cdd01777621a1658910745b90be \
    --hash=sha256:60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a \
    --hash=sha256:62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a \
    --hash=sha256:666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5 \
    --hash=sha256:675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed \
    --hash=sha256:679b0076f67ecc0138fd2ede3a8fd196dddc2ad3254069bcb9faf9a79b1cebcd \
    --hash=sha256:7349ab0fa0c429c82442a27a9673fc802ffdb7c7775fad780226cb234965e53c \
    --hash=sha256:7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e \
    --hash=sha256:7e50d0a0cc3189f9cb0aeb3a6a6af18c16f59f004b866cd2be1c14b36134a4a0 \
    --hash=sha256:95a7476c59002f2f6c590b9b7b998306fba6a5aa646b1e22ddfeaf8f78c3a29c \
    --hash=sha256:96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a \
    --hash=sha256:9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b \
    --hash=sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0 \
    --hash=sha256:a354325ee03388678242a4d7ebcd08b5c727033fcff3b2f536aea978e15ee9e6 \
    --hash=sha256:a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2 \
    --hash=sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a \
    --hash=sha256:afedb719a9dcfc7eaf2287b839d8198e06dcd4cb5d276a3df279231138e83d30 \
    --hash=sha256:b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218 \
    --hash=sha256:b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5 \
    --hash=sha256:bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07 \
    --hash=sha256:cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2 \
    --hash=sha256:d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4 \
    --hash=sha256:d5241e0a80d808d70546c697135da2c613f30e28251ff8307eb72ba696945764 \
    --hash=sha256:edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef \
    --hash=sha256:f870204a840a60da0b12273ef34f7051e98c3b5961b61b0c2c1be6dfd64fbcd3 \
    --hash=sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f
    # via -r requirements.in
# this works:
#./.env/bin/pip-compile --output-file=requirements.txt requirements.in

# ~/dev/anisse/pex-minimal-repros on  main [?] via  v3.10.12 took 5s
:; pex -V
2.3.1

# ~/dev/anisse/pex-minimal-repros on  main [?] via  v3.10.12
:; pex -r requirements.txt -- -c 'import numpy; print(numpy.__file__)'
/home/jsirois/.pex/installed_wheels/675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed/numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/numpy/__init__.py

@jsirois jsirois closed this as completed May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jsirois @xsteadfastx @staticaland @kwlzn @kevinkjt2000