Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using a combination of --generate-hashes, -c constraints.txt, --resolver=backtracking and --strip-extras doesn't currently work #1752

Open
pawelad opened this issue Dec 2, 2022 · 6 comments
Open

Using a combination of --generate-hashes, -c constraints.txt, --resolver=backtracking and --strip-extras doesn't currently work #1752

pawelad opened this issue Dec 2, 2022 · 6 comments
Labels
dependency Related to a dependency resolver Related to dependency resolver

Comments

@pawelad
Copy link

pawelad commented Dec 2, 2022 •
edited
Loading

Hi,

First of all, thanks for the package and all the hard work you put into it.

I tried looking through the repo for a similar bug, but I haven't found anything. At the same time, I don't think I'm doing anything too much out of the ordinary - I'm using:

  • --generate-hashes, for obvious security reasons
  • -c constraints.txt at the top of the dev requirements files, as that's the recommended layered approach
  • --resolver=backtracking, for better package resolving (and it will be the default at some point)
  • --strip-extras, because otherwise using --resolver=backtracking failed with Constraints cannot have extras

The above scenario unfortunately produces requirement files that cannot be installed because of this:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r /home/circleci/project/requirements/prod.txt (line 269))

As I understand it, --strip-extras, which is needed for --resolver=backtracking to work with-c constraints.txt, is currently incompatible with --generate-hashes, because it needs all dependencies to have hashes and (correctly?) differences between PyJWT[crypto] and PyJWT.

My current workaround will probably involve dropping --resolver=backtracking and --strip-extras for now, but I wanted to write up this issue while I'm debugging all this.

Related issues: #398, #1092, #1300

Environment Versions

  1. macOS 12.6.1
  2. Python 3.8.13
  3. pip 22.3.1
  4. pip-compile, version 6.10.0

Steps to replicate

Given requirements files:

$ cat main.in
django==3.1.14
djangorestframework==3.12.4
drf-jwt==1.19.2
PyJWT[crypto]==2.1.0

$ cat dev.in
-c main.txt

black

$ python3 -m piptools compile --generate-hashes --strip-extras --resolver=backtracking main.in
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --generate-hashes --resolver=backtracking --strip-extras main.in
#
asgiref==3.5.2 \
    --hash=sha256:1d2880b792ae8757289136f1db2b7b99100ce959b2aa57fd69dab783d05afac4 \
    --hash=sha256:4a29362a6acebe09bf1d6640db38c1dc3d9217c68e6f9f6204d72667fc19a424
    # via django
cffi==1.15.1 \
    --hash=sha256:00a9ed42e88df81ffae7a8ab6d9356b371399b91dbdf0c3cb1e84c03a13aceb5 \
    --hash=sha256:03425bdae262c76aad70202debd780501fabeaca237cdfddc008987c0e0f59ef \
    --hash=sha256:04ed324bda3cda42b9b695d51bb7d54b680b9719cfab04227cdd1e04e5de3104 \
    --hash=sha256:0e2642fe3142e4cc4af0799748233ad6da94c62a8bec3a6648bf8ee68b1c7426 \
    --hash=sha256:173379135477dc8cac4bc58f45db08ab45d228b3363adb7af79436135d028405 \
    --hash=sha256:198caafb44239b60e252492445da556afafc7d1e3ab7a1fb3f0584ef6d742375 \
    --hash=sha256:1e74c6b51a9ed6589199c787bf5f9875612ca4a8a0785fb2d4a84429badaf22a \
    --hash=sha256:2012c72d854c2d03e45d06ae57f40d78e5770d252f195b93f581acf3ba44496e \
    --hash=sha256:21157295583fe8943475029ed5abdcf71eb3911894724e360acff1d61c1d54bc \
    --hash=sha256:2470043b93ff09bf8fb1d46d1cb756ce6132c54826661a32d4e4d132e1977adf \
    --hash=sha256:285d29981935eb726a4399badae8f0ffdff4f5050eaa6d0cfc3f64b857b77185 \
    --hash=sha256:30d78fbc8ebf9c92c9b7823ee18eb92f2e6ef79b45ac84db507f52fbe3ec4497 \
    --hash=sha256:320dab6e7cb2eacdf0e658569d2575c4dad258c0fcc794f46215e1e39f90f2c3 \
    --hash=sha256:33ab79603146aace82c2427da5ca6e58f2b3f2fb5da893ceac0c42218a40be35 \
    --hash=sha256:3548db281cd7d2561c9ad9984681c95f7b0e38881201e157833a2342c30d5e8c \
    --hash=sha256:3799aecf2e17cf585d977b780ce79ff0dc9b78d799fc694221ce814c2c19db83 \
    --hash=sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21 \
    --hash=sha256:3b926aa83d1edb5aa5b427b4053dc420ec295a08e40911296b9eb1b6170f6cca \
    --hash=sha256:3bcde07039e586f91b45c88f8583ea7cf7a0770df3a1649627bf598332cb6984 \
    --hash=sha256:3d08afd128ddaa624a48cf2b859afef385b720bb4b43df214f85616922e6a5ac \
    --hash=sha256:3eb6971dcff08619f8d91607cfc726518b6fa2a9eba42856be181c6d0d9515fd \
    --hash=sha256:40f4774f5a9d4f5e344f31a32b5096977b5d48560c5592e2f3d2c4374bd543ee \
    --hash=sha256:4289fc34b2f5316fbb762d75362931e351941fa95fa18789191b33fc4cf9504a \
    --hash=sha256:470c103ae716238bbe698d67ad020e1db9d9dba34fa5a899b5e21577e6d52ed2 \
    --hash=sha256:4f2c9f67e9821cad2e5f480bc8d83b8742896f1242dba247911072d4fa94c192 \
    --hash=sha256:50a74364d85fd319352182ef59c5c790484a336f6db772c1a9231f1c3ed0cbd7 \
    --hash=sha256:54a2db7b78338edd780e7ef7f9f6c442500fb0d41a5a4ea24fff1c929d5af585 \
    --hash=sha256:5635bd9cb9731e6d4a1132a498dd34f764034a8ce60cef4f5319c0541159392f \
    --hash=sha256:59c0b02d0a6c384d453fece7566d1c7e6b7bae4fc5874ef2ef46d56776d61c9e \
    --hash=sha256:5d598b938678ebf3c67377cdd45e09d431369c3b1a5b331058c338e201f12b27 \
    --hash=sha256:5df2768244d19ab7f60546d0c7c63ce1581f7af8b5de3eb3004b9b6fc8a9f84b \
    --hash=sha256:5ef34d190326c3b1f822a5b7a45f6c4535e2f47ed06fec77d3d799c450b2651e \
    --hash=sha256:6975a3fac6bc83c4a65c9f9fcab9e47019a11d3d2cf7f3c0d03431bf145a941e \
    --hash=sha256:6c9a799e985904922a4d207a94eae35c78ebae90e128f0c4e521ce339396be9d \
    --hash=sha256:70df4e3b545a17496c9b3f41f5115e69a4f2e77e94e1d2a8e1070bc0c38c8a3c \
    --hash=sha256:7473e861101c9e72452f9bf8acb984947aa1661a7704553a9f6e4baa5ba64415 \
    --hash=sha256:8102eaf27e1e448db915d08afa8b41d6c7ca7a04b7d73af6514df10a3e74bd82 \
    --hash=sha256:87c450779d0914f2861b8526e035c5e6da0a3199d8f1add1a665e1cbc6fc6d02 \
    --hash=sha256:8b7ee99e510d7b66cdb6c593f21c043c248537a32e0bedf02e01e9553a172314 \
    --hash=sha256:91fc98adde3d7881af9b59ed0294046f3806221863722ba7d8d120c575314325 \
    --hash=sha256:94411f22c3985acaec6f83c6df553f2dbe17b698cc7f8ae751ff2237d96b9e3c \
    --hash=sha256:98d85c6a2bef81588d9227dde12db8a7f47f639f4a17c9ae08e773aa9c697bf3 \
    --hash=sha256:9ad5db27f9cabae298d151c85cf2bad1d359a1b9c686a275df03385758e2f914 \
    --hash=sha256:a0b71b1b8fbf2b96e41c4d990244165e2c9be83d54962a9a1d118fd8657d2045 \
    --hash=sha256:a0f100c8912c114ff53e1202d0078b425bee3649ae34d7b070e9697f93c5d52d \
    --hash=sha256:a591fe9e525846e4d154205572a029f653ada1a78b93697f3b5a8f1f2bc055b9 \
    --hash=sha256:a5c84c68147988265e60416b57fc83425a78058853509c1b0629c180094904a5 \
    --hash=sha256:a66d3508133af6e8548451b25058d5812812ec3798c886bf38ed24a98216fab2 \
    --hash=sha256:a8c4917bd7ad33e8eb21e9a5bbba979b49d9a97acb3a803092cbc1133e20343c \
    --hash=sha256:b3bbeb01c2b273cca1e1e0c5df57f12dce9a4dd331b4fa1635b8bec26350bde3 \
    --hash=sha256:cba9d6b9a7d64d4bd46167096fc9d2f835e25d7e4c121fb2ddfc6528fb0413b2 \
    --hash=sha256:cc4d65aeeaa04136a12677d3dd0b1c0c94dc43abac5860ab33cceb42b801c1e8 \
    --hash=sha256:ce4bcc037df4fc5e3d184794f27bdaab018943698f4ca31630bc7f84a7b69c6d \
    --hash=sha256:cec7d9412a9102bdc577382c3929b337320c4c4c4849f2c5cdd14d7368c5562d \
    --hash=sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9 \
    --hash=sha256:d61f4695e6c866a23a21acab0509af1cdfd2c013cf256bbf5b6b5e2695827162 \
    --hash=sha256:db0fbb9c62743ce59a9ff687eb5f4afbe77e5e8403d6697f7446e5f609976f76 \
    --hash=sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4 \
    --hash=sha256:e00b098126fd45523dd056d2efba6c5a63b71ffe9f2bbe1a4fe1716e1d0c331e \
    --hash=sha256:e229a521186c75c8ad9490854fd8bbdd9a0c9aa3a524326b55be83b54d4e0ad9 \
    --hash=sha256:e263d77ee3dd201c3a142934a086a4450861778baaeeb45db4591ef65550b0a6 \
    --hash=sha256:ed9cb427ba5504c1dc15ede7d516b84757c3e3d7868ccc85121d9310d27eed0b \
    --hash=sha256:fa6693661a4c91757f4412306191b6dc88c1703f780c8234035eac011922bc01 \
    --hash=sha256:fcd131dd944808b5bdb38e6f5b53013c5aa4f334c5cad0c72742f6eba4b73db0
    # via cryptography
cryptography==3.4.8 \
    --hash=sha256:0a7dcbcd3f1913f664aca35d47c1331fce738d44ec34b7be8b9d332151b0b01e \
    --hash=sha256:1eb7bb0df6f6f583dd8e054689def236255161ebbcf62b226454ab9ec663746b \
    --hash=sha256:21ca464b3a4b8d8e86ba0ee5045e103a1fcfac3b39319727bc0fc58c09c6aff7 \
    --hash=sha256:34dae04a0dce5730d8eb7894eab617d8a70d0c97da76b905de9efb7128ad7085 \
    --hash=sha256:3520667fda779eb788ea00080124875be18f2d8f0848ec00733c0ec3bb8219fc \
    --hash=sha256:3c4129fc3fdc0fa8e40861b5ac0c673315b3c902bbdc05fc176764815b43dd1d \
    --hash=sha256:3fa3a7ccf96e826affdf1a0a9432be74dc73423125c8f96a909e3835a5ef194a \
    --hash=sha256:5b0fbfae7ff7febdb74b574055c7466da334a5371f253732d7e2e7525d570498 \
    --hash=sha256:695104a9223a7239d155d7627ad912953b540929ef97ae0c34c7b8bf30857e89 \
    --hash=sha256:8695456444f277af73a4877db9fc979849cd3ee74c198d04fc0776ebc3db52b9 \
    --hash=sha256:94cc5ed4ceaefcbe5bf38c8fba6a21fc1d365bb8fb826ea1688e3370b2e24a1c \
    --hash=sha256:94fff993ee9bc1b2440d3b7243d488c6a3d9724cc2b09cdb297f6a886d040ef7 \
    --hash=sha256:9965c46c674ba8cc572bc09a03f4c649292ee73e1b683adb1ce81e82e9a6a0fb \
    --hash=sha256:a00cf305f07b26c351d8d4e1af84ad7501eca8a342dedf24a7acb0e7b7406e14 \
    --hash=sha256:a305600e7a6b7b855cd798e00278161b681ad6e9b7eca94c721d5f588ab212af \
    --hash=sha256:cd65b60cfe004790c795cc35f272e41a3df4631e2fb6b35aa7ac6ef2859d554e \
    --hash=sha256:d2a6e5ef66503da51d2110edf6c403dc6b494cc0082f85db12f54e9c5d4c3ec5 \
    --hash=sha256:d9ec0e67a14f9d1d48dd87a2531009a9b251c02ea42851c060b25c782516ff06 \
    --hash=sha256:f44d141b8c4ea5eb4dbc9b3ad992d45580c1d22bf5e24363f2fbf50c2d7ae8a7
    # via pyjwt
django==3.1.14 \
    --hash=sha256:0fabc786489af16ad87a8c170ba9d42bfd23f7b699bd5ef05675864e8d012859 \
    --hash=sha256:72a4a5a136a214c39cf016ccdd6b69e2aa08c7479c66d93f3a9b5e4bb9d8a347
    # via
    #   -r main.in
    #   djangorestframework
    #   drf-jwt
djangorestframework==3.12.4 \
    --hash=sha256:6d1d59f623a5ad0509fe0d6bfe93cbdfe17b8116ebc8eda86d45f6e16e819aaf \
    --hash=sha256:f747949a8ddac876e879190df194b925c177cdeb725a099db1460872f7c0a7f2
    # via
    #   -r main.in
    #   drf-jwt
drf-jwt==1.19.2 \
    --hash=sha256:63c3d4ed61a1013958cd63416e2d5c84467d8ae3e6e1be44b1fb58743dbd1582 \
    --hash=sha256:660bc66f992065cef59832adcbbdf871847e9738671c19e5121971e773768235
    # via -r main.in
pycparser==2.21 \
    --hash=sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9 \
    --hash=sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206
    # via cffi
pyjwt==2.1.0 \
    --hash=sha256:934d73fbba91b0483d3857d1aff50e96b2a892384ee2c17417ed3203f173fca1 \
    --hash=sha256:fba44e7898bbca160a2b2b501f492824fc8382485d3a6f11ba5d0c1937ce6130
    # via
    #   -r main.in
    #   drf-jwt
pytz==2022.6 \
    --hash=sha256:222439474e9c98fced559f1709d89e6c9cbf8d79c794ff3eb9f8800064291427 \
    --hash=sha256:e89512406b793ca39f5971bc999cc538ce125c0e51c27941bef4568b460095e2
    # via django
sqlparse==0.4.3 \
    --hash=sha256:0323c0ec29cd52bceabc1b4d9d579e311f3e4961b98d174201d5622a23b85e34 \
    --hash=sha256:69ca804846bb114d2ec380e4360a8a340db83f0ccf3afceeb1404df028f57268
    # via django

$ python3 -m piptools compile --generate-hashes --strip-extras --resolver=backtracking dev.in
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --generate-hashes --resolver=backtracking --strip-extras dev.in
#
black==22.10.0 \
    --hash=sha256:14ff67aec0a47c424bc99b71005202045dc09270da44a27848d534600ac64fc7 \
    --hash=sha256:197df8509263b0b8614e1df1756b1dd41be6738eed2ba9e9769f3880c2b9d7b6 \
    --hash=sha256:1e464456d24e23d11fced2bc8c47ef66d471f845c7b7a42f3bd77bf3d1789650 \
    --hash=sha256:2039230db3c6c639bd84efe3292ec7b06e9214a2992cd9beb293d639c6402edb \
    --hash=sha256:21199526696b8f09c3997e2b4db8d0b108d801a348414264d2eb8eb2532e540d \
    --hash=sha256:2644b5d63633702bc2c5f3754b1b475378fbbfb481f62319388235d0cd104c2d \
    --hash=sha256:432247333090c8c5366e69627ccb363bc58514ae3e63f7fc75c54b1ea80fa7de \
    --hash=sha256:444ebfb4e441254e87bad00c661fe32df9969b2bf224373a448d8aca2132b395 \
    --hash=sha256:5b9b29da4f564ba8787c119f37d174f2b69cdfdf9015b7d8c5c16121ddc054ae \
    --hash=sha256:5cc42ca67989e9c3cf859e84c2bf014f6633db63d1cbdf8fdb666dcd9e77e3fa \
    --hash=sha256:5d8f74030e67087b219b032aa33a919fae8806d49c867846bfacde57f43972ef \
    --hash=sha256:72ef3925f30e12a184889aac03d77d031056860ccae8a1e519f6cbb742736383 \
    --hash=sha256:819dc789f4498ecc91438a7de64427c73b45035e2e3680c92e18795a839ebb66 \
    --hash=sha256:915ace4ff03fdfff953962fa672d44be269deb2eaf88499a0f8805221bc68c87 \
    --hash=sha256:9311e99228ae10023300ecac05be5a296f60d2fd10fff31cf5c1fa4ca4b1988d \
    --hash=sha256:974308c58d057a651d182208a484ce80a26dac0caef2895836a92dd6ebd725e0 \
    --hash=sha256:b8b49776299fece66bffaafe357d929ca9451450f5466e997a7285ab0fe28e3b \
    --hash=sha256:c957b2b4ea88587b46cf49d1dc17681c1e672864fd7af32fc1e9664d572b3458 \
    --hash=sha256:e41a86c6c650bcecc6633ee3180d80a025db041a8e2398dcc059b3afa8382cd4 \
    --hash=sha256:f513588da599943e0cde4e32cc9879e825d58720d6557062d1098c5ad80080e1 \
    --hash=sha256:fba8a281e570adafb79f7755ac8721b6cf1bbf691186a287e990c7929c7692ff
    # via -r dev.in
click==8.1.3 \
    --hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
    --hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
    # via black
mypy-extensions==0.4.3 \
    --hash=sha256:090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d \
    --hash=sha256:2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8
    # via black
pathspec==0.10.2 \
    --hash=sha256:88c2606f2c1e818b978540f73ecc908e13999c6c3a383daf3705652ae79807a5 \
    --hash=sha256:8f6bf73e5758fd365ef5d58ce09ac7c27d2833a8d7da51712eac6e27e35141b0
    # via black
platformdirs==2.5.4 \
    --hash=sha256:1006647646d80f16130f052404c6b901e80ee4ed6bef6792e1f238a8969106f7 \
    --hash=sha256:af0276409f9a02373d540bf8480021a048711d572745aef4b7842dad245eba10
    # via black
tomli==2.0.1 \
    --hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
    --hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
    # via black
typing-extensions==4.4.0 \
    --hash=sha256:1511434bb92bf8dd198c12b1cc812e800d4181cfcb867674e0f8279cc93087aa \
    --hash=sha256:16fa4864408f655d35ec496218b85f79b3437c829e93320c7c9215ccfd92489e
    # via black

Expected result

I'd like to be able to use generated requirements files to install the packages.

Actual result

I can't install the packages, neither with pip-sync or with pip install -r:

$ python -m piptools sync main.txt dev.txt
Collecting asgiref==3.5.2
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting black==22.10.0
  Using cached black-22.10.0-cp38-cp38-macosx_11_0_arm64.whl (1.2 MB)
Collecting cffi==1.15.1
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Preparing metadata (setup.py) ... done
Collecting cryptography==3.4.8
  Using cached cryptography-3.4.8-cp36-abi3-macosx_11_0_arm64.whl (1.9 MB)
Collecting django==3.1.14
  Using cached Django-3.1.14-py3-none-any.whl (7.8 MB)
Collecting djangorestframework==3.12.4
  Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB)
Collecting drf-jwt==1.19.2
  Using cached drf_jwt-1.19.2-py2.py3-none-any.whl (21 kB)
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting pathspec==0.10.2
  Using cached pathspec-0.10.2-py3-none-any.whl (28 kB)
Collecting platformdirs==2.5.4
  Using cached platformdirs-2.5.4-py3-none-any.whl (14 kB)
Collecting pycparser==2.21
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting pyjwt==2.1.0
  Using cached PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting pytz==2022.6
  Using cached pytz-2022.6-py2.py3-none-any.whl (498 kB)
Collecting sqlparse==0.4.3
  Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Collecting typing-extensions==4.4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Requirement already satisfied: tomli>=1.1.0 in /Users/pawelad/.pyenv/versions/3.8.13/envs/tmp/lib/python3.8/site-packages (from black==22.10.0->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 4)) (2.0.1)
Requirement already satisfied: click>=8.0.0 in /Users/pawelad/.pyenv/versions/3.8.13/envs/tmp/lib/python3.8/site-packages (from black==22.10.0->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 4)) (8.1.3)
Collecting PyJWT[crypto]<3.0.0,>=1.5.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r /var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s (line 117))
Traceback (most recent call last):
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/__main__.py", line 19, in <module>
    cli()
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/scripts/sync.py", line 177, in cli
    sync.sync(
  File "/Users/pawelad/.pyenv/versions/tmp/lib/python3.8/site-packages/piptools/sync.py", line 240, in sync
    run(  # nosec
  File "/Users/pawelad/.pyenv/versions/3.8.13/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/pawelad/.pyenv/versions/tmp/bin/python', '-m', 'pip', 'install', '-r', '/var/folders/zd/bh5ny2dj5hv8x_5vf92tnfz40000gn/T/tmp5g4e645s']' returned non-zero exit status 1.

$ pip install -r main.txt
Collecting asgiref==3.5.2
  Using cached asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting cffi==1.15.1
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Preparing metadata (setup.py) ... done
Collecting cryptography==3.4.8
  Using cached cryptography-3.4.8-cp36-abi3-macosx_11_0_arm64.whl (1.9 MB)
Collecting django==3.1.14
  Using cached Django-3.1.14-py3-none-any.whl (7.8 MB)
Collecting djangorestframework==3.12.4
  Using cached djangorestframework-3.12.4-py3-none-any.whl (957 kB)
Collecting drf-jwt==1.19.2
  Using cached drf_jwt-1.19.2-py2.py3-none-any.whl (21 kB)
Collecting pycparser==2.21
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting pyjwt==2.1.0
  Using cached PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting pytz==2022.6
  Using cached pytz-2022.6-py2.py3-none-any.whl (498 kB)
Collecting sqlparse==0.4.3
  Using cached sqlparse-0.4.3-py3-none-any.whl (42 kB)
Collecting PyJWT[crypto]<3.0.0,>=1.5.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT[crypto]<3.0.0,>=1.5.2 from https://files.pythonhosted.org/packages/40/46/505f0dd53c14096f01922bf93a7abb4e40e29a06f858abbaa791e6954324/PyJWT-2.6.0-py3-none-any.whl (from drf-jwt==1.19.2->-r main.txt (line 111))
@pawelad
Copy link
Author

pawelad commented Dec 2, 2022 •
edited
Loading

Ah, I just found the upstream issue in pip - pypa/pip#9644

I guess there's nothing we can do on pip-tools side? I wonder what should be the workaround? I went back to using the old resolver (and dropped --skip-extras), but if I understand it correctly, this will become a breaking bug when pip-tools 7 is released and the new resolver will become the default?

@AndydeCleyre
Copy link
Contributor

In your reproductions it looks like needed dependencies are not included in the compiled txts: backports.zoneinfo is absent from main.txt, though apparently required by django, and typing-extensions is absent from dev.txt, though apparently required by black.

This may be a bug, or it may be a case of mismatched environments between compilation and installation. Looking at black's pyproject.toml, I see that typing_extensions is required only when Python is < 3.10. Similarly for django, backports.zoneinfo is only required when Python is < 3.9.

So probably you are running pip-compile in an environment with Python > 3.10, but pip install in one with Python < 3.9. This is not currently expected to succeed, when the requirements differ across those environments.

See also:

@pawelad
Copy link
Author

pawelad commented Dec 2, 2022 •
edited
Loading

You're 100% right about mixing Python 3.8 and 3.10 (my project Python version vs. the temporary venv I created to reproduce), but I'm almost certain that the bigger problem still stands (as 'confirmed' by pypa/pip#9644).

I'll update the description in the coming days.

@pawelad
Copy link
Author

pawelad commented Dec 5, 2022 •
edited
Loading

@AndydeCleyre I updated issue description with pins from my production environment and making sure I'm doing everything on Python 3.8.13, pip 22.3.1 and pip-tools 6.10.0. Let me know if you still can't reproduce it.

Like I said, I believe it's a known bug in pip's new resolver (pypa/pip#9644) with known workarounds (pypa/pip#9644 (comment)) and lack of resources to fix (pypa/pip#9644 (comment))

I guess there's nothing to do on pip-tools side, but having it open here for visibility might have some worth, especially since this will become a much more pressing issue when the new resolver is gonna be enabled by default (I'd recommend postponing that until this bug is fixed).

FWIW, my current workaround is to stop using the new resolver until this gets fixed.

@atugushev atugushev added dependency Related to a dependency resolver Related to dependency resolver labels Dec 12, 2022
@jikuja jikuja mentioned this issue Dec 13, 2022
Closed
@snmishra
Copy link

My current workaround is to use --no-deps with pip install -r. Since pip-compile has specified all dependencies, --no-deps seems quite reasonable.

mrswats added a commit to mrswats/django-sqlite-backup that referenced this issue Apr 15, 2023
@mrswats
Add no no-deps flags
daf6d08 
Ref: jazzband/pip-tools#1752
@AndydeCleyre AndydeCleyre mentioned this issue May 10, 2023
Closed
@AndydeCleyre
Copy link
Contributor

Seen again as #1865 (closed as dupe of this one). The title here could probably use an update (note that --strip-extras and -c` are not necessary to reproduce).

GeoWill added a commit to DemocracyClub/UK-Polling-Stations that referenced this issue Aug 16, 2023
@GeoWill
Use pip install --no-deps in CI
bd29742 
pip-compile should have listed all required dependencies. There's
something going on with extras that I don't fully understand.
Issue here: jazzband/pip-tools#1752
GeoWill added a commit to DemocracyClub/UK-Polling-Stations that referenced this issue Aug 16, 2023
@GeoWill
Use pip install --no-deps in CI
77af0db 
pip-compile should have listed all required dependencies. There's
something going on with extras that I don't fully understand.
Issue here: jazzband/pip-tools#1752
@webknjaz webknjaz mentioned this issue Dec 1, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dependency Related to a dependency resolver Related to dependency resolver
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@snmishra @AndydeCleyre @pawelad @atugushev