Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(bzlmod) Index responses containing relative paths for wheels causes bazel downloader to fail. #2150

Closed
WillMorrisonEnspi opened this issue Aug 23, 2024 · 1 comment · Fixed by #2153
Closed

(bzlmod) Index responses containing relative paths for wheels causes bazel downloader to fail. #2150

WillMorrisonEnspi opened this issue Aug 23, 2024 · 1 comment · Fixed by #2153
Labels
help wanted type: pip

Comments

@WillMorrisonEnspi
Copy link

🐞 bug report

Affected Rule

The issue is caused by: _create_wheel_repos passing index-relative paths to whl_library instead of full URLs.

Is this a regression?

Probably not.

Description

The current implementation of whl_library passes URLs parsed from simpleapi_download verbatim to rctx.download. However, some indexes (like AWS CodeArtifact) return relative URLs. This causes the bazel downloader to choke when experimental_index_url is set to an index that returns relative URLs.

Expand to show GET response for private index vs pypi

curl --basic -u "aws:$auth_token" 'https://<redacted>.d.codeartifact.eu-central-1.amazonaws.com/pypi/packages-prod/simple/mypy-extensions/'

<!DOCTYPE html>
<html>
  <head>
    <title>Links for mypy-extensions</title>
  </head>
  <body>
    <h1>Links for mypy-extensions</h1>
      <a href="0.1.0/mypy_extensions-0.1.0.tar.gz#sha256=6942c0429e505a0cdcd0468fe897960fbd8113f84cb9bde1540120bc8b68eca9"  data-gpg-sig="false" >mypy_extensions-0.1.0.tar.gz</a>
      <br/>
      <a href="0.2.0/mypy_extensions-0.2.0.tar.gz#sha256=b8f40b3bca27843f6ca911d63550799aa2586804d40da3ab087bab8130da3ebf"  data-gpg-sig="false" >mypy_extensions-0.2.0.tar.gz</a>
      <br/>
      <a href="0.3.0/mypy_extensions-0.3.0.tar.gz#sha256=c4553f2475996c912658d3c2f398766cd47a65e0c2c6bb27ac8dea94a21640f4"  data-gpg-sig="false" >mypy_extensions-0.3.0.tar.gz</a>
      <br/>
      <a href="0.4.0/mypy_extensions-0.4.0.tar.gz#sha256=118ec27adb66c682dc842bc2ad48d6d5218867df5f90ca713f4deb3760324fd3"  data-gpg-sig="false" >mypy_extensions-0.4.0.tar.gz</a>
      <br/>
      <a href="0.4.1/mypy_extensions-0.4.1-py2.py3-none-any.whl#sha256=b16cabe759f55e3409a7d231ebd2841378fb0c27a5d1994719e340e4f429ac3e"  data-gpg-sig="false" >mypy_extensions-0.4.1-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.1/mypy_extensions-0.4.1.tar.gz#sha256=37e0e956f41369209a3d5f34580150bcacfabaa57b33a15c0b25f4b5725e0812"  data-gpg-sig="false" >mypy_extensions-0.4.1.tar.gz</a>
      <br/>
      <a href="0.4.2/mypy_extensions-0.4.2-py2.py3-none-any.whl#sha256=c77e34350c01523cd63b6dc30b36515107b2c3cbacc8392d51d33ea18d59f8f1"  data-gpg-sig="false" >mypy_extensions-0.4.2-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.2/mypy_extensions-0.4.2.tar.gz#sha256=a161e3b917053de87dbe469987e173e49fb454eca10ef28b48b384538cc11458"  data-gpg-sig="false" >mypy_extensions-0.4.2.tar.gz</a>
      <br/>
      <a href="0.4.3/mypy_extensions-0.4.3-py2.py3-none-any.whl#sha256=090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d"  data-gpg-sig="false" >mypy_extensions-0.4.3-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.3/mypy_extensions-0.4.3.tar.gz#sha256=2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"  data-gpg-sig="false" >mypy_extensions-0.4.3.tar.gz</a>
      <br/>
      <a href="0.4.4/mypy_extensions-0.4.4.tar.gz#sha256=c8b707883a96efe9b4bb3aaf0dcc07e7e217d7d8368eec4db4049ee9e142f4fd" data-requires-python="&gt;=2.7" data-gpg-sig="false" >mypy_extensions-0.4.4.tar.gz</a>
      <br/>
      <a href="1.0.0/mypy_extensions-1.0.0-py3-none-any.whl#sha256=4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d" data-requires-python="&gt;=3.5" data-gpg-sig="false" >mypy_extensions-1.0.0-py3-none-any.whl</a>
      <br/>
      <a href="1.0.0/mypy_extensions-1.0.0.tar.gz#sha256=75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782" data-requires-python="&gt;=3.5" data-gpg-sig="false" >mypy_extensions-1.0.0.tar.gz</a>
      <br/>
  </body>
</html>

curl 'https://pypi.org/simple/mypy-extensions/'

<!DOCTYPE html>
<html>
  <head>
    <meta name="pypi:repository-version" content="1.1">
    <title>Links for mypy-extensions</title>
  </head>
  <body>
    <h1>Links for mypy-extensions</h1>
<a href="https://files.pythonhosted.org/packages/77/9c/cad8e95677e005fc6d223f850e59a1751657a314e7f33aa06ffd5e209f28/mypy_extensions-0.1.0.tar.gz#sha256=6942c0429e505a0cdcd0468fe897960fbd8113f84cb9bde1540120bc8b68eca9" >mypy_extensions-0.1.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/aa/61/2c140b126809291baeb5642b626310c12373c0fa978fb7639ffeb5feaf5d/mypy_extensions-0.2.0.tar.gz#sha256=b8f40b3bca27843f6ca911d63550799aa2586804d40da3ab087bab8130da3ebf" >mypy_extensions-0.2.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/f7/51/a55d2c14e37821e6938b2e150afd50f0ed9ee4beaccb5d71b1014be63e6c/mypy_extensions-0.3.0.tar.gz#sha256=c4553f2475996c912658d3c2f398766cd47a65e0c2c6bb27ac8dea94a21640f4" >mypy_extensions-0.3.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/42/c8/3503632780b856dfd1ee4b3f6c61733833f9f58fe1a608dc8406054de99e/mypy_extensions-0.4.0.tar.gz#sha256=118ec27adb66c682dc842bc2ad48d6d5218867df5f90ca713f4deb3760324fd3" >mypy_extensions-0.4.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/4d/72/8d54e2b296631b9b14961d583e56e90d9d7fba8a240d5ce7f1113cc5e887/mypy_extensions-0.4.1-py2.py3-none-any.whl#sha256=b16cabe759f55e3409a7d231ebd2841378fb0c27a5d1994719e340e4f429ac3e" data-dist-info-metadata="sha256=15da51178c8af2bed1a429804dd2887b68ad35fdd404898e5dd429a99fa010e7" data-core-metadata="sha256=15da51178c8af2bed1a429804dd2887b68ad35fdd404898e5dd429a99fa010e7">mypy_extensions-0.4.1-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/c2/92/3cc05d1206237d54db7b2565a58080a909445330b4f90a6436302a49f0f8/mypy_extensions-0.4.1.tar.gz#sha256=37e0e956f41369209a3d5f34580150bcacfabaa57b33a15c0b25f4b5725e0812" >mypy_extensions-0.4.1.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/55/aa/2df16f9c6215b10c472fd7564b5f2c5644c13ab9c26d83b1c9373c11d97d/mypy_extensions-0.4.2-py2.py3-none-any.whl#sha256=c77e34350c01523cd63b6dc30b36515107b2c3cbacc8392d51d33ea18d59f8f1" data-dist-info-metadata="sha256=753214ffa231c23df663310f4d19d0d9fd75d729ff29de61335aa8b5cc3c1b5b" data-core-metadata="sha256=753214ffa231c23df663310f4d19d0d9fd75d729ff29de61335aa8b5cc3c1b5b">mypy_extensions-0.4.2-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/1e/2f/7bba47eb58f62a473387cd7658dedd0bedb4b0fa9d530bbbfa0a6d23034a/mypy_extensions-0.4.2.tar.gz#sha256=a161e3b917053de87dbe469987e173e49fb454eca10ef28b48b384538cc11458" >mypy_extensions-0.4.2.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl#sha256=090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d" data-dist-info-metadata="sha256=e801077942edd68f705e69e548d4aedd0790731e18c2af0e2a107af3d486d29e" data-core-metadata="sha256=e801077942edd68f705e69e548d4aedd0790731e18c2af0e2a107af3d486d29e">mypy_extensions-0.4.3-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/63/60/0582ce2eaced55f65a4406fc97beba256de4b7a95a0034c6576458c6519f/mypy_extensions-0.4.3.tar.gz#sha256=2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8" >mypy_extensions-0.4.3.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/d6/c6/7809c27b5c5dacb9f6537e9489d969b266f7091204c75a03048edcb4daf9/mypy_extensions-0.4.4.tar.gz#sha256=c8b707883a96efe9b4bb3aaf0dcc07e7e217d7d8368eec4db4049ee9e142f4fd" data-requires-python="&gt;=2.7" >mypy_extensions-0.4.4.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl#sha256=4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d" data-requires-python="&gt;=3.5" data-dist-info-metadata="sha256=617d60dffb00826a22f5662114d8111d02fa29ffb1ec67e39d6814084ef75486" data-core-metadata="sha256=617d60dffb00826a22f5662114d8111d02fa29ffb1ec67e39d6814084ef75486">mypy_extensions-1.0.0-py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/98/a4/1ab47638b92648243faf97a5aeb6ea83059cc3624972ab6b8d2316078d3f/mypy_extensions-1.0.0.tar.gz#sha256=75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782" data-requires-python="&gt;=3.5" >mypy_extensions-1.0.0.tar.gz</a><br />
</body>
</html>
<!--SERIAL 16697020-->

🔬 Minimal Reproduction

MODULE.bazel

bazel_dep(name = "rules_python", version = "0.35.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    is_default = True,
    python_version = "3.11.5",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "reqs",
    experimental_index_url = "https://<redacted>.d.codeartifact.eu-central-1.amazonaws.com/pypi/packages-prod/simple",
    python_version = "3.11.5",
    requirements_lock = "//:requirements.txt",
)
use_repo(pip, "reqs")

requirements.txt

mypy-extensions==1.0.0 \
    --hash=sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d \
    --hash=sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782

Running bazel fetch @reqs//mypy_extensions with the above files should reproduce the error below.

🔥 Exception or Error


ERROR: An error occurred during the fetch of repository 'rules_python~~pip~reqs_311_mypy_extensions_py3_none_any_4392f6c0':
   Traceback (most recent call last):
        File "/home/wmorrison/.cache/bazel/_bazel_wmorrison/0c20006a190ede26aadee6a87e8d620f/external/rules_python~/python/private/pypi/whl_library.bzl", line 216, column 31, in _whl_library_impl
                result = rctx.download(
Error in download: java.io.IOException: Bad URL: 1.0.0/mypy_extensions-1.0.0-py3-none-any.whl

🌍 Your Environment

Operating System:

Ubuntu Linux

Output of bazel version:


Bazelisk version: v1.20.0
Build label: 7.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Aug 19 16:12:50 2024 (1724083970)
Build timestamp: 1724083970
Build timestamp as int: 1724083970

Rules_python version:

  0.35.0

Anything else relevant?

Related to #1357
@aignas
Copy link
Collaborator

aignas commented Aug 23, 2024 •
edited
Loading

There was a recent PR in #2112 that fixed it for absolute URLs, fixing for this case should be very similar - whl_library should continue getting absolute URLs and the bug is in the parser.

@WillMorrison WillMorrison mentioned this issue Aug 24, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Aug 24, 2024
@WillMorrison
fix: Handle relative paths properly in _absolute_url (#2153)
e331afe 
This updates the simpleapi parser to handle indexes where wheel and
sdist may be an index_url relative path. It also organises the
conditionals with fewer negations so they're easier to read

Fixes: #2150
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted type: pip
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Handle relative paths properly in _absolute_url WillMorrison/rules_python
2 participants
@aignas @WillMorrisonEnspi