Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installing private packages from GAR in Docker build fails with 500 Internal Server Error on Poetry 1.8.0 #9026

Closed
4 tasks done
squarebridges opened this issue Feb 25, 2024 · 6 comments · Fixed by #9030
Closed
4 tasks done
Labels
kind/bug Something isn't working as expected

Comments

@squarebridges
Copy link

squarebridges commented Feb 25, 2024

  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I have a Python service whose pyproject.toml includes some dependencies from a private Google Artifact Registry Python repository. Until yesterday, this all worked fine. Poetry is installed in the container via:

# install poetry
RUN apt-get update \
    && apt-get install -y curl gnupg gcc \
    && curl -sSL https://install.python-poetry.org | python - \
    && poetry self add keyrings.google-artifactregistry-auth

and this keyring, given an environment with active application credentials, is able to authenticate me to my GAR repository and install the required dependencies. A build ran this morning and failed with the following error:

14.39   ValueError
14.39
14.39   Package('my-package', '0.0.0') is not in list
14.39
14.39   at /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:66 in package
14.39        62│         Note that this will be cached so the subsequent operations
14.39        63│         should be much faster.
14.39        64│         """
14.39        65│         try:
14.39     →  66│             index = self._packages.index(Package(name, version))
14.39        67│
14.39        68│             return self._packages[index]
14.39        69│         except ValueError:
14.39        70│             package = super().package(name, version, extras)
14.39
14.39 The following error occurred when trying to handle this error:
14.39
14.39
14.39   Stack trace:
14.39
14.39   34  /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:327 in run
14.40        325│
14.40        326│             try:
14.40      → 327│                 exit_code = self._run(io)
14.40        328│             except BrokenPipeError:
14.40        329│                 # If we are piped to another process, it may close early and send a
14.40
14.40   33  /opt/poetry/venv/lib/python3.12/site-packages/poetry/console/application.py:190 in _run
14.41        188│         self._load_plugins(io)
14.41        189│
14.41      → 190│         exit_code: int = super()._run(io)
14.41        191│         return exit_code
14.41        192│
14.41
14.41   32  /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:431 in _run
14.42        429│             io.input.interactive(interactive)
14.42        430│
14.42      → 431│         exit_code = self._run_command(command, io)
14.42        432│         self._running_command = None
14.42        433│
14.42
14.42   31  /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:473 in _run_command
14.42        471│
14.42        472│         if error is not None:
14.42      → 473│             raise error
14.42        474│
14.42        475│         return terminate_event.exit_code
14.42
14.42   30  /opt/poetry/venv/lib/python3.12/site-packages/cleo/application.py:457 in _run_command
14.43        455│
14.43        456│             if command_event.command_should_run():
14.43      → 457│                 exit_code = command.run(io)
14.43        458│             else:
14.43        459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
14.43
14.43   29  /opt/poetry/venv/lib/python3.12/site-packages/cleo/commands/base_command.py:117 in run
14.44        115│         io.input.validate()
14.44        116│
14.44      → 117│         return self.execute(io) or 0
14.44        118│
14.44        119│     def merge_application_definition(self, merge_args: bool = True) -> None:
14.44
14.44   28  /opt/poetry/venv/lib/python3.12/site-packages/cleo/commands/command.py:61 in execute
14.44         59│
14.44         60│         try:
14.44      →  61│             return self.handle()
14.44         62│         except KeyboardInterrupt:
14.44         63│             return 1
14.44
14.44   27  /opt/poetry/venv/lib/python3.12/site-packages/poetry/console/commands/install.py:153 in handle
14.45        151│         self.installer.verbose(self.io.is_verbose())
14.45        152│
14.45      → 153│         return_code = self.installer.run()
14.45        154│
14.45        155│         if return_code != 0:
14.45
14.45   26  /opt/poetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:104 in run
14.45        102│             self.verbose(True)
14.45        103│
14.45      → 104│         return self._do_install()
14.45        105│
14.45        106│     def dry_run(self, dry_run: bool = True) -> Installer:
14.45
14.45   25  /opt/poetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:241 in _do_install
14.46        239│                 source_root=self._env.path.joinpath("src")
14.46        240│             ):
14.46      → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
14.46        242│         else:
14.46        243│             self._io.write_line("Installing dependencies from lock file")
14.46
14.46   24  /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:71 in solve
14.46         69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
14.47         70│             start = time.time()
14.47      →  71│             packages, depths = self._solve()
14.47         72│             end = time.time()
14.47         73│
14.47
14.47   23  /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:154 in _solve
14.47        152│
14.47        153│         try:
14.47      → 154│             result = resolve_version(self._package, self._provider)
14.47        155│
14.47        156│             packages = result.packages
14.47
14.47   22  /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/__init__.py:18 in resolve_version
14.47         16│     solver = VersionSolver(root, provider)
14.47         17│
14.47      →  18│     return solver.solve()
14.47         19│
14.47
14.47   21  /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:175 in solve
14.48        173│             while next is not None:
14.48        174│                 self._propagate(next)
14.48      → 175│                 next = self._choose_package_version()
14.48        176│
14.48        177│             return self._result()
14.48
14.48   20  /opt/poetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
14.49        512│             package = locked
14.49        513│
14.49      → 514│         package = self._provider.complete_package(package)
14.49        515│
14.49        516│         conflict = False
14.49
14.49   19  /opt/poetry/venv/lib/python3.12/site-packages/poetry/puzzle/provider.py:489 in complete_package
14.50        487│                 dependency_package = DependencyPackage(
14.50        488│                     dependency,
14.50      → 489│                     self._pool.package(
14.50        490│                         package.pretty_name,
14.50        491│                         package.version,
14.50
14.50   18  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/repository_pool.py:198 in package
14.51        196│     ) -> Package:
14.51        197│         if repository_name:
14.51      → 198│             return self.repository(repository_name).package(
14.51        199│                 name, version, extras=extras
14.51        200│             )
14.51
14.51   17  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:70 in package
14.51         68│             return self._packages[index]
14.51         69│         except ValueError:
14.51      →  70│             package = super().package(name, version, extras)
14.51         71│             package._source_type = "legacy"
14.51         72│             package._source_url = self._url
14.51
14.51   16  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:75 in package
14.51         73│         extras: list[str] | None = None,
14.51         74│     ) -> Package:
14.51      →  75│         return self.get_release_info(canonicalize_name(name), version).to_package(
14.51         76│             name=name, extras=extras
14.51         77│         )
14.51
14.51   15  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
14.51         50│             return PackageInfo.load(self._get_release_info(name, version))
14.51         51│
14.51      →  52│         cached = self._release_cache.remember(
14.51         53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
14.51         54│         )
14.51
14.51   14  /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/cache.py:147 in remember
14.52        145│         value = self.get(key)
14.52        146│         if value is None:
14.52      → 147│             value = callback() if callable(callback) else callback
14.52        148│             self.put(key, value, minutes)
14.52        149│         return value
14.52
14.52   13  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
14.52         51│
14.52         52│         cached = self._release_cache.remember(
14.52      →  53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
14.52         54│         )
14.52         55│
14.52
14.52   12  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/legacy_repository.py:123 in _get_release_info
14.52        121│         yanked = page.yanked(name, version)
14.52        122│
14.52      → 123│         return self._links_to_data(
14.52        124│             links,
14.52        125│             PackageInfo(
14.52
14.52   11  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:360 in _links_to_data
14.53        358│
14.53        359│         # drop yanked files unless the entire release is yanked
14.53      → 360│         info = self._get_info_from_links(links, ignore_yanked=not data.yanked)
14.53        361│
14.53        362│         data.summary = info.summary
14.53
14.53   10  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:310 in _get_info_from_links
14.53        308│                 return self._get_info_from_metadata(
14.53        309│                     universal_python3_wheel
14.53      → 310│                 ) or self._get_info_from_wheel(universal_python3_wheel)
14.53        311│
14.53        312│             if universal_python2_wheel:
14.53
14.53    9  /opt/poetry/venv/lib/python3.12/site-packages/poetry/repositories/http_repository.py:123 in _get_info_from_wheel
14.54        121│             try:
14.54        122│                 package_info = PackageInfo.from_metadata(
14.54      → 123│                     metadata_from_wheel_url(link.filename, link.url, self.session)
14.54        124│                 )
14.54        125│             except HTTPRangeRequestUnsupported:
14.54
14.54    8  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:77 in metadata_from_wheel_url
14.55         75│         # After context manager exit, wheel.name will point to a deleted file path.
14.55         76│         # Add `delete_backing_file=False` to disable this for debugging.
14.55      →  77│         with LazyWheelOverHTTP(url, session) as lazy_file:
14.55         78│             metadata_bytes = lazy_file.read_metadata(name)
14.55         79│
14.55
14.55    7  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:289 in __enter__
14.56        287│     def __enter__(self: U) -> U:
14.56        288│         super().__enter__()
14.56      → 289│         self._setup_content()
14.56        290│         return self
14.56        291│
14.56
14.56    6  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:357 in _setup_content
14.57        355│         if self._length is None:
14.57        356│             logger.debug("begin fetching content length")
14.57      → 357│             self._length = self._fetch_content_length()
14.57        358│             logger.debug("done fetching content length (is: %d)", self._length)
14.57        359│             # Enable us to seek and write anywhere in the backing file up to this
14.57
14.57    5  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:508 in _fetch_content_length
14.58        506│         """
14.58        507│         initial_chunk_size = self._initial_chunk_length()
14.58      → 508│         ret_length, tail = self._extract_content_length(initial_chunk_size)
14.58        509│
14.58        510│         # Need to explicitly truncate here in order to perform the write and seek
14.58
14.58    4  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:609 in _extract_content_length
14.59        607│         try:
14.59        608│             # Initial range request for just the end of the file.
14.59      → 609│             file_length, tail = self._try_initial_chunk_request(initial_chunk_size)
14.59        610│         except HTTPError as e:
14.59        611│             resp = e.response
14.59
14.59    3  /opt/poetry/venv/lib/python3.12/site-packages/poetry/inspection/lazy_wheel.py:572 in _try_initial_chunk_request
14.60        570│
14.60        571│         self._request_count += 1
14.60      → 572│         tail = self._session.get(self._url, headers=headers, stream=True)
14.60        573│         tail.raise_for_status()
14.60        574│
14.60
14.60    2  /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/authenticator.py:267 in get
14.60        265│
14.60        266│     def get(self, url: str, **kwargs: Any) -> requests.Response:
14.60      → 267│         return self.request("get", url, **kwargs)
14.60        268│
14.60        269│     def head(self, url: str, **kwargs: Any) -> requests.Response:
14.60
14.60    1  /opt/poetry/venv/lib/python3.12/site-packages/poetry/utils/authenticator.py:245 in request
14.61        243│                 if resp.status_code not in STATUS_FORCELIST or is_last_attempt:
14.61        244│                     if raise_for_status:
14.61      → 245│                         resp.raise_for_status()
14.61        246│                     return resp
14.61        247│
14.61
14.61   HTTPError
14.61
14.61   500 Server Error: Internal Server Error for url: https://us-central1-python.pkg.dev/artifacts-downloads/namespaces/atomic-uxr-operations-jcqu/repositories/python-packages/downloads/very-long-base64-string
14.61
14.61   at /opt/poetry/venv/lib/python3.12/site-packages/requests/models.py:1021 in raise_for_status
14.62       1017│                 f"{self.status_code} Server Error: {reason} for url: {self.url}"
14.62       1018│             )
14.62       1019│
14.62       1020│         if http_error_msg:
14.62     → 1021│             raise HTTPError(http_error_msg, response=self)
14.62       1022│
14.62       1023│     def close(self):
14.62       1024│         """Releases the connection back to the pool. Once this method has been
14.62       1025│         called the underlying ``raw`` object must not be accessed again.
------

The image to this point in the build had installed Poetry 1.8.0. I then went back to this step in the build and added: poetry self update 1.7.1. The build completed successfully after doing this.

poetry install commands on my local machine, running Arch Linux (uname -r == 6.6.16-1-lts) with Python 3.12.1, complete successfully whether I have Poetry 1.7.1 or Poetry 1.8.0 installed, so this seems to only impact Docker.

This error occurs even if I am interacting with a container; if I cut off the Dockerfile to the instruction just before the poetry install and run the poetry install myself, I get the same 500 error. If I then take the URL given in the error message and curl it manually, I get a 200 response. Similarly I get a 200 response with requests in a Python interactive session.

I have tried this with multiple GAR repositories across multiple GCP projects under multiple accounts, so it is not related to a specific GAR repository or GCP account.

Here's a minimal version of the Dockerfile I'm using:

# python 3.12 slim
FROM python@sha256:d0c9bf03d80a3c00963f1b973760094c0b2070638fa64dd4f67d2247c0110efc AS base

ENV POETRY_HOME="/opt/poetry" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_NO_INTERACTION=1
ENV PATH="${POETRY_HOME}/bin:${PATH}"

# install poetry
RUN apt-get update \
    && apt-get install -y curl gnupg gcc \
    && curl -sSL https://install.python-poetry.org | python - \
    && poetry self add keyrings.google-artifactregistry-auth

# install dependencies
COPY app /home/myuser/app
RUN --mount=type=secret,id=adc,target=/root/.config/gcloud/application_default_credentials.json \
    cd /home/myuser/app && \
    poetry install

My temporary fix is to add the following to the # install poetry instruction:
&& poetry self update 1.7.1

at which point it works fine.

To summarize:

  • Inside Docker builds and Docker containers, with Poetry 1.8.0 installed, poetry install commands that install from private GAR sources throw 500 Internal Server Error.
  • In the same situation but with Poetry 1.7.1 installed, there is no error.
  • On my host machine, with Poetry 1.8.0 installed OR Poetry 1.7.1 installed, there is no error.
@squarebridges squarebridges added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Feb 25, 2024
@dimbleby
Copy link
Contributor

dimbleby commented Feb 25, 2024

500 Internal Server Error is a server error, you will want to report this to whoever is responsible for the server.

meanwhile I guess this is related to the new "lazy wheel" stuff: see blog post and release notes for discussion of what that is and how to disable it if you need to do so

@squarebridges
Copy link
Author

If I report it to the GAR team, and say that it works with Poetry 1.7.1 but fails with Poetry 1.8.0, will that not lead me back here?

@squarebridges
Copy link
Author

squarebridges commented Feb 25, 2024

In either case, adding POETRY_SOLVER_LAZY_WHEEL=0 to the environment solved this one for me. Thanks!

@dimbleby
Copy link
Contributor

If I report it to the GAR team, and say that it works with Poetry 1.7.1 but fails with Poetry 1.8.0, will that not lead me back here?

no. a server error is exactly what it says it is: a server error.

if the server (GAR) believes the client (poetry) is doing something wrong, it should return a 4XX error. 5XX is acknowledging that the problem is at their end.

sometimes the way these things go is that the maintainer of the server says: ah, that is a client error - just not one that I had remembered to handle. But at least then they have told us what it is they think the client is doing wrong.

@radoering
Copy link
Member

I might make sense to add 500 to

if code in [codes.not_implemented, codes.method_not_allowed]:

Even if good servers will return a more specific code like the ones we already catch but servers that are not able to handle it gracefully just fail with an internal error. I don't think it hurts to be more robust on our side: #9030

@abn abn removed the status/triage This issue needs to be triaged label Feb 26, 2024
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants