Installing praw from source or sdist fails due to importing runtime deps at build time #2004

CAM-Gerlach · 2023-12-15T22:19:44Z

Describe the Bug

Same issue as praw-dev/prawcore#164 and @LilSpazJoekp requested followup from praw-dev/prawcore#165

Due to the use of Flit's fallback automatic metadata version extraction needing to dynamically import the package __init__.py instead of reading it statically (see pypa/flit#386 ) , since the actual __version__ is imported from const.py, this requires the runtime dependencies be installed in the build environment when building or installing from an sdist (or the source tree).

This happens merely by chance to currently be a transitive backend dependency of flit_core and thus building in isolated mode works. However, this shouldn't be relied upon, as if either flit_core or praw's dependencies were to ever change, this would break isolated builds too. And requests isn't an exposed non-backend dependency of flit-core, so it doesn't actually get installed otherwise.

This means Requests must be manually installed (not just the explicitly pyproject,toml-specified build dependencies) if building/installing praw in any other context than pip's isolated mode, e.g. without the --no-build-isolation flag, via the legacy non-PEP 517 builder, or via other tools or most downstream ecosystems that use other build isolation mechanisms. In particular, this was an issue on conda-forge/prawcore-feedstock#14 where I was packaging the new prawcore 2.4.0 version for Conda-Forge—you can see the full Azure build log.

This can be worked around for now by manually installing requests into the build environment, but that's certainly not an ideal solution as it adds an extra build dependency (and its dependencies in turn), requires extra work by everyone installing from source (without build isolation), makes builds take longer, and is fragile and not easily maintainable/scalable long term for other runtime dependencies.

Desired Result

Praw is able to be built and installed without installing requests, or relying on it happening to be a transitive build backend dependency of flit_core and present "by accident".

While there are other ways to achieve this, as described in praw-dev/prawcore#164 , @LilSpazJoekp indicated he preferred moving __version__ to be in __init__.py directly, so Flit can find it by static inspection without triggering an import, and naturally asked for the same approach here as applied in praw-dev/prawcore#165 .

This works, with one additional complication: the USER_AGENT_FORMAT module-level constant in const.py relies in turn on __version__, which means that praw.const needs to import it from praw (__init__), and because praw.reddit.Reddit is imported in __init__ for convenience (which in turn imports praw.const), this creates an import cycle (as well as making praw expensive to import, on the order of multiple seconds on an older machine).

This cannot be trivially solved by a scope-local import, because USER_AGENT_FORMAT is a module-level constant. The simplest and likely least disruptive solution, since USER_AGENT_FORMAT is only used on place in PRAW (praw.reddit) is to just inject __version__ at the one point of use along with the user-configured user agent rather than statically, eliminating an extra layer of format replacement nesting in the process.

The one non-trivial impact from this is requiring a change to user code directly using USER_AGENT_FORMAT themselves to insert praw.__version__ along with their own user agent. After various grep.app queries, which scans the entirety of public code on GitHub, I did find a single instance of this that would be affected, in the (closed-development) nnreddit project providing a Reddit backend for the Gnus reader. However, the fix is straightforward (an extra arg to one format call) and backward-compatible with older PRAW, and the primary change here, removing, __version__ from praw.const is also equally technically backward-incompatible and was accepted in prawcore, so not sure how much of an issue that is for you.

Using a named placeholder for the PRAW version, e.g."{} PRAW/{praw_version}", would give a clearer error message as to what is needed to add to the format call, while modifying existing code (e.g. USER_AGENT_FORMAT.format(user_agent)) to support the new PRAW version (i.e. USER_AGENT_FORMAT.format(user_agent, praw_version=praw.__version__) would be fully backward compatible with USER_AGENT_FORMAT in older PRAW, as format will simply drop arguments that don't match placeholders in the string.

The other alternative as mentioned in the other issue would be simply setting the version statically in pyproject.toml. This would avoid all these issues and needing to change anything else besides the set_version.py script as well as give you the benefits of static metadata. It would mean the version is defined multiple places which you wanted to avoid, but if you're using that script to set it anyway, it shouldn't actually require any additional maintainer effort since the script could take care of it too. Of course, if that's the case we'd presumably want to go back and make that same change in prawcore too; however, it would simplify the implementation in asyncpraw (as well as asyncprawcore) and avoid this same issue with USER_AGENT_FORMAT, and for which asyncpraw.const.__version__ is used at least once in the wild.

Up to you!

Code to reproduce the bug

In a fresh venv without requests or praw installed, e.g.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip wheel

Run:

pip install flit-core  # Or flit
pip install --no-build-isolation .

My code does not include sensitive credentials

Yes, I have removed sensitive credentials from my code.

Relevant Logs

See praw-dev/prawcore#164

This code has previously worked as intended

Yes

Operating System/Environment

Any

Python Version

Any, tested 3.9-3.12

PRAW Version

7.7.2.dev0

Links, references, and/or additional comments?

No response

The text was updated successfully, but these errors were encountered:

LilSpazJoekp · 2023-12-17T23:40:44Z

When I run your reproductions steps I'm not encountering this issue with requests being required for install.

Here's what I'm doing:

Setup up the Python virtual environment.

python3.12 -m venv .venv
source .venv/bin/activate
pip install -U pip

Run code to reproduce error using local praw repo.
Command:

pip install flit-core
pip install --no-build-isolation . --no-binary .

Output:

Collecting flit-core
  Using cached flit_core-3.9.0-py3-none-any.whl.metadata (822 bytes)
Using cached flit_core-3.9.0-py3-none-any.whl (63 kB)
Installing collected packages: flit-core
Successfully installed flit-core-3.9.0
Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      Traceback (most recent call last):
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
          metadata = make_metadata(module, ini_info)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 425, in make_metadata
          md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 222, in get_info_from_module
          docstring, version = get_docstring_and_version_via_import(target)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 195, in get_docstring_and_version_via_import
          spec.loader.exec_module(m)
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/Users/jkpayne/PythonProjects/praw/praw/__init__.py", line 12, in <module>
          from .reddit import Reddit
        File "/Users/jkpayne/PythonProjects/praw/praw/reddit.py", line 15, in <module>
          from prawcore import (
      ModuleNotFoundError: No module named 'prawcore'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Install prawcore from local prawcore repo
Command:

pip install --no-build-isolation ../prawcore --no-binary ../prawcore

Output:

Processing /Users/jkpayne/PythonProjects/prawcore
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting requests<3.0,>=2.6.0 (from prawcore==2.4.1.dev0)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests<3.0,>=2.6.0->prawcore==2.4.1.dev0)
  Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl (122 kB)
Using cached idna-3.6-py3-none-any.whl (61 kB)
Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
Building wheels for collected packages: prawcore
  Building wheel for prawcore (pyproject.toml): started
  Building wheel for prawcore (pyproject.toml): finished with status 'done'
  Created wheel for prawcore: filename=prawcore-2.4.1.dev0-py3-none-any.whl size=17264 sha256=6ab3e4d2378eb16941e6d8acc6789916ca4f52a1f63f60aa89148b12b2211e8a
  Stored in directory: /private/var/folders/xf/ldd27nrn7_386m2frx69301c0000gp/T/pip-ephem-wheel-cache-5qgatd7r/wheels/aa/6d/68/a5119784a421be35656aba4aa4b2eba109e01530cb65bc4414
Successfully built prawcore
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, prawcore
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 prawcore-2.4.1.dev0 requests-2.31.0 urllib3-2.1.0

Run the code to reproduce error again
Command:

pip install --no-build-isolation . --no-binary .

Output:

Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      Traceback (most recent call last):
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
          metadata = make_metadata(module, ini_info)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 425, in make_metadata
          md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 222, in get_info_from_module
          docstring, version = get_docstring_and_version_via_import(target)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jkpayne/PythonProjects/praw/.venv/lib/python3.12/site-packages/flit_core/common.py", line 195, in get_docstring_and_version_via_import
          spec.loader.exec_module(m)
        File "<frozen importlib._bootstrap_external>", line 994, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/Users/jkpayne/PythonProjects/praw/praw/__init__.py", line 12, in <module>
          from .reddit import Reddit
        File "/Users/jkpayne/PythonProjects/praw/praw/reddit.py", line 28, in <module>
          from . import models
        File "/Users/jkpayne/PythonProjects/praw/praw/models/__init__.py", line 4, in <module>
          from .helpers import DraftHelper, LiveHelper, MultiredditHelper, SubredditHelper
        File "/Users/jkpayne/PythonProjects/praw/praw/models/helpers.py", line 10, in <module>
          from .reddit.draft import Draft
        File "/Users/jkpayne/PythonProjects/praw/praw/models/reddit/draft.py", line 9, in <module>
          from .subreddit import Subreddit
        File "/Users/jkpayne/PythonProjects/praw/praw/models/reddit/subreddit.py", line 15, in <module>
          import websocket
      ModuleNotFoundError: No module named 'websocket'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Change websockets import to be a local import.

Run the code to reproduce error again. No error occurs.
Command:

pip install --no-build-isolation . --no-binary .

Output:

Processing /Users/jkpayne/PythonProjects/praw
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: prawcore<3,>=2.1 in ./.venv/lib/python3.12/site-packages (from praw==7.7.2.dev0) (2.4.1.dev0)
Collecting update_checker>=0.18 (from praw==7.7.2.dev0)
  Using cached update_checker-0.18.0-py3-none-any.whl (7.0 kB)
Collecting websocket-client>=0.54.0 (from praw==7.7.2.dev0)
  Using cached websocket_client-1.7.0-py3-none-any.whl.metadata (7.9 kB)
Requirement already satisfied: requests<3.0,>=2.6.0 in ./.venv/lib/python3.12/site-packages (from prawcore<3,>=2.1->praw==7.7.2.dev0) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.12/site-packages (from requests<3.0,>=2.6.0->prawcore<3,>=2.1->praw==7.7.2.dev0) (2023.11.17)
Using cached websocket_client-1.7.0-py3-none-any.whl (58 kB)
Building wheels for collected packages: praw
  Building wheel for praw (pyproject.toml): started
  Building wheel for praw (pyproject.toml): finished with status 'done'
  Created wheel for praw: filename=praw-7.7.2.dev0-py3-none-any.whl size=189356 sha256=c0a0cb68afc5accf5a96847c1367aebfbd068dc2e51472fdb5e90950ae43de63
  Stored in directory: /private/var/folders/xf/ldd27nrn7_386m2frx69301c0000gp/T/pip-ephem-wheel-cache-pf5rcd8_/wheels/03/25/df/2a1cafce15246fae04b8bdc466629219cc0de6e6fd95da8b2a
Successfully built praw
Installing collected packages: websocket-client, update_checker, praw
Successfully installed praw-7.7.2.dev0 update_checker-0.18.0 websocket-client-1.7.0

CAM-Gerlach · 2023-12-18T23:18:31Z

Sorry, I copy-pasted a lot of the relevant content from the prawcore issue, and I missed updating some important bits. Namely, the error message will reference whatever dependenc(ies) of the package don't already happen to be installed in the build environment, starting from the first missing import it hits, in this case prawcore. As such, the command (or more simply for this case, pip install --no-build-isolation .) in your Step 2 does repro the error (as you found), just with a different missing dependency (in this case prawcore).

After you manually install prawcore (which works correctly due to the fix we implemented over there, and installs most of the other dependencies including requests), you then see another error due to the missing websockets dependency at build time, required for the same reasons. After patching that out, everything works (evidently because the only other PRAW-unique dependency, update_checker, happens to not be in the import chain triggered by reddit.py in __init__,py currently).

So, TL;DR, the issue is still present as reported, just with a different error message, harder to hack around and more likely to occur (due to the dependencies being much less common to happen to be installed in the build environment due to other build deps or by default). I've updated the title and initial description accordingly, thanks.

github-actions · 2024-01-18T01:04:12Z

This issue is stale because it has been open for 30 days with no activity.

Remove the Stale label or comment or this will be closed in 30 days.

CAM-Gerlach · 2024-01-30T07:52:54Z

Hey @LilSpazJoekp , have you gotten a chance to give this another look and lmk how you want to move forward here?

Simply using static metadata and having the helper script you already use to update the version other places updated it too is, after a lot of careful thought and testing is the solution I'd recommend, as it is a ≈2 line change that doesn't require touching the code at all and is fully backward-compatible, plus gets you the benefits of standardized static metadata and can be easily applied consistently across all your repos without requiring customization and testing for each.

However, I've also proposed another solution along the lines of what I originally implemented in prawcore, albeit somewhat more complicated and does have a small but non-zero backward compat impact.

Lmk, thanks!

github-actions · 2024-02-29T12:24:44Z

This issue is stale because it has been open for 30 days with no activity.

Remove the Stale label or comment or this will be closed in 30 days.

github-actions · 2024-03-30T18:17:14Z

This issue was closed because it has been stale for 30 days with no activity.

CAM-Gerlach · 2024-11-26T05:15:39Z

Hey, so seems this was automatically closed while it was still waiting for maintainer input from @LilSpazJoekp .

In any case, Conda-Forge builds (and quite probably other downstreams and builds) of PRAW 7.8.0 and above are now activately broken due to this issue (see e.g. conda-forge/praw-feedstock#26 ) and the issue is now urgent.

The hacky temporary workaround for prawcore, manually installing all the runtime dependencies at build time (a single extra package, requests, also required for the backend anyway) doesn't really scale well to PRAW given it has many dependencies with different specs.

The simplest and overall best path forward, that would be easy for me (or you) to implement on all 4 praw/prawcore/async projects with no additional changes or user-facing impact, would be to just specify the version as static metadata in pyproject.toml (as recommended anyway). The only downside is mitigated by just adding that one additional spot to the existing list in your bump_version.py so there's no additional maintainer workload going forward.

Alternatively, we could implement the other solution of refactoring const.py, but that does require several small but not completely trivial code changes (i.e. risk), entails a small but non-zero backward compat impact (which did have a few uses in the wild), and needs to be adapted to each package its used on. Therefore, at this point I'd really recommend the former.

LilSpazJoekp · 2024-12-13T01:12:58Z

Sorry this slipped through. I'm good with your recommendations. Feel free to move forward those PRs and I can get them merged in. Again, apologies for this slipping through my life has gotten way busier since my twins were born.

This was referenced Dec 15, 2023

Move version from const.py to __init__.py for Flit static introspection praw-dev/prawcore#165

Merged

Installing from prawcore sdist fails without requests in build environment praw-dev/prawcore#164

Closed

CAM-Gerlach changed the title ~~Installing praw from source or sdist fails without requests in build environment~~ Installing praw from source or sdist fails due to importing runtime deps at build time Dec 18, 2023

github-actions bot added the Stale Issue or pull request has been inactive for 20 days label Jan 18, 2024

github-actions bot removed the Stale Issue or pull request has been inactive for 20 days label Jan 30, 2024

github-actions bot added the Stale Issue or pull request has been inactive for 20 days label Feb 29, 2024

github-actions bot added the Auto-closed - Stale Automatically closed due to being stale for too long label Mar 30, 2024

github-actions bot closed this as completed Mar 30, 2024

LilSpazJoekp reopened this Dec 13, 2024

github-actions bot removed Auto-closed - Stale Automatically closed due to being stale for too long Stale Issue or pull request has been inactive for 20 days labels Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing praw from source or sdist fails due to importing runtime deps at build time #2004

Installing praw from source or sdist fails due to importing runtime deps at build time #2004

CAM-Gerlach commented Dec 15, 2023 •

edited

Loading

LilSpazJoekp commented Dec 17, 2023

CAM-Gerlach commented Dec 18, 2023 •

edited

Loading

github-actions bot commented Jan 18, 2024

CAM-Gerlach commented Jan 30, 2024

github-actions bot commented Feb 29, 2024

github-actions bot commented Mar 30, 2024

CAM-Gerlach commented Nov 26, 2024

LilSpazJoekp commented Dec 13, 2024

Installing praw from source or sdist fails due to importing runtime deps at build time #2004

Installing praw from source or sdist fails due to importing runtime deps at build time #2004

Comments

CAM-Gerlach commented Dec 15, 2023 • edited Loading

Describe the Bug

Desired Result

Code to reproduce the bug

My code does not include sensitive credentials

Relevant Logs

This code has previously worked as intended

Operating System/Environment

Python Version

PRAW Version

Links, references, and/or additional comments?

LilSpazJoekp commented Dec 17, 2023

CAM-Gerlach commented Dec 18, 2023 • edited Loading

github-actions bot commented Jan 18, 2024

CAM-Gerlach commented Jan 30, 2024

github-actions bot commented Feb 29, 2024

github-actions bot commented Mar 30, 2024

CAM-Gerlach commented Nov 26, 2024

LilSpazJoekp commented Dec 13, 2024

CAM-Gerlach commented Dec 15, 2023 •

edited

Loading

CAM-Gerlach commented Dec 18, 2023 •

edited

Loading