forked from pantsbuild/pants
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
…antsbuild#16379) Pants uses `pkgutil.get_data()` in all manner of places to load resource files that are included within the Pants Python package. `get_data()` is not supported in the PEP302 pluggable importer used by PyOxidizer. `importlib.resources`, however, _is_ supported. `importlib.resources` (as far as PyOxidizer is concerned, at the very least) has some caveats: * Resources can only be loaded from packages that contain an `__init__.py` file (i.e. are a non-namespace package) * Resources cannot be `.py` files This adds a shim function called `read_resource()` that allows reading resource files with more-or-less the same API as `pkgutil.get_data()` (in particular, allowing `read_resource(__name__, …)`, which is used commonly), but plugs into `importlib.resources.read_binary()` to allow for better portability. A symlink to `src/python/pants/VERSION` is added at `src/python/pants/_version/VERSION`, so that the resource can be loaded with the new API, without immediately breaking the hard-coded version file location required by our setup script. Finally, the python dependency parser has been renamed to `dependency_parser_py`, so that it can be loaded as a resource, and still treated as a `python_source` for Pants linting purposes. Addresses pantsbuild#7369.
- Loading branch information
Christopher Neugebauer
authored
Aug 4, 2022
1 parent
671cf3e
commit a01f375
Showing
17 changed files
with
93 additions
and
21 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md). | ||
# Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
||
# This package is used to provide a non-namespaced Python package from which | ||
# `importlib.resources.read_binary` can read our `VERSION` file. The `VERSION` file is a | ||
# symlink to the concrete `VERSION` file in `src/python/pants`. This creates a minimal package | ||
# that can be imported by `pants.version` during tests, and inferred by Pants as a dependency. | ||
# Future versions of `importlib.resources` will be able to read resources from namespace | ||
# packages, at which point, this package will no longer need to exist. | ||
|
||
python_sources(dependencies=["./VERSION:resources"]) | ||
resources(name="resources", sources=["VERSION"]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../VERSION |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 changes: 13 additions & 2 deletions
15
src/python/pants/backend/python/dependency_inference/scripts/BUILD
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,22 @@ | ||
# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md). | ||
# Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
||
resource(name="dependency_parser", source="dependency_parser.py") | ||
resource( | ||
name="dependency_parser", | ||
source="dependency_parser_py", | ||
dependencies=[ | ||
":init_py", | ||
], | ||
) | ||
resource( | ||
name="init_py", | ||
source="__init__.py", | ||
) | ||
|
||
# Also expose scripts as python sources so they get formatted/linted/checked. | ||
python_source( | ||
name="dependency_parser_source", | ||
source="dependency_parser.py", | ||
source="dependency_parser_py", | ||
# This is run with Python 2.7 and 3.5+, so we shouldn't be running pyupgrade. | ||
# skip_pyupgrade=True, | ||
) |
Empty file.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md). | ||
# Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
||
|
||
import importlib | ||
from importlib import resources | ||
from itertools import chain | ||
|
||
|
||
def read_resource(package_or_module: str, resource: str) -> bytes: | ||
"""Reads a resource file from within the Pants package itself. | ||
This helper function is designed for compatibility with `pkgutil.get_data()` wherever possible, | ||
but also allows compability with PEP302 pluggable importers such as included with PyOxidizer. | ||
This requires that resources are loaded from a valid Python package (i.e. must have an | ||
`__init__.py` file in the directory). | ||
""" | ||
|
||
a = importlib.import_module(package_or_module) | ||
package_ = a.__package__ | ||
|
||
if package_ is None: | ||
raise ValueError( | ||
"`read_resource` can only help find resources for packages or modules that live in " | ||
"a package." | ||
) | ||
|
||
resource_parts = resource.split("/") | ||
|
||
if len(resource_parts) == 1: | ||
package = package_ | ||
else: | ||
package = ".".join(chain((package_,), resource_parts[:-1])) | ||
resource = resource_parts[-1] | ||
|
||
return resources.read_binary(package, resource) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters