Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non-package modules. #258

Merged
merged 8 commits into from
Oct 7, 2022
Merged
1 change: 1 addition & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ omit =
*/_itertools.py
*/_legacy.py
*/simple.py
*/_path.py

[report]
show_missing = True
12 changes: 12 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
v5.10.0
=======

* #203: Lifted restriction on modules passed to ``files``.
Now modules need not be a package and if a non-package
module is passed, resources will be resolved adjacent to
those modules, even for modules not found in any package.
For example, ``files(import_module('mod.py'))`` will
resolve resources found at the root. The parameter to
files was renamed from 'package' to 'anchor', with a
compatibility shim for those passing by keyword.

v5.9.0
======

Expand Down
6 changes: 3 additions & 3 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ in Python packages. It provides functionality similar to ``pkg_resources``
`Basic Resource Access`_ API, but without all of the overhead and performance
problems of ``pkg_resources``.

In our terminology, a *resource* is a file tree that is located within an
importable `Python package`_. Resources can live on the file system or in a
In our terminology, a *resource* is a file tree that is located alongside an
importable `Python module`_. Resources can live on the file system or in a
zip file, with support for other loader_ classes that implement the appropriate
API for reading resources.

Expand Down Expand Up @@ -43,5 +43,5 @@ Indices and tables


.. _`Basic Resource Access`: http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access
.. _`Python package`: https://docs.python.org/3/reference/import.html#packages
.. _`Python module`: https://docs.python.org/3/glossary.html#term-module
.. _loader: https://docs.python.org/3/reference/import.html#finders-and-loaders
40 changes: 23 additions & 17 deletions docs/using.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
===========================

``importlib_resources`` is a library that leverages Python's import system to
provide access to *resources* within *packages*. Given that this library is
built on top of the import system, it is highly efficient and easy to use.
This library's philosophy is that, if you can import a package, you can access
resources within that package. Resources can be opened or read, in either
binary or text mode.
provide access to *resources* within *packages* and alongside *modules*. Given
that this library is built on top of the import system, it is highly efficient
and easy to use. This library's philosophy is that, if one can import a
module, one can access resources associated with that module. Resources can be
opened or read, in either binary or text mode.

What exactly do we mean by "a resource"? It's easiest to think about the
metaphor of files and directories on the file system, though it's important to
Expand All @@ -23,11 +23,14 @@ If you have a file system layout such as::
one/
__init__.py
resource1.txt
module1.py
resources1/
resource1.1.txt
two/
__init__.py
resource2.txt
standalone.py
resource3.txt

then the directories are ``data``, ``data/one``, and ``data/two``. Each of
these are also Python packages by virtue of the fact that they all contain
Expand All @@ -48,11 +51,14 @@ package directory, so
``data/one/resource1.txt`` and ``data/two/resource2.txt`` are both resources,
as are the ``__init__.py`` files in all the directories.

Resources are always accessed relative to the package that they live in.
``resource1.txt`` and ``resources1/resource1.1.txt`` are resources within
the ``data.one`` package, and
``two/resource2.txt`` is a resource within the
``data`` package.
Resources in packages are always accessed relative to the package that they
live in. ``resource1.txt`` and ``resources1/resource1.1.txt`` are resources
within the ``data.one`` package, and ``two/resource2.txt`` is a resource
within the ``data`` package.

Resources may also be referenced relative to another *anchor*, a module in a
package (``data.one.module1``) or a standalone module (``standalone``). In
this case, resources are loaded from the same loader that loaded that module.


Example
Expand Down Expand Up @@ -103,14 +109,14 @@ using ``importlib_resources`` would look like::
eml = files('email.tests.data').joinpath('message.eml').read_text()


Packages or package names
=========================
Anchors
=======

All of the ``importlib_resources`` APIs take a *package* as their first
parameter, but this can either be a package name (as a ``str``) or an actual
module object, though the module *must* be a package. If a string is
passed in, it must name an importable Python package, and this is first
imported. Thus the above example could also be written as::
The ``importlib_resources`` ``files`` API takes an *anchor* as its first
parameter, which can either be a package name (as a ``str``) or an actual
module object. If a string is passed in, it must name an importable Python
module, which is imported prior to loading any resources. Thus the above
example could also be written as::

import email.tests.data
eml = files(email.tests.data).joinpath('message.eml').read_text()
Expand Down
63 changes: 44 additions & 19 deletions importlib_resources/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,56 @@
import contextlib
import types
import importlib
import warnings

from typing import Union, Optional
from typing import Union, Optional, cast
from .abc import ResourceReader, Traversable

from ._compat import wrap_spec

Package = Union[types.ModuleType, str]
Anchor = Package


def files(package: Package) -> Traversable:
def package_to_anchor(func):
"""
Get a Traversable resource from a package
Replace 'package' parameter as 'anchor' and warn about the change.

Other errors should fall through.

>>> files()
Traceback (most recent call last):
TypeError: files() missing 1 required positional argument: 'anchor'
>>> files('a', 'b')
Traceback (most recent call last):
TypeError: files() takes 1 positional argument but 2 were given
"""
undefined = object()

@functools.wraps(func)
def wrapper(anchor=undefined, package=undefined):
if package is not undefined:
if anchor is not undefined:
return func(anchor, package)
warnings.warn(
"First parameter to files is renamed to 'anchor'",
DeprecationWarning,
stacklevel=2,
)
return func(package)
elif anchor is undefined:
return func()
return func(anchor)

return wrapper


@package_to_anchor
def files(anchor: Anchor) -> Traversable:
"""
Get a Traversable resource for an anchor.
"""
return from_package(get_package(package))
return from_package(resolve(anchor))


def get_resource_reader(package: types.ModuleType) -> Optional[ResourceReader]:
Expand All @@ -38,27 +74,16 @@ def get_resource_reader(package: types.ModuleType) -> Optional[ResourceReader]:


@functools.singledispatch
def resolve(cand: Package):
return cand
def resolve(cand: Anchor) -> types.ModuleType:
return cast(types.ModuleType, cand)


@resolve.register
def _(cand: str):
def _(cand: str) -> types.ModuleType:
return importlib.import_module(cand)


def get_package(package: Package) -> types.ModuleType:
"""Take a package name or module object and return the module.

Raise an exception if the resolved module is not a package.
"""
resolved = resolve(package)
if wrap_spec(resolved).submodule_search_locations is None:
raise TypeError(f'{package!r} is not a package')
return resolved


def from_package(package):
def from_package(package: types.ModuleType):
"""
Return a Traversable object for the given package.

Expand Down
15 changes: 14 additions & 1 deletion importlib_resources/tests/_compat.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,20 @@
except ImportError:
# Python 3.9 and earlier
class import_helper: # type: ignore
from test.support import modules_setup, modules_cleanup
from test.support import (
modules_setup,
modules_cleanup,
DirsOnSysPath,
CleanImport,
)


try:
from test.support import os_helper # type: ignore
except ImportError:
# Python 3.9 compat
class os_helper: # type:ignore
from test.support import temp_dir


try:
Expand Down
50 changes: 50 additions & 0 deletions importlib_resources/tests/_path.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import pathlib
import functools


####
# from jaraco.path 3.4


def build(spec, prefix=pathlib.Path()):
"""
Build a set of files/directories, as described by the spec.

Each key represents a pathname, and the value represents
the content. Content may be a nested directory.

>>> spec = {
... 'README.txt': "A README file",
... "foo": {
... "__init__.py": "",
... "bar": {
... "__init__.py": "",
... },
... "baz.py": "# Some code",
... }
... }
>>> tmpdir = getfixture('tmpdir')
>>> build(spec, tmpdir)
"""
for name, contents in spec.items():
create(contents, pathlib.Path(prefix) / name)


@functools.singledispatch
def create(content, path):
path.mkdir(exist_ok=True)
build(content, prefix=path) # type: ignore


@create.register
def _(content: bytes, path):
path.write_bytes(content)


@create.register
def _(content: str, path):
path.write_text(content)


# end from jaraco.path
####
42 changes: 42 additions & 0 deletions importlib_resources/tests/test_files.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
import typing
import unittest
import warnings
import contextlib

import importlib_resources as resources
from importlib_resources.abc import Traversable
from . import data01
from . import util
from . import _path
from ._compat import os_helper, import_helper


@contextlib.contextmanager
def suppress_known_deprecation():
with warnings.catch_warnings(record=True) as ctx:
warnings.simplefilter('default', category=DeprecationWarning)
yield ctx


class FilesTests:
Expand All @@ -25,6 +36,14 @@ def test_read_text(self):
def test_traversable(self):
assert isinstance(resources.files(self.data), Traversable)

def test_old_parameter(self):
"""
Files used to take a 'package' parameter. Make sure anyone
passing by name is still supported.
"""
with suppress_known_deprecation():
resources.files(package=self.data)


class OpenDiskTests(FilesTests, unittest.TestCase):
def setUp(self):
Expand All @@ -42,5 +61,28 @@ def setUp(self):
self.data = namespacedata01


class ModulesFilesTests(unittest.TestCase):
def setUp(self):
self.fixtures = contextlib.ExitStack()
self.addCleanup(self.fixtures.close)
self.site_dir = self.fixtures.enter_context(os_helper.temp_dir())
self.fixtures.enter_context(import_helper.DirsOnSysPath(self.site_dir))
self.fixtures.enter_context(import_helper.CleanImport())

def test_module_resources(self):
"""
A module can have resources found adjacent to the module.
"""
spec = {
'mod.py': '',
'res.txt': 'resources are the best',
}
_path.build(spec, self.site_dir)
import mod

actual = resources.files(mod).joinpath('res.txt').read_text()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is surprising to me. I would expect, a pathlib.Path('...', 'mod.py') to be returned, requiring resources.files(mod).parent.joinpath('res.txt'). res.txt is not a resource of mod, but rather of the parent module, which might not even exist if mod is top level.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing your surprise. Surely this would need some improved documentation to illustrate how files() now accepts an Anchor (instead of Package), where the anchor indicates:

  • the container (importer) from where resources can be discovered,
  • if a module is indicated, that's equivalent to indicating the container of that module,
  • if a top-level module is indicated, that will anchor on resources reachable at the top level by that same importer.

This is the behavior indicated by pkg_resources:

Note that if a module name is used, then the resource name is relative to the package immediately containing the named module.

However, I've also observed that the pkg_resources implementation also allows for loading resources against unpackaged modules. And while that behavior may have been unintentional, it works because it's a natural consequence of relying on the importer/loader to resolve the resources from the same location as the indicated package/module. That is, because the importlib mechanism supports loading modules from outside packages, so too should resources be loadable from that location.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

res.txt is not a resource of mod, but rather of the parent module, which might not even exist if mod is top level.

A parent module may not exist, but a loader for that module does exist and can load resources adjacent to that module:

 importlib_resources feature/203-non-package-modules $ touch setup.py
 importlib_resources feature/203-non-package-modules $ .tox/python/bin/python -iq
>>> import importlib_resources as res
>>> files = res.files('setup')
>>> files
PosixPath('/Users/jaraco/code/public/importlib_resources')
>>> files.joinpath('pyproject.toml').read_text()[:10]
'[build-sys'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 1f3c226, I've updated the documentation to honor an Anchor type which need not be a package.

assert actual == spec['res.txt']


if __name__ == '__main__':
unittest.main()
11 changes: 0 additions & 11 deletions importlib_resources/tests/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,17 +102,6 @@ def test_importing_module_as_side_effect(self):
del sys.modules[data01.__name__]
self.execute(data01.__name__, 'utf-8.file')

def test_non_package_by_name(self):
# The anchor package cannot be a module.
with self.assertRaises(TypeError):
self.execute(__name__, 'utf-8.file')

def test_non_package_by_package(self):
# The anchor package cannot be a module.
with self.assertRaises(TypeError):
module = sys.modules['importlib_resources.tests.util']
self.execute(module, 'utf-8.file')

def test_missing_path(self):
# Attempting to open or read or request the path for a
# non-existent path should succeed if open_resource
Expand Down