Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 52 additions & 65 deletions spec-0001/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ author:
- "Dan Schult <dschult@colgate.edu>"
discussion: https://discuss.scientific-python.org/t/spec-1-lazy-loading-for-submodules/25
endorsed-by:
shortcutDepth: 3
---

## Description
Expand Down Expand Up @@ -172,6 +173,57 @@ be immediately raised with:
linalg = lazy.load('scipy.linalg', error_on_import=True)
```

#### Type checkers

The lazy loading shown above has one drawback: static type checkers
(such as [mypy](http://mypy-lang.org) and
[pyright](https://github.com/microsoft/pyright)) will not be able to
infer the types of lazy-loaded modules and functions.
Therefore, `mypy` won't be able to detect potential errors, and
integrated development environments such as [VS
Code](https://code.visualstudio.com) won't provide code completion.

To work around this limitation, we provide an alternative way to define lazy
imports.
Instead of importing modules and functions in `__init__.py`
file with `lazy.attach`, you instead specify those imports in a `__init__.pyi`
file—called a "type stub".
Your `__init__.py` file then loads imports from the stub using `lazy.attach_stub`.

Here's an example of how to convert this `__init__.py`:

```python
# mypackage/__init__.py
import lazy_loader as lazy

__getattr__, __dir__, __all__ = lazy.attach(
__name__,
submod_attrs={
'edges': ['sobel', 'sobel_h', 'sobel_v']
}
)
```

Add a [type stub (`__init__.pyi`) file](https://mypy.readthedocs.io/en/stable/stubs.html) in the same directory as the `__init__.py`.
Type stubs are ignored at runtime, but used by static type checkers.

```python
# mypackage/__init__.pyi
from .edges import sobel, sobel_h, sobel_v
```

Replace `lazy.attach` in `mypackage/__init__.py` with a call to `attach_stub`:

```python
import lazy_loader as lazy

# this assumes there is a `.pyi` file adjacent to this module
__getattr__, __dir__, __all__ = lazy.attach_stub(__name__, __file__)
```

_Note that if you use a type stub, you will need to take additional action to add the `.pyi` file to your sdist and wheel distributions.
See [PEP 561](https://peps.python.org/pep-0561/) and the [mypy documentation](https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages) for more information._

## Implementation

Lazy loading is implemented at
Expand Down Expand Up @@ -207,71 +259,6 @@ In the mean time, we now have the necessary mechanisms to implement it ourselves

[^cannon]: Cannon B., personal communication, 7 January 2021.

### Caveats

While this pattern has benefits at runtime, it does have one specific drawback:

Static type checkers (such as [mypy](http://mypy-lang.org) and
[pyright](https://github.com/microsoft/pyright)) will not be able to infer the
types and locations dynamically-loaded modules and functions. This means that some
integrated development environments (e.g.
[VS Code](https://code.visualstudio.com)) will not be able to provide code
completion, parameter hints, documentation, or other features for any objects in your module. Nor will `mypy` be able to detect potential errors in your users' code.

You can direct static type checkers to the actual location of dynamically loaded objects in one of two ways:

1. Use an [`if typing.TYPE_CHECKING`](https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING)
clause .

```python
from typing import TYPE_CHECKING

import lazy_loader as lazy

if TYPE_CHECKING:
from .edges import sobel, sobel_h, sobel_v

__getattr__, __dir__, __all__ = lazy.attach(
__name__,
submod_attrs={
'edges': ['sobel', 'sobel_h', 'sobel_v']
}
)
```

`typing.TYPE_CHECKING` is a special variable that is set to `False` at runtime, and has negligible runtime impact.

2. Use a [type stub (`.pyi`) file](https://mypy.readthedocs.io/en/stable/stubs.html). Type stubs are ignored at runtime, but used
by static type checkers. This method entails adding a new file with a `.pyi` extension in the same directory as your actual `.py`. For example:

```python
# mypackage/__init__.py
import lazy_loader as lazy

__getattr__, __dir__, __all__ = lazy.attach(
__name__,
submod_attrs={
'edges': ['sobel', 'sobel_h', 'sobel_v']
}
)
```

```python
# mypackage/__init__.pyi
from .edges import sobel, sobel_h, sobel_v
```

_Note that if you use a type stub, you will need to take additional action to add the `.pyi` file to your sdist and wheel distributions. See [PEP 561](https://peps.python.org/pep-0561/) and the [mypy documentation](https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages) for more information._

3. If you would like to avoid the unfortunate duplication of declaring exports in _both_ a type stub and in the arguments to `lazy.attach`, the aforementioned [lazy_loader](https://github.com/scientific-python/lazy_loader) package offers a [`lazy_loader.attach_stub` function](https://github.com/scientific-python/lazy_loader#lazily-load-subpackages-and-functions-from-type-stubs) that can be used to infer your module exports directly from your stub file – which now becomes the single source of truth for your module's exports. It is used as follows:

```python
from lazy_loader import attach_stub

# this assumes there is a `.pyi` file adjacent to this module
__getattr__, __dir__, __all__ = attach_stub(__name__, __file__)
```

### Core Project Endorsement

<!--
Expand Down