-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Entrypoints #300
Entrypoints #300
Conversation
A reasonable way to test here would be to include a minimal package and .egg into the repo, and add it to sys.path during a test, checking to see if the basic codec included is automatically retrievable. Is that overkill, should I just skip testing code that is only meant for external packages? It does work for the only codec I have (https://github.com/fsspec/kerchunk/blob/main/kerchunk/grib2.py#L162 with so-far unmerged commit fsspec/kerchunk@a704c90 ). (done) |
Few quick thoughts:
|
Sorry, @cgohlke , I thought you were already on this thread, but that was, of course, for the linked issue.
I would rather do it with imagecodecs than kerchunk, the latter being more experimental and (for grib) needing cfgrib in the deps. Once imagecodecs has entrypoints, we can test against them.
As things stand, entrypoints are being used as a backup, and anything registered using the existing path will be used first. If two entrypoints share the same name, the entrypoints module will choose one, indeed probably based on where they are in sys.path . In Intake, we have warnings and clobber=bool for registering plugins, and also warn if entrypoints clash - but I don't think it's actually ended up being useful to do that. At most, I would suggest adding logging to the process, so a user can figure out what happened. |
👍
👍 |
@joshmoore , added a couple of basic logger statements. Integration testing can wait on imagecodecs. |
The imagecodecs package is still experimental/unstable. For now I published a separate package on PyPI with numcodecs entry points at imagecodecs-numcodecs: import inspect
import pprint
from setuptools import setup
import imagecodecs
import imagecodecs.numcodecs
entry_points = [
f'{cls.codec_id} = imagecodecs.numcodecs:{name}'
for name, cls in inspect.getmembers(imagecodecs.numcodecs)
if hasattr(cls, 'codec_id') and name != 'Codec'
]
pprint.pprint(entry_points)
setup(
name='imagecodecs-numcodecs',
version=imagecodecs.__version__,
description='Numcodecs entry points for the imagecodecs package',
# ...
install_requires=['numcodecs', f'imagecodecs=={imagecodecs.__version__}'],
entry_points={'numcodecs.codecs': entry_points},
) Entry points:
Some codecs are read-only or may be non-functional depending on the binary distribution (raising delayed ImportError on use). The codec names are prefixed with |
Interesting idea! So we can just include "imagecodecs-numcodecs" in requirements, and get both the codecs themselves and the entrypoints for auto import. I'm happy with that. |
re:
There is no provision for this anywhere in numcodecs. I suppose that if you were writing something that depended on a specific version of a codec, such as an intake catalog in a repo of its own, you can specify version requirements there. |
ping to all |
Sorry, was off collecting other use cases. As long as there is nothing we need to consider about the more complex use cases like versions upfront, then I'm still for moving forward, though perhaps as a 0.10.0.a1. @cgohlke: is there anything you would need from your side? |
cc @thomcom ( as this may be of interest for hooking in GPU compressors zarr-developers/community#19 (comment) ) |
Considering that there are no pressing objections, and getting 0.10.0.a1 out for others to consume. |
Thanks Martin for the PR and everyone for the review! 😄 To one of the questions Martin raised in the OP about docs, filed issue ( #302 ) to discuss & track The other question that occurred to me is what happens if an extension registers using an existing codec ID. Do we do anything about that currently? No worries if not. Just wanted to check. We can move this out to an issue if more discussion is needed 🙂 |
Programmatic registration takes precedence, entrypoints lookup is the fallback. So this should cause no surprises. |
Ok so 3rd party modules can't override existing extensions (at least not without a user doing something explicit)? |
Correct: here, we check the registry first, which will include any entries from |
Great, thanks Martin! 😄 |
* Add entrypoints to setup.py see: - conda-forge/numcodecs-feedstock#86 - #300 * Check syspath and ignore error * Temporarily disable codec test on Windows * Try adding entry test to manifest * Add `entrypoints` to `requirements.txt` * Add `entrypoints` to `requirements_dev.txt` * Remove `entrypoints` from `requirements_test.txt` * Add `entrypoints` to `requirements_rtfd.txt` * Note `numcodecs` is not `zip_safe` * xfail the test completely Co-authored-by: jakirkham <jakirkham@gmail.com>
Allows for packages to list codecs in entrypoints, so that the package in question doesn't need to be imported in before the codec can be usable.
Notes:
numcodecs/__init__
could be avoided by this.Fixes #290
TODO:
tox -e py39
passes locallytox -e docs
passes locally