-
-
Notifications
You must be signed in to change notification settings - Fork 288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Pyodide support and CI jobs for Zarr #1902
Conversation
…rs#1586) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.8.10 to 1.8.11. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.8.10...v1.8.11) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pers#1587) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 2.3.0 to 3.0.1. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](conda-incubator/setup-miniconda@v2.3.0...v3.0.1) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Prevent runtime-overhead in doing this check multiple times
Even if this is only a test, list comprehensions are faster than repeatedly call append(). Also use tuple instead of list when possible. Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [numcodecs](https://github.com/zarr-developers/numcodecs) from 0.11.0 to 0.12.1. - [Release notes](https://github.com/zarr-developers/numcodecs/releases) - [Changelog](https://github.com/zarr-developers/numcodecs/blob/main/docs/release.rst) - [Commits](zarr-developers/numcodecs@v0.11.0...v0.12.1) --- updated-dependencies: - dependency-name: numcodecs dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
…ers#1558) Co-authored-by: Joe Hamman <joe@earthmover.io>
It's faster and probably more readable. Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pymongo](https://github.com/mongodb/mongo-python-driver) from 4.5.0 to 4.6.1. - [Release notes](https://github.com/mongodb/mongo-python-driver/releases) - [Changelog](https://github.com/mongodb/mongo-python-driver/blob/master/doc/changelog.rst) - [Commits](mongodb/mongo-python-driver@4.5.0...4.6.1) --- updated-dependencies: - dependency-name: pymongo dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Starting with codespell 2.2.2, options can be specified in `pyrpoject.toml` in addition to `setup.cfg`: https://github.com/codespell-project/codespell#using-a-config-file Specifying options in a config file instead of command line options in `.pre-commit-config.yaml` ensures codespell uses the same options when run as pre-commit hook or from the command line in the repository root directory.
* Bump fsspec from 2023.10.0 to 2023.12.1 Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.10.0 to 2023.12.1. - [Commits](fsspec/filesystem_spec@2023.10.0...2023.12.1) --- updated-dependencies: - dependency-name: fsspec dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update s3fs as well * Fix s3fs --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* Add type hints to zarr.create * Use protocol for MetaArray * Use protocol for Synchronizer * Fix Path typing * Add release note * Fix dim separator typing * Ignore ... in coverage reporting * Fix chunk typing --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.1 to 5.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4.7.1...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: update pre-commit hooks updates: - https://github.com/charliermarsh/ruff-pre-commit → https://github.com/astral-sh/ruff-pre-commit - [github.com/astral-sh/ruff-pre-commit: v0.0.224 → v0.1.8](astral-sh/ruff-pre-commit@v0.0.224...v0.1.8) - [github.com/psf/black: 23.10.1 → 23.12.0](psf/black@23.10.1...23.12.0) - [github.com/codespell-project/codespell: v2.2.5 → v2.2.6](codespell-project/codespell@v2.2.5...v2.2.6) - [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0) - [github.com/pre-commit/mirrors-mypy: v1.3.0 → v1.7.1](pre-commit/mirrors-mypy@v1.3.0...v1.7.1) * Attempt to fix ruff * Use isinstance --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.8 → v0.1.9](astral-sh/ruff-pre-commit@v0.1.8...v0.1.9) - [github.com/psf/black: 23.12.0 → 23.12.1](psf/black@23.12.0...23.12.1) - [github.com/pre-commit/mirrors-mypy: v1.7.1 → v1.8.0](pre-commit/mirrors-mypy@v1.7.1...v1.8.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump fsspec from 2023.12.1 to 2023.12.2 Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.12.1 to 2023.12.2. - [Commits](fsspec/filesystem_spec@2023.12.1...2023.12.2) --- updated-dependencies: - dependency-name: fsspec dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements_dev_optional.txt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pytest-doctestplus](https://github.com/scientific-python/pytest-doctestplus) from 1.0.0 to 1.1.0. - [Release notes](https://github.com/scientific-python/pytest-doctestplus/releases) - [Changelog](https://github.com/scientific-python/pytest-doctestplus/blob/main/CHANGES.rst) - [Commits](scientific-python/pytest-doctestplus@v1.0.0...v1.1.0) --- updated-dependencies: - dependency-name: pytest-doctestplus dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.1.11](astral-sh/ruff-pre-commit@v0.1.9...v0.1.11) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Drop 3.8 and add 3.12 * Try removing line_profiler * Also bump the minimal numpy to 1.21 * Drop 3.12 again * Revert "Try removing line_profiler" This reverts commit 837854b. * Update release.rst --------- Co-authored-by: Joe Hamman <joe@earthmover.io> Co-authored-by: jakirkham <jakirkham@gmail.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.11 → v0.1.13](astral-sh/ruff-pre-commit@v0.1.11...v0.1.13) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump actions/download-artifact from 3 to 4 Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3 to 4. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Also bump upload-artifact see https://github.com/actions/download-artifact?tab=readme-ov-file#breaking-changes > Downloading artifacts that were created from action/upload-artifact@v3 and below are not supported. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
zarr-developers#1615) * Update tutorial.rst to include section about accessing Zip Files on S3 Per discussion here, add information about about accessing zip files on s3: zarr-developers#1613 * Update release.rst * Implement d-v-b's suggestions --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* doc(v3): add v3 roadmap and design document * Update v3-roadmap-and-design.md * updates after latest round of reviews * Update v3-roadmap-and-design.md Co-authored-by: Norman Rzepka <code@normanrz.com> * Update v3-roadmap-and-design.md Co-authored-by: Sanket Verma <svsanketverma5@gmail.com> --------- Co-authored-by: Norman Rzepka <code@normanrz.com> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.13 → v0.1.14](astral-sh/ruff-pre-commit@v0.1.13...v0.1.14) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This patch removes fasteners and disables zarr.sync which uses process and thread Co-authored-by: Wei Ouyang <oeway007@gmail.com>
…pers#1824) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 3.0.3 to 3.0.4. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](conda-incubator/setup-miniconda@v3.0.3...v3.0.4) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#1807) * dep(docs): deprecate experimental v3 support in docs * Apply suggestions from code review Co-authored-by: Josh Moore <josh@openmicroscopy.org> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com> --------- Co-authored-by: Josh Moore <josh@openmicroscopy.org> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
Bumps [h5py](https://github.com/h5py/h5py) from 3.10.0 to 3.11.0. - [Release notes](https://github.com/h5py/h5py/releases) - [Changelog](https://github.com/h5py/h5py/blob/master/docs/release_guide.rst) - [Commits](h5py/h5py@3.10.0...3.11.0) --- updated-dependencies: - dependency-name: h5py dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [redis](https://github.com/redis/redis-py) from 5.0.3 to 5.0.4. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](redis/redis-py@v5.0.3...v5.0.4) --- updated-dependencies: - dependency-name: redis dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… remove in v3 (zarr-developers#1801) * deprecate(stores): add deprecation warnings to DBMStore, LMDBStore, SQLiteStore, MongoDBStore, RedisStore, and ABSStore * filter warnings in pytest config * more deprecation warnings in docstrings * add release note
* use np.inf instead of PINF/NINF * update release notes
updates: - [github.com/astral-sh/ruff-pre-commit: v0.4.1 → v0.4.3](astral-sh/ruff-pre-commit@v0.4.1...v0.4.3) - [github.com/psf/black: 24.4.0 → 24.4.2](psf/black@24.4.0...24.4.2) - [github.com/pre-commit/mirrors-mypy: v1.9.0 → v1.10.0](pre-commit/mirrors-mypy@v1.9.0...v1.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Enable ruff/bugbear rules (B) As suggested by Repo-Review. * Fix ruff/bugbear issue (B007) B007 Loop control variable `key` not used within loop body https://docs.astral.sh/ruff/rules/unused-loop-control-variable/ * Fix ruff/bugbear issue (B015) B015 Pointless comparison. Did you mean to assign a value? Otherwise, prepend `assert` or remove it. https://docs.astral.sh/ruff/rules/useless-comparison/ * Fix ruff/bugbear issues (B028) B028 No explicit `stacklevel` keyword argument found https://docs.astral.sh/ruff/rules/no-explicit-stacklevel/ * Fix ruff/bugbear issues (B904) B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling https://docs.astral.sh/ruff/rules/raise-without-from-inside-except/ * Document changes in docs/release.rst * Disable ruff/bugbear rule (B017) B017 `pytest.raises(Exception)` should be considered evil https://docs.astral.sh/ruff/rules/assert-raises-exception/ --------- Co-authored-by: Joe Hamman <joe@earthmover.io>
* doc: cleanup release notes for 2.18.0
updates: - [github.com/astral-sh/ruff-pre-commit: v0.4.3 → v0.4.4](astral-sh/ruff-pre-commit@v0.4.3...v0.4.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif()
decorators.
Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.
zarr/tests/test_core.py
Outdated
try: | ||
from numcodecs.tests.common import greetings | ||
except ModuleNotFoundError: | ||
greetings = [ | ||
'¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', 'Hei maailma!', | ||
'Xin chào thế giới', 'Njatjeta Botë!', 'Γεια σου κόσμε!', | ||
'こんにちは世界', '世界,你好!', 'Helló, világ!', 'Zdravo svete!', | ||
'เฮลโลเวิลด์' | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am unaware of the reason why numcodecs.tests.common
cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.
zarr/tests/test_storage.py
Outdated
@@ -1020,6 +1020,7 @@ def create_store(self, normalize_keys=False, dimension_separator=".", **kwargs): | |||
) | |||
return store | |||
|
|||
@pytest.mark.xfail(reason="Skip under Pyodide for now, don't know how to handle umask") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.
pyproject.toml
Outdated
'numcodecs[msgpack]>=0.10.0; sys_platform == "emscripten"', # TODO: can be removed | ||
'msgpack; sys_platform == "emscripten"', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both of these lines currently do not work for some reason – I am unable to import MsgPack
from numcodecs
even after I added them, which is why I am explicitly installing msgpack
inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.
68b6f62
to
80fb6c2
Compare
@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the |
Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment. |
Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR. |
89aa103
to
bf89533
Compare
I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one. |
I have a suggestion which I hope is not unsolicited: please make the Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks! |
Important
Please view #1903 instead