Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pyodide support and CI jobs for Zarr #1902

Closed
wants to merge 97 commits into from

Conversation

agriyakhetarpal
Copy link

@agriyakhetarpal agriyakhetarpal commented May 22, 2024

Important

Please view #1903 instead

dependabot bot and others added 30 commits December 4, 2023 09:13
…rs#1586)

Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.8.10 to 1.8.11.
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.8.10...v1.8.11)

---
updated-dependencies:
- dependency-name: pypa/gh-action-pypi-publish
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pers#1587)

Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 2.3.0 to 3.0.1.
- [Release notes](https://github.com/conda-incubator/setup-miniconda/releases)
- [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md)
- [Commits](conda-incubator/setup-miniconda@v2.3.0...v3.0.1)

---
updated-dependencies:
- dependency-name: conda-incubator/setup-miniconda
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Prevent runtime-overhead in doing this check multiple times
Even if this is only a test, list comprehensions are faster than
repeatedly call append().

Also use tuple instead of list when possible.

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [numcodecs](https://github.com/zarr-developers/numcodecs) from 0.11.0 to 0.12.1.
- [Release notes](https://github.com/zarr-developers/numcodecs/releases)
- [Changelog](https://github.com/zarr-developers/numcodecs/blob/main/docs/release.rst)
- [Commits](zarr-developers/numcodecs@v0.11.0...v0.12.1)

---
updated-dependencies:
- dependency-name: numcodecs
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
It's faster and probably more readable.

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pymongo](https://github.com/mongodb/mongo-python-driver) from 4.5.0 to 4.6.1.
- [Release notes](https://github.com/mongodb/mongo-python-driver/releases)
- [Changelog](https://github.com/mongodb/mongo-python-driver/blob/master/doc/changelog.rst)
- [Commits](mongodb/mongo-python-driver@4.5.0...4.6.1)

---
updated-dependencies:
- dependency-name: pymongo
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Starting with codespell 2.2.2, options can be specified in
`pyrpoject.toml` in addition to `setup.cfg`:
https://github.com/codespell-project/codespell#using-a-config-file

Specifying options in a config file instead of command line options in
`.pre-commit-config.yaml` ensures codespell uses the same options when
run as pre-commit hook or from the command line in the repository root
directory.
* Bump fsspec from 2023.10.0 to 2023.12.1

Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.10.0 to 2023.12.1.
- [Commits](fsspec/filesystem_spec@2023.10.0...2023.12.1)

---
updated-dependencies:
- dependency-name: fsspec
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update s3fs as well

* Fix s3fs

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* Add type hints to zarr.create

* Use protocol for MetaArray

* Use protocol for Synchronizer

* Fix Path typing

* Add release note

* Fix dim separator typing

* Ignore ... in coverage reporting

* Fix chunk typing

---------

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.1 to 5.0.0.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v4.7.1...v5.0.0)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: update pre-commit hooks

updates:
- https://github.com/charliermarsh/ruff-pre-commithttps://github.com/astral-sh/ruff-pre-commit
- [github.com/astral-sh/ruff-pre-commit: v0.0.224 → v0.1.8](astral-sh/ruff-pre-commit@v0.0.224...v0.1.8)
- [github.com/psf/black: 23.10.1 → 23.12.0](psf/black@23.10.1...23.12.0)
- [github.com/codespell-project/codespell: v2.2.5 → v2.2.6](codespell-project/codespell@v2.2.5...v2.2.6)
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0)
- [github.com/pre-commit/mirrors-mypy: v1.3.0 → v1.7.1](pre-commit/mirrors-mypy@v1.3.0...v1.7.1)

* Attempt to fix ruff

* Use isinstance

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Josh Moore <josh@openmicroscopy.org>
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.8 → v0.1.9](astral-sh/ruff-pre-commit@v0.1.8...v0.1.9)
- [github.com/psf/black: 23.12.0 → 23.12.1](psf/black@23.12.0...23.12.1)
- [github.com/pre-commit/mirrors-mypy: v1.7.1 → v1.8.0](pre-commit/mirrors-mypy@v1.7.1...v1.8.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump fsspec from 2023.12.1 to 2023.12.2

Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.12.1 to 2023.12.2.
- [Commits](fsspec/filesystem_spec@2023.12.1...2023.12.2)

---
updated-dependencies:
- dependency-name: fsspec
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update requirements_dev_optional.txt

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pytest-doctestplus](https://github.com/scientific-python/pytest-doctestplus) from 1.0.0 to 1.1.0.
- [Release notes](https://github.com/scientific-python/pytest-doctestplus/releases)
- [Changelog](https://github.com/scientific-python/pytest-doctestplus/blob/main/CHANGES.rst)
- [Commits](scientific-python/pytest-doctestplus@v1.0.0...v1.1.0)

---
updated-dependencies:
- dependency-name: pytest-doctestplus
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.1.11](astral-sh/ruff-pre-commit@v0.1.9...v0.1.11)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Drop 3.8 and add 3.12

* Try removing line_profiler

* Also bump the minimal numpy to 1.21

* Drop 3.12 again

* Revert "Try removing line_profiler"

This reverts commit 837854b.

* Update release.rst

---------

Co-authored-by: Joe Hamman <joe@earthmover.io>
Co-authored-by: jakirkham <jakirkham@gmail.com>
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.11 → v0.1.13](astral-sh/ruff-pre-commit@v0.1.11...v0.1.13)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump actions/download-artifact from 3 to 4

Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v3...v4)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Also bump upload-artifact

see https://github.com/actions/download-artifact?tab=readme-ov-file#breaking-changes

> Downloading artifacts that were created from action/upload-artifact@v3 and below are not supported.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Co-authored-by: Josh Moore <josh@openmicroscopy.org>
zarr-developers#1615)

* Update tutorial.rst to include section about accessing Zip Files on S3

Per discussion here, add information about about accessing zip files on s3:
zarr-developers#1613

* Update release.rst

* Implement d-v-b's suggestions

---------

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* doc(v3): add v3 roadmap and design document

* Update v3-roadmap-and-design.md

* updates after latest round of reviews

* Update v3-roadmap-and-design.md

Co-authored-by: Norman Rzepka <code@normanrz.com>

* Update v3-roadmap-and-design.md

Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>

---------

Co-authored-by: Norman Rzepka <code@normanrz.com>
Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.13 → v0.1.14](astral-sh/ruff-pre-commit@v0.1.13...v0.1.14)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This patch removes fasteners and disables zarr.sync which uses process and thread

Co-authored-by: Wei Ouyang <oeway007@gmail.com>
jhamman and others added 16 commits April 25, 2024 06:14
…pers#1824)

Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 3.0.3 to 3.0.4.
- [Release notes](https://github.com/conda-incubator/setup-miniconda/releases)
- [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md)
- [Commits](conda-incubator/setup-miniconda@v3.0.3...v3.0.4)

---
updated-dependencies:
- dependency-name: conda-incubator/setup-miniconda
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#1807)

* dep(docs): deprecate experimental v3 support in docs

* Apply suggestions from code review

Co-authored-by: Josh Moore <josh@openmicroscopy.org>
Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>

---------

Co-authored-by: Josh Moore <josh@openmicroscopy.org>
Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
Bumps [h5py](https://github.com/h5py/h5py) from 3.10.0 to 3.11.0.
- [Release notes](https://github.com/h5py/h5py/releases)
- [Changelog](https://github.com/h5py/h5py/blob/master/docs/release_guide.rst)
- [Commits](h5py/h5py@3.10.0...3.11.0)

---
updated-dependencies:
- dependency-name: h5py
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [redis](https://github.com/redis/redis-py) from 5.0.3 to 5.0.4.
- [Release notes](https://github.com/redis/redis-py/releases)
- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES)
- [Commits](redis/redis-py@v5.0.3...v5.0.4)

---
updated-dependencies:
- dependency-name: redis
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… remove in v3 (zarr-developers#1801)

* deprecate(stores): add deprecation warnings to DBMStore, LMDBStore, SQLiteStore, MongoDBStore, RedisStore, and ABSStore

* filter warnings in pytest config

* more deprecation warnings in docstrings

* add release note
* use np.inf instead of PINF/NINF

* update release notes
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.4.1 → v0.4.3](astral-sh/ruff-pre-commit@v0.4.1...v0.4.3)
- [github.com/psf/black: 24.4.0 → 24.4.2](psf/black@24.4.0...24.4.2)
- [github.com/pre-commit/mirrors-mypy: v1.9.0 → v1.10.0](pre-commit/mirrors-mypy@v1.9.0...v1.10.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Enable ruff/bugbear rules (B)

As suggested by Repo-Review.

* Fix ruff/bugbear issue (B007)

B007 Loop control variable `key` not used within loop body

https://docs.astral.sh/ruff/rules/unused-loop-control-variable/

* Fix ruff/bugbear issue (B015)

B015 Pointless comparison. Did you mean to assign a value?
     Otherwise, prepend `assert` or remove it.

https://docs.astral.sh/ruff/rules/useless-comparison/

* Fix ruff/bugbear issues (B028)

B028 No explicit `stacklevel` keyword argument found

https://docs.astral.sh/ruff/rules/no-explicit-stacklevel/

* Fix ruff/bugbear issues (B904)

B904 Within an `except` clause, raise exceptions with `raise ... from err` or
     `raise ... from None` to distinguish them from errors in exception handling

https://docs.astral.sh/ruff/rules/raise-without-from-inside-except/

* Document changes in docs/release.rst

* Disable ruff/bugbear rule (B017)

B017 `pytest.raises(Exception)` should be considered evil

https://docs.astral.sh/ruff/rules/assert-raises-exception/

---------

Co-authored-by: Joe Hamman <joe@earthmover.io>
* doc: cleanup release notes for 2.18.0
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.4.3 → v0.4.4](astral-sh/ruff-pre-commit@v0.4.3...v0.4.4)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Copy link
Author

@agriyakhetarpal agriyakhetarpal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

Comment on lines 32 to 40
try:
from numcodecs.tests.common import greetings
except ModuleNotFoundError:
greetings = [
'¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', 'Hei maailma!',
'Xin chào thế giới', 'Njatjeta Botë!', 'Γεια σου κόσμε!',
'こんにちは世界', '世界,你好!', 'Helló, világ!', 'Zdravo svete!',
'เฮลโลเวิลด์'
]
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

@@ -1020,6 +1020,7 @@ def create_store(self, normalize_keys=False, dimension_separator=".", **kwargs):
)
return store

@pytest.mark.xfail(reason="Skip under Pyodide for now, don't know how to handle umask")
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

pyproject.toml Outdated
Comment on lines 19 to 20
'numcodecs[msgpack]>=0.10.0; sys_platform == "emscripten"', # TODO: can be removed
'msgpack; sys_platform == "emscripten"',
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

@jhamman
Copy link
Member

jhamman commented May 23, 2024

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

@agriyakhetarpal
Copy link
Author

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

@agriyakhetarpal agriyakhetarpal changed the base branch from main to v3 May 23, 2024 17:06
@agriyakhetarpal
Copy link
Author

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

@agriyakhetarpal agriyakhetarpal force-pushed the emscripten branch 2 times, most recently from 89aa103 to bf89533 Compare May 23, 2024 17:53
@agriyakhetarpal
Copy link
Author

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

@agriyakhetarpal agriyakhetarpal deleted the emscripten branch May 23, 2024 17:53
@agriyakhetarpal
Copy link
Author

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.