XL Datasets: Minimal Zarr-only dataset implementation

[cwognum]

Changelogs

• Introduce a new DatasetV2 class in a new polaris.experimental module.
• Explicitly rename the old dataset to DatasetV1, but create the Dataset alias for it.
• Extract a BaseDataset class that encompasses all functionality that is shared between V1 and V2.
• Restricted use of __getitem__ to only allow indexing a specific value or a row.
• Add a new Zarr manifest for upload integrity.
  • See Adding new Zarr manifest generation to DatasetV2 class #185

---

Checklist:

• Was this PR discussed in an issue? It is recommended to first discuss a new feature into a GitHub issue before opening a PR.
• Add tests to cover the fixed bug(s) or the newly introduced feature(s) (if appropriate).
• Update the API documentation if a new function is added, or an existing one is deleted.
• Write concise and explanatory changelogs above.
• If possible, assign one of the following labels to the PR: feature, fix, chore, documentation or test (or ask a maintainer to do it for you).

---

Closes #132

The implementation has been relatively straight-forward given the PoC. There are two points that required some further thinking.

1. Naming and code organization: I wasn't sure how to name the new class, where to introduce it and whether to rename the old class. For clarity, I decided to rename the old class, but we do maintain the Dataset alias for it to not break backwards compatibility. I also introduced a new experimental module to house the V2 implementation.
2. Indexing: With Add a converter from PDB to Zarr to the DatasetFactory #171, we made it possible for the Zarr root to contain groups. This raises the question how to index those. For these cases, I decided to explicitly introduce a index array that needs to be part of that group.

On V1 and V2 compatibility

You can think of a Pandas DataFrame as a set of named NumPy arrays. For storage, the conversion from Pandas to a Zarr archive is therefore relatively straight-forward. Note that this is limited to the storage - It's not straight-forward to implement the Pandas API on top of a Zarr archive, but this was decided to be out of scope.

An example of this conversion was implemented in this test case. A toy example would look like:

    df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})

    root = zarr.open("path/to/arhive.zarr", "w")
    for col in df.columns:
        root.array(col, data=df[col].values)

[Andrewq11]

Thanks @cwognum! Left some comments but only a couple may be blocking.

[jstlaurent]

I have a few comments and suggestions. Overall, good, solid work. 😄

[cwognum]

@jstlaurent @Andrewq11 Thank you for the review!

This ended up being a good opportunity to also review the Dataset code that was mostly written over a year ago. One of the things I realized, is that the __getitem__() method was partially broken and implementing it properly would've taken a good amount of time and complicate the code. This would've become even more tricky for the DatasetV2 class. I decided to therefore change this part of the API and simplify the type of indexing we allow.

Unfortunately, this does imply that some of the code examples we've shown in presentations are no longer accurate. See also https://github.com/polaris-hub/polaris-hub/pull/471. I am personally okay with this, since the Dataset API generally does not get a lot of attention compared to the Benchmark API. What do you think?

[Andrewq11]

Looks good on my side. Thanks for all the work here!

More progress toward XL datasets 🚀

[jstlaurent]

Looks good. Still had a couple of things to say. 😉