Zarr v3 #206

normanrz · 2023-07-20T17:22:43Z

This PR proposes to adopt the version 3 of the Zarr format for OME-Zarr.

Main changes:

Only Zarr v3 is allowed for upcoming versions of OME-Zarr.
Zarr v3 stores its metadata in zarr.json files. This is the same for arrays and groups. Attributes are now stored in a attributes object within the zarr.json files.
The chunk_key_encoding of the Zarr arrays are respected instead of mandating the / dimension separator. Basically all Zarr features are available in OME-Zarr as they get approved through the ZEP process.
Add a ome namespace within the zarr.json attributes.
Move version fields from multiscale, plate, well etc. up into the ome object.
Move the registration of labels from the labels group into the multiscale group.
Rename the document to OME-Zarr specification

This PR is currently in a draft status and builds upon the #138 PR by @bogovicj.
Check this link for a more convenient review: https://github.com/ome/ngff/pull/206/files/b92f540dc95440f2d6b7012185b09c2b862aa744..HEAD

# Conflicts: # latest/index.bs

* define pixel center coordinate system * add input/output_axes * add transform inverse flag * add affine transform type

* flesh out array space * define array indexing * add more transformation types * start transformation details section and examples * update example

# Conflicts: # latest/index.bs

* use "input" and "output" rather than '*Space' and '*Axes'

* reorder details * clean up table * add rotation * details for sequence * describe inverses * wrap examples

* rephrase matrix storage

* change to "coordinates", removing "Field" * change to "displacements", removing "Field"

* add details for transformation types * (identity, inverseOf, bijection) * describe inputAxes and outputAxes * add new examples

…amples * some clean up

* add mapIndex, mapAXis * add examples * affine stored as flat array only

* sequence does not have by-dimension behavior

* flesh out some examples

* multiscales dataset-level * xarray like coordinates / byDimension

see ome#142

# Conflicts: # latest/index.bs

* move bijection examples to separate files * add axis types to xarrayLike example

…n types

* add an example

github-actions · 2023-07-20T17:22:55Z

Automated Review URLs

d-v-b · 2023-07-20T17:26:41Z

Add a ome namespace within the zarr.json attributes.

Move version fields from multiscale, plate, well etc. up into the ome object.

Thank you for this!

imagesc-bot · 2023-08-08T12:33:37Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/adopt-zarr-v3-in-ome-zarr/84786/1

joshmoore · 2023-10-04T13:22:47Z

After a quick read, @normanrz, here are my initial thoughts (where not simply a big 👍):

Add a ome namespace within the zarr.json attributes.
Move version fields from multiscale, plate, well etc. up into the ome object.

These two together make me wonder whether or not the key couldn't be something more sophisticated, like the version itself:

"https://ngff.openmicroscopy.org/0.5": {
   ...
}

It's somewhat non standard to use a more complicated key like that, but it's a value that we will have in memory anyway to load the schema files, etc.

Move the registration of labels from the labels group into the multiscale group.

The labels "object" was definitely poorly conceived originally and so improvements welcome, but I'll note there are a handful of open issues as well as integration with the upcoming SpatialData work that might be worth considering. If it becomes more of an issue, I'd suggest splitting this out of your proposal here (if possible).

Rename the document to OME-Zarr specification

I know this is a constant source of confusion/frustration, but I think we might want to reconsider this.

normanrz · 2023-10-04T13:42:57Z

Add a ome namespace within the zarr.json attributes.
Move version fields from multiscale, plate, well etc. up into the ome object.

These two together make me wonder whether or not the key couldn't be something more sophisticated, like the version itself:
"https://ngff.openmicroscopy.org/0.5": {
   ...
}
It's somewhat non standard to use a more complicated key like that, but it's a value that we will have in memory anyway to load the schema files, etc.

Fine with me. It is a bit more verbose, but we could get rid of the additional "version" attribute.

Move the registration of labels from the labels group into the multiscale group.

The labels "object" was definitely poorly conceived originally and so improvements welcome, but I'll note there are a handful of open issues as well as integration with the upcoming SpatialData work that might be worth considering. If it becomes more of an issue, I'd suggest splitting this out of your proposal here (if possible).

Also, fine with me to split it out. But I really think label images in OME-Zarr need to be reworked.

Rename the document to OME-Zarr specification

I know this is a constant source of confusion/frustration, but I think we might want to reconsider this.

That is certainly up for debate. But my preference would be to go with OME-Zarr as title of the document, because that is what it is about.

jbms · 2023-10-05T21:36:53Z

latest/index.bs

+### Array coordinate systems
+
+Every array has a default coordinate system whose parameters need not be explicitly defined.  Its name is the path to the array
+in the container, its axes have `"type":"array"`, are unitless, and have default "name"s. The ith axis has `"name":"dim_i"`


Shouldn't the default names be those specified by the zarr v3 dimension_names property?

I see this is kind of addressed in the paragraph above, but it still might be helpful to clarify here.

jbms · 2023-10-06T16:45:32Z

latest/index.bs

+    }
+}
+```
+</div>

 "bioformats2raw.layout" (transitional) {#bf2raw}


Maybe this section should be moved to the end since it seems like it may not be of interest to most ome-zarr and is not an ideal starting point for the specification given how specialized it is.

jbms · 2023-10-06T16:47:41Z

latest/index.bs

@@ -331,24 +338,1010 @@ Conforming readers:
 - MAY ignore other groups or arrays under the root of the hierarchy.


+"coordinateSystems" metadata {#coord-sys-md}


It would help to clarify exactly where this is specified within the zarr attribute structure. The example could include the top-level "ome" key to clarify. If this isn't intended as a top-level member of the "ome" object, then perhaps it should be described in a later section, after describing its "parent" object, in order to make the specification clearer.

I think this could be addressed in #138, which this PR is based on.

normanrz · 2024-02-15T13:56:44Z

Closing this in favor of #227. Discussion will continue there.

bogovicj and others added 30 commits February 1, 2022 13:50

start of named space and transform metadata

f0da62a

fwd and inv coordinate transforms

4331deb

Merge remote-tracking branch 'upstream/main' into coord-transforms

8051e71

# Conflicts: # latest/index.bs

more transform details

180568b

* define pixel center coordinate system * add input/output_axes * add transform inverse flag * add affine transform type

start page on details and recommendations

4f48364

transformation progress

98a40de

* flesh out array space * define array indexing * add more transformation types * start transformation details section and examples * update example

change fields to camelCase

dd71953

add details for coordinate and displacement fields

1b26a8e

add details on dimension order

cd9c431

clean up array indexing section

e210027

add more details for scale, translation, displacement

d873541

merge and cleanup

63ed4ce

Merge remote-tracking branch 'upstream/main' into coord-transforms

a459492

# Conflicts: # latest/index.bs

clarify input and output dimensionality for transforms

9058b4b

* use "input" and "output" rather than '*Space' and '*Axes'

update to transformations

38ac6ac

* reorder details * clean up table * add rotation * details for sequence * describe inverses * wrap examples

rotation transformation details

3a2aae1

* rephrase matrix storage

minor rephrase intro to coordinateTransformations section

ed12192

add details for bijection ct

3ec8543

* change to "coordinates", removing "Field" * change to "displacements", removing "Field"

big changes to transformations spec

30cade5

* add details for transformation types * (identity, inverseOf, bijection) * describe inputAxes and outputAxes * add new examples

Add "coordinateSystems" and "coordinateTransformations" fields for ex…

df470d7

…amples * some clean up

coordinate transformation updates:

99f3a4e

* add mapIndex, mapAXis * add examples * affine stored as flat array only

update multiscales_example

de73c68

rename dimensionWise byDimension

163c8f4

* sequence does not have by-dimension behavior

add more transformation examples

7b35c28

byDimension examples, rm input/outputAxes fields

7baeed7

reorder, simplify sections.

162a9e9

* flesh out some examples

clean up example formatting

a90662d

add array coordinate system example

32656e5

rm duplicate editor names, typo fix

0245ca9

fix sequence example json. add inverseOf to input/output field exception

a3ff0b5

bogovicj and others added 17 commits October 3, 2022 21:26

add two new coordinate transformation examples

cd01ab7

* multiscales dataset-level * xarray like coordinates / byDimension

describe where to store coordinateTransformations

027b55b

corrections for coordinates and displacements

5bfe442

add optional "longName" field for axes

aab7469

see ome#142

Merge remote-tracking branch 'upstream/main' into coord-transforms

85808cf

# Conflicts: # latest/index.bs

Move rotation example to example directory

74466c5

Move inverseOf example to examples directory

ff2bc86

multiscales example relative typo fix

a68523f

corrections and clarifications to coordinates and displacements

3fe7aa2

* move bijection examples to separate files * add axis types to xarrayLike example

transforms MUST have "input"

f976a87

cts: remove url parameters

72b4f9c

ct: fix and clarify nested json representation for affine and rotatio…

be7d849

…n types

motivate coordinate systems

da94207

* add an example

describe discrete axes, interpolation, etc

b92f540

initial draft for Zarr3-based OME-Zarr

6d3ba9b

better directory example

0e51292

explicitly allow all Zarr features

fc6db29

d-v-b mentioned this pull request Jul 24, 2023

Single-scale images #207

Open

normanrz mentioned this pull request Aug 8, 2023

Clarify version field #210

Open

jbms reviewed Oct 5, 2023

View reviewed changes

jbms reviewed Oct 6, 2023

View reviewed changes

normanrz mentioned this pull request Feb 15, 2024

RFC-2: Zarr v3 #227

Merged

normanrz closed this Feb 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zarr v3 #206

Zarr v3 #206

normanrz commented Jul 20, 2023 •

edited

Loading

github-actions bot commented Jul 20, 2023

d-v-b commented Jul 20, 2023

imagesc-bot commented Aug 8, 2023

joshmoore commented Oct 4, 2023

normanrz commented Oct 4, 2023

jbms Oct 5, 2023

jbms Oct 6, 2023

jbms Oct 6, 2023

jbms Oct 6, 2023

normanrz Oct 9, 2023

normanrz commented Feb 15, 2024

		@@ -331,24 +338,1010 @@ Conforming readers:
		- MAY ignore other groups or arrays under the root of the hierarchy.


		"coordinateSystems" metadata {#coord-sys-md}

Zarr v3 #206

Zarr v3 #206

Conversation

normanrz commented Jul 20, 2023 • edited Loading

github-actions bot commented Jul 20, 2023

Automated Review URLs

d-v-b commented Jul 20, 2023

imagesc-bot commented Aug 8, 2023

joshmoore commented Oct 4, 2023

normanrz commented Oct 4, 2023

jbms Oct 5, 2023

Choose a reason for hiding this comment

jbms Oct 6, 2023

Choose a reason for hiding this comment

jbms Oct 6, 2023

Choose a reason for hiding this comment

jbms Oct 6, 2023

Choose a reason for hiding this comment

normanrz Oct 9, 2023

Choose a reason for hiding this comment

normanrz commented Feb 15, 2024

normanrz commented Jul 20, 2023 •

edited

Loading