Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zarr support with tensorstore backend #83

Merged
merged 6 commits into from
May 30, 2024
Merged

Conversation

sameeul
Copy link
Member

@sameeul sameeul commented May 28, 2024

No description provided.

return str(root_path.resolve()), axes_list
elif isinstance(root, zarr.hierarchy.Group):
# the top level is a group, check if this has any arrays
num_arrays = len(sorted(root.array_keys()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this the len of the sorted function instead of just taking the length of root.array_keys()?

Copy link
Member Author

@sameeul sameeul May 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

array_keys() is actually a generator function. Calling sorted on it actually generate the list and then we can get the length of all the elements produced by the generator.
Since the list contains small number of elements (~20 max), I don't expect this to be a huge overhead. I wish there is any other trick to do that.

root = zarr.open(str(root_path.resolve()), mode="r")
except zarr.errors.PathNotFoundError:
# a workaround for pre-compute slide output directory structure
root_path = self.frontend._file_path / "data.zarr"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this log a warning or is it expected behavior?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is expected behavior. Pyramid zarr outputs from precompute slide plugin puts everything inside this data.zarr and then puts this directory in a top level directory. This is something how Viv wants the folder to be organized.

self.logger.debug("__init__(): Initializing _rdr (TSTiffReader)...")
self._rdr = TSTiffReader(str(self.frontend._file_path))
self.logger.debug("__init__(): Initializing _rdr (TSReader)...")
extension = "".join(self.frontend._file_path.suffixes)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to access frontend._file_path through a public method or is this the only way?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no getters for this object at the moment. Since everything is public in Python and this is a low-level library implementation code (no user API has direct access to frontend), I opted not to add another layer of indirection.

@sameeul sameeul merged commit 1b9f072 into PolusAI:master May 30, 2024
6 checks passed
@sameeul sameeul deleted the zarr_backend branch August 7, 2024 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants