-
Notifications
You must be signed in to change notification settings - Fork 295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zstd decompression fails due to unknown frame content size #625
Comments
This Zstandard compression works: The problem is that A chunk with a correctly encode content size in the header is located at the following URL: numcodecs.js does not handle the return value of Suggested actions for remediation:
|
Thanks for the investigation!
Agreed in the typical case when the size is known, but with zarr v3 chained codecs, or future support for variable-length strings, that isn't always possible.
Agreed.
Agreed when possible. |
manzt/numcodecs.js#47 was merged earlier today and released as v0.3.2. This includes Zstd stream decompression capability that does not require a known frame content size in the Zstd header. |
@jbms I helped to address point 2 above by implementing streaming decompression in numcodecs.js 0.3.2. Bumping the version as in #639 would help alleviate the immediate problem of neuroglancer not being able to read some datasets created by tensorstore. Let me know if you think more needs to be done there. Next, I was going to try to dig into point 3 to figure out why the pledged size is not making it into the frame header when tensorstore uses Zstandard. |
Neuroglancer appears to be having difficulty decompressing a Zstd compressed dataset:
https://neuroglancer-demo.appspot.com/#!https://s3proxy.janelia.org/shroff-public/shroff_20220523_Janelia_embryo2/20220523i_Janelia_embryo2_OTO_6x6x6nm_2xbin.zstd.zarr/s6/ng.json
The Google Chrome developer tools console reports the following error:
Raw or Gzip compressed Zarr v2 datasets seem to load just fine:
Raw compressed Zarr v2 dataset:
https://neuroglancer-demo.appspot.com/#!https://s3proxy.janelia.org/shroff-public/shroff_20220523_Janelia_embryo2/20220523i_Janelia_embryo2_OTO_6x6x6nm_2xbin.raw.zarr/s6/ng.json
Gzip compressed Zarr v2 dataset:
https://neuroglancer-demo.appspot.com/#!https://s3proxy.janelia.org/shroff-public/shroff_20220523_Janelia_embryo2/20220523i_Janelia_embryo2_OTO_6x6x6nm_2xbin.gzip.zarr/s6/ng.json
The value
4294967226
corresponds to the 32-bit value0xFFFFFFBA
. The 64-bit value0xFFFFFFFFFFFFFFBA
corresponds to the Zstd error code for "Destination buffer is too small".The dataset was encoded by Tensorstore 0.1.62 via conda-forge package with build number
py312h7e2185d_0
and zstd version 1.5.6ha6fb4c9_0
The text was updated successfully, but these errors were encountered: