Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chesapeake datasets don't extract properly #59

Closed
calebrob6 opened this issue Aug 2, 2021 · 3 comments · Fixed by #282
Closed

Chesapeake datasets don't extract properly #59

calebrob6 opened this issue Aug 2, 2021 · 3 comments · Fixed by #282
Assignees
Labels
datasets Geospatial or benchmark datasets
Milestone

Comments

@calebrob6
Copy link
Member

The ChesapeakeMD dataset fails when attempting to extract the downloaded zip file as zipfile.ZipFile doesn't support the deflate64 compression type that _MD_STATEWIDE.zip uses.

@calebrob6
Copy link
Member Author

I tested the rest of them, the following don't work for the same reason:

  • ChesapeakeNY
  • ChesapeakeVA

I also directly committed a fix for the Chesapeake7 dataset to main.

@adamjstewart adamjstewart added the datasets Geospatial or benchmark datasets label Aug 2, 2021
@adamjstewart
Copy link
Collaborator

I think zipfile.ZipFile does support deflate64, you just have to explicitly pass compression=zipfile.ZIP_DEFLATED. See https://docs.python.org/3/library/zipfile.html#zipfile.ZIP_DEFLATED.

I was hoping this would be done automatically. Torchvision had some logic to inject this but it wasn't clear to me how it worked. Will investigate more.

@adamjstewart
Copy link
Collaborator

So it turns out deflate and deflate64 are very different. The latter is a proprietary compression scheme that is not supported by zipfile. We could use https://pypi.org/project/zipfile-deflate64/, or call unzip or 7z from the command line.

@adamjstewart adamjstewart added this to the 0.1.1 milestone Nov 20, 2021
@adamjstewart adamjstewart changed the title Download doesn't work with Chesapeake MD dataset Chesapeake datasets don't extract properly Dec 14, 2021
@adamjstewart adamjstewart self-assigned this Dec 14, 2021
@adamjstewart adamjstewart modified the milestones: 0.1.1, 0.1.2 Dec 19, 2021
@adamjstewart adamjstewart modified the milestones: 0.1.2, 0.2.1 Jan 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datasets Geospatial or benchmark datasets
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants