Consider hosting dataset on Huggingface & source.coop

[robmarkcole]

I'm noting very slow download times for the dataset (my connection is fast):

satlas-dataset-v1-sentinel2-small.tar                  39%[++] 183.55G  3.87MB/s

I've experienced very rapid downloads from Huggingface and suggest it as an additional location to host and distribute the dataset

Additionally https://beta.source.coop/ would be a relevant portal

[favyen2]

I will try to add it here https://huggingface.co/allenai/satlas-pretrain but it may take some time due to large size of the dataset.

[robmarkcole]

Hi @favyen2 I see you got a couple of files up which is great. Can I request you prioritise the following data? Been attempting to download since start of week, still going

satlas_explorer_datasets_2023-07-24.tar                45%          ] 430.34G  3.34MB/s    eta 32h 8m

[robmarkcole]

For the explorer dataset it took most of the week to download the tar and most of the weekend to untar. On reviewing the labelled datasets:

• marine_infra: 28GB detection json + images
• solar_farm: 19GB segmentation mask + images
• tree_cover: 963GB segmentation mask + images
• wind_turbine: 40.02 GB detection json + images

The 3 small datasets could be uploaded as individual datasets - HF has 40GB limit (TBC) per zip/tar so these should be fine. This would be a much faster experience for people who only care about one of those

[favyen2]

The dataset is now available on Hugging Face. The hand-labeled datasets for individual tasks are updated regularly and we are still deciding how to release those on an ongoing basis.