Add South Africa Crop Type Competition Dataset #1840

GeorgeHuber · 2024-02-01T03:06:35Z

This PR adds the South Africa Crop Type Competition Dataset to TorchGeo.

Similar to #1459, this dataset does not implement automatic download functions due to issues migrating data from radiantmlhub source cooperative until issue #1830 is resolved.

Example field and label in early July:

Implement the dataset extending either GeoDataset or NonGeoDataset
Add the dataset definition to torchgeo/datasets/init.py
Add a data.py script to tests/data// that generates test data with the same directory structure/file naming conventions as the new dataset
Add appropriate tests with 100% test coverage to tests/datasets/
Add the dataset to docs/api/datasets.rst
Add the dataset metadata to either docs/api/geo_datasets.csv or docs/api/non_geo_datasets.csv

yichiac

Please refer to #1459 for removing the download and checksum functions until #1830 is resolved.

torchgeo/datasets/south_africa_crop_type.py

* Documentation, satellite-specific transform and weights for additional Satlas single-image rgb&multispectral Swin-v2 models. Tests pass. * Address 3 of comments * Address comments, fix readmydocs and isort, mypy still unhappy * update * Add bands to meta dicts * Add comment about Satlas S2 RGB using TCI product * linting --------- Co-authored-by: Piper Wolters <piperw@allenai.org> Co-authored-by: Piper Wolters <piperw@prior-cirrascale-79.reviz.ai2.in>

tests/data/south_africa_crop_type/data.py

torchgeo/datasets/south_africa_crop_type.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

…eHuber/torchgeo into add-south-africa-crop-type

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

…eHuber/torchgeo into add-south-africa-crop-type

adamjstewart

I don't see any remaining major issues. Just wondering if __getitem__ can be further simplified. Also need comments in __getitem__ to explain all the non-standard stuff.

torchgeo/datasets/south_africa_crop_type.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

GeorgeHuber · 2024-03-05T16:36:47Z

I don't see any remaining major issues. Just wondering if __getitem__ can be further simplified. Also need comments in __getitem__ to explain all the non-standard stuff.

I added some comments to explain the extra stuff, let me know if you think it needs more detail. The multiple images to one label thing is really where most of the complication comes from along with the need to make sure the labels and fields are in the same order since they won't be sorted alphabetically. I thought this would be the most efficient solution as we loop over the hit filepaths once just as we would standardly but I'm open to other ideas.

adamjstewart

Probably worth iterating over the real data just to make sure there are no bugs once you move away from the synthetic data. Just as a quick personal test, doesn't need to be in CI.

torchgeo/datasets/south_africa_crop_type.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

adamjstewart · 2024-03-15T11:16:09Z

torchgeo/datasets/south_africa_crop_type.py

+        Returns:
+            data and labels at that index
+        """
+        assert isinstance(self.paths, str)


It's probably possible to relax this constraint by using filepaths to compute the correct directory for the mask. But let's do that another day, I want to get this merged.

initial commit

e8c1b8b

github-actions bot added the datasets Geospatial or benchmark datasets label Feb 1, 2024

GeorgeHuber marked this pull request as draft February 1, 2024 03:06

georgehuber added 4 commits February 1, 2024 23:11

Merge branch 'main' into add-south-africa-crop-type

abdb677

add init description

0df3cec

correct scanning of files in init

eb33e95

added random data generation

7447d0d

github-actions bot added the testing Continuous integration testing label Feb 9, 2024

georgehuber added 2 commits February 9, 2024 01:56

remove test directory

4973565

Update metadata

3590b8e

github-actions bot added the documentation Improvements or additions to documentation label Feb 9, 2024

fix formatting

b123a57

adamjstewart added this to the 0.6.0 milestone Feb 11, 2024

yichiac suggested changes Feb 13, 2024

View reviewed changes

yichiac mentioned this pull request Feb 15, 2024

Radiant MLHub Crop Type Datasets #512

Closed

4 tasks

georgehuber added 2 commits February 15, 2024 17:37

add s1 data and update regex

e8ed0ac

fix dataset and add tests

eaef967

GeorgeHuber marked this pull request as ready for review February 16, 2024 05:36

GeorgeHuber and others added 10 commits February 15, 2024 23:44

Merge branch 'main' into add-south-africa-crop-type

033ccdc

formatting

dd1a49a

mypy

d2108cc

add timeseries test and take images from july

49872f8

Merge branch 'main' into add-south-africa-crop-type

c21ae9a

fix bug in merging files across dates

27cf49f

mypy

7fa5145

Merge branch 'main' into add-south-africa-crop-type

990c013

Bump lightly from 1.4.25 to 1.5.0 in /requirements (microsoft#1894)

f281cce

adamjstewart requested changes Feb 25, 2024

View reviewed changes

Update tests/data/south_africa_crop_type/data.py

e19fa65

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

GeorgeHuber and others added 11 commits February 28, 2024 10:05

Update tests/data/south_africa_crop_type/data.py

a38f9f4

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update torchgeo/datasets/south_africa_crop_type.py

d8d9bb2

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update torchgeo/datasets/south_africa_crop_type.py

1553389

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update torchgeo/datasets/south_africa_crop_type.py

9ae8924

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Merge branch 'add-south-africa-crop-type' of https://github.com/Georg…

afea568

…eHuber/torchgeo into add-south-africa-crop-type

Update torchgeo/datasets/south_africa_crop_type.py

3c940ee

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

refactor gettitem and bug fixes

cb3f03c

Merge branch 'add-south-africa-crop-type' of https://github.com/Georg…

6525a74

…eHuber/torchgeo into add-south-africa-crop-type

typo

c59b0b4

data.py style

a560598

Merge branch 'main' into add-south-africa-crop-type

dd17b57

GeorgeHuber requested a review from adamjstewart March 1, 2024 20:31

adamjstewart reviewed Mar 2, 2024

View reviewed changes

torchgeo/datasets/south_africa_crop_type.py Outdated Show resolved Hide resolved

torchgeo/datasets/south_africa_crop_type.py Outdated Show resolved Hide resolved

GeorgeHuber and others added 4 commits March 5, 2024 10:04

Update torchgeo/datasets/south_africa_crop_type.py

e03b663

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Update torchgeo/datasets/south_africa_crop_type.py

a568af3

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

Merge branch 'microsoft:main' into add-south-africa-crop-type

dec3948

added comments for nonstandard functionality

96ce979

GeorgeHuber requested review from adamjstewart March 5, 2024 17:51

Merge branch 'main' into add-south-africa-crop-type

78d9122

adamjstewart reviewed Mar 8, 2024

View reviewed changes

torchgeo/datasets/south_africa_crop_type.py Outdated Show resolved Hide resolved

torchgeo/datasets/south_africa_crop_type.py Outdated Show resolved Hide resolved

torchgeo/datasets/south_africa_crop_type.py Outdated Show resolved Hide resolved

GeorgeHuber and others added 3 commits March 8, 2024 14:32

Update torchgeo/datasets/south_africa_crop_type.py

fcce45b

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

small change

f33e5c5

add verbose documentation

c999e89

adamjstewart approved these changes Mar 15, 2024

View reviewed changes

adamjstewart merged commit 1f6e974 into microsoft:main Mar 15, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add South Africa Crop Type Competition Dataset #1840

Add South Africa Crop Type Competition Dataset #1840

GeorgeHuber commented Feb 1, 2024 •

edited

Loading

yichiac left a comment

adamjstewart left a comment

GeorgeHuber commented Mar 5, 2024

adamjstewart left a comment •

edited

Loading

adamjstewart Mar 15, 2024

Add South Africa Crop Type Competition Dataset #1840

Add South Africa Crop Type Competition Dataset #1840

Conversation

GeorgeHuber commented Feb 1, 2024 • edited Loading

yichiac left a comment

Choose a reason for hiding this comment

adamjstewart left a comment

Choose a reason for hiding this comment

GeorgeHuber commented Mar 5, 2024

adamjstewart left a comment • edited Loading

Choose a reason for hiding this comment

adamjstewart Mar 15, 2024

Choose a reason for hiding this comment

GeorgeHuber commented Feb 1, 2024 •

edited

Loading

adamjstewart left a comment •

edited

Loading