Add IDTReeS dataset

[isaaccorley]

This PR adds the IDTReeS dataset for tree crown detection and classification. Link to paper

• Adds torchgeo.datasets.IDTReeS

Some notes on the dataset:

• The train split contains both bbox and labels per bbox. The test split task1 contains only images and task2 contains bounding boxes but no labels.
• Many of the RGB images appear to have this warped effect. I opened several in QGIS and verified this is in the raw image.
• Bounding boxes are very noisy and sometimes contain open ground and other times can contain only a small part of the tree. Maybe it's just not clear to me what part of the image is a "tree crown".
• The test split task2 images seem to only be partially labeled (missing many boxes) and some images have no boxes entirely. This would cause issues when benchmarking since your false positives may actually be true positives. I'm guessing the assumption when labeling was that you need to extract the ground truth boxes and build a classifier on top of the subset images. Who knows. See below:

TODO:

• tests
• upsample CHM and HSI to same res as RGB (200x200)
• figure out how to deal with test set being formatted dramatically different than train set will refactor in future PR if needed
• add plot method

Usage (for clarity):

from torchgeo.datasets import IDTReeS

dataset = IDTReeS(root, split="train", task="task1")
sample = ds[0]

for k, v in sample.items():
   print(k, v.shape, v.dtype)

# image torch.Size([3, 200, 200]) torch.uint8     <- RGB image
# hsi torch.Size([369, 200, 200]) torch.int16     <- Hyperspectral image
# chm torch.Size([1, 200, 200]) torch.float32     <- Canopy Height Model (CHM)
# las torch.Size([3, 5739]) torch.float64         <- Point Cloud (x, y, z)
# boxes torch.Size([12, 4]) torch.int64           <- Object Bounding Boxes (xyxy)
# label torch.Size([12]) torch.int64              <- object labels

Example images:

# without predictions
dataset.plot(sample, hsi_indices=(0, 1, 2)) # use 3 hyperspectral indices to make a false color image

# with predictions
sample["prediction_boxes"] = sample["boxes"]
sample["prediction_label"] = sample["label"]
dataset.plot(sample, hsi_indices=(0, 1, 2))

Example point cloud:

# with colormap
dataset.plot_las(index=3, colormap="BrBG")

# no colormap and no colors in .las file
dataset.plot_las(index=3, colormap=None)

# no colormap and colors exist in .las file
dataset.plot_las(index=70, colormap=None)

plot_las_example.mp4

plot_las_example_no_colors.mp4

plot_las_example_with_colors.mp4

[adamjstewart]

I'm giving a talk to a group of biologists who use drone imagery for spectral biology on Monday. If this PR is close to being ready it would be great to get this merged.

[isaaccorley]

Working on it tonight actually.

[isaaccorley]

Ok looks like tests are passing now. Had to skip the lidar plot test for python 3.9 until open3d v0.14 and on windows and mac due to segmentation faults but works fine on ubuntu 3.6-3.8.

[adamjstewart]

I assume the .las files are manually created random noise and not real data? Can you add a section to tests/data/README.md explaining how to create this data for the next person who needs to?

[adamjstewart]

I think my only remaining concerns relate to minimum dependency versions, everything else looks good!

[adamjstewart]

While debugging the recent mypy issues I noticed a lot of typing errors in this file. For example, the _load function says that it returns Tuple[List[str], Dict[int, Dict[str, Any]], Any], but many of these return values can also be None. The type: ignore here is masking this for geoms, and the Any is masking this for labels, but the real issue is that the function is not typed correctly. @isaaccorley can you fix this? We should be ignoring typing errors very sparingly for issues that are beyond our control (PyTorch is missing type hints for large parts of the library), not for things like this. For the same reason, we should try to avoid Any as it basically turns off all type checking.

[isaaccorley]

I'll refactor but so you know, this dataset has no structure to the nested dictionaries whatsoever so I opted for Dict[str, Any] in most cases. Also some of the methods return Any due to them returning an open3d object or a pandas DataFrame but I'm not able to use those types because I'm not importing them at the top level, only inside the method.

[adamjstewart]

See https://mypy.readthedocs.io/en/stable/runtime_troubles.html, it may be possible to declare the type in a string or comment instead to avoid the top-level import.

[isaaccorley]

Nice, ok I'll submit a PR.

[ashnair1]

@isaaccorley Had a question about the bounding box representation here. rasterio's index method is used to convert from (lon,lat) to pixel coordinates. However I noticed that this returns negative bbox coordinates for certain samples.

torchvision's draw_bounding_boxes states that it accepts box coordinates to be w.r.t image i.e. they must satisfy 0 <= xmin < xmax < W and 0 <= ymin < ymax < H. Is this not an issue?

I looked into this because I recently tried training an object detector (torchvision's Faster RCNN) on this dataset and the result looked a bit weird. In the docs they mention they need bboxes to satisfy the above constraint so I was wondering if this was the cause of my problem.

[ashnair1]

Think there might be an issue.

The following is a comparison of the image - MLBS_5

Left is QGIS (polygons from train_MLBS), right is plot by torchgeo

[isaaccorley]

Seems like you're right on the coords being swapped. Some of the boxes look transposed and others look like they've been cut off because they probably were outside the bounds of the image.