Standardization of the datasets

[pmeier]

This is a discussion issue which was kicked of by #1067. Some PRs that contain ideas are #1015 and #1025. I will update this comment regularly with the achieved consensus during the discussion.

Disclaimer: I have never worked with segmentation or detection datasets. If I make same wrong assumption regarding them, feel free to correct me. Furthermore, please help me to fill in the gaps.

---

Proposed Structure

This issues presents the idea to standardize the torchvision.datasets. This could be done by adding parameters to the VisionDataset (split) or by subclassing it and add task specific parameters (classes or class_to_idx) to the new classes. I imagine it something like this:

import torch.utils.data as data

class VisionDataset(data.Dataset):
    pass

class ClassificationDataset(VisionDataset):
    pass

class SegmentationDataset(VisionDataset):
    pass

class DetectionDataset(VisionDataset):
    pass

For our tests we could then have a generic_*_dataset_test as is already implement for ClassificationDatasets.

---

VisionDataset

• As discussed in Standardisation of Dataset API split argument name #1067 we could unify the argument that selects different parts of the dataset. IMO split as a str is the most general, but still clear term for this. I would implement this as positional argument within the constructor. This should work for all datasets, since in order to be useful each dataset should have at least a training and a test split. Exceptions to this are the Fakedata and ImageFolder datasets, which will be discussed separately.

• IMO every dataset should have a _download method in order to be useful for every user of this package. We could have the constructor have download=True as keyword argument and call the download method within it. As above, the Fakedata and ImageFolder datasets will be discussed below.

---

Fakedata and ImageFolder

What makes these two datasets special, is that there is nothing to download and they are not splitted in any way. IMO they are not special enough to not generalise the VisionDataset as stated above. I propose that we simply remove the split and download argument from their constructor and raise an exception if someone calls the download method.

Furthermore the Fakedata dataset is currently a ClassificationDataset. We should also create a FakeSegmentationData and a FakeDetectionData dataset.

---

ClassificationDataset

The following datasets belong to this category: CIFAR*, ImageNet, *MNIST, SVHN, LSUN, SEMEION, STL10, USPS, Caltech*

• Each dataset should return PIL.Image, int if indexed
• Each dataset should have a classes parameter, which is a tuple with all available classes in human-readable form
• Currently, some datasets have a class_to_idx parameter, which is dictionary that maps the human-readable class to its index used as target. I propose to change the direction, i.e. create a idx_to_class parameter, since IMO this is the far more common transformation.

---

SegmentationDataset

The following datasets belong to this category: VOCSegmentation

---

DetectionDataset

The following datasets belong to this category: CocoDetection, VOCDetection

---

ToDo

• The following datasets need sorting into the three categories: Caltech101, Caltech256,
  CelebA, CityScapes, Cococaptions, Flickr8k, Flickr30k, LSUN, Omniglot, PhotoTour, SBDataset (shouldn't this be just called SBD?), SBU, SEMEION, STL10, and USPS
• Add some common arguments / parameters for the SegmentationDataset and DetectionDataset

Thoughts and suggestions?

[fmassa]

Thanks a lot @pmeier for starting this discussion!

Here are my general thoughts:

• if we do not yet have enough datasets that fit in Segmentation / Detection dataset, then it might make sense to postpone implementing such abstractions, as they will lack flexibility as they will be tailored to 1 / 2 datasets.
• split as positional argument: this would be a breaking change for CocoDetection, and potentially others. Also, what if a dataset doesn't have a canonical train / test split?
• download for every dataset: agree, even though this might not be possible for some datasets. But we should strive to have as much coverage as possible

Some more specific thoughts:

ClassificationDataset

The bulk of the datasets currently available are in fact classification datasets.
For example, LSUN SEMEION, STL10, USPS are all classification datasets.
Caltech101 and Caltech256 are mostly used for classification as well, even though they have more information available.

About each dataset having a classes parameter, I wonder if this shouldn't instead be a method that returns a dict with a number of fields (with standardized names as much as possible).
For example

def get_metadata(self):
    return dict(classes=[0, 1, 2], boxes=[], ...)

Not sure if this is the right design though, but I was thinking about what other fields we might want to return. For example, the TF datasets have an _info method that returns a number of useful information about the dataset, and we could get some inspiration from it. Their need for this is different than ours though, as they need the features attribute to know how to decode their TFRecords structure.

Also worth raising the visibility of class_to_idx or alike, because it has been the source of a few user issues, like #714.

Thoughts?

Also, cc @zhangguanheng66 and @cpuhrsch as we might want to have some similarities on the dataset APIs between text and vision

[pmeier]

• if we do not yet have enough datasets that fit in Segmentation / Detection dataset, then it might make sense to postpone implementing such abstractions, as they will lack flexibility as they will be tailored to 1 / 2 datasets.

Agreed.

• split as positional argument: this would be a breaking change for CocoDetection, and potentially others.

Question(s):

What is the standard way of working with COCODetection? Does everyone simply define their own splits? How can you compare models between users?

Or is this still in challenge mode? In that case we need to decide if we want to have datasets with a single split or potentially a split without targets (see #900).

Also, what if a dataset doesn't have a canonical train / test split?

What do you mean by that? I would define a split as subset of the whole dataset. IMO each split should have the same structure.

• download for every dataset: agree, even though this might not be possible for some datasets. But we should strive to have as much coverage as possible

Can you think of a dataset besides ImageFolder and FakeData that can't / shouldn't have this method? If someone has a local dataset that he doesn't want to publish, I think its reasonable to expect him to not set download=True.

About each dataset having a classes parameter, I wonder if this shouldn't instead be a method that returns a dict with a number of fields (with standardized names as much as possible).
For example

def get_metadata(self):
    return dict(classes=[0, 1, 2], boxes=[], ...)

[...]
Also worth raising the visibility of class_to_idx or alike, because it has been the source of a few user issues, like #714.

I like the idea of classes and idx_to_class as parameter better, since every ClassificationDataset should have these. We could do get_additional_metadata() for everything else (e.g. boxes for Caltech).

---

As mentioned in #1086 should we protect all internal parameters (e.g. data for CIFAR or SVHN) to alert users to not rely on them since they might change?

[zhangguanheng66]

I would like to be very careful about adding some functions in VisionDataset, though TorchText has split/download/splits functions in the abstract dataset (https://github.com/pytorch/text/blob/master/torchtext/data/dataset.py#L16). IMO, a simple abstract dataset will give you more flexibility.

We are now testing a new pattern for TorchText. Instead of using abstract dataset (mentioned above), I move some functions to the utils and keep them orthogonal. For example, TorchText.utils.download function. We expect some feedbacks to the new pattern later before we make some significant changes to the abstract dataset.

[fmassa]

@pmeier

What is the standard way of working with COCODetection? Does everyone simply define their own splits? How can you compare models between users?

Each COCO has a different annotation file for train / val, and that is the second argument that we pass, and not split. So the ann_file plays the role of split

What do you mean by that? I would define a split as subset of the whole dataset. IMO each split should have the same structure.

Some datasets do not define a train / test split. So it's up to the user to decide how to split it, for example using torch.utils.data.random_split. Thus, having a split argument to the dataset doesn't apply there.

Can you think of a dataset besides ImageFolder and FakeData that can't / shouldn't have this method? If someone has a local dataset that he doesn't want to publish, I think its reasonable to expect him to not set download=True.

I mean, sometimes the user need to login / authenticate to download certain datasets (see #1042 (comment) for an example), so we wouldn't be able to provide a download functionality for those.

I like the idea of classes and idx_to_class as parameter better, since every ClassificationDataset should have these. We could do get_additional_metadata() for everything else (e.g. boxes for Caltech).

Maybe, but I still want to double-check a bit before.

cc @aadcock what are your thoughts on this in general?

[pmeier]

IMO, in the end it comes down to two philosophies since there are so many different datasets that we cannot foresee every possible option:

1. We have a rudimentary abstract dataset that allows a high degree of flexibility. Downside of that flexibility is that contributions need to be manually maintained to keep some standardisation.
2. We have a more complex abstract dataset that bundles functionality that most datasets possess. Downside of this is that we might need some creative solutions for the edge cases.

I would argue that datasets without split or datasets that cannot be downloaded are edge cases like ImageFolder or FakeData that need special treatment. IMO they are not important enough that the normal datasets miss out on this functionality. I think its not a problem to define something like this in those cases:

def _download():
    raise RuntimeError("Dataset cannot be downloaded")

To cite the Python Zen:

Special cases aren't special enough to break the rules.
Although practicality beats purity.

Right now we roll with option 1. and as a result of the heterogeneity we started this discussion about standardisation.

[fmassa]

@pmeier one core principle of torchvision (and PyTorch as well) is that most things should be easily understandable. So if a user wants to create / modify a new dataset, it is very easy for them to do so by reading the source code, without having to jump over a lot of abstractions.

I do think that having the right level of abstractions is a desirable thing, but we should carefully design it so that it is not a blocker for us in the future.

And it's easier to add something than to remove it, so before making all datasets have a split / download argument to it, I'd rather wait a bit more and see what the others have to say.

One could say that ideally, we would want to have a common API like

d1 = torchvision.datasets.get('coco', 'train2017', transform, download=True)
d2 = torchvision.datasets.get('voc-detection', 'train', transform)
d3 = torchvision.datasets.get('imagenet', 'train2012', transform)
d4 = torchvision.datasets.get('mnist', 'train', transform)
...

all with the same high-level API. But this makes some assumptions, and maybe we might want to enforce those assumptions/limitations at a higher level than the Dataset API.

Sure, this doesn't address what attributes each dataset should have, and that's one of the points that I want to discuss in this issue.

[zhangguanheng66]

@fmassa IMO, the abstract dataset could include at least data and transforms (field for torchtext) and full implementation of getitem, len. Those components are highly repetitive across most dataset (if my understanding is correct). something like this:

`

class TextDataset(torch.utils.data.Dataset):

    def __init__(data, fields):
        self.data = data
        self.fields = fields

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)

`

We could carefully add more stuffs later on as the abstract class evolves.

[pmeier]

@zhangguanheng66 transforms as well as the root of the dataset are already part of the abstract dataset class called VisionDataset. I agree on the other part. We could do something like this:

class VisionDataset(...):
    ...

    def __getitem__(self, index):
        image, target = self._images[index], self.targets[index]
        if self.transform is not None:
            image, target = self.transform(image, target)
        return image, target

    def __len__(self):
        return len(self._images)

    @property
    def _images(self):
        raise NotImplementedError

    @property
    def _targets(self):
        raise NotImplementedError

[zhangguanheng66]

@pmeier Sure, I agree. I'm just wondering the pros and cons to include data (e.g. images, targets) in the abstract dataset class. I guess it won't hurt us, right?

[aadcock]

I'm late to the party here and have comments on a few different pieces of this thread. Here are a few thoughts I've collected about things that I see commonly requested:

1. I also agree that @zhangguanheng66's suggestion is a good place to start since almost every dataset has these options.
2. Already mentioned, a get_metadata call of some sort on the dataset is one of the most requested items I get from the scientists I work with so that they can write code that reacts to the data that a dataset returns without having to retrieve samples or build / download the dataset. Standardizing what this returns is a bit tricky, but the call is pretty straightforward.
3. A lot of users ask about creating a canonical way to attach a sample id to each sample to use for debugging or logging information / features for a sample. This also feels like a pretty general use-case. It's not hard to create your own, but if there was a standard way of doing this, it would make it easier to write reusable code...I've solved this in the past with returning samples as dicts with fields like, "input" , "target", "metadata", but there are downstream problems with some of the composable dataset APIs.
4. @pmeier, thanks for the initial thoughts! I think they are really useful. I do think that ImageFolder is special for more than just the lack of a download. It provides an API to a datastore which could be different for each instance while ClassificationDataset et al are more like interfaces for the underlying datasets to implement. I agree they are both datasets / can be viewed in a similar framework, but my point here is that when someone is using an ImageFolder they are frequently doing something a bit different than when a user would be building a ClassificationDataset (and I think it's actually pretty common if you have any sort of proprietary data). You could even imagine using an ImageFolder to build a classification dataset, for example.
5. A minor point about the ClassificationDataset / other abstractions. A common use case that is not mentioned above with the get_metadata function / Classification datasets is that we might want to allow multi-class datasets (so you would return multiple ints). Admittedly, this quickly gets into attempting to cover every type of dataset, but this is something I work with a lot.

[pmeier]

Since there has been no more input for some days, can I regard the proposal of @zhangguanheng66 as consensus and thus add it to the top post?

[fmassa]

@pmeier we are still discussing about this internally, and we still seem to have no consensus on how to structure it yet.

[koenvandesande]

How does this approach scale to different tasks/datasets with multiple tasks? On Coco, you can do detection, human keypoint detection and segmentation (of either objects, stuff with CocoStuff or everything toghether with panoptic segmentation).

Another part of "the Python way" is duck typing: "If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck". No need to have a common superclass to implement an interface, as long as the required parts are there.