Add Kitti and Sintel datasets for optical flow

[NicolasHug]

This PR Adds the Kitti and Sintel datasets for optical flow.

Here are a few design decisions, following our initial discussion (in this PR):

• Unlike Sintel, Kitti has a built-in valid mask indicating which flow values are returned. As a resut Sintel returns img1, img2, flow while Kitti returns img1, img2, flow, valid.

• For both these datasets, the targets aren't known if split="test" so we return img1, img2, None and img1, img2, None, None in these cases.

• For both, flow is a numpy array of shape (2, H, W) and valid is a boolean numpy array of shape (H, W). The images are PIL images.

• The transforms expect to receive img1, img2, flow, valid, even for Sintel, and even in test mode.

• add tests

• write docs

cc @pmeier

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit ac4eaab (more details on the Dr. CI page):

---

• 2/2 failures introduced in this PR

---

2 failures not recognized by patterns:

Job	Step	Action
binary_linux_conda_py3.6_cu111	packaging/build_conda.sh	🔁 rerun
binary_win_conda_py3.9_cu113	Build conda packages	🔁 rerun

---

1 job timed out:

• binary_linux_conda_py3.6_cu111

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[datumbox]

Just a few thoughts:

[NicolasHug]

@datumbox @pmeier @fmassa thanks a lot for your very useful feedback.

I updated the PR accordingly (see original PR message above) and added tests and docs. This is now ready for a proper review.

[NicolasHug]

Rendered docs: https://963455-73328905-gh.circle-artifacts.com/0/docs/datasets.html

[fmassa]

Thanks!

[vadimkantorov]

There were problems with older datasets that returning tuple doesn't allow for easy adding extra fields in the future, e.g. #3608. I propose to return img, target_dict tuple (even better may be always returning a single dict for the cases of skipping actual image loading). And there is more unity when iterating between different related datasets.

[pmeier]

@vadimkantorov We are aware of that and the new prototype dataset API will always return a single dictionary per sample. There is no documentation yet, but if you want to you can have a look into torchvision/prototype/datasets to see how things will look like in the future. Plus, if you have feedback, that is very welcome!

[vadimkantorov]

If you're committed to using only tuples, please also return some internal example id, image/video file name or something of this form. Then at least people can hack around and fetch additional information

[NicolasHug]

Thanks for your input @vadimkantorov

I don't think any of the datasets return that kind of info so far (sadly), so returning a dict with extra info would make KittiFlow and Sintel outliers w.r.t. what we currently have. Returning the ids etc. is definitely useful, but such new design is better suited for a complete re-work of the datasets, which we are currently doing as @pmeier pointed out above.

[datumbox]

LGTM, thanks!

Definitely interesting to see how the ideas mentioned above can be incorporated on the new API but I think it's worth merging this PR to unblock your work on the flow models.

[pmeier]

LGTM, thanks @NicolasHug!