PoC arbitrary dispatch

[pmeier]

@NicolasHug @vfdev-5 following our offline discussion. This is just a PoC for F.equalize / transforms.RandomEqualize. I have to clean up a little, but the general structure is there.

[NicolasHug]

Thanks Philip, looks good, I'm just not sure about the need for sets

[NicolasHug]

Not sure about the pros and cons of making this None instead... Are we cool with mutable class attributes? Maybe we can think of that later

(If the answer is that this makes mypy unhappy then hmmmmmmmm)

[pmeier]

No, we can go with either.

[NicolasHug]

We can worry about all that later, but will we be able to also register kernels for simple tensors and for pil? Or are we going to hit JIT issues?

[pmeier]

I would love that, since we would remove the special casing. I don't think JIT will an issue as long as we have a

if torch.jit.is_scripting():
    return ...

at the top.

[NicolasHug]

QQs:

1. Do we expect more than one dispatcher per transform?
2. does this need really to be a set?

[pmeier]

Do we expect more than one dispatcher per transform?

Yes. For example

• https://github.com/pytorch/vision/blob/3966f9558bfc8443fc4fe16538b33805dd42812d/torchvision/transforms/v2/_color.py#L162-L177
• https://github.com/pytorch/vision/blob/3966f9558bfc8443fc4fe16538b33805dd42812d/torchvision/transforms/v2/_geometry.py#L919-L927

does this need really to be a set?

No. What would you prefer?

[NicolasHug]

Is this just

types = {}
for dispatcher in dispatchers:
    types.update(_get_supported_datapoint_classes(dispatcher))

?

and _get_supported_datapoint_classes probably doesn't need to return a set if we do that

[pmeier]

I wanted to make sure that we only go for the common ground. Meaning, we only transfer types that are support by all dispatchers. This is why I have the intersection & in there.

[NicolasHug]

It makes we wonder about the semantic and the use of _supported_datapoint_types.. what if there's a transform with multiple dispatchers where each of them handle a different part of the input? Would this transform end-up being a passthrough for everything?

[NicolasHug]

I mean typically if we're using the intersection, we can't just have a simple logic in the transforms like

if type(inpt) not in self._supported_datapoint_types:
    # pass-through inpt

We can't use the union either because that would pass some inputs to some dispatchers that don't support them (and error).

Seems like we have to implement the pass-through logic on a per-dispatcher basis instead (within the transforms).

[pmeier]

Seems like we have to implement the pass-through logic on a per-dispatcher basis instead (within the transforms).

You are right. Let me see how this fits in

[pmeier]

Most of our transforms don't touch _transformed_types at all. Here is a list of the ones that do

• Dispatcher does not support arbitrary dispatch and would error for unsupported datapoints. Thus, the transform has to filter them out
  • RandomErasing
  • FiveCrop
  • TenCrop
  • UniformTemporalSubsample
  • Normalize
  • RandomPhotometricDistort (This is only partially the right category. The transform uses many dispatchers, but also defines a "dispatcher" itself that doesn't support arbitrary dispatch. If add PermuteChannels transform pytorch/vision#7624 goes through, this transform needs to go into the "performance" category below)
• The dispatcher supports arbitrary dispatch, but the transform filters for performance reasons (not sure the second half is true, but I can't think of anything else)
  • Grayscale
  • RandomGrayscale
• Transform has no corresponding dispatcher and thus has to filter itself. We should look into why this is the case. Maybe it has a good reason, but maybe it is just an oversight?
  • LinearTransformation
• Transform only applies to a single type by design. The corresponding dispatcher has no kernels registered
  • ConvertBoundingBoxFormat
  • ClampBoundingBox
• Transform converts the actual type and thus applies only to a few specific types by design. The corresponding dispatcher has no kernels registered
  • ToTensor
  • PILToTensor
  • ToImageTensor
  • ToImagePIL
• Transform supports tensor images, but no PIL images:
  • ConvertImageDtype
  • ToDtype
  • LinearTransformation (we could implement support here similar to what we do with GaussianBlur: convert to tensor, apply the tensor kernel, and convert back)
  • Normalize

Apart from these groups, there is also Lambda. It takes the supported types as parameter. Nothing would change with the design in the PR, since it already circumvents _transformed_types.