Add validation between input_col and fn for Callable DataPipes

[ejguan]

🚀 The feature

Using inspect to validate the input_col matching fn input for Mapper, Flatmapper and other DataPipe using both input_col and fn as the arguments.

• Mapper
• Filter
• Flatmap
• BatchMap
• etc

Motivation, pitch

This would help users to identify the wrong arguments at the construction time for such IterDataPipe rather than at iteration time.

Alternatives

No response

Additional context

No response

[davidegavio]

I'd like to work on this if possible

[ejguan]

@davidegavio
Thanks for taking interest. Go for it. For Mapper and Filter, you may need to create a PR in PyTorch Core. https://github.com/pytorch/pytorch

[bushshrub]

Was this ever worked on? If not I could take a stab at this.

[ejguan]

Hi @davidegavio, do you still have plan to work on this issue?

[bushshrub]

I'll give this a try when I get home

[bushshrub]

@ejguan just to check, is this to check that fn has type annotations for returning a Sequence?

[ejguan]

If the provided fn has type annotations, we should validate the number of function's arguments matches input_col.

Let's say there is only 1 argument in fn, but input_col is specified as 1. We should raise Error.

[bushshrub]

What does input_col refer to in this scenario? Is it the dimension?

[ejguan]

Yeah. Let's say you have a DataPipe yielding tuples like (1, 2), (3, 4), ....

dp = IterableWrapper([(1, 2), (3, 4), ...])
dp = dp.map(lambda x: x + 1, input_col=1)

And, input_col would specify which element in each tuple should be applied by the fn. In this case, as input_col=1, the function will apply to 2, 4, ....

We should add inline doc to explain the behavior of input_col

[bushshrub]

Alright, I will work on this and open a PR.