GH-24868: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType

[rok]

Rationale for this change

For use cases where underlying datatype and number of dimensions in tensors are equal but not the actual shape we want to add a VariableShapeTensorType.
See #24868 and huggingface/datasets#5272

What changes are included in this PR?

This introduces definition of arrow.variable_shape_tensor extension and it's C++ implementation and a Python wrapper.

Are these changes tested?

Yes.

Are there any user-facing changes?

This introduces new extension type to the user.

• Closes: [C++] Add a Tensor logical value type with varying dimensions, implemented using ExtensionType #24868

[github-actions]

⚠️ GitHub issue #24868 has been automatically assigned in GitHub to PR creator.

[westonpace]

I just looked through the rst portion and have a few questions.

[rok]

Looking at PyTorch NestedTensor it stores strides per tensor as opposed to our proposal which stores just a permutation for the entire array which can then be used to derive strides (using tensor shape) per tensor. In PyTorch if strides are not provided to the constructor they are apparently calculated assuming row-major/C-contiguous layout.

This makes me assume strides are stored for caching (and not to allow variable data layout per tensor).

Should we adjust to store permutations (or strides) per row too or rather leave that for another extension (GeneralVariableShapeTensorType)? @lhoestq @mariosasko @AlenkaF @jorisvandenbossche @ezyang

[lhoestq]

No strong opinion regarding storing strides. Cc @thomasw21 I'm not even sure that nested tensors are used that often no ? I'm also pinging other experts to get more insights.

[AlenkaF]

How much extra complexity is added if we enable each tensor in an array to have it's own permutation defined?

[rok]

@AlenkaF Storing permutations would allow per tensor memory layout, which I'm not sure is really needed in practice and could be confusing and complex.
Storing strides (and perhaps requiring uniform permutation) could be useful as consumers wouldn't need to calculate strides when using tensors. Complexity wouldn't really increase in this case.
Either way the size of type would increase by n tensors x ndim x sizeof(int64).

[pitrou]

@jorisvandenbossche Can I let you give this the final review?

[rok]

@jorisvandenbossche can we merge this?

[jorisvandenbossche]

Sorry, one more small nitpick: the minimal metadata is actually no metadata, which is typically represented as an empty string (I am actually not fully sure if in this case the metadata key could also just not be present in the field metadata), instead of an empty json dict (I don't think we should necessarily recommend using an empty dict)

[rok]

That's a fair point! Empty string feels like the safer choice here. See my suggested change below.

[jorisvandenbossche]

Thanks a lot @rok !

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit a7fab04.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.