GH-48167: [Python][Compute] Add python bindings for scatter, inverse_permutation

[tadeja]

Rationale for this change

To close or discuss #48167
inverse_permutation and scatter functions got implemented via #44393, PR #44394.

What changes are included in this PR?

Python tests for scatter, inverse_permutation kernels and bindings for InversePermutationOptions and ScatterOptions.

Are these changes tested?

Yes, tests added in test_compute.py.

Are there any user-facing changes?

Bindings for InversePermutationOptions and ScatterOptions are added.

• GitHub issue: [Python][Compute] Add python wrappers for scatter, inverse_permutation #48167
• GitHub Issue: [Python][Compute] Add python wrappers for scatter, inverse_permutation #48167

[github-actions]

⚠️ GitHub issue #48167 has been automatically assigned in GitHub to PR creator.

[tadeja]

The background of inverse_permutation and scatter functions is nicely explained under this umbrella issue

• Github issue [C++][Compute] Support special form in expression system #47374

Does it make sense to add Python wrappers/bindings for InversePermutationOptions and ScatterOptions already at this point in time? (Or at all?) What would you say, @felipecrv, @zanmato1984 ?

Currently both InversePermutationOptions and ScatterOptions do not seem to get accepted by inverse_permutation and scatter as demonstrated in added tests of test_compute.py.

[github-actions]

⚠️ GitHub issue #48167 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #48167 has been automatically assigned in GitHub to PR creator.

[zanmato1984]

Hi @tadeja , thanks for working on this. Appreciate it. I'm glad to see we are adding python bindings for scatter and inverse_permutation. To your questions:

Does it make sense to add Python wrappers/bindings for InversePermutationOptions and ScatterOptions already at this point in time? (Or at all?) What would you say, @felipecrv, @zanmato1984 ?

I think it does make sense to have them.

Currently both InversePermutationOptions and ScatterOptions do not seem to get accepted by inverse_permutation and scatter as demonstrated in added tests of test_compute.py.

I'm not sure what you mean. The underlying C++ APIs will accept those options. So I would expect no problem for their python bindings. Are you saying that they are not working?

[tadeja]

Thank you for the input, @zanmato1984, that helped.

I'm not sure what you mean. The underlying C++ APIs will accept those options. So I would expect no problem for their python bindings. Are you saying that they are not working?

It looks like Python bindings got fixed by adding option class names to FunctionDoc in vector_swizzle.cc.

Now test_option_class_equality is failing by default with
pyarrow.lib.ArrowInvalid: Could not serialize field output_type of options type InversePermutationOptions: shared_ptr<DataType> is nullptr
unless I pass output_type there, e.g. pc.InversePermutationOptions(output_type=pa.int32()).

If output_type is NULLPTR in C++ (which happens if output_type is None in Python) then InversePermutationOptions can't be serialized.
Any advice on the approach for fixing this? (Should Python users always specify output_type, or perhaps output_type be optional in C++? Or is it fine to leave it as it is?)

See example below:

import pyarrow.compute as pc
pc.InversePermutationOptions()

>>> InversePermutationOptions(max_index=-1, output_type=<NULLPTR>)

import pyarrow.compute as pc
pc.InversePermutationOptions().serialize()

>>> Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    pc.InversePermutationOptions().serialize()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "pyarrow/_compute.pyx", line 623, in pyarrow._compute.FunctionOptions.serialize
    shared_ptr[CBuffer] c_buf = GetResultValue(res)
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
    return check_status(status)
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
    raise convert_status(status)
pyarrow.lib.ArrowInvalid: Could not serialize field output_type of options type InversePermutationOptions: shared_ptr<DataType> is nullptr

cc @pitrou

[zanmato1984]

Thanks @tadeja for the further explanation! I now see the problem. Let me check the code and get back to you soon.