Triage remaining integration test failures with other Arrow implementations

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It would be nice to have more complete integration coverage for the rust implementation, as well as more documentation and community knowledge about how to run them.

re #1398

The basic idea here is to triage remaining failures in the integration tests with the rust implementation and file tickets and/or fix them.

This is a nice project for people to work on in their spare time and would likely expose them to parts of the arrow crate.

Describe the solution you'd like

1. Setup the integration tests as described in Improve integration testing docs #1403
2. Verify that running archery --debug integration --run-flight --with-cpp=true --with-rust=true passes without error

Then, for each line like .skip_category('Rust'), in https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py,:

1. remove it from datagen.py
2. Rerun archery --debug integration --run-flight --with-cpp=true --with-rust=true
3. File figure out why it fails and fix it / file a ticket

Additional context
re #1398

[viirya]

Subtasks:

• generate_decimal128_case: (Fix generate_decimal128_case in integration test #1439)(Remove validate_decimal_precision check in DecimalBuilder.append_value #1766)
• generate_interval_case: Fix generate_interval_case integration test failure #1445
• generate_map_case: Fix generate_map_case integration test failure #1456
• generate_non_canonical_map_case: (Fix generate_non_canonical_map_case integration test failure #1475, Fix schema comparison for non_canonical_map when running flight test #1730)
• generate_nested_large_offsets_case (List equality method should work on empty offset ListArray #1817)
• generate_nested_dictionary_case (Read/write nested dictionary in ipc stream reader/writer #1565) (Fix generate_nested_dictionary_case integration test failure for Rust cases #1635) (Exclude dict_id and dict_is_ordered from equality comparison of Field #1646) (flight_client_scenarios integration test should receive schema from flight data #1669)
• generate_extension_case (no fix needed, see the comment Triage remaining integration test failures with other Arrow implementations #1404 (comment))
• generate_unions_case (Implement ArrayEqual for UnionArray #67)(Fix generate_unions_case integration test  #1676)(Fix incorrect null_count in generate_unions_case integration test #1712)(Rust IPC Read should be able to read V4 UnionType Array #1788)

[viirya]

I will work on above list one by one.

[viirya]

generate_nested_large_offsets_case seems a false negative case. The test can be passed on current master without any change.

[alamb]

generate_nested_large_offsets_case seems a false negative case. The test can be passed on current master without any change.

Perhaps we have fixed it in some previous PR

[viirya]

For generate_extension_case, although there is .skip_category('Rust') but running archery --debug integration --run-flight --with-cpp=true --with-rust=true is okay without any failure even removing the skip_category.

I don't find any extension type support in current codebase, so I suppose it is not supported.

I guess it is passed because IPC reader/writer already correctly process custom metadata in IPC message, so the metadata is preserved well.

We just don't have an explicit extension type support. But running the integration test is okay as we can read/write the extension metadata correctly to talk with C++ Arrow.

[viirya]

Okay, I fixed all integration test failures except for generate_decimal256_case. Currently the only failing test case is generate_decimal256_case which needs #131. Not sure if it is straightforward to add the Decimal 256 support, but I may take some time looking into it.

Do we need to send a patch to dev/archery/archery/integration/datagen.py in C++ Arrow repo to enable these Rust cases?

[alamb]

Okay, I fixed all integration test failures

❤️ thank you so much @viirya !!!!

except for generate_decimal256_case. Currently the only failing test case is generate_decimal256_case which needs #131. Not sure if it is straightforward to add the Decimal 256 support, but I may take some time looking into it.

Nice -- I haven't heard anyone else asking for Decimal256 support but it comes up occasionally (https://github.com/apache/arrow-rs/search?q=decimal256&type=issues) and would be great to round out support.

Do we need to send a patch to dev/archery/archery/integration/datagen.py in C++ Arrow repo to enable these Rust cases?

Yes, I think so -- apache/arrow#11238 show an example of such a PR

[viirya]

As we enable almost all test cases except for generate_decimal256_case which we have an issue #131 tracking it, maybe we can close this.

cc @alamb

[alamb]

I agree -- great job @viirya -- your work on this issue was epic. Thank you again

[viirya]

Thank you @alamb