epic: Initial implementation of this crate

[friendlymatthew]

A checklist of items that I plan to tackle. Please feel free to edit and open issues related to a certain topic

udfs to support

• variant_to_json - returns a JSON string from a VariantArray

• json_to_variant - returns a VariantArray from a JSON string

• cast_to_variant - returns a VariantArray from a column

  • cast_to_variant: Support casting from more column types #3
  • cast_to_variant: Add more unit tests #4

• variant_get(VariantArray, path) - returns the extracted type dictated by the path from the VariantArray

• is_variant_null(VariantArray) - tests whether elements in VariantArray are Variant::Null

• variant_pretty(VariantArray) - returns a human-readable version of VariantArray

• variant_schema(VariantArray) - returns the schema of a VariantArray

  • variant_schema: Initial implementation #5

• variant_object_construct(key1, value1, [keyN, valueN])

  • variant_object_construct: Initial implementation #6

• variant_object_delete(VariantArray, VariantPath)

  • variant_object_delete: Initial implementation #7

• variant_object_insert(key, value)

  • variant_object_insert: Initial implementation #8

• variant_list_construct(value1, [valueN])

  • variant_list_construct: Initial implementation #9

• variant_list_insert(VariantArray, value)

  • variant_list_insert: Initial implementation #10

• variant_explode

• variant_explode_outer

Databricks supports the following variant-related udfs: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-functions-builtin#variant-functions. I think it would be very cool to achieve 1:1 parity with their functionality

misc

• Add a examples/ directory that lists examples for every udf we support, we can make use of a sample JSONL dataset like: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
• Have integration tests, preferably using slt tests
• Add more unit tests
• Write documentation
• Write the README
• Have better error messages, especially when instructing users which arguments to pass Improve error messaging #14
• Pick a license Pick licensing #15

questions

It would also be good to compile a list of questions/ideas/limitations that arise from interfacing with arrow's parquet-variant libraries:

• Why does VariantArray not implement IntoIter?

• variant_to_json should accept an optional format configurations. Currently the library dictates how to map specialized Variant types to JSON (e.g. timestamps are always formatted as a string)

  • [Variant] Allow configurable JSON format for specialized types apache/arrow-rs#8605

• There is a lot of ceremony to go from a Variant to a VariantArray. Maybe VariantArray should impl From<IntoIterator<Item = Variant>>?

  • [Variant] Remove ceremony of going from list of Variant to VariantArray apache/arrow-rs#8606
  • [Variant] Impl Extend for VariantArrayBuilder apache/arrow-rs#8611

• When checking if 2 VariantArrays are equal, it is a bit odd that it will panic when calling .value(i) when the ith position has empty metadata and value?

  • [Variant] Remove potential panics when probing VariantArray apache/arrow-rs#8609

• Why doesn't VariantArray impl PartialEq?

  • [Variant] impl PartialEq for VariantArray apache/arrow-rs#8610

• Is there a reason why VariantArray doesn't impl Array? Seems this would be super nice

  • [Variant] Make VariantArray iterable apache/arrow-rs#8612

[alamb]

Databricks supports the following variant-related udfs: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-functions-builtin#variant-functions. I think it would be very cool to achieve 1:1 parity with their functionality

I agree this would be a great target to shoot for (maybe even put a note in the readme that the goal of this crate is to provide databricks / spark compatible variant functions)

The databricks list seems pretty similar to what is in spark: https://spark.apache.org/docs/latest/api/sql/index.html

[alamb]

misc

• Add a examples/ directory that lists examples for every udf we support, we can make use of a sample JSONL dataset like: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
• Have integration tests, preferably using slt tests

I recommend putting the examples in the crate readme / library home page as rustdocs to make sure they keep running and make them easier to discover

• Pick a license

I recommend Apache 2.0 , given it is well known and is straightforward to apply / understand, and consistent with most other DataFusion libraries.

questions

It would also be good to compile a list of questions/ideas/limitations that arise from interfacing with arrow's parquet-variant libraries:

This is a great list! I think you are probably the first person to really try and integrate Variant so this feedback is super valuable.

These would be great tickets to file in arrow-rs and attach to the epic. Would you like me to do so? Or would you rather do so yourself?

• [EPIC][Parquet] Finalize Variant Type support in Parquet apache/arrow-rs#8480

• Why does VariantArray not implement IntoIter?
• variant_to_json should accept an optional format configurations. Currently the library dictates how to map specialized Variant types to JSON (e.g. timestamps are always formatted as a string)
• There is a lot of ceremony to go from a Variant to a VariantArray. Maybe VariantArray should impl From<IntoIterator<Item = Variant>>?
• When checking if 2 VariantArrays are equal, it is a bit odd that it will panic when calling .value(i) when the ith position has empty metadata and value?
• Why doesn't VariantArray impl PartialEq?

I don't think there are great reasons for these rough edges, and I suspect we can make them all much easier

[alamb]

There is also a list of functions in the original iceberg proposal here that might be good to reference:

• https://docs.google.com/document/d/1sq70XDiWJ2DemWyA5dVB80gKzwi0CWoM0LOWM7VJVd8/edit?tab=t.0