feat(schema): Add VARIANT support to HoodieSchema

[voonhous]

Describe the issue this Pull Request addresses

Introduce the new VARIANT schema type. This will look similar to a record/struct with a logical type associated with it. The implementation is inline with parquet spec.

Linked task: #17745

Note: No reader and writer support has been added yet. It will be added in a separate PR.

Summary and Changelog

1. Add VARIANT support to HoodieSchema
2. Add VARIANT type to HoodieSchemaType enum.

Impact

Support for Variant types in accordance to parquet's spec.

Risk Level

Low

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[the-other-tim-brown]

Can we follow a similar pattern as the uuid where there is a singleton we can compare to instead of instanceof?

[voonhous]

Make sense, limit the number of object creation for types. Will eagerly initialize a singleton.

[the-other-tim-brown]

General question, can you have shredded and unshredded values in the same dataset? if so, it seems like the schema should be the same for these

[voonhous]

I don't quite understand this question. The implementation follows the parquet schema spec.

Nonetheless, will like to still understand what you're pushing towards.
Do you mean if we can have a unshredded_typed_column and shredded_typed_column in the dataset?

Or are you saying that since shredded_variant typed columns can hold unshredded data, we should just maintain the shredded type?

Unshredded

optional group variant_unshredded (VARIANT) {
  required binary metadata;
  required binary value;
}

Shredded

optional group variant_shredded (VARIANT) {
  required binary metadata;
  optional binary value;
  optional int64 typed_value;
}

So, to use shredded schema to represent unshredded, we can just make typed_value null and populate value?

[the-other-tim-brown]

I'm wondering if the column can have both shredded and unshredded in the same file.

[hudi-bot]

CI report:

• 0fff72f Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[cshuo]

Do we need include shredding information in type/schema layer? IIUC, it's more about read/write optimization mechanism, which can be inferred or fetched from configuration during reading or writing.