[MINOR] Fix logical type issue for timestamp columns

[linliu-code]

Change Logs

This pr #9743 adds more schema evolution functionality and schema processing. However, we used the InternalSchema system to do various operations such as fix null ordering, reorder, and add columns. At the time, InternalSchema only had a single Timestamp type. When converting back to avro, this was assumed to be micros. Therefore, if the schema provider had any millis columns, the processed schema would end up with those columns as micros.

In this pr to update column stats with better support for logical types: #13711, the schema issues were fixed, as well as additional issues with handling and conversion of timestamps during ingestion.

this pr aims to add functionality to spark and hive readers and writers to automatically repair affected tables.
After switching to use the 1.1 binary, the affected columns will undergo evolution from timestamp-micros to timestamp-mills. Normally a lossy evolution that is not supported, this evolution is ok because the data is actually still timestamp-millis it is just mislabeled as micros in the parquet and table schemas

Impact

When reading from a hudi table using spark or hive reader if the table schema has a column as millis, but the data schema is micros, we will assume that this column is affected and read it as a millis value instead of a micros value. This correction is also applied to all readers that the default write paths use. As a table is rewritten the parquet files will be correct. A table's latest snapshot can be immediately fixed by writing one commit with the 1.1 binary, and then clustering the entire table.

Risk level (write none, low medium or high below)

High,
extensive testing was done and functional tests were added.

Documentation Update

#14100

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[lokeshj1703]

@linliu-code We can add support for Spark 3.2 as well in this PR if it is not very difficult. The fix should ideally work for Spark 3.2 as well.
Also since local timestamp is not supported for older hudi versions, it is better to throw UnsupportedOperationException for any conversions related to it. We can remove any supporting logic for local timestamp but can still keep it in the enums to preserve ordinal numbering.

[hudi-bot]

CI report:

• 5ef5773 UNKNOWN
• 1d2d706 UNKNOWN
• 4bbecb7 UNKNOWN
• 73e1942 UNKNOWN
• ffcc9ca UNKNOWN
• 4c5a493 UNKNOWN
• 408cc29 UNKNOWN
• 0c4e026 UNKNOWN
• 8583da1 UNKNOWN
• 61285db UNKNOWN
• d264474 UNKNOWN
• a62e355 UNKNOWN
• d22894c UNKNOWN
• e8d9ca3 UNKNOWN
• d2ef7f9 UNKNOWN
• 4440884 UNKNOWN
• 721b598 UNKNOWN
• 7d0b742 UNKNOWN
• 103e3b4 UNKNOWN
• e35e8f9 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build