feat(schema): phase 17 - Remove AvroSchemaUtils usage (part 2) #17581

voonhous · 2025-12-12T11:45:52Z

Describe the issue this Pull Request addresses

Reference issue: #14282

This PR focuses on migrating usages of AvroSchemaUtils to the internal HoodieSchema abstraction and HoodieSchemaUtils.

Key Changes:

Migration: Migrated logic in key write/read paths (e.g., HoodieTable, HoodieWriteHandle, HoodieRowParquetWriteSupport) to use HoodieSchema and HoodieSchemaCompatibility.
Full Qualification: For classes where migration was not immediately feasible, calls to AvroSchemaUtils static functions have been fully qualified. This explicitly marks technical debt and makes these usages easily searchable for future refactoring.
Utility Enhancements: Added necessary bridge methods to HoodieSchemaUtils and HoodieSchemaCompatibility to support HoodieSchema objects directly.

Classes NOT included in migration (Fully Qualified)

The following classes retain AvroSchemaUtils usage but are now fully qualified:

TestAvroSchemaUtils
AvroSchemaUtils (Self)
AvroSchemaComparatorForRecordProjection
HoodieAvroUtils
HoodieSchemaCompatibility
MissingSchemaFieldException
HoodieSchemaUtils
SchemaBackwardsCompatibilityException
AvroSchemaRepair
TestAvroSchemaRepair
TestHoodieSchemaCompatibility (For consistency testing)

Specific Ignored Usages (getAvroRecordQualifiedName)

The following classes ignore AvroSchemaUtils.getAvroRecordQualifiedName:

BaseHoodieWriteClient
HoodieCatalog
HoodieHiveCatalog
HoodieTableFactory
StreamSync
TestHoodieTableFactory

Summary and Changelog

This PR is a refactoring effort to improve schema abstraction within Hudi. By moving away from raw Avro utils, we pave the way for better type safety and cleaner internal APIs.

Impact

Internal API Change: Methods in HoodieSchemaUtils and HoodieSchemaCompatibility now play a larger role in schema validation and evolution logic.
No User-Facing Change: This is a code health and refactoring PR; there are no changes to public configs or external behaviors.

Risk Level

Low

Documentation Update

none

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

voonhous · 2025-12-12T11:46:29Z

This PR is stacked on top of #17535, please ONLY merge this after #17535 is merged.

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveTypeUtils.java

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

voonhous · 2025-12-19T18:59:39Z

Rebased

hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java

the-other-tim-brown · 2025-12-21T22:12:47Z

hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java

+        .orElse(null);
+
+    if (nonNullType == null) {
+      throw new org.apache.hudi.internal.schema.HoodieSchemaException(


Similarly here, let's import the class

hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaUtils.java

the-other-tim-brown · 2025-12-21T22:16:31Z

hudi-common/src/test/java/org/apache/hudi/common/schema/TestHoodieSchemaUtils.java

+    HoodieSchema hoodieResult = HoodieSchemaUtils.resolveUnionSchema(hoodieFieldSchema, "TypeA");
+
+    // Should produce equivalent schemas
+    assertEquals(avroResult.toString(), hoodieResult.toString());


Can we assert on the objects being equal instead of the string representation by calling .toAvroSchema?

bump on this, it looks like it is not in the latest commit

Oh, removing this as we're removing HoodieAvroUtils#resolveUnionSchema. But I've verified the changes passes before removing it:

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveTypeUtils.java

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

the-other-tim-brown

Overall, LGTM. Just one minor item remaining on the tests

hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveTypeUtils.java

the-other-tim-brown · 2025-12-23T16:10:29Z

hudi-hadoop-mr/src/test/java/org/apache/hudi/hadoop/utils/TestHiveAvroSerializer.java

+    HoodieSchema schema = HoodieSchema.parse(schemaWithTimestampMicros);
+
+    // HiveTypeUtils.generateColumnTypes throws an exception for timestamp-micros since it's not supported by AvroSerDe
+    assertThrows(Exception.class, () -> {


I think for this case it will currently fall back to the underlying primitive and return a long

You're right, running the same code on master, i am getting this. Let me recheck all the logical types on master.

testGenerateColumnTypesForTimeMicros() testGenerateColumnTypesForTimeMillis() testGenerateColumnTypesForTimestampMicros()

These 3 tests are failing on master without the HiveTypeUtils migration. Let me fix them.

Modified the test + fixed the HiveTypeUtils#generateTypeInfo.

the-other-tim-brown · 2025-12-23T19:10:39Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveTypeUtils.java

-
+    HoodieSchemaType type = schema.getType();
+    if (type == DECIMAL && AvroSerDe.DECIMAL_TYPE_NAME
+        .equalsIgnoreCase((String) schema.getProp(AvroSerDe.AVRO_PROP_LOGICAL_TYPE))) {


These logical type checks seem to be very avro specific, can we just rely on the HoodieSchemaType now?

Yeap, it's doable for all the types in this function except VARCHAR and CHAR.

yihua

LGTM

…t changed

voonhous · 2025-12-24T13:33:56Z

Gosh, the CI is timing out, I'm going to increase the timeout to check if we the Azure CI can succeed without throwing any errors

NOTE: Before we merge this in, we will need to undo the last commit on this PR to revert the Azure CI timeout.

Edit to add: Not reverting Azure CI timeout increase here as increasing it to 3 hours is better than trying to get it to pass for 8 hours.

hudi-bot · 2025-12-24T15:23:14Z

CI report:

8ad0e47 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

voonhous · 2025-12-24T18:17:57Z

Azure CI + GitHub CI are all green! (FINALLY)

Merging this PR in.

…e#17581) * Remove AvroSchemaUtils from HiveAvroSerializer and HiveTypeUtils * Address comments * Address comments * Remove AvroSchemaUtils#resolveUnionSchema * Add more test to cover generateColumnTypes * Fix HiveTypeUtils#generateTypeInfo behaviour and ensure that it is not changed * Address comments * Increase Azure CI timeout

github-actions bot added the size:XL PR with lines of changes > 1000 label Dec 12, 2025

voonhous mentioned this pull request Dec 12, 2025

Phase 17: AvroSchemaUtils Method Removal #14282

Closed

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch 7 times, most recently from 64ac0dd to 1d7fdbc Compare December 16, 2025 02:38

voonhous requested a review from the-other-tim-brown December 16, 2025 02:48

github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels Dec 16, 2025

the-other-tim-brown reviewed Dec 16, 2025

View reviewed changes

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch 2 times, most recently from 311cfd7 to 4b9814f Compare December 17, 2025 11:49

voonhous requested a review from the-other-tim-brown December 18, 2025 02:29

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch from 4b9814f to 91797f9 Compare December 19, 2025 18:59

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch 2 times, most recently from f395ea6 to 2ca8f9f Compare December 20, 2025 08:05

the-other-tim-brown reviewed Dec 21, 2025

View reviewed changes

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch from 2ca8f9f to 41163c4 Compare December 22, 2025 13:43

the-other-tim-brown approved these changes Dec 22, 2025

View reviewed changes

yihua reviewed Dec 23, 2025

View reviewed changes

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch 3 times, most recently from c16ca0a to 13ac494 Compare December 23, 2025 07:37

apache deleted a comment from hudi-bot Dec 23, 2025

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch from 000b93c to b9e691a Compare December 23, 2025 14:14

voonhous mentioned this pull request Dec 23, 2025

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 1) #17599

Merged

3 tasks

the-other-tim-brown reviewed Dec 23, 2025

View reviewed changes

the-other-tim-brown approved these changes Dec 23, 2025

View reviewed changes

yihua approved these changes Dec 23, 2025

View reviewed changes

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch 2 times, most recently from badeef9 to 82775b6 Compare December 24, 2025 07:57

voonhous added 7 commits December 24, 2025 19:29

Remove AvroSchemaUtils from HiveAvroSerializer and HiveTypeUtils

ae97fc4

Address comments

7ccca27

Address comments

b7c4c62

Remove AvroSchemaUtils#resolveUnionSchema

bb57242

Add more test to cover generateColumnTypes

cbfa85a

Fix HiveTypeUtils#generateTypeInfo behaviour and ensure that it is no…

4763a70

…t changed

Address comments

03b8fb4

voonhous force-pushed the phase-17-P2-AvroSchemaUtils-removal branch from 82775b6 to 03b8fb4 Compare December 24, 2025 11:30

Increase Azure CI timeout

8ad0e47

voonhous merged commit 91c15b1 into apache:master Dec 24, 2025
390 of 417 checks passed

voonhous deleted the phase-17-P2-AvroSchemaUtils-removal branch December 24, 2025 18:19

voonhous linked an issue Jan 3, 2026 that may be closed by this pull request

Phase 17: AvroSchemaUtils Method Removal #14282

Closed

feat(schema): phase 17 - Remove AvroSchemaUtils usage (part 2) #17581

feat(schema): phase 17 - Remove AvroSchemaUtils usage (part 2) #17581

Uh oh!

Conversation

voonhous commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

voonhous commented Dec 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

voonhous commented Dec 19, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

the-other-tim-brown left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

voonhous commented Dec 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hudi-bot commented Dec 24, 2025

CI report:

Uh oh!

voonhous commented Dec 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

voonhous commented Dec 12, 2025 •

edited

Loading

voonhous Dec 23, 2025 •

edited

Loading

voonhous commented Dec 24, 2025 •

edited

Loading