feat(schema): Phase 18 - HoodieAvroUtils removal (Part 1) #17599

voonhous · 2025-12-15T15:59:27Z

Describe the issue this Pull Request addresses

Reference issue: #14283

Remove methods that were migrated to HoodieSchemaUtils, consolidate remaining Avro-specific utilities, update documentation. The scope here only covers:

hudi-cli
hudi-client-common

Specific Ignored Usages

The following classes ignore HoodieAvroUtils:

HoodieCDCLogger
KeyGenUtils
TimestampBasedAvroKeyGenerator
TestHoodieAvroParquetWriter
RawTripTestPayloadKeyGenerator

Key Changes:

Migration: Swapping out HoodieAvroUtils wherever possible.
Full Qualification: For classes where migration was not immediately feasible, calls to HoodieAvroUtils static functions have been fully qualified. This explicitly marks technical debt and makes these usages easily searchable for future refactoring.

Summary and Changelog

Swap out HoodieAvroUtils to HoodieSchema equivalent.

Impact

None

Risk Level

Low

Documentation Update

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

voonhous · 2025-12-20T19:14:04Z

@hudi-bot run azure

the-other-tim-brown · 2025-12-21T22:59:53Z

...main/java/org/apache/hudi/client/transaction/ConcurrentSchemaEvolutionTableSchemaGetter.java

+        .map(this::handlePartitionColumnsIfNeeded);
+  }
+
+  public Option<Schema> getTableAvroSchemaIfPresent(boolean includeMetadataFields, Option<HoodieInstant> instant) {


Should we mark this as deprecated?

Yes, will remove this directly.

Will also do a Avro.Schema -> HoodieSchema migration for this class.

the-other-tim-brown · 2025-12-21T23:09:57Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java

    if (!historySchemaStr.isEmpty() || Boolean.parseBoolean(config.getString(HoodieCommonConfig.RECONCILE_SCHEMA.key()))) {
      InternalSchema internalSchema;
-      Schema avroSchema = HoodieAvroUtils.createHoodieWriteSchema(config.getSchema(), config.allowOperationMetadataField());
+      HoodieSchema schema = HoodieSchemaUtils.addMetadataFields(HoodieSchema.parse(config.getSchema()), config.allowOperationMetadataField());


Should we just use the HoodieSchemaUtils#createHoodieWriteSchema here?

Yeap! Defn!

the-other-tim-brown · 2025-12-21T23:11:28Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

    return sourceFields.stream().allMatch(fieldToIndex -> {
-      Schema schema = getNestedFieldSchemaFromWriteSchema(tableSchema, fieldToIndex);
-      return isSecondaryIndexSupportedType(schema);
+      Option<Pair<String, HoodieSchemaField>> schema = HoodieSchemaUtils.getNestedField(tableSchema, fieldToIndex);


If the option is empty, should we return false here?

Yeap, no harm being more defensive here.

I'd opt for throwing an error like what the original HoodieAvroUtils#createHoodieWriteSchema does. similar to the comment for line 154.

the-other-tim-brown · 2025-12-21T23:12:07Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

    return sourceFields.stream().anyMatch(fieldToIndex -> {
-      Schema schema = getNestedFieldSchemaFromWriteSchema(tableSchema, fieldToIndex);
-      return schema.getType() != Schema.Type.RECORD && schema.getType() != Schema.Type.ARRAY && schema.getType() != Schema.Type.MAP;
+      Option<Pair<String, HoodieSchemaField>> nestedFieldOpt = HoodieSchemaUtils.getNestedField(tableSchema, fieldToIndex);


Let's throw an exception if the option is not present?

the-other-tim-brown · 2025-12-21T23:13:10Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

+      case TIME:
+        return true;
+      case TIMESTAMP:
+        // LOCAL timestamps are not supported


Should we file a follow up ticket to add support for this?

Yeap, we can. I'm just transferring the test expectations to actual code since i don't recall seeing it documented anywhere other than tests here. (Have tagged u separately for this)

the-other-tim-brown · 2025-12-21T23:25:41Z

...lient/src/main/java/org/apache/hudi/client/bootstrap/HoodieSparkBootstrapSchemaProvider.java

    String recordNamespace = "hoodie." + tableName;

-    return AvroConversionUtils.convertStructTypeToAvroSchema(parquetSchema, structName, recordNamespace);
+    return HoodieSchema.fromAvroSchema(AvroConversionUtils.convertStructTypeToAvroSchema(parquetSchema, structName, recordNamespace));


You can use HoodieSchemaConversionUtils now to convert directly to HoodieSchema

the-other-tim-brown · 2025-12-21T23:26:45Z

hudi-common/src/main/java/org/apache/hudi/avro/AvroRecordContext.java

        return null;
      }
-      Object value = currentRecord.get(field.pos());
+      Object value = currentRecord.get(fieldOpt.get().pos());


nitpick: let's create a local variable field = fieldOpt.get()

the-other-tim-brown · 2025-12-21T23:29:28Z

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java

-      Schema nullableSchema = Schema.createUnion(Schema.create(Schema.Type.NULL),fieldSchema);
+  public static HoodieSchema createSchemaWithDefaultValue(TypeDescription orcSchema, String recordName, String namespace, boolean nullable) {
+    HoodieSchema hoodieSchema = createSchemaWithNamespace(orcSchema,recordName,namespace);
+    List<HoodieSchemaField> fields = new ArrayList<>();


nitpick: when we know the size of the list, we should initialize the array list with that size

voonhous · 2025-12-23T14:46:46Z

Note: this is a stacked PR, the base of this needs to be modified after #17581 is merged.

voonhous · 2025-12-23T15:40:59Z

hudi-client/hudi-client-common/src/test/java/org/apache/hudi/index/TestHoodieIndexUtils.java

-    Schema decimal = LogicalTypes.decimal(10, 2).addToSchema(Schema.create(Schema.Type.BYTES));
-    Schema uuid = LogicalTypes.uuid().addToSchema(Schema.create(Schema.Type.STRING));
-    Schema localTimestampMillis = LogicalTypes.localTimestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
-    Schema localTimestampMicros = LogicalTypes.localTimestampMicros().addToSchema(Schema.create(Schema.Type.LONG));


@the-other-tim-brown Original tests where it expects local-timestamp to be unsupported.

the-other-tim-brown · 2025-12-28T17:39:18Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+      case DECIMAL:
+      case TIME:
+      case TIMESTAMP:


I think we should break up the logical type method now that we can handle the types in the switch statement more easily and perform the checks here

Good catch, let me add that in and replace logicalTypeEquals accordingly.

the-other-tim-brown · 2025-12-29T03:50:19Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+    if (s1IsDecimal) {
+      HoodieSchema.Decimal d1 = (HoodieSchema.Decimal) s1;
+      HoodieSchema.Decimal d2 = (HoodieSchema.Decimal) s2;
+      // Check if both use same underlying representation (FIXED vs BYTES)


If they are both fixed, the fixed size should also be compared

Sorry, i pushed to the changes to wrong branch... Can you please review again.

the-other-tim-brown · 2025-12-29T03:51:05Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+    return logicalTypeSchemaEquals(s1, s2);
+  }
+
+  private static boolean logicalTypeSchemaEquals(HoodieSchema s1, HoodieSchema s2) {


I think we should break these up into functions for each type so we can directly call them from schemaEqualsInternal

Yeap, broken them up, i pushed to the wrong branch. (origin instead of voon). Changes should be reflected now, my bad.

the-other-tim-brown · 2025-12-29T14:37:34Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+      case DATE:
+      case UUID:
+        return logicalTypeSchemaEquals(s1, s2);


Can these two be moved up to use primitiveSchemaEquals?

the-other-tim-brown · 2025-12-29T14:39:10Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+ */
+public class HoodieSchemaComparatorForSchemaEvolution {
+
+  protected HoodieSchemaComparatorForSchemaEvolution() {


Let's make this private?

the-other-tim-brown · 2025-12-29T14:48:21Z

...rc/test/java/org/apache/hudi/common/schema/TestHoodieSchemaComparatorForSchemaEvolution.java

+        HoodieSchema.parse(timeMicros)
+    ));
+  }
+}


nitpick: newline at end of file

yihua · 2025-12-29T22:54:43Z

...ent-common/src/main/java/org/apache/hudi/client/bootstrap/HoodieBootstrapSchemaProvider.java

-      Schema userSchema = new Schema.Parser().parse(writeConfig.getSchema());
-      if (!HoodieAvroUtils.getNullSchema().equals(userSchema)) {
+      HoodieSchema userSchema = HoodieSchema.parse(writeConfig.getSchema());
+      if (!HoodieSchema.create(HoodieSchemaType.NULL).equals(userSchema)) {


nit: reuse HoodieSchema.NULL_SCHEMA

yihua · 2025-12-29T23:10:10Z

hudi-client/hudi-client-common/src/test/java/org/apache/hudi/index/TestHoodieIndexUtils.java

-        .name("localTimestampMillisField").type(localTimestampMillis).noDefault()
-        .name("localTimestampMicrosField").type(localTimestampMicros).noDefault()
-        .endRecord();
+        HoodieSchemaField.of("decimalField", decimal),


Should we test both FIXED and BYTES for decimal?

yihua · 2025-12-29T23:19:13Z

hudi-client/hudi-client-common/src/test/java/org/apache/hudi/index/TestHoodieIndexUtils.java

   */
  @Test
  public void testIsEligibleForExpressionIndexWithNullableFields() {
+    // An int with default 0 must have the int type defined first.


Is this a restriction of Avro?

yihua · 2025-12-29T23:20:24Z

hudi-client/hudi-client-common/src/test/java/org/apache/hudi/index/TestHoodieIndexUtils.java

  public void testIsEligibleForExpressionIndexWithNullableFields() {
+    // An int with default 0 must have the int type defined first.
+    // If null is defined first, which HoodieSchema#createNullable does, an error will be thrown
+    HoodieSchema nullableIntWithDefault = HoodieSchema.createUnion(HoodieSchema.create(HoodieSchemaType.INT), HoodieSchema.create(HoodieSchemaType.NULL));


Does HoodieSchema.createNullable work in this case, instead of calling HoodieSchema.createUnion?

Nope, it's a restriction of Avro, here's the error if we use HoodieSchema.createNullable:

yihua · 2025-12-29T23:25:55Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+      return schema.getType().hashCode();
+    }
+  }
+}


nit: let's add a new line at the end of the file

yihua · 2025-12-29T23:30:58Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+  private boolean fixedSchemaEquals(HoodieSchema s1, HoodieSchema s2) {
+    return validateFixed(s1, s2);
+  }


This API seems redundant?

Will pop validateFixed's logic to fixedSchemaEquals, i.e. validateFixed will be removed

yihua · 2025-12-30T00:57:24Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

-        return true; // Regular LONG
      case DOUBLE:
-        return true; // Support DOUBLE type
+      case DATE:


Test says FLOAT is supported but it seems that FLOAT type check is missing here?

public void testIsEligibleForSecondaryIndexWithUnsupportedDataTypes() { // Given: A schema with unsupported data types for secondary index (Boolean, Decimal) // Note: Float and Double are now supported

Let me verify this, the comment is a little confusing. It says: Float and Double are now supported, but the test itself for Float is a test for assertThrows but for Double, it's a assertDoesNotThrow. Might need to check separately with @linliu-code what's going on here.

Since comment and configs suggests that Float is supported, i will add a case Float.

@voonhous , i do not see there are any reason that Float cannot be supported. Please add a Float case.

Sure, fixed the test and enabled Float support.

yihua · 2025-12-30T01:08:51Z

.../java/org/apache/hudi/client/transaction/TestConcurrentSchemaEvolutionTableSchemaGetter.java

  @ParameterizedTest
  @MethodSource("schemaTestParams")
-  void testGetTableAvroSchema(Schema inputSchema, boolean includeMetadataFields, Schema expectedSchema) throws Exception {
+  void testGetTableAvroSchema(HoodieSchema inputSchema, boolean includeMetadataFields, HoodieSchema expectedSchema) throws Exception {


Rename the test methods so that there's no Avro in the method names?

yihua · 2025-12-30T01:12:55Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

   * @return true if each field's data types are supported, false otherwise
   */
-  public static boolean validateDataTypeForSecondaryOrExpressionIndex(List<String> sourceFields, Schema tableSchema) {
+  public static boolean validateDataTypeForSecondaryOrExpressionIndex(List<String> sourceFields, HoodieSchema tableSchema) {


We need to follow up separately why there are two methods: validateDataTypeForSecondaryIndex, validateDataTypeForSecondaryOrExpressionIndex. From the naming, they seem to overlap.

Sure, will add a sub task for this.

#17750

yihua · 2025-12-30T01:13:36Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java

+      Option<Pair<String, HoodieSchemaField>> schema = HoodieSchemaUtils.getNestedField(tableSchema, fieldToIndex);
+      if (schema.isEmpty()) {
+        throw new HoodieException("Failed to get schema. Not a valid field name: " + fieldToIndex);
+      }


Should be wrapped into a util getFieldOrThrow?

yihua · 2025-12-30T01:16:58Z

...on/src/main/java/org/apache/hudi/common/schema/HoodieSchemaComparatorForSchemaEvolution.java

+    List<String> symbols1 = s1.getEnumSymbols();
+    List<String> symbols2 = s2.getEnumSymbols();
+
+    // Quick size check before creating sets


nit: Comment is outdated mentioning sets, but List is used.

Fixed comment.

Additional compilation error fixes - AvroOrcUtils - HoodieBootstrapSchemaProvider - HoodieSparkBootstrapSchemaProvider - TestOrcBootstrap - Enable float support in secondary index - Fix test: test Secondary Index With All DataTypes - Fix test: testValidateDataTypeForSecondaryIndex

hudi-bot · 2025-12-31T06:21:36Z

CI report:

c94921b Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

voonhous · 2025-12-31T10:25:28Z

CI green, merging this in.

Additional compilation error fixes - AvroOrcUtils - HoodieBootstrapSchemaProvider - HoodieSparkBootstrapSchemaProvider - TestOrcBootstrap - Enable float support in secondary index - Fix test: test Secondary Index With All DataTypes - Fix test: testValidateDataTypeForSecondaryIndex

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from 4d983c9 to 9df0e72 Compare December 15, 2025 16:06

github-actions bot added the size:XL PR with lines of changes > 1000 label Dec 15, 2025

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch 2 times, most recently from 96738d6 to 4aadac8 Compare December 16, 2025 02:53

voonhous changed the title ~~feat(schema): Phase - 18 hoodie avro utils removal (hudi-client-common)~~ feat(schema): Phase 18 - hoodie avro utils removal (hudi-client-common) Dec 18, 2025

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch 2 times, most recently from f440c96 to 5f6daa5 Compare December 20, 2025 09:41

voonhous changed the title ~~feat(schema): Phase 18 - hoodie avro utils removal (hudi-client-common)~~ feat(schema): Phase 18 - HoodieAvroUtils removal (hudi-client-common) Dec 20, 2025

voonhous changed the title ~~feat(schema): Phase 18 - HoodieAvroUtils removal (hudi-client-common)~~ feat(schema): Phase 18 - HoodieAvroUtils removal (Part 1) Dec 20, 2025

voonhous mentioned this pull request Dec 20, 2025

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) #17656

Closed

3 tasks

the-other-tim-brown reviewed Dec 21, 2025

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from 5f6daa5 to b8b088e Compare December 23, 2025 14:45

voonhous changed the base branch from master to phase-17-P2-AvroSchemaUtils-removal December 23, 2025 14:45

voonhous commented Dec 23, 2025

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from 5a8eb8b to 762b1e3 Compare December 23, 2025 19:09

voonhous changed the base branch from phase-17-P2-AvroSchemaUtils-removal to master December 24, 2025 06:40

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch 3 times, most recently from 89da765 to a7fd9fa Compare December 27, 2025 08:23

the-other-tim-brown reviewed Dec 28, 2025

View reviewed changes

the-other-tim-brown reviewed Dec 29, 2025

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from 7cac821 to 1235700 Compare December 29, 2025 04:00

the-other-tim-brown reviewed Dec 29, 2025

View reviewed changes

the-other-tim-brown approved these changes Dec 29, 2025

View reviewed changes

yihua reviewed Dec 29, 2025

View reviewed changes

yihua reviewed Dec 30, 2025

View reviewed changes

voonhous mentioned this pull request Dec 30, 2025

Investigate why there are validateDataTypeForSecondaryIndex and validateDataTypeForSecondaryOrExpressionIndex #17750

Open

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from c477154 to 3bd409f Compare December 30, 2025 04:11

apache deleted a comment from hudi-bot Dec 30, 2025

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch 2 times, most recently from a4939d1 to 8224033 Compare December 31, 2025 02:17

the-other-tim-brown approved these changes Dec 31, 2025

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal branch from 8224033 to c94921b Compare December 31, 2025 03:53

voonhous mentioned this pull request Dec 31, 2025

chore: Fix CI reactor miss, causing cached .m2 snapshots to be used #17760

Closed

3 tasks

voonhous merged commit 5124ad7 into apache:master Dec 31, 2025
124 of 210 checks passed

voonhous mentioned this pull request Dec 31, 2025

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) #17763

Merged

3 tasks

voonhous linked an issue Jan 3, 2026 that may be closed by this pull request

Phase 18: HoodieAvroUtils Method Removal #14283

Closed

voonhous deleted the phase-18-HoodieAvroUtils-removal branch January 15, 2026 17:11

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 1) #17599

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 1) #17599

Uh oh!

Conversation

voonhous commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

voonhous commented Dec 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous commented Dec 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

voonhous commented Dec 15, 2025 •

edited

Loading

voonhous Dec 23, 2025 •

edited

Loading

voonhous Dec 29, 2025 •

edited

Loading