feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) #17763

voonhous · 2025-12-31T14:19:47Z

Describe the issue this Pull Request addresses

Reference issue: #14283

Remove methods that were migrated to HoodieSchemaUtils, consolidate remaining Avro-specific utilities, update documentation. The scope here only covers:

hudi-hadoop-common

NOTE: Merge this after #17599

Key Changes:

Migration: Swapping out HoodieAvroUtils wherever possible.
Full Qualification: For classes where migration was not immediately feasible, calls to HoodieAvroUtils static functions have been fully qualified. This explicitly marks technical debt and makes these usages easily searchable for future refactoring.

TODOs:

Will need some suggestions on how to rename:

AvroOrcUtils
TestAvroOrcUtils

These will also need to be renamed:

hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroFileWriterFactory.java
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroHFileWriter.java
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroOrcReader.java
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroOrcWriter.java
hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroParquetSchemaConverter.java
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/HoodieAvroReadSupport.java
hudi-hadoop-common/src/main/java/org/apache/parquet/avro/NativeAvroSchemaConverter.java

I'll leave this as an exercise before actually performing the renames.

Summary and Changelog

Swap out HoodieAvroUtils to HoodieSchema equivalent.

Impact

None

Risk Level

Low

Documentation Update

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

the-other-tim-brown · 2025-12-31T14:35:43Z

hudi-common/src/main/java/org/apache/hudi/common/schema/HoodieSchemaUtils.java

+   * @return HoodieSchema containing record key and partition path fields
+   */
+  public static HoodieSchema getRecordKeyPartitionPathSchema() {
+    List<HoodieSchemaField> toBeAddedFields = new ArrayList<>();


super nitpick: initialize this with size 2

I keep forgetting about this, will try to keep this in mind in the future!

Addressed.

the-other-tim-brown · 2025-12-31T14:38:14Z

hudi-common/src/main/java/org/apache/hudi/common/util/FileFormatUtils.java

   * @return the Avro schema of the data file.
   */
-  public abstract Schema readAvroSchema(HoodieStorage storage, StoragePath filePath);
+  public abstract HoodieSchema readHoodieSchema(HoodieStorage storage, StoragePath filePath);


lets name this readSchema?

Tried doing this initially, but it conflicts with ParquetUtils#readSchema which returns a MessageType.

I will rename ParquetUtils#readSchema to ParquetUtils#readMessageType so that this abstract readHoodieSchema can be renamed to readSchema.

Addressed.

the-other-tim-brown · 2025-12-31T14:39:40Z

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java

   * @param type        ORC schema of the value Object.
   * @param colVector   The column vector to store the value Object.
-   * @param avroSchema  Avro schema of the value Object.
+   * @param schema  Avro schema of the value Object.


nit: Update the comment to fix alignment and remove Avro

the-other-tim-brown · 2025-12-31T14:43:57Z

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java

+          throw new IllegalStateException(
+              String.format("Unrecognized TimePrecision for: %s for Time type: %s", timeSchema.getPrecision(), timeSchema));
+        }
+      case TIMESTAMP:


It looks like we previously did not support translating local timestamps, should that be preserved here? They likely just get treated as longs

Yeap, we should preserve as this is a refactoring. Let's try not to change any behaviour first.

I will add a comment to this explicit handling too.

Will add a test for this too.

the-other-tim-brown · 2025-12-31T14:45:56Z

hudi-hadoop-common/src/main/java/org/apache/hudi/io/hadoop/OrcReaderIterator.java

-        .map(fieldName -> avroSchema.getField(fieldName).schema())
-        .toArray(Schema[]::new);
+    this.fieldSchemas = fieldNames.stream()
+        .map(fieldName -> this.schema.getField(fieldName).get().schema())


let's use orElseThrow if the field is missing instead of directly using get

the-other-tim-brown · 2025-12-31T14:47:34Z

hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java

+      fileSchema = Option.ofNullable(parquetUtils.readHoodieSchema(storage, path));
    }
-    return HoodieSchema.fromAvroSchema(fileSchema.get());
+    return fileSchema.get();


nit: this could be simplified to fileSchema.orElseGet(...

fileSchema is assigned to if it is empty.

Will modify it to:

fileSchema = fileSchema.or(() -> Option.ofNullable(parquetUtils.readSchema(storage, path))); return fileSchema.get();

Can you just use orElseGet?

Done, updated it to use orElseGet
.

the-other-tim-brown · 2025-12-31T14:49:38Z

...hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java

+    HoodieSchemaType type = schema.getType();
+    if (type == HoodieSchemaType.BOOLEAN) {


let's update this to a switch statement?

the-other-tim-brown · 2025-12-31T14:51:15Z

hudi-hadoop-common/src/test/java/org/apache/hudi/common/util/TestParquetUtils.java

+    List<HoodieSchemaField> toBeAddedFields = new ArrayList<>();
    for (String field: fields) {
-      Schema.Field schemaField =
-          new Schema.Field(field, METADATA_FIELD_SCHEMA, "", JsonProperties.NULL_VALUE);


Should we continue to use a constant here?

Yeap, addressed.

the-other-tim-brown · 2025-12-31T14:51:51Z

hudi-hadoop-common/src/test/java/org/apache/hudi/io/hadoop/TestHoodieBaseParquetWriter.java


-    Schema schema = new Schema.Parser().parse(HoodieTestDataGenerator.TRIP_EXAMPLE_SCHEMA);
-    HoodieAvroWriteSupport writeSupport = new HoodieAvroWriteSupport(new AvroSchemaConverter().convert(schema),
+    HoodieSchema schema = HoodieSchema.parse(HoodieTestDataGenerator.TRIP_EXAMPLE_SCHEMA);


This can be HoodieTestDataGenerator.HOODIE_SCHEMA

the-other-tim-brown · 2025-12-31T14:52:53Z

hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieHFileRecordReader.java

    reader = HoodieIOFactory.getIOFactory(HoodieStorageUtils.getStorage(path, storageConf)).getReaderFactory(HoodieRecord.HoodieRecordType.AVRO)
        .getFileReader(hoodieConfig, path, HoodieFileFormat.HFILE, Option.empty());

    //TODO boundary for now to revisit in later pr to use HoodieSchema


remove todo?

the-other-tim-brown · 2025-12-31T15:02:12Z

hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java

+    Schema repairedFileSchema = AvroSchemaRepair.repairLogicalTypes(getSchema().toAvroSchema(), schema.toAvroSchema());
    Option<Schema> promotedSchema = Option.empty();
-    if (!renamedColumns.isEmpty() || HoodieAvroUtils.recordNeedsRewriteForExtendedAvroTypePromotion(repairedFileSchema, schema)) {
+    if (!renamedColumns.isEmpty() || HoodieAvroUtils.recordNeedsRewriteForExtendedAvroTypePromotion(repairedFileSchema, schema.toAvroSchema())) {


Will recordNeedsRewriteForExtendedAvroTypePromotion be handled in another PR?

It's not handled in any of the current PRs now. I am still systematically trying to migrate what is easily migrated first.

Whatever that is left, will then evaluate if we should migrate them over. I haven't really analysed what this method entails.

the-other-tim-brown · 2025-12-31T15:04:22Z

...hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java

-      if (field.schema().getType().equals(Schema.Type.NULL)) {
+    for (HoodieSchemaField field : fields) {
+      if (field.schema().getType() == HoodieSchemaType.NULL) {
        continue; // Avro nulls are not encoded, unless they are null unions


nit: update comment to remove Avro

the-other-tim-brown · 2025-12-31T15:09:22Z

...hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java


-  private List<Type> convertFields(List<Schema.Field> fields, String schemaPath) {
+  private List<Type> convertFields(List<HoodieSchemaField> fields, String schemaPath) {
    List<Type> types = new ArrayList<Type>();


Nit: size this array to the incoming fields size

the-other-tim-brown · 2025-12-31T15:10:42Z

...hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java

-  private Schema convertFields(String name, List<Type> parquetFields, Map<String, Integer> names) {
-    List<Schema.Field> fields = new ArrayList<Schema.Field>();
+  private HoodieSchema convertFields(String name, List<Type> parquetFields, Map<String, Integer> names) {
+    List<HoodieSchemaField> fields = new ArrayList<>();


Similarly, size this list to the incoming list size

the-other-tim-brown · 2025-12-31T15:12:37Z

...hadoop-common/src/main/java/org/apache/parquet/avro/AvroSchemaConverterWithTimestampNTZ.java

+        return logicalTypeAnnotation.accept(new LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<HoodieSchema>() {
          @Override
-          public java.util.Optional<Schema> visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation listLogicalType) {
+          public java.util.Optional<HoodieSchema> visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation listLogicalType) {


While we're here, let's import Optional to shorten the line lengths in this file

We can't change this as java.util.Optional is a banned import according to our checkstyle rules.

We cannot change this to use our hudi common Option too as the method we are overriding calls for a java.util.Optional return type in: org.apache.parquet.schema.LogicalTypeAnnotation#accept

/** * Visits this logical type with the given visitor * * @param logicalTypeAnnotationVisitor the visitor to visit this type */ public abstract <T> Optional<T> accept(LogicalTypeAnnotationVisitor<T> logicalTypeAnnotationVisitor);

the-other-tim-brown · 2025-12-31T15:16:31Z

hudi-hadoop-common/src/test/java/org/apache/parquet/avro/TestAvroSchemaConverter.java

-    Schema schema = Schema.createRecord("record1", null, null, false);
-    Schema optionalIntArray = Schema.createArray(optional(Schema.create(INT)));
-    schema.setFields(Arrays.asList(new Schema.Field("myintarray", optionalIntArray, null, null)));
+    HoodieSchema optionalIntArray = HoodieSchema.createArray(HoodieSchema.createNullable(HoodieSchema.create(HoodieSchemaType.INT)));


Just a tip, you can directly use HoodieSchema.createNullable(HoodieSchemaType.INT) to shorten the setup in the tests

the-other-tim-brown · 2025-12-31T15:17:54Z

hudi-hadoop-common/src/test/java/org/apache/parquet/avro/TestAvroSchemaConverter.java

-        Arrays.asList(new Schema.Field("myarray", Schema.createArray(optional(innerRecord)), null, null)));
+    HoodieSchema optionalString = HoodieSchema.createNullable(HoodieSchema.create(HoodieSchemaType.STRING));
+    List<HoodieSchemaField> innerRecordFields = Arrays.asList(
+        HoodieSchemaField.of("s1", optionalString, null, JsonProperties.NULL_VALUE),


Should we update JsonProperties.NULL_VALUE to HoodieSchema.NULL_VALUE?

Yeap. Removed the import and updated the other references to JsonProperties.NULL_VALUE.

the-other-tim-brown · 2025-12-31T15:18:48Z

hudi-hadoop-common/src/test/java/org/apache/parquet/avro/TestAvroSchemaConverter.java

+    HoodieSchema listOfLists = HoodieSchema.createNullable(HoodieSchema.createArray(HoodieSchema.createArray(HoodieSchema.create(HoodieSchemaType.INT))));
+    HoodieSchema schema = HoodieSchema.createRecord("UnknownTwoLevelListInList", null, null, false,
+        Collections.singletonList(HoodieSchemaField.of("listOfLists", listOfLists, null, HoodieSchema.NULL_VALUE)));
    System.err.println("Avro schema: " + schema.toString(true));


let's remove this println and any others in the file while we're updating this file

Removed all:

System.err.println("Avro schema: " + schema.toString(true));

the-other-tim-brown · 2025-12-31T15:22:41Z

hudi-hadoop-common/src/test/java/org/apache/parquet/avro/TestAvroSchemaConverter.java

    assertEquals(
-        COMPATIBLE, checkReaderWriterCompatibility(fromAvro, toAvro).getType());
+        COMPATIBLE, checkReaderWriterCompatibility(fromSchema.toAvroSchema(), toSchema.toAvroSchema()).getType());


We can update this to assertTrue(HoodieSchemaCompatibility.areSchemasCompatible(fromSchema, toSchema)

the-other-tim-brown · 2026-01-01T21:56:36Z

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/AvroOrcUtils.java

   * @param unionVector       The vector to store value.
   * @param unionChildTypes   All possible types for the value Object.
-   * @param avroSchema        Avro union schema for the value Object.
+   * @param schema            Avro union schema for the value Object.


nit: remove Avro from the comment here

the-other-tim-brown · 2026-01-01T21:59:35Z

hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java


  @Override
  public ClosableIterator<IndexedRecord> getIndexedRecordIterator(HoodieSchema readerSchema, HoodieSchema requestedSchema) throws IOException {
    //TODO boundary for now to revisit in later pr to use HoodieSchema


Let's remove the TODO comments in this file

the-other-tim-brown · 2026-01-01T22:00:24Z

hudi-hadoop-common/src/main/java/org/apache/hudi/io/storage/hadoop/HoodieAvroParquetReader.java

+      fileSchema = Option.ofNullable(parquetUtils.readHoodieSchema(storage, path));
    }
-    return HoodieSchema.fromAvroSchema(fileSchema.get());
+    return fileSchema.get();


Can you just use orElseGet?

…NTZ.java

hudi-bot · 2026-01-02T09:34:49Z

CI report:

9650fda UNKNOWN
27be780 UNKNOWN
7d16660 UNKNOWN
8ddd0ef Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p3 branch from 1c8ac53 to 9650fda Compare December 31, 2025 14:26

voonhous changed the title ~~Remove HoodieAvroUtils and fixing up AvroSchemaConverterWithTimestamp…~~ feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) Dec 31, 2025

github-actions bot added the size:XL PR with lines of changes > 1000 label Dec 31, 2025

voonhous requested review from rahil-c, the-other-tim-brown and yihua and removed request for yihua December 31, 2025 14:37

the-other-tim-brown reviewed Dec 31, 2025

View reviewed changes

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p3 branch from 594dcdf to 2bc369b Compare January 1, 2026 16:54

the-other-tim-brown reviewed Jan 1, 2026

View reviewed changes

voonhous added 7 commits January 2, 2026 15:43

Remove HoodieAvroUtils and fixing up AvroSchemaConverterWithTimestamp…

989c65c

…NTZ.java

Update comments

9875e39

Fix checkstyle

6339a80

Address comments

69a35fb

Fix compilation error

00792fa

Addressed comments

3b4f3c7

Fix checkstyle

8ddd0ef

voonhous force-pushed the phase-18-HoodieAvroUtils-removal-p3 branch from bff2d71 to 8ddd0ef Compare January 2, 2026 07:43

		HoodieSchemaType type = schema.getType();
		if (type == HoodieSchemaType.BOOLEAN) {

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) #17763

Are you sure you want to change the base?

feat(schema): Phase 18 - HoodieAvroUtils removal (Part 2) #17763

Conversation

voonhous commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

voonhous Jan 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

voonhous commented Dec 31, 2025 •

edited

Loading

voonhous Jan 1, 2026 •

edited

Loading