chore(ci): Hudi-utilities test improvements #17758

the-other-tim-brown · 2025-12-30T20:45:56Z

Describe the issue this Pull Request addresses

We see occasional timeouts with the utilities tests in Azure CI so this PR aims to improve the test runtimes to ensure CI is not flakey.

Summary and Changelog

Updates the KafkaTestUtils to use a test container instead of the deprecated streamer library. With this change, the test utils are setup once per class. This cut the runtime of TestHoodieDeltaStreamer from 31 minutes to 12 minutes on my local machine.
Streamers are shutdown in tests to ensure there are no lingering threadpools or resources leftover
Minor fixes include fixing logging, avoiding printing

Impact

Improve CI stability

Risk Level

None

Documentation Update

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

the-other-tim-brown · 2025-12-31T17:08:38Z

azure-pipelines-20230430.yml

              command: 'run'
              arguments: >
                -v $(Build.SourcesDirectory):/hudi
+                -v /var/run/docker.sock:/var/run/docker.sock


This is required for running docker in docker

the-other-tim-brown · 2025-12-31T17:09:11Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/testutils/KafkaTestUtils.java

+
+import scala.Tuple2;
+
+public class KafkaTestUtils {


Drop in replacement for the existing KafkaTestUtils but does not require zookeeper

the-other-tim-brown · 2025-12-31T17:09:48Z

...ities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java

      HoodieTableMetaClient meta = createMetaClient(storage, tablePath);
      HoodieTimeline timeline = meta.getActiveTimeline().getCommitAndReplaceTimeline().filterCompletedInstants();
-      LOG.info("Timeline Instants=" + meta.getActiveTimeline().getInstants());
+      LOG.info("Timeline Instants={}", meta.getActiveTimeline().getInstants());


Fixed logging to avoid the toString calls while I was updating this file

the-other-tim-brown · 2025-12-31T17:10:09Z

hudi-tests-common/pom.xml

            <scope>compile</scope>
        </dependency>
-        <dependency>
-            <groupId>org.testcontainers</groupId>


Moved this to be where it is required

the-other-tim-brown · 2025-12-31T17:10:37Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java

+        HoodieStreamer streamer = new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+        streamer.sync();
        successTables.add(Helpers.getTableWithDatabase(context));
+        streamer.shutdownGracefully();


Ensure the instances are shutdown properly

minor. should we move this to finally block?

Yes, makes sense

the-other-tim-brown · 2025-12-31T17:11:06Z

...es/src/test/java/org/apache/hudi/utilities/deltastreamer/multisync/TestMultipleMetaSync.java

    // Initial bulk insert to ingest to first hudi table
    HoodieDeltaStreamer.Config cfg = getConfig(tableBasePath, getSyncNames("MockSyncTool1", "MockSyncTool2"));
-    new HoodieDeltaStreamer(cfg, jsc, fs, hiveServer.getHiveConf()).sync();
+    syncOnce(new HoodieDeltaStreamer(cfg, jsc, fs, hiveServer.getHiveConf()));


Lots of changes in this PR are using utils or other means to ensure the streamer is shutdown

the-other-tim-brown · 2025-12-31T17:11:26Z

...ities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java

-
-  Map<String, String> hudiOpts = new HashMap<>();
-  public KafkaTestUtils testUtils;
+  protected static KafkaTestUtils testUtils;


Tests are now only setting up kafka once to cut down on the overhead

nsivabalan

nice job on the fixes and test run time reduction

nsivabalan · 2025-12-31T18:38:52Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java

+        HoodieStreamer streamer = new HoodieStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties()));
+        streamer.sync();
        successTables.add(Helpers.getTableWithDatabase(context));
+        streamer.shutdownGracefully();


minor. should we move this to finally block?

nsivabalan · 2025-12-31T18:42:44Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/BaseTestKafkaSource.java


-  @AfterEach
-  public void cleanupClass() {
+  @AfterAll


should we keep AfterEach and add testUtils.deleteTopics(); in that?

It generally doesn't make any impact since the topics are so small but I can add it

nsivabalan · 2025-12-31T18:42:52Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestAvroKafkaSource.java


-  @AfterEach
-  public void tearDown() {
+  @AfterAll


should we keep AfterEach and add testUtils.deleteTopics(); in that?

nsivabalan · 2025-12-31T18:43:02Z

...ies/src/test/java/org/apache/hudi/utilities/sources/debezium/TestAbstractDebeziumSource.java

    testUtils.setup();
  }

-  @AfterEach


should we keep AfterEach and add testUtils.deleteTopics(); in that?

nsivabalan · 2025-12-31T18:43:12Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/helpers/TestKafkaOffsetGen.java


-  @AfterEach
-  public void teardown() throws Exception {
+  @AfterAll


same comment as above.

hudi-bot · 2025-12-31T23:00:31Z

CI report:

6dfb53d Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

github-actions bot added size:M PR with lines of changes in (100, 300] size:L PR with lines of changes in (300, 1000] and removed size:M PR with lines of changes in (100, 300] labels Dec 30, 2025

the-other-tim-brown commented Dec 31, 2025

View reviewed changes

the-other-tim-brown changed the title ~~use kafka container for tests~~ chore(ci): Hudi-utilities test improvements Dec 31, 2025

the-other-tim-brown marked this pull request as ready for review December 31, 2025 17:15

nsivabalan reviewed Dec 31, 2025

View reviewed changes

the-other-tim-brown added 8 commits December 31, 2025 16:01

use kafka container for tests

b13d578

set args for docker in docker

57b418f

just docker.sock

157de4f

create test utils replacement, remove test jar from pom

2c5cfca

fix delete topic, ensure resources shutdown gracefully for sync

21f6279

shutdown more streamers

8f13b37

address pr feedback, remove more show/print

0e21622

update azure timeouts for slow runner infra

6dfb53d

the-other-tim-brown force-pushed the test-runtime-improvements branch from c02717e to 6dfb53d Compare December 31, 2025 21:02

nsivabalan approved these changes Jan 1, 2026

View reviewed changes

nsivabalan merged commit bd0ae66 into apache:master Jan 1, 2026
72 checks passed

chore(ci): Hudi-utilities test improvements #17758

chore(ci): Hudi-utilities test improvements #17758

Conversation

the-other-tim-brown commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Dec 31, 2025

CI report:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

the-other-tim-brown commented Dec 30, 2025 •

edited

Loading