Skip to content

Conversation

@voonhous
Copy link
Member

@voonhous voonhous commented Dec 31, 2025

Describe the issue this Pull Request addresses

The CI workflow fails during various test stages with invalid compilation errors, even though the preceding "Build Project" stage completes successfully.

The issue:
Across several test jobs in bot.yml, the module list (-pl) is restricted to specific target modules (e.g., Spark or Flink modules), but the -am (also-make) flag is omitted. Consequently, Maven excludes core upstream modules like hudi-common from the active Reactor and instead resolves them as external dependencies from the local ~/.m2/repository.

Because GitHub Actions restores a cached version of the .m2 directory (IIRC, from the last successful run) at the start of the job, the test stage links against "stale" or incompatible JARs from a previous run rather than the fresh classes compiled in the current job's build stage.

Example Error from Logs

job-logs_1.txt

[ERROR] /home/runner/work/hudi/hudi/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestDataValidationCheckForLogCompactionActions.java:[188,92] incompatible types: org.apache.hudi.common.schema.HoodieSchema cannot be converted to org.apache.avro.Schema
[ERROR] /home/runner/work/hudi/hudi/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestOrcBootstrap.java:[168,24] cannot find symbol
  symbol:   method createSchemaWithDefaultValue(org.apache.orc.TypeDescription,java.lang.String,<nulltype>,boolean)

The above CI log is from:
https://github.com/apache/hudi/actions/runs/20611603231/job/59197291805?pr=17599

PR: #17599

Summary and Changelog

This PR adds the -am flag to mvn test and mvn verify commands throughout the CI workflow to ensure all project dependencies are consistently included in the Maven Reactor.

  • Before: mvn test ... -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES"
  • After: mvn test ... -am -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES"

Key Changes:

  • Updated Spark Java/Scala test jobs in .github/workflows/bot.yml to include -am in test steps.
  • Ensures Maven prioritizes freshly built classes in target/ folders over cached artifacts in ~/.m2/repository.
  • Maintains efficiency: Maven's incremental compilation skips re-building hudi-common since classes are already up to date from the previous build step.

Impact

CI reliability fix. Prevents phantom compilation failures caused by stale caches. No impact on public APIs or user-facing features.

Risk Level

None. This is a CI configuration change ensuring correct project dependency resolution.

Documentation Update

None

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

@voonhous voonhous force-pushed the fix-ci-test-compile-error branch from 3cb9de4 to 32837a7 Compare December 31, 2025 06:42
@github-actions github-actions bot added the size:S PR with lines of changes in (10, 100] label Dec 31, 2025
@hudi-bot
Copy link
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@voonhous voonhous marked this pull request as draft December 31, 2025 10:25
@voonhous voonhous closed this Dec 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S PR with lines of changes in (10, 100]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants