chore: Add hdfs feature test job #2350

wForget · 2025-09-09T03:06:35Z

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

codecov-commenter · 2025-09-09T03:28:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.44%. Comparing base (f09f8af) to head (cc529cb).
⚠️ Report is 501 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2350      +/-   ##
============================================
+ Coverage     56.12%   57.44%   +1.31%     
- Complexity      976     1297     +321     
============================================
  Files           119      147      +28     
  Lines         11743    13418    +1675     
  Branches       2251     2349      +98     
============================================
+ Hits           6591     7708    +1117     
- Misses         4012     4447     +435     
- Partials       1140     1263     +123

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wForget · 2025-09-11T09:11:43Z

.github/workflows/pr_build_linux.yml

+        with:
+          artifact_name: ${{ matrix.os }}-java-${{ matrix.java_version }}-features-${{ matrix.features.value }}-${{ github.run_id }}-${{ github.run_number }}-${{ github.run_attempt }}
+          features: ${{ matrix.features.value }}
+          maven_opts: "-Dtest=none -Dfeatures=${{ matrix.features.value }}"


-Dtest=none skip java tests

wForget · 2025-09-11T09:13:34Z

spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala

  }
+
+  def featureEnabled(feature: String): Boolean = {
+    System.getProperty("feature", "").split(",").contains(feature)


A temporary solution before #2372

should we wait #2372 ?

wForget · 2025-09-11T10:16:19Z

@parthchandra Could you please take a look?

comphead · 2025-09-11T15:17:07Z

spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromHdfsSuite.scala

+      {
+        val testFilePath = dir.toString
+        writeTestParquetFile(testFilePath)
+        withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_DATAFUSION) {


IMO we also should check native_iceberg_compat`

Thanks, I will add native_iceberg_compat mode

comphead · 2025-09-11T15:18:35Z

spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromFakeHadoopFsSuite.scala

  }

-  ignore("test native_datafusion scan on fake fs") {
+  test("test native_datafusion scan on fake fs") {


should we also test hdfs feature and native_iceberg_compat mode?

should we also test hdfs feature

hdfs feature does not seem to support non-hdfs scheme yet, as comment #2360 (comment) .

datafusion-comet/native/fs-hdfs/src/hdfs.rs

Lines 844 to 865 in 72eb0e9

fn get_namenode_uri(path: &str) -> Result<String, HdfsErr> {

match Url::parse(path) {

Ok(url) => match url.scheme() {

LOCAL_FS_SCHEME => Ok("file:///".to_string()),

HDFS_FS_SCHEME | VIEW_FS_SCHEME => {

if let Some(host) = url.host() {

let mut uri_builder = String::new();

write!(&mut uri_builder, "{}://{}", url.scheme(), host).unwrap();

if let Some(port) = url.port() {

write!(&mut uri_builder, ":{port}").unwrap();

}

Ok(uri_builder)

} else {

Err(HdfsErr::InvalidUrl(path.to_string()))

}

}

_ => Err(HdfsErr::InvalidUrl(path.to_string())),

},

Err(_) => Err(HdfsErr::InvalidUrl(path.to_string())),

}

}

Thats correct, yes

That's one of the fixes we wanted in fs-hdfs datafusion-contrib/fs-hdfs#29

Also, I think at some point it might be more reasonable to use openDAL instead of fs-hdfs (#2367)

parthchandra

Thanks for this PR @wForget. Should we wait to merge #2327 before we merge this?

parthchandra · 2025-09-12T00:43:45Z

spark/src/test/scala/org/apache/comet/parquet/ParquetReadFromFakeHadoopFsSuite.scala

  }

-  ignore("test native_datafusion scan on fake fs") {
+  test("test native_datafusion scan on fake fs") {


Also, I think at some point it might be more reasonable to use openDAL instead of fs-hdfs (#2367)

wForget · 2025-09-12T02:51:48Z

Thanks for this PR @wForget. Should we wait to merge #2327 before we merge this?

Thanks for the review. Do you mean #2372? If so, I think either way is fine — the current temporary solution won’t break CI.

parthchandra · 2025-09-13T00:49:07Z

oops. Yes, I did mean #2372. Let's get that merged and update the test in this PR.

parthchandra · 2025-09-13T00:56:46Z

Should we also be adding the test suites to pr_build_macos.yml ?

wForget · 2025-09-13T01:05:04Z

Should we also be adding the test suites to pr_build_macos.yml ?

Thanks, I will add it

comphead

it is LGTM thanks @wForget but probably should wait for #2372

wForget · 2025-09-16T06:05:03Z

Looks like a related failure https://github.com/apache/datafusion-comet/actions/runs/17752053952/job/50448541640?pr=2350, but I can't reproduce it, filed an issue #2400.

- test native_datafusion scan on fake fs *** FAILED *** (8 seconds, 838 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 6.0 failed 1 times, most recent failure: Lost task 2.0 in stage 6.0 (TID 15) (localhost executor driver): org.apache.spark.SparkException: Encountered error while reading file fake://fake-bucket/Users/runner/work/datafusion-comet/datafusion-comet/spark/target/tmp/comet_fake_3afe6cec-86bd-48ff-97a1-d354fec360327728661446900846429/data/test-file.parquet/part-00001-82d42ceb-6226-4ed6-8b34-d7108fc0b695-c000.snappy.parquet. Details:
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:864)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:296)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
	at org.apache.spark.sql.comet.CometScanExec$$anon$1.hasNext(CometScanExec.scala:271)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:60)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:165)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:164)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:212)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:164)
	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:162)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:213)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.comet.CometBatchIterator.hasNext(CometBatchIterator.java:60)
	at org.apache.comet.Native.executePlan(Native Method)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:165)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2$adapted(CometExecIterator.scala:164)
	at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:212)
	at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$1(CometExecIterator.scala:164)
	at org.apache.comet.Tracing$.withTrace(Tracing.scala:31)
	at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:162)
	at org.apache.comet.CometExecIterator.hasNext(CometExecIterator.scala:213)
	at org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriter.write(CometNativeShuffleWriter.scala:106)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.comet.CometNativeException: External: Generic IoError error: Unexpected (permanent) at , context: { expect: 8, actual: 0 } => reader got too little data
	at org.apache.comet.parquet.Native.readNextRecordBatch(Native Method)
	at org.apache.comet.parquet.NativeBatchReader.loadNextBatch(NativeBatchReader.java:812)
	at org.apache.comet.parquet.NativeBatchReader.nextBatch(NativeBatchReader.java:749)
	at org.apache.comet.parquet.NativeBatchReader.nextKeyValue(NativeBatchReader.java:707)
	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:286)
	... 36 more

parthchandra · 2025-09-16T16:27:14Z

I tried this PR earlier along with #2372 and did not see this failure either. I'll try again to see if I can reproduce it.

parthchandra

lgtm, but I cannot reproduce the failure in ci. Can anyone else try this?

wForget force-pushed the COMET-2298 branch 2 times, most recently from eefa69f to 1aa1d47 Compare September 9, 2025 10:10

wForget mentioned this pull request Sep 11, 2025

Enable feature specific unit tests #2360

Closed

wForget marked this pull request as ready for review September 11, 2025 08:01

wForget force-pushed the COMET-2298 branch from d197150 to 348342d Compare September 11, 2025 08:10

wForget commented Sep 11, 2025

View reviewed changes

comphead reviewed Sep 11, 2025

View reviewed changes

parthchandra reviewed Sep 12, 2025

View reviewed changes

mbutrovich requested review from parthchandra and comphead September 15, 2025 19:44

comphead approved these changes Sep 15, 2025

View reviewed changes

wForget added 11 commits September 16, 2025 09:57

chore: Add hdfs feature test job

3b3d704

fix and add hdfs suite

44c866b

skip java test

8c2736b

fix

1863538

disable hdrs async_file feature

4f341f5

wait for hdfs async blocking thread exit

832288b

revert disable hdrs async_file

fb8e808

fix

ac89480

clean tmp dir

a1fd391

fix comment

5004d8f

add hdfs test

2e54b7c

wForget added 5 commits September 16, 2025 09:58

fix

8e51b92

add native_iceberg_compat test

50f5873

test macos

c2f665f

fix

7211891

rebase

cc529cb

wForget force-pushed the COMET-2298 branch from d995642 to cc529cb Compare September 16, 2025 02:00

parthchandra approved these changes Sep 22, 2025

View reviewed changes

	fn get_namenode_uri(path: &str) -> Result<String, HdfsErr> {
	match Url::parse(path) {
	Ok(url) => match url.scheme() {
	LOCAL_FS_SCHEME => Ok("file:///".to_string()),
	HDFS_FS_SCHEME \| VIEW_FS_SCHEME => {
	if let Some(host) = url.host() {
	let mut uri_builder = String::new();
	write!(&mut uri_builder, "{}://{}", url.scheme(), host).unwrap();

	if let Some(port) = url.port() {
	write!(&mut uri_builder, ":{port}").unwrap();
	}
	Ok(uri_builder)
	} else {
	Err(HdfsErr::InvalidUrl(path.to_string()))
	}
	}
	_ => Err(HdfsErr::InvalidUrl(path.to_string())),
	},
	Err(_) => Err(HdfsErr::InvalidUrl(path.to_string())),
	}
	}

chore: Add hdfs feature test job #2350

Are you sure you want to change the base?

chore: Add hdfs feature test job #2350

Uh oh!

Conversation

wForget commented Sep 9, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

codecov-commenter commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wForget commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wForget Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parthchandra left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wForget commented Sep 12, 2025

Uh oh!

parthchandra commented Sep 13, 2025

Uh oh!

parthchandra commented Sep 13, 2025

Uh oh!

wForget commented Sep 13, 2025

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

wForget commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parthchandra commented Sep 16, 2025

Uh oh!

parthchandra left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov-commenter commented Sep 9, 2025 •

edited

Loading

wForget Sep 11, 2025 •

edited

Loading

wForget commented Sep 16, 2025 •

edited

Loading

parthchandra left a comment •

edited

Loading