Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Write arrow hang with OutputWriter.path #876

Closed
jackylee-ch opened this issue Apr 28, 2022 · 0 comments
Closed

Write arrow hang with OutputWriter.path #876

jackylee-ch opened this issue Apr 28, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@jackylee-ch
Copy link
Contributor

Describe the bug
While we write with arrow format, the task thread will hang with OutputWriter.path.

"Executor task launch worker for task 0.0 in stage 2.0 (TID 2)" #182 daemon prio=5 os_prio=0 tid=0x00007fe734024000 nid=0x573c runnable [0x00007fe50cfe3000]
   java.lang.Thread.State: RUNNABLE
	at com.intel.oap.spark.sql.execution.datasources.arrow.ArrowFileFormat$$anon$1$$anon$2.path(ArrowFileFormat.scala:102)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.$anonfun$write$2(FileFormatDataWriter.scala:176)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.$anonfun$write$2$adapted(FileFormatDataWriter.scala:176)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter$$Lambda$2769/907714500.apply(Unknown Source)

To Reproduce
Example code:

spark.range(0, 100).write.arrow("/tmp/output")

Expected behavior
Finish the writing task

Additional context
None

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants