This repository has been archived by the owner on Sep 18, 2023. It is now read-only.
Spark executor lost while DatasetFileWriter failed with speculation #891
Labels
bug
Something isn't working
Describe the bug
When spark speculation is true, the FinalTask may be killed as another attempt succeeded. In this case, DatasetFileWriter may failed and threw exception, which will cause executor failed.
To Reproduce
Write arrow with spark speculation and ArrowWriteExtension.
Expected behavior
Only warn log should be found, not losting executor.
Additional context
None.
The text was updated successfully, but these errors were encountered: