Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] GpuFileFormatWriter does not close the data writer #1866

Closed
jlowe opened this issue Mar 4, 2021 · 0 comments · Fixed by #4525
Closed

[BUG] GpuFileFormatWriter does not close the data writer #1866

jlowe opened this issue Mar 4, 2021 · 0 comments · Fixed by #4525
Assignees
@NVnavkumar
Labels
bug Something isn't working good first issue Good for newcomers P2 Not required for release

Comments

@jlowe
Copy link
Member

jlowe commented Mar 4, 2021

GpuFileFormatWriter has code derived from Spark's FileFormatWriter, and SPARK-30227 added code to ensure the data writer is unconditionally closed when the write completes (either successfully or due to an error). GpuFileFormatWriter should do the same.

I don't believe our GPU writers rely on the close() method being invoked to free resources, but it would be good to fix this for both consistency with the Spark code and to cover the case where someone adds cleanup logic to a data writer close() method in the future.
@jlowe jlowe added bug Something isn't working good first issue Good for newcomers ? - Needs Triage Need team to review and classify labels Mar 4, 2021
@sameerz sameerz added P2 Not required for release and removed ? - Needs Triage Need team to review and classify labels Apr 6, 2021
@NVnavkumar NVnavkumar self-assigned this Jan 12, 2022
NVnavkumar added a commit to NVnavkumar/spark-rapids that referenced this issue Jan 13, 2022
@NVnavkumar
This fixes NVIDIA#1866
b3d99a4 
Make this more consistent with implementation of FileFormatWriter change implemented in SPARK-30227 by adding finallyBlock with close() call on dataWriter

Signed-off-by: Navin Kumar <navink@nvidia.com>
@NVnavkumar NVnavkumar mentioned this issue Jan 13, 2022
Merged
sameerz pushed a commit that referenced this issue Jan 14, 2022
@NVnavkumar
This fixes #1866 (#4525)
957da55 
Make this more consistent with implementation of FileFormatWriter change implemented in SPARK-30227 by adding finallyBlock with close() call on dataWriter

Signed-off-by: Navin Kumar <navink@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NVnavkumar NVnavkumar

Labels
bug Something isn't working good first issue Good for newcomers P2 Not required for release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Properly close data writer in GpuFileFormatWriter NVnavkumar/spark-rapids
3 participants
@jlowe @sameerz @NVnavkumar