Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Support RaiseError for Spark 4.0.0 #10969

Open
razajafri opened this issue Jun 4, 2024 · 1 comment · May be fixed by #10971
Open

[FEA] Support RaiseError for Spark 4.0.0 #10969

razajafri opened this issue Jun 4, 2024 · 1 comment · May be fixed by #10971
Assignees
@razajafri
Labels
feature request New feature or request Spark 4.0+ Spark 4.0+ issues

Comments

@razajafri
Copy link
Collaborator

razajafri commented Jun 4, 2024 •
edited by mythrocks
Loading

This PR #5487 added the ability to convert a UDF that can throw SparkException into a catalyst expression with RaiseError.
Support for RaiseError was added in #5540 but Apache Spark 4.0 changed fundamentally how it throws exceptions so we have to match that change.

We'd like to have GpuRaiseError so that we can prevent a columnar-to-row transition when an error should be raised.
@razajafri razajafri added feature request New feature or request ? - Needs Triage Need team to review and classify Spark 4.0+ Spark 4.0+ issues labels Jun 4, 2024
This was referenced Jun 4, 2024
Merged
Draft
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jun 4, 2024
@mythrocks mythrocks mentioned this issue Oct 28, 2024
Open
@mythrocks
Copy link
Collaborator

mythrocks commented Oct 28, 2024 •
edited
Loading

@razajafri, could you please confirm if this issue is necessary, if we already have #10107? Can we close this as a dupe? (Or vice versa.)

mythrocks added a commit to mythrocks/spark-rapids that referenced this issue Oct 28, 2024
@mythrocks
Fix misc_expr_test for [databricks] 14.3
40639b0 
Fixes NVIDIA#11537.

This commit addresses the failure of the `test_raise_error` test
in `misc_expr_test.py` for Databricks 14.3.

This is an extension of NVIDIA#11129, where this test was skipped for
Apache Spark 4.0.  The failure on Databricks 14.3 shares the same
cause as in Spark 4.0, i.e. a backward-incompatible Spark change
in the signature of RaiseError, as introduced in
https://issues.apache.org/jira/browse/SPARK-44838.

The work to support this change in a Spark-RAPIDS shim will be
tracked in NVIDIA#10969.  This test will be skipped until that work
is completed.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Oct 28, 2024
Merged
mythrocks added a commit that referenced this issue Nov 4, 2024
@mythrocks
Fix misc_expr_test for [databricks] 14.3 (#11670)
2e16ff2 
Fixes #11537.

This commit addresses the failure of the `test_raise_error` test
in `misc_expr_test.py` for Databricks 14.3.

This is an extension of #11129, where this test was skipped for
Apache Spark 4.0.  The failure on Databricks 14.3 shares the same
cause as in Spark 4.0, i.e. a backward-incompatible Spark change
in the signature of RaiseError, as introduced in
https://issues.apache.org/jira/browse/SPARK-44838.

The work to support this change in a Spark-RAPIDS shim will be
tracked in #10969.  This test will be skipped until that work
is completed.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Nov 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
feature request New feature or request Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Support for RaiseError for Spark 4.0.0 [databricks] razajafri/spark-rapids
3 participants
@mattahrens @mythrocks @razajafri