[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUnsupportedOperationException #38387

panbingkun · 2022-10-25T11:19:30Z

What changes were proposed in this pull request?

This pr aims to replace UnsupportedOperationException with SparkUnsupportedOperationException.

Why are the changes needed?

1.When I work on https://issues.apache.org/jira/browse/SPARK-40889,
I found QueryExecutionErrors.unsupportedPartitionTransformError throw UnsupportedOperationException(but not SparkUnsupportedOperationException), it seem not to fit into the new error framework.

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala

Lines 71 to 72 in a27b459

    
           case transform => 
        
             throw QueryExecutionErrors.unsupportedPartitionTransformError(transform)

2.QueryExecutionErrors.unsupportedPartitionTransformError throw SparkUnsupportedOperationException, but UT catch UnsupportedOperationException.

spark/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala

Lines 288 to 301 in a27b459

    
               val transforms = Seq("years(ts)", "months(ts)", "days(ts)", "hours(ts)", "foo(a, 'bar', 34)") 
        
               transforms.foreach { transform => 
        
                 val query = 
        
                   s""" 
        
                      |CREATE TABLE my_tab(a INT, b STRING) USING parquet 
        
                      |PARTITIONED BY ($transform) 
        
                      """.stripMargin 
        
                 val ae = intercept[UnsupportedOperationException] { 
        
                   parseAndResolve(query) 
        
                 } 
        
                 assert(ae.getMessage 
        
                   .contains(s"Unsupported partition transform: $transform"))

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala

Lines 904 to 909 in a27b459

    
           def unsupportedPartitionTransformError( 
        
               transform: Transform): SparkUnsupportedOperationException = { 
        
             new SparkUnsupportedOperationException( 
        
               errorClass = "_LEGACY_ERROR_TEMP_2067", 
        
               messageParameters = Map("transform" -> transform.toString())) 
        
           }

spark/core/src/main/scala/org/apache/spark/SparkException.scala

Lines 144 to 154 in a27b459

    
           private[spark] class SparkUnsupportedOperationException( 
        
               errorClass: String, 
        
               messageParameters: Map[String, String]) 
        
             extends UnsupportedOperationException( 
        
               SparkThrowableHelper.getMessage(errorClass, messageParameters)) 
        
             with SparkThrowable { 
        
             override def getMessageParameters: java.util.Map[String, String] = messageParameters.asJava 
        
             override def getErrorClass: String = errorClass 
        
           }

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existed UT.

…supportedOperationException

panbingkun · 2022-10-25T11:39:52Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala

+          parseAndResolve(query)
+        },
+        errorClass = "_LEGACY_ERROR_TEMP_2067",
+        parameters = Map("transform" -> transform))


In fact, it throws SparkUnsupportedOperationException.

panbingkun · 2022-10-25T11:44:45Z

cc @MaxGekk

itholic

LGTM when CI pass

MaxGekk · 2022-10-26T08:31:46Z

core/src/main/resources/error/error-classes.json

      "Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting <autoBroadcastjoinThreshold> to -1 or increase the spark driver memory by setting <driverMemory> to a higher value<analyzeTblMsg>"
    ]
+  },
+  "_LEGACY_ERROR_TEMP_2251" : {


Let's introduce a appropriate name as you are here.

AmplabJenkins · 2022-10-26T11:41:23Z

Can one of the admins verify this patch?

…supportedOperationException

MaxGekk · 2022-10-27T07:29:17Z

core/src/main/resources/error/error-classes.json

          "Literal for '<value>' of <type>."
        ]
      },
+      "MULTIPLE_BUCKET_TRANSFORM" : {


MULTIPLE_BUCKET_TRANSFORM -> MULTIPLE_BUCKET_TRANSFORMS

MaxGekk · 2022-10-27T07:36:28Z

core/src/main/resources/error/error-classes.json

      },
+      "MULTIPLE_BUCKET_TRANSFORM" : {
+        "message" : [
+          "TRANSFORM on multiple bucket."


Multiple bucket TRANSFORMs

…supportedOperationException

MaxGekk

Waiting for CI.

MaxGekk · 2022-10-27T11:17:19Z

+1, LGTM. Merging to master.
Thank you, @panbingkun.

…supportedOperationException ### What changes were proposed in this pull request? This pr aims to replace UnsupportedOperationException with SparkUnsupportedOperationException. ### Why are the changes needed? 1.When I work on https://issues.apache.org/jira/browse/SPARK-40889, I found `QueryExecutionErrors.unsupportedPartitionTransformError` throw **UnsupportedOperationException**(but not **SparkUnsupportedOperationException**), it seem not to fit into the new error framework. https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Implicits.scala#L71-L72 2.`QueryExecutionErrors.unsupportedPartitionTransformError` throw SparkUnsupportedOperationException, but UT catch `UnsupportedOperationException`. https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala#L288-L301 https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala#L904-L909 https://github.com/apache/spark/blob/a27b459be3ca2ad2d50b9d793b939071ca2270e2/core/src/main/scala/org/apache/spark/SparkException.scala#L144-L154 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existed UT. Closes apache#38387 from panbingkun/replace_UnsupportedOperationException. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

### What changes were proposed in this pull request? This pr aims to upgrade Arrow from 14.0.2 to 15.0.0, this version fixes the compatibility issue with Netty 4.1.104.Final(GH-39265). Additionally, since the `arrow-vector` module uses `eclipse-collections` to replace `netty-common` as a compile-level dependency, Apache Spark has added a dependency on `eclipse-collections` after upgrading to use Arrow 15.0.0. ### Why are the changes needed? The new version brings the following major changes: Bug Fixes GH-34610 - [Java] Fix valueCount and field name when loading/transferring NullVector GH-38242 - [Java] Fix incorrect internal struct accounting for DenseUnionVector#getBufferSizeFor GH-38254 - [Java] Add reusable buffer getters to char/binary vectors GH-38366 - [Java] Fix Murmur hash on buffers less than 4 bytes GH-38387 - [Java] Fix JDK8 compilation issue with TestAllTypes GH-38614 - [Java] Add VarBinary and VarCharWriter helper methods to more writers GH-38725 - [Java] decompression in Lz4CompressionCodec.java does not set writer index New Features and Improvements GH-38511 - [Java] Add getTransferPair(Field, BufferAllocator, CallBack) for StructVector and MapVector GH-14936 - [Java] Remove netty dependency from arrow-vector GH-38990 - [Java] Upgrade to flatc version 23.5.26 GH-39265 - [Java] Make it run well with the netty newest version 4.1.104 The full release notes as follows: - https://arrow.apache.org/release/15.0.0.html ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #44797 from LuciferYang/SPARK-46718. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

568783b

…supportedOperationException

github-actions bot added CORE SQL labels Oct 25, 2022

panbingkun commented Oct 25, 2022

View reviewed changes

Merge branch 'master' into replace_UnsupportedOperationException

57dbbbb

itholic approved these changes Oct 26, 2022

View reviewed changes

MaxGekk requested changes Oct 26, 2022

View reviewed changes

panbingkun added 2 commits October 26, 2022 20:20

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

5b4a3a0

…supportedOperationException

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

a53338e

…supportedOperationException

panbingkun requested a review from MaxGekk October 26, 2022 12:25

panbingkun added 2 commits October 26, 2022 21:36

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

7ec8fd8

…supportedOperationException

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

cff7050

…supportedOperationException

MaxGekk reviewed Oct 27, 2022

View reviewed changes

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUn…

bda7402

…supportedOperationException

panbingkun requested a review from MaxGekk October 27, 2022 08:02

MaxGekk approved these changes Oct 27, 2022

View reviewed changes

MaxGekk closed this in e0f4410 Oct 27, 2022

panbingkun deleted the replace_UnsupportedOperationException branch November 7, 2022 01:53

LuciferYang mentioned this pull request Jan 23, 2024

[SPARK-46718][BUILD] Upgrade Arrow to 15.0.0 #44797

Closed

	case transform =>
	throw QueryExecutionErrors.unsupportedPartitionTransformError(transform)

	val transforms = Seq("years(ts)", "months(ts)", "days(ts)", "hours(ts)", "foo(a, 'bar', 34)")
	transforms.foreach { transform =>
	val query =
	s"""
	\|CREATE TABLE my_tab(a INT, b STRING) USING parquet
	\|PARTITIONED BY ($transform)
	""".stripMargin

	val ae = intercept[UnsupportedOperationException] {
	parseAndResolve(query)
	}

	assert(ae.getMessage
	.contains(s"Unsupported partition transform: $transform"))

	def unsupportedPartitionTransformError(
	transform: Transform): SparkUnsupportedOperationException = {
	new SparkUnsupportedOperationException(
	errorClass = "_LEGACY_ERROR_TEMP_2067",
	messageParameters = Map("transform" -> transform.toString()))
	}

	private[spark] class SparkUnsupportedOperationException(
	errorClass: String,
	messageParameters: Map[String, String])
	extends UnsupportedOperationException(
	SparkThrowableHelper.getMessage(errorClass, messageParameters))
	with SparkThrowable {

	override def getMessageParameters: java.util.Map[String, String] = messageParameters.asJava

	override def getErrorClass: String = errorClass
	}

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUnsupportedOperationException #38387

[SPARK-40910][SQL] Replace UnsupportedOperationException with SparkUnsupportedOperationException #38387

Uh oh!

Conversation

panbingkun commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

panbingkun Oct 25, 2022

Choose a reason for hiding this comment

Uh oh!

panbingkun Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Oct 25, 2022

Uh oh!

itholic left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk Oct 26, 2022

Choose a reason for hiding this comment

Uh oh!

panbingkun Oct 26, 2022

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Oct 26, 2022

Uh oh!

MaxGekk Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

panbingkun Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

MaxGekk Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

panbingkun Oct 27, 2022

Choose a reason for hiding this comment

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Oct 27, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

panbingkun commented Oct 25, 2022 •

edited

Loading

panbingkun Oct 25, 2022 •

edited

Loading