[SPARK-44059] Add analyzer support of named arguments for built-in functions #41864
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
| layout: global | ||
| title: DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT error class | ||
| displayTitle: DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT error class | ||
| license: | | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --- | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Call to function `<functionName>` is invalid because it includes multiple argument assignments to the same parameter name `<parameterName>`. | ||
|
|
||
| This error class has the following derived error classes: | ||
|
|
||
| ## BOTH_POSITIONAL_AND_NAMED | ||
|
|
||
| A positional argument and named argument both referred to the same parameter. | ||
|
|
||
| ## DOUBLE_NAMED_ARGUMENT_REFERENCE | ||
|
|
||
| More than one named argument referred to the same parameter. | ||
|
|
||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -456,6 +456,14 @@ Found duplicate clauses: `<clauseName>`. Please, remove one of them. | |
|
|
||
| Found duplicate keys `<keyColumn>`. | ||
|
|
||
| ### [DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT](sql-error-conditions-duplicate-routine-parameter-assignment-error-class.html) | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Call to function `<functionName>` is invalid because it includes multiple argument assignments to the same parameter name `<parameterName>`. | ||
|
|
||
| For more details see [DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT](sql-error-conditions-duplicate-routine-parameter-assignment-error-class.html) | ||
|
|
||
| ### EMPTY_JSON_FIELD_VALUE | ||
|
|
||
| [SQLSTATE: 42604](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
@@ -1202,7 +1210,13 @@ SQLSTATE: none assigned | |
|
|
||
| Not allowed to implement multiple UDF interfaces, UDF class `<className>`. | ||
|
|
||
| ### NAMED_PARAMETERS_NOT_SUPPORTED | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Named parameters are not supported for function `<functionName>`; please retry the query with positional arguments to the function call instead. | ||
|
|
||
| ### NAMED_PARAMETER_SUPPORT_DISABLED | ||
|
|
||
| SQLSTATE: none assigned | ||
|
|
||
|
|
@@ -1507,6 +1521,12 @@ Failed to rename as `<sourcePath>` was not found. | |
|
|
||
| The `<clause>` clause may be used at most once per `<operation>` operation. | ||
|
|
||
| ### REQUIRED_PARAMETER_NOT_FOUND | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Cannot invoke function `<functionName>` because the parameter named `<parameterName>` is required, but the function call did not supply a value. Please update the function call to supply an argument value (either positionally or by name) and retry the query again. | ||
|
|
||
| ### REQUIRES_SINGLE_PART_NAMESPACE | ||
|
|
||
| [SQLSTATE: 42K05](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
@@ -1710,6 +1730,12 @@ Found an unclosed bracketed comment. Please, append */ at the end of the comment | |
|
|
||
| Parameter `<paramIndex>` of function `<functionName>` requires the `<requiredType>` type, however `<inputSql>` has the type `<inputType>`. | ||
|
|
||
| ### UNEXPECTED_POSITIONAL_ARGUMENT | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Cannot invoke function `<functionName>` because it contains positional argument(s) following named argument(s); please rearrange them so the positional arguments come first and then retry the query again. | ||
|
|
||
| ### UNKNOWN_PROTOBUF_MESSAGE_TYPE | ||
|
|
||
| SQLSTATE: none assigned | ||
|
|
@@ -1740,6 +1766,12 @@ Unpivot value columns must share a least common type, some types do not: [`<type | |
|
|
||
| All unpivot value columns must have the same size as there are value column names (`<names>`). | ||
|
|
||
| ### UNRECOGNIZED_PARAMETER_NAME | ||
|
|
||
| [SQLSTATE: 4274K](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
| Cannot invoke function `<functionName>` because the function call included a named argument reference for the argument named `<argumentName>`, but this function does not include any signature containing an argument with this name. Did you mean one of the following? [`<proposal>`]. | ||
|
|
||
| ### UNRECOGNIZED_SQL_TYPE | ||
|
|
||
| [SQLSTATE: 42704](sql-error-conditions-sqlstates.html#class-42-syntax-error-or-access-rule-violation) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,7 +29,7 @@ import org.apache.spark.sql.catalyst.FunctionIdentifier | |
| import org.apache.spark.sql.catalyst.expressions._ | ||
| import org.apache.spark.sql.catalyst.expressions.aggregate._ | ||
| import org.apache.spark.sql.catalyst.expressions.xml._ | ||
| import org.apache.spark.sql.catalyst.plans.logical.{FunctionBuilderBase, Generate, LogicalPlan, OneRowRelation, Range} | ||
| import org.apache.spark.sql.catalyst.trees.TreeNodeTag | ||
| import org.apache.spark.sql.errors.QueryCompilationErrors | ||
| import org.apache.spark.sql.types._ | ||
|
|
@@ -358,8 +358,8 @@ object FunctionRegistry { | |
| // misc non-aggregate functions | ||
| expression[Abs]("abs"), | ||
| expression[Coalesce]("coalesce"), | ||
| expressionBuilder("explode", ExplodeExpressionBuilder), | ||
| expressionGeneratorBuilderOuter("explode_outer", ExplodeExpressionBuilder), | ||
| expression[Greatest]("greatest"), | ||
| expression[If]("if"), | ||
| expression[Inline]("inline"), | ||
|
|
@@ -491,7 +491,7 @@ object FunctionRegistry { | |
| expression[CollectList]("collect_list"), | ||
| expression[CollectList]("array_agg", true, Some("3.3.0")), | ||
| expression[CollectSet]("collect_set"), | ||
| expressionBuilder("count_min_sketch", CountMinSketchAggExpressionBuilder), | ||
| expression[BoolAnd]("every", true), | ||
| expression[BoolAnd]("bool_and"), | ||
| expression[BoolOr]("any", true), | ||
|
|
@@ -823,7 +823,7 @@ object FunctionRegistry { | |
| castAlias("string", StringType), | ||
|
|
||
| // mask functions | ||
| expressionBuilder("mask", MaskExpressionBuilder), | ||
|
|
||
| // csv | ||
| expression[CsvToStructs]("from_csv"), | ||
|
|
@@ -887,13 +887,42 @@ object FunctionRegistry { | |
| since: Option[String] = None): (String, (ExpressionInfo, FunctionBuilder)) = { | ||
| val (expressionInfo, builder) = FunctionRegistryBase.build[T](name, since) | ||
| val newBuilder = (expressions: Seq[Expression]) => { | ||
| if (expressions.exists(_.isInstanceOf[NamedArgumentExpression])) { | ||
| throw QueryCompilationErrors.namedArgumentsNotSupported(name) | ||
| } | ||
| val expr = builder(expressions) | ||
| if (setAlias) expr.setTagValue(FUNC_ALIAS, name) | ||
| expr | ||
| } | ||
| (name, (expressionInfo, newBuilder)) | ||
| } | ||
|
|
||
| /** | ||
| * This method will be used to rearrange the arguments provided in function invocation | ||
| * in the order defined by the function signature given in the builder instance. | ||
| * | ||
| * @param name The name of the function | ||
| * @param builder The builder of the function expression | ||
| * @param expressions The argument list passed in function invocation | ||
| * @tparam T The class of the builder | ||
| * @return An argument list in positional order defined by the builder | ||
| */ | ||
| def rearrangeExpressions[T <: FunctionBuilderBase[_]]( | ||
| name: String, | ||
| builder: T, | ||
| expressions: Seq[Expression]) : Seq[Expression] = { | ||
| val rearrangedExpressions = if (!builder.functionSignature.isEmpty) { | ||
| val functionSignature = builder.functionSignature.get | ||
| builder.rearrange(functionSignature, expressions, name) | ||
| } else { | ||
| expressions | ||
| } | ||
| if (rearrangedExpressions.exists(_.isInstanceOf[NamedArgumentExpression])) { | ||
| throw QueryCompilationErrors.namedArgumentsNotSupported(name) | ||
| } | ||
| rearrangedExpressions | ||
| } | ||
|
|
||
| private def expressionBuilder[T <: ExpressionBuilder : ClassTag]( | ||
| name: String, | ||
| builder: T, | ||
|
|
@@ -902,7 +931,8 @@ object FunctionRegistry { | |
| val info = FunctionRegistryBase.expressionInfo[T](name, since) | ||
| val funcBuilder = (expressions: Seq[Expression]) => { | ||
| assert(expressions.forall(_.resolved), "function arguments must be resolved.") | ||
| val rearrangedExpressions = rearrangeExpressions(name, builder, expressions) | ||
| val expr = builder.build(name, rearrangedExpressions) | ||
| if (setAlias) expr.setTagValue(FUNC_ALIAS, name) | ||
| expr | ||
| } | ||
|
|
@@ -935,9 +965,22 @@ object FunctionRegistry { | |
|
|
||
| private def expressionGeneratorOuter[T <: Generator : ClassTag](name: String) | ||
| : (String, (ExpressionInfo, FunctionBuilder)) = { | ||
| val (_, (info, builder)) = expression[T](name) | ||
|
Contributor
There was a problem hiding this comment. It this change necessary?
Contributor
Author
There was a problem hiding this comment. I felt it was because the name I created had the exact same name as the variable above and I didn't want to cause confusion. Also, in other example functions, builder is the variable name used for this kind of object, so I thought it be good to clear up the ambigiuity. |
||
| val outerBuilder = (args: Seq[Expression]) => { | ||
| GeneratorOuter(builder(args).asInstanceOf[Generator]) | ||
| } | ||
| (name, (info, outerBuilder)) | ||
| } | ||
|
|
||
| private def expressionGeneratorBuilderOuter[T <: ExpressionBuilder : ClassTag] | ||
| (name: String, builder: T) : (String, (ExpressionInfo, FunctionBuilder)) = { | ||
| val info = FunctionRegistryBase.expressionInfo[T](name, since = None) | ||
| val outerBuilder = (args: Seq[Expression]) => { | ||
| val rearrangedArgs = | ||
| FunctionRegistry.rearrangeExpressions(name, builder, args) | ||
| val generator = builder.build(name, rearrangedArgs) | ||
| assert(generator.isInstanceOf[Generator]) | ||
| GeneratorOuter(generator.asInstanceOf[Generator]) | ||
| } | ||
| (name, (info, outerBuilder)) | ||
| } | ||
|
|
@@ -980,6 +1023,30 @@ object TableFunctionRegistry { | |
| (name, (info, (expressions: Seq[Expression]) => builder(expressions))) | ||
| } | ||
|
|
||
| /** | ||
| * A function used for table-valued functions to return a builder that | ||
| * when given input arguments, will return a function expression representing | ||
| * the table-valued functions. | ||
| * | ||
| * @param name Name of the function | ||
| * @param builder Object which will build the expression given input arguments | ||
| * @param since Time of implementation | ||
| * @tparam T Type of the builder | ||
| * @return A tuple of the function name, expression info, and function builder | ||
| */ | ||
| def generatorBuilder[T <: GeneratorBuilder : ClassTag]( | ||
|
Contributor
There was a problem hiding this comment. please add a comment for this method |
||
| name: String, | ||
| builder: T, | ||
| since: Option[String] = None): (String, (ExpressionInfo, TableFunctionBuilder)) = { | ||
| val info = FunctionRegistryBase.expressionInfo[T](name, since) | ||
| val funcBuilder = (expressions: Seq[Expression]) => { | ||
| assert(expressions.forall(_.resolved), "function arguments must be resolved.") | ||
| val rearrangedExpressions = FunctionRegistry.rearrangeExpressions(name, builder, expressions) | ||
| builder.build(name, rearrangedExpressions) | ||
| } | ||
| (name, (info, funcBuilder)) | ||
| } | ||
|
|
||
| def generator[T <: Generator : ClassTag](name: String, outer: Boolean = false) | ||
| : (String, (ExpressionInfo, TableFunctionBuilder)) = { | ||
| val (info, builder) = FunctionRegistryBase.build[T](name, since = None) | ||
|
|
@@ -999,8 +1066,8 @@ object TableFunctionRegistry { | |
|
|
||
| val logicalPlans: Map[String, (ExpressionInfo, TableFunctionBuilder)] = Map( | ||
| logicalPlan[Range]("range"), | ||
| generatorBuilder("explode", ExplodeGeneratorBuilder), | ||
| generatorBuilder("explode_outer", ExplodeOuterGeneratorBuilder), | ||
| generator[Inline]("inline"), | ||
| generator[Inline]("inline_outer", outer = true), | ||
| generator[JsonTuple]("json_tuple"), | ||
|
|
@@ -1022,6 +1089,28 @@ object TableFunctionRegistry { | |
| val functionSet: Set[FunctionIdentifier] = builtin.listFunction().toSet | ||
| } | ||
|
|
||
| /** | ||
| * This is a trait used for scalar valued functions that defines how their expression | ||
| * representations are constructed in [[FunctionRegistry]]. | ||
| */ | ||
| trait ExpressionBuilder extends FunctionBuilderBase[Expression] | ||
|
|
||
| /** | ||
| * This is a trait used for table valued functions that defines how their expression | ||
| * representations are constructed in [[TableFunctionRegistry]]. | ||
| */ | ||
| trait GeneratorBuilder extends FunctionBuilderBase[LogicalPlan] { | ||
| override final def build(funcName: String, expressions: Seq[Expression]) : LogicalPlan = { | ||
| Generate( | ||
| buildGenerator(funcName, expressions), | ||
| unrequiredChildIndex = Nil, | ||
| outer = isOuter, | ||
| qualifier = None, | ||
| generatorOutput = Nil, | ||
| child = OneRowRelation()) | ||
| } | ||
|
|
||
| def isOuter: Boolean | ||
|
|
||
| def buildGenerator(funcName: String, expressions: Seq[Expression]) : Generator | ||
learningchess2003 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.