DBR 14.3 support

[chris-twiner]

frameless and Analyzer changes (nested object ResolveReferences uses new) - requires 4..0 rc's

[chris-twiner]

typelevel/frameless#787

[chris-twiner]

18) testProbabilityRulePass(com.sparkutils.qualityTests.RuleEngineTest)
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.Analyzer.ResolveReferences()Lorg/apache/spark/sql/catalyst/analysis/Analyzer$ResolveReferences$;
	at org.apache.spark.sql.QualitySparkUtils$.resolution(QualitySparkUtils.scala:125)
19) testSimpleProductionRules(com.sparkutils.qualityTests.RuleEngineTest)
java.lang.IllegalStateException: StructFieldsOperation.dataType should not be called.
	at com.sparkutils.quality.impl.util.StructFieldsOperation.dataType(StructFunctions.scala:66)
	at com.sparkutils.quality.impl.util.StructFieldsOperation.dataType$(StructFunctions.scala:65)
	at com.sparkutils.quality.impl.util.WithField.dataType(StructFunctions.scala:85)
	at com.databricks.sql.execution.VariantExpressionsCheck$.$anonfun$apply$4(VariantExpressionCheck.scala:58)
	at com.databricks.sql.execution.VariantExpressionsCheck$.$anonfun$apply$4$adapted(VariantExpressionCheck.scala:53)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:249)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:250)

[chris-twiner]

1) mapAggrDecimalDSLTest(com.sparkutils.qualityTests.AggregatesTest)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1511.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1511.0 (TID 19528) (ip-10-172-240-113.us-west-2.compute.internal executor driver): java.lang.ClassCastException: java.lang.Double cannot be cast to org.apache.spark.sql.types.Decimal

only one failure left.

[chris-twiner]

below is two new issues for 14.3...

https://issues.apache.org/jira/browse/SPARK-47509 has been introduced which stops the pattern used in Quality of nested subqueries with lambdas. For the intended use cases in Quality (now fails 3 tests) they should be sound:

scalarSubqueryAsOutputExpressionViaLambdaNonAttributeParam
scalarSubqueryAsOutputExpressionViaLambdaNoParam
scalarSubqueryAsOutputExpressionViaLambdaParam

these cases generate a correct join, this isn't the case for the transform example.

Additionally Databricks has further changed with another optimisation that occurs before extraOptimizations which is not yet in the oss, despite it's package name. This doesn't allow for StructFunctions to work, Quality should instead use the inbuilt ones for version > 3.1.0, which should be properly handled by Databricks and roll back the change for the previous custom version.

(these two failures do not occur on 14.3 LTS on CE - the two are not in sync).

[chris-twiner]

tests of RC3 and RC4 snapshot show full test completion on dbr 14.3 as of 19.04.24 - leaving open for RC testers issues

[chris-twiner]

closing - no issues raised on RC5, 14.3 also tests against 15.4 lts' (aside from error message differences).