Hossein/rebase master from upstream #15
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… K8S DNS Label Names ### What changes were proposed in this pull request? By default, the executor pod prefix is generated by the app name. It handles characters that match [^a-z0-9\\-] differently. The '.' and all whitespaces will be converted to '-', but other ones to empty string. Especially, characters like '_', '|' are commonly used as a word separator in many languages. According to the K8S DNS Label Names, see https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names, we can convert all special characters to `-`. For example, ``` scala> "xyz_abc_i_am_a_app_name_w/_some_abbrs".replaceAll("[^a-z0-9\\-]", "-").replaceAll("-+", "-") res11: String = xyz-abc-i-am-a-app-name-w-some-abbrs scala> "xyz_abc_i_am_a_app_name_w/_some_abbrs".replaceAll("\\s+", "-").replaceAll("\\.", "-").replaceAll("[^a-z0-9\\-]", "").replaceAll("-+", "-") res12: String = xyzabciamaappnamewsomeabbrs ``` ```scala scala> "time.is%the￥most$valuable_——————thing,it's about time.".replaceAll("[^a-z0-9\\-]", "-").replaceAll("-+", "-") res9: String = time-is-the-most-valuable-thing-it-s-about-time- scala> "time.is%the￥most$valuable_——————thing,it's about time.".replaceAll("\\s+", "-").replaceAll("\\.", "-").replaceAll("[^a-z0-9\\-]", "").replaceAll("-+", "-") res10: String = time-isthemostvaluablethingits-about-time- ``` ### Why are the changes needed? For better UX ### Does this PR introduce _any_ user-facing change? yes, the executor pod name might look better ### How was this patch tested? add new ones Closes apache#33171 from yaooqinn/SPARK-35969. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…but different sequence
### What changes were proposed in this pull request?
unionByName does not supports struct having same col names but different sequence
```
val df1 = Seq((1, Struct1(1, 2))).toDF("a", "b")
val df2 = Seq((1, Struct2(1, 2))).toDF("a", "b")
val unionDF = df1.unionByName(df2)
```
it gives the exception
`org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<c2:int,c1:int> <> struct<c1:int,c2:int> at the second column of the second table; 'Union false, false :- LocalRelation [_1#38, _2#39] +- LocalRelation _1#45, _2#46`
In this case the col names are same so this unionByName should have the support to check within in the Struct if col names are same it should not throw this exception and works.
after fix we are getting the result
```
val unionDF = df1.unionByName(df2)
scala> unionDF.show
+---+------+
| a| b|
+---+------+
| 1|{1, 2}|
| 1|{2, 1}|
+---+------+
```
### Why are the changes needed?
As per unionByName functionality based on name, does the union. In the case of struct this scenario was missing where all the columns names are same but sequence is different, so added this functionality.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added the unit test and also done the testing through spark shell
Closes apache#32972 from SaurabhChawla100/SPARK-35756.
Authored-by: SaurabhChawla <s.saurabhtim@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…e default timestamp type ### What changes were proposed in this pull request? Add a new configuration `spark.sql.timestampType`, which configures the default timestamp type of Spark SQL, including SQL DDL and Cast clause. Setting the configuration as `TIMESTAMP_NTZ` will use `TIMESTAMP WITHOUT TIME ZONE` as the default type while putting it as `TIMESTAMP_LTZ` will use `TIMESTAMP WITH LOCAL TIME ZONE`. The default value of the new configuration is TIMESTAMP_LTZ, which is consistent with previous Spark releases. ### Why are the changes needed? A new configuration for switching the default timestamp type as timestamp without time zone. ### Does this PR introduce _any_ user-facing change? No, it's a new feature. ### How was this patch tested? Unit test Closes apache#33176 from gengliangwang/newTsTypeConf. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request? This PR implemented the proposal per [design doc](https://docs.google.com/document/d/1RfFn2e9o_1uHJ8jFGsSakp-BZMizX1uRrJSybMe2a6M) for SPARK-35779. ### Why are the changes needed? Spark supports dynamic partition filtering that enables reusing parts of the query to skip unnecessary partitions in the larger table during joins. This optimization has proven to be beneficial for star-schema queries which are common in the industry. Unfortunately, dynamic pruning is currently limited to partition pruning during joins and is only supported for built-in v1 sources. As more and more Spark users migrate to Data Source V2, it is important to generalize dynamic filtering and expose it to all v2 connectors. Please, see the design doc for more information on this effort. ### Does this PR introduce _any_ user-facing change? Yes, this PR adds a new optional mix-in interface for `Scan` in Data Source V2. ### How was this patch tested? This PR comes with tests. Closes apache#32921 from aokolnychyi/dynamic-filtering-wip. Authored-by: Anton Okolnychyi <aokolnychyi@apple.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
…erations ### What changes were proposed in this pull request? Improve unit tests for data-type-based basic operations by: - removing redundant test cases - adding `astype` test for ExtensionDtypes ### Why are the changes needed? Some test cases for basic operations are duplicated after introducing data-type-based basic operations. The PR is proposed to remove redundant test cases. `astype` is not tested for ExtensionDtypes, which will be adjusted in this PR as well. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests. Closes apache#33095 from xinrong-databricks/datatypeops_test. Authored-by: Xinrong Meng <xinrong.meng@databricks.com> Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
…psWithState in Structured Streaming
### What changes were proposed in this pull request?
This PR aims to add support for specifying a user defined initial state for arbitrary structured streaming stateful processing using [flat]MapGroupsWithState operator.
### Why are the changes needed?
Users can load previous state of their stateful processing as an initial state instead of redoing the entire processing once again.
### Does this PR introduce _any_ user-facing change?
Yes this PR introduces new API
```
def mapGroupsWithState[S: Encoder, U: Encoder](
timeoutConf: GroupStateTimeout,
initialState: KeyValueGroupedDataset[K, S])(
func: (K, Iterator[V], GroupState[S]) => U): Dataset[U]
def flatMapGroupsWithState[S: Encoder, U: Encoder](
outputMode: OutputMode,
timeoutConf: GroupStateTimeout,
initialState: KeyValueGroupedDataset[K, S])(
func: (K, Iterator[V], GroupState[S]) => Iterator[U])
```
### How was this patch tested?
Through unit tests in FlatMapGroupsWithStateSuite
Closes apache#33093 from rahulsmahadev/flatMapGroupsWithState.
Authored-by: Rahul Mahadev <rahul.mahadev@databricks.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request?
Fixes decimal overflow issues for decimal average in ANSI mode, so that overflows throw an exception rather than returning null.
### Why are the changes needed?
Query:
```
scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._
scala> spark.conf.set("spark.sql.ansi.enabled", true)
scala> val df = Seq(
| (BigDecimal("10000000000000000000"), 1),
| (BigDecimal("10000000000000000000"), 1),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2),
| (BigDecimal("10000000000000000000"), 2)).toDF("decNum", "intNum")
df: org.apache.spark.sql.DataFrame = [decNum: decimal(38,18), intNum: int]
scala> val df2 = df.withColumnRenamed("decNum", "decNum2").join(df, "intNum").agg(mean("decNum"))
df2: org.apache.spark.sql.DataFrame = [avg(decNum): decimal(38,22)]
scala> df2.show(40,false)
```
Before:
```
+-----------+
|avg(decNum)|
+-----------+
|null |
+-----------+
```
After:
```
21/07/01 19:48:31 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 24)
java.lang.ArithmeticException: Overflow in sum of decimals.
at org.apache.spark.sql.errors.QueryExecutionErrors$.overflowInSumOfDecimalError(QueryExecutionErrors.scala:162)
at org.apache.spark.sql.errors.QueryExecutionErrors.overflowInSumOfDecimalError(QueryExecutionErrors.scala)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:349)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:499)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:502)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit test
Closes apache#33177 from karenfeng/SPARK-35955.
Authored-by: Karen Feng <karen.feng@databricks.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request? This is a follow up of apache#32961. This PR additionally sets the stack size in `build/mvn`. ### Why are the changes needed? We are still hitting `StackOverflowError` in Jenkins. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2/3064/console ``` [INFO] compiling 166 Scala sources and 19 Java sources to /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/sql/catalyst/target/scala-2.12/classes ... [ERROR] ## Exception when compiling 480 sources to /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2/sql/catalyst/target/scala-2.12/classes java.lang.StackOverflowError ``` This PR increases the JVM of `mvn` instead of the plugin. ``` $ MAVEN_OPTS="-XX:+PrintFlagsFinal" build/mvn clean | grep 'intx ThreadStackSize' intx ThreadStackSize = 2048 {pd product} $ MAVEN_OPTS="-Xss128m -XX:+PrintFlagsFinal" build/mvn clean | grep 'intx ThreadStackSize' intx ThreadStackSize := 131072 {pd product} ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? N/A Closes apache#33180 from dongjoon-hyun/SPARK-35825. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? Add the functionality of cleaning up files of old versions for the RocksDB instance and RocksDBFileManager. ### Why are the changes needed? Part of the implementation of RocksDB state store. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New UT added. Closes apache#32933 from xuanyuanking/SPARK-35785. Authored-by: Yuanjian Li <yuanjian.li@databricks.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
…ition coalescing ### What changes were proposed in this pull request? By default, AQE will set `COALESCE_PARTITIONS_MIN_PARTITION_NUM` to the spark default parallelism, which is usually quite big. This is to keep the parallelism on par with non-AQE, to avoid perf regressions. However, this usually leads to many small/empty partitions, and hurts performance (although not worse than non-AQE). Users usually blindly set `COALESCE_PARTITIONS_MIN_PARTITION_NUM` to 1, which makes this config quite useless. This PR adds a new config to set the min partition size, to avoid too small partitions after coalescing. By default, Spark will not respect the target size, and only respect this min partition size, to maximize the parallelism and avoid perf regression in AQE. This PR also adds a bool config to respect the target size when coalescing partitions, and it's recommended to set it to get better overall performance. This PR also deprecates the `COALESCE_PARTITIONS_MIN_PARTITION_NUM` config. ### Why are the changes needed? AQE is default on now, we should make the perf better in the default case. ### Does this PR introduce _any_ user-facing change? yes, a new config. ### How was this patch tested? new tests Closes apache#33172 from cloud-fan/aqe2. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
… precision ### What changes were proposed in this pull request? We should use `check_exact=False` because the value check in `StatsTest.test_cov_corr_meta` is too strict. ### Why are the changes needed? In some environment, the precision could be different in pandas' `DataFrame.corr` function and the test `StatsTest.test_cov_corr_meta` fails. ``` AssertionError: DataFrame.iloc[:, 0] (column name="a") are different DataFrame.iloc[:, 0] (column name="a") values are different (14.28571 %) [index]: [a, b, c, d, e, f, g] [left]: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0] [right]: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.807406715958909e-17] ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Modified tests should still pass. Closes apache#33179 from ueshin/issuse/SPARK-35981/corr. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade Apache ORC to 1.6.9. ### Why are the changes needed? This is required to bring ORC-804 in order to fix ORC encryption masking bug. ### Does this PR introduce _any_ user-facing change? No. This is not released yet. ### How was this patch tested? Pass the newly added test case. Closes apache#33189 from dongjoon-hyun/SPARK-35992. Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to publish snapshot artifacts from branch-3.2 additionally. ### Why are the changes needed? `GitHub Action`'s cronjob feature is only supported in the default branch. So, to have a daily job, we should add here. Currently, it's publishing master and 3.1. - https://github.com/apache/spark/actions/workflows/publish_snapshot.yml <img width="273" alt="Screen Shot 2021-07-02 at 10 22 41 AM" src="https://user-images.githubusercontent.com/9700541/124309380-7c407400-db1f-11eb-9aa4-30db61a72b80.png"> ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? N/A Closes apache#33192 from dongjoon-hyun/SPARK-35994. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This patch ignores the test "ensure that concurrent update and cleanup consistent versions" in apache#32933. The test is currently flaky and we will address it later. ### Why are the changes needed? Unblock other developments. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests. Closes apache#33195 from viirya/ignore-rocksdb-test. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR removes sbt-avro plugin dependency. In the current master, Build with SBT depends on the plugin but it seems never used. Originally, the plugin was introduced for `flume-sink` in SPARK-1729 (apache#807) but `flume-sink` is no longer in Spark repository. After SBT was upgraded to 1.x in SPARK-21708 (apache#29286), `avroGenerate` part was introduced in `object SQL` in `SparkBuild.scala`. It's confusable but I understand `Test / avroGenerate := (Compile / avroGenerate).value` is for suppressing sbt-avro for `sql` sub-module. In fact, Test/compile will fail if `Test / avroGenerate :=(Compile / avroGenerate).value` is commented out. `sql` sub-module contains `parquet-compat.avpr` and `parquet-compat.avdl` but according to `sql/core/src/test/README.md`, they are intended to be handled by `gen-avro.sh`. Also, in terms of Maven build, there seems to be no definition to handle `*.avpr` or `*.avdl`. Based on the above, I think we can remove `sbt-avro`. ### Why are the changes needed? If `sbt-avro` is really no longer used, it's confusable that `sbt-avro` related configurations are in `SparkBuild.scala`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? GA. Closes apache#33190 from sarutak/remove-avro-from-sbt. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to update `master` branch version to 3.3.0-SNAPSHOT. ### Why are the changes needed? Start to prepare Apache Spark 3.3.0 and the published snapshot version should not conflict with `branch-3.2`. ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Pass the CIs. Closes apache#33196 from dongjoon-hyun/SPARK-35996. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…icient ### What changes were proposed in this pull request? This PR uses 2 ideas to make `EquivalentExpressions` more efficient: 1. do not keep all the equivalent expressions, we only need a count 2. track the "height" of common subexpressions, to quickly do child-parent sort, and filter out non-child expressions in `addCommonExprs` This PR also fixes several small bugs (exposed by the refactoring), please see PR comments. ### Why are the changes needed? code cleanup and small perf improvement ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? existing tests Closes apache#33142 from cloud-fan/codegen. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
### What changes were proposed in this pull request? Fix the type hint for `pyspark.rdd .RDD.histogram`'s `buckets` argument ### Why are the changes needed? The current type hint is incomplete.  From `pyspark.rdd .RDD.histogram`'s source: ```python if isinstance(buckets, int): ... elif isinstance(buckets, (list, tuple)): ... else: raise TypeError("buckets should be a list or tuple or number(int or long)") ``` ### Does this PR introduce _any_ user-facing change? Fixed the warning displayed above. ### How was this patch tested? Fixed warning above with this change. Closes apache#33185 from tpvasconcelos/master. Authored-by: Tomas Pereira de Vasconcelos <tomasvasconcelos1@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Add `daemon={true|false}` to the POSSIBLE THREAD LEAK IN SUITE warning printed by test framework.
### Why are the changes needed?
This is to slightly accelerate interpretation of that warning, since non-daemon threads can block the process from exiting and are likely to be problematic.
Only affects test code.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually ran some tests, inspected the output log line.
Closes apache#33178 from timarmstrong/thread-leak.
Authored-by: Tim Armstrong <tim.armstrong@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? When I'm running the benchmark in GA, I met the below error. https://github.com/pingsutw/spark/runs/2867617238?check_suite_focus=true ``` java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.j ava:1692)java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 21/06/20 07:40:02 ERROR SparkContext: Error initializing SparkContext.java.lang.AssertionError: assertion failed: spark.test.home is not set! at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.deploy.worker.Worker.<init> (Worker.scala:148) at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:954) at org.apache.spark.deploy.LocalSparkCluster.$anonfun$start$2(LocalSparkCluster.scala:68) at org.apache.spark.deploy.LocalSparkCluster.$anonfun$start$2$adapted(LocalSparkCluster.scala:65) at scala.collection.immutable.Range.foreach(Range.scala:158) at org.apache.spark.deploy.LocalSparkCluster.start(LocalSparkCluster.scala:65) at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2954) at org.apache.spark.SparkContext.<init>(SparkContext.scala:559) at org.apache.spark.SparkContext.<init> (SparkContext.scala:137) at org.apache.spark.serializer.KryoSerializerBenchmark$.createSparkContext(KryoSerializerBenchmark.scala:86) at org.apache.spark.serializer.KryoSerializerBenchmark$.sc$lzycompute$1(KryoSerializerBenchmark.scala:58) at org.apache.spark.serializer.KryoSerializerBenchmark$.sc$1(KryoSerializerBenchmark.scala:58) at org.apache.spark.serializer.KryoSerializerBenchmark$.$anonfun$run$3(KryoSerializerBenchmark.scala:63) ``` ### Why are the changes needed? Set `spark.test.home` in the benchmark workflow. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Rerun the benchmark in my fork. https://github.com/pingsutw/spark/actions/runs/996067851 Closes apache#33203 from pingsutw/SPARK-36007. Lead-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
… being copied ### What changes were proposed in this pull request? Make the ANSI flag part of expressions `Sum` and `Average`'s parameter list, instead of fetching it from the sessional SQLConf. ### Why are the changes needed? For Views, it is important to show consistent results even the ANSI configuration is different in the running session. This is why many expressions like 'Add'/'Divide' making the ANSI flag part of its case class parameter list. We should make it consistent for the expressions `Sum` and `Average` ### Does this PR introduce _any_ user-facing change? Yes, the `Sum` and `Average` inside a View always behaves the same, independent of the ANSI model SQL configuration in the current session. ### How was this patch tested? Existing UT Closes apache#33186 from gengliangwang/sumAndAvg. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…types are year-month intervals
### What changes were proposed in this pull request?
This PR fixes two issues. One is that `to_json` doesn't support `map` types where value types are `year-month` interval types like:
```
spark-sql> select to_json(map('a', interval '1-2' year to month));
21/07/02 11:38:15 ERROR SparkSQLDriver: Failed in [select to_json(map('a', interval '1-2' year to month))]
java.lang.RuntimeException: Failed to convert value 14 (class of class java.lang.Integer) with the type of YearMonthIntervalType(0,1) to JSON.
```
The other issue is that even if the issue of `to_json` is resolved, `from_json` doesn't support to convert `year-month` interval string to JSON. So the result of following query will be `null`.
```
spark-sql> select from_json(to_json(map('a', interval '1-2' year to month)), 'a interval year to month');
{"a":null}
```
### Why are the changes needed?
There should be no reason why year-month intervals cannot used as map value types.
`CalendarIntervalTypes` can do it.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
Closes apache#33181 from sarutak/map-json-yminterval.
Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request? This PR proposes to upgrade sbt-antlr4 from 0.8.2 to 0.8.3 per the guides at https://github.com/ihji/sbt-antlr4 I can't find an official proper docs for this. ### Why are the changes needed? To stick to the guides in https://github.com/ihji/sbt-antlr4, and leverage the fixes included. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI in this PR should tests it out. Closes apache#33208 from HyukjinKwon/SPARK-36010. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>
### What changes were proposed in this pull request? This PR aims to upgrade Dropwizard Metrics from `4.2.0` to `4.2.2`. ### Why are the changes needed? Dropwizard `4.2.1` fixes a bug related to `JMXReporter` but `4.2.1` also contains a bug. so upgrading to `4.2.2` seems better. https://github.com/dropwizard/metrics/releases/tag/v4.2.1 https://github.com/dropwizard/metrics/releases/tag/v4.2.2 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI. Closes apache#33209 from sarutak/upgrade-metrics-4.2.2. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Current AQE has cost evaluator to decide whether to use new plan after replanning. The current used evaluator is `SimpleCostEvaluator` to make decision based on number of shuffle in the query plan. This is not perfect cost evaluator, and different production environments might want to use different custom evaluators. E.g., sometimes we might want to still do skew join even though it might introduce extra shuffle (trade off resource for better latency), sometimes we might want to take sort into consideration for cost as well. Take our own setting as an example, we are using a custom remote shuffle service (Cosco), and the cost model is more complicated. So We want to make the cost evaluator to be pluggable, and developers can implement their own `CostEvaluator` subclass and plug in dynamically based on configuration. The approach is to introduce a new config to allow define sub-class name of `CostEvaluator` - `spark.sql.adaptive.customCostEvaluatorClass`. And add `CostEvaluator.instantiate` to instantiate the cost evaluator class in `AdaptiveSparkPlanExec.costEvaluator`. ### Why are the changes needed? Make AQE cost evaluation more flexible. ### Does this PR introduce _any_ user-facing change? No but an internal config is introduced - `spark.sql.adaptive.customCostEvaluatorClass` to allow custom implementation of `CostEvaluator`. ### How was this patch tested? Added unit test in `AdaptiveQueryExecSuite.scala`. Closes apache#32944 from c21/aqe-cost. Authored-by: Cheng Su <chengsu@fb.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
… REPARTITION_BY_COL in AQE ### What changes were proposed in this pull request? Skip remove shuffle if it's shuffle origin is not `REPARTITION_BY_COL` in AQE. ### Why are the changes needed? `REPARTITION_BY_COL` doesn't guarantee the output partitioning number so we can remove it safely in AQE. For `REPARTITION_BY_NUM`, we should retain the shuffle which partition number is specified by user. For `REBALANCE_PARTITIONS_BY_COL`, it is a special shuffle used to rebalance partitions so we should not remove it. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test Closes apache#33188 from ulysses-you/SPARK-35989. Lead-authored-by: ulysses-you <ulyssesyou18@gmail.com> Co-authored-by: ulysses <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
… to Maven's ### What changes were proposed in this pull request? This PR is a followup of apache#31019 that forgot to update SBT's to match. ### Why are the changes needed? To use the same version in both Maven and SBT. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should test them. Closes apache#33207 from HyukjinKwon/SPARK-33996. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…s properly
### What changes were proposed in this pull request?
This PR fixes an issue that `from_csv/to_csv` doesn't handle year-month intervals properly.
`from_csv` throws exception if year-month interval types are given.
```
spark-sql> select from_csv("interval '1-2' year to month", "a interval year to month");
21/07/03 04:32:24 ERROR SparkSQLDriver: Failed in [select from_csv("interval '1-2' year to month", "a interval year to month")]
java.lang.Exception: Unsupported type: interval year to month
at org.apache.spark.sql.errors.QueryExecutionErrors$.unsupportedTypeError(QueryExecutionErrors.scala:775)
at org.apache.spark.sql.catalyst.csv.UnivocityParser.makeConverter(UnivocityParser.scala:224)
at org.apache.spark.sql.catalyst.csv.UnivocityParser.$anonfun$valueConverters$1(UnivocityParser.scala:134)
```
Also, `to_csv` doesn't handle year-month interval types properly though any exception is thrown.
The result of `to_csv` for year-month interval types is not ANSI interval compliant form.
```
spark-sql> select to_csv(named_struct("a", interval '1-2' year to month));
14
```
The result above should be `INTERVAL '1-2' YEAR TO MONTH`.
### Why are the changes needed?
Bug fix.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
Closes apache#33210 from sarutak/csv-yminterval.
Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request? This PR fix the incorrect comment for `TimestampNTZType`. ### Why are the changes needed? Fix the incorrect comment ### Does this PR introduce _any_ user-facing change? 'No'. ### How was this patch tested? No need. Closes apache#33218 from beliefer/SPARK-35664-followup. Authored-by: gengjiaan <gengjiaan@360.cn> Signed-off-by: Gengliang Wang <gengliang@apache.org>
…efault timestamp type ### What changes were proposed in this pull request? For the timestamp literal, it should have the following behavior. 1. When `spark.sql.timestampType` is TIMESTAMP_NTZ: if there is no time zone part, return timestamp without time zone literal; otherwise, return timestamp with local time zone literal 2. When `spark.sql.timestampType` is TIMESTAMP_LTZ: return timestamp with local time zone literal ### Why are the changes needed? When the default timestamp type is TIMESTAMP_NTZ, the result of type literal should return TIMESTAMP_NTZ when there is no time zone part in the string. From setion 5.3 "literal" of ANSI SQL standard 2011: ``` 27) The declared type of a <timestamp literal> that does not specify <time zone interval> is TIMESTAMP(P) WITHOUT TIME ZONE, where P is the number of digits in <seconds fraction>, if specified, and 0 (zero) otherwise. The declared type of a <timestamp literal> that specifies <time zone interval> is TIMESTAMP(P) WITH TIME ZONE, where P is the number of digits in <seconds fraction>, if specified, and 0 (zero) otherwise. ``` Since we don't have "timestamp with time zone", we use timestamp with local time zone instead. ### Does this PR introduce _any_ user-facing change? No, the new timestmap type and the default timestamp configuration is not released yet. ### How was this patch tested? Unit test Closes apache#33215 from gengliangwang/tsLiteral. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>
…h temporary views ### What changes were proposed in this pull request? Additional tests for pyspark tableExists with regard to views and temporary views ### Why are the changes needed? scala documentation indicates that tableExists works for tables/view and also temporary views. This unit tests try to verify that claim. While views seem ok, temporary views don't seem to work. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? tests Closes apache#33461 from dominikgehl/bug/SPARK-36243. Lead-authored-by: Dominik Gehl <dog@open.ch> Co-authored-by: Dominik Gehl <gehl@fastmail.fm> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…ct `NULL DEFINED AS` and default value should be `\N` ### What changes were proposed in this pull request? SCRIPT TRANSFORM ROW FORMAT DELIMITED should respect `NULL DEFINED AS` and default value should be `\N`   ### Why are the changes needed? Keep consistence with Hive ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added UT Closes apache#33363 from AngersZhuuuu/SPARK-36156. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…with default step ### What changes were proposed in this pull request? This PR follows up apache#33360 and add test case for `TimestampNTZ` sequence with default step. ### Why are the changes needed? Improve test coverage. ### Does this PR introduce _any_ user-facing change? 'No'. Just add test cases. ### How was this patch tested? New tests. Closes apache#33462 from beliefer/SPARK-36090-followup. Authored-by: gengjiaan <gengjiaan@360.cn> Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request? This PR proposes removing some APIs from pandas-on-Spark documentation. Because they can be easily workaround via Spark DataFrame or Column functions, so they might be removed In the future. ### Why are the changes needed? Because we don't want to expose some functions as a public API. ### Does this PR introduce _any_ user-facing change? The APIs such as `(Series|Index).spark.data_type`, `(Series|Index).spark.nullable`, `DataFrame.spark.schema`, `DataFrame.spark.print_schema`, `DataFrame.pandas_on_spark.attach_id_column`, `DataFrame.spark.checkpoint`, `DataFrame.spark.localcheckpoint` and `DataFrame.spark.explain` is removed in the documentation. ### How was this patch tested? Manually build the documents. Closes apache#33458 from itholic/SPARK-36239. Authored-by: itholic <haejoon.lee@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? This PR upgrades `lz4-java` to `1.8.0`, which includes not only performance improvement but also Darwin aarch64 support. https://github.com/lz4/lz4-java/releases/tag/1.8.0 https://github.com/lz4/lz4-java/blob/1.8.0/CHANGES.md ### Why are the changes needed? For providing better performance and platform support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI. Closes apache#33476 from sarutak/upgrade-lz4-java-1.8.0. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Gengliang Wang <gengliang@apache.org>
…s as 3.3.0 ### What changes were proposed in this pull request? As we decided to release TimestampNTZ type in Spark 3.3, we should update the versions of TimestampNTZ related changes as 3.3.0. ### Why are the changes needed? Correct the versions in documentation/code comment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing UT Closes apache#33478 from gengliangwang/updateVersion. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request? Bugfix: link to correction location of Pyspark Dataframe documentation ### Why are the changes needed? Current website returns "Not found" ### Does this PR introduce _any_ user-facing change? Website fix ### How was this patch tested? Documentation change Closes apache#33420 from dominikgehl/feature/SPARK-36209. Authored-by: Dominik Gehl <dog@open.ch> Signed-off-by: Sean Owen <srowen@gmail.com>
…ed as ANSI interval literals ### What changes were proposed in this pull request? This PR extends the way to represent `delayThreshold` with ANSI interval literals for watermark. ### Why are the changes needed? A `delayThreshold` is semantically an interval value so it's should be represented as ANSI interval literals as well as the conventional `1 second` form. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New tests. Closes apache#33456 from sarutak/delayThreshold-interval. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request? Update to the latest breeze 1.2 ### Why are the changes needed? Minor bug fixes ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests Closes apache#33449 from srowen/SPARK-35310. Authored-by: Sean Owen <srowen@gmail.com> Signed-off-by: Sean Owen <srowen@gmail.com>
…g allocations ### What changes were proposed in this pull request? Optimize some treeAggregates in MLlib by delaying allocating (thus not sending around) large arrays of zeroes This uses the same idea as in https://github.com/apache/spark/pull/23600/files ### Why are the changes needed? Allocating huge arrays of zeroes takes additional memory and network I/O which is unnecessary in some cases. It can cause operations to run out of memory that might otherwise succeed. Specifically, this should prevent the 'zero' value from having to be (pointlessly) checked for serializability, which can fail when passing through the default JavaSerializer; it would also prevent allocating and sending large 'zero' values for an empty partition in the aggregate. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closes apache#33443 from srowen/SPARK-35848. Authored-by: Sean Owen <srowen@gmail.com> Signed-off-by: Sean Owen <srowen@gmail.com>
### What changes were proposed in this pull request? This PR aims to upgrade ZSTD-JNI to 1.5.0-4. ### Why are the changes needed? ZSTD-JNI 1.5.0-3 has a packaging issue. 1.5.0-4 is recommended to be used instead. - luben/zstd-jni#181 (comment) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes apache#33483 from dongjoon-hyun/SPARK-36262. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…warnings
### What changes were proposed in this pull request?
Use `Column.__getitem__` instead of `Column.getItem` to suppress warnings.
### Why are the changes needed?
In pandas API on Spark code base, there are some places using `Column.getItem` with `Column` object, but it shows a deprecation warning.
### Does this PR introduce _any_ user-facing change?
Yes, users won't see the warnings anymore.
- before
```py
>>> s = ps.Series(list("abbccc"), dtype="category")
>>> s.astype(str)
/path/to/spark/python/pyspark/sql/column.py:322: FutureWarning: A column as 'key' in getItem is deprecated as of Spark 3.0, and will not be supported in the future release. Use `column[key]` or `column.key` syntax instead.
warnings.warn(
0 a
1 b
2 b
3 c
4 c
5 c
dtype: object
```
- after
```py
>>> s = ps.Series(list("abbccc"), dtype="category")
>>> s.astype(str)
0 a
1 b
2 b
3 c
4 c
5 c
dtype: object
```
### How was this patch tested?
Existing tests.
Closes apache#33486 from ueshin/issues/SPARK-36265/getitem.
Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
… by avoiding joins ### What changes were proposed in this pull request? Improve bool, string, numeric DataTypeOps tests by avoiding joins. Previously, bool, string, numeric DataTypeOps tests are conducted between two different Series. After the PR, bool, string, numeric DataTypeOps tests should perform on a single DataFrame. ### Why are the changes needed? A considerable number of DataTypeOps tests have operations on different Series, so joining is needed, which takes a long time. We shall avoid joins for a shorter test duration. The majority of joins happen in bool, string, numeric DataTypeOps tests, so we improve them first. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests. Closes apache#33402 from xinrong-databricks/datatypeops_diffframe. Authored-by: Xinrong Meng <xinrong.meng@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…d CategoricalIndex
### What changes were proposed in this pull request?
Add rename_categories to CategoricalAccessor and CategoricalIndex.
### Why are the changes needed?
rename_categories is supported in pandas CategoricalAccessor and CategoricalIndex. We ought to follow pandas.
### Does this PR introduce _any_ user-facing change?
Yes. `rename_categories` is supported in pandas API on Spark now.
```py
# CategoricalIndex
>>> psser = ps.CategoricalIndex(["a", "a", "b"])
>>> psser.rename_categories([0, 1])
CategoricalIndex([0, 0, 1], categories=[0, 1], ordered=False, dtype='category')
>>> psser.rename_categories({'a': 'A', 'c': 'C'})
CategoricalIndex(['A', 'A', 'b'], categories=['A', 'b'], ordered=False, dtype='category')
>>> psser.rename_categories(lambda x: x.upper())
CategoricalIndex(['A', 'A', 'B'], categories=['A', 'B'], ordered=False, dtype='category')
# CategoricalAccessor
>>> s = ps.Series(["a", "a", "b"], dtype="category")
>>> s.cat.rename_categories([0, 1])
0 0
1 0
2 1
dtype: category
Categories (2, int64): [0, 1]
>>> s.cat.rename_categories({'a': 'A', 'c': 'C'})
0 A
1 A
2 b
dtype: category
Categories (2, object): ['A', 'b']
>>> s.cat.rename_categories(lambda x: x.upper())
0 A
1 A
2 B
dtype: category
Categories (2, object): ['A', 'B']
```
### How was this patch tested?
Unit tests.
Closes apache#33471 from xinrong-databricks/category_rename_categories.
Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? This PR proposes to set the lowerbound of mypy version to use in the testing script. ### Why are the changes needed? https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141519/console ``` python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/classification.pyi:39: error: Overloaded function signatures 1 and 2 overlap with incompatible return types python/pyspark/mllib/classification.pyi:52: error: Overloaded function signatures 1 and 2 overlap with incompatible return types Found 13 errors in 4 files (checked 314 source files) 1 ``` Jenkins installed mypy at SPARK-32797 but seems the version installed is not same as GIthub Actions. It seems difficult to make the codebase compatible with multiple mypy versions. Therefore, this PR sets the lowerbound. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Jenkins job in this PR should test it out. Also manually tested: Without mypy: ``` ... flake8 checks passed. The mypy command was not found. Skipping for now. ``` With mypy 0.812: ``` ... flake8 checks passed. The minimum mypy version needs to be 0.910. Your current version is mypy 0.812. Skipping for now. ``` With mypy 0.910: ``` ... flake8 checks passed. starting mypy test... mypy checks passed. all lint-python tests passed! ``` Closes apache#33487 from HyukjinKwon/SPARK-36268. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…ssor and CategoricalIndex ### What changes were proposed in this pull request? Add `remove_unused_categories` to `CategoricalAccessor` and `CategoricalIndex`. ### Why are the changes needed? We should implement `remove_unused_categories` in `CategoricalAccessor` and `CategoricalIndex`. ### Does this PR introduce _any_ user-facing change? Yes, users will be able to use `remove_unused_categories`. ### How was this patch tested? Added some tests. Closes apache#33485 from ueshin/issues/SPARK-36261/remove_unused_categories. Authored-by: Takuya UESHIN <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Trying to adjust build memory settings and serial execution to re-enable GA. ### Why are the changes needed? GA tests are failed recently due to return code 137. We need to adjust build settings to make GA work. ### Does this PR introduce _any_ user-facing change? No, dev only. ### How was this patch tested? GA Closes apache#33447 from viirya/test-ga. Lead-authored-by: Liang-Chi Hsieh <viirya@gmail.com> Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Exposing functionExists in pyspark sql catalog ### Why are the changes needed? method was available in scala but not pyspark ### Does this PR introduce _any_ user-facing change? Additional method ### How was this patch tested? Unit tests Closes apache#33481 from dominikgehl/SPARK-36258. Authored-by: Dominik Gehl <dog@open.ch> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
…lasses ### What changes were proposed in this pull request? additional links to other classes in python documentation ### Why are the changes needed? python docstring syntax wasn't fully used everywhere ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Documentation change only Closes apache#33440 from dominikgehl/feature/python-docstrings. Authored-by: Dominik Gehl <dog@open.ch> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request? Since we have add log about commit time, I think this useful and we can make user know it directly in SQL tab's UI.  ### Why are the changes needed? Make user can directly know commit duration. ### Does this PR introduce _any_ user-facing change? User can see file commit duration in SQL tab's SQL plan graph ### How was this patch tested? Mannul tested Closes apache#31522 from AngersZhuuuu/SPARK-34399. Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com> Co-authored-by: AngersZhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…ue` in `ExternalSorter.spillMemoryIteratorToDisk` method ### What changes were proposed in this pull request? The main change of this pr is move `writer.close()` before `success = true` to ensure spill file closed before set `success = true` in `ExternalSorter.spillMemoryIteratorToDisk` method. ### Why are the changes needed? Avoid setting `success = true` first and then failure of close spill file ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass the Jenkins or GitHub Action - Add a new Test case to check `The spill file should not exists if writer close fails` Closes apache#33460 from LuciferYang/external-sorter-spill-close. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: yi.wu <yi.wu@databricks.com>
… partition
### What changes were proposed in this pull request?
This PR removes leading zeros from static number type partition when we insert into a partition table with empty partitions.
create table
CREATE TABLE `table_int` ( `id` INT, `c_string` STRING, `p_int` int)
USING parquet PARTITIONED BY (p_int);
insert
insert overwrite table table_int partition (p_int='00011')
select 1, 'c string'
where true ;
|partition|
|---------|
|p_int=11|
insert overwrite table table_int partition (p_int='00012')
select 1, 'c string'
where false ;
|partition|
|---------|
|p_int=00012|
### Why are the changes needed?
This PR creates consistent result when insert empty or non-empty partition
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add Unit test
Closes apache#33291 from dgd-contributor/35561_insert_integer_partition_fail_when_empty.
Authored-by: dgd-contributor <dgd_contributor@viettel.com.vn>
Signed-off-by: Sean Owen <srowen@gmail.com>
This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values. Jira issue report SPARK-36273. ### What changes were proposed in this pull request? This is a patch for SPARK-35546 which is needed for push-based shuffle. ### Why are the changes needed? A very minor fix of adding the reference from the other "FinalizeShuffleMerge". ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No unit tests were added. It's a pretty logical change. Closes apache#33493 from almogtavor/patch-1. Authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com> Signed-off-by: Sean Owen <srowen@gmail.com>
|
worth mentioning that there were amazingly no conflicts upon rebase. |
avenkatraman
approved these changes
Aug 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
rebasing master with the upstream of the fork to keep track of spark 3 builds