[SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data #25979

viirya · 2019-09-30T16:39:58Z

What changes were proposed in this pull request?

This patch proposes to delete old Hive external partition directory even the partition does not exist in Hive, when insert overwrite Hive external table partition.

Why are the changes needed?

When insert overwrite to a Hive external table partition, if the partition does not exist, Hive will not check if the external partition directory exists or not before copying files. So if users drop the partition, and then do insert overwrite to the same partition, the partition will have both old and new data.

For example:

withSQLConf(HiveUtils.CONVERT_METASTORE_PARQUET.key -> "false") {
  // test is an external Hive table.
  sql("INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 1")
  sql("ALTER TABLE test DROP PARTITION(name='n1')")
  sql("INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 2")
  sql("SELECT id FROM test WHERE name = 'n1' ORDER BY id") // Got both 1 and 2.
}

Does this PR introduce any user-facing change?

Yes. This fix a correctness issue when users drop partition on a Hive external table partition and then insert overwrite it.

How was this patch tested?

Added test.

viirya · 2019-09-30T18:14:16Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala

+          // insert overwrite to the same partition, the partition will have both old and new
+          // data.
+          val updatedPart = if (overwrite && table.tableType == CatalogTableType.EXTERNAL) {
+            AlterTableAddPartitionCommand(


Instead of constructing partition directory manually, I let hive add the target parititon to be overwritten. We will delete the partition directory below.

How can we rollback if the follow loadPartition operation fails?

Do we provide rollback for load partition before?

viirya · 2019-09-30T18:14:34Z

cc @cloud-fan @dongjoon-hyun @HyukjinKwon

viirya · 2019-09-30T18:14:44Z

cc @felixcheung

SparkQA · 2019-09-30T19:19:36Z

Test build #111616 has finished for PR 25979 at commit 5e5c6ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-02T08:52:03Z

How does hive handle it? Does it also produce duplicated data?

viirya · 2019-10-02T16:15:39Z

As I quickly tried on Hive, Hive does not delete existing directory with "INSERT OVERWRITE", if the external partition was dropped by "DROP PARTITION" before. Hive just moves data from staging into the directory in this case.

Whether it produces duplicated data, depending on the filename generated. I tested two version of Hive locally. I do not know the logic Hive produces data filename.

On Hive 2.1.0, two "INSERT OVERWRITE" produces data file with same name like 000000_0. The second "INSERT OVERWRITE" moves the file into and overwrite old file.

On Hive 2.3.2, the second "INSERT OVERWRITE" causes following failure when moving file with same name:

0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE test(id int) PARTITIONED BY (name string) STORED AS TEXTFILE LOCATION '/tmp/test';
No rows affected (0.074 seconds)
0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 1;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
No rows affected (1.809 seconds)
0: jdbc:hive2://localhost:10000> ALTER TABLE test DROP PARTITION(name='n1');
No rows affected (0.175 seconds)
0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 2;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: rename for src path: hdfs://namenode:8020/tmp/test/name=n1/.hive-staging_hive_2019-10-02_16-01-28_570_7296144361967433532-1/-ext-10000/000000_0 to dest path:hdfs://namenode:8020/tmp/test/name=n1/000000_0 returned false
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

So, I think it is sure that Hive does not delete existing directory when "INSERT OVERWRITE" into a non-existing partition. Whether duplicated data are seen, it depends on if filenames are unique or not between different "INSERT OVERWRITE".

If your first "INSERT OVERWRITE" produces filename "000000_0" and "000001_0", but the second "INSERT OVERWRITE" produces just "000000_0", you will read partial old data "000001_0" with new "000000_0" together.

Above tests are using MR as Hive execution engine. I am not sure if Hive has different output filename in different execution engine like Spark or Tez.

cloud-fan · 2019-10-03T03:30:49Z

Do you mean the current behavior is consistent with some Hive versions?

viirya · 2019-10-03T03:42:08Z

Currently behavior of INSERT OVERWRITE into a Hive external table partition is consistent with Hive. If the partition was dropped by DROP PARTITION before, INSERT OVERWRITE will not overwrite existing partition directory.

If there are files in the directory, conflicting files will be overwritten (Hive 2.1) or exception thrown (Hive 2.3.2). Unique files will be read as duplicated data.

felixcheung · 2019-10-03T03:54:57Z

yeah, this is a hard one, obviously the behavior is buggy, hard to detect etc. but that's how Hive is designed. I think we should log a warn in Spark at least so interested folks (like us) can detect this after the job is run

On Hive 2.1.0, two "INSERT OVERWRITE" produces data file with same name like 000000_0. The second "INSERT OVERWRITE" moves the file into and overwrite old file.

On Hive 2.3.2, the second "INSERT OVERWRITE" causes following failure when moving file with same name

we can't really rely on the name being the same to overwrite. it depends on a number of things. for instance, if the original partition has 10B row and 1M file, overwritten with new partition having 1B and 100k file, then a lot of files are not going to be overwritten (like 900k)

felixcheung · 2019-10-03T03:56:05Z

or we do

exception thrown (Hive 2.3.2)

viirya · 2019-10-03T04:48:53Z

I am not sure if this is designed behavior of Hive.

You can see a similar issue at Hive https://jira.apache.org/jira/browse/HIVE-18702. INSERT OVERWRITE should clean table directory.

The last comment in that ticket, someone left a comment describing the same case as this. He said he will create a ticket, but I do not see that.

viirya · 2019-10-03T20:35:07Z

I think the semantics of INSERT OVERWRITE should be overwriting existing data with new data. By no means it leaves old data intact with new data.

I think there are few options in this case:

delete old data like this PR currently did
no delete old data, throw an exception to user, saying there are old data. (I think this might be confusing to user. I am also wondering how user can react to this, if they do not access filesystem)

Any ideas? @cloud-fan @felixcheung

felixcheung · 2019-10-06T19:58:51Z

It looks to me we should clean the directory is what the hive community is saying.

…

________________________________ From: Liang-Chi Hsieh <notifications@github.com> Sent: Thursday, October 3, 2019 1:36:59 PM To: apache/spark <spark@noreply.github.com> Cc: Felix Cheung <felixcheung_m@hotmail.com>; Mention <mention@noreply.github.com> Subject: Re: [apache/spark] [SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data (#25979) I think the semantics of INSERT OVERWRITE should be overwriting existing data with new data. By no means it leaves old data intact with new data. I think there are few options in this case: 1. delete old data like this PR currently did 2. no delete old data, throw an exception to user, saying there are old data. (I think this might be confusing to user. I am also wondering how user can react to this, if they do not access filesystem) Any ideas? @cloud-fan<https://github.com/cloud-fan> @felixcheung<https://github.com/felixcheung> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#25979?email_source=notifications&email_token=ACENZ6ZBUB7354FYCXP5YVDQMZJWXA5CNFSM4I35N7KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAJQIZA#issuecomment-538117220>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACENZ6YVFGNNKVGFUXH5URTQMZJWXANCNFSM4I35N7KA>.

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala

SparkQA · 2019-10-08T07:05:02Z

Test build #111876 has finished for PR 25979 at commit 4df8a79.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-08T17:04:26Z

Test build #111900 has finished for PR 25979 at commit 1618e56.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-08T20:18:56Z

Test build #111914 has finished for PR 25979 at commit 78beb8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2019-10-08T20:29:42Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala

+            val splitPart = part.split("=")
+            assert(splitPart.size == 2, s"Invalid written partition path: $part")
+            ExternalCatalogUtils.unescapePathName(splitPart(0)) ->
+              ExternalCatalogUtils.unescapePathName(splitPart(1))


Regarding some partition values that get escaped, we need to unescape them.

SparkQA · 2019-10-08T20:55:15Z

Test build #111916 has finished for PR 25979 at commit 965dc4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2019-10-10T15:21:29Z

@cloud-fan @felixcheung Do you have more comments on this?

viirya · 2019-10-14T18:05:31Z

retest this please

SparkQA · 2019-10-14T20:35:05Z

Test build #112053 has finished for PR 25979 at commit 965dc4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2019-10-16T15:48:16Z

cc @dongjoon-hyun @HyukjinKwon too

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala

SparkQA · 2019-10-17T07:05:02Z

Test build #112203 has finished for PR 25979 at commit d15576f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-17T07:05:02Z

Test build #112204 has finished for PR 25979 at commit e395fac.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-17T09:43:50Z

Test build #112209 has finished for PR 25979 at commit 8337870.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-18T08:35:58Z

thanks, merging to master!

HyukjinKwon · 2019-10-24T09:17:32Z

Sorry for late response, @viirya. Looks making sense to me too.

AngersZhuuuu · 2019-11-12T10:31:00Z

As I quickly tried on Hive, Hive does not delete existing directory with "INSERT OVERWRITE", if the external partition was dropped by "DROP PARTITION" before. Hive just moves data from staging into the directory in this case.

Whether it produces duplicated data, depending on the filename generated. I tested two version of Hive locally. I do not know the logic Hive produces data filename.

On Hive 2.1.0, two "INSERT OVERWRITE" produces data file with same name like 000000_0. The second "INSERT OVERWRITE" moves the file into and overwrite old file.

On Hive 2.3.2, the second "INSERT OVERWRITE" causes following failure when moving file with same name:
0: jdbc:hive2://localhost:10000> CREATE EXTERNAL TABLE test(id int) PARTITIONED BY (name string) STORED AS TEXTFILE LOCATION '/tmp/test';
No rows affected (0.074 seconds)
0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 1;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
No rows affected (1.809 seconds)
0: jdbc:hive2://localhost:10000> ALTER TABLE test DROP PARTITION(name='n1');
No rows affected (0.175 seconds)
0: jdbc:hive2://localhost:10000> INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 2;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: rename for src path: hdfs://namenode:8020/tmp/test/name=n1/.hive-staging_hive_2019-10-02_16-01-28_570_7296144361967433532-1/-ext-10000/000000_0 to dest path:hdfs://namenode:8020/tmp/test/name=n1/000000_0 returned false
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
So, I think it is sure that Hive does not delete existing directory when "INSERT OVERWRITE" into a non-existing partition. Whether duplicated data are seen, it depends on if filenames are unique or not between different "INSERT OVERWRITE".

If your first "INSERT OVERWRITE" produces filename "000000_0" and "000001_0", but the second "INSERT OVERWRITE" produces just "000000_0", you will read partial old data "000001_0" with new "000000_0" together.

Above tests are using MR as Hive execution engine. I am not sure if Hive has different output filename in different execution engine like Spark or Tez.

Hive use mv here , so if name is same, it will be replaced, if your partition was dropped remain two files

0000000_0
0000001_0

Then you insert overwrite into this partition with one file 0000000_0, then the old data 0000001_0 still remains.

In spark, each result's file name is different....so, data dunplicted.

dongjoon-hyun · 2020-03-11T17:44:53Z

Hi, @viirya . Since this seems to be a correctness fix, could you make a backport PR to branch-2.4?

Hi, @AngersZhuuuu . Do you mean that is another corner case existing still?

viirya · 2020-03-11T18:44:51Z

@dongjoon-hyun Ok. I can prepare a backport PR for 2.4.

dongjoon-hyun · 2020-03-11T18:46:09Z

Thank you so much, @viirya .

…tion should delete old data ### What changes were proposed in this pull request? This patch proposes to delete old Hive external partition directory even the partition does not exist in Hive, when insert overwrite Hive external table partition. This is backport of #25979 to branch-2.4. ### Why are the changes needed? When insert overwrite to a Hive external table partition, if the partition does not exist, Hive will not check if the external partition directory exists or not before copying files. So if users drop the partition, and then do insert overwrite to the same partition, the partition will have both old and new data. For example: ```scala withSQLConf(HiveUtils.CONVERT_METASTORE_PARQUET.key -> "false") { // test is an external Hive table. sql("INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 1") sql("ALTER TABLE test DROP PARTITION(name='n1')") sql("INSERT OVERWRITE TABLE test PARTITION(name='n1') SELECT 2") sql("SELECT id FROM test WHERE name = 'n1' ORDER BY id") // Got both 1 and 2. } ``` ### Does this PR introduce any user-facing change? Yes. This fix a correctness issue when users drop partition on a Hive external table partition and then insert overwrite it. ### How was this patch tested? Added test. Closes #27887 from viirya/SPARK-29295-2.4. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ion path should be case insensitive ### What changes were proposed in this pull request? This is a follow up of #25979. When we inserting overwrite an external hive partitioned table with upper case dynamic partition key, exception thrown. like: ``` org.apache.spark.SparkException: Dynamic partition key P1 is not among written partition paths. ``` The root cause is that Hive metastore is not case preserving and keeps partition columns with lower cased names, see details in: https://github.com/apache/spark/blob/ddd8d5f5a0b6db17babc201ba4b73f7df91df1a3/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L895-L901 https://github.com/apache/spark/blob/e28914095aa1fa7a4680b5e4fcf69e3ef64b3dbc/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L228-L234 In this PR, we convert the dynamic partition map to a case insensitive map. ### Why are the changes needed? To fix the issue when inserting overwrite into external hive partitioned table with upper case dynamic partition key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. Closes #28765 from turboFei/SPARK-29295-follow-up. Authored-by: turbofei <fwang12@ebay.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…ion path should be case insensitive ### What changes were proposed in this pull request? This is a follow up of #25979. When we inserting overwrite an external hive partitioned table with upper case dynamic partition key, exception thrown. like: ``` org.apache.spark.SparkException: Dynamic partition key P1 is not among written partition paths. ``` The root cause is that Hive metastore is not case preserving and keeps partition columns with lower cased names, see details in: https://github.com/apache/spark/blob/ddd8d5f5a0b6db17babc201ba4b73f7df91df1a3/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L895-L901 https://github.com/apache/spark/blob/e28914095aa1fa7a4680b5e4fcf69e3ef64b3dbc/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L228-L234 In this PR, we convert the dynamic partition map to a case insensitive map. ### Why are the changes needed? To fix the issue when inserting overwrite into external hive partitioned table with upper case dynamic partition key. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. Closes #28765 from turboFei/SPARK-29295-follow-up. Authored-by: turbofei <fwang12@ebay.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 717ec5e) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Insert overwrite external table partition should not have old data.

5e5c6ef

viirya commented Sep 30, 2019

View reviewed changes

dongjoon-hyun added the SQL label Oct 1, 2019

cloud-fan reviewed Oct 7, 2019

View reviewed changes

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala Outdated Show resolved Hide resolved

Fix dynamic partition.

4df8a79

cloud-fan reviewed Oct 8, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Oct 8, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala Outdated Show resolved Hide resolved

Address comment.

1618e56

viirya added 2 commits October 8, 2019 10:57

Fix partition directory parsing.

78beb8a

Add test for escaped partition path.

965dc4a

viirya commented Oct 8, 2019

View reviewed changes

cloud-fan reviewed Oct 17, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala Outdated Show resolved Hide resolved

viirya added 2 commits October 16, 2019 23:45

Construct partition directory.

d15576f

remove unused import.

e395fac

cloud-fan reviewed Oct 17, 2019

View reviewed changes

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala Outdated Show resolved Hide resolved

Dynamic insert overwrite will update multiple partitions.

8337870

cloud-fan closed this in 5692680 Oct 18, 2019

viirya mentioned this pull request Mar 12, 2020

[SPARK-29295][SQL][2.4] Insert overwrite to Hive external table partition should delete old data #27887

Closed

turboFei mentioned this pull request Jun 9, 2020

[SPARK-29295][SQL][FOLLOWUP] Dynamic partition map parsed from partition path should be case insensitive #28765

Closed

viirya deleted the SPARK-29295 branch December 27, 2023 18:37

[SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data #25979

[SPARK-29295][SQL] Insert overwrite to Hive external table partition should delete old data #25979

Uh oh!

Conversation

viirya commented Sep 30, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

viirya Sep 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei Oct 1, 2019

Choose a reason for hiding this comment

Uh oh!

viirya Oct 1, 2019

Choose a reason for hiding this comment

Uh oh!

viirya commented Sep 30, 2019

Uh oh!

viirya commented Sep 30, 2019

Uh oh!

SparkQA commented Sep 30, 2019

Uh oh!

cloud-fan commented Oct 2, 2019

Uh oh!

viirya commented Oct 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Oct 3, 2019

Uh oh!

viirya commented Oct 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented Oct 3, 2019

Uh oh!

felixcheung commented Oct 3, 2019

Uh oh!

viirya commented Oct 3, 2019

Uh oh!

viirya commented Oct 3, 2019

Uh oh!

felixcheung commented Oct 6, 2019 via email

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

viirya Oct 8, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

viirya commented Oct 10, 2019

Uh oh!

viirya commented Oct 14, 2019

Uh oh!

SparkQA commented Oct 14, 2019

Uh oh!

viirya commented Oct 16, 2019

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Oct 17, 2019

Uh oh!

SparkQA commented Oct 17, 2019

Uh oh!

SparkQA commented Oct 17, 2019

Uh oh!

cloud-fan commented Oct 18, 2019

Uh oh!

HyukjinKwon commented Oct 24, 2019

Uh oh!

viirya Sep 30, 2019 •

edited

Loading

viirya commented Oct 2, 2019 •

edited

Loading

viirya commented Oct 3, 2019 •

edited

Loading

AngersZhuuuu commented Nov 12, 2019 •

edited

Loading