Skip to content

[SPARK-23263][TEST] CTAS should update stat if autoUpdate statistics is enabled #20430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wangyum wants to merge 7 commits into from
Closed

[SPARK-23263][TEST] CTAS should update stat if autoUpdate statistics is enabled #20430

wangyum wants to merge 7 commits into from

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Jan 30, 2018 •
edited
Loading

What changes were proposed in this pull request?

The SPARK-27403 fixed CTAS cannot update statistics even if spark.sql.statistics.size.autoUpdate.enabled is enabled, as mentioned in SPARK-23263.

This pr adds tests for that fix.

How was this patch tested?

N/A

@SparkQA
Copy link

SparkQA commented Jan 30, 2018

Test build #86790 has finished for PR 20430 at commit 08d31c0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

felixcheung
felixcheung reviewed Jan 30, 2018
View reviewed changes
Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile @cloud-fan

@cloud-fan
Copy link
Contributor

cloud-fan commented Jan 30, 2018

CC @wzhfy

@wzhfy
Copy link
Contributor

wzhfy commented Feb 1, 2018

Can we specialize this CTAS case? For data changing commands like INSERT, I think we should remove the stats if auto update is disabled, because the previous stats are inaccurate after the insertion.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 16, 2018

ping @wangyum for @wzhfy's comment.

@HyukjinKwon HyukjinKwon mentioned this pull request Jul 16, 2018
Closed
@wangyum
Copy link
Member Author

wangyum commented Jul 16, 2018

Thanks @HyukjinKwon How about close this? CTAS has other issues , as mentioned in SPARK-24766. I will try to fix it if there is a chance.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 16, 2018

Sure, please close and go ahead in the best way you could perform better.

@wangyum wangyum closed this Jul 16, 2018
@wangyum wangyum changed the title [SPARK-23263][SQL] Create table stored as parquet should update table size if automatic update table size is enabled [SPARK-23263][SQL] CTAS should update stat if autoUpdate statistics is enabled Nov 8, 2018
@wangyum wangyum mentioned this pull request Nov 8, 2018
Closed
wangyum added 2 commits May 9, 2019 10:11
@wangyum
Merge remote-tracking branch 'upstream/master' into SPARK-23263
a37d28e 
# Conflicts:
#	sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
#	sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
#	sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala
@wangyum
Add test
1c9caa0
@wangyum wangyum reopened this May 9, 2019
@wangyum wangyum changed the title [SPARK-23263][SQL] CTAS should update stat if autoUpdate statistics is enabled [SPARK-23263][TEST] CTAS should update stat if autoUpdate statistics is enabled May 9, 2019
@SparkQA
Copy link

SparkQA commented May 9, 2019

Test build #105272 has finished for PR 20430 at commit 1c9caa0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented May 9, 2019

cc @dongjoon-hyun

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented May 13, 2019

Retest this please.

dongjoon-hyun
dongjoon-hyun requested changes May 13, 2019
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wangyum . I added two comments. Could you address them?

@SparkQA
Copy link

SparkQA commented May 13, 2019

Test build #105365 has finished for PR 20430 at commit 1c9caa0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented May 20, 2019

retest this please

@SparkQA
Copy link

SparkQA commented May 20, 2019

Test build #105553 has finished for PR 20430 at commit 1c9caa0.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented May 20, 2019

retest this please

@SparkQA
Copy link

SparkQA commented May 20, 2019

Test build #105560 has finished for PR 20430 at commit 1c9caa0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented May 20, 2019

retest this please

@SparkQA
Copy link

SparkQA commented May 20, 2019

Test build #105563 has finished for PR 20430 at commit 1c9caa0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 27, 2019

Test build #105826 has finished for PR 20430 at commit 36fd737.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jun 14, 2019

@wangyum, can you replace and update? I think it's close to go.

@SparkQA
Copy link

SparkQA commented Jun 14, 2019

Test build #106518 has finished for PR 20430 at commit c9949eb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 16, 2019

Test build #106555 has finished for PR 20430 at commit d4f9035.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jun 20, 2019

Merged to master.

@wangyum wangyum deleted the SPARK-23263 branch June 20, 2019 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@felixcheung felixcheung felixcheung left review comments

@wzhfy wzhfy wzhfy left review comments

@dongjoon-hyun dongjoon-hyun dongjoon-hyun requested changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@wangyum @SparkQA @cloud-fan @wzhfy @HyukjinKwon @dongjoon-hyun @felixcheung