Skip to content

[SPARK-40798][SQL][TESTS][FOLLOW-UP] Disable ANSI mode in v2 ALTER TABLE PARTITION test #38443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 1 commit into from
Closed

[SPARK-40798][SQL][TESTS][FOLLOW-UP] Disable ANSI mode in v2 ALTER TABLE PARTITION test #38443

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ...rc/test/scala/org/apache/spark/sql/execution/command/v2/AlterTableAddPartitionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,9 @@ class AlterTableAddPartitionSuite
withNamespaceAndTable("ns", "tbl") { t =>
sql(s"CREATE TABLE $t (c int) $defaultUsing PARTITIONED BY (p int)")

withSQLConf(SQLConf.SKIP_TYPE_VALIDATION_ON_ALTER_PARTITION.key -> "true") {
withSQLConf(
SQLConf.SKIP_TYPE_VALIDATION_ON_ALTER_PARTITION.key -> "true",
SQLConf.ANSI_ENABLED.key -> "false") {
Copy link
Member Author

@HyukjinKwon HyukjinKwon Oct 31, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan @gengliangwang @ulysses-you, this is a temporary fix to recover the test case.

The root case here seems like we don't care about spark.sql.storeAssignmentPolicy in V2:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala

Line 81 in 1dd0ca2

Cast(Literal.create(raw, StringType), dt, Some(conf.sessionLocalTimeZone)).eval()

with using a plain cast (that's controlled by the ANSI flag).

Whereas seems like V1 uses the casts from PartitioningUtils.castPartitionSpec that's independent from ANSI flag (respects spark.sql.storeAssignmentPolicy). While I guess we should use the same cast, I made this fix first to make the test pass in case there's some more things to discuss about this.

Copy link
Contributor

@ulysses-you ulysses-you Oct 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to use PartitioningUtils.castPartitionSpec

Copy link
Contributor

@cloud-fan cloud-fan Oct 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, @ulysses-you can you send a PR to fix it later?

Copy link
Contributor

@ulysses-you ulysses-you Oct 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

sql(s"ALTER TABLE $t ADD PARTITION (p='aaa')")
checkPartitions(t, Map("p" -> defaultPartitionName))
sql(s"ALTER TABLE $t DROP PARTITION (p=null)")
Expand Down