Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect results with SQL MERGE INSERT * / UPDATE * when applied on a view on the target Delta table #618

Closed
tdas opened this issue Mar 11, 2021 · 0 comments
Closed

Incorrect results with SQL MERGE INSERT * / UPDATE * when applied on a view on the target Delta table #618

tdas opened this issue Mar 11, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@tdas
Copy link
Contributor

tdas commented Mar 11, 2021 •
edited
Loading

Apache Spark has a bug where MERGE's INSERT/UPDATE * expansion and resolution by Apache Spark is done by column position, not by name. This can lead to incorrect source columns be assigned to the target columns by star.

Delta should not have been affected by this bug because it tries to skip the OSS MERGE resolution to perform its own resolution with potentially schema evolution enabled. The logic to skip the resolution is based on whether the target table is a DataSourceV2Relation that supports its own schema resolution. However, this logic breaks when there is a View on top of the DataSourceV2Relation, thus causing the Apache spark's resolution to kick and do incorrect star expansion. Until the bug in this logic is fixed in Apache Spark, this cannot be fixed in OSS Delta.

This affects the following versions

  • Delta 0.7.0 on Spark 3.0.x
  • Delta 0.8.0 on Spark 3.1.x

Workarounds

  1. Remove the view.
  2. Use Scala/Python APIs.
@tdas tdas changed the title Incorrect results with MERGE INSERT * / UPDATE * when applied on a view on the target Delta table Incorrect results with SQL MERGE INSERT * / UPDATE * when applied on a view on the target Delta table Mar 11, 2021
@tdas tdas added the bug Something isn't working label Mar 11, 2021
mengtong-db pushed a commit that referenced this issue Apr 6, 2021
@tdas @mengtong-db
[SC-72276] Add a config and test
7cb5115 
This PR adds a test so that we can detect #618

Author: Tathagata Das <tathagata.das1565@gmail.com>

GitOrigin-RevId: 1af03b03f64c607c8f61b41eef678f3a72355ad7
@zsxwing zsxwing mentioned this issue Oct 30, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade Spark to 3.2.0 zsxwing/delta
1 participant
@tdas