[SPARK-54030][SQL] Add user-friendly assertion message for view metadata corruption #52732
+75
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ganeshashree
force-pushed
the
SPARK-54030
branch
from
e58cf88 to
8f67ee8
Compare
cloud-fan
reviewed
Oct 29, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
Outdated
Show resolved
Hide resolved
cloud-fan
approved these changes
Oct 29, 2025
ganeshashree
force-pushed
the
SPARK-54030
branch
from
8f67ee8 to
e709014
Compare
ganeshashree
force-pushed
the
SPARK-54030
branch
from
e709014 to
93fc54c
Compare
|
thanks, merging to master! |
Yicong-Huang
pushed a commit
to Yicong-Huang/spark
that referenced
this pull request
Oct 30, 2025
…ata corruption ### What changes were proposed in this pull request? This PR enhances error reporting for corrupted view metadata by adding a detailed, user-friendly assertion message when there's a mismatch between the number of view query column names and the number of columns in the view schema. Changes: Enhanced the assertion in SessionCatalog.scala (fromCatalogTable method) to include: The fully qualified view name The actual number of view query column names vs schema columns The list of view query column names The list of view schema column names Guidance that the metadata needs to be repaired Added a unit test in SessionCatalogSuite.scala to verify the enhanced error message is displayed correctly when corrupted view metadata is detected. ### Why are the changes needed? Currently, when view metadata is corrupted (i.e., the number of view query column names doesn't match the schema length), the assertion fails with a generic "assertion failed" message that provides no context about: Which view has the problem What the actual vs expected values are What columns are involved How to fix the issue This makes debugging production issues very difficult. The enhanced error message provides all the necessary information to quickly identify and repair the corrupted view metadata. ### Does this PR introduce _any_ user-facing change? Yes. Users will now see a detailed error message instead of a generic assertion failure when encountering corrupted view metadata: Before: `assertion failed` After: `assertion failed: Corrupted view metadata detected for view spark_catalog.db.view_name. The number of view query column names 2 does not match the number of columns in the view schema 3. View query column names: [id, name], View schema columns: [id, name, value]. This indicates corrupted view metadata that needs to be repaired.` ### How was this patch tested? Added a new unit test corrupted view metadata: mismatch between viewQueryColumnNames and schema in SessionCatalogSuite.scala that: Creates a view with intentionally corrupted metadata (2 query column names but 3 schema columns) Verifies that looking up the view throws an AssertionError Validates the error message contains all expected details Existing tests continue to pass ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52732 from ganeshashree/SPARK-54030. Authored-by: Ganesha S <ganesha.s@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR enhances error reporting for corrupted view metadata by adding a detailed, user-friendly assertion message when there's a mismatch between the number of view query column names and the number of columns in the view schema.
Changes:
Enhanced the assertion in SessionCatalog.scala (fromCatalogTable method) to include:
The fully qualified view name
The actual number of view query column names vs schema columns
The list of view query column names
The list of view schema column names
Guidance that the metadata needs to be repaired
Added a unit test in SessionCatalogSuite.scala to verify the enhanced error message is displayed correctly when corrupted view metadata is detected.
Why are the changes needed?
Currently, when view metadata is corrupted (i.e., the number of view query column names doesn't match the schema length), the assertion fails with a generic "assertion failed" message that provides no context about:
Which view has the problem
What the actual vs expected values are
What columns are involved
How to fix the issue
This makes debugging production issues very difficult. The enhanced error message provides all the necessary information to quickly identify and repair the corrupted view metadata.
Does this PR introduce any user-facing change?
Yes. Users will now see a detailed error message instead of a generic assertion failure when encountering corrupted view metadata:
Before:
assertion failedAfter:
assertion failed: Corrupted view metadata detected for view spark_catalog.db.view_name. The number of view query column names 2 does not match the number of columns in the view schema 3. View query column names: [id, name], View schema columns: [id, name, value]. This indicates corrupted view metadata that needs to be repaired.How was this patch tested?
Added a new unit test corrupted view metadata: mismatch between viewQueryColumnNames and schema in SessionCatalogSuite.scala that:
Creates a view with intentionally corrupted metadata (2 query column names but 3 schema columns)
Verifies that looking up the view throws an AssertionError
Validates the error message contains all expected details
Existing tests continue to pass
Was this patch authored or co-authored using generative AI tooling?
No