-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-279] Delta tables missing column schema in show table extended
#295
Comments
show table extended
show table extended
Actually, I'm using Spark with Iceberg tables and have the same issue. show table extended in warehouse like '*'; SQL Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: SHOW TABLE EXTENDED is not supported for v2 tables.;
Caused by: org.apache.spark.sql.AnalysisException: SHOW TABLE EXTENDED is not supported for v2 tables.; |
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days. |
Context: When generating the catalog for documentation, we switched to using one
show table extended in databasename like '*'
, instead of a separatedescribe table extended
for each individual table. See:(We could still meaningfully speed that up, by passing the specific table names into
show table extended
instead of*
. This will benefit users with many thousands of objects per Spark database, of which dbt is concerned with a small subset.)Complication: We believe that open source Delta tables fail to return their column schema in
show table extended
. This originally came up in:We believe this is the root cause of dbt-labs/dbt-docs#236. Columns are missing / have
null
names, which is not something that thedbt-docs
code is expecting to be possible.I'm not sure what action we can take here, beyond opening an issue in https://github.com/delta-io/delta. I'd prefer to avoid a return to the much slower
describe table extended
approach, or trying to finagle support for both. I also understand that Apache Spark will eventually add support for a real information schema :)The text was updated successfully, but these errors were encountered: