Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-add table owner + stats to catalog #90

Closed
jtcohen6 opened this issue May 26, 2020 · 1 comment · Fixed by #92
Closed

Re-add table owner + stats to catalog #90

jtcohen6 opened this issue May 26, 2020 · 1 comment · Fixed by #92
Labels
bug Something isn't working
Milestone

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented May 26, 2020

Background

  • @Fokko added owner (Pull the owner from the DESCRIBE EXTENDED #39) and table statistics (Add support for extracting statistics #41) to the auto-generated docs site by fleshing out the static method _parse_relation and passing it to get_catalog
  • Amid all the merging for 0.15.3 (0.15.3 upgrade #65), the eventual implementation of get_catalog called list_relations + _massage_column_for_catalog. It never called _parse_relation. The unit tests on _parse_relation have been passing, but the output of that work never made it to the actual catalog.
  • The contract of get_catalog changed in dbt v0.16.0. We fixed in dbt-spark v0.16.1 by reimplementing some private methods (fix catalog generation #77).

Next steps

This isn't a regression per se, since this feature never really worked in v0.15.3... but we sure meant for it to ship then!

We should update the catalog private method reimplementations to include owner and table stats. The basis for this work should be the latest implementation on core adapters.

@jtcohen6 jtcohen6 added the bug Something isn't working label May 26, 2020
@jtcohen6 jtcohen6 added this to the 0.17.0 milestone May 26, 2020
@beckjake
Copy link
Contributor

The code for this does still mostly exist, it's just moved into parse_describe_extended. I think the reason owner isn't showing up is because of a combination of the more general catalog issues and the fact that I lost the stuff that handles relation-wide owners (via fetch table properties).

Statistics is failing for more complicated reasons, mostly the intersection of some irritating agate behavior ('' -> None) and some bad dbt-core behavior (None not being allowed).

I've got both of these working again locally with dbt-labs/dbt-core#2489 and some minor tweaks to re-add the properties query - I'll open a PR for that in a moment, once I track down all the bugs I think it fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants