Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for catalogs #28416

Merged
merged 1 commit into from
May 16, 2024
Merged

feat: add support for catalogs #28416

merged 1 commit into from
May 16, 2024

Conversation

betodealmeida
Copy link
Member

SUMMARY

Add support for catalogs to:

  • Presto
  • Trino
  • Snowflake
  • BigQuery

We already have support for Postgres and Databricks.

@john-bodley I'm worried about the DB migration here and how it will affect AirBnB. Can you take a look? The migration needs to:

  • Read the default catalog for each of these 4 DBs and update the catalog column in a few models (SqlaTable, Query, SavedQuery, TabState, and TableSchema).
  • Update the schema_perm column for any SqlaTable and Slice models in these databases.
  • Rename existing permissions of type schema_perm.

I'm happy to hold back on this PR until we know it's safe to merge to master.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue: [SIP-95] Proposal for Catalog Support in SQL Lab #22862
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@github-actions github-actions bot added the risk:db-migration PRs that require a DB migration label May 9, 2024
@betodealmeida betodealmeida force-pushed the bigquery-catalog branch 3 times, most recently from 517da78 to bbd7ca6 Compare May 10, 2024 13:54
@betodealmeida betodealmeida marked this pull request as ready for review May 10, 2024 15:46
@betodealmeida betodealmeida requested a review from a team as a code owner May 10, 2024 15:46
@john-bodley
Copy link
Member

Thanks @betodealmeida for the ping. I think we (Airbnb) are ok with this. We will likely just skip the catalog_access as we don't use the FAB permission model for Superset entities.

@stephenlb
Copy link

Looking forward to this! We rely on using the superset interface with trino and many other databases. And full catalog with all connected sources will be meaningful to our data team 🙌❤️😄

@betodealmeida betodealmeida merged commit 5da6d2b into master May 16, 2024
31 checks passed
EnxDev pushed a commit to EnxDev/superset that referenced this pull request May 31, 2024
@rusackas rusackas deleted the bigquery-catalog branch September 27, 2024 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
risk:db-migration PRs that require a DB migration size/XL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants