Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial "wireframe" for model contracts #2890

Merged
merged 18 commits into from
Feb 27, 2023
Merged

Conversation

jtcohen6
Copy link
Collaborator

@jtcohen6 jtcohen6 commented Feb 20, 2023

resolves #2839
resolves #2594 (includes #2601)

waiting on v1.5 initialization (#2842)

What are you changing in this pull request and why?

Opening an early draft PR for discussion!

My goal right now is just to stand up some initial "wireframes" with suggestive/placeholder content. A lot of the language will need refining, and the content will need deduplicating/reorganizing. Our goal is to push up iterative changes early & often, so we can gather regular feedback. I feel good about these pages being:

  • Visible in the sidebar only if you have v1.5 selected
  • Clearly called out as "Beta functionality" and "Under construction 🚧"

Specific changes:

  • Create new section under "Docs > Collaborate with others": "Publishing models."
    • Most of the other pages under "Collaborate" are about environments, version control, documentation — not generally dbt-core project source code. But our goal in the multi-project initiative is all about "scaling collaboration"! Does this feel out of place?
    • What do we think of "Publishing models" as naming/framing? Elsewhere I've been referring to this slice of the initiative with the tagline "Models as APIs."
  • Create "narrative" pages for "Model contracts." Also create initial pages for "Model access" (Narrative: Model groups & access #2840) and "Model versions" (Narrative: Model versions #2841), but these are really just skeleton pages for now, missing a lot of important narrative.
  • Create reference pages for contract & constraints. These pages will need to be reworked as we rework the functionality & syntax—a good thing! I tried to call out explicitly places where I expect us to come back and edit/update.

Checklist

  • Add versioning components, as described in Versioning Docs
  • Add a note to the prerelease version Migration Guide
  • Review the Content style guide and About versioning so my content adheres to these guidelines.
  • Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."

Adding new pages (delete if not applicable):

  • Add page to website/sidebars.js
  • Provide a unique filename for the new page

sungchun12 and others added 4 commits February 20, 2023 00:09
* placeholder outline

* add version blocks

* add to sidebar

* remove version blocks

* add 2 sections

* postgres section example

* add more docs

* correct DDL

* correct config name

* fix data type

* add redshift docs

* Update constraints docs for Snowflake

* add model config links

* update warehouse to spark

* update not null syntax

* add more config examples

* fix ordering

* update docs based on new parsing

* add a note

* add example error messages

* Update config name on Redshift

* Update description for Spark

* add explainers

* add check

* remove fluff

---------

Co-authored-by: Dave Connors <dave.connors@fishtownanalytics.com>
Co-authored-by: Benoit Perigaud <8754100+b-per@users.noreply.github.com>
@netlify
Copy link

netlify bot commented Feb 20, 2023

Deploy Preview for docs-getdbt-com ready!

Name Link
🔨 Latest commit 4eee716
🔍 Latest deploy log https://app.netlify.com/sites/docs-getdbt-com/deploys/63fcbba117c2f500083d8ad7
😎 Deploy Preview https://deploy-preview-2890--docs-getdbt-com.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added content Improvements or additions to content guides Knowledge best suited for Guides size: large This change will more than a week to address and might require more than one person labels Feb 20, 2023
@jtcohen6 jtcohen6 changed the title Jerco/model contracts Initial "wireframe" for model contracts Feb 20, 2023
Comment on lines +51 to +57
"page": "reference/dbt-jinja-functions/local-md5",
"firstVersion": "1.4",
},
{
"page": "reference/warehouse-setups/fal-setup",
"firstVersion": "1.3",
},
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No functional change here, I just reordered the list so it's in a consistent (descending) order by firstVersion. I figure, in the future, we could remove items from this list as we deprecate older versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@matthewshaver
Copy link
Contributor

I wanted to note two particular editorial changes in case others have strong feelings about them:
"guarantees" and "pre-flight" were replaced with "parameters" and "prerequisites," respectively. Part of my duties is to prepare our documents for translation sometime in the future. These words (in their context here) may result in confusion or difficulties for readers who don't speak English as a primary language. As this is a new feature, and one that seems very important, I wanted to ensure that these documents are as clear as possible for all of our readers.

@dbeatty10
Copy link
Contributor

Smart to prepare for translation, @matthewshaver 🧠

It is feeling awkward to me when reading "prerequisite check" in context currently. Is there an option to re-phrase this in a way that simultaneously prepares for translation and reads more smoothly?

@jtcohen6
Copy link
Collaborator Author

jtcohen6 commented Feb 21, 2023

@matthewshaver @dbeatty10 Love that we're having this conversation! I will say that "parameters" and "prerequisites" don't feel quite right. Let's find the right words, both to capture the meaning and to enable eventual translation into other languages!

Parameters feels much too vague of a term. A "parameter" could be any model attribute/property/configuration, and it doesn't carry the same force of consistent truth. What I'm after here is, a set of things that are guaranteed to be true about the structure of this model and the dataset it produces. They're guaranteed because, if they're found to be untrue, the model doesn't build! "The thing I am telling you will be true, because it cannot be otherwise. Es muss sein. If it must be otherwise, then I must first change the thing that I am telling you." Other words here: "contract" (we're already using this one!), "commitment" around the "shape" or "structure" of the data...?

Prerequisite to me implies, a thing the user must do in order to be able to use this feature. There's some truth to that—the user must define the name and data_type for every column in yaml, in advance of running it — but the real feature here is a thing that dbt is doing. Before the model builds, dbt will ensure that the SQL and the yaml match up. This is the pre-flight check. Other words I could think of here: "verification," "safeguard," "precaution."

@matthewshaver
Copy link
Contributor

I went with "verification" rather than prerequisites.

Regarding the contracts, I slipped the word "guarantee" back in and clarified it with "attestations," which might be a more comfortable and broadly understood word in the software world. Let me know if that still doesn't hit right and maybe we can revert to the original form in the interest of shipping.

@dbeatty10
Copy link
Contributor

@matthewshaver

We've been using the following to phrases pretty frequently while iterating on contracts:

  • "pre-flight" check
  • "guarantees"

In the interest of shipping, how would you feel about adding those back in for now with the promise that we'll spend more time re-considering this verbiage before the final 1.5 release?

I don't really have a horse in the race for the exact phrasing here. But I do think that we'd get a lot of value out of getting and initial version of these docs published.

@matthewshaver
Copy link
Contributor

No problem. Plenty of time in the future to have these convos. It's reverted and my review is complete. Ready for merge as soon as you're ready to take it out of draft @jtcohen6 !

@jtcohen6
Copy link
Collaborator Author

Thanks for the work on this @matthewshaver @dbeatty10!

Plenty of time to have more convos & more rereads/reviews. I'm marking this as "Ready" in the spirit of getting docs live early & often, to accompany our early & often beta prereleases :)

@jtcohen6 jtcohen6 marked this pull request as ready for review February 27, 2023 13:59
@jtcohen6 jtcohen6 requested a review from a team as a code owner February 27, 2023 13:59
@matthewshaver matthewshaver merged commit d29665e into current Feb 27, 2023
@matthewshaver matthewshaver deleted the jerco/model-contracts branch February 27, 2023 14:55
@dbeatty10 dbeatty10 added the dbt-core v1.5 Docs impact for the v1.5 release (Apr 2023) label Mar 2, 2023
@dbeatty10 dbeatty10 added this to the dbt Core v1.5 Documentation milestone Apr 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content dbt-core v1.5 Docs impact for the v1.5 release (Apr 2023) guides Knowledge best suited for Guides size: large This change will more than a week to address and might require more than one person
Projects
None yet
4 participants