Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate build and docs model flags #1757

Closed
tayloramurphy opened this issue Sep 17, 2019 · 2 comments
Closed

Separate build and docs model flags #1757

tayloramurphy opened this issue Sep 17, 2019 · 2 comments
Labels
enhancement New feature or request

Comments

@tayloramurphy
Copy link

Describe the feature

Following from #1671

One weird use case we have sometimes is that we need to temporarily disable a model from building (b/c of a failing test or bad data or something) but we still want the docs to show for that model. Do you think this could be abstracted enough so that you could separate build from docs at the dbt_project.yml level?

Something like:

  netsuite_stitch:
    build: false
    docs: true
    base:
      materialized: table
    xf:
      materialized: table

but enabled: false would set both build and docs to false?

Follow up from Drew:

my broad thinking is that this should be implemented with some sort of no-op materialization? I think that strikes the right balance between:

  • preserving the model in the DAG
  • rendering the model in the documentation
  • not actually running anything against the database

This will of course fail if the destination table doesn't exist and downstream consumers depend on the model, but it sounds to me like that's a fair and reasonable tradeoff here. Definitely feel free to create a new issue if you'd like to discuss further!

To reiterate, the main goal in this request is that we want to preserve the structure of the dag but not actual run the model and its tests for a certain amount of time while data quality issues are sorted out. Our process currently is to disable a model temporarily while we fix the data bugs - this keeps the pipelines green - and then we reenable them once the data is sorted out.

I do like the proposal b/c in our case the model will pass but the tests will fail. A third option could be to have a tests: false setting as well 👀

Additional context

We're on Snowflake

Who will this benefit?

This seems like a pretty niche request. I mainly brought it up since #1671 was being worked on and it seemed like a similar use-case / part of the code base.

@tayloramurphy tayloramurphy added enhancement New feature or request triage labels Sep 17, 2019
@drewbanin drewbanin removed the triage label Sep 17, 2019
@drewbanin
Copy link
Contributor

Thanks @tayloramurphy!

Do you think that setting the test severity to WARN for these tests a viable approach for you? This would keep the tests from failing while you take the time to fix any data quality issues.

Another idea here would be to embrace the failures. Rather than try to disable a model but keep it rendering in the docs, maybe we could add a flag like broken: true (maybe we can find a more political name 😛) which will:

  1. disable the model from running, or ignore build failures, or something
  2. render the docs for the model, but show a big warning that it's broken

I don't know how good an idea this is, but I do think it's worth discussing!

@drewbanin
Copy link
Contributor

I don't think this is something we're planning on prioritizing in the next 6 months, so I'm going to close it. @tayloramurphy if you still feel strongly about this LMK and we can re-open it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants