Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] dbt snapshot --defer #4110

Closed
1 task done
iljau opened this issue Oct 21, 2021 · 5 comments · Fixed by #4296 or #4514
Closed
1 task done

[Feature] dbt snapshot --defer #4110

iljau opened this issue Oct 21, 2021 · 5 comments · Fixed by #4296 or #4514
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors! state Stateful selection (state:modified, defer)

Comments

@iljau
Copy link

iljau commented Oct 21, 2021

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Implement "dbt snapshot --defer", which would behave similar to "dbt run --defer".

Describe alternatives you've considered

No response

Who will this benefit?

When trying to set up "slim ci", one step in my case would be to run snapshots based on other dbt models.

dbt snapshot --select 'dbt.*,state:modified' --state manifests/staging

When running this command, generated references point to current schema, which doesn't have these tables, as they weren't generated during preceding run command:
dbt run --models state:modified --defer --state manifests/airflow/staging

Are you interested in contributing this feature?

No response

Anything else?

https://docs.getdbt.com/reference/node-selection/defer
https://docs.getdbt.com/docs/guides/best-practices#run-only-modified-models-to-test-changes-slim-ci

dbt version: 0.20.1

@iljau iljau added enhancement New feature or request triage labels Oct 21, 2021
@jtcohen6
Copy link
Contributor

@iljau Thanks for opening!

The steps here are quite straightforward. I'm going to mark this a good first issue, and I'd welcome a PR from a community member.

  1. Include snapshot in the subcommands that get the --defer flag:

    dbt-core/core/dbt/main.py

    Lines 1091 to 1092 in f79a968

    # --defer
    _add_defer_argument(run_sub, test_sub, build_sub)

  2. Remove defer_to_manifest override in SnapshotTask, so as to inherit the implementation from RunTask:

    def defer_to_manifest(self, adapter, selected_uids):
    # snapshots don't defer
    return

  3. Then, we'd want to add/adjust a test case combining snapshots + deferral:

    # these should work without --defer!
    self.run_dbt(['snapshot'])
    # make sure these commands don't work with --defer
    with pytest.raises(SystemExit):
    self.run_dbt(['seed', '--defer'])
    with pytest.raises(SystemExit):
    self.run_dbt(['snapshot', '--defer'])

Is this a change you'd be interested in contributing?

one step further

At this point, there's no good reason not to support --defer for compile, too, and thereby docs generate as well (#3741). (The only reason I can come up with: we might eventually want to rename/rework the UX of deferral (#2968)).

For that support:

  • Add the --defer flag to the compile and docs subparsers, too
  • Move defer_to_manifest up from RunTask into CompileTask (its parent)

alternative

The build task, new in v0.21, supports both snapshots and deferral:

$ dbt build --resource-type snapshot --defer --state manifests/airflow/staging`

So, that's something you can use in the meantime.

@jtcohen6 jtcohen6 added good_first_issue Straightforward + self-contained changes, good for new contributors! state Stateful selection (state:modified, defer) and removed triage labels Oct 21, 2021
@kadero
Copy link
Contributor

kadero commented Nov 16, 2021

Hello @jtcohen6 @iljau 👋,

Thank you for opening this issue and providing a step by step solution 🙂.
Happy to work on this feature and to raise a PR for it. Ok for you?

🙏

@iljau
Copy link
Author

iljau commented Nov 16, 2021

I switched to using dbt build and don't have immediate plans to work on a fix.

@kadero kadero mentioned this issue Nov 17, 2021
4 tasks
@ciejer
Copy link

ciejer commented Dec 5, 2021

Hi @jtcohen6 - just checking in.
Is there another issue you're aware of that covers your "one step further" note from above? We're missing deferral in catalog generation (dbt docs generate); followed the link here from #3741 but the PR for this does not appear to close that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors! state Stateful selection (state:modified, defer)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants
@iljau @ciejer @jtcohen6 @kadero and others