Microbatch: batched execution #10677

MichelleArk · 2024-09-06T20:56:31Z

Resolves #10700

Problem

For microbatch models, it should be possible to:

Split the execution of a microbatch model into batches, regardless of whether it is a full-refresh or incremental run
Run a single adapter (main) query per batch (i.e. partition)
Fail gracefully if a single batch fails

Solution

Move start/end time computation from provider to MicrobatchBuilder
Build start/end + batches during ModelRun execution for microbatch models
Execute each batch:
- Populate jinja context vars for recompilation of refs for each batch
- Re-compile model + update jinja context vars like is_incremental + should_full_refresh
- Create run result

Checklist

I have read the contributing guide and understand what's expected of me.
I have run this code in development, and it appears to resolve the stated issue.
This PR includes tests, or tests are not required or relevant for this PR.
This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
This PR includes type annotations for new and modified functions.

🎩 Example batch-level failure:

…ime usage

…ack + batch_size work

…me_filter

…time objects

github-actions · 2024-09-06T20:56:46Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

codecov · 2024-09-06T22:25:03Z

Codecov Report

Attention: Patch coverage is 98.63014% with 2 lines in your changes missing coverage. Please review.

Project coverage is 88.95%. Comparing base (c6b8f7e) to head (737ae3d).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10677      +/-   ##
==========================================
+ Coverage   88.90%   88.95%   +0.05%     
==========================================
  Files         180      181       +1     
  Lines       22856    22959     +103     
==========================================
+ Hits        20319    20423     +104     
+ Misses       2537     2536       -1

Flag	Coverage Δ
integration	`86.14% <88.35%> (+0.03%)`	⬆️
unit	`62.37% <58.21%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Unit Tests	`62.37% <58.21%> (-0.06%)`	⬇️
Integration Tests	`86.14% <88.35%> (+0.03%)`	⬆️

core/dbt/task/run.py

MichelleArk · 2024-09-16T05:40:16Z

🤷 clicking into the failing codecov check, things look okay! perhaps the GH response is stale somehow.

pairing.md

QMalcolm

Requesting changes and am gonna follow up by making said changes 😅

QMalcolm · 2024-09-17T20:37:19Z

core/dbt/task/run.py

+        microbatch_builder = MicrobatchBuilder(
+            model=model,
+            is_incremental=self._is_incremental(model),
+            event_time_start=getattr(self.config.args, "EVENT_TIME_START", None),
+            event_time_end=getattr(self.config.args, "EVENT_TIME_END", None),
+        )
+        end = microbatch_builder.build_end_time()
+        start = microbatch_builder.build_start_time(end)


When I was reading through MicrobatchBuilder I found it odd that build_end_time and build_start_time weren't private functions that the __init__ would then call to default the event_time_start/event_time_end. With this bit of code here, I find myself still thinking so. Basically, the start/end are always going to be specific to a given MicrobatchBuilder instance. Perhaps as a fast follow to this PR we should investigate if this can be reduced to that. The code here would then become

... microbatch_builder = MicrobatchBuilder( model=model, is_incremental=self._is_incremental(model), event_time_start=getattr(self.config.args, "EVENT_TIME_START", None), event_time_end=getattr(self.config.args, "EVENT_TIME_END", None), ) batches = microbatch_builder.build_batches() ...

The alternative would be to take the MicrobatchBuilder class to be less specific to a model, which also seems like a valid approach. Right now though we seem to be somewhere in the middle with a class that is specific to a model, but methods that we're expected to call which should never change their return given the associated model.

QMalcolm · 2024-09-17T20:38:21Z

core/dbt/task/run.py

+        # TODO: Remove. This is a temporary method. We're working with adapters on
+        # a strategy to ensure we can access the `is_incremental` logic without drift


We should follow up with this on @mikealfare about the possible timeline

core/dbt/task/run.py

QMalcolm · 2024-09-17T20:46:06Z

core/dbt/task/run.py

+            if (
+                os.environ.get("DBT_EXPERIMENTAL_MICROBATCH")
+                and model.config.materialized == "incremental"
+                and model.config.incremental_strategy == "microbatch"
+            ):
+                batch_results = self._execute_microbatch_materialization(
+                    model, manifest, context, materialization_macro
+                )
+            else:
+                result = MacroGenerator(
+                    materialization_macro, context, stack=context["context_macro_stack"]
+                )()
+                for relation in self._materialization_relations(result, model):
+                    self.adapter.cache_added(relation.incorporate(dbt_created=True))


I know we talked about this work, and I absolutely understand why we're doing it this way. Something feels smelly about it though, and I can't exactly put my finger on it. My best guess is that we're doing a conditional, exiting the conditional, and then basically re-entering the conditional on lines 396-399. For instance line 399 should never be hit if we enter line 384. However because of the split conditionals, this isn't immediately apparent. I wonder if we should be calling into two separate private functions just before the try on line 378, and only one or the other function would ever be called depending on if we're doing microbatch stuff or not.

QMalcolm · 2024-09-17T20:46:49Z

core/dbt/task/run.py

+            status=RunStatus.Success,
+            timing=[],
+            thread_id=threading.current_thread().name,
+            # TODO -- why isn't this getting propagated to logs?


The execution_time isn't making it to the logs? 🤔 That's odd....

core/dbt/materializations/incremental/microbatch.py

QMalcolm

My remaining open comments can be addressed at a later date. Let's move forward 🚀

MichelleArk and others added 18 commits August 22, 2024 16:34

initial rough-in with CLI flags

c4930e9

dbt-adapters testing against event-time-ref-filtering

4c8528b

Merge branch 'main' into event-time-ref-filtering

3bb6807

fix TestList

f5d5bb6

Checkpoint

19ad7c6

fix tests

a57481f

add event_time_start params to build

25c10f7

rename configs

699179f

Gate resolve_event_time_filter via micro batch strategy and fix strpt…

2d19d1c

…ime usage

Add unit test for resolve_event_time_filter

57b1353

Additional unit tests for resolve_event_time_filter to ensure lookb…

e0bae27

…ack + batch_size work

Remove extraneous comments and print statements from resolve_event_ti…

1313aff

…me_filter

Fixup microbatch functional tests to use microbatch strategy

7307c02

Gate microbatch functionality behind env_var while in beta

838a0aa

Add comment about how _is_incremental should be removed

43715de

Improve event_time_start/end cli parameters to auto convert to date…

e38ff47

…time objects

for testing: dbt-postgres 'microbatch' strategy

457698c

rough in: chunked backfills

3f8369f

cla-bot bot added the cla:yes label Sep 6, 2024

MichelleArk added 2 commits September 6, 2024 18:16

partial failure of microbatch runs

c64b538

decouple run result methods

62f7675

initial refactor

601e333

MichelleArk mentioned this pull request Sep 9, 2024

Add EventTimeFilter and BaseRelation.render_event_time_filtered dbt-labs/dbt-adapters#285

Merged

MichelleArk commented Sep 9, 2024

View reviewed changes

core/dbt/task/run.py Outdated Show resolved Hide resolved

QMalcolm force-pushed the event-time-ref-filtering branch from e4138c5 to 3a6c739 Compare September 11, 2024 16:58

Base automatically changed from event-time-ref-filtering to main September 12, 2024 22:16

Merge branch 'main' into microbatch-chunked-backfill

dc7d4b7

MichelleArk changed the title ~~rough in: chunked backfills~~ Microbatch: batched execution Sep 14, 2024

MichelleArk force-pushed the microbatch-chunked-backfill branch from c6b2ccc to 71a526c Compare September 14, 2024 02:32

rename configs to __dbt_internal

a31e703

MichelleArk force-pushed the microbatch-chunked-backfill branch from 71a526c to a31e703 Compare September 14, 2024 02:58

MichelleArk added 9 commits September 13, 2024 23:01

update compiled_code in context after re-compilation

2fe7ba0

finish rename of context vars

4f69b83

changelog entry

0b6fccf

Merge branch 'main' into microbatch-chunked-backfill

636f1aa

fix patch_microbatch_end_time

a7aef47

refactor into MicrobatchBuilder

ffaccc8

fix provider unit tests + add unit tests for MicrobatchBuilder

4d462d7

add TestMicrobatchJinjaContextVarsAvailable

1f0f7be

unit test offset + truncate timestamp methods

9f16fd6

MichelleArk marked this pull request as ready for review September 16, 2024 05:40

MichelleArk requested a review from a team as a code owner September 16, 2024 05:40

MichelleArk requested a review from QMalcolm September 16, 2024 05:41

QMalcolm reviewed Sep 17, 2024

View reviewed changes

pairing.md Outdated Show resolved Hide resolved

QMalcolm requested changes Sep 17, 2024

View reviewed changes

QMalcolm added 3 commits September 17, 2024 15:50

Remove pairing.md file

c0832d2

Add tying to microbatch specific functions added in task/run.py

2e44489

Add doc strings to microbatch.py functions and classes

3fdd88e

QMalcolm approved these changes Sep 17, 2024

View reviewed changes

QMalcolm added 4 commits September 18, 2024 00:11

Merge branch 'main' into microbatch-chunked-backfill

5e66486

Set microbatch node status to ERROR if all batches for node failed

fc3f75e

Fire an event for batch exceptions instead of directly printing

c1f8461

Fix firing of failed microbatch log event

737ae3d

QMalcolm merged commit 8fe5ea1 into main Sep 18, 2024
61 of 62 checks passed

QMalcolm deleted the microbatch-chunked-backfill branch September 18, 2024 16:46

QMalcolm mentioned this pull request Sep 18, 2024

Split out model vs microbatch execution #10737

Merged

5 tasks

QMalcolm mentioned this pull request Nov 7, 2024

Add microbatch strategy dbt-labs/dbt-redshift#924

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Microbatch: batched execution #10677

Microbatch: batched execution #10677

MichelleArk commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 6, 2024

codecov bot commented Sep 6, 2024 •

edited

Loading

MichelleArk commented Sep 16, 2024 •

edited

Loading

QMalcolm left a comment

QMalcolm Sep 17, 2024

QMalcolm Sep 17, 2024

QMalcolm Sep 17, 2024

QMalcolm Sep 17, 2024

QMalcolm left a comment

		# TODO: Remove. This is a temporary method. We're working with adapters on
		# a strategy to ensure we can access the `is_incremental` logic without drift

Microbatch: batched execution #10677

Microbatch: batched execution #10677

Conversation

MichelleArk commented Sep 6, 2024 • edited Loading

Problem

Solution

Checklist

github-actions bot commented Sep 6, 2024

codecov bot commented Sep 6, 2024 • edited Loading

Codecov Report

MichelleArk commented Sep 16, 2024 • edited Loading

QMalcolm left a comment

Choose a reason for hiding this comment

QMalcolm Sep 17, 2024

Choose a reason for hiding this comment

QMalcolm Sep 17, 2024

Choose a reason for hiding this comment

QMalcolm Sep 17, 2024

Choose a reason for hiding this comment

QMalcolm Sep 17, 2024

Choose a reason for hiding this comment

QMalcolm left a comment

Choose a reason for hiding this comment

MichelleArk commented Sep 6, 2024 •

edited

Loading

codecov bot commented Sep 6, 2024 •

edited

Loading

MichelleArk commented Sep 16, 2024 •

edited

Loading