[Bug] Microbatch can cause dbt runs to hang

[QMalcolm]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

You can cause microbatch models to hang the dbt runtime. This is most easily reproduced by providing the --fail-fast flag with a microbatch that has an error in it, full reproduction steps can be found https://github.com/dbt-labs/TMP_MICROBATCH_HANGING_REPRO. The hanging has been observed in other manners, but they are not as easily reproduced.

Expected Behavior

Microbatch models shouldn't cause the dbt runtime to hang.

Steps To Reproduce

https://github.com/dbt-labs/TMP_MICROBATCH_HANGING_REPRO

Relevant log output

Environment

- OS: MacOS 15.1.1
- Python: 3.11.9
- dbt: 1.9.2

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

[john-rotenstein]

We experienced a similar issue in dbt Cloud when using Microbatching and Slim CI. dbt acknowledged the problem: "the root cause of this issue has been identified and a fix is being implemented"

[jhsb25]

We experienced a similar issue in dbt Cloud when using Microbatching and Slim CI. dbt acknowledged the problem: "the root cause of this issue has been identified and a fix is being implemented"

@john-rotenstein one workaround is to increase your threads for your DBT job significantly. I think this makes the pool of thread big enough that the main tthread is never utilised and blocked. Anyway 70 was chosen as an arbitraru number for us and it seems to have done the job

[markostefanovic1]

one workaround is to increase your threads for your DBT job significantly. I think this makes the pool of thread big enough that the main tthread is never utilised and blocked. Anyway 70 was chosen as an arbitraru number for us and it seems to have done the job

This is true for me too. When I was initially testing microbatch models, I was using a single thread, so when building a single model, the execution would hang after the first batch was finished building. When I increased the number of threads to 2, it started working normally. (dbt-core==1.9.6, dbt-snowflake==1.9.4)

But now I am encountering a problem when I do the full build and multiple microbatch models start executing simultaneously - 2 microbatch models start executing, finish the first batch and then hang.

I guess I'll try increasing the thread number like you said to a crazy number and hope for this to be fixed soon.

[matangwbr]

I'm running my project on version 1.10.4 with snowflake adapter version 1.10.0, and it seems the bug isn't resolved.
I'm trying to run multiple batches (more than 10), If I'm using 1 thread it get stuck after one batch, If I'm using more threads, more batches are executed but still get stuck after some.
I think you should reopen the bug

[glqstrauss]

I am experiencing the same issue as @matangwbr running with dbt=1.10.6, snowflake adapter=1.10.0, with 4 threads and 6 microbatch models. Hangs after completing the first batch for 4 models (so presumably using all 4 threads there). Let me know if I should open a new issue if this one is not going to be reopened. Thanks!

[QMalcolm]

I am experiencing the same issue as @matangwbr running with dbt=1.10.6, snowflake adapter=1.10.0, with 4 threads and 6 microbatch models. Hangs after completing the first batch for 4 models (so presumably using all 4 threads there). Let me know if I should open a new issue if this one is not going to be reopened. Thanks!

The current tracking issue can be found here #11420.

The original issue was resolved. The issue was that if the number of batches being run ever matched number of available threads, whether allowing for concurrent batches or not, then control was never properly returned to the main thread.

The new issue is that when a microbatch model is run with concurrent batches, sometimes one of the batches fails to ever report as "done". Thus that model never realizes it is complete and causes execution to hang.

Given there is already an open issue, #11420, a separate one shouldn't be opened (unless different circumstances of microbatch hanging arise). The workaround is to set concurrent_batches=false on the model experiencing the issue, which is unfortunate.