Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] dbt-snowflake opens a new connection per each model #1254

Open
2 tasks done
TiansuYu opened this issue Nov 27, 2024 · 1 comment
Open
2 tasks done

[Bug] dbt-snowflake opens a new connection per each model #1254

TiansuYu opened this issue Nov 27, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@TiansuYu
Copy link

Is this a new bug in dbt-snowflake?

  • I believe this is a new bug in dbt-snowflake
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The way I am currently using dbt-snowflake locally is that, I have a dbt profile as follows:

my-snowflake:
  target: dev
  outputs:
    dev:
        type: snowflake
        account: some-account
        authenticator: externalbrowser
        
        database: xyz
        user: xyz
        role: xyz
        warehouse: xyz
        schema: ""

        threads: 4

However, running dbt test creates a snowflake connection per each model. This fires a huge amount of connection request on my browser, and I barely have time to click on Duo for 2FA requests. After a few try, because I wasn't responding enough on Duo, the 2FA would lock me out.

I am wondering what is the connection management practices in this repo looks like? Has the connection pool been properly managed? https://docs.snowflake.com/en/developer-guide/node-js/nodejs-driver-connect#creating-a-connection-pool

Expected Behavior

Each dbt run should create a proper connection, ideally one per each thread, and whatever dbt jobs running should pick up connections from the pool, instead of firing auth request like it is free candy.

Steps To Reproduce

  1. Create a dbt profile in ~/.dbt/profile.yml
  2. set authenticator: externalbrowser
  3. Use an example dbt project with a lot of models and run dbt deps && dbt test.

Relevant log output

No response

Environment

- OS: MacOs
- Python: 3.12
- dbt-core: 1.8.8
- dbt-snowflake: 1.8.4

Additional Context

No response

@TiansuYu TiansuYu added bug Something isn't working triage labels Nov 27, 2024
@amychen1776
Copy link

@TiansuYu Have you enabled token caching? We have some links in our docs here https://docs.getdbt.com/docs/cloud/connect-data-platform/connect-snowflake#snowflake-mfa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants