feat(snowflake_hook): Adding oauth support for SnowflakeHook #47073

Sharashchandra · 2025-02-25T17:34:04Z

This PR enhances the SnowflakeHook in the Airflow repo by completing the OAuth authentication flow. It adds logic to fetch the access token using the refresh_token, client_id, and client_secret, which are already managed in the extras of the Snowflake connection object.

Changes & Fixes:
Implements logic to exchange the refresh_token for a new access token before establishing a connection.
Resolves an issue where using refresh_token, client_id, and client_secret directly with SnowflakeHook results in the error:
main/newsfragments).

snowflake.connector.errors.DatabaseError: 250001 (08001): None: Failed to connect to DB: <account>.snowflakecomputing.com:443. Invalid OAuth access token.

bugraoz93

Looks good! Thanks for the changes! Only have a small nit.

bugraoz93 · 2025-02-25T18:16:34Z

providers/snowflake/src/airflow/providers/snowflake/hooks/snowflake.py

+            access_token = self.get_oauth_token(conn_config=conn_config)
+            conn_config["token"] = access_token


Suggested change

access_token = self.get_oauth_token(conn_config=conn_config)

conn_config["token"] = access_token

conn_config["token"] = self.get_oauth_token(conn_config=conn_config)

Not a blocker but a small nit.

Made this change and pushed.

bugraoz93

Amazing thanks!

potiuk · 2025-02-25T22:56:58Z

conflicts :(

Sharashchandra · 2025-02-26T02:14:52Z

Merge conflicts have been resolved!

…n_config dict instead of via variable

…OAuth

* Edge worker maintenance mode can be set from cli * apply review findings * add misc * delete outdated code part --------- Co-authored-by: Majoros Donat (XC-DX/EET2-Bp) <donat.majoros2@hu.bosch.com>

The apache#47105 wrongly assumed that the new ydb version caused the problem with grpc.io, it turned out however that the root cause was wrong grpcio specification for yandexcloud - where grpcio requirement did not match runtime check for version. Excluding the versions of yandexcloud (and asking them to yank those versions) should help in solving the problem and it should not limit users to upgrade to newer versions of yandexcloud (when they are not using apache.beam or other libraries that limit grpcio to < 1.68.1)

* Add Mapped Task Summary page * Remove mapped task instance lookup in dag breadcrumb

`airflow webserver` was added to the list of legacy commands, which means users will be informed of the replacement instead of just getting a generic error: ``` $ airflow webserver Usage: airflow [-h] GROUP_OR_COMMAND ... ... airflow command error: argument GROUP_OR_COMMAND: Command `airflow webserver` has been removed. Please use `airflow api-server`, see help above. ```

The intent behind this isinstance check was to deal with this, but it was missed that you can't import it on Airflow 3. Oops Closes apache#47139

…ty (apache#47145)

…#47153)

This included renaming the following args to better match fastapi/uvicorn: - `--hostname` -> `--host` - `--debug` -> `--dev` The following config options were removed (there isn't an equivalent currently): - `[webserver] error_logfile` - `[webserver] access_logformat` The following config options were moved/renamed to the `api` section. - `[webserver] web_server_host` has been moved to `[api] host` - `[webserver] web_server_port` has been moved to `[api] port` - `[webserver] workers` has been moved to `[api] workers` - `[webserver] web_server_worker_timeout` has been moved to `[api] worker_timeout` - `[webserver] web_server_ssl_cert` has been moved to `[api] ssl_cert` - `[webserver] web_server_ssl_key` has been moved to `[api] ssl_key` - `[webserver] access_logfile` has been moved to `[api] access_logfile`

…pache#47155) When provider.yaml or pypyroject.toml changes for a provider, that should trigger CI-image bound static checks - because only there we verify provider.yaml and it might trigger CI image build to get new dependencies that might fail the CI image build.

* WIP: List Backfills * Resolve review comments * Removed id and split dates into from-to * Updated duration * Append s to duration field * Remove sorting and adding new columns * Updated icon in dag

The Tasksdk datamodel codegen has not been fixed and when new version was released it caused pre-commit to fail. Fixing it, to specific version should add reproducibility, at the same time dependabot of ours will create PRs when new version is released so we are not loosing auto-upgrade features.

…ache#47148) * Add methodtools as a dependency of the common-sql provider Since `apache-airflow-providers-common-sql==1.23.0`, released on 26 February 2025, users started facing the error: ``` from cosmos.operators._asynchronous.bigquery import DbtRunAirflowAsyncBigqueryOperator cosmos/operators/_asynchronous/bigquery.py:8: in <module> from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator ../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/bigquery.py:32: in <module> from airflow.providers.common.sql.operators.sql import ( # type: ignore[attr-defined] # for _parse_boolean ../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/operators/sql.py:[29](https://github.com/astronomer/astronomer-cosmos/actions/runs/13544957799/job/37854234771#step:6:30): in <module> from airflow.providers.common.sql.hooks.sql import DbApiHook, fetch_all_handler, return_single_query_results ../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.12-2.9/lib/python3.12/site-packages/airflow/providers/common/sql/hooks/sql.py:37: in <module> from methodtools import lru_cache E ModuleNotFoundError: No module named 'methodtools' ``` It seems this started happening since apache#41327. This PR aims to solve this issue. Closes: apache#47147 * Fix linting * Revert change to provider.yaml and applyc hange to pyproject.toml

…e#47170)

This makes the process of developing and debugging much more plesant, as without this we get 0 logging (just a "500" access log line) in case of an unhandled exception.

* Update INTHEWILD.md This PR adds Blue3 Investimentos to the INTHEWILD.md file, which tracks organizations using this project in production. Blue3 Investimentos is a financial investment company that has incorporated this tool into their workflow. * Update INTHEWILD.md Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * sort correct --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

Common.sql self-deprecated itself after apache#43747 by importing handlers from the old location.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

Sharashchandra · 2025-02-28T04:13:16Z

Closing this PR since I messed up with updating the branch, will fix this and reopen the PR.

Sharashchandra requested review from mik-laj and potiuk as code owners February 25, 2025 17:34

boring-cyborg bot added the provider:snowflake Issues related to Snowflake provider label Feb 25, 2025

bugraoz93 approved these changes Feb 25, 2025

View reviewed changes

Sharashchandra force-pushed the snowflake_hook_oauth_no_merge_commit branch from 90fba16 to 2791f73 Compare February 26, 2025 02:14

Sharashchandra force-pushed the snowflake_hook_oauth_no_merge_commit branch 3 times, most recently from e130f3f to 64c4213 Compare February 26, 2025 17:20

Sharashchandra requested a review from bugraoz93 February 26, 2025 17:34

bugraoz93 approved these changes Feb 27, 2025

View reviewed changes

Sharashchandra added 2 commits February 27, 2025 17:43

feat(snowflake_hook): Adding oauth support for SnowflakeHook

bc1323a

refactor(snowflake_hook): adding the access_token directly to the con…

437b17a

…n_config dict instead of via variable

Sharashchandra force-pushed the snowflake_hook_oauth_no_merge_commit branch from 64c4213 to 437b17a Compare February 27, 2025 12:13

Sharashchandra and others added 14 commits February 27, 2025 17:54

docs(snowflake_connection): Updating the docs to include changes for …

6c70f48

…OAuth

Edge worker maintenance mode can be set from cli (apache#47140)

7bc7d64

* Edge worker maintenance mode can be set from cli * apply review findings * add misc * delete outdated code part --------- Co-authored-by: Majoros Donat (XC-DX/EET2-Bp) <donat.majoros2@hu.bosch.com>

Add Mapped Task Summary page (apache#47120)

897cdb8

* Add Mapped Task Summary page * Remove mapped task instance lookup in dag breadcrumb

bugfix: cannot import name 'workloads' for Airflow v2 (apache#47152)

94770ac

The intent behind this isinstance check was to deal with this, but it was missed that you can't import it on Airflow 3. Oops Closes apache#47139

Fix standalone command by removing the usage of webserver_port proper…

cbe3cf2

…ty (apache#47145)

Invalidate GetDag query on toggle pause to keep pages in sync (apache…

17db9c9

…#47153)

Fix Task Mapping with XCOM arguments from other Tasks (apache#47141)

905b8c4

doc: fix the typo for glob regex apache#47116 (apache#47134)

6788815

fix broken link (apache#47127)

c11b4fc

Update bug report template for Airflow 3 (apache#47144)

b0b41a9

vatsrahul1001 and others added 12 commits February 28, 2025 09:39

Fix Operator link for TriggerDagRunOperator (apache#47051)

c5b4c5a

AIP-38 Fix log key warning (apache#47157)

7e7d4f3

Create list backfills tab in dag page (apache#46772)

95cd658

* WIP: List Backfills * Resolve review comments * Removed id and split dates into from-to * Updated duration * Append s to duration field * Remove sorting and adding new columns * Updated icon in dag

Fix section for base_url in FAB auth manager (apache#47173)

1c8b069

Disable past/future Clear TI buttons when logical date is null (apach…

48f8b2c

…e#47170)

Add in 500 error logging for the ExecutionAPI app (apache#47159)

b120a24

This makes the process of developing and debugging much more plesant, as without this we get 0 logging (just a "500" access log line) in case of an unhandled exception.

doc: fix contribution doc ports (apache#47176) (apache#47177)

6d504ef

fIx deprecation warnings in common.sql (apache#47169)

6cfdef0

Common.sql self-deprecated itself after apache#43747 by importing handlers from the old location.

Add bundle name arg to list dags cli command (apache#45779)

edb798c

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

Sharashchandra requested review from ashb, bbovenzi, eladkal, ephraimbuddy, hussein-awala, jedcunningham, jscheffl, pierrejeambrun and ryanahamilton as code owners February 28, 2025 04:09

Sharashchandra closed this Feb 28, 2025

Sharashchandra mentioned this pull request Feb 28, 2025

Adding OAuth support for SnowflakeHook #47191

Merged

This was referenced May 5, 2025

Status of testing Providers that were prepared on May 05, 2025 #50189

Closed

Status of testing Providers that were prepared on May 08, 2025 #50356

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(snowflake_hook): Adding oauth support for SnowflakeHook #47073

feat(snowflake_hook): Adding oauth support for SnowflakeHook #47073

Uh oh!

Sharashchandra commented Feb 25, 2025

Uh oh!

bugraoz93 left a comment

Uh oh!

bugraoz93 Feb 25, 2025

Uh oh!

Sharashchandra Feb 25, 2025

Uh oh!

bugraoz93 left a comment

Uh oh!

potiuk commented Feb 25, 2025

Uh oh!

Sharashchandra commented Feb 26, 2025

Uh oh!

Sharashchandra commented Feb 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants

		access_token = self.get_oauth_token(conn_config=conn_config)
		conn_config["token"] = access_token

	access_token = self.get_oauth_token(conn_config=conn_config)
	conn_config["token"] = access_token
	conn_config["token"] = self.get_oauth_token(conn_config=conn_config)

feat(snowflake_hook): Adding oauth support for SnowflakeHook #47073

feat(snowflake_hook): Adding oauth support for SnowflakeHook #47073

Uh oh!

Conversation

Sharashchandra commented Feb 25, 2025

Uh oh!

bugraoz93 left a comment

Choose a reason for hiding this comment

Uh oh!

bugraoz93 Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

Sharashchandra Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

bugraoz93 left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Feb 25, 2025

Uh oh!

Sharashchandra commented Feb 26, 2025

Uh oh!

Sharashchandra commented Feb 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

18 participants