OAUTHv2 Authentication

[ozerpekozcan]

Description

Hi,

Currently, Airflow supports simple HTTP authentication. However, it would be beneficial to have built-in OAuth authentication, where the token URL, scope, and other HTTP parameters can be extracted from the connection. This would allow authentication using a token before making requests to the actual endpoint.

Additionally, if the token is expired, Airflow should automatically fetch a new one to prevent timeout errors. The implementation should retain all features of HttpOperator and HttpSensor, including support for deferrable execution.

This enhancement would improve security and reliability when integrating with OAuth-protected APIs.

Thanks,
Ozer

Use case/motivation

A customizable OAuth authentication mechanism would allow Airflow users to:

• Define OAuth-specific parameters in Airflow Connections (e.g., token URL, scopes, client ID, client secret, grant type).
• Automatically handle token renewal when expired, avoiding job failures.
• Support both synchronous and deferrable execution, improving resource efficiency.
• Extend the existing HttpHook, HttpOperator, and HttpSensor to support OAuth-based authentication without altering existing API logic.

Motivation:

• Standardization – A unified OAuth authentication method ensures consistency across different APIs without custom coding for each integration.
• Security – Storing OAuth credentials securely in Airflow Connections reduces exposure and enhances compliance with security policies.
• Reduced Operational Overhead – Automating token retrieval and refresh eliminates the need for manual token management in DAGs.
• Deferrable Execution Support – Avoids resource-intensive polling by deferring execution while waiting for authentication responses.
• Scalability – Easily extendable to new APIs by simply updating connection details instead of modifying DAG logic.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[amolsr]

I don't think we should be needing this. The airflow runs as internal project. Why would i be needing the provide higher authentication standards.

[mik-laj]

@ozerpekozcan Could you give more details about which OAuth2 flows you would like to support? OAuth2 in popular flows expects interaction with the user and the browser.

[ozerpekozcan]

Hello @mik-laj,
Browser integration isn’t actually necessary. The main difference between Basic and OAuth authentication is that with OAuth, you need to POST your credentials to obtain a Bearer token before you're allowed to access certain APIs. This token is then used to authorize subsequent API operations.

[mik-laj]

Do you mean to implement Client Credentials Flow?

We can add it, but currently Airflow doesn't have any storage to store temporary keys, so in practice this means that a new key has to be generated every time it's needed, so it is a rare case of Airflow using a token that has already expired.

If you are interested in contributing I can review it, but please add documentation and specify which specific flow is supported by Airflow.

[hendrix04]

I am actualy really surprised that Airflow doesn't support the Client Credentials flow for Oauth as I would have assumed this would have been the first implimentation. It is LITERALLY meant for machine to machine communication. Since Airflow is used for ETL, which is all machine to machine communication, the fact that this thread exists blows my mind.

I am weary of @ozerpekozcan's explination of the "main difference" between Basic Auth and OAuth as that definitely isn't the main difference. That said, if @ozerpekozcan has only ever been intorduced to the Client Credentials flow then I could understand their confusion.

Some connectors already support setting refresh tokens and retriving / caching access tokens. The problem here is that while it may work for a lot of people, refresh tokens have wildly variable shelf lives and in some regulated industries, they last a couple of days at most. In this case, whoever owns that connection would have to create a new refresh token every day or two and then update the airflow connection to have the new refresh token. You could imagine how this is not a great experience for an end user.

@mik-laj, I am in no way an Airflow expert, but I am surprised that Airflow wouldn't have a decent way to cache this data. XComs has a specific purpose so someone wouldn't want to hijack that. The other quick option that I found while not great, is environment variables.

I absolutely hate both of those options. @mik-laj, would you have any suggestions on where to start here? It seems like where / how this cache should live is the biggest hurdle to jump over.

[ozerpekozcan]

@mik-laj, @hendrix04 maybe persisting the token in back-end database (postgres or mysql) in encrypted format, including expiry timestamp can be an option. But I don't know if it's a secure way. Actually airflow allows the user login using client credentials. For instance it's possible to use Google authentication. Therefore, required libraries are already available.

[vincbeck]

This issue has been resolved by #59245. Feel free to re-open the issue if you think the issue is still there