HttpHook.run() does not use passed in extra_options if the connection doesn't have Extra set #51685
Description
Apache Airflow Provider(s)
http
Versions of Apache Airflow Providers
apache-airflow-providers-http==5.3.0
Apache Airflow version
2.11.0
Operating System
debian bookworm
Deployment
Official Apache Airflow Helm Chart
Deployment details
not relevant
What happened
A HttpHook was instantiated: hook = HttpHook(...)
The Connection that the hook uses has no Extra defined.
When calling run('/some-endpoint', extra_options={"max_redirects": 0}), the max_redirects is not used in the request.
What you think should happen instead
The extra_options dict passed in to run() should be used, regardless of whether or not the Connection has any Extra values configured.
How to reproduce
import airflow
from airflow.providers.standard.operators.python import PythonOperator
from airflow.providers.http.hooks.http import HttpHook
def http_hook_test():
extra_options = {"max_redirects": 0}
# tinyurl is a HTTP Connection with Host=https://tinyurl.com (and no Extra values)
http_hook = HttpHook(method='GET', http_conn_id='tinyurl')
response = http_hook.run(
# https://tinyurl.com/3ssbpdcj redirects to https://google.com
endpoint="/3ssbpdcj",
extra_options=extra_options,
)
raise RuntimeError("This code should not be reached, because requests.exceptions.TooManyRedirects was raised")
with airflow.DAG(
dag_id='http_hook_test_dag',
schedule_interval=None,
start_date=airflow.utils.dates.days_ago(1),
catchup=False,
) as dag:
test_http_hook = PythonOperator(
task_id='test_http_hook',
python_callable=http_hook_test,
)
Anything else
This also seems to apply to the Airflow 3.0.2 HttpHook.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct