Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Apr 24, 2025

api/ssl_cert has a default value of "" so has_option is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag.

This PR only touches airflow core -- there will be another PR for fixing FAB
manager.

Fixes #49495


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

…cure

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag
@ashb ashb requested a review from vincbeck as a code owner April 24, 2025 14:00
@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API kind:documentation labels Apr 24, 2025
@ashb ashb added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Apr 24, 2025
@Dev-iL
Copy link
Collaborator

Dev-iL commented Apr 24, 2025

I can confirm that the change to the simple auth manager works with docker!

ashb added a commit to astronomer/airflow that referenced this pull request Apr 24, 2025
… token cookie as secure

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

Provider version of apache#49721
@ashb ashb merged commit 76edd92 into apache:main Apr 24, 2025
100 checks passed
@ashb ashb deleted the fix-simple-auth-infinite-redir branch April 24, 2025 15:06
github-actions bot pushed a commit that referenced this pull request Apr 24, 2025
…cookie as secure (#49721)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag
(cherry picked from commit 76edd92)

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
@github-actions
Copy link

Backport successfully created: v3-0-test

Status Branch Result
v3-0-test PR Link

dheerajturaga added a commit to dheerajturaga/airflow that referenced this pull request Apr 24, 2025
potiuk pushed a commit that referenced this pull request Apr 24, 2025
… token cookie as secure (#49724)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

Provider version of #49721
vincbeck pushed a commit that referenced this pull request Apr 24, 2025
vincbeck pushed a commit that referenced this pull request Apr 24, 2025
…cookie as secure (#49721) (#49733)

* [v3-0-test] Fix infinite redirect caused by mistakenly setting token cookie as secure (#49721)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag
(cherry picked from commit 76edd92)

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

* Update airflow-core/docs/core-concepts/auth-manager/index.rst

---------

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
@brechtvb
Copy link

can this be manually applied on a 3.0.0 with docker-compose?

@Dev-iL
Copy link
Collaborator

Dev-iL commented Apr 24, 2025

can this be manually applied on a 3.0.0 with docker-compose?

@brechtvb Yes it can. One has to edit the image (spin up a container, access it via the cli, apply the diff, exit the container, commit), then modify docker-compose if the image name was changed and re-run.

@brechtvb
Copy link

@Dev-iL i tried to do so, i can modify the py file, but i have no idea where to find the .rst file.

when only modifying the .py file, the docker-airflow-apiserver fails to boot

@Dev-iL
Copy link
Collaborator

Dev-iL commented Apr 24, 2025

@brechtvb disregard that one, it's just docs. I only modified the py, though I'm using the simple auth manager.

Perhaps you edited the py file incorrectly (e.g. add tab instead of spaces making it an invalid script)?

@brechtvb
Copy link

brechtvb commented Apr 24, 2025

very odd - not working 2 times from scratch. I'm familiar with python and docker.

when running python login.py (the to be modified file) there were no messages or errors, so syntax was fine.

INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 93, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
                                          ^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 64, in _jwt_validator
    **get_sig_validation_args(make_secret_key_if_needed=False),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 578, in get_sig_validation_args
    return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 535, in get_signing_key
    raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!

ERROR:    Application startup failed. Exiting.
INFO:     Waiting for child process [352]

@Dev-iL
Copy link
Collaborator

Dev-iL commented Apr 24, 2025

@brechtvb does this happen with a clean, or a migrated db? Did you prune volumes in between attempts?

@brechtvb
Copy link

yes, so i used the docker-compose setup and i do this:

docker compose down --volumes --remove-orphans

i make sure all previous containers are gone.

@Dev-iL
Copy link
Collaborator

Dev-iL commented Apr 24, 2025

@brechtvb which auth manager are you using? Did you try the other one?

kaxil pushed a commit that referenced this pull request Apr 24, 2025
…cure (#49721) (#49733)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag
(cherry picked from commit 76edd92)

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>

* Update airflow-core/docs/core-concepts/auth-manager/index.rst

---------

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
@brechtvb
Copy link

i just use this: https://airflow.apache.org/docs/apache-airflow/3.0.0/docker-compose.yaml

and modified the airflow container as the diff describes, then committed that to (local) tag: 3.0.0-fixed

and updated that in the docker-compose.yaml

@vincbeck
Copy link
Contributor

very odd - not working 2 times from scratch. I'm familiar with python and docker.

when running python login.py (the to be modified file) there were no messages or errors, so syntax was fine.

INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
    result = await _enter(cm)
             ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 93, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
                                          ^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 64, in _jwt_validator
    **get_sig_validation_args(make_secret_key_if_needed=False),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 578, in get_sig_validation_args
    return {"secret_key": get_signing_key("api_auth", "jwt_secret", make_secret_key_if_needed)}
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py", line 535, in get_signing_key
    raise ValueError(f"The value {section}/{key} must be set!")
ValueError: The value api_auth/jwt_secret must be set!

ERROR:    Application startup failed. Exiting.
INFO:     Waiting for child process [352]

Looking at the error message you need to set the config [api_auth] jwt_secret

@brechtvb
Copy link

@vincbeck that would be odd - it is nowhere mentioned in the deployment docs.

All i did was fixing the infinite loop redirect in the docker container.

@ashb
Copy link
Member Author

ashb commented Apr 25, 2025

A possible work around for anyone eager to try out 3.0.0 before we get it released in 3.0.1:

FROM apache/airflow:3.0.0

USER root

RUN apt update && apt install -y patch patchutils

RUN set -ex; \
    cd /home/airflow/.local/lib/python3.12/site-packages/airflow; \
    curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49721.patch \
    | filterdiff -p1 -i 'airflow-core/src/airflow/*' | patch -p4 -u --verbose

USER airflow

@brechtvb
Copy link

brechtvb commented Apr 25, 2025

This does not fix it for me - i guess i'm doing something wrong. Still in a loop when logging in with airflow:airflow

new container listed & updated in the docker-compose.yaml (to use the new one)

REPOSITORY       TAG            IMAGE ID       CREATED          SIZE
apache/airflow   3.0.0-fixed    36a65204eca0   15 minutes ago   1.83GB
apache/airflow   3.0.0          07fc276fd638   2 days ago       1.8GB

image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.0-fixed}

@ashb
Copy link
Member Author

ashb commented Apr 25, 2025

@brechtvb What Auth Manager are you using? What are the URLs you get redirected through please?

@brechtvb
Copy link

Very sorry, the black boxes is the hostname (no domain included):

image

is this a networking problem on my side.

@ashb
Copy link
Member Author

ashb commented Apr 25, 2025

If you are using the FAB auth manger, then you will need the patch for #49724 too:

FROM apache/airflow:3.0.0

USER root

RUN apt update && apt install -y patch patchutils

RUN set -ex; \
    cd /home/airflow/.local/lib/python3.12/site-packages/airflow; \
    curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49721.patch \
    | filterdiff -p1 -i 'airflow-core/src/airflow/*' | patch -p4 -u --verbose; \
    curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49724.patch \
    | patch -p5 -u --verbose; \
    curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49581.patch \
    | filterdiff -p1 -i 'providers/fab/src/airflow/*' | patch -p5 -u --verbose

USER airflow

@brechtvb
Copy link

@ashb it'w working! thank you very much.

i believe there is small typo, the double /

providers/fab//src/airflow

prabhusneha pushed a commit to astronomer/airflow that referenced this pull request Apr 25, 2025
…cure (apache#49721)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

I have also updated the Simple security manager to look at X-Forwarded-Proto
so that if there is a reverse proxy in front (and if Airflow is configured to
trust that) then it sets the secure flag
prabhusneha pushed a commit to astronomer/airflow that referenced this pull request Apr 25, 2025
… token cookie as secure (apache#49724)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

Provider version of apache#49721
prabhusneha pushed a commit to astronomer/airflow that referenced this pull request Apr 25, 2025
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
… token cookie as secure (apache#49724)

`api/ssl_cert` has a default value of "" so `has_option` is always returning
true, so we have to find in a slightly more complex way of telling if this
setting is turned on.

Provider version of apache#49721
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
@ashb ashb mentioned this pull request May 7, 2025
2 tasks
@diegoscarabelli
Copy link

diegoscarabelli commented May 8, 2025

@ashb With your solution for FAB auth manager, I am finding this error applying the second patch:

0.128 + cd /home/airflow/.local/lib/python3.12/site-packages/airflow
0.128 + curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49721.patch
0.128 + filterdiff -p1 -i 'airflow-core/src/airflow/*'
0.128 + patch -p4 -u --verbose
0.132   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0.132                                  Dload  Upload   Total   Spent    Left  Speed
100  2879  100  2879    0     0   8448      0 --:--:-- --:--:-- --:--:--  8467
0.474 Hmm...  Looks like a unified diff to me...
0.474 The text leading up to this was:
0.474 --------------------------
0.474 |diff --git a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/routes/login.py b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/routes/login.py
0.474 |index c901692d8f9c8..82875ceb1239c 100644
0.474 |--- a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/routes/login.py
0.474 |+++ b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/routes/login.py
0.474 --------------------------
0.490 patching file api_fastapi/auth/managers/simple/routes/login.py
0.490 Using Plan A...
0.490 Hunk #1 succeeded at 17.
0.490 Hunk #2 succeeded at 57.
0.490 done
0.490 + curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49724.patch
0.490 + patch -p5 -u --verbose
0.499   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0.499                                  Dload  Upload   Total   Spent    Left  Speed
100  9270  100  9270    0     0  29393      0 --:--:-- --:--:-- --:--:-- 29428
0.817 Hmm...  I can't seem to find a patch in there anywhere.
0.817 patch: **** Only garbage was found in the patch input.
------
failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c set -ex;     cd /home/airflow/.local/lib/python3.12/site-packages/airflow;     curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49721.patch     | filterdiff -p1 -i 'airflow-core/src/airflow/*' | patch -p4 -u --verbose;     curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49724.patch     | patch -p5 -u --verbose;     curl https://patch-diff.githubusercontent.com/raw/apache/airflow/pull/49581.patch     | filterdiff -p1 -i 'providers/fab/src/airflow/*' | patch -p5 -u --verbose" did not complete successfully: exit code: 2

@Dev-iL
Copy link
Collaborator

Dev-iL commented May 8, 2025

@diegoscarabelli can you try 3.0.1-rc1 and see if the problem is fixed?

@sopanawit
Copy link

@diegoscarabelli can you try 3.0.1-rc1 and see if the problem is fixed?

I tried this version in my docker compose file and confirmed it worked!

@diegoscarabelli
Copy link

@diegoscarabelli can you try 3.0.1-rc1 and see if the problem is fixed?

Yes, 3.0.1rc1 works. Hopefully becomes official soon. Thanks!

@kaxil
Copy link
Member

kaxil commented May 8, 2025

@diegoscarabelli can you try 3.0.1-rc1 and see if the problem is fixed?

Yes, 3.0.1rc1 works. Hopefully becomes official soon. Thanks!

Yup, next Monday (12 May) is a likely date

@brechtvb
Copy link

brechtvb commented May 8, 2025

Maybe off topic:

Is the docker compose deployment deemed fit for production environments?

@kaxil
Copy link
Member

kaxil commented May 8, 2025

Maybe off topic:

Is the docker compose deployment deemed fit for production environments?

Nope, check https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#:~:text=This%20procedure%20can,Airflow%20in%20production. to know why :)

@gtsiatsios
Copy link

gtsiatsios commented May 9, 2025

Trying to use 3.0.1-rc1 in Docker-compose but I get
Screenshot 2025-05-09 at 12 38 00

FYR #49721 (comment) has patch files that don't exist

@brechtvb
Copy link

Maybe off topic:
Is the docker compose deployment deemed fit for production environments?

Nope, check https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#:~:text=This%20procedure%20can,Airflow%20in%20production. to know why :)

But this link talks about production docker images

https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-production-docker-images

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Infinite redirection loop when attempting to access a dockerized Airflow webserver using hostname

8 participants