Unable to Integrate AWS SAML 2.0 with Airflow 3.0.1 Using AWS Auth Manager

[ashrafhulakoti]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I’m attempting to integrate AWS SAML 2.0 authentication with Apache Airflow 3.0.1 using the new Auth Manager framework.

🔧 Setup Details:
Airflow Version: 3.0.1

Authentication Provider: AWS IAM Identity Center (formerly AWS SSO)

Authorization Provider: Amazon Verified Permissions

I have followed the official documentation for configuring the AWS Auth Manager, including setting the relevant environment variables.

What you think should happen instead?

On successful authentication, appropriate roles should be assigned via Verified Permissions

Users should be redirected back to the Airflow UI with proper access

How to reproduce

Follow the steps mentioned in the doc:
Authentication Provider: AWS IAM Identity Center (formerly AWS SSO)

Authorization Provider: Amazon Verified Permissions

Operating System

linux(Kubernetes Helm Chart)(apache/airflow:3.0.1)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Helm chart 3.0.1 version

Anything else?

INFO: 10.200.5.116:36966 - "GET /login_callback HTTP/1.1" 200 OK INFO: 10.200.5.116:36966 - "GET /ui/config HTTP/1.1" 401 Unauthorized INFO: 10.200.5.116:36966 - "GET /api/v2/auth/login?next=https%3A%2F%2Fexample.com%2Flogin_callback HTTP/1.1" 307 Temporary Redirect INFO: 10.200.5.116:36966 - "GET /auth/login?next=https://example.com/login_callback HTTP/1.1" 307 Temporary Redirect INFO: 10.200.5.116:36966 - "GET /ui/config HTTP/1.1" 401 Unauthorized INFO: 10.200.5.116:36966 - "GET /api/v2/auth/login?next=https%3A%2F%2Fexample.com%2Flogin_callback HTTP/1.1" 307 Temporary Redirect INFO: 10.200.5.116:36966 - "GET /ui/auth/menus HTTP/1.1" 401 Unauthorized INFO: 10.200.5.116:3092 - "GET /api/v2/plugins HTTP/1.1" 401 Unauthorized

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[ashrafhulakoti]

@vincbeck, I see you've worked on this recently. Could you please help me resolve this issue?

[vincbeck]

Looks like the integration works. Just the user you are using does not have permissions associated to it. Can you please share you Amazon verified permissions policies/configurations

[ashrafhulakoti]

policy:
permit(principal, action, resource);

schema:
{
    "Airflow": {
        "actions": {
            "Asset.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Asset"
                    ]
                }
            },
            "AssetAlias.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "AssetAlias"
                    ]
                }
            },
            "Backfill.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Backfill"
                    ]
                }
            },
            "Backfill.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Backfill"
                    ]
                }
            },
            "Backfill.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Backfill"
                    ]
                }
            },
            "Connection.DELETE": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Connection"
                    ]
                }
            },
            "Connection.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Connection"
                    ]
                }
            },
            "Connection.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Connection"
                    ]
                }
            },
            "Connection.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Connection"
                    ]
                }
            },
            "Custom.DELETE": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Custom"
                    ]
                }
            },
            "Custom.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Custom"
                    ]
                }
            },
            "Custom.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Custom"
                    ]
                }
            },
            "Custom.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Custom"
                    ]
                }
            },
            "Configuration.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Configuration"
                    ]
                }
            },
            "Dag.DELETE": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Dag"
                    ],
                    "context": {
                        "attributes": {
                            "dag_entity": {
                                "type": "String",
                                "required": false
                            }
                        },
                        "type": "Record"
                    }
                }
            },
            "Dag.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Dag"
                    ],
                    "context": {
                        "attributes": {
                            "dag_entity": {
                                "required": false,
                                "type": "String"
                            }
                        },
                        "type": "Record"
                    }
                }
            },
            "Dag.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Dag"
                    ],
                    "context": {
                        "attributes": {
                            "dag_entity": {
                                "required": false,
                                "type": "String"
                            }
                        },
                        "type": "Record"
                    }
                }
            },
            "Dag.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Dag"
                    ],
                    "context": {
                        "attributes": {
                            "dag_entity": {
                                "required": false,
                                "type": "String"
                            }
                        },
                        "type": "Record"
                    }
                }
            },
            "Menu.MENU": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Menu"
                    ]
                }
            },
            "Pool.DELETE": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Pool"
                    ]
                }
            },
            "Pool.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Pool"
                    ]
                }
            },
            "Pool.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Pool"
                    ]
                }
            },
            "Pool.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Pool"
                    ]
                }
            },
            "Variable.DELETE": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Variable"
                    ]
                }
            },
            "Variable.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Variable"
                    ]
                }
            },
            "Variable.POST": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Variable"
                    ]
                }
            },
            "Variable.PUT": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "Variable"
                    ]
                }
            },
            "View.GET": {
                "appliesTo": {
                    "principalTypes": [
                        "User"
                    ],
                    "resourceTypes": [
                        "View"
                    ]
                }
            }
        },
        "entityTypes": {
            "Asset": {},
            "AssetAlias": {},
            "Backfill": {},
            "Configuration": {},
            "Connection": {},
            "Custom": {},
            "Dag": {},
            "Menu": {},
            "Pool": {},
            "Group": {},
            "User": {
                "memberOfTypes": [
                    "Group"
                ]
            },
            "Variable": {},
            "View": {}
        }
    }
}

[vincbeck]

Weird, everything looks good

[vincbeck]

Could you enable debug logging? I would like to see the response from Amazon Verified Permissions

[ashrafhulakoti]

[2025-05-15T20:06:38.847+0000] {_core.py:339} DEBUG - registered value JWTValidator(jwks=None, issuer=None, required_claims=frozenset({'exp', 'aud', 'iat'}), audience=['urn:airflow.apache.org:task'], algorithm=['HS512'], leeway=10) for service type airflow.api_fastapi.auth.tokens.JWTValidator
Stack (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/lib/python3.12/multiprocessing/spawn.py", line 135, in _main
    return self._bootstrap(parent_sentinel)
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
    target(sockets=sockets)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/supervisors/multiprocess.py", line 63, in target
    return self.real_target(sockets)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/server.py", line 66, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/lifespan/on.py", line 86, in main
    await app(scope, self.receive, self.send)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 152, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 19, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 77, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 100, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 48, in __call__
    await self.app(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 723, in app
    await self.lifespan(scope, receive, send)
  File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 692, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 133, in merged_lifespan
    async with original_context(app) as maybe_original_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/app.py", line 62, in lifespan
    await stack.enter_async_context(
  File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context
    result = await _enter(cm)
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 134, in merged_lifespan
    async with nested_context(app) as maybe_nested_state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/fastapi.py", line 66, in __call__
    async with self.registry, cm(app, self.registry) as state:
  File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/app.py", line 94, in lifespan
    registry.register_value(JWTValidator, _jwt_validator(), ping=JWTValidator.status)
  File "/home/airflow/.local/lib/python3.12/site-packages/svcs/_core.py", line 339, in register_value
    log.debug(
INFO:     Application startup complete.
INFO:     10.200.21.221:52762 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.21.221:52778 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.21.221:42870 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.21.221:42868 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.4.86:7708 - "GET /health HTTP/1.1" 404 Not Found
INFO:     10.200.5.116:48874 - "GET /health HTTP/1.1" 404 Not Found
INFO:     10.200.21.221:59796 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.21.221:59798 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     10.200.4.86:23302 - "GET /login_callback HTTP/1.1" 200 OK
INFO:     10.200.4.86:23302 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     10.200.4.86:23302 - "GET /api/v2/auth/login?next=https%3A%2F%2Fexample.com%2Flogin_callback HTTP/1.1" 307 Temporary Redirect
INFO:     10.200.4.86:23302 - "GET /auth/login?next=https://example.com/login_callback HTTP/1.1" 307 Temporary Redirect

[ashrafhulakoti]

There is a debug log wrt JWTValidator, which I think is irrelevant to the issue.
There is no additional debug log for the login failure.

[ashrafhulakoti]

There are some debug logs before the application startup as well.

[2025-05-15T20:06:36.981+0000] {parsers.py:250} DEBUG - Response headers: {'x-amzn-RequestId': '***', 'Content-Type': 'text/xml', 'Content-Length': '2439', 'Date': 'Thu, 15 May 2025 20:06:36 GMT'}
[2025-05-15T20:06:36.981+0000] {parsers.py:251} DEBUG - Response body:
b'<AssumeRoleWithWebIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">\n  <AssumeRoleWithWebIdentityResult>\n    <Audience>sts.amazonaws.com</Audience>\n    <AssumedRoleUser>\n      <AssumedRoleId>***</AssumedRoleId>\n      <Arn>arn:aws:sts::***:assumed-role/***/***</Arn>\n    </AssumedRoleUser>\n    <Provider>arn:aws:iam::***:oidc-provider/***</Provider>\n    <Credentials>\n      <AccessKeyId>***</AccessKeyId>\n      <SecretAccessKey>***</SecretAccessKey>\n      <SessionToken>***</SessionToken>\n      <Expiration>2025-05-15T21:06:36Z</Expiration>\n    </Credentials>\n    <SubjectFromWebIdentityToken>***</SubjectFromWebIdentityToken>\n  </AssumeRoleWithWebIdentityResult>\n  <ResponseMetadata>\n    <RequestId>***</RequestId>\n  </ResponseMetadata>\n</AssumeRoleWithWebIdentityResponse>\n'
[2025-05-15T20:06:36.982+0000] {hooks.py:238} DEBUG - Event needs-retry.sts.AssumeRoleWithWebIdentity: calling handler <botocore.retryhandler.RetryHandler object at 0xffff7dedcf80>
[2025-05-15T20:06:36.983+0000] {retryhandler.py:211} DEBUG - No retry needed.
[2025-05-15T20:06:36.985+0000] {credentials.py:625} DEBUG - Retrieved credentials will expire at: 2025-05-15 21:06:36+00:00
[2025-05-15T20:06:36.986+0000] {auth.py:431} DEBUG - Calculating signature using v4 auth.
[2025-05-15T20:06:36.986+0000] {auth.py:432} DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:verifiedpermissions.ap-southeast-1.amazonaws.com
x-amz-date:20250515T200636Z
x-amz-security-token:***
x-amz-target:VerifiedPermissions.GetSchema

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
***
[2025-05-15T20:06:36.986+0000] {auth.py:434} DEBUG - StringToSign:
AWS4-HMAC-SHA256
20250515T200636Z
20250515/ap-southeast-1/verifiedpermissions/aws4_request
***
[2025-05-15T20:06:36.986+0000] {auth.py:436} DEBUG - Signature:
***
[2025-05-15T20:06:36.987+0000] {hooks.py:238} DEBUG - Event request-created.verifiedpermissions.GetSchema: calling handler <function add_retry_headers at 0xffff89c17240>
[2025-05-15T20:06:36.987+0000] {endpoint.py:263} DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://verifiedpermissions.ap-southeast-1.amazonaws.com/, headers={'X-Amz-Target': b'VerifiedPermissions.GetSchema', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.37.3 ...', 'X-Amz-Date': b'20250515T200636Z', 'X-Amz-Security-Token': b'***'}

[ashrafhulakoti]

I feel the issue is related to #49581

[vincbeck]

Do not you have a log starting with Authorization response: ? If yes, I would be interested in this one :) It should be emitted from facade.py