Missing api secret_key capabilities to replace webserver for 3.0+

[albundy83]

Official Helm Chart version

1.17.0 (latest released)

Apache Airflow version

3.0.2

Kubernetes Version

1.32.5

Helm Chart configuration

Hello,
When I deploy helm chart, I have the following warning:

/home/airflow/.local/lib/python3.12/site-packages/airflow/configuration.py:858 DeprecationWarning: The secret_key option in [webserver] has been moved to the secret_key option in [api] - the old setting has been used, but please update your config.

But we can't create an api one equivalent to webserver one.

Docker Image customizations

No response

What happened

Just the warning

What you think should happen instead

No response

How to reproduce

Deploy helm chart using the following values:

createUserJob:
  serviceAccount:
    automountServiceAccountToken: false
  useHelmHooks: false
  applyCustomEnv: false
migrateDatabaseJob:
  serviceAccount:
    automountServiceAccountToken: false
  useHelmHooks: false
  applyCustomEnv: false
  jobAnnotations:
    argocd.argoproj.io/hook: Sync
useStandardNaming: true
defaultAirflowRepository: docker.io/apache/airflow
defaultAirflowTag: 3.0.2
airflowVersion: 3.0.2
registry:
  connection:
    user: easi-app
    pass: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    host: airflow.youpi.fr
executor: CeleryExecutor,KubernetesExecutor
fernetKeySecretName: airflow-fernet-key
jwtSecretName: airflow-jwt-secret
config:
  core:
    test_connection: Enabled
logs:
  persistence:
    enabled: false
triggerer:
  serviceAccount:
    automountServiceAccountToken: false
  persistence:
    enabled: false
    size: 10Gi
    storageClassName: gp2-retain-immediate
statsd:
  serviceAccount:
    automountServiceAccountToken: false
redis:
  serviceAccount:
    automountServiceAccountToken: false
  passwordSecretName: airflow-redis-password
  persistence:
    enabled: false
    size: 1Gi
    storageClassName: gp2-retain-immediate
  terminationGracePeriodSeconds: 30
postgresql:
  enabled: false
pgbouncer:
  enabled: false
data:
  brokerUrlSecretName: airflow-broker-url
  metadataSecretName: airflow-db-cnx
workers:
  serviceAccount:
    automountServiceAccountToken: true
  safeToEvict: true
  persistence:
    enabled: true
    size: 10Gi
    storageClassName: gp2-retain-immediate
  resources:
    limits:
      cpu: 1
      ephemeral-storage: 2Gi
      memory: 4Gi
    requests:
      cpu: 100m
      ephemeral-storage: 100Mi
      memory: 128Mi
scheduler:
  serviceAccount:
    automountServiceAccountToken: true
dags:
  gitSync:
    enabled: true
    repo: https://airflow.youpi.fr/gitlab/dags/airflow.git
    branch: main
    rev: HEAD
    credentialsSecret: git-credentials
    subPath: dags
ingress:
  apiServer:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    ingressClassName: nginx
    hosts:
      - name: airflow.doca-easi-multicloud.fr
        tls:
          enabled: true
          secretName: airflow-tls
apiServer:
  serviceAccount:
    automountServiceAccountToken: false
  env:
    - name: CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: airflow-api-keycloak
          key: CLIENT_ID
    - name: CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: airflow-api-keycloak
          key: CLIENT_SECRET
    - name: OIDC_ISSUER
      valueFrom:
        secretKeyRef:
          name: airflow-api-keycloak
          key: OIDC_ISSUER
    - name: AIRFLOW__API__BASE_URL
      valueFrom:
        secretKeyRef:
          name: airflow-api-keycloak
          key: AIRFLOW__API__BASE_URL
  apiServerConfig: |
    from airflow.providers.fab.auth_manager.security_manager.override import FabAirflowSecurityManagerOverride
    from flask_appbuilder.security.manager import AUTH_OAUTH
    from base64 import b64decode
    from cryptography.hazmat.primitives import serialization
    from flask import redirect, session
    from flask_appbuilder import expose
    from flask_appbuilder.security.views import AuthOAuthView
    import jwt
    import logging
    import os
    import requests

    log = logging.getLogger(__name__)
    CSRF_ENABLED = True
    AUTH_TYPE = AUTH_OAUTH
    AUTH_USER_REGISTRATION = True
    AUTH_ROLES_SYNC_AT_LOGIN = True
    AUTH_USER_REGISTRATION_ROLE = 'Public'
    PERMANENT_SESSION_LIFETIME = 43200

    # Make sure you create these roles on Keycloak
    AUTH_ROLES_MAPPING = {
        'airflow_admin': ['Admin'],
        'airflow_op': ['Op'],
        'airflow_public': ['Public'],
        'airflow_user': ['User'],
        'airflow_viewer': ['Viewer'], 
    }
    PROVIDER_NAME = 'keycloak'
    CLIENT_ID = os.getenv('CLIENT_ID')
    CLIENT_SECRET = os.getenv('CLIENT_SECRET')
    AIRFLOW__API__BASE_URL = os.getenv('AIRFLOW__API__BASE_URL')
    OIDC_ISSUER = os.getenv('OIDC_ISSUER')
    OIDC_BASE_URL = f'{OIDC_ISSUER}/protocol/openid-connect'
    OIDC_TOKEN_URL = f'{OIDC_BASE_URL}/token'
    OIDC_AUTH_URL = f'{OIDC_BASE_URL}/auth'
    OIDC_METADATA_URL = f'{OIDC_ISSUER}/.well-known/openid-configuration'
    OAUTH_PROVIDERS = [{
        'name': PROVIDER_NAME,
        'token_key': 'access_token',
        'icon': 'fa-circle-o',
        'remote_app': {
            'api_base_url': OIDC_BASE_URL,
            'access_token_url': OIDC_TOKEN_URL,
            'authorize_url': OIDC_AUTH_URL,
            'server_metadata_url': OIDC_METADATA_URL,
            'request_token_url': None,
            'client_id': CLIENT_ID,
            'client_secret': CLIENT_SECRET,
            'client_kwargs': {
                'scope': 'email profile',
                'code_challenge_method': 'S256',
                'response_type': 'code',
            },
        }
    }]

    # Fetch public key
    req = requests.get(OIDC_ISSUER)
    key_der_base64 = req.json()['public_key']
    key_der = b64decode(key_der_base64.encode())
    public_key = serialization.load_der_public_key(key_der)

    class CustomOAuthView(AuthOAuthView):
        @expose('/logout/', methods=['GET', 'POST'])
        def logout(self):
            session.clear()
            return redirect(f'{OIDC_ISSUER}/protocol/openid-connect/logout?post_logout_redirect_uri={AIRFLOW__API__BASE_URL}&client_id={CLIENT_ID}')

    class CustomSecurityManager(FabAirflowSecurityManagerOverride):
        authoauthview = CustomOAuthView

        def get_oauth_user_info(self, provider, response):
            if provider == 'keycloak':
                token = response['access_token']
                me = jwt.decode(token, public_key, algorithms=['HS256', 'RS256'], audience=CLIENT_ID)

                # Extract roles from resource access
                groups = me.get('resource_access', {}).get(CLIENT_ID, {}).get('roles', [])

                log.info(f'groups: {groups}')

                if not groups:
                    groups = ['Viewer']

                userinfo = {
                    'username': me.get('preferred_username'),
                    'email': me.get('email'),
                    'first_name': me.get('given_name'),
                    'last_name': me.get('family_name'),
                    'role_keys': groups,
                }

                log.info(f'user info: {userinfo}')

                return userinfo
            else:
                return {}

    # Make sure to replace this with your own implementation of AirflowSecurityManager class
    SECURITY_MANAGER_CLASS = CustomSecurityManager

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[machiiine]

There is the option to:

• create a secret: AIRFLOW__API__SECRET_KEY for example in a secret workload "airflow-secret"
• import it with

    extraEnvFrom: |
      - secretRef:
          name: 'airflow-secret'

• disable the generation of the previous one:

    enableBuiltInSecretEnvVars:
      AIRFLOW__WEBSERVER__SECRET_KEY: false

I think it did the trick on my side.