Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forward-auth for traefik does not work only with embedded outpost #3566

Closed
toxic0berliner opened this issue Sep 8, 2022 · 3 comments
Closed
Labels
bug Something isn't working

Comments

@toxic0berliner
Copy link

*Describe the bug
Traefik forward auth is not working properly with the embedded outpost.

To Reproduce
Deploy something like this :

compose.yml (click to expand)
version: "3"
services:
  traefik:
    container_name: traefik
    environment:
      - OVH_ENDPOINT=${OVH_ENDPOINT}
      - OVH_APPLICATION_KEY=${OVH_APPLICATION_KEY}
      - OVH_APPLICATION_SECRET=${OVH_APPLICATION_SECRET}
      - OVH_CONSUMER_KEY=${OVH_CONSUMER_KEY}
      - TZ=${TZ}
    hostname: traefik
    image: traefik:2.6
    labels:
      traefik.http.services.traefik.loadbalancer.server.port: 8080
    ports:
      - 443:443/tcp
      - 80:80/tcp
      - 8080:8080/tcp
    restart: always
    volumes:
      - traefik-config:/etc/traefik
      - traefik-log:/var/log/traefik
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${BASE_VOLUMES}/${STACKNAME}/traefik/acme.json:/acme.json
    networks:
      proxynetwork:
        aliases: 
          - traefik
  authentik-postgresql:
    container_name: authentik-postgresql
    hostname: authentik-postgresql
    image: postgres:12-alpine
    restart: unless-stopped
    labels:
      traefik.enable: false
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    networks:
      proxynetwork:
        aliases: 
          - authentik-postgresql
    volumes:
      - authentik-db:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=${PG_PASS:?database password required}
      - POSTGRES_USER=${PG_USER:-authentik}
      - POSTGRES_DB=${PG_DB:-authentik}
  authentik-redis:
    container_name: authentik-redis
    hostname: authentik-redis
    image: redis:alpine
    restart: unless-stopped
    labels:
      traefik.enable: false
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    networks:
      proxynetwork:
        aliases: 
          - authentik-redis
  authentik:
    container_name: authentik
    hostname: authentik
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2022.8.2}
    restart: unless-stopped
    labels:
      traefik.http.routers.authentik.rule: HostRegexp(`auth.amato.top`)
      traefik.http.routers.authentik.priority: 15
      traefik.http.services.authentik.loadbalancer.server.port: 9000
      traefik.http.routers.authentik.rule: Host(`auth.example.com`)
      traefik.http.middlewares.authentik.forwardauth.address: http://authentik:9000/outpost.goauthentik.io/auth/traefik
      traefik.http.middlewares.authentik.forwardauth.trustForwardHeader: true
      traefik.http.middlewares.authentik.forwardauth.authResponseHeaders: X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid,X-authentik-jwt,X-authentik-meta-jwks,X-authentik-meta-outpost,X-authentik-meta-provider,X-authentik-meta-app,X-authentik-meta-version
    command: server
    environment:
      AUTHENTIK_REDIS__HOST: authentik-redis
      AUTHENTIK_POSTGRESQL__HOST: authentik-postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_ERROR_REPORTING__ENABLED: ${AUTHENTIK_ERROR_REPORTING__ENABLED}
      # WORKERS: 2
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_EMAIL__HOST: ${AUTHENTIK_EMAIL__HOST}
      AUTHENTIK_EMAIL__PORT: ${AUTHENTIK_EMAIL__PORT}
      AUTHENTIK_EMAIL__USERNAME: ${AUTHENTIK_EMAIL__USERNAME}
      AUTHENTIK_EMAIL__PASSWORD: ${AUTHENTIK_EMAIL__PASSWORD}
      AUTHENTIK_EMAIL__USE_TLS: ${AUTHENTIK_EMAIL__USE_TLS}
      AUTHENTIK_EMAIL__USE_SSL: ${AUTHENTIK_EMAIL__USE_SSL}
      AUTHENTIK_EMAIL__TIMEOUT: ${AUTHENTIK_EMAIL__TIMEOUT}
      AUTHENTIK_EMAIL__FROM: ${AUTHENTIK_EMAIL__FROM}
      GEOIPUPDATE_ACCOUNT_ID: ${GEOIPUPDATE_ACCOUNT_ID}
      GEOIPUPDATE_LICENSE_KEY: ${GEOIPUPDATE_LICENSE_KEY}
      AUTHENTIK_AUTHENTIK__GEOIP: ${AUTHENTIK_AUTHENTIK__GEOIP}
      AUTHENTIK_PORT_HTTP: ${AUTHENTIK_PORT_HTTP}
      AUTHENTIK_PORT_HTTPS: ${AUTHENTIK_PORT_HTTPS}
      DOMAIN: ${DOMAIN}
      AUTHENTIK_LOG_LEVEL: trace
    volumes:
      - authentik-media:/media
      - authentik-custom-templates:/templates
      - authentik-geoip:/geoip
    # following env_file is needed due to #3547 (https://github.com/goauthentik/authentik/issues/3547)
    env_file:
      - stack.env
    networks:
      proxynetwork:
        aliases: 
          - authentik
  authentik-worker:
    container_name: authentik-worker
    hostname: authentik-worker
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2022.8.2}
    restart: unless-stopped
    labels:
      traefik.enable: false
    command: worker
    environment:
      AUTHENTIK_REDIS__HOST: authentik-redis
      AUTHENTIK_POSTGRESQL__HOST: authentik-postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    user: root
    volumes:
      - authentik-media:/media
      - authentik-certs:/certs
      - /var/run/docker.sock:/var/run/docker.sock
      - authentik-custom-templates:/templates
      - authentik-geoip:/geoip
    # following env_file is needed due to #3547 (https://github.com/goauthentik/authentik/issues/3547)
    env_file:
      - stack.env
    networks:
      proxynetwork:
        aliases: 
          - authentik-worker
  authentik-geoipupdate:
    container_name: authentik-geoipupdate
    hostname: authentik-geoipupdate
    image: "maxmindinc/geoipupdate:latest"
    labels:
      traefik.enable: false
    volumes:
      - "authentik-geoip:/usr/share/GeoIP"
    environment:
      GEOIPUPDATE_EDITION_IDS: "GeoLite2-City"
      GEOIPUPDATE_ACCOUNT_ID: ${GEOIPUPDATE_ACCOUNT_ID}
      GEOIPUPDATE_LICENSE_KEY: ${GEOIPUPDATE_LICENSE_KEY}
      GEOIPUPDATE_FREQUENCY: "8"
    networks:
      proxynetwork:
        aliases: 
          - authentik-geoipupdate
  hello-world:
    image: nginxdemos/hello
    labels:
      traefik.http.services.hello-world.loadbalancer.server.port: 80
      traefik.http.routers.hello-world.rule: "HostRegexp(`hello.example.com`)"
      traefik.http.routers.hello-world.middlewares: authentik@docker
    networks:
      proxynetwork:
        aliases: 
          - hello
volumes:
  traefik-config:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/traefik/etc-traefik
  traefik-log:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/traefik/log
  authentik-media:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/authentik/media
  authentik-custom-templates:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/authentik/custom-templates
  authentik-db:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/authentik/db
  authentik-geoip:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/authentik/geoip
  authentik-certs:
    driver: local-persist
    driver_opts:
      mountpoint: ${BASE_VOLUMES}/${STACKNAME}/authentik/certs
networks:
  proxynetwork:
    name: proxynetwork
    driver: bridge
    external: true

and have only traefik listen to 80 and 444 ports.
browse to auth.example.com/if/flow/initial-setup and create an admin user.
Add a proxy provider at domain level using

  • auth-url : https://auth.example.com
  • cookie-domain: example.com
  • unauthenticated urls : auth.example.com in http, https and with&without .* as path

Add an app, name it example.com and select the provider you just created.

Expected behavior
browsing hello.example.com should redirect to auth.example.com and trigger auth, sucessfull auth should redirect to hello.example.com
But in fact it redirects to this :
https://auth.example.comhhttps//auth.example.comthttps://auth.example.comthttps://auth.example.comphttps://auth.example.com:https://auth.example.com/https://auth.example.com/https://auth.example.comlhttps://auth.example.comohttps://auth.example.comchttps://auth.example.comahttps://auth.example.comlhttps://auth.example.comhhttps://auth.example.comohttps://auth.example.comshttps://auth.example.comthttps://auth.example.com:https://auth.example.com8https://auth.example.com0https://auth.example.com0https://auth.example.com0https://auth.example.com/https://auth.example.comahttps://auth.example.comphttps://auth.example.comphttps://auth.example.comlhttps://auth.example.comihttps://auth.example.comchttps://auth.example.comahttps://auth.example.comthttps://auth.example.comihttps://auth.example.comohttps://auth.example.comnhttps://auth.example.com/https://auth.example.comohttps://auth.example.com/https://auth.example.comahttps://auth.example.comuhttps://auth.example.comthttps://auth.example.comhhttps://auth.example.comohttps://auth.example.comrhttps://auth.example.comihttps://auth.example.comzhttps://auth.example.comehttps://auth.example.com/https://auth.example.com?client_id=somethinglookingprivate&redirect_uri=https%3A%2F%2Fauth.example.com%2Foutpost.goauthentik.io%2Fcallback%3FX-authentik-auth-callback%3Dtrue&response_type=code&scope=ak_proxy+email+profile+openid&state=somethinglookinglessprivate

(no it's not a copy-paste problem...)

Logs
When I call hello.example.com, traefik debug logs show this :

time="2022-09-09T00:28:26+02:00" level=debug msg="Remote error http://authentik:9000/outpost.goauthentik.io/auth/traefik. StatusCode: 302" middlewareName=authentik@docker middlewareType=ForwardedAuthType

So traefik did in fact contact the outpost to authorize the request.
Then docker logs authentik show this :

(click to expand)
{
    "app": "example",
    "event": "Found app based on cookie domain",
    "host": "hello.example.com",
    "level": "debug",
    "logger": "authentik.outpost.proxyv2",
    "timestamp": "2022-09-09T00:22:20+02:00"
}{
    "event": "passing to application mux",
    "host": "hello.example.com",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2",
    "timestamp": "2022-09-09T00:22:20+02:00"
}{
    "event": "tracing headers for debug",
    "header": {
        "Accept": ["*/*"],
        "User-Agent": ["curl/7.81.0"],
        "X-Forwarded-For": ["10.10.10.10"],
        "X-Forwarded-Host": ["hello.example.com"],
        "X-Forwarded-Method": ["GET"],
        "X-Forwarded-Port": ["443"],
        "X-Forwarded-Proto": ["https"],
        "X-Forwarded-Server": ["traefik"],
        "X-Forwarded-Uri": ["/"],
        "X-Real-Ip": ["10.10.10.10"]
    },
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "name": "example",
    "timestamp": "2022-09-09T00:22:20+02:00"
} {
    "event": "traefik forwarded url",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "name": "example",
    "timestamp": "2022-09-09T00:22:20+02:00",
    "url": "https://hello.example.com/"
} {
    "event": "Matching URL against allow list",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "match": false,
    "name": "example",
    "regex": "https://auth.example.com/.*",
    "timestamp": "2022-09-09T00:22:20+
    02:00",
    "url": "https://hello.example.com/"
}{
    "event": "Matching URL against allow list",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "match": false,
    "name": "example",
    "regex": "http://auth.example.com/.*",
    "timestamp": "2022-09-09T00:22:20+0
    2:00",
    "url": "https://hello.example.com/"
} {
    "event": "Matching URL against allow list",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "match": false,
    "name": "example",
    "regex": "https://auth.example.com/",
    "timestamp": "2022-09-09T00:22:20+02:00",
    "url": "https://hello.example.com/"
}{
    "event": "Matching URL against allow list",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "match": false,
    "name": "example",
    "regex": "http://auth.example.com/",
    "timestamp": "2022-09-09T00:22:20+02:
    00",
    "url": "https://hello.example.com/"
} {
    "event": "Matching URL against allow list",
    "level": "trace",
    "logger": "authentik.outpost.proxyv2.application",
    "match": false,
    "name": "example",
    "regex": "http://authentik:9000/.*",
    "timestamp": "2022-09-09T00:22:20+02:00",
    "url": "https://hello.example.com/"
}{
    "event": "/outpost.goauthentik.io/auth/traefik",
    "host": "hello.example.com",
    "level": "info",
    "logger": "authentik.outpost.proxyv2.application",
    "method": "GET",
    "name": "example",
    "remote": "10.10.10.10",
    "runtime": "4.120",
    "scheme": "https",
    "size": 1388,
    "status": 302,
    "timestamp": "2022-09-09T00:22:20+02:00",
    "upstream": "",
    "user_agent": "curl/7.81.0"
}

And the user gets redirected to this wrong url... browsers don't like malformed urls...

Version and Deployment (please complete the following information):

  • authentik version: 2022.8.2
  • Deployment: portainer (docker-compose)

Additional context
I have looked at issue #2180 which put me on the track and pushed me to create a separate outpost.
But given there is a single container for authentik and it's outpost, I can't see how to apply any kind of other priorities. Maybe the outpost is running on port 8000 and could be accessed by traefik differently but that would contradict the docs and other examples found online (https://gist.github.com/LukasForst/9898e08b6290821d6ccfcd0b6ad4f376)

Anyway, when I create a new outpost named traefik, using the docker local integration, and giving it the following config :

log_level: trace
docker_labels:
  traefik.http.routers.authentik-outpost-traefik.priority: "50"
authentik_host: http://authentik:9000
docker_network: proxynetwork
container_image: null
docker_map_ports: false
kubernetes_replicas: 1
kubernetes_namespace: default
authentik_host_browser: https://auth.example.com
object_naming_template: authentik-outpost-%(name)s
authentik_host_insecure: true
kubernetes_service_type: ClusterIP
kubernetes_image_pull_secrets: []
kubernetes_disabled_components: []
kubernetes_ingress_annotations: {}
kubernetes_ingress_secret_name: authentik-outpost-tls

Then update the embedded outpost to unselect the example provider from it and enabling it in the newly created traefik outpost.

Then modify my whole compose file to change the definition of the middleware like this :

      # Remove this line :
      # traefik.http.middlewares.authentik.forwardauth.address: http://authentik:9000/outpost.goauthentik.io/auth/traefik
      # and replace by this line :
      traefik.http.middlewares.authentik.forwardauth.address: http://authentik-outpost-traefik:9000/outpost.goauthentik.io/auth/traefik

Suddenly browsing to hello.example.com works just fine, I get a login page from authentik, and end up with the nging hello page displayed and a cookie stored for example.com

I even tried copy-pasting the config of this outpost (that works) to the config for the embedded outpost, knowing that not all values would apply, but anyway, no luck...

Trace logs are not helpful here sadly, I feel the embedded outpost is itself being redirected to auth itself but I don't get it as it shows as connected and "viewed" a few second ago...

Any help here is welcome ! I would really like to remove access from the docker socket and avoid having an extra container running just to run the outpost...

Thanks in advance

@toxic0berliner toxic0berliner added the bug Something isn't working label Sep 8, 2022
@ToshY
Copy link

ToshY commented Sep 9, 2022

As someone who struggled setting up the proxy provider with Traefik as well, maybe I can try to help by showing on how I set it up. I'll show you both my Authentik settings, docker compose (partial) and Traefik middleware.

Authentik

Application
image

Provider

image

Outpost (default)

image

docker-compose.yml

services:
  authentik-server:
    image: ghcr.io/goauthentik/server:${AUTHENTIK_SERVER_IMAGE_VERSION}
    <<: *restart-stopped
    command: server
    <<: *authentik-environment
    volumes:
      - media:/media
      - geoip:/geoip
    labels:
      traefik.enable: true
      traefik.docker.network: proxy
      traefik.http.routers.authentik.tls: true
      traefik.http.routers.authentik.priority: 50
      traefik.http.routers.authentik.entrypoints: websecure
      traefik.http.routers.authentik.service: authentik
      traefik.http.routers.authentik.rule: Host(`authentik.${APP_DOMAIN}`, `www.authentik.${APP_DOMAIN}`) || HostRegexp(`{subdomain:[a-z0-9-]+}.${APP_DOMAIN}`) && PathPrefix(`/outpost.goauthentik.io/`)
      traefik.http.services.authentik.loadbalancer.server.port: 9000
    networks:
      - authentik
      - proxy

Traefik

/dynamic/http.yml

http:
  middlewares:
    authentik:
      forwardAuth:
        address: '{{ env "AUTHENTIK_AUTH_ENDPOINT" }}'
        trustForwardHeader: true
        authResponseHeaders:
          - X-Authentik-Username
          - X-Authentik-Groups
          - X-Authentik-Email
          - X-Authentik-Name
          - X-Authentik-Uid
          - X-Authentik-Jwt
          - X-Authentik-Meta-Jwks
          - X-Authentik-Meta-Outpost
          - X-Authentik-Meta-Provider
          - X-Authentik-Meta-App
          - X-Authentik-Meta-Version

Here, AUTHENTIK_AUTH_ENDPOINT is http://authentik-server:9000/outpost.goauthentik.io/auth/traefik (replace authentik-server with your authentik service name, authentik).

Now using it in a whoami container

  whoami:
    image: traefik/whoami:latest
    restart: unless-stopped
    labels:
      traefik.enable: true
      traefik.docker.network: proxy
      traefik.http.routers.whoami.tls: true
      traefik.http.routers.whoami.entrypoints: websecure
      traefik.http.routers.whoami.rule: Host(`whoami.${APP_DOMAIN}`, `www.whoami.${APP_DOMAIN}`)
      traefik.http.routers.whoami.middlewares: authentik@file #remove @file if you dont setup with file
      traefik.http.routers.whoami.service: whoami
      traefik.http.services.whoami.loadbalancer.server.port: 80
    networks:
      - proxy

This will redirect you to Authentik, and after logging in, you'll get redirected to the whoami page.


Maybe this will (partially) help you debug your problem.

@toxic0berliner
Copy link
Author

Thanks a lot for your answer, I am still struggling to see what I have done differently but for sure I will try again as it seems you have successfully

  • run authentik with only the embedded outpost
  • used it as auth middleware in traefik

Thanks a lot for the screenshot of your provider and app, I'll need a while to try those settings again from a fresh install.

@toxic0berliner
Copy link
Author

I got this more or less working today, had some very strange behaviors, at one point I had hello.example.com under auth by traefik but it didn't redirect to hello.example.com once auth was sucessfull, it always got back to auth.example.com, even if the cookie was set properly when browsing to hello.example.com....

Not sure what I did that fixed it, restarted traefik mainly, but hey, seems it works now...
Going to try to see for some custom policies now, like auto-login based on IP adresses.

The fun part that got me a bit frightened is that I apply the middleware for the entire traefik entrypoint, so authentik itself is behind it's own auth middleware, but thankfully that works if I add https://auth.example.com/.* as well as wss://auth.example.com/.* under the "unauthenticated urls"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants