Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Failed to proxy to backend" #5059

Closed
Jdavid77 opened this issue Mar 23, 2023 · 8 comments
Closed

"Failed to proxy to backend" #5059

Jdavid77 opened this issue Mar 23, 2023 · 8 comments
Labels
bug Something isn't working wontfix

Comments

@Jdavid77
Copy link

Jdavid77 commented Mar 23, 2023 •
edited
Loading

Describe the bug
Since upgrading to the latest version , i've been getting this error:

{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-23T17:38:43Z"}

And it makes authentik restart continuously and the pod gets shutdown everytime.

To Reproduce
Steps to reproduce the behavior:

  1. Uprade authentik to the latest version.

My Current helm configuration is as follows:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: authentik
  namespace: idp
spec:
  interval: 5m
  chart:
    spec:
      # renovate: registryUrl=https://charts.goauthentik.io
      chart: authentik
      version: 2023.3.1
      sourceRef:
        kind: HelmRepository
        name: authentik-charts
        namespace: flux-system
      interval: 5m
  values:
    image:
      repository: ghcr.io/goauthentik/server
      tag: 2023.3.1
      pullPolicy: IfNotPresent
    serviceAccount:
      create: false
    ingress:
      enabled: true
      annotations:
      hosts:
        - host: authentik.${SECRET_DOMAIN}
          paths:
            - path: /
      tls:
        - hosts:
            - authentik.${SECRET_DOMAIN}
    worker:
      replicas: 2
    # Password in clear text because this is a random test instance
    authentik:
      secret_key: ${AUTHENTIK_SECRET}
      log_level: debug
      error_reporting:
        enabled: false
      avatars: none
      postgresql:
        # Should be present on the secret created by the postgres operator
        host: ${POSTGRES_HOST}
        name: ${POSTGRES_DB}
        user: ${POSTGRES_USER}
      email:
        # -- SMTP Server emails are sent from, fully optional
        host: "${SMTP_MAIL_SERVER}"
        port: 587
        # -- SMTP credentials, when left empty, not authentication will be done
        username: "${SMTP_MAIL_USERNAME}"
        # -- SMTP credentials, when left empty, not authentication will be done
        password: "${SMTP_MAIL_PASSWORD}"
        # -- Enable either use_tls or use_ssl, they can't be enabled at the same time.
        use_tls: true
        # -- Enable either use_tls or use_ssl, they can't be enabled at the same time.
        use_ssl: false
        # -- Connection timeout
        timeout: 30
        # -- Email from address, can either be in the format "foo@bar.baz" or "authentik <foo@bar.baz>"
        from: ${SMTP_FROM}

    postgresql:
      enabled: false

    redis:
      enabled: true
      persistence:
        enabled: false
      auth:
        enabled: false
    prometheus:
      serviceMonitor:
        create: false
      rules:
        create: false
    livenessProbe:
      # -- enables or disables the livenessProbe
      enabled: true
      httpGet:
        # -- liveness probe url path
        path: /-/health/live/
        port: http
      initialDelaySeconds: 60
      periodSeconds: 15
    readinessProbe:
      enabled: true
      httpGet:
        path: /-/health/ready/
        port: http
      initialDelaySeconds: 60
      periodSeconds: 15
  valuesFrom:
    - kind: Secret
      name: postgres-outputs
      targetPath: authentik.postgresql.password
      valuesKey: authentik_password

P.S - Usings SOPS for variable substitution

Expected behavior
A clear and concise description of what you expected to happen.

Version and Deployment (please complete the following information):

  • authentik version: 2023.3.1
  • Deployment: helm chart version 2023.3.1

Additional context
Zero clue how to procede with this error it never happened before, all our previous updates have been smooth
@Jdavid77 Jdavid77 added the bug Something isn't working label Mar 23, 2023
@BeryJu
Copy link
Member

BeryJu commented Mar 23, 2023

Can you post the full pod logs somewhere?

Also you're configuring 2 workers in your deployment, which I don't think does what you intend it to, you probably want to set replicas to 2 and not worker.replicas

@Jdavid77
Copy link
Author 

{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "257e3bde4590471186bf6b9bb055e61a", "runtime": 159, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.782543", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "fdb05baef60446678f57f09c3e98bdaf", "runtime": 164, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.808899", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "17b104fc7bcb4f659bc56b54c299a73d", "runtime": 158, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.862961", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "5cdb3fe4f60e4f4a9b3e0ac272a296d0", "runtime": 168, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.932561", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "568d65e3f40c4113bfeea43f0eae717e", "runtime": 153, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.937691", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "06d6c714b8de45119e89daf82857411c", "runtime": 152, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.938652", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "e88f71842e4c4a71b4206379daefbadc", "runtime": 146, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.939265", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "f0c03eb6a3374d81baade2043dc02422", "runtime": 153, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.955555", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "7b29af536da94a0d9728e0ecc604af99", "runtime": 150, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.957363", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "0ab038654d4f4b04afdd3c52fb1717b6", "runtime": 151, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.958189", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "a223adc8992f464abad0dd15b1322b71", "runtime": 158, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.961170", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "e9c574c7fb6841f5987cd907ed733159", "runtime": 158, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:47.976356", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "61eae13ad29145eaa09fc0e145ecd073", "runtime": 178, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.049590", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/stages/all/?name=default-source-enrollment-prompt", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "596375fe63784ca9bb421d57a223e00d", "runtime": 154, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.109582", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "69a75499397343768d70d4b2864acb96", "runtime": 155, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.110137", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/stages/all/?name=default-source-enrollment-login", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "46468d4afdf9435081e9e5297a8ed0be", "runtime": 139, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.113573", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/stages/all/?name=default-source-enrollment-write", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "beed3246f13c415183cfc4917ee237f3", "runtime": 144, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.116004", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/policies/expression/f00c7179-e513-4a1d-b68b-8cdaf4bc0299/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "16bd53175d5048b49f41d65122f00351", "runtime": 142, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.117981", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "b57e3824ffc54ff09afbfb90e6855c39", "runtime": 174, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.118952", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/neoception-enrollment-flow/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "5c3832415b1a42799b0dffe26b38edd4", "runtime": 143, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.121298", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "721b4ad5c5c6443699ac871c66d843e7", "runtime": 170, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.123115", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/policies/expression/416e4f45-6dce-49b7-a695-4541533a7835/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "df270b3974324bc9bdf73f3b6ac8524b", "runtime": 151, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.137234", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-source-authentication", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "d851584a2e954c7abd7856d2db643342", "runtime": 133, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.191964", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/policies/expression/b1e2cb8b-328a-45b1-9cfd-8a7b59d49bae/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "3e1c4d358a3e43d8a8b408f60a72d161", "runtime": 83, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.202244", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/stages/captcha/91cbc6b9-93e8-4433-9041-6a0eefba1315/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "f077642bf26a4842b2ac8cd31aae73ff", "runtime": 140, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.294942", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/core/groups/eef48c98-6afe-41dd-b60b-1f33048a3963/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "5349faf1b6854738bc810a6009658ade", "runtime": 153, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.327545", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "fc6e0b4a229b41bfbac42a6eb0e643dc", "runtime": 150, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.332194", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/policies/expression/e62c9da3-f651-4078-854a-5d2eae5fba0b/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "2163ebf3b1ca4dd29569464b2ad603c9", "runtime": 153, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.333717", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/core/groups/8b52c250-b235-4ddf-84f5-c06466bede10/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "188ddc9161f8422abd8622b336daa757", "runtime": 156, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.335712", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/core/groups/0dfb4631-b4e2-4667-b29d-9b823d1e08e4/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "9bab9ddc23dc497eb6bf3921efaa04f6", "runtime": 157, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.337982", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/core/groups/e8778a15-df22-457c-8156-67cfd4231c1c/", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "3855bc72902f4967bb5a99c9edffa702", "runtime": 158, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.343285", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "3e1e0e4b7f5644e9b186b56cea2b967a", "runtime": 175, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.352923", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "7ef97411bb0140938a21df425f755da9", "runtime": 163, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.363716", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "05026e2439d8495aa967d29ac4001752", "runtime": 170, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.383726", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "f21c305496074a218fdd4f00f3d15f6a", "runtime": 193, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.497440", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "4ef7397fe1ee4f0a98f392031a9a0c5b", "runtime": 160, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.510491", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "6d659df05fec4947a0c88cbe6dd8e36f", "runtime": 170, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.511792", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "df15f7ac0dcd4bbfa8f1d4644d41f0c7", "runtime": 166, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.513545", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "52ded84f17d14e27b47420a53dbfd06d", "runtime": 155, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.522620", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "146cf0d556aa4e84aac8e987e87a5aec", "runtime": 156, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.528253", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "6a41e04633a74438a5dda711473a71de", "runtime": 161, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.532248", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "ae46d176a15f4ae5928d553db052330f", "runtime": 167, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.533452", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "3f39b6ca6fb44afca4ccd0db1ee8edfa", "runtime": 174, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.547781", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "5c9cfb36aaeb4dd6af1b95f888bdbbde", "runtime": 184, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.571627", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "741a96e5b24447ce907c023c66a05591", "runtime": 242, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.746184", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "cfdb34a8754344a6b6e5154b22fa5e97", "runtime": 267, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.812574", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "6db48b06cb1e41bc9726d7be094d2663", "runtime": 271, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.815180", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "3b67fb0da51345cf90bc9ac6dede0122", "runtime": 271, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.815463", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "5805c461c7ef46bb8eecc8b65abf39e8", "runtime": 279, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.837324", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "39318345ba684d8bbb2a844d4775353b", "runtime": 264, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.840729", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "175c7a442b7640a7b3f7a2009c81910d", "runtime": 287, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.849978", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "64142be992fb458b99f7d0d0a84e9b99", "runtime": 266, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.851311", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "7cc508b021934615bfdf4b508806b46e", "runtime": 289, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.855719", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "36445e862dc943758772adc180d2b39a", "runtime": 278, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.857476", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "b86687ad4170496694812985f0098ac1", "runtime": 211, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.963093", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "3862fe1186154f28baa72ce9633b569b", "runtime": 175, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:48.996053", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "11cf26164dcf4cfcb71f8b8f06f94eab", "runtime": 176, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.004764", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "1b1f7e85361a4298b883ec83c61ce28c", "runtime": 174, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.007710", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "137dc3b03c904251b61ed51f0fccfec0", "runtime": 162, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.013109", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "841d7d88d94d4871816018d789473bd4", "runtime": 164, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.015337", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "2105069ed67a43c2aeac8f50e80ec009", "runtime": 148, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.016616", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "961832c7913646f5976f9a4b44fb11d9", "runtime": 156, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.023159", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "7d77bd9575564cdb8f7cfce80c38cc71", "runtime": 160, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.031686", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "728d357ee3454addb5e91f45c3e735df", "runtime": 174, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.047244", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "adf6c244b65748a6acc5758c20eaa9e9", "runtime": 197, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.170414", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "3e25b07fad1f47a3851e0be37a681b2d", "runtime": 197, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.198142", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "822dfd993e8841a7b575b8cd0f8f0eb9", "runtime": 183, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.201204", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "e205c24e5435432da9fbdbe3465830a3", "runtime": 180, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.203394", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "2cfac0314cf348f1a23c358690b0fc43", "runtime": 187, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.209770", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "59873b4a0b8b4de4a8624fbe42e20706", "runtime": 172, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.228096", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "cf2c863e0f244f31a5e25194f7cef3e5", "runtime": 175, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.231495", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "3fab53dcc2af4fbfbf1d8cf070e73efd", "runtime": 176, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.233323", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "a1bee34db681444fa1804f6f9363c376", "runtime": 178, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.237409", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "20c3e12076774f1c9b531b0a59103fe9", "runtime": 180, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.249900", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "877be65b0ea24e2b98feddd250e5dea6", "runtime": 160, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.341895", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "9ba3ebdbd68a4182b484f3a9bdbd59f6", "runtime": 141, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.356724", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "ca8d5731601e4536951e01b4622d9b44", "runtime": 135, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.359135", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "69e9d2ca1d3144f9b6a9a1d1adfcac2f", "runtime": 139, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.360410", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "2d4e8e5af23145188fe93affe5dc98c1", "runtime": 135, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.363617", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/crypto/certificatekeypairs/?include_details=true&name=authentik+Self-signed+Certificate", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.17", "request_id": "f2fec04b69514b4896227ba7e5934a64", "runtime": 130, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.363837", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/propertymappings/scope/?managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-email&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-openid&managed=goauthentik.io%2Fproviders%2Foauth2%2Fscope-profile", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "e68cad21a2d64f77aab6748f81cbb23c", "runtime": 125, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.366767", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/api/v3/flows/instances/?slug=default-provider-authorization-implicit-consent", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.17", "request_id": "568a8db4dfcd4e36aad4d4ae007df330", "runtime": 128, "scheme": "https", "status": 403, "timestamp": "2023-03-23T18:20:49.368230", "user": "", "user_agent": "authentik-terraform@2022.10.0"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.12", "request_id": "4e117f062af346dc8248e1107df3be81", "runtime": 40, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:20:57.741273", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "127.0.0.1", "request_id": "d245635b38c24e778cbb5e886a655f72", "runtime": 32, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:21:10.791951", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "8b413d934e9640bd991fa8a366900742", "runtime": 21, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:21:12.722157", "user": "", "user_agent": "kube-probe/1.25"}
{"event":"updating tenant certificates","level":"info","logger":"authentik.router.tenant_tls","timestamp":"2023-03-23T18:21:17Z"}
{"auth_via": "unauthenticated", "event": "Task published", "host": "localhost:8000", "level": "info", "logger": "authentik.root.celery", "pid": 3633, "request_id": "797e67348a6d408392058f5bba5e0f4e", "task_id": "87d1a570-3f8c-4f03-81b3-deb1714e2c70", "task_name": "authentik.providers.scim.tasks.scim_signal_direct", "timestamp": "2023-03-23T18:21:17.674147"}
{"auth_via": "secret_key", "event": "/api/v3/core/tenants/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "127.0.0.1", "request_id": "797e67348a6d408392058f5bba5e0f4e", "runtime": 65, "scheme": "http", "status": 200, "timestamp": "2023-03-23T18:21:17.682244", "user": "ak-outpost-1a46ceed775540afa3336f2d2bdd2618", "user_agent": "goauthentik.io/outpost/2023.3.1"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "1a270db5b4fc4f0bbda2b31c99dcd1ed", "runtime": 455, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:21:28.155968", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "dropping exception", "exc": "Http404('No Application matches the given query.')", "host": "authentik.internal.neoception.dev", "level": "debug", "logger": "authentik.lib.sentry", "pid": 4973, "request_id": "3524b321bc274bfdaa64559b142bdd4e", "timestamp": "2023-03-23T18:21:37.127908"}
{"auth_via": "unauthenticated", "event": "/application/o/weavegitops-ils/.well-known/openid-configuration", "host": "authentik.internal.neoception.dev", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "20.101.182.118", "request_id": "3524b321bc274bfdaa64559b142bdd4e", "runtime": 261, "scheme": "https", "status": 404, "timestamp": "2023-03-23T18:21:37.130089", "user": "", "user_agent": "Go-http-client/2.0"}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "127.0.0.1", "request_id": "00c4e09b9f7a4e4cb24bc1808bf80908", "runtime": 1630, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:21:42.390010", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-23T18:21:43Z"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "21e679bfe27b481ca02c7ddce39be094", "runtime": 1182, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:21:43.882890", "user": "", "user_agent": "kube-probe/1.25"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-23T18:21:58Z"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "2e872f4b86d2442eaffe6d608bb365c2", "runtime": 4896, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:02.597455", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "127.0.0.1", "request_id": "5d30a3dc41fd42599ca2709a0552aa20", "runtime": 20, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:10.779707", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "df88dd0247b243fd9a68978014b0552d", "runtime": 32, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:12.733790", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 3633, "remote": "10.192.0.12", "request_id": "679d4c508c754822b4f05ef03bf0bef8", "runtime": 33, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:27.734541", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "127.0.0.1", "request_id": "72854e8a2879446cad495e0810a9683e", "runtime": 44, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:40.803616", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "c40deae5a29548bf9cf4fa1ca999a2cc", "runtime": 33, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:42.733838", "user": "", "user_agent": "kube-probe/1.25"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.192.0.97:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 4973, "remote": "10.192.0.12", "request_id": "06c391f1ff384c8b99038c82e005ceca", "runtime": 34, "scheme": "http", "status": 204, "timestamp": "2023-03-23T18:22:57.735087", "user": "", "user_agent": "kube-probe/1.25"}

@Jdavid77
Copy link
Author

@BeryJu any idea on what it could be causing this issue?

@BeryJu
Copy link
Member

BeryJu commented Mar 24, 2023

So it seems the core service is running correctly for some time and only then do these timeouts happen.

The reason this error happens is because the server container runs two processes actually, the core authentik server (which is python) and a small go proxy, that serves the static files, provides TLS, and also runs the embedded outpost. The main ports 9000 and 9443 point to that go proxy, which then proxies API requests to the actual backend. This error is caused by that proxying from the go proxy to the backend timing out.

The main reason this can happen is high load on the system or in the container, or it's also possible that there's a bug in the core service that causes it's threads to hang (the logs don't point to anything like that, but I won't fully exclude it from being the cause)

Do you use CPU throttling in kubernetes? (It doesn't look like it from the config above but there might be other things modifying the resource limits)

Also as I said above, set the worker replicas to 1 and the server replica to 2, which should better distribute the load.

@rafaribe
Copy link

No resource limits set for Authentik, we increased the replicas. @Jdavid77 will report back after a while to see if the problem keeps happening or not. We also suspected some problem with the PostgreSQL connection but it seems a bit unlikely.

@MrOwen
Copy link

MrOwen commented Mar 25, 2023

I'm also receiving this error on a fresh install of authentik using Helm (2023.3.1). I use an external database provisioned by Stackgres and it is ssl enabled. Initial install went perfectly (all migrations ran and such) and then something weird is happening after. Here's the logs from when I start seeing these errors:

{"event":"updating tenant certificates","level":"info","logger":"authentik.router.tenant_tls","timestamp":"2023-03-25T08:48:04Z"}
{"auth_via": "unauthenticated", "event": "Task published", "host": "localhost:8000", "level": "info", "logger": "authentik.root.celery", "pid": 21, "request_id": "4419d55472cf4d34b1e0e3261a0271f2", "task_id": "9e4d3353-00b5-434a-b24e-5cfa86757c4f", "task_name": "authentik.providers.scim.tasks.scim_signal_direct", "timestamp": "2023-03-25T08:48:04.669357"}
{"auth_via": "secret_key", "event": "/api/v3/core/tenants/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 21, "remote": "127.0.0.1", "request_id": "4419d55472cf4d34b1e0e3261a0271f2", "runtime": 49, "scheme": "http", "status": 200, "timestamp": "2023-03-25T08:48:04.679533", "user": "ak-outpost-ea7da355a60b44edb27a312590eadca7", "user_agent": "goauthentik.io/outpost/2023.3.1"}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 21, "remote": "127.0.0.1", "request_id": "55f19e5a88c248ca8d8d8062f8b338cf", "runtime": 19, "scheme": "http", "status": 204, "timestamp": "2023-03-25T08:48:33.273904", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.42.2.214:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 20, "remote": "10.42.2.1", "request_id": "7a782d4940c14cf5ab213e09de6e3e92", "runtime": 14, "scheme": "http", "status": 204, "timestamp": "2023-03-25T08:48:45.196429", "user": "", "user_agent": "kube-probe/1.25"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:49:36Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:49:46Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:49:56Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:06Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:16Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:26Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:36Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:44Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:46Z"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:50:56Z"}
{"event":"updating tenant certificates","level":"info","logger":"authentik.router.tenant_tls","timestamp":"2023-03-25T08:51:04Z"}
{"event": "Internal Server Error: /", "exc_info": ["<class 'django.db.utils.OperationalError'>", "OperationalError('connection to server at \"pg.internal.domain.com\" (192.0.0.0), port 5432 failed: Connection timed out\\n\\tIs the server running on that host and accepting TCP/IP connections?\\n')", "<traceback object at 0x7fd9da506d80>"], "level": "error", "logger": "django.request", "timestamp": 1679734264.5825372}
{"event": "Internal Server Error: /-/health/ready/", "exc_info": ["<class 'django.db.utils.OperationalError'>", "OperationalError('connection to server at \"pg.internal.domain.com\" (192.0.0.0), port 5432 failed: Connection timed out\\n\\tIs the server running on that host and accepting TCP/IP connections?\\n')", "<traceback object at 0x7fd9da65a480>"], "level": "error", "logger": "django.request", "timestamp": 1679734264.5825543}
{"auth_via": "unauthenticated", "event": "/", "host": "auth.owsa.io", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 21, "remote": "172.22.40.22", "request_id": "d994fb01690a490db37fa4ff28de6f77", "runtime": 130667, "scheme": "https", "status": 500, "timestamp": "2023-03-25T08:51:04.641385", "user": "", "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0"}
{"auth_via": "unauthenticated", "event": "/-/health/ready/", "host": "10.42.2.214:9000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 20, "remote": "10.42.2.1", "request_id": "0ddd385a002f4a0b878a12ec02ebf47d", "runtime": 129491, "scheme": "http", "status": 500, "timestamp": "2023-03-25T08:51:04.672273", "user": "", "user_agent": "kube-probe/1.25"}
{"error":"context canceled","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2023-03-25T08:51:06Z"}
{"event": "Internal Server Error: /-/health/live/", "exc_info": ["<class 'django.db.utils.OperationalError'>", "OperationalError('connection to server at \"pg.internal.domain.com\" (192.0.0.0), port 5432 failed: Connection timed out\\n\\tIs the server running on that host and accepting TCP/IP connections?\\n')", "<traceback object at 0x7fd9da5f2b00>"], "level": "error", "logger": "django.request", "timestamp": 1679734272.774382}
{"auth_via": "unauthenticated", "event": "/-/health/live/", "host": "localhost:8000", "level": "info", "logger": "authentik.asgi", "method": "GET", "pid": 20, "remote": "127.0.0.1", "request_id": "8a73b665a2024d33929fda19268c1b67", "runtime": 129520, "scheme": "http", "status": 500, "timestamp": "2023-03-25T08:51:12.775766", "user": "", "user_agent": "goauthentik.io/proxy/healthcheck"}

A really odd thing is that Authentik connected to the db server initially over ssl successfully and installation ran fine. In my setup, pg.internal.domain.com resolves to something like 172.99.40.246 internally but you'll notice in the logs that pg.internal.domain.com is actually resolving to 192.0.0.0 which is my root domain public A record (it's not actually that specific IP fyi). I don't know why all of the sudden after installation that Authentik decides to use the publicly resolvable IP of my postgres server instead of the internal IP address I have set up for it (and which is seemingly uses correctly during installation). On any of the nodes, I can run nslookup pg.internal.domain.com and it returns the expected value so just the resolution inside of the container is messed up. I'm going to try 2023.2 and 2023.1 and see if I run into the issue.

@MrOwen
Copy link

MrOwen commented Mar 25, 2023

It's always the dns... always. Just needed to add an additional zone to my coredns config (so that the domain.com zone resolves to my internal dns server) and all is well. Hopefully this helps anyone else in similar position:
CoreDNS config line: forward domain.com 172.22.40.250
https://blog.voltane.eu/k3s-local-dns/

@stale
Copy link

stale bot commented May 25, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label May 25, 2023
@zolero zolero mentioned this issue Jun 6, 2023
Closed
@stale stale bot closed this as completed Jun 9, 2023
@dark-vex dark-vex mentioned this issue Jan 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@MrOwen @BeryJu @rafaribe @Jdavid77