Error in app 'security' when starting Caddy server with oAuth2 with Google

[sidbena]

Describe the issue

Since today (it seems), I'm getting the below error while trying to spin up my Caddy2 instance. This config has worked flawlessly for at least a few weeks.
{"level":"error","ts":1657485825.2251232,"logger":"security","msg":"failed provisioning app server instance","app":"security","error":"server initialization failed: failed configuring identity provider: failed to fetch metadata for OAuth 2.0 authorization server: Get \"https://accounts.google.com/.well-known/openid-configuration\": dial tcp: lookup accounts.google.com: i/o timeout"}
Attempting to contact accounts.google.com works just fine with curl or wget.

Configuration

Paste full Caddyfile below:

{
        order authenticate before respond
        order authorize before basicauth

        security {
                oauth identity provider google {
                        realm <redacted>
                        driver google
                        client_id <redacted>.apps.googleusercontent.com
                        client_secret <redacted>
                }

                authentication portal secureportal {
                        crypto default token lifetime 3600
                        crypto key <redacted> sign-verify <redacted>
                        enable identity provider google
                        cookie domain <redacted>
                        transform user {
                                suffix match email @<redacted>
                                add role allowed
                        }
                }

                authorization policy securelogin {
                        set auth url https://<redacted>
                        crypto key <redacted> verify <redacted>
                        allow roles allowed
                }
        }

        dynamic_dns {
                provider cloudflare {env.CLOUDFLARE_DYNDNS_API_TOKEN}
                check_interval 1s
                domains {
                       <redacted> home proxy
                }
                ip_source upnp
                ip_source simple_http https://api64.ipify.org
                ip_source simple_http https://api.ipify.org
                ip_source simple_http https://icanhazip.com
        }
}

(dns) {
        tls <redacted> {
                dns cloudflare {env.CLOUDFLARE_DYNDNS_API_TOKEN}
        }
}

(encoding) {
        encode zstd gzip
}

(logging) {
        log {
                output file /config/logs/access.{args.0}.log
        }
}

(proxy) {
        @{args.0} host {args.0}.<redacted>
        reverse_proxy @{args.0} {args.1}
}

*.<redacted> {
        @external not remote_ip 192.168.0.0/16
        route @external {
                authorize with securelogin
        }
        import dns
        import encoding
        import logging services

        # import all proxy services
        import proxy dns https://<redacted>:443
        import proxy duplicati duplicati:8200
        import proxy homebridge homebridge.lan:8080
        import proxy ipmi ipmi.lan:80
        import proxy plex plex.lan:32400
        import proxy nas https://<redacted>:443
        import proxy unifi https://unifi.lan:8443
}

<redacted> {
        @external not remote_ip 192.168.0.0/16
        route @external {
                authorize with securelogin
        }
        import dns
        import logging proxy
        import encoding
        templates
        file_server
        root * <redacted>
}

<redacted> {
        import dns
        import encoding
        import logging sale
        file_server
        root * <redacted>
}

<redacted>, <redacted> {
        import dns
        import encoding
        import logging nextcloud
        header {
                # Enable HTTP Strict Transport Security (HSTS) to force clients to always
                # connect via HTTPS (do not use if only testing)
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
                # Enable cross-site filter (XSS) and tell browser to block detected attacks
                X-XSS-Protection "1; mode=block"
                # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
                X-Content-Type-Options nosniff
                # Disallow the site to be rendered within a frame (clickjacking protection)
                X-Frame-Options SAMEORIGIN
                # Remove Server field
                -Server
                # Upgrade insecure requests
                Content-Security-Policy "upgrade-insecure-requests"
                # Don't include referrer info in requests
                # Referrer-Policy "no-referrer"
        }
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301
        redir /.well-known/webfinger /index.php/.well-known/webfinger 301
        reverse_proxy https://<redacted>.:443
}

<redacted> {
        import encoding
        import logging auth
        authenticate with secureportal
}

Version Information

Provide output of caddy list-modules -versions | grep git below:

v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

dns.providers.cloudflare v0.0.0-20210607183747-91cf700356a1
dynamic_dns v0.0.0-20220312031409-f638ea80fe56
dynamic_dns.ip_sources.simple_http v0.0.0-20220312031409-f638ea80fe56
dynamic_dns.ip_sources.upnp v0.0.0-20220312031409-f638ea80fe56
http.authentication.providers.authorizer v1.1.14
http.handlers.authenticator v1.1.14
http.handlers.webdav v0.0.0-20210914165325-f7b67f8ca1e6
security v1.1.14

Expected behavior

I'd like the server to run smoothly again.

[greenpau]

@sidbena , thank you for the issue! There were no code changes that might have the described impact. Is it possible that you server traffic is proxied or firewalled?

[sidbena]

Yeah, that’s what’s a little odd, I haven’t touched the setup in weeks, since moving to the ‘security’ plug-in from the old setup that required two plugins, like ‘jwt’ and one other.
I haven’t upgraded the caddy version, no ‘pkg’ updates on the freebsd side either.

I really don’t know what’s changed, really frustrating.
The traffic to the server is port-forwarded (this firewall config has been unchanged for well over a year) but apart from that no other ‘filters’ are involved.
Is there perhaps more info I can gather with some verbose logging or so?

[greenpau]

@sidbena , you could add “debug” to your config. The error io/timeout suggests that your tcp connection fails.

[sidbena]

This is really ridiculous ... I tried adding debug to the general portion of my caddyfile today (been on-and-off for a bit) and the server just spun up, no errors, no problems ...
I have no idea what's changed. I have done nothing to the config, besides adding debug.
I've since tried to remove it too, and it keeps working, so I must've had some issue on the network or so.
I really can't explain it. Sorry to have taken up some of your time!
Awesome plugin, btw!

[greenpau]

@sidbena , it could have been something on the network side. Glad it worked out!

[greenpau]

@sidbena , I am looking to add testimonial sections to https://authcrunch.com. Could you please write one and send it to me at greenpau@outlook.com?