[Generic] GET request for userdata_url built incorrectly

[lyh16]

Bug description

oauthenticator keeps adding a colon(":") to the end of the specified userdata_url during OAuth authentication, causing a 403 error and ultimately leading to an Error 500 on the-littlest-jupyterhub.

Expected behaviour

OAuth authentication should be done successfully.

GET https://<REDACTED>/application/o/userinfo/

Actual behaviour

the-littlest-jupyterhub fails to login via GenericOAuthenticator.

(Emphasis on colon at the end)
GET https://<REDACTED>/application/o/userinfo/:

How to reproduce

1. Populate config.yaml accordingly.
2. Add a few customizations to jupyterhub_config.d.
3. Run tljh-config reload

Your personal set up

• OS:
  Ubuntu 20.04
• Version(s):
  Jupyterhub v4.0.1, Python v3.8.10
• SSO Server:
  Authentik v2023.4.1 (Docker) - separate server from tljh
• Jupyterhub:
  the-littlest-jupyterhub v.latest (Docker) - separate server from SSO

Full environment

# paste output of `pip freeze` or `conda list` here
alembic==1.11.1
async-generator==1.10
attrs==23.1.0
certifi==2019.11.28
certipy==0.1.3
cffi==1.15.1
chardet==3.0.4
cryptography==41.0.2
dbus-python==1.2.16
distro==1.4.0
distro-info===0.23ubuntu1
greenlet==2.0.2
idna==2.8
importlib-metadata==6.8.0
importlib-resources==6.0.0
Jinja2==3.1.2
jsonschema==4.18.4
jsonschema-specifications==2023.7.1
jupyter-telemetry==0.1.0
jupyterhub==4.0.1
Mako==1.2.4
MarkupSafe==2.1.3
oauthenticator==16.0.3
oauthlib==3.2.2
packaging==23.1
pamela==1.1.0
pkgutil-resolve-name==1.3.10
prometheus-client==0.17.1
pycparser==2.21
PyGObject==3.36.0
pyOpenSSL==23.2.0
python-apt==2.0.1+ubuntu0.20.4.1
python-dateutil==2.8.2
python-json-logger==2.0.7
referencing==0.30.0
requests==2.22.0
requests-unixsocket==0.2.0
rpds-py==0.9.2
ruamel.yaml==0.17.32
ruamel.yaml.clib==0.2.7
six==1.14.0
SQLAlchemy==2.0.19
ssh-import-id==5.10
tornado==6.3.2
traitlets==5.9.0
typing-extensions==4.7.1
unattended-upgrades==0.1
urllib3==1.25.8
zipp==3.16.2

Configuration

# config.yaml
users:
  admin:
  - <REDACTED>
user_environment:
  default_app: jupyterlab
services:
  cull:
    timeout: 2630000
auth:
  type: oauthenticator.generic.GenericOAuthenticator

# jupyterhub_config.d/admin_access.py
c.JupyterHub.admin_access = True

# jupyterhub_config.d/authentik.py
c.GenericOAuthenticator.client_id = "<REDACTED>"
c.GenericOAuthenticator.client_secret = "<REDACTED>"
c.GenericOAuthenticator.oauth_callback_url = "https://<REDACTED>/hub/oauth_callback"

c.GenericOAuthenticator.authorize_url = "https://<REDACTED>/application/o/authorize/"
c.GenericOAuthenticator.token_url = "https://<REDACTED>/application/o/token/"
c.GenericOAuthenticator.userdata_url = "https://<REDACTED>/application/o/userinfo/"
c.GenericOAuthenticator.logout_redirect_url = "<REDACTED>/application/o/<REDACTED>/end-session/"

c.LocalAuthenticator.create_system_users = True

c.GenericOAuthenticator.login_service = "Authentik"
c.GenericOAuthenticator.username_key = "username"
c.GenericOAuthenticator.allowed_groups = {"<REDACTED>", "<REDACTED>"}
c.GenericOAuthenticator.admin_users = {"<REDACTED>"}

Logs

# paste relevant logs here, if any
Jul 20 05:50:47 hostname systemd[1]: Started jupyterhub.service.
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.023 JupyterHub app:2859] Running JupyterHub version 4.0.1
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.024 JupyterHub app:2889] Using Authenticator: oauthenticator.generic.GenericOAuthenticator-16.0.3
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.024 JupyterHub app:2889] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.024 JupyterHub app:2889] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-1.1.0
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.037 JupyterHub app:1664] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.356 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.512 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.580 JupyterHub app:2928] Initialized 0 spawners in 0.068 seconds
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.592 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.596 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.seven_days
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.598 JupyterHub metrics:278] Found 0 active users in the last ActiveUserPeriods.thirty_days
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.599 JupyterHub app:3142] Not starting proxy
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.604 JupyterHub app:3178] Hub API listening on http://127.0.0.1:15001/hub/
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.604 JupyterHub app:3189] Starting managed service cull-idle
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.604 JupyterHub service:385] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=2630000', '--cull-every=60', '--concurrency=5', '--max-age=0']
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.605 JupyterHub service:133] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=2630000 --cull-every=60 --concurrency=5 --max-age=0
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.611 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://127.0.0.1:15001/hub/
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.854 JupyterHub log:191] 200 GET /hub/api/ (cull-idle@127.0.0.1) 34.77ms
Jul 20 05:50:54 hostname python3[7758]: [I 2023-07-20 05:50:54.891 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 35.09ms
Jul 20 05:50:55 hostname python3[7758]: [I 2023-07-20 05:50:55.277 JupyterHub log:191] 200 GET /hub/api (@127.0.0.1) 1.21ms
Jul 20 05:51:11 hostname python3[7758]: [I 2023-07-20 05:51:11.770 JupyterHub log:191] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@172.24.0.1) 0.84ms
Jul 20 05:51:12 hostname python3[7758]: [I 2023-07-20 05:51:12.066 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2F (@172.24.0.1) 68.60ms
Jul 20 05:51:13 hostname python3[7758]: [I 2023-07-20 05:51:13.935 JupyterHub oauth2:102] OAuth redirect: https://<REDACTED>/hub/oauth_callback
Jul 20 05:51:13 hostname python3[7758]: [I 2023-07-20 05:51:13.936 JupyterHub log:191] 302 GET /hub/oauth_login?next=%2Fhub%2F -> https://<REDACTED>/application/o/authorize/?response_type=code&redirect_uri=https%3A%2F%2Faica.imtl.re.kr%2Fhub%2Foauth_callback&client_id=KR1LQrhMOKA6StFviUxt1iB3zjIsrUznTxScSpwZ&state=[secret] (@172.24.0.1) 1.70ms
Jul 20 05:51:16 hostname python3[7758]: [E 2023-07-20 05:51:16.351 JupyterHub oauth2:620] Error Fetching user info... 403 GET https://<REDACTED>/application/o/userinfo/:
Jul 20 05:51:16 hostname python3[7758]: [E 2023-07-20 05:51:16.351 JupyterHub web:1871] Uncaught exception GET /hub/oauth_callback?code=1693c6b2b52c41b98a5b8c8323193d3c&state=eyJzdGF0ZV9pZCI6ICI0MjFlODQyMWFjMWM0OWMxYTNjYzJlZGU3YTkzMmYyNyIsICJuZXh0X3VybCI6ICIvaHViLyJ9 (172.24.0.1)
Jul 20 05:51:16 hostname python3[7758]:     HTTPServerRequest(protocol='http', host='<REDACTED>', method='GET', uri='/hub/oauth_callback?code=1693c6b2b52c41b98a5b8c8323193d3c&state=eyJzdGF0ZV9pZCI6ICI0MjFlODQyMWFjMWM0OWMxYTNjYzJlZGU3YTkzMmYyNyIsICJuZXh0X3VybCI6ICIvaHViLyJ9', version='HTTP/1.1', remote_ip='172.24.0.1')
Jul 20 05:51:16 hostname python3[7758]:     Traceback (most recent call last):
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/tornado/web.py", line 1786, in _execute
Jul 20 05:51:16 hostname python3[7758]:         result = await result
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 202, in get
Jul 20 05:51:16 hostname python3[7758]:         user = await self.login_user()
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 826, in login_user
Jul 20 05:51:16 hostname python3[7758]:         authenticated = await self.authenticate(data)
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user
Jul 20 05:51:16 hostname python3[7758]:         authenticated = await maybe_future(self.authenticate(handler, data))
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 958, in authenticate
Jul 20 05:51:16 hostname python3[7758]:         user_info = await self.token_to_user(token_info)
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 873, in token_to_user
Jul 20 05:51:16 hostname python3[7758]:         return await self.httpfetch(
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 655, in httpfetch
Jul 20 05:51:16 hostname python3[7758]:         return await self.fetch(
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 621, in fetch
Jul 20 05:51:16 hostname python3[7758]:         raise e
Jul 20 05:51:16 hostname python3[7758]:       File "/opt/tljh/hub/lib/python3.8/site-packages/oauthenticator/oauth2.py", line 600, in fetch
Jul 20 05:51:16 hostname python3[7758]:         resp = await self.http_client.fetch(req, **kwargs)
Jul 20 05:51:16 hostname python3[7758]:     tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
Jul 20 05:51:16 hostname python3[7758]:     
Jul 20 05:51:16 hostname python3[7758]: [E 2023-07-20 05:51:16.442 JupyterHub log:183] {
Jul 20 05:51:16 hostname python3[7758]:       "Host": "<REDACTED>",
Jul 20 05:51:16 hostname python3[7758]:       "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
Jul 20 05:51:16 hostname python3[7758]:       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
Jul 20 05:51:16 hostname python3[7758]:       "Accept-Encoding": "gzip",
Jul 20 05:51:16 hostname python3[7758]:       "Accept-Language": "en-US,en;q=0.5",
Jul 20 05:51:16 hostname python3[7758]:       "Cdn-Loop": "cloudflare",
Jul 20 05:51:16 hostname python3[7758]:       "Cf-Connecting-Ip": "<REDACTED>",
Jul 20 05:51:16 hostname python3[7758]:       "Cf-Ipcountry": "<REDACTED>",
Jul 20 05:51:16 hostname python3[7758]:       "Cf-Ray": "7e98ee68ac2219c9-KIX",
Jul 20 05:51:16 hostname python3[7758]:       "Cf-Visitor": "{\"scheme\":\"https\"}",
Jul 20 05:51:16 hostname python3[7758]:       "Cf-Warp-Tag-Id": "2f5ca5a4-957d-4be2-95c5-fc080a93c01d",
Jul 20 05:51:16 hostname python3[7758]:       "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
Jul 20 05:51:16 hostname python3[7758]:       "Dnt": "1",
Jul 20 05:51:16 hostname python3[7758]:       "Priority": "u=1",
Jul 20 05:51:16 hostname python3[7758]:       "Sec-Fetch-Dest": "document",
Jul 20 05:51:16 hostname python3[7758]:       "Sec-Fetch-Mode": "navigate",
Jul 20 05:51:16 hostname python3[7758]:       "Sec-Fetch-Site": "same-site",
Jul 20 05:51:16 hostname python3[7758]:       "Sec-Gpc": "1",
Jul 20 05:51:16 hostname python3[7758]:       "Upgrade-Insecure-Requests": "1",
Jul 20 05:51:16 hostname python3[7758]:       "X-Forwarded-For": "<REDACTED>",
Jul 20 05:51:16 hostname python3[7758]:       "X-Forwarded-Host": "<REDACTED>",
Jul 20 05:51:16 hostname python3[7758]:       "X-Forwarded-Port": "80",
Jul 20 05:51:16 hostname python3[7758]:       "X-Forwarded-Proto": "http",
Jul 20 05:51:16 hostname python3[7758]:       "X-Forwarded-Server": "hostname",
Jul 20 05:51:16 hostname python3[7758]:       "X-Real-Ip": "<REDACTED>"
Jul 20 05:51:16 hostname python3[7758]:     }
Jul 20 05:51:16 hostname python3[7758]: [E 2023-07-20 05:51:16.442 JupyterHub log:191] 500 GET /hub/oauth_callback?code=[secret]&state=[secret] (@172.24.0.1) 1103.29ms

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[mangecoeur]

I'm am also having problems with userdata fetching but I checked the code and it is not adding a colon, if you look at the code in oatuh2.py:620

app_log.error(f"Error {label} {e.code} {req.method} {url}: {message}")

the colon is part of the error message. The problem is there is no message added which might better explain the error.

[consideRatio]

Thanks @mangecoeur for figuring that out! Okay, so there is an issue with userdata fetching, but unrelated to the colon added as part of an error message.

I think at this point I lack information to debug the actual issue then.

[mangecoeur]

P.S I at least found out why I am not getting an error message: it seems most Oauth providers return errors as Json with some information, but in my case connecting to Microsoft ADFS based server it returns an HTTP error 401 with no body. I was able to extract the access token and test it out in an HTTP client which gives access to the WWW-Authenticate header that contains more info (error MSIS9921 in my case)

[mangecoeur]

P.P.S not sure this is the best place to store this knowledge but with the help of
https://serverfault.com/questions/1109792/setting-up-oidc-with-adfs-invalid-userinfo-request I figured out that you need to define resource=urn:microsoft:userinfo for the inital authorize request. I finally found the (somewhat hidden) extra_authorize_params setting so now I have c.GenericOAuthenticator.extra_authorize_params = {"resource": "urn:microsoft:userinfo"} and it works!