Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] - Hub pod fails if keycloak doesn't starts before it #2547

Open
aktech opened this issue Jun 27, 2024 · 0 comments
Open

[BUG] - Hub pod fails if keycloak doesn't starts before it #2547

aktech opened this issue Jun 27, 2024 · 0 comments
Labels
type: bug 🐛 Something isn't working

Comments

@aktech
Copy link
Member

aktech commented Jun 27, 2024

Describe the bug

The hub pod Errors out if keycloak isn't started already.

│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Authenticator: builtins.KeyCloakOAuthenticator                                                                                                                                                                         │
│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Spawner: jhub_apps.spawner.spawner_creation.JHubSpawner                                                                                                                                                                │
│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.0.0                                                                                                                                                                    │
│ [I 2024-06-27 10:02:20.801 JupyterHub <string>:109] Loading managed roles                                                                                                                                                                                                    │
│ [E 2024-06-27 10:02:20.811 JupyterHub app:3852]                                                                                                                                                                                                                              │
│     Traceback (most recent call last):                                                                                                                                                                                                                                       │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 3849, in launch_instance_async                                                                                                                                                                   │
│         await self.initialize(argv)                                                                                                                                                                                                                                          │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 3332, in initialize                                                                                                                                                                              │
│         await self.init_role_creation()                                                                                                                                                                                                                                      │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 2286, in init_role_creation                                                                                                                                                                      │
│         managed_roles = await self.authenticator.load_managed_roles()                                                                                                                                                                                                        │
│       File "<string>", line 114, in load_managed_roles                                                                                                                                                                                                                       │
│       File "<string>", line 221, in _get_token                                                                                                                                                                                                                               │
│     tornado.httpclient.HTTPClientError: HTTP 503: Service Unavailable

This is due to the fact that jupyterhub calls keycloak API on startup to get roles into JupyterHub (see load_managed_roles)

Expected behavior

Hub waits (retries for some time) for keycloak to start before dying out.

OS and architecture in which you are running Nebari

NA

How to Reproduce the problem?

Kill keycloak pod and restart Hub pod

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

@aktech aktech added type: bug 🐛 Something isn't working needs: triage 🚦 Someone needs to have a look at this issue and triage labels Jun 27, 2024
@Adam-D-Lewis Adam-D-Lewis removed the needs: triage 🚦 Someone needs to have a look at this issue and triage label Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug 🐛 Something isn't working
Projects
Status: New 🚦
Development

No branches or pull requests

2 participants