[bug] Idle culler doesn't appear to be working properly

[kcpevey]

Describe the bug

• We have the idle culler set to 30 minutes. But after this time period, the server is still running. This happens when the browser is minimized and when there isn't anything obviously actively running.

Expected behaviour

• I expect that after 30 minutes of not touching my browser window (leaving it minimized), my server should be culled

Actual behaviour

• after 30 minutes of not touching my browser window (leaving it minimized), my server is not culled

How to reproduce

For example,

1. load up a server
2. run a cell in a notebook with only imports (nothing obviously continuously running).
3. minimize the window
4. come back after the idle-culller-timeout
5. run another cell in that notebook
6. the server is still live.

Your personal set up

OS

• * OSX

[danlester]

Note that I did notice idle culler was missing from the jupyterhub image, and added it in here recently:

#959

I haven't studied things in enough detail yet to see if that solves all of your problems...

Be aware that by default cdsdashboards actually tries to 'fake' activity: https://cdsdashboards.readthedocs.io/en/stable/chapters/troubleshooting.html#dashboards-and-servers-keep-dying

I'm sure this has come up before though.

[leej3]

I think another source of the problem is that the idle culler has different behavior for jupyterlab. From the readme:

Note that cull_connected can be tricky to understand for JupyterLab as a browser having a web-socket
 connection to a kernel or not isn't as obvious as it was in the classical Jupyter notebook UI.
 [See this issue for more details](https://github.com/jupyterlab/jupyterlab/issues/6893).

[dharhas]

@magsol this probably needs to be investigated for the 0.4 release. From what I can tell it is an issue on the demo deployment I have access to.

[costrouc]

@viniciusdc is going to test that the idle culler has been working in 0.4 and detailing in the docs better what the idle culler does.

[viniciusdc]

The ide culler is working as expected, but it has some issues when running alongside jupyterhub, as the browser windows actually open a WebSocket connection that generates a false positive, not idle status, as pointed out by @leej3 above.

I am currently working on a fix to address kernel and idle connections for v0.4... Besides that, login out from the server definitely triggers the culling process after the timeout period (the example bellow had its cull_timeout variable modified for testing):

[I 2022-02-25 20:22:46.932 JupyterHub log:189] 200 GET /hub/api/users (cull-idle@127.0.0.1) 29.73ms
[I 220225 20:22:46 __init__:192] Culling server bob (inactive for 00:27:49)
[I 2022-02-25 20:22:46.939 JupyterHub proxy:309] Removing user bob from proxy (/user/bob/)
[I 2022-02-25 20:22:46.942 JupyterHub spawner:2620] Deleting pod dev/jupyter-bob
[I 2022-02-25 20:22:48.867 JupyterHub base:1116] User bob server took 1.928 seconds to stop
[I 2022-02-25 20:22:48.869 JupyterHub log:189] 204 DELETE /hub/api/users/bob/server (cull-idle@127.0.0.1) 1934.65ms