User symlink files in /run/ are not deleted when server is stopped, causing 500 internal server error

[gmboyer]

Bug:

If a user stops their JupyterHub server through their control panel and then attempts to start it again, they will receive an error when systemdspawner spawns the user:

"500 : Internal Server Error Error in Authenticator.pre_spawn_start: FileExistsError [Errno 17] File exists: '/run/jupyter-username-singleuser'

The user cannot connect until I manually delete the user's symlink file in /run/.

Perhaps this is a bug where the user's symlink isn't automatically deleted from /run/ when they stop their server. When the spawner tries to start their server again, it erroneously thinks the server is already running and throws the 500 error.

Additional information:

I'm using a GitLab OAuthenticator with systemdspawner's dynamic_spawning set to true, if that helps.

Ubuntu 20.04
JupyterHub 1.2.2
Latest version of systemdspawner as of Dec. 15, 2020

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[yuvipanda]

@minrk do you think this is related to the recent fix to hide tokens from the process list?

[bcbnz]

I encountered the same problem (except in my case the links were at /run/jupyterhub-username). I have temporarily solved it by adding the following code to systemdspawner/systemdspawner.py at line 219 (just after the 'check if service failed' block):

        runlink = '/run/jupyterhub-{}'.format(self.user.name)
        if os.path.lexists(runlink) and not os.path.exists(runlink):
            self.log.info('Removing stale link %s', runlink)
            os.unlink(runlink)

os.path.exists returns False for broken symlinks while os.path.lexists returns True, so this should avoid trying to remove the link if its target still exists.

This is too hard-coded to be a general solution (especially since the path in my case is different to the original report), but may be useful to others until a proper solution is developed.

Also, I see the following messages in the logs when a user server is started up. I'm not sure if the migration message is relevant to this issue.

Jan 05 12:14:33 image systemd[1]: Started /bin/bash -c cd /var/lib/jupyterhub_users/username && exec jupyterhub-singleuser --port=47567.
Jan 05 12:14:33 image jupyterhub[108]: Running as unit: jupyterhub-username.service
Jan 05 12:14:33 image systemd[110]: Found pre-existing public RuntimeDirectory= directory /run/jupyterhub-username, migrating to /run/private/jupyterhub-username.
Jan 05 12:14:33 image systemd[110]: Apparently, service previously had DynamicUser= turned off, and has now turned it on.

[gmboyer]

@bcbnz Your solution worked like a charm. Thank you for taking the time to post this workaround. For those who have the same symlink format I did, modify the first line of bcbnz's code to:

runlink = '/run/jupyter-{}-singleuser'.format(self.user.name)

[fracaron]

For me adding c.SystemdSpawner.unit_extra_properties = {'RuntimeDirectoryPreserve': 'no'} to jupyterhub config resolved the issue. The user's environment file was deleted after restarting the jupyterhub service while the symlink to it was preserved.

The default set by systemdspawner 'RuntimeDirectoryPreserve': 'restart' will only work if used with c.JupyterHub.cleanup_servers = False. cleanup_servers = False will avoid shuting down the single-user servers with jupyterhub restart.

Using Ubuntu 20.04 and jupyterhub 1.3.0

[doyeonkp]

change the code in systemd.py
Currently it looks like this

def ensure_environment_directory(environment_file_directory):
    """Ensure directory for environment files exists and is private"""
    # ensure directory exists
    os.makedirs(environment_file_directory, mode=0o700, exist_ok=True)
    # validate permissions
    mode = os.stat(environment_file_directory).st_mode
    if mode & 0o077:
        warnings.warn(
            f"Fixing permissions on environment directory {environment_file_directory}: {oct(mode)}",
            RuntimeWarning,
        )
        os.chmod(environment_file_directory, 0o700)

The log said [Error17] File exists, so I assume os.makedirs(environment_file_directory, mode=0o700, exist_ok=True) this line is cause of the error.

so changed as

def ensure_environment_directory(environment_file_directory):
    """Ensure directory for environment files exists and is private"""
    # ensure directory exists
    if not os.path.isdir(environment_file_directory):
        os.makedirs(environment_file_directory, mode=0o700, exist_ok=True)
    # validate permissions
    mode = os.stat(environment_file_directory).st_mode
    if mode & 0o077:
        warnings.warn(
            f"Fixing permissions on environment directory {environment_file_directory}: {oct(mode)}",
            RuntimeWarning,
        )
        os.chmod(environment_file_directory, 0o700)

so check the file exist then create.
Now mine is work fine. Hope it helps someone face the same issue like me

[adsche]

Hey all, I believe the more straightforward solution might be to just set RUN_ROOT='/run/private if DynamicUser==True. This works for me and is in accordance with the systemd documentation on DynamicUser [1]. Sure, skipping the check as in the above solutions also works, but it will also modify the mode of the symlink (causing a warning message from systemd) instead of the actual state dir in /run/private...

[1] https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html - but look for the section starting with RuntimeDirectory= ...