Directory listing chokes on non-ASCII chars

[nkeim]

In a directory listing, a non-ASCII character in a filename (including notebook names) abruptly terminates the listing, making all files below it invisible.

Steps to reproduce are in the linked gist, though if you are a Mac user you can just type Option-m in the Terminal to put a µ in a filename, then check the Jupyter directory listing. The character in the linked example is "µ" (micro sign), but it appears that any byte outside range(128) will do it.

https://gist.github.com/nkeim/5798b1211d52ed47993b

This has to be run with Python 2; Python 3 insists on ASCII for the filename.

Note that the IPython version is just "3.2.0-dev". @ellisonbg may be able to tell you the precise commit, if that matters.

Background: I ended up with Unicode characters in my notebook names courtesy of IPython 2.x, which seemed to handle them perfectly.

[minrk]

Hm. This may be a configuration or environment issue. I can list, create, and open unicode filenames without issue on IPython 3.x and 4.x on both Python 2 and 3. I am on OS X 10.10.4 with Pythons from conda.

[nkeim]

Thanks! The environment for the notebook server spawned by JupyterHub could easily be the problem. It appears that LANG is not set by JupyterHub.

Within the world of the notebook server, the Python 3 interpreter (Ubuntu 14.04.1 LTS /usr/bin/python3) just doesn't like Unicode in filenames. open('µm.txt', 'w').write('hi') raises a UnicodeEncodeError. But when I run the same interpreter in my login environment, there's no problem... until I take away its LANG=en_US.UTF-8.

Strangely I cannot reproduce this with Python 3 from Anaconda on my Mac, even if I deliberately give it LANG=en_US.ASCII.

[takluyver]

I think that makes sense - it's using the locale encoding to convert filenames to bytes. I think it only comes up on Linux, because OS X and Windows define how filenames are encoded independent of the locale.

Jupyterhub should probably ensure that the single-user server is started with appropriate locale settings so that it treats the filesystem as UTF-8.

The notebook server should fail more gracefully when there's a unicode problem with a filename. I'll have a look at that.

[minrk]

@nkeim what version of JupyterHub? The Hub passes its LANG, LC env variables to the notebook children by default. Do you have any custom config?

[nkeim]

Deferring to @ellisonbg for the JupyterHub version — he maintains it here.

[minrk]

@nkeim in that case, I know exactly what the problem is. The launch config (in supervisord) is not setting the LANG environment variable. This is in /etc/supervisor/conf.d/jupyterhub.conf The environment line should add:

environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8,...

to ensure that server processes get the right env.

[nkeim]

@minrk Great, thanks!

[takluyver]

#229 should stop it from breaking the rest of the list in such cases.