Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook) #25

mjbright · 2015-09-09T19:16:29Z

OK, so this is more of a question ... than an issue.

I wanted to add bash_kernel to jupyter/minimal-notebook.

Not quite sure about how start-notebook.sh is being used I decided to make a modified copy into which I added the following lines before the final exec line.

    CBIN=/opt/conda/bin                                                                                                                            

    conda install pip                                                                                                                              

    sudo -u $NB_USER $CBIN/pip install --user bash_kernel                                                                                          
    sudo -u $NB_USER $CBIN/python -m bash_kernel.install

However these commands don't seem to be executed when invoked by the CMD of the Dockerfile.
Connecting to the docker container and rerunning /start-notebook.sh installs the kernel as expected.

Why is this manual step necessary, and how what should I do so that it's not necessary?

The text was updated successfully, but these errors were encountered:

parente · 2015-09-10T11:55:29Z

I haven't tried it, but I think I would opt to put all additional install steps in a new Dockerfile that derives FROM the minimal notebook like so:

FROM jupyter/minimal-notebook:4.0

/opt/conda/bin/pip install --user bash_kernel
/opt/conda/bin/python -m bash_kernel.install

I would docker build -t jupyter/bash-notebook . and spawn a container from that image.

The start-notebook.sh script is really about late-creation of the jovyan user and housekeeping to ensure permissions are correct for that user in the home directory and the conda directory.

mjbright · 2015-09-10T12:27:32Z

Yes, that was the initial approach I'd taken until I realized that this
wasn't working as bash_kernel was installing only for the current, i.e.
root user, so I moved it after the jovyan user creation (so out of the
Dockerfile into start-notebook.sh).

Whilst bash_kernel could install system-wide, that would solve the problem-
I think would be interesting to be able to do derivations after the
creation of the jovyan user.

I guess there were good reasons to move the jovyan user creation out of the
Dockerfile ...

On 10 September 2015 at 13:55, Peter Parente notifications@github.com
wrote:

I haven't tried it, but I think I would opt to put all additional install
steps in a new Dockerfile that derives FROM the minimal notebook like so:

FROM jupyter/docker-stacks:4.0

/opt/conda/bin/pip install --user bash_kernel
/opt/conda/bin/python -m bash_kernel.install

I would docker build -t jupyter/bash-notebook . and spawn a container
from that image.

The start-notebook.sh script is really about late-creation of the jovyan
user and housekeeping to ensure permissions are correct for that user in
the home directory and the conda directory.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

parente · 2015-09-10T12:32:20Z

Whilst bash_kernel could install system-wide, that would solve the problem-

Ah, yes, removing --user would do the trick. I see it now.

I think would be interesting to be able to do derivations after the
creation of the jovyan user.

Do you have a use case in mind? I'm trying to think of one and have come up short (but it's still early in the day ;)

I guess there were good reasons to move the jovyan user creation out of the
Dockerfile ...

Yes. See issue #16. I've heard rumors that Docker 1.9 might finally have host/container user mapping. If so, and if it works on reasonable kernel versions, we might be able to remove the bandaid.

parente · 2015-09-10T12:38:14Z

Another possibility.

Undo the user creation in script and move it back to Dockerfile so that the username / home direct exist for commands in the Dockerfile to work as expected.
Change the start-notebook script so that instead of late-creating the user there, it instead changes in the UID/GID of the jovyan user and fixes permissions (e.g., https://muffinresearch.co.uk/linux-changing-uids-and-gids-for-user/)

Pros

User exists at docker build time
UID can be configured to fix host/container permission problems

Cons

Have to scan the entire container to fix perms at start time OR limit to specific directories to scan by default (e.g., /home/jovyan, /opt/conda)
Bandaid for docker shortcoming that may get resolved in a real-soon-now version

mjbright · 2015-09-10T12:48:10Z

I'm not aware of the uid/permission problem, but I do like the sound of
doing things cleanly in the Dockerfile, even if it requires bandaid to
workaround some hopefully temporary Docker issues.

Although my 'bash_kernel' issue is probably a feature of that kernel -
which could be fixed to install system wide, I think it would be better to
be able to install things as the target user where possible.

On 10 September 2015 at 14:38, Peter Parente notifications@github.com
wrote:

Another possibility.

Undo the user creation in script and move it back to Dockerfile so
that the username / home direct exist for commands in the Dockerfile to
work as expected.

Change the start-notebook script so that instead of late-creating
the user there, it instead changes in the UID/GID of the jovyan user and
fixes permissions (e.g.,
https://muffinresearch.co.uk/linux-changing-uids-and-gids-for-user/)

Pros

User exists at docker build time

UID can be configured to fix host/container permission problems

Cons

Have to scan the entire container to fix perms at start time OR
limit to specific directories to scan by default (e.g., /home/jovyan,
/opt/conda)

Bandaid for docker shortcoming that may get resolved in a
real-soon-now version

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

parente · 2015-09-10T12:50:38Z

I can give the create jovyan early / fix uid later approach a shot to see if it's terribly slow or if there are other surprises.

/cc @costerwi @rgbkrk for other perspectives

parente · 2015-09-10T12:52:04Z

But for the moment, the workaround of doing /opt/conda/bin/pip install bash_kernel to install it global is not that terrible. The container is for a single user anyway, imho.

mjbright · 2015-09-10T12:56:35Z

Sure ... but

that has to be done after creation of that user
for some reason this wasn't getting "done" on the first invocation of
start-notebook.sh (?)
I had to manually run my modified start-notebook.sh once the container
was up.

Which I can keep doing, it's just ugly and not-automated.

Thanks for the suggestions.

On 10 September 2015 at 14:52, Peter Parente notifications@github.com
wrote:

But for the moment, the workaround of doing /opt/conda/bin/pip install
bash_kernel to install it global is not that terrible. The container is
for a single user anyway, imho.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

parente · 2015-09-10T12:58:22Z

that has to be done after creation of that user

Even without the --user flag? I thought that would work fine without the user in existence and install to /usr/local/ or some other global PREFIX.

mjbright · 2015-09-10T13:11:03Z

Oops, sorry I'll have to try that (sorry didn't know the --user flag and
assumed it did something quite different !)

Will try tonight.

On 10 September 2015 at 14:58, Peter Parente notifications@github.com
wrote:

that has to be done after creation of that user

Even without the --user flag? I thought that would work fine without the
user in existence and install to /usr/local/ or some other global PREFIX.

—
Reply to this email directly or view it on GitHub
#25 (comment)
.

parente · 2015-09-10T13:12:54Z

I just tried it and the kernel spec didn't install anywhere wtihout --user. And now I see why: system wide install is not yet supported takluyver/bash_kernel#31.

jovyan@c0e462d94b33:/$ jupyter --paths                                                                                             
config:                                                                                                                            
    /home/jovyan/.jupyter                                                                                                          
    /opt/conda/etc/jupyter                                                                                                         
    /usr/local/etc/jupyter                                                                                                         
    /etc/jupyter                                                                                                                   
data:                                                                                                                              
    /home/jovyan/.local/share/jupyter                                                                                              
    /opt/conda/share/jupyter                                                                                                       
    /usr/local/share/jupyter                                                                                                       
    /usr/share/jupyter                                                                                                             
runtime:                                                                                                                           
    /home/jovyan/.local/share/jupyter/runtime

costerwi · 2015-09-10T14:04:17Z

I'm not sure why the modified start-notebook.sh didn't work. Can you check docker logs for that image?

As a different ugly workaround, a derived container may be able to install or copy misbehaving packages like this into /etc/skel and allow the existing start-notebook.sh to move them into jovyan's home.

parente · 2015-09-11T03:37:05Z

With PR #31, this Dockerfile now works:

FROM jupyter/minimal-notebook

USER jovyan

# Make sure not to create a cache dir else NB_UID switching
# will hit issues.
RUN /opt/conda/bin/pip install --no-cache-dir bash_kernel
RUN /opt/conda/bin/python -m bash_kernel.install

USER root

parente · 2015-09-13T02:11:09Z

After the PR #31 merge,

docker pull jupyter/minimal-notebook:latest
# in a folder containing the Dockerfile from https://github.com/jupyter/docker-stacks/issues/25#issuecomment-139444841
docker build --rm -t jupyter/bash-notebook .
docker run --rm -it --p 8888:8888 jupyter/bash-notebook

Also works fine if I change the UID of jovyan (-e NB_UID=1005).

mjbright · 2015-09-13T07:45:16Z

Thanks a lot Peter, I really appreciate the work you put in on this!

mjbright · 2015-09-13T08:00:56Z

... but ... any idea why this worked the first time (first build and run) but then I keep getting the following errors on subsequent runs?

(My host is a Fedora 22, I'm running docker 1.8.1 by the way)

2015-09-13 07:42:50,019 CRIT Supervisor running as root (no user in config file)
2015-09-13 07:42:50,019 WARN Included extra file "/etc/supervisor/conf.d/notebook.conf" during parsing
2015-09-13 07:42:50,034 INFO RPC interface 'supervisor' initialized
2015-09-13 07:42:50,034 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-13 07:42:50,034 INFO supervisord started with pid 1
2015-09-13 07:42:51,039 INFO spawned: 'notebook' with pid 9
2015-09-13 07:42:51,050 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:52,053 INFO spawned: 'notebook' with pid 10
2015-09-13 07:42:52,061 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:54,066 INFO spawned: 'notebook' with pid 11
2015-09-13 07:42:54,077 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:57,082 INFO spawned: 'notebook' with pid 12
2015-09-13 07:42:57,092 INFO exited: notebook (exit status 127; not expected)
2015-09-13 07:42:58,094 INFO gave up: notebook entered FATAL state, too many start retries too quickly

mjbright · 2015-09-13T08:45:16Z

OK, found the problem.

Sunday driver - I was doing a bad mount on my docker run ....

hey, it's Sunday morning ;-)

rgbkrk · 2015-09-13T19:35:45Z

😄

Secure the oauth cookie better

mjbright mentioned this issue Sep 10, 2015

start-notebook.sh should be split into config-notebook.sh and start-notebook.sh #27

Closed

costerwi mentioned this issue Sep 10, 2015

Global install possible as root takluyver/bash_kernel#35

Merged

This was referenced Sep 11, 2015

datascience-notebook: Julia kernel not available for selection #30

Closed

Recursively set group permission on $CONDA_DIR #28

Closed

parente mentioned this issue Sep 11, 2015

Create jovyan in Dockerfile, reset UID at startup #31

Merged

parente closed this as completed Sep 13, 2015

parente mentioned this issue Sep 25, 2015

bash kernel not loading. Need to update to two step process jupyter/docker-demo-images#64

Closed

rochaporto pushed a commit to rochaporto/docker-stacks that referenced this issue Jan 23, 2019

Merge pull request jupyter#25 from yuvipanda/secure-cookies

0fd4100

Secure the oauth cookie better

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook) #25

Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook) #25

mjbright commented Sep 9, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

costerwi commented Sep 10, 2015

parente commented Sep 11, 2015

parente commented Sep 13, 2015

mjbright commented Sep 13, 2015

mjbright commented Sep 13, 2015

mjbright commented Sep 13, 2015

rgbkrk commented Sep 13, 2015

Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook) #25

Unable to automate installation of bash_kernel (on top of jupyter/minimal-notebook) #25

Comments

mjbright commented Sep 9, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

mjbright commented Sep 10, 2015

parente commented Sep 10, 2015

costerwi commented Sep 10, 2015

parente commented Sep 11, 2015

parente commented Sep 13, 2015

mjbright commented Sep 13, 2015

mjbright commented Sep 13, 2015

mjbright commented Sep 13, 2015

rgbkrk commented Sep 13, 2015