Skip to content

Commit

Permalink
Merge pull request #961 from parente/sudo-recipe
Browse files Browse the repository at this point in the history 
Add sudo recipe
  • Loading branch information
@parente
parente authored Nov 16, 2019
2 parents 45f07a1 + 0b2ced3 commit 213760e
Showing 1 changed file with 105 additions and 41 deletions.
146 changes: 105 additions & 41 deletions docs/using/recipes.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,32 @@
# Contributed Recipes

Users sometimes share interesting ways of using the Jupyter Docker Stacks. We encourage users to [contribute these recipes](../contributing/recipes.md) to the documentation in case they prove useful to other members of the community by submitting a pull request to `docs/using/recipes.md`. The sections below capture this knowledge.
Users sometimes share interesting ways of using the Jupyter Docker Stacks. We encourage users to
[contribute these recipes](../contributing/recipes.md) to the documentation in case they prove
useful to other members of the community by submitting a pull request to `docs/using/recipes.md`.
The sections below capture this knowledge.

## Using `sudo` within a container

Password authentication is disabled for the `NB_USER` (e.g., `jovyan`). This choice was made to
avoid distributing images with a weak default password that users ~might~ will forget to change
before running a container on a publicly accessible host.

You can grant the within-container `NB_USER` passwordless `sudo` access by adding
`-e GRANT_SUDO=yes` and `--user root` to your Docker command line or appropriate container
orchestrator config.

For example:

```
docker run -it -e GRANT_SUDO=yes --user root jupyter/minimal-notebook
```

**You should only enable `sudo` if you trust the user and/or if the container is running on an
isolated host.**

## Using `pip install` or `conda install` in a Child Docker image

Create a new Dockerfile like the one shown below.
Create a new Dockerfile like the one shown below.

```dockerfile
# Start from a core stack version
Expand All @@ -19,8 +41,8 @@ Then build a new image.
docker build --rm -t jupyter/my-datascience-notebook .
```

To use a requirements.txt file, first create your `requirements.txt` file
with the listing of packages desired. Next, create a new Dockerfile like the one shown below.
To use a requirements.txt file, first create your `requirements.txt` file with the listing of
packages desired. Next, create a new Dockerfile like the one shown below.

```dockerfile
# Start from a core stack version
Expand All @@ -44,11 +66,14 @@ RUN conda install --yes --file /tmp/requirements.txt && \
fix-permissions /home/$NB_USER
```

Ref: [docker-stacks/commit/79169618d571506304934a7b29039085e77db78c](https://github.com/jupyter/docker-stacks/commit/79169618d571506304934a7b29039085e77db78c#commitcomment-15960081)
Ref:
[docker-stacks/commit/79169618d571506304934a7b29039085e77db78c](https://github.com/jupyter/docker-stacks/commit/79169618d571506304934a7b29039085e77db78c#commitcomment-15960081)

## Add a Python 2.x environment

Python 2.x was removed from all images on August 10th, 2017, starting in tag `cc9feab481f7`. You can add a Python 2.x environment by defining your own Dockerfile inheriting from one of the images like so:
Python 2.x was removed from all images on August 10th, 2017, starting in tag `cc9feab481f7`. You can
add a Python 2.x environment by defining your own Dockerfile inheriting from one of the images like
so:

```
# Choose your desired base image
Expand All @@ -70,33 +95,43 @@ $CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python
USER $NB_USER
```

Ref: [https://github.com/jupyter/docker-stacks/issues/440](https://github.com/jupyter/docker-stacks/issues/440)
Ref:
[https://github.com/jupyter/docker-stacks/issues/440](https://github.com/jupyter/docker-stacks/issues/440)

## Run JupyterLab

JupyterLab is preinstalled as a notebook extension starting in tag [c33a7dc0eece](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history).
JupyterLab is preinstalled as a notebook extension starting in tag
[c33a7dc0eece](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history).

Run jupyterlab using a command such as `docker run -it --rm -p 8888:8888 jupyter/datascience-notebook start.sh jupyter lab`
Run jupyterlab using a command such as
`docker run -it --rm -p 8888:8888 jupyter/datascience-notebook start.sh jupyter lab`

## Let's Encrypt a Notebook server

See the README for the simple automation here [https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy](https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy) which includes steps for requesting and renewing a Let's Encrypt certificate.
See the README for the simple automation here
[https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy](https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy)
which includes steps for requesting and renewing a Let's Encrypt certificate.

Ref: [https://github.com/jupyter/docker-stacks/issues/78](https://github.com/jupyter/docker-stacks/issues/78)
Ref:
[https://github.com/jupyter/docker-stacks/issues/78](https://github.com/jupyter/docker-stacks/issues/78)

## Slideshows with Jupyter and RISE

[RISE](https://github.com/damianavila/RISE) allows via extension to create live slideshows of your notebooks, with no conversion, adding javascript Reveal.js:
[RISE](https://github.com/damianavila/RISE) allows via extension to create live slideshows of your
notebooks, with no conversion, adding javascript Reveal.js:

```
# Add Live slideshows with RISE
RUN conda install -c damianavila82 rise
```
Credit: [Paolo D.](https://github.com/pdonorio) based on [docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43)

Credit: [Paolo D.](https://github.com/pdonorio) based on
[docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43)

## xgboost

You need to install conda's gcc for Python xgboost to work properly. Otherwise, you'll get an exception about libgomp.so.1 missing GOMP_4.0.
You need to install conda's gcc for Python xgboost to work properly. Otherwise, you'll get an
exception about libgomp.so.1 missing GOMP_4.0.

```
%%bash
Expand All @@ -110,20 +145,31 @@ import xgboost

Sometimes it is useful to run the Jupyter instance behind a nginx proxy, for instance:

- you would prefer to access the notebook at a server URL with a path (`https://example.com/jupyter`) rather than a port (`https://example.com:8888`)
- you may have many different services in addition to Jupyter running on the same server, and want to nginx to help improve server performance in manage the connections
- you would prefer to access the notebook at a server URL with a path
(`https://example.com/jupyter`) rather than a port (`https://example.com:8888`)
- you may have many different services in addition to Jupyter running on the same server, and want
to nginx to help improve server performance in manage the connections

Here is a [quick example NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2) to get started. You'll need a server, a `.crt` and `.key` file for your server, and `docker` & `docker-compose` installed. Then just download the files at that gist and run `docker-compose up -d` to test it out. Customize the `nginx.conf` file to set the desired paths and add other services.
Here is a [quick example NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2)
to get started. You'll need a server, a `.crt` and `.key` file for your server, and `docker` &
`docker-compose` installed. Then just download the files at that gist and run `docker-compose up -d`
to test it out. Customize the `nginx.conf` file to set the desired paths and add other services.

## Host volume mounts and notebook errors

If you are mounting a host directory as `/home/jovyan/work` in your container and you receive permission errors or connection errors when you create a notebook, be sure that the `jovyan` user (UID=1000 by default) has read/write access to the directory on the host. Alternatively, specify the UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the [Common Features, Docker Options section](../using/common.html#Docker-Options)
If you are mounting a host directory as `/home/jovyan/work` in your container and you receive
permission errors or connection errors when you create a notebook, be sure that the `jovyan` user
(UID=1000 by default) has read/write access to the directory on the host. Alternatively, specify the
UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the
[Common Features, Docker Options section](../using/common.html#Docker-Options)

Ref: [https://github.com/jupyter/docker-stacks/issues/199](https://github.com/jupyter/docker-stacks/issues/199)
Ref:
[https://github.com/jupyter/docker-stacks/issues/199](https://github.com/jupyter/docker-stacks/issues/199)

## Manpage installation

Most containers, including our Ubuntu base image, ship without manpages installed to save space. You can use the following dockerfile to inherit from one of our images to enable manpages:
Most containers, including our Ubuntu base image, ship without manpages installed to save space. You
can use the following dockerfile to inherit from one of our images to enable manpages:

```dockerfile
# Choose your desired base image
Expand All @@ -148,7 +194,10 @@ RUN echo "MANPATH_MAP ${CONDA_DIR}/bin ${CONDA_DIR}/man" >> /etc/manpath.config
USER $NB_UID
```

Adding the documentation on top of an existing singleuser image wastes a lot of space and requires reinstalling every system package, which can take additional time and bandwidth; the `datascience-notebook` image has been shown to grow by almost 3GB when adding manapages in this way. Enabling manpages in the base Ubuntu layer prevents this container bloat:
Adding the documentation on top of an existing singleuser image wastes a lot of space and requires
reinstalling every system package, which can take additional time and bandwidth; the
`datascience-notebook` image has been shown to grow by almost 3GB when adding manapages in this way.
Enabling manpages in the base Ubuntu layer prevents this container bloat:

```Dockerfile
# Ubuntu 18.04 (bionic) from 2018-05-26
Expand Down Expand Up @@ -178,18 +227,24 @@ We also have contributed recipes for using JupyterHub.

### Use JupyterHub's dockerspawner

In most cases for use with DockerSpawner, given any image that already has a notebook stack set up, you would only need to add:
In most cases for use with DockerSpawner, given any image that already has a notebook stack set up,
you would only need to add:

1. install the jupyterhub-singleuser script (for the right Python)
1. install the jupyterhub-singleuser script (for the right Python)
2. change the command to launch the single-user server

Swapping out the `FROM` line in the `jupyterhub/singleuser` Dockerfile should be enough for most cases.
Swapping out the `FROM` line in the `jupyterhub/singleuser` Dockerfile should be enough for most
cases.

Credit: [Justin Tyberg](https://github.com/jtyberg), [quanghoc](https://github.com/quanghoc), and [Min RK](https://github.com/minrk) based on [docker-stacks/issues/124](https://github.com/jupyter/docker-stacks/issues/124) and [docker-stacks/pull/185](https://github.com/jupyter/docker-stacks/pull/185)
Credit: [Justin Tyberg](https://github.com/jtyberg), [quanghoc](https://github.com/quanghoc), and
[Min RK](https://github.com/minrk) based on
[docker-stacks/issues/124](https://github.com/jupyter/docker-stacks/issues/124) and
[docker-stacks/pull/185](https://github.com/jupyter/docker-stacks/pull/185)

### Containers with a specific version of JupyterHub

To use a specific version of JupyterHub, the version of `jupyterhub` in your image should match the version in the Hub itself.
To use a specific version of JupyterHub, the version of `jupyterhub` in your image should match the
version in the Hub itself.

```
FROM jupyter/base-notebook:5ded1de07260
Expand All @@ -198,8 +253,8 @@ RUN pip install jupyterhub==0.8.0b1

Credit: [MinRK](https://github.com/jupyter/docker-stacks/issues/423#issuecomment-322767742)


Ref: [https://github.com/jupyter/docker-stacks/issues/177](https://github.com/jupyter/docker-stacks/issues/177)
Ref:
[https://github.com/jupyter/docker-stacks/issues/177](https://github.com/jupyter/docker-stacks/issues/177)

## Spark

Expand All @@ -215,7 +270,7 @@ import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages "org.apache.hadoop:hadoop-aws:2.7.3" pyspark-shell'

import pyspark
myAccessKey = input()
myAccessKey = input()
mySecretKey = input()

spark = pyspark.sql.SparkSession.builder \
Expand All @@ -240,7 +295,7 @@ from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

hadoopConf = sc._jsc.hadoopConfiguration()
myAccessKey = input()
myAccessKey = input()
mySecretKey = input()
hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
hadoopConf.set("fs.s3.awsAccessKeyId", myAccessKey)
Expand All @@ -249,7 +304,8 @@ hadoopConf.set("fs.s3.awsSecretAccessKey", mySecretKey)
df = sqlContext.read.parquet("s3://myBucket/myKey")
```

Ref: [https://github.com/jupyter/docker-stacks/issues/127](https://github.com/jupyter/docker-stacks/issues/127)
Ref:
[https://github.com/jupyter/docker-stacks/issues/127](https://github.com/jupyter/docker-stacks/issues/127)

### Using Local Spark JARs

Expand All @@ -267,13 +323,18 @@ directKafkaStream.pprint()
ssc.start()
```

Ref: [https://github.com/jupyter/docker-stacks/issues/154](https://github.com/jupyter/docker-stacks/issues/154)
Ref:
[https://github.com/jupyter/docker-stacks/issues/154](https://github.com/jupyter/docker-stacks/issues/154)

### Using spark-packages.org

If you'd like to use packages from [spark-packages.org](https://spark-packages.org/), see [https://gist.github.com/parente/c95fdaba5a9a066efaab](https://gist.github.com/parente/c95fdaba5a9a066efaab) for an example of how to specify the package identifier in the environment before creating a SparkContext.
If you'd like to use packages from [spark-packages.org](https://spark-packages.org/), see
[https://gist.github.com/parente/c95fdaba5a9a066efaab](https://gist.github.com/parente/c95fdaba5a9a066efaab)
for an example of how to specify the package identifier in the environment before creating a
SparkContext.

Ref: [https://github.com/jupyter/docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43)
Ref:
[https://github.com/jupyter/docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43)

### Use jupyter/all-spark-notebooks with an existing Spark/YARN cluster

Expand Down Expand Up @@ -339,17 +400,17 @@ RUN jupyter toree install --sys-prefix --spark_opts="--master yarn --deploy-mode
USER $NB_USER
```

Credit: [britishbadger](https://github.com/britishbadger) from [docker-stacks/issues/369](https://github.com/jupyter/docker-stacks/issues/369)
Credit: [britishbadger](https://github.com/britishbadger) from
[docker-stacks/issues/369](https://github.com/jupyter/docker-stacks/issues/369)

## Run Jupyter Notebook/Lab inside an already secured environment (i.e., with no token)

(Adapted from [issue 728](https://github.com/jupyter/docker-stacks/issues/728))

The default security is very good. There are use cases, encouraged by
containers, where the jupyter container and the system it runs within, lie
inside the security boundary. In these use cases it is convenient to launch the
server without a password or token. In this case, you should use the `start.sh`
script to launch the server with no token:
The default security is very good. There are use cases, encouraged by containers, where the jupyter
container and the system it runs within, lie inside the security boundary. In these use cases it is
convenient to launch the server without a password or token. In this case, you should use the
`start.sh` script to launch the server with no token:

For jupyterlab:

Expand All @@ -358,13 +419,15 @@ docker run jupyter/base-notebook:6d2a05346196 start.sh jupyter lab --LabApp.toke
```

For jupyter classic:

```
docker run jupyter/base-notebook:6d2a05346196 start.sh jupyter notebook --NotebookApp.token=''
```

## Enable nbextension spellchecker for markdown (or any other nbextension)

NB: this works for classic notebooks only

```
# Update with your base image of choice
FROM jupyter/minimal-notebook:latest
Expand All @@ -377,4 +440,5 @@ RUN pip install jupyter_contrib_nbextensions && \
jupyter nbextension enable spellchecker/main --user
```

Ref: [https://github.com/jupyter/docker-stacks/issues/675](https://github.com/jupyter/docker-stacks/issues/675)
Ref:
[https://github.com/jupyter/docker-stacks/issues/675](https://github.com/jupyter/docker-stacks/issues/675)

0 comments on commit 213760e

Please sign in to comment.