From 0b2ced306b222973eab4976505fe048bbbf5ceee Mon Sep 17 00:00:00 2001 From: Peter Parente Date: Sat, 9 Nov 2019 16:06:08 -0500 Subject: [PATCH] Add sudo recipe --- docs/using/recipes.md | 146 ++++++++++++++++++++++++++++++------------ 1 file changed, 105 insertions(+), 41 deletions(-) diff --git a/docs/using/recipes.md b/docs/using/recipes.md index 3bb07b158f..37fb0d475e 100644 --- a/docs/using/recipes.md +++ b/docs/using/recipes.md @@ -1,10 +1,32 @@ # Contributed Recipes -Users sometimes share interesting ways of using the Jupyter Docker Stacks. We encourage users to [contribute these recipes](../contributing/recipes.md) to the documentation in case they prove useful to other members of the community by submitting a pull request to `docs/using/recipes.md`. The sections below capture this knowledge. +Users sometimes share interesting ways of using the Jupyter Docker Stacks. We encourage users to +[contribute these recipes](../contributing/recipes.md) to the documentation in case they prove +useful to other members of the community by submitting a pull request to `docs/using/recipes.md`. +The sections below capture this knowledge. + +## Using `sudo` within a container + +Password authentication is disabled for the `NB_USER` (e.g., `jovyan`). This choice was made to +avoid distributing images with a weak default password that users ~might~ will forget to change +before running a container on a publicly accessible host. + +You can grant the within-container `NB_USER` passwordless `sudo` access by adding +`-e GRANT_SUDO=yes` and `--user root` to your Docker command line or appropriate container +orchestrator config. + +For example: + +``` +docker run -it -e GRANT_SUDO=yes --user root jupyter/minimal-notebook +``` + +**You should only enable `sudo` if you trust the user and/or if the container is running on an +isolated host.** ## Using `pip install` or `conda install` in a Child Docker image -Create a new Dockerfile like the one shown below. +Create a new Dockerfile like the one shown below. ```dockerfile # Start from a core stack version @@ -19,8 +41,8 @@ Then build a new image. docker build --rm -t jupyter/my-datascience-notebook . ``` -To use a requirements.txt file, first create your `requirements.txt` file -with the listing of packages desired. Next, create a new Dockerfile like the one shown below. +To use a requirements.txt file, first create your `requirements.txt` file with the listing of +packages desired. Next, create a new Dockerfile like the one shown below. ```dockerfile # Start from a core stack version @@ -44,11 +66,14 @@ RUN conda install --yes --file /tmp/requirements.txt && \ fix-permissions /home/$NB_USER ``` -Ref: [docker-stacks/commit/79169618d571506304934a7b29039085e77db78c](https://github.com/jupyter/docker-stacks/commit/79169618d571506304934a7b29039085e77db78c#commitcomment-15960081) +Ref: +[docker-stacks/commit/79169618d571506304934a7b29039085e77db78c](https://github.com/jupyter/docker-stacks/commit/79169618d571506304934a7b29039085e77db78c#commitcomment-15960081) ## Add a Python 2.x environment -Python 2.x was removed from all images on August 10th, 2017, starting in tag `cc9feab481f7`. You can add a Python 2.x environment by defining your own Dockerfile inheriting from one of the images like so: +Python 2.x was removed from all images on August 10th, 2017, starting in tag `cc9feab481f7`. You can +add a Python 2.x environment by defining your own Dockerfile inheriting from one of the images like +so: ``` # Choose your desired base image @@ -70,33 +95,43 @@ $CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python USER $NB_USER ``` -Ref: [https://github.com/jupyter/docker-stacks/issues/440](https://github.com/jupyter/docker-stacks/issues/440) +Ref: +[https://github.com/jupyter/docker-stacks/issues/440](https://github.com/jupyter/docker-stacks/issues/440) ## Run JupyterLab -JupyterLab is preinstalled as a notebook extension starting in tag [c33a7dc0eece](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history). +JupyterLab is preinstalled as a notebook extension starting in tag +[c33a7dc0eece](https://github.com/jupyter/docker-stacks/wiki/Docker-build-history). -Run jupyterlab using a command such as `docker run -it --rm -p 8888:8888 jupyter/datascience-notebook start.sh jupyter lab` +Run jupyterlab using a command such as +`docker run -it --rm -p 8888:8888 jupyter/datascience-notebook start.sh jupyter lab` ## Let's Encrypt a Notebook server -See the README for the simple automation here [https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy](https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy) which includes steps for requesting and renewing a Let's Encrypt certificate. +See the README for the simple automation here +[https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy](https://github.com/jupyter/docker-stacks/tree/master/examples/make-deploy) +which includes steps for requesting and renewing a Let's Encrypt certificate. -Ref: [https://github.com/jupyter/docker-stacks/issues/78](https://github.com/jupyter/docker-stacks/issues/78) +Ref: +[https://github.com/jupyter/docker-stacks/issues/78](https://github.com/jupyter/docker-stacks/issues/78) ## Slideshows with Jupyter and RISE -[RISE](https://github.com/damianavila/RISE) allows via extension to create live slideshows of your notebooks, with no conversion, adding javascript Reveal.js: +[RISE](https://github.com/damianavila/RISE) allows via extension to create live slideshows of your +notebooks, with no conversion, adding javascript Reveal.js: ``` # Add Live slideshows with RISE RUN conda install -c damianavila82 rise ``` -Credit: [Paolo D.](https://github.com/pdonorio) based on [docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43) + +Credit: [Paolo D.](https://github.com/pdonorio) based on +[docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43) ## xgboost -You need to install conda's gcc for Python xgboost to work properly. Otherwise, you'll get an exception about libgomp.so.1 missing GOMP_4.0. +You need to install conda's gcc for Python xgboost to work properly. Otherwise, you'll get an +exception about libgomp.so.1 missing GOMP_4.0. ``` %%bash @@ -110,20 +145,31 @@ import xgboost Sometimes it is useful to run the Jupyter instance behind a nginx proxy, for instance: -- you would prefer to access the notebook at a server URL with a path (`https://example.com/jupyter`) rather than a port (`https://example.com:8888`) -- you may have many different services in addition to Jupyter running on the same server, and want to nginx to help improve server performance in manage the connections +- you would prefer to access the notebook at a server URL with a path + (`https://example.com/jupyter`) rather than a port (`https://example.com:8888`) +- you may have many different services in addition to Jupyter running on the same server, and want + to nginx to help improve server performance in manage the connections -Here is a [quick example NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2) to get started. You'll need a server, a `.crt` and `.key` file for your server, and `docker` & `docker-compose` installed. Then just download the files at that gist and run `docker-compose up -d` to test it out. Customize the `nginx.conf` file to set the desired paths and add other services. +Here is a [quick example NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2) +to get started. You'll need a server, a `.crt` and `.key` file for your server, and `docker` & +`docker-compose` installed. Then just download the files at that gist and run `docker-compose up -d` +to test it out. Customize the `nginx.conf` file to set the desired paths and add other services. ## Host volume mounts and notebook errors -If you are mounting a host directory as `/home/jovyan/work` in your container and you receive permission errors or connection errors when you create a notebook, be sure that the `jovyan` user (UID=1000 by default) has read/write access to the directory on the host. Alternatively, specify the UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the [Common Features, Docker Options section](../using/common.html#Docker-Options) +If you are mounting a host directory as `/home/jovyan/work` in your container and you receive +permission errors or connection errors when you create a notebook, be sure that the `jovyan` user +(UID=1000 by default) has read/write access to the directory on the host. Alternatively, specify the +UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the +[Common Features, Docker Options section](../using/common.html#Docker-Options) -Ref: [https://github.com/jupyter/docker-stacks/issues/199](https://github.com/jupyter/docker-stacks/issues/199) +Ref: +[https://github.com/jupyter/docker-stacks/issues/199](https://github.com/jupyter/docker-stacks/issues/199) ## Manpage installation -Most containers, including our Ubuntu base image, ship without manpages installed to save space. You can use the following dockerfile to inherit from one of our images to enable manpages: +Most containers, including our Ubuntu base image, ship without manpages installed to save space. You +can use the following dockerfile to inherit from one of our images to enable manpages: ```dockerfile # Choose your desired base image @@ -148,7 +194,10 @@ RUN echo "MANPATH_MAP ${CONDA_DIR}/bin ${CONDA_DIR}/man" >> /etc/manpath.config USER $NB_UID ``` -Adding the documentation on top of an existing singleuser image wastes a lot of space and requires reinstalling every system package, which can take additional time and bandwidth; the `datascience-notebook` image has been shown to grow by almost 3GB when adding manapages in this way. Enabling manpages in the base Ubuntu layer prevents this container bloat: +Adding the documentation on top of an existing singleuser image wastes a lot of space and requires +reinstalling every system package, which can take additional time and bandwidth; the +`datascience-notebook` image has been shown to grow by almost 3GB when adding manapages in this way. +Enabling manpages in the base Ubuntu layer prevents this container bloat: ```Dockerfile # Ubuntu 18.04 (bionic) from 2018-05-26 @@ -178,18 +227,24 @@ We also have contributed recipes for using JupyterHub. ### Use JupyterHub's dockerspawner -In most cases for use with DockerSpawner, given any image that already has a notebook stack set up, you would only need to add: +In most cases for use with DockerSpawner, given any image that already has a notebook stack set up, +you would only need to add: -1. install the jupyterhub-singleuser script (for the right Python) +1. install the jupyterhub-singleuser script (for the right Python) 2. change the command to launch the single-user server -Swapping out the `FROM` line in the `jupyterhub/singleuser` Dockerfile should be enough for most cases. +Swapping out the `FROM` line in the `jupyterhub/singleuser` Dockerfile should be enough for most +cases. -Credit: [Justin Tyberg](https://github.com/jtyberg), [quanghoc](https://github.com/quanghoc), and [Min RK](https://github.com/minrk) based on [docker-stacks/issues/124](https://github.com/jupyter/docker-stacks/issues/124) and [docker-stacks/pull/185](https://github.com/jupyter/docker-stacks/pull/185) +Credit: [Justin Tyberg](https://github.com/jtyberg), [quanghoc](https://github.com/quanghoc), and +[Min RK](https://github.com/minrk) based on +[docker-stacks/issues/124](https://github.com/jupyter/docker-stacks/issues/124) and +[docker-stacks/pull/185](https://github.com/jupyter/docker-stacks/pull/185) ### Containers with a specific version of JupyterHub -To use a specific version of JupyterHub, the version of `jupyterhub` in your image should match the version in the Hub itself. +To use a specific version of JupyterHub, the version of `jupyterhub` in your image should match the +version in the Hub itself. ``` FROM jupyter/base-notebook:5ded1de07260 @@ -198,8 +253,8 @@ RUN pip install jupyterhub==0.8.0b1 Credit: [MinRK](https://github.com/jupyter/docker-stacks/issues/423#issuecomment-322767742) - -Ref: [https://github.com/jupyter/docker-stacks/issues/177](https://github.com/jupyter/docker-stacks/issues/177) +Ref: +[https://github.com/jupyter/docker-stacks/issues/177](https://github.com/jupyter/docker-stacks/issues/177) ## Spark @@ -215,7 +270,7 @@ import os os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages "org.apache.hadoop:hadoop-aws:2.7.3" pyspark-shell' import pyspark -myAccessKey = input() +myAccessKey = input() mySecretKey = input() spark = pyspark.sql.SparkSession.builder \ @@ -240,7 +295,7 @@ from pyspark.sql import SQLContext sqlContext = SQLContext(sc) hadoopConf = sc._jsc.hadoopConfiguration() -myAccessKey = input() +myAccessKey = input() mySecretKey = input() hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") hadoopConf.set("fs.s3.awsAccessKeyId", myAccessKey) @@ -249,7 +304,8 @@ hadoopConf.set("fs.s3.awsSecretAccessKey", mySecretKey) df = sqlContext.read.parquet("s3://myBucket/myKey") ``` -Ref: [https://github.com/jupyter/docker-stacks/issues/127](https://github.com/jupyter/docker-stacks/issues/127) +Ref: +[https://github.com/jupyter/docker-stacks/issues/127](https://github.com/jupyter/docker-stacks/issues/127) ### Using Local Spark JARs @@ -267,13 +323,18 @@ directKafkaStream.pprint() ssc.start() ``` -Ref: [https://github.com/jupyter/docker-stacks/issues/154](https://github.com/jupyter/docker-stacks/issues/154) +Ref: +[https://github.com/jupyter/docker-stacks/issues/154](https://github.com/jupyter/docker-stacks/issues/154) ### Using spark-packages.org -If you'd like to use packages from [spark-packages.org](https://spark-packages.org/), see [https://gist.github.com/parente/c95fdaba5a9a066efaab](https://gist.github.com/parente/c95fdaba5a9a066efaab) for an example of how to specify the package identifier in the environment before creating a SparkContext. +If you'd like to use packages from [spark-packages.org](https://spark-packages.org/), see +[https://gist.github.com/parente/c95fdaba5a9a066efaab](https://gist.github.com/parente/c95fdaba5a9a066efaab) +for an example of how to specify the package identifier in the environment before creating a +SparkContext. -Ref: [https://github.com/jupyter/docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43) +Ref: +[https://github.com/jupyter/docker-stacks/issues/43](https://github.com/jupyter/docker-stacks/issues/43) ### Use jupyter/all-spark-notebooks with an existing Spark/YARN cluster @@ -339,17 +400,17 @@ RUN jupyter toree install --sys-prefix --spark_opts="--master yarn --deploy-mode USER $NB_USER ``` -Credit: [britishbadger](https://github.com/britishbadger) from [docker-stacks/issues/369](https://github.com/jupyter/docker-stacks/issues/369) +Credit: [britishbadger](https://github.com/britishbadger) from +[docker-stacks/issues/369](https://github.com/jupyter/docker-stacks/issues/369) ## Run Jupyter Notebook/Lab inside an already secured environment (i.e., with no token) (Adapted from [issue 728](https://github.com/jupyter/docker-stacks/issues/728)) -The default security is very good. There are use cases, encouraged by -containers, where the jupyter container and the system it runs within, lie -inside the security boundary. In these use cases it is convenient to launch the -server without a password or token. In this case, you should use the `start.sh` -script to launch the server with no token: +The default security is very good. There are use cases, encouraged by containers, where the jupyter +container and the system it runs within, lie inside the security boundary. In these use cases it is +convenient to launch the server without a password or token. In this case, you should use the +`start.sh` script to launch the server with no token: For jupyterlab: @@ -358,6 +419,7 @@ docker run jupyter/base-notebook:6d2a05346196 start.sh jupyter lab --LabApp.toke ``` For jupyter classic: + ``` docker run jupyter/base-notebook:6d2a05346196 start.sh jupyter notebook --NotebookApp.token='' ``` @@ -365,6 +427,7 @@ docker run jupyter/base-notebook:6d2a05346196 start.sh jupyter notebook --Notebo ## Enable nbextension spellchecker for markdown (or any other nbextension) NB: this works for classic notebooks only + ``` # Update with your base image of choice FROM jupyter/minimal-notebook:latest @@ -377,4 +440,5 @@ RUN pip install jupyter_contrib_nbextensions && \ jupyter nbextension enable spellchecker/main --user ``` -Ref: [https://github.com/jupyter/docker-stacks/issues/675](https://github.com/jupyter/docker-stacks/issues/675) +Ref: +[https://github.com/jupyter/docker-stacks/issues/675](https://github.com/jupyter/docker-stacks/issues/675)