Merge pull request #112 from rkdarst/dev

Integration branch for next release

.travis.yml

@@ -2,33 +2,32 @@
 language: python
 sudo: false
 python:
+    - 3.7-dev
     - 3.6
     - 3.5
     - 3.4
 env:
     - JHUB_VER=0.7.2
     - JHUB_VER=0.8.1
-    - JHUB_VER=0.9.0b2
+    - JHUB_VER=0.9.1
 matrix:
     include:
      - python: 3.7-dev
-       env: JHUB_VER=0.9.0b2
-     - python: 3.6
        env: JHUB_VER=master
     exclude:
      - python: 3.4
-       env: JHUB_VER=0.9.0b2
+       env: JHUB_VER=0.9.1
     allow_failures:
       - env: JHUB_VER=master
-      - python: 3.7-dev
 
 before_install:
     - npm install -g configurable-http-proxy
     - git clone --quiet --depth 1 https://github.com/minrk/travis-wheels travis-wheels
     - git clone --quiet --branch $JHUB_VER https://github.com/jupyter/jupyterhub.git jupyterhub
 install:
 # Don't let requirements pull in tornado 5 yet except for jupyterhub master
-    - if [ $JHUB_VER != "master" -a $JHUB_VER != "0.9.0b2" ]; then pip install "tornado<5.0"; fi
+    - if [ $JHUB_VER != "master" -a $JHUB_VER != "0.9.1" ]; then pip install "tornado<5.0"; fi
+    - pip install jsonschema!=3.0.0a1   # issue #110, remove later
     - pip install --pre -f travis-wheels/wheelhouse -r jupyterhub/dev-requirements.txt
     - pip install --pre -e jupyterhub
 

README.md

@@ -24,6 +24,8 @@ This package formerly included WrapSpawner and ProfilesSpawner, which provide me
 
 ## Batch Spawners
 
+For information on the specific spawners, see [SPAWNERS.md](SPAWNERS.md).
+
 ### Overview
 
 This file contains an abstraction layer for batch job queueing systems (`BatchSpawnerBase`), and implements
@@ -82,6 +84,19 @@ to run Jupyter notebooks on an academic supercomputer cluster.
    c.TorqueSpawner.state_exechost_exp = r'int-\1.mesabi.xyz.edu'
    ```
 
+### Security
+
+Unless otherwise stated for a specific spawner, assume that spawners
+*do* evaluate shell environment for users and thus the [security
+requiremnts of JupyterHub security for untrusted
+users](https://jupyterhub.readthedocs.io/en/stable/reference/websecurity.html)
+are not fulfilled because some (most?) spawners *do* start a user
+shell which will execute arbitrary user environment configuration
+(``.profile``, ``.bashrc`` and the like) unless users do not have
+access to their own cluster user account.  This is something which we
+are working on.
+
+
 ## Provide different configurations of BatchSpawner
 
 ### Overview
@@ -141,11 +156,33 @@ clusters, as well as an option to run a local notebook directly on the jupyterhu
 
 ### dev (requires minimum JupyterHub 0.7.2 and Python 3.4)
 
+Added (user)
+
 * Add Jinja2 templating as an option for all scripts and commands.  If '{{' or `{%` is used anywhere in the string, it is used as a jinja2 template.
-* Update Slurm batch script.  Now, the single-user notebook is run in a job step, with a wrapper of `srun`.  This may need to be removed if you don't want environment variables limited.
 * Add new option exec_prefix, which defaults to `sudo -E -u {username}`.  This replaces explicit `sudo` in every batch command - changes in local commands may be needed.
+* New option: `req_keepvars_extra`, which allows keeping extra variables in addition to what is defined by JupyterHub itself (addition of variables to keep instead of replacement).  #99
+* Add `req_prologue` and `req_epilogue` options to scripts which are inserted before/after the main jupyterhub-singleuser command, which allow for generic setup/cleanup without overriding the entire script.  #96
+* SlurmSpawner: add the `req_reservation` option.  #
+
+Added (developer)
+
 * Add many more tests.
+* Add a new page `SPAWNERS.md` which information on specific spawners.  Begin trying to collect a list of spawner-specific contacts.  #97
+
+Changed
+
 * Update minimum requirements to JupyterHub 0.8.1 and Python 3.4.
+* Update Slurm batch script.  Now, the single-user notebook is run in a job step, with a wrapper of `srun`.  This may need to be removed using `req_srun=''` if you don't want environment variables limited.
+* Pass the environment dictionary to the queue and cancel commands as well.  This is mostly user environment, but may be useful to these commands as well in some cases. #108, #111  If these envioronment variables were used for authentication as an admin, be aware that there are pre-existing security issues because they may be passed to the user via the batch submit command, see #82.
+
+Fixed
+
+* Improve debugging on failed submission by raising errors including error messages from the commands.  #106
+* Many other non-user or developer visible changes.  #107 #106 #100
+* In Travis CI, blacklist jsonschema=3.0.0a1 because it breaks tests
+
+Removed
+
 
 ### v0.8.1 (bugfix release)
 

SPAWNERS.md

@@ -0,0 +1,60 @@
+# Notes on specific spawners
+
+## `TorqueSpawner`
+
+Maintainers:
+
+
+## `MoabSpawner`
+
+Subclass of TorqueSpawner
+
+Maintainers:
+
+
+## `SlurmSpawner`
+
+Maintainers: @rkdarst
+
+This spawner enforces the environment if `srun` is used to wrap the
+spawner command, which is the default.  If you *do* want user
+environment to be used, set `req_srun=''`.  However, this is not
+perfect: there is still a bash shell begun as the user which could run
+arbitrary startup, define shell aliases for `srun`, etc.
+
+Use of `srun` is required to gracefully terminate.
+
+
+## `GridengineSpawner`
+
+Maintainers:
+
+
+## `CondorSpawner`
+
+Maintainers:
+
+
+## `LsfSpawner`
+
+Maintainers:
+
+
+# Checklist for making spawners
+
+Please document each of these things under the spawner list above, -
+even if it is "OK", we need to track status of all spawners.  If it is
+a bug, users really need to know.
+
+- Does your spawner read shell environment before starting?  (See
+  [Jupyterhub
+  Security](https://jupyterhub.readthedocs.io/en/stable/reference/websecurity.html).
+
+- Does your spawner send SIGTERM to the jupyterhub-singleuser process
+  before SIGKILL?  It should, so that the process can terminate
+  gracefully.  Add `echo "terminated gracefully"` to the end of the
+  batch script - if you see this in your singleuser server output, you
+  know that you DO receive SIGTERM and terminate gracefully.  If your
+  batch system can not automatically send SIGTERM before SIGKILL, PR
+  #75 might help here, ask for it to be finished.
+