1.8.13

* #193: Support for JupyterHub Proxy
man-group · May 20, 2020 · 2ed34ab · 2ed34ab
1 parent d277dc7
commit 2ed34ab
Show file tree

Hide file tree

Showing 35 changed files with 507 additions and 71 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -66,7 +66,7 @@ python: &python
       CIRCLE_ARTIFACTS: /tmp/circleci-artifacts
       CIRCLE_TEST_REPORTS: /tmp/circleci-test-results
       CODECOV_TOKEN: b0d35139-0a75-427a-907b-2c78a762f8f0
-      VERSION: 1.8.12
+      VERSION: 1.8.13
       PANDOC_RELEASES_URL: https://github.com/jgm/pandoc/releases
     steps:
       - attach_workspace:

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,5 +1,8 @@
 ## Changelog
 
+### 1.8.13 (2020-5-20)
+ * [#193](https://github.com/man-group/dtale/issues/193): Support for JupyterHub Proxy
+
 ### 1.8.12 (2020-5-15)
  * [#196](https://github.com/man-group/dtale/issues/196): dataframes that have datatime indexes without a name
  * Added the ability to apply formats to all columns of same dtype

diff --git a/README.md b/README.md
@@ -37,6 +37,7 @@ D-Tale was the product of a SAS to Python conversion.  What was originally a per
 - [Getting Started](#getting-started)
   - [Python Terminal](#python-terminal)
   - [Jupyter Notebook](#jupyter-notebook)
+  - [Jupyterhub w/ Jupyter Server Proxy](#jupyterhub-w-jupyter-server-proxy)
   - [Jupyterhub w/ Kubernetes](https://github.com/man-group/dtale/blob/master/docs/JUPYTERHUB_KUBERNETES.md)
   - [Google Colab & Kaggle](#google-colab--kaggle)
   - [R with Reticulate](#r-with-reticulate)
@@ -147,6 +148,58 @@ One thing of note is that a lot of the modal popups you see in the standard brow
 |:------:|:------:|:------:|:------:|:------:|
 |![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/Column_menu.png)|![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/correlations_popup.png)|![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/describe_popup.png)|![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/histogram_popup.png)|![](https://raw.githubusercontent.com/aschonfeld/dtale-media/master/images/instances_popup.png)|
 
+### JupyterHub w/ Jupyter Server Proxy
+
+JupyterHub has an extension that allows to proxy port for user, [JupyterHub Server Proxy](https://github.com/jupyterhub/jupyter-server-proxy)
+
+To me it seems like this extension might be the best solution to getting D-Tale running within kubernetes.  Here's how to use it:
+
+```python
+import pandas as pd
+
+import dtale
+import dtale.app as dtale_app
+
+dtale_app.JUPYTER_SERVER_PROXY = True
+
+dtale.show(pd.DataFrame([1,2,3]))
+```
+
+Notice the command `dtale_app.JUPYTER_SERVER_PROXY = True` this will make sure that any D-Tale instance will be served with the jupyter server proxy application root prefix:
+
+`/user/{jupyter username}/proxy/{dtale instance port}/`
+
+One thing to note is that if you try to look at the `_main_url` of your D-Tale instance in your notebook it will not include the hostname or port:
+
+```python
+import pandas as pd
+
+import dtale
+import dtale.app as dtale_app
+
+dtale_app.JUPYTER_SERVER_PROXY = True
+
+d = dtale.show(pd.DataFrame([1,2,3]))
+d._main_url # /user/johndoe/proxy/40000/dtale/main/1
+```
+
+ This is because it's very hard to promgramatically figure out the host/port that your notebook is running on.  So if you want to look at `_main_url` please be sure to preface it with:
+
+ `http[s]://[jupyterhub host]:[jupyterhub port]`
+
+If for some reason jupyterhub changes their API so that the application root changes you can also override D-Tale's application root by using the `app_root` parameter to the `show()` function:
+
+```python
+import pandas as pd
+
+import dtale
+import dtale.app as dtale_app
+
+dtale.show(pd.DataFrame([1,2,3]), app_root='/user/johndoe/proxy/40000/`)
+```
+
+Using this parameter will only apply the application root to that specific instance so you would have to include it on every call to `show()`.
+
 ### JupyterHub w/ Kubernetes
 
 Please read this [post](https://github.com/man-group/dtale/blob/master/docs/JUPYTERHUB_KUBERNETES.md)
@@ -157,7 +210,7 @@ These are hosted notebook sites and thanks to the work of [flask_ngrok](https://
 
 **DISCLAIMER:** It is import that you set `USE_NGROK` to true when using D-Tale within these two services.  Here is an example:
 
-```
+```python
 import pandas as pd
 
 import dtale
@@ -998,6 +1051,7 @@ Contributors:
  * [Fernando Saravia Rajal](https://github.com/fersarr)
  * [Dominik Christ](https://github.com/DominikMChrist)
  * [Reza Moshksar](https://github.com/reza1615)
+ * [Bertrand Nouvel](https://github.com/bnouvelbmll)
  * [Chris Boddy](https://github.com/cboddy)
  * [Jason Holden](https://github.com/jasonkholden)
  * [Tom Taylor](https://github.com/TomTaylorLondon)

diff --git a/docker/dtale.env b/docker/dtale.env
@@ -1,2 +1,2 @@
-VERSION=1.8.12
+VERSION=1.8.13
 TZ=America/New_York
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -64,9 +64,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'1.8.12'
+version = u'1.8.13'
 # The full version, including alpha/beta/rc tags.
-release = u'1.8.12'
+release = u'1.8.13'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/dtale/app.py b/dtale/app.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import, print_function
 
+import getpass
 import os
 import random
 import socket
@@ -23,7 +24,8 @@
 from dtale import dtale
 from dtale.cli.clickutils import retrieve_meta_info_and_version, setup_logging
 from dtale.utils import (DuplicateDataError, build_shutdown_url, build_url,
-                         dict_merge, get_host, running_with_flask_debug)
+                         dict_merge, fix_url_path, get_host,
+                         is_app_root_defined, running_with_flask_debug)
 from dtale.views import DtaleData, head_data_id, is_up, kill, startup
 
 if PY3:
@@ -34,6 +36,7 @@
 logger = getLogger(__name__)
 
 USE_NGROK = False
+JUPYTER_SERVER_PROXY = False
 ACTIVE_HOST = None
 ACTIVE_PORT = None
 
@@ -69,9 +72,8 @@ def get(self, *args, **kwargs):
         :param args: Optional arguments to be passed to :meth:`flask:flask.FlaskClient.get`
         :param kwargs: Optional keyword arguments to be passed to :meth:`flask:flask.FlaskClient.get`
         """
-        return super(DtaleFlaskTesting, self).get(
-            base_url='http://{host}:{port}'.format(host=self.host, port=self.port), *args, **kwargs
-        )
+        self.application.config['SERVER_NAME'] = '{host}:{port}'.format(host=self.host, port=self.port)
+        return super(DtaleFlaskTesting, self).get(url_scheme='http', *args, **kwargs)
 
 
 class DtaleFlask(Flask):
@@ -86,7 +88,7 @@ class DtaleFlask(Flask):
     :param kwargs: Optional keyword arguments to be passed to :class:`flask:flask.Flask`
     """
 
-    def __init__(self, import_name, reaper_on=True, url=None, *args, **kwargs):
+    def __init__(self, import_name, reaper_on=True, url=None, app_root=None, *args, **kwargs):
         """
         Constructor method
         :param reaper_on: whether to run auto-reaper subprocess
@@ -97,8 +99,21 @@ def __init__(self, import_name, reaper_on=True, url=None, *args, **kwargs):
         self.base_url = url
         self.shutdown_url = build_shutdown_url(url)
         self.port = None
+        self.app_root = app_root
         super(DtaleFlask, self).__init__(import_name, *args, **kwargs)
 
+    def update_template_context(self, context):
+        super(DtaleFlask, self).update_template_context(context)
+        if self.app_root is not None:
+            context['url_for'] = self.url_for
+
+    def url_for(self, endpoint, *args, **kwargs):
+        if self.app_root is not None and endpoint == 'static':
+            if 'filename' in kwargs:
+                return fix_url_path('{}/{}'.format(self.app_root, kwargs["filename"]))
+            return fix_url_path('{}/{}'.format(self.app_root, args[0]))
+        return url_for(endpoint, *args, **kwargs)
+
     def run(self, *args, **kwargs):
         """
         :param args: Optional arguments to be passed to :meth:`flask:flask.run`
@@ -110,7 +125,7 @@ def run(self, *args, **kwargs):
         self.build_reaper()
         super(DtaleFlask, self).run(use_reloader=kwargs.get('debug', False), *args, **kwargs)
 
-    def test_client(self, reaper_on=False, port=None, *args, **kwargs):
+    def test_client(self, reaper_on=False, port=None, app_root=None, *args, **kwargs):
         """
         Overriding Flask's implementation of test_client so we can specify ports for testing and
         whether auto-reaper should be running
@@ -125,6 +140,10 @@ def test_client(self, reaper_on=False, port=None, *args, **kwargs):
         :rtype: :class:`dtale.app.DtaleFlaskTesting`
         """
         self.reaper_on = reaper_on
+        self.app_root = app_root
+        if app_root is not None:
+            self.config['APPLICATION_ROOT'] = app_root
+            self.jinja_env.globals['url_for'] = self.url_for
         self.test_client_class = DtaleFlaskTesting
         return super(DtaleFlask, self).test_client(*args, **dict_merge(kwargs, dict(port=port)))
 
@@ -172,21 +191,28 @@ def get_send_file_max_age(self, name):
         return super(DtaleFlask, self).get_send_file_max_age(name)
 
 
-def build_app(url, host=None, reaper_on=True, hide_shutdown=False, github_fork=False):
+def build_app(url, host=None, reaper_on=True, hide_shutdown=False, github_fork=False, app_root=None):
     """
     Builds :class:`flask:flask.Flask` application encapsulating endpoints for D-Tale's front-end
 
     :return: :class:`flask:flask.Flask` application
     :rtype: :class:`dtale.app.DtaleFlask`
     """
 
-    app = DtaleFlask('dtale', reaper_on=reaper_on, static_url_path='', url=url, instance_relative_config=False)
+    app = DtaleFlask('dtale', reaper_on=reaper_on, static_url_path='', url=url, instance_relative_config=False,
+                     app_root=app_root)
     app.config['SECRET_KEY'] = 'Dtale'
     app.config['HIDE_SHUTDOWN'] = hide_shutdown
     app.config['GITHUB_FORK'] = github_fork
 
     app.jinja_env.trim_blocks = True
     app.jinja_env.lstrip_blocks = True
+
+    if app_root is not None:
+        app.config['APPLICATION_ROOT'] = app_root
+        app.jinja_env.globals['url_for'] = app.url_for
+    app.jinja_env.globals['is_app_root_defined'] = is_app_root_defined
+
     app.register_blueprint(dtale)
 
     compress = Compress()
@@ -209,7 +235,7 @@ def favicon():
 
         :return: image/png
         """
-        return redirect(url_for('static', filename='images/favicon.ico'))
+        return redirect(app.url_for('static', filename='images/favicon.ico'))
 
     @app.route('/missing-js')
     def missing_js():
@@ -299,7 +325,7 @@ def has_no_empty_params(rule):
             # Filter out rules we can't navigate to in a browser
             # and rules that require parameters
             if "GET" in rule.methods and has_no_empty_params(rule):
-                url = url_for(rule.endpoint, **(rule.defaults or {}))
+                url = app.url_for(rule.endpoint, **(rule.defaults or {}))
                 links.append((url, rule.endpoint))
         return jsonify(links)
 
@@ -404,9 +430,23 @@ def is_port_in_use(port):
     return base
 
 
+def build_startup_url_and_app_root(app_root=None):
+    url = build_url(ACTIVE_PORT, ACTIVE_HOST)
+    final_app_root = app_root
+    if final_app_root is None and JUPYTER_SERVER_PROXY:
+        final_app_root = '/user/{}/proxy/'.format(getpass.getuser())
+    if final_app_root is not None:
+        if JUPYTER_SERVER_PROXY:
+            final_app_root = fix_url_path('{}/{}'.format(final_app_root, ACTIVE_PORT))
+            return final_app_root, final_app_root
+        else:
+            return fix_url_path('{}/{}'.format(url, final_app_root)), final_app_root
+    return url, final_app_root
+
+
 def show(data=None, host=None, port=None, name=None, debug=False, subprocess=True, data_loader=None,
          reaper_on=True, open_browser=False, notebook=False, force=False, context_vars=None, ignore_duplicate=False,
-         **kwargs):
+         app_root=None, **kwargs):
     """
     Entry point for kicking off D-Tale :class:`flask:flask.Flask` process from python process
 
@@ -452,7 +492,7 @@ def show(data=None, host=None, port=None, name=None, debug=False, subprocess=Tru
 
         ..link displayed in logging can be copied and pasted into any browser
     """
-    global ACTIVE_HOST, ACTIVE_PORT, USE_NGROK
+    global ACTIVE_HOST, ACTIVE_PORT, USE_NGROK, JUPYTER_SERVER_PROXY
 
     try:
         logfile, log_level, verbose = map(kwargs.get, ['logfile', 'log_level', 'verbose'])
@@ -469,10 +509,11 @@ def show(data=None, host=None, port=None, name=None, debug=False, subprocess=Tru
         else:
             initialize_process_props(host, port, force)
 
-        url = build_url(ACTIVE_PORT, ACTIVE_HOST)
-        instance = startup(url, data=data, data_loader=data_loader, name=name, context_vars=context_vars,
+        app_url = build_url(ACTIVE_PORT, ACTIVE_HOST)
+        startup_url, final_app_root = build_startup_url_and_app_root(app_root)
+        instance = startup(startup_url, data=data, data_loader=data_loader, name=name, context_vars=context_vars,
                            ignore_duplicate=ignore_duplicate)
-        is_active = not running_with_flask_debug() and is_up(url)
+        is_active = not running_with_flask_debug() and is_up(app_url)
         if is_active:
             def _start():
                 if open_browser:
@@ -484,7 +525,8 @@ def _start():
                 thread.start()
 
             def _start():
-                app = build_app(url, reaper_on=reaper_on, host=ACTIVE_HOST)
+                app = build_app(app_url, reaper_on=reaper_on, host=ACTIVE_HOST,
+                                app_root=final_app_root)
                 if debug and not USE_NGROK:
                     app.jinja_env.auto_reload = True
                     app.config['TEMPLATES_AUTO_RELOAD'] = True
@@ -513,7 +555,7 @@ def _start():
             if notebook:
                 instance.notebook()
         else:
-            logger.info('D-Tale started at: {}'.format(url))
+            logger.info('D-Tale started at: {}'.format(app_url))
             _start()
 
         return instance
@@ -554,7 +596,8 @@ def get_instance(data_id):
     """
     data_id_str = str(data_id)
     if global_state.get_data(data_id_str) is not None:
-        return DtaleData(data_id_str, build_url(ACTIVE_PORT, ACTIVE_HOST))
+        startup_url, _ = build_startup_url_and_app_root()
+        return DtaleData(data_id_str, startup_url)
     return None