[MRG] Configurable server extension applications

[Zsailer]

This is an initial implementation of a "Jupyter server extension application" class. Pinging @SylvainCorlay @takluyver @kevin-bates @lresende @rolweber @maartenbreddels for review.

What?

This PR introduces the ExtensionApp to the Jupyter server, a base class for creating configurable, front-end client applications as extensions of the jupyter server.

Details

ExtensionApp subclasses JupyterApp from jupyter_core. Extensions that subclass this class can be:

• launched and configured from the command line with jupyter <extension_name> ...
• configured by a file named jupyter_<extension_name>_config.py.

When an extension application is launched, it

1. starts and launches an instance of the Jupyter server.
2. calls self.load_jupyter_server_extension() method.
3. loads configuration from config files+command line.
4. appends the extension's handlers to the main web application.
5. appends a /static/<extension_name>/ endpoint where it serves static files (i.e. js, css, etc.).

Why?

There are various frontends that depend on the classic Notebook Server (i.e. notebook, nteract, lab, and voila). However, the Notebook frontend is deeply coupled with the Notebook server, forcing other clients to work around it. Jupyter Server separates out the server from the notebook, so all front-end applications are first-class citizens to the server.

This PR makes it much easier to write client applications that depend on the Jupyter Server.

How?

1. Subclass the ExtensionApp. Set the following traits:

   • extension_name: name of the extension.
   • static_paths: path to static files directory
   • template_paths: path to templates directory.

   and define (optional) the following methods:

   • initialize_handlers(): append handlers to self.handlers.
   • initialize_templates(): setup your html templating options.
   • initialize_settings(): add extensions settings to the webapp using self.settings.

   Example

   # application.py
   from jupyter_server.extension.application import ExtensionApp
   
   class MyExtension(ExtensionApp):
       
       # Name of the extension
       extension_name = "my_extension"
   
       # Local path to static files directory.
       static_paths = [
           "/path/to/static/dir/"
       ]
   
       # Local path to templates directory.
       template_paths = [
           "/path/to/template/dir/"
       ]
   
       def initialize_handlers(self):
           self.handlers.append(
               (r'/myextension', MyExtensionHandler)
           )
   
       def initialize_templates(self):
           ...
   
       def initialize_settings(self):
           ...

2. Define handlers by subclassing the ExtensionHandler.

   Example

   # handler.py
   from jupyter_server.extension.handler import ExtensionHandler
   
   
   class MyExtensionHandler(ExtensionHandler):
       
       def get(self):
           ...

3. Point Jupyter server to the extension. We need to define two things:

   • _jupyter_server_extension_paths(): a function defining what module to load the extension from.
   • load_jupyter_server_extension(): point there server to the extension's loading function mechansim.

   Example

   # __init__.py
   from .extension import MyExtension
   
   
   EXTENSION_NAME = "my_extension"
   
   def _jupyter_server_extension_paths():
       return [{"module": EXTENSION_NAME}]
   
   load_jupyter_server_extension = MyExtension.load_jupyter_server_extension

4. Add the following configuration to your jupyter configuration directory to enable the extension.

   {
     "ServerApp": {
       "jpserver_extensions": {
         "notebook": true
       }
     }
   }

5. Add entry point in setup.py.

   setup(
       name='my_extension',
       version='0.1',
       data_files=[
           ('etc/jupyter/jupyter_server_config.d', ['etc/jupyter/jupyter_server_config.d./my_extension.json']),
       ],
       ...
       'entry_points': {
           'console_scripts': [
                'jupyter-myext = my_extension.application:MyExtension.launch_instance'
           ]
       },
       ...
   )

[Zsailer]

@rgbkrk and @mpacer might also be interested in this PR.

I'm currently working on making the nteract_on_jupyter application work with Jupyter server using this new application class.

[blink1073]

This looks great! I pushed a bit of docs cleanup.

[rolweber]

So you want to stick with the term "extension"? Fine with me. I guess calling it "server app" would be somewhat misleading, because the extension point (plug-in point?) is meant for front-ends.

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/rename-this-category-to-jupyter-server/1130/6

[Zsailer]

@rolweber ServerApp is already taken by the jupyter server application itself (replacing the NotebookApp in the classic notebook).

I definitely designed this class with front-end applications in mind, but it's valuable to any extension that might want to use traitlet's configurability APIs.

In my head, the ServerApp is the core object with a defined set of endpoints+REST API (kernels, kernelspec, sessions, contents, etc.), and other endpoints added by notebook, lab, gateway, etc. "extend" the core server. That's my original reason for naming it "ExtensionApp". Though perhaps extension is a loaded term... Do you have something else in mind?

[blink1073]

Maybe call the base class BaseServerApp and this one ServerApp?

[Zsailer]

@blink1073 Using Base prefix might be a little misleading too, because ExtensionApp class doesn't actually inherit the ServerApp. I usually think of the Base prefix as a signal for inheritance?

Instead, the ExtensionApp is a separate object that creates an instance of the Server on the fly and appends itself after server initialization (following our current server extension model). That way, multiple extensions can be loaded simultaneously.

[SylvainCorlay]

+1 on ExtensionApp or ServerExtensionApp.

[blink1073]

I think the issue is that the term extension carries a lot of history and baggage. But it really does extend the server by adding handlers, so I'm torn. ServerEntryPointApp 😄?

[kevin-bates]

Great discussion. Names are hard, but important.

Question. If you were to run ExtensionApp directly, do you get the same functionality as running ServerApp directly? (I expect so.)

Since ExtensionApp will always serve as a base class for configurable applications to "extend" ServerApp, I think calling ExtensionApp BaseExtensionApp or ExtensionAppBase is reasonable. It would be great to have 'server' in the name though. What about ServerAppExtensionBase?

[rolweber]

Since this class is meant to be subclassed, but isn't used as a base class inside the package, I like Base better as a suffix than a prefix. Do we want "App" in the name? Probably yes, because it indicates something that can be started, in analogy to "NotebookApp".
Do we want/need "Server" in the name? The class will be imported from jupyter_server, is that enough?

How about avoiding "extension" as a noun?

• ExtendedAppBase
• ExtendedServerAppBase

[blink1073]

👍 for ExtendedServerAppBase, verbose, but explicit.

[kevin-bates]

FWIW, I don't really care for the used of Extended here. To me that describes the result and not necessarily what this class is. IMHO, this class is a base class for ServerApp extensions - thus ServerAppExtensionBase. Yes, verbose (I was even going to mention that in my original suggestion but didn't want to sway opinion), but I think it more clearly describes the intention of this base class.

I feel like when one reads ExtendedServerAppBase they would ask - "Where is the ServerAppBase class that this class extends?"

[rolweber]

Good point. Too bad we cannot use braces in classnames... (ExtendedServer)AppBase :-)

I'm OK with ServerAppExtensionBase, too. Besides, I'm not a native speaker, so my suggestions must be considered with caution.

[blink1073]

ServerAppExtensionBase sounds good to me too. Surely we can add a FactoryProviderResource in there as well? 😉

[Zsailer]

Okay, great! I'm on board with ServerAppExtensionBase. For the base handler class, should we use ServerAppExtensionHandlerBase?

Before we merge this, I'd like to get this mostly working with the Notebook front-end, JupyterLab, nteract_on_jupyter, and (maybe) voila. These projects cover a wide range of use-cases the ServerAppExtensionBase needs to support. I've got the notebook and nteract_on_jupyter working! 🎉 I'll update you when I get the other two working.

[kevin-bates]

Oh, great, you had to bring that up! 😃

I suppose if we apply the same transformation to ExtensionHandler that we applied to ExtensionApp we'd get ServerHandlerExtensionBase. 😄 We save a syllable!

I'm okay either way - or other suggestions as well.

[Zsailer]

Ok, the old notebook server and jupyter server are out-of-sync enough to make testing this PR on a notebook extension a pain. I've got it working, but I had to port pieces of jupyter server to make it work. The jupyter server is just too far behind the notebook right now.

So... I'd really like to update jupyter server to changes in notebook. Anyone opposed to this? I'm happy to tackle this myself, but if someone has more experience doing this, it might go faster.

[blink1073]

@SylvainCorlay did the initial porting, and it required re-mapping directories by hand iirc. I see two options, both requiring Sylvain to give some pointers:

• Port all of the relevant PRs since the last go-around
• Re-create what Sylvain did using the current state of the notebook repo

[rolweber]

AIUI, Apps need to be extensions, because only one app is launched at a time. But the app can have as many handlers as needed, and those are just handlers, not extensions of another handler?

ServerAppExtensionHandlerBase - a base for handlers defined by/for a ServerApp extension

ServerHandlerExtensionBase - a base for extending a ServerHandler?

I prefer the first suggestion :-)

[kevin-bates]

@rolweber - well, when you say it like that, ServerAppExtensionHandlerBase makes a lot of sense - especially the keeping of Extension with ServerApp and it really is a handler base class.

I guess I retract my suggestion and I apologize for the detour.

[blink1073]

Hmm, thinking of my Zen of Python (re: namespacing). We're already in a module named jupyter_server. Do we really need all these modifiers? Hell, it could be called just ExtensionApp and ExtensionHandler.

[maartenbreddels]

No Zen-expert, but I agree. Python has namespaces, no need to repeat them in class names.

[Zsailer]

Great, rolling back to the original names!

[kevin-bates]

If the purpose of these is to act as base classes then shouldn't that be part of their name. I thought that was a convention. If so, I'd prefer it be the suffix.

[blink1073]

We have a JupyterApp that is the base class of all of the Jupyter applications. I think it is fair to say that an ExtensionApp is a thing, and I subclass that to a more specific one.

[kevin-bates]

Understood. Thanks.

[SylvainCorlay]

@Zsailer sorry for not coming here earlier.

This is looking pretty good to me. Any reason why this is still marked as [WIP]?

[Zsailer]

@SylvainCorlay I think this is ready for merge (and iteration). I'll work on documentation in a following PR.

[Zsailer]

In case anyone is interested, I applied this new ExtensionApp class to Voila in voila-dashboards/voila#270. Check it out. Voila was a perfect candidate for testing this API since it depends solely on the jupyter_server (not notebook).

[rolweber]

lgtm

[SylvainCorlay]

Yeah!

[Zsailer]

Thanks, @rolweber !