Sync Notebook and Jupyter Server.

[Zsailer]

Opening this issue to track the porting of relevant Notebook PRs to Jupyter Server.

List to Port:

• properly handle unicode characters in home dirs jupyter/notebook#3664
• flush iopub before sending restarting/dead status messages jupyter/notebook#3759
• [FIX] notebookapp, auth: get_secure_cookie kwargs jupyter/notebook#3778
• modify contributing instructions to use pytest for testing jupyter/notebook#3789
• Fill in Checkpoints section jupyter/notebook#3799
• indent code block jupyter/notebook#3806
• ip_address only accepts unicode on Python 2 jupyter/notebook#3809
• Fix tornado max_body_size & max_buffer_size settings jupyter/notebook#3829
• Get the list of exporters from entrypoints jupyter/notebook#3879
• Describe problems and solutions involving CSP headers jupyter/notebook#3883
• Allow access control headers to be overriden in jupyter_notebook_conf… jupyter/notebook#3886
• Prevent access through 'NoneType' when closing activity_stream jupyter/notebook#3907
• When adapting also log the version jupyter_client is expecting. jupyter/notebook#4034
• Added metrics for currently running terminals and labeled by type kernels jupyter/notebook#4036
• Include status types when logging activity jupyter/notebook#4063
• Allow more contents manager functions to return futures jupyter/notebook#4099
• Fix copy/paste-ability of default URL presented on startup jupyter/notebook#4103
• Enable login mode for terminal - to automatically source the /etc/profile script jupyter/notebook#4112
• Fix default value method for allow_remote_access when ip is '*' jupyter/notebook#4139
• Embed NB2KG into Notebook server jupyter/notebook#4161 (via Port gateway from notebook #92)
• Enable kernel message filtering jupyter/notebook#4210
• Update session_exists() to account for invalid sessions due to culling jupyter/notebook#4219
• Allow ?no_track_activity=1 to opt-out of activity tracking jupyter/notebook#4235
• Launch the browser with a redirect file jupyter/notebook#4260
• More selective filename test in list_running_servers jupyter/notebook#4284
• List hidden files if allowed jupyter/notebook#4328
• Fix case sensitive notebook file extension  jupyter/notebook#4355
• surface config.d nbserver_extensions to the NotebookApp config object jupyter/notebook#4376
• Call tornado WebSocketHandler.get() as a coroutine jupyter/notebook#4392
• Enable restart_kernel for async usage jupyter/notebook#4412
• remove unused import of os jupyter/notebook#4426
• Update gateway support with recent changes jupyter/notebook#4431 (via Port gateway from notebook #92)
• tornado 6 compatibility jupyter/notebook#4449
• Finish transition to our own maybe_future jupyter/notebook#4453
• Fix incorrect MIME/Content-Type for JavaScript on misconfigured Windows systems jupyter/notebook#4468
• [doc] fix some linkcheck errors and warnings jupyter/notebook#4519
• explicitly ignore windows registry for mimetypes jupyter/notebook#4528
• Changed the default protocol jupyter/notebook#4531
• Fix binary message issue when configured for gateway forwarding jupyter/notebook#4576 (via Port gateway from notebook #92)
• fix default file name for make a copy jupyter/notebook#4618
• Added control channel to ZMQChannelsHandler jupyter/notebook#4672
• bugfix: shutdown_server returns True when pid exists jupyter/notebook#4674
• Port NB2KG PRs to Gateway jupyter/notebook#4701 (via Port gateway from notebook #92)
• Missing yield statement on WebSocketChannelsHandler jupyter/notebook#4724 (via Port gateway from notebook #92)
• Remove verbose debug statement in gateway mode jupyter/notebook#4725 (via Port gateway from notebook #92)
• Update jupyter_client jupyter/notebook#4756
• Attempt to re-establish websocket connection to Gateway jupyter/notebook#4777 (via Port gateway from notebook #92)
• Make FilesHandler.get() a decorated coroutine. Closes #4869 jupyter/notebook#4891

My strategy is to just manually move PRs that affect files in the Server. Is that the best strategy here? @SylvainCorlay @blink1073

[blink1073]

As a last resort, sure. I'm wondering if Sylvain had some git-foo that he used last time. 😀

[Zsailer]

@SylvainCorlay, Sensei, teach us your git-foo ways! 😃

[kevin-bates]

Assuming this is the place to talk about NB PRs we want in server..

The Gateway related PRs are needed: jupyter/notebook#4161, jupyter/notebook#4431 and the yet-to-be-merged jupyter/notebook#4576. These will enable the ability to manage kernels on computer cluster edge nodes, separated from the notebooks themselves.

I also believe jupyter server should introduce async kernel management, so (minor, but merged) PR jupyter/notebook#4412 should probably be taken, but we should strongly consider the yet-to-be-merged PR jupyter/notebook#4479 (which depends on jupyter client PR jupyter/jupyter_client#428). These last two enable the ability to change management class hierarchies from sync to async and I would propose we have the default be async (the PR uses sync as the default since it was targeting Notebook). Actually, this topic should probably become a separate issue.

If necessary, I'd be happy to help triage various notebook PRs for their applicability into server.

Thanks.

[Zsailer]

Absolutely. I will definitely make sure to port these PRs over. I think the intention is to port anything that touches server-side code in the notebook.

Before you spend too much time triaging notebook PRs, let's see if @SylvainCorlay has any tricks up his sleeve...

[lresende]

I believe @SylvainCorlay mentioned in the workshop that @Carreau has a tool that helps identifies all prs that touch a set of files or something like that which could help us identify the changes that need to be moved.

[SylvainCorlay]

I don't think that there is much magic we can do besides manually porting the PRs at this point.

[Zsailer]

Okay, I think it's time to start building a list of PRs to port. I'll edit the top post with a running checklist. If anyone wants to begin triaging PRs in notebook and listing them here, I'll update the checklist as we go.

[Zsailer]

Also, if anyone wants to port a PR, go ahead! 😄 I'll start working through this list, but any help is appreciated.

[Zsailer]

Also, here's a list of notebook PRs that were merged after @SylvainCorlay did the initial fork. Many of these PRs are frontend specific.

[Zsailer]

I've updated the top comment to include the full list of merged PRs in notebook that touch server_side code. Many of the changes are small.

What's the best strategy to move forward?

My plan was to

1. cherry-pick each of these changes in one major PR.
2. starting tracking notebook repo for server-side changes.
3. manually port PRs to server as they land in notebook.

Any better strategies?

[kevin-bates]

Thanks Zach. I think steps 2 and 3 are fine. Step 1 is probably the real question. One massive PR seems like it could be a nightmare if issues arise. I'm wondering if it would be better to chunk things up given they were independent changes previously. Perhaps a "series" of 3 to 5 PRs? Would that approach be helpful for checkpointing progress?

I'm curious if you found changes that spanned into what would be the "nb classic" server extension?

Btw, the various Gateway PRs are addressed in #92. I'll update the comment tomorrow - checking the applicable entries (with a reference to #92) if you're okay with that.

[Zsailer]

chunk things up given they were independent changes previously

I think you're right. Let's break it up. I'm planning to move through these pretty fast, so I'll open multiple batches of PRs.

I didn't track any PRs that were nbclassic specific. I'll probably rebase jupyter/notebook#4653, allowing us to start fresh on tracking PRs for nbclassic.

Btw, the various Gateway PRs are addressed in #92. I'll update the comment tomorrow - checking the applicable entries (with a reference to #92) if you're okay with that.

Great! #92 is what inspired me to build the above checklist. I'd love to stay on top of notebook PRs moving forward. Thanks for doing this great work!

[kevin-bates]

@Zsailer - I've updated the comment to indicate those gateway-related PRs handled in #92. There's also one gateway-related commit from #4449 that you'll find already done (be35a37).

[Zsailer]

Question for the group—should we squash merge the ported PRs? Or should we retain the full commit history?

[lresende]

I personally really don't like the option to see the PR history in master, it makes it very messy, particularly as a lot of people add committs to a pr like 'fix typo' or 'another try'... Also, if someone is really interested in the pr lifecycle, comments, history, this is still available on the pr.

Having said that, in this particular case, this is cherry-picking multiple commits, so I would just rebase and merge which would maintain the history.

[kevin-bates]

I agree. I think moving forward, we should try to adopt a squash and merge approach for multi-commit PRs. But, as Luciano says, since these are essentially multi-PR PRs, where each PR is potentially multiple commits, I think it makes sense to not squash in this case.

[Zsailer]

Great, thanks everyone. Check out #96 for our plan to move forward.

[vidartf]

I'm mostly against squashing due to it making git blame (especially here in github) a lot harder to navigate. However, I'm for it under these circumstances:

• The individual commits change some code, then changes it back to the original value (reversion). In these cases squashes make for a cleaner history.
• The total line changes are small (implying that it doesn't conflate changes that are logically different).

[Zsailer]

🎉 🎉 I think we've successfully synced jupyter_server with the classic notebook server (huge thank you to @kevin-bates for reviewing these PRs)! 🎉 🎉

I'll be tracking notebook to make sure we port PRs here as they land in notebook.

[kevin-bates]

Thanks for the great work Zach! Time to rock this thing! 🚀