Cell code selection dropdowns disappear when notebook is reopened

[aolney]

On a clean install, using jupyterlab-sos 0.4.3, code selection dropdowns (on each cell) work properly when a notebook is created.

However, when reopening notebook, all but the first dropdown disappear.

[BoPeng]

How do you re-open the notebook? Do you mean

1. Press Enter from the address bar to refresh the page, or
2. Close the tab and open from the browser, or
3. Close the tab, shutdown the notebook, and open from the browser

[BoPeng]

Also please confirm that you are using JupyterLab 1.0.0 alpha.

[aolney]

I can confirm jupyterlab 1.0.0a3

I was intrigued by the different options you mentioned for reopening, so I tried them all:

1. Press Enter from the address bar to refresh the page, or -->bug occurs as reported
2. Close the tab and open from the browser, or -->closed Jupyterlab tab and reopened from Jupyterlab explorer; bug occurs as reported
3. Close the tab, shutdown the notebook, and open from the browser -->shutdown notebook and notebook server, restart everything; bug occurs as reported

[BoPeng]

I am using this extension right now but could not reproduce the problem. Could you please send me your test notebook?

[BoPeng]

It is most likely that SoS failed to add the dropdown box for some reason and stopped adding it to other cells. If you open the browser console (Shift + Right click -> Inspect -> Console for chrome), you might be able to see some error messages, which would be useful for debugging.

[aolney]

For some reason Chrome wasn't displaying any errors, but I tried Chrome Beta and got the following:

vendors~main.f5e8638b1bed17cfc01e.js:171930 Download the React DevTools for a better development experience: https://fb.me/react-devtools
vendors~main.f5e8638b1bed17cfc01e.js:195887 Starting application in workspace: /lab
vendors~main.f5e8638b1bed17cfc01e.js:55675 JupyterLab extension sos-extension is activated!
vendors~main.f5e8638b1bed17cfc01e.js:236748 Creating a new notebook info
vendors~main.f5e8638b1bed17cfc01e.js:148652 Starting WebSocket: ws://127.0.0.1:8888/api/kernels/d9d4fe05-29fc-4e6a-940a-547914b85fc0
vendors~main.f5e8638b1bed17cfc01e.js:55586 kernel changed to sos
vendors~main.f5e8638b1bed17cfc01e.js:55565 session ready with kernel sos
vendors~main.f5e8638b1bed17cfc01e.js:150026 Kernel: connected (d9d4fe05-29fc-4e6a-940a-547914b85fc0)
vendors~main.f5e8638b1bed17cfc01e.js:55610 connected to sos kernel
vendors~main.f5e8638b1bed17cfc01e.js:55498 sos comm registered
vendors~main.f5e8638b1bed17cfc01e.js:205043 executor patched
vendors~main.f5e8638b1bed17cfc01e.js:55260 Received notebook-version
vendors~main.f5e8638b1bed17cfc01e.js:55260 Received kernel-list
vendors~main.f5e8638b1bed17cfc01e.js:148720 TypeError: base_mode.toLowerCase is not a function
    at changeStyleOnKernel (vendors~main.f5e8638b1bed17cfc01e.js:259270)
    at Object.updateCellStyles (vendors~main.f5e8638b1bed17cfc01e.js:259287)
    at on_frontend_msg (vendors~main.f5e8638b1bed17cfc01e.js:55263)
    at DefaultKernel.<anonymous> (vendors~main.f5e8638b1bed17cfc01e.js:149564)
    at Generator.next (<anonymous>)
    at vendors~main.f5e8638b1bed17cfc01e.js:148597
    at new Promise (<anonymous>)
    at push.YC29.__awaiter (vendors~main.f5e8638b1bed17cfc01e.js:148593)
    at DefaultKernel._handleCommMsg (vendors~main.f5e8638b1bed17cfc01e.js:149554)
    at DefaultKernel.<anonymous> (vendors~main.f5e8638b1bed17cfc01e.js:149621)
(anonymous) @ vendors~main.f5e8638b1bed17cfc01e.js:148720
default.js:64 Starting WebSocket: ws://127.0.0.1:8888/api/kernels/d9d4fe05-29fc-4e6a-940a-547914b85fc0
default.js:1438 Kernel: connected (d9d4fe05-29fc-4e6a-940a-547914b85fc0)

The offending line in selectors.js is

    if (!base_mode || base_mode.toLowerCase() === 'sos') {
        cell.inputArea.editorWidget.editor.setOption('mode', 'sos');
    }

[BoPeng]

Thanks. It looks like SoS tried to syntax highlight your F# cell, and the basemode of F# is not a string.

It is troublesome to install F# on a mac so let me try this on a windows machine and see if I can reproduce and fix it.

[aolney]

The bug initially appeared on a workbook with Bash, F#, and Java cells; I used F# in reporting the bug as a minimal working example. I can try other languages if you think some aren't affected. Thanks for all your efforts 👍

[BoPeng]

It would be easier to spot the problem if your screenshot contains these cells with the dropdown and stops at a F# cell, but I think this should be the culprit. I will fix it soon.

BTW, SoS supports Bash, there is a third party language module for Java, but none for F# yet. I have recently updated our tutorial on how to write a language module so I would invite you to write one if you would like to enable data exchange between F# and other kernels. I will be of course happy to help (although I know nothing about F#).

[aolney]

I'd be happy to contribute an F# module. BTW here is a minimum working example with multiple languages, f# in the last cell.

[BoPeng]

Ohmm, that particular toLowerCase was fixed in 5ef2de5#diff-a1df64701656b00fe9cd4cdc832bdaabL130, which should be in jupyterlab-sos 0.4.3. Could you check if you are actually running 0.4.3 with the following command?

$ jupyter labextension list

I also have just released jupyterlab-sos 0.4.4 with a fix that works only with the trunk of JupyterLab, but you can try to upgrade to this version and try again.

[aolney]

I seem to be running the version you indicated:

$jupyter labextension list
JupyterLab v1.0.0a3
Known labextensions:
   app dir: /vm/tf113/share/jupyter/lab
        jupyterlab-sos v0.4.3  enabled  OK
        transient-display-data v0.2.1  enabled  OK

Updating to 0.4.4 using jupyter labextension update jupyterlab-sos solved the problem, thanks 👍

[BoPeng]

This is strange because 0.4.3 contains the fix according to its source code. Anyway, I am glad that it works now and will close the ticket.

It is great that you are interesting in contributing a language module for F#, I can set up a framework with basic features in vatlab and invite you to the repo or you can create from scratch in your account. Just let me know how you would like to proceed.

[aolney]

Inviting to the repo seems the best approach for long term continuity

[BoPeng]

Thanks! I have created a repo and invited you. The repo has all the basic stuff and I have confirmed that it provides the F# language to SoS. The best way for you to proceed, I think, is

1. clone the repo
2. pip install -e . to do a local installation so that you do not have to reinstall with changes to it
3. Follow the language module tutorial to add the missing pieces
4. Start a jupyter server, restart sos kernel with each change to sos-fsharp and test
5. Record the completed features to tests or write tests first if you prefer
6. ping me if you get any question

Please also update sos-notebook to the latest version (0.19.6) because it contains an updated %capture magic which is useful for the development of language modules.

[aolney]

Got it, will do 😎

[aolney]

Is there a recommended testing procedure besides the full blown test suite? I've made a first attempt at kernel.py and am trying to test %sessioninfo, but it seems to be failing silently. I'm not strong in Python, so I'm probably doing lots of n00b things.

Code is:

    def sessioninfo(self):
        response = self.sos_kernel.get_response(
            'match System.AppDomain.CurrentDomain.GetAssemblies() |> Seq.map( fun a -> a.GetName()) |> Seq.tryFind( fun name -> name.Name = \"FSharp.Core\") with |Some(x) -> x.ToString()| None -> \"No session information is available\"',
            ('stream',),
            name=('stdout',))[0]
        return response[1]['text']

The F# code returns the following string when run from a REPL:

"FSharp.Core, Version=4.4.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[BoPeng]

If you would like test the F# language module, you can use the selenium-based tests described here, with examples for sos-r and sos-matlab. Although the tests use classic jupyter, it should be enough to test language modules. Note that we are porting the test mechanism to jupyter lab (#31 ).

[BoPeng]

Note that we used to test sos_kernel directly but got trapped into all the messages and timing stuff. Selenium-based tests basically test what you can see from a browser and are much easier to write.

[aolney]

Sorry for the delay - this isn't forgotten

[BoPeng]

All my tests are failing. Webdriver works when called from python (launches Chrome), but Chrome isn't launching when I run pytest. Any suggestions?

The test suite was designed for classic jupyter, not JupyterLab. Your command line seems to be starting the notebook, so

JupyterLab extension loaded from /z/aolney/repos/sos-fsharp/sosvenv/lib/pyth

might not be the source of problem.

Chrome should not start if you run pytest without setting JUPYTER_TEST_BROWSER. However, if you run

export JUPYTER_TEST_BROWSER=live

You should see the browser and can hopefully see what has gone wrong. I might be able to see what went wrong if you push your module and tests to https://github.com/vatlab/sos-fsharp.

[aolney]

Apologies, I deleted the comment and posted a new one, but somehow the second comment did not get saved.

I was able to get Chrome working but it seems the problem is that the F# kernel is not getting selected in the dropdown, so all the F# calls are going to the SoS kernel.

I've cloned the sos-r repo to run the tests there, and 30/39 of those are passing properly. Now that I see the sos-r tests running, I wish I'd done this first to better understand the runtime behavior sos-fsharp is supposed to achieve (magics are a bit strange to me, coming from Beaker to Jupyter).

My code is pushed to branch v1 if you want to take a look.

[aolney]

I tried replacing all instances of "F#" in sos-fsharp with the Jupyter plugin name ifsharp. This makes the kernel selectable from pytest but seems to hide the work I've done with magics (unless I've done them so incorrectly that its not a question of being hidden).

[BoPeng]

When you see ifsharp in the kernel list, it means the language module is not properly loaded. Let me check out the v1 branch and see if I can reproduce/fix the problem.

[BoPeng]

1. With latest sos, sos-notebook, master or v1 of sos-fsharp, I can see F# in the language list.

2. I opened a F# notebook, and copy/paste your init_statement in, and I get error messages starting with

Unexpected symbol '|' in expression

which means your statement will not be correctly executed when the ifsharp kernel is loaded.

3. I switch back to the master of sos-fsharp, run

set JUPYTER_TEST_BROWSER=live

I can see that the chrome starts, but kernel='F#' fails to switch to the kernel. I am checking what is going on.

[BoPeng]

OK, the problem was with the selenium test that tries to select the F# option with a CSS selector that looks something like option[value=F#], which should have been option[value='F#'] because of the special character #. This has been fixed in the trunk of sos-notebook so please update your copy of sos-notebook to the trunk version and try again.

[aolney]

Thank you for being so responsive - I'm sure you're very busy, so I don't expect < 24hrs for a response.

I tried to pip install from the trunk with pip install git+https://github.com/vatlab/sos-notebook.git. However, F# still doesn't appear in the kernel dropdown (ifsharp instead).
This is after I reverted my commit changing F# to ifsharp.
I also repeated pytest after trying the following, but F# did not appear in the dropdown:

• Installing sos from the trunk with pip install git+https://github.com/vatlab/sos.git
• Trying from the master branch, not v1
• Reinstalled ifsharp (v3.0.1)

pip list shows the following sos packages:

sos (0.19.14)
sos-fsharp (0.1.0, /z/aolney/repos/sos-fsharp/src)
sos-notebook (0.19.10)

I've pushed my current v1 in case you need to look at it.

[BoPeng]

There are a bunch of loopholes when you have multiple conda environments and versions of sos-notebook etc, and even I sometimes got confused. Please check

1. verify if your conda environment is activated.
2. verify if sos kernel is installed with command jupyter kernelspec list --json. Make sure that the argv of sos kernel is using the python of your env (something like /Users/bpeng1/anaconda3/envs/sos/bin/python). Otherwise you could be using a python of another env.
3. verify if you only have one version of sos-notebook installed. That it to say, check directories or files in the patten of ..../anaconda3/envs/sos/lib/python3.6/site-packages/sos-notebook*. If you see multiple of them, remove all and run pip install git+https://github.com/vatlab/sos-notebook.git. Just to make sure, run python -m sos_notebook.install after it.

For sos-fsharp, I recommend that you remove existing installation and reinstall with command pip install -e . if this is not the way you are installing it. The -e . installs the module from your source code so that you do not have to install again after source code change. In Jupyter, a restart kernel will pick up your change in sos-fsharp.

Let me know if your test runs after these steps.

[BoPeng]

Also, if you run set SOS_DEBUG=ALL (windows) before you start jupyter or lab, you will see a bunch of debug information in ~/.sos/sos_debug.log, which could help from time to time.

[aolney]

I tried all the above but the kernel is not switching to F# in the test. On a positive note, the sos_debug.log is showing the correct kernel being found:

2019-07-01 14:20:46,036 - DEBUG - KERNEL - Found registered language R
2019-07-01 14:20:46,147 - DEBUG - KERNEL - Found registered language F#

Also, as you suggested, the sos kernelspec was listing a different virtualenv. FWIW I'm not using conda; I'm using python3 with virtualenv on Ubuntu. I'm about to catch a flight but will try to investigate tomorrow.

[aolney]

OK, I got it working (though tests are failing). It seemed to be a convoluted problem with my virtualenvs as you suggested.

BTW would it make sense to migrate this issue to sos-fsharp? It may help others, but they won't be looking for it here. I think I will close this issue and start a new one there if I encounter additional issues. Please reopen if you prefer to continue discussion here.

[BoPeng]

Yes, we should move the discussions there.