Add JS tests for Jupyter NBs #549

etpinard · 2016-08-18T16:42:54Z

Behold: JS tests in Jupyter Notebooks are coming to plotly.py

My strategy was the following:

Have a few ipynb fixture files with plotly.py code cells but no outputs
Execute them using the jupyter execute api
Convert the resulting ipynb JSON to HTML using nbconver api
Add JS test bundle to the resulting HTML file
Run the JS tests in a browser!

Step 5 was a little harder than I thought. There's a ton of test runners that take a JS bundle, shim it into an HTML page and run the tests. But, I couldn't find any test runner that takes an already made
HTML page and runs the test there. A few runners allow you to add HTML to their default test page, but I couldn't get the results I wanted, so I decided to build my little test runner see lib/server.js

TODO:

add more test fixtures / cases
get this to work on CircleCI
make sure this works in all python versions we support

@chriddyp @theengineear @cldougl @mdtusz

etpinard · 2016-08-18T16:44:38Z

plotly/tests/test_core/test_jupyter/test_jupyter.py

+
+
+class PlotlyJupyterTestCase(TestCase):
+    def setUp(self):


Execute fixture NB and convert results to HTML

🐄 set_up or setup for python 🐍_case

Looks like setUp is what we want here

but I'm no python expert.

mdtusz · 2016-08-18T17:16:00Z

This looks great! Will be good to get some more solid tests in for nb's.

theengineear · 2016-08-18T18:14:50Z

So awesome, @etpinard!! Nice work, man!

theengineear · 2016-12-28T21:25:01Z

@etpinard I'm assuming this just got back-burnered with other work, but this would be really nice to test. Status update?

etpinard · 2017-01-04T16:28:33Z

@theengineear I got everything working locally in both python 2.7 and 3.4

Upgrading optonal-requirements.txt didn't go down so well on CircleCI. If you're up for a challenge, I could use some help.

etpinard · 2017-01-04T16:31:00Z

optional-requirements.txt

-## ipython dependencies ##
-ipython[all]==3.0.0
+## ipython ##
+ipython


@theengineear this didn't go down so smooth on Circle.

theengineear · 2017-01-05T05:43:55Z

Yup. I just upgraded our test framework to use tox. I'll fiddle around with it tomorrow am and let you know how it goes!

Challenge accepted 😸

- add coverage to optional requirements - pip install optional requirements and no need for nothing else

- make lib/server.js handle arbitrary html index and js test file (to test multiple ipynb fixutres) - use ipykernel to determine which python version nosetest is launched w/ - use domready to make sure that tests are ran after DOM is ready

- one testing offline connected=True - one testing offline connected=False (the default)

theengineear · 2017-01-05T18:37:30Z

@etpinard, the tests are failing for me locally because they're looking for a custom.css file? Adding a blank custom.css ends up creating two seemingly OK html files in /fixtures.

Initial error:

After adding `custom.css`:

$ nosetests plotly/tests/test_optional/test_jupyter/test_jupyter.py
FF
======================================================================
FAIL: test_js (test_jupyter.PlotlyJupyterConnectedFalseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ee/PyCharmProjects/python-api/plotly/tests/test_optional/test_jupyter/test_jupyter.py", line 55, in test_js
    self.fail('One or more javascript test failed')
AssertionError: One or more javascript test failed
-------------------- >> begin captured logging << --------------------
root: INFO: Executing notebook with kernel: python2
root: DEBUG: Native kernel (python2) available from /Users/ee/venvs/python-api/lib/python2.7/site-packages/ipykernel/resources
root: DEBUG: Starting kernel: ['/Users/ee/venvs/python-api/bin/python2.7', '-m', 'ipykernel', '-f', u'/var/folders/r8/xjmvqr192r589r78rrx8_ft80000gn/T/tmp7aYXtD.json']
root: DEBUG: Connecting to: tcp://127.0.0.1:63173
root: DEBUG: connecting shell channel to tcp://127.0.0.1:63170
root: DEBUG: Connecting to: tcp://127.0.0.1:63170
root: DEBUG: connecting iopub channel to tcp://127.0.0.1:63171
root: DEBUG: Connecting to: tcp://127.0.0.1:63171
root: DEBUG: connecting stdin channel to tcp://127.0.0.1:63172
root: DEBUG: Connecting to: tcp://127.0.0.1:63172
root: DEBUG: connecting heartbeat channel to tcp://127.0.0.1:63174
root: DEBUG: Executing cell:
from plotly.offline import plot, iplot, init_notebook_mode
import plotly.graph_objs as go

# Make plotly work with Jupyter notebook
init_notebook_mode()

keys=['one','two','three']
values=[1,2,3]

iplot({
    "data": [go.Bar(x=keys, y=values)],
    "layout": go.Layout(title="Sample Bar Chart")
})
root: DEBUG: output: status
root: DEBUG: output: execute_input
root: DEBUG: output: display_data
root: DEBUG: output: display_data
root: DEBUG: output: status
root: DEBUG: Executing cell:

root: DEBUG: output: status
root: DEBUG: output: execute_input
root: DEBUG: output: status
root: DEBUG: Applying preprocessor: coalesce_streams
root: DEBUG: Applying preprocessor: CSSHTMLHeaderPreprocessor
root: DEBUG: Applying preprocessor: HighlightMagicsPreprocessor
root: DEBUG: Attempting to load template full.tpl
root: DEBUG: Loaded template full.tpl
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_js (test_jupyter.PlotlyJupyterConnectedTrueTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ee/PyCharmProjects/python-api/plotly/tests/test_optional/test_jupyter/test_jupyter.py", line 55, in test_js
    self.fail('One or more javascript test failed')
AssertionError: One or more javascript test failed
-------------------- >> begin captured logging << --------------------
root: INFO: Executing notebook with kernel: python2
root: DEBUG: Native kernel (python2) available from /Users/ee/venvs/python-api/lib/python2.7/site-packages/ipykernel/resources
root: DEBUG: Starting kernel: ['/Users/ee/venvs/python-api/bin/python2.7', '-m', 'ipykernel', '-f', u'/var/folders/r8/xjmvqr192r589r78rrx8_ft80000gn/T/tmpWTCkTH.json']
root: DEBUG: Connecting to: tcp://127.0.0.1:63196
root: DEBUG: connecting shell channel to tcp://127.0.0.1:63193
root: DEBUG: Connecting to: tcp://127.0.0.1:63193
root: DEBUG: connecting iopub channel to tcp://127.0.0.1:63194
root: DEBUG: Connecting to: tcp://127.0.0.1:63194
root: DEBUG: connecting stdin channel to tcp://127.0.0.1:63195
root: DEBUG: Connecting to: tcp://127.0.0.1:63195
root: DEBUG: connecting heartbeat channel to tcp://127.0.0.1:63197
root: DEBUG: Executing cell:
from plotly.offline import plot, iplot, init_notebook_mode
import plotly.graph_objs as go

# Make plotly work with Jupyter notebook
init_notebook_mode(connected=True)

keys=['one','two','three']
values=[1,2,3]

iplot({
    "data": [go.Bar(x=keys, y=values)],
    "layout": go.Layout(title="Sample Bar Chart")
})
root: DEBUG: output: status
root: DEBUG: output: execute_input
root: DEBUG: output: display_data
root: DEBUG: output: display_data
root: DEBUG: output: status
root: DEBUG: Executing cell:

root: DEBUG: output: status
root: DEBUG: output: execute_input
root: DEBUG: output: status
root: DEBUG: Applying preprocessor: coalesce_streams
root: DEBUG: Applying preprocessor: CSSHTMLHeaderPreprocessor
root: DEBUG: Applying preprocessor: HighlightMagicsPreprocessor
root: DEBUG: Attempting to load template full.tpl
root: DEBUG: Loaded template full.tpl
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 2 tests in 12.386s

FAILED (failures=2)

It's not clear to me why this is failing, thoughts?

theengineear · 2017-01-05T18:41:28Z

Also, I did a little (emphasis on little) about testing jupyter notebooks:

Not sure if you saw either of these, but it might be sweet to try and use a 3rd party tool so we don't have to maintain (ipython_nose). I'm not 100% sure if we can test what we need to test from inside a notebook cell, but I might take some time to check that out.

theengineear · 2017-01-05T18:48:04Z

^^ nvm, all good. helps if you npm install 😸

theengineear · 2017-01-05T20:34:33Z

Holy intermittent image server tests... getting a ton of:

ERROR: test_image_get_returns_valid_image_test(jpeg, 300, 300, None) (plotly.tests.test_core.test_image.test_image.TestImage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/plotly.py/plotly/tests/test_core/test_image/test_image.py", line 27, in test
    image = py.image.get(self.data, image_format, width, height, scale)
  File "/home/ubuntu/plotly.py/plotly/plotly/plotly.py", line 715, in get
    raise exceptions.PlotlyError("The response "
nose.proxy.PlotlyError: The response from plotly could not be translated.

theengineear · 2017-01-05T20:55:52Z

#649

^^ for the image server fails...

Just wrapping them in a loop so that we can retry as needed for now.

theengineear · 2017-01-05T22:55:56Z

Aight, gonna merge this one in while everything is 🎏.

//cc @etpinard @chriddyp

etpinard reviewed Aug 18, 2016
View reviewed changes

etpinard mentioned this pull request Oct 19, 2016

Animations/Frames in Python API #584

Merged

etpinard commented Jan 4, 2017

View reviewed changes

etpinard and others added 7 commits January 5, 2017 10:08

first pass js jupyter tests

eba8bd7

un-ignore fixture ipynb files

673d0a3

standardize optional requirements for CircleCI

f02b6a7

- add coverage to optional requirements - pip install optional requirements and no need for nothing else

add requirements for jupyter test on circleci

cedb7f1

improve jupyter test design:

590f485

- make lib/server.js handle arbitrary html index and js test file (to test multiple ipynb fixutres) - use ipykernel to determine which python version nosetest is launched w/ - use domready to make sure that tests are ran after DOM is ready

split ipynb fixture and js test file into two cases:

2892049

- one testing offline connected=True - one testing offline connected=False (the default)

Move test_jupyter into *optional* test cases.

27966e5

Add jupyter to optional tox deps.

28e2fe0

theengineear force-pushed the test-jupyter branch from 0511421 to c5df158 Compare January 5, 2017 19:28

Cache jupyter node_modules deps.

8850feb

theengineear force-pushed the test-jupyter branch from c5df158 to 8850feb Compare January 5, 2017 19:59

theengineear added 2 commits January 5, 2017 14:30

Quick fix for intermittent image server tests.

053cdd6

Just wrapping them in a loop so that we can retry as needed for now.

Use node a version that Circle has preinstalled.

7da7ed2

theengineear force-pushed the test-jupyter branch from 59e6cc0 to 7da7ed2 Compare January 5, 2017 22:30

theengineear merged commit 8fdd412 into master Jan 5, 2017

theengineear deleted the test-jupyter branch January 5, 2017 22:56

This was referenced Jan 6, 2017

Add docs for Jupyter NB test suite #653

Merged

Improved compatibility with ipywidgtes and jupyter notebook #641

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add JS tests for Jupyter NBs #549

Add JS tests for Jupyter NBs #549

etpinard commented Aug 18, 2016 •

edited by theengineear

Loading

etpinard Aug 18, 2016 •

edited

Loading

mdtusz Aug 18, 2016

etpinard Aug 18, 2016

mdtusz commented Aug 18, 2016

theengineear commented Aug 18, 2016

theengineear commented Dec 28, 2016

etpinard commented Jan 4, 2017

etpinard Jan 4, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

Add JS tests for Jupyter NBs #549

Add JS tests for Jupyter NBs #549

Conversation

etpinard commented Aug 18, 2016 • edited by theengineear Loading

etpinard Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

mdtusz Aug 18, 2016

Choose a reason for hiding this comment

etpinard Aug 18, 2016

Choose a reason for hiding this comment

mdtusz commented Aug 18, 2016

theengineear commented Aug 18, 2016

theengineear commented Dec 28, 2016

etpinard commented Jan 4, 2017

etpinard Jan 4, 2017

Choose a reason for hiding this comment

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

Initial error:

After adding custom.css:

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

theengineear commented Jan 5, 2017

etpinard commented Aug 18, 2016 •

edited by theengineear

Loading

etpinard Aug 18, 2016 •

edited

Loading

After adding `custom.css`: