global: upgrade jupyter env tag to 2.0.0

[mvidalgarcia]

Back to a single notebook file with Python3 kernel.

closes reanahub/reana-env-jupyter#6

To test:

$ reana-dev run-example -c r-d-worldpopulation
...
OK

[tiborsimko]

It works all well with updated r-e-jupyter PR for me.

One issue is the warnings.

With py3 we get a number of warnings:

 Input Notebook:  code/worldpopulation.ipynb
Output Notebook: /dev/null
Generating grammar tables from /usr/local/lib/python3.6/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /.cache/black/21.7b0/Grammar3.6.8.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/.cache/black/21.7b0/tmpowtry046'
Generating grammar tables from /usr/local/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /.cache/black/21.7b0/PatternGrammar3.6.8.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/.cache/black/21.7b0/tmp9ucq_f38'
Executing:   0%|          | 0/7 [00:00<?, ?cell/s]/usr/local/lib/python3.6/site-packages/IPython/paths.py:67: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
/usr/local/lib/python3.6/site-packages/papermill/iorw.py:120: UserWarning: the file is not specified with any extension : null
  "the file is not specified with any extension : " + os.path.basename(path)
/usr/local/lib/python3.6/site-packages/IPython/paths.py:67: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
Executing notebook with kernel: python3
/usr/local/lib/python3.6/site-packages/papermill/iorw.py:120: UserWarning: the file is not specified with any extension : null
  "the file is not specified with any extension : " + os.path.basename(path)
Executing: 100%|██████████| 7/7 [00:03<00:00,  2.16cell/s]

Perhaps we could hunt these down? It might need to amend the image to create necessary cache directories, or set up working directory, etc.

It seems that the py2 example was also showing some of these:

  from pandas._libs import algos as _algos, reshape as _reshape
/usr/lib64/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  import pandas._libs.parsers as parsers
Input Notebook:  code/worldpopulation.ipynb
Output Notebook: /dev/null
/usr/lib/python2.7/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
/usr/lib/python2.7/site-packages/IPython/paths.py:69: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
100%|##########| 7/7 [00:01<00:00,  6.55it/s]

We could also address some of these later... Just musing out lound about possible image amendments before merging and tagging reanahub/reana-env-jupyter:2.0.0.

[mvidalgarcia]

Looks it's a problem related to black caches creation (psf/black#1143, nteract/papermill#498), which is a papermill dep.

[mvidalgarcia]

A workaround would be adding something like this to the Dockerfile to generate the cache folders manually:

RUN black /usr/local/lib/python3.6/site-packages/black/cache.py

Not ideal IMO, as black folder layout can change over time and it's not pinned in papermill requirements.

@tiborsimko

[tiborsimko]

Not ideal IMO, as black folder layout can change over time and it's not pinned in papermill requirements.

Yes, the requirements are kind of light... But since we pin =papermill= already, we are owning the upgrade process... So perhaps we can also pin =black=, and use your workaround (whilst keeping a comment in Dockerfile as to why it;s there, and we should be all good. Then, when we shall be upgrading papermill and friends in the future, we shall revisit the black dependency version and cache issue. WDYT?

[mvidalgarcia]

Unfortunately, RUN black /usr/local/lib/python3.6/site-packages/black/cache.py doesn't generate the right cache folder, it only generates the one for the root user in /root/.cache/black. When the workflow is submitted to REANA, uid is 1000 so it expects the cache in /.cache/black.

It can be solve by running this instead:

RUN mkdir -m 770 -p /.cache/black/21.7b0/

As black is pinned, it should have the same effect.

[mvidalgarcia]

After the last changes in reanahub/reana-env-jupyter#10 the logs look like before:

Input Notebook:  code/worldpopulation.ipynb
Output Notebook: /dev/null
Generating grammar tables from /usr/local/lib/python3.6/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /.cache/black/21.7b0/Grammar3.6.8.final.0.pickle
Generating grammar tables from /usr/local/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /.cache/black/21.7b0/PatternGrammar3.6.8.final.0.pickle
Executing:   0%|          | 0/7 [00:00<?, ?cell/s]/usr/local/lib/python3.6/site-packages/IPython/paths.py:67: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
/usr/local/lib/python3.6/site-packages/papermill/iorw.py:120: UserWarning: the file is not specified with any extension : null
  "the file is not specified with any extension : " + os.path.basename(path)
/usr/local/lib/python3.6/site-packages/IPython/paths.py:67: UserWarning: IPython parent '/' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
Executing notebook with kernel: python3
/usr/local/lib/python3.6/site-packages/papermill/iorw.py:120: UserWarning: the file is not specified with any extension : null
  "the file is not specified with any extension : " + os.path.basename(path)
Executing: 100%|██████████| 7/7 [00:02<00:00,  2.52cell/s]


Completed