Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unable to plot in jupyter notebook using R kernel #210

Closed
fyears opened this issue May 20, 2016 · 53 comments
Closed

unable to plot in jupyter notebook using R kernel #210

fyears opened this issue May 20, 2016 · 53 comments
Labels
tag:Upstream A problem with one of the upstream packages installed in the docker images type:Bug A problem with the definition of one of the docker images maintained here

Comments

@fyears
Copy link

fyears commented May 20, 2016

I am using default configuration of jupyter/datascience-notebook.

By accessing http://docker.local:8888/, I could start a notebook with R kernel, but I could not generate plots inside the notebook.

The simple official example:

require(stats) # for lowess, rpois, rnorm
plot(cars)
lines(lowess(cars))

capture

The #54 might relevant, but I am using the most up-to-todate image (20160519) .

@parente
Copy link
Member

parente commented May 20, 2016

Thanks for the report. Definitely looks like #54 again. libxrender moved from datascience-notebook to minimal-notebook (https://github.com/jupyter/docker-stacks/blob/master/minimal-notebook/Dockerfile#L33) so it should still be available. Must be something else wrong now.

@parente parente added the type:Bug A problem with the definition of one of the docker images maintained here label May 20, 2016
@michael-erasmus
Copy link

michael-erasmus commented May 25, 2016

I'm running into the exact same issue.

Not sure if this helps, but if I run start-notebook --debug is see the following output whenever I try plotting anything. It also creates PDF files on disk.

[1] "Got unhandled msg_type:" "comm_info_request"
[1] "Got unhandled msg_type:" "comm_open"
[1] "Got unhandled msg_type:" "comm_open"
dev.new(): using pdf(file="Rplots1.pdf")

@michael-erasmus
Copy link

I also spotted this in the output, right before the Kernel crashed:

Warning messages:
1: In doTryCatch(return(expr), name, parentenv, handler) :
  unable to load shared object '/opt/conda/lib64/R/modules//R_X11.so':
  libjpeg.so.8: cannot open shared object file: No such file or directory
2: In png(tf, width, height, "in", pointsize, bg, res, antialias = antialias) :
  unable to load shared object '/opt/conda/lib64/R/library/grDevices/libs//cairo.so':
  libjpeg.so.8: cannot open shared object file: No such file or directory
3: In png(tf, width, height, "in", pointsize, bg, res, antialias = antialias) :
  failed to load cairo DLL
4: In file(con, "rb") :
  cannot open file '/tmp/RtmpjYzEfu/file183dd7bd4f.png': No such file or directory
5: In svg(tf, width, height, pointsize, FALSE, family, bg, antialias) :
  failed to load cairo DLL
6: In file(con, "rb") :
  cannot open file '/tmp/RtmpjYzEfu/file181f35e31f.svg': No such file or directory
[I 04:52:09.095 NotebookApp] Kernel shutdown: 9fefd796-4a2b-4980-9141-a663fb75e4d6
[D 04:52:09.097 NotebookApp] 204 DELETE /api/sessions/02a18c5b-b2d3-4b7a-a4a6-491b0615858e (192.168.99.1) 106.02ms

@michael-erasmus
Copy link

Did some digging, it might be related to this issue? conda/conda#2020

@parente
Copy link
Member

parente commented May 25, 2016

I played one round of bisect over recent builds listed here https://github.com/jupyter/docker-stacks/wiki/Docker-build-history

  1. May. 16, 2016, jupyter/datascience-notebook:dc6ae8bd8209 (latest), plots don't work
  2. Apr. 13, 2016, jupyter/datascience-notebook:2d878db5cbff (4 builds ago), plots work
  3. May. 05, 2016, jupyter/datascience-notebook:2bcf753a6695 (3 builds ago), plots don't work

The only explicit diff between the last build that plots and the first build that doesn't is the update to use Notebook 4.2 and ipywidgets 5.1:

2bcf753a6695

Other implicit version changes may have snuck in during the docker image build, but we only allow most packages to vary in patch release number, not major or minor version. Plus, our builds are all done on the same box starting with a pull of the last build to gain the benefit of docker layer caching. This means anything before the notebook 4.2 version change in the Dockerfile will not have changed (e.g., version of libxrender pulled from debian).

@takluyver @jakirkham have you seen any problems with IRkernel in notebook 4.2 with or without conda?

@parente
Copy link
Member

parente commented May 25, 2016

@michael-erasmus hit the nail on the head.

Comparing the last working build to the first non-working one, I see a third-order dependency version changed and broke things. In 2d878db, the libjpeg v8 is present:

jovyan@8eac9da0d24d:/opt/conda/lib$ ls -l | grep jpeg
-rw-r--r--  2 jovyan users  1769836 Sep 11  2012 libjpeg.a
-rwxr-xr-x  1 jovyan users      918 Apr 14 01:33 libjpeg.la
lrwxrwxrwx  1 jovyan users       16 Apr 14 01:33 libjpeg.so -> libjpeg.so.8.4.0
lrwxrwxrwx  1 jovyan users       16 Apr 14 01:33 libjpeg.so.8 -> libjpeg.so.8.4.0
-rwxr-xr-x  2 jovyan users   973500 Sep 11  2012 libjpeg.so.8.4.0

In 2bcf753 and newer, v8 is gone and v9 is there:

jovyan@910e6e4609dd:~/work$ ls -l /opt/conda/lib | grep libjpeg
-rw-r--r--  2 jovyan users  2078420 Apr 15 18:42 libjpeg.a
-rwxr-xr-x  1 jovyan users      917 May  5 16:52 libjpeg.la
lrwxrwxrwx  1 jovyan users       16 May  5 16:52 libjpeg.so -> libjpeg.so.9.2.0
lrwxrwxrwx  1 jovyan users       16 May  5 16:52 libjpeg.so.9 -> libjpeg.so.9.2.0
-rwxr-xr-x  2 jovyan users  1073308 Apr 15 18:42 libjpeg.so.9.2.0

@jakirkham Is there a conda-forge feedstock for irkernel that needs to be updated? Or a R_X11 feedstock?

@parente parente added the tag:Upstream A problem with one of the upstream packages installed in the docker images label May 25, 2016
@jakirkham
Copy link
Member

@jakirkham Is there a conda-forge feedstock for irkernel that needs to be updated? Or a R_X11 feedstock?

Not yet. Someone started to get the ball rolling on R, but they are generally very busy and I believe there were a few tricks so it came to a stop.

Though we do provide JPEG 9. So, it is possible that some of these R packages are not properly pinned against JPEG 8. Hence we JPEG is upgraded and that breaks them. I would recommend adding a pinned file to the /opt/conda/conda-meta directory, which simply includes the line jpeg 8*. After running an update, this should bring JPEG back down to the right version and things will start working again. Though it is possible though that we may need a more exact pinning.

@parente
Copy link
Member

parente commented May 25, 2016

I would recommend adding a pinned file to the /opt/conda/conda-meta directory

Do you think we should add it to the images as a temporary fix and do a new build? Or do you think users should add it as a workaround until it's fixed upstream?

@jakirkham
Copy link
Member

Let's ask Continuum about fixing these packages to be properly pinned. If it looks like a long turnaround, we should probably pin it ourselves. Do we know exactly which packages were using JPEG?

@parente
Copy link
Member

parente commented Jun 1, 2016

Do we know exactly which packages were using JPEG?

I do not. Is there a canonical place to look to figure it out?

@michael-erasmus
Copy link

Just wanted to confirm that @jakirkham's suggestion did indeed work for me. I managed to get plotting to work by using a Dockerfile like this:

FROM jupyter/datascience-notebook
RUN echo "jpeg 8*" >> /opt/conda/conda-meta/pinned
RUN conda update --all -y

@jakirkham
Copy link
Member

jakirkham commented Jun 4, 2016

Glad to hear that @michael-erasmus. Could you please share with us what happened during conda update --all -y? It will help us as we talk with Continuum about what packages we need to have rebuilt.

@jakirkham
Copy link
Member

@msarahan, do you know who/where we should notify someone about this pinning issue with R packages? Basically, the stack in this docker container jupyter/datascience-notebook ends up being broken w.r.t. to R because of JPEG 9. Fortunately, pinning to a JPEG 8 seems tenable so it may just be an issue of R packages that are missing pins on Linux.

@msarahan
Copy link

msarahan commented Jun 7, 2016

CC @mingwandroid, who handles our R ecosystem.

SternAndrew pushed a commit to SternAndrew/docker-stacks that referenced this issue Jun 23, 2016
… issues by pulling in newer versions of rpy2 and ipywidgets.

1) Added freetds to allow access to database instances using odbc.
2) Create a shell .odbc.ini file that can be filled out to access a database instance.
3) Updated the version of rpy2 since the %R and %%R magic wouldn't work from the Python kernels so that R can be used in cells within a Python sheet.
4) Updated ipywidgets so that the widgets can be used within a Python3 sheet. Without this update it wouldn't show the controls in a Python3 sheet after running the sheet
5) Pinned the conda to jpeg 8 since the ggplot2 graphs wouldn't showup in R sheets.  See jupyter#210
@parente
Copy link
Member

parente commented Jul 9, 2016

This appears to be fixed in the latest build (tag: 9f67ac2ec481).

@fyears, @michael-erasmus, please re-open if you still have problems once you update.

@parente
Copy link
Member

parente commented Jul 14, 2016

New problem with plotting. This time, pinning to jpeg8 post-hoc does not fix the issue. Looking into what we might do.

@msarahan
Copy link

Sorry I missed this one. Pinging @mingwandroid who does the r stuff.

@parente
Copy link
Member

parente commented Jul 14, 2016

The latest (3aa2f70cb868) r-notebook image is OK, but datascience-notebook and all-spark-notebook both have the problem. Something in the scipy-notebook image is triggering the upgrade to libjpeg9 and so R in the two child images have the problem.

I can do the pinning in the scipy-notebook for the time being as a workaround until the R package(s) declare the libjpeg8 dependency properly. Or until we bump versions on all of the R packages so they all work with libjpeg9. I'll try the latter first.

@parente
Copy link
Member

parente commented Jul 14, 2016

No luck upgrading all the R libs. One of them still wants libjpeg8. So pinning it is.

parente added a commit to parente/docker-stacks that referenced this issue Jul 14, 2016
Pin libjpeg to v8

(c) Copyright IBM Corp. 2016
parente added a commit that referenced this issue Jul 15, 2016
@jakirkham
Copy link
Member

I can't believe I missed this. Seems the problem is cairo is linked to jpeg. While it doesn't look like cairo itself requires jpeg, it may have discovered it via a dependency that got linked to jpeg. Maybe R is a total red herring in all of this.

unable to load shared object '/opt/conda/lib64/R/library/grDevices/libs//cairo.so':
  libjpeg.so.8: cannot open shared object file: No such file or directory

@ianabc
Copy link

ianabc commented Aug 17, 2016

Not sure if this helps, but 54838ed and 7001472 pick up different revisions of r-base

[root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:54838ed4acb1 conda list | grep r-base
r-base                    3.3.1                         1    r
r-base64enc               0.1_3                  r3.3.1_0    r
[root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:7001472997fa conda list | grep r-base
r-base                    3.3.1                         3    r
r-base64enc               0.1_3                  r3.3.1_0    r

@mingwandroid
Copy link

Please do conda info and conda list

@ianabc
Copy link

ianabc commented Aug 17, 2016

On 54838ed...

root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:54838ed4acb1 conda info
Current conda install:

             platform : linux-64
        conda version : 4.1.11
    conda-env version : 2.5.2
  conda-build version : not installed
       python version : 3.5.2.final.0
     requests version : 2.10.0
     root environment : /opt/conda  (writable)
  default environment : /opt/conda
     envs directories : /opt/conda/envs
        package cache : /opt/conda/pkgs
         channel URLs : https://conda.anaconda.org/r/linux-64/
                        https://conda.anaconda.org/r/noarch/
                        https://repo.continuum.io/pkgs/free/linux-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /home/jovyan/.condarc
         offline mode : False
    is foreign system : False

[root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:54838ed4acb1 conda list
# packages in environment at /opt/conda:
#
bokeh                     0.11.1                   py35_0  
bzip2                     1.0.6                         3  
cairo                     1.12.18                       6  
cloudpickle               0.1.1                    py35_0  
conda                     4.1.11                   py35_0  
conda-env                 2.5.2                    py35_0  
backports.shutil_get_terminal_size 1.0.0                    py35_0    conda-forge
blas                      1.1                    openblas    conda-forge
cycler                    0.10.0                   py35_0    conda-forge
decorator                 4.0.10                   py35_0    conda-forge
dill                      0.2.5                    py35_0    conda-forge
entrypoints               0.2.2                    py35_0    conda-forge
ipykernel                 4.3.1                    py35_1    conda-forge
ipython                   5.0.0                    py35_0    conda-forge
ipywidgets                5.1.5                    py35_0    conda-forge
jinja2                    2.8                      py35_1    conda-forge
jsonschema                2.5.1                    py35_0    conda-forge
jupyter_client            4.3.0                    py35_0    conda-forge
jupyter_core              4.1.0                    py35_0    conda-forge
libiconv                  1.14                          3    conda-forge
libpng                    1.6.23                        0    conda-forge
libsodium                 1.0.10                        0    conda-forge
markupsafe                0.23                     py35_0    conda-forge
mistune                   0.7.3                    py35_0    conda-forge
mpmath                    0.19                     py35_0    conda-forge
nbconvert                 4.2.0                    py35_0    conda-forge
nbformat                  4.0.1                    py35_0    conda-forge
notebook                  4.2.2                    py35_0    conda-forge
numpy                     1.10.4          py35_blas_openblas_202  [blas_openblas]  conda-forge
openblas                  0.2.18                        4    conda-forge
pandas                    0.18.1              np110py35_1    conda-forge
pexpect                   4.2.0                    py35_1    conda-forge
pickleshare               0.7.3                    py35_0    conda-forge
pillow                    3.2.0                    py35_0    conda-forge
prompt_toolkit            1.0.4                    py35_0    conda-forge
ptyprocess                0.5.1                    py35_0    conda-forge
pygments                  2.1.3                    py35_0    conda-forge
pyparsing                 2.1.1                    py35_0    conda-forge
python-dateutil           2.5.3                    py35_0    conda-forge
pytz                      2016.6.1                 py35_0    conda-forge
pyzmq                     15.3.0                   py35_1    conda-forge
scikit-learn              0.17.1          np110py35_blas_openblas_201  [blas_openblas]  conda-forge
scipy                     0.17.1          np110py35_blas_openblas_202  [blas_openblas]  conda-forge
simplegeneric             0.8.1                    py35_0    conda-forge
sip                       4.18                     py35_0    conda-forge
six                       1.10.0                   py35_0    conda-forge
statsmodels               0.6.1               np110py35_0    conda-forge
sympy                     1.0                      py35_0    conda-forge
terminado                 0.6                      py35_0    conda-forge
tornado                   4.4.1                    py35_1    conda-forge
traitlets                 4.2.2                    py35_0    conda-forge
wcwidth                   0.1.7                    py35_0    conda-forge
widgetsnbextension        1.2.6                    py35_3    conda-forge
zeromq                    4.1.5                         0    conda-forge
curl                      7.49.0                        1  
cython                    0.23.5                   py35_0  
fastcache                 1.0.2                    py35_1  
fontconfig                2.11.1                        6  
freetype                  2.5.5                         1  
h5py                      2.5.0               np110py35_4  
hdf5                      1.8.15.1                      3  
icu                       54.1                          0  
ipython_genutils          0.1.0                    py35_0  
jbig                      2.1                           0  
jpeg                      8d                            1  
jupyterhub                0.5.0                     <pip>
libffi                    3.2.1                         0  
libgfortran               3.0.0                         1  
libtiff                   4.0.6                         2  
libxml2                   2.9.2                         0  
llvmlite                  0.8.0                    py35_0  
matplotlib                1.5.1               np110py35_0  
mkl                       11.3.3                        0  
networkx                  1.11                     py35_0  
numba                     0.23.1              np110py35_0  
numexpr                   2.5.2               np110py35_1  
openssl                   1.0.2h                        1  
pamela                    0.2.1                     <pip>
patsy                     0.4.1                    py35_0  
pcre                      8.39                          0  
pip                       8.1.2                    py35_0  
pixman                    0.32.6                        0  
pycosat                   0.6.1                    py35_1  
pycrypto                  2.6.1                    py35_4  
python                    3.5.2                         0  
pyyaml                    3.11                     py35_4  
glib                      2.43.0                        2    r
gsl                       1.16                          1    r
harfbuzz                  0.9.35                        6    r
libgcc                    4.8.5                         1    r
ncurses                   5.9                           8    r
pango                     1.36.8                        3    r
r                         3.3.1                  r3.3.1_0    r
r-assertthat              0.1                    r3.3.1_2    r
r-base                    3.3.1                         1    r
r-base64enc               0.1_3                  r3.3.1_0    r
r-bh                      1.60.0_2               r3.3.1_0    r
r-bitops                  1.0_6                  r3.3.1_2    r
r-boot                    1.3_18                 r3.3.1_0    r
r-car                     2.1_2                  r3.3.1_0    r
r-caret                   6.0_70                 r3.3.1_0    r
r-catools                 1.17.1                 r3.3.1_2    r
r-class                   7.3_14                 r3.3.1_0    r
r-cluster                 2.0.4                  r3.3.1_0    r
r-codetools               0.2_14                 r3.3.1_0    r
r-colorspace              1.2_6                  r3.3.1_0    r
r-curl                    0.9.7                  r3.3.1_0    r
r-dbi                     0.4_1                  r3.3.1_0    r
r-devtools                1.11.1                 r3.3.1_0    r
r-dichromat               2.0_0                        1a    r
r-digest                  0.6.9                  r3.3.1_0    r
r-dplyr                   0.4.3                  r3.3.1_0    r
r-evaluate                0.9                    r3.3.1_0    r
r-foreach                 1.4.3                  r3.3.1_0    r
r-forecast                7.1                    r3.3.1_0    r
r-foreign                 0.8_66                 r3.3.1_0    r
r-formatr                 1.4                    r3.3.1_0    r
r-fracdiff                1.4_2                  r3.3.1_2    r
r-ggplot2                 2.1.0                  r3.3.1_0    r
r-git2r                   0.15.0                 r3.3.1_0    r
r-gtable                  0.2.0                  r3.3.1_0    r
r-highr                   0.6                    r3.3.1_0    r
r-htmltools               0.3.5                  r3.3.1_0    r
r-httpuv                  1.3.3                  r3.3.1_0    r
r-httr                    1.2.0                  r3.3.1_0    r
r-irdisplay               0.3                    r3.3.1_2    r
r-irkernel                0.6.1                  r3.3.1_0    r
r-iterators               1.0.8                  r3.3.1_0    r
r-jsonlite                0.9.22                 r3.3.1_0    r
r-kernsmooth              2.23_15                r3.3.1_0    r
r-knitr                   1.13                   r3.3.1_0    r
r-labeling                0.3                    r3.3.1_2    r
r-lattice                 0.20_33                r3.3.1_0    r
r-lazyeval                0.2.0                  r3.3.1_0    r
r-lme4                    1.1_12                 r3.3.1_0    r
r-magrittr                1.5                    r3.3.1_2    r
r-markdown                0.7.7                  r3.3.1_2    r
r-mass                    7.3_45                 r3.3.1_0    r
r-matrix                  1.2_6                  r3.3.1_0    r
r-matrixmodels            0.4_1                  r3.3.1_0    r
r-memoise                 1.0.0                  r3.3.1_0    r
r-mgcv                    1.8_12                 r3.3.1_0    r
r-mime                    0.4                    r3.3.1_0    r
r-minqa                   1.2.4                  r3.3.1_2    r
r-munsell                 0.4.3                  r3.3.1_0    r
r-nlme                    3.1_128                r3.3.1_0    r
r-nloptr                  1.0.4                  r3.3.1_2    r
r-nnet                    7.3_12                 r3.3.1_0    r
r-nycflights13            0.2.0                  r3.3.1_0    r
r-openssl                 0.9.4                  r3.3.1_0    r
r-pbdzmq                  0.2_3                  r3.3.1_0    r
r-pbkrtest                0.4_6                  r3.3.1_0    r
r-plyr                    1.8.4                  r3.3.1_0    r
r-quadprog                1.5_5                  r3.3.1_2    r
r-quantreg                5.26                   r3.3.1_0    r
r-r6                      2.1.2                  r3.3.1_0    r
r-randomforest            4.6_12                 r3.3.1_0    r
r-rcolorbrewer            1.1_2                        1a    r
r-rcpp                    0.12.5                 r3.3.1_0    r
r-rcpparmadillo           0.7.100.3.1            r3.3.1_0    r
r-rcppeigen               0.3.2.8.1              r3.3.1_0    r
r-rcurl                   1.95_4.8               r3.3.1_0    r
r-recommended             3.3.1                  r3.3.1_0    r
r-repr                    0.7                    r3.3.1_0    r
r-reshape2                1.4.1                  r3.3.1_2    r
r-rmarkdown               0.9.6                  r3.3.1_0    r
r-rpart                   4.1_10                 r3.3.1_0    r
r-rsqlite                 1.0.0                  r3.3.1_2    r
r-rstudioapi              0.5                    r3.3.1_0    r
r-scales                  0.4.0                  r3.3.1_0    r
r-shiny                   0.13.2                 r3.3.1_0    r
r-sparsem                 1.7                    r3.3.1_0    r
r-spatial                 7.3_11                 r3.3.1_0    r
r-stringi                 1.1.1                  r3.3.1_0    r
r-stringr                 1.0.0                  r3.3.1_0    r
r-survival                2.39_4                 r3.3.1_0    r
r-tibble                  1.0                    r3.3.1_0    r
r-tidyr                   0.5.1                  r3.3.1_0    r
r-timedate                3012.100               r3.3.1_2    r
r-tseries                 0.10_35                r3.3.1_0    r
r-uuid                    0.1_2                  r3.3.1_0    r
r-whisker                 0.3_2                  r3.3.1_2    r
r-withr                   1.0.2                  r3.3.1_0    r
r-xtable                  1.8_2                  r3.3.1_0    r
r-yaml                    2.1.13                 r3.3.1_2    r
r-zoo                     1.7_13                 r3.3.1_0    r
rpy2                      2.8.1              py35r3.3.1_1    r
readline                  6.2                           2  
requests                  2.10.0                   py35_0  
ruamel_yaml               0.11.14                  py35_0  
scikit-image              0.11.3              np110py35_0  
seaborn                   0.7.1                    py35_0  
setuptools                23.0.0                   py35_0  
sqlalchemy                1.0.13                   py35_0  
SQLAlchemy                1.0.14                    <pip>
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
wheel                     0.29.0                   py35_0  
xz                        5.2.2                         0  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3  

and on 7001472

[root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:7001472997fa  conda info
Current conda install:

             platform : linux-64
        conda version : 4.1.11
    conda-env version : 2.5.2
  conda-build version : not installed
       python version : 3.5.2.final.0
     requests version : 2.10.0
     root environment : /opt/conda  (writable)
  default environment : /opt/conda
     envs directories : /opt/conda/envs
        package cache : /opt/conda/pkgs
         channel URLs : https://conda.anaconda.org/r/linux-64/
                        https://conda.anaconda.org/r/noarch/
                        https://repo.continuum.io/pkgs/free/linux-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /home/jovyan/.condarc
         offline mode : False
    is foreign system : False

root@jupyter-eldap ~]# docker run --rm -it docker.io/jupyter/datascience-notebook:7001472997fa  conda list
# packages in environment at /opt/conda:
#
bokeh                     0.11.1                   py35_0  
bzip2                     1.0.6                         3  
cairo                     1.12.18                       6  
cloudpickle               0.1.1                    py35_0  
conda                     4.1.11                   py35_0  
conda-env                 2.5.2                    py35_0  
backports.shutil_get_terminal_size 1.0.0                    py35_0    conda-forge
blas                      1.1                    openblas    conda-forge
cycler                    0.10.0                   py35_0    conda-forge
decorator                 4.0.10                   py35_0    conda-forge
dill                      0.2.5                    py35_0    conda-forge
entrypoints               0.2.2                    py35_0    conda-forge
h5py                      2.6.0               np110py35_6    conda-forge
hdf5                      1.8.17                        2    conda-forge
ipykernel                 4.3.1                    py35_1    conda-forge
ipython                   5.0.0                    py35_0    conda-forge
ipywidgets                5.1.5                    py35_0    conda-forge
jinja2                    2.8                      py35_1    conda-forge
jsonschema                2.5.1                    py35_0    conda-forge
jupyter_client            4.3.0                    py35_0    conda-forge
jupyter_core              4.1.0                    py35_0    conda-forge
libiconv                  1.14                          3    conda-forge
libpng                    1.6.23                        0    conda-forge
libsodium                 1.0.10                        0    conda-forge
markupsafe                0.23                     py35_0    conda-forge
mistune                   0.7.3                    py35_0    conda-forge
mpmath                    0.19                     py35_0    conda-forge
nbconvert                 4.2.0                    py35_0    conda-forge
nbformat                  4.0.1                    py35_0    conda-forge
notebook                  4.2.2                    py35_0    conda-forge
numpy                     1.10.4          py35_blas_openblas_202  [blas_openblas]  conda-forge
openblas                  0.2.18                        4    conda-forge
pandas                    0.18.1              np110py35_1    conda-forge
pexpect                   4.2.0                    py35_1    conda-forge
pickleshare               0.7.3                    py35_0    conda-forge
pillow                    3.2.0                    py35_0    conda-forge
prompt_toolkit            1.0.4                    py35_0    conda-forge
ptyprocess                0.5.1                    py35_0    conda-forge
pygments                  2.1.3                    py35_0    conda-forge
pyparsing                 2.1.7                    py35_0    conda-forge
python-dateutil           2.5.3                    py35_0    conda-forge
pytz                      2016.6.1                 py35_0    conda-forge
pyzmq                     15.3.0                   py35_1    conda-forge
scikit-learn              0.17.1          np110py35_blas_openblas_201  [blas_openblas]  conda-forge
scipy                     0.17.1          np110py35_blas_openblas_202  [blas_openblas]  conda-forge
simplegeneric             0.8.1                    py35_0    conda-forge
sip                       4.18                     py35_0    conda-forge
six                       1.10.0                   py35_0    conda-forge
statsmodels               0.6.1               np110py35_0    conda-forge
sympy                     1.0                      py35_0    conda-forge
terminado                 0.6                      py35_0    conda-forge
tornado                   4.4.1                    py35_1    conda-forge
traitlets                 4.2.2                    py35_0    conda-forge
wcwidth                   0.1.7                    py35_0    conda-forge
widgetsnbextension        1.2.6                    py35_3    conda-forge
zeromq                    4.1.5                         0    conda-forge
curl                      7.49.0                        1  
cython                    0.23.5                   py35_0  
fastcache                 1.0.2                    py35_1  
fontconfig                2.11.1                        6  
freetype                  2.5.5                         1  
icu                       54.1                          0  
ipython_genutils          0.1.0                    py35_0  
jbig                      2.1                           0  
jpeg                      8d                            1  
jupyterhub                0.5.0                     <pip>
libffi                    3.2.1                         0  
libgfortran               3.0.0                         1  
libtiff                   4.0.6                         2  
libxml2                   2.9.2                         0  
llvmlite                  0.8.0                    py35_0  
matplotlib                1.5.1               np110py35_0  
mkl                       11.3.3                        0  
networkx                  1.11                     py35_0  
numba                     0.23.1              np110py35_0  
numexpr                   2.5.2               np110py35_1  
openssl                   1.0.2h                        1  
pamela                    0.2.1                     <pip>
patsy                     0.4.1                    py35_0  
pcre                      8.39                          0  
pip                       8.1.2                    py35_0  
pixman                    0.32.6                        0  
pycosat                   0.6.1                    py35_1  
pycrypto                  2.6.1                    py35_4  
python                    3.5.2                         0  
pyyaml                    3.11                     py35_4  
glib                      2.43.0                        2    r
gsl                       1.16                          1    r
harfbuzz                  0.9.35                        6    r
libgcc                    4.8.5                         1    r
ncurses                   5.9                           8    r
pango                     1.36.8                        3    r
r                         3.3.1                  r3.3.1_0    r
r-assertthat              0.1                    r3.3.1_2    r
r-base                    3.3.1                         3    r
r-base64enc               0.1_3                  r3.3.1_0    r
r-bh                      1.60.0_2               r3.3.1_0    r
r-bitops                  1.0_6                  r3.3.1_2    r
r-boot                    1.3_18                 r3.3.1_0    r
r-car                     2.1_2                  r3.3.1_0    r
r-caret                   6.0_70                 r3.3.1_0    r
r-catools                 1.17.1                       1a    r
r-class                   7.3_14                 r3.3.1_0    r
r-cluster                 2.0.4                  r3.3.1_0    r
r-codetools               0.2_14                 r3.3.1_0    r
r-colorspace              1.2_6                  r3.3.1_0    r
r-curl                    0.9.7                  r3.3.1_0    r
r-dbi                     0.4_1                  r3.3.1_0    r
r-devtools                1.11.1                 r3.3.1_0    r
r-dichromat               2.0_0                        1a    r
r-digest                  0.6.9                  r3.3.1_0    r
r-dplyr                   0.4.3                  r3.3.1_0    r
r-evaluate                0.9                    r3.3.1_0    r
r-foreach                 1.4.3                  r3.3.1_0    r
r-forecast                7.1                    r3.3.1_0    r
r-foreign                 0.8_66                 r3.3.1_0    r
r-formatr                 1.4                    r3.3.1_0    r
r-fracdiff                1.4_2                  r3.3.1_2    r
r-ggplot2                 2.1.0                  r3.3.1_0    r
r-git2r                   0.15.0                 r3.3.1_0    r
r-gtable                  0.2.0                  r3.3.1_0    r
r-highr                   0.6                    r3.3.1_0    r
r-htmltools               0.3.5                  r3.3.1_0    r
r-httpuv                  1.3.3                  r3.3.1_0    r
r-httr                    1.2.0                  r3.3.1_0    r
r-irdisplay               0.3                    r3.3.1_2    r
r-irkernel                0.6.1                  r3.3.1_0    r
r-iterators               1.0.8                  r3.3.1_0    r
r-jsonlite                0.9.22                 r3.3.1_0    r
r-kernsmooth              2.23_15                r3.3.1_0    r
r-knitr                   1.13                   r3.3.1_0    r
r-labeling                0.3                    r3.3.1_2    r
r-lattice                 0.20_33                r3.3.1_0    r
r-lazyeval                0.2.0                  r3.3.1_0    r
r-lme4                    1.1_12                 r3.3.1_0    r
r-magrittr                1.5                    r3.3.1_2    r
r-markdown                0.7.7                  r3.3.1_2    r
r-mass                    7.3_45                 r3.3.1_0    r
r-matrix                  1.2_6                  r3.3.1_0    r
r-matrixmodels            0.4_1                  r3.3.1_0    r
r-memoise                 1.0.0                  r3.3.1_0    r
r-mgcv                    1.8_12                 r3.3.1_0    r
r-mime                    0.4                    r3.3.1_0    r
r-minqa                   1.2.4                  r3.3.1_2    r
r-munsell                 0.4.3                  r3.3.1_0    r
r-nlme                    3.1_128                r3.3.1_0    r
r-nloptr                  1.0.4                  r3.3.1_2    r
r-nnet                    7.3_12                 r3.3.1_0    r
r-nycflights13            0.2.0                  r3.3.1_0    r
r-openssl                 0.9.4                  r3.3.1_0    r
r-pbdzmq                  0.2_3                  r3.3.1_0    r
r-pbkrtest                0.4_6                  r3.3.1_0    r
r-plyr                    1.8.4                  r3.3.1_0    r
r-quadprog                1.5_5                  r3.3.1_2    r
r-quantreg                5.26                   r3.3.1_0    r
r-r6                      2.1.2                  r3.3.1_0    r
r-randomforest            4.6_12                 r3.3.1_0    r
r-rcolorbrewer            1.1_2                  r3.3.1_2    r
r-rcpp                    0.12.5                 r3.3.1_0    r
r-rcpparmadillo           0.7.100.3.1            r3.3.1_0    r
r-rcppeigen               0.3.2.8.1              r3.3.1_0    r
r-rcurl                   1.95_4.8               r3.3.1_0    r
r-recommended             3.3.1                  r3.3.1_0    r
r-repr                    0.7                    r3.3.1_0    r
r-reshape2                1.4.1                  r3.3.1_2    r
r-rmarkdown               0.9.6                  r3.3.1_0    r
r-rpart                   4.1_10                 r3.3.1_0    r
r-rsqlite                 1.0.0                  r3.3.1_2    r
r-rstudioapi              0.5                    r3.3.1_0    r
r-scales                  0.4.0                  r3.3.1_0    r
r-shiny                   0.13.2                 r3.3.1_0    r
r-sparsem                 1.7                    r3.3.1_0    r
r-spatial                 7.3_11                 r3.3.1_0    r
r-stringi                 1.1.1                  r3.3.1_0    r
r-stringr                 1.0.0                  r3.3.1_0    r
r-survival                2.39_4                 r3.3.1_0    r
r-tibble                  1.0                    r3.3.1_0    r
r-tidyr                   0.5.1                  r3.3.1_0    r
r-timedate                3012.100               r3.3.1_2    r
r-tseries                 0.10_35                r3.3.1_0    r
r-uuid                    0.1_2                  r3.3.1_0    r
r-whisker                 0.3_2                  r3.3.1_2    r
r-withr                   1.0.2                  r3.3.1_0    r
r-xtable                  1.8_2                  r3.3.1_0    r
r-yaml                    2.1.13                 r3.3.1_2    r
r-zoo                     1.7_13                 r3.3.1_0    r
rpy2                      2.8.1              py35r3.3.1_1    r
readline                  6.2                           2  
requests                  2.10.0                   py35_0  
ruamel_yaml               0.11.14                  py35_0  
scikit-image              0.11.3              np110py35_0  
seaborn                   0.7.1                    py35_0  
setuptools                23.0.0                   py35_0  
sqlalchemy                1.0.13                   py35_0  
SQLAlchemy                1.0.14                    <pip>
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
wheel                     0.29.0                   py35_0  
xz                        5.2.2                         0  
yaml                      0.1.6                         0  
zlib                      1.2.8                         3  

@karenyyng
Copy link

Besides the r-base version, there are several other packages that are changed between the two commits.

Since I do not think the other upgraded packages are related to the graphics capabilities, I think r-rcolorbrewer is the most suspicious of breaking the plotting capability.

$ docker run --rm -it docker.io/jupyter/datascience-notebook:7001472997fa  conda list > 7001472997fa.txt
$ docker run --rm -it docker.io/jupyter/datascience-notebook:54838ed4acb1 conda list > 54838ed4acb1.txt
$ diff 7001472997fa.txt 54838ed4acb1.txt
15,16d14
< h5py                      2.6.0               np110py35_6    conda-forge
< hdf5                      1.8.17                        2    conda-forge
42c40
< pyparsing                 2.1.7                    py35_0    conda-forge
---
> pyparsing                 2.1.1                    py35_0    conda-forge
63a62,63
> h5py                      2.5.0               np110py35_4
> hdf5                      1.8.15.1                      3
97c97
< r-base                    3.3.1                         3    r
---
> r-base                    3.3.1                         1    r
104c104
< r-catools                 1.17.1                       1a    r
---
> r-catools                 1.17.1                 r3.3.1_2    r
160c160
< r-rcolorbrewer            1.1_2                  r3.3.1_2    r
---
> r-rcolorbrewer            1.1_2                        1a    r

@parente
Copy link
Member

parente commented Sep 5, 2016

@takluyver any thoughts on what in the delta above might be causing capabilities() to now show jpeg, png, tiff, and cairo support all to be FALSE?

@jakirkham I tried removing the pin for libjpeg and installing jpeg9 from conda-forge, but to no avail. I get the same plot without title.

@parente
Copy link
Member

parente commented Sep 10, 2016

The change that broke plotting happened in r-base between 3.3.1-1 and 3.3.1-3.

As a workaround, conda install -c r r-base='3.3.1 1' gets things working again.

@parente
Copy link
Member

parente commented Sep 10, 2016

@mingwandroid do you have any thoughts on what might have changed between r-base 3.3.1 build number 1 (conda-archive/conda-recipes@601fd5d) and build number 3 (conda-archive/conda-recipes@dfe2d55) that could have flipped jpeg, png, and tiff capabilities from TRUE to FALSE as shown in #210 (comment)? I'm looking but not coming up with much.

parente added a commit to parente/docker-stacks that referenced this issue Sep 10, 2016
@mingwandroid
Copy link

On a clean install on latest ArchLinux my capabilities include all of those. Can you try the same and report back?

@parente
Copy link
Member

parente commented Sep 10, 2016

@mingwandroid, sure. Here's my bash history starting from a fresh docker debian container:

apt-get update
apt-get install wget
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.1.11-Linux-x86_64.sh
apt-get install bzip2
bash Miniconda3-4.1.11-Linux-x86_64.sh -b
/root/miniconda3/bin/conda install jupyter
/root/miniconda3/bin/conda install -c r r-base
. /root/miniconda3/bin/activate
R --quiet -e 'capabilities()'

Here's the output from the last command:

(root) root@13ae5375c38c:/# R --quiet -e 'capabilities()'
> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
      FALSE       FALSE       FALSE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE       FALSE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE

@parente
Copy link
Member

parente commented Sep 14, 2016

I pushed a new image with the older r-base build for the time being. It's tagged 54552d36cf0a and is currently latest.

If someone else can try it, that would be great.

@poplav
Copy link
Contributor

poplav commented Sep 14, 2016

This issue is becoming quite large, so to keep the fire going 😄 or at least make note of it.

It looks like currently this issue appears when a user tries to plot. Although, come the next IRkernel release/version this issue will appear on the first execution of a code cell such as 1+1 and kill the kernel instantly.
It will happen because of the changes this past May IRkernel/IRkernel#340 when initing null devices in the shadow environment commits are here and here. This initialization originates when the executor gets initialized which happens when the kernel is initialized. This is not yet in the 0.6.1 release that is in docker stacks to clarify.

This was discovered when running docker_dev, which is running master in this same docker stacks environment.

cc'ing @flying-sheep as you are familiar with this or at least making sure you are aware.

For reference the error when running 1+1 on master was:

Error in .External2(C_X11, paste("png::", filename, sep = ""), g$width,  : 
  unable to start device PNG
Calls: <Anonymous> ... evaluate -> dev.new -> do.call -> <Anonymous> -> ok_device
In addition: Warning message:
In ok_device(filename, ...) : unable to open connection to X11 display ''
Execution halted

In conclusion this is not a big issue yet, but will be later.

@flying-sheep
Copy link

wait: when exactly will this occur?

AFAIK only when there is an R system so extremely broken that a very basic call to a non-interactive device like pdf or png fails, right?

or am i wrong?

@ianabc
Copy link

ianabc commented Oct 6, 2016

@parente datascience-notebook:54552d36cf0a works for me. I can plot again!

@parente
Copy link
Member

parente commented Oct 7, 2016

@flying-sheep With irkernel 0.7 and conda r-base 3.3.1-3, I can crash the kernel by simply entering x <- 1 in a new notebook. The error on the console is the same as what @poplav stated above.

Switching back to r-base 3.3.1-1 resolves the issue. It seems that conda 3.3.1-3 build is the culprit again.

@flying-sheep
Copy link

thanks for the help, all of you!

see ContinuumIO/anaconda-issues#1063 for more

@parente
Copy link
Member

parente commented Oct 7, 2016

@flying-sheep Thanks for filing this upstream.

I'm going to close this (massive) issue now to avoid giving the impression that plotting is still an issue here. When the anaconda issue closes upstream, we can unpin the R version here.

@parente parente closed this as completed Oct 7, 2016
@mingwandroid
Copy link

mingwandroid commented Oct 21, 2016

@parente, would it be possible for you to test a new preview build of r-base?

conda update -c r r-essentials
conda update -c rdonnellyr r-base

Thanks!

@parente
Copy link
Member

parente commented Jan 13, 2017

PR #324 unpins jpeg and updates to r-base 3.3.2 which does not suffer from the original problems.

@ghost
Copy link

ghost commented Mar 7, 2019

Hi, I have exactly the same problem.
Hello,

I have problems creating charts in the jupyter notebook, sparmagic and R.

This is my set up:

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Db2 Warehouse
Matrix products: default
BLAS: /mnt/blumeta0/R-Install/lib64/R/lib/libRblas.so
LAPACK: /mnt/blumeta0/R-Install/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.5.2
Cairo packages:
cairo.x86_64 1.14.8-2.el7 @base
cairo-devel.x86_64 1.14.8-2.el7 @base
I use jupyter notebook with livy server and sparkmagic
After I installed cairo, I reinstalled R.

I used pip install everything
I invoke the notebook like this:
in the docker container:
jupyter notebook --ip=0.0.0.0 --allow-root
the steps:
options(bitmapType='cairo') --> output: SparkSession available as 'spark'.
library(ggplot2) --> output: null device 1
p <- ggplot(data.frame(x=1:5, y= 1:5, aes(x,y)) geom_point() ---> output: null device 1
p --> output: Error in (function = if (onefile) "Rplots.pdf" else "Rplots%3d.pdf"
cannot open file Rplots.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tag:Upstream A problem with one of the upstream packages installed in the docker images type:Bug A problem with the definition of one of the docker images maintained here
Projects
None yet
Development

No branches or pull requests