"ImportError: libgeos-3.4.2.so" on read the docs

[jorisvandenbossche]

The geopandas doc are built on read the docs using the feature to install packages with conda. But when I updated the build today, I got the following error from importing fiona (previously it worked correctly):

  File "/home/docs/checkouts/readthedocs.org/user_builds/geopandas/conda/latest/lib/python3.4/site-packages/geopandas-0.2+6.g46e50fe-py3.4.egg/geopandas/io/file.py", line 3, in <module>
    import fiona
  File "/home/docs/checkouts/readthedocs.org/user_builds/geopandas/conda/latest/lib/python3.4/site-packages/fiona/__init__.py", line 69, in <module>
    from fiona.collection import Collection, BytesCollection, vsi_path
  File "/home/docs/checkouts/readthedocs.org/user_builds/geopandas/conda/latest/lib/python3.4/site-packages/fiona/collection.py", line 7, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: libgeos-3.4.2.so: cannot open shared object file: No such file or directory

See for the build log: https://readthedocs.org/projects/geopandas/builds/4250850/

In the environment.yml file conda-forge is listed as channel (https://github.com/geopandas/geopandas/blob/master/doc/environment.yml). But, I am not sure if I can see if the fiona package is correctly fetched from conda-forge or is coming from the defaults channel.

I can also not reproduce this locally (on linux-64). In the build log, it can be seen that different packages are downgraded (eg jpeg is downgraded from 9b-0 to 8d-1) because of readthedocs installing certain versions of packages (eg pillow 3.0.0). Also if I replicate this behaviour locally, I don't get the error.

[ocefpaf]

I just checked with conda search -c conda-forge --platform linux-64 geos

geos                         3.3.3                         0  defaults        
                             3.4.2                         0  defaults        
                             3.4.2                         0  conda-forge/linux-64 
                          .  3.4.2                         1  conda-forge/linux-64 
                          *  3.4.2                         2  conda-forge/linux-64 
                             3.5.0                         0  defaults

And I noticed that defaults updated their geos to 3.5.0. We need to:

1. Pin geos in all of conda-forge to avoid this from happeneing again.
2. Update our geos 😄

[jorisvandenbossche]

So it's because of readthedocs specifyin pillow to 3.0 (conda-forge has pilow 3.2), that the one from defaults is installed, causing a whole rist of other packages to be up/downgraded, ending with a broken install:

(geopandas_release_test2) joris@joris-XPS-13-9350:~$ conda install pillow=3.0
Fetching package metadata ...........
Solving package specifications: ..........

Package plan for installation in environment /home/joris/miniconda3/envs/geopandas_release_test2:

The following NEW packages will be INSTALLED:

    geotiff:    1.4.1-0                      
    jbig:       2.1-0                        
    libgdal:    2.1.0-0                      
    pillow:     3.0.0-py34_1                 
    proj4:      4.9.2-0                      

The following packages will be UPDATED:

    fontconfig: 2.11.1-3          conda-forge --> 2.11.1-6         
    geos:       3.4.2-2           conda-forge --> 3.5.0-0          
    hdf4:       4.2.11-4          conda-forge --> 4.2.12-0         
    libnetcdf:  4.4.0-2           conda-forge --> 4.4.1-0          
    qt:         4.8.7-0           conda-forge --> 4.8.7-4          

The following packages will be DOWNGRADED due to dependency conflicts:

    cairo:      1.12.18-8         conda-forge --> 1.12.18-6        
    freetype:   2.6.3-1           conda-forge --> 2.5.5-1          
    gdal:       2.1.1-np111py34_1 conda-forge --> 2.1.0-py34_0     
    jpeg:       9b-0              conda-forge --> 8d-1             
    libtiff:    4.0.6-6           conda-forge --> 4.0.6-2          
    matplotlib: 1.5.2-np111py34_4 conda-forge --> 1.5.1-np111py34_0
    pango:      1.40.1-0          conda-forge --> 1.39.0-0         

But it can also easily be caused by doing a conda install geos=3.5, which does not know that fiona needs a specific version of geos.

Pinning geos to 3.4.2 in the environment.yml file does also solve it for now.

[ocefpaf]

So it's because of readthedocs specifyin pillow to 3.0 (conda-forge has pilow 3.2), that the one from defaults is installed, causing a whole rist of other packages to be up/downgraded, ending with a broken install

That is fine as long as everything is properly pinned. We never pinned geos b/c defaults had an old one and conda-forge's version was always chosen. Now we need to do it 😄

Pinning geos to 3.4.2 in the environment.yml file does also solve it for now.

That is fine but if we update to 3.5.0 you will need to change that too. (Don't worry b/c it won't happen anytime soon.)

I am fixing all the pinnings and, if I got all the packages, you won't need that. But it may take until tomorrow for me to finish re-building everything (mostly gdal which is always a pain).

[ocefpaf]

Should be fixed in #105 but let me know if that works for you.

Thanks @jorisvandenbossche for the report. I am working on a recipe for geos 3.5.0 soon.

[jorisvandenbossche]

Retriggered the doc build on rtd, and it's working now! Thanks @ocefpaf!

[ocefpaf]

Thanks for testing!

[fmaussion]

This pillow 3 problem is driving me crazy :( I know you guys aren't responsible for what's happening on RTD, but I can't seem to find a correct pinning strategy which works.

I have been able to pin my packages so that oggm now builds on RTD. Here is the environment.yml file:

name: oggm
channels:
  - conda-forge
dependencies:
  - python=3.5
  - numpy=1.12.0
  - scipy=0.18.1
  - gdal=2.1.0
  - matplotlib=2.0.0
  - pandas=0.19.2
  - geopandas=0.2.1
  - Pillow=3.0.0  # because RTD installs this version anyway
  - joblib=0.11
  - netCDF4=1.2.7
  - scikit-image=0.12.3
  - configobj
  - pyproj=1.9.5.1
  - krb5
  - geos=3.5.0 # https://github.com/conda-forge/fiona-feedstock/issues/21
  - rasterio=1.0a7
  - xarray=0.9.1
  - filelock
  - numpydoc
  - ipython=5.3.0
  - sphinx=1.3.5  # because RTD installs this version anyway

HOWEVER, now I'd like to add pyproj to this list. And this is where I can't find a suitable combination of packages version which works fine. Cartopy always complains about proj4 versions not being right, no matter which version I tried. I also tried to build cartopy via pip, but I get these kinds of errors:

Collecting cartopy==0.14.2
  Downloading Cartopy-0.14.2.tar.gz (8.4MB)
    Complete output from command python setup.py egg_info:
    /tmp/pip-build-xeomev86/cartopy/setup.py:174: UserWarning: Unable to determine GEOS version. Ensure you have 3.3.3 or later installed, or installation may fail.
      '.'.join(str(v) for v in GEOS_MIN_VERSION), ))
    Proj4 4.9.0 must be installed.

Any idea on how I could get this going? Any strategy or advice would be helpful...

[ocefpaf]

I have a few question to try to understand what is going on. (Bare in mind that I don't know anything about RTD.)

- Pillow=3.0.0 # because RTD installs this version anyway

Any idea why this happens? How does RTD install its pillow? pip or conda? Do they live in the same env or can you safely ignore it and install the latest one?

- geos=3.5.0 # https://github.com/conda-forge/fiona-feedstock/issues/21

This pin should no longer be needed.

Can you try a more relaxed version of your env? Something like:

name: oggm
channels:
  - conda-forge
dependencies:
  - python=3.5
  - numpy=1.12.0
  - scipy=0.18.1
  - geopandas=0.2.1  # let geopandas install mpl, pandas, gdal
  - joblib=0.11
  - netCDF4=1.2.7
  - scikit-image=0.12.3
  - configobj
  - cartopy
  - krb5
  - rasterio=1.0a7
  - xarray=0.9.1
  - filelock
  - numpydoc
  - ipython=5.3.0

[fmaussion]

Thanks @ocefpaf !

How does RTD install its pillow?

I don't know why they do this, but they do it after our installation. If you look at the build log, the first two commands are:

conda env create --name latest --file /home/docs/checkouts/readthedocs.org/user_builds/gis-conda-rtd-compat/checkouts/latest/docs/environment.yml
conda install --yes --name latest sphinx==1.3.5 Pygments==2.1.1 docutils==0.12 mock pillow==3.0.0 sphinx_rtd_the

So the first one is under our control, the second isn't...

I've made a toy project to test these issues in a simple environment:

• github: https://github.com/fmaussion/gis-conda-rtd-compat
• rtd builds: https://readthedocs.org/projects/gis-conda-rtd-compat/builds/

What I've tried so far:

• unpinning everything: doesn't work, because RTD wants to downgrade pillow after our successful installation.
• pinning pillow to 3.0.0: doesn't work, dependency issues:

UnsatisfiableError: The following specifications were found to be in conflict:
  - cartopy
  - geopandas
  - pillow 3.0.0*
Use "conda info <package>" to see the dependencies for each package.

• your suggestion: fails like above (log)

Should I fill an issue on RTD? My experience with the RTD folks is that they aren't very keen to continue conda support (in favor of docker), an I'm afraid that opening a new issue might comfort them in this direction. (see this issue for the latest large conda thread).

Would you be interested in getting commit rights to my test repo?

[ocefpaf]

@fmaussion I am starting to get it. The problem is that everything in conda-forge is built with jpeg 9* and we never had pillow 3.0.0 (we started at 3.2.0).

If the second conda call in RTD, that installs pillow 3.0.0, is preserving the conda-forge channel and channel preference config, we can solve this by adding the old pillow 3.0.0 to conda-forge.

I am away from home for a while but I can do this Monday and we can test it. Please bug me in case I forget.

[ocefpaf]

@fmaussion I got some free time and added pillow 3.0.0, with jpeg 9*, to conda-forge. Can you re-try RTD and let me know if that helps?

[fmaussion]

@ocefpaf brilliant, thanks so much! Everything now builds:

• doc build: http://gis-conda-rtd-compat.readthedocs.io/en/latest/
• environment.yml: https://github.com/fmaussion/gis-conda-rtd-compat/blob/master/docs/environment.yml

For the record, it is important to add pillow=3.0.0 in the projects environment.yml, otherwise the subsequent conda install by RTD will fail (I guess because they don't use conda-forge).

One last question: would you recommend to pin the packages for later builds, or leave them open? I traditionally pin them to a certain version which works (i.e. this one).

[ocefpaf]

@ocefpaf <https://github.com/ocefpaf> brilliant, thanks so much! No problem. Glad itis resolved. For the record, it is important to add pillow=3.0.0 in the projects environment.yml, otherwise the subsequent conda install by RTD will fail (I guess because they don't use conda-forge). Makes sense. The second command cannot know the channels used in the env file. So ensuring the right pillow is installed you avoid the channel issued. One last question: would you recommend to pin the packages for later builds, or leave them open? I traditionally pin them to a certain version which works (i.e. this one). Good question. My policy is to pin only when it is really needed. But pinning to a working set does make your life easier.