Can't build autofunction because unable to install a dependency (GDAL) listed in requirements.txt

[PratyushTripathy]

Details

I am trying to build documentation for my python package, which depends on another module "GDAL". When I list this in the requirements.txt, the build fails, without listing GDAL, the build passes. The problem really is that I can not use autofunction and linkcode if my module is not correctly installed in the virtual environment.

• Read the Docs project URL: https://readthedocs.org/projects/pyrsgis/
• Build URL (if applicable):
  when I add GDAL in the requirements.txt: https://readthedocs.org/projects/pyrsgis/builds/14320709/
  when I remove GDAL from requirements.txt: https://readthedocs.org/projects/pyrsgis/builds/14320681/
• Read the Docs username (if applicable): pratyush

Expected Result

I eventually want to build documentation such that my module's API pages on readthedocs can link to the source code. I do not really care if GDAL is installed in the virtual environment or not. Basically I want to know:

1. how I can install GDAL in the virtual environment so that autofunction works? Installation on PC is little tricky, one needs to install binaries separately and add it to the path.
2. if there is a way to use linkcode without installing all the dependencies of my module? (I know this doesn't sound right)

Actual Result

linkcode does not work because the module can not be imported because it is unable to install GDAL in the backend. On my local machine I am able to successfully build it because GDAL is installed--autofunction works on my PC build.

[astrojuanlu]

Hello @PratyushTripathy ! https://pypi.org/project/GDAL/ does not seem to provide precompiled Python wheels for any platform. This means that, to install it, you will also need libgdal from your operative system, which provides the gdal-config script that is missing in https://readthedocs.org/projects/pyrsgis/builds/14320709/:

    Could not find gdal-config. Make sure you have installed the GDAL native library and development headers.

To fix this issue, you can add libgdal-dev to the build.apt_packages key of your .readthedocs.yaml file (see more information in our blog).

However, I notice that you're specifying gdal>=3.0.0 in your requirements.txt. This will still lead to problems because our base image uses Ubuntu 18.04, which provides GDAL 2.2.3. Therefore, your options are either:

• Use conda to install your dependencies, rather than pip, or
• Make your package compatible with older versions of GDAL.

Let us know about your progress, happy to help.

[PratyushTripathy]

Hi @astrojuanlu,

Thank you for your prompt response. My package does not really rely on new functionalities of GDAL so I think v2.2.3 should just work fine. Thanks for pointing that out.

I specified the version in the requirements.txt and added the some packages under apt_packages in the `.readthedocs.yaml' file. I took whatever was seemed necessary from the blog link you shared (geoserver-rest). The build fails with a new error. https://readthedocs.org/projects/pyrsgis/builds/14323674/

Btw, the Use system packages in the advanced settings is checked. Hope that is okay.

Sorry if I am doing something silly, I am new to docs.

[astrojuanlu]

No need to apologize @PratyushTripathy :) You are not alone: https://gis.stackexchange.com/a/74060/99665

The problem is that, apparently, to fix the missing cpl_port.h issue you either need to add custom flags to pip (which can't be done at the moment) or custom environment variables (which can't be done at the moment either #6311 yes, it can be done but only from the UI, see below).

Therefore, my recommendation is two switch to conda, as explained in https://docs.readthedocs.io/en/stable/guides/conda.html

[astrojuanlu]

Sorry, I was mistaken @PratyushTripathy : you can set custom environment variables on https://readthedocs.org/dashboard/pyrsgis/environmentvariables/.

[PratyushTripathy]

It worked!!

I have been thinking about this for a while now. Had almost reached a point where I felt that this can not be solved and I have to let go the temptation.

You re-motivated me to build a good documentation. :)

Thank you so much!

[astrojuanlu]

Glad to hear! 🎉 Could you please write here what environment variables did you add to your UI, for future reference?

[PratyushTripathy]

I created two variables and did not check the Expose this environment variable in PR builds? box. These are the two variables:

Name: CPLUS_INCLUDE_PATH
Value: /usr/include/gdal

Name: C_INCLUDE_PATH
Value: /usr/include/gdal

[astrojuanlu]

Perfect! You might want to expose them in PR builds if you are accepting external contributions. Otherwise, installation will fail for PRs.

Closing.

[PratyushTripathy]

Hi @astrojuanlu,

This was working fine until today when the build failed. It says error in GDAL setup command: use_2to3 is invalid. In addition, I see the following:
ERROR: Could not find a version that satisfies the requirement gdal==2.2.3 (from versions: 1.5.0, 1.5.2, 1.6.0, 1.6.1, 1.7.0, 1.7.1, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.11.0, 1.11.1, 1.11.2, 2.0.0, 2.0.1, 2.1.0, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.2, 2.4.3, 2.4.4, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.3.0, 3.3.1, 3.3.2, 3.3.3)
ERROR: No matching distribution found for gdal==2.2.3

Is there a change in the GDAL version that you use in the base image? I built the documentation offline on my PC and it seems to be working fine.

[astrojuanlu]

According to OSGeo/gdal#4467, you'll have to pin setuptools

[PratyushTripathy]

I tried to pin setuptools, tried several older versions one by one (57.5.0, 57.4.0 and so on), none worked. Then I pinned it to the version that I have in PC (47.1.0 - it is quite old though) because the build is working fine on my PC, but that didn't work either. By the way, I am adding this only in the docs_requirements.txt file, hope that is okay?

[stsewd]

@PratyushTripathy for that to work, you'll need to have a separate requirements file with that dependency, and put it just above the one you have https://github.com/PratyushTripathy/pyrsgis/blob/8938fc3431d944c54bd3484f1132d047b30f1c5b/.readthedocs.yaml#L24-L25, that way rtd will first install that version of setuptools and then install the other requirements using that version of setuptools, otherwise it will keep installing all the requirements with the latest version of setuptools. Making it a two-step installation is needed to make it work.

[PratyushTripathy]

@stsewd It worked. Thank you! Here is the updated .readthedocs.yaml file.

[PratyushTripathy]

Hi @stsewd @astrojuanlu, just wanted to check if there is a change in setting up GDAL that readthedocs uses in the backend. The recent pyrsgis build failed. I have only committed this change since the last successful build and this commit doesn't look problematic to me.

[astrojuanlu]

Hi @PratyushTripathy , this has nothing to do with how RTD sets up GDAL:

AttributeError: module 'setuptools.dist' has no attribute 'invalid_unless_false'

But then, there's also this...

/home/docs/checkouts/readthedocs.org/user_builds/pyrsgis/envs/master/bin/python -m pip install --exists-action=w --no-cache-dir -r docs_requirements_setuptools.txt
Collecting setuptools==47.1.0
  Downloading setuptools-47.1.0-py3-none-any.whl (583 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Uninstalling setuptools-59.6.0:
      Successfully uninstalled setuptools-59.6.0
Successfully installed setuptools-58.2.0

😳

Please consider raising this to GDAL, pip, or setuptools.

[PratyushTripathy]

Removed the part that specified the setuptools version. Now the error is about the GDAL version. Tried a few other things related to specifying the GDAL version in the requirements.txt, that didn't work either.

[astrojuanlu]

@PratyushTripathy I think you're back to the initial error: old gdal is incompatible with newer setuptools. If you keep experiencing the setuptools installation errors, you should report them to pip or setuptools. Read the Docs uses the latest pip. There is not much else we can do for you, sorry.

[PratyushTripathy]

@astrojuanlu No worries. Thank you for the information. Will report the issue to pip/setuptools.