BackendUnavailable while building the documentation

[ziirish]

Details

• Read the Docs project URL: https://readthedocs.org/projects/burp-ui
• Build URL (if applicable): https://readthedocs.org/projects/burp-ui/builds/8509365/
• Read the Docs username (if applicable): ziirish

Expected Result

The documentation used to build without any issue.

Actual Result

For a few weeks, every build attempt fails in an early step whereas nothing changed in my requirements.

Here is the failing step:

/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/bin/python -m pip install --exists-action=w --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/burp-ui/.cache/pip -r requirements.txt

And here is the traceback:

Collecting trio==0.10.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/e6/20/37be7b5f47db6a9fbf905b5de5386e5b7193c45d07becb750db6f03cd117/trio-0.10.0.tar.gz (402kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
Exception:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 349, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
pip._vendor.pep517.wrappers.BackendUnavailable

The problem started with the commit 2618ba00be7ed7a2fa483dfc21bfb7bb46038d90 of my project. As you can see here there aren't any change in the requirements between the last known functioning commit (2c1b4bb7efeaeea2dbdef2e0c7096ac4f534b020) and this one.

During my early troubleshooting I found those issues that might be related: pypa/setuptools#1644 and pypa/pip#6164
Those issues are talking about both pip 19.0 and python <=3.6.5.
I noticed the readthedocs builder run python 3.6.4.

I'm using the following readthedocs.yml:

# https://docs.readthedocs.io/en/latest/yaml-config.html
build:
  image: latest
python:
  version: 3.6
  pip_install: true
  extra_requirements:
    - rtd

Let me know if you need further details.
Thanks

[stsewd]

Can you try wiping the enviroment https://docs.readthedocs.io/en/stable/guides/wipe-environment.html?

Also, if this bug isn't present in python 3.5 or 3.7, you can use a different version https://docs.readthedocs.io/en/stable/config-file/v2.html#python

(note that is using the v2 of the config file)

[ziirish]

Thanks for the hints.

The wipe trick didn't solve the issue. I'll have a look at python 3.7 then config v2.

I cannot try 3.5 though because I'm using format strings which were introduced in 3.6.
I couldn't use 3.7 because I had variables called async which is a reserved keyword since 3.7 but this issue should be fixed by now.

[stsewd]

You can also use a previous version of pip that was working with that version of python. You can create a requirements file with pip only and put it before the requirements of your project https://docs.readthedocs.io/en/stable/config-file/v2.html#requirements-file

[stsewd]

Also, we install the latest version of pip by default, so if the problem is solved in the pip side, it would be fixed here as soon as they make a new release.

[ziirish]

I have updated my .readthedocs.yml to the v2 format and I'm now using python 3.7 instead of 3.6 but I am hitting a new issue still in the early build stages.

Config:

# https://docs.readthedocs.io/en/latest/yaml-config.html
version: 2
build:
  image: latest
python:
  version: 3.7
  install:
    - method: pip
      path: .
      extra_requirements:
        - rtd

Traceback:

Processing /home/docs/checkouts/readthedocs.org/user_builds/burp-ui/checkouts/latest
Collecting trio==0.10.0 (from burp-ui==0.7.0.dev0)
  Using cached https://files.pythonhosted.org/packages/e6/20/37be7b5f47db6a9fbf905b5de5386e5b7193c45d07becb750db6f03cd117/trio-0.10.0.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  Complete output from command /home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/bin/python /home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmprmehga8o:
  Traceback (most recent call last):
    File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 48, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  
  ----------------------------------------
Command "/home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/bin/python /home/docs/checkouts/readthedocs.org/user_builds/burp-ui/envs/latest/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmprmehga8o" failed with error code 1 in /tmp/pip-install-y4lpxq8a/trio

It looks like this issue has been fixed in setuptools v40.8.0
I'll try to add another requirement forcing that version of setuptools prior to installing the rest of my dependencies.

[ziirish]

Adding an explicit setuptools==40.8.0 seems to have fixed the above error for python 3.7.

[stsewd]

Thanks, looks like we need to install the latest version of setuptools to play nice with the latest version of pip. I'll do a PR soon.

[ziirish]

Awesome! Thanks for your hard work.