Azure Pipelines failing due to pip error

[stephenmcgruer]

At time of filing, the last ~10 runs on https://dev.azure.com/web-platform-tests/wpt/_build?definitionId=1&_a=summary fail multiple sub-commands due to:

/bin/bash --noprofile --norc /Users/runner/runners/2.164.7/work/_temp/39d98253-aff3-4b3a-b181-a39efc0fca53.sh
Traceback (most recent call last):
  File "/Users/runner/runners/2.164.7/work/1/s/_venv/bin/pip", line 6, in <module>
    from pip._internal.cli.main import main
ImportError: No module named main
CRITICAL:tools.wpt.utils:('/Users/runner/runners/2.164.7/work/1/s/_venv/bin/pip', 'install', '--prefer-binary', u'requests') exited with return code 1
CRITICAL:tools.wpt.utils:
Traceback (most recent call last):
  File "./wpt", line 12, in <module>
    wpt.main()
  File "/Users/runner/runners/2.164.7/work/1/s/tools/wpt/wpt.py", line 144, in main
    venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
  File "/Users/runner/runners/2.164.7/work/1/s/tools/wpt/wpt.py", line 122, in setup_virtualenv
    venv.install(name)
  File "/Users/runner/runners/2.164.7/work/1/s/tools/wpt/virtualenv.py", line 111, in install
    call(self.pip_path, "install", "--prefer-binary", *requirements)
  File "/Users/runner/runners/2.164.7/work/1/s/tools/wpt/utils.py", line 49, in call
    return subprocess.check_output(args)
  File "/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('/Users/runner/runners/2.164.7/work/1/s/_venv/bin/pip', 'install', '--prefer-binary', u'requests')' returned non-zero exit status 1

Not sure yet whether this is an Azure Pipelines problem or a WPT problem. cc @mustjab in case, and @foolip mentioned that @jgraham may have recently mentioned dealing with a problem like this elsewhere?

[foolip]

I don't see any recent changes on master that could obviously have caused this, but maybe the version of pip installed on Azure Pipelines changed.

The recent failure I remember had something to do with pypa/pip#7621 I think.

[foolip]

https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=41863&view=logs&j=bf8f5026-3d1f-5955-26be-17f9e1ef0bb3&t=ba27e54b-6052-5ce8-7e19-8f300f61f1be&l=19 shows that pip 19.3.1 is in use.

Unfortunately we don't have the version information for previous runs so we can't tell if it changed recently or not. However, it's doubtful it's because of pypa/pip#7621.

[foolip]

https://stackoverflow.com/q/60004431 had a similar error and involved pip in a virtualenv, which is also the case here.

@stephenmcgruer I'd suggest experimenting with installing or reinstalling pip to see if that has any effect.

[stephenmcgruer]

So 18a3e16 attempted to upgrade pip itself during the run. It was upgraded from pip 19.3.1 to pip-20.0.2.

That did not hit ImportError: No module named main but instead hit:

Successfully built wptrunner
Installing collected packages: wptrunner
  Attempting uninstall: wptrunner
    Found existing installation: wptrunner 1.14
    Can't uninstall 'wptrunner'. No files were found to uninstall.
  ERROR: Can't roll back wptrunner; was not uninstalled
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/requirements'
Consider using the `--user` option or check the permissions.

[stephenmcgruer]

Uninstalling and reinstalling pip appeared to have the same effect as upgrading it; permission denied trying to access /usr/local/requirements. It's not clear to me why its going after /usr/local/requirements, rather than .tox/py27/requirements/...

[stephenmcgruer]

@foolip I'm heading out of the office now, but the above contains the very little I've learnt over the last hour and a bit. #21697 has a azure pipelines run with a pip reinstall and lots of verbosity turned on in tox, but I still can't make much sense of it.

[foolip]

I'm trying #21722, it seemed to work on an experiment branch.