Python pip download action no longer builds local link into a wheel

[jpassaro]

Description:

Prior to version 20.3, pip download -r requirements.txt logged a message when requirements.txt would reference a directory:

Processing <directory>
  Link is a directory, ignoring download_dir

PipRunner used this log message to detect when a directory needed to be treated as an sdist and built into a wheel.

Unfortunately this logging is no longer present in pip version 20.3 and up, so the lambda builder no longer includes the local package as one of the artifacts.

Steps to reproduce the issue:
See aws/aws-sam-cli#2967 for the file setup.

Observed result:

$ pip --version
pip 21.1.2 from /Users/jpassaro/code/aws-lambda-builders/venv/lib/python3.8/site-packages/pip (python 3.8)
$ lambda-builders-dev '{"jsonschema":2,"id":"abcd","params":{"__protocol_version":"0.3","capability":{"language":"python","dependency_manager":"pip","application_framework":null},"source_dir":".","manifest_path":"./requirements.txt","artifacts_dir":"/tmp/lambda-build-pip-21.1.2/artifacts","scratch_dir":"/tmp/lambda-build-pip-21.1.2/scratch","runtime":"python3.8","optimizations":null,"mode":"debug","options":null},"method":"LambdaBuilder.build"}'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
{"jsonrpc": "2.0", "id": "abcd", "result": {"artifacts_dir": "/tmp/lambda-build-pip-21.1.2/artifacts"}}
$ ls -l /tmp/lambda-build-pip-21.1.2/artifacts
total 32
drwxr-xr-x   6 jpassaro  wheel   192 Jun 18 12:33 certifi
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:33 certifi-2021.5.30.dist-info
drwxr-xr-x  42 jpassaro  wheel  1344 Jun 18 12:33 chardet
drwxr-xr-x   9 jpassaro  wheel   288 Jun 18 12:33 chardet-3.0.4.dist-info
drwxr-xr-x  10 jpassaro  wheel   320 Jun 18 12:33 idna
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:33 idna-2.10.dist-info
-rw-r--r--   1 jpassaro  wheel   220 Jun 18 11:34 install-me.py
drwxr-xr-x  20 jpassaro  wheel   640 Jun 18 12:33 requests
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:33 requests-2.23.0.dist-info
-rw-r--r--   1 jpassaro  wheel   128 Jun 17 15:21 requirements.txt
-rw-r--r--   1 jpassaro  wheel   290 Jun 17 15:25 setup.cfg
-rw-r--r--   1 jpassaro  wheel    38 Jun 17 15:21 setup.py
drwxr-xr-x   3 jpassaro  wheel    96 Jun 18 12:33 src
drwxr-xr-x  16 jpassaro  wheel   512 Jun 18 12:33 urllib3
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:33 urllib3-1.25.11.dist-info

Expected result:

$ pip install -U pip==20.2.24
<snip>
$ lambda-builders-dev '{"jsonschema":2,"id":"abcd","params":{"__protocol_version":"0.3","capability":{"language":"python","dependency_manager":"pip","application_framework":null},"source_dir":".","manifest_path":"./requirements.txt","artifacts_dir":"/tmp/lambda-build-pip-20.2.4/artifacts","scratch_dir":"/tmp/lambda-build-pip-20.2.4/scratch","runtime":"python3.8","optimizations":null,"mode":"debug","options":null},"method":"LambdaBuilder.build"}'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
{"jsonrpc": "2.0", "id": "abcd", "result": {"artifacts_dir": "/tmp/lambda-build-pip-20.2.4/artifacts"}}
$ ls -l /tmp/lambda-build-pip-20.2.4/artifacts
total 32
drwxr-xr-x   6 jpassaro  wheel   192 Jun 18 12:31 certifi
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:31 certifi-2021.5.30.dist-info
drwxr-xr-x  42 jpassaro  wheel  1344 Jun 18 12:31 chardet
drwxr-xr-x   9 jpassaro  wheel   288 Jun 18 12:31 chardet-3.0.4.dist-info
drwxr-xr-x  10 jpassaro  wheel   320 Jun 18 12:31 idna
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:31 idna-2.10.dist-info
-rw-r--r--   1 jpassaro  wheel   220 Jun 18 11:34 install-me.py
drwxr-xr-x   4 jpassaro  wheel   128 Jun 18 12:31 mycode
drwxr-xr-x   6 jpassaro  wheel   192 Jun 18 12:31 mycode-0.0.0.dist-info
drwxr-xr-x  20 jpassaro  wheel   640 Jun 18 12:31 requests
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:31 requests-2.23.0.dist-info
-rw-r--r--   1 jpassaro  wheel   128 Jun 17 15:21 requirements.txt
-rw-r--r--   1 jpassaro  wheel   290 Jun 17 15:25 setup.cfg
-rw-r--r--   1 jpassaro  wheel    38 Jun 17 15:21 setup.py
drwxr-xr-x   3 jpassaro  wheel    96 Jun 18 12:31 src
drwxr-xr-x  16 jpassaro  wheel   512 Jun 18 12:31 urllib3
drwxr-xr-x   7 jpassaro  wheel   224 Jun 18 12:31 urllib3-1.25.11.dist-info

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Mac, python3.8, running lambda-builders-dev from dev installation of aws_lambda_builders

Debug log output with pip 20.2.4:

$ LAMBDA_BUILDERS_LOG_LEVEL=10 lambda-builders-dev '{"jsonschema":2,"id":"abcd","params":{"__protocol_version":"0.3","capability":{"language":"python","dependency_manager":"pip","application_framework":null},"source_dir":".","manifest_path":"./requirements.txt","artifacts_dir":"/tmp/lambda-build-pip-20.2.4/artifacts","scratch_dir":"/tmp/lambda-build-pip-20.2.4/scratch","runtime":"python3.8","optimizations":null,"mode":"debug","options":null},"method":"LambdaBuilder.build"}'
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r ./requirements.txt --dest /tmp/lambda-build-pip-20.2.4/scratch --exists-action i
calling pip wheel --no-deps --wheel-dir /tmp/lambda-build-pip-20.2.4/scratch /Users/jpassaro/code/aws-sam-bug-repro/code
Full dependency closure: {certifi==2021.5.30(wheel), mycode==0.0.0(wheel), requests==2.23.0(wheel), urllib3==1.25.11(wheel), chardet==3.0.4(wheel), idna==2.10(wheel)}
initial compatible: {certifi==2021.5.30(wheel), mycode==0.0.0(wheel), requests==2.23.0(wheel), urllib3==1.25.11(wheel), chardet==3.0.4(wheel), idna==2.10(wheel)}
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: {certifi==2021.5.30(wheel), mycode==0.0.0(wheel), requests==2.23.0(wheel), urllib3==1.25.11(wheel), chardet==3.0.4(wheel), idna==2.10(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {certifi==2021.5.30(wheel), mycode==0.0.0(wheel), requests==2.23.0(wheel), urllib3==1.25.11(wheel), chardet==3.0.4(wheel), idna==2.10(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {certifi==2021.5.30(wheel), mycode==0.0.0(wheel), requests==2.23.0(wheel), urllib3==1.25.11(wheel), chardet==3.0.4(wheel), idna==2.10(wheel)}
Final compatible: {requests==2.23.0(wheel), mycode==0.0.0(wheel), chardet==3.0.4(wheel), urllib3==1.25.11(wheel), certifi==2021.5.30(wheel), idna==2.10(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
{"jsonrpc": "2.0", "id": "abcd", "result": {"artifacts_dir": "/tmp/lambda-build-pip-20.2.4/artifacts"}}

debug output with pip 21.1.2 (note the call to pip wheel is missing):

$ LAMBDA_BUILDERS_LOG_LEVEL=10 lambda-builders-dev '{"jsonschema":2,"id":"abcd","params":{"__protocol_version":"0.3","capability":{"language":"python","dependency_manager":"pip","application_framework":null},"source_dir":".","manifest_path":"./requirements.txt","artifacts_dir":"/tmp/lambda-build-pip-21.1.2/artifacts","scratch_dir":"/tmp/lambda-build-pip-21.1.2/scratch","runtime":"python3.8","optimizations":null,"mode":"debug","options":null},"method":"LambdaBuilder.build"}'
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r ./requirements.txt --dest /tmp/lambda-build-pip-21.1.2/scratch --exists-action i
Full dependency closure: {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), urllib3==1.25.11(wheel), requests==2.23.0(wheel), idna==2.10(wheel)}
initial compatible: {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), urllib3==1.25.11(wheel), requests==2.23.0(wheel), idna==2.10(wheel)}
initial incompatible: set()
Downloading missing wheels: set()
compatible wheels after second download pass: {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), urllib3==1.25.11(wheel), requests==2.23.0(wheel), idna==2.10(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), urllib3==1.25.11(wheel), requests==2.23.0(wheel), idna==2.10(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), urllib3==1.25.11(wheel), requests==2.23.0(wheel), idna==2.10(wheel)}
Final compatible: {chardet==3.0.4(wheel), certifi==2021.5.30(wheel), requests==2.23.0(wheel), idna==2.10(wheel), urllib3==1.25.11(wheel)}
Final incompatible: set()
Final missing wheels: set()
PythonPipBuilder:ResolveDependencies succeeded
Running PythonPipBuilder:CopySource
PythonPipBuilder:CopySource succeeded
{"jsonrpc": "2.0", "id": "abcd", "result": {"artifacts_dir": "/tmp/lambda-build-pip-21.1.2/artifacts"}}

[jpassaro]

I found some logging in pip that seems to have replaced what's being missed here. Submitted a PR: #249