Using pip 23.1 produces invalid deployment file when package references other local packages

[nejcskofic]

Description:

I have python package which depends on another local python package (monorepo setup). Building deployment archive in python environment containing pip 23.1 produces archive which does not have local dependency. Downgrading pip to 23.0.x produces expected result.

Steps to reproduce:

• Create two python packages (package_a and package_b), package_b should depend on package_a
• Create requirements.txt file with package_b
• Create deployment archive using previously created requirements.txt (I'm using custom tool which directly uses aws-lambda-builders pypi package, but should be reproducible with sam build as well)

Observed result:

There are no errors. Resulting deployment package does not contain package_a (but does contain any additional dependencies package_a might have).

Expected result:

Resulting deployment package contains package_a.

Details

Based on debugging the root cause seems to be that pip download command changed output format. Setting output to debug I get the following output when running on pip 23.1 (only relevant portion shown):

calling pip download -r /tmp/tmpuads70j_/local_requirements.txt --dest /tmp/tmpuads70j_/deps --exists-action i
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/common (from core@ file:///<omitted>/projects/api/../core->api==2.25.1->-r /tmp/tmpuads70j_/local_requirements.txt (line 2))
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/api/dist/api-2.25.1-py3-none-any.whl (from -r /tmp/tmpuads70j_/local_requirements.txt (line 2))
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/core (from api==2.25.1->-r /tmp/tmpuads70j_/local_requirements.txt (line 2))

When I downgrade pip to 23.0.x, I get following output:

calling pip download -r /tmp/tmpuads70j_/local_requirements.txt --dest /tmp/tmpuads70j_/deps --exists-action i
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/common
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/api/dist/api-2.25.1-py3-none-any.whl
calling pip wheel --no-deps --wheel-dir /tmp/tmpuads70j_/deps ./projects/core

If I take only first command I get the following output for pip 23.1:

# pip download -r /tmp/tmpmsrvj9ii/local_requirements.txt --dest /tmp/tmpmsrvj9ii/deps --exists-action i
Processing <omitted>/projects/handlers/dist/handlers-2.25.1-py3-none-any.whl (from -r /tmp/tmpmsrvj9ii/local_requirements.txt (line 2))
Processing <omitted>/projects/core (from handlers==2.25.1->-r /tmp/tmpmsrvj9ii/local_requirements.txt (line 2))
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing <omitted>/projects/common (from core@ file:///<omitted>/projects/handlers/../core->handlers==2.25.1->-r /tmp/tmpmsrvj9ii/local_requirements.txt (line 2))
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

If run the same command on pip 23.0.x:

# pip download -r /tmp/tmpmsrvj9ii/local_requirements.txt --dest /tmp/tmpmsrvj9ii/deps --exists-action i
Processing ./projects/handlers/dist/handlers-2.25.1-py3-none-any.whl
Processing ./projects/core
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Processing ./projects/common
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

Culprit seems to be function download_all_dependencies, specifically this block:

        stdout = out.decode()
        wheel_package_paths = set()
        for pattern in self._LINK_IS_DIR_PATTERNS:
            for match in re.finditer(pattern, stdout):
                wheel_package_paths.add(str(match.group(1)))

        for wheel_package_path in wheel_package_paths:
            # Looks odd we do not check on the error status of building the
            # wheel here. We can assume this is a valid package path since
            # we already passed the pip download stage. This stage would have
            # thrown a PackageDownloadError if any of the listed packages were
            # not valid.
            # If it fails the actual build step, it will have the same behavior
            # as any other package we fail to build a valid wheel for, and
            # complain at deployment time.
            self.build_wheel(wheel_package_path, directory)

We are decoding lines that start with Processing and take whatever is after it to mean package path. This is then used to build wheel. With pip 23.1, content after Processing is no longer just a package path, which results in invalid package path which silently fails, producing deployment archive which is missing local package dependencies.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

I'm using Ubuntu 22.04.2 LTS, latest released aws-lambda-builders pypi package (1.30.0).

[lucashuy]

Thanks for making this detailed bug report! I can reproduce the issue you are experiencing where starting from pip 23.1.x, some more text is appended to the end of the package name, causing some parsing to fail. This was done using sam build, which like you suggested is affected since we consume the Lambda Builders library.