Bug: UnsupportedPackageError when tests run under setuptools 66+

[jfly]

Description:

On setuptools 66+, I see some tests in tests/functional/workflows/python_pip/test_packager.py::TestSdistMetadataFetcher start to fail with a aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: sdist error.

This is because some a few tests set up packages with version numbers that do not conform to PEP 440, and setuptools 66 dropped support for non-conforming versions: https://setuptools.pypa.io/en/stable/history.html#v66-0-0

Steps to reproduce:

Simply install the latest version of setuptools (67.6.0 at time of writing) and run the relevant tests:

$ pip install -U setuptools==67.6.0
Requirement already satisfied: setuptools in ./.direnv/python-3.10.10/lib/python3.10/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-67.6.0-py3-none-any.whl (1.1 MB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
Successfully installed setuptools-67.6.0

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

$ pytest tests/functional/workflows/python_pip/test_packager.py::TestSdistMetadataFetcher::test_both_tar_bz2
=================================================== test session starts ====================================================
platform linux -- Python 3.10.10, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/jeremy/src/github.com/aws/aws-lambda-builders
plugins: cov-4.0.0
collected 1 item

tests/functional/workflows/python_pip/test_packager.py F                                                             [100%]

========================================================= FAILURES =========================================================
________________________________________ TestSdistMetadataFetcher.test_both_tar_bz2 ________________________________________

self = <tests.functional.workflows.python_pip.test_packager.TestSdistMetadataFetcher object at 0x7f3141c82710>
osutils = <aws_lambda_builders.workflows.python_pip.utils.OSUtils object at 0x7f3141c82ce0>
sdist_reader = <aws_lambda_builders.workflows.python_pip.packager.SDistMetadataFetcher object at 0x7f3141c82da0>

    def test_both_tar_bz2(self, osutils, sdist_reader):
        setup_py = self._SETUP_PY % (self._BOTH, "foo-bar", "1.0-2b")
        with osutils.tempdir() as tempdir:
            filepath = self._write_fake_sdist(setup_py, tempdir, "tar.bz2")
>           name, version = sdist_reader.get_package_name_and_version(filepath)

tests/functional/workflows/python_pip/test_packager.py:958:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
aws_lambda_builders/workflows/python_pip/packager.py:625: in get_package_name_and_version
    pkg_info_filepath = self._get_pkg_info_filepath(package_dir)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aws_lambda_builders.workflows.python_pip.packager.SDistMetadataFetcher object at 0x7f3141c82da0>
package_dir = '/run/user/1000/tmps9vqsywg/sdist'

    def _get_pkg_info_filepath(self, package_dir):
        setup_py = self._osutils.joinpath(package_dir, "setup.py")
        script = self._SETUPTOOLS_SHIM % setup_py

        cmd = [sys.executable, "-c", script, "--no-user-cfg", "egg_info", "--egg-base", "egg-info"]
        egg_info_dir = self._osutils.joinpath(package_dir, "egg-info")
        self._osutils.makedirs(egg_info_dir)
        p = subprocess.Popen(
            cmd, cwd=package_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=self._osutils.original_environ()
        )
        _, stderr = p.communicate()
        info_contents = self._osutils.get_directory_contents(egg_info_dir)
        if p.returncode != 0:
            LOG.debug("Non zero rc (%s) from the setup.py egg_info command: %s", p.returncode, stderr)
        if info_contents:
            pkg_info_path = self._osutils.joinpath(egg_info_dir, info_contents[0], "PKG-INFO")
        else:
            # This might be a pep 517 package in which case this PKG-INFO file
            # should be available right in the top level directory of the sdist
            # in the case where the egg_info command fails.
            LOG.debug("Using fallback location for PKG-INFO file in package directory: %s", package_dir)
            pkg_info_path = self._osutils.joinpath(package_dir, "PKG-INFO")
        if not self._osutils.file_exists(pkg_info_path):
>           raise UnsupportedPackageError(self._osutils.basename(package_dir))
E           aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package: sdist

aws_lambda_builders/workflows/python_pip/packager.py:608: UnsupportedPackageError
================================================= short test summary info ==================================================
FAILED tests/functional/workflows/python_pip/test_packager.py::TestSdistMetadataFetcher::test_both_tar_bz2 - aws_lambda_builders.workflows.python_pip.packager.UnsupportedPackageError: Unable to retrieve name/version for package:...
==================================================== 1 failed in 0.14s =====================================================

[moelasmar]

thanks @jfly for raising this issue and for your participation. I reviewed your PR and looks good to me.