A CVE vulnerability that has been fixed by ubuntu, can still be found with trivy scanner.

[jellehelsen]

According to my trivy scan CVE-2022-40897 is still present in my newly built image, build with paketobuildpacks/builder-jammy-base. A fix was already released by Canonical a while ago (https://ubuntu.com/security/CVE-2022-40897) but the vulnerability can still be found.

According to you docs security updates are picked up from Canonical on mere hours. Why can this vulnerability can still be found?

[robdimsdale]

I think there's a few things going on here.

Firstly, we don't ship the python distribution that is provided by canonical - we compile from source (see here). So you'd actually need to look at the upstream source.

That being said, I'm assuming that the upstream cpython source has the fix which means we would expect to see that fix in the buildpack. The Python buildpacks provide the latest two patch releases of all supported minors.

Which then leads me to ask what version of the python buildpack are you actually using? You can see this information in the output of the build (e.g. pack build).

Once we have the version of the buildpack we can talk about specific versions of python that are present.

[jellehelsen]

Relevant part of the build log:

[builder] Paketo Buildpack for CPython 1.9.0
[builder]   Resolving CPython version
[builder]     Candidate version sources (in priority order):
[builder]       pyproject.toml -> "^3.11"
[builder]                      -> ""
[builder]       <unknown>      -> ""
[builder] 
[builder]     Selected CPython version (using pyproject.toml): 3.11.5
[builder] 
[builder]   Executing build process
[builder]     Installing CPython 3.11.5
[builder]       Completed in 3.331s

Is this the info you need?

[robdimsdale]

Yup, that's helpful, thanks. You also answered my next question I was going to ask - the version of python available.

So you can see that you're using Python 3.11.5 - the latest available.

I think in order to track down this CVE report you'll need to look at your scanner to determine which file on the filesystem is actually contributing to that scan.

We see a lot of false positives with scanners. I'm not necessarily saying (yet) that this is one of them, but I don't think the issue is an out of date version of Python.

[jellehelsen]

From inside the built container, showing the bad version.

cnb@2a0a352d462c:/$ ls -l /layers/paketo-buildpacks_cpython/cpython/lib/python3.11/site-packages/
total 32
-rw-r--r-- 1 1001 cnb  119 Jan  1  1980 README.txt
drwxr-xr-x 3 1001 cnb 4096 Jan  1  1980 _distutils_hack
-rw-r--r-- 1 1001 cnb  151 Jan  1  1980 distutils-precedence.pth
drwxr-xr-x 5 1001 cnb 4096 Jan  1  1980 pip
drwxr-xr-x 2 1001 cnb 4096 Jan  1  1980 pip-23.1.2.dist-info
drwxr-xr-x 5 1001 cnb 4096 Jan  1  1980 pkg_resources
drwxr-xr-x 8 1001 cnb 4096 Jan  1  1980 setuptools
drwxr-xr-x 2 1001 cnb 4096 Jan  1  1980 setuptools-65.5.0.dist-info

As it happens, I also use the poetry buildpack, and in the virtual enviroment that poetry creates the version of setuptools is way higher:

cnb@2a0a352d462c:/$ ls -l /layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages/ | grep setuptools
drwxr-xr-x 7 1001 cnb   4096 Jan  1  1980 setuptools
drwxr-xr-x 2 1001 cnb   4096 Jan  1  1980 setuptools-68.2.2.dist-info

So the vulnerable version is still installed, albeit not used.

Hope this helps.

[robdimsdale]

Ah, setuptools.

The only place in the buildpack toolchain that we explicitly interact with setup tools is in the pip buildpack, where we download it as part of the pre-compiled pip dependency.

When I look at the contents of the most recent pip dependency (https://artifacts.paketo.io/pip/pip_23.2.0_noarch_ba051f2f.tgz) I see the following:

 ❯ ls -alh | grep setuptools
-rw-r--r--   1 ubuntu ubuntu 2.1M Aug  2 14:11 setuptools-68.0.0.tar.gz

So if you're getting 65.5.0 then I'm guess either you're using an old version of the pip buildpack (which is a dependency of poetry) or it's a transitive dependency of your application

[jellehelsen]

I don't agree. I'm using the latest version of the pip buildpack (0.18.0). It being a dependency of my application does not really makes sense to me. All dependencies of my application are installed with poetry and are installed inside the poetry virtualenv. As I've shown in my previous message, the version of setuptools in the virtualenv is 68.2.2. Only the version in the system python is wrong.

Complete builder log:

[builder] Timer: Builder started at 2023-09-21T09:29:20Z
[builder] 
[builder] Paketo Buildpack for CA Certificates 3.6.3
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] Paketo Buildpack for CPython 1.9.0
[builder]   Resolving CPython version
[builder]     Candidate version sources (in priority order):
[builder]       pyproject.toml -> "^3.11"
[builder]                      -> ""
[builder]       <unknown>      -> ""
[builder] 
[builder]     Selected CPython version (using pyproject.toml): 3.11.5
[builder] 
[builder]   Executing build process
[builder]     Installing CPython 3.11.5
[builder]       Completed in 3.331s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_cpython/cpython
[builder]       Completed in 0s
[builder] 
[builder] 
[builder]   Configuring build environment
[builder]     PYTHONPATH          -> "/layers/paketo-buildpacks_cpython/cpython"
[builder]     PYTHONPYCACHEPREFIX -> "/tmp"
[builder] 
[builder]   Configuring launch environment
[builder]     PYTHONPATH -> "/layers/paketo-buildpacks_cpython/cpython"
[builder] 
[builder] Paketo Buildpack for Pip 0.18.0
[builder]   Resolving Pip version
[builder]     Candidate version sources (in priority order):
[builder]        -> ""
[builder] 
[builder]     Selected Pip version (using ): 23.2.1
[builder] 
[builder]   Executing build process
[builder]     Installing Pip 23.2.1
[builder]       Completed in 10.589s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_pip/pip
[builder]       Completed in 0s
[builder] 
[builder]   Configuring build environment
[builder]     PIP_FIND_LINKS -> "$PIP_FIND_LINKS /layers/paketo-buildpacks_pip/pip-source"
[builder] 
[builder]   Configuring build environment
[builder]     PYTHONPATH -> "/layers/paketo-buildpacks_pip/pip/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder]   Configuring launch environment
[builder]     PYTHONPATH -> "/layers/paketo-buildpacks_pip/pip/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder] Paketo Buildpack for Poetry 0.6.5
[builder]   Resolving Poetry version
[builder]     Candidate version sources (in priority order):
[builder]       <unknown> -> ""
[builder] 
[builder]     Selected Poetry version (using <unknown>): 1.4.2
[builder] 
[builder]   Executing build process
[builder]     Installing Poetry 1.4.2
[builder]       Completed in 10.939s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_poetry/poetry
[builder]       Completed in 0s
[builder] 
[builder]   Configuring build environment
[builder]     PYTHONPATH -> "/layers/paketo-buildpacks_poetry/poetry/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder]   Configuring launch environment
[builder]     PYTHONPATH -> "/layers/paketo-buildpacks_poetry/poetry/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder] Paketo Buildpack for Poetry Install 0.3.17
[builder]   Executing build process
[builder]     Running 'POETRY_CACHE_DIR=/layers/paketo-buildpacks_poetry-install/cache POETRY_VIRTUALENVS_PATH=/layers/paketo-buildpacks_poetry-install/poetry-venv poetry install'
[builder]       Creating virtualenv kubecost-function-xS3fZVNL-py3.11 in /layers/paketo-buildpacks_poetry-install/poetry-venv
[builder]       Installing dependencies from lock file
[builder]       
[builder]       Package operations: 15 installs, 0 updates, 0 removals
[builder]       
[builder]         • Installing six (1.16.0)
[builder]         • Installing certifi (2023.7.22)
[builder]         • Installing charset-normalizer (2.1.1)
[builder]         • Installing idna (3.4)
[builder]         • Installing iniconfig (2.0.0)
[builder]         • Installing packaging (23.1)
[builder]         • Installing pluggy (1.2.0)
[builder]         • Installing python-dateutil (2.8.2)
[builder]         • Installing typing-extensions (4.7.1)
[builder]         • Installing urllib3 (1.26.16)
[builder]         • Installing datadog-api-client (2.15.0)
[builder]         • Installing httpretty (1.1.4)
[builder]         • Installing markdown (3.4.4)
[builder]         • Installing pytest (7.4.0)
[builder]         • Installing requests (2.28.1)
[builder]       
[builder]       Installing the current project: kubecost-function (0.1.25)
[builder]       Completed in 4.786s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_poetry-install/poetry-venv
[builder]       Completed in 17ms
[builder] 
[builder]   Configuring build environment
[builder]     PATH                    -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/bin:$PATH"
[builder]     POETRY_VIRTUALENVS_PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv"
[builder]     PYTHONPATH              -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder]   Configuring launch environment
[builder]     PATH                    -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/bin:$PATH"
[builder]     POETRY_VIRTUALENVS_PATH -> "/layers/paketo-buildpacks_poetry-install/poetry-venv"
[builder]     PYTHONPATH              -> "/layers/paketo-buildpacks_poetry-install/poetry-venv/kubecost-function-xS3fZVNL-py3.11/lib/python3.11/site-packages:$PYTHONPATH"
[builder] 
[builder] Paketo Buildpack for Poetry Run 0.4.21
[builder]   Assigning launch processes:
[builder]     web (default): poetry run kubecost
[builder] 
[builder] 
[builder] Paketo Buildpack for Image Labels 4.5.2
[builder]   https://github.com/paketo-buildpacks/image-labels
[builder]   Build Configuration:
[builder]     $BP_IMAGE_LABELS                                                                       arbitrary image labels
[builder]     $BP_OCI_AUTHORS                                                                        the org.opencontainers.image.authors image label
[builder]     $BP_OCI_CREATED                                                                        the org.opencontainers.image.created image label
[builder]     $BP_OCI_DESCRIPTION                                                                    the org.opencontainers.image.description image label
[builder]     $BP_OCI_DOCUMENTATION                                                                  the org.opencontainers.image.documentation image label
[builder]     $BP_OCI_LICENSES                                                                       the org.opencontainers.image.licenses image label
[builder]     $BP_OCI_REF_NAME                                                                       the org.opencontainers.image.ref.name image label
[builder]     $BP_OCI_REVISION                                                                       the org.opencontainers.image.revision image label
[builder]     $BP_OCI_SOURCE         https://github.com/vlaamsemilieumaatschappij/kubecost-function  the org.opencontainers.image.source image label
[builder]     $BP_OCI_TITLE                                                                          the org.opencontainers.image.title image label
[builder]     $BP_OCI_URL                                                                            the org.opencontainers.image.url image label
[builder]     $BP_OCI_VENDOR                                                                         the org.opencontainers.image.vendor image label
[builder]     $BP_OCI_VERSION                                                                        the org.opencontainers.image.version image label
[builder]   Image labels:
[builder]     org.opencontainers.image.source
[builder] Timer: Builder ran for 29.790927[588](https://github.com/vlaamsemilieumaatschappij/kubecost-function/actions/runs/6259807481/job/16996552356#step:6:589)s and ended at 2023-09-21T09:29:49Z

[jellehelsen]

To make things more clear I've created a test repo @ https://github.com/jellehelsen/cpython-buildpack-test . No dependencies, just cpython-3.11. The only code is print("hello world"). Still the vulnerable setuptools version is used.
You can check the build log showing to error here.

[thitch97]

Hi @jellehelsen, I think I see what the issue is here. As you've shown here (& verified by downloading the source distribution for python 3.11.5 from here) the offending version of setuptools is shipped with v3.11.5 by default. There are ways to force the removal of that setuptools version, such as invoking pip install --force-reinstall, but as you are using poetry (and because there isn't currently any such cleanup logic in the buildpack), v65.5.0 sticks around in the cpython layer.

I downloaded python v3.12 and did not find a setuptools version in the same place as in v3.11.5, so I wonder whether simply upgrading to v3.12 would solve the scanning problem.

[robdimsdale]

TL;DR

1. Python 3.12 removes setuptools entirely, so this problem does not exist when building with 3.12 (or, presumably, 3.13+)
2. Python 3.11 and earlier has some version of setuptools bundled within it. We could potentially upgrade that to a later version, but doing so would be modifying the source distribution in a way that is outside of the scope of the buildpacks project (see below).

Why not upgrade setuptools on Python 3.11.x and earlier?

We've identified that the source distribution of python 3.1 (1and earlier) contains an old version of setuptools with known CVEs. We build from source, so we provide that version of setuptools. We've identified that python 3.12 contains a newer version of setuptools which doesn't have that CVE.

If you zoom out a level, what we're saying is:

• python 3.11.5 has some known CVEs
• python 3.12 has fixes for those CVEs

When I look at it that way, I don't think there's any action we should take. In general the buildpacks do not attempt to fix CVEs in upstream distributions of the dependencies. The buildpacks provide a lot of value to application developers and platform operators, but patching the upstream dependencies is outside of the scope of what we provide.

The fact that Canonical (and other re-distributors like Red Hat) have fixed this CVE in their distributions of python doesn't mean that we can take on a similar burden in the buildpacks. These organizations have far more resources available to them to test the patches they make to software they redistribute. We don't have the bandwidth to make a similar committment.

Setuptools on Python 3.12

Setuptools was removed from Python 3.12 (source):

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

Building with Python 3.12 results in the following:

 ❯ docker run --rm -it --entrypoint=launcher <resultant app image SHA> ls -alh /layers/paketo-buildpacks_cpython/cpython/lib/python3.12/site-packages/
total 20K
drwxr-xr-x  4 1001 cnb 4.0K Jan  1  1980 .
drwxr-xr-x 39 1001 cnb 4.0K Jan  1  1980 ..
-rw-r--r--  1 1001 cnb  119 Jan  1  1980 README.txt
drwxr-xr-x  5 1001 cnb 4.0K Jan  1  1980 pip
drwxr-xr-x  2 1001 cnb 4.0K Jan  1  1980 pip-23.2.1.dist-info

Upgrading setuptools as part of the build process on Python 3.10

After quite a bit of hacking around, I was able to run pip3 install --force-reinstall setuptools as part of the cpython build process, and it successfully upgraded the version of setuptools:

❯ docker run --rm -it --entrypoint=launcher <resultant image SHA> ls -alh /layers/paketo-buildpacks_cpython/cpython/lib/python3.10/site-packages/
total 40K
drwxr-xr-x  8 1001 cnb 4.0K Jan  1  1980 .
drwxr-xr-x 36 1001 cnb 4.0K Jan  1  1980 ..
-rw-r--r--  1 1001 cnb  119 Jan  1  1980 README.txt
drwxr-xr-x  2 1001 cnb 4.0K Jan  1  1980 _distutils_hack
-rw-r--r--  1 1001 cnb  151 Jan  1  1980 distutils-precedence.pth
drwxr-xr-x  5 1001 cnb 4.0K Jan  1  1980 pip
drwxr-xr-x  2 1001 cnb 4.0K Jan  1  1980 pip-23.0.1.dist-info
drwxr-xr-x  4 1001 cnb 4.0K Jan  1  1980 pkg_resources
drwxr-xr-x  7 1001 cnb 4.0K Jan  1  1980 setuptools
drwxr-xr-x  2 1001 cnb 4.0K Jan  1  1980 setuptools-68.2.2.dist-info

Unfortunately, this has two problems:

1. It won't work in an offline setting, as there's no source for the most recent version of setuptools inside the cpython distribution
2. It breaks caching, because we're modifying the filesystem in a way that cannot be predicted ahead of time and so violates the caching logic (i.e. we can't know what version of setuptools we're installing to).

We could potentially work around both of these issues by performing the pip install --force-reinstall logic during the compilation process of the cpython distribution, rather than during the build phase.

However, this gets into the question of what should the supported surface area of the buildpacks actually be, and I don't think we can commit to modifying the upstream source distribution.

Setuptools 65.5.0 in Python 3.11.5

Searching through the source of Python 3.11.5 for the string 65.5 shows us that Python 3.11.5 only contains setuptools v65.5.0. There is no later version provided.

❯ rg -i -. '65\.5'
Lib/ensurepip/__init__.py
13:_SETUPTOOLS_VERSION = "65.5.0"

<...>

Misc/NEWS
1736:  versions 22.3 and 65.5.0 respectively.

Some additional notes

In order to get the pip3 install --force-reinstall setuptools command working I had to rewrite the first line of the pip3 command to point to the correct location of the python3 interpreter. Otherwise you encounter the following error:

/layers/paketo-buildpacks_cpython/cpython/bin/pip3: /tmp/tmp.7dyEKmx8Dm/bin/python3.10: bad interpreter: No such file or directory

If you open pip3 (it's just a text file) you see:

#!/tmp/tmp.7dyEKmx8Dm/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

This /tmp path is the path that the cpython distribution was compiled under as part of the compilation process of the cpython distribution itself.

In order to invoke pip3 as part of the cpython build phase, we'll have to edit this file to point to the resultant location of the python distribution on the filesystem (i.e. /layers/paketo-buildpacks_cpython/cpython/bin/python3.

For the purposes of this investigation, I edited it during the build phase. We could/should make this change during the actual compilation if we wanted to invoke pip during the cpython build phase.

[jericop]

I have created a pull request to address this issue because there is no other clean way to address this CVE.

The main driver for this is to address a CVE that cannot be fixed by changing dependencies in requirement.txt files. The only way to get around this is issue is to modify the cpython layer with an inline buildpack (at build time) or to modify the container after build. Both are not good solutions.

• Why not upgrade setuptools on Python 3.11.x and earlier?
  • @robdimsdale says we should not do this and I completely agree.
  • My solution is not a patching approach, but rather a complete fix. It avoids the need for patching by removing setup tools entirely.
  • It also prevents cache issues by implementing the fix in the cpython buildpack directly.
• Setuptools on Python 3.12
  • Python stops shipping setuptools, and this change is in line with that.
• Some additional notes
  • The pip executables do not function, so my change just removes them so there are no broken scripts/executables in the final container image.

I agree that generally buildpacks shouldn't address CVEs, but this is a case where the CVE can't be addressed without modifying the buildpack or by modifying the image created with the buildpack. No changes to requirements.txt will prevent the CVE if you are using older versions of python.

[jericop]

To further demonstrate that this issue can only be addressed in the cpython bulidpack, I have built the following sample app and run the Trivy scan to show the CVE. Note that the sample app I'm building has no external (pip) dependencies and it installs the default version of cpython (3.10.x), which is impacted by the CVE.

git clone https://github.com/paketo-buildpacks/samples.git

pack build no-package-manager-sample --path samples/python/no_package_manager --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder-jammy-base

trivy image no-package-manager-sample --severity HIGH,CRITICAL --scanners vuln

Trivy scan output

2024-02-28T12:00:30.761-0500	INFO	Vulnerability scanning is enabled
2024-02-28T12:00:30.826-0500	INFO	Detected OS: ubuntu
2024-02-28T12:00:30.826-0500	INFO	Detecting Ubuntu vulnerabilities...
2024-02-28T12:00:30.827-0500	INFO	Number of language-specific files: 4
2024-02-28T12:00:30.827-0500	INFO	Detecting gobinary vulnerabilities...
2024-02-28T12:00:30.828-0500	INFO	Detecting python-pkg vulnerabilities...

no-package-manager-sample (ubuntu 22.04)

Total: 0 (HIGH: 0, CRITICAL: 0)

2024-02-28T12:00:30.829-0500	INFO	Table result includes only package filenames. Use '--format json' option to get the full path to the package file.

Python (python-pkg)

Total: 1 (HIGH: 1, CRITICAL: 0)

┌───────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────┐
│        Library        │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                         Title                         │
├───────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────┤
│ setuptools (METADATA) │ CVE-2022-40897 │ HIGH     │ 65.5.0            │ 65.5.1        │ pypa-setuptools: Regular Expression Denial of Service │
│                       │                │          │                   │               │ (ReDoS) in package_index.py                           │
│                       │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-40897            │
└───────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────┘