Once you've implemented supported for Chainguard Images, it's time to verify that your scanner is producing the correct results during image scans.
Chainguard provides special test images that are designed to help scanner developers validate their implementation. And these same images help Chainguard verify that a given scanner has successfully implemented support for Chainguard Images.
These images are organized into test cases.
We can measure the "readiness" of a scanner's support for Chainguard Images by testing if the scanner produces the expected the set of vulnerabilities for each specially crafted "test case".
Each test case has two variation of the same image — one with the wolfi distro ID, and one with the chainguard distro ID. These images have tags suffixed with -wolfi and -chainguard, respectively. Aside from this difference in distro identification, the two variations within each test case are equivalent, and should produce an identical set of vulnerabilities.
Each test case and its images are listed below, along with an explanation of why we've included that test case.
The "answer sheet" of the correct vulnerability IDs expected to be surfaced for each test case are stored in JSON in this repository, at data/answers.json.
ghcr.io/chainguard-images/scanner-test:no-software-wolfighcr.io/chainguard-images/scanner-test:no-software-chainguard
This image contains no executable software. The only packages installed are wolfi-baselayout and ca-certificates-bundle, which together provide a minimal directory structure and basic data files for the image, such as /etc/os-release and root CA certificates.
ghcr.io/chainguard-images/scanner-test:fixed-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:fixed-vulnerabilities-chainguard
This image includes a few software packages, like busybox and glibc, but none of the packages in this image currently have any known vulnerabilities. Some of these packages have had vulnerabilities in the past, but Chainguard has resolved the vulnerabilities and noted the fixes in the secdb.
ghcr.io/chainguard-images/scanner-test:fixed-language-package-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:fixed-language-package-vulnerabilities-chainguard
This image includes a distro package (ko) that includes several language ecosystem packages (Go modules, in this case). These language ecosystem packages have been the source of vulnerabilities in the past, but those have been resolved in the current version of the distro package.
ghcr.io/chainguard-images/scanner-test:false-positive-language-package-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:false-positive-language-package-vulnerabilities-chainguard
This image is similar to :fixed-language-package-vulnerabilities-wolfi, but in this case, there have been false positives reported against the language ecosystem packages contained within the distro package. Those false positives are disclaimed in Chainguard's security feeds, and scanners should consequently not surface the false positives if they would have otherwise.
ghcr.io/chainguard-images/scanner-test:false-positives-from-lock-files-wolfighcr.io/chainguard-images/scanner-test:false-positives-from-lock-files-chainguard
This image is related to :false-positive-language-package-vulnerabilities, in that it tests if a scanner is surfacing avoidable false positives. But this image focuses on false positives that occur when scanners try to use "lock files" (e.g. gemfile.lock, package-lock.json, etc.) as evidence that a language package is installed. Using lock files is often a flawed strategy for package detection because lock files tend to include references to software packages that have not actually been installed. Vulnerabilities matched against these "not actually installed" packages should not be reported to users.
ghcr.io/chainguard-images/scanner-test:subpackage-fixed-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:subpackage-fixed-vulnerabilities-chainguard
This image contains the latest version of libcrypto3, which is a subpackage of openssl. While there is vulnerability data about openssl in the secdb that applies to libcrypto3, the installed version of libcrypto3 is the latest version and contains patches for all vulnerabilities listed in the secdb entry.
ghcr.io/chainguard-images/scanner-test:unfixed-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:unfixed-vulnerabilities-chainguard
This image contains an intentionally outdated version of glibc, to see if scanners recognize that there's a vulnerability in this package that's fixed in a later version of the package.
ghcr.io/chainguard-images/scanner-test:unfixed-vulnerabilities-chainguard-chainguard
This image contains an intentionally outdated version of aws-ebs-csi-driver-1.19, to see if scanners recognize that there's a vulnerability in this package that's fixed in a later version of the package.
Unlike in the previous unfixed-vulnerabilities test images, the package only exists in Chainguard's package repository, and has never existed in Wolfi. This means that there is no data at all about this package in Wolfi, and all information about the package is only in Chainguard's advisory feed.
ghcr.io/chainguard-images/scanner-test:subpackage-unfixed-vulnerabilities-wolfighcr.io/chainguard-images/scanner-test:subpackage-unfixed-vulnerabilities-chainguard
This image contains an intentionally outdated version of libcrypto3. libcrypto3 is a subpackage of openssl, and the relevant secdb data is filed under openssl. This image tests that scanners understand how to relate subpackages to origin package data to find unfixed vulnerabilities.
Note: While the intent is to keep this list of images and their vulnerabilities as correct as possible, new vulnerabilities are published frequently, and there may be cases where a novel CVE correctly shows in a scan result even though it's not listed here. This data was last verified on October 7, 2024, so treat scan result items for vulnerabilities published after this date with caution.
Feedback is welcome! If you have suggestions for test cases that would be useful to you, please feel free to file an issue. Better collaboration leads to higher quality scans. 🚀