[Discussion] Findings on Discrepancy Assessments within the SBOM Ecosystem.

[dw763j]

Assessments results on discrepancy of SBOM ecosystem and some suggestions

Background

As SBOM can be widely used in software software chain management, the capability and issues within SBOM ecosystem can influence the employment of users, thus accurately assessments of the current SBOM state is important. To this end, we have conducted a series of assessments on key characteristics in SBOM applications to reveal the potential discrepancies hindering usage.

Questions

We asked 3 questions:
1. Compliance: Do SBOM tools generate outputs that adhere to user requirements and standards?
2. Consistency: Do SBOM tools maintain consistency in transforming the produced SBOM?
3. Accuracy: How accurate are the SBOM produced by tools in reflecting the objective software?

Upon 9970 SBOM documents generated from 6 SBOM tools (sbom-tool, ort, syft, gh-sbom, cdxgen and scancode) in both SPDX and CycloneDX on 1162 GitHub repositories, we assess these questions. To evaluate accuracy, 100 repositories are annotated for benchmark, comprising 660 components and 4,000 data fields.

Results

This table shows average results across all the 6 tools, results are all in package level. Note that in the results for information of software itself is quite poor, for instance, we have 89.59% repositories contain licenses while only a minority are identified.

Attr.	pkg_name	version	author	purl	license	copyright
Compliance	79.61%	74.99%	17.84%	67.53%	32.34%	14.17%
Consistency	18.44%	22.24%	0.11%	24.99%	2.12%	-
Accuracy	25.81%	10.66%	4.94%	-	10.66%	-

The findings indicate that while SBOM tools 100% support mandatory standards requirements (including Doc.: specVersion, License, Namespace, Creator; Comp.: Name, Identifier, downloadLocation, verificationCode), their performance in user case support is at 49.37% and the consistency within these supported use cases is on average of 17.63% (as the table shows). Accuracy assessments reveal significant discrepancies, with accuracy rates of 8.62%, 25.81%, and 12.3% as in software metadata, identified denpendent components, and detailed component information, underscoring substantial areas for improvement within the SBOM ecosystem.

Suggestions

1. In component sections, some tools record the package name with their information sources like pip, maven, npm, etc., while others do not. In version tools varing in recording like whether add a 'V' before the version string this will lead to problems in utilizing SBOM from different SBOM tools. We suggest to require tools to specify their pattern in recoring information without the standard's explicit specification.
2. The meaning of NOASSERTION ,NONE and Nonecould be confusing in specific data fields. For instance, version can naturally be empty in packages as the developers didn't record them in the software, tools deal empty ones into empty string or the three forms, which could lead to inconsistency for further exchange. We suggest to provide specific marks for these natually empty data fields.
3. For hashes, we found that in different tools that using the same hash algorithm on the same single file have different checksums in SPDX, there is even no consistent checksums across all the software and packages. While in CycloneDX, the hashes even does not specify the object the hash is performed on. We suggest to demand tools in creating checksums explicitly illustrate their process for creating the checksums, e.g. salt value or other preprocessing.

We hope our findings can help promote the SBOM ecosystem, any questions or discussions are welcomed.

Fast check on code

We provide a fast check code at here based on part of our dataset.

Examples:

For checksum, here are examples that the file, hash algorithm are both matched, yet they still didn's get the same checksums:

[jkowalleck]

some example idea to showcase threads.

New threads can be created by scrolling all to the bottom and starting a comment.

New replies to existing treads can be added by writing a reply.

[jkowalleck]

some example answer to showcase threads

[jkowalleck]

This thread is not intended to receive any answers — it is just a showcase how threads might be used :-)

if you have something to say, that does not has a thread yet, open a new one in this conversation.

[jkowalleck]

In version tools varing in recording like whether add a 'V' before the version string this will lead to problems in utilizing SBOM from different SBOM tools. We suggest to require tools to specify their pattern in recoring information without the standard's explicit specification.

I suggest: use the actual version that applies, and do not “normalize”(modify) in any means.
And here is why:

For the PHP ecosystem composer, a version prefix v might be transparent. Meaning v1.33.7 might be treated as the same as 1.33.7 on callout, only if the original requested version does not exist. Internally, it might normalize the version, and still it honors the actual (deducted) version.

Furthermore, In this case, the v is not a simple prefix, but has actual meaning.

Usage Example:
If one runs composer require 'my-package-a:v1.33.7' and it has no such version but a 1.33.7, it will switch to it and lock to version 1.33.7.
If one runs composer require 'my-package-b:1.33.7' and it has no such version but a v1.33.7, it will switch to it and lock to version v1.33.7.
If one runs composer require 'my-package-c:v1.33.7' and it has such version v1.33.7 and also 1.33.7, it will use version v1.33.7 and lock to it.
If one runs composer require 'my-package-c:1.33.7' and it has such version 1.33.7 and also v1.33.7, it will use version 1.33.7 and lock to it.

Real-world Example:

• https://packagist.org/packages/cyclonedx/cyclonedx-library uses versions like v3.2.0 - which is the real version name with an actual v prefix.
• https://packagist.org/packages/phpunit/phpunit uses versions like 11.1.1 - which is the real version name without any v prefix.

[nscuro]

Similar story for Go (golang/go#32945). The prefix has a meaning, and mixing prefixed with non-prefixed version creates more ambiguity than it solves.

In Go, requesting a module with v1.0.0 will work, but 1.0.0 will not. Why include a broken version in the BOM?

[dw763j]

Similar story for Go (golang/go#32945). The prefix has a meaning, and mixing prefixed with non-prefixed version creates more ambiguity than it solves.

I don't mean to change the version, as in the SBOMs, some SBOM tools indeed change it ,version of github/actions in gh-sbom are all removed v, for instance. My suggestion is provide a description field for recording their operations.

[nscuro]

Upon 9970 SBOM documents generated from 6 SBOM tools (sbom-tool, ort, syft, gh-sbom, cdxgen and scancode) in both SPDX and CycloneDX on 1162 GitHub repositories, we assess these questions.

If you are going to do these kinds of assessments, and advertise their results, you really need to provide more information on the methodology used. I mean down to the actual CLI commands you ran, and the repositories you ran them against. You should also publish the raw results so people can verify your results.

Many of these tools have a wide range of flags and options that can impact results in one way or another. Trade-offs with those options are usually documented. "We ran tool X" is not sufficient information.

[...] for instance, we have 89.59% repositories contain licenses while only a minority are identified

So the tools didn't identify the licenses, but how did you? Did you use automation? Was it a manual review? Were those licenses well-known ones from the SPDX catalogue, or custom ones?

Accuracy assessments reveal significant discrepancies [...]

How was that assessment done across the 9970 repositories? Did you do it manually? How did you verify that something is accurate?

[jkowalleck]

@nscuro this actually looks like 3 different questions/topics. Answering all three in one single thread might become messy. maybe you could split your questions?

[nscuro]

@jkowalleck I guess I'm less asking for answers to specific questions, and more pointing out that the presented results severely lack foundation.

[dw763j]

If you are going to do these kinds of assessments, and advertise their results, you really need to provide more information on the methodology used. I mean down to the actual CLI commands you ran, and the repositories you ran them against. You should also publish the raw results so people can verify your results.

We are working on reorganizing our codes and dataset to release it.

How was that assessment done across the 9970 repositories? Did you do it manually? How did you verify that something is accurate?

Sorry for the misunderstanding, I missed the introduce of benchmark, it is a 100 size repos that created based on PyPI and GitHub with some further mannually deal.

[dw763j]

If you are going to do these kinds of assessments, and advertise their results, you really need to provide more information on the methodology used. I mean down to the actual CLI commands you ran, and the repositories you ran them against. You should also publish the raw results so people can verify your results.

Many of these tools have a wide range of flags and options that can impact results in one way or another. Trade-offs with those options are usually documented. "We ran tool X" is not sufficient information.

Our assessments are conducted in python environment, with github repositories as the input dirs for tools to analysis. To ensure accurate evaluations across all SBOM tools used, we standardized the SBOM generation process. This involved setting the input as the directory and configuring the tools to generate SBOMs in accordance with SBOM standard based on the official guidelines of each tool's documents.

For consistency, we predominantly exported the documents in JSON format.

[...] for instance, we have 89.59% repositories contain licenses while only a minority are identified

So the tools didn't identify the licenses, but how did you? Did you use automation? Was it a manual review? Were those licenses well-known ones from the SPDX catalogue, or custom ones?

The number of 89.59% is based on results of GitHub API. We did construct a benchmark with both automation and manual review for missing details.

How was that assessment done across the 9970 repositories? Did you do it manually? How did you verify that something is accurate?

The compliance and consistency are across the 9970 SBOMs, while the accuracy is contucted based on the 100 annotated repositories.

[pombredanne]

I do not understand what this discussion is about.

• Which study is this? URL!
• Where are the data? No data, no discussion

This post feels to me like being both confused and confusing.

Me wants data!

[dw763j]

We have released some fast check code and test dataset to support further analysis.

[pombredanne]

@dw763j Do you have an URL?

[dw763j]

Fast check on code

We provide a fast check code at here based on part of our dataset.

Here.

[prabhu]

I am yawning. cdxgen v9 was used instead of v10. Also, there are a number of parameters and prerequisites that needs to be followed correctly. Let me know if you're interested in a proper evaluation which requires a payment, since we don't offer consulting services for free.