Add support for SBOM analyzing at Binary-Artifacts stage by integrating sbom-scorecard

[Dentrax]

Is your feature request related to a problem? Please describe.
We should Detect if SBOMs generated (by @david-a-wheeler), and then we can scan the SBOMs to enrich the overall scorecard score using 3rd-party packages.

P.S.: scan in this context, does not mean vulnerability scanning. We can think of it like SBOM best practices & compliance scanning.

Describe the solution you'd like
Integrate the sbom-scorecard for calculating SBOM scores and aggregate the score result to Binary-Artifacts score.

See the getScore() function for more information.

Describe alternatives you've considered
Do not scan SBOMs if it's out-of-context of scorecard.

Additional context
-

/cc @developer-guy @justinabrahms

[justinabrahms]

I'm quite happy to help this become a thing. Reach out with any questions you may have. :)

[laurentsimon]

Sounds like a good idea. Any chance you could join a bi-weekly Scorecard sync to propose the idea?
In addition to the score, we are working on providing finer-grained results, an example can be found #2584, so hopefully SBOM check can be expressed with this format.

[idunbarh]

Excited to see this added to Scorecard. I think @david-a-wheeler's suggestion to review at https://www.ntia.gov/sites/default/files/publications/sbom_minimum_elements_report_0.pdf and ensure these minimum criteria are included in the score. Looks like you are 90% there.

[justinabrahms]

This is tracked at eBay/sbom-scorecard#34 and likely shouldn't be a blocker for progressing on integration.

[laurentsimon]

btw, what do you mean by Binary-Artifacts stage?

[david-a-wheeler]

I think @david-a-wheeler's suggestion to review at https://www.ntia.gov/sites/default/files/publications/sbom_minimum_elements_report_0.pdf and ensure these minimum criteria are included in the score. Looks like you are 90% there.

BTW, I don't think this is a blocker for use by OpenSSF Scorecard, it was just a suggestion for how to make sbom-scorecard even better. I think it helps if you can point to what some customers are already asking for, so you can say, "yes, we cover that!".

[Dentrax]

btw, what do you mean by Binary-Artifacts stage?

Oh, I think I should have said evaluation instead: https://github.com/ossf/scorecard/blob/main/checks/evaluation/binary_artifacts.go

[laurentsimon]

OK. What's the reasoning behind checking for SBOM for binaries in the repo (Binary-Artifacts - since we encourage users not to store binaries in the repo) rather than for released binary assets of the project?

I'm also wondering whether users will be able to remediate if the SBOM is of poor quality: do existing SBOM generation tools meet the criteria of "good SBOMs"?

Is a "good SBOM" solely a property of the SBOM generation tool, the project maintainers, or both?

[justinabrahms]

I've not looked into whether the tools are capable of generating "good SBOMs" in practice, but there's nothing stopping it technically. It's mostly "have the dependencies added the relevant metadata in the right places". I would say "good sbom" is both the tool and the maintainers.

For my $0.02, I think that you should put the sbom along the artifact, not in source control.

[Dentrax]

What's the reasoning behind checking for SBOM for binaries in the repo (Binary-Artifacts - since we encourage users not to store binaries in the repo) rather than for released binary assets of the project?

I think that you should put the sbom along the artifact, not in source control.

SBOMs should NOT be stored in the repo - Probably I dropped the wrong source code - since I'm not familiar with the codebase - so that you thought in this way. SBOMs should be an artifact that served along with the binaries in the release page or smth.

Should I revise the issue title to avoid misunderstanding?

[justinabrahms]

In the meeting today, we discussed that this should happen within the context of release, not binary artifacts.

We will need a mechanism to detect if something is an sbom given a (for instance) github release. Google's osv-scanner may provide a way to do that. We believe that this will probably look something like looking through the release artifacts for matching files (either as filename searching or attempting to parse the data and see if there's a formatting match).

[spencerschrock]

+1 to if Scorecard starts checking for SBOMs, it would fit better with releases, which makes the following checks candidates: Signed-Releases and Packaging.

My only reservation is around diluting the score of the Signed-Releases. Currently points are awarded as:
8 for signing
2 for provenance

Any points to SBOM would come out of this, and if we wanted more than just a basic "do you have an SBOM?" it would involve multiple such points.

[justinabrahms]

I agree that something titled "signed releases" should heavily favor signing. It does seem like there's more to a release than just signing. Is it possible for us to just name this "releases" and then include signing as a subcomponent?

…

On Mon, May 15, 2023, at 2:30 PM, Spencer Schrock wrote: +1 to if Scorecard starts checking for SBOMs, it would fit better with releases, which makes the following checks candidates: `Signed-Releases` and `Packaging`. My only reservation is around diluting the score of the `Signed-Releases`. Currently points are awarded as: 8 for signing 2 for provenance Any points to SBOM would come out of this, and if we wanted more than just a basic "do you have an SBOM?" it would involve multiple such points. — Reply to this email directly, view it on GitHub <#2605 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAA6DMO3S4MKIWBG5WXW5TXGKOA3ANCNFSM6AAAAAAT77GEGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lucasgonze]

I think we’re getting away from the original goal, which is to incorporate SBOMs as a signal, to encourage their generation, and to encourage higher-quality SBOMs. The relevant questions are:

1. Does an SBOM exist?
2. How good is it?

If the SBOM is stored in the repo instead of packaged with a release, that is a suboptimal practice, but it doesn’t negate either the existence of the SBOM or its quality.

I propose that a check for question 1 might look for SBOM generation in any detectable form. A check for question 2 then hinges on whether the SBOM is available to be graded.

This is enough complexity to warrant a new section.

[spencerschrock]

Is it possible for us to just name this "releases" and then include signing as a subcomponent?

Yes and no. For backwards compatibility reasons, the name can change in documentation, but any reference from CLI or results can't.

This is enough complexity to warrant a new section.

I could see an argument for a new (experimental?) check with low risk, similar to License. I unfortunately can't make tomorrow's sync, but will look for any discussion.

I think ultimately I agree with Laurent from #1476 (comment).

I think we need a way for scorecard to indicate "something to look at / think about", instead of a results that penalizes projects.

[lucasgonze]

I researched methods to document SBOM generation using a Github Action:
https://gist.github.com/lucasgonze/b84a2c52c697bf8f686a005080c369dd

Findings:

1. There are a number of SBOM generators that can be deployed using a Github Action. Generating an SBOM in this way is trivial.
2. There is a strong order-of-magnitude popularity curve. The most popular is 128 stars, the next is 28 stars, the rest are below 20. I imagine the most popular (Anchore) accounts for the vast majority of generated SBOMs.
3. Every generator could be detected unambiguously. None of them were multipurpose tools that might or might not be generating an SBOM.
4. Every Action could be detected by examining code under /.github/workflows/
5. Outputs varied widely. Naming conventions were inconsistent. Most allowed output files to be named using a parameter.
6. Only two (the most and least popular, Anchore and Diggity) specifically targeted release artifacts

[github-actions]

Stale issue message - this issue will be closed in 7 days

[Dentrax]

/remove-lifecycle stale

[github-actions]

This issue is stale because it has been open for 60 days with no activity.