Implement Syft2, 3 and CycloneDX 1.3 SBOM formats #312

fg-j · 2022-03-24T15:25:06Z

Summary

This PR pulls in a lot of borrowed code from anchore/syft and CycloneDX/cyclonedx-go, because both of these libraries do not currently support multiple versions of their SBOM schema. I've had to yank sections of old versions of these codebases and embed them here.

As much as possible, I've attempted to document the sources of the code – and where things diverge from their original content.

It's my intent not to change the API of packit's SBOM package at all with this PR, given how experimental this featureset is. The hope is that if this is added to packit and this change to the lifecycle is released, Paketo buildpacks can pin to CycloneDX 1.3 SBOMs.

Supercedes #325 as well.

Partially addresses #302 , starting with requested / backward-incompatible SBOM formats. Ideally, I'd like to push these implementations of the Syft sbom.Format interface into syft itself. Our use of them in packit can serve as a proof of concept of the viability of supporting multiple SBOM formats.

Checklist

I have viewed, signed, and submitted the Contributor License Agreement.
I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
I have added an integration test, if necessary.
I have reviewed the styleguide for guidance on my code quality.
I'm happy with the commit history on this PR (I have rebased/squashed as needed).

fg-j · 2022-03-24T15:34:00Z

There are some TODOs in here for tweaking parts of the implementation. Would love feedback on approaching those! I've been staring at it all too long. Need some fresh eyes to help me tighten things up.

fg-j · 2022-03-25T18:23:51Z

Merging this is blocked on the release of the corresponding lifecycle feature -- since we'll need to use the feature to update the sbom-formats array in buildpack.tomls to pin to CycloneDX 1.3 SBOM.

sbom/sbom_format.go

sophiewigmore · 2022-03-31T16:00:40Z

sbom/sbom_format.go

+	return sbomFormatByID(selected)
+}
+
+func sbomFormatByID(id sbom.FormatID) (sbomFormat, error) {


I've been looking at this function for a bit now, but I really can't think of a better way to do it than what you've laid out

sophiewigmore · 2022-03-31T18:26:22Z

@fg-j as discussed, it seems like the path of this PR is subject to change based off of our conversation with the Syft team at their working group meeting today. Do you want to pause on reviews until we have a clearer picture of what will happen in Syft first?

ryanmoran · 2022-03-31T19:01:55Z

I think we'll still want to merge this PR or something like it in the short-term so that we can unblock other issues, but we have a much higher degree of confidence that this debt we will be incurring can be paid down at some point in the medium-term future. That being said, I think the goals for what we should try to support here are to lock down the formats we currently support as outputs so that we can bump the Syft library forward to the latest versions without breaking our API contract.

Specifically, we need support for:

CycloneDX 1.3
Syft 3.0.1 (used in Buildpacks)
Syft 2.0.2 (used in Stacks)
SPDX 2.2

We get Syft 3 and SPDX 2.2 for now with what is on the latest versions of Syft, but there is the concern that it might be broken at some point in the future. Should we duplicate support in our own package to make sure the API stays stable, or should we just keep up with checking that the schema versions don't change for those formats?

fg-j · 2022-03-31T20:31:08Z

@ryanmoran Given that syft has shown some interest in exposing APIs that'll make it easier to create SBOM format implementations (without copying in their internal packages), I'm inclined to do the minimum (i.e. add Syft 3.0.1) in the short term and wait to see if we can unload some tech debt before we add our own CycloneDX 1.4 and SPDX 2.2 implementations.

I will update this PR to make CycloneDX 1.3 the default SBOM version generated when no version parameter is set on the media type so that consuming buildpacks will stay pinned to a CycloneDX minor version.

sophiewigmore · 2022-04-05T14:32:30Z

@fg-j you mentioned that when new Syft versions come out we'll occasionally have to change our implementation like in 7335806. Once this is merged in, what kind of Syft changes do we need to be cognizant of in releases? I'd imagine it will differ from what we're on the lookout for now (backwards incompatible schema changes).

fg-j · 2022-04-05T14:47:24Z

@sophiewigmore – For CycloneDX 1.4 and SPDX 2.2, since this PR still relies on Syft's implementations of some schema versions, we'll still want to assert that the SBOMs generated with the syft Formats have the expected versions.

We test this here for CycloneDX 1.4 and here for SPDX 2.2

When syft changes the underlying structs it uses to represent packages, we may need to change the code that converts from the syft representation to the JSON output representations. The entrypoint for this logic is in the to_format_model.go file in each format subdirectory (e.g. in syft 3.0.1). The encode_test.go for each format should fail if the syft structs change in a way that makes them convert differently into JSON.

If syft changes its internal representation of packages significantly, we may need to write bespoke converters, like ConvertImageMetadata, which peels fields off syft's current source.ImageMetadata type and returns them in a struct that matches the legacy representation of the same data.

fg-j · 2022-04-05T14:53:25Z

Removing the blocked label because this implementation defaults to producing CycloneDX 1.3 and Syft 3.0.1 SBOMs, which are the same formats we currently generate. Merging this will not change the schema of buildpack generated SBOMs. No longer blocked on the release of the lifecycle change allowing schema versions.

sophiewigmore · 2022-04-05T17:44:55Z

@fg-j I've gone through the code a few times and it looks good to me! It makes sense and I think covers all of our bases. @ryanmoran tagging you for a review as well before we merge it in to make sure it aligns with what you expected in #302 / to provide more technical expertise on the code structure.

I've also tried it out with the defaults as you mentioned, and the output looks as expected.

so that they can be used in syft.Encode() to generate pinned SBOM schema versions. Upgrades to syft 0.42.2 NOTE: upgrading syft required me to upgrade to go.1.18 on my machine We'll need to update our pipelines accordingly

ryanmoran · 2022-04-06T18:38:13Z

I'm trying to get npm-install to run with this branch. I'm having some trouble understanding how I can specify what format I want for a sbom.FormattedReader. For example, how do I specify the right formats for these test values?

https://github.com/paketo-buildpacks/npm-install/blob/28857f234fee9c4bbbc67cf4fa4117a4bfe776c2/build_test.go#L160-L173

fg-j · 2022-04-07T15:42:19Z

@ryanmoran The latest commit here should fix those test failures -- the NewFormattedReader() now does more to convert the inputted packit.sbom.Format into a syft.sbom.Format, respecting the schema version defaults we've set in packit.

fg-j requested a review from a team as a code owner March 24, 2022 15:25

fg-j requested review from ForestEckhardt and paketo-bot-reviewer March 24, 2022 15:25

fg-j closed this Mar 24, 2022

fg-j reopened this Mar 24, 2022

fg-j added the semver:minor A change requiring a minor version bump label Mar 24, 2022

fg-j force-pushed the sbom-formats branch from 625ce40 to 6814c9a Compare March 24, 2022 19:32

fg-j added the status/blocked This issue has been triaged and resolving it is blocked on some other issue label Mar 25, 2022

fg-j mentioned this pull request Mar 31, 2022

Bump github.com/anchore/syft from 0.41.1 to 0.42.4 #315

Closed

sophiewigmore reviewed Mar 31, 2022

View reviewed changes

sbom/sbom_format.go Show resolved Hide resolved

sophiewigmore reviewed Mar 31, 2022

View reviewed changes

fg-j mentioned this pull request Apr 4, 2022

Bump github.com/anchore/syft from 0.41.1 to 0.43.0 #319

Closed

fg-j changed the title ~~Implement Syft2 and CycloneDX 1.3 SBOM formats~~ Implement Syft2, 3 and CycloneDX 1.3 SBOM formats Apr 4, 2022

fg-j force-pushed the sbom-formats branch from 17b10eb to 7335806 Compare April 4, 2022 20:33

fg-j requested review from sophiewigmore and ryanmoran April 4, 2022 20:34

fg-j removed the status/blocked This issue has been triaged and resolving it is blocked on some other issue label Apr 5, 2022

sophiewigmore previously approved these changes Apr 5, 2022

View reviewed changes

Frankie Gallina-Jones added 3 commits April 6, 2022 13:57

add cyclonedx13 and syft2 implementations of Syft sbom.Format

3f1626e

so that they can be used in syft.Encode() to generate pinned SBOM schema versions. Upgrades to syft 0.42.2 NOTE: upgrading syft required me to upgrade to go.1.18 on my machine We'll need to update our pipelines accordingly

address linter complaints

52c0f7f

add syft 3.0.1; unit tests passing

ad3ff1f

Frankie Gallina-Jones added 2 commits April 6, 2022 13:58

register syft3.0.1 format, update default schema versions

ad76504

remove dead code; get tests passing with upgraded syft 0.43.0

3d94612

ryanmoran dismissed sophiewigmore’s stale review via 3d94612 April 6, 2022 17:58

ryanmoran force-pushed the sbom-formats branch from 536c8c8 to 3d94612 Compare April 6, 2022 17:58

adjust FormattedReader implementation to preserve test compatibility

80e9fab

upgrade to latest syft

7a01517

fg-j mentioned this pull request Apr 7, 2022

Bump github.com/anchore/syft from 0.41.1 to 0.43.2 #325

Closed

ryanmoran approved these changes Apr 8, 2022

View reviewed changes

ryanmoran merged commit e3745cd into v2 Apr 8, 2022

ryanmoran deleted the sbom-formats branch April 8, 2022 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Syft2, 3 and CycloneDX 1.3 SBOM formats #312

Implement Syft2, 3 and CycloneDX 1.3 SBOM formats #312

fg-j commented Mar 24, 2022 •

edited

Loading

fg-j commented Mar 24, 2022

fg-j commented Mar 25, 2022

sophiewigmore Mar 31, 2022

sophiewigmore commented Mar 31, 2022

ryanmoran commented Mar 31, 2022 •

edited by fg-j

Loading

fg-j commented Mar 31, 2022

sophiewigmore commented Apr 5, 2022 •

edited

Loading

fg-j commented Apr 5, 2022 •

edited

Loading

fg-j commented Apr 5, 2022

sophiewigmore commented Apr 5, 2022 •

edited

Loading

ryanmoran commented Apr 6, 2022

fg-j commented Apr 7, 2022

Implement Syft2, 3 and CycloneDX 1.3 SBOM formats #312

Implement Syft2, 3 and CycloneDX 1.3 SBOM formats #312

Conversation

fg-j commented Mar 24, 2022 • edited Loading

Summary

Checklist

fg-j commented Mar 24, 2022

fg-j commented Mar 25, 2022

sophiewigmore Mar 31, 2022

Choose a reason for hiding this comment

sophiewigmore commented Mar 31, 2022

ryanmoran commented Mar 31, 2022 • edited by fg-j Loading

fg-j commented Mar 31, 2022

sophiewigmore commented Apr 5, 2022 • edited Loading

fg-j commented Apr 5, 2022 • edited Loading

fg-j commented Apr 5, 2022

sophiewigmore commented Apr 5, 2022 • edited Loading

ryanmoran commented Apr 6, 2022

fg-j commented Apr 7, 2022

fg-j commented Mar 24, 2022 •

edited

Loading

ryanmoran commented Mar 31, 2022 •

edited by fg-j

Loading

sophiewigmore commented Apr 5, 2022 •

edited

Loading

fg-j commented Apr 5, 2022 •

edited

Loading

sophiewigmore commented Apr 5, 2022 •

edited

Loading