Update Syft formats for SyftJson #752

Toure · 2022-01-18T19:28:28Z

This change will introduce omitempty struct tag to PackageCustomData.
This struct tag will cause null and empty values to be dropped on serialization
for consumers downstream.

Signed-off-by: Toure Dunnon toure.dunnon@anchore.com

github-actions · 2022-01-18T19:31:31Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.45ms ± 2%    1.67ms ± 6%  +14.89%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.49ms ±11%    3.77ms ± 2%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.02ms ± 2%    1.18ms ± 1%  +15.40%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         912µs ± 2%    1032µs ± 2%  +13.18%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                    1.06ms ± 3%    1.19ms ± 1%  +12.17%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      980µs ± 2%    1065µs ± 2%   +8.76%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      15.8ms ± 7%    17.1ms ± 2%     ~     (p=0.056 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.44ms ±10%    1.55ms ± 3%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          1.91µs ± 1%    2.33µs ± 2%  +21.66%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               253kB ± 0%     253kB ± 0%   -0.23%  (p=0.032 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            1.07MB ± 0%    1.06MB ± 0%     ~     (p=0.548 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     230kB ± 0%     230kB ± 0%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         208kB ± 0%     207kB ± 0%     ~     (p=0.056 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     254kB ± 0%     253kB ± 0%   -0.32%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      235kB ± 0%     235kB ± 0%     ~     (p=0.151 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.78MB ± 0%    3.78MB ± 0%   -0.20%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.30MB ± 0%    1.30MB ± 0%   -0.04%  (p=0.032 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            560B ± 0%      560B ± 0%     ~     (all equal)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               6.33k ± 0%     6.33k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             21.4k ± 0%     21.4k ± 0%     ~     (p=0.444 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     6.06k ± 0%     6.06k ± 0%   -0.02%  (p=0.016 n=4+5)
ImagePackageCatalogers/javascript-package-cataloger-2         5.34k ± 0%     5.34k ± 0%     ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     7.07k ± 0%     7.07k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpmdb-cataloger-2                      6.80k ± 0%     6.80k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       74.7k ± 0%     74.7k ± 0%     ~     (p=0.302 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      7.36k ± 0%     7.36k ± 0%     ~     (all equal)
ImagePackageCatalogers/go-module-binary-cataloger-2            13.0 ± 0%      13.0 ± 0%     ~     (all equal)

misclick

spiffcs · 2022-01-18T19:33:03Z

@Toure is there more context for this change or a reference for why we need to have these values omitted?

In what cases are you seeing these show up as null on the JSON output?

There are also some cases (should we make this change) that need to be updated on the CLI test side

Vijay-P · 2022-01-18T19:37:20Z

        {
            "id": "2666e846-97b0-4f6a-8701-f0bd19ac5e83",
            "name": "Authlib",
            "version": "0.15.2",
            "type": "python",
            "foundBy": "python-index-cataloger",
            "locations": [
                {
                    "path": "/Users/vijay/Documents/enterprise/requirements-test.txt"
                }
            ],
            "licenses": [],
            "language": "python",
            "cpes": [
                "cpe:2.3:a:python_Authlib:python-Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python_Authlib:python_Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python-Authlib:python-Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python-Authlib:python_Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python-Authlib:Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:Authlib:python-Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:Authlib:python_Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python_Authlib:Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python:python_Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python:python-Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:Authlib:Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python:Authlib:0.15.2:*:*:*:*:python:*:*",
                "cpe:2.3:a:python-Authlib:python-Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python_Authlib:python_Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python-Authlib:python_Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python_Authlib:python-Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python_Authlib:Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:Authlib:python-Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python-Authlib:Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:Authlib:python_Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python:python_Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python:python-Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:Authlib:Authlib:0.15.2:*:*:*:*:*:*:*",
                "cpe:2.3:a:python:Authlib:0.15.2:*:*:*:*:*:*:*"
            ],
            "purl": "pkg:pypi/Authlib@0.15.2",
            "metadataType": "",
            "metadata": null
        },

Toure · 2022-01-18T19:42:45Z

@spiffcs
When scanning Anchore Enterprise the output SBoM will contain a null value for metadata and metadataType attribute, as an example, Python_Authlib returns a null value for metadata which causes document imports to fail, this change would allow us to omit those fields.

spiffcs · 2022-01-18T19:48:35Z

Nice thanks for the example and comments! I think it's fine to drop the metadata field specifically since it would always be a nulll type and not something like "" or 0.

I will defer approval in this case to @wagoodman because maybe he has context or a good reason why we might want to update the UnMarshalJSON function to account for "metadataType": "", and set a value for metadata rather than drop the field completely from the output.

I do think metadataType: "" is useful and in this case dropping the field might not be what we're looking for in the final JSON output.

spiffcs

I'll sync with the tools team about the above comments. I might just be being too careful here about dropping fields from the output if empty.

spiffcs · 2022-01-18T19:56:22Z

The CLI tests are still failing - so those will need a quick update to reflect the new desired output.

wagoodman · 2022-01-18T20:11:20Z

@Toure I can help with the json schema update if you'd like (where CLI tests are failing)

This change will introduce omitempty struct tag to PackageCustomData. This struct tag will cause null and empty values to be dropped on serialization for consumers downstream. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com>

test coverage. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com>

…hub.com/hectorj2f/syft into hectorj2f/add_dependencies_to_cyclonedx * 'hectorj2f/add_dependencies_to_cyclonedx' of https://github.com/hectorj2f/syft: (29 commits) Improve CycloneDX format output (#710) Add additional PHP metadata (#753) Update Syft formats for SyftJson (#752) Add support for "file" source type in syftjson unmarshaling (#750) remove contains file from spdx dependency generation support .sar for java ecosystem (#748) Start developer documentation (#746) Align SPDX export more with SPDX 2.2 specification (#743) Replace distro type (#742) update goreleaser with windows checksums (#740) bump stereoscope version to remove old containerd (#741) Add support for multiple output files in different formats (#732) Add support for searching for jars within archives (#734) 683 windows filepath (#735) Fix CPE encode/decode when it contains special chars (#714) support .par for java ecosystems (#727) Add arm64 support to install script (#729) Revert "bump goreleaser to v1.2 (#720)" (#731) Add a version flag (#722) Add lpkg as java package format (#694) ...

* Update Syft formats for SyftJson This change will introduce omitempty struct tag to PackageCustomData. This struct tag will cause null and empty values to be dropped on serialization for consumers downstream. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com> * Updated the golden files for syftjson to allow for proper test coverage. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com> Signed-off-by: fsl <1171313930@qq.com>

* Update Syft formats for SyftJson This change will introduce omitempty struct tag to PackageCustomData. This struct tag will cause null and empty values to be dropped on serialization for consumers downstream. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com> * Updated the golden files for syftjson to allow for proper test coverage. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com> Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>

* Update Syft formats for SyftJson This change will introduce omitempty struct tag to PackageCustomData. This struct tag will cause null and empty values to be dropped on serialization for consumers downstream. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com> * Updated the golden files for syftjson to allow for proper test coverage. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com>

Toure requested review from wagoodman and spiffcs January 18, 2022 19:28

spiffcs previously approved these changes Jan 18, 2022

View reviewed changes

spiffcs self-requested a review January 18, 2022 19:31

Toure requested a review from Vijay-P January 18, 2022 19:34

spiffcs reviewed Jan 18, 2022

View reviewed changes

Update Syft formats for SyftJson

c8299f3

This change will introduce omitempty struct tag to PackageCustomData. This struct tag will cause null and empty values to be dropped on serialization for consumers downstream. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com>

Toure force-pushed the PackageCustomData_update branch from 5a16025 to c8299f3 Compare January 18, 2022 20:23

Updated the golden files for syftjson to allow for proper

af54e00

test coverage. Signed-off-by: Toure Dunnon <toure.dunnon@anchore.com>

Toure force-pushed the PackageCustomData_update branch from 8050502 to af54e00 Compare January 18, 2022 21:52

wagoodman approved these changes Jan 18, 2022

View reviewed changes

wagoodman merged commit 814f2bf into main Jan 18, 2022

wagoodman deleted the PackageCustomData_update branch January 18, 2022 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Syft formats for SyftJson #752

Update Syft formats for SyftJson #752

Toure commented Jan 18, 2022

github-actions bot commented Jan 18, 2022 •

edited

Loading

spiffcs commented Jan 18, 2022 •

edited

Loading

Vijay-P commented Jan 18, 2022

Toure commented Jan 18, 2022 •

edited

Loading

spiffcs commented Jan 18, 2022 •

edited

Loading

spiffcs left a comment

spiffcs commented Jan 18, 2022

wagoodman commented Jan 18, 2022 •

edited

Loading

Update Syft formats for SyftJson #752

Update Syft formats for SyftJson #752

Conversation

Toure commented Jan 18, 2022

github-actions bot commented Jan 18, 2022 • edited Loading

Benchmark Test Results

spiffcs commented Jan 18, 2022 • edited Loading

Vijay-P commented Jan 18, 2022

Toure commented Jan 18, 2022 • edited Loading

spiffcs commented Jan 18, 2022 • edited Loading

spiffcs left a comment

Choose a reason for hiding this comment

spiffcs commented Jan 18, 2022

wagoodman commented Jan 18, 2022 • edited Loading

github-actions bot commented Jan 18, 2022 •

edited

Loading

spiffcs commented Jan 18, 2022 •

edited

Loading

Toure commented Jan 18, 2022 •

edited

Loading

spiffcs commented Jan 18, 2022 •

edited

Loading

wagoodman commented Jan 18, 2022 •

edited

Loading