Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor spdx tooling test to reduce intermittent failures #1707

Merged
merged 1 commit into from
Apr 3, 2023

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Apr 3, 2023

We've seen several failures of the SPDX tooling test, but haven't been able to determine exactly why:

Screenshot 2023-04-03 at 1 33 24 PM

I have a couple educated guesses and made some targeted refactors, but have no conclusive evidence if this will fix the underlying issue (seems to only happen in CI every once in a while).

I've also made a couple of improvements:

  • uses the existing test fixture facilities, as to leverage any preloaded CI cache without requiring a re build of the image.
  • uses a temp dir per test Run() call instead of sharing directories.
  • passes the explicit file for syft to write to and passes this file path to the verification tool

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman requested a review from a team April 3, 2023 18:24
@wagoodman wagoodman marked this pull request as ready for review April 3, 2023 18:24
@github-actions
Copy link

github-actions bot commented Apr 3, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                          │ ./.tmp/benchmark-4dcb0bb.txt │%0A                                                          │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   11.78m ± 26%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             851.5µ ±  2%25%0AImagePackageCatalogers/python-package-cataloger-2                           3.069m ±  1%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   693.6µ ±  2%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       359.0µ ±  1%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   511.3µ ±  3%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   496.3µ ±  2%25%0AImagePackageCatalogers/java-cataloger-2                                     11.25m ±  3%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     8.501µ ±  4%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    560.4µ ±  2%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         18.69µ ±  1%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              971.8µ ±  1%25%0AImagePackageCatalogers/portage-cataloger-2                                  339.8µ ±  3%25%0AImagePackageCatalogers/sbom-cataloger-2                                     105.5µ ±  1%25%0AImagePackageCatalogers/binary-cataloger-2                                   184.0µ ±  1%25%0Ageomean                                                                     471.1µ%0A%0A                                                          │ ./.tmp/benchmark-4dcb0bb.txt │%0A                                                          │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   5.063Mi ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             123.8Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                           947.0Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   155.8Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       90.62Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   144.8Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   170.1Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                     2.722Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.555Ki ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    129.2Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         3.133Ki ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              314.8Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                  77.21Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                     13.06Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                   29.03Ki ± 0%25%0Ageomean                                                                     108.5Ki%0A%0A                                                          │ ./.tmp/benchmark-4dcb0bb.txt │%0A                                                          │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    86.71k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.049k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                            15.49k ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    3.458k ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        1.205k ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    2.646k ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    3.759k ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                      38.26k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                       40.00 ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     3.437k ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                           101.0 ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               5.011k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                   1.539k ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                       392.0 ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                     872.0 ± 0%25%0Ageomean                                                                      2.223k

@wagoodman wagoodman enabled auto-merge (squash) April 3, 2023 18:33
err = buildCmd.Run()
require.NoError(t, err)
},
assertions: []traitAssertion{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, I removed the use of trait assertions since this test really needs this to be 0, not just assert it. The triat assertions are meant for blackbox testing, not for prerequirements for validations (which should fail immediately if not met).

@wagoodman wagoodman merged commit 8a574c9 into main Apr 3, 2023
@wagoodman wagoodman deleted the refactor-spdx-tooling-cli-test branch April 3, 2023 18:43
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
)

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants