Speed up cataloging by replacing globs searching with index lookups #1510

wagoodman · 2023-01-24T18:01:00Z

Leverages anchore/stereoscope#154 to improve cataloging speeds. Searching by full path globs is used everywhere, however, in images that have a large number of files this kind of operation is very intensive. This PR introduces new searching objects that leverage basename indexes in stereoscope in order to vastly improve the initial lookup against the index and then additionally prune potential results with additional requirements.

Today's behavior:

time go run ./cmd/syft ~/images/virt-manager.tar -vvv  
# ...
real	4m34.067s
user	5m49.291s
sys	0m6.189s

With the new indexes being leveraged:

time go run ./cmd/syft ~/images/virt-manager.tar -vvv
# ...
real	38.155s
user	43.73s
sys	12.88s

Specific changes:

Adds several tests around all source.FileResover implementations to show that they behave very similarly (this PR addresses several inconsistencies)
The indexing done in the source.directoryResolver has been split out into another object entirely: source.directoryIndexer
Refactors the source.directoryResolver to use the new stereoscope filetree.Index and filetree.Searcher to best leverage the new performance related enhancements from Add additional catalog indexes for performance stereoscope#154
Fixes the directory resolve to not visit the same file twice during indexing (once from the real path and another from one or more virtual paths).
Adds a new form of testing around the catalogers: glob match testing. This is done within each package where a cataloger is implemented and ensures that the cataloger is wired appropriately to select all possible files that can be fetched from the cataloger via globs without testing specific parsing logic.
Replaces the source.FileMetadata object with the stereoscope file.Metadata object instead.
Replaces the source.FileType object with the stereoscope file.Type object instead.

Closes #1328
Closes #1480

github-actions · 2023-01-24T18:03:14Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

goos: linux
goarch: amd64
pkg: github.com/anchore/syft/test/integration
cpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
                                                          │ ./.tmp/benchmark-ba8be9b.txt │
                                                          │            sec/op            │
ImagePackageCatalogers/alpmdb-cataloger-2                                   12.03m ± 29%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             855.6µ ±  2%
ImagePackageCatalogers/python-package-cataloger-2                           2.152m ±  8%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   669.7µ ±  2%
ImagePackageCatalogers/javascript-package-cataloger-2                       346.6µ ±  1%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   480.9µ ±  2%
ImagePackageCatalogers/rpm-db-cataloger-2                                   446.2µ ±  2%
ImagePackageCatalogers/java-cataloger-2                                     10.90m ±  1%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     7.963µ ±  2%
ImagePackageCatalogers/apkdb-cataloger-2                                    466.2µ ±  1%
ImagePackageCatalogers/go-module-binary-cataloger-2                         18.00µ ±  1%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              945.6µ ±  1%
ImagePackageCatalogers/portage-cataloger-2                                  288.4µ ±  2%
ImagePackageCatalogers/sbom-cataloger-2                                     103.9µ ±  1%
ImagePackageCatalogers/binary-cataloger-2                                   139.5µ ±  0%
geomean                                                                     430.0µ

                                                          │ ./.tmp/benchmark-ba8be9b.txt │
                                                          │             B/op             │
ImagePackageCatalogers/alpmdb-cataloger-2                                   5.060Mi ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                             141.9Ki ± 0%
ImagePackageCatalogers/python-package-cataloger-2                           767.5Ki ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                   155.9Ki ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                       95.83Ki ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                   144.7Ki ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                   170.8Ki ± 0%
ImagePackageCatalogers/java-cataloger-2                                     2.723Mi ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                     1.523Ki ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                    122.2Ki ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                         3.102Ki ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                              314.3Ki ± 0%
ImagePackageCatalogers/portage-cataloger-2                                  75.51Ki ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                     13.06Ki ± 0%
ImagePackageCatalogers/binary-cataloger-2                                   20.55Ki ± 0%
geomean                                                                     105.2Ki

                                                          │ ./.tmp/benchmark-ba8be9b.txt │
                                                          │          allocs/op           │
ImagePackageCatalogers/alpmdb-cataloger-2                                    86.71k ± 0%
ImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.159k ± 0%
ImagePackageCatalogers/python-package-cataloger-2                            10.35k ± 0%
ImagePackageCatalogers/php-composer-installed-cataloger-2                    3.458k ± 0%
ImagePackageCatalogers/javascript-package-cataloger-2                        1.253k ± 0%
ImagePackageCatalogers/dpkgdb-cataloger-2                                    2.646k ± 0%
ImagePackageCatalogers/rpm-db-cataloger-2                                    3.759k ± 0%
ImagePackageCatalogers/java-cataloger-2                                      38.26k ± 0%
ImagePackageCatalogers/graalvm-native-image-cataloger-2                       40.00 ± 0%
ImagePackageCatalogers/apkdb-cataloger-2                                     3.234k ± 0%
ImagePackageCatalogers/go-module-binary-cataloger-2                           101.0 ± 0%
ImagePackageCatalogers/dotnet-deps-cataloger-2                               5.011k ± 0%
ImagePackageCatalogers/portage-cataloger-2                                   1.487k ± 0%
ImagePackageCatalogers/sbom-cataloger-2                                       392.0 ± 0%
ImagePackageCatalogers/binary-cataloger-2                                     609.0 ± 0%
geomean                                                                      2.112k

syft/pkg/cataloger/alpm/parse_alpm_db_test.go

syft/pkg/cataloger/apkdb/cataloger.go

syft/pkg/cataloger/binary/cataloger.go

syft/pkg/cataloger/deb/cataloger.go

syft/pkg/cataloger/generic/cataloger.go

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

spiffcs

Small comments about if values could be nil and cause a panic. I'm going to take another look at the resolver in a second pass since those had the most changes.

syft/source/image_all_layers_resolver.go

syft/source/location.go

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

…indexes Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

…nchore#1510) * replace raw globs with index equivelent operations Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add cataloger test for alpm cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix import sorting for binary cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting for mock resolver Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * separate portage cataloger parser impl from cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * enhance cataloger pkgtest utils to account for resolver responses Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for alpm cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for apkdb cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for dpkg cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for cpp cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for dart cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for dotnet cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for elixir cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for erlang cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for golang cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for haskell cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for java cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for javascript cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for php cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for portage cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for python cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for rpm cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for rust cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for sbom cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for swift cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * allow generic catloger to run all mimetype searches at once Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove stutter from php and javascript cataloger constructors Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump stereoscope Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add tests for generic.Search Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add exceptions for java archive git ignore entries Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * enhance basename and extension resolver methods to be variadic Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * dont allow * prefix on extension searches Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add glob-based cataloger tests for ruby cataloger Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove unnecessary string casting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * incorporate surfacing of leaf link resolitions from stereoscope results Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * [wip] switch to stereoscope file metadata Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * [wip + failing] revert to old globs but keep new resolvers Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * index files, links, and dirs within the directory resolver Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix several resolver bugs and inconsistencies Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * move format testutils to internal package Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update syft json to account for file type string normalization Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * split up directory resolver from indexing Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update docs to include details about searching Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * [wip] bump stereoscope to development version Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * adjust symlinks fixture to be fixed to digest Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix all-locations resolver tests Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix test fixture reference Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename file.Type Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump stereoscope Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix PR comment to exclude extra * Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump to dev version of stereoscope Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * bump to final version of stereoscope Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * move observing resolver to pkgtest Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman added the WIP work in progress / do not merge label Jan 24, 2023

Mikcl mentioned this pull request Jan 24, 2023

Refactor the Generic Cataloger re-uses filesystem indexing #1463

Closed

wagoodman closed this Jan 25, 2023

wagoodman force-pushed the add-basename-catalog-indexes branch from a9193c9 to 7f3382f Compare January 25, 2023 13:38

wagoodman deleted the add-basename-catalog-indexes branch January 25, 2023 13:41

wagoodman restored the add-basename-catalog-indexes branch January 25, 2023 13:41

wagoodman reopened this Jan 25, 2023

wagoodman force-pushed the add-basename-catalog-indexes branch from 85b7f66 to cba0e45 Compare January 25, 2023 19:04

wagoodman requested a review from a team January 25, 2023 20:53

wagoodman mentioned this pull request Jan 25, 2023

Add additional catalog indexes for performance anchore/stereoscope#154

Merged

wagoodman added enhancement New feature or request and removed WIP work in progress / do not merge labels Jan 25, 2023

wagoodman changed the title ~~Replace raw globs with index equivalent operations~~ Speed up cataloging by replacing globs searching with index lookups Jan 25, 2023

wagoodman mentioned this pull request Jan 26, 2023

Very long cataloging process #1328

Closed

kzantow reviewed Jan 26, 2023

View reviewed changes

kzantow mentioned this pull request Jan 26, 2023

Improve syft performance by memoizing filetree #1480

Closed

kzantow linked an issue Jan 30, 2023 that may be closed by this pull request

Improve syft performance by memoizing filetree #1480

Closed

wagoodman added 12 commits January 30, 2023 16:59

replace raw globs with index equivelent operations

25f1a9c

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add cataloger test for alpm cataloger

ebddc92

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

fix import sorting for binary cataloger

e69d1b9

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

fix linting for mock resolver

f123b6b

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

separate portage cataloger parser impl from cataloger

e35245b

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

enhance cataloger pkgtest utils to account for resolver responses

fc2c846

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for alpm cataloger

e9ac344

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for apkdb cataloger

67b131c

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for dpkg cataloger

33a902a

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for cpp cataloger

3ea801d

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for dart cataloger

46d9f6e

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

add glob-based cataloger tests for dotnet cataloger

a70bef6

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

spiffcs reviewed Feb 8, 2023

View reviewed changes

syft/source/image_all_layers_resolver.go Show resolved Hide resolved

syft/source/location.go Show resolved Hide resolved

move observing resolver to pkgtest

f246744

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman mentioned this pull request Feb 9, 2023

Update Stereoscope to 5a306f07f2e73a1498267d40c144ba4c10487a45 #1557

Closed

wagoodman linked an issue Feb 9, 2023 that may be closed by this pull request

Very long cataloging process #1328

Closed

spiffcs approved these changes Feb 9, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into add-basename-catalog-…

89f40fc

…indexes Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

wagoodman enabled auto-merge (squash) February 9, 2023 16:13

wagoodman merged commit 988041b into main Feb 9, 2023

wagoodman deleted the add-basename-catalog-indexes branch February 9, 2023 16:19

mathrock mentioned this pull request Feb 20, 2023

Encountering "cycle during symlink resolution" with syft version 0.71.0 onwards #1586

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up cataloging by replacing globs searching with index lookups #1510

Speed up cataloging by replacing globs searching with index lookups #1510

wagoodman commented Jan 24, 2023 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading

spiffcs left a comment

Speed up cataloging by replacing globs searching with index lookups #1510

Speed up cataloging by replacing globs searching with index lookups #1510

Conversation

wagoodman commented Jan 24, 2023 • edited Loading

github-actions bot commented Jan 24, 2023 • edited Loading

Benchmark Test Results

spiffcs left a comment

Choose a reason for hiding this comment

wagoodman commented Jan 24, 2023 •

edited

Loading

github-actions bot commented Jan 24, 2023 •

edited

Loading