Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add file metadata cataloger #339

Merged
merged 29 commits into from
Mar 23, 2021
Merged

Add file metadata cataloger #339

merged 29 commits into from
Mar 23, 2021

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Mar 8, 2021 •
edited
Loading

This PR primarily adds the new file metadata cataloger to syft. There are several other changes that were made because of this:

  • The root command is being deprecated in favor of the packages subcommand. The root command is still in place and is an alias for the packages subcommand, additionally showing a deprecation warning when invoked.
  • Adds a new power-user subcommand, which exposes the new file-metadata cataloger functionality.
  • Expands on the application configuration to account for power-user functionality and to remove command-specific validations (e.g. presenter options, which should be lazily validated)
  • Refactors the presenter go packages to be oriented based on the per-subcommand they represent. Additionally migrates the syft/presenters to internal/presenters since they are command-specific and not generically helpful for library consumers.
  • Migrate the existing syft/cataloger package to syft/pkg/cataloger since there are now semantically more than one "kind" of cataloger, and the existing ones are tailored just for software packages.
  • Updates the JSON schema to reflect the additional (optional) fields exposed by the power-user command.
  • Removes the scope option from the image metadata section in the JSON presenters and adds application configuration to the JSON presenter Descriptor object (currently for the power-user subcommand only).
  • Pulls in the latest stereoscope changes (TarIndex for content fetching)
  • Removes the Multi*-oriented methods from the Resolver and renames Resolver to FileResolver, and clarifies the names of the other constituent resolver interfaces.
  • Adds CLI testing
  • Migrates JSON Schema testing from integration tests to CLI tests
  • Adds Benchmark testing

Closes #335

@wagoodman wagoodman added the WIP work in progress / do not merge label Mar 8, 2021
@wagoodman wagoodman force-pushed the add-file-metadata-indexer-invert-control branch 8 times, most recently from 756f51c to a68af76 Compare March 18, 2021 14:58
@wagoodman wagoodman changed the title Add file metadata indexer Add file metadata cataloger Mar 18, 2021
@wagoodman wagoodman marked this pull request as ready for review March 18, 2021 17:34
@wagoodman wagoodman force-pushed the add-file-metadata-indexer-invert-control branch from a68af76 to 19f646e Compare March 18, 2021 17:34
@github-actions
Copy link

github-actions bot commented Mar 18, 2021 •
edited
Loading

Benchmark Test Results

Benchmark results from the latest changes vs base branch 
name                                                   time/op

@wagoodman wagoodman requested a review from a team March 18, 2021 19:56
@wagoodman wagoodman removed the WIP work in progress / do not merge label Mar 18, 2021
@wagoodman wagoodman mentioned this pull request Mar 19, 2021
Merged
alfredodeza
alfredodeza reviewed Mar 19, 2021
View reviewed changes
cmd/power_user.go Outdated Show resolved Hide resolved
alfredodeza
alfredodeza reviewed Mar 19, 2021
View reviewed changes
cmd/cmd.go Outdated Show resolved Hide resolved
@wagoodman wagoodman force-pushed the add-file-metadata-indexer-invert-control branch from fde39e6 to b5900e7 Compare March 19, 2021 16:46
wagoodman added 10 commits March 22, 2021 10:46
@wagoodman
migrate syft/cataloger to syft/pkg/cataloger
4666ca8 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
migrate syft/presenter to internal/presenter
ff4ed40 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add poweruser concerns to the application config
f22d7d2 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove multi* content fetching from resolvers
a6cba5d 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add poweruser command and alias root to packages subcommand
97f0f83 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update json schema with optional poweruser data shape
6a960ec 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update event handlers relative to new command structure
18dd295 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update import to require patching image metadata source field
ae32942 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update integration and acceptance tests for new resolvers
efcd8a8 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove inline-compare testing
68d698e 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add cli testing
cf516ad 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update pipeline with new levels of testing
1d87f07 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add file-metadata config options to docs
4bde850 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove benchmark test event filter in validations pipeline
b1b57f6 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
improve config parsing + fix command deprecation warning
f180d1c 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove token usage from benchmark sticky comment action
abca2c5 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
adjust jsom schema version + adopt java pom properies test fixtures
36e4af1 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman force-pushed the add-file-metadata-indexer-invert-control branch from 3740d6f to 36e4af1 Compare March 22, 2021 15:29
@wagoodman
simplify command alias logic + remove deprecation warning for root co…
46636cc 
…mmand

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
upgrade goreleaser + constrain pipeline tool cache
e3b1522 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
luhring
luhring reviewed Mar 23, 2021
View reviewed changes
README.md Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
syft/pkg/cataloger/java/test-fixtures/pom/colon-delimited-with-equals.pom.properties
@@ -2,4 +2,4 @@
#Tue Jul 07 18:59:56 GMT 2020
groupId:org.anchore
artifactId: example-java=app-maven
version: 0.1.0=something
version: 0.1.0=something
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit / future: It'd be nice to find easy ways to prevent these newline removals from adding noise to the diff. (For myself, I just check a box in my editor)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd vote adding precommit hooks over synchronizing IDE settings (something I've been wanting to add to the repo anyway)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either/both is good w/ me 👍

syft/pkg/cataloger/python/package_cataloger.go Outdated Show resolved Hide resolved
wagoodman added 10 commits March 23, 2021 10:28
@wagoodman
bump coverage threshold + use ubuntu for snapshot builds
77e4c89 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
arrange power-user tasks ahead of parsing the source image
0e9c1c1 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
stabilize json file metadata presenter order
4019909 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add tests around new file metadata cataloger
d420368 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove prealloc nolint rule in catalogers
929b78e 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
update documentation to reflect root vs packages subcommand
68c4bf8 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
remove docker layer cache from validation pipeline
12f4191 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
pull in stereoscope cache fix + add test cache makefile target helpers
3ef30f9 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
cache mac acceptance image
8854d83 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
add file-type mix as golden image
1b7c755 
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
luhring
luhring approved these changes Mar 23, 2021
View reviewed changes
Copy link
Contributor

@luhring luhring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice addition! Thanks for the thoughtful review sessions ❤️

@wagoodman wagoodman merged commit e256e86 into main Mar 23, 2021
@wagoodman wagoodman deleted the add-file-metadata-indexer-invert-control branch March 23, 2021 21:02
@wagoodman wagoodman mentioned this pull request Mar 29, 2021
Merged
@wagoodman wagoodman mentioned this pull request May 12, 2021
Merged
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
@wagoodman
Merge pull request anchore#339 from anchore/add-file-metadata-indexer…
7f22906 
…-invert-control

Add file metadata cataloger
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alfredodeza alfredodeza alfredodeza left review comments

@luhring luhring luhring approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add file metadata cataloger
3 participants
@wagoodman @alfredodeza @luhring