Initial commit of OSV record linter #243
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
``` go run cmd/osv/main.go record lint test_data/nointroduced-CVE-2023-41045.json ``` Signed-off-by: Andrew Pollock <apollock@google.com>
andrewpollock
force-pushed
the
osv_linter_prototype
branch
from
2fcd981
to
ca67153
Compare
oliverchang
reviewed
May 24, 2024
- Generate the maps of checks and collections dynamically, to reduce future maintenance burden - Define an interface for Checks - Refactor existing code accordingly Signed-off-by: Andrew Pollock <apollock@google.com>
Add more docstrings Signed-off-by: Andrew Pollock <apollock@google.com>
(actual directory walking support coming soon) Still functional: ``` $ go run cmd/osv/main.go record lint test_data/nointroduced-CVE-2023-41045.json Running "osv.dev" check collection on &["test_data/nointroduced-CVE-2023-41045.json"] Running "introduced-event-exists" check on "test_data/nointroduced-CVE-2023-41045.json" 2024/05/29 07:07:55 "test_data/nointroduced-CVE-2023-41045.json": "introduced-event-exists": []checks.CheckError{checks.CheckError{Code:"R0001", Message:"missing 'introduced' object in event at index 0"}} 2024/05/29 07:07:55 found errors exit status 1 ``` Signed-off-by: Andrew Pollock <apollock@google.com>
I'm still getting the hang of GJSON's query syntax and how to operate on results from it. I'm at least now more confident about the behaviour of this check. Signed-off-by: Andrew Pollock <apollock@google.com>
Signed-off-by: Andrew Pollock <apollock@google.com>
oliverchang
reviewed
Jun 7, 2024
Revert the interface, on the premise that there's only going to be one known implementation at this time. Rename the types, do away with the custom string and map. Move more of the definitional variable to the same place as the code. Simply how checks are inventoried by adding another "ALL" collection. Signed-off-by: Andrew Pollock <apollock@google.com>
Use an historically incorrectly generated record that was flagged in google/osv.dev#1984 ``` $ go run ./cmd/osv record lint test_data/CVE-2018-5407.json test_data/nondistinct-CVE-2018-5407.json Running "osv.dev" check collection on &["test_data/CVE-2018-5407.json" "test_data/nondistinct-CVE-2018-5407.json"] Running "introduced-event-exists" check on "test_data/CVE-2018-5407.json" Running "range-is-distinct" check on "test_data/CVE-2018-5407.json" Running "introduced-event-exists" check on "test_data/nondistinct-CVE-2018-5407.json" Running "range-is-distinct" check on "test_data/nondistinct-CVE-2018-5407.json" 2024/07/19 05:22:54 "test_data/nondistinct-CVE-2018-5407.json": "range-is-distinct": []checks.CheckError{checks.CheckError{Code:"R0002", Message:": overlapping event: \"e818b74be2170fbe957a07b0da4401c2b694b3b8\} 2024/07/19 05:22:54 found errors exit status 1 ``` Signed-off-by: Andrew Pollock <apollock@google.com>
For more conciseness, and because the linter is expected to be run on records before they're processed by OSV.dev, not after. Signed-off-by: Andrew Pollock <apollock@google.com>
oliverchang
reviewed
Jul 29, 2024
Add checks for package and package version existence. Add end-to-end support for two ecosystems: PyPI and Go
- Normalize ecosystems with colons in them down to their base - Cache the existence/nonexistence of a package (in a normalized ecosystem) to reduce duplicate network checks - Correct the test data for CVE-2018-5407 to be the current live record without overlapping ranges present (this shouldn't fail range validation)
It is valid to not have any range at all, as seen in the likes of GHSA-9v2f-6vcg-3hgv, which was being flagged incorrectly.
Remove some println() debugging
The Go module proxy seems to not support package names with uppercase in their name. GitHub URLs are known to be case-insensitive, so it's safe to explicitly lowercase these. I dare say it'll be safe to lowercase everything, but I wanted to start conservatively for now. Also treat Go toolchain vulnerabilities the same way as stdlib ones so they aren't flagged as a non-existent package.
Overwrite, don't append so that only the check requested gets run
They don't get returned by the Go proxy. Also support versions with or without the "v" prefix. All existing published Go vulnerabilities with the exception of GO-2024-3012 now pass validation. Signed-off-by: Andrew Pollock <apollock@google.com>
oliverchang
reviewed
Aug 8, 2024
|
@cuixq would you mind reviewing this also? (In particular, the package and version validation, but overall as well) |
another-rex
reviewed
Aug 8, 2024
oliverchang
reviewed
Aug 8, 2024
andrewpollock
changed the title
oliverchang
reviewed
Aug 12, 2024
Unnecessary, default to the ALL one. Signed-off-by: Andrew Pollock <apollock@google.com>
I was following examples in the GJSON documentation, but I can't really see a reason for them. Signed-off-by: Andrew Pollock <apollock@google.com>
return the result of the comparison instead of branching to return a boolean Signed-off-by: Andrew Pollock <apollock@google.com>
New project, no reason not to Signed-off-by: Andrew Pollock <apollock@google.com>
General wisdom seems to be not to include these in the repo Signed-off-by: Andrew Pollock <apollock@google.com>
1.23.0 isn't available for gLinux yet Signed-off-by: Andrew Pollock <apollock@google.com>
Addresses reviewer feedback Signed-off-by: Andrew Pollock <apollock@google.com>
Address reviewer feedback Signed-off-by: Andrew Pollock <apollock@google.com>
- use better Go package names - break out Go packages more granularly - make some functions private Signed-off-by: Andrew Pollock <apollock@google.com>
The callers need to do this as they access the response Signed-off-by: Andrew Pollock <apollock@google.com>
Signed-off-by: Andrew Pollock <apollock@google.com>
oliverchang
approved these changes
Aug 19, 2024
| @@ -0,0 +1,308 @@ | |||
| { | |||
There was a problem hiding this comment.
Most of these test files don't seem to be referenced anywhere as part of an automated test. Are you planning to add them in another PR?
There was a problem hiding this comment.
There's so much extra scaffolding to add around CI/CD type stuff... Yes, that'll be in a future PR. Right now, these test files are for running the code against manually, per the PR description.
oliverchang
reviewed
Aug 19, 2024
another-rex
reviewed
Aug 19, 2024
This reduces failure noise for ecosystems yet to be implemented Signed-off-by: Andrew Pollock <apollock@google.com>
Addresses user expectation feedback from @another-rex Signed-off-by: Andrew Pollock <apollock@google.com>
cuixq
reviewed
Aug 28, 2024
Thanks to @cuixq alerting me to golang.org/x/mod/{module,semver} I can check for psuedoversions and mess with semver versions somewhat more cleanly Signed-off-by: Andrew Pollock <apollock@google.com>
This variable has no need to be public Signed-off-by: Andrew Pollock <apollock@google.com>
This handles all the idiosyncrasies of PEP440 for versions, and normalizes package names per the documented guidelines. Signed-off-by: Andrew Pollock <apollock@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is reasonably functional at this point, with multiple checks of two different aspects:
Ranges:
introducedexistsPackages:
Part of google/osv.dev#2187