Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve export command #1571

Merged
merged 19 commits into from
Sep 12, 2024
Merged

Improve export command #1571

merged 19 commits into from
Sep 12, 2024

Conversation

pombredanne
Copy link
Collaborator

@pombredanne pombredanne commented Aug 27, 2024

This PR improves the "export" command for FederatedCode

  1. the export now uses less memory and does not load the whole DB at once
  2. we no longer export DB ids
  3. the output schema is simplified
  4. there is a new "aboutcode.hashid" package that is meant to be reused in FederatedCode

This is ready for review. I will likely use Serializers in the future

This helps with usage in FederatedCode, PurlDB and VulnerableCode

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Use the more accurate affected_by_vulnerabilities
Create an alias to ensure compatbility

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Use a single queryset

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is the CWE-prefixed value that we want to commonly display.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
In PackageQuerySet
- Align processing in for_purls and for_purl manager methods
- Correctly reference Package and not Vulnerability

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
- Improve memroy usage of main querysets
- Do not leak internal ids in serialized data
- Work towards reusing serializers

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
We use the main docker instead

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
And also license-expression, packageurl and cwe2

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Reference: #1579
Reported-by: tdruez <tdruez@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne pombredanne changed the title Improved export command Improve export command Sep 11, 2024
@pombredanne
Copy link
Collaborator Author

@keshav-space do you mind to look into this? this is needed for FederatedCode

Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pombredanne, LGTM!

@pombredanne
Copy link
Collaborator Author

The docs build is failing on URL validation. I am merging anyway.

@pombredanne pombredanne merged commit ed17dbd into main Sep 12, 2024
7 of 9 checks passed
@pombredanne pombredanne deleted the improved-export-command branch September 12, 2024 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants