[GSOC23] - C - Implement a StAX parser for OVAL files #7510

HoussemNasri · 2023-09-07T21:36:50Z

What does this PR change?

Introduce a StAX parser (instead of the current JAXB parser) for OVAL files to increase memory efficiency when parsing large OVAL files.

Useful Links

The RFC

GUI diff

No difference.

Before:

After:

DONE

Documentation

No documentation needed: add explanation. This can't be used if there is a GUI diff
No documentation needed: only internal and user invisible changes
Documentation issue was created: Link for SUSE Manager contributors, Link for community contributors.
API documentation added: please review the Wiki page Writing Documentation for the API if you have any changes to API documentation.
(OPTIONAL) Documentation PR
DONE

Test coverage

No tests: add explanation
No tests: already covered
Unit tests were added
Cucumber tests were added
DONE

Links

Fixes #
Tracks # add downstream PR, if any

DONE

Changelogs

Make sure the changelogs entries you are adding are compliant with https://github.com/uyuni-project/uyuni/wiki/Contributing#changelogs and https://github.com/uyuni-project/uyuni/wiki/Contributing#uyuni-projectuyuni-repository

If you don't need a changelog check, please mark this checkbox:

No changelog needed

If you uncheck the checkbox after the PR is created, you will need to re-run changelog_test (see below)

Re-run a test

If you need to re-run a test, please mark the related checkbox, it will be unchecked automatically once it has re-run:

Re-run test "changelog_test"
Re-run test "backend_unittests_pgsql"
Re-run test "java_pgsql_tests"
Re-run test "schema_migration_test_pgsql"
Re-run test "susemanager_unittests"
Re-run test "javascript_lint"
Re-run test "spacecmd_unittests"

github-actions · 2023-09-07T21:37:23Z

Suggested tests to cover this Pull Request

srv_docker_cve_audit
min_cve_audit

java/code/src/com/suse/oval/OvalParser.java

java/code/src/com/suse/oval/parser/OvalParser.java

github-actions · 2023-10-26T20:09:44Z

👋 Hello! Thanks for contributing to our project.
Acceptance tests will take some time (aprox. 1h), please be patient ☕
You can see the progress at the end of this page and at https://github.com/uyuni-project/uyuni/pull/7510/checks
Once tests finish, if they fail, you can check 👀 the cucumber report. See the link at the output of the action.
You can also check the artifacts section, which contains the logs at https://github.com/uyuni-project/uyuni/pull/7510/checks.

If you are unsure the failing tests are related to your code, you can check the "reference jobs". These are jobs that run on a scheduled time with code from master. If they fail for the same reason as your build, it means the tests or the infrastructure are broken. If they do not fail, but yours do, it means it is related to your code.

Reference tests:

KNOWN ISSUES

Sometimes the build can fail when pulling new jar files from download.opensuse.org . This is a known limitation. Given this happens rarely, when it does, all you need to do is rerun the test. Sorry for the inconvenience.

For more tips on troubleshooting, see the troubleshooting guide.

Happy hacking!
⚠️ You should not merge if acceptance tests fail to pass. ⚠️

java/code/src/com/suse/oval/parser/OvalParser.java

github-actions · 2023-12-26T01:42:58Z

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2024-01-05T01:48:08Z

This PR was closed because it has been stalled for 10 days with no activity.

mcalmer · 2024-01-05T07:16:13Z

@HoussemNasri @parlt91 i think this is still needed, right?

HoussemNasri · 2024-01-05T18:03:53Z

@HoussemNasri @parlt91 i think this is still needed, right?

Yes, the A and B pull requests need to be reviewed first before we can move to this one in case there was some changes that would affect this one (all PRs are kind of stacked on each other). Right now, pr A is under review.

mcalmer · 2024-01-05T19:36:02Z

Than we better reopen this request

web/html/src/branding/css/susemanager/components/cveaudit.less

Etheryte

The issues that were pointed out regarding styles and how they're used in the previous pull request are still present in this one. The comments are marked as resolved, but the issues have not been addressed, see e.g. #7509 (comment) and #7509 (comment). Please do not mark issues as resolved if they have not been addressed.

java/code/src/com/suse/oval/parser/OvalParser.java

Etheryte

No frontend changes in the latest summa summarum changeset.

java/code/src/com/suse/oval/OVALCleaner.java

parlt91

I also did some testing around the changes. There seems to be an issue with parsing. A lot of info seems to be overridden. If I had to guess I'd say it looks like each bulk overrides the last one instead of adding onto it. Could you have a look into it?
I'll do another review round once these issues are addressed.

java/code/src/com/suse/oval/parser/OvalParser.java

java/code/src/com/suse/oval/OVALCleaner.java

java/code/src/com/suse/oval/parser/OVALDefinitionsBulkHandler.java

- Also referenced the table from suseOVALPlatformVulnerablePackage

…in the database

…a given OS product

java/code/src/com/suse/oval/parser/OvalParser.java

+     * @param bulkHandler an operation to applied on every bulk of parsed OVAL definitions.
+     * */
+    public void parseDefinitionsInBulk(File ovalFile, OVALDefinitionsBulkHandler bulkHandler) {
+        XMLInputFactory xmlInputFactory = XMLInputFactory.newInstance();


java/code/src/com/suse/oval/parser/OvalParser.java

parlt91

Could you go a bit into detail on your though process for adding the ovalOsProduct table? IIUC you use it to determine what data we need to delete from ovalPlatfromVulnerablePackage on resync of the oval data to avoid conflicting data.

parlt91 · 2024-11-14T11:34:13Z

schema/spacewalk/common/tables/suseOVALOsProduct.sql

+ * Red Hat trademarks are not licensed under GPLv2. No permission is
+ * granted to use or replicate Red Hat trademarks that are incorporated
+ * in this software or its documentation.
+ */


You need to use -- instead of * as comment here.

parlt91 · 2024-11-19T08:47:32Z

java/code/src/com/suse/oval/OVALCachingFactory.java

-        productVulnerablePackages.stream()
-                .collect(groupingBy(ProductVulnerablePackages::getProductCpe))
-                .keySet().forEach(OVALCachingFactory::clearOVALMetadataByPlatform);
+//        productVulnerablePackages.stream()


I assume you used this for testing the sync multiple times, but we should enable it again before we merge.

parlt91 · 2024-11-19T08:58:52Z

java/code/src/com/redhat/rhn/common/db/datasource/xml/oval_queries.xml

+        <query params="os_product_family, os_product_version">
+            DELETE
+            FROM suseOVALPlatformVulnerablePackage pvp
+            WHERE pvp.platform_id = (SELECT id


Instead of deleting the OvalPlatformVulnerablePackage entries manually, couldn't we just delete the associated OvalOsProduct entry and have a cascading delete? I'd guess this would also perform better then deleting using a sub query.

parlt91 · 2024-11-19T09:26:58Z

...hema-5.1.1-to-susemanager-schema-5.1.2/003-map-oval-vulnerability-metadata-by-os-product.sql

+    DROP CONSTRAINT suse_oval_platform_vulnerable_pkg_id_pk;
+ALTER TABLE suseOVALPlatformVulnerablePackage
+    ADD CONSTRAINT suse_oval_platform_vulnerable_pkg_id_pk PRIMARY KEY (product_os_id, platform_id, cve_id, vulnerable_pkg_id);
+


If I see this correctly we miss a foreign key for product_os_id in the migration script.

alter table suseovalplatformvulnerablepackage add constraint suseovalplatformvulnerablepackage_os_product_id_fkey foreign key (product_os_id) references suseovalosproduct(id) on delete cascade;

We should also cascade on delete

github-actions bot added java java_lint_checkstyle java_pgsql_tests API schema_migration_test_pgsql database schema_migration_test_oracle ui javascript_lint data-model xmlrpc old-ui labels Sep 7, 2023

github-actions bot added the needs-translation label Sep 7, 2023

github-advanced-security bot found potential problems Sep 7, 2023

View reviewed changes

java/code/src/com/suse/oval/OvalParser.java Fixed Show fixed Hide fixed

HoussemNasri force-pushed the oval-stax-parser branch from 8dc2d93 to 6d6dc0f Compare September 14, 2023 17:10

github-advanced-security bot found potential problems Sep 17, 2023

View reviewed changes

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show fixed Hide fixed

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show fixed Hide fixed

HoussemNasri force-pushed the oval-stax-parser branch from a279e94 to 7a5f2c7 Compare September 17, 2023 18:01

HoussemNasri mentioned this pull request Sep 24, 2023

[WIP][GSOC23] - E - Update CVE Audit documentation uyuni-project/uyuni-docs#2471

Open

2 tasks

HoussemNasri mentioned this pull request Oct 25, 2023

[GSOC23] - A - Implement a fully functional CVE auditing feature based on OVAL data #7466

Merged

17 tasks

github-advanced-security bot found potential problems Oct 26, 2023

View reviewed changes

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show fixed Hide fixed

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show fixed Hide fixed

github-actions bot added the Stale label Dec 26, 2023

github-actions bot closed this Jan 5, 2024

mcalmer reopened this Jan 5, 2024

github-actions bot added the API label Sep 15, 2024

Etheryte reviewed Sep 18, 2024

View reviewed changes

web/html/src/branding/css/susemanager/components/cveaudit.less Outdated Show resolved Hide resolved

Etheryte requested changes Sep 18, 2024

View reviewed changes

HoussemNasri force-pushed the oval-stax-parser branch 3 times, most recently from 9a8b53c to 07266bb Compare October 8, 2024 11:31

github-advanced-security bot found potential problems Oct 8, 2024

View reviewed changes

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show resolved Hide resolved

java/code/src/com/suse/oval/parser/OvalParser.java Fixed Show resolved Hide resolved

Etheryte previously approved these changes Oct 9, 2024

View reviewed changes

parlt91 reviewed Oct 23, 2024

View reviewed changes

java/code/src/com/suse/oval/OVALCleaner.java Outdated Show resolved Hide resolved

parlt91 requested changes Oct 23, 2024

View reviewed changes

HoussemNasri added 12 commits November 10, 2024 17:01

Add support for SLE Micro OVAL-based CVE auditing

72d6796

Parse OVAL states and objects with StAX

2e38e4c

Parse OVAL tests with StAX

dfe4638

Parse OVAL definition's metadata with StAX

9e0fd78

Parse OVAL definition's criteria tree with StAX

0c40c39

Support parsing Dpkg OVAL with StAX

de20aca

Rename 'OVALLookupHelper' to 'OVALResourceCache'

8a82f5e

Parse and sync OVAL definitions in bulk

128843d

Introduce suseOVALOsProduct table

ea54803

- Also referenced the table from suseOVALPlatformVulnerablePackage

Specify the OS product to which the OVAL vulnerability entry belongs …

9526ce3

…in the database

Rename 'OVALProduct' to 'OVALOsProduct'

5cd00b8

Clear existing OVAL metadata correctly before inserting new data for …

0401bee

…a given OS product

HoussemNasri dismissed Etheryte’s stale review via 0401bee November 10, 2024 21:02

HoussemNasri force-pushed the oval-stax-parser branch from 07266bb to 0401bee Compare November 10, 2024 21:02

Remove redundant comment

5bbaefb

github-advanced-security bot found potential problems Nov 10, 2024

View reviewed changes

Use OVAL Stax parser in tests

495fe03

HoussemNasri force-pushed the oval-stax-parser branch from d69ee50 to 495fe03 Compare November 10, 2024 21:47

parlt91 reviewed Nov 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GSOC23] - C - Implement a StAX parser for OVAL files #7510

[GSOC23] - C - Implement a StAX parser for OVAL files #7510

HoussemNasri commented Sep 7, 2023 •

edited

Loading

github-actions bot commented Sep 7, 2023 •

edited

Loading

github-actions bot commented Oct 26, 2023 •

edited

Loading

github-actions bot commented Dec 26, 2023

github-actions bot commented Jan 5, 2024

mcalmer commented Jan 5, 2024

HoussemNasri commented Jan 5, 2024

mcalmer commented Jan 5, 2024

Etheryte left a comment

Etheryte left a comment

parlt91 left a comment

parlt91 left a comment

parlt91 Nov 14, 2024

parlt91 Nov 19, 2024

parlt91 Nov 19, 2024

parlt91 Nov 19, 2024

[GSOC23] - C - Implement a StAX parser for OVAL files #7510

Are you sure you want to change the base?

[GSOC23] - C - Implement a StAX parser for OVAL files #7510

Conversation

HoussemNasri commented Sep 7, 2023 • edited Loading

What does this PR change?

Useful Links

GUI diff

Documentation

Test coverage

Links

Changelogs

Re-run a test

github-actions bot commented Sep 7, 2023 • edited Loading

github-actions bot commented Oct 26, 2023 • edited Loading

github-actions bot commented Dec 26, 2023

github-actions bot commented Jan 5, 2024

mcalmer commented Jan 5, 2024

HoussemNasri commented Jan 5, 2024

mcalmer commented Jan 5, 2024

Etheryte left a comment

Choose a reason for hiding this comment

Etheryte left a comment

Choose a reason for hiding this comment

parlt91 left a comment

Choose a reason for hiding this comment

parlt91 left a comment

Choose a reason for hiding this comment

parlt91 Nov 14, 2024

Choose a reason for hiding this comment

parlt91 Nov 19, 2024

Choose a reason for hiding this comment

parlt91 Nov 19, 2024

Choose a reason for hiding this comment

parlt91 Nov 19, 2024

Choose a reason for hiding this comment

HoussemNasri commented Sep 7, 2023 •

edited

Loading

github-actions bot commented Sep 7, 2023 •

edited

Loading

github-actions bot commented Oct 26, 2023 •

edited

Loading