Generate the CPE Dictionary dynamically #6304

yuumasato · 2020-10-27T20:11:02Z

Description:

This PR attempts to simplify definition syntax of CPE
- Removes the products' static cpe-dictionary and generate them dynamically based on the platform elements of the content in the shorthand xml.
- The build system identifies the required CPEs and generates the appropriate product cpe-dictionary.
The the main product doesn't need to contain cpe-items related to derivate product anymore.
- CPEs for derivative products are injected in the derivative product's cpe-dictionary.
Product related CPEs are now defined in their respective product.yml file.
Rule related CPEs are sourced from cpes_root defined in products.yml file.
- For the CPEs defined in upstream, cpes_root is shared/applicability/cpes.yml.
Remove constants XCCDF_PLATFORM_TO_CPE and PRODUCT_TO_CPE_MAPPING.
These mapping are now generated dynamically, based on the cpes defined to the product.

Rationale:

Simplify CPE declaration
Avoids the need to manually add a new cpe-item for each product.

openshift-ci-robot · 2020-10-27T20:11:04Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openscap-ci · 2020-10-27T20:11:22Z

Changes identified:
Others:
Changes in Python files.

Show details

Others:
Python abstract syntax tree change found in build-scripts/cpe_generate.py.
Python abstract syntax tree change found in build-scripts/enable_derivatives.py.
Python abstract syntax tree change found in build-scripts/yaml_to_shorthand.py.
Python abstract syntax tree change found in ssg/build_cpe.py.
Python abstract syntax tree change found in ssg/build_derivatives.py.
Python abstract syntax tree change found in ssg/build_yaml.py.
Python abstract syntax tree change found in ssg/constants.py.
Python abstract syntax tree change found in ssg/products.py.
Python abstract syntax tree change found in tests/ssg_test_suite/common.py.

Recommended tests to execute:
(cd build && cmake ../ && ctest -j4)

shared/applicability/cpes.yml

redhatrises · 2020-10-27T20:45:15Z

ssg/constants.py

+    ],
+    "centos8": [
+        "cpe:/o:centos:centos:8",
+    ],


all cpes need to be dropped from constants.py

Actually a lot of CPE mappings can be dropped from ssg/constants.py.
I'm not sure if it would better be done in this PR or a subsequent one.
I lean towards doing it in a separate PR.

sounds good

PRODUCT_TO_CPE_MAPPING was removed

redhatrises · 2020-10-27T20:47:43Z

shared/applicability/cpes.yml

+  "cpe:/a:redhat:enterprise_virtualization_manager:4":
+    id: rhvm4
+    title: "Red Hat Virtualization 4 Manager"
+    check_id: installed_app_is_rhv4


Overall structure needs to be cleaner and clearer like so:

- id: rhvm4 title: "Red Hat Virtualization 4 Manager" cpe: "cpe:/a:redhat:enterprise_virtualization_manager:4" check_id: installed_app_is_rhv4

there also needs to be an additional key called additional_cpes or include_cpes that maps to shared/applicability/cpes.yml

there also needs to be an additional key called additional_cpes or include_cpes that maps to shared/applicability/cpes.yml

Why, if the CPEs are picked up automatically based on the content included in the benchmark?

To not only make the content readable and understandable, but also remove the abstraction of the build system. Makes it easy for content authors and everyone to read and understand additional cpe is being added in product.yml

I find it rather cumbersome to, after selecting a rule in the profile, or adding to a rule's prodtype, to have to add any cpe-item it uses to the product.yml.
I'd expect the build-system to be handle that automatically for me (include the CPE in the dictionary of the product).

I think the readability and understandability should be solved in another way.
Maybe "compiled" products yamls? Build logs?

... having the cpe in the product.yml makes it easy for content authors to understand...

Imagine that there is a CPE in product.yml, and I think that it got there by mistake, or that the reason why it was introduced is not valid any more. What would I have to do in order to resolve those doubts?

Maybe I don't understand where you are heading...

If a CPE entry in a product.yml was not trivial, I would expect comment lines, if there weren't any I'd look into the commit that added it.

Everything has a cost - if the automation performs some magic behind the content author's back, that can have a cost, especially when the magic doesn't work right, or if the author would like something slightly different.

Conversely, if there is no automation, the obvious cost is that you have to add config items s.a. CPEs manually. But that's not where it ends - you get risk of copy-paste errors, and you also have to deal with removal of items that are no longer needed or that have been removed by refactoring.

I think I understand @redhatrises concern of automating too much, and I think that we should look both ways - what are the risks of automating too little?
So far it seems that we should require non-trivial manually added CPEs to be described by comments.

I agree that automation can make it hard to do something slightly different. But that was not the original point of the question, and I think it can be handled in a different way.

So far it seems that we should require non-trivial manually added CPEs to be described by comments.

Now I'm not sure if your previous question was about:

the manually added CPE for the product in product.yml or

the automatically added CPEs in the product's cpe-dictionary.

To clarify, the automation doesn't add anything to product.yml file, it will include cpe-item in the cpe-dictionary, as the files removed by this PR (rhel8/cpe/rhel8-cpe-dictionary.xml).
And to check the CPEs added by the build system one can check the built dictionary. It should have the same complexity as reviewing the old static files.

My question was about items that would have to be added to product.yml if there is no automation. The automation is supposed to make sure that only and only CPEs that are needed get added. Conversely, when a CPE is added manually, it has to be kept in sync with the actual content.

You have answered the question, thanks.

pep8speaks · 2020-10-29T13:46:49Z

Hello @yuumasato! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file build-scripts/enable_derivatives.py:

Line 113:100: E501 line too long (109 > 99 characters)

In the file ssg/build_cpe.py:

Line 100:66: E225 missing whitespace around operator
Line 100:100: E501 line too long (103 > 99 characters)
Line 103:5: E303 too many blank lines (2)
Line 108:17: E201 whitespace after '['
Line 108:64: E202 whitespace before ']'

Comment last updated at 2020-11-06 14:22:44 UTC

shared/applicability/cpes.yml

yuumasato · 2020-10-30T11:56:05Z

I'm currently updating documentation around CPEs.
Don't merge yet, please.

chromium/product.yml

redhatrises · 2020-11-03T21:56:01Z

jre/product.yml

+  jre-oracle:
+    name: "cpe:/a:oracle:jre:"
+    title: "Oracle's Java Runtime Environment"
+    check_id: installed_app_is_java


do we need jre-oracle? Why not skip it and start with - name?

The product CPES don't need to have these jre-oracle and jre-ibm... They are like IDs for the CPEs, but are not actually used by the build system.

But, the IDs of the CPEs in shared/applicability/cpes.yml are needed, these are IDs are referenced by rules and profiles.
I just tried to keep same declaration format.

The product CPES don't need to have these jre-oracle and jre-ibm... They are like IDs for the CPEs, but are not actually used by the build system.

But, the IDs of the CPEs in shared/applicability/cpes.yml are needed, these are IDs are referenced by rules and profiles.
I just tried to keep same declaration format.

So, if I am understanding you correctly, jre-oracle, jre-ibm, etc. aren't needed by the build system in product.yml but are needed by the build system in cpes.yml?

Well, the IDs are there for any content to refer to them.
But as they are already declared in the product.yml they are for, nothing else references them.
Rules and profiles refer to the IDs of cpes declared in cpes.yml.

I'd rather keep the syntax the same for cpes in product.yml and cpes.yml.

Actually, the CPE IDs are needed.
The ocp4 profiles reference CPE's defined in the product.yml.

redhatrises · 2020-11-03T22:01:04Z

chromium/product.yml

+  chromium:
+    name: "cpe:/a:google:chromium-browser"
+    title: "Google Chromium Browser"
+    check_id: installed_app_is_chromium


Just like the profiles_root, I want to have additional_cpes in this file, because as a content author, I don't want to choose to add cpes.yml or not. I might come with my own cpes.yml file that doesn't contain what we have now.

So you'd like to be able to choose the file to source the CPE's from?

So if product.yml specified a directory, and any yml (or even xml) file there was sourced. Would it satisfy your needs?

cpes_root: "../shared/applicability"

So you'd like to be able to choose the file to source the CPE's from?

So if product.yml specified a directory, and any yml (or even xml) file there was sourced. Would it satisfy your needs?

cpes_root: "../shared/applicability"

It definitely would address the needs!

Possibility to load multiple source CPE yml files added in 05506be

To avoid the need to manually add a new cpe-item for each product, let's generate the list of cpe-items based on the platforms included in the Benchmark (i.e. shorthand). The cpe-item for the derivate product is injected by the enable_derivatives.py script. The the main product doesn't need to contain cpe-items related to derivate product anymore.

Do no derivate the reference OVAL file from the cpe reference. They may be the same, but let's not carry this assumption.

Refactored the CPE loader to understand the new format and simpler.

One definition was overwriting the other.

The mapping is not static anymore, it is sourced directly from the cpes.yml file.

This should make management of product related CPEs easier.

From feedback, adding "-" improves readability of the CPEs. This changes the value of 'cpes' from a dictonary to a list of dictionaries.

Add parameter to product.yml to specify source directory of CPEs. As the source of CPEs can vary from product to product, now the "CPE loader" is product specific (via ProductCPEs class).

Let's not use the statically geenerated mapping, and use ProductCPEs to get the CPE names.

This mapping is now provided by class ProductCPEs via get_product_cpe_names() call.

This should make it easy to manage the CPEs.

redhatrises · 2020-11-11T19:36:13Z

At this point, looks good to me. I also like how you broke out the shared CPEs into categories. I think that that is a nice touch.

…ructure.

openshift-ci-robot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Oct 27, 2020

yuumasato force-pushed the unify_cpe_dictionaries branch from 80f0d77 to 22438ae Compare October 27, 2020 20:19

redhatrises suggested changes Oct 27, 2020

View reviewed changes

yuumasato force-pushed the unify_cpe_dictionaries branch from 6de663d to a1f18b5 Compare October 29, 2020 20:15

redhatrises reviewed Oct 29, 2020

View reviewed changes

shared/applicability/cpes.yml Outdated Show resolved Hide resolved

yuumasato force-pushed the unify_cpe_dictionaries branch from a1f18b5 to 892ee23 Compare October 29, 2020 21:50

yuumasato marked this pull request as ready for review October 29, 2020 22:01

openshift-ci-robot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Oct 29, 2020

openshift-ci-robot added the needs-rebase Used by openshift-ci bot. label Nov 1, 2020

redhatrises reviewed Nov 3, 2020

View reviewed changes

chromium/product.yml Outdated Show resolved Hide resolved

redhatrises reviewed Nov 3, 2020

View reviewed changes

chromium/product.yml Outdated Show resolved Hide resolved

redhatrises reviewed Nov 3, 2020

View reviewed changes

yuumasato force-pushed the unify_cpe_dictionaries branch from 9bce03a to 05506be Compare November 5, 2020 17:29

openshift-ci-robot removed the needs-rebase Used by openshift-ci bot. label Nov 5, 2020

yuumasato marked this pull request as draft November 5, 2020 17:41

openshift-ci-robot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Nov 5, 2020

yuumasato marked this pull request as ready for review November 5, 2020 21:12

openshift-ci-robot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Nov 5, 2020

yuumasato added 7 commits November 6, 2020 14:03

Decouple the CPE ref from the OVAL filename

9c0cdb4

Do no derivate the reference OVAL file from the cpe reference. They may be the same, but let's not carry this assumption.

Reformat cpes.yml for readability

804d558

Refactored the CPE loader to understand the new format and simpler.

Sort the cpe-items before generating the cpe-list

4dddc85

Fix definition of ocp4 CPES

38f72dd

One definition was overwriting the other.

Drop use of XCCDF_PLATFORM_TO_CPE

7d5d50c

The mapping is not static anymore, it is sourced directly from the cpes.yml file.

Move product CPEs to their product.ymls

269f018

This should make management of product related CPEs easier.

yuumasato added 9 commits November 6, 2020 14:05

Add cpes key to cpes.yml file for readability

8b3c61b

Document CPEsx

1918acd

Change CPE declaration syntax for readability

379ed72

From feedback, adding "-" improves readability of the CPEs. This changes the value of 'cpes' from a dictonary to a list of dictionaries.

Parametrize CPEs source file

e1b099b

Add parameter to product.yml to specify source directory of CPEs. As the source of CPEs can vary from product to product, now the "CPE loader" is product specific (via ProductCPEs class).

Extend CPE documentation

ff84f34

Fix insertion of derivative CPE into dictionary

f7d2228

Use ProductCPEs when dumping Profiles to xml

52d1d81

Let's not use the statically geenerated mapping, and use ProductCPEs to get the CPE names.

Remove product to CPE names mapping

ca3ad15

This mapping is now provided by class ProductCPEs via get_product_cpe_names() call.

Mention cpes_root in the examples readme

004a9aa

openshift-ci-robot added the needs-rebase Used by openshift-ci bot. label Nov 6, 2020

Split cpes.yml into multiple file

077eb49

This should make it easy to manage the CPEs.

yuumasato force-pushed the unify_cpe_dictionaries branch from f4a00cc to 077eb49 Compare November 6, 2020 14:22

openshift-ci-robot removed the needs-rebase Used by openshift-ci bot. label Nov 6, 2020

redhatrises merged commit 27db2b8 into ComplianceAsCode:master Nov 11, 2020

yuumasato deleted the unify_cpe_dictionaries branch November 12, 2020 08:55

yuumasato added this to the 0.1.54 milestone Nov 12, 2020

ggbecker added a commit to ggbecker/content that referenced this pull request Nov 16, 2020

Port devel documentation from ComplianceAsCode#6304 to readthedocs st…

ecceef9

…ructure.

ggbecker added a commit to ggbecker/content that referenced this pull request Nov 16, 2020

Port devel documentation from ComplianceAsCode#6304 to readthedocs st…

a5e6e36

…ructure.

This was referenced Dec 2, 2020

Translate CPE check ids for derivative products. #6447

Merged

Issue generating new content #6398

Closed

ggbecker mentioned this pull request Jan 8, 2021

Documentation on how to create new product is outdated with regards to new automated CPE mechanism #6518

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate the CPE Dictionary dynamically #6304

Generate the CPE Dictionary dynamically #6304

yuumasato commented Oct 27, 2020 •

edited

Loading

openshift-ci-robot commented Oct 27, 2020

openscap-ci commented Oct 27, 2020 •

edited

Loading

redhatrises Oct 27, 2020

yuumasato Oct 28, 2020

redhatrises Oct 29, 2020

yuumasato Nov 5, 2020

redhatrises Oct 27, 2020

redhatrises Oct 27, 2020

yuumasato Oct 28, 2020

redhatrises Oct 29, 2020

yuumasato Oct 29, 2020 •

edited

Loading

matejak Oct 30, 2020

yuumasato Nov 2, 2020

matejak Nov 2, 2020

yuumasato Nov 2, 2020

matejak Nov 3, 2020

pep8speaks commented Oct 29, 2020 •

edited

Loading

yuumasato commented Oct 30, 2020

redhatrises Nov 3, 2020

yuumasato Nov 4, 2020

redhatrises Nov 4, 2020

yuumasato Nov 5, 2020

yuumasato Nov 5, 2020

redhatrises Nov 3, 2020

yuumasato Nov 4, 2020 •

edited

Loading

redhatrises Nov 4, 2020

yuumasato Nov 5, 2020

redhatrises commented Nov 11, 2020

Generate the CPE Dictionary dynamically #6304

Generate the CPE Dictionary dynamically #6304

Conversation

yuumasato commented Oct 27, 2020 • edited Loading

Description:

Rationale:

openshift-ci-robot commented Oct 27, 2020

openscap-ci commented Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuumasato Oct 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Oct 29, 2020 • edited Loading

Comment last updated at 2020-11-06 14:22:44 UTC

yuumasato commented Oct 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuumasato Nov 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

redhatrises commented Nov 11, 2020

yuumasato commented Oct 27, 2020 •

edited

Loading

openscap-ci commented Oct 27, 2020 •

edited

Loading

yuumasato Oct 29, 2020 •

edited

Loading

pep8speaks commented Oct 29, 2020 •

edited

Loading

yuumasato Nov 4, 2020 •

edited

Loading