Introduce a JSON build manifest #10761

matejak · 2023-06-27T15:23:29Z

Description:

Summarize product's built content in a JSON file.
The format isn't definitive, but the PR illustrates that it is easy to provide certain key information about content from compiled artifacts.

Rationale:

Although right now, a datastream can be used as a single source of truth, it is difficult to parse it. The JSON format is much better in this regard, and it allows e.g. machine-readable high-level comparison of two different content releases.
An accompanying tool able to compare contents of two versions of ComplianceAsCode will be added to this PR later.

Review Hints:

Build a product, and examine the manifest.json in the build directory.

openshift-ci · 2023-06-27T15:23:33Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

github-actions · 2023-06-27T15:25:32Z

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

jan-cerny

My main concern is that the process of generating a data stream is complex and therefore there can be some difference between the manifest generated this way and the actual contents of data stream.

jan-cerny · 2023-06-28T08:22:24Z

build-scripts/generate_manifest.py

+            expected_filename = os.path.join(main_dir, expected_filename_template.format(id=rid))
+            if os.path.exists(expected_filename):
+                contents.add(content_id)
+        output_dict[rid] = list(contents)


For some reason, the list under the rule id is often empty, even if the remediation files exist.

I can't reproduce this, could you provide an example?

I can reproduce it using ./build_product rhel9 . There is a large amount of examples basically you can pick any rule that has this list empty in the manifest, eg. package_sudo_installed, aide_periodic_cron_checking, partition_for_tmp, and you will find that in fact there exists a bash or ansible.

jan-cerny · 2023-06-28T08:24:28Z

cmake/SSGCommon.cmake

+    add_custom_command(
+        OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/manifest.json"
+        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/generate_manifest.py" --output "${CMAKE_CURRENT_BINARY_DIR}/manifest.json" --build-root "${CMAKE_CURRENT_BINARY_DIR}"
+	DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/cpe-oval-unlinked.xml"


Why is the dependency cpe-oval-unlinked?

I mean: shouldn't this cmake command depend on a different target?

Because the generator uses compiled artifacts. While most of artifacts are compiled at the beginning of the build, the per-rule OVALs are made available only after that step that is marked as a dependency.
I will add a comment to clarify that, it is indeed confusing.
Moreover, it doesn't appear to be working.

matejak · 2023-06-28T14:54:42Z

My main concern is that the process of generating a data stream is complex and therefore there can be some difference between the manifest generated this way and the actual contents of data stream.

Yes, this manifest definitely reflects capabilities of the project. Although we are pretty sure that whatever is compiled makes its way to the datastream, it is not guaranteed.
OTOH, the manifest can remain stable regardless of what output types are produced, and datastreams won't be the default output forever. On the top of it, manifests are extremely easy to produce and to parse, which makes them attractive.

Collect built artifacts, and save the collection to a json file that represents the structure of the content available for the built product in a form that is more easy to process than the scanner-facing artifacts s.a. datastreams.

... to the manifest

This commit introduces a new script compare_versions.py which can show differences between two ComplianceAsCode versions. Lists added or removed rules, profiles, changes in profile composition and changes in remediations and platforms. For comparison, you can use git tags or ComplianceAsCode JSON manifest files directly. This new feature leverages the ComplianceAsCode JSON manifests, which have been introduced by ComplianceAsCode#10761

jan-cerny · 2023-06-30T07:26:54Z

cmake/SSGCommon.cmake

+        OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/manifest-${PRODUCT}.json"
+        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${SSG_BUILD_SCRIPTS}/generate_manifest.py" --output "${CMAKE_CURRENT_BINARY_DIR}/manifest-${PRODUCT}.json" --build-root "${CMAKE_CURRENT_BINARY_DIR}"
+        # The manifest requires compiled artifacts on right places and also per-rule OVAL.
+	# It is not clear when those things assume their places, so manifest is compiled late


Indentation problem. It causes a CI fail in the cmakelint test.

It may take a lot of steps to get things in place

codeclimate · 2023-06-30T09:36:25Z

Code Climate has analyzed commit f215139 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 53.5% (0.0% change).

View more on Code Climate.

jan-cerny

I have generated and reviewed the manifest for RHEL 9 and the data in there seem to be correct.

This commit introduces a new script compare_versions.py which can show differences between two ComplianceAsCode versions. Lists added or removed rules, profiles, changes in profile composition and changes in remediations and platforms. For comparison, you can use git tags or ComplianceAsCode JSON manifest files directly. This new feature leverages the ComplianceAsCode JSON manifests, which have been introduced by ComplianceAsCode#10761

This commit introduces a new script compare_versions.py which can show differences between two ComplianceAsCode versions. Lists added or removed rules, profiles, changes in profile composition and changes in remediations and platforms. For comparison, you can use git tags or ComplianceAsCode JSON manifest files directly. This new feature leverages the ComplianceAsCode JSON manifests, which have been introduced by ComplianceAsCode/content#10761

matejak added this to the 0.1.69 milestone Jun 27, 2023

openshift-ci bot added the do-not-merge/work-in-progress Used by openshift-ci bot. label Jun 27, 2023

jan-cerny reviewed Jun 28, 2023

View reviewed changes

matejak force-pushed the manifest branch from 966168e to a5ef8e5 Compare June 28, 2023 14:51

matejak added 3 commits June 29, 2023 10:59

Introduce a script summarizing contents to json

6751431

Collect built artifacts, and save the collection to a json file that represents the structure of the content available for the built product in a form that is more easy to process than the scanner-facing artifacts s.a. datastreams.

Add build of a JSON manifest file to the build process

baacc83

Introduce rule platform names and profile values

9f7095a

... to the manifest

matejak force-pushed the manifest branch from c33120e to 9f7095a Compare June 29, 2023 08:59

jan-cerny mentioned this pull request Jun 29, 2023

Introduce a script to compare ComplianceAsCode versions #10768

Merged

matejak force-pushed the manifest branch from 39a96f6 to b70403c Compare June 29, 2023 16:38

jan-cerny self-assigned this Jun 30, 2023

jan-cerny added the Highlight This PR/Issue should make it to the featured changelog. label Jun 30, 2023

jan-cerny reviewed Jun 30, 2023

View reviewed changes

matejak added 2 commits June 30, 2023 10:53

Defer creation of the manifest

fde8ce6

It may take a lot of steps to get things in place

Allow populating one content type by more means

f215139

matejak force-pushed the manifest branch from b70403c to f215139 Compare June 30, 2023 08:53

matejak marked this pull request as ready for review June 30, 2023 08:59

openshift-ci bot removed the do-not-merge/work-in-progress Used by openshift-ci bot. label Jun 30, 2023

jan-cerny approved these changes Jun 30, 2023

View reviewed changes

jan-cerny merged commit e2bfb40 into ComplianceAsCode:master Jun 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce a JSON build manifest #10761

Introduce a JSON build manifest #10761

matejak commented Jun 27, 2023

openshift-ci bot commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

jan-cerny left a comment

jan-cerny Jun 28, 2023

matejak Jun 28, 2023

jan-cerny Jun 29, 2023

jan-cerny Jun 28, 2023

jan-cerny Jun 28, 2023

matejak Jun 28, 2023 •

edited

Loading

matejak commented Jun 28, 2023

jan-cerny Jun 30, 2023

codeclimate bot commented Jun 30, 2023

jan-cerny left a comment

Introduce a JSON build manifest #10761

Introduce a JSON build manifest #10761

Conversation

matejak commented Jun 27, 2023

Description:

Rationale:

Review Hints:

openshift-ci bot commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

jan-cerny left a comment

Choose a reason for hiding this comment

jan-cerny Jun 28, 2023

Choose a reason for hiding this comment

matejak Jun 28, 2023

Choose a reason for hiding this comment

jan-cerny Jun 29, 2023

Choose a reason for hiding this comment

jan-cerny Jun 28, 2023

Choose a reason for hiding this comment

jan-cerny Jun 28, 2023

Choose a reason for hiding this comment

matejak Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

matejak commented Jun 28, 2023

jan-cerny Jun 30, 2023

Choose a reason for hiding this comment

codeclimate bot commented Jun 30, 2023

jan-cerny left a comment

Choose a reason for hiding this comment

matejak Jun 28, 2023 •

edited

Loading