Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mark some scenarios as specific to SCE #12052

Merged
merged 2 commits into from
Jun 10, 2024

Conversation

jan-cerny
Copy link
Collaborator

@jan-cerny jan-cerny commented Jun 7, 2024

Description:

We introduce a new test scenarios header check that allows to mark test scenarios as specific to a single check engine type. For example, adding header check = sce to a test scenario marks this test scenario as specific only to SCE. If SCE check isn't available, such scenario will be skipped. If the check header isn't specified in a test scenario, the test scenario will work with any check type.

For SCE test scenarios we will also check if SCE support is installed in OpenSCAP in the back end and we will skip the test if SCE support isn't available there.

Rationale:

Fixes: #12030

Review Hints:

  1. ./build_product -d rhel9
  2. python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel9 set_nftables_table
  3. ADDITIONAL_CMAKE_OPTIONS="-DSSG_SCE_ENABLED=ON" ./build_product -d rhel9
  4. repeat the automatus command in step 2
  5. remove openscap-engine-sce from the virtual machine
  6. repeat the automatus command in step 2

We introduce a new test scenarios header `check` that allows
to mark test scenarios as specific to a single check engine
type. For example, adding header `check = sce` to a test scenario
marks this test scenario as specific only to SCE. If SCE check
isn't available, such scenario will be skipped. If the `check`
header isn't specified in a test scenario, the test scenario
will work with any check type.

Fixes: ComplianceAsCode#12030
@jan-cerny jan-cerny added this to the 0.1.74 milestone Jun 7, 2024
Copy link

github-actions bot commented Jun 7, 2024

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment
Open in Gitpod

Oracle Linux 8 Environment
Open in Gitpod

Copy link

github-actions bot commented Jun 7, 2024

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:12052
This image was built from commit: 6602496

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:12052

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:12052 make deploy-local

@jan-cerny
Copy link
Collaborator Author

/test 4.15-images

@Mab879 Mab879 self-assigned this Jun 7, 2024
@Mab879
Copy link
Member

Mab879 commented Jun 7, 2024

$  ADDITIONAL_CMAKE_OPTIONS="-DSSG_SCE_ENABLED=ON" ./build_product rhel9
$ python3 tests/automatus.py rule  --libvirt qemu:///system automatus_rhel9_4 set_nftables_table 
Setting console output to log level INFO
INFO - The data stream contains 3 Benchmarks
INFO - 0 - scap_org.open-scap_cref_ssg-rhel9-xccdf.xml
INFO - 1 - scap_org.open-scap_cref_rhel9-checks-sce-set_nftables_table.sh
INFO - 2 - scap_org.open-scap_cref_rhel9-checks-sce-ssh_keys_passphrase_protected.sh
INFO - Selected Benchmark is 0
INFO - To select a different Benchmark, use --xccdf-id-number option.
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/mburket/Developer/ComplianceAsCode/content/logs/rule-custom-2024-06-07-1527/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_set_nftables_table
ERROR - Script nftables_incorrect_family.fail.sh using profile (all) found issue:
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_no_tables.fail.sh using profile (all) found issue:
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage 
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_table_present.pass.sh using profile (all) found issue:
ERROR - Rule evaluation resulted in notchecked, instead of expected pass during initial stage 

I'm still getting some errors, after this fix. Is there something I'm missing?

@jan-cerny
Copy link
Collaborator Author

@Mab879 I think you're missing support for SCE in OpenSCAP running in your virtual machine. Check if openscap-engine-sce is installed in the VM and oscap --version contains the SCE capability.

This revealed a problem that this PR doesn't address the situation when the SCE check is going to be tested but OpenSCAP doesn't support SCE. We need to think how to address this situation.

For SCE test scenarios we will check if SCE is supported by
OpenSCAP installed in the testing back end. If OpenSCAP on the
back end doesn't support SCE we will display a warning and skip
the tests.

SCE support will be verified only once, during the back end
initialization.
@jan-cerny
Copy link
Collaborator Author

I have add logic that checks if the SCE tests are supported by OpenSCAP on the back end.

Copy link

codeclimate bot commented Jun 10, 2024

Code Climate has analyzed commit 6602496 and detected 1 issue on this pull request.

Here's the issue category breakdown:

Category Count
Complexity 1

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.4% (0.0% change).

View more on Code Climate.

Copy link
Member

@Mab879 Mab879 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new behavior works correctly.

Thanks.

@Mab879
Copy link
Member

Mab879 commented Jun 10, 2024

Waving the Rawhide tests as it appears to be failure in dnf.

Codeclimate finding can be waived for the moment.

@Mab879 Mab879 merged commit 5c526df into ComplianceAsCode:master Jun 10, 2024
111 of 113 checks passed
@Mab879 Mab879 added the Infrastructure Our content build system label Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Infrastructure Our content build system
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test scenarios fail for SCE-only rules if built without SCE
2 participants