Use of applicability_check in OVAL content results in fail/false rather than 'not checked' or 'not applicable'

[redhatrises]

I talked to @iankko on IRC about this.

• Uploaded some highlights of the irc chat -
  irc_chat.txt

Essentially adding applicability_check="true" attribute to criteria/criterion/extend_definition results in fail/false. I would expect that if applicability_check evaluates as false it would return 'not checked' or 'not applicable'. Otherwise if applicability_check evaluates as true, evaluate the other criteria as true (if configured correctly) or false (if not configured correctly).

[GaryGapinski]

Having had the recent misfortune to look closely at the SCAP 1.2 normative publications, I can assert that the applicability_check attribute is unfortunately not defined with respect to any effect on OVAL or XCCDF result logic and values. In particular, the schema for http://oval.mitre.org/XMLSchema/oval-results-5 ResultEnumerationdoes not allow a result to be affected by applicability_check, and NIST IR 7275r4 §6.4.4.4 tables 12, 13, and 14 as well as §6.6.4.2 table 26 have a scope of just XCCDF.

NIST SP 800-126r2 has an OVAL to XCCDF result mapping in §4.5.2 that does not address that attribute and precludes any result indicating applicability (it also lacks normative mapping for class="miscellaneous").

The OVAL specification would have to be changed to link ResultEnumeration "not applicable" to criteria/criterion/extend_definition applicability_check="true". That seems eminently reasonable, which will adversely affect the chances of such happening. Worse, an additional result of "applicable" would be needed along with an additional mapping from OVAL result to XCCDF result. It may be easier to devise alternative, improved checklist and evaluation languages.

[jan-cerny]

@redhatrises I agree with @GaryGapinski. From what I understand from the specification, the applicability_check attribute is only informational and should not affect the definition result. It seems to me similar to the affected element, which also does not perform any check and does not influence the result. I agree that it seems confusing, because using common sense, I would expect the result of not applicable.

[redhatrises]

I can see having a result of 'not applicable' being a valuable enhancement especially if a check is targeted towards a specific operating system in the same profile. For example, checking if the operating system is RHEL7 in the STIG profile to validate FIPS requirements should evaluate as 'true/false' on RHEL7, but 'not applicable' on CentOS 7.

An additional result of 'applicable' would not at all need to be added, because if the check is applicable, it will evaluate at 'true/false/error...' and not 'applicable'.

[isimluk]

I have had similar misfortune experience as @GaryGapinski, when I dived down to the papers. That being said, I agree with all the conclusions above.

• The OVAL language specifies when we (oscap) can use notapplicable result. Details at http://oval.mitre.org/language/version5.11/ovalresults/documentation/oval-results-schema.html#ResultEnumeration
• The current specification does not allow applicability_check to have an effect on the result.
• Hence, oscap does not implement any effects.
• The introduction of applicability_check to scap-security-guide will (sadly) have no effects.

My understanding of OVAL process is of today, is that right path to proceed with this would be to start thread at https://github.com/OVALProject/Language/issues

[solind]

I just happened to be looking at the applicability_check myself and stumbled upon this discussion! I agree, this attribute does not seem to be well-specified. To the extent that any information can be found pertaining to it, the latest discussion does suggest that the attribute is purely informational; see:
http://making-security-measurable.1364806.n2.nabble.com/applicability-check-question-td7578733.html

However, there doesn't seem to be any conclusive discussion of the functionality that was ultimately supposed to be adopted. That puts the attribute in a gray area, in my opinion.

There is currently no official home-page for the OVAL language, although we (the community) are actively working on a new one. The Github site you referenced (OVALProject/Language) was set up by MITRE and is not actively maintained by the community.

If you're interested in engaging the larger OVAL developer community to discuss this issue, and actually even make a change to the language itself, I'd suggest using the official OVAL developer email list: http://lists.cisecurity.org/mailman/listinfo/oval_developer_lists.cisecurity.org