Issue with extraneous-implemented-requirements constraint

[Telos-sa]

This relates to ...

• the FedRAMP OSCAL Registry
• the FedRAMP OSCAL baselines
• the Guide to OSCAL-based FedRAMP Content
• the Guide to OSCAL-based FedRAMP System Security Plans (SSP)
• the Guide to OSCAL-based FedRAMP Security Assessment Plans (SAP)
• the Guide to OSCAL-based FedRAMP Security Assessment Results (SAR)
• the Guide to OSCAL-based FedRAMP Plan of Action and Milestones (POA&M)
• the FedRAMP SSP OSCAL Template (JSON or XML Format)
• the FedRAMP SAP OSCAL Template (JSON or XML Format)
• the FedRAMP SAR OSCAL Template (JSON or XML Format)
• the FedRAMP POA&M OSCAL Template (JSON or XML Format)
• the FedRAMP OSCAL Validations

What happened?

We are getting some new validation errors with the enhanced oscal-cli (v2.4.0). We haven't modified the structure of our implemented-requirements, but oscal-cli is now yielding the following error (for hundreds of controls) when validating against FedRAMP external constraints. This is an example of the error for ac-1:

[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[1]] extraneous-implemented-requirements: A FedRAMP SSP MUST NOT include extraneous controls outside of the FedRAMP baseline. Extraneous control: (ac-1).

I believe this might be related to #981 which appears to have reintroduced this constraint.

Relevant log output

[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[1]] extraneous-implemented-requirements: A FedRAMP SSP MUST NOT include extraneous controls outside of the FedRAMP baseline. Extraneous control: (ac-1).

How do we replicate this issue?

1. In the telos-fedramp-pilot repository we shared, execute the validation command:

oscal-cli validate Hogwarts\ SSP\ -\ 1.0\ \(2024-12-11T115022Z\).json -c fedramp_constraints/fedramp-external-allowed-values.xml fedramp_constraints/fedramp-external-constraints.xml

Where, exactly?

• enhanced oscal-cli v2.4.0
• most up to date fedramp-external-allowed-values.xml and fedramp-external-constraints.xml
• OSCAL SSP v1.1.2

Other relevant details

Our OSCAL SSP is one patch behind (v1.1.2 vs v1.1.3) - I have not been able to investigate if thats why this is occurring.

[aj-stein-gsa]

@wandmagic, you and I are going to have triage this one tomorrow.

[wandmagic]

Can you provide an example profile that can reproduce this error?

[Telos-sa]

@wandmagic I went ahead and invited you to the repository that I mentioned - this has the OSCAL SSP that generates these errors:

[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[1]] extraneous-implemented-requirements: A FedRAMP SSP MUST NOT include extraneous controls outside of the FedRAMP baseline. Extraneous control: (ac-1).

The import-profile>href is set to https://raw.githubusercontent.com/GSA/fedramp-automation/master/dist/content/rev5/baselines/xml/FedRAMP_rev5_LOW-baseline-resolved-profile_catalog.xml

[wandmagic]

I'll add this the unit test suite, and fix the constraint tommorow morning

[aj-stein-gsa]

I'll add this the unit test suite, and fix the constraint tommorow morning

@wandmagic just make sure we talk about that before you commit and push first please.

[wandmagic]

I tried running the content against the latest develop and was unable to reproduce the error, I did see several other validation error, especially a long one where has-data-flow-diagram-link-href-target produces an exception because we have two data-flow diagrams.

[wandmagic]

I added a github action to your repo to test against latest versions of the constraints, please take a look at the test output, to see whats left to get your pilot SSP to validate, its possible that an older version of fedramp constraints was used.

[Telos-sa]

@wandmagic
When I validate it with the oscal-external-constraints.xml file, it yields an error for import-profile:

An error occurred while evaluating the expression 'resolve-profile(doc(resolve-uri(import-profile/@href)))/catalog'. Unable to execute function 'fn:doc(arg1 as meta:string?) as document-node()'

When I run it with the fedramp-external-constraints.xml and fedramp-external-allowed-values.xml files (updated 2 hours ago) I still get the extraneous controls error:

[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[156]] extraneous-implemented-requirements: A FedRAMP SSP MUST NOT include extraneous controls outside of the FedRAMP baseline. Extraneous control: (sr-12).

Should import-profile>href be set to something else? It is https://raw.githubusercontent.com/GSA/fedramp-automation/master/dist/content/rev5/baselines/xml/FedRAMP_rev5_LOW-baseline-resolved-profile_catalog.xml currently

[wandmagic]

@Telos-sa please take a look at the pull request i made in your private repo, it contains a github action with results of the validation, where i was not able to reproduce this error it is possible that there is a network issue when the oscal CLI attempts to retrieve the baseline and interpret on your local machine. That is the correct uri to place in there

[aj-stein-gsa]

I would check your environment and/or use the container we provide, I am unable to get the extraneous error anymore. Please cross-check not just the branches, but also the current commit (git rev-parse HEAD) for our repos.

me@computer telos-fedramp-pilot % oscal-cli --version
oscal-cli 2.4.0 built at 2024-11-26 17:07 from branch dba6d9c570f0aa42022d9754df42d1dc5fc295d4 (dba6d9c) at https://github.com/metaschema-framework/oscal-cli
liboscal-java  built at 2024-11-26 16:40 from branch 2f3a394fa856e2bc90b74425c639c9bc107ea4e6 (2f3a394) at https://github.com/metaschema-framework/liboscal-java
oscal v1.1.3 built at 2024-11-26 16:40 from branch b123c11bd12c8b8f1bcc8bf85763e5775c0423e9 (b123c11) at https://github.com/usnistgov/OSCAL.git
metaschema-java 2.1.0 built at 2024-11-26T16:21:47+0000 from branch 462da0c64add5b369af740f4d2057643ac72b056 (462da0c) at https://github.com/metaschema-framework/metaschema-java
metaschema 2.1.0 built at 2024-11-26T16:21:47+0000 from branch b6601f7430f83f1a53a11bf32575b69e131bc912 (b6601f7) at https://github.com/metaschema-framework/metaschema.git
me@computer telos-fedramp-pilot % git branch --show-current
main
me@computer telos-fedramp-pilot % git rev-parse HEAD
82e80b82d9908857f44cb95949859262c4e5cc82
me@computer telos-fedramp-pilot % pushd /home/me/fedramp-automation
/home/me/fedramp-automation /home/me/telos-fedramp-pilot
me@computer fedramp-automation % git branch --show-current      
develop
me@computer fedramp-automation % git rev-parse HEAD             
48dd7f54ced994a03e5e455e545d686dc814c5c5
me@computer fedramp-automation % oscal-cli validate '/home/me/telos-fedramp-pilot/OSCAL SSP/Hogwarts SSP - 1.0 (2024-11-19T093555Z).json' -c '/home/me/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml' -c '/home/me/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml' -c '/home/me/fedramp-automation/src/validations/constraints/oscal-external-constraints.xml'
Loading 'file:/home/me/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml'
Loading 'file:/home/me/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml'
Loading 'file:/home/me/fedramp-automation/src/validations/constraints/oscal-external-constraints.xml'
Validating 'file:///home/me/telos-fedramp-pilot/OSCAL%20SSP/Hogwarts%20SSP%20-%201.0%20(2024-11-19T093555Z).json' as JSON.
me@computer fedramp-automation % oscal-cli validate '//home/me/telos-fedramp-pilot/OSCAL SSP/Hogwarts SSP - 1.0 (2024-11-19T093555Z).json' -c '//home/me/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml' -c '//home/me/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml' -c '//home/me/fedramp-automation/src/validations/constraints/oscal-external-constraints.xml'
Loading 'file://home/me/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml'
Loading 'file://home/me/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml'
Loading 'file://home/me/fedramp-automation/src/validations/constraints/oscal-external-constraints.xml'
Validating 'file:////home/me/telos-fedramp-pilot/OSCAL%20SSP/Hogwarts%20SSP%20-%201.0%20(2024-11-19T093555Z).json' as JSON.
Validation identified the following issues:
[ERROR] [/system-security-plan/system-implementation[1]/component[7]] component-has-authentication-method: A FedRAMP SSP MUST include at least one authentication method for each leveraged system.
[ERROR] [/system-security-plan/system-implementation[1]/component[8]] component-has-authentication-method: A FedRAMP SSP MUST include at least one authentication method for each leveraged system.
[ERROR] [/system-security-plan/system-implementation[1]/component[7]] component-has-non-provider-responsible-role: A FedRAMP SSP MUST have each component describing leveraged systems, interconnections, or authorized services identify at least one responsible role other than "provider".
[ERROR] [/system-security-plan/system-implementation[1]/component[8]] component-has-non-provider-responsible-role: A FedRAMP SSP MUST have each component describing leveraged systems, interconnections, or authorized services identify at least one responsible role other than "provider".
[ERROR] [/system-security-plan/system-implementation[1]/component[7]] component-has-provider-responsible-role: A FedRAMP SSP MUST have each component describing leveraged systems, interconnections, or authorized services identify a "provider" role that references one responsible party.
[ERROR] [/system-security-plan/system-implementation[1]/component[8]] component-has-provider-responsible-role: A FedRAMP SSP MUST have each component describing leveraged systems, interconnections, or authorized services identify a "provider" role that references one responsible party.
[ERROR] [/system-security-plan/system-implementation[1]/leveraged-authorization[1]/prop[3]/@value] leveraged-authorization-has-valid-impact-level: A FedRAMP SSP MUST define the appropriate FIPS-199 impact level (low, moderate, or high) for each leveraged authorization.
[ERROR] [/system-security-plan] non-provider-responsible-role-references-user: A FedRAMP SSP MUST have each component describing leveraged systems, interconnections, or authorized services reference at least one user with an authorized privilege and function performed via the "privilege-uuid" property.
[ERROR] [/system-security-plan/metadata[1]] data-center-alternate: There MUST be one or more alternate data center(s).
[ERROR] [/system-security-plan/metadata[1]] data-center-count: There MUST be at least two (2) data centers listed.
[ERROR] [/system-security-plan/metadata[1]] data-center-primary: There MUST be a single primary data center.
[ERROR] [/system-security-plan/metadata[1]] responsible-party-prepared-by-location-valid: A FedRAMP SSP MUST have a responsible party for preparing the document, and that party MUST define an address.
[ERROR] [/system-security-plan/metadata[1]] fedramp-version: A FedRAMP document's metadata MUST define a valid FedRAMP version.
[ERROR] [/system-security-plan/metadata[1]] marking: A FedRAMP document MUST have a marking that defines its data classification.
[WARNING] [/system-security-plan/metadata[1]/party[3]/telephone-number[1]] Value '1800-123-4567' did not match the pattern '^[0-9]{3}[0-9]{1,12}$' at path '/system-security-plan/metadata[1]/party[3]/telephone-number[1]'
[WARNING] [/system-security-plan/metadata[1]/party[5]/telephone-number[1]] Value '001-46785-115547' did not match the pattern '^[0-9]{3}[0-9]{1,12}$' at path '/system-security-plan/metadata[1]/party[5]/telephone-number[1]'
[WARNING] [/system-security-plan/metadata[1]/party[8]/telephone-number[1]] Value '12345678522-2' did not match the pattern '^[0-9]{3}[0-9]{1,12}$' at path '/system-security-plan/metadata[1]/party[8]/telephone-number[1]'
[WARNING] [/system-security-plan/metadata[1]/party[10]/telephone-number[1]] Value '123456789-4452' did not match the pattern '^[0-9]{3}[0-9]{1,12}$' at path '/system-security-plan/metadata[1]/party[10]/telephone-number[1]'
[ERROR] [/system-security-plan/system-characteristics[1]] has-cloud-service-model-remarks: A FedRAMP SSP with a cloud service model of "other" MUST supply remarks to explain this choice.
[ERROR] [/system-security-plan/system-characteristics[1]] has-fully-operational-date: A FedRAMP SSP MUST define the system's fully operational date.
[ERROR] [/system-security-plan/system-characteristics[1]] has-system-id: A FedRAMP SSP MUST have a FedRAMP system identifier.
[ERROR] [/system-security-plan/system-characteristics[1]/authorization-boundary[1]/diagram[1]/link[1]] has-authorization-boundary-diagram-link-href-target: A FedRAMP SSP MUST include an authorization boundary diagram.
[ERROR] [/system-security-plan/system-characteristics[1]/network-architecture[1]/diagram[1]/link[1]] has-network-architecture-diagram-link-href-target: A FedRAMP SSP MUST include a network architecture diagram.
[ERROR] [/system-security-plan/system-characteristics[1]/data-flow[1]/diagram[1]/link[1]] has-data-flow-diagram-link-href-target: A FedRAMP SSP MUST include a data flow diagram.
[ERROR] [/system-security-plan/system-characteristics[1]/data-flow[1]/diagram[2]/link[1]] has-data-flow-diagram-link-href-target: A FedRAMP SSP MUST include a data flow diagram.
[ERROR] [/system-security-plan/system-implementation[1]] has-inventory-items: A FedRAMP SSP system implementation section MUST have at least two inventory items.
[ERROR] [/system-security-plan/system-implementation[1]/component[8]] inter-boundary-component-direction-incoming-has-ipv-uri: Component a0d10f42-41eb-5407-97ae-bd35b3d2e455 (/system-security-plan/system-implementation[1]/component[8]) MUST have at least one local ipv4 address, ipv6 address, or a URI to an API.
[ERROR] [/system-security-plan/system-implementation[1]/component[8]] network-component-has-connection-security-prop: All network components in a FedRAMP SSP system implementation MUST define at least one interconnection security property.
[ERROR] [/system-security-plan/system-implementation[1]/component[9]] network-component-has-implementation-point: A FedRAMP SSP with service components and CLI software components performing cross-boundary network communication MUST indicate exactly one time if the point of implementation is internal or external to the system.
[ERROR] [/system-security-plan/system-implementation[1]/component[10]] network-component-has-implementation-point: A FedRAMP SSP with service components and CLI software components performing cross-boundary network communication MUST indicate exactly one time if the point of implementation is internal or external to the system.
[ERROR] [/system-security-plan/system-implementation[1]/component[11]] network-component-has-implementation-point: A FedRAMP SSP with service components and CLI software components performing cross-boundary network communication MUST indicate exactly one time if the point of implementation is internal or external to the system.
[ERROR] [/system-security-plan/system-implementation[1]/component[12]] network-component-has-implementation-point: A FedRAMP SSP with service components and CLI software components performing cross-boundary network communication MUST indicate exactly one time if the point of implementation is internal or external to the system.
[ERROR] [/system-security-plan/system-implementation[1]/component[8]/prop[5]/@value] external-system-nature-of-agreement: Value 'Contract' doesn't match one of 'contract, eula, isa, license, mou, other, or sla' at path '/system-security-plan/system-implementation[1]/component[8]/prop[5]/@value'
[WARNING] [/system-security-plan/system-implementation[1]/component[10]/protocol[1]] It is a best practice to provide a UUID.
[WARNING] [/system-security-plan/system-implementation[1]/component[11]/protocol[1]] It is a best practice to provide a UUID.
[WARNING] [/system-security-plan/system-implementation[1]/component[12]/protocol[1]] It is a best practice to provide a UUID.
[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[82]/statement[1]] missing-response-components: A FedRAMP SSP MUST identify how the system implements each control requirement implemented at the per-statement level and reference any component used to implement it.
[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[82]/statement[2]] missing-response-components: A FedRAMP SSP MUST identify how the system implements each control requirement implemented at the per-statement level and reference any component used to implement it.
[ERROR] [/system-security-plan/control-implementation[1]/implemented-requirement[82]] The cardinality '0' is below the required minimum '1' for items matching './/by-component'.
[ERROR] [/system-security-plan/back-matter[1]] has-fedramp-citations: A FedRAMP MUST be have exactly one resource with a link to the FedRAMP Laws, Regulations, Standards and Guidance, but 0 found.
[ERROR] [/system-security-plan/back-matter[1]/resource[6]/prop[1]/@value] Value 'separation-of-duties-matrix' doesn't match one of 'acronyms, administrators-guide, agreement, artifact, citation, evidence, external-guidance, image, interview-notes, law, logo, plan, policy, procedure, questionnaire, raw-data, regulation, report, rules-of-behavior, screen-shot, standard, system-guide, tool-output, or users-guide' at path '/system-security-plan/back-matter[1]/resource[6]/prop[1]/@value'
The file 'file:////home/me/telos-fedramp-pilot/OSCAL%20SSP/Hogwarts%20SSP%20-%201.0%20(2024-11-19T093555Z).json' is invalid.
me@computer fedramp-automation % oscal-cli validate '//home/me/telos-fedramp-pilot/OSCAL SSP/Hogwarts SSP - 1.0 (2024-11-19T093555Z).json' -c '//home/me/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml' -c '//home/me/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml' -c '//home/me/fedramp-automation/src/validations/constraints/oscal-external-constraints.xml' --no-color 2>&1| grep --only-matching --color -i extraneous
# No output returned

[aj-stein-gsa]

I will close for now, thanks to @wandmagic for helping you integrate a testing harness. If something else comes up that we are missing, reopen and let us now.

[Telos-sa]

@wandmagic @aj-stein-gsa
It was indeed a network issue - it completely slipped my mind that the VPN I'm on might be affecting any requests that are happening behind the scenes with the validator. My apologies!

[aj-stein-gsa]

Makes sense. If there's a long hang before constraints run and then you see control related issues it is likely network iff you are using a remote URL for the profiles and catalogs instead of a relative path. It's convenient but that's the trade off.

Let us know if you need anything else with new issues.

[wandmagic]

@wandmagic @aj-stein-gsa It was indeed a network issue - it completely slipped my mind that the VPN I'm on might be affecting any requests that are happening behind the scenes with the validator. My apologies!

to get around it you might have make local copies of nist catalog and baseline, glad we found the issue!