Refactor test data for constraints for conflicting test requirements

[aj-stein-gsa]

This is a ...

improvement - something could be better

This relates to ...

• the FedRAMP OSCAL Validations

User Story

As a FedRAMP developer, to better add or change tests for the constraints, I would like a tests to be able to effectively run with the right amount of test data, especially when two or more tests have conflicting test requirements that cannot exist (or not exist) in a singular test data file.

Goals

• Make tests and test data more manageable
• Make tests and their state operate independently
• Limit cognitive load and minimize conflicts (cognitive, tooling issues a la git merge) to reduce friction on development and maintenance of rules

Dependencies

N/A

Acceptance Criteria

• An architectural decision record (ADR) is made to explain what will stay the same or change given dev team discussions on the topic
• Tooling and CI/CD change issues will be added to the list below once the ADR has been review and team plans completion steps.
  • Adjust dev test tool to scaffold independent content files, not just modify ALL-INVALID.xml in Refactor test data for constraints for conflicting test requirements #691
• Review all other tests not covered by General cleanup related to 'system-characteristics' constraints #710 refactor and deprecate use of ALL-INVALID.xml, require each test to maintain its own test fixture file -INVALID.xml per-test, and delete ALL-INVALID.xml.

Once the second sublist is complete, this architecture decision will be complete and read to merge.

Other information

Today, the FR Automation Team devs had a meeting and posed an interesting challenge with the tests as they exist today in the feature branch, soon to be develop branch tests. When performing negative tests when an assembly or field should not exist, and an improper value must be tested, having all the tests target a single exhaustive list of invalid test vectors in one large SSP file exhibits certain challenges. We have decided to review this approach, consider alternative options, and act accordingly.

[aj-stein-gsa]

@Gabeblis thanks for volunteering to pick this one up. I hope the goals and background are clear for the ADR portion of this.

[Gabeblis]

@Gabeblis thanks for volunteering to pick this one up. I hope the goals and background are clear for the ADR portion of this.

Thank you!

[aj-stein-gsa]

The refactoring of the scripts in the test infrastructure is ready to move forward since #710 proved we can have a lot of broken tests that are hard to track. #690 should be ready for final review integration if work on it is complete.

/cc @wandmagic for awareness

[aj-stein-gsa]

And @Gabeblis, thanks for working through this and offering to pick up the "deprecate use of ALL-INVALID.xml" tasks to bring this to a close. @wandmagic, let us know when the updated test scaffolding is completely ready for final review (not urgent, ping Monday or early next week).

Have a good weekend, y'all. 🫡 👋

[wandmagic]

I think #690 could use a good review including testing of the dev-constraint test. according to my tests it seems to work pretty good @Gabeblis feel free to check it out to see if it meets the needs of the ADR. I've allowed people to still have the option to use all-invalid but marked the option as (deprecated), the default option is to generate a new XML invalid based on the constraint context

[Gabeblis]

I think #690 could use a good review including testing of the dev-constraint test. according to my tests it seems to work pretty good @Gabeblis feel free to check it out to see if it meets the needs of the ADR. I've allowed people to still have the option to use all-invalid but marked the option as (deprecated), the default option is to generate a new XML invalid based on the constraint context

Awesome, I'll start messing with it now. I'll aim to give you some feedback later today.

[aj-stein-gsa]

It seems we should have closed this issue before. I had reviewed the relevant PRs long ago and things are looking good now. One of us should have closed it sooner.