-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Problem Statement
The Trace Matrix logic is a little convoluted.
Proposed Solution
The TraceMatrix needs to provide the following capabilities:
- Export documentation showing for each requirement the test metrics (linked, passed, failed, not-executed)
- Export documentation showing for each test the requirement and metrics (passed, failed)
- Provide the list of unsatisfied requirements
It should satisfy these capabilities in the clearest and most concise way - with a design so obvious that its correct behavior is immediately obvious. Firstly we can recognize some basic facts:
- Every requirement ID is unique
- Any TestOutcome that is not 'IsExecuted' should be ignored
- The combination of TestFileBaseName and TestResult.Name may not be unique because they may come from different TestResult.ClassName, but these duplicate results can be collapsed to a single record counting the passes and fails.
The implementation should therefore be to create a record of TestExecution consisting of:
- FileNameBase (the base name of the test file)
- Name (the test name)
- Passes (number of passes in the file matching the test name)
- Fails (number of fails in the file matching the test name)
For fast access it's possible to assemble a Dictionary[TestName,List[TestExecution]]. This then becomes exceedingly easy to process - a simple FindTestExecutions method given the test name (with or without test source filter) can be:
- Split into test source and test name
- Look up the test name in the dictionary
- Iterate through the list of TestExecution gathering those tests matching the test source
The three core capabilities can be built on top of this core FindTestExecutions functionality.
Alternatives Considered
No response
Usage Examples
Benefits
Exceedingly easy processing, and using a dictionary by test name should improve performance. In almost all projects the number of unique tests is far larger than the number of test files, so the implementation optimizes in the important area.
Priority
Medium - Would improve my workflow
Willingness to Contribute
Yes, I can submit a pull request
Additional Context
Remember to update the architecture.
Checklist
- I have searched existing issues to ensure this is not a duplicate
- I have provided a clear description of the feature
- I have explained why this feature would be useful