Fi-1930: Custom Result Rollup #516

vanessuniq · 2024-07-15T14:29:30Z

Summary

This PR enhances the result rollup feature to support custom result logic through a user-defined block. The changes include:

New ResultCollection class: manages a collection of Inferno::Entities::Result objects
Updated TestGroup and TestSuite Classes: now has a customize_passing_result method that sets/ gets the custom block to define the passing criteria.
Updated Inferno::ResultSummarizer and Inferno::TestRunner: now takes custom result block into account when determining the overall result of a TestGroup or TestSuite.

Testing Guidance

Review changes.
Run unit tests and ensure passing
Run the new dev suite Custom Result Suite and ensure correct behavior

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

codecov · 2024-07-15T14:31:35Z

Codecov Report

Attention: Patch coverage is 96.61017% with 6 lines in your changes missing coverage. Please review.

Project coverage is 79.83%. Comparing base (9fab4c0) to head (f5e70ee).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...v_suites/dev_custom_results/custom_result_suite.rb	95.94%	3 Missing ⚠️
lib/inferno/dsl/messages.rb	85.71%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #516      +/-   ##
==========================================
+ Coverage   79.66%   79.83%   +0.17%     
==========================================
  Files         249      252       +3     
  Lines       13179    13305     +126     
  Branches     1283     1289       +6     
==========================================
+ Hits        10499    10622     +123     
- Misses       1931     1934       +3     
  Partials      749      749

Flag	Coverage Δ
backend	`91.87% <96.61%> (+0.18%)`	⬆️
frontend	`74.54% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

…mework/inferno-core into fi-1930-custom-result-rollup

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

…mework/inferno-core into fi-1930-custom-result-rollup

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm · 2024-08-02T12:10:46Z

dev_suites/dev_custom_results/custom_result_suite.rb

+    title 'Passing Group'
+    description 'Criteria: Passes if test 1 passes, or test 2 and test 3 pass'
+
+    customize_passing_result do |results|


I don't think this is quite there yet.

customize_passing_result is a confusing name.

This doesn't allow groups to do all of the things with results that tests can do, like add messages and summary messages

It would be great if assertions and the rest of the test dsl could be used here

What if groups had the option of containing a run block like tests do, and the results were made available via an attr_accessor?

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

vanessuniq · 2024-08-09T13:57:17Z

Test suite and group result now have messages and summary message. Only summary message for group is shown on the UI. messages are no shown.

Jammjammjamm · 2024-08-20T17:36:17Z

lib/inferno/entities/test_group.rb

-      def_delegators 'self.class', :title, :id, :groups, :inputs, :outputs, :tests
+      def_delegators 'self.class', :title, :id, :block, :groups, :inputs, :outputs, :tests
+
+      attr_accessor :result_message, :child_results


You think it's worth calling this child_results rather than just results?

I just wanted to add clarity that these are the children's results being used, but I'm totally fine simplifying it to results since it only handles those anyway.

Jammjammjamm · 2024-08-20T17:41:53Z

spec/inferno/result_collection_spec.rb

+  end
+
+  describe '#each' do
+    it 'iterates over each result in the collection' do


it 'returns.... You should check the return type of #each as well.

We need to test that it returns the correct thing when it's given a block, also.

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm · 2024-08-29T17:14:30Z

spec/inferno/result_collection_spec.rb

+      expect(result_collection.each).to be_a(Enumerator)
+    end
+
+    it 'returns the `results` attribute of the collection when a block is given' do


#each typically returns self, and I think it should here also, so that any further chaining happens on a ResultCollection rather than an Array.

true!! updated

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm · 2024-09-04T11:44:30Z

lib/inferno/test_runner.rb


+      parent_instance = parent.new
+      parent_instance.results << child_results


Is this actually tested? This should create a nested array, in contrast to the flat array created above.

ResultCollection#<< flattens the ResultCollection#results array.

vanessuniq added 2 commits July 15, 2024 10:16

Adding method to customize passing children required for a parent to …

8dac414

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize result rollup behavior

bc45caf

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

vanessuniq requested a review from Jammjammjamm July 15, 2024 14:29

vanessuniq self-assigned this Jul 15, 2024

vanessuniq marked this pull request as draft July 19, 2024 15:05

vanessuniq added 7 commits July 29, 2024 09:31

Implemented ResultCollection to manage a collection of results

cd53e63

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize passing criteria for suite/group

68358e2

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update to lookup results by full runnable id and partial id

c03ca6d

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Integrate custom result to result summarizer

7499e2e

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Upddated unit tests

6d233d7

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update logic of when parent's result need to be updated

d122c38

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Added dev suite for custom results

1968517

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

vanessuniq requested a review from arscan July 30, 2024 01:29

vanessuniq added 10 commits July 29, 2024 21:30

Adding method to customize passing children required for a parent to …

0e0f0d1

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize result rollup behavior

6e6ba6f

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Implemented ResultCollection to manage a collection of results

5389914

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize passing criteria for suite/group

98983d3

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update to lookup results by full runnable id and partial id

69fba7b

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Integrate custom result to result summarizer

3fbcda5

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Upddated unit tests

b7c9219

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update logic of when parent's result need to be updated

90ba917

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Added dev suite for custom results

14aebe2

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Merge branch 'fi-1930-custom-result-rollup' of github.com:inferno-fra…

ff898bc

…mework/inferno-core into fi-1930-custom-result-rollup

vanessuniq marked this pull request as ready for review July 30, 2024 01:36

vanessuniq added 5 commits July 30, 2024 10:37

Adding method to customize passing children required for a parent to …

803c7d3

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize result rollup behavior

a4ff6d6

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Implemented ResultCollection to manage a collection of results

3556267

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize passing criteria for suite/group

d00112b

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update to lookup results by full runnable id and partial id

4d0c76c

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

vanessuniq added 12 commits July 30, 2024 10:37

Upddated unit tests

aef4215

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Update logic of when parent's result need to be updated

d04b73b

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Added dev suite for custom results

f04dcdf

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Adding method to customize passing children required for a parent to …

77880cd

…pass Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize result rollup behavior

597b85f

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

customize passing criteria for suite/group

f9d0b41

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Integrate custom result to result summarizer

5e1ce58

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Upddated unit tests

23a2796

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Merge branch 'fi-1930-custom-result-rollup' of github.com:inferno-fra…

442753a

…mework/inferno-core into fi-1930-custom-result-rollup

cleanup

87aeb7b

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Added ResultCollection and custom result behavior unit tests

f3d6cd0

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

updated test_runner_spec

f7d7ce9

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm reviewed Aug 2, 2024

View reviewed changes

vanessuniq added 3 commits August 9, 2024 09:14

Refactored logic for custom result block

9004016

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

cleanup

530acb5

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

custom result suite minor uodate

48a1e1e

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

vanessuniq requested a review from Jammjammjamm August 9, 2024 15:38

Jammjammjamm reviewed Aug 20, 2024

View reviewed changes

vanessuniq and others added 5 commits August 21, 2024 12:46

testing the return type of ResultCollection#each

941a0e8

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

updated test suite and group to

5ce2374

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Merge branch 'main' into fi-1930-custom-result-rollup

985774d

Merge branch 'main' into fi-1930-custom-result-rollup

b9d4f41

updated result collection unit test

7075edb

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm reviewed Aug 29, 2024

View reviewed changes

Updated ResultCollection#each and ResultCollection#<< to return self

f5e70ee

Signed-off-by: Vanessa Fotso <vfotso@mitre.org>

Jammjammjamm reviewed Sep 4, 2024

View reviewed changes

Jammjammjamm approved these changes Sep 4, 2024

View reviewed changes

vanessuniq merged commit 586d3b8 into main Sep 4, 2024
10 checks passed

rpassas mentioned this pull request Sep 11, 2024

Inferno Core v0.4.40 #523

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fi-1930: Custom Result Rollup #516

Fi-1930: Custom Result Rollup #516

vanessuniq commented Jul 15, 2024 •

edited

Loading

codecov bot commented Jul 15, 2024 •

edited

Loading

Jammjammjamm Aug 2, 2024

vanessuniq commented Aug 9, 2024

Jammjammjamm Aug 20, 2024

vanessuniq Aug 21, 2024

Jammjammjamm Aug 20, 2024

vanessuniq Aug 21, 2024

Jammjammjamm Aug 28, 2024

Jammjammjamm Aug 29, 2024

vanessuniq Aug 29, 2024

Jammjammjamm Sep 4, 2024

vanessuniq Sep 4, 2024 •

edited

Loading


		parent_instance = parent.new
		parent_instance.results << child_results

Fi-1930: Custom Result Rollup #516

Fi-1930: Custom Result Rollup #516

Conversation

vanessuniq commented Jul 15, 2024 • edited Loading

Summary

Testing Guidance

codecov bot commented Jul 15, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

vanessuniq commented Aug 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vanessuniq Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

vanessuniq commented Jul 15, 2024 •

edited

Loading

codecov bot commented Jul 15, 2024 •

edited

Loading

vanessuniq Sep 4, 2024 •

edited

Loading