feat: update test_results_parser version and start computing names #742

joseph-sentry · 2024-09-25T20:37:12Z

This commit updates the version of the test_results_parser library
that we are using:

no more support for parsing vitest.json or pytest-reportlog files
parse_junit_xml now returns a ParsingInfo class, the type signature
looks something like:

class ParsingInfo:
testruns: list[Testrun]
framework: framework

where Framework is:

class Framework(Enum):
Pytest
Jest
Vitest
PHPUnit
Testrun also now contains the filename field which is an optional
string and a classname field

We've also started computing the name of the test which just means
depending on the framework we've detected we transform the raw name
and classname from the JUnit file to something more readable and if
that computed name is available it's what we should show in the PR
comment

github-actions · 2024-09-25T20:37:26Z

This PR includes changes to shared. Please review them here: codecov/shared@403d5a2...106b0ae

codecov · 2024-09-25T20:46:17Z

Codecov Report

Attention: Patch coverage is 94.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 98.00%. Comparing base (3aa88f7) to head (6c6cb0b).
Report is 9 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
tasks/test_results_processor.py	89.15%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #742      +/-   ##
==========================================
- Coverage   98.03%   98.00%   -0.03%     
==========================================
  Files         438      438              
  Lines       36469    36451      -18     
==========================================
- Hits        35751    35725      -26     
- Misses        718      726       +8

Flag	Coverage Δ
integration	`98.00% <94.00%> (-0.03%)`	⬇️
unit	`98.00% <94.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <89.88%> (-0.06%)`	⬇️
OutsideTasks	`98.00% <100.00%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
database/models/reports.py	`99.53% <100.00%> (+<0.01%)`	⬆️
services/test_results.py	`90.00% <100.00%> (-0.06%)`	⬇️
services/tests/test_test_results.py	`100.00% <ø> (ø)`
tasks/test_results_finisher.py	`93.60% <ø> (ø)`
tasks/tests/unit/test_test_results_finisher.py	`100.00% <100.00%> (ø)`
...sks/tests/unit/test_test_results_processor_task.py	`100.00% <100.00%> (ø)`
tasks/test_results_processor.py	`94.27% <89.15%> (-4.08%)`	⬇️

... and 1 file with indirect coverage changes

codecov-staging · 2024-09-25T20:46:19Z

Codecov Report

Attention: Patch coverage is 94.00000% with 9 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
tasks/test_results_processor.py	89.15%	9 Missing ⚠️

@@            Coverage Diff             @@
##             main     #742      +/-   ##
==========================================
- Coverage   98.03%   98.00%   -0.03%     
==========================================
  Files         438      438              
  Lines       36469    36451      -18     
==========================================
- Hits        35751    35725      -26     
- Misses        718      726       +8

Flag	Coverage Δ
integration	`98.00% <94.00%> (-0.03%)`	⬇️
unit	`98.00% <94.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <89.88%> (-0.06%)`	⬇️
OutsideTasks	`98.00% <100.00%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
database/models/reports.py	`99.53% <100.00%> (+<0.01%)`	⬆️
services/test_results.py	`90.00% <100.00%> (-0.06%)`	⬇️
services/tests/test_test_results.py	`100.00% <ø> (ø)`
tasks/test_results_finisher.py	`93.60% <ø> (ø)`
tasks/tests/unit/test_test_results_finisher.py	`100.00% <100.00%> (ø)`
...sks/tests/unit/test_test_results_processor_task.py	`100.00% <100.00%> (ø)`
tasks/test_results_processor.py	`94.27% <89.15%> (-4.08%)`	⬇️

... and 1 file with indirect coverage changes

codecov-qa · 2024-09-25T20:46:29Z

Codecov Report

Attention: Patch coverage is 94.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 98.00%. Comparing base (3aa88f7) to head (6c6cb0b).
Report is 9 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
tasks/test_results_processor.py	89.15%	9 Missing ⚠️

@@            Coverage Diff             @@
##             main     #742      +/-   ##
==========================================
- Coverage   98.03%   98.00%   -0.03%     
==========================================
  Files         438      438              
  Lines       36469    36451      -18     
==========================================
- Hits        35751    35725      -26     
- Misses        718      726       +8

Flag	Coverage Δ
integration	`98.00% <94.00%> (-0.03%)`	⬇️
unit	`98.00% <94.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <89.88%> (-0.06%)`	⬇️
OutsideTasks	`98.00% <100.00%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
database/models/reports.py	`99.53% <100.00%> (+<0.01%)`	⬆️
services/test_results.py	`90.00% <100.00%> (-0.06%)`	⬇️
services/tests/test_test_results.py	`100.00% <ø> (ø)`
tasks/test_results_finisher.py	`93.60% <ø> (ø)`
tasks/tests/unit/test_test_results_finisher.py	`100.00% <100.00%> (ø)`
...sks/tests/unit/test_test_results_processor_task.py	`100.00% <100.00%> (ø)`
tasks/test_results_processor.py	`94.27% <89.15%> (-4.08%)`	⬇️

... and 1 file with indirect coverage changes

codecov-public-qa · 2024-09-25T20:46:34Z

Codecov Report

Attention: Patch coverage is 94.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 98.00%. Comparing base (3aa88f7) to head (6c6cb0b).
Report is 1 commits behind head on main.

✅ All tests successful. No failed tests found.

@@            Coverage Diff             @@
##             main     #742      +/-   ##
==========================================
- Coverage   98.03%   98.00%   -0.03%     
==========================================
  Files         438      438              
  Lines       36469    36451      -18     
==========================================
- Hits        35751    35725      -26     
- Misses        718      726       +8

Flag	Coverage Δ
integration	`98.00% <94.00%> (-0.03%)`	⬇️
unit	`98.00% <94.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <89.88%> (-0.06%)`	⬇️
OutsideTasks	`98.00% <100.00%> (+<0.01%)`	⬆️

Files	Coverage Δ
database/models/reports.py	`99.53% <100.00%> (+<0.01%)`	⬆️
services/test_results.py	`90.00% <100.00%> (-0.06%)`	⬇️
services/tests/test_test_results.py	`100.00% <ø> (ø)`
tasks/test_results_finisher.py	`93.60% <ø> (ø)`
tasks/tests/unit/test_test_results_finisher.py	`100.00% <100.00%> (ø)`
...sks/tests/unit/test_test_results_processor_task.py	`100.00% <100.00%> (ø)`
tasks/test_results_processor.py	`94.27% <89.15%> (-4.08%)`	⬇️

... and 1 file with indirect coverage changes

This commit updates the version of the test_results_parser library that we are using: - no more support for parsing vitest.json or pytest-reportlog files - parse_junit_xml now returns a ParsingInfo class, the type signature looks something like: class ParsingInfo: testruns: list[Testrun] framework: framework where Framework is: class Framework(Enum): Pytest Jest Vitest PHPUnit - Testrun also now contains the filename field which is an optional string and a classname field We've also started computing the name of the test which just means depending on the framework we've detected we transform the raw name and classname from the JUnit file to something more readable and if that computed name is available it's what we should show in the PR comment

github-actions · 2024-10-03T15:21:45Z

This PR includes changes to shared. Please review them here: https://github.com/codecov/shared/compare/88117b96a4b420d88549b8df2649c3eb9c61c2a5...

…k-detection

github-actions · 2024-10-08T15:57:35Z

This PR includes changes to shared. Please review them here: https://github.com/codecov/shared/compare/88117b96a4b420d88549b8df2649c3eb9c61c2a5...

…k-detection

github-actions · 2024-10-09T18:33:10Z

This PR includes changes to shared. Please review them here: https://github.com/codecov/shared/compare/b9e83b2c2f072749bae36c18b80d384b8cc50d94...

michelletran-codecov

Just a few comments

michelletran-codecov · 2024-10-09T19:21:32Z

tasks/test_results_processor.py

+                print("BBBBBBBBBBBBBBBBBBBBBBBBBBBBB")
        except ParserError as e:
+            print("AAAAAAAAAAAAAAAAAAAAAAAAAAAAA")


Remove print statements

michelletran-codecov · 2024-10-09T19:26:18Z

tasks/tests/samples/sample_test_network.txt

@@ -0,0 +1 @@
+{"network_files": ["api/temp/calculator/test_calculator.py"], "test_results_files": [{"filename": "codecov-demo/temp.junit.xml", "format": "base64+compressed", "data": "eJy1VMluwjAQvfMVI1dCoBbHZiklJEFVS4V66Kkqx8okBqw6i2KHwt/XWSChnCrRXDLjefNm8Uuc2T6UsOOpEnHkIooJAh75cSCijYsyve49oJnnaK60yoR5NWyIWMhdlBzyE5OWpnGqXGQY1kzILOXGoQjUl0gSHhSBItdFQ2OJPJdgMuqXjtIsTFzUJ/1Bj9IeuX+n1KZjmwwxoZSMDWwbK13W/HhZvC1fn5eLCZaxzyQq2/KZ4uBLplQJY4nAmocJNhA/k0zHKc5xn7WPqimKYxYEjc6Iad66DrHKVjplvv4f9jCTWiTy0GQnV2MPxE4E/Lxzz6muGMzFKbbJaZXiqQajIHBdIHjUvqFkCrcA31tugEUA2lJP11nkayM3eKobKIsA00D2lAz9CV9NSHujpx16B/3uievI9mceU/sChryAr6ExZKdrtwpw+VQjXeSVPVVjtubn6HoBp8iVlnDGd9VFtFpGFFYuCqsWgfVLFDg52ANiw2Mxpyk3zz94x6qU4DnWUW2VWfwkmrbyfgBbcXMH", "labels": ""}], "metadata": {}}


nit: Can we format these json better? This will make it easier to read and debug errors.

michelletran-codecov · 2024-10-09T19:27:47Z

tasks/test_results_processor.py

+        # TODO: this is bad
+        self.network = data.get("network_files")


Can you add more context into why this is bad? I see that network_files is supposed to be in the uploaded test file, but I don't have much context into why getting this here is a bad thing?

ah, this was a note for myself to revisit, the alternative to this is to return the network from the function, which i think makes more sense, but i wanted to quickly make sure it worked so i just set it as an attribute on the task itself, will fix this with the next commit

- change test_results_processor to pass network through to compute_name instead of setting it as an attribute on the Task - change sample_test files from txt to json

michelletran-codecov

Generally LGTM! Just a question about one of the class names.

michelletran-codecov · 2024-10-09T20:38:52Z

tasks/test_results_processor.py

+                name = raw_name
+                return name


simplification suggestion:

Suggested change

name = raw_name

return name

return raw_name

michelletran-codecov · 2024-10-09T20:44:50Z

tasks/test_results_processor.py

        test_instance_data = []
        test_flag_bridge_data = []
-        daily_totals = dict()
+        daily_totals: dict[str, dict[str, str | int | list[str]]] = dict()


Not something to do in this PR, but I would consider creating a custom object/type/dataclass to simplify this complex type annotation.

michelletran-codecov · 2024-10-09T20:46:30Z

tasks/test_results_processor.py

+
+
+@dataclass
+class argProcessingResult:


Is this supposed to be ProcessingResult or ArgProcessingResult?

joseph-sentry marked this pull request as draft September 25, 2024 20:37

joseph-sentry requested a review from a team October 1, 2024 18:41

joseph-sentry added 2 commits October 3, 2024 11:19

deps: update shared

99b60ae

joseph-sentry force-pushed the joseph/framework-detection branch from cf09fe4 to 1df0658 Compare October 3, 2024 15:21

joseph-sentry marked this pull request as ready for review October 3, 2024 15:28

joseph-sentry marked this pull request as draft October 3, 2024 15:30

joseph-sentry added 2 commits October 3, 2024 11:40

fix: properly indent if statement

cccdfcf

Merge branch 'main' of github.com:codecov/worker into joseph/framewor…

0f4f276

…k-detection

joseph-sentry marked this pull request as ready for review October 8, 2024 15:57

Merge branch 'main' of github.com:codecov/worker into joseph/framewor…

0d22285

…k-detection

michelletran-codecov reviewed Oct 9, 2024

View reviewed changes

fix: address feedback

4d4eede

- change test_results_processor to pass network through to compute_name instead of setting it as an attribute on the Task - change sample_test files from txt to json

joseph-sentry requested a review from michelletran-codecov October 9, 2024 20:35

michelletran-codecov reviewed Oct 9, 2024

View reviewed changes

fix: address feedback

6c6cb0b

michelletran-codecov approved these changes Oct 9, 2024

View reviewed changes

joseph-sentry added this pull request to the merge queue Oct 15, 2024

Merged via the queue into main with commit 39fb207 Oct 15, 2024
17 of 27 checks passed

joseph-sentry deleted the joseph/framework-detection branch October 15, 2024 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: update test_results_parser version and start computing names #742

feat: update test_results_parser version and start computing names #742

joseph-sentry commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

codecov bot commented Sep 25, 2024 •

edited

Loading

codecov-staging bot commented Sep 25, 2024 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Sep 25, 2024 •

edited

Loading

codecov-public-qa bot commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 9, 2024

michelletran-codecov left a comment

michelletran-codecov Oct 9, 2024

michelletran-codecov Oct 9, 2024

michelletran-codecov Oct 9, 2024

joseph-sentry Oct 9, 2024

michelletran-codecov left a comment

michelletran-codecov Oct 9, 2024

michelletran-codecov Oct 9, 2024

michelletran-codecov Oct 9, 2024

		@@ -0,0 +1 @@
		{"network_files": ["api/temp/calculator/test_calculator.py"], "test_results_files": [{"filename": "codecov-demo/temp.junit.xml", "format": "base64+compressed", "data": "eJy1VMluwjAQvfMVI1dCoBbHZiklJEFVS4V66Kkqx8okBqw6i2KHwt/XWSChnCrRXDLjefNm8Uuc2T6UsOOpEnHkIooJAh75cSCijYsyve49oJnnaK60yoR5NWyIWMhdlBzyE5OWpnGqXGQY1kzILOXGoQjUl0gSHhSBItdFQ2OJPJdgMuqXjtIsTFzUJ/1Bj9IeuX+n1KZjmwwxoZSMDWwbK13W/HhZvC1fn5eLCZaxzyQq2/KZ4uBLplQJY4nAmocJNhA/k0zHKc5xn7WPqimKYxYEjc6Iad66DrHKVjplvv4f9jCTWiTy0GQnV2MPxE4E/Lxzz6muGMzFKbbJaZXiqQajIHBdIHjUvqFkCrcA31tugEUA2lJP11nkayM3eKobKIsA00D2lAz9CV9NSHujpx16B/3uievI9mceU/sChryAr6ExZKdrtwpw+VQjXeSVPVVjtubn6HoBp8iVlnDGd9VFtFpGFFYuCqsWgfVLFDg52ANiw2Mxpyk3zz94x6qU4DnWUW2VWfwkmrbyfgBbcXMH", "labels": ""}], "metadata": {}}

feat: update test_results_parser version and start computing names #742

feat: update test_results_parser version and start computing names #742

Conversation

joseph-sentry commented Sep 25, 2024

github-actions bot commented Sep 25, 2024

codecov bot commented Sep 25, 2024 • edited Loading

Codecov Report

codecov-staging bot commented Sep 25, 2024 • edited by codecov-notifications bot Loading

Codecov Report

codecov-qa bot commented Sep 25, 2024 • edited Loading

Codecov Report

codecov-public-qa bot commented Sep 25, 2024 • edited Loading

Codecov Report

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 9, 2024

michelletran-codecov left a comment

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

joseph-sentry Oct 9, 2024

Choose a reason for hiding this comment

michelletran-codecov left a comment

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

michelletran-codecov Oct 9, 2024

Choose a reason for hiding this comment

codecov bot commented Sep 25, 2024 •

edited

Loading

codecov-staging bot commented Sep 25, 2024 •

edited by codecov-notifications bot

Loading

codecov-qa bot commented Sep 25, 2024 •

edited

Loading

codecov-public-qa bot commented Sep 25, 2024 •

edited

Loading