Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update cth_surefire to report errors in end_per_testcase #4433

Merged
merged 1 commit into from
Dec 13, 2024

Conversation

chrzaszcz
Copy link
Member

@chrzaszcz chrzaszcz commented Dec 12, 2024

Since the CI build fails if there is any fail in end_per_testcase, our cth_surefire plugin generating the report should behave the same way.

I ran the unit test manually to confirm that the change is working correctly:

big_tests $ rebar3 eunit -m cth_surefire -v
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling mongoose_tests
===> Performing EUnit tests...
======================== EUnit ========================
module 'cth_surefire'
  cth_surefire: report_test...[1.908 s] ok
  cth_surefire: report_skipped_group_test...[1.698 s] ok
  cth_surefire: report_two_groups_test...[1.906 s] ok
  cth_surefire: report_skipped_suite_test...[1.679 s] ok
  [done in 7.203 s]
=======================================================
  All 4 tests passed.

@mongoose-im

This comment was marked as outdated.

Copy link

codecov bot commented Dec 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.36%. Comparing base (0f2536c) to head (37836b8).
Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4433      +/-   ##
==========================================
+ Coverage   84.84%   85.36%   +0.52%     
==========================================
  Files         549      549              
  Lines       33875    33875              
==========================================
+ Hits        28740    28919     +179     
+ Misses       5135     4956     -179     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@chrzaszcz chrzaszcz force-pushed the fix-cth-surefire branch 2 times, most recently from f16c5d3 to c4d16bc Compare December 13, 2024 08:15
@chrzaszcz chrzaszcz changed the title Update cth_surefire to report errors in end_per_testcase [WIP] Update cth_surefire to report errors in end_per_testcase Dec 13, 2024
Note: the 4th argument of post_end_per_testcase (Result) should be
      {fail, ...} (as documented), but it is actually {failed, ...}.
@mongoose-im

This comment was marked as outdated.

@mongoose-im

This comment was marked as outdated.

@mongoose-im

This comment was marked as outdated.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Dec 13, 2024

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / 37836b8
Reports root/ big
OK: 472 / Failed: 0 / User-skipped: 49 / Auto-skipped: 0


small_tests_26 / small_tests / 37836b8
Reports root / small


small_tests_27 / small_tests / 37836b8
Reports root / small


small_tests_27_arm64 / small_tests / 37836b8
Reports root / small


ldap_mnesia_26 / ldap_mnesia / 37836b8
Reports root/ big
OK: 2353 / Failed: 0 / User-skipped: 912 / Auto-skipped: 0


ldap_mnesia_27 / ldap_mnesia / 37836b8
Reports root/ big
OK: 2352 / Failed: 1 / User-skipped: 912 / Auto-skipped: 0

pubsub_SUITE:tree+basic:publish_only_retract_items_scope_test
{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {pubsub_SUITE,'-publish_only_retract_items_scope_test/1-fun-0-',2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_SUITE.erl"},
            {line,685}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1303}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1235}]}]}}

Report log


internal_mnesia_27 / internal_mnesia / 37836b8
Reports root/ big
OK: 2495 / Failed: 0 / User-skipped: 770 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 37836b8
Reports root/ big
OK: 4761 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 37836b8
Reports root/ big
OK: 4761 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0


dynamic_domains_mysql_redis_27 / mysql_redis / 37836b8
Reports root/ big
OK: 4726 / Failed: 0 / User-skipped: 154 / Auto-skipped: 0


pgsql_cets_27 / pgsql_cets / 37836b8
Reports root/ big
OK: 4850 / Failed: 0 / User-skipped: 188 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_27 / odbc_mssql_mnesia / 37836b8
Reports root/ big
OK: 4756 / Failed: 0 / User-skipped: 124 / Auto-skipped: 0


mysql_redis_27 / mysql_redis / 37836b8
Reports root/ big
OK: 5130 / Failed: 0 / User-skipped: 149 / Auto-skipped: 0


cockroachdb_cets_27 / cockroachdb_cets / 37836b8
Reports root/ big
OK: 4850 / Failed: 0 / User-skipped: 188 / Auto-skipped: 0


pgsql_mnesia_26 / pgsql_mnesia / 37836b8
Reports root/ big
OK: 5151 / Failed: 0 / User-skipped: 128 / Auto-skipped: 0


pgsql_mnesia_27 / pgsql_mnesia / 37836b8
Reports root/ big
OK: 5151 / Failed: 0 / User-skipped: 128 / Auto-skipped: 0


mssql_mnesia_27 / odbc_mssql_mnesia / 37836b8
Reports root/ big
OK: 5146 / Failed: 0 / User-skipped: 133 / Auto-skipped: 0


ldap_mnesia_27 / ldap_mnesia / 37836b8
Reports root/ big
OK: 117 / Failed: 0 / User-skipped: 2 / Auto-skipped: 0

@chrzaszcz chrzaszcz marked this pull request as ready for review December 13, 2024 09:08
Copy link
Contributor

@gustawlippa gustawlippa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update, looks good! 👍

@gustawlippa gustawlippa merged commit 952be3c into master Dec 13, 2024
4 checks passed
@gustawlippa gustawlippa deleted the fix-cth-surefire branch December 13, 2024 12:08
@jacekwegr jacekwegr added this to the 6.3.1 milestone Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants