HtmlFormatter only includes retried scenarios when surefire.rerunFailingTestsCount is used

[class101]

First, my deepest respects for the work you have done with cucumber.
It is a simply fascinating tool.

I love it so much that when I arrived on the project on which I am today,
the use that my team made of it challenged me.

They executed their 1450 scenarios in 6 hours, after studying all your
documentations and innovations such as @ScenarioScope, we achieved a runtime
of 17min, for the same 1450 tests!

Well ok I admit that they also used too much Thread.sleep, which I transformed
into Selenium's Fluent Wait ;)

It's one of the accomplishments I'm most proud of in my career and I wanted to
share it with you ladies and gentlemen, keep up the excellent work :)

👓 What did you see?

When I use the following Maven property failsafe.rerunFailingTestsCount,
property automatically recognized by the maven-failsafe-plugin,
I get the following actual incomplete report :

✅ What did you expect to see?

When I remove the property, the report is as expected, 8 errors in
total :

As a comparison, we're also using the excellent tool maven-cucumber-reporting
by @damianszczepanik and it seems to support the failsafe property just fine as
you can see below :)

📦 Which tool/library version are you using?

I observe the problem on the last 2 versions, i.e. 7.11.0-SNAPSHOT and
7.10.1, we are using jDK 11, Maven 3.8.7 and below the other libraries

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install 7.11.0-SNAPSHOT
2. Create any feature
3. In this feature, place 2 failing scenarios and 1 success
4. Setup failsafe, this is our setting

<!--
  Tests d'intégration IT (Selenium/Cucumber)
  ==========================================
-->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <executions>
        <execution>
            <id>integration-tests</id>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
            <configuration>
                <argLine>${jacoco.argLine.coverage} ${jacoco.argLine}</argLine>
                <skip>${skipITs}</skip>
                <useUnlimitedThreads>false</useUnlimitedThreads>
                <threadCount>${crelog.cucumber.thread.max}</threadCount>
                <perCoreThreadCount>false</perCoreThreadCount>
                <parallel>classesAndMethods</parallel>
                <includes>
                    <include>**/AllSeleniumITs.java</include>
                </includes>
            </configuration>
        </execution>
    </executions>
</plugin>

<properties>
  <!--
      Options liées au paralèllisme des TI remontées ici pour mise en évidence simplifiée
      (voir doc un peu plus bas ci-dessous pour usage)
  -->
  <cucumber.execution.parallel.enabled>true</cucumber.execution.parallel.enabled>
  <crelog.webdriver.scope>scenario</crelog.webdriver.scope>
  <!--
      Quelques durées observées :
      ===========================
        - 20x threads   : [INFO] cl-extranet-web .................................... SUCCESS [21:00 min]
        - 16x threads   : [INFO] cl-extranet-web .................................... SUCCESS [19:51 min]
        - 14x threads   : [INFO] cl-extranet-web .................................... SUCCESS [21:02 min]
        - 16x sans logs : [INFO] cl-extranet-web .................................... SUCCESS [17:36 min]
      ===========================
      Nombre de threads recommandé :
         - 16
      ===========================
      Mode de lancement recommandé :
         - 'mvn clean install' par cmd.exe, IDEA et toutes les applications de la VM fermées
      ===========================
      (tests réalisés sur VM 4x CPU 8Go sans antivirus, Trend-Micro est obervé causer un temps de compilation
       Maven 5x à 6x fois plus long par rapport à un temps de compilation sans antivirus)
  -->
  <crelog.cucumber.thread.max>16</crelog.cucumber.thread.max>
  <!--suppress UnresolvedMavenProperty
      ===========================
      Jeux de tests au choix (thread safe et non thread safe)
  -->
  <!--
    1386x scénarios threadsafe
  -->
  <cucumber.filter.tags>not @ignore and not @notThreadSafe</cucumber.filter.tags>
  <!--
    56x scénarios non-threadsafe à corriger
  -->
  <!--<cucumber.filter.tags>not @ignore and @notThreadSafe</cucumber.filter.tags>-->
  <!--
     Autres options Crelog des tests IT
     ==================================
     # headless.mode          : Si true, le navigateur sera masque [defaut: true]                                                  #
     # max.wait               : En secondes, le temps maximum d'attente pour un element attendu existant [defaut: 10]              #
     # max.wait.missing       : En millisecondes, le temps maximum d'attente pour un element attendu non-existant [defaut: 750]    #
     # webdriver.type         : Le navigateur a tester, chrome ou edge [defaut: chrome]                                            #
     # webdriver.scope        : singleton|scenario = un navigateur pour tous les tests ou un par scenario [defaut: singleton]      #
     #                            - si monothread  (cucumber...parallel.enabled=false) utiliser singleton (rapide) ou scenario (lent) #
     #                            - si multithread (cucumber...parallel.enabled=true) utiliser scenario (recommande)               #
     # webdriver.login.always : true=chaque scenario effectuera une authentification extranet (impact performance) [defaut: false] #
     # *.webdriver.path       : L'emplacement du driver navigateur [defaut: src/test/resources/drivers/*]                          #
     # user.agent             : Le user agent personnalise [def aut: vide]                                                         #
     #                          Exemples : Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko                   #
     #                                     Mozilla/4.0 (compatible; MSIE 11.0; Windows NT 6.3; Trident/7.0; .NET4.0E; .NET4.0C)    #
  -->
  <crelog.headless.mode>true</crelog.headless.mode>
  <crelog.max.wait>10</crelog.max.wait>
  <crelog.max.wait.missing>750</crelog.max.wait.missing>
  <crelog.webdriver.retry>30</crelog.webdriver.retry>
  <crelog.webdriver.type>chrome</crelog.webdriver.type>
  <crelog.webdriver.login.always>false</crelog.webdriver.login.always>
  <crelog.chrome.webdriver.logging.level>0</crelog.chrome.webdriver.logging.level> <!-- INFO -->
  <crelog.chrome.webdriver.path>src/test/resources/drivers/chromedriver.exe</crelog.chrome.webdriver.path>
  <crelog.edge.webdriver.path>src/test/resources/drivers/msedgedriver.exe</crelog.edge.webdriver.path>
  <crelog.user.agent />
  <crelog.list.navigateur.autorise>7,31-11-2020;8;9;10;11</crelog.list.navigateur.autorise>
  <crelog.list.navigateur.autorise.autre>MICROSOFT EDGE;CHROME</crelog.list.navigateur.autorise.autre>
  <crelog.current.role.id>superAdmin,orgaProd,recouvrement,orgaAccess,orgaNoAccess,interneAccess,interneNoAccess,ctxAccess,ctxNoAccess,reseauProd,interneToutFrontToutBack,orgaRnaa1</crelog.current.role.id>
  <!--
     Autres options cucumber centralisées ici pour simplifier car il est nécessaire qu'elle soit synchronisées
     dans 2 fichiers : cucumber.properties et junit-platform.properties, c'est donc le maven resources plugin
     qui fait les remplacements (filtering=true)
  -->
  <cucumber.execution.parallel.config.strategy>custom</cucumber.execution.parallel.config.strategy>
  <cucumber.execution.parallel.config.custom.class>fr.cl.extranet.it.config.ITParallelConfiguration</cucumber.execution.parallel.config.custom.class>
  <cucumber.glue>fr.cl.extranet.it.config</cucumber.glue>
  <cucumber.junit-platform.naming-strategy>long</cucumber.junit-platform.naming-strategy>
  <cucumber.publish.enabled>false</cucumber.publish.enabled>
  <cucumber.publish.quiet>true</cucumber.publish.quiet>
  <cucumber.plugin.stable>summary,json:target/cucumber-reporting-plugin-reports/inputJSON/AllSeleniumITs.json,html:target/cucumber-official-reports/index.html</cucumber.plugin.stable>
  <cucumber.plugin>pretty,${cucumber.plugin.stable}</cucumber.plugin> <!-- pretty est désactivé en ligne de commande (hors IDEA) car inutile et trop verbeux -->
  <failsafe.rerunFailingTestsCount>2</failsafe.rerunFailingTestsCount>
  <debug.port>0</debug.port>
</properties>

6. Add the offending maven property

<failsafe.rerunFailingTestsCount>2</failsafe.rerunFailingTestsCount>

📚 Any additional context?

We have tests that sometimes pass after the 2nd or 3rd attempt, that's why we
would like to keep the support of the failsafe option, it avoids all the
problems of false-positive test results that we had before, our
cycle of execution is now 100% stable thanks to rerunFailingTestsCount

We also think your work on cucumber's responsive HTML reporting interface is
fantastic, great job.

Lastly, I don't think you have any problems reproducing the problem, but if I
can ever help by trying to find the cause of the problem in the sources, I
will be happy to help you digging into the source code, I have all set here to
debug anywhere needed or by participating in a merge request,
I am quite comfortable with the use of git.

I have taken care to send you as much information as possible so that the
resolution is the most obvious. Maybe a little too much haha.

Cheers from France and happy new year 2023 everyone!

-- Arnaud Dovi
HN Services / Crédit-Logement

[mpkorstanje]

On behalve of everyone who's contributed to this I'd like thank you for your praise. Personally I'm happy to see that you've been able to achieve such a significant performance improvement. I'm glad you've found it useful.

As for your issue.

Unfortunately the way Surefire implements rerunFailingTestsCount is by starting a new execution of Cucumber, just for those failed tests. Currently Cucumber can't tell the difference and will happily overwrite any files it created in the first run, with the files from the second run.

Once JUnit 5 provides the ability for a test engine to store information in a session that spans both executions (i.e. junit-team/junit5#2816) it should in theory be possible to write the report only once with the reruns included.

As a comparison, we're also using the excellent tool maven-cucumber-reporting by damianszczepanik and it seems to support the failsafe property just fine as you can see below :)

So this is quite puzzling. From what I understand, the plugin reads the json files produced by Cucumber. Those should have been overwritten. Perhaps there is a more subtle bug going on? From memory, the plugin merges multiple report files. Could it be possible that you are looking at json files produces by previous tests or different tests?

[mpkorstanje]

And a few notes for the future:

1. Surefire currently closes the launcher and launcher session for the initial attempt and every rerun. This means that even if JUnit 5 were to support sessions for test engines, Cucumber would not be aware of them. As Maven plugins tend to release and develop at a glacial speed, I would not expect it to be possible to report on reran scenarios 1-2 years until after JUnit implements support for it.

2. The core of Cucumber JVM currently does not support retrying tests. There is no good default strategy for retrying or rerunning tests, nor is there sensible way to report on those retried test. I'm loathe to deal with the fractal of complexity that comes with implementing one or more specific implementation of a rerun strategy

   So while I'm happy to help make it possible to run the same scenario multiple times, I'd rather support this at a programmatic level only.

3. Cucumber JS currently supports rerunning, so the html report and the messages used to create the report should too.