Tests runs only ends in timeout despite npx start-test-run --finish command responds ok

[florianlefeuvre]

Describe the bug
At work we have a Cypress stack using Gherkin and your reporter, and our tests are launched in parallel through cypress-parallel dependency.

Everything works fine until we try to finish the runs through npx start-test-run --finish. This command seems to work as intended in the logs (it correctly responds Run <run-id> was finished), but on Testomat.io the run doesn't finish (until the 30 minutes timeout setup in our project settings).

To Reproduce
We have a .gitlab-ci.yml with a test:cypress job containing :

variables :
    TESTOMATIO: ${TESTOMATIO_TOKEN}
    TESTOMATIO_TITLE: 'Report ${env} at ${CI_PIPELINE_CREATED_AT} (pipeline: #${CI_PIPELINE_ID})'
    TESTOMATIO_RUN: ${TESTOMATIO_RUN_ID}
    TESTOMATIO_PROCEED: 1
    TESTOMATIO_ENV: ${env}
    SEVERITY_LEVEL: ''
before_script:
    - export TESTOMATIO_RUN=$(TESTOMATIO=${TESTOMATIO_TOKEN} npx start-test-run --launch | tail -1)
    - echo $TESTOMATIO_RUN > RUN_ID
script:
    - npm run parallel:$env$SEVERITY_LEVEL -- --threads ${THREADS_NUMBER}
after_script:
    - export TESTOMATIO_RUN=$(cat RUN_ID)
    - TESTOMATIO=${TESTOMATIO_TOKEN} npx start-test-run --finish

Exemple of script that can be launched through package.json :

    "run:preprod:critical": "cypress run --browser chrome --headless --env ENV=preprod,TAGS=\"not @ignore and not @exclude-from-preprod and @critical\"",
    "parallel:preprod:critical": "cypress-parallel --script run:preprod:critical --threads 4 --specsDir \"cypress/<path-to-features>/**/*.feature\"",

Expected behavior
Tests runs are launched in parallel (using different threads). When the job finishes the after_script launches npx start-test-run --finish, then the tests run on testomat.io should be flagged as finished. Instead, test run is still pending and finishes with a 30 minutes timeout.

Screenshots

• Logs from CI (job finishes) :
  
• Job status from Gitlab (finished 7 minutes ago)
  
• In testomat.io, test run is still running but all tests are finished and correctly uploaded (finish state is not triggered)
  

Additional context
We didn't have this issue before. It appeared weeks ago (despite not having updated anything in our stack). We then did try to update the reporter dependency (to 1.4.6) but it didn't help.

[DavertMik]

@florianlefeuvre

As a first guess I can assume TESTOMATIO_PROCEED=1 is used after_script so it completely blocks finishing run as it specifies that run should be kept

Move this variable to script so it would be used only on that section.

If this doesn't help please add the following variable

TESTOMATIO_DISABLE_BATCH_UPLOAD=1

This disables async processing of jobs which can also cause the similar issues

In the next Testomat.io update we will handle such cases so TESTOMATIO_DISABLE_BATCH_UPLOAD won't be needed

[DavertMik]

I re-checked this issue. Latest reporter 1.5.0 should handle this case correctly.

https://www.npmjs.com/package/@testomatio/reporter/v/1.5.0

[MathieuJouis]

Hello @DavertMik,

After upgrading to latest reporter v1.5.0, we see no change in the behavior.

The run is clearly indicated as finished on CI :

 🤩 Testomat.io Reporter v1.5.0
Finishing Run on Testomat.io...
[TESTOMATIO] Pipes: Testomatio Reporter
[TESTOMATIO] Testomatio Reporter v1.5.0
Run dbeaec1c was finished

Despite this, in testomat.io, the run remains in progress :

[olexandr13]

@florianlefeuvre could you reporoduce this problem locally?

[TetianaKhomenko]

Hi @MathieuJouis @florianlefeuvre! Does the issue still reproduce?

[MathieuJouis]

Hi @TetianaKhomenko ,

Unfortunately yes.

I tried with version 1.5.1 of the reporter, but the problem persists