e2e-cypress quickstarter hangs on 'verify'

[jchaves]

Describe the bug
While building a repository created with Cypress quickstart on ODS4, it fails to run

Cypress seems to try and launch a self-verification on install, and that causes the whole build to crahs after the default timeout of 30000 miliseconds

To Reproduce

1. Provision a Cypress quickstarter project
2. trigger the pipeline
3. See error

Expected behavior
The default quickstart pipeline succeeds

Screenshots

Affected version (please complete the following information):

• OpenShift: 3.11
• OpenDevStack 4.x

Log Output (ensure to remove any confidential information like tokens, project names, etc.

> cypress run --config-file cypress-installation.json --browser chrome

It looks like this is your first time using Cypress: 7.5.0


Cypress verification timed out.

This command failed with the following output:

/home/jenkins/.cache/Cypress/7.5.0/Cypress/Cypress --no-sandbox --smoke-test --ping=312

----------

Command timed out after 30000 milliseconds: /home/jenkins/.cache/Cypress/7.5.0/Cypress/Cypress --no-sandbox --smoke-test --ping=312
Timed out

----------

Platform: linux (Red Hat Linux - 7.9)
Cypress Version: 7.5.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! e2e@0.0.0 cypress:run-installation: `cypress run --config-file cypress-installation.json --browser chrome`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the e2e@0.0.0 cypress:run-installation script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2021-11-19T17_25_46_603Z-debug.log
ERROR: "cypress:run-installation" exited with 1.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! e2e@0.0.0 e2e: `npm-run-all lint build delete-junit-results cypress:run-installation cypress:run-integration cypress:run-acceptance`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the e2e@0.0.0 e2e script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/jenkins/.npm/_logs/2021-11-19T17_25_46_622Z-debug.log

[jchaves]

@segator @braisvq1996

[clemensutschig]

@jchaves - Can you cat /home/jenkins/.npm/_logs/2021* then we see the output

Does this happen on qa only?
Maybe the site under test is down ?

https://swingli.com/how-to-fix-cypress-verification-timed-out-exception/#

It's weird - working super nicely in the ci build - proxy issue maybe (just thinking out loud)

https://stackoverflow.com/questions/63667880/cypress-verification-timed-out-after-30000-milliseconds contains a bunch of ideas

[segator]

We already tried to increment the timeout to 2 minutes and same issue, no cpu usage so far.
the command just get stucked, we tried the container outisde the cluster and works nice.
Seems not a proxy issue we tried on the terminal of the pod curl to site under test and working..

[clemensutschig]

@segator - what happens if you run the build again - same error?

Can you try to modify the jenkinsfile with npx verify as written on stackoverflow?

Lastly this works in dev or?

[clemensutschig]

cypress-io/cypress#7158 has a couple of ideas (including chmod on /home/jenkins) - maybe here lies our problem..

[BraisVQ]

I have tested cypress component in July in our DEV environment and it worked, I have just re-run it and it fails with the same error as in QA

[metmajer]

@braisvq1996 @segator we can't properly analyse without logs. Will you provide them?

[glyk0n]

• Does this happen on qa only? Maybe the site under test is down ?
  No, it also happens in dev. We think it is at the cluster level

• https://swingli.com/how-to-fix-cypress-verification-timed-out-exception/#
  This was tested with no successful result

• Attached log you request
  2021-11-22T09_44_45_072Z-debug.log

[metmajer]

Cheers @glyk0n! @braisvq1996 can you try the suggestion?

[clemensutschig]

Hmm @cschweikert can you help here maybe with turning on more logs? @ManuelFeller @m-apsolon

[BraisVQ]

I have run 4.x cypress quickstarter in PROD and it works, there must be something in the DEV/QA cluster that is making this fail

[clemensutschig]

WTF - this is really scary

[cschweikert]

Hi everybody, here are some more ideas:

• Are all envs pulling and using the exact same version of jenkins-agent-nodejs12:4.x? There was a fix in 4.x lately (nodejs 12 jenkins agent build failing on ODS 4.x #642).
• Could it be a memory issue? What if some Chrome process or so crashes while it requires more memory. Try adding this to the Jenkinsfile:

odsComponentPipeline(
  ...
  resourceRequestMemory: '2Gi',
  resourceLimitMemory: '4Gi',
  ...
)

• I was wondering about the --no-sandox wtihin the log (see inside the actual error message). This is at least different from other screenshots like in the blog mentioned above (https://swingli.com/how-to-fix-cypress-verification-timed-out-exception/#).

The shown behavior with the timeout reminds me on issues I've seen in another test framework. There the Chrome process died underneath due to a lack of memory.

@clemensutschig Unfortunately the verify command doesn't provide a verbose-like parameter or has better debug support in this case.

I'll try reproducing this issue on our cluster now.

[BraisVQ]

I have build the same image for both clusters (QA and PROD) and in PROD works but not in QA.
Both are using default resources for the agent (1 core, 2 GiB memory)

[cschweikert]

package-lock.json

The quickstarter doesn't come along with a package-lock.json. Please make sure to make builds by also having the package-lock.json under version control. Otherwise two builds are basaically not comparable.

postinstall

I can't see the npm install part inside the logs from above. Did the postinstallation of cypress work well? Should look a bit like this:

> cypress@3.8.3 postinstall /tmp/workspace/bixeap-cd/bixeap-cd-cypress-cs-master/node_modules/cypress
> node index.js --exec install

Installing Cypress (version: 3.8.3)

[13:53:07]  Downloading Cypress     [started]
[13:53:10]  Downloading Cypress     [completed]
[13:53:10]  Unzipping Cypress       [started]
[13:53:24]  Unzipping Cypress       [completed]
[13:53:24]  Finishing Installation  [started]
[13:53:24]  Finishing Installation  [completed]

You can now open Cypress by running: node_modules/.bin/cypress open
...

what else

If the agent images are the same (bitwise identical, check the hashes), if the default memory settings on both clusters are identical, if the code (at least Jenkinsfile, the package.json and package-lock.json) is the same, there is not much more you can do for getting everything identical. The only thing that still comes in my mind might be differences in Jenkins (version, configuration, plugins, executable/access rights, ...) or some crazy network rules/issues.

The following is only a wild guess, but maybe cypress verify already tries to reach the target url. In this case it would be example.com. Maybe this is then blocked.

[cschweikert]

Hi, there had been a bigger update on the e2e-cypress quickstarter (#770) as well as on the build agent (#772).

@jchaves @braisvq1996 Would be cool to know, if the new quickstarter resolves this issue.

[BraisVQ]

@cschweikert sorry, the root of the problem for this issue was Aqua and it has already been solved, I will close this issue.

Also, we have not yet installed/tested the latest changes in GitHub in our ODS installation