cypress run hangs when executed by TeamCity in 3.5.0

[hinfaits]

Current behavior:

• Since upgrading to 3.5.0, when our TeamCity Server launches runs on remote agents the execution run always hangs when using Electron.
• We have one build agent local to the TeamCity Server (agent and server on same VM) and everything works in 3.5.0.
• Our full setup continues to work fine when using Cypress 3.3.2
• Our full setup also works fine using Chrome in 3.5.0
• Our tests run fine launched from the agent manually

This is a full log where the build hangs while running an empty test (our config includes a global hook to load the application and modify localStorage in the support file, but this test is empty). Some stuff has been redacted slightly. - https://pastebin.com/ULeiF83U

When running a normal test, it seems like things go wrong after these lines are spat out. After this there's more messages from the browser and I can see my application idling, making some requests and logging some messages but it doesn't look like any further test steps are executed. It looks like in both cases there's a drop in connectivity with the browser but they may be different issues.

[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.274Z cypress:https-proxy Not making direct connection { url: 'qa.internaldomain.com:443' }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.274Z cypress:https-proxy Making intercepted connection to 58393
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.274Z cypress:https-proxy getting proxy URL { port: 58393, serverPort: 58391, sniPort: 58393, url: 'https://localhost:58393' }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.276Z cypress:network:connect successfully connected { opts: { port: 58393, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.276Z cypress:https-proxy received upstreamSocket callback for request { port: 58393, hostname: 'localhost', err: undefined }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.277Z cypress:network:connect successfully connected { opts: { port: 58393, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.277Z cypress:https-proxy received upstreamSocket callback for request { port: 58393, hostname: 'localhost', err: undefined }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.279Z cypress:network:connect successfully connected { opts: { port: 58393, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.279Z cypress:https-proxy received upstreamSocket callback for request { port: 58393, hostname: 'localhost', err: undefined }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.683Z cypress:server:server Getting remote state: { auth: null, props: { port: '443', tld: 'com', domain: 'internaldomain' }, origin: 'https://qa.internaldomain.com', strategy: 'http', visiting: false, domainName: 'internaldomain.com', fileServer: null }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.683Z cypress:server:proxy handling proxied request { url: '/url/url/36486', proxiedUrl: 'https://qa.internaldomain.com/url/url/36486', headers: { host: 'qa.internaldomain.com', connection: 'keep-alive', 'content-length': '156', accept: 'application/json, text/javascript, */*; q=0.01', origin: 'https://qa.internaldomain.com', 'x-csrf-token': '83180abc2aa0db8a656c9e5849c5a10f', 'x-requested-with': 'XMLHttpRequest', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.5.0 Chrome/73.0.3683.121 Electron/5.0.10 Safari/537.36', 'content-type': 'application/json', referer: 'https://qa.internaldomain.com/pm', 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'en-US', cookie: 'JSESSIONID=16da25bf-b600-4171-9744-ff7d5843ac9d; ... }, remoteState: { auth: null, props: { port: '443', tld: 'com', domain: 'internaldomain' }, origin: 'https://qa.internaldomain.com', strategy: 'http', visiting: false, domainName: 'internaldomain.com', fileServer: null } }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.683Z cypress:server:stream_buffer appending chunk to buffer { bytesWritten: 0, chunkLength: 156 }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.684Z cypress:network:agent addRequest called { isHttps: true, href: 'https://qa.internaldomain.com/url/url/36486' }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.684Z cypress:network:agent got family { family: 4, href: 'https://qa.internaldomain.com/url/url/36486' }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.684Z cypress:server:stream_buffer stream buffer writeable final called
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.818Z cypress:server:request received status code on request { requestId: 'request237', statusCode: 200 }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.818Z cypress:server:request successful response received { requestId: 'request237' }
[13:29:05] :	 [Step 2/2] 2019-10-31T20:29:05.818Z cypress:server:proxy received response for { url: 'https://qa.internaldomain.com/url/url/36486', headers: { 'content-type': 'application/json;charset=UTF-8', 'transfer-encoding': 'chunked', connection: 'keep-alive', date: 'Thu, 31 Oct 2019 20:29:05 GMT', server: 'Apache-Coyote/1.1', 'cache-control': 'no-cache, no-store', pragma: 'no-cache', expires: '-1', 'x-client-ip': '207.194.85.138', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'SAMEORIGIN', message: 'patientProcedure.success.create', 'x-cache': 'Miss from cloudfront', via: '1.1 0bb842bd5868b86440983b936bb0adef.cloudfront.net (CloudFront)', 'x-amz-cf-pop': 'SEA19-C1', 'x-amz-cf-id': 'Euop7rfWiV6A1A-nfDas-qQ93kj3WzGM1Wn94Bdhrp47i4Px9r8WDQ==' }, statusCode: 200, isGzipped: undefined, wantsInjection: false, wantsSecurityRemoved: false }
[13:29:06] :	 [Step 2/2] 2019-10-31T20:29:06.025Z cypress:network:connect retries exhausted, bubbling up error { iteration: 4, err: { Error: connect ECONNREFUSED 127.0.0.1:8844 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8844 } }
[13:29:06] :	 [Step 2/2] 2019-10-31T20:29:06.025Z cypress:https-proxy received upstreamSocket callback for request { port: '8844', hostname: 'localhost', err: { Error: connect ECONNREFUSED 127.0.0.1:8844 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8844 } }
[13:29:06] :	 [Step 2/2] 2019-10-31T20:29:06.025Z cypress:https-proxy received error on client browserSocket { err: { Error: connect ECONNREFUSED 127.0.0.1:8844 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8844 }, url: 'localhost:8844' }
[13:29:06] :	 [Step 2/2] 2019-10-31T20:29:06.079Z cypress:network:connect received error on connect, retrying { iteration: 0, delay: 0, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:07] :	 [Step 2/2] 2019-10-31T20:29:07.083Z cypress:network:connect received error on connect, retrying { iteration: 1, delay: 100, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:08] :	 [Step 2/2] 2019-10-31T20:29:08.187Z cypress:network:connect received error on connect, retrying { iteration: 2, delay: 200, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:09] :	 [Step 2/2] 2019-10-31T20:29:09.391Z cypress:network:connect received error on connect, retrying { iteration: 3, delay: 200, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.595Z cypress:network:connect retries exhausted, bubbling up error { iteration: 4, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.595Z cypress:https-proxy received upstreamSocket callback for request { port: '8443', hostname: 'localhost', err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.595Z cypress:https-proxy received error on client browserSocket { err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 }, url: 'localhost:8443' }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.898Z cypress:server:server Got CONNECT request from localhost:8443
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.898Z cypress:https-proxy Writing browserSocket connection headers { url: 'localhost:8443', headLength: 0, headers: { host: 'localhost:8443', 'proxy-connection': 'keep-alive', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.5.0 Chrome/73.0.3683.121 Electron/5.0.10 Safari/537.36' } }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:https-proxy Got first head bytes { url: 'localhost:8443', head: '\u0016\u0003\u0001\u0000�\u0001\u0000\u0000�\u0003\u0003��З����N0|�y���\u0003�L����щ�\u00022\u0002�(A\u0000\u0000\u001c���+�/�,�0̨̩�\u0013�\u0014\u0000�\u0000�' }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:server:cors Parsed URL { port: '8443', tld: 'localhost', domain: '' }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:server:server HTTPS request does not match URL: https://localhost:8443 with props: { port: '443', tld: 'com', domain: 'internaldomain' }
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:server:server HTTPS request https://localhost:8443 matches blacklist? undefined
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:https-proxy Making connection to localhost:8443
[13:29:10] :	 [Step 2/2] 2019-10-31T20:29:10.900Z cypress:https-proxy getting proxy URL { port: 8443, serverPort: 58391, sniPort: 58393, url: 'https://localhost:8443' }
[13:29:11] :	 [Step 2/2] 2019-10-31T20:29:11.904Z cypress:network:connect received error on connect, retrying { iteration: 0, delay: 0, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:12] :	 [Step 2/2] 2019-10-31T20:29:12.910Z cypress:network:connect received error on connect, retrying { iteration: 1, delay: 100, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:14] :	 [Step 2/2] 2019-10-31T20:29:14.013Z cypress:network:connect received error on connect, retrying { iteration: 2, delay: 200, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:15] :	 [Step 2/2] 2019-10-31T20:29:15.218Z cypress:network:connect received error on connect, retrying { iteration: 3, delay: 200, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:16] :	 [Step 2/2] 2019-10-31T20:29:16.421Z cypress:network:connect retries exhausted, bubbling up error { iteration: 4, err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:16] :	 [Step 2/2] 2019-10-31T20:29:16.421Z cypress:https-proxy received upstreamSocket callback for request { port: '8443', hostname: 'localhost', err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 } }
[13:29:16] :	 [Step 2/2] 2019-10-31T20:29:16.421Z cypress:https-proxy received error on client browserSocket { err: { Error: connect ECONNREFUSED 127.0.0.1:8443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1061:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 8443 }, url: 'localhost:8443' }
[13:30:35] :	 [Step 2/2] 2019-10-31T20:30:35.129Z cypress:server:browsers:electron console message: { column: 2114031, level: 'warning', line: 1, source: 'console-api', text: 'RabbitMQ connection lost! Trying to reconnect ( 1 ) in 1000 ms. url:wss://qa-rabbit.internaldomain.com:15671/ws, code:1000, reason:STOMP died', url: 'https://qa.internaldomain.com/assets/AscendCode-c50712637c55188576f0dae8b777313f.js' }
[13:30:54] :	 [Step 2/2] 2019-10-31T20:30:54.991Z cypress:server:server Got CONNECT request from qa.internaldomain.com:443
[13:30:54] :	 [Step 2/2] 2019-10-31T20:30:54.991Z cypress:https-proxy Writing browserSocket connection headers { url: 'qa.internaldomain.com:443', headLength: 0, headers: { host: 'qa.internaldomain.com:443', 'proxy-connection': 'keep-alive', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/3.5.0 Chrome/73.0.3683.121 Electron/5.0.10 Safari/537.36' } }

Desired behavior:

Builds do not hang.

Steps to reproduce: (app code and test code)

This is happening consistently but I cannot seem to reproduce this outside of specifically launching our tests on the TeamCity agent from TeamCity Server. When using PsExec to run the tests as the system user (our TeamCity agent uses the system user) everything executes as normal.

I'm not sure how to deliver a full reproducible example as it looks like something specific to our environment. But I will try my best to help debug.

Versions

Cypress 3.5.0
Electron
Windows 8 + Windows 10

[GregLahaye]

Having same issue here. Cypress hangs after running a single spec file when run from TeamCity but works when run manually on the same build agent. Using Cypress 3.6.0.
My log files look almost identical, Cypress gets stuck on line 'cypress:server:browsers killing browser process' but works fine with Chrome. May try testing with a downgraded version of Cypress.

[axehigh]

Having same issue. Was using 3.5.0. Downgrading to 3.3.2 works fine.
Thanks

[jennifer-shehane]

@hinfaits I see in the logs that you're passing

--env configFile=qa.json --reporter teamcity --spec .\cypress\integration\SPEC-999.js

Then

cli starts with arguments ["C:\\Program Files\\nodejs\\node.exe","C:\\BuildAgent\\work\\7f314bce6463a58b\\node_modules\\cypress\\bin\\cypress","run","--env","configFile=qa.json","--reporter","teamcity","--spec",".\\cypress\\integration\\SPEC-999.js.js"]

So this eventually looks like it's being read as:

parsed cli options { 
spec: '.\\cypress\\integration\\SPEC-999.js.js',
reporter: 'teamcity',
env: 'configFile=qa.json' 
}

So, I'm not sure why the SPEC-999.js.js has a double .js

Anyway, I don't think this matters cause it seems to find the spec file to run later:

cypress:server:specs found 1 spec files: [ { name: 'SPEC-999.js', relative: 'cypress\\integration\\SPEC-999.js.js', absolute: 'C:/BuildAgent/work/7f314bce6463a58b/cypress/integration/SPEC-999.js' } ]

So I'm just making it to the end of your logfile and I actually do not see Cypress hanging. I am seeing Cypress issue an exit code of 1, since 1 test file failed, so I do not see how this is an issue with Cypress.

[luis-m-gonzalez]

For the record - I'm having the same issue - except with Cypress 3.6.0

[hinfaits]

@jennifer-shehane, in the logs the double .js is likely because I was using find/replace to remove some internal data (just to be on the safe side).

In the logs where you've identified exit with code 1, that happens after I've sent selected Stop Build from TeamCity Server and now it's killed what's running on the agent. Before that, it's hung for 6 minutes trying to close the browser.

[jennifer-shehane]

Interesting. Do you happen to have any on beforeunload listeners in your code as detailed in this issue? #2118

Does passing the --browser flag to run in chrome make any difference?

[hinfaits]

We don't have beforeunload listeners generally. We do have some in an area we do not touch with our Cypress based automation.

Running with chrome solves all our hanging. Unfortunately it introduces some other problems so it isn't a viable workaround for us.

Please let me know if there's any more information you'd like to debugging I can do.

[DamienCassou]

Interesting. Do you happen to have any on beforeunload listeners in your code as detailed in this issue? #2118

I have a similar issue and a very simple test suite. No beforeunload is involved but I have a listener on window:before:load.

Does passing the --browser flag to run in chrome make any difference?

it fixes my problem!

[fidgety]

We have a window:before:load which causes the process to hang, it only happens when being executed though the TeamCity agent; running against the same user locally on the box works fine. We're a paying customer so we can run parallel and this is stopping us being able to use it at present. Switching to Chrome makes the tests run significantly slower.

[ronnyek]

This is happening in vsts/azure devops as well.

[jennifer-shehane]

Unfortunately we'll have to close this issue as there is not enough information to reproduce the problem. This does not mean that your issue is not happening - it just means that we do not have a path to move forward.

Please comment in this issue with a reproducible example so we can look into the issue further.

[jennifer-shehane]

Since this issue hasn't had activity in a while, we'll close the issue until we can confirm this is still happening. Please comment if there is new information to provide concerning the original issue and we'd be happy to reopen.