test: fix flaky deploy-cordapp-jars-to-nodes-v4.8-express

[petermetz]

Describe the bug

The test at packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.8-express.test.ts is flaky meaning that most of the time it succeeds but occasionally it fails which is bad.

To Reproduce

Run it multiple times until it fails.

Expected behavior

The test case should be stable e.g. always passing or always failing.

Logs/Stack traces

Attached within the zip as: build-node-16/8_Run .toolsci.sh.txt
logs_11388.zip
logs_11522.zip

2021-11-29T16:58:42.7153857Z     [2021-11-29T16:49:36.167Z] DEBUG (CordaConnectorContainer): [ghcr.io/hyperledger/cactus-connector-corda-server:2021-11-11] '\x1B[2m2021-11-29 16:49:36.163\x1B[0;39m \x1B[32mDEBUG\x1B[0;39m \x1B[35m9\x1B[0;39m \x1B[2m---\x1B[0;39m \x1B[2m[nio-8080-exec-3]\x1B[0;39m \x1B[36mApiPluginLedgerConnectorCordaServiceImpl\x1B[0;39m \x1B[2m:\x1B[0;39m Invoking flow with timeout of 600000 ms ...\r\n'
2021-11-29T16:58:42.7159001Z     [2021-11-29T16:49:36.168Z] DEBUG (CordaConnectorContainer): [ghcr.io/hyperledger/cactus-connector-corda-server:2021-11-11] '\x1B[2m2021-11-29 16:49:36.167\x1B[0;39m \x1B[32mDEBUG\x1B[0;39m \x1B[35m9\x1B[0;39m \x1B[2m---\x1B[0;39m \x1B[2m[nio-8080-exec-3]\x1B[0;39m \x1B[36mApiPluginLedgerConnectorCordaServiceImpl\x1B[0;39m \x1B[2m:\x1B[0;39m Starting to wait for flow completion now...\r\n'
2021-11-29T16:58:42.7161144Z     # test count(22) != plan(null)
2021-11-29T16:58:42.7161629Z     # failed 1 of 22 tests
2021-11-29T16:58:42.7163919Z not ok 13 - ./packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.8-express.test.ts # time=900132.042ms
2021-11-29T16:58:42.7165677Z   ---
2021-11-29T16:58:42.7166035Z   env:
2021-11-29T16:58:42.7166516Z     TS_NODE_COMPILER_OPTIONS: "{}"
2021-11-29T16:58:42.7168527Z   file: ./packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.8-express.test.ts
2021-11-29T16:58:42.7170099Z   timeout: 900000
2021-11-29T16:58:42.7170724Z   command: /opt/hostedtoolcache/node/16.9.1/x64/bin/node
2021-11-29T16:58:42.7171260Z   args:
2021-11-29T16:58:42.7171761Z     - -r
2021-11-29T16:58:42.7172618Z     - /home/runner/work/cactus/cactus/node_modules/ts-node/register/index.js
2021-11-29T16:58:42.7173665Z     - --max-old-space-size=4096
2021-11-29T16:58:42.7175842Z     - ./packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.8-express.test.ts
2021-11-29T16:58:42.7177389Z   stdio:
2021-11-29T16:58:42.7177911Z     - 0
2021-11-29T16:58:42.7178619Z     - pipe
2021-11-29T16:58:42.7179106Z     - 2
2021-11-29T16:58:42.7179594Z   cwd: /home/runner/work/cactus/cactus
2021-11-29T16:58:42.7180075Z   failures:
2021-11-29T16:58:42.7180675Z     - tapError: no plan
2021-11-29T16:58:42.7181132Z   exitCode: null
2021-11-29T16:58:42.7181823Z   signal: SIGTERM
2021-11-29T16:58:42.7182210Z   ...
2021-11-29T16:58:42.7182434Z 
2021-11-29T16:58:42.7184517Z Bail out! ./packages/cactus-plugin-ledger-connector-corda/src/test/typescript/integration/deploy-cordapp-jars-to-nodes-v4.8-express.test.ts
2021-11-29T16:59:26.6726477Z [2021-11-29T16:59:26.661Z] ERROR (InvokeContractJsonObjectEndpoint): Crash while serving post - /api/v1/plugins/@hyperledger/cactus-plugin-ledger-connector-quorum/invoke-contract-json-object Error: PluginLedgerConnectorQuorum#transactGethKeychain() Failed to invoke web3.eth.personal.sendTransaction(). InnerException: Error: Returned error: nonce too low
2021-11-29T16:59:26.6730539Z     at Object.ErrorResponse (/home/runner/work/cactus/cactus/node_modules/web3-core-requestmanager/node_modules/web3-core-helpers/lib/errors.js:28:19)
2021-11-29T16:59:26.6732321Z     at /home/runner/work/cactus/cactus/node_modules/web3-core-requestmanager/lib/index.js:302:36
2021-11-29T16:59:26.6734298Z     at XMLHttpRequest.request.onreadystatechange (/home/runner/work/cactus/cactus/node_modules/web3-providers-http/lib/index.js:98:13)
2021-11-29T16:59:26.6737961Z     at XMLHttpRequestEventTarget.dispatchEvent (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request-event-target.ts:44:13)
2021-11-29T16:59:26.6740297Z     at XMLHttpRequest._setReadyState (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:219:8)
2021-11-29T16:59:26.6742132Z     at XMLHttpRequest._onHttpResponseEnd (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:345:8)
2021-11-29T16:59:26.6743904Z     at IncomingMessage.<anonymous> (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:311:39)
2021-11-29T16:59:26.6745022Z     at IncomingMessage.emit (node:events:406:35)
2021-11-29T16:59:26.6745818Z     at IncomingMessage.emit (node:domain:475:12)
2021-11-29T16:59:26.6746714Z     at endReadableNT (node:internal/streams/readable:1343:12)
2021-11-29T16:59:26.6747734Z     at processTicksAndRejections (node:internal/process/task_queues:83:21)
2021-11-29T16:59:26.6750607Z     at PluginLedgerConnectorQuorum.transactGethKeychain (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-quorum/src/main/typescript/plugin-ledger-connector-quorum.ts:6:7572)
2021-11-29T16:59:26.6754654Z     at async PluginLedgerConnectorQuorum.invokeContract (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-quorum/src/main/typescript/plugin-ledger-connector-quorum.ts:6:4539)
2021-11-29T16:59:26.6758918Z     at async InvokeContractJsonObjectEndpoint.handleRequest (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-quorum/src/main/typescript/web-services/invoke-contract-endpoint-json-object.ts:3:727)
2021-11-29T16:59:28.7661426Z [2021-11-29T16:59:28.764Z] ERROR (InvokeContractJsonObjectEndpoint): Crash while serving post - /api/v1/plugins/@hyperledger/cactus-plugin-ledger-connector-quorum/invoke-contract-json-object Error: Returned error: nonce too low
2021-11-29T16:59:28.7676026Z     at Object.ErrorResponse (/home/runner/work/cactus/cactus/node_modules/web3-core-requestmanager/node_modules/web3-core-helpers/lib/errors.js:28:19)
2021-11-29T16:59:28.7701961Z     at /home/runner/work/cactus/cactus/node_modules/web3-core-requestmanager/lib/index.js:302:36
2021-11-29T16:59:28.7704115Z     at XMLHttpRequest.request.onreadystatechange (/home/runner/work/cactus/cactus/node_modules/web3-providers-http/lib/index.js:98:13)
2021-11-29T16:59:28.7706642Z     at XMLHttpRequestEventTarget.dispatchEvent (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request-event-target.ts:44:13)
2021-11-29T16:59:28.7710176Z     at XMLHttpRequest._setReadyState (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:219:8)
2021-11-29T16:59:28.7712394Z     at XMLHttpRequest._onHttpResponseEnd (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:345:8)
2021-11-29T16:59:28.7714526Z     at IncomingMessage.<anonymous> (/home/runner/work/cactus/cactus/node_modules/xhr2-cookies/xml-http-request.ts:311:39)
2021-11-29T16:59:28.7717338Z     at IncomingMessage.emit (node:events:406:35)
2021-11-29T16:59:28.7718364Z     at IncomingMessage.emit (node:domain:475:12)
2021-11-29T16:59:28.7719136Z     at endReadableNT (node:internal/streams/readable:1343:12)
2021-11-29T16:59:28.7720148Z     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
2021-11-29T16:59:28.7720792Z   data: null
2021-11-29T16:59:28.7721120Z }

Screenshots

N/A

Cloud provider or hardware configuration:

Github Actions Runner

Operating system name, version, build:

Ubuntu 20.04

Hyperledger Cactus release version or commit (git rev-parse --short HEAD):

1.0.0-rc.2

Hyperledger Cactus Plugins/Connectors Used

Corda

Additional context

• This happened during the CI run of an unrelated PR.
• Will send a separate PR that skips this test until we figure out what the real solution would be to make it stable.

[petermetz]

Closing because this has not been spotted for a while (for a few months at least)
Please re-open if it becomes frequent enough again to warrant more attention.