Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript #1471

Closed
petermetz opened this issue Oct 21, 2021 · 0 comments · Fixed by #2628
Closed

test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript #1471

petermetz opened this issue Oct 21, 2021 · 0 comments · Fixed by #2628
Assignees
@petermetz
Labels
bug Something isn't working dependencies Pull requests that update a dependency file Developer_Experience Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) good-first-issue Good for newcomers good-first-issue-400-expert Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. help wanted Extra attention is needed Tests Anything related to tests be that automatic or manual, integration or unit, etc.
Milestone
vT.B.D

Comments

@petermetz
Copy link
Contributor

petermetz commented Oct 21, 2021 •
edited
Loading

Describe the bug

We've got a flaky one at:
packages/cactus-plugin-ledger-connector-fabric/src/test/typescript/integration/fabric-v2-2-x/deploy-cc-from-javascript-source.test.ts

2021-10-19T08:07:31.364Z - error: [DiscoveryHandler]: compareProposalResponseResults[undefined] - read/writes result sets do not match index=1
2021-10-19T08:07:31.365Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
    peer=undefined, status=grpc, message=Peer endorsements do not match
[2021-10-19T08:07:31.366Z] ERROR (PluginLedgerConnectorFabric): transact() crashed:  Error: No valid responses from any peers. Errors:
    peer=undefined, status=grpc, message=Peer endorsements do not match
    at newEndorsementError (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/transaction.ts:69:9)
    at getResponsePayload (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/transaction.ts:22:17)
    at Transaction.submit (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/transaction.ts:294:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async PluginLedgerConnectorFabric.transact (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/plugin-ledger-connector-fabric.ts:172:75)
    at async RunTransactionEndpointV1.handleRequest (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/run-transaction/run-transaction-endpoint-v1.ts:3:1283) {
  responses: [],
  errors: [
    Error: Peer endorsements do not match
        at DiscoveryHandler._endorse (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/node_modules/fabric-common/lib/DiscoveryHandler.js:295:14)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at async Endorsement.send (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/node_modules/fabric-common/lib/Proposal.js:410:15)
        at async Transaction.submit (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/node_modules/fabric-network/src/transaction.ts:292:28)
        at async PluginLedgerConnectorFabric.transact (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/plugin-ledger-connector-fabric.ts:172:75)
        at async RunTransactionEndpointV1.handleRequest (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/run-transaction/run-transaction-endpoint-v1.ts:3:1283) {
      endorsements: [Array]
    }
  ]
}
[2021-10-19T08:07:31.416Z] ERROR (run-transaction-endpoint-v1): RunTransactionEndpointV1#handleRequest() failed to serve request Error: PluginLedgerConnectorFabric#transact() Unable to run transaction: No valid responses from any peers. Errors:
    peer=undefined, status=grpc, message=Peer endorsements do not match
    at PluginLedgerConnectorFabric.transact (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/plugin-ledger-connector-fabric.ts:173:81)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async RunTransactionEndpointV1.handleRequest (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/run-transaction/run-transaction-endpoint-v1.ts:3:1283)
TypeError [ERR_INVALID_CHAR]: Invalid character in statusMessage
    at new NodeError (node:internal/errors:371:5)
    at ServerResponse.writeHead (node:_http_server:323:11)
    at ServerResponse._implicitHeader (node:_http_server:266:8)
    at write_ (node:_http_outgoing:763:9)
    at ServerResponse.end (node:_http_outgoing:852:5)
    at ServerResponse.send (/home/runner/work/cactus/cactus/node_modules/express/lib/response.js:221:10)
    at ServerResponse.json (/home/runner/work/cactus/cactus/node_modules/express/lib/response.js:267:15)
    at RunTransactionEndpointV1.handleRequest (/home/runner/work/cactus/cactus/packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/run-transaction/run-transaction-endpoint-v1.ts:3:1626)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'ERR_INVALID_CHAR'
}
    [2021-10-19T08:07:21.099Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] 'time="2021-10-19T08:07:21.098239687Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ccef22b5fafcb85bc871020341a66c9710f87345eed3c8f29d941dbf86993595 pid=8108\r\n'
    [2021-10-19T08:07:21.100Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] "2021-10-19 08:07:21,100 DEBG 'dockerd' stderr output:\r\n"
    [2021-10-19T08:07:21.101Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] 'time="2021-10-19T08:07:21.098239687Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ccef22b5fafcb85bc871020341a66c9710f87345eed3c8f29d941dbf86993595 pid=8108\r\n'
    [2021-10-19T08:07:21.102Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] '\r\n'
    [2021-10-19T08:07:21.140Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] 'time="2021-10-19T08:07:21.131972625Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/44c89c9cada869351d3b0e667e13e3b5131d1a2d7142fa76c7dc999174ed942e pid=8127\r\n'
    [2021-10-19T08:07:21.140Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] "2021-10-19 08:07:21,138 DEBG 'dockerd' stderr output:\r\n"
    [2021-10-19T08:07:21.140Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] 'time="2021-10-19T08:07:21.131972625Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/44c89c9cada869351d3b0e667e13e3b5131d1a2d7142fa76c7dc999174ed942e pid=8127\r\n'
    [2021-10-19T08:07:21.140Z] DEBUG (FabricTestLedgerV1): [ghcr.io/hyperledger/cactus-fabric2-all-in-one:2021-09-02--fix-876-supervisord-retries] '\r\n'
    [2021-10-19T08:07:31.047Z] DEBUG (run-transaction-endpoint-v1): POST /api/v1/plugins/@hyperledger/cactus-plugin-ledger-connector-fabric/run-transaction
    [2021-10-19T08:07:31.051Z] DEBUG (PluginLedgerConnectorFabric): discovery={ enabled: true, asLocalhost: true }
    [2021-10-19T08:07:31.051Z] DEBUG (PluginLedgerConnectorFabric): eventHandlerOptions={
      commitTimeout: 300,
      endorseTimeout: 300,
      strategy: [Function: NETWORK_SCOPE_ALLFORTX] {
        [length]: 2,
        [name]: 'NETWORK_SCOPE_ALLFORTX'
      }
    }
    [2021-10-19T08:07:31.244Z] DEBUG (PluginLedgerConnectorFabric): transact() gateway connection established OK
    # test count(4) != plan(null)
    # failed 1 of 4 tests

To Reproduce

Keep running the test and eventually you'll see it.

Expected behavior

Tests are stable, not flaky.

Logs/Stack traces

Logs with additional details:
2023-08-17-cacti-ci-fabric-flaky-test-crash-extra-hfc-logging.log

The CI logs from the GHA runner as of this morning:
logs_10074.zip

Cloud provider or hardware configuration:

GHA runner

Operating system name, version, build:

Ubuntu 20.04

Hyperledger Cactus release version or commit (git rev-parse --short HEAD):

1.0.0-rc.1

Hyperledger Cactus Plugins/Connectors Used

Fabric

Additional context

I'll send a separate PR that skips this test to avoid flakiness in the meantime.
@petermetz petermetz added bug Something isn't working good-first-issue Good for newcomers help wanted Extra attention is needed dependencies Pull requests that update a dependency file Developer_Experience Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. good-first-issue-400-expert Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) Tests Anything related to tests be that automatic or manual, integration or unit, etc. labels Oct 21, 2021
petermetz added a commit to petermetz/cacti that referenced this issue Oct 21, 2021
@petermetz
test: skip flaky fabric 2.2.x deploy cc from JS test
e4b246f 
This is related to hyperledger-cacti#1471 (does not fix it though). In order to enable
faster CI/PR turnaround times we are aggressively eliminating (skipping)
flaky tests and creating issues in the backlog for fixing them later on.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz mentioned this issue Oct 21, 2021
Merged
petermetz added a commit to petermetz/cacti that referenced this issue Oct 22, 2021
@petermetz
test: skip flaky fabric 2.2.x deploy cc from JS test
4e6f1c3 
This is related to hyperledger-cacti#1471 (does not fix it though). In order to enable
faster CI/PR turnaround times we are aggressively eliminating (skipping)
flaky tests and creating issues in the backlog for fixing them later on.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Oct 26, 2021
@petermetz @Leeyoungone
test: skip flaky fabric 2.2.x deploy cc from JS test
2ae4acf 
This is related to hyperledger-cacti#1471 (does not fix it though). In order to enable
faster CI/PR turnaround times we are aggressively eliminating (skipping)
flaky tests and creating issues in the backlog for fixing them later on.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
RafaelAPB pushed a commit to RafaelAPB/blockchain-integration-framework that referenced this issue Mar 9, 2022
@petermetz @RafaelAPB
test: skip flaky fabric 2.2.x deploy cc from JS test
87f64bc 
This is related to hyperledger-cacti#1471 (does not fix it though). In order to enable
faster CI/PR turnaround times we are aggressively eliminating (skipping)
flaky tests and creating issues in the backlog for fixing them later on.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
outSH added a commit to outSH/cactus that referenced this issue Jul 4, 2022
@outSH
test(cactus): remove flaky tests from main CI execution
c34e857 
- Remove flaky tests from main CI execution.
- Add separate TAP and jest configuration for running only flaky tests.
- Add test:*:flaky scripts in the main package.json
- Add new step to github CI script to run flaky tests as optional
  (i.e. don't fail if flaky test fails)

Closes: hyperledger-cacti#2112

Related: hyperledger-cacti#2104
Related: hyperledger-cacti#1626
Related: hyperledger-cacti#2019
Related: hyperledger-cacti#1625
Related: hyperledger-cacti#1543
Related: hyperledger-cacti#1598
Related: hyperledger-cacti#1528
Related: hyperledger-cacti#1521
Related: hyperledger-cacti#1495
Related: hyperledger-cacti#1485
Related: hyperledger-cacti#1473
Related: hyperledger-cacti#1471
Related: hyperledger-cacti#1469
Related: hyperledger-cacti#1150

Signed-off-by: Michal Bajer <michal.bajer@fujitsu.com>
@outSH outSH mentioned this issue Jul 4, 2022
Closed
@petermetz petermetz self-assigned this Aug 18, 2023
@petermetz petermetz added this to the vT.B.D milestone Aug 18, 2023
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
810e2f5 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz mentioned this issue Aug 18, 2023
Closed
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
443b660 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
b5d21d0 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
5378ce9 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
e8e366f 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
0afb71d 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 18, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
d4bcc90 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 19, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
09ce959 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 19, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
2f40a8b 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 20, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
eb6b5fe 
WORK IN PROGRESS - just thinned down the CI for now

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 20, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
22aecbc 
Primary changes:
---------------

1. Added robust debug logging to the express middleware of the API server
so that problems like this are easier to debug in the future (e.g.
trouble with the Open API spec documents themselves)
2. Refactored the ci.yaml document so that there are separate jobs for
each tape based Fabric connector integration test. This has reduced
(or potentially eliminated) the presence of the flake that we were
encountering during the contract deployment test cases. It also has the
upside of faster CI because of more parallelism in the CI jobs.
3. Updated the assertions in a couple of test cases where the newer Fabric
versions were causing the assertions to fail because the returned JSON
response in the example asset transfer chaincode is now referring to the
asset owner via a property named "Owner" not "owner" (casing difference)

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
@petermetz petermetz mentioned this issue Aug 20, 2023
Merged
petermetz added a commit to petermetz/cacti that referenced this issue Aug 20, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
26dc510 
Primary changes:
---------------

1. Added robust debug logging to the express middleware of the API server
so that problems like this are easier to debug in the future (e.g.
trouble with the Open API spec documents themselves)
2. Refactored the ci.yaml document so that there are separate jobs for
each tape based Fabric connector integration test. This has reduced
(or potentially eliminated) the presence of the flake that we were
encountering during the contract deployment test cases. It also has the
upside of faster CI because of more parallelism in the CI jobs.
3. Updated the assertions in a couple of test cases where the newer Fabric
versions were causing the assertions to fail because the returned JSON
response in the example asset transfer chaincode is now referring to the
asset owner via a property named "Owner" not "owner" (casing difference)

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Aug 21, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
7362183 
Primary changes:
---------------

1. Added robust debug logging to the express middleware of the API server
so that problems like this are easier to debug in the future (e.g.
trouble with the Open API spec documents themselves)
2. Refactored the ci.yaml document so that there are separate jobs for
each tape based Fabric connector integration test. This has reduced
(or potentially eliminated) the presence of the flake that we were
encountering during the contract deployment test cases. It also has the
upside of faster CI because of more parallelism in the CI jobs.
3. Updated the assertions in a couple of test cases where the newer Fabric
versions were causing the assertions to fail because the returned JSON
response in the example asset transfer chaincode is now referring to the
asset owner via a property named "Owner" not "owner" (casing difference)

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit that referenced this issue Aug 21, 2023
@petermetz
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
08afe1a 
Primary changes:
---------------

1. Added robust debug logging to the express middleware of the API server
so that problems like this are easier to debug in the future (e.g.
trouble with the Open API spec documents themselves)
2. Refactored the ci.yaml document so that there are separate jobs for
each tape based Fabric connector integration test. This has reduced
(or potentially eliminated) the presence of the flake that we were
encountering during the contract deployment test cases. It also has the
upside of faster CI because of more parallelism in the CI jobs.
3. Updated the assertions in a couple of test cases where the newer Fabric
versions were causing the assertions to fail because the returned JSON
response in the example asset transfer chaincode is now referring to the
asset owner via a property named "Owner" not "owner" (casing difference)

Fixes #1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES pushed a commit to sandeepnRES/cacti that referenced this issue Dec 21, 2023
@petermetz @sandeepnRES
test(connector-fabric): fix flaky test 2.2.x deploy-cc-from-javascript
e20c6f4 
Primary changes:
---------------

1. Added robust debug logging to the express middleware of the API server
so that problems like this are easier to debug in the future (e.g.
trouble with the Open API spec documents themselves)
2. Refactored the ci.yaml document so that there are separate jobs for
each tape based Fabric connector integration test. This has reduced
(or potentially eliminated) the presence of the flake that we were
encountering during the contract deployment test cases. It also has the
upside of faster CI because of more parallelism in the CI jobs.
3. Updated the assertions in a couple of test cases where the newer Fabric
versions were causing the assertions to fail because the returned JSON
response in the example asset transfer chaincode is now referring to the
asset owner via a property named "Owner" not "owner" (casing difference)

Fixes hyperledger-cacti#1471

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@petermetz petermetz

Labels
bug Something isn't working dependencies Pull requests that update a dependency file Developer_Experience Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) good-first-issue Good for newcomers good-first-issue-400-expert Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. help wanted Extra attention is needed Tests Anything related to tests be that automatic or manual, integration or unit, etc.
Projects
None yet
Milestone
vT.B.D
1 participant
@petermetz