Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka Producers unable to reconnect after being disconnected in Central-Ledger Integration-Tests #3067

Closed
mdebarros opened this issue Dec 12, 2022 · 1 comment
Closed

Kafka Producers unable to reconnect after being disconnected in Central-Ledger Integration-Tests #3067

mdebarros opened this issue Dec 12, 2022 · 1 comment
Assignees
@geka-evk
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it

Comments

@mdebarros
Copy link
Member

Summary:
Kafka Producers unable to reconnect after being disconnected in Central-Ledger Integration-Tests.

Example of issue: https://app.circleci.com/pipelines/github/mojaloop/central-ledger/1471/workflows/6c3b5c2d-8a6c-449a-8760-80cbbb761a2a/jobs/18871

2022-12-12T11:04:26.640Z - error: uncaughtException: Producer not connected
    FSPIOPError: Producer not connected
        at createFSPIOPError (/home/circleci/project/node_modules/@********/central-services-error-handling/src/factory.js:198:12)
        at Object.reformatFSPIOPError (/home/circleci/project/node_modules/@********/central-services-error-handling/src/factory.js:333:12)
        at Object.produceMessage (/home/circleci/project/node_modules/@********/central-services-stream/src/util/producer.js:79:32)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at async Object.produceGeneralMessage (/home/circleci/project/node_modules/@********/central-services-shared/src/util/kafka/index.js:238:3)
        at async Object.addLimitAndInitialPosition (/home/circleci/project/src/domain/participant/index.js:296:5)
        at async Object.exports.prepareLimitAndInitialPosition (/home/circleci/project/test/integration/helpers/participantLimit.js:50:5)
        at async prepareTestData (/home/circleci/project/test/integration/handlers/transfers/handlers.test.js:165:42)
        at async Test.<anonymous> (/home/circleci/project/test/integration/handlers/transfers/handlers.test.js:385:18)

Severity:
Medium

Priority:
Medium

Expected Behavior

Kafka Producers should reconnect when Producer.connect is invoked after being disconnected before.

This can be seen in the Central-Ledger Integration-Tests

Steps to Reproduce

  1. Create a PR based on mojaloop/central-ledger@77cb536
  2. Test Integration CI Job should fail with error: uncaughtException: Producer not connected

Specifications

  • Component (if known): Central-Services-Stream (CSL), Central-Ledger
  • Version: CSL v11.0.0, CL v16.3.2
  • Platform: CI
  • Subsystem: Integration Tests
  • Type of testing: CI
  • Bug found/raised by: @mdebarros

Notes:

  • Severity when opened: Medium
  • Priority when opened: Medium
@mdebarros mdebarros added bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it labels Dec 12, 2022
@mdebarros mdebarros mentioned this issue Dec 12, 2022
Merged
mdebarros added a commit to mojaloop/central-ledger that referenced this issue Dec 12, 2022
@mdebarros
fix(mojaloop/#2644): missing error-code for the transfer in the centr…
7e49456 
…al-ledger db (#929)

fix(mojaloop/#2644): missing error-code for the transfer in the central-ledger db - mojaloop/project#2644
- added payer_transfer_timeout.json test-case to feature_tests/transfer_negative_scenarios
- updated dependencies
- Added work-around for Kafka Producer Client disconnect/connect and created a following story to investigate this further as I believe its an issue with the Kafka Producer Client Lib: mojaloop/project#3067
- Added a new Helper `KafkaHelper` to manage connect/disconnect in an effort to resolve the above issue by implementing re-usable helper functions
- Updated docker-compose to include kafka-init to pre-create Kafka topics to improve stability of Integration Tests
- Fixed issue with Wait Retry helper functions which did not set the Timeout correct (i.e. it was hard-coded to 2, instead of being set by the input parameter)
- Added env config (TST_RETRY_COUNT, TST_RETRY_TIMEOUT) for Integration Tests to set input params for Wait Retry helper functions being called by the `./transfers/handlers.test.js` test-cases
- Updated `test:int` to include env configs for TST_RETRY_COUNT, TST_RETRY_TIMEOUT and UV_THREADPOOL_SIZE (increased avaialable threads for NODE-RDKAFKA). This should include consistency for Integration Tests.
@geka-evk geka-evk self-assigned this Apr 25, 2024
@geka-evk
Copy link

PR: mojaloop/central-services-stream#139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@geka-evk geka-evk

Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mdebarros @geka-evk @chris-me-law