CL very high latency in transfer processing

[vijayg10]

Summary:
Observed very high latency in transfer processing when running central-ledger handlers with version v17.0.4 (https://github.com/mojaloop/central-ledger/releases/tag/v17.0.4).
Due to this high latency, the websocket connections in the performance k6 tests are timing out before receiving callbacks.

Observations:
The problem got introduced in v17.0.4. Tested v17.0.3 and there is no issue with that.
Found the following errors in the prepare handler related to kafka

2023-09-21T16:56:01.835Z - error: "Consumer::onError()[topics='topic-transfer-position'] - Error: Broker: Unknown topic or partition\n    at Function.createLibrdkafkaError [as create] (/opt/app/node_modules/node-rdkafka/lib/error.js:456:10)\n    at /opt/app/node_modules/node-rdkafka/lib/kafka-consumer.js:468:29)"
2023-09-21T16:56:01.935Z - error: "Consumer::consume() - onRecursive - error Error: Broker: Unknown topic or partition"
2023-09-21T16:56:32.248Z - error: "Consumer::onError()[topics='topic-transfer-position'] - Error: Broker: Unknown topic or partition\n    at Function.createLibrdkafkaError [as create] (/opt/app/node_modules/node-rdkafka/lib/error.js:456:10)\n    at /opt/app/node_modules/node-rdkafka/lib/kafka-consumer.js:468:29)"
2023-09-21T16:56:32.349Z - error: "Consumer::consume() - onRecursive - error Error: Broker: Unknown topic or partition"

Severity:
(Low, Medium, High)

Priority:
(Critical, Medium, Low)

Expected Behavior

• There should be no websocket timeouts and k6 test should pass.
• There should be no errors related to kafka in handlers

Steps to Reproduce

1. Download the repository mojaloop/ml-core-test-harness and switch to main branch
2. Change the CENTRAL_LEDGER_VERSION version in .env file to v17.0.4
3. Change number of replicas to 1 for all the services in .env file for making it easy to debug
4. Start the services using following comand

   docker compose --project-name ml-core -f docker-compose-perf.yml --profile transfers-test --profile 8dfsp --profile ttk-provisioning-transfers up -d
   

5. Wait for the provisioning collection to be completed
6. Run a k6 test for a single transfer

   env K6_SCRIPT_CONFIG_FILE_NAME=fspiopSingleTransfer.json docker compose --project-name load -f docker-compose-load.yml up
   

7. You will notice an error with websocket WS timed-out on URL: ws://sim-perffsp
8. Observe the logs in prepare handler and you will some logs related to kafka like Error: Broker: Unknown topic or partition

Specifications

• Component (if known): central-ledger
• Version: v17.0.4
• Platform: ML Core Testharness
• Subsystem:
• Type of testing:
• Bug found/raised by: @vijayg10 @mdebarros @sri-miriyala

Notes:

• Severity when opened:
• Priority when opened:

[PaulGregoryBaker]

17.3 is working 17.4 is broken.
This change is realted to NodeJS upgrade and the use of the new Kafka lib.

[PaulGregoryBaker]

Hey team! Please add your planning poker estimate with Zenhub @aaronreynoza @kleyow @oderayi @sri-miriyala @vijayg10

[vijayg10]

Observed this issue mainly in prepare handler.

In the below image, there are three test runs. The first wave is with latest version of CL (v17.2.0) and other two runs are with older version(v17.0.3).