Helm Test Intermittent failure with 'Generic ID not found' #2925

mdebarros · 2022-09-09T09:45:52Z

Summary:
Helm Test Intermittent failure with 'Generic ID not found' when executing transfers via the Mojaloop-Simulator's API for POST /scenarios requests related tests, failing with below error; possibly indicating an issue with lookup (either a provisioning issue or bug - after some investigation the source of the issue seems to occur at the Mojaloop-Simulator)

{ errorCode: '3200', errorDescription: 'Generic ID not found' }

Example reports:

Work-around:

Re-run the helm test, specifically re-run the following Test Collections:

$DEPLOYMENT_NAME-ml-ttk-test-setup <-- Re-provisioning script to re-create MISSING Party data on Mojaloop-Simulators
$DEPLOYMENT_NAME-ml-ttk-test-val-gp <-- Golden Path Test Collection

helm -n $NAMESPACE test $DEPLOYMENT_NAME --filter 'name=$DEPLOYMENT_NAME-ml-ttk-test-setup, name=$DEPLOYMENT_NAME-ml-ttk-test-val-gp'

Severity:
Low

Priority:
Low

Expected Behavior

helm tests should consistently pass without the ID not found with the Parties being correctly provisioned on the Mojaloop Simulators.

Steps to Reproduce

Deploy v14.0.0 Mojaloop Helm Release
Wait for all services to report healthy (i.e. 6-8m)
Execute several watch -n 300 helm test moja2 --filter "name=moja2-ml-ttk-test-setup,name=moja2-ml-ttk-test-val-gp"
Monitor test results, and cancel the watch command to stop running further tests

Specifications

Component (if known): Mojaloop GP Tests (run via Helm)
Version: v14.0.0
Platform: Helm
Subsystem:
Type of testing: N/A
Bug found/raised by: N/A

Dependency:

Improve Mojaloop Simulator logging #2928

Tasks for fix:

Using the updated ml simulator from Improve Mojaloop Simulator logging #2928 , deploy moja1 or moja2 with new simulators
Investigation and attempt to reproduce
Cron job / manual runs

Note: If issue not found move to closed or blocked until issue re-occurs and then revive the story

Notes:

Severity when opened: Low
Priority when opened: Low

The text was updated successfully, but these errors were encountered:

elnyry-sam-k · 2022-11-14T14:42:16Z

Hey team! Please add your planning poker estimate with Zenhub @mdebarros @sri-miriyala @vijayg10 @kleyow

kleyow · 2022-11-16T20:33:40Z

https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-16T20:31:35.199Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T00:47:19.760Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T01:00:19.346Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T01:07:27.254Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T13:27:07.171Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T14%3A46%3A53.209Z.html
https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-11-17T15%3A49%3A08.698Z.html

kleyow · 2022-11-17T16:00:22Z

After running the tests for multiple hours I have not observed the specific error labeled in this story. I've added failed reports that stem mostly from 5xx errors.

mojaloop/testing-toolkit-test-cases#91 may have resolved this issue.

Moving to blocked.
To be re assigned if observed in daily cron job again.
To be closed if not observed for a length period of time.

elnyry-sam-k · 2022-11-29T09:19:12Z

WIl re-open this if there's a repro, but closed for now as it is not reproducible

kleyow · 2023-02-22T01:24:09Z

Issue seems to present as "Id not found" and not "Generic ID not found".
We believe this issue is still present with the latest helm v14.1.1 release.
Issue seems to be related to either/both provisioning onboarding of parties for the mojaloop simulators tesfsp1-4 or the simulators forgeting the addition of parties since they are stored in memory.

mdebarros · 2023-03-07T16:59:16Z

PartyId that is missing from the testfsp4's backend (i.e. mojaloop/mojaloop-simulator:v13.0.1):

17891239872

Log from moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc:

{"ts":"2023-03-07T14:55:56.235Z","msg":"Found handler","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"handler":"[Function: getPartiesByTypeAndId]"}}
{"ts":"2023-03-07T14:55:56.237Z","msg":"Executing HTTP GET","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"reqOpts":{"method":"GET","uri":"http://moja4-sim-testfsp4-backend:3000/parties/MSISDN/17891239872","headers":{"Content-Type":"application/json","Accept":"application/json","Date":"Tue, 07 Mar 2023 14:55:56 GMT"}}}}
{"ts":"2023-03-07T14:55:56.239Z","msg":"Request processed","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"}}}
{"ts":"2023-03-07T14:55:56.240Z","msg":"Cache message received on channel __keyevent@0__:set. Making callback with id 0","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","component":"cache"}}
{"ts":"2023-03-07T14:55:56.240Z","msg":"Received Redis keyevent notification","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-test-api","component":"websocket-server","key":"request_17891239872","channel":"__keyevent@0__:set","id":"0"}}
{"ts":"2023-03-07T14:55:56.248Z","msg":"Error in getParties","ctx":{"simulator":"payeefsp","hostname":"moja4-sim-testfsp4-scheme-adapter-b9687468c-mjfqc","app":"mojaloop-connector-inbound-api","component":"api","request":{"id":"abandoned-orange-deafening-yak","path":"/parties/MSISDN/17891239872","method":"GET"},"err":"{\"msg\":\"Request returned non-success status code 404\",\"res\":{\"statusCode\":404,\"headers\":{\"content-type\":\"application/json; charset=utf-8\",\"content-length\":\"46\",\"date\":\"Tue, 07 Mar 2023 14:55:56 GMT\",\"connection\":\"keep-alive\",\"keep-alive\":\"timeout=5\"},\"data\":{\"statusCode\":\"3200\",\"message\":\"ID not found\"},\"originalRequest\":{\"method\":\"GET\",\"host\":\"moja4-sim-testfsp4-backend\",\"port\":\"3000\",\"path\":\"/parties/MSISDN/17891239872\",\"headers\":{\"Content-Type\":\"application/json\",\"Accept\":\"application/json\",\"Date\":\"Tue, 07 Mar 2023 14:55:56 GMT\"},\"agent\":\"[REDACTED]\",\"body\":null}}}"}}

Request to backend to list all parties

curl 'http://moja4-sim-testfsp4-backend:3003/repository/parties' -H 'traceparent: 00-aabbe4cf5b5aa1b531408d4bbddbcd5c-0123456789abcdef0-00' -H 'user-agent: axios/0.27.2' --compressed

Response with missing 17891239872 party

{
  "status": 200,
  "statusText": "OK",
  "body": [
    {
      "displayName": "testfsp4BankFname testfsp4Bankname",
      "firstName": "testfsp4BankFname",
      "middleName": "",
      "lastName": "testfsp4BankLname",
      "dateOfBirth": "1985-05-13",
      "idType": "MSISDN",
      "idValue": "27713813917",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "476036885579",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "124241848903",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "258100991099",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "511367992055",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    },
    {
      "displayName": "fsp4WalletFname{{$timestamp}} fsp4WalletLname{{$timestamp}}",
      "firstName": "fsp4WalletFname{{$timestamp}}",
      "middleName": "",
      "lastName": "fsp4WalletLname{{$timestamp}}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "901216703790",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    },
    {
      "displayName": "fsp4BankFname{$timestamp} fsp4BankLname{$timestamp}",
      "firstName": "fsp4BankFname{$timestamp}",
      "middleName": "",
      "lastName": "fsp4BankLname{$timestamp}",
      "dateOfBirth": "2023-03-07",
      "idType": "MSISDN",
      "idValue": "654117426119",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Bank"
        }
      ]
    }
  ],
  "headers": {
    "vary": "Origin",
    "content-type": "application/json; charset=utf-8",
    "content-length": "2051",
    "date": "Tue, 07 Mar 2023 16:25:32 GMT",
    "connection": "keep-alive",
    "keep-alive": "timeout=5"
  }
}

GET http://moja4-sim-testfsp4-backend-9db946c89-49v5h:3000/parties/MSISDN/17891239872

{"statusCode":"3200","message":"ID not found"}

Log from moja4-sim-testfsp4-backend-9db946c89-49v5h:

{
  app: �[32m'simulator'�[39m,
  request: {
    id: �[32m'steep-calm-helpful-furniture'�[39m,
    path: �[32m'/parties/MSISDN/17891239872'�[39m,
    method: �[32m'GET'�[39m
  },
  response: {
    body: { statusCode: �[32m'3200'�[39m, message: �[32m'ID not found'�[39m },
    status: �[33m404�[39m
  },
  msg: �[32m'Request processed'�[39m,
  timestamp: �[32m'2023-03-07T14:55:56.246Z'�[39m
}

Archive.zip

mdebarros · 2023-03-07T17:01:15Z

Re-running the API operation to create the Party on the Mojaloop-Simulator after a restart:

curl 'http://moja4-sim-testfsp4-backend:3003/repository/parties' -H 'content-type: application/json' -H 'traceparent: 00-aabbe4cf5b5aa1b531408d4bbddbcd90-0123456789abcdef0-00' -H 'user-agent: axios/0.27.2' --data-binary '{"displayName":"testfsp4WalletFname testfsp4WalletLname","firstName":"testfsp4WalletFname","middleName":"","lastName":"testfsp4WalletLname","dateOfBirth":"1985-05-13","idType":"MSISDN","idValue":"17891239872","extensionList":[{"key":"accountType","value":"Wallet"}]}' --compresse

{
  "status": 200,
  "statusText": "OK",
  "body": [
    {
      "displayName": "testfsp4WalletFname testfsp4WalletLname",
      "firstName": "testfsp4WalletFname",
      "middleName": "",
      "lastName": "testfsp4WalletLname",
      "dateOfBirth": "1985-05-13",
      "idType": "MSISDN",
      "idValue": "17891239872",
      "extensionList": [
        {
          "key": "accountType",
          "value": "Wallet"
        }
      ]
    }
  ],
  "headers": {
    "vary": "Origin",
    "access-control-allow-origin": "",
    "content-type": "application/json; charset=utf-8",
    "content-length": "268",
    "date": "Tue, 07 Mar 2023 16:56:28 GMT",
    "connection": "keep-alive",
    "keep-alive": "timeout=5"
  }
}

mdebarros · 2023-03-07T17:21:56Z

Unable to determine the cause of the missing participant, except that it is seems to be occuring during the onboarding setup...

The provisioning script hub/provisioning/CGS_Specific/add-users-to-new-sims-and-als-registration.json still shows the operation as being successful:

  �[36mAdd Users to new Sims ; ALS registration -> [testfsp4, Wallet] POST /parties  {{SIM4_MSISDN}}�[39m
	�[32m[ SUCCESS ]�[39m	�[32mresposne code is 200�[39m

Will need to re-look at this once mojaloop-simulator/releases/tag/v13.1.0 or later is packaged in a release. This version will also include audits for incoming repository requests which will help identify the cause of the issue.

mdebarros added the bug Something isn't working or it has wrong behavior on a Mojaloop Core service label Sep 9, 2022

mdebarros mentioned this issue Sep 9, 2022

Failures in daily cron job running GP tests #2734

Closed

elnyry-sam-k added oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it core-dev-squad labels Sep 9, 2022

chris-me-law assigned kleyow Nov 16, 2022

chris-me-law closed this as completed Nov 28, 2022

mdebarros reopened this Mar 7, 2023

mdebarros self-assigned this Mar 7, 2023

mojaloopci mentioned this issue Jul 9, 2024

feat(auto): release candidate for Zambezi v16.1.0 mojaloop/helm#640

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Helm Test Intermittent failure with 'Generic ID not found' #2925

Helm Test Intermittent failure with 'Generic ID not found' #2925

mdebarros commented Sep 9, 2022 •

edited

Loading

elnyry-sam-k commented Nov 14, 2022

kleyow commented Nov 16, 2022 •

edited

Loading

kleyow commented Nov 17, 2022

elnyry-sam-k commented Nov 29, 2022

kleyow commented Feb 22, 2023

mdebarros commented Mar 7, 2023 •

edited

Loading

mdebarros commented Mar 7, 2023

mdebarros commented Mar 7, 2023 •

edited

Loading

Helm Test Intermittent failure with 'Generic ID not found' #2925

Helm Test Intermittent failure with 'Generic ID not found' #2925

Comments

mdebarros commented Sep 9, 2022 • edited Loading

elnyry-sam-k commented Nov 14, 2022

kleyow commented Nov 16, 2022 • edited Loading

kleyow commented Nov 17, 2022

elnyry-sam-k commented Nov 29, 2022

kleyow commented Feb 22, 2023

mdebarros commented Mar 7, 2023 • edited Loading

mdebarros commented Mar 7, 2023

mdebarros commented Mar 7, 2023 • edited Loading

mdebarros commented Sep 9, 2022 •

edited

Loading

kleyow commented Nov 16, 2022 •

edited

Loading

mdebarros commented Mar 7, 2023 •

edited

Loading

mdebarros commented Mar 7, 2023 •

edited

Loading