Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures in daily cron job running GP tests #2734

Closed
elnyry-sam-k opened this issue Apr 5, 2022 · 1 comment
Closed

Failures in daily cron job running GP tests #2734

elnyry-sam-k opened this issue Apr 5, 2022 · 1 comment
Assignees
@mdebarros
Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Milestone
Sprint 19.2

Comments

@elnyry-sam-k
Copy link
Member

elnyry-sam-k commented Apr 5, 2022 •
edited by mdebarros
Loading

Summary:

There are 18 total failures in the Mojaloop TTK GP tests that run daily.

Results: Total Assertions: 2535, TOTAL PASSED TESTS: 2517, TOTAL FAILED TESTS: 18

Most recent test results: here

  1. All failures except one are in POST /scenarios requests related tests, most of them failing with below error; possibly indicating issue with lookup (either a provisioning issue or bug)
errorCode: '3200', errorDescription: 'Generic ID not found' }
  1. and one failure in interchange fee settlement tests

. [Create Interchange-Fee Settlement for a single Window](https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-09-06T17%3A26%3A05.367Z.html#) fails intermittently (probably timing issue)
Ref: https://mojaloop-oss-qa-results.s3.us-west-2.amazonaws.com/ttk-tests/reports/TTK-Assertion-Report-multi-2022-09-06T17%3A26%3A05.367Z.html

{
"errorInformation": {
  "errorCode": "3100",
  "errorDescription": "Generic validation error - Inapplicable windows 35"
}
}

This bug is to capture these failures and not specific to the root cause.

Severity:
High

Priority:
Medium

Expected Behavior
The cron job runs with 100% pass rate (or zero failures, after running all the tests)

Steps to Reproduce

  1. Daily cron job running on a Mojaloop deployment
  2. Review test results (currently on the #announcements channel on Mojaloop slack)

Specifications

  • Component (if known): Lookup, Settlement
  • Version: v13.1.0 of Mojaloop
  • Platform: AWS (moja1)
  • Subsystem: lookup
  • Type of testing: Daily cron job
  • Bug found/raised by: elnyry-sam-k (logged)

Notes:

  • Severity when opened: High
  • Priority when opened: Medium
@elnyry-sam-k elnyry-sam-k added oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it bug Something isn't working or it has wrong behavior on a Mojaloop Core service labels Apr 5, 2022
@elnyry-sam-k elnyry-sam-k changed the title Failures in daily corn job running GP tests Failures in daily cron job running GP tests Apr 5, 2022
@elnyry-sam-k elnyry-sam-k added this to the Sprint 19.2 milestone Aug 17, 2022
@elnyry-sam-k elnyry-sam-k mentioned this issue Aug 17, 2022
Closed
22 tasks
@mdebarros mdebarros mentioned this issue Sep 6, 2022
Merged
mdebarros added a commit to mojaloop/testing-toolkit-test-cases that referenced this issue Sep 7, 2022
@mdebarros
fix(mojaloop/#2734): failures in daily cron job running GP tests
a87e552 
fix(mojaloop/#2734): failures in daily cron job running GP tests - mojaloop/project#2734
- added re-try logic to close interchange-fee settlement window as part of CGS test-suite
@mdebarros mdebarros mentioned this issue Sep 7, 2022
Merged
mdebarros added a commit to mojaloop/testing-toolkit-test-cases that referenced this issue Sep 8, 2022
@mdebarros
feat(mojaloop/#2880): simplify liquidity cover check and fix issue wi…
2bd14f7 
…th validation (#84)

feat(mojaloop/#2880): simplify liquidity cover check and fix issue with validation - mojaloop/project#2880
- fixes to collections/hub/golden_path/feature_tests/block_transfers
  - aligned liquidity and position calculations with central-ledger logic
  - cleaned up unnecessary requests (i.e. why have two requests to fetch settlement and position accounts when it can be done in one?)
- re-factored `[new sims] transfers, positions CGS Window1 tests (OTC-649, OTC-645)` @ collections/hub/golden_path/settlement_cgs/newsetcgs
  - standardised naming convention
  - re-aligned position, settlement and interchange checks with central-ledger logic, and expanded on checks dynamically based on the transfer amounts (`Tx1`, `Tx2`, `Tx3`, `Tx4`, `Tx5`, `Tx6`) instead of using "hard-coded" values
  - consolidated separate `GET /account` test-case for settlement, and positions into a single test-case (i.e. only need one call to admin API to retrieve balances and do validation checks)
  - added new OPTIONAL env vars for:
    1. `SETTLEMENT_CGS_TX_AMOUNT_1` (defaults: 15), `SETTLEMENT_CGS_TX_AMOUNT_2` (defaults: 87.88), `SETTLEMENT_CGS_TX_AMOUNT_3` (defaults: 158.36), `SETTLEMENT_CGS_TX_AMOUNT_4` (defaults: 211.11), `SETTLEMENT_CGS_TX_AMOUNT_5` (defaults: 14.05), `SETTLEMENT_CGS_TX_AMOUNT_6` (defaults: 150) representing the transfer amounts for each Transaction. These correlate to `Tx1`...`Tx6`
    2. `SETTLEMENT_RULE_INTERCHANGE_FEE` (defaults: 0.006) which is the Interchange Fee % used to calculate the aggregate Fee

fix(mojaloop/#2734): failures in daily cron job running GP tests - mojaloop/project#2734
- added re-try logic to create interchange-fee settlements as part of CGS test-suite
- bumped up RETRY_MAX_ATTEMPTS (10 to 20) & RETRY_MAX_WAIT_IN_MS (200 to 250) values to improve stability
mdebarros added a commit to mojaloop/testing-toolkit-test-cases that referenced this issue Sep 8, 2022
@mdebarros
feat(mojaloop/#2880): simplify liquidity cover check and fix issue wi…
c547f2a 
…th validation (master) (#86)

feat(mojaloop/#2880): simplify liquidity cover check and fix issue with validation - mojaloop/project#2880
- fixes to collections/hub/golden_path/feature_tests/block_transfers
  - aligned liquidity and position calculations with central-ledger logic
  - cleaned up unnecessary requests (i.e. why have two requests to fetch settlement and position accounts when it can be done in one?)
- re-factored `[new sims] transfers, positions CGS Window1 tests (OTC-649, OTC-645)` @ collections/hub/golden_path/settlement_cgs/newsetcgs
  - standardised naming convention
  - re-aligned position, settlement and interchange checks with central-ledger logic, and expanded on checks dynamically based on the transfer amounts (`Tx1`, `Tx2`, `Tx3`, `Tx4`, `Tx5`, `Tx6`) instead of using "hard-coded" values
  - consolidated separate `GET /account` test-case for settlement, and positions into a single test-case (i.e. only need one call to admin API to retrieve balances and do validation checks)
  - added new OPTIONAL env vars for:
    1. `SETTLEMENT_CGS_TX_AMOUNT_1` (defaults: 15), `SETTLEMENT_CGS_TX_AMOUNT_2` (defaults: 87.88), `SETTLEMENT_CGS_TX_AMOUNT_3` (defaults: 158.36), `SETTLEMENT_CGS_TX_AMOUNT_4` (defaults: 211.11), `SETTLEMENT_CGS_TX_AMOUNT_5` (defaults: 14.05), `SETTLEMENT_CGS_TX_AMOUNT_6` (defaults: 150) representing the transfer amounts for each Transaction. These correlate to `Tx1`...`Tx6`
    2. `SETTLEMENT_RULE_INTERCHANGE_FEE` (defaults: 0.006) which is the Interchange Fee % used to calculate the aggregate Fee

fix(mojaloop/#2734): failures in daily cron job running GP tests - mojaloop/project#2734
- added re-try logic to create interchange-fee settlements as part of CGS test-suite
- bumped up RETRY_MAX_ATTEMPTS (10 to 20) & RETRY_MAX_WAIT_IN_MS (200 to 250) values to improve stability
mdebarros added a commit to mojaloop/mojaloop-simulator that referenced this issue Sep 8, 2022
@mdebarros
fix(mojaloop/#2734): failures in daily cron job running gp tests (#146)
50ebcad 
fix(mojaloop/#2734): failures in daily cron job running GP tests - mojaloop/project#2734
- fixed logging to correctly display request logs. this is needed to help debug issues
- re-factored config-loading as this was badly being being handled, which impacted unit/integration tests etc.
- added unit tests for config-loading
- added support for `CONFIG_OVERRIDE` env var to support using example.env for unit/integration tests, with default being `.env` when running `npm start`.

chore: maintenance
- updated dependencies
- updated husky to the latest version, and moved config from package.json to .husky folder
- replaced module-alias with native import alias's
@mdebarros
Copy link
Member

mdebarros commented Sep 9, 2022 •
edited
Loading

Two follow-up bug reports have been filed as a result of this story:

  1. Helm Test Intermittent failure with Generic ID not found - Helm Test Intermittent failure with 'Generic ID not found' #2925
  2. Helm Test Intermittent failure with Inapplicable windows - Investigate: helm Test Intermittent failure with 'Inapplicable windows' #2926

Additional stability enhancements to the TTK GP Test Collections have been made as part of this release: https://github.com/mojaloop/testing-toolkit-test-cases/releases/tag/v14.0.0.1

mdebarros added a commit to mojaloop/helm that referenced this issue Sep 9, 2022
@mdebarros
feat(mojaloop/#2880): simplify liquidity cover check and fix issue wi…
8ece499 
…th validation (#496)

feat(mojaloop/[#2880](https://github.com/mdebarros/helm/issues/2880)): simplify liquidity cover check and fix issue with validation - mojaloop/project#2880
- upgraded mojaloop-simulator to v13.0.1
- upgraded central-settlement to v15.0.0
- upgraded central-ledger to v15.1.2.1
- updated TTK Test-Collection to v14.0.0.1

fix(mojaloop/#2734): Failures in daily cron job running GP tests - mojaloop/project#2734
- Included stability fixes in TTK Test-Collection to v14.0.0.1 release
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdebarros mdebarros

Labels
bug Something isn't working or it has wrong behavior on a Mojaloop Core service core-dev-squad oss-core This is an issue - story or epic related to a feature on a Mojaloop core service or related to it
Projects
None yet
Milestone
Sprint 19.2
Development

No branches or pull requests
2 participants
@elnyry-sam-k @mdebarros