Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proposal: Remove external dependencies that are tightly integrated into the Mojaloop Helm Chart #69

Closed
2 tasks
mdebarros opened this issue Nov 19, 2020 · 3 comments
Closed
2 tasks

Proposal: Remove external dependencies that are tightly integrated into the Mojaloop Helm Chart #69

mdebarros opened this issue Nov 19, 2020 · 3 comments
Assignees
@mdebarros
Labels
theme-non-functional-support Improvements in DFSP Onboarding time, Cybersecurity & Performance as a Pillar
Milestone
PI 12

Comments

@mdebarros
Copy link
Member

mdebarros commented Nov 19, 2020 •
edited by godfreykutumela
Loading

One Line Summary:

I propose that all external dependencies are removed from the Mojaloop helm charts, and are instead deployed as a separate pre-requisites.

The rationale behind this being:

  1. We currently do not see any Mojaloop switch implementors using these dependencies in QA/Prod environments. They instead have chosen to either deploy their own dependencies or rely on managed services instead.
  2. The tightly coupled external dependencies are proving to be an issue for the Mojaloop Helm chart upgradability as we do not have any control over them, by externalizing them we remove this tight coupling on Mojaloop Helm deployments. This ensures that the onus is on the deployer to ensure that all dependencies are deployed as necessary.
  3. This also ensures that any Mojaloop Helm deployment can be easily undeployed without impacting the persistence layer (i.e. data stores, persistent message streams, etc). This is useful for deploying new versions that do not support upgrades (i.e. similar to this issue Helm upgrade v10.4.0 -> v11.0.0 upgrade issues project#1759).
  4. The following points will most likely break the upgradability of the charts in the future similar (i.e. similar to this issue Helm upgrade v10.4.0 -> v11.0.0 upgrade issues project#1759):
    • Official Helm repos for stable and incubation charts are being deprecated, and we will have to either migrate repos or find alternative helm charts for external dependencies that are currently not maintained.
    • Many charts are also not being maintained due to the deprecation of Helm v2, and instead, integrators are choosing to create Helm v3 charts that are not backwards compatible.
    • Many charts are moving to a Kubernetes Operator model, which does not play well when tightly integrated as a dependency on a parent chart.

We would need to enhance the "Deployment Guide" documentation to provide proper guidance on to deploy the dependencies and configure the Mojaloop Helm chart appropriately.

Request Details:

Artifacts:

Dependencies:

Accountability:

Decision(s):

  1. Dependencies will be removed
  2. Documentation deployment guide updated to include them as pre-requisite dependencies
  3. Documentation deployment guide updated to include information on how to deploy the pre-requisite dependencies
  4. Separate initiative will need to occur to simplify Mojaloop's deployment steps into a "one-click" experience

Details

  • Actual decision made as a result of discussion

Follow-up:

  • Actions to implement the decisions
@mdebarros mdebarros self-assigned this Nov 19, 2020
@godfreykutumela godfreykutumela added this to the PI 12 milestone Nov 19, 2020
@godfreykutumela godfreykutumela added the theme-non-functional-support Improvements in DFSP Onboarding time, Cybersecurity & Performance as a Pillar label Nov 19, 2020
@mdebarros mdebarros mentioned this issue Nov 25, 2020
Merged
mdebarros added a commit to mojaloop/helm that referenced this issue Nov 25, 2020
@mdebarros
v11.0.0-PI12 (#401)
87373ca 
## Helm release changes

### 1. Maintenance updates
1. GitHub issue: mojaloop/project#1617
2. FSP Interoperability API v1.1 Swagger file used from here: https://github.com/mojaloop/mojaloop-specification/tree/master/documents/v1.1-document-set
3. For breaking changes, please review the "Breaking Changes" section below for for Mojaloop Helm v11.0.0
4. Notes on simulators 
	- [Mojaloop-Simulator](https://github.com/mojaloop/mojaloop-simulator) is enabled by default (six instances used). 
	- Ensure that correct Postman Scripts are used if you wish to test against the Mojaloop-Simulators:
        	-  A single MojaSim collection is now available after reconciliation between two collections that existed previously.
        	- Setup Mojaloop Hub : [MojaloopHub_Setup](https://github.com/mojaloop/postman/blob/master/MojaloopHub_Setup.postman_collection.json)
        	- Setup Mojaloop Simulators for testing : [MojaloopSims_Onboarding](https://github.com/mojaloop/postman/blob/master/MojaloopSims_Onboarding.postman_collection.json)
        	- Golden path tests: [Golden_Path_Mojaloop](https://github.com/mojaloop/postman/blob/master/Golden_Path_Mojaloop.postman_collection.json)
	- Legacy Simulators are still required and deployed by default; disabling this will cause issues since there is Account Lookup directory mocking functionality in this service. To run Golden path tests using the Legacy simulator:
                - Setup Mojaloop Hub, test FSPs : [ML_OSS_Setup_LegacySim](https://github.com/mojaloop/postman/blob/master/ML_OSS_Setup_LegacySim.postman_collection.json)
        	- Golden path tests: [ML_OSS_Golden_Path_LegacySim](https://github.com/mojaloop/postman/blob/master/ML_OSS_Golden_Path_LegacySim.postman_collection.json)
	- In order to run the Legacy Simulator postman collection or the Mojaloop-Simulator collection, their corresponding setup collection needs to be run first; this can be done without redeployment.
5. Upgrades to all core and supporting services to support 11.0.0 release _(see Application versions and release notes below)_
6. The current Golden Path collection expects
 `quoting-service.config.simple_routing_mode_enabled` to be set to `false` to get 100% pass-rate, if this flag is disabled, there will be failures in several quoting-service tests (7 expected failures). This command can be used to override the default config: `helm install moja ./mojaloop/ -n demo --set quoting-service.config.simple_routing_mode_enabled=false`
7. Along with this configuration for on-us transfers is disabled on both default deployments and postman (ON_US_TRANSFERS_ENABLED on postman) collections.
8. Moving to SemVer as the standard is now adopted. This will start with v11.0.0 and follow the general semantic versioning guidance after that.
9. Updated Promfana chart with "Prometheus: v2.20.1; Grafana: v7.1.1"
10. Updated EFK chart with “appVersion: "elasticsearch: 7.7.1; kibana: 7.7.1; apm-server: 7.7.1; fluentd-elasticsearch: 3.0.4"
11. Charts now support deployments on Kubernetes v1.16.x onwards - Fix for issue #219
12. Removed deprecated ingress-nginx chart (#350). Please use [Nginx official chart](https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/).
13. Updated official Helm repo URIs for stable & incubator dependencies and documentation: 
	- Stable: https://kubernetes-charts.storage.googleapis.com/ --> https://charts.helm.sh/stable
	- Incubator: http://storage.googleapis.com/kubernetes-charts-incubator --> https://charts.helm.sh/incubator
13. Updated `innodb_flush_log_at_trx_commit` to the value of `1` for MySQL servers to ensure ACID compliance (ref: https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit)
14. Renamed Central-Event-Processor's (CEP) MongoDB instance dependency to: `nameOverride: cep-mongodb`. This will ensure that the CEP has its own MongoDB instance.
15. Renamed Mojaloop-Self-Testing-Toolkit's (TTK) MongoDB instance dependency to: `nameOverride: ttk-mongodb`. This will ensure that the TTK has its own MongoDB instance.


### 2. New Features
1. Along with changes from v10.4.0 related to the version v1.1 of the Mojaloop FSP Interoperability API, additional follow-up changes are implemented and a few fixes made. The changes are listed in the [tracking issue](mojaloop/mojaloop-specification#52) of the specification and there is additional context offered [here](https://mojaloop.io/mojaloop-fsp-interoperability-api-version-1-1/). Epic: mojaloop/project#1333
2. Added TTK CLI chart to ml-testing-toolkit (#380).
3. Added the option to pass a template as the rules.json (#353) in quoting service
4. Updates for bulk transfers support in MojaSims/Scheme-Adapter (mojaloop/project/issues/1386)
5. Added charts for bulk-get-handler: mojaloop/project/issues/1496
6. Package development versions (#354)
7. Update helm charts to support kubernetes deployments1.15 and later versions: mojaloop/project/issues/1463 (PR#349)
8. Added TTK CLI chart to ml-testing-toolkit (#380)
9. Modified the testing toolkit CLI chart to enabled scheduling (#381) 


### 3. Bug Fixes
1. Incorrect error behaviour on transfer fulfilment with future timestamp [1708](mojaloop/project#1708)
2. ACCEPT header returned in the PUT from switch for the timeout scenario [1710](mojaloop/project#1710) 
3. Date header missing from the Switch signature in timeout scenario [1709](mojaloop/project#1709)  
4. Valid transactionRequests callback in SDK responding with error response [1647](mojaloop/project#1647)  
5. Support test currencies XXX, XTS in ml-api-adapter [1652](mojaloop/project#1652)

## 4. Application versions
Application versions that are supported for this update:
1. ml-api-adapter: v10.5.0 -> **v11.1.2**
2. central-ledger: v10.5.1 -> **v11.3.1**
3. account-lookup-service: v10.4.2 -> **v11.1.2**
4. quoting-service: v10.5.5 -> **v11.1.4**
5. central-settlement: **v10.5.0** (no change)
6. central-event-processor: **v10.5.0** (no change)
7. bulk-api-adapter: v10.5.0 -> **v11.0.2**
8. email-notifier: **v9.5.0** (no change)
9. als-oracle-pathfinder: v9.4.0 --> **v10.2.0**
10. transaction-requests-service: v10.4.0 -> **v11.1.2**
11. finance-portal-ui: **v10.4.0** (no change)
12. finance-portal-backend-service: **v10.4.0** (no change)
13. settlement-management: **v8.8.2** (no change)
14. operator-settlement: **v9.2.1**  (no change)
15. event-sidecar: **v9.5.1**  (no change)
16. event-stream-processor: **v9.5.0-snapshot**
17. simulator: v10.5.3 -> **v11.0.2**
18. mojaloop-simulator: v10.4.1 -> **v11.2.1**
19. sdk-scheme-adapter: v10.5.0 -> **v11.8.0**
20. ml-testing-toolkit: v10.4.1 -> **v11.5.0**
21. ml-testing-toolkit-ui: v10.4.1 -> **v11.5.0**
22. ml-testing-toolkit-cli: **v11.3.0** (new)
 
## 5. Application release notes
1. ml-api-adapter - https://github.com/mojaloop/ml-api-adapter/releases/tag/v11.1.2
2. central-ledger - https://github.com/mojaloop/central-ledger/releases/tag/v11.3.1
3. account-lookup-service - https://github.com/mojaloop/account-lookup-service/releases/tag/v11.1.2
4. quoting-service - https://github.com/mojaloop/quoting-service/releases/tag/v11.1.4
5. central-settlement- https://github.com/mojaloop/central-settlement/releases/tag/v10.5.0
6. central-event-processor - https://github.com/mojaloop/central-event-processor/releases/tag/v10.5.0
7. bulk-api-adapter - https://github.com/mojaloop/bulk-api-adapter/releases/tag/v11.0.2
8. email-notifier - https://github.com/mojaloop/email-notifier/releases/tag/v9.5.0
9. als-oracle-pathfinder - https://github.com/mojaloop/als-oracle-pathfinder/releases/tag/v10.2.0
10. transaction-requests-service - https://github.com/mojaloop/transaction-requests-service/releases/tag/v11.1.2
11. finance-portal-ui - https://github.com/mojaloop/finance-portal-ui/releases/tag/v10.4.0
12. finance-portal-backend-service - https://github.com/mojaloop/finance-portal-backend-service/releases/tag/v10.4.0
13. settlement-management - https://github.com/mojaloop/settlement-management/releases/tag/v8.8.2
14. operator-settlement - https://github.com/mojaloop/operator-settlement/releases/tag/v9.2.1
15. event-sidecar - https://github.com/mojaloop/event-sidecar/releases/tag/v9.5.1
16. event-stream-processor - https://github.com/mojaloop/event-stream-processor/releases/v9.5.0-snapshot
17. simulator - https://github.com/mojaloop/simulator/releases/tag/v11.0.2
18. mojaloop-simulator - https://github.com/mojaloop/mojaloop-simulator/releases/tag/v11.2.1
19. sdk-scheme-adapter - https://github.com/mojaloop/sdk-scheme-adapter/releases/tag/v11.8.0
20. ml-testing-toolkit - https://github.com/mojaloop/ml-testing-toolkit/releases/tag/v11.5.0
21. ml-testing-toolkit-ui - https://github.com/mojaloop/ml-testing-toolkit-ui/releases/tag/v11.5.0
22. ml-testing-toolkit-cli - https://github.com/mojaloop/ml-testing-toolkit-ui/releases/tag/v11.3.0
 
## 6. Operational Chart versions
- EFK: **v7.1.0**  -> **v11.0.0**
- Promfana: **v8.4.0** -> **v11.0.0**

## 7. Breaking changes
- Helm release v10.4.0 is not upgradable to v11.0.0 - mojaloop/project#1759

## 8. Known Issues
- Proposal to remove external dependencies from Mojaloop Helm Charts - mojaloop/design-authority-project#69 **(_Note: this will be a breaking change in the next major release_)**
- ~Testing Toolkit Post-Hook Install tests are not compatible with Postman (_Note: this will not impact default installations_) - mojaloop/project#1864


## 9. Contributors:

- Contributing organizations: BMGF, CrossLake, MBX 
- Crosslake: @lewisdaly
- ModusBox: @amarmodus, @elnyry-sam-k , @mdebarros, @oderayi, @rmothilal , @vgenev , @vijayg10, @msk-, @vbarzokas, @aaronreynoza, @KamuelaFranco
- Individuals:  @vorburger
_Note: companies in alphabetical order_

---
## PR Dependencies
- mojaloop/project#1866
- mojaloop/postman#188
@mdebarros mdebarros mentioned this issue Nov 25, 2020
Closed
29 tasks
@godfreykutumela
Copy link
Contributor

Hi, @mdebarros the DA has approved this decision. The updating of the deployment guide and enabling of a one-click deployment should be addressed as separate issues which can be started parallel to this one and will not require any further DA approval. Thanks and please let me know should you need anything on this.

@mdebarros mdebarros mentioned this issue Jan 11, 2021
Open
28 tasks
@lewisdaly
Copy link
Contributor

@mdebarros I wonder if we can go one step further and make our helm charts even more modular...

I'm thinking more of a "plugin" style approach, where users can install the bare-bones mojaloop by default, and then can install additional components that are managed in their own helm charts.

@mdebarros
Copy link
Member Author

@mdebarros I wonder if we can go one step further and make our helm charts even more modular...

I'm thinking more of a "plugin" style approach, where users can install the bare-bones mojaloop by default, and then can install additional components that are managed in their own helm charts.

It's worth noting that one can already do that. You can install each chart individually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdebarros mdebarros

Labels
theme-non-functional-support Improvements in DFSP Onboarding time, Cybersecurity & Performance as a Pillar
Projects
None yet
Milestone
PI 12
Development

No branches or pull requests
3 participants
@lewisdaly @mdebarros @godfreykutumela