Develop (#8)

* added timestamps in the datamodels and notification interval control * removed vsvode env directory from the central repo * Message template changes (#1) * Message template changes * added file ti gitignore * delete package-lock.json * Gitignore .idea * delete file * delete file * delete file * delete file * delete file * delete file * delete file * delete file * delete file * delete file * updated a typo * Fix/config (#4) * added readme.md and config to be available from environmental variables * added config doc * Added database docs (#5) * Added database docs * Added link in readme.md * Fix/env var (#6) * added readme.md and config to be available from environmental variables * added config doc * added config to take envvars for sensitive data, fixed directory name typo * fixed a config issue * added healthcheck * Feature/dockerize (#7) * Added Dockerfile * Added Dockerfile and dockerignore
mojaloop · Nov 27, 2018 · d4cef0c · d4cef0c
1 parent 050d9ff
commit d4cef0c
Show file tree

Hide file tree

Showing 34 changed files with 1,718 additions and 455 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,18 @@
+deploy/
+coverage/
+node_modules/
+.dockerignore
+.editorconfig
+.git/
+.gitignore
+.istanbul.yml
+circle.yml
+docker-compose.circle.yml
+docker-compose.dev.yml
+docker-compose.functional.yml
+docker-compose.yml
+Dockerfile
+LICENSE
+README.md
+
+sonar-project.properties
diff --git a/.gitignore b/.gitignore
@@ -10,6 +10,8 @@ pids
 *.pid
 *.seed
 *.pid.lock
+.vscode
+package-lock.json
 
 # Directory for instrumented libs generated by jscoverage/JSCover
 lib-cov
@@ -35,12 +37,14 @@ build/Release
 # Dependency directories
 node_modules/
 jspm_packages/
+package-lock.json
 
 # TypeScript v1 declaration files
 typings/
 
 # Optional npm cache directory
 .npm
+.idea
 
 # Optional eslint cache
 .eslintcache

diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,23 @@
+FROM mhart/alpine-node:8.9.4
+USER root
+
+WORKDIR /opt/central-event-processor
+COPY src /opt/central-event-processor/src
+COPY config /opt/central-event-processor/config
+COPY package.json /opt/central-event-processor/
+COPY app.js /opt/central-event-processor/
+COPY docs /opt/central-event-processor/docs
+
+RUN apk --no-cache add git
+RUN apk add --no-cache -t build-dependencies make gcc g++ python libtool autoconf automake \
+    && cd $(npm root -g)/npm \
+    && npm config set unsafe-perm true \
+    && npm install -g node-gyp
+
+RUN npm install --production && \
+  npm uninstall -g npm
+
+RUN apk del build-dependencies
+
+EXPOSE 3080
+CMD node app.js
diff --git a/README.md b/README.md
@@ -1,115 +1,130 @@
+# Notification service
+
+## Contents
+<!-- vscode-markdown-toc -->
+* 1. [Stories](#Stories)
+* 2. [Tasks](#Tasks)
+* 3. [Reacts on](#Reactson)
+* 4. [Used technologies](#Usedtechnologies)
+* 5. [Local storage](#Localstorage)
+* 6. [Architecture overview](#Architectureoverview)
+* 7. [General process overview](#Generalprocessoverview)
+	* 7.1. [enums](#enums)
+	* 7.2. [Rules](#Rules)
+  * 7.3  [Config](#Config)
+* 8. [Limit Adjustment Rules flow](#LimitAdjustmentRulesflow)
+* 9. [Limit Position Threshold Breach flow](#LimitPositionThresholdBreachflow)
+* 10. [Actions Agent flow](#ActionsAgentflow)
+* 11. [Notifier flow (separate service)](#Notifierflowseparateservice)
+
+<!-- vscode-markdown-toc-config
+	numbering=true
+	autoSave=true
+	/vscode-markdown-toc-config -->
+<!-- /vscode-markdown-toc -->
+
+##  1. <a name='Stories'></a>Stories
+=======
 Central Event Processor
 ====================
 
 stories
 -------
+
 * [#517 - Notification for changes to NetDebitCap or Position adjustments](https://github.com/mojaloop/project/issues/517)
 * [#518 - Notification for approaching Net Debit Cap Threshold](https://github.com/mojaloop/project/issues/518)
 
-tasks
---------
-* receive urgent notifications and ?requests?
-* sends notifications based on configs to dfsps
-* receives config updates
-* receives position updates on the smallest period of warnings into the configs for all dfsps (json view)
-* logging? what and where?
-
-reacts on 
-----------------------
-* notification configs changes - the data won't serve any other purpose but to config notification engines so why not JSON?
-  * settings for different threshold alarms, channels of delivery etc.
-* urgent notifications - kafka msg routed to the engine based on the lowest critical level stored in the central database 
-* positions/limits/NDC or any other type of changes in sensitive data - why not get it as a big JSON view document with all dfsps
-  * get it by direct central database access / kafka-admin-topic / http request to central-ledger
-
-data
-----
-* configs
-* views
-* urgent notifications
-
-local storage and central database
------------------------------------
-* can we use mysql json data type on central database or string in mysql?
-* the view can be generated by running the same query over and over again, which is optimised by sql db enginge.
-* central database `limits and threshold` table to store only the value of critical levesl(% of NCP/value?) for urgent notifications and id/address of the latest received config or config itself
-* if we decide to have local storage it can be key:value or document storage
-  * we can use the view idea for reporting on later stage
-  * we can deliver views for different dashboard apps for admin and troubleshooting purposes to the dfsps. HUB admin might be connected to the central database for live data
-  * we can use this storage for enums/pointers or other references to the central database to offload it a bit (no updates of pointers for example, no enum queries from different functionalities)
-
-actors
--------------------
-* central-ledger
-* notification-router
-* notification-engine
-
-technologies
-------------
-* [napajs](https://github.com/Microsoft/napajs) for process management into the notification services
-* [node-schedule](https://www.npmjs.com/package/node-schedule) for scheduling tasks
-
-
-connections
--------------------
-|central-ledger| connection |notification-router| connection |notification-engine|
-|-|-|-|-|-|
-|kafka-admin-topic| <->|config|||              
-|kafka-notification-topic|<->|urgent notificaions|||
-|central-database |<-| creating dashboard view||| 
-||| local storage for configs and view | <->| sends notifications based on delta |
-|||napajs? sockets? kafka?| <->| receives and acts on urgent notifications|
-
-* central-ledger
-  * API to support setting threshold(s) for Notifications/Alerts (config)
-  * compare current critical level with new critical level from config and act on it
-  * produce urgent notifications if critical level is reached
-
-* notification-router
-  * connected to central-ledger
-    * kafka (admin topics) - config changes, json dashboard data?
-    * kafka notification topic
-    * local storage
-    * central database access (for view or maybe view from kafka admin topic? not really time reliable)
-  * connected to notification-engine for sending urgent messages (napajs/kafka/http)
-
-* notification-engine(s)
-  * spawn notification-engine per dfsp as a worker process (napajs) or another docker container (how it will communicate with router in that case)
-  * connected to the local storage
-    * without local storage we need connection to the router for requests to the central database or direct connection to the central database
-  * runs scheduled processes - based on config, requests data received by the router with the dashboard json view and reacts on urgent notifications
-    * [node-schedule](https://www.npmjs.com/package/node-schedule)
-  * operates different channels to reach dfsp (email/sms/etc)
-
-View - example 
---------------
+##  2. <a name='Tasks'></a>Tasks
+* send notifications when current position breaches the limit threshold value after a successful transfer was commited
+* send notifications when the limit was adjusted 
+* make it extendable and flexible
+
+##  3. <a name='Reactson'></a>Reacts on 
+* messages consumed from the notification topic
+* data read from the central-ledger API
+
+##  4. <a name='Usedtechnologies'></a>Used technologies
+* [RxJS](https://github.com/ReactiveX/rxjs)
+* [json-rule-engine](https://github.com/cachecontrol/json-rules-engine)
+* [mongoose](https://github.com/Automattic/mongoose)
+
+##  5. <a name='Localstorage'></a>Local storage
+* Mongo DB
+* Mongoose is used for schema validations and ORM functions
+* to set up connection the following environmental variables might be used: `CEP_DATABASE_URI` and `CEP_DATABASE_NAME`
+* full database documentation can be found [here](docs/database/Mojaloop_central-notifications_Db_ver1.0.html)
+
+##  6. <a name='Architectureoverview'></a>Architecture overview
+![architecture](docs/images/1.png)
+
+
+This is standalone service which is connected to Kafka Notification topic into the mojaloop environemnt and monitors the topic for messages which match certain rules and takes actions accordingly. 
+
+The service is developed using [RxJS](https://github.com/ReactiveX/rxjs) for observing the system and acting accordingly. The decissions for actions are taken by the [json-rule-engine](https://github.com/cachecontrol/json-rules-engine). 
+
+##  7. <a name='Generalprocessoverview'></a>General process overview
+![process](docs/images/2.png)
+
+The rules validations are triggered upon commited transfers. As soon as a commited transfer notification is produced from the central-ledger to the notification topic, the central-notifications service picks it up, gathers more information, runs few rules validations and acts based on rules engine outcome.
+
+The data for performing rules validation is requested from the central-ledger admin API calls using observables, available [here](src/observables/centralLedgerAPI.js) some mapping and wiring is done through below [enums](src/lib/enum.js) properties:
+
+###  7.1. <a name='enums'></a>enums
 ```
-{
-  timestamp: 
-  dfspsViews: [
-    dfsp1: {
-      curentNDC:
-      position:
-      ......
-
-    },
-    dfsp2: {
-
-    }
-  ]
+const notificationActionMap = {
+  NET_DEBIT_CAP_THRESHOLD_BREACH_EMAIL: {
+    enum: 'NET_DEBIT_CAP_THRESHOLD_BREACH_EMAIL',
+    action: 'sendEmail',
+    templateType: 'breach',
+    language: 'en'
+  },
+  NET_DEBIT_CAP_ADJUSTMENT_EMAIL: {
+    enum: 'NET_DEBIT_CAP_ADJUSTMENT_EMAIL',
+    action: 'sendEmail',
+    templateType: 'adjustment',
+    language: 'en'
+  }
+}
+
+const limitNotificationMap = {
+  NET_DEBIT_CAP: {
+    enum: 'NET_DEBIT_CAP',
+    NET_DEBIT_CAP_THRESHOLD_BREACH_EMAIL: notificationActionMap.NET_DEBIT_CAP_THRESHOLD_BREACH_EMAIL,
+    NET_DEBIT_CAP_ADJUSTMENT_EMAIL: notificationActionMap.NET_DEBIT_CAP_ADJUSTMENT_EMAIL
+  }
 }
 ```
-urgent notifications flow overview
-----------------------------------
-1. central ledger creates urgent notification on kafka-notification-topic
-2. router receives the notification and sends to particular engine
-3. engine acts based on config (maybe not send same notificaion more than 3 times in an hour)
-
-view processing flow overview
------------------------------
-1. router gets the view from central database on given period and stores it into local storage
-2. enginge gets the view from local storage on its own period (might be bigger)
-3. engine compares old view with the new one and gives delta
-4. engine process delta and config and creates actions based on positives
-5. engine acts
 
+###  7.2. <a name='Rules'></a>Rules
+Currently two separate Rules are validated: 
+1. Limit Adjustment Rule [here](src/observables/rules/ndcAdjustment.js)
+2. Limit Position Threshold Breach Rule [here](src/observables/rules/ndcBreach.js) 
+In the current implementation for each separate rule, an observable has to be created, like the couple above, and configured when and how to trigger it into the [setup](src/setup.js)
+The Rules outputs should be chained to common Action Agent.
+
+###  7.3. <a name='Config'></a>Config
+
+The default config is available [here](config/config.json)
+To use Environmental Variables for MongoDB URI and database name use: 
+`CEP_MONGO_URI` and `CEP_MONGO_DATABASE`
+
+##  8. <a name='LimitAdjustmentRulesflow'></a>Limit Adjustment Rules flow
+![limitAdjustment](docs/images/3.png)
+
+This rule is triggered on each limit response from the central-ledger admin API. 
+
+
+##  9. <a name='LimitPositionThresholdBreachflow'></a>Limit Position Threshold Breach flow
+![limitPositionThresholdBreach](docs/images/4.png)
+
+This rule is triggered when all data for the participants in the current transfer is received.
+
+##  10. <a name='ActionsAgentflow'></a>Actions Agent flow
+![actionAgent](docs/images/5.png)
+
+The Action Agent - [here](src/observables/actions) - takes care of action preparation regrding the data from central-ledger admin API and various settings.
+
+##  11. <a name='Notifierflowseparateservice'></a>Notifier flow (separate service)
+![notifier](docs/images/6.png)
+
+Email notifier service is a separate app, that observes the same topic for messages with field *from* = `SYSTEM`. Its code is available [here](https://github.com/mojaloop/email-notifier)
diff --git a/app.js b/app.js
@@ -21,9 +21,13 @@
  * Deon Botha <deon.botha@modusbox.com>
  --------------
  ******/
-
 'use strict'
 
 const setup = require('./src/setup').setup
+const Logger = require('@mojaloop/central-services-shared').Logger
 
-setup()
+try {
+  setup()
+} catch (err) {
+  Logger.info(`Notifier throws an error ${err}`)
+}
diff --git a/config/custom-environment-variables.json b/config/custom-environment-variables.json
@@ -0,0 +1,6 @@
+{
+  "mongo": {
+    "uri": "CEP_DATABASE_URI",
+    "database": "CEP_DATABASE_NAME"
+  }
+}
diff --git a/config/config.json → config/default.json b/config/config.json → config/default.json
@@ -3,6 +3,7 @@
     "resetPeriod": 60,
     "notificationInterval": 3
   },
+  "healthCheckPort": 3080,
   "swaggerOptions": {
     "info": {
       "title": "Central notification system API Documentation",
@@ -11,7 +12,7 @@
     "expanded": "full"
   },
   "mongo": {
-    "url": "localhost:27017",
+    "uri": "localhost:27017",
     "database": "mojaloop",
     "ruleCollection": "Rule",
     "netDebitCapPositionCollection": "NDCPosition",

diff --git a/docs/database/Mojaloop_central-notifications_Db_ver1.0.html b/docs/database/Mojaloop_central-notifications_Db_ver1.0.html
diff --git a/docs/images/1.png b/docs/images/1.png
diff --git a/docs/images/2.png b/docs/images/2.png
diff --git a/docs/images/3.png b/docs/images/3.png
diff --git a/docs/images/4.png b/docs/images/4.png
diff --git a/docs/images/5.png b/docs/images/5.png
diff --git a/docs/images/6.png b/docs/images/6.png