Notification for approaching Net Debit Cap Threshold

[elnyry-sam-k]

Goal:
As a Hub Operator
I would like to provide Notification(s) for approaching NDC Threshold that is configurable
so that participating FSPs and the Hub Operator can be informed

Tasks:

• Finalize the the CEP engine to be used [@vgenev, @bothadeon ]

• Design notification strategy for notifications when a threshold on Net Debit Cap is reached

• API to support setting threshold(s) for Notifications/Alerts

  • Investigate the best available CEP to be used for Nodejs [@vgenev, @bothadeon ]
  • Setup and testing of chosen CEP (RxJs) on local environment [@vgenev, @bothadeon ]
  • Investigate, setup and initial testing CEP observables and events [@vgenev, @bothadeon ]
  • Start implementing observables on Topic monitor and test [@vgenev, @bothadeon ]

• Scheduler

  • Investigate existing scheduler[@vgenev, @bothadeon ]
  • Implement scheduler[@vgenev, @bothadeon ]
  • Test Implemented scheduler[@vgenev, @bothadeon ]

• central-notification

  • Build a separate container and handler to be able to monitor the topics in current Kafka implementation[@vgenev, @bothadeon ]
  • Test the CEP monitor on the notifications topic for transfers[@vgenev, @bothadeon ]
  • Refine the filtering of the message and the observables when the transfer messages gets loaded onto the notification topic[@vgenev, @bothadeon ]

• Persitence store as per diagram

  • Investigate and decide on the best persitence store for the JSOn messages , Views and rules for the rules engine [@vgenev, @bothadeon ]
  • Build schemas for the JSON persistence store and test [@vgenev, @bothadeon ]
  • Test and deploy persitence store in a Docker container [@vgenev, @bothadeon ]
  • _DB design to support setting multiple alarm thresholds [@vgenev, @bothadeon ]
  • Design Mechanism to trigger a notification for breaching the threshold [@vgenev, @bothadeon ]

• Rules engine

  • Investigate and adopt JSON rules engine.[@vgenev, @bothadeon ]
  • Implement and test rules engine capabilities.[@vgenev, @bothadeon ]
  • Design and test dfsp notification rules and persitence into store.[@vgenev, @bothadeon ]
  • Implement event monititor and results from rules engine.[@vgenev, @bothadeon ]

• Central services database

  • Design participant notification preferences[@vgenev, @bothadeon ]
  • Implement the notification preference for dfsp with endpoints[@vgenev, @bothadeon ]
  • Add an endpoint to the Admin - Central Ledger API to get the current Net Debit Cap threshold value for the DFSP .[@vgenev, @bothadeon ]
  • Add an endpoint to the Admin - Central Ledger API to update the current Net Debit Cap threshold value for the DFSP .[@vgenev, @bothadeon ]

• Notifier

  • Email - primary means of notifications (for Net Debit Cap Threshold breaches)[@vgenev, @bothadeon ]
  • Finalize mechanism to notify an FSP[@vgenev, @bothadeon ]
  • Implement the Notifier(Mailer)[@vgenev, @bothadeon ]
  • Implement the mail message template builder (Mailer)[@vgenev, @bothadeon ]
  • Provision to reset the counter at a configurable time (at the start of the day? ) [alarm counter - to limit the number of times a breach on the same threshold is reported]

• On-Boarding

  • Tasks necessary to make sure that as part of on-boarding, FSPs configure an e-mail ID for notifications on threshold breaches (on Net Debit Cap .[@vgenev, @bothadeon ]

• Add JSDocs .[@vgenev, @bothadeon ]

• Automated Tests (Unit tests covered in Automated testing - unit, etc tests for email-notifier component #559 )

• Release feature

• Release feature

  • Deployment on dev environment
  • Dockerize the repo .[ @bothadeon , @vgenev ]
  • Implement the health checks endpoint for Kubernetis .[ @bothadeon , @vgenev ]
  • Develop Helm chart for deployment.[ @bothadeon , @vgenev ]
  • Kubernetes installation and deployment test.[ @bothadeon , @vgenev ]
  • AWS Deployment.[ @bothadeon , @vgenev ]

• Notification system documentation

• Deployment (Circle CI & Github Repo)

• Interim DA review on progress and decisions made. (Demo)

Acceptance Criteria:

• FSPs should be able to set a threshold on the Net Debit Cap, so that they receive a notification when it is reached.
• Based on the threshold set, a notification (or notifications) is (/are) sent to an FSP when that threshold is set
• Both FSPs and the Hub Operator receive these alerts/notifications

PRs

• [ ]

Dependencies:

• None currently

Accountability:

• Owner
• QA/Review @wdcmb @RobReeve

Notes:

• For a given threshold, during a given period - limit the number of notifications to 3. This number to be reset a the configured time.
• For the MVP, a single threshold is sufficient
• As a future extension to this, multiple thresholds need to be supported and notifications based on that.
• Review email functionality done earlier for Settlements to see if it can be re-used

[vgenev]

Notification service

stories

• #517 - Notification for changes to NetDebitCap or Position adjustments
• #518 - Notification for approaching Net Debit Cap Threshold

tasks

• receive urgent notifications and ?requests?
• sends notifications based on configs to dfsps
• receives config updates
• receives position updates on the smallest period of warnings into the configs for all dfsps (json view)
• logging? what and where?

reacts on

• notification configs changes - the data won't serve any other purpose but to config notification engines so why not JSON?
  • settings for different threshold alarms, channels of delivery etc.
• urgent notifications - kafka msg routed to the engine based on the lowest critical level stored in the central database
• positions/limits/NDC or any other type of changes in sensitive data - why not get it as a big JSON view document with all dfsps
  • get it by direct central database access / kafka-admin-topic / http request to central-ledger

data

• configs
• views
• urgent notifications

local storage and central database

• can we use mysql json data type on central database or string in mysql?
• the view can be generated by running the same query over and over again, which is optimised by sql db enginge.
• central database limits and threshold table to store only the value of critical levesl(% of NCP/value?) for urgent notifications and id/address of the latest received config or config itself
• if we decide to have local storage it can be key:value or document storage
  • we can use the view idea for reporting on later stage
  • we can deliver views for different dashboard apps for admin and troubleshooting purposes to the dfsps. HUB admin might be connected to the central database for live data
  • we can use this storage for enums/pointers or other references to the central database to offload it a bit (no updates of pointers for example, no enum queries from different functionalities)

actors

• central-ledger
• notification-router
• notification-engine

technologies

• napajs for process management into the notification services
• node-schedule for scheduling tasks

connections

central-ledger	connection	notification-router	connection	notification-engine
kafka-admin-topic	<->	config
kafka-notification-topic	<->	urgent notificaions
central-database	<-	creating dashboard view
		local storage for configs and view	<->	sends notifications based on delta
		napajs? kafka?	<->	receives and acts on urgent notifications

• central-ledger

  • API to support setting threshold(s) for Notifications/Alerts (config)
  • compare current critical level with new critical level from config and act on it
  • produce urgent notifications if critical level is reached

• notification-router

  • connected to central-ledger
    • kafka (admin topics) - config changes, json dashboard data?
    • kafka notification topic
    • local storage
    • central database access (for view or maybe view from kafka admin topic? not really time reliable)
  • connected to notification-engine for sending urgent messages (napajs/kafka/http)

• notification-engine(s)

  • spawn notification-engine per dfsp as a worker process (napajs) or another docker container (how it will communicate with router in that case)
  • connected to the local storage
    • without local storage we need connection to the router for requests to the central database or direct connection to the central database
  • runs scheduled processes - based on config, requests data received by the router with the dashboard json view and reacts on urgent notifications
    • node-schedule
  • operates different channels to reach dfsp (email/sms/etc)

View

{
  timestamp: 
  dfspsViews: [
    dfsp1: {
      curentNDC:
      position:
      ......

    },
    dfsp2: {

    }
  ]
}

basic aglorithm for view processing

1. router gets the view from central database on given period and stores it into local storage
2. enginge gets the view from local storage on its own period (might be bigger)
3. engine compares old view with the new one and gives delta
4. engine process delta and config and creates actions based on positives
5. engine acts

urgent notifications

1. central ledger creates urgent notification on kafka-notification-topic
2. router receives the notification and sends to particular engine
3. engine acts based on config (maybe not send same notificaion more than 3 times in an hour)

[elnyry-sam-k]

Thanks Valentin. This looks good to get out discussion started, possibly in an hour or so.. I think the prerequisite is to first get the e-mail/notification details and have the provisions to set those up as part of onboarding (both for Hub & FSPs)

[vgenev]

Yes, they can be prepared as an engine that act only on critical messages and be developed first together with the central-ledger notification sendings as a minimum.

…

On Thu, 25 Oct 2018 at 12:14, Sam ***@***.***> wrote: Thanks Valentin. This looks good to get out discussion started, possibly in an hour or so.. I think the prerequisite is to first get the e-mail/notification details and have the provisions to set those up as part of onboarding (both for Hub & FSPs) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#518 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLGY1Mk1_gozkH2faK7JBEEQKxDAZTSks5uoYDfgaJpZM4XtCfa> .

[vgenev]

tasks

• element

  • MVP
    • Endgame

• views

  • null
    *create view model

• config

  • create basic config for MVP (critical level settings)
    • extend config with more rules

• central-ledger

  • add API endpoint for push/put/get config
  • change database & DAO, if needed, to support one level (critical) of notifications
  • create "urgent" notifications on position adjustments based on the critical levels in config

• notification-router

  • create consumer of notification and admin topics
  • forward urgent messages to related dfsp as napajs object
    • support local storage
    • create configs into the local storage
    • get data from central database or central-ledger (via request or kafka message) for the latest adjustments and alerts periodically and store it into local storage

• notification-engine

  • receives urgent notifications from the router
  • sends notificaionts to dfsp as email
    • get configs and views from local storage
    • create schedules based on configs and latest view received
    • implement view comparison algorithm
    • add other notification channels, if necessary

[bothadeon]

Screenshot of the design path to follow as per DA meeting

[bothadeon]

Pull request for the changes to the get participant limits to return the threshold alarm percentage: mojaloop/central-ledger#206

[bothadeon]

Example of the new payload layout for http://localhost:3001/participants/{{participant}}/limits
POST /participants/testfsp/limits HTTP/1.1
{
"currency": "USD",
"limit": {
"type": "NET_DEBIT_CAP",
"value": 1000,
"thresholdAlarmPercentage": 55
}
}

[bothadeon]

1 st Pull requests after implementing Circle CI :
mojaloop/email-notifier#3
mojaloop/central-event-processor#1

[bothadeon]

Pull request for the two new endpoint types (seeds) :
mojaloop/central-ledger#209

[elnyry-sam-k]

Moving to 4.1 to complete automated tests and deployment related tasks

[elnyry-sam-k]

Increased estimate because of the tasks involving helm charts/helm, integration with email-notifier and other services. testing on the AWS deployment, etc.