Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

204 monitoring infrastructure #225

Merged
merged 3 commits into from
Sep 10, 2019
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions rest-api/monitoring/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Monitoring a live deployment of Hedera Mirror Node

This code runs on an external server outside of the Hedera Beta MirrorNode deployment, and periodically polls the REST APIs exposed by the Hedera mirror node to ensure that the deployed APIs are working.
It also provides a simple dashboard to monitor the status.

## Overview

Hedera mirror nodes REST APIs expose /transactions, /balances and /accounts endpoints.
To monitor a live deployment of a Hedera mirror node, this code consists of monitoring APIs and monitoring dashboard as described below.

#### Monitoring APIs:
A process runs that periodically polls the APIs exposed by the deployment of a Hedera mirror node.
It then checks the responses using a few simple checks for /transactions, /balances and /accounts APIs.
The results of these checks are exposed as a set of REST APIs exposed by this monitoring service as follows:

| API | HTTP return code | Description |
|-----| -----------------| ------------|
|/api/v1/status | 200(OK) | Provides a list of results of all tests run on all servers |
|/api/v1/status/{id} | 200 (OK) | If all tests pass for a server, then it returns the results |
| | 4xx | If any tests fail for a server, or if the server is not running, then it returns a 4xx error code to make it easy to integrate with alerting systems |


#### Monitoring dashboard:

A dashboard polls the above-mentioned APIs and displays the results.

----

## Quickstart

### Requirements

- [ ] List of addresses of Hedera mirror nodes that you want to monitor
- [ ] An external server where you want to run this code to monitor the mirror node. You will need two TCP ports on the server.
- [ ] npm and pm2


```
git clone git@github.com:hashgraph/hedera-mirror-node.git
cd hedera-mirror-node/rest-apis/monitoring
```

To run the monitor_apis backend:
```
cd monitor_apis
cp config/sample.serverlist.json config/serverlist.json // Start with the sample configuration file
nano config/serverlist.json // Insert the mirror node deployments you want to monitor
npm install // Install npm dependencies
PORT=xxxx npm start // To start the monitoring server on port xxxx (Note: please enter a number for xxxx)
```
The server will start polling Hedera mirror nodes specified in the config/serverlist.json file.
The default timeout to populate the data is 2 minutes. After 2 minutes, you can verify the output using `curl <ip-address-where-you-run-monitoring-service>:<port>/api/v1/status` command.


To run the dashboard (from hedera-mirror-node/rest-apis/monitoring directory):
```
cd monitor_dashboard
nano js/main.js // Change the server: 'localhost:3000' line to point to the ip-address/name and port of the server where you are running the monitoring backed as described in the above tests.
pm2 serve . yyyy // Serve the dashboard html pages on another port...(Note: please enter a number for yyyy)
```

Using your browser, connect to `http:<ip-address-where-you-run-monitoring-service>:<yyyy>/index.html`

----

## Contributing

Refer to [CONTRIBUTING.md](CONTRIBUTING.md)

## License

Apache License 2.0, see [LICENSE](LICENSE).

2 changes: 2 additions & 0 deletions rest-api/monitoring/monitor_apis/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
serverlist.json
90 changes: 90 additions & 0 deletions rest-api/monitoring/monitor_apis/accounts.monitor.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
/*-
* ‌
* Hedera Mirror Node
* ​
* Copyright (C) 2019 Hedera Hashgraph, LLC
* ​
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
* ‍
*/

'use strict';

const fetch = require("node-fetch");
const common = require('./common.js');

let testAccounts;

/**
* Base url for the accounts API
* @param {Object} server The server to run the test against
* @return {String} Base query string for Hedera mirror node REST API
*/
const getBaseUrl = (server) => {
return (`http://${server.ip}:${server.port}/api/v1/accounts`);
}

/**
* Executes the /accounts API with no parameters
* Expects the response to have 1000 entries, and timestamp in
* the FRESHNESS_EXPECTATION minutes
* @param {Object} server The server to run the test against
* @return {} None. updates testAccounts variable
*/
const getAccountsNoParams = async (server) => {
const FRESHNESS_EXPECTATION = 20; // minutes
mike-burrage-hedera marked this conversation as resolved.
Show resolved Hide resolved
const url = getBaseUrl(server);
const response = await fetch(url);
const data = await response.json();

common.logResult (server, url, 'getAccountsNoParams',
(data.accounts.length === 1000) ?
{result: true, msg: 'Received 1000 accounts'} :
{result: false, msg: 'Received less than 1000 accounts'});

const txSec = data.accounts[0].balance.timestamp.split('.')[0];
const currSec = Math.floor(new Date().getTime() / 1000);
mike-burrage-hedera marked this conversation as resolved.
Show resolved Hide resolved
const delta = currSec - txSec;

common.logResult (server, url, 'getAccountsNoParams',
(delta < (60 * FRESHNESS_EXPECTATION)) ?
{result: true, msg: `Freshness: Received accounts from ${delta} seconds ago`} :
{result: false, msg: `Freshness: Got stale accounts from ${delta} seconds ago`}
);

testAccounts = data.accounts;
}

/**
* Executes the /accounts API for querying one single account
* Expects the account id to match the requested account id
* @param {Object} server The server to run the test against
* @return {} None.
*/
const getOneAccount = async (server) => {
const accId = testAccounts[0].account;
const url = getBaseUrl(server) + '/' + accId;
const response = await fetch(url);
const data = await response.json();

common.logResult (server, url, 'getOneAccount',
(data.account === accId) ?
{result: true, msg: 'Received correct account'} :
{result: false, msg: 'Did not receive correct account'});
}

module.exports = {
testAccounts: testAccounts,
getAccountsNoParams: getAccountsNoParams,
getOneAccount: getOneAccount
}
120 changes: 120 additions & 0 deletions rest-api/monitoring/monitor_apis/balances.monitor.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
/*-
* ‌
* Hedera Mirror Node
* ​
* Copyright (C) 2019 Hedera Hashgraph, LLC
* ​
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
* ‍
*/

'use strict';

const fetch = require("node-fetch");
const common = require('./common.js');

let testBalances;
let testBalancesTimestamp;

/**
* Base url for the balances API
* @param {Object} server The server to run the test against
* @return {String} Base query string for Hedera mirror node REST API
*/
const getBaseUrl = (server) => {
return (`http://${server.ip}:${server.port}/api/v1/balances`);
}

/**
* Executes the /balances API with no parameters
* Expects the response to have 1000 entries, and timestamp in
* the FRESHNESS_EXPECTATION minutes
* @param {Object} server The server to run the test against
* @return {} None. updates testBalances variable
*/
const getBalancesNoParams = async (server) => {
const FRESHNESS_EXPECTATION = 20; // minutes
const url = getBaseUrl(server);
const response = await fetch(url);
const data = await response.json();

common.logResult (server, url, 'getBalancesNoParams',
(data.balances.length === 1000) ?
{result: true, msg: 'Received 1000 balances'} :
{result: false, msg: 'Received less than 1000 balances'});

const balancesSec = data.timestamp.split('.')[0];
const currSec = Math.floor(new Date().getTime() / 1000);
const delta = currSec - balancesSec;

common.logResult (server, url, 'getBalancesNoParams',
(delta < (60 * FRESHNESS_EXPECTATION)) ?
{result: true, msg: `Freshness: Received balances from ${delta} seconds ago`} :
{result: false, msg: `Freshness: Got stale balances from ${delta} seconds ago`}
);

testBalances = data.balances;
testBalancesTimestamp = data.timestamp;
}

/**
* Executes the /balances API with timestamp filter
* Expects the response to have 1000 entries, and to have a timestamp
* that is within BALANCE_TS_QUERY_TOLERANCE minutes
* @param {Object} server The server to run the test against
* @return {} None. updates testBalances variable
*/
const checkBalancesWithTimestamp = async (server) => {
const BALANCE_TS_QUERY_TOLERANCE = 30; // minutes
const url = getBaseUrl(server) + '?timestamp=lt:' + testBalancesTimestamp;
const response = await fetch(url);
const data = await response.json();

common.logResult (server, url, 'checkBalancesWithTimestamp',
(data.balances.length === 1000) ?
{result: true, msg: 'Received 1000 balances'} :
{result: false, msg: 'Received less than 1000 balances'});

common.logResult (server, url, 'checkBalancesWithTimestamp',
((data.timestamp < testBalancesTimestamp) &&
((testBalancesTimestamp - data.timestamp) < (60 * BALANCE_TS_QUERY_TOLERANCE)))
?
{result: true, msg: 'Received older balances correctly'} :
{result: false, msg: 'Did not receive older balances correctly'});
}

/**
* Executes the /balances API for querying one single account
* Expects the account id to match the requested account id
* @param {Object} server The server to run the test against
* @return {} None.
*/
const getOneBalance = async (server) => {
const accId = testBalances[0].account;
const url = getBaseUrl(server) + '/?account.id=' + accId;
const response = await fetch(url);
const data = await response.json();

common.logResult (server, url, 'getOneBalance',
((data.balances.length === 1) &&
(data.balances[0].account === accId)) ?
{result: true, msg: 'Received correct account balance'} :
{result: false, msg: 'Did not receive correct account balance'});
}

module.exports = {
testBalances: testBalances,
getBalancesNoParams: getBalancesNoParams,
checkBalancesWithTimestamp: checkBalancesWithTimestamp,
getOneBalance: getOneBalance
}
Loading