Skip to content

Commit

Permalink
Merge pull request #862 from mDuo13/health_check_method
Browse files Browse the repository at this point in the history
Health Check method
  • Loading branch information
mDuo13 authored Jul 13, 2020
2 parents 18b0cb8 + 08782a3 commit cbf104f
Show file tree
Hide file tree
Showing 11 changed files with 322 additions and 17 deletions.
2 changes: 2 additions & 0 deletions content/concepts/consensus-network/amendments/amendments.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ The amendments that a `rippled` server is configured to vote for or against have

If your server is amendment blocked, you must [upgrade to a new version](install-rippled.html) to sync with the network.

It is also possible to be amendment blocked because you connected your server to a [parallel network](parallel-networks.html) that has different amendments enabled. For example, the XRP Ledger Devnet typically has upcoming and experimental amendments enabled. If you are using the latest production release, your server is likely to be amendment blocked when connecting to Devnet. You could resolve this issue by upgrading to an unstable pre-release or nightly build, or you could [connect to a different network such as Testnet](connect-your-rippled-to-the-xrp-test-net.html) instead.


#### How to Tell If Your `rippled` Server Is Amendment Blocked

Expand Down
2 changes: 1 addition & 1 deletion content/concepts/the-rippled-server/peer-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ ip = 0.0.0.0
protocol = peer
```

The peer protocol port also serves the [special Peer Crawler API method](peer-crawler.html).
The peer protocol port also serves [special peer port methods](peer-port-methods.html).

## Node Key Pair

Expand Down
109 changes: 109 additions & 0 deletions content/references/rippled-api/peer-port-methods/health-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Health Check
[[Source]](https://github.com/ripple/rippled/blob/de0c52738785de8bf837f9124da65c7905e7bb5a/src/ripple/overlay/impl/OverlayImpl.cpp#L1084-L1168 "Source")

The Health Check is a special [peer port method](peer-port-methods.html) for reporting on the health of an individual `rippled` server. This method is intended for use in automated monitoring to recognize outages and prompt automated or manual interventions such as restarting the server. [New in: rippled 1.6.0][]

This method checks several metrics to see if they are in ranges generally considered healthy. If all metrics are in normal ranges, this method reports that the server is healthy. If any metric is outside normal ranges, this method reports that the server is unhealthy and reports the metric(s) that are unhealthy. Since some metrics may rapidly fluctuate into and out of unhealthy ranges, you should not raise alerts unless the health check fails multiple times in a row.

**Note:** Since the health check is a [peer port method](peer-port-methods.html), it is not available when testing the server in [stand-alone mode](rippled-server-modes.html#reasons-to-run-a-rippled-server-in-stand-alone-mode).


## Request Format

To request the Health Check information, make the following HTTP request:

- **Protocol:** https
- **HTTP Method:** GET
- **Host:** (any `rippled` server, by hostname or IP address)
- **Port:** (the port number where the `rippled` server uses the Peer Protocol, typically 51235)
- **Path:** `/health`
- **Security:** Most `rippled` servers use a self-signed certificate to respond to the request. By default, most tools (including web browsers) flag or block such responses for being untrusted. You must ignore the certificate checking (for example, if using cURL, add the `--insecure` flag) to display a response from those servers.

<!-- TODO: link a tutorial for how to run rippled with a non-self-signed TLS cert -->

## Example Response

<!-- MULTICODE_BLOCK_START -->

*Healthy*

```json
HTTP/1.1 200 OK
Server: rippled-1.6.0-b8
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked

{
"info": {}
}
```

*Warning*

```json
HTTP/1.1 503 Service Unavailable
Server: rippled-1.6.0
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked

{
"info": {
"server_state": "connected",
"validated_ledger": -1
}
}
```

*Critical*

```json
HTTP/1.1 500 Internal Server Error
Server: rippled-1.6.0
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked

{
"info": {
"peers": 0,
"server_state": "disconnected",
"validated_ledger":-1
}
}
```

<!-- MULTICODE_BLOCK_END -->

## Response Format

The response's HTTP status code indicates the health of the server:

| Status Code | Health Status | Description |
|:------------------------------|:--------------|:-----------------------------|
| **200 OK** | Healthy | All health metrics are within acceptable ranges. |
| **503 Service Unavailable** | Warning | One or more metric is in the warning range. Manual intervention may or may not be necessary. |
| **500 Internal Server Error** | Critical | One or more metric is in the critical range. There is a serious problem that probably needs manual intervention to fix. |

The response body is a JSON object with a single `info` object at the top level. The `info` object contains values for each metric that is in a warning or critical range. The response omits metrics that are in a healthy range, so a fully healthy server has an empty object.

The `info` object may contain the following fields:

| `Field` | Value | Description |
|:--------------------|:--------|:---------------------------------------------|
| `amendment_blocked` | Boolean | _(May be omitted)_ If `true`, the server is [amendment blocked](amendments.html#amendment-blocked) and must be upgraded to remain synced with the network; this state is critical. If the server is not amendment blocked, this field is omitted. |
| `load_factor` | Number | _(May be omitted)_ A measure of the overall load the server is under. This reflects I/O, CPU, and memory limitations. This is a warning if the load factor is over 100, or critical if the load factor is 1000 or higher. |
| `peers` | Number | _(May be omitted)_ The number of [peer servers](peer-protocol.html) this server is connected to. This is a warning if connected to 7 or fewer peers, and critical if connected to zero peers. |
| `server_state` | String | _(May be omitted)_ The current [server state](rippled-server-states.html). This is a warning if the server is in the `tracking`, `syncing`, or `connected` states. This is critical if the server is in the `disconnected` state. |
| `validated_ledger` | Number | _(May be omitted)_ The number of seconds since the last time a ledger was validated by [consensus](intro-to-consensus.html). If there is no validated ledger available ([as during the initial sync period when starting the server](server-doesnt-sync.html#normal-syncing-behavior)), this is the value `-1` and is considered a warning. This metric is also a warning if the last validated ledger was at least 7 seconds ago, or critical if the last validated ledger was at least 20 seconds ago. |

## See Also

For guidance interpreting the results of the health check, see [Health Check Interventions](health-check-interventions.html).


<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Peer Crawler

The Peer Crawler is a special API endpoint for reporting on the health and topology of the peer-to-peer network. This API method is available by default on a non-privileged basis through the [Peer Protocol](peer-protocol.html) port, which is also used for `rippled` servers' peer-to-peer communications about consensus, ledger history, and other necessary information.
The Peer Crawler is a special [peer port method](peer-port-methods.html) for reporting on the health and topology of the peer-to-peer network. This API method is available by default on a non-privileged basis through the [Peer Protocol](peer-protocol.html) port, which is also used for `rippled` servers' peer-to-peer communications about consensus, ledger history, and other necessary information.

The information reported by the peer crawler is effectively public, and can be used to report on the overall XRP Ledger network, its health, and topology.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Connect Your rippled to an XRPL Altnet
# Connect Your rippled to a Parallel Network

Ripple has created [alternative test and development networks](parallel-networks.html) for developers to test their apps on the latest non-production version of the XRP Ledger (Testnet) or to test and experiment with features on the latest beta version (Devnet). **The funds used on these networks are not real funds and are intended for testing only.** You can connect your [`rippled` server](the-rippled-server.html) to either the Testnet or Devnet.

**Note:** The XRP Testnet and Devnet ledger and balances are reset on a regular basis.
**Caution:** The Devnet frequently has new and experimental [amendments](amendments.html) enabled, so the latest production release version is likely to be amendment blocked when connecting to Devnet. You should use a pre-release or nightly build when connecting to Devnet.

To connect your `rippled` server to the XRP Testnet or Devnet, set the following configurations:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,17 +53,21 @@ For development purposes, run `rippled` as a non-admin user, not using `sudo`.
$ git clone git@github.com:ripple/rippled.git
$ cd rippled

0. By default, cloning puts you on the `develop` branch. Use this branch if you are doing development work and want the latest set of untested features.
0. Switch to the appropriate branch for the software version you want:

If you want the latest stable release, checkout the `master` branch.
For the latest stable release, use the `master` branch.

$ git checkout master

If you want to test out the latest release candidate, checkout the `release` branch:
For the latest release candidate, use the `release` branch:

$ git checkout release

Or, you can checkout one of the tagged releases listed on [GitHub](https://github.com/ripple/rippled/releases).
For the latest in-progress version, use the `develop` branch:

$ git checkout develop

Or, you can checkout one of the tagged releases listed on [GitHub](https://github.com/ripple/rippled/releases).

0. In the `rippled` directory you just cloned, create your build directory and access it. For example:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -177,17 +177,19 @@ Memory requirements are mainly a function of the `node_size` configuration setti

#### Network

Any enterprise or carrier-class data center should have substantial network bandwidth to support running `rippled` servers.
Any enterprise or carrier-class data center should have substantial network bandwidth to support running `rippled` servers. The actual bandwidth necessary varies significantly based on the current transaction volume in the network. Server behavior (such as backfilling [ledger history](ledger-history.html)) also affects network use.

During exceptionally high periods of transaction volume, some operators have reported that their `rippled` servers have completely saturated a 100MBit/s network link. Therefore, a gigabit network interface is required for reliable performance.

Here are examples of observed network bandwidth use for common `rippled` tasks:

| Task | Transmit/Receive |
|:------------------------------------------------|:---------------------------|
| Process current transaction volumes | 2Mbps transmit, 2 Mbps receive |
| Process average transaction volumes | 2Mbps transmit, 2 Mbps receive |
| Process peak transaction volumes | >100Mbps transmit |
| Serve historical ledger and transaction reports | 100Mbps transmit |
| Start up `rippled` | 20Mbps receive |


## See Also

- **Concepts:**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,16 @@ Before you install `rippled`, you must meet the [System Requirements](system-req

1. Install the Ripple RPM repository:

Choose the appropriate RPM repository for the stability of releases you want:

- `stable` for the latest production release (`master` branch)
- `unstable` for pre-release builds (`release` branch)
- `nightly` for experimental/development builds (`develop` branch)

<!-- MULTICODE_BLOCK_START -->

*Stable*

$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-stable]
name=XRP Ledger Packages
Expand All @@ -24,6 +34,33 @@ Before you install `rippled`, you must meet the [System Requirements](system-req
repo_gpgcheck=1
REPOFILE

*Pre-release*

$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-unstable]
name=XRP Ledger Packages
baseurl=https://repos.ripple.com/repos/rippled-rpm/unstable/
enabled=1
gpgcheck=0
gpgkey=https://repos.ripple.com/repos/rippled-rpm/unstable/repodata/repomd.xml.key
repo_gpgcheck=1
REPOFILE

*Development*

$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-nightly]
name=XRP Ledger Packages
baseurl=https://repos.ripple.com/repos/rippled-rpm/nightly/
enabled=1
gpgcheck=0
gpgkey=https://repos.ripple.com/repos/rippled-rpm/nightly/repodata/repomd.xml.key
repo_gpgcheck=1
REPOFILE

<!-- MULTICODE_BLOCK_START -->


2. Fetch the latest repo updates:

$ sudo yum -y update
Expand Down
Loading

0 comments on commit cbf104f

Please sign in to comment.