Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add health state and management server for coordinators #908

Merged
merged 10 commits into from
Jul 24, 2024
Merged

Add health state and management server for coordinators #908

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d0b561e
Update README.md
kgolubic Jul 4, 2024
1f27f1f
Merge branch 'main' into memgraph-2-19
kgolubic Jul 9, 2024
442c557
add health output docs
antoniofilipovic Jul 12, 2024
33fab5e
change sentence
antoniofilipovic Jul 12, 2024
da8dd36
address PR comments
antoniofilipovic Jul 18, 2024
43ba7f6
Merge branch 'memgraph-2-18-1' into add-health-output
kgolubic Jul 18, 2024
271b386
add example of correct approach
antoniofilipovic Jul 19, 2024
8ae4189
Merge branch 'add-health-output' of github.com:memgraph/documentation…
antoniofilipovic Jul 19, 2024
c54b0f1
rewrite show instances follower part
antoniofilipovic Jul 22, 2024
d256c3a
add suggestion to use dns
antoniofilipovic Jul 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,14 +95,19 @@ an [issue](https://github.com/memgraph/documentation/issues).

### Contributing guide

If you want to change the documentation, create a new branch and make
the appropriate changes. Then, create a pull request to merge these changes into the
If you want to change the documentation, create a new branch and make the
appropriate changes. Then, create a pull request to merge these changes into the
`main` branch.

The pull request should describe the changes it's proposing and all checks must be completed.
The pull request should describe the changes it's proposing and all checks must
be completed.

Add an appropriate label to the PR, either `status: draft` if you are still working on the PR, or `status: ready` if the PR is ready for review.
Add an appropriate label to the PR, either `status: draft` if you are still
working on the PR, or `status: ready` if the PR is ready for review.

When the PR is reviewed and approved, the label will be changed to `status: ship it` and merged into the main by the repo admins.
When the PR is reviewed and approved, the label will be changed to `status: ship
it` and merged into the main by the repo admins.

If the PR requires changes, the label will be changed to `status: change`. Address the comments and change the documentation appropriately, then re-request a review and change the label to `status: ready` again.
If the PR requires changes, the label will be changed to `status: change`.
Address the comments and change the documentation appropriately, then re-request
a review and change the label to `status: ready` again.
63 changes: 29 additions & 34 deletions pages/clustering/high-availability.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ One coordinator instance is the leader whose job is always to ensure one writeab
own Raft log. Operations saved into the Raft log are those that are related to cluster management. Memgraph doesn't have its implementation of the Raft protocol.
For this task, Memgraph uses an industry-proven library [NuRaft](https://github.com/eBay/NuRaft).

You can start the coordinator instance by specifying `--coordinator-id` and
`--coordinator-port` flags. The coordinator instance only responds to
You can start the coordinator instance by specifying `--coordinator-id`,
`--coordinator-port` and `--management-port` flags. The `--management-port` is used on follower coordinators to get health state of cluster from leader coordinator. The coordinator instance only responds to
antoniofilipovic marked this conversation as resolved.
Show resolved Hide resolved
queries related to high availability, so you cannot execute any data-oriented query on it. The coordinator port is used for the Raft protocol, which
all coordinators use to ensure the consistency of the cluster's state. Data instances are distinguished from coordinator instances by
specifying `--management-port` flag. This port is used for RPC network communication between the coordinator and data
specifying only `--management-port` flag. This port is used for RPC network communication between the coordinator and data
instances. When started by default, the data instance is MAIN. The coordinator will ensure that no data inconsistency can happen during and after the instance's
restart. Once all instances are started, the user can start adding data instances to the cluster.

Expand Down Expand Up @@ -104,13 +104,15 @@ docker run --name coord1 -p 7690:7687 -p 7445:7444 memgraph/memgraph-mage
--coordinator-id=1
--experimental-enabled=high-availability
--coordinator-hostname=localhost
--management-port=10121
```

Coordinator IDs serve as identifiers, while the coordinator port is used for synchronization and log replication between coordinators. Coordinator IDs and coordinator ports must be different for all coordinators.
Coordinator IDs serve as identifiers, the coordinator port is used for synchronization and log replication between coordinators and management port is used to get health state of cluster from leader coordinator.
Coordinator IDs, coordinator ports and management ports must be different for all coordinators.

Configuration option `--coordinator-hostname` must be set on all coordinator instances. It is used only in the `SHOW INSTANCES` query to correctly return networking information about
the first leader (the coordinator instance on which you run registration queries). Setting this configuration flag to some random value will not cause issues in the cluster's capabilities
but you will get misleading output when running `SHOW INSTANCES` query.
Configuration option `--coordinator-hostname` must be set on all coordinator instances. It is used on followers to ping leader coordinator on correct IP address and return health state about
antoniofilipovic marked this conversation as resolved.
Show resolved Hide resolved
the cluster. Setting this configuration flag to some random value will cause issues in the cluster's capabilities
antoniofilipovic marked this conversation as resolved.
Show resolved Hide resolved
and you will get misleading output when running `SHOW INSTANCES` query.

### Env flags

Expand Down Expand Up @@ -150,6 +152,7 @@ export MEMGRAPH_BOLT_PORT=7692
export MEMGRAPH_HA_DURABILITY=true
export MEMGRAPH_NURAFT_LOG_FILE="<path-to-log-file>"
export MEMGRAPH_COORDINATOR_HOSTNAME="localhost"
export MEMGRAPH_MANAGEMENT_PORT=10121
```

When using any of these environment variables, flags for `--bolt-port`, `--coordinator-port`, `--coordinator-id`,`--coordinator-hostname` and `--experimental-enabled` will be ignored.
Expand Down Expand Up @@ -267,21 +270,13 @@ This operation will result in writing to the Raft log.
You can check the state of the whole cluster using the `SHOW INSTANCES` query. The query will show which instances are visible in the cluster,
which network ports they are using for managing cluster state, whether they are considered alive from the coordinator's
perspective, their role (are they MAIN, REPLICA, LEADER, FOLLOWER or unknown if not alive) and the time elapsed in ms since the last successful response
from instances to the leader's ping. This query can be run both on leader and followers with the sole difference that followers cannot know the
health status of other instances and the time of last response since only the leader is pinging instances for their status.
from instances to the leader's ping. This query can be run both on leader and followers. Followers will try to contact leader to get state of the cluster.
antoniofilipovic marked this conversation as resolved.
Show resolved Hide resolved
In case contacting leader fails, followers will return data with time elapsed in ms since the last successful response set to 0, and health state unknown.
antoniofilipovic marked this conversation as resolved.
Show resolved Hide resolved

```plaintext
SHOW INSTANCES;
```

<Callout type="warning">

When `SHOW INSTANCES` is being run, bolt server of the 1st coordinator (coordinator on which you run registration queries) will always
show IP address set to 0.0.0.0 instead of machine's IP address (container/pod when running in Docker/Kubernetes). All communication will
still work but the displayed output is wrong. This is a known bug and we are working on improving this.

</Callout>

## Setting config for highly-available cluster

There are several flags that you can use for managing the cluster. Flag `--management-port` is used for distinguishing data instances
Expand All @@ -296,7 +291,7 @@ for replication and high availability tasks.

</Callout>

Flags `--coordinator-id` and `--coordinator-port` need to be unique and specified on coordinator instances. They will cause the creation of a Raft
Flags `--coordinator-id`, `--coordinator-port` and `--management-port` need to be unique and specified on coordinator instances. They will cause the creation of a Raft
server that coordinator instances use for communication. Flag `--instance-health-check-frequency-sec` specifies how often should leader coordinator
should check the status of the replication instance to update its status. Flag `--instance-down-timeout-sec` gives the user the ability to control how much time should
pass before the coordinator starts considering the instance to be down.
Expand Down Expand Up @@ -544,8 +539,8 @@ You can directly use initialization file `HA_register.cypher`:

```

ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "coord2:7691", "coordinator_server": "coord2:10112"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "coord3:7692", "coordinator_server": "coord3:10113"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "coord2:7691", "coordinator_server": "coord2:10112", "management_server": "coord2:10122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "coord3:7692", "coordinator_server": "coord3:10113", "management_server": "coord3:10123"};

REGISTER INSTANCE instance_1 WITH CONFIG {"bolt_server": "instance1:7687", "management_server": "instance1:10011", "replication_server": "instance1:10001"};
REGISTER INSTANCE instance_2 WITH CONFIG {"bolt_server": "instance2:7688", "management_server": "instance2:10012", "replication_server": "instance2:10002"};
Expand Down Expand Up @@ -575,7 +570,7 @@ services:
- ./HA_register.cypher:/tmp/init/HA_register.cypher:ro
environment:
- MEMGRAPH_HA_CLUSTER_INIT_QUERIES=/tmp/init/HA_register.cypher
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord1", "--log-file=/tmp/coord1.log", "--also-log-to-stderr", "--coordinator-id=1", "--coordinator-port=10111", "--coordinator-hostname=coord1", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord1", "--log-file=/tmp/coord1.log", "--also-log-to-stderr", "--coordinator-id=1", "--coordinator-port=10111", "--management-port=10121", "--coordinator-hostname=coord1", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.4
Expand All @@ -585,7 +580,7 @@ services:
container_name: coord2
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord2", "--log-file=/tmp/coord2.log", "--also-log-to-stderr", "--coordinator-id=2", "--coordinator-port=10112", "--coordinator-hostname=coord2", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord2", "--log-file=/tmp/coord2.log", "--also-log-to-stderr", "--coordinator-id=2", "--coordinator-port=10112", "--management-port=10122", "--coordinator-hostname=coord2", "--experimental-enabled=high-availability"]
networks:
memgraph_ha:
ipv4_address: 172.21.0.2
Expand All @@ -595,7 +590,7 @@ services:
container_name: coord3
volumes:
- ./license.cypher:/tmp/init/license.cypher:ro
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord3", "--log-file=/tmp/coord3.log", "--also-log-to-stderr", "--coordinator-id=3", "--coordinator-port=10113", "--coordinator-hostname=coord3", "--experimental-enabled=high-availability"]
command: [ "--init-file=/tmp/init/license.cypher", "--log-level=TRACE", "--data-directory=/tmp/mg_data_coord3", "--log-file=/tmp/coord3.log", "--also-log-to-stderr", "--coordinator-id=3", "--coordinator-port=10113", "--management-port=10123", "--coordinator-hostname=coord3", "--experimental-enabled=high-availability"]

networks:
memgraph_ha:
Expand Down Expand Up @@ -645,17 +640,17 @@ This example will show how to set up a highly available cluster in Memgraph usin

1. Start coordinator1:
```plaintext
docker run --name coord1 -p 7690:7690 -p 7444:7444 memgraph/memgraph-mage --bolt-port=7690 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord1.log --also-log-to-stderr --coordinator-id=1 --coordinator-port=10111 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord1 -p 7690:7690 -p 7444:7444 memgraph/memgraph-mage --bolt-port=7690 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord1.log --also-log-to-stderr --coordinator-id=1 --coordinator-port=10111 --management-port=10121 --experimental-enabled=high-availability --coordinator-hostname=localhost
```

2. Start coordinator2:
```plaintext
docker run --name coord2 -p 7691:7691 -p 7445:7444 memgraph/memgraph-mage --bolt-port=7691 --log-level=TRACE --data-directory=/tmp/mg_data_coord2 --log-file=/tmp/coord2.log --nuraft-log-file=/tmp/nuraft/coord2.log --also-log-to-stderr --coordinator-id=2 --coordinator-port=10112 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord2 -p 7691:7691 -p 7445:7444 memgraph/memgraph-mage --bolt-port=7691 --log-level=TRACE --data-directory=/tmp/mg_data_coord2 --log-file=/tmp/coord2.log --nuraft-log-file=/tmp/nuraft/coord2.log --also-log-to-stderr --coordinator-id=2 --coordinator-port=10112 --management-port=10122 --experimental-enabled=high-availability --coordinator-hostname=localhost
```

3. Start coordinator3:
```plaintext
docker run --name coord3 -p 7692:7692 -p 7446:7444 memgraph/memgraph-mage --bolt-port=7692 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord3.log --nuraft-log-file=/tmp/nuraft/coord1.log --also-log-to-stderr --coordinator-id=3 --coordinator-port=10113 --experimental-enabled=high-availability --coordinator-hostname=localhost
docker run --name coord3 -p 7692:7692 -p 7446:7444 memgraph/memgraph-mage --bolt-port=7692 --log-level=TRACE --data-directory=/tmp/mg_data_coord3 --log-file=/tmp/coord3.log --nuraft-log-file=/tmp/nuraft/coord1.log --also-log-to-stderr --coordinator-id=3 --coordinator-port=10113 --management-port=10123 --experimental-enabled=high-availability --coordinator-hostname=localhost
```

4. Start instance1:
Expand All @@ -682,8 +677,8 @@ mgconsole --port=7690
2. Connect the other two coordinator instances to the cluster.

```plaintext
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "localhost:7691", "coordinator_server": "localhost:10112"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "localhost:7692", "coordinator_server": "localhost:10113"};
ADD COORDINATOR 2 WITH CONFIG {"bolt_server": "localhost:7691", "coordinator_server": "localhost:10112", "management_server": "localhost:10122"};
ADD COORDINATOR 3 WITH CONFIG {"bolt_server": "localhost:7692", "coordinator_server": "localhost:10113", "management_server": "localhost:10123"};
```

3. Register 3 data instances as part of the cluster:
Expand All @@ -710,9 +705,9 @@ SET INSTANCE instance_3 TO MAIN;

| name | bolt_server | coordinator_server | management_server | health | role | last_succ_resp_ms |
| ------------- | -------------- | ------------------ | ----------------- | ------ | -------- | ---------------- |
| coordinator_1 | localhost:7690 | localhost:10111 | "" | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | "" | up | follower | 16 |
| coordinator_3 | localhost:7692 | localhost:10113 | "" | up | follower | 25 |
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:10121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:10122 | up | follower | 16 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:10123 | up | follower | 25 |
| instance_1 | localhost:7687 | "" | localhost:10011 | up | replica | 39 |
| instance_2 | localhost:7688 | "" | localhost:10012 | up | replica | 21 |
| instance_3 | localhost:7689 | "" | localhost:10013 | up | main | 91 |
Expand All @@ -724,9 +719,9 @@ that and automatically promote the first alive REPLICA to become the new MAIN. T

| name | bolt_server | coordinator_server | management_server | health | role | last_succ_resp_ms |
| ------------- | -------------- | ------------------ | ----------------- | ------ | -------- | ------------------|
| coordinator_1 | localhost:7690 | localhost:10111 | "" | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | "" | up | follower | 34 |
| coordinator_3 | localhost:7692 | localhost:10113 | "" | up | follower | 28 |
| coordinator_1 | localhost:7690 | localhost:10111 | localhost:10121 | up | leader | 0 |
| coordinator_2 | localhost:7691 | localhost:10112 | localhost:10122 | up | follower | 34 |
| coordinator_3 | localhost:7692 | localhost:10113 | localhost:10123 | up | follower | 28 |
| instance_1 | localhost:7687 | "" | localhost:10011 | up | main | 61 |
| instance_2 | localhost:7688 | "" | localhost:10012 | up | replica | 74 |
| instance_3 | localhost:7689 | "" | localhost:10013 | down | unknown | 71222 |
Expand Down