Problem adding peers to the existing cluster

[Xswer]

What was made:
At the moment we have an existing NATS cluster, consisting of 3 Peers. Current requirement is to add 2 new Peers to the cluster. To do that we have deployed 2 new Peers with cluster.routes pointing to the existing cluster peers + new peer. After that the existing cluster was redeployed with the new configuration (but store was persisted). Now each peer has a route to all the other peers in cluster.

Current state:
All the peers connect to each other on port 6222. I can see the successful connections in logs. But then nothing happens - two new peers are ignored - one of them gets stuck in "Candidate" status, as if it was trying to elect itself as Leader.

Our config for the existing cluster - please note, that the config varies, depending on particular instance (routes and peers)
`
port: 4222
http_port: 8222
debug: true
cluster {
listen: 0.0.0.0:6222
routes: [
"nats://nats-b.nats-test-old-cluster:6222",
"nats://nats-c.nats-test-old-cluster:6222",
"nats://nats-d.nats-test-cluster:6222",
"nats://nats-e.nats-test-cluster:6222",
]
}

streaming {
id: testOldCluster
store: file
dir: /mnt/natsdata/a/store
sd: true
cluster {
node_id: "a"
peers: ["b", "c", "d", "e"]
log_path: /mnt/natsdata/a/log
}
store_limits: {
max_channels: 1024
max_msgs: 0
max_bytes: 0
max_age: "336h"
}
}
`

Our config for two new peers:
`
port: 4222
http_port: 8222
cluster {
listen: 0.0.0.0:6222
routes: [
"nats://nats-e.nats-test-cluster:6222",
"nats://nats-a.nats-test-old-cluster:6222",
"nats://nats-b.nats-test-old-cluster:6222",
"nats://nats-c.nats-test-old-cluster:6222",
]
}

streaming {
id: testOldCluster
store: file
sd: false
cluster {
node_id: "d"
peers: ["e","a","b","c"]
log_path: /mnt/efs/d/log
}
dir: /mnt/efs/d/store
store_limits: {
max_channels: 1024
max_msgs: 0
max_bytes: 0
max_age: "336h"
}
}
`

Problem:
The new peers are ignored - no sync happens between the cluster and new peers.

[kozlovic]

@Xswer You cannot grow/shrink a cluster just by updating the configuration files. The RAFT cluster information is stored in the RAFT log itself. Only starting v0.19.0 we added a feature to add/remove nodes from the cluster. Check PR: #1090 for more information.
Basically you would need to enable the feature with a configuration option and then send NATS messages to add/remove nodes.
Of course, you should still update your configuration file with the new peers, etc.., but the point is that this is not enough.

[Xswer]

@kozlovic thank you for the fast response. Your explanation has helped a lot, peers are now in sync.

[GhOsTMZ]

@kozlovic means we have no way to grow/shrink a cluster for version 0.17.0? Or we have another way of doing that?

[kozlovic]

@GhOsTMZ Ability to "administratively" add/remove peers from a cluster was indeed added only at v0.19.0, so you can't do that with 0.17.0