Fix Zookeeper persistence

[solsson]

Topic information, though not the contents kept in Kafka, would be lost if all zookeeper pods had been down at the same time. Only the snapshots were actually saved to the persistent volume.

According to https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html#sc_dataFileManagement "ZooKeeper can recover using this snapshot".

The regression probably dates back to ccb9e5d which was released with v2.0.0.

[solsson]

It looks like a change in mount path can not be kubectl applyd, which means that an upgrade can't restart pod by pod. Thus upgrades must probably try to preserve data with the new mount path. I've tested the following flow once:

kubectl replace -f zookeeper/10zookeeper-config.yml
kubectl -n kafka exec zoo-1 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl -n kafka exec zoo-0 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl delete -f zookeeper/51zoo.yml && kubectl apply -f zookeeper/51zoo.yml

kubectl -n kafka exec pzoo-2 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl -n kafka exec pzoo-1 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl -n kafka exec pzoo-0 -- bash -c 'cd /var/lib/zookeeper && mkdir data/data && mv data/myid data/version-2 data/data/ && cp -r log data/'
kubectl delete -f zookeeper/50pzoo.yml && kubectl apply -f zookeeper/50pzoo.yml

It could be a good idea to stop all kafka brokers before doing this. I found no method to stop zk in a way that didn't trigger pod restart.

[solsson]

@pavel-agarkov Care to test the above?

[pavel-agarkov]

Sure! But probably tomorrow since it is already midnight in my timezone.

[solsson]

The upgrade path would have been smoother if log dir was put inside snapshot dir, but they recommend against that in https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.html#sc_dataFileManagement. It could be that some setups add a separate volume for /var/lib/zookeeper/log instead.

Maybe the safest way is to back up /var/lib/zookeeper, add a sleep infinity at the top of init.sh and restart. Back up from inside the cluster is probably preferred, but kubectl -n kafka cp zoo-1:/var/lib/zookeeper zoo-1 is also possible. Also note that it's best to experiment with the zoo statefulset first as it has a minority of nodes.

[pavel-agarkov]

Took me a while to make it work on my single node setup.
I don't know why it worked previously but now it didn't until I changed

# maxClientCnxns changed from 1 to 2
zookeeper.properties: |-
  ...
  maxClientCnxns=2
  ...

and some other fixes probably also related to the single node setup.

But I will know for sure how it works only after a few days of natural nodes killing 😅

EDIT: looks like you have removed this line but it somehow reappeared in my fork after the merge...

[solsson]

@pavel-agarkov

maxClientCnxns=2

The line is still there. See #230. I should make the change you suggested and release again... but... Did you see any error message that was specific about hitting this limit? That's what I wanted to experience before I started raising the limit.

[pavel-agarkov]

Yes, the whole zookeeper's log was filled with:

[2018-12-03 08:39:29,911] WARN Too many connections from /10.40.21.8 - max is 1 (org.apache.zookeeper.server.NIOServerCnxnFactory)

here is what was before that warning:

[2018-12-03 08:31:33,892] INFO Established session 0x100024b03900000 with negotiated timeout 6000 for client /10.40.21.8:44568 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-12-03 08:31:43,680] INFO Got user-level KeeperException when processing sessionid:0x100024b03900000 type:setData cxid:0x41 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/config/topics/__consumer_offsets Error:KeeperErrorCode = NoNode for /config/topics/__consumer_offsets (org.apache.zookeeper.server.PrepRequestProcessor)

[solsson]

Do you know which kind of pods that came from, like 10.40.21.8 in your log output?

[pavel-agarkov]

I haven't investigated it at that time and I failed to find it in logs now.

[pavel-agarkov]

It still works well after a week of pods killing. Topics are not being lost any more.
Thank you.