Scaling out docs seem to fail

[jmwilkinson]

I am attempting to strictly follow the docs listed here:

https://clickhouse.com/docs/en/architecture/horizontal-scaling

This involves copy-pasting the configs and then updating hostnames and the cluster name, but essentially the configs are the same.

I can run the SHOW clusters; query successfully, but as soon as I run CREATE DATABASE db1 ON CLUSTER cluster_2S_1R;, the query simply hangs for three minutes then times out.

I have two questions around this:

1. The zookeeper config is necessary event when using clickhouse keeper, not zookeeper, correct? Assuming this is true, it would be nice to explicitly call this out in the docs, as the fact that the config block is labelled "zookeeper" but we are not actually using zookeeper (and want to avoid using it) is quite confusing.
2. Can someone confirm that the docs as written, if strictly followed, work? If they do not, obviously they should be updated. If they are working, I will need to troubleshoot how my implementation has gone wrong.

Additional context:

• I am using docker compose with the docker image to manage the cluster.
• I am mapping the configs into the docker containers. They load as expected.
• In general, the startup sequencing seems to be successful. If I have a config error, I can see it in the err.log file.
• I have run a number of queries showing the nodes appear to be clustered and functioning, but it is not entirely clear to me what clustered even means to ClickHouse.

[jmwilkinson]

The output of the clickhouse keeper commands is interesting (node names and general ip addresses changed).

[root@node03 ~]# echo ruok | nc localhost 9181
imok

Keeper at least thinks it is ok.

[root@node03 ~]# echo srvr | nc localhost 9181
ClickHouse Keeper version: v24.10.2.80-stable-96b80057159a167b4c697cf2676afa9319f87f21
Latency min/avg/max: 0/0/1
Received: 41998
Sent: 41999
Connections: 1
Outstanding: 0
Zxid: 0x1ec23
Mode: leader
Node count: 15

Initially that node count seemed high, but apparently that is just the total number of nodes in the data tree that zk is keeping synchronized.

[root@node03 ~]# echo stat | nc localhost 9181
ClickHouse Keeper version: v24.10.2.80-stable-96b80057159a167b4c697cf2676afa9319f87f21
Clients:
 10.10.3.1:56432(recved=0,sent=0)
 x.x.x.x:58988(recved=42001,sent=42002)

Latency min/avg/max: 0/0/1
Received: 42001
Sent: 42002
Connections: 1
Outstanding: 0
Zxid: 0x1ec2c
Mode: leader
Node count: 15

The 10.10 address is a docker internal IP. I don't really like that client list; the x.x.x.x is a valid different node, but the docker IP seems like it could be a problem.

If I run the command on each node, transforming the IP into the node name, I get this:

Node 1 (follower, container IP 10.10.5.2):

Clients:
 10.10.5.1:57724(recved=0,sent=0)
 [node3]:56968(recved=42077,sent=42078)

Node 2 (follower, container IP 10.10.4.2):

Clients:
 10.10.4.1:34042(recved=0,sent=0)
 [node1]:47790(recved=42262,sent=42263)

Node 3 (leader, container IP 10.10.3.2):

Clients:
 10.10.3.1:56432(recved=0,sent=0)
 [node2]:58988(recved=42001,sent=42002)

So every keeper instance seems to have a single valid client reference, and then another IP which is almost, but not quite, their own container IP. This all might be a red herring, but it certainly looks wrong to me.

That being said, the output from the docs is:

Clients:
 192.168.1.1:52852(recved=0,sent=0)
 192.168.1.1:52042(recved=24,sent=48)

The IP/port combos do not let me distinguish which client is associated with which node, but given the packet counts for the first client, it seems as though it may be self-referential. That would be interesting, because it means of the two other nodes, only one is connected as a client, which is the same as my setup.

...

Ok, additional research seems to indicate this is normal for zookeeper, that clients "chain" together like this.

Which means the keeper clustering appears to be functioning as expected, and it is still unclear why running ON CLUSTER command timeout.

[jmwilkinson]

The most obvious error appears in the err.log:

2024.12.02 22:21:48.138465 [ 67 ] {a50f7d26-d33a-4f91-9c0b-4b352f2b0ab2} <Error> executeQuery: Code: 159. DB::Exception: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000003 is not finished on 3 of 3 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 180.620260965 seconds, which is longer than distributed_ddl_task_timeout. (TIMEOUT_EXCEEDED) (version 24.10.2.80 (official build)) (from x.x.x.x:51184) (in query: CREATE DATABASE db1 ON CLUSTER test_cluster), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cf7c73b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007ea788c
2. DB::DDLQueryStatusSource::generate() @ 0x0000000011720595
3. DB::ISource::tryGenerate() @ 0x00000000129a8b1b
4. DB::ISource::work() @ 0x00000000129a8827
5. DB::ExecutionThreadContext::executeTask() @ 0x00000000129c2fa7
6. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000129b6550
7. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x00000000129b5824
8. DB::CompletedPipelineExecutor::execute() @ 0x00000000129b40d2
9. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&, String const&, std::shared_ptr<DB::Context const> const&, std::optional<DB::FormatSettings> const&)>) @ 0x000000001173aa68
10. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128a6f23
11. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128aae85
12. DB::HTTPServerConnection::run() @ 0x000000001294aa1d
13. Poco::Net::TCPServerConnection::start() @ 0x000000001580b827
14. Poco::Net::TCPServerDispatcher::run() @ 0x000000001580bcb9
15. Poco::PooledThread::run() @ 0x00000000157d8821
16. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000157d6ddd
17. ? @ 0x00007efc59529609
18. ? @ 0x00007efc5944e353

I am able to see the task in the ddl queue, but I have no idea why there are no workers executing it.