启动 broker集群报错，无法连接zookeeper集群，其他客户端可以联系该zk集群

[GitForLipeng]

Describe the bug
根据官方文档步骤部署pulsar集群 ， 使用已有zookeeper集群，
启动broker集群时报错

To Reproduce
Steps to reproduce the behavior:
1：初始化元数据-正常
2：bookkeeper集群启动/测试正常
3：启动broker集群--报错
错误信息如图

Expected behavior

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Linux hejt-31.novalocal 3.10.0-1127.13.1.el7.x86_64 javascript client #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Additional context
补充一下具体的报错信息

20:21:01.364 [main-SendThread(10.21.171.58:2183)] WARN org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 20018ms for sessionid 0x0
20:21:01.466 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
20:21:01.466 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
20:21:01.466 [main] ERROR org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Failed to create zookeeper client to xx.xx.xx.xx:2183,xx.xx.xx.xx:2183,xx.xx.xx.xx:2183
//这里将真实的ip替换成了 xx
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0.jar:2.7.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:81) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:61) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.(ManagedLedgerClientFactory.java:80) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:461) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:277) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:346) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
20:21:01.471 [main] ERROR org.apache.bookkeeper.client.BookKeeper - Encountered metadata exceptions on initializing metadata client driver
org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 10.21.171.58:2183,10.21.171.59:2183,10.21.171.60:2183
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:217) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) [org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:81) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:61) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.(ManagedLedgerClientFactory.java:80) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:461) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:277) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:346) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0.jar:2.7.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
... 9 more
20:21:01.471 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to initialize metadata client driver
java.io.IOException: Failed to initialize metadata client driver
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:464) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:81) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:61) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.(ManagedLedgerClientFactory.java:80) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:461) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:277) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:346) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 10.21.171.58:2183,10.21.171.59:2183,10.21.171.60:2183
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:217) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
... 7 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0.jar:2.7.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
... 7 more
20:21:01.472 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.io.IOException: Failed to initialize metadata client driver
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:611) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:277) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:346) [org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
Caused by: java.io.IOException: Failed to initialize metadata client driver
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:464) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:81) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:61) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.(ManagedLedgerClientFactory.java:80) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:461) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
... 2 more
Caused by: org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to create zookeeper client to 10.21.171.58:2183,10.21.171.59:2183,10.21.171.60:2183
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:217) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:81) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:61) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.ManagedLedgerClientFactory.(ManagedLedgerClientFactory.java:80) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:461) ~[org.apache.pulsar-pulsar-broker-2.7.0.jar:2.7.0]
... 2 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[org.apache.pulsar-pulsar-zookeeper-2.7.0.jar:2.7.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase.waitForConnection(ZooKeeperWatcherBase.java:159) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$Builder.build(ZooKeeperClient.java:260) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.initialize(ZKMetadataDriverBase.java:197) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.meta.zk.ZKMetadataClientDriver.initialize(ZKMetadataClientDriver.java:63) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper.(BookKeeper.java:454) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:307) ~[org.apache.bookkeeper-bookkeeper-server-4.12.0.jar:4.12.0]
at org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKee

[BewareMyPower]

It's better to give your detail steps to reproduce. Also for simple, you can try to deploy a single node cluster first, i.e. a ZK cluster with one node, a BK cluster with only one node and a Pulsar cluster with one node.

Here is my script to deploy a single node cluster in localhost (see the script's comments for config changes): https://gist.github.com/BewareMyPower/b7cc6a2030d54bdea5c48a66121b32da

[codelipenghui]

Seems the problem is related to the broker can't connect to the zookeeper. It should to check the zookeeper whether available and the network between the broker and zookeeper(Firewall or others). Close this issue first since no update for a long time.

[dojiao]

I meet this issue too and i just provide a solution in my situation. Try to run netstat and check if it very slow. If it is, the problem is the DNS resolution.

[sword0307]

I meet the same issue

[kwenzh]

I meet the same issue