org.apache.kafka.clients.NetworkClient Error connecting to node broker:29092 (id: -1 rack: null) java.net.UnknownHostException: broker: Name or service not known

[dash7ou]

control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
rest-proxy | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
rest-proxy | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501070911, tries=1, nextAllowedTryMs=1606501071012) timed out at 1606501070912 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
ksqldb-server | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
ksqldb-server | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501071531, tries=1, nextAllowedTryMs=1606501071632) timed out at 1606501071532 after 1 attempt(s)
ksqldb-server | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
schema-registry | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
schema-registry | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501072795, tries=1, nextAllowedTryMs=1606501072897) timed out at 1606501072797 after 1 attempt(s)
schema-registry | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker: Name or service not known
control-center | at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
control-center | at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
control-center | at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
control-center | at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
control-center | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501079587, tries=1, nextAllowedTryMs=1606501079698) timed out at 1606501079598 after 1 attempt(s)
control-center | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
rest-proxy | [main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
rest-proxy | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
rest-proxy | at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
rest-proxy | at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)

[dash7ou]

@ybyzek I have an issue when running docker-compose file for cp-all-in-one folder, on ec2 amazon, it runs fine for me locally.

[cosbor11]

This happens for me too

[konfortes]

Same here for local run.
osx 10.15.5
Docker 3.0.3(51017)

[wscardua]

me too

osx 10.15.7
docker 3.0.4 (51218)

[ybyzek]

Things to check:

1. Is the broker up? (docker-compose ps)
2. Can the other container ping the broker container? (docker-compose exec control-center ping broker)
3. Are there any errors in the logs (search through docker-compose logs)
4. Which branch of this repo is checked out? Make sure to run latest (latest or 6.0.1-post)

[mhbrk]

docker-compose logs -f broker

Attaching to broker
broker | ===> User
broker | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
broker | ===> Configuring ...
broker | ===> Running preflight checks ...
broker | ===> Check if /var/lib/kafka/data is writable ...
broker | ===> Check if Zookeeper is healthy ...
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=broker
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=11.0.9.1
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu11-ca
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/share/java/cp-base-new/paranamer-2.8.jar:/usr/share/java/cp-base-new/netty-resolver-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-library-2.13.2.jar:/usr/share/java/cp-base-new/metrics-core-2.2.0.jar:/usr/share/java/cp-base-new/audience-annotations-0.5.0.jar:/usr/share/java/cp-base-new/kafka-clients-6.0.1-ccs.jar:/usr/share/java/cp-base-new/commons-cli-1.4.jar:/usr/share/java/cp-base-new/slf4j-api-1.7.30.jar:/usr/share/java/cp-base-new/jopt-simple-5.0.4.jar:/usr/share/java/cp-base-new/jackson-databind-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-module-scala_2.13-2.10.5.jar:/usr/share/java/cp-base-new/netty-transport-native-unix-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-logging_2.13-3.9.2.jar:/usr/share/java/cp-base-new/jackson-dataformat-yaml-2.10.5.jar:/usr/share/java/cp-base-new/argparse4j-0.7.0.jar:/usr/share/java/cp-base-new/gson-2.8.6.jar:/usr/share/java/cp-base-new/scala-collection-compat_2.13-2.1.6.jar:/usr/share/java/cp-base-new/jackson-datatype-jdk8-2.10.5.jar:/usr/share/java/cp-base-new/jackson-annotations-2.10.5.jar:/usr/share/java/cp-base-new/utility-belt-6.0.1.jar:/usr/share/java/cp-base-new/netty-buffer-4.1.48.Final.jar:/usr/share/java/cp-base-new/common-utils-6.0.1.jar:/usr/share/java/cp-base-new/scala-java8-compat_2.13-0.9.1.jar:/usr/share/java/cp-base-new/zookeeper-jute-3.5.8.jar:/usr/share/java/cp-base-new/zookeeper-3.5.8.jar:/usr/share/java/cp-base-new/kafka_2.13-6.0.1-ccs.jar:/usr/share/java/cp-base-new/zstd-jni-1.4.4-7.jar:/usr/share/java/cp-base-new/jackson-core-2.10.5.jar:/usr/share/java/cp-base-new/lz4-java-1.7.1.jar:/usr/share/java/cp-base-new/netty-handler-4.1.48.Final.jar:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar:/usr/share/java/cp-base-new/jackson-module-paranamer-2.10.5.jar:/usr/share/java/cp-base-new/snappy-java-1.1.7.3.jar:/usr/share/java/cp-base-new/netty-common-4.1.48.Final.jar:/usr/share/java/cp-base-new/jackson-dataformat-csv-2.10.5.jar:/usr/share/java/cp-base-new/snakeyaml-1.26.jar:/usr/share/java/cp-base-new/netty-transport-native-epoll-4.1.48.Final.jar:/usr/share/java/cp-base-new/netty-codec-4.1.48.Final.jar:/usr/share/java/cp-base-new/scala-reflect-2.13.2.jar
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.19.121-linuxkit
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/appuser
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=120MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=1992MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=126MB
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@a7e666
broker | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
broker | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
broker | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/172.24.0.2:2181: Connection refused
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.24.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.24.0.3:59678, server: zookeeper/172.24.0.2:2181
broker | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.24.0.2:2181, sessionid = 0x1000024ccb10000, negotiated timeout = 40000
broker | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1000024ccb10000 closed
broker | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1000024ccb10000
broker | ===> Launching ...
broker | ===> Launching kafka ...
broker | [2021-01-17 21:17:02,315] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
broker | [2021-01-17 21:17:03,638] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
broker | [2021-01-17 21:17:03,679] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092
broker | at kafka.utils.CoreUtils$.validate$1(CoreUtils.scala:260)
broker | at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:271)
broker | at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:2297)
broker | at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2376)
broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:2354)
broker | at kafka.server.KafkaConfig.(KafkaConfig.scala:1807)
broker | at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
broker | at kafka.Kafka$.main(Kafka.scala:68)
broker | at kafka.Kafka.main(Kafka.scala)
broker exited with code 1

[mhbrk]

also it doesn't have - "29092:29092" in this part: https://github.com/confluentinc/cp-all-in-one/blob/6.0.1-post/cp-all-in-one/docker-compose.yml#L20

[ybyzek]

@mhbrk from the error logs:

broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092

It looks like 9092 is reused twice, this is not allowed as described in the message. So it appears your local docker-compose.yml file has been modified. Please update it to match what is in GitHub (which is validated to work) and try again.

[DavoBR]

about the port 29092, i am facing with the following error

container console output:

confluentinc_cp-server_6.1.0.txt

[OneCricketeer]

also it doesn't have - "29092:29092" in this part

broker hostname isn't resolvable by the host, anyway, so a port forward is not the correct location to debug the problem

As the logs say, your listeners config cannot use the same port. Without this, your broker dies as well as all other services, such as control center, that depend on it

You may explicitly define KAFKA_LISTENERS, otherwise the default behavior in the Docker image is to use the advertised listeners, and replace the host names with 0.0.0.0

These get printed in the logs when the container starts https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/cub.py#L469

[tooptoop4]

@dash7ou did u solve?

[nicolepastrana]

This is what worked for me in my M1 Chip:

• zookeeper image: confluentinc/cp-zookeeper:latest.arm64
• broker image: confluentinc/cp-server:latest.arm64
• schema-registry image: confluentinc/cp-schema-registry:latest.arm64
• connect image: cnfldemos/cp-server-connect-datagen:0.5.0-6.2.0 [I haven't tested with the latest cnfldemos/cp-server-connect-datagen:0.5.3-7.1.0]
• control-center image: confluentinc/cp-enterprise-control-center:latest.arm64
• ksqldb-server image: confluentinc/cp-ksqldb-server:latest.arm64
• ksqldb-cli image: confluentinc/cp-ksqldb-cli:latest.arm64
• ksql-datagen image: confluentinc/ksqldb-examples:latest.arm64
• rest-proxy image: confluentinc/cp-kafka-rest:latest.arm64

https://github.com/confluentinc/cp-all-in-one/blob/7.1.1-post/cp-all-in-one/docker-compose.yml