CMAK 에서 Add cluster 시 KeeperErrorCode = Unimplemented for /kafka-manager/mutex 에러 발생하는 경우

[occidere]

CMAK 에서 Cluster 생성 시 KeeperErrorCode = Unimplemented for /kafka-manager/mutex 에러 발생하는 경우

상황

• CMAK (kafka manager) 에서 Add cluster 를 했을 때 KeeperErrorCode = Unimplemented for /kafka-manager/mutex 에러가 발생
• 

로그

2021-08-06 14:33:11,283 - [ERROR] - from kafka.manager.KafkaManager in pool-1-thread-2
Failed on input : KMAddCluster(ClusterConfig(test,CuratorConfig(localhost:2181,10,100,1000),true,2.4.0,false,None,None,false,true,false,false,true,true,Some(ClusterTuning(Some(30),Some(2),Some(100),Some(2),Some(100),Some(2),Some(100),Some(30),Some(5),Some(16),Some(1000),Some(16),Some(1000),Some(16),Some(1000),Some(30000),Some(1000000),Some(7))),PLAINTEXT,None,None))
org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /kafka-manager/mutex
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:746)
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
        at org.apache.curator.framework.recipes.locks.StandardLockInternalsDriver.createsTheLock(StandardLockInternalsDriver.java:54)
        at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:217)
        at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:232)
        at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:108)
        at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.internalAcquire1Lease(InterProcessSemaphoreV2.java:344)
        at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:284)
        at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:246)
        at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex.acquire(InterProcessSemaphoreMutex.java:52)
        at kafka.manager.actor.KafkaManagerActor.$anonfun$modify$2(KafkaManagerActor.scala:150)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
        at scala.util.Success.$anonfun$map$1(Try.scala:255)
        at scala.util.Success.map(Try.scala:213)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

원인

• 이 코멘트 에 의하면 현재 kafka가 cmak 에선 지원하지 않는 zookeeper 를 사용하고 있기 때문

해결

zookeeper 에 접속한 뒤 ./bin/zkCli.sh 로 주키퍼 클라이언트 실행 후 아래와 같이 /kafka-manager 경로에 3개 폴더 생성

# docker exec -it my-zookeeper /bin/bash
root@ec5ea8a1e52b:/opt/zookeeper-3.4.13# ./bin/zkCli.sh
...
[zk: localhost:2181(CONNECTED) 0] ls /kafka-manager
[configs, clusters, deleteClusters]
[zk: localhost:2181(CONNECTED) 1] create /kafka-manager/mutex ""
Created /kafka-manager/mutex
[zk: localhost:2181(CONNECTED) 2] create /kafka-manager/mutex/locks ""
Created /kafka-manager/mutex/locks
[zk: localhost:2181(CONNECTED) 3] create /kafka-manager/mutex/leases ""
Created /kafka-manager/mutex/leases

이후 cmak 재실행하면 정상적으로 생성되는 것 확인 가능

참고

• zkCli 참고: Failed to add cluster on 3.0.0.1 yahoo/CMAK#731 (comment)
• 증상 참고: Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again. yahoo/CMAK#748