nebula-metad fails to start up: KVBasedClusterIdMan.h: Put failed

[george-hopkins]

Describe the bug

nebula-metad 3.2.0 fails to start on Ubuntu 18.04 (Docker image ubuntu:bionic)

Your Environments

• OS: Ubuntu 18.04

How To Reproduce

Steps to reproduce the behavior:

root@docker0:/tmp# dpkg -i nebula-graph-3.2.0.ubuntu1804.amd64.deb 
Selecting previously unselected package nebula-graph.
(Reading database ... 4051 files and directories currently installed.)
Preparing to unpack nebula-graph-3.2.0.ubuntu1804.amd64.deb ...
Unpacking nebula-graph (3.2.0) ...
Setting up nebula-graph (3.2.0) ...
root@docker0:/tmp# /usr/local/nebula/scripts/nebula.service start all
[INFO] Starting nebula-metad...
[INFO] Done
[INFO] Starting nebula-graphd...
[INFO] Done
[INFO] Starting nebula-storaged...
[INFO] Done
root@docker0:/tmp# /usr/local/nebula/scripts/nebula.service status all
[INFO] nebula-metad(ef6d6a0): Exited
[INFO] nebula-graphd(ef6d6a0): Running as 118, Listening on 9669
[INFO] nebula-storaged(ef6d6a0): Running as 157, Listening on 9779

Expected behavior

nebula-metad keeps running

Additional context

root@docker0:/tmp# cat /usr/local/nebula/logs/nebula-metad.INFO 
Log file created at: 2022/07/28 15:32:06
Running on machine: docker0
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20220728 15:32:06.154403   322 MetaDaemon.cpp:135] localhost = "127.0.0.1":9559
I20220728 15:32:06.173408   322 NebulaStore.cpp:51] Start the raft service...
I20220728 15:32:06.180752   322 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20220728 15:32:06.290015   322 RaftexService.cpp:46] Start raft service on 9560
I20220728 15:32:06.290253   322 NebulaStore.cpp:85] Scan the local path, and init the spaces_
I20220728 15:32:06.290391   322 NebulaStore.cpp:92] Scan path "/usr/local/nebula/data/meta/nebula/0"
I20220728 15:32:06.290422   322 NebulaStore.cpp:271] Init data from partManager for "127.0.0.1":9559
I20220728 15:32:06.290454   322 NebulaStore.cpp:387] Create data space 0
I20220728 15:32:06.365542   322 RocksEngine.cpp:97] open rocksdb on /usr/local/nebula/data/meta/nebula/0/data
I20220728 15:32:06.383934   322 NebulaStore.cpp:459] Space 0, part 0 has been added, asLearner 0
I20220728 15:32:06.384073   322 NebulaStore.cpp:78] Register handler...
I20220728 15:32:06.384095   322 MetaDaemonInit.cpp:101] Waiting for the leader elected...
I20220728 15:32:06.384115   322 MetaDaemonInit.cpp:113] Leader has not been elected, sleep 1s
I20220728 15:32:07.384330   322 KVBasedClusterIdMan.h:109] There is no clusterId existed in kvstore!
I20220728 15:32:07.384493   322 MetaDaemonInit.cpp:121] I am leader, create cluster Id
I20220728 15:32:07.384542   322 KVBasedClusterIdMan.h:36] Create ClusterId 5910483749134832870
I20220728 15:32:07.384703   322 KVBasedClusterIdMan.h:139] Put failed, error -4
E20220728 15:32:07.384757   322 MetaDaemonInit.cpp:125] Persist cluster failed!
I20220728 15:32:07.385987   322 NebulaStore.cpp:321] Stop the raft service...
I20220728 15:32:07.386029   322 RaftexService.cpp:67] Stopping the raftex service on port 9560
I20220728 15:32:07.386077   322 RaftexService.cpp:77] All partitions have stopped
I20220728 15:32:07.386127   322 NebulaStore.cpp:324] Stop kv engine...
I20220728 15:32:07.386535   322 NebulaStore.cpp:39] Cut off the relationship with meta client
I20220728 15:32:07.388005   322 Part.h:59] [Port: 9560, Space: 0, Part: 0] ~Part()
I20220728 15:32:07.388074   322 RocksEngine.h:196] Release rocksdb on /usr/local/nebula/data/meta/nebula/0
I20220728 15:32:07.393414   322 NebulaStore.cpp:47] ~NebulaStore()
E20220728 15:32:07.393594   322 MetaDaemon.cpp:144] Init kv failed!

[wenhaocs]

Hi @george-hopkins Apparently, the KVStore write failed. Do you mind set the --v =4 in metad.config and restart again?

[george-hopkins]

I20220728 18:26:53.484944   424 FileBasedWal.cpp:514] [Port: 9560, Space: 0, Part: 0] Failed to appendLogs because of no more space

Thank you! Nebula works flawlessly after assigning a larger disk.