Bookie went down on the power outage on the cluster #22706
Unanswered
sindhushreem25
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Bookies went down when 2 out of 3 worker nodes went down due to some power outage .Post the machines and Kubernetes cluster is up bookies are running but one of the topics stats is showing the below exception and data flow is not happening to that topic. Any ways to recover this topic ?
./pulsar-admin topics stats persistent://public/default/profile1_9-partition-1
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such entry
Reason:
--- An unexpected error occurred in the server ---
Message: No such entry
Stacktrace:
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such entry
On further debugging in the broker logs we could see that read entry was failing .
Then we ran bookie ledger metadata and readledger below are the ouput .
metadata :
2024-05-16T04:48:22,297 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=2, writeQuorumSize=2, ackQuorumSize=2, state=CLOSED, length=8336855, lastEntryId=474, digestType=CRC32C, password=base64:, ensembles={0=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181], 475=[itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-0.itomdipulsar-bookkeeper.chotest:3181]}, customMetadata={component=base64:bWFuYWdlZC1sZWRnZXI=, pulsar/managed-ledger=base64:cHVibGljL2RlZmF1bHQvcGVyc2lzdGVudC9wcm9maWxlMV85LXBhcnRpdGlvbi0x, application=base64:cHVsc2Fy}}
read ledger :
ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L315020253 E474-E474, Sent to [itomdipulsar-bookkeeper-1.itomdipulsar-bookkeeper.chotest:3181, itomdipulsar-bookkeeper-2.itomdipulsar-bookkeeper.chotest:3181], Heard from [] : bitset = {}, Error = 'No such entry'. First unread entry is (-1, rc = null)
Beta Was this translation helpful? Give feedback.
All reactions