broker healthcheck endpoint reports No such ledger exists on Metadata Server - ledger endlessly

[wallacepeng]

2024-04-21T03:14:20,474+0000 [pulsar-io-4-7] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck] [null] Failed to create producer: {"errorMsg":"org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=3596046 - operation=Failed to open ledger caused by org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=3596046 - operation=Failed to open ledger","reqId":3493733038286036253, "remote":"pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/172.20.203.179:6650", "local":"/172.20.203.179:58124"}
2024-04-21 03:14:20.474
2024-04-21T03:14:20,474+0000 [pulsar-io-4-7] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xdce5010a, L:/172.20.203.179:58124 - R:pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/172.20.203.179:6650] Received error from server: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=3596046 - operation=Failed to open ledger caused by org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server - ledger=3596046 - operation=Failed to open ledger

this is caused by that internal healthcheck cannot create producer for topic (it seems have null producerName)
persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck .
it reports No such ledger exists on Metadata Server . All the others work except the healthcheck .

i have bookkeeper clusters, i decommission the default one then it got this error . does it still try to create ledger on decommisisoned bookie ?

bin/pulsar-admin topics stats-internal persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck
No such ledger exists on Metadata Server

Reason: No such ledger exists on Metadata Server

correct log from other pulsar cluster looks like

[wallacepeng]

I ended up with setting up another broker cluster, then i shutdown old broker cluster, clean up the namespaces , managed-ledgers then store old cluster , decommission new broker cluster. it magically fixed .