ISSUE-13238: Should introduce deleting state for ledger deletion

[sijie]

Original Issue: apache#13238

---

Is your enhancement request related to a problem? Please describe.
In current ledger deletion, we divided it into two separate steps.

1. Remove all the waiting to delete ledgers from ledger list and update the newest ledger list into meta store.
2. In the meta store update callback operation, delete the waiting to delete ledgers from storage system, such as BookKeeper or Tiered storage.

Due to the separate step, we can't ensure the ledger deletion transaction. If the first step succeed and the second step failed, it will lead to ledgers can't be deleted from storage system forever. The second step maybe fail by broker restart or storage system deletion failed.

https://github.com/apache/pulsar/blob/bd68b6f05f9749328701c59bdaf3cddda2254d39/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2532-L2561

In our customer's environment, we have found many orphan ledgers cause by the above reason.

Describe the solution you'd like
We'd better introduce a deleting state for the waiting to delete ledgers. The desired ledger deletion should follow the three steps.

1. Remove all the waiting to delete ledgers from ledger list and update the newest ledger list into meta store.
2. Write the waiting to delete ledgers into meta store marked as deleting state.
3. Use another thread to periodically check the deleting state ledgers, and do ledger delete operation on storage system. Each topic's owner control their own topics' ledger deletion.
4. After the ledger deletion complete, remove the ledger from meta store.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.