[fix] [broker] Fix race-condition causing repeated delete topic

[poorbarcode]

Fixes #23474

Motivation

Background

A lock named isClosingOrDeleting prevents concurrently deleting/closing topic

---

Issue 1: repeated delete a topic caused by a race condition

The code that deleting the topic is like below

CompletableFuture deleteFuture = () -> {
    CompletableFuture res = new CompletableFuture();
    ....
    if (success) {
        res.complete();
    } else {
        res.failed();
        // Highlight: release the lock `isClosingOrDeleting`.
        //Release the lock the first time.
        unfenceTopicToResume();
    }
};

deleteFuture.whenComplete((__ ,ex) -> {
    if (ex != null) {
        // Highlight: release the lock `isClosingOrDeleting`.
        //Release the lock the second time.
        unfenceTopicToResume();
    }
});

The code in PersistentTopic.delete acquires the lock once, but releases it twice, which breaks the locker, and leads to multiple deleting runs at the same time.

---

Issue 2: MetaStoreImpl.getException wrap a CompletionException to MetaStoreException deactivates the checker of deleting ML in the method PersistentTopic.delete

There is a checker that broker will assumed the ML deleting is succeed if it has been deleted before, such as bellow:

if (exception.getCause()  instanceof MetadataStoreException.NotFoundException) {
    future.complete(ctx);
} 

But MetaStoreImpl.getException wrap a CompletionException to MetaStoreException, causing the catch block to get an error like this MetaStoreException -> CompletionException -> NotFoundException, it breaks the checker above.

Modifications

• Fix two issues
• Revert [fix][ci] Temporarily move OneWayReplicatorTestBase tests to flaky group to unblock CI #23478, which marked multi tests as flaky

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

LGTM

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 57.14286% with 6 lines in your changes missing coverage. Please review.

Project coverage is 74.34%. Comparing base (bbc6224) to head (36cc57f).
Report is 701 commits behind head on master.

Files with missing lines	Patch %	Lines
...sar/broker/service/persistent/PersistentTopic.java	50.00%	4 Missing ⚠️
.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java	66.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23522      +/-   ##
============================================
+ Coverage     73.57%   74.34%   +0.77%     
- Complexity    32624    34938    +2314     
============================================
  Files          1877     1943      +66     
  Lines        139502   147011    +7509     
  Branches      15299    16199     +900     
============================================
+ Hits         102638   109299    +6661     
- Misses        28908    29284     +376     
- Partials       7956     8428     +472     

Flag	Coverage Δ
inttests	27.64% <7.14%> (+3.05%)	⬆️
systests	24.42% <7.14%> (+0.10%)	⬆️
unittests	73.71% <57.14%> (+0.86%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../apache/bookkeeper/mledger/impl/MetaStoreImpl.java	85.36% <66.66%> (-0.55%)	⬇️
...sar/broker/service/persistent/PersistentTopic.java	79.93% <50.00%> (+1.47%)	⬆️

... and 652 files with indirect coverage changes