[fix][broker] Fix the broker registery cannot recover from the metadata node deletion

[BewareMyPower]

Motivation

#23298 introduce a regression that once the metadata node of this broker was deleted (e.g. by session timeout), the broker register would never have a chance to recover. In this case, the clients whose owner is this broker would never be able to produce or consume.

Modifications

Add a default listener to BrokerRegisterImpl that will register itself again in an asynchronous way if the deleted node is the current broker. Add a new state Unregistering to prevent the broker from registering itself again after unregister() is called.

Add BrokerRegistryIntegrationTest to verify this fix and the behavior introduced from #23298

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[heesung-sn]

I wonder if we need to periodically check this broker registration lock in the monitor thread as well for better fault tolerance.

[BewareMyPower]

periodically check this broker registration lock in the monitor thread as well for better fault tolerance.

Yeah it can handle the case when there is a bug with the metadata store client. But even if it has a bug, we can easily find it from the alerts and fix it by manually restarting the broker or creating the metadata node. Therefore, I neither support nor oppose this idea.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 71.42857% with 8 lines in your changes missing coverage. Please review.

Project coverage is 74.56%. Comparing base (bbc6224) to head (5c720d8).
Report is 606 commits behind head on master.

Files with missing lines	Patch %	Lines
...ker/loadbalance/extensions/BrokerRegistryImpl.java	71.42%	5 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23359      +/-   ##
============================================
+ Coverage     73.57%   74.56%   +0.99%     
- Complexity    32624    33942    +1318     
============================================
  Files          1877     1934      +57     
  Lines        139502   145035    +5533     
  Branches      15299    15848     +549     
============================================
+ Hits         102638   108150    +5512     
+ Misses        28908    28605     -303     
- Partials       7956     8280     +324     

Flag	Coverage Δ
inttests	27.75% <46.42%> (+3.16%)	⬆️
systests	24.56% <0.00%> (+0.24%)	⬆️
unittests	73.91% <71.42%> (+1.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ker/loadbalance/extensions/BrokerRegistryImpl.java	81.41% <71.42%> (-3.35%)	⬇️

... and 600 files with indirect coverage changes

[lhotari]

This PR introduced a flaky test #23365, @BewareMyPower do you have a chance to fix it?