[improve][broker] GetPartitionMetadata fail also can produce messages

[congbobo184]

Motivation

GetPartitionMetadata fail also can produce messages

• 'autoUpdatePartitionsInterval' will get partition metadata and will regularly detect partition changes

• if GetPartitionMetadata will return ServiceNotReady, client receive ServiceNotReady will close cnx

• if close the current cnx, all producers and consumers witch use this cnx will close and reconnect

(

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java

Lines 323 to 345 in 5c6602c

	public void channelInactive(ChannelHandlerContext ctx) throws Exception {
	super.channelInactive(ctx);
	connectionsClosedCounter.increment();
	lastDisconnectedTimestamp = System.currentTimeMillis();
	log.info("{} Disconnected", ctx.channel());
	if (!connectionFuture.isDone()) {
	connectionFuture.completeExceptionally(new PulsarClientException("Connection already closed"));
	}
	ConnectException e = new ConnectException(
	"Disconnected from server at " + ctx.channel().remoteAddress());
	// Fail out all the pending ops
	pendingRequests.forEach((key, future) -> {
	if (pendingRequests.remove(key, future) && !future.isDone()) {
	future.completeExceptionally(e);
	}
	});
	waitingLookupRequests.forEach(pair -> pair.getRight().getRight().completeExceptionally(e));
	// Notify all attached producers/consumers so they have a chance to reconnect
	producers.forEach((id, producer) -> producer.connectionClosed(this, Optional.empty(), Optional.empty()));
	consumers.forEach((id, consumer) -> consumer.connectionClosed(this, Optional.empty(), Optional.empty()));

)

• this will influence a lot of producers and consumers and if current time the zk not available and bundle cache not exist the topic's bundle metadata, the client can't send messages to broker because the producer lookup will fail

Modifications

GetPartitionMetadata return MetadataError when throw MetadataStoreException

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[github-actions]

@congbobo184 Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.46%. Comparing base (bbc6224) to head (6a937ce).
Report is 461 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23050      +/-   ##
============================================
- Coverage     73.57%   73.46%   -0.12%     
- Complexity    32624    33187     +563     
============================================
  Files          1877     1915      +38     
  Lines        139502   143945    +4443     
  Branches      15299    15726     +427     
============================================
+ Hits         102638   105746    +3108     
- Misses        28908    30093    +1185     
- Partials       7956     8106     +150     

Flag	Coverage Δ
inttests	27.62% <0.00%> (+3.03%)	⬆️
systests	24.80% <0.00%> (+0.47%)	⬆️
unittests	72.51% <100.00%> (-0.33%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...va/org/apache/pulsar/broker/service/ServerCnx.java	72.50% <100.00%> (+0.35%)	⬆️

... and 511 files with indirect coverage changes