Skip to content

Commit 7237831

Browse files
committed
Do not close the socket when the broker failed to acquire ownership for namespace bundle
### Motivation When the broker failed to acquire the ownership of a namespace bundle by `LockBusyException`. It means there is another broker that has acquired the metadata store path and didn't release that path. For example: Broker 1: ``` 2024-01-24T23:35:36,626+0000 [metadata-store-10-1] WARN org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup <role> for topic persistent://<tenant>/<ns>/<topic> with error org.apache.pulsar.broker.PulsarServerException: Failed to acquire ownership for namespace bundle <tenant>/<ns>/0x50000000_0x51000000 Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$LockBusyException: Resource at /namespace/<tenant>/<ns>/0x50000000_0x51000000 is already locked ``` Broker 2: ``` 2024-01-24T23:35:36,650+0000 [broker-topic-workers-OrderedExecutor-3-0] INFO org.apache.pulsar.broker.PulsarService - Loaded 1 topics on <tenant>/<ns>/0x50000000_0x51000000 -- time taken: 0.044 seconds ``` After broker 2 released the lock at 23:35:36,650, the lookup request to broker 1 should tell the client that namespace bundle 0x50000000_0x51000000 is currently being unloaded and in the next retry the client will connect to the new owner broker. Here is another typical error: ``` 2024-01-24T23:57:57,264+0000 [pulsar-io-4-5] INFO org.apache.pulsar.broker.lookup.TopicLookupBase - Failed to lookup <role> for topic persistent://<tenant>/<ns>/<topic> with error Namespace bundle <tenant>/<ns>/0x0d000000_0x0e000000 is being unloaded ``` Though after apache/pulsar#21211, the server error becomes `MetadataError` rather than `ServiceNotReady`. However, since the `ServerError` is `ServiceNotReady`, the client will close the connection. If there are many other producers or consumers on the same connection, they will all reestablish connection to the broker, which is unnecessary and brings much pressure to broker side. ### Modifications In `checkServerError`, when the error code is `ServiceNotReady`, check the error message as well, if it hit the case in `handleLookupError`, do not close the connection.
1 parent d1dd08b commit 7237831

File tree

2 files changed

+10
-5
lines changed

2 files changed

+10
-5
lines changed

lib/ClientConnection.cc

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1469,10 +1469,15 @@ Future<Result, SchemaInfo> ClientConnection::newGetSchema(const std::string& top
14691469
return promise.getFuture();
14701470
}
14711471

1472-
void ClientConnection::checkServerError(ServerError error) {
1472+
void ClientConnection::checkServerError(ServerError error, const std::string& message) {
14731473
switch (error) {
14741474
case proto::ServerError::ServiceNotReady:
1475-
close(ResultDisconnected);
1475+
// See
1476+
// https://github.com/apache/pulsar/blob/1952f94769d9dc80908d159be6e6ce1ff48b83fb/pulsar-broker/src/main/java/org/apache/pulsar/broker/lookup/TopicLookupBase.java#L334
1477+
// These errors should be retryable and we should not close the connection
1478+
if (message.find("Failed to lookup") != std::string::npos) {
1479+
close(ResultDisconnected);
1480+
}
14761481
break;
14771482
case proto::ServerError::TooManyRequests:
14781483
// TODO: Implement maxNumberOfRejectedRequestPerConnection like
@@ -1573,7 +1578,7 @@ void ClientConnection::handlePartitionedMetadataResponse(
15731578
<< partitionMetadataResponse.request_id()
15741579
<< " error: " << partitionMetadataResponse.error()
15751580
<< " msg: " << partitionMetadataResponse.message());
1576-
checkServerError(partitionMetadataResponse.error());
1581+
checkServerError(partitionMetadataResponse.error(), partitionMetadataResponse.message());
15771582
lookupDataPromise->setFailed(
15781583
getResult(partitionMetadataResponse.error(), partitionMetadataResponse.message()));
15791584
} else {
@@ -1650,7 +1655,7 @@ void ClientConnection::handleLookupTopicRespose(
16501655
LOG_ERROR(cnxString_ << "Failed lookup req_id: " << lookupTopicResponse.request_id()
16511656
<< " error: " << lookupTopicResponse.error()
16521657
<< " msg: " << lookupTopicResponse.message());
1653-
checkServerError(lookupTopicResponse.error());
1658+
checkServerError(lookupTopicResponse.error(), lookupTopicResponse.message());
16541659
lookupDataPromise->setFailed(
16551660
getResult(lookupTopicResponse.error(), lookupTopicResponse.message()));
16561661
} else {

lib/ClientConnection.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -404,7 +404,7 @@ class PULSAR_PUBLIC ClientConnection : public std::enable_shared_from_this<Clien
404404

405405
friend class PulsarFriend;
406406

407-
void checkServerError(ServerError error);
407+
void checkServerError(ServerError error, const std::string& message);
408408

409409
void handleSendReceipt(const proto::CommandSendReceipt&);
410410
void handleSendError(const proto::CommandSendError&);

0 commit comments

Comments
 (0)