Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RIP-63] Queue Selection Strategy Optimization #6568

Merged
merged 48 commits into from
Aug 29, 2023

Conversation

GenerousMan
Copy link
Contributor

Make sure set the target branch to develop

What is the purpose of the change

[RIP-63] Issue #6567 .
Optimize the proxy's and client's selection strategy for brokers when sending messages, and use multiple selection strategies as a pipeline to filter suitable queues.

Brief changelog

Client:

  • Modified the files in org.apache.rocketmq.client.latency, such as MQFaultStrategy and LatencyFaultTolerance. These changes implemented queues's filters, which will provide more flexible selecting strategies.
  • Added some switches and configs in ClientConfig, to control this strategy.
  • Modified the updateFaultItem() method in the sending process. This method is designed to help record the brokers' available status. Now it will also record whether brokers are reachable or not.

Proxy:

  • Added the MQFaultStrategy in the topicRouteService. It will provide a similar queue selection mechanism in the client.
  • Added some switches and configs in ProxyConfig, to control this strategy.
  • Implemented the selectOneByPipeline() in MessageQueueSelector. It will call the filters in mqFaultStrategy to select suitable queue while sending messages.
  • Modified the implement of select() in SendMessageQueueSelector, now it will call the selectOneByPipeline(), but not the selectOne() before.(PS: If the switch is not opened, it will also run as selectOne()).
  • Added updateFaultItem() in the ProducerProcessor's sendMessage to record brokers' status, according to sending results.

Verifying this change

XXXX

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

  • Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
  • Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
  • Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

public void reset() {
int index = Math.abs(random.nextInt());
if (index < 0)
index = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code redundant

@@ -79,6 +89,10 @@ public class ClientConfig {
*/
protected boolean enableStreamRequestType = false;

private boolean sendLatencyEnable = Boolean.parseBoolean(System.getProperty(SEND_LATENCY_ENABLE, "false"));
Copy link
Contributor

@fuyou001 fuyou001 Apr 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be prudent to incorporate a warning concerning order message within the document,may be order message is disturbed

@@ -2846,6 +2849,37 @@ public void setMessageRequestMode(final String brokerAddr, final String topic, f
throw new MQClientException(response.getCode(), response.getRemark());
}
}
public Properties queryRemoteClientConfig(long timeoutMillis)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be increased namesrv server load

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I will remove this design.

* Remoting Client Common configuration
*/
public class RemoteClientConfig {
private final static Logger log = LoggerFactory.getLogger(MQClientInstance.class);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems more appropriate to write it like this:
private final static Logger log = LoggerFactory.getLogger(RemoteClientConfig.class);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, It should be. But I think I will remove this RemoteClientConfig...
The switches in clientConfig are enough.

@GenerousMan
Copy link
Contributor Author

# Conflicts:
#	client/src/main/java/org/apache/rocketmq/client/ClientConfig.java
#	proxy/src/main/java/org/apache/rocketmq/proxy/processor/ProducerProcessor.java
#	proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java
#	proxy/src/test/java/org/apache/rocketmq/proxy/service/BaseServiceTest.java
#	proxy/src/test/java/org/apache/rocketmq/proxy/service/sysmessage/HeartbeatSyncerTest.java
#	proxy/src/test/java/org/apache/rocketmq/proxy/service/transaction/ClusterTransactionServiceTest.java
# Conflicts:
#	proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java
# Conflicts:
#	client/src/main/java/org/apache/rocketmq/client/latency/MQFaultStrategy.java
# Conflicts:
#	client/src/main/java/org/apache/rocketmq/client/impl/producer/DefaultMQProducerImpl.java
#	proxy/src/main/java/org/apache/rocketmq/proxy/config/ProxyConfig.java
#	proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java
xdkxlk
xdkxlk previously approved these changes Aug 25, 2023
Copy link
Contributor

@xdkxlk xdkxlk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

return sendResultList;
}, this.executor);
}, this.executor)
.exceptionally(t -> {
Copy link
Contributor

@xdkxlk xdkxlk Aug 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use whenComplete is better. When an exception occurs, the upper layer function will get null in thenApply

@xdkxlk xdkxlk dismissed their stale review August 25, 2023 07:10

Some methods need to be modified

final int i = this.randomItem.incrementAndGet() % half;
return tmpList.get(i).getName();
Collections.shuffle(tmpList);
//Collections.sort(tmpList);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

annotation could be removed

}, this.executor);
}, this.executor)
.whenComplete((result, exception) -> {
endTimestamp.set(System.currentTimeMillis());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to use AtomicLong for endTimestamp, just a local variable would be fine

drpmma
drpmma previously approved these changes Aug 29, 2023
Copy link
Contributor

@drpmma drpmma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@drpmma drpmma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@drpmma drpmma merged commit b028277 into apache:develop Aug 29, 2023
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants