Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial best practise for broker,produer and consumer #1

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,13 @@ docs:
- title: Best Practice
children:
- title: "Broker"
url: /docs/motivation/
url: /docs/best-practice-broker/
- title: "Producer"
url: /docs/motivation/
url: /docs/best-practice-producer/
- title: "Consumer"
url: /docs/core-concept/
url: /docs/best-practice-consumer/
- title: "NameServer"
url: /docs/best-practice-namesvr/
- title: "Virtualization"
url: /docs/cli-admin-tool/
- title: FAQ
Expand Down
23 changes: 23 additions & 0 deletions _docs/best-practice-broker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
title: "Best Practice For Broker"
permalink: /docs/best-practice-broker/
modified: 2016-12-24T15:01:43-04:00
---

Some useful tips for users.

{% include toc %}

## Broker Role
Broker Role is ASYNC_MASTER, SYNC_MASTER or SLAVE.
If you cannot tolerate message missing, we suggest you deploy SYNC_MASTER and attach a SLAVE to it.
If you feel ok about missing, but you want the Broker to be always available, you may deploy ASYNC_MASTER with SLAVE.
If you just want to make it easy, you may only need a ASYNC_MASTER without SLAVE.
## FlushDiskType
ASYNC_FLUSH is recommended, for SYNC_FLUSH is expensive and will cause too much performance loss. If you want reliability, we recommend you use SYNC_MASTER with SLAVE.
## ReentrantLock vs CAS
to be finished
## os.sh
to be finished


39 changes: 39 additions & 0 deletions _docs/best-practice-consumer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: "Best Practice For Consumer"
permalink: /docs/best-practice-consumer/
modified: 2016-12-24T15:01:43-04:00
---

Some useful tips for users.

{% include toc %}
## Consumer Group and Subscriptions
The first thing you should be aware of is that different Consumer Group can consume the same topic independently, each of the group will have their own consuming offsets.
And make sure each Consumer within the same Group to subscribe the same topics.
## MessageListener
### Orderly
The Consumer will lock each MessageQueue to make sure it is consumed one by one orderly. This will cause performance loss, but it is useful when you are care about the order of the messages.
It is not recommended to throw exception, you can return ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT instead.
### Concurrently
As the name tells, the Consumer will consume the messages concurrently. It is recommended to use this for achieving good performance.
It is not recommended to throw exception, you can return ConsumeConcurrentlyStatus.RECONSUME_LATER instead.
### Consume Status
For MessageListenerConcurrently, you can return RECONSUME_LATER to tell the consumer that you can not consume it right now and want to reconsume it later. Then you can continue to consume other messages.
For MessageListenerOrderly, as that you care about the order, so you can not jump over the message, but you can return SUSPEND_CURRENT_QUEUE_A_MOMENT to tell the consumer to hold on for a moment.
### Blocking
It is not recommend to block the Listener, for in return it will block the thread pool, and finally the consuming process may get stuck.
## Thread Number
The consumer use a ThreadPoolExecutor to process consuming internally. So you can tune it by using setConsumeThreadMin or setConsumeThreadMax.
## ConsumeFromWhere
When a new Consumer Group is established, it will need to decide whether it need to consume the historical messages which had already existed in the Broker.
CONSUME_FROM_LAST_OFFSET will ignore the historical messages, and consume any newly produced.
CONSUME_FROM_FIRST_OFFSET will consume every message existed in the Broker.
You can also use CONSUME_FROM_TIMESTAMP to consume messages produced after the specified timestamp.
## Duplication
Many circumstances could cause duplication, such as:
* Producer resend messages(i.e, in case of FLUSH_SLAVE_TIMEOUT)
* Consumer shutdown with some offsets not updated to the Broker in time.


So you may need to do some external work to handle this if your application cannot tolerate. For example, you may check the primary key of your DB.

14 changes: 14 additions & 0 deletions _docs/best-practice-namesvr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: "Best Practice For NameServer"
permalink: /docs/best-practice-namesvr/
modified: 2016-12-24T15:01:43-04:00
---

Some useful tips for users.

{% include toc %}

## Ordered Message
to be finished


45 changes: 45 additions & 0 deletions _docs/best-practice-producer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: "Best Practice For Producer"
permalink: /docs/best-practice-producer/
modified: 2016-12-24T15:01:43-04:00
---

Some useful tips for users.

{% include toc %}

## SendStatus
When sending a message, you will get SendResult and it will contain the SendStatus. Firstly, we assume that Message's isWaitStoreMsgOK=true(default is true). If not, we will always get SEND_OK if no exception is thrown.
Follow are the descriptions about each status.
### FLUSH_DISK_TIMEOUT
If the Broker set MessageStoreConfig's FlushDiskType=SYNC_FLUSH(default is ASYNC_FLUSH), and the Broker dose not finish flushing disk within MessageStoreConfig's syncFlushTimeout(default is 5 secs), you will get such status.
### FLUSH_SLAVE_TIMEOUT
If the Broker's role is SYNC_MASTER(default is ASYNC_MASTER), and the slave Broker dose not finish synchronizing with the master within the MessageStoreConfig's syncFlushTimeout(default is 5 secs), you will get such status.
### SLAVE_NOT_AVAILABLE
If the Broker's role is SYNC_MASTER(default is ASYNC_MASTER), but no slave Broker is configured, you will get such status.
### SEND_OK
You should be aware that SEND_OK does not mean it is reliable. If you cannot tolerate message missing, you should also enable SYNC_MASTER or SYNC_FLUSH.
### Duplication or Missing
If you get FLUSH_DISK_TIMEOUT, FLUSH_SLAVE_TIMEOUT or SLAVE_NOT_AVAILABLE, and the Broker happens to shutdown right the moment, you may get your message missing.
At this time, you have two choices, one is letting it go, which may get message missing; another is resending, which may get message duplication.
Often we suggest resend and make a way to handle the duplication removal when consuming. Unless you feel it does not matter when some messages are missed.
## Timeout
The Client send requests to Broker, and wait the responses, but if the max wait time is elapsed and no response is return, the Client will throw a RemotingTimeoutException.
The default wait time is 3 seconds.You can also pass timeout argument using send(msg, timeout) instead of send(msg).
Note that we do not suggest the value to be too small, for the Broker need some time to flush disk or synchronize with slave. Also the value may have little effect if it is too bigger than syncFlushTimeout for Broker may return a response with FLUSH_SLAVE_TIMEOUT or FLUSH_SLAVE_TIMEOUT before the timeout.
## Message Size
We suggest the message should be no more than 512K.
## Async Sending
Default send(msg) will block until the response is return. So if you care about performance, we suggest you use send(msg, callback) which will act in a async way.
## Producer Group
Normally, the producer group has no effects. But if you use transaction, you should take care of it.
In default, you can only create only one producer with the same producer group in the same JVM. Usually, this is enough.
## Thread Safety
The producer is thread-safe, you can just use it in your business logic.
## Performance
If you want more than one producer in one JVM, maybe for big data processing, we suggest you:
* use async sending with a few producers(3~5 is enough)
* setInstanceName for each producer



2 changes: 1 addition & 1 deletion content/about/contact/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@


<meta property="og:type" content="article">
<meta property="article:published_time" content="2016-12-29T16:56:36+08:00">
<meta property="article:published_time" content="2016-12-29T17:33:39+08:00">



Expand Down
2 changes: 1 addition & 1 deletion content/about/team/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@


<meta property="og:type" content="article">
<meta property="article:published_time" content="2016-12-29T16:56:36+08:00">
<meta property="article:published_time" content="2016-12-29T17:33:39+08:00">



Expand Down
52 changes: 52 additions & 0 deletions content/archive-layout-with-content/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -731,6 +731,58 @@ <h2 class="archive__item-title" itemprop="headline">
</article>
</div>

<div class="list__item">
<article class="archive__item" itemscope="" itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-broker/" rel="permalink">Best Practice For Broker</a>

</h2>


</article>
</div>

<div class="list__item">
<article class="archive__item" itemscope="" itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-consumer/" rel="permalink">Best Practice For Consumer</a>

</h2>


</article>
</div>

<div class="list__item">
<article class="archive__item" itemscope="" itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-namesvr/" rel="permalink">Best Practice For NameServer</a>

</h2>


</article>
</div>

<div class="list__item">
<article class="archive__item" itemscope="" itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-producer/" rel="permalink">Best Practice For Producer</a>

</h2>


</article>
</div>

<div class="list__item">
<article class="archive__item" itemscope="" itemtype="http://schema.org/CreativeWork">

Expand Down
88 changes: 88 additions & 0 deletions content/collection-archive/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,94 @@ <h2 class="archive__item-title" itemprop="headline">
</div>







<div class="list__item">
<article class="archive__item" itemscope itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-broker/" rel="permalink">Best Practice For Broker
</a>

</h2>

<p class="archive__item-excerpt" itemprop="description">Some useful tips for users.

</p>
</article>
</div>







<div class="list__item">
<article class="archive__item" itemscope itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-consumer/" rel="permalink">Best Practice For Consumer
</a>

</h2>

<p class="archive__item-excerpt" itemprop="description">Some useful tips for users.

</p>
</article>
</div>







<div class="list__item">
<article class="archive__item" itemscope itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-namesvr/" rel="permalink">Best Practice For NameServer
</a>

</h2>

<p class="archive__item-excerpt" itemprop="description">Some useful tips for users.

</p>
</article>
</div>







<div class="list__item">
<article class="archive__item" itemscope itemtype="http://schema.org/CreativeWork">

<h2 class="archive__item-title" itemprop="headline">

<a href="/docs/best-practice-producer/" rel="permalink">Best Practice For Producer
</a>

</h2>

<p class="archive__item-excerpt" itemprop="description">Some useful tips for users.

</p>
</article>
</div>





Expand Down
Loading