Skip to content

Commit fbf073b

Browse files
urfreespaceurfreespace
authored
[website][upgrade]feat: docs migration - Cookbooks (#12319)
* [website][upgrade]feat: docs migration - Cookbooks Signed-off-by: LiLi <urfreespace@gmail.com> * upadte Signed-off-by: LiLi <urfreespace@gmail.com>
1 parent 1ae131a commit fbf073b

30 files changed

+4672
-0
lines changed

‎site2/website-next/docs/cookbooks-bookkeepermetadata.md

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
---
2+
id: cookbooks-bookkeepermetadata
3+
title: BookKeeper Ledger Metadata
4+
---
5+
6+
import Tabs from '@theme/Tabs';
7+
import TabItem from '@theme/TabItem';
8+
9+
10+
Pulsar stores data on BookKeeper ledgers, you can understand the contents of a ledger by inspecting the metadata attached to the ledger.
11+
Such metadata are stored on ZooKeeper and they are readable using BookKeeper APIs.
12+
13+
Description of current metadata:
14+
15+
| Scope | Metadata name | Metadata value |
16+
| ------------- | ------------- | ------------- |
17+
| All ledgers | application | 'pulsar' |
18+
| All ledgers | component | 'managed-ledger', 'schema', 'compacted-topic' |
19+
| Managed ledgers | pulsar/managed-ledger | name of the ledger |
20+
| Cursor | pulsar/cursor | name of the cursor |
21+
| Compacted topic | pulsar/compactedTopic | name of the original topic |
22+
| Compacted topic | pulsar/compactedTo | id of the last compacted message |
23+
24+

‎site2/website-next/docs/cookbooks-compaction.md

+144
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
id: cookbooks-compaction
3+
title: Topic compaction
4+
sidebar_label: Topic compaction
5+
---
6+
7+
import Tabs from '@theme/Tabs';
8+
import TabItem from '@theme/TabItem';
9+
10+
11+
Pulsar's [topic compaction](concepts-topic-compaction.md#compaction) feature enables you to create **compacted** topics in which older, "obscured" entries are pruned from the topic, allowing for faster reads through the topic's history (which messages are deemed obscured/outdated/irrelevant will depend on your use case).
12+
13+
To use compaction:
14+
15+
* You need to give messages keys, as topic compaction in Pulsar takes place on a *per-key basis* (i.e. messages are compacted based on their key). For a stock ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the key (more on this [below](#when-should-i-use-compacted-topics)). Messages without keys will be left alone by the compaction process.
16+
* Compaction can be configured to run [automatically](#configuring-compaction-to-run-automatically), or you can manually [trigger](#triggering-compaction-manually) compaction using the Pulsar administrative API.
17+
* Your consumers must be [configured](#consumer-configuration) to read from compacted topics ([Java consumers](#java), for example, have a `readCompacted` setting that must be set to `true`). If this configuration is not set, consumers will still be able to read from the non-compacted topic.
18+
19+
20+
> Compaction only works on messages that have keys (as in the stock ticker example the stock symbol serves as the key for each message). Keys can thus be thought of as the axis along which compaction is applied. Messages that don't have keys are simply ignored by compaction.
21+
22+
## When should I use compacted topics?
23+
24+
The classic example of a topic that could benefit from compaction would be a stock ticker topic through which consumers can access up-to-date values for specific stocks. Imagine a scenario in which messages carrying stock value data use the stock symbol as the key (`GOOG`, `AAPL`, `TWTR`, etc.). Compacting this topic would give consumers on the topic two options:
25+
26+
* They can read from the "original," non-compacted topic in case they need access to "historical" values, i.e. the entirety of the topic's messages.
27+
* They can read from the compacted topic if they only want to see the most up-to-date messages.
28+
29+
Thus, if you're using a Pulsar topic called `stock-values`, some consumers could have access to all messages in the topic (perhaps because they're performing some kind of number crunching of all values in the last hour) while the consumers used to power the real-time stock ticker only see the compacted topic (and thus aren't forced to process outdated messages). Which variant of the topic any given consumer pulls messages from is determined by the consumer's [configuration](#consumer-configuration).
30+
31+
> One of the benefits of compaction in Pulsar is that you aren't forced to choose between compacted and non-compacted topics, as the compaction process leaves the original topic as-is and essentially adds an alternate topic. In other words, you can run compaction on a topic and consumers that need access to the non-compacted version of the topic will not be adversely affected.
32+
33+
34+
## Configuring compaction to run automatically
35+
36+
Tenant administrators can configure a policy for compaction at the namespace level. The policy specifies how large the topic backlog can grow before compaction is triggered.
37+
38+
For example, to trigger compaction when the backlog reaches 100MB:
39+
40+
```bash
41+
42+
$ bin/pulsar-admin namespaces set-compaction-threshold \
43+
--threshold 100M my-tenant/my-namespace
44+
45+
```
46+
47+
Configuring the compaction threshold on a namespace will apply to all topics within that namespace.
48+
49+
## Triggering compaction manually
50+
51+
In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example:
52+
53+
```bash
54+
55+
$ bin/pulsar-admin topics compact \
56+
persistent://my-tenant/my-namespace/my-topic
57+
58+
```
59+
60+
The `pulsar-admin` tool runs compaction via the Pulsar {@inject: rest:REST:/} API. To run compaction in its own dedicated process, i.e. *not* through the REST API, you can use the [`pulsar compact-topic`](reference-cli-tools.md#pulsar-compact-topic) command. Here's an example:
61+
62+
```bash
63+
64+
$ bin/pulsar compact-topic \
65+
--topic persistent://my-tenant-namespace/my-topic
66+
67+
```
68+
69+
> Running compaction in its own process is recommended when you want to avoid interfering with the broker's performance. Broker performance should only be affected, however, when running compaction on topics with a large keyspace (i.e when there are many keys on the topic). The first phase of the compaction process keeps a copy of each key in the topic, which can create memory pressure as the number of keys grows. Using the `pulsar-admin topics compact` command to run compaction through the REST API should present no issues in the overwhelming majority of cases; using `pulsar compact-topic` should correspondingly be considered an edge case.
70+
71+
The `pulsar compact-topic` command communicates with [ZooKeeper](https://zookeeper.apache.org) directly. In order to establish communication with ZooKeeper, though, the `pulsar` CLI tool will need to have a valid [broker configuration](reference-configuration.md#broker). You can either supply a proper configuration in `conf/broker.conf` or specify a non-default location for the configuration:
72+
73+
```bash
74+
75+
$ bin/pulsar compact-topic \
76+
--broker-conf /path/to/broker.conf \
77+
--topic persistent://my-tenant/my-namespace/my-topic
78+
79+
# If the configuration is in conf/broker.conf
80+
$ bin/pulsar compact-topic \
81+
--topic persistent://my-tenant/my-namespace/my-topic
82+
83+
```
84+
85+
#### When should I trigger compaction?
86+
87+
How often you [trigger compaction](#triggering-compaction-manually) will vary widely based on the use case. If you want a compacted topic to be extremely speedy on read, then you should run compaction fairly frequently.
88+
89+
## Consumer configuration
90+
91+
Pulsar consumers and readers need to be configured to read from compacted topics. The sections below show you how to enable compacted topic reads for Pulsar's language clients. If the
92+
93+
### Java
94+
95+
In order to read from a compacted topic using a Java consumer, the `readCompacted` parameter must be set to `true`. Here's an example consumer for a compacted topic:
96+
97+
```java
98+
99+
Consumer<byte[]> compactedTopicConsumer = client.newConsumer()
100+
.topic("some-compacted-topic")
101+
.readCompacted(true)
102+
.subscribe();
103+
104+
```
105+
106+
As mentioned above, topic compaction in Pulsar works on a *per-key basis*. That means that messages that you produce on compacted topics need to have keys (the content of the key will depend on your use case). Messages that don't have keys will be ignored by the compaction process. Here's an example Pulsar message with a key:
107+
108+
```java
109+
110+
import org.apache.pulsar.client.api.Message;
111+
import org.apache.pulsar.client.api.MessageBuilder;
112+
113+
Message<byte[]> msg = MessageBuilder.create()
114+
.setContent(someByteArray)
115+
.setKey("some-key")
116+
.build();
117+
118+
```
119+
120+
The example below shows a message with a key being produced on a compacted Pulsar topic:
121+
122+
```java
123+
124+
import org.apache.pulsar.client.api.Message;
125+
import org.apache.pulsar.client.api.MessageBuilder;
126+
import org.apache.pulsar.client.api.Producer;
127+
import org.apache.pulsar.client.api.PulsarClient;
128+
129+
PulsarClient client = PulsarClient.builder()
130+
.serviceUrl("pulsar://localhost:6650")
131+
.build();
132+
133+
Producer<byte[]> compactedTopicProducer = client.newProducer()
134+
.topic("some-compacted-topic")
135+
.create();
136+
137+
Message<byte[]> msg = MessageBuilder.create()
138+
.setContent(someByteArray)
139+
.setKey("some-key")
140+
.build();
141+
142+
compactedTopicProducer.send(msg);
143+
144+
```

‎site2/website-next/docs/cookbooks-deduplication.md

+156
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
---
2+
id: cookbooks-deduplication
3+
title: Message deduplication
4+
sidebar_label: Message deduplication
5+
---
6+
7+
import Tabs from '@theme/Tabs';
8+
import TabItem from '@theme/TabItem';
9+
10+
11+
When **Message deduplication** is enabled, it ensures that each message produced on Pulsar topics is persisted to disk *only once*, even if the message is produced more than once. Message deduplication is handled automatically on the server side.
12+
13+
To use message deduplication in Pulsar, you need to configure your Pulsar brokers and clients.
14+
15+
## How it works
16+
17+
You can enable or disable message deduplication at the namespace level or the topic level. By default, it is disabled on all namespaces or topics. You can enable it in the following ways:
18+
19+
* Enable deduplication for all namespaces/topics at the broker-level.
20+
* Enable deduplication for a specific namespace with the `pulsar-admin namespaces` interface.
21+
* Enable deduplication for a specific topic with the `pulsar-admin topics` interface.
22+
23+
## Configure message deduplication
24+
25+
You can configure message deduplication in Pulsar using the [`broker.conf`](reference-configuration.md#broker) configuration file. The following deduplication-related parameters are available.
26+
27+
Parameter | Description | Default
28+
:---------|:------------|:-------
29+
`brokerDeduplicationEnabled` | Sets the default behavior for message deduplication in the Pulsar broker. If it is set to `true`, message deduplication is enabled on all namespaces/topics. If it is set to `false`, you have to enable or disable deduplication at the namespace level or the topic level. | `false`
30+
`brokerDeduplicationMaxNumberOfProducers` | The maximum number of producers for which information is stored for deduplication purposes. | `10000`
31+
`brokerDeduplicationEntriesInterval` | The number of entries after which a deduplication informational snapshot is taken. A larger interval leads to fewer snapshots being taken, though this lengthens the topic recovery time (the time required for entries published after the snapshot to be replayed). | `1000`
32+
`brokerDeduplicationProducerInactivityTimeoutMinutes` | The time of inactivity (in minutes) after which the broker discards deduplication information related to a disconnected producer. | `360` (6 hours)
33+
34+
### Set default value at the broker-level
35+
36+
By default, message deduplication is *disabled* on all Pulsar namespaces/topics. To enable it on all namespaces/topics, set the `brokerDeduplicationEnabled` parameter to `true` and re-start the broker.
37+
38+
Even if you set the value for `brokerDeduplicationEnabled`, enabling or disabling via Pulsar admin CLI overrides the default settings at the broker-level.
39+
40+
### Enable message deduplication
41+
42+
Though message deduplication is disabled by default at the broker level, you can enable message deduplication for a specific namespace or topic using the [`pulsar-admin namespaces set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication) or the [`pulsar-admin topics set-deduplication`](reference-pulsar-admin.md#topic-set-deduplication) command. You can use the `--enable`/`-e` flag and specify the namespace/topic.
43+
44+
The following example shows how to enable message deduplication at the namespace level.
45+
46+
```bash
47+
48+
$ bin/pulsar-admin namespaces set-deduplication \
49+
public/default \
50+
--enable # or just -e
51+
52+
```
53+
54+
### Disable message deduplication
55+
56+
Even if you enable message deduplication at the broker level, you can disable message deduplication for a specific namespace or topic using the [`pulsar-admin namespace set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication) or the [`pulsar-admin topics set-deduplication`](reference-pulsar-admin.md#topic-set-deduplication) command. Use the `--disable`/`-d` flag and specify the namespace/topic.
57+
58+
The following example shows how to disable message deduplication at the namespace level.
59+
60+
```bash
61+
62+
$ bin/pulsar-admin namespaces set-deduplication \
63+
public/default \
64+
--disable # or just -d
65+
66+
```
67+
68+
## Pulsar clients
69+
70+
If you enable message deduplication in Pulsar brokers, you need complete the following tasks for your client producers:
71+
72+
1. Specify a name for the producer.
73+
1. Set the message timeout to `0` (namely, no timeout).
74+
75+
The instructions for Java, Python, and C++ clients are different.
76+
77+
<Tabs
78+
defaultValue="Java clients"
79+
values={[
80+
{
81+
"label": "Java clients",
82+
"value": "Java clients"
83+
},
84+
{
85+
"label": "Python clients",
86+
"value": "Python clients"
87+
},
88+
{
89+
"label": "C++ clients",
90+
"value": "C++ clients"
91+
}
92+
]}>
93+
<TabItem value="Java clients">
94+
95+
To enable message deduplication on a [Java producer](client-libraries-java.md#producers), set the producer name using the `producerName` setter, and set the timeout to `0` using the `sendTimeout` setter.
96+
97+
```java
98+
99+
import org.apache.pulsar.client.api.Producer;
100+
import org.apache.pulsar.client.api.PulsarClient;
101+
import java.util.concurrent.TimeUnit;
102+
103+
PulsarClient pulsarClient = PulsarClient.builder()
104+
.serviceUrl("pulsar://localhost:6650")
105+
.build();
106+
Producer producer = pulsarClient.newProducer()
107+
.producerName("producer-1")
108+
.topic("persistent://public/default/topic-1")
109+
.sendTimeout(0, TimeUnit.SECONDS)
110+
.create();
111+
112+
```
113+
114+
</TabItem>
115+
<TabItem value="Python clients">
116+
117+
To enable message deduplication on a [Python producer](client-libraries-python.md#producers), set the producer name using `producer_name`, and set the timeout to `0` using `send_timeout_millis`.
118+
119+
```python
120+
121+
import pulsar
122+
123+
client = pulsar.Client("pulsar://localhost:6650")
124+
producer = client.create_producer(
125+
"persistent://public/default/topic-1",
126+
producer_name="producer-1",
127+
send_timeout_millis=0)
128+
129+
```
130+
</TabItem>
131+
<TabItem value="C++ clients">
132+
133+
To enable message deduplication on a [C++ producer](client-libraries-cpp.md#producer), set the producer name using `producer_name`, and set the timeout to `0` using `send_timeout_millis`.
134+
135+
```cpp
136+
137+
#include <pulsar/Client.h>
138+
139+
std::string serviceUrl = "pulsar://localhost:6650";
140+
std::string topic = "persistent://some-tenant/ns1/topic-1";
141+
std::string producerName = "producer-1";
142+
143+
Client client(serviceUrl);
144+
145+
ProducerConfiguration producerConfig;
146+
producerConfig.setSendTimeout(0);
147+
producerConfig.setProducerName(producerName);
148+
149+
Producer producer;
150+
151+
Result result = client.createProducer(topic, producerConfig, producer);
152+
153+
```
154+
</TabItem>
155+
156+
</Tabs>

0 commit comments

Comments
 (0)