Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] Add *processing guarantee for connector* #5224

Merged
merged 4 commits into from
Sep 27, 2019
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 95 additions & 2 deletions site2/docs/io-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,102 @@ Common sinks include other messaging systems and SQL and NoSQL databases.

For the complete list of Pulsar built-in sink connectors, see [sink connector](io-connectors.md#sink-connector).

## Processing guarantee

Processing guarantees are used to handle errors when writing messages to Pulsar topics.

> Pulsar connectors and Functions use the **same** processing guarantees as below.

Delivery semantic | Description
:------------------|:-------
`at-most-once` | Each message sent to a connector is to be **processed once** or **not to be processed**.
`at-least-once` | Each message sent to a connector is to be **processed once** or **more than once**.
`effectively-once` | Each message sent to a connector has **one output associated** with it.

> Processing guarantees for connectors not just rely on Pulsar guarantee but also **relate to external systems**, that is, **the implementation of source and sink**.

* Source: Pulsar ensures that writing messages to Pulsar topics respects to the processing guarantees. It is within Pulsar's control.

* Sink: the processing guarantees rely on the sink implementation. If the sink implementation does not handle retries in an idempotent way, the sink does not respect to the processing guarantees.

### Set

When creating a connector, you can set the processing guarantee with the following semantics:

* ATLEAST_ONCE

* ATMOST_ONCE

* EFFECTIVELY_ONCE

> If `--processing-guarantees` is not specified when creating a connector, the default semantic is `ATLEAST_ONCE`.

Here takes **Admin CLI** as an example. For more information about **REST API** or **JAVA Admin API**, see [here](io-use.md#create).

<!--DOCUSAURUS_CODE_TABS-->

<!--Source-->

```bash
$ bin/pulsar-admin sources create \
--processing-guarantees ATMOST_ONCE \
# Other source configs
```

For more information about the options of `pulsar-admin sources create`, see [here](reference-connector-admin.md#create).

<!--Sink-->

```bash
$ bin/pulsar-admin sinks create \
--processing-guarantees EFFECTIVELY_ONCE \
# Other sink configs
```

For more information about the options of `pulsar-admin sinks create`, see [here](reference-connector-admin.md#create-1).

<!--END_DOCUSAURUS_CODE_TABS-->

### Update

After creating a connector, you can update the processing guarantee with the following semantics:

* ATLEAST_ONCE

* ATMOST_ONCE

* EFFECTIVELY_ONCE

Here takes **Admin CLI** as an example. For more information about **REST API** or **JAVA Admin API**, see [here](io-use.md#create).

<!--DOCUSAURUS_CODE_TABS-->

<!--Source-->

```bash
$ bin/pulsar-admin sources update \
--processing-guarantees EFFECTIVELY_ONCE \
# Other source configs
```

For more information about the options of `pulsar-admin sources update`, see [here](reference-connector-admin.md#update).

<!--Sink-->

```bash
$ bin/pulsar-admin sinks update \
--processing-guarantees ATMOST_ONCE \
# Other sink configs
```

For more information about the options of `pulsar-admin sinks update`, see [here](reference-connector-admin.md#update-1).

<!--END_DOCUSAURUS_CODE_TABS-->


## Work with connector

You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands.
You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands.

Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via **Pulsar admin CLI** (that is, [Connector CLI](reference-connector-admin.md) and [Functions CLI](functions-cli.md)), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately).
Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately).