Clarify the different behaviors of pulsar sink when auto-ack is true/false

[truong-hua]

What issue do you find in Pulsar docs?

The doc does not clearly describe the behavior of a sink when the auto-ack is disabled and whether the auto-ack = true is always reliable for any cases and should be the only option or not.

In reality, we are facing some missed data in sink while our cluster is changing. We are only using rabbitmq as sink destination so we are now would like to understand more about the auto-ack methodology of pulsar to better understand the situation and somehow resolve our problem.

What is your suggestion?

We would like to know more what should we add to a sink if auto-ack is set to false.

Secondly, we would like to know if auto-ack = true, when a sink ack a message and when it nack a message, and whether it's 100% reliable or not? Let's say there is no any problem with the sink destination service (in our case it's RabbitMQ).

Do you have any references?

Nope

Would you like to fix this issue?

Nope

Note

• I have researched my question.

[shibd]

This is related to the function framework.

When Guarantees == ATMOST_ONCE and autoAck == true, The message will be ack before it enters sink connector.

In other cases auto-ack has no effect on the sink connector, the sink connector needs to ack himself.

auto-ack is a confusing configuration, you can see this PIP-166, it will plan deprecated that config.

[shibd]

If you use RabbitMQ sink connector from pulsar repository, When Guarantees != ATMOST_ONCE, auto-ack configuration has no effect on message ack, because the current implementation is internally ack every time.

pulsar/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSink.java

Lines 85 to 88 in 0134908

	String routingKey = record.getProperties().get("routingKey");
	rabbitMQChannel.basicPublish(exchangeName,
	StringUtils.isEmpty(routingKey) ? defaultRoutingKey : routingKey, null, value);
	record.ack();

[nlu90]

1. when you are confident the message is successfully processed, you should ack it.
2. if auto-ack is true, the message is acked when it's first received by the function/connector. No nack is invoked IIRC.

[shibd]

if auto-ack is true, the message is acked when it's first received by the function/connector. No nack is invoked IIRC.

And also need Guarantees == ATMOST_ONCE?

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[momo-jun]

Closed the issue since it has been addressed. Feel free to reopen it when it's relevant.