In Pulsar topic, we have retention policy to control the acknowledged message lifetime. For the unacknowledged messages, we have a separate mechanism to control the message lifetime, which is called TTL. The TTL is a time-to-live value for the message, which is controlled by ttlDurationDefaultInSeconds. The message will be automatically acknowledged if it is not consumed within the TTL value.
In Pulsar, we have two kinds of topics, system topic and normal topic. The system topics are used for internal purposes, such as transaction internal topics. The system topics are not supposed to be consumed by the users. However, the system topics are still subject to the TTL check. If the system topics are not consumed within the TTL value, the messages in the system topics will be automatically acknowledged. This is not the expected behavior for the system topics and may lead to data loss.
For normal topics, we also has two kinds of subscriptions, system subscription and normal subscription. The system subscription is used for internal purposes, such as compaction service or third-party plugins. The system subscription is not supposed to be used by the users. However, the system subscription is still subject to the TTL check. If the system subscription is not consumed within the TTL value, the messages in the system subscription will be automatically acknowledged. This is not the expected behavior for the system subscription.
We had one PR #21865 to filter the compaction service cursors for TTL check, but it doesn't cover other system cursors. To provide a general solution and support third-party plugin cursors not impacted by TTL, I proposed to add an additionalSystemCursorNames ignore list to filter the TTL check.
Add an additionalSystemCursorNames ignore list to filter the TTL check for additional system subscriptions except for compaction service subscription. The additionalSystemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions. The compaction service subscription is a system subscription and should not be impacted by TTL. To reduce the risk of data loss after enabled compaction service, we will add the compaction service subscription to the TTL ignore list by default and can't be removed.
Add a additionalSystemCursorNames ignore list to filter the TTL check for system subscriptions. The additionalSystemCursorNames ignore list is an optional configuration, and the default value is empty. Pulsar broker will filter the TTL check for the additionalSystemCursorNames subscriptions.
This change is fully compatible.
- Mailing List discussion thread: https://lists.apache.org/thread/xgcworz4j8rjlqwr476s7sqn9do43f1t
- Mailing List voting thread: https://lists.apache.org/thread/xs3g2y6fgjpfjr8fhf1qghcxkrt3yby7