Pulsar SQL is not able to read the latest message

[sijie]

Describe the bug

Pulsar SQL is always lacking one message when querying pulsar topics.

Expected behavior

"read-my-write" semantic.

if I write a bunch of messages to pulsar topics, and issue a query after "my-write", I should be able to query all the messages I written.

Reason

The problem is due to broker doesn't advance LAC, so when pulsar sql bypasses broker to query the data, it can only read entries up to the LAC that all the bookies learned.

Fixes

Enable periodically write lac on broker.

[jerrypeng]

@sijie yup have discussed this issue before with @merlimat but we decided not to take any action at that time because we reasoned that Pulsar is a streaming system thus we expect users to continuously pump data into pulsar and not just one time batch thus this wouldn't be an issue since LAC will eventually get updated if there is continuous data

[sijie]

@jerrypeng this is not just about not being able to read the event, but also about the latency (how fast it can query the data). current assumption is assuming there are are events "eventually" arrive and advance the LAC, which will significantly delay the messages being visible to users if the events are delayed. so I think we still need to enable explicit write lac on broker, so the lac will be advanced within a SLA.

The feature is already in bookkeeper. It is just a matter to enable it and make sure it is working as expected. If pulsar broker doesn't want this feature to be turned on by default, we can make it configurable and turned off by default. If people want this feature, they can turn it on. I don't think we should just ignore this fact and try to explain to people who don't really understand how the underlying stuff works.

[jerrypeng]

@sijie I totally agree we should add something like this. I was just presenting some previous thoughts we have had on this problem.

[merlimat]

Yes, we should have a way to configure the explicit LAC update on the topics and have some reasonable default.

Though, should this be for 2.3.1? I think it would more be appropriate for 2.4 since it's not a bug.. but rather a current limitation.

[sijie]

@merlimat we can move this to 2.4.0

[atris]

Hi, If nobody is working on this, can I please be assigned this?

[sijie]

@atris the bookkeeper change has made to Pulsar. #5822 is enabling the explicit lac configuration. once that is fixed, this task is done.

[codelipenghui]

close via #5822

[codelipenghui]

This issue is fixed by #6908 and #6908 is released in 2.5.2, So I add release/2.5.2 tag for this issue.