Constant lag on newer version of code, 1.5.2 has no lag.

[Smadarl]

I have a lot of consumers I've written (20+) over the past 4 years or so. Just in the past few months (though it's possibly been longer than that) I've noticed that some of them that I've updated recently no longer have a lag of 0 most of the time anymore. I looked into a lot of things trying to figure out what was causing the lag. All my consumers run against the same kafka cluster, so it's not the cluster.

Finally, I made a new, very simple consumer using the latest version (2.1.1) and had it read from a single topic, and it still had lag. After trying a few more things, I downgraded to 1.5.2, and the lag disappeared. I know that the lag exists in 1.9.2 also, but beyond that, I don't know where the change happens. Somewhere between 1.5.2 and 1.9.2.

Has anyone else experienced this? Is this expected? Is there some configuration change I need to make in newer versions?

[Smadarl]

I tried compiling against a few more different versions (https://github.com/confluentinc/confluent-kafka-go/releases). 1.6.1 has no lag, 1.7.0 does. So something between those two releases is causing the lag.

[Smadarl]

Found the issue. https://github.com/confluentinc/librdkafka/releases/tag/v1.7.0
"Statistics: consumer_lag is now using the committed_offset, while the new consumer_lag_stored is using stored_offset (offset to be committed)."

I needed to change the specific stat field I was looking at.