at-least-once acknowledgements are not processed

[cosmok]

My setup is like this:

forwarder -> aggregator
td-agent 0.14.10 on both

When I set require_ack_response to true in my forwarder,

I can see acks being logged on the aggregator:

2016-12-23 11:20:06 +1100 [trace]: [Fluent::Plugin::ForwardInput] sent response to fluent socket address="10.18.68.219" response={"ack"=>"VESFkVa4eEpn+/hwFcOpLw==\n"}

But the forwarder still thinks ACKs have not been received
2016-12-23 11:20:36 +1100 [warn]: [Fluent::Plugin::ForwardOutput] failed to flush the buffer chunk, timeout to commit. chunk_id="5444859156b8784a67fbf87015c3a92f" flushed_at=2016-12-23 11:20:06 +1100
[more of these..]
2016-12-23 11:29:28 +1100 [warn]: [Fluent::Plugin::ForwardOutput] failed to flush the buffer chunk, timeout to commit. chunk_id="5444859156b8784a67fbf87015c3a92f" flushed_at=2016-12-23 11:28:57 +1100

Chunk id matches:
bin2hex(base64_decode('VESFkVa4eEpn+/hwFcOpLw==')); => "5444859156b8784a67fbf87015c3a92f"

And I can see that the buffer is not cleared until the retry_limit in the file system.

so far I have tried:

• increasing/decreasing flush_interval on the forwarder
• increasing/decreasing buffer_chunk_limit in the forwarder
• increasing/decreasing retry_wait in the forwarder
• increasing/decreasing ack_response_timeout in the forwarder

tcpdump is confirming traffic from the aggregator to forwarder.

Is there another setting that needs to be looked at?

[nieldomingo]

looks like i'm getting this same behaviour

[repeatedly]

Thanks for the report.
We will check it later.
Does this problem happen with older v0.14?

[nieldomingo]

reverted to 0.14.9, it seems to be ok.

[tagomoris]

I identified the bug that cause this problem, and am now fixing it.

[tagomoris]

@cosmok @nieldomingo can you check whether the branch fix-bug-forward-ack-reponse solve your problem or not?

[tagomoris]

v0.14.11 has been released with the fix for this problem. Please try it.

[cosmok]

@tagomoris v0.14.11 has this issue fixed.
Thanks!