retry with expected sequence token

[kaz080]

Hello,

Sometime I had this kind of exception continuously:

2015-01-15 10:37:06 +0000 [warn]: fluent/output.rb:353:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-01-15 10:37:09 +0000 error_class="Aws::CloudWatchLogs::Errors::InvalidSequenceTokenException" error="The given sequenceToken is invalid. The next expected sequenceToken is: 49542672486808773264381019708032531348811120384399118754" plugin_id="object:3ff007cf4b1c"

The message includes expected sequenceToken, so I wrote retry code using it.

Thanks

[ryotarai]

I'm not sure why the error occurs. Did you figure out?

[nritholtz]

+1, experiencing same issue:

2015-10-19 17:21:30 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-10-19 17:37:57 +0000 error_class="Aws::CloudWatchLogs::Errors::InvalidSequenceTokenException" error="The given sequenceToken is invalid. The next expected sequenceToken is: 49540110655877395126812203274137863573824741149218191138" plugin_id="object:3fc2be6ffe58"

[ryotarai]

Hmm, I understand the problem still exist but retrieving token from an error message is not a good way...
Is there any other way?

[nritholtz]

I think the issue I was experiencing was due to logging to the same log_stream from different fluentd instances.

As a quick fix, I've created a unique suffix to the log_stream_name for each fluentd instance, so as to avoid logging to the same log_stream.

[ryotarai]

I think the issue I was experiencing was due to logging to the same log_stream from different fluentd instances.

Ah, I see. As you said, multiple processes should not insert logs into one log stream.