Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Fluent::Plugin::Base#after_start to detect end of #start #1190

Merged
merged 3 commits into from
Aug 29, 2016

Conversation

tagomoris
Copy link
Member

This method (and flag which shows this method is called properly) helps threads to start its actual process when all preparations are done.
This fixes #1187.

This is to make a detection that #start method completely finished.
Without this method, flush/enqueue threads will run without consideration even if Output#start takes
long time to finish. Output#start may need long time to finish if it tries network connection or
service detection over internet.
@tagomoris tagomoris added bug Something isn't working v0.14 labels Aug 29, 2016
@tagomoris
Copy link
Member Author

@repeatedly Please check this change. This should be included in next release.

@@ -922,6 +927,11 @@ def enqueue_thread_run
interval = @buffer_config.flush_thread_interval
end

while !self.after_started? && !self.stopped?
sleep interval
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is actual interval value needed for this check?
sleep 1 is not good?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sleep 1 works well... right, longer interval is not good. I'll update code.

@tagomoris
Copy link
Member Author

It's green on CI. I'll merge this later.

@tagomoris tagomoris merged commit e9ac872 into master Aug 29, 2016
@tagomoris tagomoris deleted the add-after_start-to-detect-end-of-start branch August 29, 2016 15:01
@beng-beng
Copy link

Does this addition require plugins to call after_start?

I was testing things using Fluentd v0.14.12 and the Elasticsearch output plugin v1.9.2 (which extends from ObjectBufferedOutput) and write never seemed to be called. It also seemed like flush_interval was being ignored. Instead, chunks only adhered to buffer_chunk_limit.

As a result, this would eventually lead to a BufferOverflowError. Adding the following to the plugin seemed to resolve the problem, but I just wanted to confirm whether or not this an appropriate usage of after_start within a plugin. Thanks.

def start
  super
  self.after_start
end

@repeatedly
Copy link
Member

No. elasitcsearch plugin v1.9.2 uses v0.12 API and v0.14 uses compat layer for it.
after_start is not needed in v0.12 API so the problem seems not plugin itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v0.14
Projects
None yet
Development

Successfully merging this pull request may close these issues.

v0.14 call write before start finished in BufferedOutputPlugin
3 participants