Beats plugin does not get restarted after unrecoverable error

[thehhugg]

Logstash doesn't properly restart the beats plugin if it fails. Restarting logstash resolves the problem, but that's not an ideal solution.

{:timestamp=>"2016-12-01T18:33:51.729000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Beats port=>5044, codec=><LogStash::Codecs::Plain charset=>\"UTF-8\">, host=>\"0.0.0.0\", ssl=>false, ssl_verify_mode=>\"none\", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>\"message\", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>[\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\", \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\", \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384\", \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\"], client_inactivity_timeout=>60>\n Error: event executor terminated", :level=>:error}

[jordansissel]

Hmm, that indeed looks like a bug.

@ph thoughts?

[ph]

@thehhugg This indeed look like a bug, can you add the logstash version and the beats input version you are currently using? Also is this possible to include your configuration?

Is there any other error before that one? The event executor terminated, make me things that plugin#stop was called.

You can get the beats version by running this command

bin/logstash-plugin list --verbose beats

[thehhugg]

Sure!

Logstash version: 2.4.1
Beats version: logstash-input-beats (3.1.8)

Config, other notes, etc:

input {
  beats {
    port => 5044
  }
}
filter {
  (redacted)
}

output {
  amazon_es {
    hosts => ["(var name redacted)"]
    region => "(var name redacted)"
    index => "(var name redacted)"
    aws_access_key_id => "(var name redacted)"
    aws_secret_access_key => "(var name redacted)"
  }
  stdout {
    codec => rubydebug
  }
}

Linux ~ 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 GNU/Linux

Docker version 1.12.3, build 6b644ec

• Logstash is running as a Marathon app in a Mesos-managed container.
• There weren't any errors preceding this one 😞

For what it's worth, it hasn't fallen over again, but if it does, I'll see if I can get you guys some more verbose output.

[ph]

@thehhugg Thanks for the info, we had another report of a similar error in #172 I will do a bit more testing on my side if I can come up with a reproducible use case or at least provide more debug info.

[mrbanzai]

I have the same sort of issue, when trying to launch multiple beats inputs on different ports. Perhaps that's the common thread?

[baurmatt]

We're too. If we can provide further information please let me know.

[matamouros]

Same here. From one day to the next, this started happening. Restarting Logstash, the filebeat shipper, etc, doesn't do anything. When I restart Logstash this starts happening after a few seconds.

[2017-02-12T10:52:35,829][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Beats port=>5044, id=>"63edd083fe1245e2b83c698477e59284bc990786-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_77a98a69-b962-4e8f-987d-145cdced8135",
enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, congestion_threshold=>5, target_field_for_codec=>"m
essage", tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_E
CDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60>
  Error: event executor terminated

My relevant config (working as of yesterday evening):

input {
    beats {
        port => 5044
    }
}

filter {
    [SNIP...]
}

output {
    # Uncomment the following for debugging into a local file
    file {
        path => "/tmp/logstash.out"
    }

    elasticsearch {
        hosts => "localhost:9200"
        index => "myindex-%{+YYYY.MM.dd}"
        template => "/etc/logstash/myindex.json"
        template_name => "myindex"
        template_overwrite => true
        document_type => "%{[@metadata][type]}"
    }

Any help much appreciated, will happily provide more details.

[jaijhala]

I'm having this issue as well. Using logstash 5.2.0

[fegul]

Also having this issue, using logstash 5.3.0

[jordansissel]

Error: event executor terminated

This is on my radar to look into soon. I don't' have any updates yet. The errors reported are helpful in pointing out where the problem might be, thank you all :)

[matamouros]

Please @fegul and @jaijhala and all looking this thread, make sure you already don't have :5044 bound and are trying to start Logstash again. I think I got this when for some reason the port was already taken.

[fegul]

@matamouros so it looks like the issue was in the confs I was building to parse different types of logs. In the one for Apache that's mentioned in this guide, it specifies having the following in the file:

input { beats { port => "5043" } }

What I was doing was copying the Apache conf and just changing the filter to specify the grok pattern I needed. That was when I started getting this error.

To eliminate the error, I removed the input section entirely from all confs except for the Apache one and it works just fine now.

[masip85]

@fegul , you are completely right. I can't manage multiple config files. With one unique config file works soomthly. What should we do to manage more than one?

[oussaka]

I found the solution here https://github.com/elastic/logstash/issues/6279

[cagdaskarabag]

Hello all,
below you can see my setup.
../bin/logstash-plugin list --verbose beats
logstash-input-beats (3.1.12)
../bin/logstash --version
logstash 5.2.2

i am having the same error even though i use only 1 config file.
and @oussaka the link you've provided is not working anymore. what was the solution? could you please share it? thank you.

[oussaka]

@cagdaskarabag the link works. (copy/past url in new tab).

[mrjcleaver]

i.e. elastic/logstash#6279

[praseodym]

This might be fixed in #289.