fluentd docker-compose error: `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

[sbilgil]

Describe the bug

I have created docker-compose.yml and fluentd/etc + fluentd/fluent.conf with respect to the documentation on https://docs.fluentd.org/v/0.12/container-deployment/docker-compose. When i run cmd docker-compose up it failed with an error :"/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)". I have searched net and found some tips but they didnt work on my environment.

To Reproduce

just create docker-compose.yml / fluent.conf and Dockerfile wrt https://docs.fluentd.org/v/0.12/container-deployment/docker-compose

Expected behavior

get fluentd working without error

Your Environment

- Fluentd version: fluent/fluentd:v0.12-debian
- TD Agent version: td4
- Operating system: Ubuntu 18.04.4 LTS
- Kernel version:5.3.0-28-generic

Your Configuration

fluent.conf file:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<match *.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>
  <store>
    @type stdout
  </store>
</match>

Your Error Log

web_1            | WARNING: no logs are available with the 'fluentd' log driver
fluentd_1        | 2023-02-06 12:04:19 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
fluentd_1        | 2023-02-06 12:04:19 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
fluentd_1        | 2023-02-06 12:04:19 +0000 [info]: gem 'fluentd' version '1.12.0'
fluentd_1        | /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)
fluentd_1        | 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1        | 	from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/elasticsearch_fallback_selector.rb:1:in `<top (required)>'
fluentd_1        | 	from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `require_relative'
fluentd_1        | 	from /usr/local/bundle/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/out_elasticsearch.rb:31:in `<top (required)>'
fluentd_1        | 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1        | 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:102:in `block in search'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:99:in `each'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:99:in `search'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/registry.rb:44:in `lookup'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin.rb:160:in `new_impl'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin.rb:109:in `new_output'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin/multi_output.rb:72:in `block in configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin/multi_output.rb:63:in `each'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin/multi_output.rb:63:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin/out_copy.rb:38:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/plugin.rb:178:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:132:in `add_match'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:74:in `block in configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:64:in `each'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/agent.rb:64:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/label.rb:31:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/root_agent.rb:143:in `block in configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/root_agent.rb:143:in `each'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/root_agent.rb:143:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/engine.rb:105:in `configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/engine.rb:80:in `run_configure'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/supervisor.rb:641:in `run_supervisor'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/lib/fluent/command/fluentd.rb:341:in `<top (required)>'
fluentd_1        | 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1        | 	from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
fluentd_1        | 	from /usr/local/bundle/gems/fluentd-1.12.0/bin/fluentd:8:in `<top (required)>'
fluentd_1        | 	from /usr/local/bundle/bin/fluentd:23:in `load'
fluentd_1        | 	from /usr/local/bundle/bin/fluentd:23:in `<main>'
main_fluentd_1 exited with code 1

Additional context

No response

[prein]

Related: https://stackoverflow.com/questions/71120621/efk-system-is-build-on-docker-but-fluentd-cant-start-up

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[ashie]

The document is outdated since it's for fluentd v0.12 which is already deprecated.
Please see fluent/fluentd-docs-gitbook#391