Dynamic increasing/decreasing servers of out_forward

[toyama0919]

Hi.

I wanted auto scale Aggregator for out_forward plugin.

I think server list can dynamically get with plugin.

Example

<match **>
  type forward
  buffer_type file
  buffer_path /var/log/td-agent/buffer/aggregate
  flush_interval 60s

  # dynamically define
  <servers>
    type ec2
    region ap-northeast-1
    port 24224
    tag { "Stages":"production", "Roles":"fluentd-aggregator" }
  </servers>

  # static define
  <server>
    host aggregator01
    port 24224
  </server>
</match>

Plugin Example(Amazon EC2 Plugin)

module Fluent
  class Ec2Servers < Servers
    Plugin.register_servers('ec2', self)

    include Configurable

    config_param :region, :string
    config_param :port, :string
    config_param :tag, :hash, :default => {}

    def initialize
      super
      require 'aws-sdk'
    end

    def configure(conf)
      super
      ....
    end

    def servers
      nodes = []

      AWS.ec2.client.describe_instances.each do |i|
        ##
        ## some process
        ##

        failure = FailureDetector.new(@heartbeat_interval, @hard_timeout, Time.now.to_i.to_f)

        node_conf = NodeConfig.new(ec2_name_tag, ec2_private_ip_address, @port, weight, standby, failure,
          @phi_threshold, recover_sample_size, @expire_dns_cache, @phi_failure_detector, @dns_round_robin)

        if @heartbeat_type == :none
          nodes << NoneHeartbeatNode.new(log, node_conf)
        else
          nodes << Node.new(log, node_conf)
        end
      end
      nodes
    end
  end
end

Other plugin Example

• gce
• azure
• softlayer
• sakura

Not need restart is Best, but need restart is simply😊

[tagomoris]

If you can restart Fluentd processes, following 2 options seem to satisfy your requirements:

• use template engine (ex: erb) to generate configuration files w/ fetching recent node list by ruby snippet
• use Ruby DSL to write configurations

How do you think about these?

[sonots]

DNS round robin probably satisfies most of cases, and it is already available.

[toyama0919]

Thank you for reply😊

Sorry , Missing description.
My Use case is during the morning is High traphic.
Afternoon is middle traphic.

@tagomoris

use template engine (ex: erb) to generate configuration files w/ fetching recent node list by ruby snippet.

But this is need deploy.
Every deploy is High cost...

use Ruby DSL to write configurations

Ruby DSL is not defacto standard.
Ruby DSL Will it be supported in future?

@sonots
DNS round robin is well.
But needs ELB for AWS.

These use case is demand think Auto scale..
http://hb.matsumoto-r.jp/entry/2015/11/04/155848
http://qiita.com/sfujiwara/items/f8b5e004abff0c7c36aa

If possible, I create pull request..
Do you think?

sorry..😭

[sonots]

ELB is not required for DNS round robin, but DNS is required. AWS provides route53, but dnsmasq is a simple dns server.

Your request requires new kinds of plugins, we must think of it carefully. At least, I think such plugin should provide auto-refresh without restarting.

[toyama0919]

At least, I think such plugin should provide auto-refresh without restarting.

I think so too..
But DNS round robin is not dynamic Increase and decrease aggregator.