Amazon DynamoDB output plugin for Fluentd event collector

Requirements

fluent-plugin-influxdb	fluentd	ruby
>= 0.2.0	>= v0.14.0	>= 2.1
< 0.2.0	>= v0.12.0	>= 1.9

Installation

$ fluent-gem install fluent-plugin-dynamodb

Configuration

DynamoDB

First of all, you need to create a table in DynamoDB. It's easy to create via Management Console.

Specify table name, hash attribute name and throughput as you like. fluent-plugin-dynamodb will load your table schema and write event-stream out to your table.

Fluentd

<match dynamodb.**>
  @type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint https://dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>

• aws_key_id (optional) - AWS access key id. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
• aws_sec_key (optional) - AWS secret key. This parameter is required when your agent is not running on EC2 instance with an IAM Instance Profile.
• proxy_uri (optional) - your proxy url.
• dynamo_db_endpoint (required) - end point of dynamodb. see Regions and Endpoints
• dynamo_db_table (required) - table name of dynamodb.

TIPS

retrieving data

fluent-plugin-dynamo will add time attribute and any other attributes of record automatically. For example if you read apache's access log via fluentd, structure of the table will have been like this.

id (Hash Key)	time	host	path	method	referer	code	agent	size
"a937f980-b304-11e1-bc96-c82a14fffef2"	"2012-06-10T05:26:46Z"	"192.168.0.3"	"/index.html"	"GET"	"-"	"200"	"Mozilla/5.0"	"4286"
"a87fc51e-b308-11e1-ba0f-5855caf50759"	"2012-06-10T05:28:23Z"	"192.168.0.4"	"/sample.html"	"GET"	"-"	"200"	"Mozilla/5.0"	"8933"

Item can be retrieved by the key, but fluent-plugin-dynamo uses UUID as a primary key. There is no simple way to retrieve logs you want. By the way, you can write scan-filter with AWS SDK like this, but Hive on EMR is the best practice I think.

multiprocessing

If you need high throughput and if you have much provisioned throughput and abudant buffer, you can setup multiprocessing. fluent-plugin-dynamodb uses multi workers, so you can launch 6 workers as follows.

<match dynamodb.**>
  @type dynamodb
  aws_key_id AWS_ACCESS_KEY
  aws_sec_key AWS_SECRET_ACCESS_KEY
  proxy_uri http://user:password@192.168.0.250:3128/
  dynamo_db_endpoint https://dynamodb.ap-northeast-1.amazonaws.com
  dynamo_db_table access_log
</match>
<system>
  workers 6
</system>

multi-region redundancy

As you know fluentd has copy output plugin. So you can easily setup multi-region redundancy as follows.

<match dynamo.**>
  @type copy
  <store>
    @type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint https://dynamodb.ap-northeast-1.amazonaws.com
  </store>
  <store>
    @type dynamodb
    aws_key_id AWS_ACCESS_KEY
    aws_sec_key AWS_SECRET_ACCESS_KEY
    dynamo_db_table test
    dynamo_db_endpoint https://dynamodb.ap-southeast-1.amazonaws.com
  </store>
</match>

TODO

• auto-create table
• tag_mapped

Copyright

Copyright	Copyright (c) 2012- Takashi Matsuno
License	Apache License, Version 2.0