Hosted on Docker Hub
This container is extremely ECS-friendly, although it may be used for other environments.
| Environment Variable | Purpose |
|---|---|
| DEBUG | Set to true if local development or non EC2 instance host |
| LOG_ENVIRONMENT | Logging environment name (e.g. QA or Production). This will be a field in the resulting log entries. Defaults to development if not set |
| AWS_REGION | Set AWS region for CloudWatch. Defaults to us-east-1 if not set |
| AWS_ACCESS_KEY_ID | AWS Access Key (Optimally, you can instead use Instance Profiles |
| AWS_SECRET_ACCESS_KEY | AWS Secret Access Key (Optimally, you can instead use Instance Profiles |
By running this container with the following command, one can aggregate the logs of Docker containers running on the same host:
docker run -d --restart=always -e "LOG_ENVIRONMENT=qa" -v /var/lib/docker/containers:/var/lib/docker/containers -v /var/run/docker.sock:/var/run/docker.sock:ro --name=fluentd nritholtz/docker-fluentdThe container logs will forwarded to CloudWatch, with the log_group_name as the Task definition's family, followed by the set LOG_ENVIRONMENT environment variable. The log_stream_name will be the ECS container's name, followed by the container's ID (this is necessary due to issues with having more than 1 source for a single log_stream, e.g. the stored sequence parameter).
The outputted entries looks exactly like Docker's JSON formatted logs, except it will contain additional fields depending on which environment you are using for starting up your other Docker containers.
The following scenarios are supported and evaluated in the order that they appear:
| Environment | Results |
|---|---|
| ECS | Log entries will use the running ECS Task Definition's Family and container name for the container_project and container_name log fields, respectively. In addition, a field task_definition_version will be added to the entries as well. |
| Docker Compose | Log entries will use the running Docker Compose's Project and Service for the container_project and container_name log fields, respectively. |
| Default | If container does not fall within either of the previous categories, will not add either of the fields. |
Docker Compose example
{"log":"2015-10-15T11:44:27Z 54 TID-gomn1ol8g Appworker/Worker/4cab697c-7e67-40ee-a443-b8f52a034183 INFO: completed in: 232.16888 ms\n","stream":"stdout","container_image":"app_web","container_id":"a8c0128c0b47d5a7a5ed702be3c5f57cd62dc73371d5d7743b63a49ac1e09074","container_project":"app","container_service":"worker","time": "2015-10-23T13:00:42Z","ip_address": "10.212.151.112"}docker-fluentd uses Fluentd inside to tail log files that are mounted on /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log. It uses the tail input plugin to tail JSON-formatted log files that each Docker container emits.
Then, Fluentd adds the additional fields using the included record_transformer plugin, before writing out to local files using the file output plugin. It will also using the Fluentd CloudWatch Plugin when sending logs to CloudWatch.
In addition, docker-fluentd registers any new containers started from the Docker service on the same host, by using Docker Gen. All of which are managed by a supervisord process.
Fluentd has a lot of plugins and can probably support your data output destination.