Skip to content

Latest commit

 

History

History
232 lines (183 loc) · 16.1 KB

configuration.md

File metadata and controls

232 lines (183 loc) · 16.1 KB
title
Configuration

Configuring the Docker Driver

The Docker daemon on each machine has a default logging driver and each container will use the default driver unless configured otherwise.

Change the logging driver for a container

The docker run command can be configured to use a different logging driver than the Docker daemon's default with the --log-driver flag. Any options that the logging driver supports can be set using the --log-opt <NAME>=<VALUE> flag. --log-opt can be passed multiple times for each option to be set.

The following command will start Grafana in a container and send logs to Grafana Cloud, using a batch size of 400 entries and no more than 5 retries if a send fails.

docker run --log-driver=loki \
    --log-opt loki-url="https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push" \
    --log-opt loki-retries=5 \
    --log-opt loki-batch-size=400 \
    grafana/grafana

Note: The Loki logging driver still uses the json-log driver in combination with sending logs to Loki, this is mainly useful to keep the docker logs command working. You can adjust file size and rotation using the respective log option max-size and max-file. You can deactivate this behavior by setting the log option no-file to true.

Change the default logging driver

If you want the Loki logging driver to be the default for all containers, change Docker's daemon.json file (located in /etc/docker on Linux) and set the value of log-driver to loki:

{
  "debug": true,
  "log-driver": "loki"
}

Options for the logging driver can also be configured with log-opts in the daemon.json:

{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

Note: log-opt configuration options in daemon.json must be provided as strings. Boolean and numeric values (such as the value for loki-batch-size in the example above) must therefore be enclosed in quotes (").

After changing daemon.json, restart the Docker daemon for the changes to take effect. All newly created containers from that host will then send logs to Loki via the driver.

Configure the logging driver for a Swarm service or Compose

You can also configure the logging driver for a swarm service directly in your compose file. This also applies for docker-compose:

version: "3.7"
services:
  logger:
    image: grafana/grafana
    logging:
      driver: loki
      options:
        loki-url: "https://<user_id>:<password>@logs-prod-us-central1.grafana.net/loki/api/v1/push"

You can then deploy your stack using:

docker stack deploy my_stack_name --compose-file docker-compose.yaml

Or with docker-compose:

docker-compose -f docker-compose.yaml up

Once deployed, the Grafana service will send its logs to Loki.

Note: stack name and service name for each swarm service and project name and service name for each compose service are automatically discovered and sent as Loki labels, this way you can filter by them in Grafana.

Labels

Loki can received a set of labels along with log line. These labels are used to index log entries and query back logs using LogQL stream selector.

By default, the Docker driver will add the following labels to each log line:

  • filename: where the log is written to on disk
  • host: the hostname where the log has been generated
  • container_name: the name of the container generating logs
  • swarm_stack, swarm_service: added when deploying from Docker Swarm.

Custom labels can be added using the loki-external-labels, loki-pipeline-stages, loki-pipeline-stage-file, labels, env, and env-regex options. See the next section for all supported options.

Pipeline stages

While you can provide loki-pipeline-stage-file it can be hard to mount the configuration file to the driver root filesystem. This is why another option loki-pipeline-stages is available allowing your to pass a list of stages inlined.

The example docker-compose below configures 2 stages, one to extract level values and one to set it as a label:

version: "3"
services:
  nginx:
    image: grafana/grafana
    logging:
      driver: loki
      options:
        loki-url: http://host.docker.internal:3100/loki/api/v1/push
        loki-pipeline-stages: |
          - regex:
              expression: '(level|lvl|severity)=(?P<level>\w+)'
          - labels:
              level:
    ports:
      - "3000:3000"

Note the loki-pipeline-stages: | allowing to keep the indentation correct.

When using docker run you can also pass the value via a string parameter like such:

read -d '' stages << EOF
- regex:
     expression: '(level|lvl|severity)=(?P<level>\\\w+)'
- labels:
    level:
EOF

docker run --log-driver=loki \
    --log-opt loki-url="http://host.docker.internal:3100/loki/api/v1/push" \
    --log-opt loki-pipeline-stages="$stages" \
    -p 3000:3000 grafana/grafana

This is a bit more difficult as you need to properly escape bash special characters. (note \\\w+ for \w+)

Providing both loki-pipeline-stage-file and loki-pipeline-stages will cause an error.

Relabeling

You can use Prometheus relabeling configuration to modify labels discovered by the driver. The configuration must be passed as a YAML string like the pipeline stages.

For example the configuration below will rename the label swarm_stack and swarm_service to respectively namespace and service.

version: "3"
services:
  nginx:
    image: grafana/grafana
    logging:
      driver: loki
      options:
        loki-url: http://host.docker.internal:3100/loki/api/v1/push
        loki-relabel-config: |
          - action: labelmap
            regex: swarm_stack
            replacement: namespace
          - action: labelmap
            regex: swarm_(service)
    ports:
      - "3000:3000"

Supported log-opt options

To specify additional logging driver options, you can use the --log-opt NAME=VALUE flag.

Option Required? Default Value Description
loki-url Yes Loki HTTP push endpoint.
loki-external-labels No container_name={{.Name}} Additional label value pair separated by , to send with logs. The value is expanded with the Docker tag template format. (eg: container_name={{.ID}}.{{.Name}},cluster=prod)
loki-timeout No 10s The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
loki-batch-wait No 1s The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
loki-batch-size No 102400 The maximum size of a log batch to send.
loki-min-backoff No 100ms The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
loki-max-backoff No 10s The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
loki-retries No 10 The maximum amount of retries for a log batch.
loki-pipeline-stage-file No The location of a pipeline stage configuration file (example). Pipeline stages allows to parse log lines to extract more labels. see documentation
loki-pipeline-stages No The pipeline stage configuration provided as a string see and see documentation
loki-relabel-config No A Prometheus relabeling configuration allowing you to rename labels see
loki-tenant-id No Set the tenant id (http headerX-Scope-OrgID) when sending logs to Loki. It can be overrides by a pipeline stage.
loki-tls-ca-file No Set the path to a custom certificate authority.
loki-tls-cert-file No Set the path to a client certificate file.
loki-tls-key-file No Set the path to a client key.
loki-tls-server-name No Name used to validate the server certificate.
loki-tls-insecure-skip-verify No false Allow to skip tls verification.
loki-proxy-url No Proxy URL use to connect to Loki.
no-file No false This indicates the driver to not create log files on disk, however this means you won't be able to use docker logs on the container anymore. You can use this if you don't need to use docker logs and you run with limited disk space. (By default files are created)
keep-file No false This indicates the driver to keep json log files once the container is stopped. By default files are removed, this means you won't be able to use docker logs once the container is stopped.
max-size No -1 The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the docker log command working.
max-file No 1 The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
labels No Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container.
env No Comma-separated list of keys of environment variables to be included in message if they specified for a container.
env-regex No A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message.

Troubleshooting

Plugin logs can be found as docker daemon log. To enable debug mode refer to the Docker daemon documentation.

The standard output (stdout) of a plugin is redirected to Docker logs. Such entries are prefixed with plugin=.

To find out the plugin ID of the Loki logging driver, use docker plugin ls and look for the loki entry.

Depending on your system, location of Docker daemon logging may vary. Refer to Docker documentation for Docker daemon log location for your specific platform.