Skip to content

Latest commit

 

History

History
111 lines (91 loc) · 6.44 KB

README.md

File metadata and controls

111 lines (91 loc) · 6.44 KB
Raw
Deis Workflow is no longer maintained.
Please read the announcement for more detail.
09/07/2017 Deis Workflow v2.18 final release before entering maintenance mode
03/01/2018 End of Workflow maintenance: critical patches no longer merged
Hephy is a fork of Workflow that is actively developed and accepts code contributions.

Deis Logger

Build Status codecov.io Go Report Card Docker Repository on Quay

Deis (pronounced DAY-iss) Workflow is an open source Platform as a Service (PaaS) that adds a developer-friendly layer to any Kubernetes cluster, making it easy to deploy and manage applications on your own servers.

Deis Graphic

For more information about the Deis Workflow, please visit the main project page at https://github.com/deis/workflow.

We welcome your input! If you have feedback, please submit an issue. If you'd like to participate in development, please read the "Development" section below and submit a pull request.

Description

A system logger for use in the Deis Workflow open source PaaS.

This Docker image is based on quay.io/deis/base image. You can see what version we are currently using in the Dockerfile

The new v2 logger implementation has seen a simplification from the last rewrite. While it still uses much of that code it no longer depends on etcd. Instead, we will use kubernetes service discovery to determine where logger is running.

We have also decided to not use logspout as the mechanism to get logs from each container to the logger component. Now we will use fluentd which is a widely supported logging framework with hundreds of plugins. This will allow the end user to configure multiple destinations such as Elastic Search and other Syslog compatible endpoints like papertrail.

Configuration

The following environment variables can be used to configure logger:

Name Default Value
STORAGE_ADAPTER "redis"
NUMBER_OF_LINES (per app) "1000"
AGGREGATOR_TYPE "nsq"
DEIS_NSQD_SERVICE_HOST ""
DEIS_NSQD_SERVICE_PORT_TRANSPORT 4150
NSQ_TOPIC logs
NSQ_CHANNEL consume
NSQ_HANDLER_COUNT 30
AGGREGATOR_STOP_TIMEOUT_SEC 1
DEIS_LOGGER_REDIS_SERVICE_HOST ""
DEIS_LOGGER_REDIS_SERVICE_PORT 6379
DEIS_LOGGER_REDIS_PASSWORD ""
DEIS_LOGGER_REDIS_DB 0
DEIS_LOGGER_REDIS_PIPELINE_LENGTH 50
DEIS_LOGGER_REDIS_PIPELINE_TIMEOUT_SECONDS 1

Development

The only assumption this project makes about your environment is that you have a working docker host to build the image against.

Building binary and image

To build the binary and image run the following make command:

DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make build
IMAGE_PREFIX=myaccount make build
DEV_REGISTRY=myhost:5000 make build

Pushing the image

The makefile assumes that you are pushing the image to a remote repository like quay or dockerhub. So you will need to supply the REGISTRY environment variable.

DEV_REGISTRY=quay.io IMAGE_PREFIX=myaccount make push
IMAGE_PREFIX=myaccount make push
DEV_REGISTRY=myhost:5000 make push

Kubernetes interactions

  • make install - Install the recently built docker image into the kubernetes cluster
  • make upgrade - Upgrade a currently installed image
  • make uninstall - Uninstall logger from a kubernetes cluster

Architecture Diagram

                        ┌────────┐
                        │ Router │                  ┌────────┐
                        └────────┘                  │ Logger │
                            │                       └────────┘
                        Log file                        │
                            │                           │
                            ▼                           ▼
┌────────┐             ┌─────────┐    logs/metrics   ┌─────┐
│App Logs│──Log File──▶│ fluentd │───────topics─────▶│ NSQ │
└────────┘             └─────────┘                   └─────┘
                                                        │
                                                        │
┌─────────────┐                                         │
│ HOST        │                                         ▼
│  Telegraf   │───┐                                ┌────────┐
└─────────────┘   │                                │Telegraf│
                  │                                └────────┘
┌─────────────┐   │                                    │
│ HOST        │   │    ┌───────────┐                   │
│  Telegraf   │───┼───▶│ InfluxDB  │◀────Wire ─────────┘
└─────────────┘   │    └───────────┘   Protocol
                  │          ▲
┌─────────────┐   │          │
│ HOST        │   │          ▼
│  Telegraf   │───┘    ┌──────────┐
└─────────────┘        │ Grafana  │
                       └──────────┘