Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: clarify distinction between single binary and microservices. #4374

Merged
merged 3 commits into from
Sep 29, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 19 additions & 22 deletions docs/sources/architecture/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,50 +4,47 @@ weight: 1000
---
# Loki's Architecture

This document will expand on the information detailed in the [Loki
Overview](../overview/).
## Multi-tenancy

## Multi Tenancy

All data - both in memory and in long-term storage - is partitioned by a
All data, both in memory and in long-term storage, may be partitioned by a
tenant ID, pulled from the `X-Scope-OrgID` HTTP header in the request when Loki
is running in multi-tenant mode. When Loki is **not** in multi-tenant mode, the
header is ignored and the tenant ID is set to "fake", which will appear in the
index and in stored chunks.

## Modes of Operation
## Modes of operation

![modes_diagram](modes_of_operation.png)

Loki has a set of components (defined below in [Components](#components)) which
Loki has a set of [components](#components), which
are internally referred to as modules. Each component spawns a gRPC server for
internal traffic and an HTTP/1 server for external API requests. All components
come with an HTTP/1 server, but most only expose readiness, health, and metrics
endpoints.

Which component Loki runs is determined by either the `-target` flag at the
command line or the `target: <string>` section in Loki's config file. When the
value of `target` is `all`, Loki will run all of its components in a single
process. This is referred to as "single process", "single binary", or monolithic
mode. Monolithic mode is the default deployment of Loki when Loki is installed
using Helm.
Loki can run as a single binary; all components are part of the same process.
Or, Loki can run components as microservices.
As microservices, the cluster is horizontally scalable.

When `target` is _not_ set to `all` (i.e., it is set to `querier`, `ingester`,
`query-frontend`, or `distributor`), then Loki is said to be running in "horizontally scalable",
or microservices, mode.
When invoked, the `-target` flag on the
command line or the `target: <string>` configuration determines
the components' mode: single binary or microservices.
A `target` value of `all` runs Loki in single binary mode.
A `target` value of one of the components invokes that component
as its own microservice.

Each component of Loki, such as the ingesters and distributors, communicate with
one another over gRPC using the gRPC listen port defined in the Loki config.
When running components in monolithic mode, this is still true: each component,
one another over gRPC using the gRPC listen port defined in the Loki configuration.
When running components as a single binary, this is still true: each component,
although running in the same process, will connect to each other over the local
network for inter-component communication.

Single process mode is ideally suited for local development, small workloads,
and for evaluation purposes. Monolithic mode can be scaled with multiple
The single process mode is ideally suited for local development, small workloads,
and for evaluation purposes. The single binary mode can be scaled to multiple
processes with the following limitations:

1. Local index and local storage cannot currently be used when running
monolithic mode with more than one replica, as each replica must be able to
1. Local index and local storage cannot currently be used
with more than one replica, as each replica must be able to
access the same storage backend, and local storage is not safe for concurrent
access.
1. Individual components cannot be scaled independently, so it is not possible
Expand Down
5 changes: 3 additions & 2 deletions docs/sources/configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,9 @@ Pass the `-config.expand-env` flag at the command line to enable this way of set
### Supported contents and default values of `loki.yaml`

```yaml
# The module to run Loki with. Supported values
# all, compactor, distributor, ingester, querier, query-frontend, table-manager.
# A comma-separated list of components to run. The default value "all" runs
# Loki in single binary mode.
# Supported values: all, compactor, distributor, ingester, querier, query-frontend, table-manager.
[target: <string> | default = "all"]

# Enables authentication through the X-Scope-OrgID header, which must be present
Expand Down
3 changes: 3 additions & 0 deletions docs/sources/installation/docker.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
---
title: Docker
weight: 30
---
# Install Loki with Docker or Docker Compose

You can install Loki and Promtail with Docker or Docker Compose if you are evaluating, testing, or developing Loki.
For production, we recommend installing with Tanka or Helm.

The configuration acquired with these installation instructions run Loki as a single binary.

## Prerequisites

- [Docker](https://docs.docker.com/install)
Expand Down
3 changes: 3 additions & 0 deletions docs/sources/installation/helm.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
---
title: Helm
weight: 20
---
# Install Loki with Helm

The Helm installation runs the Loki cluster as a single binary.

## Prerequisites

Make sure you have Helm [installed](https://helm.sh/docs/using_helm/#installing-helm).
Expand Down
1 change: 1 addition & 0 deletions docs/sources/installation/install-from-source.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: Build from source
weight: 50
---
# Build from source

Expand Down
7 changes: 5 additions & 2 deletions docs/sources/installation/local.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
---
title: Local
weight: 40
---
# Install and run Loki locally

In order to log events with Loki, you must download and install both Promtail and Loki.
In order to log events with Loki, download and install both Promtail and Loki.
- Loki is the logging engine.
- Promtail sends logs to Loki.

## Install and run
The configuration specifies running Loki as a single binary.

## Install

1. Navigate to the [release page](https://github.com/grafana/loki/releases/).
2. Scroll down to the Assets section under the version that you want to install.
Expand Down
3 changes: 3 additions & 0 deletions docs/sources/installation/tanka.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
---
title: Tanka
weight: 10
---
# Install Loki with Tanka

[Tanka](https://tanka.dev) is a reimplementation of
[Ksonnet](https://ksonnet.io) that Grafana Labs created after Ksonnet was
deprecated. Tanka is used by Grafana Labs to run Loki in production.

The Tanka installation runs the Loki cluster in microservices mode.

## Prerequisites

Install the latest version of Tanka (version v0.17.1 or a more recent version) for the `tk env`
Expand Down
16 changes: 7 additions & 9 deletions docs/sources/overview/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ and allows for efficient query execution.
- **Multi-tenancy**

Loki allows multiple tenants to utilize a single Loki instance.
The data of distinct tenants is completely isolated from other tentants.
The data of distinct tenants is completely isolated from other tenants.
Multi-tenancy is configured by assigning a tenant ID in the agent.

- **LogQL, Loki's query language**
Expand All @@ -53,15 +53,13 @@ and allows for efficient query execution.

- **Scalability**

Loki works well at small scale.
In single process mode, all required microservices run in one process.
Single process mode is great for testing Loki,
running it locally, or running it at a small scale.
Loki can be run as a single binary;
all the components run in one process.

Loki is also designed to scale out for large scale installations.
Each of the Loki's microservice components can be broken out into
separate processes, and configuration permits individual scaling
of the components.
Loki is designed for scalability,
as each of Loki's components can be run as microservices.
Configuration permits scaling the microservices individually,
permitting flexible large-scale installations.

- **Flexibility**

Expand Down