Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: updated changelog and references to 2.3 #4112

Merged
merged 2 commits into from
Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
299 changes: 293 additions & 6 deletions CHANGELOG.md

Large diffs are not rendered by default.

10 changes: 2 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,8 @@ Loki differs from Prometheus by focusing on logs instead of metrics, and deliver

### Documentation

* [master](https://grafana.com/docs/loki/latest/)
* [v1.5.0](https://github.com/grafana/loki/tree/v1.5.0/docs/README.md)
* [v1.4.1](https://github.com/grafana/loki/tree/v1.4.1/docs/README.md)
* [v1.4.0](https://github.com/grafana/loki/tree/v1.4.0/docs/README.md)
* [v1.3.0](https://github.com/grafana/loki/tree/v1.3.0/docs/README.md)
* [v1.2.0](https://github.com/grafana/loki/tree/v1.2.0/docs/README.md)
* [v1.1.0](https://github.com/grafana/loki/tree/v1.1.0/docs/README.md)
* [v1.0.0](https://github.com/grafana/loki/tree/v1.0.0/docs/README.md)
* [Most Recent Release](https://grafana.com/docs/loki/latest/)
* [Unreleased](https://grafana.com/docs/loki/next/)

Commonly used sections:

Expand Down
18 changes: 9 additions & 9 deletions docs/sources/installation/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ For production, we recommend installing with Tanka or Helm.
Copy and paste the commands below into your command line.

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.2.1 -config.file=/mnt/config/loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.2.1 -config.file=/mnt/config/promtail-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.3.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.3.0 -config.file=/mnt/config/loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.3.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
docker run -v $(pwd):/mnt/config -v /var/log:/var/log grafana/promtail:2.3.0 -config.file=/mnt/config/promtail-config.yaml
```

When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files.
Expand All @@ -36,10 +36,10 @@ Copy and paste the commands below into your terminal. Note that you will need to

```bash
cd "<local-path>"
wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run -v <local-path>:/mnt/config -p 3100:3100 grafana/loki:2.2.1 --config.file=/mnt/config/loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
docker run -v <local-path>:/mnt/config -v /var/log:/var/log grafana/promtail:2.2.1 --config.file=/mnt/config/promtail-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.3.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run -v <local-path>:/mnt/config -p 3100:3100 grafana/loki:2.3.0 --config.file=/mnt/config/loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.3.0/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
docker run -v <local-path>:/mnt/config -v /var/log:/var/log grafana/promtail:2.3.0 --config.file=/mnt/config/promtail-config.yaml
```

When finished, `loki-config.yaml` and `promtail-config.yaml` are downloaded in the directory you chose. Docker containers are running Loki and Promtail using those config files.
Expand All @@ -51,6 +51,6 @@ Navigate to http://localhost:3100/metrics to view the output.
Run the following commands in your command line. They work for Windows or Linux systems.

```bash
wget https://raw.githubusercontent.com/grafana/loki/v2.2.1/production/docker-compose.yaml -O docker-compose.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.3.0/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up
```
2 changes: 1 addition & 1 deletion docs/sources/maintaining/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ Loki docs are versioned. Follow the below steps to version Loki docs for this re
>NOTE: Here $LOCAL_LOKI_PATH is your local path where Loki is checked out with correct $VERSION

1. Clone Grafana website [repo](https://github.com/grafana/website)
1. Create new branch `git checkout -b $VERSION` (replace `$VERSION` with current release version. e.g: `v2.2.1`)
1. Create new branch `git checkout -b $VERSION` (replace `$VERSION` with current release version. e.g: `v2.3.0`)
1. Run `mv content/docs/loki/next content/docs/loki/next.main`
1. Run `mkdir content/docs/loki/next`
1. Run `cp -R $LOCAL_LOKI_PATH/docs/sources/* content/docs/loki/next`
Expand Down
85 changes: 75 additions & 10 deletions docs/sources/release-notes/v2-3.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,86 @@ title: V2.3

# Version 2.3 release notes

Prose here about 2.3.
The Loki team is excited to announce the release of Loki 2.3!

It's been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.

2.3 is also the first version of Loki released under the AGPLv3 license. You can [read more about our licensing here](https://grafana.com/licensing/).

Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details please read [LICENSING.md](https://github.com/grafana/loki/blob/main/LICENSING.md).

## Features and enhancements

List of features here.
* Loki now has the ability to apply [custom retention](../../operations/storage/retention/) based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.
* Coming along hand in hand with storing logs for longer durations is the ability to [delete log streams](../../operations/storage/logs-deletion/). The initial implementation lets you submit delete request jobs which will be processed after 24 hours.
* A very exciting new LogQL parser has been introduced: the [pattern parser](../../logql/#parser-expression). Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the [Common Log Format](https://en.wikipedia.org/wiki/Common_Log_Format). This is now Loki's fastest parser so try it out on any of your log lines!
* Extending on the work of Alerting Rules, Loki now accepts [recording rules](../../rules/#recording-rules). This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.
* LogQL can understand [IP addresses](../../logql/ip/)! This enables filtering on IP addresses and subnet ranges.

For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.

* We created an [index gateway](../../operations/storage/boltdb-shipper/#index-gateway) which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.
* Ingester queriers [are now shardable](https://github.com/grafana/loki/pull/3852), this is a significant performance boost for high volume log streams when querying recent data.
* Instant queries can now be [split and sharded](https://github.com/grafana/loki/pull/3984) making them just as fast as range queries.

A very common feature requested has also been included in 2.3:

* [3919](https://github.com/grafana/loki/pull/3919) **github-vincent-miszczak**: Add ingester.autoforget-unhealthy-timeout opt-in feature

Without revisiting the decisions and discussions around the somewhat controversial behavior of _unhealthy_ ingesters, you can now decided how you would like them to be handled: manually or automatically.

Lastly several useful additions to the LogQL query language have been included:

* More text/template functions are included for `label_format` and `line_format` with PR [3515](https://github.com/grafana/loki/pull/3515), please check out the [documentation for template functions](https://grafana.com/docs/loki/latest/logql/template_functions/).
* Also support for math functions withing `label_format` and `line_format` was included with [3434](https://github.com/grafana/loki/pull/3434).
* Two additional metric functions with some interesting use cases `first_over_time` and `last_over_time` were added in PR [3050](https://github.com/grafana/loki/pull/3050). These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.

## Upgrade considerations

The path from 2.2.1 to 2.3.0 should be smooth, as always, please check out the [Upgrade Guide](https://github.com/grafana/loki/blob/master/docs/sources/upgrading/_index.md#230) for important upgrade guidance.

One change we consider noteworthy however is:

* [3216](https://github.com/grafana/loki/pull/3216) **sandeepsukhani**: check for stream selectors to have at least one equality matcher.

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

`{namespace=~".*"}`

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

`{cluster="us-east-1",namespace=~".*"}`

Use `.+` instead of `.*`

`{namespace=~".+"}`

This difference may seem subtle but if we break it down `.` matches any character, `*` matches zero or more of the preceding character and `+` matches one or more of the preceding character. The `.*` case will match empty values where `.+` will not, this is the important difference. `{namespace=""}` is an invalid request (unless you add another equals label matcher like the example above).

The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.



## Security fixes

List of security fixes for 2.3.x.

### 2.3.0 security fixes

2.3.0 contains an important security fix:

* [4020](https://github.com/grafana/loki/pull/4020) **simonswine**: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).

**Note** Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an `X-Scope-OrgID` header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the `X-Scope-OrgID`.

- Pattern parser. [poor docs link to Parser expression](../../logql/#parser-expression)
- Windows event log scraping. [docs link](../../clients/promtail/scraping/#windows-event-log)
- Recording rules.
- Index Gateway.
- Retention.
- `promtail_instance` label.

## Bug fixes

Lists of bug fixes for v2.3.x.
Lists of bug fixes for 2.3.x.

### 2.3.0 bug fixes

### V2.3.0 bug fixes
* An important fix for leaking resources was patched with [3733](https://github.com/grafana/loki/pull/3733), when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.
* [3686](https://github.com/grafana/loki/pull/3686) fixes a panic with the frontend when use with downstream URL. **Note** we recommend using the [GRPC Pull Model](https://grafana.com/docs/loki/latest/configuration/query-frontend/#grpc-mode-pull-model), better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.
27 changes: 27 additions & 0 deletions docs/sources/upgrading/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,33 @@ If possible try to stay current and do sequential updates. If you want to skip v

-_add changes here which are unreleased_

## 2.3.0

### Loki

#### Query restriction introduced for queries which do not have at least one equality matcher

PR [3216](https://github.com/grafana/loki/pull/3216) **sandeepsukhani**: check for stream selectors to have at least one equality matcher

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

`{namespace=~".*"}`

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

`{cluster="us-east-1",namespace=~".*"}`

Use `.+` instead of `.*`

`{namespace=~".+"}`

This difference may seem subtle but if we break it down `.` matches any character, `*` matches zero or more of the preceding character and `+` matches one or more of the preceding character. The `.*` case will match empty values where `.+` will not, this is the important difference. `{namespace=""}` is an invalid request (unless you add another equals label matcher like the example above)

The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.


## 2.2.0

### Loki
Expand Down
4 changes: 2 additions & 2 deletions production/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ networks:

services:
loki:
image: grafana/loki:2.2.1
image: grafana/loki:2.3.0
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki

promtail:
image: grafana/promtail:2.2.1
image: grafana/promtail:2.3.0
volumes:
- /var/log:/var/log
command: -config.file=/etc/promtail/config.yml
Expand Down