Skip to content
This repository was archived by the owner on Aug 23, 2023. It is now read-only.

Doc updates #927

Merged
merged 2 commits into from
Jun 7, 2018
Merged

Doc updates #927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3837841
consolidation doc fixes
Dieterbe May 30, 2018
f3fdae4
update wording in readme, conveying no longer an alpha project
Dieterbe May 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 33 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,51 +1,53 @@
# Metrictank

[![Circle CI](https://circleci.com/gh/grafana/metrictank.svg?style=shield)](https://circleci.com/gh/grafana/metrictank)
[![Go Report Card](https://goreportcard.com/badge/github.com/grafana/metrictank)](https://goreportcard.com/report/github.com/grafana/metrictank)
[![GoDoc](https://godoc.org/github.com/grafana/metrictank?status.svg)](https://godoc.org/github.com/grafana/metrictank)

## Introduction

## introduction
*Metrictank is a cassandra-backed, metrics2.0 based, multi-tenant timeseries database for Graphite and friends*
Metrictank is a multi-tenant timeseries engine for Graphite and friends.
It provides long term storage, high availability, efficient storage, retrieval and processing for large scale environments.

metrictank is a timeseries database, inspired by the [Facebook gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf).
Most notably, it employs the compression mechanism described in the paper to dramatically lower storage overhead,
as well as data chunking to lower the load on cassandra. Graphite users are first-class citizens.
[GrafanaLabs](http://grafana.com) has been running metrictank in production since December 2015.
It currently requires an external datastore like Cassandra, and we highly recommend using Kafka to support clustering, as well
as a clustering manager like Kubernetes. This makes it non-trivial to operate, though GrafanaLabs has an on-premise product
that makes this process much easier.

## status
## Features

While [GrafanaLabs](http://grafana.com) has been running metrictank in production since december 2015, there are still plenty of kinks to work out
and bugs to fix. It should be considered an *alpha* project.
* 100% open source
* Inspired by the [Facebook gorilla paper](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf).
Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements.
* Writeback RAM cache, serving most data out of memory.
* Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
* Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)
* Can also be used with Prometheus (but the experience won't be as good as something built just for prometheus, which we're also working on)
* Accurate, flexible rollups by storing min/max/sum/count (which also gives us average).
So we can do consolidation (combined runtime+archived) accurately and correctly,
[unlike most other graphite backends like whisper](https://grafana.com/blog/2016/03/03/25-graphite-grafana-and-statsd-gotchas/#runtime.consolidation)
* Flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
* Input options: carbon, metrics2.0, kafka, Prometheus (soon: json or msgpack over http)
* Guards against excessive data requests

## limitations
## Limitations

* no performance/availability isolation between tenants per instance. (only data isolation)
* minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
* Backlog replaying and query responses, should be faster. [A Go GC issue may occasionally inflate response times](https://github.com/golang/go/issues/14812).
* we use metrics2.0 in native input protocol and indexes, but [barely do anything with it yet](https://github.com/grafana/metrictank/blob/master/docs/tags.md).
* No performance/availability isolation between tenants per instance. (only data isolation)
* Minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
* Backlog replaying and queries can be made faster. [A Go GC issue may occasionally inflate response times](https://github.com/golang/go/issues/14812).
* We use metrics2.0 in native input protocol and indexes, but [barely do anything with it yet](https://github.com/grafana/metrictank/blob/master/docs/tags.md).
* can't overwrite old data. We support reordering the most recent time window but that's it. (unless you restart MT)

## interesting design characteristics (feature or limitation.. up to you)
## Interesting design characteristics (feature or limitation... up to you)

* upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and re-assigning the primary role.
* Upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and possibly re-assigning the primary role.
Otherwise data loss of current chunks will be incurred. See [operations guide](https://github.com/grafana/metrictank/blob/master/docs/operations.md)
* clustering works best with an orchestrator like kubernetes. MT itself does not automate master promotions. See [clustering](https://github.com/grafana/metrictank/blob/master/docs/clustering.md) for more.
* only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression),
* Only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression),
No text support.
* only uint32 unix timestamps in second resolution. For higher resolution, consider [streaming directly to grafana](https://grafana.com/blog/2016/03/31/using-grafana-with-intels-snap-for-ad-hoc-metric-exploration/)
* no data locality: doesn't seem needed yet to put related series together.
* Only uint32 unix timestamps in second resolution. For higher resolution, consider [streaming directly to grafana](https://grafana.com/blog/2016/03/31/using-grafana-with-intels-snap-for-ad-hoc-metric-exploration/)
* No data locality: doesn't seem needed yet to put related series together.

## main features

* 100% open source
* graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
* can also be used with Prometheus (but the experience won't be as good as something built just for prometheus, which we're also working on)
* accurate, flexible rollups by storing min/max/sum/count (which also gives us average).
So we can do consolidation (combined runtime+archived) accurately and correctly,
[unlike most other graphite backends like whisper](https://grafana.com/blog/2016/03/03/25-graphite-grafana-and-statsd-gotchas/#runtime.consolidation)
* metrictank acts as a writeback RAM cache for recent data.
* flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
* input options: carbon, metrics2.0, kafka. (soon: json or msgpack over http)
* guards against excessive data requests
* efficient data compression and efficient use of Cassandra.

## Docs

Expand Down
8 changes: 2 additions & 6 deletions docs/consolidation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,8 @@ It supports min, max, sum, average.
Metrictank uses a function called `alignRequests` which will:

* look at all the requested timeseries and the time range
* figure out the optimal input data (out of raw and rollups) and wether or not runtime consolidation is needed, and which settings to use if so
* so that we can return all series in the same, optimal interval. Optimal meaning highest resolution but still <= maxDataPoints, using least system resources.

Note:
* this function ignores TTL values. It is assumed you've configured sensible TTL's that are long enough to cover the timeframes during which the given bands will be chosen.
* metrictank supports different raw intervals for different metrics, but aggregation (rollup) settings are currently global for all metrics.
* figure out the optimal input data (out of raw and rollups) and whether or not runtime consolidation is needed, and which settings to use if so
* so that we can return all series in the same, but as-high-resolution-as-possible interval (covering the timerange), subject to maxPointsPerReq{Hard,Soft} settings

Terminology:

Expand Down