Loki HTTP/JSON Model Layer #1022

joe-elliott · 2019-09-16T19:42:40Z

What this PR does / why we need it:
This PR adds an http model layer between the internal protobuf components and json/http representations. This gives Loki the flexibility to support multiple json/http representations of internal objects. Ultimately this allows Loki to support the existing "legacy" protocol while also migrating forward to the prometheus compatible "v1" protocol.

v1 endpoints:

/loki/api/v1/query_range
/loki/api/v1/query
/loki/api/v1/label
/loki/api/v1/label/{name}/value
/loki/api/v1/tail

legacy endpoints:

/api/prom/query
/api/prom/label
/api/prom/label/{name}/values
/api/prom/tail

Response formats are documented in the tests.

Which issue(s) this PR fixes:
Fixes #273
It also partially addresses #1007

Special notes for your reviewer:
notes:

formats for all endpoints are documented and tested under pkg/logql/marshal and pkg/logql/marshal/legacy
old endpoints have been manually tested with logcli and grafana
logcli updated to use new model objects

Checklist

Documentation added
Tests updated

rfratto

Idiomatically I think the pkg/loghttp/v1 package should be just pkg/loghttp, you generally don't introduce a version path unless it's not version 1. (and when you do, the package name at pkg/loghttp/v2 would still be loghttp)

rfratto

This is looking really good! Sorry about all the nitpicky comments.

I don't know if you were planning on it already but I'd (personally) like for all the new exported functions to use valid godoc strings.

pkg/loghttp/legacy/marshal.go

pkg/loghttp/legacy/marshal_test.go

pkg/loghttp/legacy/query.go

pkg/loghttp/v1/query.go

pkg/querier/http.go

sh0rez · 2019-09-18T14:50:27Z

CI builds are failing with undefined references, maybe check generated protos or so?

joe-elliott · 2019-09-19T19:11:11Z

Editing the original comment to reflect the state of this PR.

pkg/logql/marshal/marshal.go

pkg/querier/tail.go

rfratto

LGTM! Although I may be a bit biased since I asked for a lot of changes. @cyriltovena could you take a second look?

cyriltovena · 2019-09-20T00:06:33Z

I will I think Joe is not done ;)

Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott · 2019-09-20T21:18:26Z

@cyriltovena @rfratto
I feel like this is complete. Removing WIP from title and updating Description.

Signed-off-by: Joe Elliott <number101010@gmail.com>

cyriltovena · 2019-09-23T14:58:45Z

pkg/logcli/query/query.go

-	d := q.resultsDirection()
-	i := iter.NewStreamsIterator(streams, d)
+	// sort and display entries
+	allEntries := make([]streamEntryPair, 0)


Good move, this wasn't really required as streams were already deduplicated by the querier.

cyriltovena

LGTM

/loki/api/v1/push was added in grafana#1010 but was temporarily reverted. grafana#1022 reintroduced changes from grafana#1010 but inadvertently did not reintroduce the new push endpoint.

@cyriltovena

* docs: create structure of docs overhaul This commit removes all old docs and lays out the table of contents and framework for how the new documentation will be intended to be read. * docs: add design docs back in * docs: add community documentation * docs: add LogQL docs * docs: port existing operations documentation * docs: add new placeholder file for promtail configuration docs * docs: add TOC for operations/storage * docs: add Loki API documentation * docs: port troubleshooting document * docs: add docker-driver documentation * docs: link to configuration from main docker-driver document * docs: update API for new paths * docs: fix broken links in api.md and remove json marker from examples * docs: incorporate api changes from #1009 * docs: port promtail documentation * docs: add TOC to promtail configuration reference * docs: fix promtail spelling errors * docs: add loki configuration reference * docs: add TOC to configuration * docs: add loki configuration example * docs: add Loki overview with brief explanation about each component * docs: add comparisons document * docs: add info on table manager and update storage/README.md * docs: add getting started * docs: incorporate config yaml changes from #755 * docs: fix typo in releases url for promtail * docs: add installation instructions * docs: add more configuration examples * docs: add information on fluentd client fluent-bit has been temporarily removed until the PR for it is merged. * docs: PR review feedback * docs: add architecture document * docs: add missing information from old docs * `localy` typo Co-Authored-By: Ed Welch <ed@oqqer.com> * docs: s/ran/run/g * Typo * Typo * Tyop * Typo * docs: fixed typo * docs: PR feedback * docs: @cyriltovena PR feedback * docs: add more details to promtail url config option * docs: expand promtail's pipelines document with extra detail * docs: remove reference to Stage interface in pipelines.md * docs: fixed some spelling * docs: clarify promtail configuration and scraping * docs: attempt #2 at explaining promtail's usage of machine hostname * docs: spelling fixes * docs: add reference to promtail custom metrics and fix silly typo * docs: cognizant -> aware * docs: typo * docs: typos * docs: add which components expose which API endpoints in microservices mode * docs: change ksonnet installation to tanka * docs: address most @pracucci feedback * docs: fix all spelling errors so reviewers don't have to keep finding them :) * docs: incorporate changes to API endpoints made in #1022 * docs: add missing loki metrics * docs: add missing promtail metrics * docs: @pstribrany feedback * docs: more @pracucci feedback * docs: move metrics into a table * docs: update push path references to /loki/api/v1/push * docs: add detail to further explain limitations of monolithic mode * docs: add alternative names to modes_of_operation diagram * docs: add log ordering requirement * docs: add procedure for updating docs with latest version * docs: separate out stages documentation into one document per stage * docs: list supported stores in storage documentation * docs: add info on duplicate log lines in pipelines * docs: add line_format as key feature to fluentd * docs: hopefully final commit :)

joe-elliott requested review from rfratto, cyriltovena and slim-bean September 16, 2019 19:44

rfratto reviewed Sep 16, 2019

View reviewed changes

rfratto mentioned this pull request Sep 19, 2019

Documentation Rewrite #982

Merged

joe-elliott force-pushed the http-client branch from df1c4d5 to 3450620 Compare September 19, 2019 18:59

rfratto reviewed Sep 19, 2019

View reviewed changes

pkg/logql/marshal/marshal.go Outdated Show resolved Hide resolved

rfratto reviewed Sep 19, 2019

View reviewed changes

pkg/querier/tail.go Outdated Show resolved Hide resolved

rfratto approved these changes Sep 19, 2019

View reviewed changes

joe-elliott added 17 commits September 20, 2019 17:10

First pass data model

fe23eff

Signed-off-by: Joe Elliott <number101010@gmail.com>

Use prom model b/c we're serializing promql objects

cd2dab3

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added legacy query support and tests

1ba0d7f

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added legacy label test

80d4aba

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added tail response marshalling and tests

bb72d46

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed marshallers and test

e39f920

Signed-off-by: Joe Elliott <number101010@gmail.com>

Expanded legacy test cases

26e9915

Signed-off-by: Joe Elliott <number101010@gmail.com>

Dropped streams nano test

528b08c

Signed-off-by: Joe Elliott <number101010@gmail.com>

First pass v1 new objects

9201f03

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added failing tail response tests

38ae5e9

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added failing tailresponse test

64f0a86

Signed-off-by: Joe Elliott <number101010@gmail.com>

Partial v1 tests

ab342ef

Signed-off-by: Joe Elliott <number101010@gmail.com>

Improved legacy labels test

fee712f

Signed-off-by: Joe Elliott <number101010@gmail.com>

Improved legacy query tests

d9efb3a

Signed-off-by: Joe Elliott <number101010@gmail.com>

Moved all legacy tests to new method

5f03769

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added v1 tests and fixed stream marshalling bug

8127ba5

Signed-off-by: Joe Elliott <number101010@gmail.com>

First pass new Model

a57f76e

Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott added 17 commits September 20, 2019 17:10

Cleaned up linting issues

7174fe7

Signed-off-by: Joe Elliott <number101010@gmail.com>

Minor comment cleanup

f4683ea

Signed-off-by: Joe Elliott <number101010@gmail.com>

Adjusted GOGC to make CircleCI happy

edea13f

Signed-off-by: Joe Elliott <number101010@gmail.com>

Changed legacy => loghttp for consistency

df2a539

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed matrix error message to be correct

e4e36fe

Signed-off-by: Joe Elliott <number101010@gmail.com>

Moved label query over to loghttp response

3c70563

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added marshal loop tests

1ea9b96

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added response type test

f22281f

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed tail response marshal/unmarshal

819874d

Signed-off-by: Joe Elliott <number101010@gmail.com>

Passing unmarshal/marshal queryresponse tests

1a7b475

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed vector and matrix

0398d9e

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added output support for streams minus ordering

24e1ae3

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed tailing

e555529

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed output tests

e47f605

Signed-off-by: Joe Elliott <number101010@gmail.com>

Fixed query tests

1596d37

Signed-off-by: Joe Elliott <number101010@gmail.com>

Order log output

3346cec

Signed-off-by: Joe Elliott <number101010@gmail.com>

Use labels instead of stream

45413c8

Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott force-pushed the http-client branch from 2b50653 to 45413c8 Compare September 20, 2019 21:10

joe-elliott changed the title ~~WIP: Loki HTTP/JSON Model Layer~~ Loki HTTP/JSON Model Layer Sep 20, 2019

Lowered parallelization for CircleCI

51c558a

Signed-off-by: Joe Elliott <number101010@gmail.com>

cyriltovena reviewed Sep 23, 2019

View reviewed changes

cyriltovena approved these changes Sep 23, 2019

View reviewed changes

cyriltovena merged commit 9614b02 into grafana:master Sep 23, 2019

rfratto mentioned this pull request Sep 23, 2019

loki: add back new /loki/api/v1/push endpoint #1046

Closed

rfratto added a commit to rfratto/loki that referenced this pull request Sep 25, 2019

docs: incorporate changes to API endpoints made in grafana#1022

d0026d5

joe-elliott mentioned this pull request Sep 27, 2019

Query endpoints fail with 404 #1079

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Loki HTTP/JSON Model Layer #1022

Loki HTTP/JSON Model Layer #1022

joe-elliott commented Sep 16, 2019 •

edited

Loading

rfratto left a comment

rfratto left a comment

sh0rez commented Sep 18, 2019

joe-elliott commented Sep 19, 2019

rfratto left a comment

cyriltovena commented Sep 20, 2019

joe-elliott commented Sep 20, 2019

cyriltovena Sep 23, 2019

cyriltovena left a comment

Loki HTTP/JSON Model Layer #1022

Loki HTTP/JSON Model Layer #1022

Conversation

joe-elliott commented Sep 16, 2019 • edited Loading

rfratto left a comment

Choose a reason for hiding this comment

rfratto left a comment

Choose a reason for hiding this comment

sh0rez commented Sep 18, 2019

joe-elliott commented Sep 19, 2019

rfratto left a comment

Choose a reason for hiding this comment

cyriltovena commented Sep 20, 2019

joe-elliott commented Sep 20, 2019

cyriltovena Sep 23, 2019

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

joe-elliott commented Sep 16, 2019 •

edited

Loading