Collapse multiline logs based on a start line. #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cyriltovena
reviewed
Nov 26, 2020
cyriltovena
reviewed
Nov 26, 2020
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* doc: Add missing backticks Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Use absolute path to positions.yaml file Signed-off-by: Jack Baldry <jack.baldry@grafana.com> * doc: Remove .. from URL paths I'm not sure what the `..` means exactly. Signed-off-by: Jack Baldry <jack.baldry@grafana.com>
* Introducing go pipelines to promtail. Based off @jeschkies idea. WIP Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * More tests and code cleanup. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Wip breaking things. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixing tests, adding Stop to the interface. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes all tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes more test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Moar fixes for tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Close correctly client before reading. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use defer. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add some comments. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes lint issues Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Attempt to fix journald without linux. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fixes journald json test. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Add missing stop in the filetarget. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Update pkg/logentry/stages/match_test.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * First set of feeback review fixes. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Use newEntry as suggested by Karsten. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * Fix tests. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> * lint. Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>
Summary: This is a very simple approach based on grafana#1380 to provide multiline or block log entries in promtail. A `multiline` stage is added to pipelines. This stages matches a start line. Once a start line is matched all following lines are appended to an entry and not passed on to downstream stages. Once a new start line is matched the former block of multilines is sent. If now new line arrives withing `max_wait_time` the block is flushed to the next stage and a new block is started.
jeschkies
force-pushed
the
karsten/multiline-2
branch
from
6040b4d
to
dc715cb
Compare
|
Moved to grafana#3024 |
jeschkies
referenced
this pull request
in jeschkies/loki
Dec 21, 2021
* linting, cleaning up * CI: update golangci-lint version, remove go test -v flag * linting, add gofumpt and golangci-lint to makefile
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This is a very simple approach based on grafana#1380 to provide multiline
or block log entries in promtail.
A
multilinestage is added to pipelines. This stages matches a startline. Once a start line is matched all following lines are appended
to an entry and not passed on to downstream stages. Once a new start
line is matched the former block of multilines is sent.
If now new line arrives withing
max_wait_timethe block is flushed tothe next stage and a new block is started.