Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collapse multiline logs based on a start line. #3024

Merged
merged 15 commits into from
Dec 3, 2020
Merged

Collapse multiline logs based on a start line. #3024

merged 15 commits into from
Dec 3, 2020

Commits on Dec 3, 2020

  1. Collapse multiline logs based on a start line. 

    Summary:
This is a very simple approach based on grafana#1380 to provide multiline
or block log entries in promtail.

A `multiline` stage is added to pipelines. This stages matches a start
line. Once a start line is matched all following lines are appended
to an entry and not passed on to downstream stages. Once a new start
line is matched the former block of multilines is sent.

If now new line arrives withing `max_wait_time` the block is flushed to
the next stage and a new block is started.
    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    6e70704 View commit details
    Browse the repository at this point in the history

  2. Test multiline stage process.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    7f70762 View commit details
    Browse the repository at this point in the history

  3. Format code.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    3db919d View commit details
    Browse the repository at this point in the history

  4. Flush multiline block after max_lines.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    851b91e View commit details
    Browse the repository at this point in the history

  5. Capture internal state of the stage.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    503a0d2 View commit details
    Browse the repository at this point in the history

  6. Process different multiline streams in parallel.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    dc715cb View commit details
    Browse the repository at this point in the history

  7. Start documenting multiline stage.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    6ac6375 View commit details
    Browse the repository at this point in the history

  8. Give an example configuration for multiline stage.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    48b09d6 View commit details
    Browse the repository at this point in the history

  9. Make linter happy.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    79c8c27 View commit details
    Browse the repository at this point in the history

  10. Pass through entries until first start line.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    f9ed133 View commit details
    Browse the repository at this point in the history

  11. Update pkg/logentry/stages/multiline.go 

    Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com>
    @jeschkies @cyriltovena
    jeschkies and cyriltovena authored Dec 3, 2020
    Configuration menu
    Copy the full SHA
    ca7ad81 View commit details
    Browse the repository at this point in the history

  12. Default maxz wait time to 3 seconds.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    c07875e View commit details
    Browse the repository at this point in the history

  13. Update pkg/logentry/stages/multiline.go

    @cyriltovena
    cyriltovena authored Dec 3, 2020
    Configuration menu
    Copy the full SHA
    2c4dc8e View commit details
    Browse the repository at this point in the history

  14. Wrap all debug logs.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    bb1cf59 View commit details
    Browse the repository at this point in the history

  15. Format code.

    @jeschkies
    jeschkies committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    138371b View commit details
    Browse the repository at this point in the history