Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Instrument aws-s3 with metrics #25711

Merged
merged 2 commits into from
May 17, 2021
Merged

[Filebeat] Instrument aws-s3 with metrics #25711

merged 2 commits into from
May 17, 2021

Commits on May 17, 2021

  1. Instrument aws-s3 with metrics 

    Diagnosing performance issues with the aws-s3 input is difficult so this instruments it with some metrics to make this easier.
These are the metrics that are added.

- Number of SQS messages received (not necessarily processed fully).
- Number of SQS visibility timeout extensions.
- Number of SQS messages inflight (gauge).
- Number of SQS message returned to queue (happens on errors implicitly after visibility timeout passes).
- Number of SQS messages deleted.
- Histogram of the elapsed SQS processing times in nanoseconds (time of receipt to time of delete/return).
- Number of S3 objects downloaded.
- Number of S3 bytes processed.
- Number of events created from processing S3 data.
- Number of S3 objects inflight (gauge).
- Histogram of the elapsed S3 object processing times in nanoseconds (start of download to completion of parsing).

The metrics are structured as:

    dataset.<input-id>:
        id=<input id>
        input=aws-s3
        sqs_messages_received_total
        sqs_visibility_timeout_extensions_total
        sqs_messages_inflight_gauge
        sqs_messages_returned_total
        sqs_messages_deleted_total
        sqs_message_processing_time.histogram
        s3_objects_requested_total
        s3_bytes_processed_total
        s3_events_created_total
        s3_objects_inflight_gauge
        s3_object_processing_time.histogram

The v2 input logger was updated to include the input ID to make correlation with metrics possible when an explicit `id` is not set in the input config
    @andrewkroh
    andrewkroh committed May 17, 2021
    Configuration menu
    Copy the full SHA
    4ccf431 View commit details
    Browse the repository at this point in the history

  2. Format docs

    @andrewkroh
    andrewkroh committed May 17, 2021
    Configuration menu
    Copy the full SHA
    c51e2ef View commit details
    Browse the repository at this point in the history