filebeat/inputs/filestream: add metric for messages truncated

[mauri870]

Proposed commit message

While investigating an SDH, I noticed that although we add the truncated label to log fields, there is no feedback on the amount of messages that are truncated. It often happens that almost all messages are truncated because of a misconfigured input.

This PR adds a input metric that counts the total number of truncated messages.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Create a log file with multiline logs:

cat <<EOF > /tmp/in.log
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index]
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver\$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
    at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
EOF

Configure filebeat to parse the multiline logs with max_lines:

filebeat.inputs:
- type: filestream
  id: foo-bar
  enabled: true
  paths:
    - "/tmp/in.log"
  parsers:
    - multiline:
        type: pattern
        pattern: '^\['
        negate: true
        match: after
        max_lines: 3

output.discard:
  enabled: true
http.enabled: true

Check the metrics:

$ curl -XGET 'localhost:5066/inputs/' | jq
[
  {
    "bytes_processed_total": 596,
    "events_processed_total": 1,
    "files_active": 1,
    "files_closed_total": 0,
    "files_opened_total": 1,
    "id": "foo-bar",
    "input": "filestream",
    "messages_read_total": 1,
    "messages_truncated_total": 1,
    "processing_errors_total": 0,
    "processing_time": {
      "histogram": {
        "count": 1,
        "max": 5000516503,
        "mean": 5000516503,
        "median": 5000516503,
        "min": 5000516503,
        "p75": 5000516503,
        "p95": 5000516503,
        "p99": 5000516503,
        "p999": 5000516503,
        "stddev": 0
      }
    }
  }
]

Related issues

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mauri870? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

LGTM. I didn't have time to test it today, but the code is pretty simple. If it has not been merged by tomorrow morning, I'll test it.

[belimawr]

The code looks good, but I'm not managing to test it... Filebeat won't start.

Here is the full filebeat.yml I'm using:

filebeat.inputs:
- type: filestream
  id: foo-bar
  enabled: true
  paths:
    - "/tmp/flog.log"
  parsers:
    - multiline:
        type: pattern
        pattern: '^LINESTART$"),'
        negate: true
        match: after
        max_bytes: 20

output.discard:
  enabled: true
http.enabled: true

Whenever I run, I get the following error:

Exiting: Failed to start crawler: starting input failed: error while initializing input: error while parsing multiline parser config: error parsing regexp: unexpected ): `^LINESTART$"),` accessing 'filebeat.inputs.0.parsers.0.multiline.pattern' (source:'filebeat.yml') accessing 'filebeat.inputs.0' (source:'filebeat.yml')
exit status 1

[cmacknz]

One thing I found interesting is that curl localhost:5066/inputs does not work, it needs a trailing slash (inputs/), is that expected?

Probably not an intentional omission

Edit: if /inputs/ is the listing of inputs it might be intentional in that it behaves more like a directory listing.

[mauri870]

Edit: if /inputs/ is the listing of inputs it might be intentional in that it behaves more like a directory listing.

It is a bit counter intuitive when thinking about REST but it makes sense if it is supposed to be a directory listing. The docs also use /inputs/ so it is probably deliberate.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-multine-trunc-log upstream/filebeat-multine-trunc-log
git merge upstream/main
git push upstream filebeat-multine-trunc-log

[mauri870]

/test

[mauri870]

For some reason, the linter is failing and pointing out issues in the metricbeat mongo module. I don't plan to modify these files to avoid introducing additional linting problems.

[mauri870]

@elastic/obs-infraobs-integrations Kindly requesting a review. Thank you!

[jlind23]

@lalit-satapathy @devamanv can we get someone's eyes on this please?

[lalit-satapathy]

@lalit-satapathy @devamanv can we get someone's eyes on this please?

@ishleenk17 will be taking a look.

[ishleenk17]

For some reason, the linter is failing and pointing out issues in the metricbeat mongo module. I don't plan to modify these files to avoid introducing additional linting problems.

Why has the mongdb module been touched for this change? @mauri870
If you are doing that for the linter errors, you can ignore that.
Our team will take care of it in any corresponding mongod module related change.

[mauri870]

Why has the mongdb module been touched for this change?

Thanks, I forgot I had this small change for the struct annotations, I just reverted it.