Fix add_kubernetes_metadata matcher: support rotated logs when 'resource_type: pod'

[tetianakravchenko]

Signed-off-by: Tetiana Kravchenko tetiana.kravchenko@elastic.co

What does this PR do?

According to https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html#file-log-rotation, there might be cases when we need to read rotated log files:

Make sure Filebeat is configured to read from all rotated logs

This PR add possibility to get pod uid for the rotated log files when resource_type is set to pod.

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• https://discuss.elastic.co/t/filebeat-add-kubernetes-metadata-pod-uid-matcher-doesnt-match-poduid-for-rotated-file-on-aks/298938

Use cases

Context:
According to [kubernetes documentation]:

Kubernetes is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that.

In most cases where k8s uses docker as runtime the strategy is what docker sets through docker log-opt:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  }
}

In some cases it is needed to set paths to /var/log/pods/*/*/*.log*: in some occasions, there are a very high volume of logs which caused the files to be quickly rotated without filebeat having time to harvest them.
For this reason was removed check strings.HasSuffix(source, ".log") to support rotated log files.

At the same time compressed log files are not supported for now, *.gz files are excluded from the processing when resource_type: pod
example of /var/log/pods/*/*/*.log* content:

/var/log/pods/kube-system_metricbeat-56nf2_64e94e05-c377-4bcb-9cc2-5184c4f4b978/metricbeat# ls -l | awk -F " " '{print $9}'

0.log
0.log.20220221-203122.gz
0.log.20220221-203252.gz
0.log.20220221-210526.gz
0.log.20220221-210912
1.log
1.log.20220308-190811.gz
1.log.20220308-190916.gz
1.log.20220308-191043.gz
1.log.20220308-191159

in this case only 0.log, 0.log.20220221-210912, 1.log and 1.log.20220308-191159 should be processed

Screenshots

Logs

[mergify]

This pull request does not have a backport label. Could you fix it @tetianakravchenko? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
[](https://ci-stats.elastic.co/app/apm/services/beats-ci/transactions/view?rangeFrom=2022-03-08T20:45:02.579Z&rangeTo=2022-03-08T21:05:02.579Z&transactionName=BUILD Beats/beats/PR-{number}&transactionType=job&latencyAggregationType=avg&traceId=90bdc36beaca5dcc3ca36de778b4107b&transactionId=b336fec2db6aa932)

Expand to view the summary

Build stats

• Start Time: 2022-03-08T20:55:02.579+0000

• Duration: 94 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9657
Skipped	1319
Total	10976

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[tetianakravchenko]

/test

[ChrsMark]

lgtm! It would help if we could have 2 small examples in the description of the PR explaining what type of files are now handled and what are excluded since I had to read the tests mainly to understand the expected behaviour.

[tetianakravchenko]

/test

[tetianakravchenko]

@ChrsMark

It would help if we could have 2 small examples in the description of the PR explaining what type of files are now handled and what are excluded since I had to read the tests mainly to understand the expected behaviour.

I've updated description