Cherry-pick #19619 to 7.x: Init journald input

[urso]

Cherry-pick of PR #19619 to 7.x branch. Original message:

• Enhancement

What does this PR do?

This changes introduces the journald input in filebeat based on the
input v2 API. Most of the reading and field transformation code is
shared with journalbeat

Integration tests are not yet implemented. I'd sugest to reimplement the existing journalbeat python tests in this package as more simple go based tests. Most of the reading and event transformation support is already shared with journalbeat, which gives us at least some confidence.

Note: this is just the input implementation. The API is not fully hooked up yet in Filebeat, which means that the input is not available to be configured yet.

Why is it important?

Provide journald input to the Agent

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Configure input with journald type. The input settings match the ones currently found for journalbeat.

Sample configuration:

filebeat.inputs:
- type: journald
  include_matches:
  - syslog.identifier=test


output.console:
  codec.format:
    string: '%{[@timestamp]} - %{[syslog.identifier]}: %{[message]}'

use systemd-cat to write logs:

echo "<message>" | systemd-cat -t test

• Check output that logs are collected (Note: default input backoff is max 1min, might take a while if filebeat is started first)
• Restart and check old logs are not collected
• add more log entries and check they are collected.

Related issues

• Relates Filebeat input v2 API #15324

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Expand to view the summary

Build stats

• Build Cause: [Branch indexing]

• Start Time: 2020-07-13T09:39:56.591+0000

• Duration: 123 min 53 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8362
Skipped	1564
Total	9926

Steps errors

Expand to view the steps failures

• Name: Mage build unitTest

  • Description: mage build unitTest

  • Duration: 3 min 49 sec

  • Start Time: 2020-07-13T10:09:00.757+0000

  • log

• Name: Notifies GitHub of the status of a Pull Request

  • Description:

  • Duration: 23 min 18 sec

  • Start Time: 2020-07-13T10:17:36.424+0000

  • log

• Name: Recursively delete the current directory from the workspace

  • Description: Ran out of retries for URL: https://api.github.com/users/elastic

  • Duration: 0 min 8 sec

  • Start Time: 2020-07-13T10:43:20.510+0000

  • log

• Name: Notifies GitHub of the status of a Pull Request

  • Description: Ran out of retries for URL: https://api.github.com/users/elastic

  • Duration: 23 min 44 sec

  • Start Time: 2020-07-13T10:18:09.812+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-07-13T10:17:35.088+0000

  • log

• Name: Notifies GitHub of the status of a Pull Request

  • Description: Ran out of retries for URL: https://api.github.com/users/elastic

  • Duration: 21 min 56 sec

  • Start Time: 2020-07-13T10:19:58.720+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 24 sec

  • Start Time: 2020-07-13T10:13:06.331+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-07-13T10:15:30.952+0000

  • log

• Name: Notifies GitHub of the status of a Pull Request

  • Description: Ran out of retries for URL: https://api.github.com/users/elastic

  • Duration: 18 min 55 sec

  • Start Time: 2020-07-13T10:22:59.708+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-07-13T10:56:53.780+0000

  • log

Log output

Expand to view the last 100 lines of log output

[2020-07-13T11:42:10.460Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-13T11:42:10.460Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-13T11:42:10.460Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-13T11:42:10.460Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-13T11:42:10.460Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-13T11:42:11.152Z] Failed in branch Filebeat x-pack
[2020-07-13T11:42:12.191Z] Failed in branch Filebeat oss
[2020-07-13T11:42:14.863Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats
[2020-07-13T11:42:15.315Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-13T11:42:15.365Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Lint
[2020-07-13T11:42:15.613Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-13T11:42:15.860Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-13T11:42:16.165Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-13T11:42:16.649Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-13T11:42:16.914Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-13T11:42:17.119Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-13T11:42:17.340Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-13T11:42:17.629Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-13T11:42:18.130Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-13T11:42:19.173Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-13T11:42:19.709Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-13T11:42:20.197Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-13T11:42:20.714Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-13T11:42:21.175Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-13T11:42:21.581Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-13T11:42:21.967Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-13T11:42:22.684Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-13T11:42:23.170Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-13T11:42:23.676Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-13T11:42:24.066Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-13T11:42:24.343Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-13T11:42:24.605Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-13T11:42:24.876Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-13T11:42:25.175Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-13T11:42:25.621Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-13T11:42:25.959Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-13T11:42:26.293Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-13T11:42:26.737Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-13T11:42:27.166Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-13T11:42:27.589Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Libbeat-oss
[2020-07-13T11:42:27.963Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-13T11:42:28.316Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-07-13T11:42:28.783Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-13T11:42:29.231Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-07-13T11:42:29.579Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-13T11:42:30.144Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-13T11:42:30.377Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-13T11:42:30.597Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-13T11:42:30.785Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Filebeat-oss
[2020-07-13T11:42:31.296Z] + cat
[2020-07-13T11:42:31.296Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-13T11:42:31.296Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-13T11:42:37.945Z] runbld>>> runbld started
[2020-07-13T11:42:37.945Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-13T11:42:39.346Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19782' in order of occurrence in the config (last value wins).
[2020-07-13T11:42:40.749Z] runbld>>> Debug logging enabled.
[2020-07-13T11:42:40.749Z] runbld>>> Storing result
[2020-07-13T11:42:41.021Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-13T11:42:41.021Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713114240-BB8D3531
[2020-07-13T11:42:41.021Z] runbld>>> Adding system facts.
[2020-07-13T11:42:41.976Z] runbld>>> Adding vcs info for the latest commit:  96062a97cb6bf5d946fffa86aa730a1c01dc1791
[2020-07-13T11:42:41.976Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-13T11:42:41.976Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-13T11:42:41.976Z] Processing JUnit reports with runbld...
[2020-07-13T11:42:41.976Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-13T11:42:42.239Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-13T11:42:42.239Z] runbld>>> DURATION: 26ms
[2020-07-13T11:42:42.239Z] runbld>>> STDOUT: 40 bytes
[2020-07-13T11:42:42.239Z] runbld>>> STDERR: 49 bytes
[2020-07-13T11:42:42.239Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-13T11:42:42.239Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats
[2020-07-13T11:42:43.344Z] runbld>>> Storing build metadata: 
[2020-07-13T11:42:43.344Z] runbld>>> Adding test report.
[2020-07-13T11:42:43.344Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats
[2020-07-13T11:42:44.296Z] runbld>>> Found 105 test output files
[2020-07-13T11:42:44.569Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-13T11:42:44.569Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-13T11:42:44.569Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-13T11:42:44.569Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-13T11:42:44.569Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-13T11:42:45.978Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-13T11:42:45.978Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-19782/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-13T11:42:46.239Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 9780 Skipped: 1329
[2020-07-13T11:42:46.499Z] runbld>>> Storing result
[2020-07-13T11:42:46.499Z] runbld>>> FAILURES: 0
[2020-07-13T11:42:46.760Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-13T11:42:46.760Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713114240-BB8D3531
[2020-07-13T11:42:46.760Z] runbld>>> Email notification disabled by environment variable.
[2020-07-13T11:42:46.760Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-13T11:42:47.565Z] Body did not finish within grace period; terminating with extreme prejudice
[2020-07-13T11:42:47.682Z] ERROR: runbld post build action failed.
[2020-07-13T11:42:47.682Z] Timeout has been exceeded
[2020-07-13T11:42:47.993Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19782
[2020-07-13T11:42:48.235Z] [INFO] getVaultSecret: Getting secrets
[2020-07-13T11:42:48.346Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-13T11:42:49.597Z] + chmod 755 generate-build-data.sh
[2020-07-13T11:42:49.598Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19782/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19782/runs/1 ABORTED 7372731
[2020-07-13T11:42:49.598Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19782/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-13T11:42:56.080Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19782/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-13T11:42:56.339Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19782/runs/1/log/ -o pipeline-log.txt

[kaiyan-sheng]

LGTM