Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.x](backport #27351) [Filebeat] Enable journald input in releases #27851

Merged
merged 1 commit into from
Sep 13, 2021

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Sep 9, 2021

This is an automatic backport of pull request #27351 done by Mergify.
Cherry-pick of 6b41742 has failed:

On branch mergify/bp/7.x/pr-27351
Your branch is up to date with 'origin/7.x'.

You are currently cherry-picking commit 6b417428d3.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   dev-tools/mage/common.go
	modified:   dev-tools/mage/crossbuild.go
	modified:   dev-tools/mage/pkg.go
	new file:   dev-tools/mage/semver.go
	modified:   filebeat/docs/filebeat-options.asciidoc
	new file:   filebeat/docs/inputs/input-journald.asciidoc
	modified:   filebeat/input/default-inputs/inputs_linux.go
	new file:   filebeat/input/journald/input_stub.go
	modified:   filebeat/input/v2/simplemanager.go
	modified:   filebeat/magefile.go
	new file:   filebeat/scripts/mage/build.go
	modified:   x-pack/elastic-agent/pkg/agent/program/supported.go
	modified:   x-pack/elastic-agent/spec/filebeat.yml
	modified:   x-pack/filebeat/magefile.go
	modified:   x-pack/libbeat/persistentcache/store.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   go.sum

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

@mergify mergify bot added backport conflicts There is a conflict in the backported pull request labels Sep 9, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 9, 2021
@botelastic
Copy link

botelastic bot commented Sep 9, 2021

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 9, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-10T18:38:43.039+0000

  • Duration: 144 min 27 sec

  • Commit: 87ab96f

Test stats 🧪

Test Results
Failed 0
Passed 53964
Skipped 5268
Total 59232

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 1

Expand to view the steps failures

filebeat-goIntegTest - mage goIntegTest
  • Took 5 min 36 sec . View more details on here
  • Description: mage goIntegTest

Log output

Expand to view the last 100 lines of log output

[2021-09-10T21:01:17.576Z] b6d86f8eae13: Preparing
[2021-09-10T21:01:17.576Z] b379042c76d1: Preparing
[2021-09-10T21:01:17.576Z] b544b17712ac: Preparing
[2021-09-10T21:01:17.576Z] b65333f43c3d: Preparing
[2021-09-10T21:01:17.576Z] f2f77537f65b: Preparing
[2021-09-10T21:01:17.576Z] 46e2e0c790dc: Preparing
[2021-09-10T21:01:17.576Z] 79f9f51e18e9: Preparing
[2021-09-10T21:01:17.576Z] 4ebb1fa9cae3: Preparing
[2021-09-10T21:01:17.576Z] 82cc7c3a0183: Preparing
[2021-09-10T21:01:17.576Z] 4a45220ae2a0: Preparing
[2021-09-10T21:01:17.576Z] e7ed17121dee: Preparing
[2021-09-10T21:01:17.576Z] 785573c4b945: Preparing
[2021-09-10T21:01:17.576Z] f2f77537f65b: Waiting
[2021-09-10T21:01:17.576Z] 46e2e0c790dc: Waiting
[2021-09-10T21:01:17.576Z] 4ebb1fa9cae3: Waiting
[2021-09-10T21:01:17.576Z] 79f9f51e18e9: Waiting
[2021-09-10T21:01:17.576Z] e7ed17121dee: Waiting
[2021-09-10T21:01:17.576Z] 785573c4b945: Waiting
[2021-09-10T21:01:17.576Z] 82cc7c3a0183: Waiting
[2021-09-10T21:01:17.835Z] 881790885a0c: Layer already exists
[2021-09-10T21:01:17.835Z] b65333f43c3d: Layer already exists
[2021-09-10T21:01:17.835Z] b379042c76d1: Layer already exists
[2021-09-10T21:01:17.835Z] b544b17712ac: Layer already exists
[2021-09-10T21:01:17.835Z] b6d86f8eae13: Layer already exists
[2021-09-10T21:01:17.835Z] 46e2e0c790dc: Layer already exists
[2021-09-10T21:01:17.835Z] f2f77537f65b: Layer already exists
[2021-09-10T21:01:17.835Z] 4ebb1fa9cae3: Layer already exists
[2021-09-10T21:01:17.835Z] 785573c4b945: Layer already exists
[2021-09-10T21:01:17.835Z] e7ed17121dee: Layer already exists
[2021-09-10T21:01:17.835Z] 79f9f51e18e9: Layer already exists
[2021-09-10T21:01:18.095Z] 82cc7c3a0183: Layer already exists
[2021-09-10T21:01:18.095Z] 4a45220ae2a0: Layer already exists
[2021-09-10T21:01:19.033Z] 87ab96f0214df5ecd5ab8476cc2c38ef6fc100f5-amd64: digest: sha256:fa5bb2867fc18fe32c07ee156f5ee3604e0c36389459bcdd29c644c71311dfd8 size: 3031
[2021-09-10T21:01:19.154Z] Timeout set to expire in 5 min 0 sec
[2021-09-10T21:01:19.456Z] Cleaning up /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e
[2021-09-10T21:01:19.456Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-10T21:01:19.456Z] ++ id -u
[2021-09-10T21:01:19.456Z] ++ id -g
[2021-09-10T21:01:19.456Z] + docker run -v /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-10T21:01:19.456Z] Unable to find image 'alpine:3.4' locally
[2021-09-10T21:01:20.025Z] 3.4: Pulling from library/alpine
[2021-09-10T21:01:20.284Z] c1e54eec4b57: Pulling fs layer
[2021-09-10T21:01:20.543Z] c1e54eec4b57: Verifying Checksum
[2021-09-10T21:01:20.543Z] c1e54eec4b57: Download complete
[2021-09-10T21:01:20.804Z] c1e54eec4b57: Pull complete
[2021-09-10T21:01:20.804Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-09-10T21:01:20.804Z] Status: Downloaded newer image for alpine:3.4
[2021-09-10T21:01:22.711Z] + set -e
[2021-09-10T21:01:22.711Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-10T21:01:22.711Z] Change permissions with write access of all files inside the specific folder
[2021-09-10T21:01:22.711Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e
[2021-09-10T21:01:23.704Z] Running in /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e/src/github.com/elastic/beats/build
[2021-09-10T21:01:24.001Z] + rm -rf ve
[2021-09-10T21:01:24.001Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-09-10T21:01:24.318Z] + python .ci/scripts/pre_archive_test.py
[2021-09-10T21:01:39.223Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-09-10T21:01:39.223Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-09-10T21:01:39.223Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-09-10T21:01:39.223Z] Copy ./x-pack/filebeat/build/package/filebeat/filebeat-linux-amd64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat/filebeat-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.223Z] Copy ./x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-amd64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.223Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-09-10T21:01:39.223Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-09-10T21:01:39.224Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/metricbeat/build/package/metricbeat-ubi8/metricbeat-linux-amd64.docker/docker-build into build/x-pack/metricbeat/build/package/metricbeat-ubi8/metricbeat-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/metricbeat/build/package/metricbeat/metricbeat-linux-amd64.docker/docker-build into build/x-pack/metricbeat/build/package/metricbeat/metricbeat-linux-amd64.docker/docker-build
[2021-09-10T21:01:39.224Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-09-10T21:01:39.253Z] Running in /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e/src/github.com/elastic/beats/build
[2021-09-10T21:01:39.273Z] Recording test results
[2021-09-10T21:01:43.003Z] None of the test reports contained any result
[2021-09-10T21:01:43.013Z] [Checks API] No suitable checks publisher found.
[2021-09-10T21:01:43.464Z] + go clean -modcache
[2021-09-10T21:01:46.826Z] Timeout set to expire in 5 min 0 sec
[2021-09-10T21:01:47.172Z] Cleaning up /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e
[2021-09-10T21:01:47.172Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-09-10T21:01:47.172Z] ++ id -u
[2021-09-10T21:01:47.172Z] ++ id -g
[2021-09-10T21:01:47.172Z] + docker run -v /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1171:1172 {} \;'
[2021-09-10T21:02:56.935Z] + set -e
[2021-09-10T21:02:56.935Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-09-10T21:02:56.935Z] Change permissions with write access of all files inside the specific folder
[2021-09-10T21:02:56.935Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e
[2021-09-10T21:02:57.009Z] Running in /var/lib/jenkins/workspace/PR-27851-5-eb4027f0-fee6-49aa-8ea0-0ef45d9cf26e
[2021-09-10T21:03:02.983Z] + gsutil --version
[2021-09-10T21:03:04.450Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-09-10T21:03:04.779Z] + gcloud auth activate-service-account --key-file ****
[2021-09-10T21:03:05.349Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-09-10T21:03:05.677Z] + gsutil -m -q cp eC1wYWNrL2VsYXN0aWMtYWdlbnQtcGFja2FnaW5nLWxpbnV4ODdhYjk2ZjAyMTRkZjVlY2Q1YWI4NDc2Y2MyYzM4ZWY2ZmMxMDBmNQ gs://beats-ci-temp/ci/cache/
[2021-09-10T21:03:07.594Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-09-10T21:03:07.684Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27851/src/github.com/elastic/beats
[2021-09-10T21:03:09.002Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27851
[2021-09-10T21:03:09.070Z] [INFO] getVaultSecret: Getting secrets
[2021-09-10T21:03:09.122Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-10T21:03:09.953Z] + chmod 755 generate-build-data.sh
[2021-09-10T21:03:09.953Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27851/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27851/runs/5 FAILURE 8666646
[2021-09-10T21:03:09.953Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27851/runs/5/steps/?limit=10000 -o steps-info.json
[2021-09-10T21:03:24.769Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27851/runs/5/tests/?status=FAILED -o tests-errors.json
[2021-09-10T21:03:25.680Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27851/runs/5/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 53964
Skipped 5268
Total 59232

@mergify
Copy link
Contributor Author

mergify bot commented Sep 10, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.x/pr-27351 upstream/mergify/bp/7.x/pr-27351
git merge upstream/7.x
git push upstream mergify/bp/7.x/pr-27351

Use the main-debian8 variant of golang-crossbuilder to build Filebeat
packages for linux/amd64 and linux/368. Previously a debian7 image was
used, but debian7 does not have the systemd libraries needed for journald.

Debian 8 uses glibc 2.19 and debian 7 uses 2.13. We used debian7 because
the binaries produced worked on RHEL 6 which has glibc 2.12.

To ensure that the Filebeat binaries for linux/amd64 and linux/386 continue
to work on RHEL 6 the build process will check the glibc requirement to
ensure it is <=2.12.

To keep the linked glibc version compatible with RHEL 6 I had to upgrade to
github.com/dgraph-io/badger/v3 from v2. v3 removed a bunch of cgo usages which
reduced the need on various libc functions.

badger adopted golang/glog for logging. This library is adds global CLI
flags that conflict with Beats own logging CLI flags (like '-v'). This is
a common problem for glog (k8s encountered this). To address the problem
I forked golang/glog and made the flag.FlagSet used by the library configurable.
@andrewkroh
Copy link
Member

run tests

@andrewkroh
Copy link
Member

/package

@andrewkroh
Copy link
Member

Packaging passed.

@andrewkroh andrewkroh merged commit f0c7c2b into 7.x Sep 13, 2021
@mergify mergify bot deleted the mergify/bp/7.x/pr-27351 branch September 13, 2021 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport conflicts There is a conflict in the backported pull request needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants