Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat] Remove global loggers from libbbeat/dashboard #18541

Merged

Conversation

fearful-symmetry
Copy link
Contributor

@fearful-symmetry fearful-symmetry commented May 14, 2020

What does this PR do?

See #18500 for the meta-issue. This remove any global loggers. I was a tad hesitant to turn "soft" errors into hard errors, so I worked around some of the error handling for the sake of not breaking anything. This still technically contains "breaking" changes, so not sure if we wanna backport it.

Why is it important?

We want to remove the global logger from libbeat.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@fearful-symmetry fearful-symmetry added the Team:Integrations Label for the Integrations team label May 14, 2020
@fearful-symmetry fearful-symmetry requested review from urso and a team May 14, 2020 18:37
@fearful-symmetry fearful-symmetry self-assigned this May 14, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 14, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 14, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 1
Passed 9213
Skipped 1542
Total 10756

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Windows / test_close_renamed – test_harvester.Test

    • Status: FAILED
    • Age: 1
    • Duration: 0.564
    • Error Details: [Errno 13] Permission denied: 'C:\Users\jenkins\workspace\Beats_beats-beats-mbp_PR-18541\src\github.com\elastic\beats\filebeat\build\system-tests\run\test_harvester.Test.test_close_renamed\registry/filebeat/data.json'

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Result: FAILURE

    • Duration: 10 min 47 sec

    • Start Time: 2020-05-19T21:49:02.001+0000

    • log

  • Name: Install docker-compose 1.21.0

    • Description: .ci/scripts/install-docker-compose.sh

    • Result: FAILURE

    • Duration: 1 min 32 sec

    • Start Time: 2020-05-19T21:42:36.565+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Result: FAILURE

    • Duration: 2 min 22 sec

    • Start Time: 2020-05-19T21:45:17.288+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-19T22:38:38.312Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-19T22:38:38.312Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-19T22:38:38.312Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-19T22:38:38.312Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-19T22:38:38.312Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-19T22:38:38.312Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-19T22:38:38.804Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats
[2020-05-19T22:38:39.124Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-19T22:38:39.137Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Lint
[2020-05-19T22:38:39.219Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-05-19T22:38:39.295Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Winlogbeat-oss
[2020-05-19T22:38:39.367Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-05-19T22:38:39.440Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Dockerlogbeat
[2020-05-19T22:38:39.514Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Journalbeat-oss
[2020-05-19T22:38:39.587Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-05-19T22:38:39.661Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-05-19T22:38:39.733Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-19T22:38:39.808Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-19T22:38:39.882Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-19T22:38:39.956Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-19T22:38:40.203Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-05-19T22:38:40.276Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Heartbeat-oss
[2020-05-19T22:38:40.351Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-19T22:38:40.427Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-19T22:38:40.497Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-05-19T22:38:40.567Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-05-19T22:38:40.636Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Libbeat-x-pack
[2020-05-19T22:38:40.704Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Auditbeat-Linux
[2020-05-19T22:38:40.773Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-05-19T22:38:40.841Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-Windows
[2020-05-19T22:38:40.909Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-05-19T22:38:40.979Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-05-19T22:38:41.049Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-05-19T22:38:41.118Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-05-19T22:38:41.187Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Packetbeat-oss
[2020-05-19T22:38:41.256Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-05-19T22:38:41.325Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-19T22:38:41.393Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-19T22:38:41.463Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Auditbeat-Mac-OS-X
[2020-05-19T22:38:41.531Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-05-19T22:38:41.600Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Filebeat-oss
[2020-05-19T22:38:41.668Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Heartbeat-Windows
[2020-05-19T22:38:41.737Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Functionbeat-Windows
[2020-05-19T22:38:41.805Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Libbeat-oss
[2020-05-19T22:38:41.873Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-19T22:38:41.943Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-19T22:38:42.011Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-05-19T22:38:42.078Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-05-19T22:38:42.145Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Auditbeat-Windows
[2020-05-19T22:38:42.214Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-05-19T22:38:42.282Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-05-19T22:38:42.349Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-19T22:38:42.702Z] + cat
[2020-05-19T22:38:42.702Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-19T22:38:42.702Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-19T22:38:49.311Z] runbld>>> runbld started
[2020-05-19T22:38:49.311Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-19T22:38:51.235Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18541' in order of occurrence in the config (last value wins).
[2020-05-19T22:38:52.627Z] runbld>>> Debug logging enabled.
[2020-05-19T22:38:52.627Z] runbld>>> Storing result
[2020-05-19T22:38:52.627Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-19T22:38:52.627Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200519223852-B60F2EFA
[2020-05-19T22:38:52.627Z] runbld>>> Adding system facts.
[2020-05-19T22:38:53.576Z] runbld>>> Adding vcs info for the latest commit:  7393570b1fc339554870d3476c92f956d3817967
[2020-05-19T22:38:53.837Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-19T22:38:53.837Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-19T22:38:53.837Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-19T22:38:53.837Z] Processing JUnit reports with runbld...
[2020-05-19T22:38:54.100Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-19T22:38:54.100Z] runbld>>> DURATION: 19ms
[2020-05-19T22:38:54.100Z] runbld>>> STDOUT: 40 bytes
[2020-05-19T22:38:54.100Z] runbld>>> STDERR: 49 bytes
[2020-05-19T22:38:54.100Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-19T22:38:54.100Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats
[2020-05-19T22:38:55.505Z] runbld>>> Storing build metadata: 
[2020-05-19T22:38:55.505Z] runbld>>> Adding test report.
[2020-05-19T22:38:55.505Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats
[2020-05-19T22:38:56.459Z] runbld>>> Found 111 test output files
[2020-05-19T22:38:57.406Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-19T22:38:57.407Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-19T22:38:57.407Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-19T22:38:57.407Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-19T22:38:57.407Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-19T22:38:57.407Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-19T22:38:57.980Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-19T22:38:57.981Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-19T22:38:58.554Z] runbld>>> Test output logs contained: Errors: 1 Failures: 0 Tests: 10606 Skipped: 1304
[2020-05-19T22:38:58.554Z] runbld>>> Storing result
[2020-05-19T22:38:58.554Z] runbld>>> FAILURES: 1
[2020-05-19T22:38:59.125Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-19T22:38:59.125Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200519223852-B60F2EFA
[2020-05-19T22:38:59.125Z] runbld>>> Email notification disabled by environment variable.
[2020-05-19T22:38:59.125Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-19T22:39:05.244Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18541
[2020-05-19T22:39:05.518Z] [INFO] getVaultSecret: Getting secrets
[2020-05-19T22:39:05.571Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-19T22:39:06.402Z] + chmod 755 generate-build-data.sh
[2020-05-19T22:39:06.402Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18541/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18541/runs/4 FAILURE 4738498
[2020-05-19T22:39:06.953Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18541/runs/4/steps/?limit=10000 -o steps-info.json
[2020-05-19T22:39:08.297Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18541/runs/4/tests/?status=FAILED -o tests-errors.json

@fearful-symmetry fearful-symmetry requested review from urso and a team May 17, 2020 21:03
@@ -153,6 +157,6 @@ func (loader KibanaLoader) statusMsg(msg string, a ...interface{}) {
if loader.msgOutputter != nil {
loader.msgOutputter(msg, a...)
} else {
logp.Debug("dashboards", msg, a...)
loader.defaultLogger.Debugf("%s %v", msg, a)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the old implementation looks like "statusMsg" implements a Printf like interface (better named would have been statusMsgf.

s/loader.defaultLogger.Debugf("%s %v", msg, a)/loader.defaultLogger.debugf(msg, a...)/

Is MsgOutputer used somewhere? It kind of looks like a 'duplicate' interface for logging.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yah, it was one of those things where whatever calls the loader also asks for that message handler option, so it seems to go fairly high up the call stack and I was hesitant to mess with it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.

Let's change this line to this (to restore the behavior from before this PR):

loader.defaultLogger.Debugf(msg, a...)

@fearful-symmetry fearful-symmetry merged commit 0b12bb4 into elastic:master May 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants