Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add memory metrics into compute googlecloud #18802

Merged
merged 4 commits into from
May 28, 2020
Merged

Add memory metrics into compute googlecloud #18802

merged 4 commits into from
May 28, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

This PR is to add 4 memory metrics(only for e2 family VMs) into googlecloud compute metricset and also added the missing network.sent_packets_count metric.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 27, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this May 27, 2020
@kaiyan-sheng kaiyan-sheng added Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels May 27, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 27, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 27, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18802 updated]

  • Start Time: 2020-05-28T16:09:18.946+0000

  • Duration: 61 min 13 sec

Test stats 🧪

Test Results
Failed 0
Passed 3536
Skipped 811
Total 4347

Steps errors

Expand to view the steps failures

  • Name: Mage goIntegTest
    • Description: mage goIntegTest

    • Duration: 1 min 47 sec

    • Start Time: 2020-05-28T16:32:19.026+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-28T17:08:31.498Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-05-28T17:08:31.498Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-05-28T17:08:31.498Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-05-28T17:08:31.498Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-05-28T17:08:31.498Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-05-28T17:08:31.498Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-05-28T17:08:31.498Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-05-28T17:08:31.498Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-05-28T17:08:31.498Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-05-28T17:08:31.498Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-05-28T17:08:31.498Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-05-28T17:08:31.498Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-05-28T17:08:31.498Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-05-28T17:08:31.498Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-05-28T17:08:31.498Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-05-28T17:08:31.498Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-05-28T17:08:31.498Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-05-28T17:08:31.498Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-05-28T17:08:31.498Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-05-28T17:08:31.498Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-05-28T17:08:31.498Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-05-28T17:08:31.498Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-05-28T17:08:31.498Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-05-28T17:08:31.868Z] + curl -sSLo codecov https://codecov.io/bash
[2020-05-28T17:08:32.129Z] + FILE=auditbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=filebeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-28T17:08:32.129Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-28T17:08:32.129Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-28T17:10:09.226Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats
[2020-05-28T17:10:09.531Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-28T17:10:09.542Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Lint
[2020-05-28T17:10:09.619Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-28T17:10:09.689Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-28T17:10:09.756Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-28T17:10:09.825Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-28T17:10:09.894Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-28T17:10:09.960Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-28T17:10:10.030Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-05-28T17:10:10.098Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-28T17:10:10.172Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-28T17:10:10.244Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-28T17:10:10.594Z] + cat
[2020-05-28T17:10:10.594Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-28T17:10:10.594Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-28T17:10:17.180Z] runbld>>> runbld started
[2020-05-28T17:10:17.180Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-28T17:10:18.565Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18802' in order of occurrence in the config (last value wins).
[2020-05-28T17:10:19.505Z] runbld>>> Debug logging enabled.
[2020-05-28T17:10:19.505Z] runbld>>> Storing result
[2020-05-28T17:10:19.767Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-28T17:10:19.767Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200528171019-FBCADCD8
[2020-05-28T17:10:19.767Z] runbld>>> Adding system facts.
[2020-05-28T17:10:20.711Z] runbld>>> Adding vcs info for the latest commit:  ae1e1e783d6fdabdb85c8cd50fcb8cd3e41d2004
[2020-05-28T17:10:20.971Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-28T17:10:20.971Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-28T17:10:20.971Z] Processing JUnit reports with runbld...
[2020-05-28T17:10:20.971Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-28T17:10:21.233Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-28T17:10:21.233Z] runbld>>> DURATION: 16ms
[2020-05-28T17:10:21.233Z] runbld>>> STDOUT: 40 bytes
[2020-05-28T17:10:21.233Z] runbld>>> STDERR: 49 bytes
[2020-05-28T17:10:21.233Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-28T17:10:21.233Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats
[2020-05-28T17:10:22.622Z] runbld>>> Storing build metadata: 
[2020-05-28T17:10:22.622Z] runbld>>> Adding test report.
[2020-05-28T17:10:22.622Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802/src/github.com/elastic/beats
[2020-05-28T17:10:23.565Z] runbld>>> Found 17 test output files
[2020-05-28T17:10:24.953Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 4347 Skipped: 729
[2020-05-28T17:10:24.953Z] runbld>>> Storing result
[2020-05-28T17:10:24.953Z] runbld>>> FAILURES: 0
[2020-05-28T17:10:25.215Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-28T17:10:25.215Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200528171019-FBCADCD8
[2020-05-28T17:10:25.215Z] runbld>>> Email notification disabled by environment variable.
[2020-05-28T17:10:25.215Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-28T17:10:31.039Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18802
[2020-05-28T17:10:31.282Z] [INFO] getVaultSecret: Getting secrets
[2020-05-28T17:10:31.323Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-28T17:10:32.096Z] + chmod 755 generate-build-data.sh
[2020-05-28T17:10:32.096Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18802/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18802/runs/4 FAILURE 3672886
[2020-05-28T17:10:32.097Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18802/runs/4/steps/?limit=10000 -o steps-info.json
[2020-05-28T17:10:32.647Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18802/runs/4/tests/?status=FAILED -o tests-errors.json
[2020-05-28T17:10:33.198Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18802/runs/4/log/ -o pipeline-log.txt

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

- `instance_name`: The name of the VM instance.

* `instance.memory.balloon.swap_out_bytes_count`: The amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family.
- `instance_name`: The name of the VM instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are you generating this list of metrics? I find it a bit redundant with metricbeat/docs/fields.asciidoc. If we feel the need of having the metrics descriptions in two places maybe we have to rethink how we generate these docs.
In any case they should be auto-generated from the fields.yml files, so we only have to maintain the descriptions in one place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm manually copying these fields in here so it can be displayed on individual metricset doc: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-googlecloud-compute.html

Good point! I shouldn't have to copy paste these descriptions twice and maintain in two different places. Maybe in the doc, I should just display x-pack/metricbeat/module/googlecloud/compute/_meta/fields.yml instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already a link to exported fields in the metricset docs.
Captura de pantalla de 2020-05-28 20-37-49

But the one for this module seems quite empty: https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-googlecloud.html

We will have to review this. If we think that the fields should appear directly in the metricset docs page, we should generate them automatically there. If we want to continue using the separated page we will have to check why the fields of this module doesn't appear there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an old issue discussing about fields descriptions in docs: #9288

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I will create a separate PR to fix the exported fields section.

- `instance_name`: The name of the VM instance.

* `instance.memory.balloon.swap_in_bytes_count`: The amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family.
- `instance_name`: The name of the VM instance.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this instance_name repeated in all metrics?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instance_name is label that comes with these metrics, like a default dimension in gcp.

- "compute.googleapis.com/instance/network/received_bytes_count"
- "compute.googleapis.com/instance/network/received_packets_count"
- "compute.googleapis.com/instance/network/sent_bytes_count"
- "compute.googleapis.com/instance/network/sent_packets_count"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@jsoriano
Copy link
Member

Oh, btw, before merging please remember to add a changelog entry.

@jsoriano
Copy link
Member

Failure in CI can be ignored, it is related to #18741

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! We can leave by now the discussions about fields descriptions.

@kaiyan-sheng kaiyan-sheng added v7.9.0 and removed needs_backport PR is waiting to be backported to other branches. labels May 28, 2020
kaiyan-sheng added a commit that referenced this pull request May 29, 2020
#18831)

* Add memory metrics into compute googlecloud (#18802)

* Add memory metrics into compute googlecloud

* update unit test and data.json for memory metrics

(cherry picked from commit 1eb4e79)

* update changelog
kaiyan-sheng added a commit that referenced this pull request May 29, 2020
* Add memory metrics into compute googlecloud

* update unit test and data.json for memory metrics

(cherry picked from commit 1eb4e79)
kaiyan-sheng added a commit that referenced this pull request May 29, 2020
v1v added a commit to v1v/beats that referenced this pull request Jun 2, 2020
…-stage-level

* upstream/master: (30 commits)
  Add a GRPC listener service for Agent (elastic#18827)
  Disable host.* fields by default for iptables module (elastic#18756)
  [WIP] Clarify capabilities of the Filebeat auditd module (elastic#17068)
  fix: rename file and remove extra separator (elastic#18881)
  ci: enable JJBB (elastic#18812)
  Disable host.* fields by default for Checkpoint module (elastic#18754)
  Disable host.* fields by default for Cisco module (elastic#18753)
  Update latest.yml testing env to 7.7.0 (elastic#18535)
  Upgrade k8s.io/client-go and k8s keystore tests (elastic#18817)
  Add missing Jenkins stages for Auditbeat (elastic#18835)
  [Elastic Log Driver] Create a config shim between libbeat and the user (elastic#18605)
  Use indexers and matchers in config when defaults are enabled (elastic#18818)
  Fix panic on `metricbeat test modules` (elastic#18797)
  [CI] Fix permissions in MacOSX agents (elastic#18847)
  [Ingest Manager] When not port are specified and the https is used fallback to 443 (elastic#18844)
  [Ingest Manager] Fix install service script for windows (elastic#18814)
  [Metricbeat] Fix getting compute instance metadata with partial zone/region config (elastic#18757)
  Improve error messages in s3 input (elastic#18824)
  Add memory metrics into compute googlecloud (elastic#18802)
  include bucket name when logging error (elastic#18679)
  ...
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metricbeat Metricbeat review Team:Platforms Label for the Integrations - Platforms team v7.8.0 v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants