gcp: import metrics metricset #1570

endorama · 2021-09-02T14:27:37Z

What does this PR do?

Migrate GCP metrics metricset to integration.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2021-09-02T14:36:50Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-15T07:22:49.054+0000
Duration: 7 min 58 sec
Commit: 42331d7

Trends 🧪

Steps errors

Expand to view the steps failures

`Check integration: gcp`

Took 0 min 0 sec . View more details here
Description: ../../build/elastic-package check -v

`Collect internal fleet-server logs`

Took 0 min 0 sec . View more details here
Description: docker exec -t elastic-package-stack_fleet-server_1 sh -c "find data/logs/default/fleet-server-json* -printf '%p\n' -exec cat {} \;" > build/elastic-stack-dump/gcp/logs/fleet-server-internal.log

`Google Storage Download`

Took 0 min 0 sec . View more details here

Log output

Expand to view the last 100 lines of log output

[2021-09-15T07:28:52.343Z] go: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
[2021-09-15T07:28:52.343Z] go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible
[2021-09-15T07:28:52.343Z] go: downloading github.com/golang/protobuf v1.5.2
[2021-09-15T07:28:52.343Z] go: downloading github.com/google/go-cmp v0.5.5
[2021-09-15T07:28:52.343Z] go: downloading github.com/json-iterator/go v1.1.11
[2021-09-15T07:28:52.605Z] go: downloading github.com/modern-go/reflect2 v1.0.1
[2021-09-15T07:28:52.605Z] go: downloading github.com/google/gofuzz v1.1.0
[2021-09-15T07:28:52.605Z] go: downloading github.com/go-logr/logr v0.4.0
[2021-09-15T07:28:52.605Z] go: downloading google.golang.org/protobuf v1.26.0
[2021-09-15T07:28:52.606Z] go: downloading github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535
[2021-09-15T07:28:52.606Z] go: downloading github.com/go-openapi/errors v0.19.2
[2021-09-15T07:28:52.606Z] go: downloading github.com/mitchellh/mapstructure v1.4.1
[2021-09-15T07:28:52.606Z] go: downloading go.mongodb.org/mongo-driver v1.1.2
[2021-09-15T07:28:52.606Z] go: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
[2021-09-15T07:28:52.606Z] go: downloading k8s.io/utils v0.0.0-20210707171843-4b05e18ac7d9
[2021-09-15T07:28:52.606Z] go: downloading k8s.io/kube-openapi v0.0.0-20210421082810-95288971da7e
[2021-09-15T07:28:52.871Z] go: downloading github.com/davecgh/go-spew v1.1.1
[2021-09-15T07:28:52.871Z] go: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
[2021-09-15T07:28:52.871Z] go: downloading github.com/google/btree v1.0.1
[2021-09-15T07:28:52.871Z] go: downloading gopkg.in/inf.v0 v0.9.1
[2021-09-15T07:28:52.871Z] go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
[2021-09-15T07:28:52.871Z] go: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d
[2021-09-15T07:28:52.871Z] go: downloading k8s.io/component-base v0.21.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/PaesslerAG/jsonpath v0.1.1
[2021-09-15T07:28:52.871Z] go: downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
[2021-09-15T07:28:52.871Z] go: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
[2021-09-15T07:28:52.871Z] go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
[2021-09-15T07:28:52.871Z] go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
[2021-09-15T07:28:52.871Z] go: downloading github.com/stretchr/testify v1.7.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
[2021-09-15T07:28:52.871Z] go: downloading github.com/go-errors/errors v1.0.1
[2021-09-15T07:28:52.871Z] go: downloading github.com/go-openapi/spec v0.19.5
[2021-09-15T07:28:52.871Z] go: downloading github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd
[2021-09-15T07:28:52.871Z] go: downloading github.com/russross/blackfriday v1.5.2
[2021-09-15T07:28:52.871Z] go: downloading github.com/PaesslerAG/gval v1.0.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/pmezard/go-difflib v1.0.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/mitchellh/go-wordwrap v1.0.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/moby/term v0.0.0-20201216013528-df9cb8a40635
[2021-09-15T07:28:52.871Z] go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
[2021-09-15T07:28:52.871Z] go: downloading github.com/go-stack/stack v1.8.0
[2021-09-15T07:28:52.871Z] go: downloading github.com/go-openapi/jsonreference v0.19.5
[2021-09-15T07:28:52.871Z] go: downloading github.com/go-openapi/swag v0.19.14
[2021-09-15T07:28:52.871Z] go: downloading github.com/moby/spdystream v0.2.0
[2021-09-15T07:28:53.136Z] go: downloading github.com/go-openapi/jsonpointer v0.19.5
[2021-09-15T07:28:53.136Z] go: downloading github.com/PuerkitoBio/purell v1.1.1
[2021-09-15T07:28:53.136Z] go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
[2021-09-15T07:28:53.136Z] go: downloading github.com/mailru/easyjson v0.7.6
[2021-09-15T07:28:53.136Z] go: downloading github.com/josharian/intern v1.0.0
[2021-09-15T07:29:40.027Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations/packages/gcp
[2021-09-15T07:29:40.325Z] + ../../build/elastic-package check -v
[2021-09-15T07:29:40.325Z] elastic-package has been installed.
[2021-09-15T07:29:40.325Z] 2021/09/15 07:29:40 DEBUG Enable verbose logging
[2021-09-15T07:29:40.325Z] 2021/09/15 07:29:40 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-09-15T07:29:40.325Z] Format the package
[2021-09-15T07:29:40.325Z] Error: checking package failed: formatting the integration failed (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations/packages/gcp, failFast: true): walking through the integration files failed: formatting file failed (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations/packages/gcp/manifest.yml): file is not formatted (path: /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations/packages/gcp/manifest.yml)
[2021-09-15T07:29:40.365Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations
[2021-09-15T07:29:40.383Z] Archiving artifacts
[2021-09-15T07:29:41.146Z] ‘build/test-results/*.xml’ doesn’t match anything: ‘build’ exists but not ‘build/test-results/*.xml’
[2021-09-15T07:29:41.146Z] No artifacts found that match the file pattern "build/test-results/*.xml". Configuration error?
[2021-09-15T07:29:41.158Z] Recording test results
[2021-09-15T07:29:41.563Z] None of the test reports contained any result
[2021-09-15T07:29:41.563Z] [Checks API] No suitable checks publisher found.
[2021-09-15T07:29:41.879Z] + build/elastic-package stack dump -v --output build/elastic-stack-dump/gcp
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41  WARN CommitHash is undefined, in both /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/version and the compiled binary, config may be out of date.
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG Enable verbose logging
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG Dump Elastic stack data
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG Dump stack logs (location: build/elastic-stack-dump/gcp)
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG Dump stack logs for elasticsearch
[2021-09-15T07:29:41.879Z] 2021/09/15 07:29:41 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elasticsearch
[2021-09-15T07:29:42.828Z] 2021/09/15 07:29:42 DEBUG Dump stack logs for elastic-agent
[2021-09-15T07:29:42.828Z] 2021/09/15 07:29:42 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elastic-agent
[2021-09-15T07:29:43.401Z] 2021/09/15 07:29:43 DEBUG Dump stack logs for fleet-server
[2021-09-15T07:29:43.401Z] 2021/09/15 07:29:43 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs fleet-server
[2021-09-15T07:29:44.348Z] 2021/09/15 07:29:44 DEBUG output command: /usr/bin/docker exec -t elastic-package-stack_fleet-server_1 sh -c find data/logs/default/fleet-server-json* -printf '%p\n' -exec cat {} \;
[2021-09-15T07:29:44.348Z] 2021/09/15 07:29:44 ERROR can't fetch internal logs (service: fleet-server): docker exec failed: exec failed (stderr="Error: No such container: elastic-package-stack_fleet-server_1\n"): exit status 1
[2021-09-15T07:29:44.348Z] 2021/09/15 07:29:44 DEBUG Dump stack logs for kibana
[2021-09-15T07:29:44.348Z] 2021/09/15 07:29:44 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs kibana
[2021-09-15T07:29:45.296Z] 2021/09/15 07:29:44 DEBUG Dump stack logs for package-registry
[2021-09-15T07:29:45.296Z] 2021/09/15 07:29:44 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs package-registry
[2021-09-15T07:29:45.870Z] Path to stack dump: build/elastic-stack-dump/gcp
[2021-09-15T07:29:45.870Z] Done
[2021-09-15T07:29:46.171Z] + docker exec -t elastic-package-stack_fleet-server_1 sh -c find data/logs/default/fleet-server-json* -printf '%p\n' -exec cat {} \;
[2021-09-15T07:29:46.172Z] Error: No such container: elastic-package-stack_fleet-server_1
[2021-09-15T07:29:46.217Z] Failed in branch gcp
[2021-09-15T07:29:46.288Z] Running in /var/lib/jenkins/workspace/est-manager_integrations_PR-1570/src/github.com/elastic/integrations
[2021-09-15T07:29:46.413Z] Error when executing always post condition:
[2021-09-15T07:29:46.414Z] java.lang.NullPointerException
[2021-09-15T07:29:46.414Z] 
[2021-09-15T07:29:46.537Z] Terminated
[2021-09-15T07:29:46.587Z] Terminated
[2021-09-15T07:29:46.959Z] Running on Jenkins in /var/lib/jenkins/workspace/est-manager_integrations_PR-1570
[2021-09-15T07:29:47.043Z] [INFO] getVaultSecret: Getting secrets
[2021-09-15T07:29:47.078Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-09-15T07:29:47.749Z] + chmod 755 generate-build-data.sh
[2021-09-15T07:29:47.749Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1570/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1570/runs/11 FAILURE 418401
[2021-09-15T07:29:47.749Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1570/runs/11/steps/?limit=10000 -o steps-info.json
[2021-09-15T07:29:48.660Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Ingest-manager/integrations/PR-1570/runs/11/tests/?status=FAILED -o tests-errors.json
[2021-09-15T07:29:48.660Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-09-15T07:29:49.571Z] Retry 2/3 exited 22, retrying in 2 seconds...

sayden · 2021-09-05T18:50:42Z

packages/gcp/data_stream/metrics/fields/fields.yml

@@ -0,0 +1,4 @@
+- name: gcp.


OH MY! Thanks for fixing my CI 😆

endorama · 2022-02-01T15:01:10Z

This PR is too old, too many changes have happened and I'm not sure is worth migrating the entire metrics metricset now that we are implementing more specific ones. I'm closing this, we will migrate it eventually in future iterations.

endorama added enhancement New feature or request Team:Integrations Label for the Integrations team labels Sep 2, 2021

endorama self-assigned this Sep 2, 2021

sayden reviewed Sep 5, 2021

View reviewed changes

endorama mentioned this pull request Sep 7, 2021

Misleading error message when a data stream name contains a trailing dot elastic/kibana#111421

Closed

endorama added 8 commits September 13, 2021 11:52

gcp: import metrics data stream

c7aae7f

gcp: add gcp/metrics input

7adface

gcp: bump to 0.4.0

c5564e6

add template path

f8e8a76

remove dot

423632d

create new policy template

7a45774

try fixing

3b5bf37

try fixing 2

42331d7

nchaulet mentioned this pull request Sep 15, 2021

[Fleet] Support input level variable for granular integrations elastic/kibana#112272

Closed

endorama closed this Feb 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gcp: import metrics metricset #1570

gcp: import metrics metricset #1570

endorama commented Sep 2, 2021 •

edited

Loading

elasticmachine commented Sep 2, 2021 •

edited

Loading

Build stats

Trends 🧪

`Check integration: gcp`

`Collect internal fleet-server logs`

`Google Storage Download`

sayden Sep 5, 2021

endorama Sep 7, 2021

endorama commented Feb 1, 2022

gcp: import metrics metricset #1570

gcp: import metrics metricset #1570

Conversation

endorama commented Sep 2, 2021 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Sep 2, 2021 • edited Loading

💔 Build Failed

Build stats

Trends 🧪

Steps errors

Check integration: gcp

Collect internal fleet-server logs

Google Storage Download

Log output

sayden Sep 5, 2021

Choose a reason for hiding this comment

endorama Sep 7, 2021

Choose a reason for hiding this comment

endorama commented Feb 1, 2022

endorama commented Sep 2, 2021 •

edited

Loading

elasticmachine commented Sep 2, 2021 •

edited

Loading

`Check integration: gcp`

`Collect internal fleet-server logs`

`Google Storage Download`