Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Add GCP CloudSQL region filter #32943

Merged
merged 14 commits into from
Sep 19, 2022
Merged

[Metricbeat] Add GCP CloudSQL region filter #32943

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
e9914b6
add cloudsql region filter
gpop63 Aug 31, 2022
b4d42c2
add changelog entry
gpop63 Aug 31, 2022
0e82c01
fix comment for golangci-lint
gpop63 Aug 31, 2022
e3ea3b8
fix get filter for metric tests
gpop63 Aug 31, 2022
117cfa7
better AlignersMapToGCP comment
gpop63 Sep 1, 2022
7353ae3
Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql_re…
gpop63 Sep 10, 2022
1e85834
revert to zone and add comment
gpop63 Sep 10, 2022
b02b11f
Revert "fix get filter for metric tests"
gpop63 Sep 10, 2022
0b7d558
Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql_re…
gpop63 Sep 11, 2022
69a3921
Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql_re…
gpop63 Sep 15, 2022
4e92cfb
add cloudsql region label
gpop63 Sep 15, 2022
35196aa
add regions filter
gpop63 Sep 15, 2022
46e33df
fix changelog
gpop63 Sep 19, 2022
2be3f1c
remove comment from metrics requester
gpop63 Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
*Metricbeat*

- Fix and improve AWS metric period calculation to avoid zero-length intervals {pull}32724[32724]
- Add GCP CloudSQL region filter {pull}32943[32943]

*Packetbeat*

Expand Down
3 changes: 2 additions & 1 deletion x-pack/metricbeat/module/gcp/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ const (
ServiceStorage = "storage"
ServiceFirestore = "firestore"
ServiceDataproc = "dataproc"
ServiceCloudSQL = "cloudsql"
)

//Paths within the GCP monitoring.TimeSeries response, if converted to JSON, where you can find each ECS field required for the output event
Expand Down Expand Up @@ -76,7 +77,7 @@ const (
LabelMetadata = "metadata"
)

// Available perSeriesAligner map
// AlignersMapToGCP map contains available perSeriesAligner
// https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#Aligner
var AlignersMapToGCP = map[string]monitoringpb.Aggregation_Aligner{
"ALIGN_NONE": monitoringpb.Aggregation_ALIGN_NONE,
Expand Down
7 changes: 6 additions & 1 deletion x-pack/metricbeat/module/gcp/metrics/metrics_requester.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,11 @@ func (r *metricsRequester) getFilterForMetric(serviceName, m string) string {
}

f = fmt.Sprintf(`%s AND resource.labels.location = "%s"`, f, r.config.Region)
case gcp.ServiceCloudSQL:
if r.config.Region != "" {
region := strings.TrimSuffix(r.config.Region, "*")
f = fmt.Sprintf(`%s AND resource.labels.region = starts_with("%s")`, f, region)
}
default:
if r.config.Region != "" && r.config.Zone != "" {
r.logger.Warnf("when region %s and zone %s config parameter "+
Expand All @@ -152,7 +157,7 @@ func (r *metricsRequester) getFilterForMetric(serviceName, m string) string {
// }

region := strings.TrimSuffix(r.config.Region, "*")
f = fmt.Sprintf(`%s AND resource.labels.zone = starts_with("%s")`, f, region)
f = fmt.Sprintf(`%s AND resource.labels.region = starts_with("%s")`, f, region)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not include this change here, as it's not related to CloudSQL. This change is also a breaking change, so we need to take another approach.
I would add a comment, so we can remove this in the future: FIXME: this is using zone but should use region. This is a breaking change.

} else if r.config.Zone != "" {
// zone := r.config.Zone
// if strings.HasSuffix(r.config.Zone, "*") {
Expand Down
8 changes: 4 additions & 4 deletions x-pack/metricbeat/module/gcp/metrics/metrics_requester_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ func TestGetFilterForMetric(t *testing.T) {
"compute",
"compute.googleapis.com/firewall/dropped_bytes_count",
metricsRequester{config: config{Region: "us-east1"}, logger: logger},
"metric.type=\"compute.googleapis.com/firewall/dropped_bytes_count\" AND resource.labels.zone = starts_with(\"us-east1\")",
"metric.type=\"compute.googleapis.com/firewall/dropped_bytes_count\" AND resource.labels.region = starts_with(\"us-east1\")",
},
{
"pubsub service with region in config",
Expand All @@ -70,14 +70,14 @@ func TestGetFilterForMetric(t *testing.T) {
"compute",
"compute.googleapis.com/firewall/dropped_bytes_count",
metricsRequester{config: config{Region: "us-central1", Zone: "us-central1-a"}, logger: logger},
"metric.type=\"compute.googleapis.com/firewall/dropped_bytes_count\" AND resource.labels.zone = starts_with(\"us-central1\")",
"metric.type=\"compute.googleapis.com/firewall/dropped_bytes_count\" AND resource.labels.region = starts_with(\"us-central1\")",
},
{
"compute uptime with partial region",
"compute",
"compute.googleapis.com/instance/uptime",
metricsRequester{config: config{Region: "us-west"}, logger: logger},
"metric.type=\"compute.googleapis.com/instance/uptime\" AND resource.labels.zone = starts_with(\"us-west\")",
"metric.type=\"compute.googleapis.com/instance/uptime\" AND resource.labels.region = starts_with(\"us-west\")",
},
{
"compute uptime with partial zone",
Expand All @@ -91,7 +91,7 @@ func TestGetFilterForMetric(t *testing.T) {
"compute",
"compute.googleapis.com/instance/uptime",
metricsRequester{config: config{Region: "us-*"}, logger: logger},
"metric.type=\"compute.googleapis.com/instance/uptime\" AND resource.labels.zone = starts_with(\"us-\")",
"metric.type=\"compute.googleapis.com/instance/uptime\" AND resource.labels.region = starts_with(\"us-\")",
},
{
"compute uptime with wildcard in zone",
Expand Down