GCP Ops agent metrics not working

[zahidadeel-zazmic]

Hi. I am trying to use the following policy with a GCP project to discover the percent usage of disks using ops agent metrics agent.googleapis.com/disk/percent_used. When I execute the following policy, it doesn't show me anything except an empty list.

The same metrics work fine for me in Metrics Explorer on GCP.

As per my observation, the metric-key value is not correct. I just started exploring Cloud Custodian, so I am not quite sure about the usage of metric-key.

policies:
  - name: list-all-disks
    resource: gcp.disk
    filters:
      - type: metrics
        name: agent.googleapis.com/disk/percent_used
        aligner: ALIGN_MEAN
        days: 30
        metric-key: resource.labels.instance_id
        filter: ' metric.labels.state = "used"'
        value: 0
        op: greater-than
        group-by-fields:
          - resource.labels.instance_id
          - metric.labels.device

I looked into c7n_gcp.client.ServiceClient.execute_query code. For the mentioned policy, I observed that the constructed request URL looks like below:

https://monitoring.googleapis.com/v3/projects/DUMMY-NAME-376517/timeSeries?filter=metric.type+=+"agent.googleapis.com/disk/percent_used"+AND+(+resource.labels.instance_id+=+"feapis"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-0189f03f-9g07"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-0189f03f-tj1m"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-376517-gke-pool-1-9570d1c8-omxt"+OR+resource.labels.instance_id+=+"instance-1"+OR+resource.labels.instance_id+=+"vm-2"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-daa14302-0o7i"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-daa14302-knvb"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-376517-gke-pool-1-5dacc53e-x4f0"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-2d93465a-9l6h"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-3-DUMMY-NAME-3-2d93465a-qfuz"+OR+resource.labels.instance_id+=+"gke-DUMMY-NAME-376517-gke-pool-1-8ccfba1c-u4lh"+OR+resource.labels.instance_id+=+"instance-stage"+)++AND++metric.labels.state+=+"used"&interval.startTime=2024-01-08T18:26:19Z&interval.endTime=2024-02-07T18:26:19Z&aggregation.alignmentPeriod=2592000.0s&aggregation.perSeriesAligner=ALIGN_MEAN&aggregation.crossSeriesReducer=REDUCE_NONE&aggregation.groupByFields=resource.labels.instance_id&aggregation.groupByFields=metric.labels.device&view=FULL&alt=json

If I debug and remove the metric-key based filter from the request URL, it can fetch the metric results and the modified request URL looks like below.

https://monitoring.googleapis.com/v3/projects/DUMMY-NAME-376517/timeSeries?filter=metric.type+=+"agent.googleapis.com/disk/percent_used"+AND++metric.labels.state+=+"used"&interval.startTime=2024-01-08T18:33:15Z&interval.endTime=2024-02-07T18:33:15Z&aggregation.alignmentPeriod=2592000.0s&aggregation.perSeriesAligner=ALIGN_MEAN&aggregation.crossSeriesReducer=REDUCE_NONE&aggregation.groupByFields=resource.labels.instance_id&aggregation.groupByFields=metric.labels.device&view=FULL&alt=json

Does it mean, the written metrics are fine but the metric-key is not correct?
I wonder why this metric-key is required and how to use it properly.
any help would be much appreciated.

Thanks.