Backport of Add exporting of Envoy metrics to HCP for linked clusters into release/1.4.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #370 to be assessed for backporting due to the inclusion of the label backport/1.4.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@loshz
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul-dataplane/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Overview

This builds off the work of the Consul PR: hashicorp/consul#20257

We want to let users send Envoy metrics to HCP for use in service-level dashboards and coming service topology map features. We also want to support that without requiring that users deploy a dedicated Consul Telemetry Collector.

The behavior added to Consul Dataplane in this PR is:

1. Watch for changes to TelemetryState in Consul via the resource API
2. On creation of a TelemetryState resource, create an HTTP client to push metrics to HCP's Consul metrics endpoint
3. Start periodically (minutely) scraping Envoy's stats endpoint
4. Convert those Prometheus metrics to OTLP metrics
5. POST the metrics to HCP

More details are available in the RFC: https://docs.google.com/document/d/1gh3_zUtdIDeDPIH7z3d0-kkx1scB2oZLi7yuYg88PK8

Testing

Unit/integration Testing

$go test -timeout 30s -tags integration github.com/hashicorp/consul-dataplane/pkg/hcp/telemetry

ok  	github.com/hashicorp/consul-dataplane/pkg/hcp/telemetry	0.788s	coverage: 90.7% of statements

go test -timeout 30s -tags integration github.com/hashicorp/consul-dataplane/pkg/hcp/telemetry/otlphttp

ok  	github.com/hashicorp/consul-dataplane/pkg/hcp/telemetry/otlphttp	0.651s	coverage: 88.9% of statements

E2E Testing with HCP

To test this with real values I:

1. created a resource in consul-core by calling the resource API with grpcurl (using a real endpoint, filters, labels, and service principal)
2. validated that metrics were pushed through the HCP metrics endpoint and available for querying

# create a resource manually
grpcurl -d @ \
  -plaintext \
  -protoset pkg/consul.protoset \
  127.0.0.1:8502 \
  hashicorp.consul.resource.ResourceService.Write \
<<EOF
  {
    "resource": {
      "id": {
        "type": {
          "group": "hcp",
          "group_version": "v2",
          "kind": "TelemetryState"
        },
        "name": "global"
      },
      "metadata": {
        "consul.io/hcp/telemetry-state/debug/skip-deletion": "true"
      },
      "data": {
        "@type": "types.googleapis.com/hashicorp.consul.hcp.v2.TelemetryState",
        "resource_id": "organization/de9edc98-06f2-45b3-96ba-e4b0ced62717/project/a2c4c2d9-0087-4a1b-a9d6-b6030d55ea5c/hashicorp.consul.global-network-manager.cluster/josh-test-cdp-metrics",
        "client_id": "foo",
        "client_secret": "barr",
        "hcp_config": {
          "auth_url": "https://auth.idp.hcp.dev"
        },
        "proxy": {
          "http_proxy": "http://192.168.0.135:8080"
        },
        "metrics": {
          "endpoint": "http://consul-telemetry.hcp.dev/otlp/v1/metrics",
          "include_list": [".+"],
          "disabled": false,
          "labels": {
            "foo": "bar"
          }
        }
      }
    }
  }
EOF

E2E Testing with HCP and a Proxy Server

TelemetryState has an HTTP proxy for use in environments where egress to the internet (ie HCP) is restricted to a single point of egress. This is similar to the Datadog agent's proxy settings.

message TelemetryState {
  ProxyConfig proxy = 5;
}

message ProxyConfig {
  // HttpProxy configures the http proxy to use for HTTP (non-TLS) requests.
  string http_proxy = 1;

  // HttpsProxy configures the http proxy to use for HTTPS (TLS) requests.
  string https_proxy = 2;

  // NoProxy can be configured to include domains which should NOT be forwarded through the configured http proxy
  repeated string no_proxy = 3;
}

To test pushing of metrics through a proxy, I set up Nginx locally as a forward proxy:

http {
...
    server {
        listen               8080;
        server_name  192.168.0.135;

        location /otlp/v1/metrics {
            proxy_ssl_verify off;
            proxy_pass https://consul-telemetry.hcp.dev/otlp/v1/metrics;
        }

And confirmed that metrics were pushed thru it to CTGW (both in logs in metrics in Grafana):

192.168.0.135 - - [25/Jan/2024:19:56:51 -0500] "POST http://consul-telemetry.hcp.dev/otlp/v1/metrics HTTP/1.1" 200 23 "-" "consul-dataplane/1.4.0-dev (linux/arm64)" "-"

---

Overview of commits

• e83b3db - ac3aac8 - 12ffa70 - 685aaa9 - 2d3743e - 643752e - 7ea6df8 - 902d030 - be5eea4 - 7feae84 - c7d0a27 - c72f13d - a98b860 - cc5bf0e - d14b874 - ef69491 - bd9acd6 - 470cdd7 - fae579e - 84ef4f9 - 670adce - f4d2eac - c03f915 - 57b22c5 - e00c344 - 45dcc16 - 9a74994

[hashicorp-cla]

All committers have signed the CLA.

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR