Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up Prometheus labels scraped from proxy #633

Merged
merged 1 commit into from
Mar 27, 2018
Merged

Clean up Prometheus labels scraped from proxy #633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions cli/cmd/testdata/install_default.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,18 +569,23 @@ data:
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
# special case k8s' "job" label, to not interfere with prometheus' "job"
# label
# __meta_kubernetes_pod_label_conduit_io_proxy_job=foo =>
# k8s_job=foo
- source_labels: [__meta_kubernetes_pod_label_conduit_io_proxy_job]
action: replace
target_label: k8s_job
# __meta_kubernetes_pod_label_conduit_io_proxy_deployment=foo =>
# k8s_deployment=foo
# deployment=foo
- action: labelmap
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
replacement: k8s_$1
# drop all labels that we just made copies of in the previous labelmap
- action: labeldrop
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
# __meta_kubernetes_pod_label_foo=bar => k8s_foo=bar
# __meta_kubernetes_pod_label_foo=bar => foo=bar
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
replacement: k8s_$1

### Grafana ###
---
Expand Down
13 changes: 9 additions & 4 deletions cli/cmd/testdata/install_output.golden
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -572,18 +572,23 @@ data:
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
# special case k8s' "job" label, to not interfere with prometheus' "job"
# label
# __meta_kubernetes_pod_label_conduit_io_proxy_job=foo =>
# k8s_job=foo
- source_labels: [__meta_kubernetes_pod_label_conduit_io_proxy_job]
action: replace
target_label: k8s_job
# __meta_kubernetes_pod_label_conduit_io_proxy_deployment=foo =>
# k8s_deployment=foo
# deployment=foo
- action: labelmap
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
replacement: k8s_$1
# drop all labels that we just made copies of in the previous labelmap
- action: labeldrop
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
# __meta_kubernetes_pod_label_foo=bar => k8s_foo=bar
# __meta_kubernetes_pod_label_foo=bar => foo=bar
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
replacement: k8s_$1

### Grafana ###
---
Expand Down
13 changes: 9 additions & 4 deletions cli/install/template.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,18 +384,23 @@ data:
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
# special case k8s' "job" label, to not interfere with prometheus' "job"
# label
# __meta_kubernetes_pod_label_conduit_io_proxy_job=foo =>
# k8s_job=foo
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been on both sides of the fence on this, but the fact that we have to prefix job to avoid conflicts makes me kinda think we should go back to prefixing all resources for consistency...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree it's unfortunate that k8s_job ends up being a special case. I think in general it will make our code slightly more readable. An example I just ran into:
#627 (comment)
...where we're referring to pod-template-hash when talking to Kubernetes and then k8s_pod_template_hash immediately after when querying Prometheus.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I'll note that Googling around yields a direct k8s->prom label mapping to be pretty standard, and what folks would naturally expect when querying Prometheus for Kubernetes data:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L266-L267

- source_labels: [__meta_kubernetes_pod_label_conduit_io_proxy_job]
action: replace
target_label: k8s_job
# __meta_kubernetes_pod_label_conduit_io_proxy_deployment=foo =>
# k8s_deployment=foo
# deployment=foo
- action: labelmap
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this action do when replacement is not specified?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config, replacement defaults to the matched group $1:

[ replacement: <string> | default = $1 ]

regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
replacement: k8s_$1
# drop all labels that we just made copies of in the previous labelmap
- action: labeldrop
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
# __meta_kubernetes_pod_label_foo=bar => k8s_foo=bar
# __meta_kubernetes_pod_label_foo=bar => foo=bar
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
replacement: k8s_$1

### Grafana ###
---
Expand Down
18 changes: 9 additions & 9 deletions controller/script/simulate-proxy/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,13 @@ var (
// for reference: https://github.com/runconduit/conduit/blob/master/doc/proxy-metrics.md#labels
labels = []string{
// kubeResourceTypes
"k8s_daemon_set",
"k8s_deployment",
"daemon_set",
"deployment",
"k8s_job",
"k8s_replication_controller",
"k8s_replica_set",
"replication_controller",
"replica_set",

"k8s_pod_template_hash",
"pod_template_hash",
"namespace",

// constantLabels
Expand Down Expand Up @@ -196,10 +196,10 @@ func (s *simulatedProxy) generateProxyTraffic() {
// newConduitLabel creates a label map to be used for metric generation.
func (s *simulatedProxy) newConduitLabel(destinationPod string, isResponseLabel bool) prom.Labels {
labelMap := prom.Labels{
"direction": randomRequestDirection(),
"k8s_deployment": s.deploymentName,
"authority": "world.greeting:7778",
"namespace": s.namespace,
"direction": randomRequestDirection(),
"deployment": s.deploymentName,
"authority": "world.greeting:7778",
"namespace": s.namespace,
}
if labelMap["direction"] == "outbound" {
labelMap["dst_deployment"] = destinationPod
Expand Down
13 changes: 9 additions & 4 deletions doc/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,23 @@ rich telemetry data to your cluster. Simply add the following item to your
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
# special case k8s' "job" label, to not interfere with prometheus' "job"
# label
# __meta_kubernetes_pod_label_conduit_io_proxy_job=foo =>
# k8s_job=foo
- source_labels: [__meta_kubernetes_pod_label_conduit_io_proxy_job]
action: replace
target_label: k8s_job
# __meta_kubernetes_pod_label_conduit_io_proxy_deployment=foo =>
# k8s_deployment=foo
# deployment=foo
- action: labelmap
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
replacement: k8s_$1
# drop all labels that we just made copies of in the previous labelmap
- action: labeldrop
regex: __meta_kubernetes_pod_label_conduit_io_proxy_(.+)
# __meta_kubernetes_pod_label_foo=bar => k8s_foo=bar
# __meta_kubernetes_pod_label_foo=bar => foo=bar
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
replacement: k8s_$1
```

That's it! Your Prometheus cluster is now configured to scrape Conduit's
Expand Down
27 changes: 13 additions & 14 deletions doc/proxy-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,15 @@ will correspond to `k8s_*` Prometheus labels:
* `job`: The Prometheus job responsible for the collection, typically
`conduit-proxy`.
* `namespace`: Kubernetes namespace that the pod belongs to.
* `k8s_deployment`: The deployment that the pod belongs to (if applicable).
* `deployment`: The deployment that the pod belongs to (if applicable).
* `k8s_job`: The job that the pod belongs to (if applicable).
* `k8s_replica_set`: The replica set that the pod belongs to (if applicable).
* `k8s_daemon_set`: The daemon set that the pod belongs to (if applicable).
* `k8s_replication_controller`: The replication controller that the pod belongs
to (if applicable).
* `k8s_pod_template_hash`: Corresponds to the
[pod-template-hash][pod-template-hash] Kubernetes
label. This value changes during redeploys and
rolling restarts.
* `replica_set`: The replica set that the pod belongs to (if applicable).
* `daemon_set`: The daemon set that the pod belongs to (if applicable).
* `replication_controller`: The replication controller that the pod belongs to
(if applicable).
* `pod_template_hash`: Corresponds to the [pod-template-hash][pod-template-hash]
Kubernetes label. This value changes during redeploys and
rolling restarts.

Here's a concrete example, given the following pod snippet:

Expand All @@ -98,11 +97,11 @@ The resulting Prometheus labels will look like this:
```
request_total{
namespace="emojivoto",
k8s_app="vote-bot",
k8s_conduit_io_control_plane_ns="conduit",
k8s_deployment="vote-bot",
k8s_pod_template_hash="3957278789",
k8s_test="vote-bot-test",
app="vote-bot",
conduit_io_control_plane_ns="conduit",
deployment="vote-bot",
pod_template_hash="3957278789",
test="vote-bot-test",
instance="10.1.3.93:4191",
job="conduit-proxy"
}
Expand Down