-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add filtering by job in stat, tap, top; fix panic #1904
Conversation
Filtering by Kubernetes job was not supported. Also filtering by any unknown type caused a panic. Add filtering support by Kubernetes job, with special case mapping `job` to `k8s_job`, to not conflict with Prometheus' job label. Fix panic when unknown type specified as a `--from` or `--to` flag. Fix `job` label from `linkerd-proxy` overwriting Prometheus `job` label at collection time. This caused all metrics collected by proxy sidecars in Kubernetes jobs to be collected into an incorrect Prometheus job, rather than the expected `linkerd-proxy` Prometheus job. Fix `unsupported resource type` tap error message incorrectly printing the target resource rather than the destination. Set `--controller-log-level debug` in `install_test.go` for easier debugging. Expose `slow-cooker`'s metrics via a k8s service in the tap integration test, to validate proxy requests with a job as destination. Fixes #1872 Part of #627 Signed-off-by: Andrew Seigner <siggy@buoyant.io>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⭐️ Huge improvement, and hooray for more tests! I just had a few small wording nits, but otherwise lgtm.
cli/cmd/tap.go
Outdated
* deployments | ||
* namespaces | ||
* pods | ||
* replicationcontrollers | ||
* services (only supported as a "--to" resource)`, | ||
* services (only supported as a --to resource) | ||
* jobs (only supported as a --from or --to)`, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For consistency with the services wording, I'd write this as:
- jobs (only supported as a --from or --to resource)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, on closer inspection, it looks like tap and top only support a --to
flag.
bin/linkerd -l kltest top deploy -n kltest-tap-test --from job/slow-cooker
Error: unknown flag: --from
Offhand it seems like it should be possible for them to support a --from
flag, but I don't think we should implement that as part of this PR. In the meantime, can you change this to:
- jobs (only supported as a --to resource)
@@ -491,6 +491,9 @@ data: | |||
- source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job] | |||
action: replace | |||
target_label: k8s_job | |||
# drop __meta_kubernetes_pod_label_linkerd_io_proxy_job | |||
- action: labeldrop | |||
regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Wow, nice catch.
cli/cmd/top.go
Outdated
* deployments | ||
* namespaces | ||
* pods | ||
* replicationcontrollers | ||
* services (only supported as a "--to" resource)`, | ||
* services (only supported as a --to resource) | ||
* jobs (only supported as a --from or --to)`, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment here as in tap.go:
- jobs (only supported as a --to resource)
if err != nil { | ||
return pb.Resource{}, err | ||
} | ||
|
||
return res[0], err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😱
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Filtering by Kubernetes job was not supported. Also filtering by any unknown
type caused a panic.
Add filtering support by Kubernetes job, with special case mapping
job
tok8s_job
, to not conflict with Prometheus' job label.Fix panic when unknown type specified as a
--from
or--to
flag.Fix
job
label fromlinkerd-proxy
overwriting Prometheusjob
label atcollection time. This caused all metrics collected by proxy sidecars in
Kubernetes jobs to be collected into an incorrect Prometheus job, rather than
the expected
linkerd-proxy
Prometheus job.Fix
unsupported resource type
tap error message incorrectly printing thetarget resource rather than the destination.
Set
--controller-log-level debug
ininstall_test.go
for easier debugging.Expose
slow-cooker
's metrics via a k8s service in the tap integration test, tovalidate proxy requests with a job as destination.
Fixes #1872
Part of #627
Signed-off-by: Andrew Seigner siggy@buoyant.io
Validating commands
Deploy integration tests
bin/fast-build bin/test-run `pwd`/bin/linkerd
Send traffic to a Kubernetes job
View traffic to/from Kubernetes job