Support dogstatd with tags #165
Comments
|
It may but I don't know. Someone else implemented the statsd support. Can you give an example ? |
|
In it's simplest form: I also found a dogstatsd client written in Go. https://github.com/ooyala/go-dogstatsd Instead of having a bunch of metrics named explicitly per consul agent, we would have them sending with the same metric name and use tags to identify host, etc. |
|
Is that compatible with the standard statsd client? |
|
I found one written by Datadog themselves. https://github.com/DataDog/datadog-go/tree/master/statsd https://github.com/DataDog/datadog-go/tree/master/statsd I’m pretty certain this supports traditional statsd as well as extended tagging support for Dogstatsd users. I will confirm with Datadog. |
|
To my chagrin, it doesn't support standard statsd. The datagram format is different. http://docs.datadoghq.com/guides/dogstatsd/#datagram-format I'm discussing some plausible solutions with a couple other engineers. |
|
If they are substantially different then we can just add it as another metric provider. If they were compatible then it could have made sense to just keep one of the two. You want to try a PR or want me to work on this? |
|
The reporter you might be looking for? Both of these look like they woud do the job: https://github.com/ooyala/go-dogstatsd and https://github.com/DataDog/datadog-go . I'm a hack of a Go dev :) Perhaps if you have the cycles you can try? Thanks again for all the hard work on this project. Looking forward to pushing some serious traffic at a tier of Fabio LBs! |
|
ok, but then you'll have to wait a couple of days since I'm reworking the the parser for the configuration to support the path stripping that people seem to really want - besides other things. |
|
That sounds fantastic. Thanks! |
|
Hey folks, is there any news on this functionality? |
|
Not yet. Just back from vacation and digging through the backlog. |
|
Any love for this? :D |
|
I should have some time this week. I'll wrap up the other change I'm working on and then have a look. You're up for testing?
|
|
@magiconair Yep, I can test! :) |
|
Same here
|
|
Looks like there is an open issue on the go-metrics library to add tags support: rcrowley/go-metrics#135 but that isn't going anywhere. We need to define an option to define the tags per route. What about Since the |
|
Right now I am assuming that the tags are static per route. Is that a valid assumption? |
|
I've written up a small statsd proxy that can convert fabio statsd path into tagged metrics; https://github.com/bownty/statsd-rewrite-proxy :) |
|
@magiconair for me the tags would be per route, ala https://github.com/bownty/statsd-rewrite-proxy/blob/master/rules.go#L5-L7 |
|
tagged metrics is a bit tricky, since they also require the actual statsd path name to change, for it to make sense within UIs like datadog update not sure if its relevant in this project or upstream somewhere, but while writing the tool I noticed the following (invalid) metric being exposed looks like the float is badly encoded |
|
@jippi Created pubnub/go-metrics-statsd#1 to fix #207 |
This code is a proof-of-concept to implement statsd metrics with tags which work on dogstatd. This is not the final code and will change before being merged.
|
@jippi and @killcity I've pushed a proof-of-concept into the branch which hopefully does what you want. You need to add This is not the final code since I've hacked this into the pubnub/statsd implementation. I'd just like to check that I got the format right and that it does what it should do. Could you have a look please? |
|
@magiconair for tagged metrics to be useful inside datadog UI, the path need to be "tag" less. e.g. then in the UI you can select So for me, tagged metrics is about a new path structure where the "value" from the normal statsd path is moved into named tags |
|
Got it. Thx
|
This code is a proof-of-concept to implement statsd metrics with tags which work on dogstatd. This is not the final code and will change before being merged.
|
I think we would want to auto publish the following tags (which would be used in conjuction with the paths that appear in Datadog):
and the following paths:
Can you guys think of any others? Does this make sense? One thing I'm not totally sure of is if we want to use:
We would need to auto publish all the various response codes along with the ones listed above sans "code:". |
|
@killcity so you're also in favor of pushing the status code as a tag instead of the metric name just like @jippi suggested? The main reason the change takes a bit longer is that the go-metrics library doesn't natively support tags which means that depending on the metrics backend fabio needs to report different metric names and put values into different places (e.g. service name, urlprefix, status code, ...) I'm mulling over adding a full metrics abstraction layer which handles this or add a handful |
|
Sorry for the long delay. Yes, I am also in favor of pushing the status code as a separate tag. Thanks for working on this one. |
This patch refactors the metrics support within fabio to achieve several goals: * support labels for DogstatD, Prometheus and others * support raw events for statsd and others without aggregation * support multiple backends simultaneously to support migration * make integration of new metrics backends and/or libraries simple * being fully backwards compatible to not break existing setups One of the main challenges is that fabio generates names for the metrics targets through templates and that these templates are configurable. A metrics backend which supports labels needs to generate a different name for the same metric than one that does not support labels. Combining this with the need to support multiple different metrics backends at runtime means that none of the existing metrics libraries like go-kit/metrics and hashicorp/go-metrics is a good fit. However, they can be used as drivers which reduces the need for testing and makes integration of new backends simpler since fabio does not need to rely on a single metrics library. This patch is work in progress with the new metrics code in the metrics4 package and the old metrics code in the 'metrics_old' package where it is kept for reference until it has been migrated or removed. The basic architecture should be good enough to add more providers and functionality so that the community can help. Right now there are two providers "flat" and "label" to demonstrate the concepts. They provide counter and timers in statsd/dogstatd format and can be configured with -metrics.target=flat[,label]. Other providers should be added in the same way. The metrics_old code should be converted to the new provider scheme. The go-kit/metrics library currently supports most of the necessary drivers and is the preferred way of integrating new drivers. The list below is a probably incomplete list of things that need to be done: * The configuration is not used except for metrics.target * The drivers from the metrics_old package need to be migrated * Additional drivers from go-kit need to be added * The old rcrowley/go-metrics code relies on 'registries' for namespacing. After the routing table has changed the registry needs to be pruned of services which no longer exist so that go-metrics stops reporting them. * The RX/TX counters in the TCP and TCP+SNI proxy are probably sub-optimal or broken. * Some counters may not have been initialized properly, e.g. WSConn * WSConn should probably be a gauge which also needs to be set to 0 on startup. * The approach of injecting a noop metrics provider if none has been set has already bitten me once since some metrics are reported while others are not. I need to think about this a bit more. Fixes #126 Fixes #165 Fixes #211 Fixes #253 Fixes #326 Fixes #327 Fixes #371 Closes #329 Closes #331 Closes #334 Closes #335
|
Please see #211 (comment) |
|
@leprechau This is another metrics-related topic that would be good to review, as datadog is quite popular and the metrics integration is already possible via the existing statsd interface. |
This patch refactors the metrics support within fabio to achieve several goals: * support labels for DogstatD, Prometheus and others * support raw events for statsd and others without aggregation * support multiple backends simultaneously to support migration * make integration of new metrics backends and/or libraries simple * being fully backwards compatible to not break existing setups One of the main challenges is that fabio generates names for the metrics targets through templates and that these templates are configurable. A metrics backend which supports labels needs to generate a different name for the same metric than one that does not support labels. Combining this with the need to support multiple different metrics backends at runtime means that none of the existing metrics libraries like go-kit/metrics and hashicorp/go-metrics is a good fit. However, they can be used as drivers which reduces the need for testing and makes integration of new backends simpler since fabio does not need to rely on a single metrics library. This patch is work in progress with the new metrics code in the metrics4 package and the old metrics code in the 'metrics_old' package where it is kept for reference until it has been migrated or removed. The basic architecture should be good enough to add more providers and functionality so that the community can help. Right now there are two providers "flat" and "label" to demonstrate the concepts. They provide counter and timers in statsd/dogstatd format and can be configured with -metrics.target=flat[,label]. Other providers should be added in the same way. The metrics_old code should be converted to the new provider scheme. The go-kit/metrics library currently supports most of the necessary drivers and is the preferred way of integrating new drivers. The list below is a probably incomplete list of things that need to be done: * The configuration is not used except for metrics.target * The drivers from the metrics_old package need to be migrated * Additional drivers from go-kit need to be added * The old rcrowley/go-metrics code relies on 'registries' for namespacing. After the routing table has changed the registry needs to be pruned of services which no longer exist so that go-metrics stops reporting them. * The RX/TX counters in the TCP and TCP+SNI proxy are probably sub-optimal or broken. * Some counters may not have been initialized properly, e.g. WSConn * WSConn should probably be a gauge which also needs to be set to 0 on startup. * The approach of injecting a noop metrics provider if none has been set has already bitten me once since some metrics are reported while others are not. I need to think about this a bit more. Fixes #126 Fixes #165 Fixes #211 Fixes #253 Fixes #326 Fixes #327 Fixes #371 Closes #329 Closes #331 Closes #334 Closes #335 Remove 'Register' methods and create objects directly. Add Gauge and use it for websocket connections Vendoring in github.com/alexcesaro/statsd First stab at integrating raw statsd from #335 Rebase off master 10/23/2020 - NJ
This patch refactors the metrics support within fabio to achieve several goals: * support labels for DogstatD, Prometheus and others * support raw events for statsd and others without aggregation * support multiple backends simultaneously to support migration * make integration of new metrics backends and/or libraries simple * being fully backwards compatible to not break existing setups One of the main challenges is that fabio generates names for the metrics targets through templates and that these templates are configurable. A metrics backend which supports labels needs to generate a different name for the same metric than one that does not support labels. Combining this with the need to support multiple different metrics backends at runtime means that none of the existing metrics libraries like go-kit/metrics and hashicorp/go-metrics is a good fit. However, they can be used as drivers which reduces the need for testing and makes integration of new backends simpler since fabio does not need to rely on a single metrics library. This patch is work in progress with the new metrics code in the metrics4 package and the old metrics code in the 'metrics_old' package where it is kept for reference until it has been migrated or removed. The basic architecture should be good enough to add more providers and functionality so that the community can help. Right now there are two providers "flat" and "label" to demonstrate the concepts. They provide counter and timers in statsd/dogstatd format and can be configured with -metrics.target=flat[,label]. Other providers should be added in the same way. The metrics_old code should be converted to the new provider scheme. The go-kit/metrics library currently supports most of the necessary drivers and is the preferred way of integrating new drivers. The list below is a probably incomplete list of things that need to be done: * The configuration is not used except for metrics.target * The drivers from the metrics_old package need to be migrated * Additional drivers from go-kit need to be added * The old rcrowley/go-metrics code relies on 'registries' for namespacing. After the routing table has changed the registry needs to be pruned of services which no longer exist so that go-metrics stops reporting them. * The RX/TX counters in the TCP and TCP+SNI proxy are probably sub-optimal or broken. * Some counters may not have been initialized properly, e.g. WSConn * WSConn should probably be a gauge which also needs to be set to 0 on startup. * The approach of injecting a noop metrics provider if none has been set has already bitten me once since some metrics are reported while others are not. I need to think about this a bit more. Fixes #126 Fixes #165 Fixes #211 Fixes #253 Fixes #326 Fixes #327 Fixes #371 Closes #329 Closes #331 Closes #334 Closes #335 Remove 'Register' methods and create objects directly. Add Gauge and use it for websocket connections Vendoring in github.com/alexcesaro/statsd First stab at integrating raw statsd from #335 Rebase off master 10/23/2020 - NJ
This patch refactors the metrics support within fabio to achieve several goals: * support labels for DogstatD, Prometheus and others * support raw events for statsd and others without aggregation * support multiple backends simultaneously to support migration * make integration of new metrics backends and/or libraries simple * being fully backwards compatible to not break existing setups One of the main challenges is that fabio generates names for the metrics targets through templates and that these templates are configurable. A metrics backend which supports labels needs to generate a different name for the same metric than one that does not support labels. Combining this with the need to support multiple different metrics backends at runtime means that none of the existing metrics libraries like go-kit/metrics and hashicorp/go-metrics is a good fit. However, they can be used as drivers which reduces the need for testing and makes integration of new backends simpler since fabio does not need to rely on a single metrics library. This patch is work in progress with the new metrics code in the metrics4 package and the old metrics code in the 'metrics_old' package where it is kept for reference until it has been migrated or removed. The basic architecture should be good enough to add more providers and functionality so that the community can help. Right now there are two providers "flat" and "label" to demonstrate the concepts. They provide counter and timers in statsd/dogstatd format and can be configured with -metrics.target=flat[,label]. Other providers should be added in the same way. The metrics_old code should be converted to the new provider scheme. The go-kit/metrics library currently supports most of the necessary drivers and is the preferred way of integrating new drivers. The list below is a probably incomplete list of things that need to be done: * The configuration is not used except for metrics.target * The drivers from the metrics_old package need to be migrated * Additional drivers from go-kit need to be added * The old rcrowley/go-metrics code relies on 'registries' for namespacing. After the routing table has changed the registry needs to be pruned of services which no longer exist so that go-metrics stops reporting them. * The RX/TX counters in the TCP and TCP+SNI proxy are probably sub-optimal or broken. * Some counters may not have been initialized properly, e.g. WSConn * WSConn should probably be a gauge which also needs to be set to 0 on startup. * The approach of injecting a noop metrics provider if none has been set has already bitten me once since some metrics are reported while others are not. I need to think about this a bit more. Fixes #126 Fixes #165 Fixes #211 Fixes #253 Fixes #326 Fixes #327 Fixes #371 Closes #329 Closes #331 Closes #334 Closes #335 Remove 'Register' methods and create objects directly. Add Gauge and use it for websocket connections Vendoring in github.com/alexcesaro/statsd First stab at integrating raw statsd from #335 Rebase off master 10/23/2020 - NJ
This patch refactors the metrics support within fabio to achieve several goals: * support labels for DogstatD, Prometheus and others * support raw events for statsd and others without aggregation * support multiple backends simultaneously to support migration * make integration of new metrics backends and/or libraries simple * being fully backwards compatible to not break existing setups One of the main challenges is that fabio generates names for the metrics targets through templates and that these templates are configurable. A metrics backend which supports labels needs to generate a different name for the same metric than one that does not support labels. Combining this with the need to support multiple different metrics backends at runtime means that none of the existing metrics libraries like go-kit/metrics and hashicorp/go-metrics is a good fit. However, they can be used as drivers which reduces the need for testing and makes integration of new backends simpler since fabio does not need to rely on a single metrics library. This patch is work in progress with the new metrics code in the metrics4 package and the old metrics code in the 'metrics_old' package where it is kept for reference until it has been migrated or removed. The basic architecture should be good enough to add more providers and functionality so that the community can help. Right now there are two providers "flat" and "label" to demonstrate the concepts. They provide counter and timers in statsd/dogstatd format and can be configured with -metrics.target=flat[,label]. Other providers should be added in the same way. The metrics_old code should be converted to the new provider scheme. The go-kit/metrics library currently supports most of the necessary drivers and is the preferred way of integrating new drivers. The list below is a probably incomplete list of things that need to be done: * The configuration is not used except for metrics.target * The drivers from the metrics_old package need to be migrated * Additional drivers from go-kit need to be added * The old rcrowley/go-metrics code relies on 'registries' for namespacing. After the routing table has changed the registry needs to be pruned of services which no longer exist so that go-metrics stops reporting them. * The RX/TX counters in the TCP and TCP+SNI proxy are probably sub-optimal or broken. * Some counters may not have been initialized properly, e.g. WSConn * WSConn should probably be a gauge which also needs to be set to 0 on startup. * The approach of injecting a noop metrics provider if none has been set has already bitten me once since some metrics are reported while others are not. I need to think about this a bit more. Fixes #126 Fixes #165 Fixes #211 Fixes #253 Fixes #326 Fixes #327 Fixes #371 Closes #329 Closes #331 Closes #334 Closes #335 Remove 'Register' methods and create objects directly. Add Gauge and use it for websocket connections Vendoring in github.com/alexcesaro/statsd First stab at integrating raw statsd from #335 Rebase off master 10/23/2020 - NJ
Will Fabio eventually support (or does it already?) tagging statsd metrics? This is key when using Datadog.
The text was updated successfully, but these errors were encountered: