nsqd: dogstatsd support with generic tagged metrics

[dm03514]

Dogstatsd is a superset of statsd which supports tagging metrics. Tagging metrics allow for more generic metric names with specific values being differentiated by tags.

For example in the message_count metric the metric is specific to the topic name and channel name "topic.%s.channel.%s.message_count"

The dogstats-ish approach could be a more generic metric name:

nsq.message_count

and would then assign tags topic:{{ TOPIC_NAME }} and channel:{{ CHANNEL_NAME}}.

Tagging allows for really generic datadog dashboards, wildcard filtering, and more generic alerts.

I was thinking of a new standalone datadog provider, enabled by command line opt?

[mreiferson]

Thanks for the question.

I wrote https://github.com/mreiferson/ddstatsd to handle the translation of dot-namespaced to native datadog tags. It's unlikely we'll build in direct support for datadog into nsq.

[dm03514]

@mreiferson if I contribute the feature you wouldn't be open to incorporating it in the project?

[mreiferson]

Open to hear what @jehiah and @ploxiln think, but I don't think we should be directly integrating with enterprise vendor-specific protocols.

[dm03514]

Ah, sorry, didn't think of it that way :p

[ploxiln]

Agreed, but I'd be somewhat open to a datadog output module, if we had a more modular/pluggable stats interface and e.g. an influxdb output module as well. So, if it was an optional module among a few others, that didn't affect the core code, and which we would classify as "contrib" and "not really supported", I would be OK with it.

[dm03514]

Is this a good forum to discuss the contrib architecture and how it fits in with nsq core? My employer is a pretty large user of NSQ and I would love to contribute something!

[mreiferson]

Sure, I'll re-open.

[mreiferson]

Thanks for your interest 😍

[ploxiln]

I should clarify that there is not yet a "contrib" architecture. What I meant was just a modular architecture for stats exporting in which (eventually) one module could be less "maintained". (Also, that was just my opinion.)

But, it is definitely possible to contribute to nsq without any "contrib" classification or architecture! It just requires finding a good fit between what you need and have the time for, and what the project needs.

[ploxiln]

... I think @mreiferson thought you might be interested in trying to write this "modular stats export" architecture. And see also an analogous issue for nsqadmin: #732

[dm03514]

Ah I read #732, def interested in what a modular stats architecture may look like, or ways to decouple it from core. We have workarounds where we query for nsq statsd metrics and tag them using a generic metric using dogstats tags and @mreiferson ddstatsd utility looks like it fulfills the requirements, but we already have a pretty large nsqd deployment, and if it's possible for nsqd to generate the datadog stats it would be a gigantic win to reduce our operational complexity.

What could a plugin architecture look like? (i'm not sure of how this would look in GO since i'm really new to GO).

It looks like the touchpoints for the statsd support are:

• nsqd.go enables statsd loop when opts are present
• apps/nsqd/nsqd.go flag
• nsqd/options.go

Would pluggable stats ideally be maintained in a different repo?

If pluggable contrib modules are in nsqio/nsq could you imagine a setup similar to statsd? Where statsd is in the nsqd package and implements a method on then *NSQD instance, and has access to all of its variable? Or a more generic and controlled access framework??

[ploxiln]

I don't think we would want separate repositories or separate plugin libraries/files, I think we would prefer these "modules" to be in this repo and compiled into nsqd / nsqadmin. The idea is for the code for them to be relatively isolated and with a relatively simple and generic interface. There's an example of what I mean in how https://github.com/bitly/oauth2_proxy handles the various providers (but I think it may be possible to do better in terms of cmdline/config options).

But those are just my thoughts ...

[dm03514]

Would contrib modules only have access to nsqd public methods? or would they have access to private ones too? It looks like statsd and dogstats poll NSQD.GetStats(), would that hold up for other contrib modules?? Could they be passed the instance of *NSQD? and poll it however they want? If that's possible would a contrib logger have to be created?

Also would contrib modules have to define their own options, independent of nsqd options to help reduce coupling? Finally (sorry :p) would you want contrib to be generic enough to operate on a GetStats() interface? Or would it be ok to have NSQD specific contrib and nsqlookupd/nsqadmin specific contrib?