Allows defining custom metrics

[unbit]

Currently uWSGI exports in its various stats subsystems only a static set of metrics.

It would be great to allow for custom defined metrics.

Metrics could be created with

metric = foo,bar,x,y,x

Then you can increase/decrese with

uwsgi.metric_inc('foo')
uwsgi.metric_dec('foo')
uwsgi.metric_set('foo', 0)

metric are 64bit unsigned values (or it is better to allow for negative values ?)

metrics could be 'attached' to loglines, so for example it could be interesting to count the number of error 500:

log-metric = foo (HTTP/1.\d 500)

this will increase 'foo' metric automagically.

Metric are automatically exported to stats server and external plugin like the carbon one

Other ideas ?

[prymitive]

uwsgi.metric_incr/dec(key, amount=1) would be better, so You could increase metric by 8 in one call.

Great idea, now uWSGI plugins could generate custom metrics without the need to patch carbon plugin. And if this will be available for apps in uWSGI API than apps could generate custom stats and that is very very useful.

[unbit]

Update:

metrics can be automatically gathered from:

• counting log line (like counting the number of listen queue errors or failed writes)
• parsing log lines using pcre grouping: you can get a specific value to use as a setter or incrementer (like adding the amount of transferred bytes)
• setting from file (like reading a cgroup memory usage)
• setting from space-splitted file (like /proc/load_avg)
• from a command (via pipe)

[prymitive]

Where will log parsing happen? Will it slow down request handling in case of complex regexp rules in metrics?
If it does happen in logger plugin than if request rate (and so log messages rate) will be higher than what logger and pcre can handle with single core (since we don't have multithreaded logger), than what will happen? Will logger drop some logs or is it going to eat memory until we hit OOM?

[unbit]

Log parsing happens here: https://github.com/unbit/uwsgi/blob/master/core/master.c#L264

(that function runs in a thread when threaded-logger if enabled)

If the logger queue is full, logs are skipped, so no parsing will happens. But eventually we can track full queues too in the metrics system.

[unbit]

i just realized that having processes/threads constantly writing to the same memory area will be overkill in terms of lock contention.

Maybe the best solution will be having a master-governed thread that takes stats updates from a non-blocking file descriptor (like happens with the offloading subsystem).

Even if it will be heavier in terms of syscall usage, we do not need to be worried about deadlocks.

We could easily implement multi-updates (a single packet can contain multiple metrics) too and we could even allows remote updates (immagine a cluster of centralized stats server)

[unbit]

Ok, let's revamp discussion (it is now an RFC). The metric subsystem is the last feature before we can start releasing 1.5 RCs.

Things to do:

First of all rrdtool and carbon plugins should became "stats pusher" plugins (so, stats pusher api should be improved)

The mongodb stats pusher plugin should be a good base. It is based on a json input, this is another thing that should change soon. (json should be generated dinamically by some kind of high-performance internal structure)

New metrics can be registered by plugins:

uwsgi_register_metric(char *name, uint64_t *value, uint64_t flags);

(name is in the form foo.bar.one.two, value is a pointer to a 64bit memory area holding the metric value)

or by options:

metric = foo.bar.one.two 30 file:/proc/loadavg

(read from /proc/loadavg every 30 seconds and put the result in the metric foo.bar.one.two)

Metrics can be "grouped" to have statistics in various time-slot/resolutions

metric-group = 30
metric-group = 3600

will hold metrics with 30 seconds resolution and 1 hour resolution.

Things returned by the stats server should be registered as metrics, so that we can dinamically generate the json chunk (it will be very hard to write such a versatile api)

Maybe we can start thinking about a passive mode (in substitution of the current stats server system) so we could return different kind of datas (in addition to json). This is low-priority.

Metric are lost after a reload, should we implement something to optionally avoid that (reloading last metrics from some storage before fully starting the instance) ?

[prymitive]

Metric are lost after a reload, should we implement something to optionally avoid that (reloading last metrics from some storage before fully starting the instance) ?

If we could create separate cache (or storage) programmatically, than metric subsystem could create it's own file based local cache/storage for metrics. Just like current persistent cache.

[unbit]

i am still unsure if it is better to allows for multiple caches (as already requested by other users) or to use key prefixes.... as the caching is now used by various areas (like ssl sessions) maybe we should make a choice...

[prymitive]

Single cache won't fit all uses since ssl works best with certain bloc ksize, while user apps or other plugins might require cache with different block size. Also we might end with various parts fighting for cache space and it's easier to figure out what's going on in each cache, shared one will be more tricky.
I'm voting for separate caches.

[unbit]

moved it to uWSGI 1.6. This should allow us to directly use the new generation caching framework as metrics storage.

[unbit]

In addition to name based mappings, like uwsgi.workers.total_requests we need to have a numeric (oid) corresponding mapping to be SNMP friendly.

uwsgi.workers.total_requests would be something like xxx.1.1.1 or uwsgi.workers.worker1.total_requests to xxx.1.1.2.1 and so on.

So, the metric will need to be build as a tree with each leaf being a value. Plugins need to register their branch on the tree.

[prymitive]

Little note so that recent mailing list thread when user noticed that 1 second carbon frequency generates gaps will not be forgotten:

To have more non-blocking approach, metrics producers (like generating metrics for requests, rss usage and so) and pushers (carbon plugin) should run in dedicated threads with queue in between.
Metrics producer should generate them and put into queue, removing oldest entry if queue is full.
Pusher should fetch data from queue and push it upstream.

[unbit]

Another update, the caching framework can now store 64bit numbers, so we could have persistent storage for free

[unbit]

Metrics subsystem has been pushed and it is almost complete. Time to write docs (and close this issue)