feature request: graphite templating - fields

[roobert]

Hi,

As far as I can work out it's not possible to use templates to create fields.

I would like to store data in graphite form in the influxdb way, for example, with the following graphite input:

sensu.metric.net.$hostname.$interface.rx_bytes <value>
sensu.metric.net.$hostname.$interface.tx_bytes <value>

I would expect to be able to translate it into the following line protocol equivalent using templates:

metric.net,host=$hostname,interface=$interface rx_bytes=<value>
metric.net,host=$hostname,interface=$interface tx_bytes=<value>

Introducing a new key word of 'field' might make sense so my template could look something like:

.measurement.measurement.hostname.interface.field

But it seems like the templates are limited in that they only allow setting of a measurement name with a single 'field' of 'value', is this correct? Apologies if I'm missing something!

Cheers!

[roobert]

I've implemented a basic prototype which works in the sense that now you can specify field names for metrics, however I've hit a limitation which I've clarified with the UDP input:

Test:

mbp0 /home/rw > echo "rob,host=mbp0 key1=v 1442853876" | nc -u localhost 8087
^C
mbp0 /home/rw > echo "rob,host=mbp0 key2=2 1442853876" | nc -u localhost 8087
^C

Same query run before and after test:

> select * from rob
name: rob
---------
time       host     key1
1442853876 mbp0     1
> select * from rob
name: rob
---------
time       host     key1     key2
1442853876 mbp0              2

This implies that there is no way to UPDATE a timestamp with additional field data.

What this means is that unless the graphite input is modified to do some sort of caching/buffering (to allow time for a group of metrics to be grouped into a single transaction) - there is no way for influxdb to store graphite metrics in the 'influxdb way' where a single timestamp has tags with multiple fields.

Is this something that is lightly to happen? Would a third party application infront of influxdb that translates graphite data to influxdb line protocol (similar to statsd) be a better idea..?

[otoolep]

This implies that there is no way to UPDATE a timestamp with additional field data.

That is correct. This is not possible within the InfluxDB system.

[pauldix]

When 0.9.5 ships, updates to a single field will be possible.

[roobert]

OK great, thanks peeps!

Would love some feedback on implementation detail in the linked PR. I should be able to finish off the patch/tests/docs by the time the new influxdb storage engine is added to the nightly builds so with any luck we may be able to get this feature into 0.9.5..? :)

[otoolep]

Sounds like a reasonable idea. Any comment @jwilder ?

[roobert]

Main questions regarding implementation:

• should multiple field tags be allowed, in the same way as the measurement tag is?
• is something like this valid: mytag.measurement*.field
• should field* be implemented

[jwilder]

Personally, I would model these measurements slightly differently. Instead of a measurement named metric.net with multiple fields, I would suggest using a measurement name of net_rx_bytes and net_tx_bytes with a single field of value.

The reason I usually suggest that is for discoverability/usability as well as performance. For example, when you run SHOW MEASUREMENTS, you will get both measurements back and the query is always the same: select value from xyz where ....

If the measurement is metric.net, SHOW MEASUREMENTS returns metric.net and you need to run SHOW FIELD KEYS FROM metric.net to get the list of fields and then build your query from there. I've found that much more cumbersome to use in practice.

From a storage perspective, multiple fields need to scanned and read even if they are not used in the query. If there is a single value field, only that series data is read. Whether this has an impact for you is dependent on your environment, but you won't know until you have a lot of data written. The new storage engine should make this point moot though.

In any event, a minimalist feature set would be better here since each feature adds latency to each write request that is handled:

• multiple field tags sounds reasonable and can be implemented efficiently
• fields after a wildcard measurement seems problematic. I would disallow that for now.
• field*also seems problematic if combined with measurement*. e.g. service.measurement*.field* would be non-deterministic on a measurement like sensu.metric.net.host.interface.eth0.tx_bytes. I would not support field* for now.

[roobert]

@jwilder, thanks very much for your response. I'm all in favour of simplicity.

Personally I prefer taking the performance hit and using fields. We've been storing a lot of metrics in this way using the native line protocol and so the reason for this feature request is that I'd like organisational consistency between our app metrics and system metrics (which are mostly created by 3rd party graphite plugins).

In terms of query building, we do it through grafana and so the extra steps aren't really a problem for us.

I'll finish the patch ASAP.

Cheers,

[jwilder]

Fixed via #4178