InfluxDB Database/Measurement Settings Based on MQTT Topic

[orcunbaslak]

Feature Request

Opening a feature request kicks off a discussion.

Proposal: Telegraf will parse the topic message and move it to appropriate database/measurement based on the MQTT topic. Like /db_1/measurement_1/# will forward all the messages in that topic to those databases/measurements.

Current behavior: Not that I know of.

Desired behavior: As I explained above.

Use case:

Well, I plant to use MQTT as a communication hub for 1.000 IoT devices. I cannot go to telegraf.conf each time and create a configuration for each topic to be parsed by namepass= and moved to their respective databases/measurments. That should be done on the fly.

Thanks.

[GeorgeMac]

@orcunbaslak could you give an example of how this would differ from the current behavior of the mqtt_consumer input plugin:

https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer#tags

Or perhaps some more details on your usecase, so that we can understand how the current plugin doesn't support it.

[orcunbaslak]

Hello,

Let me try to make it more clear.

Let's say I have 1.000 IoT devices.
Device #1 sends its metrics to topic : /db1/device1/
Device #2 sends its metrics to topic : /db1/device2/
Device #3 sends its metrics to topic : /db2/device3/
Device #4 sends its metrics to topic : /db2/device4/

So I want [[inputs.mqtt_consumer]] to listen to '#' and when it receives a message from device1 in the topic /db1/device2, It sends this message to [[outputs.influxdb]] with the appropriate database and measurement.

However with the name_pass setting; measurement is the same for all incoming messages which is something like mqtt_consumer_xxxx and database is what we've set in [[outputs.influxdb]].

I'd like to extract measurement name and database name from the topic and use the data shared in that topic go to exactly to that database/measurement pair (like passing a variable to # database_tag = "").

I don't know If that's doable in this current state but README files doesnt give the documentation needed for such config if it is possible.

[danielnelson]

The mqtt_consumer does not support setting the measurement from the topic. Your best bet is to use an input data_format that supports setting the measurement as part of the message. Most of the formats support this but one notably exception I commonly see used with MQTT is data_format="value".

You could however split the topic into two tags using the regex processor, and then use the results with the database_tag:

[[processors.regex]]
  [[processors.regex.tags]]
    key = "topic"
    pattern = '^/(\w+)/(\w+)/$'
    replacement = "${1}"
    result_key = "influxdb_database"
  [[processors.regex.tags]]
    key = "topic"
    pattern = '^/(\w+)/(\w+)/$'
    replacement = "${2}"

[bamne123]

Hi @danielnelson ,

Can we use Processor to Create Fields in db?
Above code will set the Tags,
What changes are required if I want to store those values to db into different column?

Eg. I have device_id in the topic, Which will be split using regex processor. Now rather than setting into tags now i want to store it in Device_ID column.

[danielnelson]

@bamne123 The regex processor can operate on tags or fields, but it can't convert tags to fields. What you cando is use the converter processor as a second step. Use the order option to control the order in which the processors are applied.

If you have more questions please ask at the InfluxData Community site.