processors.regex doesn't works well. or works unstable #4787

Dees7 · 2018-10-02T11:29:18Z

Relevant telegraf.conf:

[global_tags]                       
[agent]                             
  interval = "5s"                   
  round_interval = true             
  metric_batch_size = 500           
  metric_buffer_limit = 1000        
  collection_jitter = "0s"          
  flush_interval = "10s"            
  flush_jitter = "0s"               
  precision = ""                    
  debug = true                      
  quiet = false                     
  logfile = ""                      
  hostname = ""                     
  omit_hostname = true              
[[processors.printer]]              
[[processors.regex]]                
    [[processors.regex.tags]]       
      key = "topic"                 
      pattern = ".*/(.*)/.*"        
      replacement = "${1}"          
      result_key = "topicname"      
[[inputs.mqtt_consumer]]            
  servers = ["tcp://localhost:1883"]
  qos = 0                           
  connection_timeout = "30s"        
  topics = ["test/#"]               
  persistent_session = true         
  client_id = "telegraftest"        
  username = "name"                 
  password = "pass"    
  name_override = "mqttc"           
  tag_keys = ["hostname"]           
  data_format = "json"              
[[outputs.file]]                    
   files = ["stdout"]               
   data_format = "graphite"

System info:

Telegraf unknown (git: master 3268937)
Description: Raspbian GNU/Linux 9.4 (stretch)
mosquitto is an MQTT v3.1 broker.

Steps to reproduce:

Satart telegraf.
publish topoc. all ok
restart telegraf
publish topic. processors.regex doesnt works
..
n. processors.regex begins work again after 5-10 telegram restart.

Expected behavior:

tag topicname=sonoff

Actual behavior:

no tag "topicname" after restart

Additional info:

publishing:

mosquitto_pub -u 'user' -P 'pass' -t 'test/sonoff/RESULT' -m '{"POWER1":13,"TEMP":41.1,"hostname":"mine"}'
mosquitto_pub -u 'user' -P 'pass' -t 'test/sonoff/RESULT' -m '{"POWER1":13,"TEMP":41.1,"hostname":"mine"}'

testing as client subscribtion with mosquitto_sub:

test/sonoff/RESULT {"POWER1":13,"TEMP":41.1,"hostname":"mine"}
test/sonoff/RESULT {"POWER1":13,"TEMP":41.1,"hostname":"mine"}

Telegraf log (looks good topicname=sonoff)

[root@duino:influxdb]# /usr/bin/telegraf --debug -config /etc/telegraf/telegraftest.conf
2018-10-02T10:51:52Z D! Attempting connection to output: file
2018-10-02T10:51:52Z D! Successfully connected to output: file
2018-10-02T10:51:52Z I! Starting Telegraf
2018-10-02T10:51:52Z I! Loaded inputs: inputs.mqtt_consumer
2018-10-02T10:51:52Z I! Loaded aggregators:
2018-10-02T10:51:52Z I! Loaded processors: regex printer
2018-10-02T10:51:52Z I! Loaded outputs: file
2018-10-02T10:51:52Z I! Tags enabled:
2018-10-02T10:51:52Z I! Agent Config: Interval:5s, Quiet:false, Hostname:"", Flush Interval:10s
2018-10-02T10:51:55Z I! MQTT Client Connected
2018-10-02T10:51:55Z I! MQTT Client Connected
2018-10-02T10:51:55Z E! Error in plugin [inputs.mqtt_consumer]: E! MQTT Connection lost
error: EOF
MQTT Client will try to reconnect
2018-10-02T10:51:55Z E! Error in plugin [inputs.mqtt_consumer]: E! MQTT Connection lost
error: read tcp [::1]:57884->[::1]:1883: use of closed network connection
MQTT Client will try to reconnect
2018-10-02T10:51:55Z I! MQTT Client Connected
2018-10-02T10:51:55Z E! Error in plugin [inputs.mqtt_consumer]: E! MQTT Connection lost
error: EOF
MQTT Client will try to reconnect
2018-10-02T10:51:55Z I! MQTT Client Connected
2018-10-02T10:51:56Z I! MQTT Client Connected
2018-10-02T10:51:56Z E! Error in plugin [inputs.mqtt_consumer]: E! MQTT Connection lost
error: read tcp [::1]:57892->[::1]:1883: use of closed network connection
MQTT Client will try to reconnect
2018-10-02T10:51:56Z E! Error in plugin [inputs.mqtt_consumer]: E! MQTT Connection lost
error: EOF
MQTT Client will try to reconnect
2018-10-02T10:51:56Z I! MQTT Client Connected
2018-10-02T10:51:56Z I! MQTT Client Connected
mqttc,hostname=mine,topic=test/sonoff/RESULT,topicname=sonoff TEMP=41.1,POWER1=13 1538477518180205009
2018-10-02T10:52:05Z D! Output [file] buffer fullness: 1 / 1000 metrics.
mine.test-sonoff-RESULT.sonoff.mqttc.TEMP 41.1 1538477518
mine.test-sonoff-RESULT.sonoff.mqttc.POWER1 13 1538477518
2018-10-02T10:52:05Z D! Output [file] wrote batch of 1 metrics in 487.031µs
2018-10-02T10:52:06Z I! Hang on, flushing any cached metrics before shutdown
2018-10-02T10:52:06Z D! Output [file] buffer fullness: 0 / 1000 metrics.

Telegraf log with no topicname

[root@duino:influxdb]# /usr/bin/telegraf --debug -config /etc/telegraf/telegraftest.conf
2018-10-02T10:52:10Z D! Attempting connection to output: file
2018-10-02T10:52:10Z D! Successfully connected to output: file
2018-10-02T10:52:10Z I! Starting Telegraf
2018-10-02T10:52:10Z I! Loaded inputs: inputs.mqtt_consumer
2018-10-02T10:52:10Z I! Loaded aggregators:
2018-10-02T10:52:10Z I! Loaded processors: printer regex
2018-10-02T10:52:10Z I! Loaded outputs: file
2018-10-02T10:52:10Z I! Tags enabled:
2018-10-02T10:52:10Z I! Agent Config: Interval:5s, Quiet:false, Hostname:"", Flush Interval:10s
2018-10-02T10:52:15Z I! MQTT Client Connected
2018-10-02T10:52:15Z I! MQTT Client Connected
mqttc,hostname=mine,topic=test/sonoff/RESULT POWER1=13,TEMP=41.1 1538477535301600165
2018-10-02T10:52:21Z I! Hang on, flushing any cached metrics before shutdown
2018-10-02T10:52:21Z D! Output [file] buffer fullness: 1 / 1000 metrics.
mine.test-sonoff-RESULT.sonoff.mqttc.POWER1 13 1538477535
mine.test-sonoff-RESULT.sonoff.mqttc.TEMP 41.1 1538477535
2018-10-02T10:52:21Z D! Output [file] wrote batch of 1 metrics in 267.291µs

mosquito.log

Tue 2 Oct 15:51:55 +05 2018 1538477515: Received SUBSCRIBE from telegraftest
Tue 2 Oct 15:51:55 +05 2018 1538477515: test/# (QoS 0)
Tue 2 Oct 15:51:55 +05 2018 1538477515: Sending SUBACK to telegraftest
Tue 2 Oct 15:51:56 +05 2018 1538477516: Sending CONNACK to telegraftest (1, 0)
Tue 2 Oct 15:51:56 +05 2018 1538477516: Sending CONNACK to telegraftest (1, 0)
Tue 2 Oct 15:51:56 +05 2018 1538477516: Sending CONNACK to telegraftest (1, 0)
Tue 2 Oct 15:51:58 +05 2018 1538477518: Sending CONNACK to mosqpub/24949-duino (0, 0)
Tue 2 Oct 15:51:58 +05 2018 1538477518: Received PUBLISH from mosqpub/24949-duino (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:51:58 +05 2018 1538477518: Sending PUBLISH to mosqsub/13070-duino (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:51:58 +05 2018 1538477518: Sending PUBLISH to telegraftest (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:51:58 +05 2018 1538477518: Received DISCONNECT from mosqpub/24949-duino



Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending CONNACK to telegraftest (1, 0)
Tue 2 Oct 15:52:15 +05 2018 1538477535: Received SUBSCRIBE from telegraftest
Tue 2 Oct 15:52:15 +05 2018 1538477535: test/# (QoS 0)
Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending SUBACK to telegraftest
Tue 2 Oct 15:52:15 +05 2018 1538477535: Received SUBSCRIBE from telegraftest
Tue 2 Oct 15:52:15 +05 2018 1538477535: test/# (QoS 0)
Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending SUBACK to telegraftest
Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending CONNACK to mosqpub/25036-duino (0, 0)
Tue 2 Oct 15:52:15 +05 2018 1538477535: Received PUBLISH from mosqpub/25036-duino (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending PUBLISH to mosqsub/13070-duino (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:52:15 +05 2018 1538477535: Sending PUBLISH to telegraftest (d0, q0, r0, m0, 'test/sonoff/RESULT', ... (43 bytes))
Tue 2 Oct 15:52:15 +05 2018 1538477535: Received DISCONNECT from mosqpub/25036-duino
```

The text was updated successfully, but these errors were encountered:

Dees7 · 2018-10-02T11:30:54Z

I tested Telegraf 1.8.0. The same behavior

danielnelson · 2018-10-04T01:29:54Z

I think this is the same bug as in #4594, lets discuss further on that issue.

Dees7 · 2018-10-04T08:11:13Z

No. This is other bug.
MQTT works well. and data recived but processed wrong

#first start
mqttc,hostname=mine,topic=test/sonoff/RESULT,topicname=sonoff TEMP=41.1,POWER1=13 1538477518180205009

#restart
mqttc,hostname=mine,topic=test/sonoff/RESULT POWER1=13,TEMP=41.1 1538477535301600165

There is no topicname=sonoff after restart
As we can see in log TEMP=41.1 and other are recived by telegraf

danielnelson · 2018-10-04T19:18:49Z

This may just be an issue of which order the two processors run in. Unless you add the order option it is undefined. Can you retest with these additions:

[[processors.printer]]
  order = 2
[[processors.regex]]
  order = 1
  [[processors.regex.tags]]
    key = "topic"
    pattern = ".*/(.*)/.*"
    replacement = "${1}"
    result_key = "topicname"

Dees7 · 2018-10-06T15:56:19Z

Yes. It helps.
Is it possible to set "order" option in [[outputs.influxdb]] ?

glinton · 2018-10-08T15:49:28Z

@Dees7 there is no "order" config for the influxdb output, are you asking if you can define the order of the tags/fields being output?

Dees7 · 2018-10-08T17:29:20Z

Yes, I ask if I can define order of tagging and output to influxdb. Something like:

[[inputs.mqtt_consumer]]
  order = 1
  ..
  tag_keys = ["hostname","POWER","POWER1"]
[[processors.converter]]
  [processors.converter.tags]
    order = 2
    float = ["POWER*"]
[[processors.regex]]
  [[processors.regex.tags]]
    order = 3
    key = "topic"
    pattern = ".*/(.*)/.*"
    replacement = "${1}"
    result_key = "topicname"
[[processors.printer]]
  order = 4
  ..
[[outputs.influxdb]]
  order = 5
  ..

Because I have no "topicname" in output to influxdb (and POWERs were added as tag not float) if it was not printed with processors.printer.
So now I have in DB:
POWER(float) POWER1(float) POWER1_1(tag) POWER2(float) POWER2_1(tag) (cause bug-order?)

danielnelson · 2018-10-08T18:02:44Z

Inputs and Outputs do not have an order option because they run before and after the processors and so there is no need for them to be ordered.

If you are still having issues after adding the order option to the processors, can you show the full config after the new changes?

Dees7 · 2018-10-10T04:33:04Z

Ok. Now it looks as unstable connection to mqtt.
I cant catch the bug. Issue can be closed.
Thank you.

danielnelson · 2018-10-10T18:03:06Z

Keep an eye on #4594 for updates on the MQTT issue.

erkexzcx · 2020-09-22T08:15:15Z

Using below options should be part of documentation... Would be such a time saver!

order = 1

glinton added the area/mqtt label Oct 2, 2018

Dees7 changed the title ~~processors.regex doesn't works well. or work unstable~~ processors.regex doesn't works well. or works unstable Oct 3, 2018

danielnelson closed this as completed Oct 4, 2018

Dees7 mentioned this issue Oct 4, 2018

MQTT input plugin disconnecting from Mosquitto #4594

Closed

danielnelson reopened this Oct 4, 2018

danielnelson closed this as completed Oct 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

processors.regex doesn't works well. or works unstable #4787

processors.regex doesn't works well. or works unstable #4787

Dees7 commented Oct 2, 2018 •

edited

Loading

Dees7 commented Oct 2, 2018

danielnelson commented Oct 4, 2018

Dees7 commented Oct 4, 2018

danielnelson commented Oct 4, 2018

Dees7 commented Oct 6, 2018

glinton commented Oct 8, 2018

Dees7 commented Oct 8, 2018

danielnelson commented Oct 8, 2018

Dees7 commented Oct 10, 2018

danielnelson commented Oct 10, 2018

erkexzcx commented Sep 22, 2020

processors.regex doesn't works well. or works unstable #4787

processors.regex doesn't works well. or works unstable #4787

Comments

Dees7 commented Oct 2, 2018 • edited Loading

Relevant telegraf.conf:

System info:

Steps to reproduce:

Expected behavior:

Actual behavior:

Additional info:

Dees7 commented Oct 2, 2018

danielnelson commented Oct 4, 2018

Dees7 commented Oct 4, 2018

danielnelson commented Oct 4, 2018

Dees7 commented Oct 6, 2018

glinton commented Oct 8, 2018

Dees7 commented Oct 8, 2018

danielnelson commented Oct 8, 2018

Dees7 commented Oct 10, 2018

danielnelson commented Oct 10, 2018

erkexzcx commented Sep 22, 2020

Dees7 commented Oct 2, 2018 •

edited

Loading