Updated t128_tank to parse the messages #63

shriyanshk128T · 2023-10-17T17:14:14Z

Description

Converting t128_tank data to relevant fields and tags with the help of an inbuilt parser and added a configurable field.

Testing

Manual Testing
Screengrabs
Robot Testing - This robot testing is also testing the changes made for t128_peer_path in telegraf and also addressing the change of adding 1s timeout to [t128_cpu collector in plugins](https://github.com/Juniper-SSN/ssn-ssr-plugins/pull/581). The robot branch is release/5.6.12

Screengrabs

Config

[inputs]
[[inputs.t128_tank]]
topic="events"
sequence_number_field="test_index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

"fields": {
    "event_detail": "node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697556861.010:7237): pid=17448 uid=0 auid=1000 ses=59 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/1 res=success'",
    "permitted": true,
    "test_index": "423",
    "user": "root"
  },
  "name": "events",
  "tags": {
    "collector_id": "auditd",
    "host": "t137-dut2.openstacklocal",
    "node": "westA",
    "subtype": "authentication",
    "type": "admin"
  },
  "timestamp": 1697556861
}

{
  "fields": {
    "record": "{\"flows\":[{\"protocol\":6,\"direction\":\"forward\",\"destPort\":16385,\"ingressInterfaces\":[{\"device\":\"wan3\",\"network\":\"wan3\"}],\"srcAddr\":\"10.0.3.4\",\"destAddr\":\"10.0.3.23\",\"srcPort\":16384}],\"sourcePeerName\":\"T137_NorthEast\",\"identification\":{\"service\":\"_bgp_T137_West_loopback\"},\"description\":\"SecurityNotFound: No Security Policy matching , 1\",\"closeReason\":\"ServiceAreaSecurityNotFound\",\"protocol\":6,\"sessionID\":\"57a2ab51-b0bf-439a-abd9-7059c9b22c7e\",\"srcAddr\":\"10.0.0.4\",\"destAddr\":\"10.0.0.2\",\"tenant\":\"_bgp_speaker_\",\"srcPort\":41241,\"sourceType\":\"INTER_ROUTER\",\"status\":\"FAILURE\",\"destPort\":179}",
    "test_index": "12"
  },
  "name": "session-record",
  "tags": {
    "host": "t137-dut2.openstacklocal",
    "recordType": "ERROR"
  },
  "timestamp": 1697508714
}

When Index File is Provided

from specified as end

config

[inputs]
[[inputs.t128_tank]]
topic="events"
from="end"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T15:50:10Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:50:10Z I! Loaded inputs: t128_tank
2023-10-18T15:50:10Z I! Loaded aggregators:
2023-10-18T15:50:10Z I! Loaded processors:
2023-10-18T15:50:10Z I! Loaded outputs: file
2023-10-18T15:50:10Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:50:10Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:50:10Z I! [inputs.t128_tank] starting events tank read from index 581
2023-10-18T15:50:10Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643402.691:8496): pid=27054 uid=0 auid=4294967295 ses=4294967295 msg='op=success acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=success'","index_value":"582","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643402}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643513.116:8521): pid=27838 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"583","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643513}

from specified as start

config

[inputs]
[[inputs.t128_tank]]
topic="events"
from="start"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T15:52:27Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:52:27Z I! Loaded inputs: t128_tank
2023-10-18T15:52:27Z I! Loaded aggregators:
2023-10-18T15:52:27Z I! Loaded processors:
2023-10-18T15:52:27Z I! Loaded outputs: file
2023-10-18T15:52:27Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:52:27Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:52:27Z I! [inputs.t128_tank] starting events tank read from index 581
2023-10-18T15:52:27Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643402.685:8493): pid=27054 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_faillock,pam_unix acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=172.18.15.253 addr=172.18.15.253 terminal=ssh res=success'","index_value":"581","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643402}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643402.691:8496): pid=27054 uid=0 auid=4294967295 ses=4294967295 msg='op=success acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=success'","index_value":"582","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643402}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643513.116:8521): pid=27838 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"583","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643513}

from is empty

config

[inputs]
[[inputs.t128_tank]]
topic="events"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

index_file contains 580

2023-10-18T15:48:48Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:48:48Z I! Loaded inputs: t128_tank
2023-10-18T15:48:48Z I! Loaded aggregators:
2023-10-18T15:48:48Z I! Loaded processors:
2023-10-18T15:48:48Z I! Loaded outputs: file
2023-10-18T15:48:48Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:48:48Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:48:48Z I! [inputs.t128_tank] starting events tank read from index 581
2023-10-18T15:48:48Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643402.685:8493): pid=27054 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_faillock,pam_unix acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=172.18.15.253 addr=172.18.15.253 terminal=ssh res=success'","index_value":"581","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643402}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643402.691:8496): pid=27054 uid=0 auid=4294967295 ses=4294967295 msg='op=success acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=success'","index_value":"582","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643402}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643513.116:8521): pid=27838 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"583","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643513}
...
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643939.699:8671): pid=31369 uid=0 auid=1000 ses=97 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/1 res=success'","index_value":"590","permitted":true,"user":"root"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643939}

When Index File is not Provided

from specified as end

config

[inputs]
[inputs]
[[inputs.t128_tank]]
topic="events"
from="end"
sequence_number_field="index_value"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T15:43:21Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:43:21Z I! Loaded inputs: t128_tank
2023-10-18T15:43:21Z I! Loaded aggregators:
2023-10-18T15:43:21Z I! Loaded processors:
2023-10-18T15:43:21Z I! Loaded outputs: file
2023-10-18T15:43:21Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:43:21Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:43:21Z I! [inputs.t128_tank] starting events tank read from index 18446744073709551615
2023-10-18T15:43:21Z I! [inputs.t128_tank] Command started successfully

Logged out and Logged in the system to produce the event

[root@t137-dut2 centos]# tail -f tank.json
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643828.576:8647): pid=30586 uid=0 auid=1000 ses=97 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/1 res=success'","index_value":"588","permitted":true,"user":"root"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643828}

from specified as start

config

[inputs]
[[inputs.t128_tank]]
topic="events"
from="start"
sequence_number_field="index_value"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

2023-10-18T15:40:39Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:40:39Z I! Loaded inputs: t128_tank
2023-10-18T15:40:39Z I! Loaded aggregators:
2023-10-18T15:40:39Z I! Loaded processors:
2023-10-18T15:40:39Z I! Loaded outputs: file
2023-10-18T15:40:39Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:40:39Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:40:39Z I! [inputs.t128_tank] starting events tank read from index 0
2023-10-18T15:40:39Z I! [inputs.t128_tank] Command started successfully

[root@t137-dut2 centos]# tail -f tank.json
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=SYSCALL msg=audit(1697641211.914:8229): arch=c000003e syscall=159 success=yes exit=0 a0=55933ff72980 a1=1 a2=0 a3=55934086a28c items=0 ppid=1 pid=7601 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm=\"ntpd\" exe=\"/usr/sbin/ntpd\" key=\"128T\" old_time=2023-10-18T15:00:11.914000Z new_time=2023-10-18T15:00:11Z","index_value":"556","new_date_time":"2023-10-18T15:00:11Z","user":"ntp"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"ntp_adjustment","type":"system"},"timestamp":1697641211}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697642148.468:8242): pid=17980 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"557","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697642148}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697642148.474:8243): pid=17980 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"558","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697642148}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697642150.143:8244): pid=17980 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_faillock,pam_unix acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=172.18.15.253 addr=172.18.15.253 terminal=ssh res=success'","index_value":"559","permitted":true,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697642150}
...
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643516.133:8541): pid=27868 uid=0 auid=1000 ses=102 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/3 res=success'","index_value":"587","permitted":true,"user":"root"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643516}

from is empty

config

[inputs]
[[inputs.t128_tank]]
topic="events"
sequence_number_field="index_value"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T15:45:15Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:45:15Z I! Loaded inputs: t128_tank
2023-10-18T15:45:15Z I! Loaded aggregators:
2023-10-18T15:45:15Z I! Loaded processors:
2023-10-18T15:45:15Z I! Loaded outputs: file
2023-10-18T15:45:15Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:45:15Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:45:15Z I! [inputs.t128_tank] starting index from {%!s(uint64=18446744073709551615)}
2023-10-18T15:45:15Z I! [inputs.t128_tank] starting events tank read from index 18446744073709551615
2023-10-18T15:45:15Z I! [inputs.t128_tank] Command started successfully

[root@t137-dut2 centos]# tail -f tank.json
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643931.028:8659): pid=31307 uid=0 auid=1000 ses=97 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/1 res=success'","index_value":"589","permitted":true,"user":"root"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643931}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697643939.699:8671): pid=31369 uid=0 auid=1000 ses=97 msg='op=PAM:authentication grantors=pam_rootok acct=\"root\" exe=\"/usr/bin/su\" hostname=t137-dut2.openstacklocal addr=? terminal=pts/1 res=success'","index_value":"590","permitted":true,"user":"root"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697643939}

Boundary Fault in Index File

index-file contains 5000

from specified as end

config

[inputs]
[[inputs.t128_tank]]
topic="events"
from="end"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T15:58:56Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:58:56Z I! Loaded inputs: t128_tank
2023-10-18T15:58:56Z I! Loaded aggregators:
2023-10-18T15:58:56Z I! Loaded processors:
2023-10-18T15:58:56Z I! Loaded outputs: file
2023-10-18T15:58:56Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:58:56Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:58:56Z I! [inputs.t128_tank] starting events tank read from index 5001
2023-10-18T15:58:56Z I! [inputs.t128_tank] Command started successfully
2023-10-18T15:58:56Z E! [inputs.t128_tank] encountered error while parsing lines in events output, will not attempt to parse more lines: parsed boundary fault: next available index is 556
2023-10-18T15:58:56Z E! [inputs.t128_tank] events read routine exited with error: parsed boundary fault: next available index is 556
2023-10-18T15:59:01Z I! [inputs.t128_tank] starting events tank read from index 556
2023-10-18T15:59:01Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=SYSCALL msg=audit(1697641211.914:8229): arch=c000003e syscall=159 success=yes exit=0 a0=55933ff72980 a1=1 a2=0 a3=55934086a28c items=0 ppid=1 pid=7601 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm=\"ntpd\" exe=\"/usr/sbin/ntpd\" key=\"128T\" old_time=2023-10-18T15:00:11.914000Z new_time=2023-10-18T15:00:11Z","index_value":"556","new_date_time":"2023-10-18T15:00:11Z","user":"ntp"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"ntp_adjustment","type":"system"},"timestamp":1697641211}

from specified as start

config

[inputs]
[[inputs.t128_tank]]
topic="events"
from="start"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

2023-10-18T15:57:12Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T15:57:12Z I! Loaded inputs: t128_tank
2023-10-18T15:57:12Z I! Loaded aggregators:
2023-10-18T15:57:12Z I! Loaded processors:
2023-10-18T15:57:12Z I! Loaded outputs: file
2023-10-18T15:57:12Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T15:57:12Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T15:57:12Z I! [inputs.t128_tank] starting events tank read from index 5001
2023-10-18T15:57:12Z I! [inputs.t128_tank] Command started successfully
2023-10-18T15:57:12Z E! [inputs.t128_tank] encountered error while parsing lines in events output, will not attempt to parse more lines: parsed boundary fault: next available index is 556
2023-10-18T15:57:12Z E! [inputs.t128_tank] events read routine exited with error: parsed boundary fault: next available index is 556
2023-10-18T15:57:17Z I! [inputs.t128_tank] starting events tank read from index 556
2023-10-18T15:57:17Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=SYSCALL msg=audit(1697641211.914:8229): arch=c000003e syscall=159 success=yes exit=0 a0=55933ff72980 a1=1 a2=0 a3=55934086a28c items=0 ppid=1 pid=7601 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm=\"ntpd\" exe=\"/usr/sbin/ntpd\" key=\"128T\" old_time=2023-10-18T15:00:11.914000Z new_time=2023-10-18T15:00:11Z","index_value":"556","new_date_time":"2023-10-18T15:00:11Z","user":"ntp"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"ntp_adjustment","type":"system"},"timestamp":1697641211}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697642148.468:8242): pid=17980 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"557","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697642148}

from is empty

config

[inputs]
[[inputs.t128_tank]]
topic="events"
sequence_number_field="index_value"
index_file="/home/centos/filer.index"
[outputs]
[[outputs.file]]
files=["/home/centos/tank.json"]
data_format = "json"

[root@t137-dut2 centos]# telegraf --config sample.conf
2023-10-18T16:00:10Z I! Starting Telegraf 1.22.6-c3cc6fc3
2023-10-18T16:00:10Z I! Loaded inputs: t128_tank
2023-10-18T16:00:10Z I! Loaded aggregators:
2023-10-18T16:00:10Z I! Loaded processors:
2023-10-18T16:00:10Z I! Loaded outputs: file
2023-10-18T16:00:10Z I! Tags enabled: host=t137-dut2.openstacklocal
2023-10-18T16:00:10Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"t137-dut2.openstacklocal", Flush Interval:10s
2023-10-18T16:00:10Z I! [inputs.t128_tank] starting index from {%!s(uint64=0)}
2023-10-18T16:00:10Z I! [inputs.t128_tank] starting events tank read from index 5001
2023-10-18T16:00:10Z I! [inputs.t128_tank] Command started successfully
2023-10-18T16:00:10Z E! [inputs.t128_tank] encountered error while parsing lines in events output, will not attempt to parse more lines: parsed boundary fault: next available index is 556
2023-10-18T16:00:10Z E! [inputs.t128_tank] events read routine exited with error: parsed boundary fault: next available index is 556
2023-10-18T16:00:15Z I! [inputs.t128_tank] starting events tank read from index 556
2023-10-18T16:00:15Z I! [inputs.t128_tank] Command started successfully

{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=SYSCALL msg=audit(1697641211.914:8229): arch=c000003e syscall=159 success=yes exit=0 a0=55933ff72980 a1=1 a2=0 a3=55934086a28c items=0 ppid=1 pid=7601 auid=4294967295 uid=38 gid=38 euid=38 suid=38 fsuid=38 egid=38 sgid=38 fsgid=38 tty=(none) ses=4294967295 comm=\"ntpd\" exe=\"/usr/sbin/ntpd\" key=\"128T\" old_time=2023-10-18T15:00:11.914000Z new_time=2023-10-18T15:00:11Z","index_value":"556","new_date_time":"2023-10-18T15:00:11Z","user":"ntp"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"ntp_adjustment","type":"system"},"timestamp":1697641211}
{"fields":{"event_detail":"node=t137-dut2.openstacklocal type=USER_AUTH msg=audit(1697642148.468:8242): pid=17980 uid=0 auid=4294967295 ses=4294967295 msg='op=pubkey acct=\"centos\" exe=\"/usr/sbin/sshd\" hostname=? addr=172.18.15.253 terminal=ssh res=failed'","fail_reason":"pubkey","index_value":"557","permitted":false,"user":"centos"},"name":"events","tags":{"collector_id":"auditd","host":"t137-dut2.openstacklocal","node":"westA","subtype":"authentication","type":"admin"},"timestamp":1697642148}

Index file before and after

before
[root@t112-dut1 centos]# cat fileindex.index
775

after 
[root@t112-dut1 centos]# cat fileindex.index
796

Validates a Standard Valid Config :: Validate completes with no er... | PASS |
------------------------------------------------------------------------------
Validates a Nonstandard Valid Config :: Validate completes with no... | PASS |
------------------------------------------------------------------------------
Generates a Standard Valid Config :: Generate has no errors and cr... | PASS |
------------------------------------------------------------------------------
Generates a Nonstandard Valid Config :: Generate has no errors and... | PASS |
------------------------------------------------------------------------------
Configures a Standard Valid Config :: Configure has no errors and ... | PASS |
------------------------------------------------------------------------------
Configures a Nonstandard Valid Config :: Configure has no errors a... | PASS |
------------------------------------------------------------------------------
Runs a Standard Test Input :: Test input runs with no errors and p... | PASS |
------------------------------------------------------------------------------
Runs a Nonstandard Test Input :: Test input runs with no errors an... | PASS |
------------------------------------------------------------------------------
Runs a Standard Run Once :: Run once executes with no errors and p... | PASS |
------------------------------------------------------------------------------
Runs a Nonstandard Run Once :: Run once executes with no errors an... | PASS |
------------------------------------------------------------------------------
Produces a Standard Sample Agent Config :: Produces a sample agent... | PASS |
------------------------------------------------------------------------------
Produces a Nonstandard Sample Agent Config :: Produces a sample ag... | PASS |
------------------------------------------------------------------------------
Produces a Standard List of Available Inputs And Outputs :: Lists ... | PASS |
------------------------------------------------------------------------------
Produces a Nonstandard List of Available Inputs And Outputs :: Lis... | PASS |
------------------------------------------------------------------------------
Produces an Example Config for a Standard Input :: Produces a samp... | PASS |
------------------------------------------------------------------------------
Produces an Example Config for a Nonstandard Input :: Produces a s... | PASS |
------------------------------------------------------------------------------
Stops Standard Running Services :: Running services are stopped by... | PASS |
------------------------------------------------------------------------------
Stops Nonstandard Running Services :: Running services are stopped... | PASS |
------------------------------------------------------------------------------
Stops Only Services for Specified Config :: Unrelated services are... | PASS |
------------------------------------------------------------------------------
MA Service Runs Agent Setup On Start :: Unrelated services are lef... | PASS |
------------------------------------------------------------------------------
Validates Old Config are Removed :: Validate completes with no errors | PASS |
------------------------------------------------------------------------------
Verify t128_metrics Sample By Name :: Verifies that the t128_metri... | PASS |
------------------------------------------------------------------------------
Verify t128_metrics configured output :: Verifies the configured o... | PASS |
------------------------------------------------------------------------------
Verify t128_metrics file output :: Verify the t128_metrics configu... | PASS |
------------------------------------------------------------------------------
Verify t128_metrics kafka output :: Verifies the t128_metrics conf... | PASS |
------------------------------------------------------------------------------
Verify Nonstandard kafka output :: Verifies the t128_metrics confi... | PASS |
------------------------------------------------------------------------------
Verify t128_device_state Executable :: Verifies the t128_device_st... | PASS |
------------------------------------------------------------------------------
Verify t128_device_state Sample By Name :: Verifies that the t128_... | PASS |
------------------------------------------------------------------------------
Verify t128_device_state configured output :: Verifies the configu... | PASS |
------------------------------------------------------------------------------
Verify t128_device_state file output :: Verify the t128_peer_path ... | PASS |
------------------------------------------------------------------------------
Verify t128_device_state kafka output :: Verifies the t128_device_... | PASS |
------------------------------------------------------------------------------
Verify t128_peer_path Executable :: Verifies the t128_peer_path py... | PASS |
------------------------------------------------------------------------------
Verify t128_peer_path Sample By Name :: Verifies that the t128_pee... | PASS |
------------------------------------------------------------------------------
Verify t128_peer_path configured output :: Verifies the configured... | PASS |
------------------------------------------------------------------------------
Verify t128_peer_path file output :: Verify the t128_peer_path con... | PASS |
------------------------------------------------------------------------------
Verify t128_peer_path kafka output :: Verifies the t128_peer_path ... | PASS |
------------------------------------------------------------------------------
Verify t128_top_analytics Executable :: Verifies the t128_top_anal... | PASS |
------------------------------------------------------------------------------
Verify t128_top_analytics Sample By Name :: Verifies that the t128... | PASS |
------------------------------------------------------------------------------
Verify t128_top_analytics configured output :: Verifies the config... | PASS |
------------------------------------------------------------------------------
Verify t128_top_analytics file output :: Verify the t128_top_analy... | PASS |
------------------------------------------------------------------------------
Verify t128_top_analytics kafka output :: Verifies the t128_top_an... | PASS |
------------------------------------------------------------------------------
Verify t128_arp_state Executable :: Verifies the lytics t128_arp_s... | PASS |
------------------------------------------------------------------------------
Verify t128_arp_state Sample By Name :: Verifies that the t128_arp... | PASS |
------------------------------------------------------------------------------
Verify t128_arp_state configured output :: Verifies the configured... | PASS |
------------------------------------------------------------------------------
Verify t128_arp_state file output :: Verify the t128_arp_state con... | PASS |
------------------------------------------------------------------------------
Verify t128_arp_state kafka output :: Verifies the t128_arp_state ... | PASS |
------------------------------------------------------------------------------
Verify t128_graphql Sample By Name :: Verifies that the t128_graph... | PASS |
------------------------------------------------------------------------------
Verify t128_graphql configured output :: Verifies the configured o... | PASS |
------------------------------------------------------------------------------
Verify t128_graphql file output :: Verify the t128_graphql configu... | PASS |
------------------------------------------------------------------------------
Verify t128_graphql kafka output :: Verifies the t128_graphql conf... | PASS |
------------------------------------------------------------------------------
Verify t128_cpu Sample By Name :: Verifies that the t128_cpu sampl... | PASS |
------------------------------------------------------------------------------
Verify t128_cpu configured output :: Verifies the configured outpu... | PASS |
------------------------------------------------------------------------------
Verify t128_cpu file output :: Verify the t128_cpu configured outp... | PASS |
------------------------------------------------------------------------------
Verify t128_cpu kafka output :: Verifies the t128_cpu configured o... | PASS |
------------------------------------------------------------------------------
Verify t128_events Sample By Name :: Verifies that the t128_events... | PASS |
------------------------------------------------------------------------------

WAN-2496 #time 30m

plugins/inputs/t128_tank/t128_tank.go

WAN-2496 #time 30m

plugins/inputs/t128_tank/t128_tank.go

…-2496-fix-tank-parser-and-add-user-config-field

WAN-2496 #time 20m

plugins/inputs/t128_tank/t128_tank.go

plugins/inputs/t128_tank/reader.go

WAN-2496 #time 5m

WAN-2496 #time 15m

plugins/inputs/t128_tank/reader.go

WAN-2496 #time 30m

gregschrock

please add unit tests for this; it should not be hard to test

plugins/inputs/t128_tank/t128_tank.go

plugins/inputs/t128_tank/reader.go

WAN-2496 #time 1h

plugins/inputs/t128_tank/reader.go

plugins/inputs/t128_tank/t128_tank.go

WAN-2496 #time 20m

plugins/inputs/t128_tank/reader.go

plugins/inputs/t128_tank/t128_tank.go

plugins/inputs/t128_tank/t128_tank_test.go

WAN-2496 #time 15m

plugins/inputs/t128_tank/t128_tank.go

WAN-2496 #time 1h

* Sqaushed commits WAN-2321 #time 10m * addressed pr comments and added a testcase WAN-2321 #time 10m * Added a new Telegraf input for LTE (#56) * Added a new Telegraf input for events and session-records (#58) * Updated T128_peer_path Tags (#64) * Updated t128_tank to parse the messages (#63) * added a parser to parse data and send as metrics WAN-2496 #time 30m * added a new config option and updated the logic WAN-2496 #time 30m * addressed more comments and simplified logic WAN-2496 #time 20m * addressed comments WAN-2496 #time 5m * updated the index-file config WAN-2496 #time 5m * fixed an issue with last index not being saved WAN-2496 #time 15m * addressed pr comments WAN-2496 #time 30m * added data precision WAN-2496 #time 30m * addressed more comments WAN-2496 #time 1h * addressed comments and test for lastobserved value WAN-2496 #time 20m * added testcase WAN-2496 #time 20m * addressed comments WAN-2496 #time 15m * Fix timestamp precision adjustments WAN-2496 #time 1h --------- Co-authored-by: Greg Schrock <gschrock@juniper.net> * Update README.md for lte-collector * Update README.md for peer path collector --------- Co-authored-by: Shriyansh Kothari <shriyanshk@juniper.net> Co-authored-by: Kaushik Agrawal <60372242+agrawalkaushik@users.noreply.github.com> Co-authored-by: Shriyansh Kothari <100544034+shriyanshk128T@users.noreply.github.com>

added a parser to parse data and send as metrics

c3cc6fc

WAN-2496 #time 30m

shriyanshk128T requested review from BenMatase, gregschrock, agrawalkaushik and jiangsunan October 17, 2023 17:16

gregschrock reviewed Oct 17, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

added a new config option and updated the logic

8887ea1

WAN-2496 #time 30m

shriyanshk128T requested review from gregschrock and BenMatase October 18, 2023 16:02

BenMatase reviewed Oct 18, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

plugins/inputs/t128_tank/t128_tank.go Show resolved Hide resolved

gregschrock reviewed Oct 18, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

shriyanshk128T added 2 commits October 18, 2023 17:57

Merge branch 'feature/new-plugins-to-reduce-python-overhead' into WAN…

6a9b32f

…-2496-fix-tank-parser-and-add-user-config-field

addressed more comments and simplified logic

5245bfd

WAN-2496 #time 20m

shriyanshk128T requested review from BenMatase and gregschrock October 19, 2023 13:01

BenMatase approved these changes Oct 19, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

gregschrock reviewed Oct 19, 2023

View reviewed changes

plugins/inputs/t128_tank/reader.go Outdated Show resolved Hide resolved

addressed comments

a41c3cb

WAN-2496 #time 5m

shriyanshk128T requested review from gregschrock and BenMatase October 19, 2023 20:29

updated the index-file config

cbc2962

WAN-2496 #time 5m

gregschrock approved these changes Oct 20, 2023

View reviewed changes

BenMatase approved these changes Oct 20, 2023

View reviewed changes

fixed an issue with last index not being saved

1b4cff8

WAN-2496 #time 15m

gregschrock requested changes Oct 23, 2023

View reviewed changes

plugins/inputs/t128_tank/reader.go Outdated Show resolved Hide resolved

plugins/inputs/t128_tank/reader.go Outdated Show resolved Hide resolved

addressed pr comments

89f4942

WAN-2496 #time 30m

shriyanshk128T requested a review from gregschrock October 24, 2023 14:00

added data precision

3ae649a

WAN-2496 #time 30m

gregschrock reviewed Oct 24, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

gregschrock requested changes Oct 24, 2023

View reviewed changes

plugins/inputs/t128_tank/reader.go Show resolved Hide resolved

plugins/inputs/t128_tank/reader.go Show resolved Hide resolved

plugins/inputs/t128_tank/reader.go Outdated Show resolved Hide resolved

plugins/inputs/t128_tank/reader.go Outdated Show resolved Hide resolved

addressed more comments

d10c595

WAN-2496 #time 1h

gregschrock requested changes Oct 25, 2023

View reviewed changes

shriyanshk128T added 2 commits October 25, 2023 09:15

addressed comments and test for lastobserved value

3f918e6

WAN-2496 #time 20m

added testcase

198e280

WAN-2496 #time 20m

gregschrock requested changes Oct 25, 2023

View reviewed changes

addressed comments

4e05389

WAN-2496 #time 15m

shriyanshk128T requested review from BenMatase and gregschrock October 25, 2023 20:11

gregschrock requested changes Oct 27, 2023

View reviewed changes

plugins/inputs/t128_tank/t128_tank.go Outdated Show resolved Hide resolved

Fix timestamp precision adjustments

65dcd5a

WAN-2496 #time 1h

gregschrock approved these changes Oct 30, 2023

View reviewed changes

BenMatase approved these changes Oct 31, 2023

View reviewed changes

agrawalkaushik merged commit 866ffae into feature/new-plugins-to-reduce-python-overhead Nov 1, 2023

gregschrock deleted the WAN-2496-fix-tank-parser-and-add-user-config-field branch November 1, 2023 14:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated t128_tank to parse the messages #63

Updated t128_tank to parse the messages #63

shriyanshk128T commented Oct 17, 2023 •

edited

Loading

This comment was marked as resolved.

gregschrock left a comment

Updated t128_tank to parse the messages #63

Updated t128_tank to parse the messages #63

Conversation

shriyanshk128T commented Oct 17, 2023 • edited Loading

Description

Testing

Screengrabs

Config

When Index File is Provided

from specified as end

config

from specified as start

config

from is empty

config

When Index File is not Provided

from specified as end

config

Logged out and Logged in the system to produce the event

from specified as start

config

from is empty

config

Boundary Fault in Index File

from specified as end

config

from specified as start

config

from is empty

config

Index file before and after

This comment was marked as resolved.

gregschrock left a comment

Choose a reason for hiding this comment

shriyanshk128T commented Oct 17, 2023 •

edited

Loading