Events producer #954

renukamanavalan · 2022-03-01T22:44:20Z

SONiC to stream events occurring in SONiC switch (e.g BGP flap, process not running, ...) in real time via gNMI subscription model.
The events are defined using versioned YANG schema.
For more details, please refer the doc.

Repo	PR title	State
sonic-buildimage	Event libswsscommon deps
sonic-buildimage	Event libswsscommon deps
sonic-buildimage	Streaming structured events implementation
sonic-buildimage	Fix PR build failure
sonic-buildimage	Add events to host and create rsyslog_plugin deb pkg
sonic-buildimage	Add Structured Events w/ YANG Models
sonic-buildimage	Add Yang Models for structured events
sonic-buildimage	Add YANG model and unit tests for additional structured events
sonic-buildimage	Publish additional events
sonic-buildimage	Add rsyslog plugin regex for select operation failure
sonic-buildimage	Add YANG model for alpm parity error
sonic-swss	Publish identified events via structured-events channel
sonic-gnmi	Telemetry support for streaming events
sonic-gnmi	Events client: Ensure all go routines exit upon client disconnect.
sonic-gnmi	Structured events: Publish as JsonIetfVal instead of StringVal
sonic-gnmi	gnmi_cli - Tool update
sonic-gnmi	Streaming events URL support "not to use cache"
sonic-utilities	Event Counters CLI
sonic-swss-common	APIs to support streaming structured events
sonic-swss-common	Events: APIs to set/get global options

venkatmahalingam · 2022-06-17T06:47:59Z

I was thinking about 2 approaches in the meeting today. @renukamanavalan @bandaru-viswanath, @dgsudharsan @praveen-li we can discuss the pros and cons in the next meeting.

1) Introduce event specific container and reference to the list used for the config-DB table.
For example, in the port table YANG file, we can do the following,

https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-port.yang

       container PORT {    -> This is 1:1 representation of config-DB schema
		description "PORT part of config_db.json";

		list PORT_LIST {
                       key "name";
                       leaf name {
				type string {
					length 1..128;
				}
	           } 
                    ..............
                    }
        }
        **container PORT_STATE {  -> This container can be used for events, if PORT_STATE table name is not good, we can use 
                                                     better name.
            config false;
             list PORT_LIST {
                 key "name";
               leaf name {
                             type leafref {
                                  path "../../../PORT/PORT_LIST/name;
			}
               leaf oper_status { .. }
       }**

}

2) Enhance the container e.g PORT defined already for the config-DB schema to include state information e.g port_state_info as well.

grouping port_state_info {
<<<<< read-only attributes>>>>>>
leaf oper_status { .. }
}
container PORT {
description "PORT part of config_db.json";

		list PORT_LIST {
                       key "name";
                       leaf name {
				type string {
					length 1..128;
				}
	           } 
                    ..............
                      uses port_state_info;
                    }
        }

renukamanavalan · 2022-06-21T21:08:57Z

I was thinking about 2 approaches in the meeting today. @renukamanavalan @bandaru-viswanath, @dgsudharsan @praveen-li we can discuss the pros and cons in the next meeting.

1) Introduce event specific container and reference to the list used for the config-DB table. For example, in the port table YANG file, we can do the following,

https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-port.yang
       container PORT {    -> This is 1:1 representation of config-DB schema
		description "PORT part of config_db.json";

		list PORT_LIST {
                       key "name";
                       leaf name {
				type string {
					length 1..128;
				}
	           } 
                    ..............
                    }
        }
        **container PORT_STATE {  -> This container can be used for events, if PORT_STATE table name is not good, we can use 
                                                     better name.
            config false;
             list PORT_LIST {
                 key "name";
               leaf name {
                             type leafref {
                                  path "../../../PORT/PORT_LIST/name;
			}
               leaf oper_status { .. }
       }**
}

2) Enhance the container e.g PORT defined already for the config-DB schema to include state information e.g port_state_info as well.

grouping port_state_info { <<<<< read-only attributes>>>>>> leaf oper_status { .. } } container PORT { description "PORT part of config_db.json";
		list PORT_LIST {
                       key "name";
                       leaf name {
				type string {
					length 1..128;
				}
	           } 
                    ..............
                      uses port_state_info;
                    }
        }

Thanks Venkat!
Here is a 3rd option.

module sonic-events-swss {
…
        container if-state { 
               leaf ifname {
                     type string;
                     description "Interface name";
               }
               type leafref {
                      path "/port:sonic-port/port:PORT/port:PORT_LIST/port:name";
               }

The following are the reasons behind.

The consumer of these events are not running inside the switch, but external and have no access to config.
The state reported by these events are not saved inside the switch and hence never accessed from within the switch.
As events are from switches running multiple different versions, we put a restriction that YANG events module should not break backward compatibility, but config schema does not have such restriction and no need either.
Multiple different events reported an object may not fit logically in the config/state YANG model for the corresponding object.
For example, PFC-storm & dhcp-relay events on an i/f does not fit in the YANG model of interface object.
The consumer are looking for events across all switches for correlating & reporting and count of switches is often few tens of thousands in numbers. Also, the config among the switches could be using multiple different YANG versions, potentially.

renukamanavalan · 2022-06-23T16:18:21Z

@venkatmahalingam, can we conclude on this discussion above?

venkatmahalingam · 2022-06-26T03:09:05Z

@venkatmahalingam, can we conclude on this discussion above?

A gNMI client could subscribe for events with optional filter on event source in streaming mode. Below shows the command & o/p for subscribing all, and receiving BGP events.

gnmic --target events --path "/events/" --mode STREAM --stream-mode ON_CHANGE

The instance data would indicate YANG module path & revision that is required for validation.

o/p
{
"sonic-events-bgp:bgp-state": {
"timestamp": "2022-08-17T02:39:21.286611Z",
"ip": "100.126.188.90",
"status": "down"
}
}

Instead of doing the above generic events subscription, if we can follow either approach 1 or 2, it will be cleaner to subscribe particular table state information, right? in the future, we can enhance the subscription for config changes as well.

I agree that if it's just an event, we dont need to store it in the DB, we can use another event YANG but what we use is the state information (i.e interface admin/oper status, BGP session status..etc), also, we should support get operation as well on the state information, right?

renukamanavalan · 2022-06-27T22:27:17Z

ta would indica

An explicit decision is made to support only one gNMI client - Please refer the section "Event exporting".
At the rate of 10k events/second, with offline cache support & missed count tracking, supporting multiple can be expensive.
On the other hand, looking at the consumer requirements/design model, a client receiving at 10K/sec should be light weighted and end consumers are often looking at events across thousands of switches. Hence the receiving client, which is 1:1 with SONiC switch is expected to dump the events into an external storage and the events consumers query/watch for updates at the storage. Often the consumers, need to collect info from multiple connected switches to make a call.

Hence gNMI client can only subscribe for /events which is all.

The "state" is not the only event reported on an i/f, e.g. PFC-Storm. With the absolute path provided as leafref, helps consumer identify object where needed.

venkatmahalingam · 2022-07-01T00:00:53Z

ta would indica

An explicit decision is made to support only one gNMI client - Please refer the section "Event exporting".
At the rate of 10k events/second, with offline cache support & missed count tracking, supporting multiple can be expensive.
On the other hand, looking at the consumer requirements/design model, a client receiving at 10K/sec should be light weighted and end consumers are often looking at events across thousands of switches. Hence the receiving client, which is 1:1 with SONiC switch is expected to dump the events into an external storage and the events consumers query/watch for updates at the storage. Often the consumers, need to collect info from multiple connected switches to make a call.

Hence gNMI client can only subscribe for /events which is all.

The "state" is not the only event reported on an i/f, e.g. PFC-Storm. With the absolute path provided as leafref, helps consumer identify object where needed.

I understand that state is not the only event being reported, I think, we can report non-state information (absolutely doesnt make sense to have it in DB) in event YANG model.

What's the design to handle both the cases 1) interface oper. status changes (stored in the STATE_DB) 2) PFC storm (not in DB)?

For the first case, enhance the existing YANG to have state information as well and for the second case, introduce new event specific YANGs and leafref to existing tables if applicable.

@renukamanavalan @bandaru-viswanath, @dgsudharsan any concerns on the above design?

renukamanavalan · 2022-07-01T00:04:11Z

ta would indica

An explicit decision is made to support only one gNMI client - Please refer the section "Event exporting".
At the rate of 10k events/second, with offline cache support & missed count tracking, supporting multiple can be expensive.
On the other hand, looking at the consumer requirements/design model, a client receiving at 10K/sec should be light weighted and end consumers are often looking at events across thousands of switches. Hence the receiving client, which is 1:1 with SONiC switch is expected to dump the events into an external storage and the events consumers query/watch for updates at the storage. Often the consumers, need to collect info from multiple connected switches to make a call.

Hence gNMI client can only subscribe for /events which is all.
The "state" is not the only event reported on an i/f, e.g. PFC-Storm. With the absolute path provided as leafref, helps consumer identify object where needed.

I understand that state is not the only event being reported, I think, we can report non-state information (absolutely doesnt make sense to have it in DB) in event YANG model.

What's the design to handle both the cases 1) interface oper. status changes (stored in the STATE_DB) 2) PFC storm (not in DB)?

For the first case, enhance the existing YANG to have state information as well and for the second case, introduce new event specific YANGs and leafref to existing tables if applicable.

@renukamanavalan @bandaru-viswanath, @dgsudharsan any concerns on the above design?

We persist neither in DB.

bandaru-viswanath · 2022-07-01T10:31:56Z

ta would indica

An explicit decision is made to support only one gNMI client - Please refer the section "Event exporting".
At the rate of 10k events/second, with offline cache support & missed count tracking, supporting multiple can be expensive.
On the other hand, looking at the consumer requirements/design model, a client receiving at 10K/sec should be light weighted and end consumers are often looking at events across thousands of switches. Hence the receiving client, which is 1:1 with SONiC switch is expected to dump the events into an external storage and the events consumers query/watch for updates at the storage. Often the consumers, need to collect info from multiple connected switches to make a call.

Hence gNMI client can only subscribe for /events which is all.
The "state" is not the only event reported on an i/f, e.g. PFC-Storm. With the absolute path provided as leafref, helps consumer identify object where needed.

I understand that state is not the only event being reported, I think, we can report non-state information (absolutely doesnt make sense to have it in DB) in event YANG model.

What's the design to handle both the cases 1) interface oper. status changes (stored in the STATE_DB) 2) PFC storm (not in DB)?

For the first case, enhance the existing YANG to have state information as well and for the second case, introduce new event specific YANGs and leafref to existing tables if applicable.

@renukamanavalan @bandaru-viswanath, @dgsudharsan any concerns on the above design?

Looks good to me @venkatmahalingam

Added sample gnmi commands

zhangyanzhao · 2022-11-11T23:01:13Z

@renukamanavalan can you please add the code PRs by referring to #806? Thanks

praveen-li · 2022-11-14T17:21:46Z

Hi Zain Nice report for all PR under event producer. Thanks, Regards Praveen

…

renukamanavalan added 5 commits February 16, 2022 16:36

Create events-producer.md

6f913d2

Update events-producer.md

873632e

Update events-producer.md

327c49b

Update events-producer.md

8c32e00

Update events-producer.md

f7263d8

renukamanavalan self-assigned this Mar 1, 2022

renukamanavalan added 10 commits March 2, 2022 16:46

Update events-producer.md

4879c71

Update events-producer.md

37bc1cd

Update events-producer.md

a843a42

Update events-producer.md

606aba9

Update events-producer.md

13482ba

Update events-producer.md

c9646cf

Update events-producer.md

c3244b0

Update events-producer.md

1df7ca9

Update events-producer.md

848e414

Update events-producer.md

438e005

renukamanavalan mentioned this pull request Mar 12, 2022

Stream structured log data to telemetry's UDP listener sonic-net/sonic-buildimage#10226

Closed

6 tasks

renukamanavalan added 13 commits March 15, 2022 15:34

Update events-producer.md

d763726

Merge branch 'Azure:master' into events-producer

fb50e64

Update events-producer.md

4a7b301

Update events-producer.md

c191f72

Update events-producer.md

f991bce

Update events-producer.md

b4e4079

Update events-producer.md

7adcf3f

Update events-producer.md

aec5111

Update events-producer.md

40c366d

Update events-producer.md

fe428ab

Update events-producer.md

e5507ac

Update events-producer.md

52aeba4

Update events-producer.md

7bc772c

renukamanavalan added 4 commits May 12, 2022 15:49

Update events-producer.md

dd6e136

Update events-producer.md

dc430b2

Update events-producer.md

2d62e7f

Update events-producer.md

a1a9f2c

Update events-producer.md

9decb8c

zbud-msft mentioned this pull request Jul 9, 2022

Add YANG models for structured events sonic-net/sonic-buildimage#11404

Closed

7 tasks

This was referenced Jul 13, 2022

APIs to support streaming structured events sonic-net/sonic-swss-common#649

Closed

telemetry client updated to stream structured events sonic-net/sonic-telemetry#111

Open

renukamanavalan mentioned this pull request Jul 29, 2022

telemetry client updated to stream structured events sonic-net/sonic-gnmi#5

Closed

6 tasks

renukamanavalan mentioned this pull request Aug 5, 2022

New events client to support Streaming structured events. renukamanavalan/sonic-telemetry#3

Closed

zbud-msft mentioned this pull request Aug 8, 2022

APIs to support streaming structured events sonic-net/sonic-swss-common#667

Merged

zbud-msft mentioned this pull request Aug 26, 2022

Streaming structured events implementation sonic-net/sonic-buildimage#11848

Merged

6 tasks

renukamanavalan mentioned this pull request Sep 12, 2022

[orchagent]: Publish identified events via structured-events channel sonic-net/sonic-swss#2446

Merged

renukamanavalan added 2 commits October 4, 2022 14:14

Update events-producer.md

4ced908

Added sample gnmi commands

Update events-producer.md

5677242

zbud-msft approved these changes Nov 11, 2022

View reviewed changes

Merge branch 'master' into events-producer

1b66b9d

renukamanavalan merged commit 42a9067 into sonic-net:master Nov 11, 2022

renukamanavalan deleted the events-producer branch December 4, 2022 23:39

jeff-yin mentioned this pull request Sep 7, 2023

Event/alarm framework HLD update. Integrate with event producer framework. #1409

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Events producer #954

Events producer #954

renukamanavalan commented Mar 1, 2022 •

edited by zbud-msft

Loading

venkatmahalingam commented Jun 17, 2022 •

edited

Loading

renukamanavalan commented Jun 21, 2022 •

edited

Loading

renukamanavalan commented Jun 23, 2022

venkatmahalingam commented Jun 26, 2022

renukamanavalan commented Jun 27, 2022

venkatmahalingam commented Jul 1, 2022 •

edited

Loading

renukamanavalan commented Jul 1, 2022

bandaru-viswanath commented Jul 1, 2022

zhangyanzhao commented Nov 11, 2022

praveen-li commented Nov 14, 2022 via email

Events producer #954

Events producer #954

Conversation

renukamanavalan commented Mar 1, 2022 • edited by zbud-msft Loading

venkatmahalingam commented Jun 17, 2022 • edited Loading

renukamanavalan commented Jun 21, 2022 • edited Loading

renukamanavalan commented Jun 23, 2022

venkatmahalingam commented Jun 26, 2022

renukamanavalan commented Jun 27, 2022

venkatmahalingam commented Jul 1, 2022 • edited Loading

renukamanavalan commented Jul 1, 2022

bandaru-viswanath commented Jul 1, 2022

zhangyanzhao commented Nov 11, 2022

praveen-li commented Nov 14, 2022 via email

renukamanavalan commented Mar 1, 2022 •

edited by zbud-msft

Loading

venkatmahalingam commented Jun 17, 2022 •

edited

Loading

renukamanavalan commented Jun 21, 2022 •

edited

Loading

venkatmahalingam commented Jul 1, 2022 •

edited

Loading