Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[action] [PR:20024] Fix flaky capture service start (#20024) #20130

Merged
merged 1 commit into from
Sep 4, 2024

Conversation

mssonicbld
Copy link
Collaborator

Why I did it

Dropping control character (message sent when XSUB connects to XPUB as part of ZMQ Proxy setup to notify that subscription has been made) in do capture has been flaky since control character is not guaranteed to be the first message sent if there are events (like event-down-ctr) being published to XSUB.

Scenarios

  1. Control character is sent and is first message when starting capture service

eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1
eventd#eventd#eventd: :- do_capture: Received subscription message when XSUB connects to XPUB

  1. Events like event-down ctr is sent before control character

eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 17 sonic-events-host
eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 0 0 4 0 0 0 1 d 103 {"sonic-events-host:event-stopped-ctr":{"ctr_name":"EVENTD","timestamp":"2024-08-27T00:02:51.407518Z"}} 1 r 36 3357542f-bae1-458f-a804-660e620d21f5 1 s 1 9 1 t 19 1724716971407591080
heartbeat_ctrl: Set heartbeat_ctrl pause=1
do_capture: Received subscription message when XSUB connects to XPUB

  1. Control character is not sent at all

eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1

  1. Control character is delayed and not caught when starting capture service, but is then caught after causing deserialize error.

do_capture: Receiving event from source: 22 serialization::archive 18 17 sonic-events-host, will read second part of event
deserialize: deserialize Failed: input stream errorstr[0:64]:(#001) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&
zmq_read_part: Failed to deserialize part rc=-2
zmq_read_part: last:errno=11
zmq_message_read: Failure to read part1 rc=-2
zmq_message_read: last:errno=11

We can cover these scenarios by just dropping the control character inside zmq_message_read as part of events_common in swsscommon (different PR). In this PR we will remove such handling logic and make sure that empty events that will be sent by control character are ignored.

Work item tracking
  • Microsoft ADO (number only):28728116

How I did it

Remove logic for handling control character

How to verify it

UT and sonic-mgmt test cases.

#### Why I did it

Dropping control character (message sent when XSUB connects to XPUB as part of ZMQ Proxy setup to notify that subscription has been made) in do capture has been flaky since control character is not guaranteed to be the first message sent if there are events (like event-down-ctr) being published to XSUB.

Scenarios

1) Control character is sent and is first message when starting capture service

`eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1`
`eventd#eventd#eventd: :- do_capture: Received subscription message when XSUB connects to XPUB`

2) Events like event-down ctr is sent before control character

`eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 17 sonic-events-host`
`eventd#eventd#eventd: :- run: Dropping Message: 22 serialization::archive 18 0 0 4 0 0 0 1 d 103 {"sonic-events-host:event-stopped-ctr":{"ctr_name":"EVENTD","timestamp":"2024-08-27T00:02:51.407518Z"}} 1 r 36 3357542f-bae1-458f-a804-660e620d21f5 1 s 1 9 1 t 19 1724716971407591080`
`heartbeat_ctrl: Set heartbeat_ctrl pause=1`
`do_capture: Received subscription message when XSUB connects to XPUB`

3) Control character is not sent at all

`eventd#eventd#eventd: :- heartbeat_ctrl: Set heartbeat_ctrl pause=1`

4) Control character is delayed and not caught when starting capture service, but is then caught after causing deserialize error.

`do_capture: Receiving event from source: 22 serialization::archive 18 17 sonic-events-host, will read second part of event`
`deserialize: deserialize Failed: input stream errorstr[0:64]:(sonic-net#1) data type: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&`
`zmq_read_part: Failed to deserialize part rc=-2`
`zmq_read_part: last:errno=11`
`zmq_message_read: Failure to read part1 rc=-2`
`zmq_message_read: last:errno=11`

We can cover these scenarios by just dropping the control character inside zmq_message_read as part of events_common in swsscommon (different PR). In this PR we will remove such handling logic and make sure that empty events that will be sent by control character are ignored.

##### Work item tracking
- Microsoft ADO **(number only)**:28728116

#### How I did it

Remove logic for handling control character

#### How to verify it

UT and sonic-mgmt test cases.
@mssonicbld
Copy link
Collaborator Author

Original PR: #20024

@mssonicbld mssonicbld mentioned this pull request Sep 4, 2024
11 tasks
@mssonicbld mssonicbld merged commit e9fec15 into sonic-net:202311 Sep 4, 2024
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants