[healthd] Use unix_socket_path instead of loopback ip #14843

vivekrnv · 2023-04-25T13:37:15Z

Why I did it

interfaces-config service restarts networking service, which in-turn results in loopback interface address is being removed and reassigned back

If the system-health happens to start during that instance expections and logs like this are seen:

Apr 15 18:14:49.357869 r-panther-20 ERR healthd: update system status exception:Unable to connect to redis: Cannot assign requested address
Apr 15 18:14:49.429778 r-panther-20 ERR healthd: subscribe_statedb exited- Unable to connect to redis: Cannot assign requested address
Apr 15 18:14:52.218594 r-panther-20 ERR healthd: system_service_Map_base::at
Apr 15 18:14:52.219714 r-panther-20 ERR healthd: system_service_Map_base::at
Apr 15 18:14:55.218636 r-panther-20 ERR healthd: system_service_Map_base::at
Apr 15 18:14:55.218722 r-panther-20 ERR healthd: system_service_Map_base::at

How I did it

use unix socket path

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2023-04-25T15:48:59Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2023-04-25T15:49:01Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2023-04-25T15:49:15Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2023-04-28T14:58:39Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2023-04-28T14:58:41Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2023-04-28T14:58:54Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2023-05-01T08:50:00Z

@qiluo-msft could you please help to review?

moshemos · 2023-05-03T15:01:03Z

@qiluo-msft can you please help merge it, we need this fix for our 202211 release

liat-grozovik · 2023-05-08T10:55:29Z

@Junchao-Mellanox could you please help to review the concept?

Junchao-Mellanox · 2023-05-09T01:32:12Z

Try to understand the issue. Is system-health the only service that will connect redis during loopback interface flapping?

Maybe we should fix this line https://github.com/sonic-net/sonic-buildimage/blob/a738c39328b3e069a83162fe036ae693a8bd3d27/src/system-health/health_checker/sysmonitor.py#LL34C31-L34C42 . Use unix socket instead of tcp socket.

vivekrnv · 2023-05-09T03:32:58Z

Try to understand the issue. Is system-health the only service that will connect redis during loopback interface flapping?

Maybe we should fix this line https://github.com/sonic-net/sonic-buildimage/blob/a738c39328b3e069a83162fe036ae693a8bd3d27/src/system-health/health_checker/sysmonitor.py#LL34C31-L34C42 . Use unix socket instead of tcp socket.

Regarding the first proposal, even other processes of system health will be using the loopback to connect to redis. So, the problem can still be seen even if just the system ready is updated.

There were a few instances of the same kind of issue reported. Using unix socket apparently causes some permission issues #10179 (comment)

@qiluo-msft can clarify on that.

Junchao-Mellanox · 2023-05-09T09:06:50Z

system health is a service managed by systemd, it is not a CLI. So I assume the permission issue is not relevant.

vivekrnv · 2023-05-09T10:26:40Z

system health is a service managed by systemd, it is not a CLI. So I assume the permission issue is not relevant.

Hmm, FYI another instance of where this approach is used. c93716a. Seems to be a common practice across sonic.

Junchao-Mellanox · 2023-05-09T10:33:04Z

rsyslog service is a different case. It uses UDP which depends on LO interface. System health does not rely on UDP/TCP. So, I don't see an issue to use unix socket here.

vivekrnv · 2023-05-09T10:36:12Z

rsyslog service is a different case. It uses UDP which depends on LO interface. System health does not rely on UDP/TCP. So, I don't see an issue to use unix socket here.

So you suggest moving all DBconnector initialization in system-health to Unix socket?

This reverts commit 341108e.

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…dimage into intf_cfg_dep_healthd

vivekrnv · 2023-05-10T03:06:36Z

@Junchao-Mellanox, updated with your recommendation. Can you review?

vivekrnv · 2023-05-16T17:18:43Z

@judyjoseph to review for multi-asic.

@qiluo-msft Since this service runs in host, it is same behavior for multi-asic device too.

One question I have is which user is system-health service running on ? If it is not "root" we will have permissions issues accessing the unix socket path ?
cat ./system-health.service
[Unit]
Description=SONiC system health monitor
Requires=database.service updategraph.service
After=database.service updategraph.service

[Service]
ExecStart=/usr/local/bin/healthd
Restart=always

[Install]
WantedBy=multi-user.target

This comment might help answer your question. #14843 (comment)

vivekrnv · 2023-05-18T05:00:50Z

@judyjoseph any other comments?

judyjoseph

LGTM. And please make sure it is validated in a JIT based environment -- so that there is no permission issues.

vivekrnv · 2023-05-19T23:05:48Z

LGTM. And please make sure it is validated in a JIT based environment -- so that there is no permission issues.

Can you expand on what JIT means? #Resolved

keboliu · 2023-05-26T02:47:12Z

Hi @qiluo-msft would you please merge this since it's approved?

keboliu · 2023-05-29T03:11:57Z

@StormLiangMS would you please help to cherry-pick?

mssonicbld · 2023-05-29T11:30:39Z

Cherry-pick PR to 202211: #15249

- Why I did it interfaces-config service restarts networking service, which in-turn results in loopback interface address is being removed and reassigned back If the system-health happens to start during that instance expections and logs like this are seen: Apr 15 18:14:49.357869 r-panther-20 ERR healthd: update system status exception:Unable to connect to redis: Cannot assign requested address Apr 15 18:14:49.429778 r-panther-20 ERR healthd: subscribe_statedb exited- Unable to connect to redis: Cannot assign requested address Apr 15 18:14:52.218594 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:52.219714 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:55.218636 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:55.218722 r-panther-20 ERR healthd: system_service_Map_base::at - How I did it use unix socket path Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2023-06-07T16:03:21Z

Issue was also seen on 202205 as reported here #15364 and thus request to backport to 202205

bingwang-ms · 2023-06-07T16:15:34Z

MSFT ADO link https://msazure.visualstudio.com/One/_workitems/edit/24204612/

- Why I did it interfaces-config service restarts networking service, which in-turn results in loopback interface address is being removed and reassigned back If the system-health happens to start during that instance expections and logs like this are seen: Apr 15 18:14:49.357869 r-panther-20 ERR healthd: update system status exception:Unable to connect to redis: Cannot assign requested address Apr 15 18:14:49.429778 r-panther-20 ERR healthd: subscribe_statedb exited- Unable to connect to redis: Cannot assign requested address Apr 15 18:14:52.218594 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:52.219714 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:55.218636 r-panther-20 ERR healthd: system_service_Map_base::at Apr 15 18:14:55.218722 r-panther-20 ERR healthd: system_service_Map_base::at - How I did it use unix socket path Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

mssonicbld · 2023-06-15T16:17:15Z

Cherry-pick PR to 202205: #15480

vivekrnv and others added 2 commits April 21, 2023 16:45

[health] Start system-health after interfaces-config

341108e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'sonic-net:master' into intf_cfg_dep_healthd

e5b0456

vivekrnv requested a review from lguohan as a code owner April 25, 2023 13:37

keboliu added the Request for 202211 Branch label Apr 26, 2023

liat-grozovik requested a review from qiluo-msft May 2, 2023 07:31

dgsudharsan previously approved these changes May 9, 2023

View reviewed changes

vivekrnv added 3 commits May 10, 2023 02:32

Revert "[health] Start system-health after interfaces-config"

4a8a3e4

This reverts commit 341108e.

Use unix_socket_path in system-health

a7f6bf9

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'intf_cfg_dep_healthd' of github.com:vivekrnv/sonic-buil…

551544b

…dimage into intf_cfg_dep_healthd

vivekrnv dismissed dgsudharsan’s stale review via 551544b May 10, 2023 03:04

dgsudharsan approved these changes May 10, 2023

View reviewed changes

vivekrnv changed the title ~~[healthd] Start system-health after interfaces-config~~ [healthd] Use unix_socket_path instead of loopback ip in system-health May 10, 2023

vivekrnv changed the title ~~[healthd] Use unix_socket_path instead of loopback ip in system-health~~ [healthd] Use unix_socket_path instead of loopback ip May 10, 2023

Junchao-Mellanox approved these changes May 10, 2023

View reviewed changes

judyjoseph approved these changes May 19, 2023

View reviewed changes

liat-grozovik merged commit bc9c054 into sonic-net:master May 26, 2023

StormLiangMS added the Approved for 202211 Branch label May 29, 2023

mssonicbld added the Created PR to 202211 Branch label May 29, 2023

mssonicbld mentioned this pull request May 29, 2023

[action] [PR:14843] [healthd] Use unix_socket_path instead of loopback ip #15249

Merged

10 tasks

mssonicbld added a commit that referenced this pull request May 29, 2023

[healthd] Use unix_socket_path instead of loopback ip (#14843) (#15249)

24daa8a

mssonicbld added Included in 202211 Branch and removed Approved for 202211 Branch Created PR to 202211 Branch labels May 29, 2023

vivekrnv mentioned this pull request Jun 7, 2023

healthd flooding ERR system_service_Map_base::at #15364

Closed

vivekrnv added the Request for 202205 Branch label Jun 7, 2023

yxieca added the Approved for 202205 Branch label Jun 15, 2023

mssonicbld added the Created PR to 202205 Branch label Jun 15, 2023

mssonicbld mentioned this pull request Jun 15, 2023

[action] [PR:14843] [healthd] Use unix_socket_path instead of loopback ip #15480

Merged

10 tasks

mssonicbld added a commit that referenced this pull request Jun 15, 2023

[healthd] Use unix_socket_path instead of loopback ip (#14843) (#15480)

7ce21a6

mssonicbld added Included in 202205 Branch and removed Approved for 202205 Branch Created PR to 202205 Branch labels Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[healthd] Use unix_socket_path instead of loopback ip #14843

[healthd] Use unix_socket_path instead of loopback ip #14843

vivekrnv commented Apr 25, 2023 •

edited

Loading

vivekrnv commented Apr 25, 2023

mssonicbld commented Apr 25, 2023

azure-pipelines bot commented Apr 25, 2023

vivekrnv commented Apr 28, 2023

mssonicbld commented Apr 28, 2023

azure-pipelines bot commented Apr 28, 2023

liat-grozovik commented May 1, 2023

moshemos commented May 3, 2023

liat-grozovik commented May 8, 2023

Junchao-Mellanox commented May 9, 2023 •

edited

Loading

vivekrnv commented May 9, 2023 •

edited

Loading

Junchao-Mellanox commented May 9, 2023

vivekrnv commented May 9, 2023 •

edited

Loading

Junchao-Mellanox commented May 9, 2023

vivekrnv commented May 9, 2023

vivekrnv commented May 10, 2023

vivekrnv commented May 16, 2023

vivekrnv commented May 18, 2023

judyjoseph left a comment

vivekrnv commented May 19, 2023 •

edited

Loading

keboliu commented May 26, 2023

keboliu commented May 29, 2023

mssonicbld commented May 29, 2023

vivekrnv commented Jun 7, 2023

bingwang-ms commented Jun 7, 2023

mssonicbld commented Jun 15, 2023

[healthd] Use unix_socket_path instead of loopback ip #14843

[healthd] Use unix_socket_path instead of loopback ip #14843

Conversation

vivekrnv commented Apr 25, 2023 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

vivekrnv commented Apr 25, 2023

mssonicbld commented Apr 25, 2023

azure-pipelines bot commented Apr 25, 2023

vivekrnv commented Apr 28, 2023

mssonicbld commented Apr 28, 2023

azure-pipelines bot commented Apr 28, 2023

liat-grozovik commented May 1, 2023

moshemos commented May 3, 2023

liat-grozovik commented May 8, 2023

Junchao-Mellanox commented May 9, 2023 • edited Loading

vivekrnv commented May 9, 2023 • edited Loading

Junchao-Mellanox commented May 9, 2023

vivekrnv commented May 9, 2023 • edited Loading

Junchao-Mellanox commented May 9, 2023

vivekrnv commented May 9, 2023

vivekrnv commented May 10, 2023

vivekrnv commented May 16, 2023

vivekrnv commented May 18, 2023

judyjoseph left a comment

Choose a reason for hiding this comment

vivekrnv commented May 19, 2023 • edited Loading

keboliu commented May 26, 2023

keboliu commented May 29, 2023

mssonicbld commented May 29, 2023

vivekrnv commented Jun 7, 2023

bingwang-ms commented Jun 7, 2023

mssonicbld commented Jun 15, 2023

vivekrnv commented Apr 25, 2023 •

edited

Loading

Junchao-Mellanox commented May 9, 2023 •

edited

Loading

vivekrnv commented May 9, 2023 •

edited

Loading

vivekrnv commented May 9, 2023 •

edited

Loading

vivekrnv commented May 19, 2023 •

edited

Loading