system-health service_checker should check containers based on asic presence #13497

spilkey-cisco · 2023-01-24T22:12:15Z

Why I did it

On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created.

system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file).

system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity.

How I did it

Port container_checker logic from #11442 into service_checker for system-health.

How to verify it

Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

linux-foundation-easycla · 2023-01-24T22:12:18Z

The committers listed above are authorized under a signed CLA.

✅ login: spilkey-cisco (3342a31)

bmridul · 2023-02-01T21:55:53Z

@abdosi Pls assign folks for review.

shivuv · 2023-02-08T21:24:43Z

@abdosi Pls assign folks for review.

@abdosi @gechiang : Can you have someone review this?

abdosi · 2023-02-08T21:38:02Z

@prgeor can you please review/approve this.

prgeor · 2023-02-09T02:13:06Z

@spilkey-cisco have you tested the scenario where if the fabric card once inserted, can still have hardware issues causing ASIC from detecting? or this is not a valid scenario?

spilkey-cisco · 2023-02-09T02:17:54Z

@spilkey-cisco have you tested the scenario where if the fabric card once inserted, can still have hardware issues causing ASIC from detecting? or this is not a valid scenario?

That scenario should be covered by the checks in pcied, validating that expected devices exist.

shivuv · 2023-02-10T21:58:44Z

@abdosi @gechiang @prgeor : If you are ok with this change, please approve and merge it. Also requesting to cherry pick this into 202205 branch.

gechiang · 2023-02-10T23:05:07Z

@lguohan None of us has permission to merge. Please help review and if approved, merge the PR.
Thanks!

…onic-net#13497) Why I did it On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created. system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file). system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity. How I did it Port container_checker logic from sonic-net#11442 into service_checker for system-health. How to verify it Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

mssonicbld · 2023-02-17T13:53:02Z

Cherry-pick PR to 202211: #13857

gechiang · 2023-02-17T19:57:14Z

@yxieca can you help cherry-pick this to 202205 branch?
Thanks!

…13497) Why I did it On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created. system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file). system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity. How I did it Port container_checker logic from #11442 into service_checker for system-health. How to verify it Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

mssonicbld · 2023-02-21T16:37:51Z

@spilkey-cisco PR conflicts with 202205 branch

gechiang · 2023-02-21T17:27:33Z

@spilkey-cisco please submit a separate PR for this in 202205 branch as it could not be cherry-picked cleanly...

…onic-net#13497) Why I did it On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created. system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file). system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity. How I did it Port container_checker logic from sonic-net#11442 into service_checker for system-health. How to verify it Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

spilkey-cisco · 2023-02-23T20:13:50Z

@spilkey-cisco please submit a separate PR for this in 202205 branch as it could not be cherry-picked cleanly...

Created #13966

…13497) (#13966) Why I did it On a supervisor card in a chassis, syncd/teamd/swss/lldp etc dockers are created for each Switch Fabric card. However, not all chassis would have all the switch fabric cards present. In this case, only dockers for Switch Fabrics present would be created. system-health indicates errors in this scenario as it is expecting dockers for all Switch Fabrics (based on NUM_ASIC defined in asic.conf file). system-health process error messages were also altered to indicate which container had the issue; multiple containers may run processes with the same name, which can result in identical system-health error messages, causing ambiguity. How I did it Port container_checker logic from #11442 into service_checker for system-health. How to verify it Bringup Supervisor card with one or more missing fabric cards. Execute 'show system-health summary'. The command should not report failure due to missing dockers for the asics on the fabric cards which are not present.

…rmat (#7649) What is the motivation for this PR? test_service_checker_with_process_exit failed due to this change: sonic-net/sonic-buildimage#13497 How did you do it? Modify format of verfiying log. How did you verify/test it? Run test Signed-off-by: Yaqiang Zhu <yaqiangzhu@microsoft.com>

StormLiangMS · 2023-03-08T01:37:22Z

@spilkey-cisco this PR break the test case test_service_checker_with_process_exit, because the log format change, pls run sonic-mgmt test suite if any print log changes, that could break test cases.

+@rlhui for vis

spilkey-cisco · 2023-03-08T02:03:12Z

@spilkey-cisco this PR break the test case test_service_checker_with_process_exit, because the log format change, pls run sonic-mgmt test suite if any print log changes, that could break test cases.

+@rlhui for vis

I was not aware of this requirement or impact, I will keep it in mind in the future. Perhaps this PR should not have been approved or merged without requesting these test results. Are there any contributor guidelines you can point me to? I'd like to be sure I meet all these requirements before raising PRs or asking for approval in the future.

gechiang · 2023-03-08T03:04:47Z

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

spilkey-cisco · 2023-03-08T03:09:08Z

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

The log format change was done because the same process name can exist in multiple containers, especially for multi-asic. It certainly can be reverted, but the system-health errors become ambiguous. Please let me know if it needs to be reverted and if you need me to open a PR for that, or if you will handle the change.

StormLiangMS · 2023-03-08T12:47:35Z

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

The log format change was done because the same process name can exist in multiple containers, especially for multi-asic. It certainly can be reverted, but the system-health errors become ambiguous. Please let me know if it needs to be reverted and if you need me to open a PR for that, or if you will handle the change.

@spilkey-cisco I think the container name is already in, why we need the later one? It could cause the difference among different releases, pls revert this back.
self.set_object_not_ok('Process', '{}:{}'.format(container_name, process_name), "Process '{}' in container '{}' is not running".format(process_name, container_name))

StormLiangMS · 2023-03-08T12:50:02Z

@yaqiangz when this one is reverted, I think we should change the testcases back too?

@StormLiangMS Thanks for catching this. Can you confirm if this is also causing "syslog" format change as well? If so, we will need to revert the portion of code that made the log format change.

I thought it changes the syslog, but from what Yaqiang found, it is a DB info change. @gechiang

… new format (#7681) What is the motivation for this PR? Test failed introduced by this PR sonic-net/sonic-buildimage#13497 has been fixed by this PR #7649. But in DUT with old image, it will failed. How did you do it? Add support for check_system_health_info to support both two formats How did you verify/test it? Run test Signed-off-by: Yaqiang Zhu <yaqiangzhu@microsoft.com>

spilkey-cisco · 2023-03-08T17:40:11Z

@spilkey-cisco I think the container name is already in, why we need the later one? It could cause the difference among different releases, pls revert this back. self.set_object_not_ok('Process', '{}:{}'.format(container_name, process_name), "Process '{}' in container '{}' is not running".format(process_name, container_name))

The container name was not displayed in the system-health CLI, only the final string argument '<process>' is not running. On multi-asic systems especially where duplicates of containers exist, the CLI (ie. show system-health summary) does not display the container name:

	     'syncd' is not running
	     'teammgrd' is not running
	     'syncd' is not running
	     'syncd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'teammgrd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'syncd' is not running
	     'vxlanmgrd' is not running
	     'vlanmgrd' is not running
	     'orchagent' is not running
	     'teammgrd' is not running
	     'teammgrd' is not running

With the log change, the CLI tells us which container they belong to:

	     Process 'syncd' in container 'syncd0' is not running
	     Process 'syncd' in container 'syncd12' is not running
	     Process 'teammgrd' in container 'teamd0' is not running
	     Process 'teammgrd' in container 'teamd1' is not running
	     Process 'syncd' in container 'syncd1' is not running
	     Process 'syncd' in container 'syncd13' is not running
	     Process 'teamsyncd' in container 'teamd12' is not running
	     Process 'teammgrd' in container 'teamd12' is not running
	     Process 'teammgrd' in container 'teamd13' is not running
	     Process 'vxlanmgrd' in container 'swss12' is not running
	     Process 'vlanmgrd' in container 'swss12' is not running
	     Process 'vxlanmgrd' in container 'swss1' is not running
	     Process 'vlanmgrd' in container 'swss1' is not running
	     Process 'vxlanmgrd' in container 'swss0' is not running
	     Process 'vlanmgrd' in container 'swss0' is not running
	     Process 'vxlanmgrd' in container 'swss13' is not running
	     Process 'vlanmgrd' in container 'swss13' is not running

… new format (#7681) What is the motivation for this PR? Test failed introduced by this PR sonic-net/sonic-buildimage#13497 has been fixed by this PR #7649. But in DUT with old image, it will failed. How did you do it? Add support for check_system_health_info to support both two formats How did you verify/test it? Run test Signed-off-by: Yaqiang Zhu <yaqiangzhu@microsoft.com>

Related work items: sonic-net#276, sonic-net#305, sonic-net#332, sonic-net#338, sonic-net#339, sonic-net#1188, sonic-net#1192, sonic-net#1197, sonic-net#1206, sonic-net#1685, sonic-net#1690, sonic-net#1696, sonic-net#1699, sonic-net#1709, sonic-net#1727, sonic-net#1737, sonic-net#1741, sonic-net#1742, sonic-net#2511, sonic-net#2512, sonic-net#2532, sonic-net#2559, sonic-net#2626, sonic-net#2638, sonic-net#2645, sonic-net#2649, sonic-net#2660, sonic-net#2669, sonic-net#2670, sonic-net#2678, sonic-net#10084, sonic-net#11442, sonic-net#11873, sonic-net#12047, sonic-net#12110, sonic-net#12207, sonic-net#12529, sonic-net#12678, sonic-net#13235, sonic-net#13287, sonic-net#13372, sonic-net#13395, sonic-net#13456, sonic-net#13497, sonic-net#13522, sonic-net#13545, sonic-net#13547, sonic-net#13552, sonic-net#13569, sonic-net#13572, sonic-net#13578, sonic-net#13591, sonic-net#13611, sonic-net#13647, sonic-net#13649, sonic-net#13660, sonic-net#13710, sonic-net#13716, sonic-net#13724, sonic-net#13726, sonic-net#13732, sonic-net#13735, sonic-net#13739, sonic-net#13757, sonic-net#13786, sonic-net#13792, sonic-net#13800, sonic-net#13801, sonic-net#13802, sonic-net#13805, sonic-net#13806, sonic-net#13812, sonic-net#13814, sonic-net#13822, sonic-net#13831, sonic-net#13834, sonic-net#13847, sonic-net#13870, sonic-net#13882, sonic-net#13884, sonic-net#13885, sonic-net#13894, sonic-net#13895, sonic-net#13926, sonic-net#13932, sonic-net#13935, sonic-net#13942, sonic-net#13951, sonic-net#13953, sonic-net#13964

Add asic presence filtering for container checking in system-health

3342a31

spilkey-cisco requested a review from lguohan as a code owner January 24, 2023 22:12

abdosi approved these changes Feb 8, 2023

View reviewed changes

abdosi requested a review from prgeor February 8, 2023 21:36

abdosi added the Request for 202205 Branch label Feb 8, 2023

gechiang approved these changes Feb 8, 2023

View reviewed changes

prgeor added the Request for 202211 Branch label Feb 9, 2023

rlhui merged commit ad679a0 into sonic-net:master Feb 11, 2023

StormLiangMS added the Approved for 202211 Branch label Feb 17, 2023

mssonicbld added the Created PR to 202211 Branch label Feb 17, 2023

mssonicbld mentioned this pull request Feb 17, 2023

[action] [PR:13497] system-health service_checker should check containers based on asic presence #13857

Merged

mssonicbld added Included in 202211 Branch and removed Approved for 202211 Branch Created PR to 202211 Branch labels Feb 17, 2023

yxieca added the Approved for 202205 Branch label Feb 21, 2023

mssonicbld added the Cherry Pick Conflict_202205 label Feb 21, 2023

spilkey-cisco mentioned this pull request Feb 23, 2023

system-health service_checker should check containers based on asic presence (cherry-pick #13497) #13966

Merged

yxieca added the Included in 202205 Branch label Feb 28, 2023

yaqiangz mentioned this pull request Mar 6, 2023

[system_health] Fix test_system_health failed due to change of log format sonic-net/sonic-mgmt#7649

Merged

6 tasks

yaqiangz mentioned this pull request Mar 8, 2023

[system_health] Add support for check_system_health_info both old and new format sonic-net/sonic-mgmt#7681

Merged

6 tasks

kartik-arista mentioned this pull request Mar 23, 2023

system_health/test_system_health.py::test_service_checker_with_process_exit fails with "AssertionError: ... is not recorded" sonic-net/sonic-mgmt#7832

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

system-health service_checker should check containers based on asic presence #13497

system-health service_checker should check containers based on asic presence #13497

spilkey-cisco commented Jan 24, 2023

linux-foundation-easycla bot commented Jan 24, 2023 •

edited

Loading

bmridul commented Feb 1, 2023

shivuv commented Feb 8, 2023

abdosi commented Feb 8, 2023

prgeor commented Feb 9, 2023

spilkey-cisco commented Feb 9, 2023

shivuv commented Feb 10, 2023

gechiang commented Feb 10, 2023

mssonicbld commented Feb 17, 2023

gechiang commented Feb 17, 2023

mssonicbld commented Feb 21, 2023

gechiang commented Feb 21, 2023

spilkey-cisco commented Feb 23, 2023

StormLiangMS commented Mar 8, 2023

spilkey-cisco commented Mar 8, 2023

gechiang commented Mar 8, 2023

spilkey-cisco commented Mar 8, 2023

StormLiangMS commented Mar 8, 2023

StormLiangMS commented Mar 8, 2023 •

edited

Loading

spilkey-cisco commented Mar 8, 2023 •

edited

Loading

system-health service_checker should check containers based on asic presence #13497

system-health service_checker should check containers based on asic presence #13497

Conversation

spilkey-cisco commented Jan 24, 2023

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

linux-foundation-easycla bot commented Jan 24, 2023 • edited Loading

bmridul commented Feb 1, 2023

shivuv commented Feb 8, 2023

abdosi commented Feb 8, 2023

prgeor commented Feb 9, 2023

spilkey-cisco commented Feb 9, 2023

shivuv commented Feb 10, 2023

gechiang commented Feb 10, 2023

mssonicbld commented Feb 17, 2023

gechiang commented Feb 17, 2023

mssonicbld commented Feb 21, 2023

gechiang commented Feb 21, 2023

spilkey-cisco commented Feb 23, 2023

StormLiangMS commented Mar 8, 2023

spilkey-cisco commented Mar 8, 2023

gechiang commented Mar 8, 2023

spilkey-cisco commented Mar 8, 2023

StormLiangMS commented Mar 8, 2023

StormLiangMS commented Mar 8, 2023 • edited Loading

spilkey-cisco commented Mar 8, 2023 • edited Loading

linux-foundation-easycla bot commented Jan 24, 2023 •

edited

Loading

StormLiangMS commented Mar 8, 2023 •

edited

Loading

spilkey-cisco commented Mar 8, 2023 •

edited

Loading