[Services] Restart Platform-monitor service upon unexpected critical process exit. #3689

yozhao101 · 2019-10-30T23:04:14Z

What I did
Restart Platform-monitor service if one of critical processes running in Platform-monitor container exited or crashed abnormally.
How I did it
Generally I follow the framework created by Joe to implement this feature in Platform-monitor container.
First, add supervisor-proc-exit-listener event listener option in Supervisord configuration file in Platform-monitor docker container. Supervisord will read a list of critical processes for which to monitor the unexpected crashed and exited.
Second, configure Platform-monitor.service to always auto-restart the service if it stops, with a delay of 30 seconds. Also set a rate limit of 3 restarts within 20 minutes (1200 seconds).
How to verify it
On your switch device, please use docker ps command to list all running docker containers.
Then use docker exec -it container_id bash to login target container. Typing top command
on the shell will display all the processes dynamically and you will spot the process id of one
of the critical processes. Finally type the command kill -9 process_id to terminate one process.
After exiting the container, you can use watch -n 1 docker ps to dynamically see the restart
of Platform-monitor container.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

in supervisord conf file. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

critical processes file into dockerfile.j2. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

platform-monitor if it attempts to restart this container more than 3 times in 20 minutes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

…CRIPT) into shared Makefile docker-platform-monitor.mk. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

…s exit. (sonic-net#3667) Signed-off-by: Yong Zhao <yozhao@microsoft.com> [Services] Restart Platform-monitor service upon unexpected critical process exit. (sonic-net#3689) Signed-off-by: Yong Zhao <yozhao@microsoft.com> Signed-off-by: Sangita Maity <sangitamaity0211@gmail.com> RB=2126600 G=lnos-reviewers R=pchaudha,pmao,vapatil,zxu A=zxu

…atically (#21194) #### Why I did it src/sonic-utilities ``` * 2f8508f1 - (HEAD -> master, origin/master, origin/HEAD) Revert PR: Add scope to field validator (#3689) (3 days ago) [xwjiang-ms] * 5481d0ef - Optimize techsupport reducing number of vtysh calls in scale sceario (#3605) (5 days ago) [Sudharsan Dhamal Gopalarathnam] * 7e2e59c5 - add sonic_asic_platform in the render context when rendering docker_image_ctl.j2 (#3672) (5 days ago) [Changrong Wu] * d85e1db3 - Disable routeCheck monit as part of config reload/minigraph stop service and enable it back as part of service start. (#3682) (6 days ago) [abdosi] * 139983ad - platform/marvell renaming to platform/marvell-prestera (#3622) (6 days ago) [krismarvell] * e3991d54 - Add scope to field validator. (#3675) (7 days ago) [Xincun Li] ``` #### How I did it #### How to verify it #### Description for the changelog

…atically (sonic-net#21194) #### Why I did it src/sonic-utilities ``` * 2f8508f1 - (HEAD -> master, origin/master, origin/HEAD) Revert PR: Add scope to field validator (sonic-net#3689) (3 days ago) [xwjiang-ms] * 5481d0ef - Optimize techsupport reducing number of vtysh calls in scale sceario (sonic-net#3605) (5 days ago) [Sudharsan Dhamal Gopalarathnam] * 7e2e59c5 - add sonic_asic_platform in the render context when rendering docker_image_ctl.j2 (sonic-net#3672) (5 days ago) [Changrong Wu] * d85e1db3 - Disable routeCheck monit as part of config reload/minigraph stop service and enable it back as part of service start. (sonic-net#3682) (6 days ago) [abdosi] * 139983ad - platform/marvell renaming to platform/marvell-prestera (sonic-net#3622) (6 days ago) [krismarvell] * e3991d54 - Add scope to field validator. (sonic-net#3675) (7 days ago) [Xincun Li] ``` #### How I did it #### How to verify it #### Description for the changelog

yozhao101 added 5 commits October 30, 2019 11:46

[docker-platform-monitor] Create a file named critical_processes.

cb2fdbc

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[docker-platform-monitor] Make event listener autostart by adding option

82d73b2

in supervisord conf file. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[docker-platform-monitor] Add paths of supervisord listener script and

aaac842

critical processes file into dockerfile.j2. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[docker-platform-monitor] Configure systemd to stop restarting

b2c04a1

platform-monitor if it attempts to restart this container more than 3 times in 20 minutes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[docker-platform-monitor] Add macro $(SUPERVISOR_PROC_EXIT_LISTENER_S…

2073c9f

…CRIPT) into shared Makefile docker-platform-monitor.mk. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 marked this pull request as ready for review November 1, 2019 23:14

yozhao101 requested a review from jleveque November 1, 2019 23:15

jleveque added the Enhancement ➕ label Nov 5, 2019

jleveque approved these changes Nov 5, 2019

View reviewed changes

jleveque merged commit 4fa3a1e into sonic-net:master Nov 5, 2019

yozhao101 mentioned this pull request Jun 25, 2020

[doc] Monitoring and Auto-mitigating the unhealthy of docker containers in SONiC sonic-net/SONiC#564

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Services] Restart Platform-monitor service upon unexpected critical process exit. #3689

[Services] Restart Platform-monitor service upon unexpected critical process exit. #3689

yozhao101 commented Oct 30, 2019

[Services] Restart Platform-monitor service upon unexpected critical process exit. #3689

[Services] Restart Platform-monitor service upon unexpected critical process exit. #3689

Conversation

yozhao101 commented Oct 30, 2019