Fixed determine/process reboot-cause service dependency #17406

anamehra · 2023-12-04T23:59:35Z

Signed-off-by: anamehra anamehra@cisco.com

Why I did it

Fixes #16990 for 202305/202205 branch

Note: This PR is for 202305 and 202205. For master, a new PR will be raised with a new field (Uphold=) provided by debian bookworm to handle the dependency failure restartability of the processes.

determine-reboot-cause and process-reboot-cause service does not start If the database service fails to restart in the first attempt. Even if the Database service succeeds in the next attempt, these reboot-cause services do not start.
The process-reboot-cause service also does not restart if the docker or database service restarts, which leads to an empty reboot-cause history
deploy-mg from sonic-mgmt also triggers the docker service restart. The restart of the docker service caused the issue stated in 2 above. The docker restart also triggers determine-reboot-cause to restart which creates an additional reboot-cause file in history and modifies the last reboot-cause.

This PR fixes these issues by making both processes start again when dependency meets after dependency failure, making both processes restart when the database service restarts, and preventing duplicate processing of the last reboot reason.

Work item tracking

Microsoft ADO 25892856

How I did it

Modified systemd unit files to make determine-reboot-cause and process-reboot-cause services restartable when the database service restarts.
On the restart, the determine-reboot-cause service should not recreate a new reboot-cause entry in the database. Added check for first start or restart to skip entry for restart case.

How to verify it

On single asic pizza box:

Installed the image and check reboot-cause history
restart database service and verify that determine-reboot-cause and process-reboot-cause services also restart. Verify that reboot-cause shows correct data and no new entry is created for restart.

On Chassis:

Installed the image and check reboot-cause history
restart the database service and verify that determine-reboot-cause and process-reboot-cause services also restart. Verify that reboot-cause shows correct data and no new entry is created for restart.
Reboot LC. On Supervicor, stop database-chassis service.
Let database service on LC fail the first time. determine-reboot-cause and process-reboot-cause would fail to start due to dependency failure
start database-chassis on Supervisor. Database service on LC should now start successfully.
Verify determine-reboot-cause and process-reboot-cause also starts
Verify show reboot-cause history output

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: anamehra <anamehra@cisco.com>

anamehra · 2023-12-05T00:00:37Z

Hi @prgeor , @gechiang , for your review. Thanks

prgeor · 2023-12-05T21:01:51Z

@StormLiangMS can you merge this for 202305

prgeor · 2023-12-05T21:02:59Z

@yxieca please help cherry pick to 202205

yxieca · 2023-12-06T00:04:46Z

@anamehra please help raise a ticket for 202205 branch directly. I don't think the automation will cherry-pick PR from feature branch to feature branch.

anamehra · 2023-12-08T17:32:27Z

@anamehra please help raise a ticket for 202205 branch directly. I don't think the automation will cherry-pick PR from feature branch to feature branch.

Please review following for 202205
#17462

gechiang · 2023-12-10T02:39:58Z

@StormLiangMS Please help review/merge this to 202305.
This is a bug fix for all platforms.

anamehra · 2024-06-19T18:00:42Z

@abdosi , @rlhui , may we have this in 202405? Thanks

gechiang · 2024-06-19T22:54:55Z

@anamehra , looks like automation is not able to cleanly pick this PR into 202405 branch. can you please submit a separate PR directly under the 202405 branch and link this PR to it.

anamehra · 2024-06-20T05:50:39Z

@anamehra , looks like automation is not able to cleanly pick this PR into 202405 branch. can you please submit a separate PR directly under the 202405 branch and link this PR to it.

Hi @gechiang, done!

gechiang · 2024-06-21T22:26:59Z

Hi @anamehra , I just realized that this PR you raised (#17406)
was not on the "master" branch but of the 202305 branch.
Ideally you should raise this PR on the master branch and then request backport to other older branches.
Can I ask you to convert your 202404 specific PR (sonic-net/sonic-host-services#132) to the master branch instead?
Then on the Master branch based PR, we can ask for the backport to 202405 again and I think it should be able to cleanly picked since master and 202405 are pretty close at this moment.
If we don't fix this, you will end up keep catching up whenever a new release is created...
Thanks!

anamehra · 2024-06-21T23:41:11Z

Hi @anamehra , I just realized that this PR you raised (#17406) was not on the "master" branch but of the 202305 branch. Ideally you should raise this PR on the master branch and then request backport to other older branches. Can I ask you to convert your 202404 specific PR (sonic-net/sonic-host-services#132) to the master branch instead? Then on the Master branch based PR, we can ask for the backport to 202405 again and I think it should be able to cleanly picked since master and 202405 are pretty close at this moment. If we don't fix this, you will end up keep catching up whenever a new release is created... Thanks!

Hi @gechiang , I raised this for 202405 only due to a discussion in the community forum. There is PR with 'Uphold' fix and I am waiting on responses to my queries on that before I raise a PR on master. My original PR in master was initially rejected to use uphold.
@abdosi , @prgeor , if. you agree, I can raise this PR #17406 on master. Thanks

…utomatically (#19415) #### Why I did it src/sonic-host-services ``` * 02d9b55 - (HEAD -> master, origin/master, origin/HEAD) Added support to render template format of `delayed` flag on Feature Table. (#135) (28 hours ago) [abdosi] * 60fdfea - Fixed determine/process reboot-cause service dependency (#17406) (#132) (13 days ago) [anamehra] ``` #### How I did it #### How to verify it #### Description for the changelog

…utomatically (#19551) #### Why I did it src/sonic-host-services ``` * aea0bef - (HEAD -> 202405, origin/202405) Ignore sonic_platform package fileNotFoundError on non-chassis vs platforms (#133) (#140) (4 minutes ago) [mssonicbld] * 0e7e4d5 - Added support to render template format of `delayed` flag on Feature Table. (#135) (#137) (11 days ago) [mssonicbld] * 235c2a4 - Fixed determine/process reboot-cause service dependency (#17406) (#132) (13 days ago) [anamehra] ``` #### How I did it #### How to verify it #### Description for the changelog

…utomatically (sonic-net#19415) #### Why I did it src/sonic-host-services ``` * 02d9b55 - (HEAD -> master, origin/master, origin/HEAD) Added support to render template format of `delayed` flag on Feature Table. (sonic-net#135) (28 hours ago) [abdosi] * 60fdfea - Fixed determine/process reboot-cause service dependency (sonic-net#17406) (sonic-net#132) (13 days ago) [anamehra] ``` #### How I did it #### How to verify it #### Description for the changelog

…utomatically (sonic-net#19415) #### Why I did it src/sonic-host-services ``` * 02d9b55 - (HEAD -> master, origin/master, origin/HEAD) Added support to render template format of `delayed` flag on Feature Table. (#135) (28 hours ago) [abdosi] * 60fdfea - Fixed determine/process reboot-cause service dependency (sonic-net#17406) (#132) (13 days ago) [anamehra] ``` #### How I did it #### How to verify it #### Description for the changelog

Fixed determine/process reboot-cause service dependency

520bb0d

Signed-off-by: anamehra <anamehra@cisco.com>

anamehra requested a review from lguohan as a code owner December 4, 2023 23:59

prgeor approved these changes Dec 5, 2023

View reviewed changes

prgeor added the Request for 202205 Branch label Dec 5, 2023

anamehra mentioned this pull request Dec 5, 2023

Fixed determine/process reboot-cause service dependency sonic-net/sonic-host-services#88

Closed

yxieca added Request for 202205 Branch and removed Request for 202205 Branch labels Dec 5, 2023

gechiang approved these changes Dec 10, 2023

View reviewed changes

gechiang mentioned this pull request Dec 10, 2023

Fixed determine/process reboot-cause service dependency #17462

Merged

11 tasks

StormLiangMS merged commit 4595db4 into sonic-net:202305 Dec 17, 2023
18 checks passed

abdosi mentioned this pull request Jun 11, 2024

Add Process Reboot Cause Service as Upholds of Database Service #18772

Open

13 tasks

wenyiz2021 added the Request for 202405 Branch label Jun 19, 2024

mssonicbld added the Cherry Pick Conflict_202405 label Jun 19, 2024

anamehra mentioned this pull request Jun 20, 2024

Fixed determine/process reboot-cause service dependency (#17406) sonic-net/sonic-host-services#132

Merged

bingwang-ms removed Cherry Pick Conflict_202405 Request for 202405 Branch labels Aug 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed determine/process reboot-cause service dependency #17406

Fixed determine/process reboot-cause service dependency #17406

anamehra commented Dec 4, 2023 •

edited

Loading

anamehra commented Dec 5, 2023

prgeor commented Dec 5, 2023

prgeor commented Dec 5, 2023

yxieca commented Dec 6, 2023

anamehra commented Dec 8, 2023

gechiang commented Dec 10, 2023

anamehra commented Jun 19, 2024

gechiang commented Jun 19, 2024

anamehra commented Jun 20, 2024

gechiang commented Jun 21, 2024

anamehra commented Jun 21, 2024

Fixed determine/process reboot-cause service dependency #17406

Fixed determine/process reboot-cause service dependency #17406

Conversation

anamehra commented Dec 4, 2023 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

anamehra commented Dec 5, 2023

prgeor commented Dec 5, 2023

prgeor commented Dec 5, 2023

yxieca commented Dec 6, 2023

anamehra commented Dec 8, 2023

gechiang commented Dec 10, 2023

anamehra commented Jun 19, 2024

gechiang commented Jun 19, 2024

anamehra commented Jun 20, 2024

gechiang commented Jun 21, 2024

anamehra commented Jun 21, 2024

anamehra commented Dec 4, 2023 •

edited

Loading