Check platform reboot cause to see if any reset happened during fast/warm-reboot #7920

sujinmkang · 2021-06-19T08:40:36Z

Why I did it

Check platform reboot cause to see if any reset happened during fast/warm-reboot
While starting dockers with boot_type, it's only checking the /proc/cmdline.
If any unexpected Platform reset happens during fast/warm-reboot, it should be detected as platform reboot
and "FAST_REBOOT|system" should be set in that case.

How I did it

Add the platform reboot cause parsing when determining the boot_type

How to verify it

Check the logic with reboot
Convert the device with fast-reload and see if the boot_type is detected correctly

Which release branch to backport (provide reason below if selected)

201811
201911
202006
202012

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

fast/warm-reboot

jleveque · 2021-06-22T01:01:24Z

/azp run

azure-pipelines · 2021-06-22T01:01:40Z

Azure Pipelines successfully started running 1 pipeline(s).

jleveque · 2021-06-24T00:16:23Z

/azp run

jleveque · 2021-06-24T00:24:48Z

files/build_templates/docker_image_ctl.j2

+function isPlatformReset()
+{
+    output=$(python3 -c "import sonic_platform.platform; p = sonic_platform.platform.Platform(); c = p.get_chassis(); hw_rc_major, hw_rc_minor = c.get_reboot_cause(); print(hw_rc_major)" 2>/dev/null)
+    echo "${output}"


Maybe this should echo more context, rather than just the reboot cause?

@sujinmkang while testing in 201811 image, This line(66) causing changes in getBootType() function.
(ie) BOOT_TYPE = "reboot cause" value instead of "cold", "warm", "fastfast", etc

Removing this line (echo "${output}") resolved the issue.

I commented about it yesterday but it said pending status.

jleveque · 2021-06-24T00:25:03Z

files/build_templates/docker_image_ctl.j2

+    if [[ "${output}" != "Non-Hardware" ]]; then
+        return 0
+    else
+        return 1


Maybe add a comment here to explain this logic?

azure-pipelines · 2021-06-24T00:31:31Z

Azure Pipelines successfully started running 1 pipeline(s).

sujinmkang · 2021-08-04T21:49:34Z

/azp run

azure-pipelines · 2021-08-04T21:49:51Z

Azure Pipelines successfully started running 1 pipeline(s).

santhosh-kt · 2021-08-30T15:31:58Z

Uploaded UT test logs in #8024 for your reference.
(OS9 to SONIC - cold reset.txt)
(OS9 to SONIC - fastboot.txt)

santhosh-kt · 2021-08-31T12:44:06Z

files/build_templates/docker_image_ctl.j2

@@ -60,6 +60,16 @@ function getMountPoint()
    echo $1 | python -c "import sys, json, os; mnts = [x for x in json.load(sys.stdin)[0]['Mounts'] if x['Destination'] == '/usr/share/sonic/hwsku']; print '' if len(mnts) == 0 else os.path.abspath(mnts[0]['Source'])" 2>/dev/null
 }

+function isPlatformReset()
+{
+    output=$(python3 -c "import sonic_platform.platform; p = sonic_platform.platform.Platform(); c = p.get_chassis(); hw_rc_major, hw_rc_minor = c.get_reboot_cause(); print(hw_rc_major)" 2>/dev/null)


@sujinmkang : On checking the logs double time, found out one of the process(database.sh) is mentioning it as cold reset despite fast-boot happened. I suspect this is because either determine-reboot-cause or process-reboot-cause function is not up until that time?
Will there be any impact if database is still considering the BOOT_TYPE as cold?

sonic database.sh[680]: BOOT_TYPE we received cold sonic lldp.sh[2314]: BOOT_TYPE we received fast sonic pmon.sh[2308]: BOOT_TYPE we received fast sonic-s6100-01 bgp.sh[2886]: BOOT_TYPE we received fast sonic-s6100-01 swss.sh[2887]: BOOT_TYPE we received fast sonic-s6100-01 teamd.sh[3655]: BOOT_TYPE we received fast sonic-s6100-01 snmp.sh[3658]: BOOT_TYPE we received fast sonic-s6100-01 radv.sh[3651]: BOOT_TYPE we received fast sonic-s6100-01 syncd.sh[3656]: BOOT_TYPE we received fast sonic-s6100-01 dhcp_relay.sh[4441]: BOOT_TYPE we received fast

@santhosh-kt we're not using determine-reboot-cause or process-reboot-cause for this determination but only using platform api. Probably does the database always consider the boot_type as cold even without this change? Since we're sending the same boot_type to all container starting script. Right?

Sorry! I misunderstood. Except database.sh script all other daemon scripts are called lately if I understood.

So all those scripts(except database.sh) could have easily interpreted the boot type since get_reboot_cause() is showing correct value as track_reboot_reason.sh script got executed before these scripts.

But for database.sh, this get_reboot_cause() is showing wrong value as it runs even before s6100_platform.sh itself.
My question is, will wrong BOOT_TYPE has any impact in database.sh script?

@santhosh-kt can we detect the platform driver got initialized or not? Or should we check the exception from platform api call to see the platform driver initialization status? Any other suggestion?

@santhosh-kt I checked the database container start code, I don't think the boot_type is be used for database container starts.

@yxieca I don't think the database container is starting differently based on the boot-type. do you have any comments or concern?

Database container start code has a special handling during warm reboot: https://github.com/Azure/sonic-buildimage/blob/master/files/build_templates/docker_image_ctl.j2#L87

Add additional reboot cause information to track the unexpected reboot cases

Since platform driver is not ready before database container, we cannot determine if there is any platform reset happens when database starts. So Delete the setting before starting swss if any platform reset happened, so that the swss/orchagent can perform loading phy firmware downloading before it hits the phy connection timeout.

sujinmkang · 2021-09-20T16:21:46Z

@yxieca Since the platform driver is not ready by database container starts and we are sure the platform driver is ready before swss starting, I changed the code to Delete "FAST_REBOOT|system" key when swss starts instead of checking the platform reset in database starts. What do you think this change?

…e comments.

yxieca · 2021-09-20T21:20:29Z

@yxieca Since the platform driver is not ready by database container starts and we are sure the platform driver is ready before swss starting, I changed the code to Delete "FAST_REBOOT|system" key when swss starts instead of checking the platform reset in database starts. What do you think this change?

Is this question related to this PR? I didn't find the change matches this question?

…wss docker start script

available.

sujinmkang · 2021-10-05T08:49:17Z

@santhosh-kt Can you please verify this new PR with the scenario?
@yxieca Can you please review the logic to detect the platform reset/reboot cause during fast-reload/reboot case?

santhosh-kt · 2021-10-06T05:27:49Z

Hi @sujinmkang : Following changes can be tried out in master image but as I mentioned earlier, latest master raw image will expand upto 3GB and it will cause disk space issue while conversion.
I see to try the same in 201811 but we don't have determine-reboot-cause changes as well as no support on previous-reboot-cause.json.

sujinmkang · 2021-10-06T16:23:46Z

@santhosh-kt I created a new PR for 201811 : #8912

sujinmkang · 2021-10-06T22:27:57Z

@santhosh-kt Can you please give me some more information about the 3G image issue? Is the issue still existing without my PR or does it happen with my PR?

santhosh-kt · 2021-10-07T05:09:43Z

@sujinmkang This issue is seen even before your changes. Below commit increased the raw disk size since docker file system got increased in size as well as "no disk space left" build error was seen when sonic-broadcom.raw build was generated with 2GB as RAW SIZE LIMIT.

cbe948e#diff-a681207dd6c606f49b8ab351137a2c0238f02908652615a58b73cf74865e55b1

yxieca · 2021-10-11T03:27:23Z

files/build_templates/docker_image_ctl.j2

+            if [[ $REBOOT_CAUSE =~ $REG_BOOT_TYPE ]]; then
+                if [[ "${EXTRA_CAUSE}" != "${CAUSE_NO_AVAIL}" ]]; then
+                    # Delete the FAST_REBOOT|system db setting
+                    $SONIC_DB_CLI STATE_DB DEL "FAST_REBOOT|system" &>/dev/null


How can you guarantee that syncd didn't take fast-reboot approach? syncd could have read this value before swss deletes it.

yxieca · 2021-10-11T03:27:49Z

files/build_templates/docker_image_ctl.j2

+    if [[ "$BOOT_TYPE" == "fast" ]] && [[ -d /host/fast-reboot ]]; then
+        if [[ -f /host/reboot-cause/previous-reboot-cause.json ]]; then
+            REG_BOOT_TYPE="fast*"
+	    CAUSE_NO_AVAIL="\"N/A\""


can you use white space instead of tab?

sujinmkang · 2022-01-13T04:21:42Z

Closing this PR since we don't need this PR for master branch

Check platform reboot cause to see if any reset happened during

0cef668

fast/warm-reboot

sujinmkang requested a review from lguohan as a code owner June 19, 2021 08:40

jleveque reviewed Jun 24, 2021

View reviewed changes

review comments

61192e8

santhosh-kt reviewed Aug 31, 2021

View reviewed changes

Platform reboot cause can be referred to set the "FAST_REBOOT|system"

286e115

Add additional reboot cause information to track the unexpected reboot cases

sujinmkang closed this Sep 17, 2021

sujinmkang reopened this Sep 17, 2021

Add redirection of stdout/stderr for db delete command and change som…

fa40a2f

…e comments.

sujinmkang added 3 commits September 29, 2021 16:38

move the fast reboot flag removal from post action to pre action of s…

f4a07cc

…wss docker start script

Add additional check for sonic_platform import error

c9c135c

Use previous-reboot-cause.json to check if any platform reboot-cause

3358e50

available.

yxieca reviewed Oct 11, 2021

View reviewed changes

sujinmkang closed this Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check platform reboot cause to see if any reset happened during fast/warm-reboot #7920

Check platform reboot cause to see if any reset happened during fast/warm-reboot #7920

sujinmkang commented Jun 19, 2021

jleveque commented Jun 22, 2021

azure-pipelines bot commented Jun 22, 2021

jleveque commented Jun 24, 2021

jleveque Jun 24, 2021

santhosh-kt Aug 24, 2021

santhosh-kt Aug 24, 2021

santhosh-kt Aug 25, 2021

jleveque Jun 24, 2021

azure-pipelines bot commented Jun 24, 2021

sujinmkang commented Aug 4, 2021

azure-pipelines bot commented Aug 4, 2021

santhosh-kt commented Aug 30, 2021

santhosh-kt Aug 31, 2021

sujinmkang Sep 1, 2021

santhosh-kt Sep 2, 2021

sujinmkang Sep 14, 2021

sujinmkang Sep 14, 2021

sujinmkang Sep 14, 2021

yxieca Sep 14, 2021

sujinmkang commented Sep 20, 2021

yxieca commented Sep 20, 2021

sujinmkang commented Oct 5, 2021

santhosh-kt commented Oct 6, 2021

sujinmkang commented Oct 6, 2021 •

edited

Loading

sujinmkang commented Oct 6, 2021

santhosh-kt commented Oct 7, 2021

yxieca Oct 11, 2021

yxieca Oct 11, 2021

sujinmkang commented Jan 13, 2022

Check platform reboot cause to see if any reset happened during fast/warm-reboot #7920

Check platform reboot cause to see if any reset happened during fast/warm-reboot #7920

Conversation

sujinmkang commented Jun 19, 2021

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

jleveque commented Jun 22, 2021

azure-pipelines bot commented Jun 22, 2021

jleveque commented Jun 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azure-pipelines bot commented Jun 24, 2021

sujinmkang commented Aug 4, 2021

azure-pipelines bot commented Aug 4, 2021

santhosh-kt commented Aug 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sujinmkang commented Sep 20, 2021

yxieca commented Sep 20, 2021

sujinmkang commented Oct 5, 2021

santhosh-kt commented Oct 6, 2021

sujinmkang commented Oct 6, 2021 • edited Loading

sujinmkang commented Oct 6, 2021

santhosh-kt commented Oct 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sujinmkang commented Jan 13, 2022

sujinmkang commented Oct 6, 2021 •

edited

Loading