Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SONiC upgrade via warm-reboot requires force flag for warm-reboot script #5972

Closed
stepanblyschak opened this issue Nov 19, 2020 · 1 comment
Closed

SONiC upgrade via warm-reboot requires force flag for warm-reboot script #5972

stepanblyschak opened this issue Nov 19, 2020 · 1 comment
Assignees
@vaibhavhd

Comments

@stepanblyschak
Copy link
Collaborator

Description

SONiC upgrade via warm-reboot requires to add force flag to warm-reboot script

Steps to reproduce the issue:

  1. Install SONiC image wich has new FW/SDK.
  2. Execute 'sudo warm-reboot'
  3. Observe error: 'ASIC config may have changed".

A W/A we used is to add '-f' to warm-reboot script, however that might be unsafe because forcing the warm-reboot means the orchagent pause check will be ignored too.

Describe the results you received:

SONiC upgrade via warm-reboot requires '-f'.

Describe the results you expected:

SONiC upgrade via warm-reboot should not be with force option.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**

SONiC Software Version: SONiC.201911.27-dirty-20201118.102410
Distribution: Debian 9.13
Kernel: 4.9.0-11-2-amd64
Build commit: 500395c5
Build date: Wed Nov 18 10:06:31 UTC 2020
Built by: sw-r2d2-bot@r-build-sonic-ci02

Platform: x86_64-mlnx_msn2010-r0
HwSKU: ACS-MSN2010
ASIC: mellanox
Serial Number: MT1749X10061
Uptime: 08:47:02 up 12:08,  1 user,  load average: 0.58, 1.57, 1.77

Docker images:
REPOSITORY                    TAG                               IMAGE ID            SIZE
docker-syncd-mlnx             201911.27-dirty-20201118.102410   f5a4cac8f89e        397MB
docker-syncd-mlnx             latest                            f5a4cac8f89e        397MB
docker-sonic-telemetry        201911.27-dirty-20201118.102410   a7e9b923a610        353MB
docker-sonic-telemetry        latest                            a7e9b923a610        353MB
docker-router-advertiser      201911.27-dirty-20201118.102410   b4dcead770bb        289MB
docker-router-advertiser      latest                            b4dcead770bb        289MB
docker-platform-monitor       201911.27-dirty-20201118.102410   0fe534d40320        664MB
docker-platform-monitor       latest                            0fe534d40320        664MB
docker-fpm-frr                201911.27-dirty-20201118.102410   bc28400beffb        335MB
docker-fpm-frr                latest                            bc28400beffb        335MB
docker-teamd                  201911.27-dirty-20201118.102410   3ee8fd635d96        315MB
docker-teamd                  latest                            3ee8fd635d96        315MB
docker-lldp-sv2               201911.27-dirty-20201118.102410   0e85e478d60c        312MB
docker-lldp-sv2               latest                            0e85e478d60c        312MB
docker-dhcp-relay             201911.27-dirty-20201118.102410   ebb16cb91b89        299MB
docker-dhcp-relay             latest                            ebb16cb91b89        299MB
docker-database               201911.27-dirty-20201118.102410   df165a740c6e        289MB
docker-database               latest                            df165a740c6e        289MB
docker-snmp-sv2               201911.27-dirty-20201118.102410   8afe13fa3d85        348MB
docker-snmp-sv2               latest                            8afe13fa3d85        348MB
docker-orchagent              201911.27-dirty-20201118.102410   5f3f8d603a9a        333MB
docker-orchagent              latest                            5f3f8d603a9a        333MB
docker-sflow                  201911.27-dirty-20201118.102410   f5df67d2338b        315MB
docker-sflow                  latest                            f5df67d2338b        315MB
docker-nat                    201911.27-dirty-20201118.102410   658aaa9aafdf        316MB
docker-nat                    latest                            658aaa9aafdf        316MB
docker-sonic-mgmt-framework   201911.27-dirty-20201118.102410   94dc20bbc374        427MB
docker-sonic-mgmt-framework   latest                            94dc20bbc374        427MB


**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
@vaibhavhd vaibhavhd self-assigned this Nov 23, 2020
@vaibhavhd vaibhavhd mentioned this issue Dec 5, 2020
Merged
vaibhavhd added a commit to sonic-net/sonic-utilities that referenced this issue Dec 7, 2020
@vaibhavhd
Fast-reboot: add a new flag to ignore ASIC config checksum verificati…
326e534 
…on failures (#1292)

To fix the issue sonic-net/sonic-buildimage#5972
warm-reboot with force flag ignores ASIC config checksum mismatch along with orchagent RESTARTCHECK failure.
This commit accounts for a use case when checksum-verification should be ignored but orchagent pause check should not be ignored.
The change is to add a new option in fast-reboot script to ignore ASIC checksum verification failures.
@vaibhavhd
Copy link
Contributor

Hi @stepanblyschak, thanks for reporting this. I have added a new option (-i) that ignores only ASIC configuration checksum verification failures and kept -f to ignore only orchagent restart crash.

This is the PR: sonic-net/sonic-utilities#1292

Can you please confirm that this resolves the issue raised here?

anand-kumar-subramanian pushed a commit to anand-kumar-subramanian/sonic-utilities that referenced this issue Mar 2, 2021
@vaibhavhd @anand-kumar-subramanian
Fast-reboot: add a new flag to ignore ASIC config checksum verificati…
7b91f9c 
…on failures (sonic-net#1292)

To fix the issue sonic-net/sonic-buildimage#5972
warm-reboot with force flag ignores ASIC config checksum mismatch along with orchagent RESTARTCHECK failure.
This commit accounts for a use case when checksum-verification should be ignored but orchagent pause check should not be ignored.
The change is to add a new option in fast-reboot script to ignore ASIC checksum verification failures.
malletvapid23 added a commit to malletvapid23/Sonic-Utility that referenced this issue Aug 3, 2023
@malletvapid23
Fast-reboot: add a new flag to ignore ASIC config checksum verificati…
c26bf95 
…on failures (#1292)

To fix the issue sonic-net/sonic-buildimage#5972
warm-reboot with force flag ignores ASIC config checksum mismatch along with orchagent RESTARTCHECK failure.
This commit accounts for a use case when checksum-verification should be ignored but orchagent pause check should not be ignored.
The change is to add a new option in fast-reboot script to ignore ASIC checksum verification failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vaibhavhd vaibhavhd

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vaibhavhd @stepanblyschak