Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fast-reboot] fast reboot upgrade from 201911 to 202205 has downtime >30 sec #16699

Closed
stepanblyschak opened this issue Sep 26, 2023 · 2 comments · Fixed by #16733
Closed

[fast-reboot] fast reboot upgrade from 201911 to 202205 has downtime >30 sec #16699

stepanblyschak opened this issue Sep 26, 2023 · 2 comments · Fixed by #16733
Assignees
Labels
MSFT Triaged this issue has been triaged

Comments

@stepanblyschak
Copy link
Collaborator

stepanblyschak commented Sep 26, 2023

Description

During boot from an old image, FAST_REBOOT|system flag should be set in STATE DB, then db_migrator translates it into FAST_RESTART_ENABLE_TABLE|system flag that every component reads to determine whether system is in fast boot mode.

The code that sets FAST_REBOOT|system flag was removed by https://github.com/sonic-net/sonic-buildimage/pull/15685/files#diff-130e5ab75471398db06b73a4c6e4b56517f42e3c1f17870b5309620fc0e2fafbL254.
A comment was posted in PR after merge with explanation - https://github.com/sonic-net/sonic-buildimage/pull/15685/files#r1295535606.

Steps to reproduce the issue:

  1. Install latest 201911
  2. Install latest 202205
  3. Do fast-reboot
  4. After fast-reboot has finished, check that syncd starts in cold mode:
$ cat dump/reboot.cause
User issued 'fast-reboot' command [User: admin, Time: Mon Sep 25 11:56:20 UTC 2023]
$ cat dump/ps.aux | grep 'syncd -s'
root        6981  2.9  1.9 1415360 158540 pts/0  Sl   12:00   0:43 /usr/bin/syncd -s -p /tmp/sai.profile

Describe the results you received:

Syncd started in cold mode.

Describe the results you expected:

Syncd starts in fast mode.

Output of show version:

SONiC Software Version: SONiC.202205_3_rc.8-f00856b1a_Internal
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: f00856b1a
Build date: Fri Sep 22 22:06:28 UTC 2023
Built by: sw-r2d2-bot@r-build-sonic-ci03-244

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700-D48C8
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1822K07815
Model Number: MSN2700-CS2FO
Hardware Revision: A1
Uptime: 12:24:49 up 24 min,  1 user,  load average: 0.90, 0.77, 0.88
Date: Mon 25 Sep 2023 12:24:49

Docker images:
REPOSITORY                    TAG                                IMAGE ID       SIZE
docker-macsec                 latest                             edc3b9918675   332MB
docker-dhcp-relay             latest                             e7f46c58210d   321MB
docker-syncd-mlnx             202205_3_rc.8-f00856b1a_Internal   5758c6a421d9   905MB
docker-syncd-mlnx             latest                             5758c6a421d9   905MB
docker-sonic-telemetry        202205_3_rc.8-f00856b1a_Internal   3642e9684947   394MB
docker-sonic-telemetry        latest                             3642e9684947   394MB
docker-teamd                  202205_3_rc.8-f00856b1a_Internal   c8552ab8c66c   330MB
docker-teamd                  latest                             c8552ab8c66c   330MB
docker-snmp                   202205_3_rc.8-f00856b1a_Internal   4242a44b9217   364MB
docker-snmp                   latest                             4242a44b9217   364MB
docker-router-advertiser      202205_3_rc.8-f00856b1a_Internal   4bf4a260c7ad   314MB
docker-router-advertiser      latest                             4bf4a260c7ad   314MB
docker-platform-monitor       202205_3_rc.8-f00856b1a_Internal   a769ae6b56c3   750MB
docker-platform-monitor       latest                             a769ae6b56c3   750MB
docker-orchagent              202205_3_rc.8-f00856b1a_Internal   bdc3dafeedc7   347MB
docker-orchagent              latest                             bdc3dafeedc7   347MB
docker-mux                    202205_3_rc.8-f00856b1a_Internal   6c4f636f2ce4   362MB
docker-mux                    latest                             6c4f636f2ce4   362MB
docker-lldp                   202205_3_rc.8-f00856b1a_Internal   d586bb6d050b   356MB
docker-lldp                   latest                             d586bb6d050b   356MB
docker-fpm-frr                202205_3_rc.8-f00856b1a_Internal   96d5bad2ae7c   359MB
docker-fpm-frr                latest                             96d5bad2ae7c   359MB
docker-database               202205_3_rc.8-f00856b1a_Internal   3e354f4fd60e   314MB
docker-database               latest                             3e354f4fd60e   314MB
docker-sonic-mgmt-framework   202205_3_rc.8-f00856b1a_Internal   2884af3951b8   433MB
docker-sonic-mgmt-framework   latest                             2884af3951b8   433MB
docker-sflow                  202205_3_rc.8-f00856b1a_Internal   22948a0972a9   303MB
docker-sflow                  latest                             22948a0972a9   303MB
docker-nat                    202205_3_rc.8-f00856b1a_Internal   6386cd493360   305MB
docker-nat                    latest                             6386cd493360   305MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

sonic_dump_arc-switch1025_20230925_122433.tar.gz

@stepanblyschak
Copy link
Collaborator Author

@vaibhavhd Could you please take a look?

@judyjoseph
Copy link
Contributor

@stepanblyschak can you check if this fixes the isse :#16669.

@vaibhavhd is this fix needed in 202205 also?

@judyjoseph judyjoseph added Triaged this issue has been triaged MSFT labels Sep 27, 2023
yxieca pushed a commit that referenced this issue Sep 28, 2023
…t fast-reboot from older images (#16733)

Why I did it
Fix: #16699

Fast reboot is failing from old OS versions (eg., 201911 image) to latest (eg., master branch) after PR #15685

The system wide flag for FAST_REBOOT is still required when the base OS version does not support the new fast-reboot reconciliation logic (no db dump)
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Sep 28, 2023
…t fast-reboot from older images (sonic-net#16733)

Why I did it
Fix: sonic-net#16699

Fast reboot is failing from old OS versions (eg., 201911 image) to latest (eg., master branch) after PR sonic-net#15685

The system wide flag for FAST_REBOOT is still required when the base OS version does not support the new fast-reboot reconciliation logic (no db dump)
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Sep 28, 2023
…t fast-reboot from older images (sonic-net#16733)

Why I did it
Fix: sonic-net#16699

Fast reboot is failing from old OS versions (eg., 201911 image) to latest (eg., master branch) after PR sonic-net#15685

The system wide flag for FAST_REBOOT is still required when the base OS version does not support the new fast-reboot reconciliation logic (no db dump)
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Sep 28, 2023
…t fast-reboot from older images (sonic-net#16733)

Why I did it
Fix: sonic-net#16699

Fast reboot is failing from old OS versions (eg., 201911 image) to latest (eg., master branch) after PR sonic-net#15685

The system wide flag for FAST_REBOOT is still required when the base OS version does not support the new fast-reboot reconciliation logic (no db dump)
mssonicbld pushed a commit that referenced this issue Oct 20, 2023
…t fast-reboot from older images (#16733)

Why I did it
Fix: #16699

Fast reboot is failing from old OS versions (eg., 201911 image) to latest (eg., master branch) after PR #15685

The system wide flag for FAST_REBOOT is still required when the base OS version does not support the new fast-reboot reconciliation logic (no db dump)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MSFT Triaged this issue has been triaged
Projects
None yet
3 participants