Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

201911 -> 202211 fast/warm-upgrade failure due to "unknown" compression #13568

Closed
stepanblyschak opened this issue Jan 31, 2023 · 2 comments · Fixed by #13837
Closed

201911 -> 202211 fast/warm-upgrade failure due to "unknown" compression #13568

stepanblyschak opened this issue Jan 31, 2023 · 2 comments · Fixed by #13837
Assignees
@stepanblyschak
Labels
Triaged this issue has been triaged

Comments

@stepanblyschak
Copy link
Collaborator

Description

Warm reboot fails due to failure in attempt to mount next image squashfs. This is needed for several things including asic config check, FW upgrade on Nvidia platform, etc. The compression algorithm was changed to zstd in 202211 but in 201911 this compression is not supported leading to errors and unability to perform warm reboot. Commit that introduced zstd: d536592

Steps to reproduce the issue:

  1. boot 201911 image
  2. install 202211 image
  3. sudo warm-reboot -vvvf

Describe the results you received:

root@r-tigris-04:/home/admin# sonic_installer install 202211.bin -y
Image SONiC-OS-202211_RC_2.2-02e5d77ca_Internal is already installed. Setting it as default...
Command: grub-set-default --boot-directory=/host 1

Command: sync;sync;sync

Command: sleep 3

Done
root@r-tigris-04:/home/admin# fast-reboot
mount: wrong fs type, bad option, bad superblock on /dev/loop2,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

ERROR: no such file: /tmp/image-202211_RC_2.2-02e5d77ca_Internal-fs//etc/mlnx/fw-SPC2.mfa

root@r-tigris-04:/home/admin# echo $?
1
root@r-tigris-04:/home/admin#

dmesg output:

admin@str-msn2700-22:~$ sudo dmesg | tail
[34260.312537] squashfs: SQUASHFS error: Filesystem uses "unknown" compression. This is not supported

Describe the results you expected:

Successful warm-reboot operation

Output of show version:

SONiC-OS-202211_RC_2.2-02e5d77ca_Internal

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@stepanblyschak
Copy link
Collaborator Author

Seems it is the same issue as in #12171

@gechiang
Copy link
Collaborator

gechiang commented Feb 1, 2023

@stepanblyschak this is a known issue. The upgrade path from 201911 to 202211 is not supported currently. it needs to be done step wise by first upgrade to 202012.

@stepanblyschak please consider move the firmware extraction to somewhere else as a resolution for this issue.

@gechiang gechiang added the Triaged this issue has been triaged label Feb 1, 2023
This was referenced Feb 16, 2023
Merged
Merged
liat-grozovik pushed a commit that referenced this issue Feb 22, 2023
@stepanblyschak
[202205][Mellanox] Place FW binaries under platform directory instead…
70e2ea1 
… of squashfs (#13838)

Fixes #13568
Backport of #13837

Upgrade from old image always requires squashfs mount to get the next image FW binary. This can be avoided if we put FW binary under platform directory which is easily accessible after installation:

admin@r-spider-05:~$ ls /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
/host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
admin@r-spider-05:~$ ls -al /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa
lrwxrwxrwx 1 root root 66 Feb  8 17:57 /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa -> /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa

- Why I did it
202211 and above uses different squashfs compression type that 201911 kernel can not handle. Therefore, we avoid mounting squashfs altogether with this change.

- How I did it
Place FW binary under /host/image-/platform/mlnx/, soft links in /etc/mlnx are created to avoid breaking existing scripts/automation.
/etc/mlnx/fw-SPCX.mfa is a soft link always pointing to the FW that should be used in current image
mlnx-fw-upgrade.sh is updated to prefer /host/image-/platform/mlnx location and fallback to /etc/mlnx in squashfs in case new location does not exist. This is necessary to do image downgrade.

- How to verify it
Upgrade from 201911 to 202205
202205 to 201911 downgrade
202205 -> 202205 reboot
ONIE -> 202205 boot (First FW burn)

Signed-off-by: Stepan Blyschak <stepanb@nvidia.com>
liat-grozovik pushed a commit that referenced this issue Mar 6, 2023
@stepanblyschak
[Mellanox] Place FW binaries under platform directory instead of squa…
f908dfe 
…shfs (#13837)

Fixes #13568

Upgrade from old image always requires squashfs mount to get the next image FW binary. This can be avoided if we put FW binary under platform directory which is easily accessible after installation:

admin@r-spider-05:~$ ls /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
/host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
admin@r-spider-05:~$ ls -al /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa
lrwxrwxrwx 1 root root 66 Feb  8 17:57 /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa -> /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa

- Why I did it
202211 and above uses different squashfs compression type that 201911 kernel can not handle. Therefore, we avoid mounting squashfs altogether with this change.

- How I did it
Place FW binary under /host/image-/platform/mlnx/, soft links in /etc/mlnx are created to avoid breaking existing scripts/automation.
/etc/mlnx/fw-SPCX.mfa is a soft link always pointing to the FW that should be used in current image
mlnx-fw-upgrade.sh is updated to prefer /host/image-/platform/mlnx location and fallback to /etc/mlnx in squashfs in case new location does not exist. This is necessary to do image downgrade.

- How to verify it
Upgrade from 201911 to master
master to 201911 downgrade
master -> master reboot
ONIE -> master boot (First FW burn)
Which release branch to backport (provide reason below if selected)
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this issue Mar 7, 2023
@stepanblyschak @mssonicbld
[Mellanox] Place FW binaries under platform directory instead of squa…
aa03e69 
…shfs (sonic-net#13837)

Fixes sonic-net#13568

Upgrade from old image always requires squashfs mount to get the next image FW binary. This can be avoided if we put FW binary under platform directory which is easily accessible after installation:

admin@r-spider-05:~$ ls /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
/host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
admin@r-spider-05:~$ ls -al /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa
lrwxrwxrwx 1 root root 66 Feb  8 17:57 /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa -> /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa

- Why I did it
202211 and above uses different squashfs compression type that 201911 kernel can not handle. Therefore, we avoid mounting squashfs altogether with this change.

- How I did it
Place FW binary under /host/image-/platform/mlnx/, soft links in /etc/mlnx are created to avoid breaking existing scripts/automation.
/etc/mlnx/fw-SPCX.mfa is a soft link always pointing to the FW that should be used in current image
mlnx-fw-upgrade.sh is updated to prefer /host/image-/platform/mlnx location and fallback to /etc/mlnx in squashfs in case new location does not exist. This is necessary to do image downgrade.

- How to verify it
Upgrade from 201911 to master
master to 201911 downgrade
master -> master reboot
ONIE -> master boot (First FW burn)
Which release branch to backport (provide reason below if selected)
mssonicbld pushed a commit that referenced this issue Mar 8, 2023
@stepanblyschak @mssonicbld
[Mellanox] Place FW binaries under platform directory instead of squa…
969166d 
…shfs (#13837)

Fixes #13568

Upgrade from old image always requires squashfs mount to get the next image FW binary. This can be avoided if we put FW binary under platform directory which is easily accessible after installation:

admin@r-spider-05:~$ ls /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
/host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
admin@r-spider-05:~$ ls -al /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa
lrwxrwxrwx 1 root root 66 Feb  8 17:57 /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa -> /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa

- Why I did it
202211 and above uses different squashfs compression type that 201911 kernel can not handle. Therefore, we avoid mounting squashfs altogether with this change.

- How I did it
Place FW binary under /host/image-/platform/mlnx/, soft links in /etc/mlnx are created to avoid breaking existing scripts/automation.
/etc/mlnx/fw-SPCX.mfa is a soft link always pointing to the FW that should be used in current image
mlnx-fw-upgrade.sh is updated to prefer /host/image-/platform/mlnx location and fallback to /etc/mlnx in squashfs in case new location does not exist. This is necessary to do image downgrade.

- How to verify it
Upgrade from 201911 to master
master to 201911 downgrade
master -> master reboot
ONIE -> master boot (First FW burn)
Which release branch to backport (provide reason below if selected)
xumia pushed a commit to xumia/sonic-buildimage-1 that referenced this issue Mar 10, 2023
@stepanblyschak @xumia
[Mellanox] Place FW binaries under platform directory instead of squa…
d5d24c3 
…shfs (sonic-net#13837)

Fixes sonic-net#13568

Upgrade from old image always requires squashfs mount to get the next image FW binary. This can be avoided if we put FW binary under platform directory which is easily accessible after installation:

admin@r-spider-05:~$ ls /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
/host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa
admin@r-spider-05:~$ ls -al /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa
lrwxrwxrwx 1 root root 66 Feb  8 17:57 /tmp/image-fw-new-loc.0-dirty-20230208.193534-fs/etc/mlnx/fw-SPC.mfa -> /host/image-fw-new-loc.0-dirty-20230208.193534/platform/fw-SPC.mfa

- Why I did it
202211 and above uses different squashfs compression type that 201911 kernel can not handle. Therefore, we avoid mounting squashfs altogether with this change.

- How I did it
Place FW binary under /host/image-/platform/mlnx/, soft links in /etc/mlnx are created to avoid breaking existing scripts/automation.
/etc/mlnx/fw-SPCX.mfa is a soft link always pointing to the FW that should be used in current image
mlnx-fw-upgrade.sh is updated to prefer /host/image-/platform/mlnx location and fallback to /etc/mlnx in squashfs in case new location does not exist. This is necessary to do image downgrade.

- How to verify it
Upgrade from 201911 to master
master to 201911 downgrade
master -> master reboot
ONIE -> master boot (First FW burn)
Which release branch to backport (provide reason below if selected)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stepanblyschak stepanblyschak

Labels
Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
2 participants
@stepanblyschak @gechiang