Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Boot type during config reload after fast reboot is invalid #3697

Closed
nazariig opened this issue Nov 1, 2019 · 3 comments · Fixed by #3741 or sonic-net/sonic-sairedis#529
Closed

Boot type during config reload after fast reboot is invalid #3697

nazariig opened this issue Nov 1, 2019 · 3 comments · Fixed by #3741 or sonic-net/sonic-sairedis#529
Assignees

Comments

@nazariig
Copy link
Collaborator

nazariig commented Nov 1, 2019

Description
After the fast reboot, on the next config reload, the system behaves in the same way as during the fast reboot flow.

It seems that the boot type detection logic in sonic-buildimage and sonic-sairedis are not aligned:

sonic-buildimage/files/scripts/syncd.sh

function getBootType()
{
    # same code snippet in files/build_templates/docker_image_ctl.j2
    case "$(cat /proc/cmdline)" in
    *SONIC_BOOT_TYPE=warm*)
        TYPE='warm'
        ;;
    *SONIC_BOOT_TYPE=fastfast*)
        TYPE='fastfast'
        ;;
    *SONIC_BOOT_TYPE=fast*|*fast-reboot*)
        TYPE=$(awk '{ if ($1 <= 180) print "fast"; else print "cold" }' /proc/uptime)
        ;;
    *)
        TYPE='cold'
    esac
    echo "${TYPE}"
}

sonic-sairedis/syncd/scripts/syncd_init_common.sh

case "$(cat /proc/cmdline)" in
  *SONIC_BOOT_TYPE=fastfast*)
    if [ -e /var/warmboot/warm-starting ]; then
        FASTFAST_REBOOT='yes'
    fi
    ;;
  *SONIC_BOOT_TYPE=fast*|*fast-reboot*)
     FAST_REBOOT='yes'
    ;;
  *)
     FAST_REBOOT='no'
     FASTFAST_REBOOT='no'
    ;;
esac

...

function set_start_type()
{
    if [ x"$WARM_BOOT" == x"true" ]; then
        CMD_ARGS+=" -t warm"
    elif [ x"$FAST_REBOOT" == x"yes" ]; then
        CMD_ARGS+=" -t fast"
    elif [ x"$FASTFAST_REBOOT" == x"yes" ]; then
        CMD_ARGS+=" -t fastfast"
    fi
}

Steps to reproduce the issue:

  1. fast-reboot
  2. config reload -y

Describe the results you received:

root@sonic:/home/admin# docker exec -ti syncd bash
root@sonic:/# ps -aux | grep [s]yncd
root        23  5.1  2.5 1265152 207064 pts/0  Sl   13:15   0:04 /usr/bin/syncd -u -p /tmp/sai.profile -t fast -r -m /usr/share/sonic/hwsku/port_config.ini

Describe the results you expected:

root@sonic:/home/admin# docker exec -ti syncd bash
root@sonic:/# ps -aux | grep [s]yncd
root        23  5.1  2.5 1265152 207064 pts/0  Sl   13:15   0:04 /usr/bin/syncd -u -p /tmp/sai.profile -t cold -r -m /usr/share/sonic/hwsku/port_config.ini

Additional information you deem important (e.g. issue happens only occasionally):

  • N/A
@stepanblyschak
Copy link
Collaborator

@pavel-shirshov Instead of checking uptime? Can we check /proc/cmdline once on database postStartAction and put a mark in new state DB table (lets say 'FAST_REBOOT|system' entry to be aligned with warm reboot). Every service doing start checks the database. On config reload the state DB is cleared. What do you think?

@pavel-shirshov
Copy link
Contributor

pavel-shirshov commented Nov 7, 2019

@stepanblyschak

I think that it wouldn't be enough.
If someone restart bgp container without reloading config, bgpd will think we're still in fast-reboot mode.
I think we should disable FAST-REBOOT mode in 3 minutes.

We can as you propose put the key into the DB and remove it in 3 minutes (or by config reload).
In the scripts we will check the key.
Will it work for you?

@stepanblyschak
Copy link
Collaborator

@pavel-shirshov I wasn't aware about BGP case, a new fastboot-finalizer service (similarly to warmboot-service) can remove the key after 3 min. It will work for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants