Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Warm reboot: restore the database docker with content saved #2216

Merged
merged 8 commits into from
Nov 2, 2018
Merged

Warm reboot: restore the database docker with content saved #2216

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ace9553
Database service warm start
qiluo-msft Oct 9, 2018
fa331f9
Update files/build_templates/docker_image_ctl.j2
yxieca Oct 31, 2018
504d691
Update files/build_templates/docker_image_ctl.j2
yxieca Oct 31, 2018
2ed1ebe
Update files/build_templates/docker_image_ctl.j2
yxieca Oct 31, 2018
f6c7a64
Fix sudo, and exit immediately if any failure
qiluo-msft Nov 1, 2018
33a1c9b
Fix syntax
qiluo-msft Nov 1, 2018
40788df
Fix redisLoadAndDelete argument, and refactor
qiluo-msft Nov 1, 2018
0030740
Fix: sudo, ping through unix socket
qiluo-msft Nov 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 42 additions & 5 deletions files/build_templates/docker_image_ctl.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,56 @@

function getMountPoint()
{
echo $1 | python -c "import sys, json, os; mnts = [x for x in json.load(sys.stdin)[0]['Mounts'] if x['Destination'] == '/usr/share/sonic/hwsku']; print '' if len(mnts) == 0 else os.path.basename(mnts[0]['Source'])" 2>/dev/null
echo $1 | python -c "import sys, json, os; mnts = [x for x in json.load(sys.stdin)[0]['Mounts'] if x['Destination'] == '/usr/share/sonic/hwsku']; print '' if len(mnts) == 0 else os.path.basename(mnts[0]['Source'])" 2>/dev/null
}

function getBootType()
{
local TYPE
case "$(cat /proc/cmdline)" in
*SONIC_BOOT_TYPE=fast*)
TYPE='fast'
;;
*SONIC_BOOT_TYPE=warm*)
TYPE='warm'
;;
*)
TYPE="normal"
esac
echo $TYPE
}

function postStartAction()
{
{%- if docker_container_name == "database" %}
until [[ $(/usr/bin/docker exec database redis-cli ping | grep -c PONG) -gt 0 ]]; do
REDIS_SOCK="/var/run/redis/redis.sock"
until [[ $(/usr/bin/docker exec database redis-cli -s $REDIS_SOCK ping | grep -c PONG) -gt 0 ]]; do
sleep 1;
done
WARM_DIR=/host/warmboot
SUDO="sudo -n"
BOOT_TYPE=`getBootType`
if [[ "$BOOT_TYPE" == "warm" && -d $WARM_DIR ]]; then
function redisLoadAndDelete()
{
Copy link
Contributor

@yxieca yxieca Oct 31, 2018 •
edited by qiluo-msft
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function needs to also take database ID as a parameter #Resolved

DB="$1"
FILENAME="$2"
test -e $FILENAME || { echo "No file $FILENAME" >&2; exit 10; }
$SUDO redis-load -s $REDIS_SOCK -d $DB -e EMPTY $FILENAME || { echo "Failed to redis-load $FILENAME" >&2; exit 11; }
$SUDO rm $FILENAME || exit 12
}
# Load applDB from /host/warm-reboot/appl_db.json
redisLoadAndDelete 0 $WARM_DIR/appl_db.json
# Load configDB from /host/warm-reboot/config_db.json
redisLoadAndDelete 4 $WARM_DIR/config_db.json
# Load stateDB from /host/warm-reboot/state_db.json
redisLoadAndDelete 6 $WARM_DIR/state_db.json
# Load asicDB from /host/warm-reboot/asic_db.json
redisLoadAndDelete 1 $WARM_DIR/asic_db.json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing came to my mind: I think we should test all file existence before proceeding with restoration. If any file is missing, there is something wrong. We should restore all or nothing. Do you agree?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current implementation treat this case as a service start failure. Later we can refine the case with robust recovery.

fi
{%- elif docker_container_name == "swss" %}
docker exec swss rm -f /ready # remove cruft
if [[ -d /host/fast-reboot ]];
then
if [[ "$BOOT_TYPE" == "fast" && -d /host/fast-reboot ]]; then
test -e /host/fast-reboot/fdb.json && docker cp /host/fast-reboot/fdb.json swss:/
test -e /host/fast-reboot/arp.json && docker cp /host/fast-reboot/arp.json swss:/
test -e /host/fast-reboot/default_routes.json && docker cp /host/fast-reboot/default_routes.json swss:/
Expand Down Expand Up @@ -58,7 +95,7 @@ start() {
echo "Starting existing {{docker_container_name}} container with HWSKU $HWSKU"
docker start {{docker_container_name}}
postStartAction
exit 0
exit $?
fi

# docker created with a different HWSKU, remove and recreate
Expand Down