Fix for fast/cold-boot: call db_migrator only after old config is loa…

…ded (sonic-net#14933)

Why I did it
Fix the issue where db_migrator is called before DB is loaded w/ config. This leads to db_migrator:

Not finding anything, and resumes to incorrectly migrate every missing config
This is not expected. migration should happen after the old config is loaded and only new schema changes need migration.
Since DB does not have anything when migrator is called, db_migrator fails when some APIs return None.
The reason for incorrect call is that:

database service starts db_migrator as part of startup sequence.
config-setup service loads data from old-config/minigraph. However, since it has Requires=database.service.
Hence, config-setup starts only when database service is started. And database service is started when db_migrator is completed.
Fixed by:

Check if this is first time boot by checking pending_config_migration flag.
If pending_config_migration is enabled, then do not call db_migrator as part of database service startup.
Let database service start which triggers config-setup service to start.
Now call db_migrator after when config-setup service loads old-config/minigraph

files/build_templates/docker_image_ctl.j2

@@ -257,10 +257,18 @@ function postStartAction()
             $SONIC_DB_CLI CONFIG_DB SET "CONFIG_DB_INITIALIZED" "1"
         fi
 
-        if [[ -x /usr/local/bin/db_migrator.py ]]; then
-            # Migrate the DB to the latest schema version if needed
-            if [ -z "$DEV" ]; then
-                /usr/local/bin/db_migrator.py -o migrate
+        if [ -e /tmp/pending_config_migration ]; then
+            # this is first boot to a new image, config-setup execution is pending.
+            # For fast/cold reboot case, DB contains nothing at this point
+            # Call db_migrator after config-setup loads the config (from old config or minigraph)
+            echo "Delaying db_migrator until config migration is over"
+        else
+            # this is not a first time boot to a new image. Datbase container starts w/ old pre-existing config
+            if [[ -x /usr/local/bin/db_migrator.py ]]; then
+                # Migrate the DB to the latest schema version if needed
+                if [ -z "$DEV" ]; then
+                    /usr/local/bin/db_migrator.py -o migrate
+                fi
             fi
         fi
         # Add redis UDS to the redis group and give read/write access to the group

files/image_config/config-setup/config-setup

@@ -304,6 +304,16 @@ check_all_config_db_present()
     return 0   
 }
 
+# DB schema is subject to change between two images
+# Perform DB schema migration after loading backup config from previous image
+do_db_migration()
+{
+    if [[ -x /usr/local/bin/db_migrator.py ]]; then
+        # Migrate the DB to the latest schema version if needed
+        /usr/local/bin/db_migrator.py -o migrate
+    fi
+}
+
 # Perform configuration migration from backup copy.
 #  - This step is performed when a new image is installed and SONiC switch boots into it
 do_config_migration()
@@ -326,16 +336,19 @@ do_config_migration()
     if [ x"${WARM_BOOT}" == x"true" ]; then
         echo "Warm reboot detected..."
         disable_updategraph
+        do_db_migration
         rm -f /tmp/pending_config_migration
         exit 0
     elif check_all_config_db_present; then
         echo "Use config_db.json from old system..."
         reload_configdb
+        do_db_migration
         # Disable updategraph
         disable_updategraph
     elif [ -r ${MINGRAPH_FILE} ]; then
         echo "Use minigraph.xml from old system..."
         reload_minigraph
+        do_db_migration
         # Disable updategraph
         disable_updategraph
     else