NC | Online Upgrade | add host config dir version to system.json | Health - add blocked hosts check

[romayalon]

Explain the changes

1. Added config_dir_version info per host in system.json.

> sudo cat /etc/noobaa.conf.d/system.json | jq .
{
  "config_directory": {
    "config_dir_version": "1.0.0",
    "upgrade_package_version": "5.18.0",
    "phase": "CONFIG_DIR_UNLOCKED",
    "upgrade_history": {
      "successful_upgrades": []
    }
  },
  "hostname1": {
    "current_version": "5.20.0",
    "config_dir_version": "3.0.0", // <- this is the new added property
    "upgrade_history": {
      "successful_upgrades": [
        {
          "timestamp": 1735114113164,
          "from_version": "5.18.0",
          "to_version": "5.20.0"
        }
      ]
    }
  }
}

3. Health script - added a check that reports on hosts that are blocked for config directory updates based on the comparison of the new property config_dir_version per hosts and the config_directory.config_dir_version. Both compared items are read from system.json

> noobaa-cli diagnose health 2>/dev/null
{
  "response": {
    "code": "HealthStatus",
    "reply": {
      "service_name": "noobaa",
      "status": "NOTOK",
      ......
        },
        "config_directory_status": {
          "phase": "CONFIG_DIR_UNLOCKED",
          "config_dir_version": "1.0.0",
          "upgrade_package_version": "5.18.0",
          "upgrade_status": {
            "message": "there is no in-progress upgrade"
          },
          "blocked_hosts": { // <- this is the new check
            "hostaname1": {
              "host_version": "5.20.0",
              "host_config_dir_version": "3.0.0",
              "error": "running code config_dir_version=3.0.0 is higher than the config dir version mentioned in system.json=1.0.0, any updates to the config directory are blocked until the config dir upgrade"
            }
          }
        }
      }
    }
  }

Issues: Fixed #xxx / Gap #xxx

1. Fixed partially NC | Online upgrade improvements #8586

Testing Instructions:

• Automatic tests -

sudo  jest --testRegex=jest_tests/test_config_fs.test.js
sudo  jest --testRegex=jest_tests/test_nc_upgrade_manager.test.js
sudo node ./node_modules/mocha/bin/mocha src/test/unit_tests/test_nc_health.js

• Manual tests -

1. start noobaa 5.17.z - sudo node noobaa-core/src/cmd/nsfs.js --debug, stop noobaa by CTRL+C
2. git checkout 5.18.0
3. start noobaa 5.18.0 - sudo node noobaa-core/src/cmd/nsfs.js --debug, stop noobaa by CTRL+C
4. run Health CLI - noobaa-cli diagnose health 2>/dev/null
5. expect to see the following blocked hosts in the health response -

"config_directory_status": {
          "error": "config directory data is missing, must upgrade config directory",
          "blocked_hosts": {
            "hostname1": {
              "host_version": "5.18.0",
              "host_config_dir_version": "1.0.0",
              "error": "host's config_dir_version is 1.0.0, system's config_dir_version is undefined"
            }
          }
        }

6. upgrade the config directory - noobaa-cli upgrade start --expected_version 5.18.0 --expected_hosts hostname1 2>/dev/null
7. run again Health CLI - noobaa-cli diagnose health 2>/dev/null
8. expect to not see the blocked host in the new health response -

"config_directory_status": {
    "phase": "CONFIG_DIR_UNLOCKED",
    "config_dir_version": "1.0.0",
    "upgrade_package_version": "5.18.0",
    "upgrade_status": {
      "message": "there is no in-progress upgrade"
    }
}

• Doc added/updated
• Tests added

[shirady]

LGTM