NSFS | NC | Health script | distinguish between temporary and persistent errors

[naveenpaul1]

Explain the changes

1. Heath script updated with error_type and for temporary errors multiple retries are added
2. Health issue with missing accounts and buckets folder in the config root path
3. Rsyslog start-up error fixed by removing unwanted default value configuration in noobaa_rsyslog.config

Issues: Fixed #xxx / Gap #xxx

1. NSFS | NC | Health script | distinguish between temporary and persistent errors & retries addition #7714
2. health command is returning empty and the return code is 2  #7730

Testing Instructions:

1. run the health script and verify the health response structure similar to below

  "service_name": "nsfs",
  "status": "NOTOK",
  "memory": "88.6M",
  "error": {
    "error_code": "RSYSLOG_SERVICE_FAILED",
    "error_message": "RSYSLOG service is not started properly, Please verify the service with status command."
  },
  "checks": {
    "services": [
      {
        "name": "nsfs",
        "service_status": "active",
        "pid": "1204",
        "error_type": "PERSISTENCE"
      },
      {
        "name": "rsyslog",
        "service_status": "inactive",
        "pid": "0",
        "error_type": "PERSISTENCE"
      }
    ],
    "endpoint": {
      "endpoint_state": {
        "response": {
          "response_code": 200,
          "response_message": "Endpoint running successfuly."
        },
        "total_fork_count": 1,
        "running_workers": [
          "1"
        ]
      },
      "error_type": "TEMPORARY"
    },
    "accounts_status": {
      "invalid_accounts": [
        {
          "name": "naveen",
          "storage_path": "/tmp/nsfs_root_invalid/",
          "code": "STORAGE_NOT_EXIST"
        }
      ],
      "valid_accounts": [
        {
          "name": "naveen",
          "storage_path": "/tmp/nsfs_root"
        }
      ],
      "error_type": "PERSISTENCE"
    },
    "buckets_status": {
      "invalid_buckets": [
        {
          "name": "bucket1.json",
          "config_path": "/etc/noobaa.conf.d/buckets/bucket1.json",
          "code": "INVALID_CONFIG"
        },
        {
          "name": "bucket3",
          "storage_path": "/tmp/nsfs_root/bucket3",
          "code": "STORAGE_NOT_EXIST"
        }
      ],
      "valid_buckets": [
        {
          "name": "bucket2",
          "storage_path": "/tmp/nsfs_root/bucket2"
        }
      ],
      "error_type": "PERSISTENCE"
    }
  }
} ```

- [X] Doc added/updated
- [X] Tests added

[ps20renar]

Is it possible to enhance these check also to add additional syslog services like syslogng instead rsyslog only?

[romayalon]

@naveenpaul1 see the last comment, can we do it?

[naveenpaul1]

@romayalon @ps20renar yes, can do a similar check depending on the user input.