[Backport release/1.7] CRAYSAT-1937: Add the first functional test fo…

…r SAT (#654)

CRAYSAT-1937: Add the first functional test for SAT

Add the first functional test for the System Admin Toolkit (SAT).  This
is a proof of concept for future tests for SAT.

This test simply executes the `sat --version` command on the NCN and
verifies that the output matches the value in the file used to set the
version.

SAT tests are implemented in Python using the `unittest` standard
library package.  This provides an easy way to structure tests, add set
up and tear down logic for each test, and run the tests from a script.
The Python code for the SAT tests is all under a `sat_functional`
subpackage, and individual subpackages and modules beneath that
subpackage will implement tests for specific SAT functionality. Tests
will be organized by the `sat` command they are testing.

The `__main__.py` in the `sat_testing` module simply passes along the
arguments, which should be a list of relative test paths, to the
`unittest.TextTestRunner`, which then runs the tests.

SAT tests are added in Gossfiles. Currently a single test file for SAT
tests, `goss-sat-version.yaml`, is created. A test suite is created in
the goss file `ncn-sat-functional-tests.yaml`. This test suite is
limited to run on only `ncn-m001`. It will eventually include all SAT
functional tests.

Test Description:
Downloaded the RPM built on Jenkins to drax. Extracted the RPM using
`rpm2cpio $RPM_FILE | cpio -idvmu`. Modified the path to the Python
virtual environment in the `test_sat_functional` script in the `bin`
directory of the virtualenv to match the extraction path, set
`GOSS_BASE` to the extracted Goss base directory, and then ran the Goss
test file and test suite using `goss -g $GOSS_FILE validate`.

(cherry picked from commit 539a1ea)

Co-authored-by: Ryan Haasken <ryan.haasken@hpe.com>

goss-testing/suites/ncn-sat-functional-tests.yaml

@@ -0,0 +1,41 @@
+#
+# MIT License
+#
+# (C) Copyright 2024 Hewlett Packard Enterprise Development LP
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+
+# This is a suite of functional tests for validating the functionality of the
+# System Admin Toolkit (SAT). Tests are split up across multiple gossfiles,
+# which are then included by this suite.
+#
+# Most `sat` commands are just making requests to the CSM APIs, so it would be
+# redundant to run each test on every node. Instead, we just run them on ncn-m001.
+
+# Use hostname from variable file, if set.
+# Otherwise use hostname from environment variable (or blank, if that is also not set)
+{{ $env_hostname := getEnv "HOSTNAME" }}
+{{ $vars_hostname := get .Vars "this_node_name" }}
+{{ $this_node_name := default $env_hostname $vars_hostname }}
+gossfile:
+  {{if eq $this_node_name "ncn-m001"}}
+  ../tests/goss-sat-version.yaml: {}
+  {{ end }}
+

goss-testing/tests/ncn/goss-sat-version.yaml

@@ -0,0 +1,41 @@
+#
+# MIT License
+#
+# (C) Copyright 2024 Hewlett Packard Enterprise Development LP
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+
+{{ $scripts := .Env.GOSS_BASE | printf "%s/scripts" }}
+{{ $logrun := $scripts | printf "%s/log_run.sh" }}
+{{ $sat_test := $scripts | printf "%s/python/sat_functional" }}
+command:
+    {{ $test_label := "sat_version" }}
+    {{$test_label}}:
+        title: Test the 'sat --version' command
+        meta:
+            desc: Tests that the 'sat --version' command returns the expected version
+            sev: 0
+        exec: |-
+            "{{$logrun}}" -l "{{$test_label}}" \
+                "{{$sat_test}}" "version.test_version"
+        exit-status: 0
+        stderr:
+        - OK
+        timeout: 5000 # timeout in milliseconds

src/csm_testing/tests/sat_functional/__main__.py

@@ -0,0 +1,97 @@
+#
+# MIT License
+#
+# (C) Copyright 2024 Hewlett Packard Enterprise Development LP
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+"""
+Functional tests for the System Admin Toolkit (SAT).
+
+This script is a thin wrapper around the unittest module. This script is used
+instead of directly using `python -m unittest` because the build scripts in this
+repository install this script in the Python virtual environment in which the
+csm_testing module is installed and then create a symlink to the script. This
+script will thus be able to automatically find the unittest test cases defined
+in the directories within this directory.
+
+This script also prepends the arguments it's given with the parent package name
+in order to shorten the calls made in the Goss test files.
+
+For example:
+
+  `python -m unittest csm_testing.tests.sat_functional.version.TestVersion.test_version`
+
+becomes:
+
+  `sat_functional version.TestVersion.test_version`
+"""
+import argparse
+import sys
+import unittest
+
+
+def create_parser() -> argparse.ArgumentParser:
+    """Create a parser that accepts a list of test identifiers to run.
+
+    Returns:
+        The parser object.
+    """
+    parser = argparse.ArgumentParser(
+        description='Run the specified functional tests for the System Admin Toolkit (SAT).')
+    parser.add_argument(
+        'tests', nargs='+',
+        help='The names of the tests to run. These should be the names of the '
+             'test modules, test classes, or test methods to run, relative to the '
+             'csm_testing.tests.sat_functional package.'
+    )
+    return parser
+
+
+def main() -> None:
+    """Execute the specified unit tests and exit with the appropriate status code.
+
+    Raises:
+        SystemExit: The exit code is set to 0 if all tests pass, 1 otherwise.
+    """
+    parser = create_parser()
+    args = parser.parse_args()
+
+    suite = unittest.TestSuite()
+
+    for test in args.tests:
+        # Prefix each test name with the path to the sat_functional package
+        tests = unittest.TestLoader().loadTestsFromName(
+            f'csm_testing.tests.sat_functional.{test}'
+        )
+        suite.addTests(tests)
+
+    # Run the test suite with a verbosity of 2 so it will print the name of each
+    # test being executed.
+    runner = unittest.TextTestRunner(verbosity=2)
+    result = runner.run(suite)
+
+    if result.wasSuccessful():
+        sys.exit(0)
+    else:
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()

src/csm_testing/tests/sat_functional/version/test_version.py

@@ -0,0 +1,69 @@
+#
+# MIT License
+#
+# (C) Copyright 2024 Hewlett Packard Enterprise Development LP
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+# OTHER DEALINGS IN THE SOFTWARE.
+#
+"""
+Simple test for the functionality of the `sat --version` command. This should
+just print the semantic version of the `sat` command.
+"""
+import shlex
+import subprocess
+from typing import Optional
+import unittest
+
+SAT_VERSION_FILE = '/opt/cray/etc/sat/version'
+
+
+def get_version_file_contents() -> Optional[str]:
+    """Get the contents of the file /opt/cray/etc/sat/version if it exists
+
+    Returns:
+        The version string from the file, or None if the file does not exist
+    """
+    try:
+        with open(SAT_VERSION_FILE, 'r', encoding='utf-8') as version_file:
+            return version_file.read().strip()
+    except FileNotFoundError:
+        return None
+
+
+class TestVersion(unittest.TestCase):
+    """Test the `sat --version` command."""
+
+    def test_version_command(self):
+        """Test that `sat --version` returns the semantic version."""
+        command = 'sat --version'
+
+        # Use stdout and stderr instead of capture_output=True for Python 3.6 compatibility
+        proc = subprocess.run(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                              check=True)
+        version_output = proc.stdout.decode().strip()
+
+        expected_version = get_version_file_contents()
+        if expected_version:
+            # The output is of the form "sat x.y.z"
+            self.assertEqual(f'sat {expected_version}', version_output)
+        else:
+            # If the file does not exist, the `sat` podman wrapper script uses a csm-latest
+            # tag from the registry. We don't know what the version is, but we can check the
+            # format of the output at least.
+            self.assertRegex(version_output, r'^sat \d+\.\d+\.\d+$')