[FEATURE REQUEST] Cleaner output when using Saltcheck

[pgporada]

Is your feature request related to a problem? Please describe.
#57327 has been triaged and is working well for me.

Describe the solution you'd like
I'd like the following features

• To specify that only failing tests be outputted
• Tests showed up "prettier" when using table output

Additional context
The output is pretty verbose and doesn't lend itself well to review. If I could show just the failing tests that would be an immediate improvement.

salt 'server' saltcheck.run_highstate_tests saltenv=dev 
server:
    |_
      ----------
      server.mytests:
          ----------
          program1_is_serving_metrics:
              ----------
              duration:
                  0.428
              status:
                  Pass
          program2_is_serving_metrics:
              ----------
              duration:
                  0.4296
              status:
                  Pass
          program3_is_serving_metrics:
              ----------
              duration:
                  0.3912
              status:
                  Pass
          program4_is_serving_metrics:
              ----------
              duration:
                  0.3872
              status:
                  Pass
          program5_is_serving_metrics:
              ----------
              duration:
                  0.4159
              status:
                  Pass
    |_
      ----------
      program.install:
          ----------
          verify_package_version:
              ----------
              duration:
                  5.7908
              status:
                  Pass
    |_
      ----------
      program.feature:
          ----------
    |_
      ----------
      cert.installer:
          ----------
    |_
      ----------
      firewall:
          ----------
    |_
      ----------
      ldap:
          ----------
    |_
      ----------
      yum_repo:
          ----------
    |_
      ----------
      metrics.collector:
          ----------
    |_
      ----------
      thing.thing5:
          ----------
    |_
      ----------
      some.thing:
          ----------
    |_
      ----------
      TEST RESULTS:
          ----------
          Execution Time:
              7.8427
          Failed:
              0
          Missing Tests:
              8
          Passed:
              6
          Skipped:
              0

Here's the same thing but with table output.

salt 'server' saltcheck.run_highstate_tests saltenv=dev --out=table
server:
----------
    ------------------------------------------------------
    |           server.mytests                             |
    ------------------------------------------------------
    |    {'program5_is_serving_metrics':    |
    |      {'status': 'Pass', 'duration': 0.4063},       |
    |     'program5_is_serving_metrics':     |
    | {'status': 'Fail: 1 is not equal to ', 'duration': |
    | 0.416}, 'program2_is_serving_metrics': |
    | {'status': 'Fail: 1 is not equal to ', 'duration': |
    |                      0.4083},                      |
    |     'program3_is_serving_metrics':     |
    | {'status': 'Fail: 1 is not equal to ', 'duration': |
    |                      0.4191},                      |
    |     'program4_is_serving_metrics':     |
    | {'status': 'Fail: 1 is not equal to ', 'duration': |
    |                      0.4027}}                      |
    ------------------------------------------------------
    |   {'verify_package_version': {'status': 'Pass',    |
    |                'duration': 5.8119}}                |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    |                         {}                         |
    ------------------------------------------------------
    | {'Execution Time': 7.8643, 'Passed': 2, 'Failed':  |
    |        4, 'Skipped': 0, 'Missing Tests': 8}        |
    ------------------------------------------------------

[garethgreenaway]

@mchugh19 FYI.

[mchugh19]

Rather than supporting every output case in the module, you could also achieve similar results by outputting json and filtering the results.

Given output

{
  "local": [
    {
      "os": {
        "test_public_html_present": {
          "assertion1": {
            "status": "Pass",
            "module.function [args]": "cmd.run [\"ls /etc/salt/master\"]",
            "saltcheck assertion": "IS NOT EMPTY /etc/salt/master"
          },
          "assertion2": {
            "status": "Pass",
            "module.function [args]": "cmd.run [\"ls /etc/salt/master\"]",
            "saltcheck assertion": "master IN /etc/salt/master"
          },
          "status": "Pass",
          "duration": 1.4072
        },
        "failing_test": {
          "assertion1": {
            "status": "Fail: /etc/salt/master is not empty"
          },
          "status": "Fail",
          "duration": 0.3072
        }
      }
    },
    {
      "TEST RESULTS": {
        "Execution Time": 1.7144,
        "Passed": 1,
        "Failed": 1,
        "Skipped": 0,
        "Missing Tests": 0
      }
    }
  ]
}

You can use jq to filter where status = Fail

salt-call --local saltcheck.run_state_tests sample_test --output json | jq '.local[] |  to_entries[] | .value  | with_entries(select(.value.status == "Fail"))' 2>/dev/null
{
  "failing_test": {
    "assertion1": {
      "status": "Fail: /etc/salt/master is not empty"
    },
    "status": "Fail",
    "duration": 0.3072
  }
}

If you are running saltcheck as part of a CI build, you could use a python analysis step rather than simple jq to determine build status.

[mchugh19]

This feature request should be fixed by #57824. It provides the functionality to run salt-call saltcheck.run_state_tests os.files,os only_fails=True to only display test failures.

[sagetherage]

Thank you @mchugh19 I am putting this into Magnesium as already "in progress" and will attempt to get it reviewed and merged soon.