Conditional Asserts Enhancement

[stevelipinski]

Curious if it would be difficult to add conditional asserts...?

I was investigating using the unittest plugin to come up with a set of tests that validate some helm chart "rules" to be imposed on multiple charts. Thinking along the lines of a compliance audit test suite that ensures all the charts we develop abide by certain rules or best-practices.

To accomplish this, the asserts would need to be conditional, maybe using another assert as the condition.

Just as an example, if I wanted to check that all container images referenced used an internal registry (foo.com), I could create the following asserts:

tests:
- it: tests statefulsets use local foo.com registry
  asserts:
  - isNotNull:
      path: spec.template.spec.containers[0].image
      optional: true    # indicating it fails silently if this fails and/or the path does not exist
  - matchRegex:
      path: spec.template.spec.containers[0].image
      pattern: foo.com/*
      conditional: 0    # the idx of the assert to condition this assert on. 
                        # If assert[0] is successful, then this assert applies

That is a contrived example, and it could be implemented in a handful of different ways - like instead of referencing the other assert by index, an assert could set some "variables", similar to ansible's register:.

I wish I was more familiar with Go as I could try to contribute to something like this...

[quintush]

Hello @stevelipinski,

Thanks for the feedback.
I will look into it.

I noticed there are some options.

Suggested solution:
I will try to start with an output variabele, which can be used in other assertions.
If an output variabele is set, the assertion itself will not fail.It will only put the outcome in the output variabele.

As a later enhancement the test can send a warning if a output variabele is set, but not used in other assertions.

Do you think it would work?

Greetings
@quintush

[stevelipinski]

Yeah, I think an output variable would work just as well, and then would there be a when added to the asserts? Essentially working like ansible, right?:

  - asserts:
    - isNotNull:
        path: spec.template.spec.containers[0].image
        register: container0_image
    - matchRegex:
        path: spec.template.spec.containers[0].image
        pattern: foo.com/*
        when: container0_image == true

[avaakash]

I had a similar use-case where I would want certain test cases to throw warnings and their failure would not result in the overall failure of the test suite. This will help add some test cases which are optional to pass.

[stevelipinski]

@avaakash - FYI, I think mainline work on helm-unittest has moved back to helm-unittest/helm-unittest. You may want to comment there.
I see two nearly equivalent issues already, so I'm not going to create another copy:
helm-unittest#251
helm-unittest#203

[avaakash]

Sure thanks @stevelipinski