Fail on warn should exit with different rc

[nirroz93]

Hi!

Currently, running conftest test --fail-on-warn return 1 when there is a warning or an error.

Different return code when there are no errors (just warn) will be nice and, for me, it seems like the "normal" behaviour (meaning: when I was surprised it didn't :-) )

[garethr]

I like the idea. What exit codes would you propose?

[nirroz93]

IMHO the most intuitive rc 0 OK 1 WARN (with no errors) 2 ERROR (and error+warn), or leave some place for other future levels, but this may cause backward compatibility if someone is using the current return code (and I don't see a reason for using ==1 and not !=0, but if you think there is so 1 error 2 warn).

[jpreese]

@nirroz93 I'd be curious about your use case. I'm not necessarily opposed to providing more information to the user about the results, but admittedly I do like the current behavior.

Without --fail-on-warn in CI you'll most likely want WARN to continue to process, and on your local computer, you'll see the warnings in the output window.

With --fail-on-warn in CI your warnings will fail, which should be expected given the argument name.

[nirroz93]

@jpreese
I want to do other action items for warns and error. For example, on warn I want to put the Jenkins job as "unstable"+mail, but for other deny (from other checks) I want to fail the job.

In the current state I need to catch the output and parse it (or run conftest twice).

[jpreese]

When possible, I like to mimic the behavior of OPA. I feel the experience shouldn't diverge too much, because contest just builds onto all the niceties that OPA provides.

That said, in their eval command they have 0 = success, 1 = undefined, 2 = error. For our purposes, we could consider "undefined" as no errors exist, but a warning was found. (open-policy-agent/opa#981)

I would agree with you @nirroz93 that the best course of action would most likely be:

(without --fail-on-warn)
0 -> warnings. no errors.
1 -> error occurred

(with --fail-on-warn)
0 -> no warnings or errors
1 -> no error occurred. has at least 1 warning.
2 -> an error occurred. (could also have warnings)

I would suspect most CI checks for != 0, so this has low impact to breaking current implementations, gives you the data you need, and is at least somewhat in line with OPA.

[jpreese]

Resolved via #378