-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add actionable failure messaging to evaluate
#264
Comments
discussed this with @earmenda , and @ThomasLaPiana need your eyes on the below proposal. The purpose of this issue is to provide actionable failure messaging about why the evaluation failed, however, because the evaluation failure is due to a violating combination of privacy attributes (uses, subjects, category, qualifier) at run time, it doesn't make sense to just include any less than 4 of the attributes in the failure messaging. So we tried putting all 4 attributes into a sentence in the "details" value in the json object, and it looks really long and unreadable. As such, wanting to update the evaluate failure messaging to be something to this tune (gisty approximation, don't use this for actual requirements here 😉 )
Aaaand maybe if the CLI user can include a flag to only make this for a human user using the CLI to be able to read without the json bloat (i know aws cli uses
sidebar: it's worth mentioning that in the future, when/if we can support storing or comparing state of the past resource commits, we would be able to definitively point at a singular violating privacy attribute (ie: data category) within a declaration. For example, last week, a system was not using I don't want to blow up the scope for this issue, but wanted to make sure we just get a few eyes on this proposal before proceeding. |
Rather than include a Separately, if we so desire, we can include a "fail fast" option, to only display the first violation for a given failed evaluation. It's hard for me to say off-hand if this would actually speed up the |
I think people using the CLI can read JSON, so I don't see the need for another CLI flag. As long as the UI version is human-readable I think we're fine. @iamkelllly I love the idea of having the violating attributes there! Overall my suggestions here are:
|
All of the rules and entities are fetched before doing the evaluation, so wouldn't make a difference from a computational standpoint. I don't really see the need for this flag right now, with that in mind |
While the failure messaging for
evaluate
is descriptive with the policy declaration, system, and rule that's being violated.Current state:
Providing actionable/meaningful error and failure messaging helps an implementer clearly know next steps to take whether they
This issue is to add additional logging to the details node of the failure message here to understand what exactly was failing, for example:
There are likely edge cases where this format might not work, but would like to explore the edges of this with the demo/quickstart repo.
The text was updated successfully, but these errors were encountered: