Add task to suite to alert user to CRITICAL issues from MIP Convert tasks and report them

[matthew-mizielinski]

Migrated Jira Issue CDDSO-165
Jira creation date 06/07/2022 10:30

It would be possible to add a task to the CDDS suites before QC to flag up issues in a very obvious way. At present we rely on the user to check by grepping, which some don’t do.

I imagine a couple of changes to implement this;

• The critical_issues.log file should be created on a per stream basis (change to the MIP Convert wrapper
• A task should be added to the suite before the qc task that
  • presents a summary of the critical errors (variable, error, number of occurrences out of number of job steps
  • fails if there are critical errors

If this “critical_check” task fails it is up to the user to decide if they care about the variables affected. They can then mark the task as succeeded to proceed or stop and investigate further.