proportionality.qmd

---
format:
  html:
    css: aqua_style.css
    html-table-processing: none
---

::: {.callout-important}
This version of the AQuA book is a preliminary ALPHA draft.  It is still in development, and we are still working to ensure that it meets user needs. 

The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. 
:::

# Proportionality

All analysis shall be assured, and the assurance should be proportionate to the potential impact, size and complexity of the analysis. The level of assurance should be guided by a structured assessment of the business risks. 

The assurer and the analyst shall be independent. The degree of separation depends on many factors including the importance of the output, and the size and complexity of the analysis. This does not mean that the analyst should not undertake assurance, rather that there shall also be some formal independent assurance. 

## Introduction

Think about and deliver appropriate (proportionate) levels of assurance for your analysis.  There is a need to be confident in analysis delivered, but there is no point spending months assuring simple analysis that will inform a decision that will make minimal impact.  

Table 3-1 provides a list of key factors that should be considered when determining what level of assurance is appropriate. 

Further detail and considerations may be found on the Data Quality Hub's [Quality Questions and Red Flags](https://dataqualityhub.github.io/resources-for-quality-analysis-external/) page.


| Factor  | Comments  |
|:--------|:----------|
| Business criticality | Different issues will impact on business criticality such as financial, legal, operational, political and reputational impacts. |
| Relevance of the analysis to the decision making process | When analysis forms only one component of a broad evidence base, less assurance is required for that specific analysis than if the decision is heavily dependent on the analysis alone. Significant assurance is still likely to be required for the evidence base.  |
| Type and complexity of analysis | Highly complex analysis requires more effort to assure. The nature of that analysis may also require the engagement of appropriate subject matter experts. |
| Novelty of approach | A previously untried method requires more assurance. Confidence will grow as the technique is repeatedly tested. |
| Reusing or repurposing existing work | Reusing work that was carried out previously may require validation and verification to confirm that original approach - method, assumptions data etc. are still appropriate for the new requirement.|
| Level of precision required in outputs | Lower precision analysis often uses simplified assumption, models and data. The assurance approach is the same but will take less time than more precise analysis.|
| Amount of resource available for the analysis and assurance | The value for money of any additional assurance must be balanced alongside the benefits and risk appetite that exists. Analysis that is used for many purposes (e.g. population projections) may require greater levels of QA than might be suggested by any individual decision they support.  |
| Longevity of the analysis | Ongoing analysis will require robust change control and regular review. |
| Public impact | Analysis which will have a significant impact on the public may require more assurance. |
| Repeat runs for the same analysis | The focus for repeat analysis will be on version control and assurance of data and parameters for each run. |

: Table 3-1 - Factors for determining appropriate assurance {.striped}


Figure 3-1 shows some assurance techniques that might be considered for different levels of analysis complexity and business risk. The key message is the need for more assurance interventions increases with the complexity of, and the business risk associated with analysis.

![Figure 3-1 - Types of quality assurance - darker shades indicate the need for need for extra assurance activities and greater separation between the analyst and the assurer. The contours indicate the groups of activities that may be carried out for a particular level of business risk/complexity.](Figure 3-1 Types of Assurance Alternative 3.jpg){fig-alt="Figure 3-1 is diagram showing the relationship between risk, complexity and the requirement for assurance activity. There are two axes on the diagram. The X axis goes from simple analysis on the left to highly complex analysis on the right. The y axis goes from low business risk at the bottom to high business risk at the top. As risk and complexity increase there is a need for extra assurance activities as well as a higher degree of separation between the analyst and the assurer. For complex, high risk analysis this might include external peer review or audit. On the diagram, the increasing level of risk as we move from bottom left to top right is represented by darker shades."}

The interventions in Figure 3-1 must not be viewed in isolation. The interventions should build on each other, for example some complex and risky analysis that would benefit from an external review should also use interventions closer to the axes, for example version control and analyst led testing. 

The total elimination of risk will never be achievable, so a balance needs to be found that reduces the overall business risk to an acceptable level. The diagram indicates a few practical assurance techniques. In practice there are many different techniques that need to be considered and implemented as appropriate.

Many of these interventions are mentioned elsewhere in the AQUA Book, and are not repeated here.

## Structured assessment of business risk and complexity

To guide what assurance is needed it is necessary to take a structured approach when reviewing business risks. Business risk should be viewed as the combination of the potential impact of analysis errors, and the likelihood of errors occurring. In situations where the potential business impact is high, it is more important that the likelihood of errors is reduced. 

This can be visualised by considering the situation as a risk matrix, illustrated in Table 3-2. The impact of the analysis will usually be beyond the control of the analyst to change, so there will be few options to move an assessment down the table. However, there will usually be treatments (or mitigations), involving additional assurance measures, that will allow the assessed business risk to move to the left.


<table class="riskTable">
<caption>
Table 3-2 - Example of a risk matrix
</caption>
<tr>
  <th></th>
  <th></th>
  <th colspan = "5">Likelihood of errors occurring</th>
<tr>
  <th rowspan = "7">Impact of errors occurring</th>
  <td></td>
  <td></td>
  <th> Highly unlikely </th>
  <th> Unlikely </th>
  <th> Realistic possibility </th>
  <th> Likely or probably </th>
  <th> Highly likely </th>
</tr>
<tr>
  <td> </td>
  <th> Critical </th>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
  <td class="high_risk"> High </td>
  <td class="high_risk"> High </td>
  <td class="high_risk"> High </td>
</tr>
<tr>
 <td> </td>
  <th> Severe </th>
    <td class="low_risk"> Low </td>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
  <td class="high_risk"> High </td>
  <td class="high_risk"> High </td>
 </tr>
 <tr>
  <td></td>
  <th> Major </th>
    <td class="low_risk"> Low </td>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
  <td class="high_risk"> High </td>
</tr>
<tr>
  <td></td>
  <th> Moderate </th>
  <td class="very_low_risk"> Very Low </td>
  <td class="low_risk"> Low </td>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
  <td class="medium_risk"> Medium </td>
</tr>
<tr>
  <td></td>
  <th> Minor </th>
  <td class="very_low_risk"> Very Low </td>
  <td class="very_low_risk"> Very Low </td>
  <td class="low_risk"> Low </td>
  <td class="low_risk"> Low </td>
  <td class="medium_risk"> Medium </td>
</tr>
</table>

Table 3-3 shows appropriate responses to a risk assessment. Where business risk is high, appropriate treatment(s) must be considered to reduce the probability of errors occurring. The choice of treatment will depend on the mitigations already in place and on the complexity of the analysis (see Figure 3-1). 

For a situation where simple analysis is being employed, a review by an appropriate expert may be sufficient as the additional mitigation. However, for complex analysis that is already employing a wide range internal assurance measures, options like external peer review may be necessary. 

In cases where there is a need for analysis, but there are also significant time and/or resource constraints, it may not be possible to do as much assurance as usual. In these situations, the focus should be on areas of greatest risk. These risks and limitations must also be communicated, along with appropriate caveats.

| Assessed risk    | Mitigations to consider                              |
|------------------|------------------------------------------------------|
| High     | The risk should not be tolerated. New assurance measures must be considered to treat (mitigate) the likelihood of errors occurring.  If treatment isn't an option, consideration must be be given to terminating or transferring the (analysis) risk. If it remains necessary to tolerate the risk the SRO needs to fully understand the risk.  |
| Medium   | The risk should not be tolerated without SRO agreement. New assurance measures should be put in place to treat (mitigate) the likelihood of errors occurring. Continue with planned or existing mitigations. |
| Low      | The risk can be tolerated. Existing or planned mitigations should be continued, and new treatments may be considered.  |
| Very Low | The risk can be tolerated. Existing/planned mitigations measures should be continued.  |

: Table 3-3 - Responses to risk assessment levels {.striped}

For further guidance on risk management, refer to the [Orange book](https://www.gov.uk/government/publications/orange-book) which covers risk management principles and risk control frameworks.

## Externally commissioned work

Proportionate assurance of externally commissioned work is just as important as for internally produced analysis. For the commissioner, to use the work they should be fully informed of the business risk associated with it. This should be provided by an appropriate mix of documented risk assessments provided as part of the work, and by joint risk assessments planned throughout the life of the project. For commissioned work the options for mitigation will be similar to those for internal analysis.

The difference will be in ensuring the assessment of risks and the applied mitigations are fully understood by the commissioner.

## Artificial intelligence and business risk

Increasingly analysis may be underpinned by Artificial Intelligence (AI). With AI-informed analysis the need to understand business risk remains, and the same structured approach to assessing business risk should be taken. The challenges in providing this assessment will be in ensuring the transparency of the analysis, availability of a suitable mix of experts, and developing understanding of what mitigations are possible.