At this stage you should have considered where the uncertainties lie. You will now need to consider whether it is possible and appropriate to quantify this uncertainty. In a highly uncertain context, detailed assessment of data-driven uncertainty may not be worthwhile. This is an analytical judgement to be made in light of the full consideration of sources of uncertainty.
This section presents some approaches at a high level then in more detail for quantifying and measuring the uncertainty in your analysis.
We have highlighted ways to think about the uncertainty coming from specific sources. We now bring this together into approaches that can be applied to understand the size and distribution of these uncertainties and how you might include this information in your analysis. In most cases, the approach to uncertainty quantification is limited by the data and time available to you, though you should also consider where the major sources of uncertainty are in your analysis and pay most attention to these.
In addition, the following approaches can be useful in understanding the size and distribution of the uncertainties in your analytical inputs.
Consider discrete and continuous distributions…
A probability distribution describes the probability of occurrences of different outcomes. Generally, there are two types of probability distribution; discrete distributions and continuous distributions.
Consider whether you have information about the underlying distribution of the parameter. Often data from other sources will be provided with confidence intervals (or standard errors, etc) that can be used to quantify uncertainty. Where such information is not provided, you may be able to approximate these with knowledge of the sample size and design.
data sources may include confidence intervals…
Distributions can also be created using what you know about error from previous models. Consider the performance of previous forecasts against outturn results. The distribution of previous errors can provide the uncertainty distribution for the current forecast. Additionally, you might be able to use one source to validate another or to help estimate the uncertainty in the original source.
or analysis of past model performance to create distributions…
If no quantitative data on the underlying population is available, you may be able to elicit this information from experts . For example, using the Delphi Method, a structured facilitation method, which relies on a panel of (ideally external) experts to build a combined judgement, for example, about a particular number. This method can be used to ask a panel of experts to estimate the range of uncertainty and use the aggregated responses to produce a distribution. Consider tools to overcome biases, e.g. Brier score. These measure the mean squared difference between the predicted and actual outcomes. The lower the Brier score is for a set of predictions, the better the predictions are calibrated, which has been shown to improve judgement
Without quantitative data, use expert judgement
A range is similar to a probability distribution, in that it considers the possible outcomes but does not consider the probability of each outcome occurring. If there are data or resource limitations a range can be a simple way to illustrate the uncertainty in a parameter.
Ranges can work in absence of distributions…
Historical data can be used to quantify a range. Consider how the parameter has changed over a suitable time period. The maximum and minimum values could provide a sensible range. When using historical data be aware that you will only be able to assess ‘business as usual’ uncertainty. If there are future shocks to the system this may fall outside your historic range.
Historical data can inform the range,
For parameters that have been the subject of academic studies a literature review can be used to create a range. Consider why different studies may result in different outcomes, and which studies are the most suitable for what you are trying to measure.
You can also use a literature review…
If no quantitative data is available, consider whether there are relevant policy/operational constraints that will limit your range. Judgement from experts can be also be used to create sensible ranges.
Expert steers can inform the range
In some situations, it is not possible to create a probability distribution or a range. In such cases, make a qualitative assessment of uncertainty. This is still useful to analysts and customers to consider the magnitude of uncertainty.
You can make qualitative assessments yourself, and by using expert judgement. A simple approach is to RAG rate the likelihood and impact of uncertainty in your parameters. This qualitative assessment should be considered when thinking about the analytical results. If data is categorised as highly uncertain and having a large impact on results, then final outputs will be subject to large uncertainty. Care in presentation is needed when using these more qualitative methods, however, as it’s potentially easier for decision makers to misunderstand the relevance of qualitative assessments of uncertainty for their decision compared to quantitative assessments. It is particularly important to explain the impact of uncertainty on the analytical result and the decision to be made when using these types of qualitative assessment.
RAG rate your parameters
A further option to support decision makers is to use break-even analysis. This is useful to understand at which point a saving becomes a cost or possibly at which point you would take a different decision.
Break-even analysis can help re-frame the question
Some decision makers will be used to seeing a range around a central estimate. Break-even analysis works backwards – if we were to break-even what would the input be? This could help bring the policy alive and help assumption owners to really consider how realistic the assumption is.
Break-even analysis helps people understand how much the input has to change before you reach a break-even point so they can consider the probability of this occurring.
Now that we have explored how to assess uncertainties in individual analytical inputs, we turn to commonly used techniques for analysing overall uncertainty in analytical outputs, moving from approaches to use when we have a good numerical understanding of input uncertainties, though to approaches which can be used when we know less about how input uncertainties are quantified.
If all significant sources of uncertainty can be quantified, along with the correlations between them, then probabilistic methods can provide a picture of the range of possible outcomes and the likelihood of each.
This should be repeated until the key outputs are stable and
reproducible at the level of rounding that will be used when
communicating the results
If there are a small number of uncertainty sources, then it may be possible to combine them mathematically, which can be quicker than using probabilistic methods
Example:
Any situation where you’re combining two
probability distributions using simple mathematical operations
(addition, subtraction, multiplication)
If a forecast has been produced repeatedly over many years, it may be possible to use the accuracy of all the previous forecasts to estimate the uncertainty in the new forecast
Outline:
Start by looking at the variance of all of
the previous forecasts after one time period, and create a distribution
of these variances. This distribution can be used to estimate the
uncertainty of the new forecast after one time period.
Repeat
for the variance at each subsequent time period to produce e.g. a fan
forecast (smoothing if needed).
Example:
MoJ prison population forecasting
If one source of uncertainty has a much greater impact than all the others, then the uncertainty due to this one factor might be a reasonable proxy for the overall uncertainty
Outline:
Consider all the sources of uncertainty
affecting the analysis and quantify the impact that each has on the
overall uncertainty (even if this is approximate or subjective – see
section on ‘Defining and Identifying Uncertainty’). If one source has
substantially more impact that the others, then simply looking at the
uncertainty in this one factor might be a pragmatic approximation for
the overall uncertainty measure, ignoring other sources.
This
is not a robust way of estimating overall uncertainty, as it ignores all
but one source. It should only be used if previous techniques are
inappropriate, and there is genuinely one source of uncertainty that has
an impact that is an order of magnitude larger than the others. However,
when time is tight this may be a pragmatic way of estimating
uncertainty.
Example:
The Accuracy Tracking Tool (link to DfE
Accuracy tracking tool to come) can be used to estimate the residual
uncertainty once the dominant uncertainty has been modelled. This tool
assesses the accuracy of different forecast elements and allows you to
see the percentage which both the dominant and residual uncertainty
contribute to the total error.
If there are many sources of uncertainty with complex correlations between them and the likelihood of them occurring is unknown, then it may be more sensible to use a set of scenarios to illustrate the range of plausible outcomes.
Example:
Forecasting where a range of policy options
are being considered, particularly where the likelihood of an event
occurring is unknown, for example early analysis on Exiting the EU
scenarios
If the previous methods are unfeasible then you could make a subjective estimate of the overall uncertainty using expert elicitation techniques.
Outline:
If the previous methods are unfeasible, then
adding a subjective estimate of the overall of uncertainty is an option.
Where there is too little information or time to do a quantified
analysis, it may be better to provide a judgement on the uncertainty
than nothing at all.
Ideally this should be a group decision,
using formal expert elicitation methods (e.g. Delphi) to avoid
group-think and arrive at a consensus, agreement, or an average
(depending on the technique used). If formal expert elicitation isn’t
possible, then the judgement of an individual is better than nothing,
though less rigorous.
However the uncertainty is estimated,
make sure that it is clear that it is a subjective opinion rather than
results of analysis to prevent it being misused.
Example:
Providing context around a high priority
figure that needs to be submitted quickly.
Analysis based on a
data source of unknown reliability
Analysis where the expected
range of results would lead to the same outcome
It is best practice to test the outputs of the uncertainty analysis before using/presenting
After modelling uncertainty, you should always test the outputs of the analysis before sharing the results. This minimises the risk of errors in your analysis and helps you to understand the detailed outputs fully, including the level of the extreme or the most likely values.
Uncertainty analysis may produce ‘extreme outcomes’, so that implausible results or scenarios are given. These can be identified through having a sensible checklist of validation rules (e.g. no negative values allowed) and aided with visualisation and filtering. These implausible outcomes could indicate an issue with the setup conditions of your analysis.
Unusual results in uncertainty analysis may also indicate a weakness in how you have used your chosen technique. For example, if using the Monte Carlo technique, there may be unknown correlation which hasn’t been accounted for, or you might have used an inappropriate distribution for a parameter.
Unusual results may indicate a weakness in the use of the technique
One element you may not be able to test in your analysis is the effect of unknown system shocks, such as a recession. It is not always useful or practical to investigate and analyse these types of epistemic or ontological uncertainty, but they should be accounted for in presenting your analysis, for example by highlighting the risks associated with the analysis and the decision it will inform.
Accounting for uncertainties that analysis cannot capture