This text is a summary from the Supporting Opinion of the Uncertainty Analysis Guidance SO 4
Quantitative and qualitative expression
An expression of uncertainty requires two components:
expression of the range of possible true answers to a question of interest, or a range of possible true values for a quantity of interest, and
expression of the probabilities of the different answers or values.
A complete quantitative expression of uncertainty would specify all the answers or values that are considered possible and probabilities for them all.
Partial quantitative expression provides only partial information on the probabilities and in some cases partial information on the possibilities (specifying a selection of possible answers or values).
Partial quantitative expression requires less information or judgements but may be sufficient for decision-making in some assessments, whereas other cases may require fuller quantitative expression.
Individual values: Uncertainty partially quantified by specifying some possible values, without specifying what other values are possible or setting upper or lower limits.
Bound: Uncertainty partially quantified by specifying either an upper limit or a lower limit on a quantitative scale, but not both.
Range: Uncertainty partially quantified by specifying both a lower and upper limit on a quantitative scale, without expressing the probabilities of different values within the limits.
Probability: Uncertainty about a binary outcome (including the answer to a yes/no question) fully quantified by specifying the probability or approximate probability of both possible outcomes.
Probability bound: Uncertainty about a non-variable quantity partially quantified by specifying a bound or range with an accompanying probability or approximate probability. A probability interval, two-sided or one-sided, is according to this definition a probability bound.
Distribution: Uncertainty about a non-variable quantity fully quantified by specifying the probability of all possible values on a quantitative scale.
They may rank the magnitudes of different uncertainties, and are sometimes given numeric labels, but they do not quantify the magnitudes of the uncertainties nor their impact on an assessment conclusion.
Descriptive expression: Uncertainty described in narrative text or characterised using verbal terms without any quantitative definition.
Ordinal scale: Uncertainty described by ordered categories, where the magnitude of the difference between categories is not quantified.
The role of quantitative expressions of uncertainty
The Codex Working Principles on Risk Analysis (Codex, 2016) state that:
Expression of uncertainty or variability in risk estimates may be qualitative or quantitative, but should be quantified to the extent that is scientifically achievable.
A similar statement is included in EFSA’s (2009) guidance on transparency. Having considered the advantages of quantitative expression, and addressed the concerns (presented below), the Scientific Committee concludes that assessors should express in quantitative terms the combined effect of as many as possible of the identified sources of uncertainty, while recognising that how this is reported must be compatible with the requirements of decision-makers and legislation.
Any sources of uncertainty that assessors are unable to include in their quantitative expression, for whatever reason, must be documented qualitatively and reported alongside it, because they will have significant implications for decision-making.
Together, the quantified uncertainty and the description of unquantified uncertainties provide the overall characterisation of uncertainty, and express it as unambiguously as is possible.
The phrase ‘scientifically achievable’ requires careful interpretation. It does not mean that uncertainties should be quantified using the most sophisticated scientific methods available (e.g. a fully probabilistic analysis); this would be inefficient in cases where simpler methods of quantification would provide sufficient information on uncertainty for decision-making. Rather, scientifically achievable should be interpreted as referring to including as many as possible of the identified sources of uncertainty within the quantitative assessment of overall uncertainty, and omitting only those which the assessors are unable to quantify.
This recommended approach is thus consistent with the requirement of the FAO/WHO Codex Working Principles for Risk Analysis and the EFSA Guidance on Transparency from 2010, which state that uncertainty be ‘quantified to the extent that is scientifically achievable’.
- ‘scientifically achievable’ should be interpreted as referring to including as many as possible of the identified sources of uncertainty within the quantitative assessment of overall uncertainty, and omitting only those which the assessors are unable to quantify.
- ‘scientifically achievable’ does not mean that uncertainties should be quantified using the most sophisticated scientific methods available.
It is important to note that overall uncertainty cannot and does not include any information about unknown unknowns, i.e. uncertainties not known to the assessors. These are always potentially present, but cannot be included in assessment, as the assessors are unaware of them. Furthermore, it must be remembered that the characterisation of uncertainty is conditional on the assessors who provide it, and on the evidence, time and resources available to them. These things should be understood by decision makers and taken into account by them when interpreting and using the assessment conclusions.
The recommended approach does not imply a requirement to quantify ‘unknown unknowns’ or ignorance. These type of sources of uncertainty are always potentially present, but cannot be included in assessment, as the assessors are unaware of them
The principal reasons for preferring quantitative expressions of uncertainty
Qualitative expressions are ambiguous.
Decision-making often depends on quantitative comparisons, for example, whether a risk exceeds some acceptable level, or whether benefits outweigh costs.
If assessors provide only a single answer or estimate and a qualitative expression of the uncertainty, decision-makers will have to make their own quantitative interpretation of how different the real answer or value might be. This judgement is better made by assessors, since they are better placed to understand the sources of uncertainty affecting the assessment and judge their effect on its conclusion.
Qualitative expressions often imply, or may be interpreted as implying, judgements about the implications of uncertainty for decision-making, which are outside the remit of EFSA.
Assessors may assess uncertainty differently yet agree on a single qualitative expression, because they interpret it differently.
Expressing uncertainties in terms of their quantitative impact on the assessment conclusion will reveal differences of opinion between experts working together on an assessment, enabling a more rigorous discussion and hence improving the quality of the final conclusion.
It has been demonstrated that people often perform poorly at judging combinations of probabilities. This implies they may perform poorly at judging how multiple uncertainties in an assessment combine. It may therefore be more reliable to divide the uncertainty analysis into parts and quantify uncertainty separately for those parts containing important sources of uncertainty, so that they can be combined by calculation.
Quantifying uncertainty enables decision-makers to weigh the probabilities of different consequences against other relevant considerations.
The role of qualitative expressions of uncertainty
Qualitative methods in uncertainty analysis are specifically recommended for the following purposes:
As a simple approach for prioritising uncertainties.
At intermediate points in an uncertainty analysis, to characterise individual sources of uncertainty qualitatively, as an aid to quantifying their combined impact by probability judgement. This may be useful either for individual parts of the uncertainty analysis, or as a preliminary step when characterising the overall uncertainty of the conclusion.
When quantifying uncertainty by expert judgement, and when communicating the results of that, it may in some cases be helpful to use an approximate probability scale with accompanying qualitative descriptors.
At the end of uncertainty analysis, for describing uncertainties that the assessors are unable to include in their quantitative evaluation.
When reporting the assessment, for expressing the assessment conclusion in qualitative terms when this is required by decision-makers or legislation.