Methods to quantify uncertainty

Expressing uncertainty using probability and alternatives to probability

For yes/no questions or binary quantities, uncertainty can be quantitatively expressed by assigning probabilities to the two possible answers. The probabilities must sum to 100%, determining the probability for the other answer. Uncertainty for a non-variable quantity can be fully quantified by specifying a probability distribution, indicating the probability of the true value falling within any given range. Partial quantification of uncertainty can be achieved by specifying a credible interval, a range of values of interest along with the probability of the true value lying within that range. Additional ranges and probabilities provide a more complete quantification. Probabilities and distributions can be derived from expert judgment, statistical analysis of data, or calculations involving other probabilities. Approximate probabilities, expressed as ranges, can be used to simplify the specification process. Probability bounds are special cases of credible intervals, allowing for approximate probabilities. Probability bounds are useful for combining uncertainties of multiple quantities in a deterministic model. Expert judgment and statistical analysis are both valid approaches for obtaining probabilities, with statistical analysis preferred when applicable. Combining statistical results with expert judgment is often recommended. Calculations based on models are discussed for combining uncertainties expressed using probabilities.

Quantifying uncertainty about a variable quantity is more challenging than quantifying uncertainty about a quantity with a single uncertain true value. The first step is to define the variable and specify its context or scope. Fully quantifying uncertainty about a variable involves modeling its variability, typically using a statistical model. This model can be a family of probability distributions or a more complex model of variable relationships. Uncertainty about the variability is expressed by using probability distributions to represent uncertainty about parameters in the statistical model. The choice of statistical model also introduces uncertainty, which should be considered in the analysis. For example, in a linear regression model, uncertainty about the parameters affects the uncertainty of percentiles or individual response values. By expressing uncertainty about the parameters using a joint probability distribution, the result is a probability distribution that represents uncertainty about percentiles or individual responses based on the covariate. Partial expression of uncertainty about a variable is possible but may require specialized knowledge and is less commonly used. Full quantification is necessary when the entire distribution of variability is of interest, while partial quantification can be used for specific aspects, such as specified percentiles. The approach taken to address uncertainty about variables has significant implications for calculating uncertainty about the output of a model. This is discussed further in Section 11.4.

When dealing with a categorical question, if probability is not used to quantify uncertainty, the alternative options are qualitative expression or including the uncertainty in a later expression that combines multiple sources. For an uncertain quantity, the minimum quantitative expression of uncertainty is specifying a range of values, which can be an upper or lower bound. However, a range by itself does not indicate the probability of including the true value or the relative likelihood of different values within the range. To provide a complete expression of uncertainty, a probability or approximate probability for the range must also be provided. If uncertainty is quantified with a range alone, the missing probability information should be provided later in the process, such as when quantifying overall uncertainty. In cases where absolute upper or lower limits are derived from theoretical considerations, such as a concentration not exceeding 100%, a range with absolute limits implies a probability content of 100%. If using such a range, the probability judgment should be explicitly stated, making it a probabilistic expression. Deterministic methods for working with bounds and ranges are discussed in section 11.6.

Possibility theory, along with fuzzy logic and fuzzy sets, has been proposed as an alternative approach to quantify uncertainty. It has been used in conjunction with probabilistic methods, such as Monte Carlo, in risk assessment applications. While fuzzy methods have been applied in various contexts, their benefits compared to probability-based methods are still uncertain. The IPCS (2014) Guidance Document briefly discusses fuzzy methods, acknowledging their ability to handle uncertainties arising from vagueness or incomplete information but noting their inability to provide precise estimates of uncertainty or handle random sampling error. Furthermore, the fuzzy/possibility measure lacks an operational definition comparable to subjective probability, as defined by de Finetti (1937) and Savage (1954). Consequently, these methods are not included in our comprehensive assessment of methods.

Obtaining probabilities by statistical analysis of data

Confidence intervals

Confidence intervals are suitable for application across EFSA in situations where standard statistical models are used in order to quantify uncertainty separately about individual statistical model parameters using intervals.
The quantification provided is not directly suitable for combining with other uncertainties in probabilistic calculations although expert judgement may be applied in order to support such uses.

Obtaining probabilities by expert judgement

Semi-formal EKE

The method has a high applicability in Working Groups and boards of EFSA and should be applied to quantify uncertainties in all situations:
1. where empirical data from experiments/surveys, literature are limited;
2. where the purpose of the risk assessment does not require the performance of a full formal EKE;
3. or where restrictions in the resources (e.g. in urgent situations) forces EFSA to apply a simplified procedure.
The method is applicable in all steps of the risk assessment, esp. to summarise the overall uncertainty of the conclusion. Decisions on the risk assessment methods (e.g. risk models, factors, sources of uncertainties) could be judged qualitatively with quantitative elements (e.g. subjective probabilities on appropriateness, what-if scenarios).
The method should not substitute the use of empirical data, experiments, surveys or literature, when these are already available or could be retrieved with corresponding resources.
In order to enable an EFSA Working Group to perform expert elicitations, all experts should have basic knowledge in probabilistic judgements and some experts of the Working Group should be trained in steering expert elicitations according to the EFSA Guidance.
Detailed guidance for semi-formal EKE should be developed to complement the existing guidance for formal EKE (EFSA, 2014a,b), applicable to a range of judgement types (quantitative and categorical questions, approximate probabilities, probability bounds, etc.).

Combining uncertainties for model inputs by probability calculations

Deterministic models (calculations)

The methods described in SO Sections 9, 11.2 and 11.3 can be used to quantify uncertainty about inputs to the model in the form of probability distributions or probability bounds. The mathematics of probability then leads in principle to an expression of uncertainty about the output using probability. Calculating that expression is easier in some situations than others.

Probabilistic models

Some probabilistic models are really just deterministic models with variable inputs. They can be handled as described in SO Section 11.4.2. Other models are more innately probabilistic and Monte Carlo simulation has a fundamental role in representing the processes involved, as well as quantifying variable inputs. Examples of this might include models of disease transmission, infection and recovery in a mixed population of susceptible and resistant individuals, or probabilistic modelling of cumulative exposures in a population of individuals to multiple contaminants via multiple routes. While there may be possibilities for some special form of Probability Bounds Analysis in such cases, it is likely to be easier to embed the model in a 2D Monte Carlo analysis or a Bayesian graphical model (see Section 11.5.2) in order to calculate uncertainty for the output of the model from uncertainties expressed about inputs using probability distributions.

Probability calculations for logic models

This is potentially an important tool for EFSA as it provides a way to structure logical arguments involving yes/no conclusions and to calculate the combined uncertainty about a conclusion based on uncertainty about underlying yes/no questions expressed using probability.

Deterministic methods for quantifying uncertainty

Uncertainty tables for quantitative questions

A template for listing sources of uncertainty affecting a quantitative question and assessing their individual and combined impacts on the uncertainty of the assessment conclusion.

This method is applicable to all types of uncertainty affecting quantities of interest, in all areas of scientific assessment. It is flexible and can be adapted to fit within the time available, including urgent situations.
The method is a framework for documenting expert judgements and making them transparent. It is generally used for semi-formal expert judgements, but formal techniques (see SO Annex B.9) could be incorporated where appropriate, e.g. when the uncertainties considered are critical to decision-making.
The method uses expert judgement to combine multiple uncertainties. The results of this will be less reliable than calculation, it would be better to use uncertainty tables as a technique for facilitating and documenting expert judgement of quantitative ranges for combination by interval analysis. However, uncertainty tables using +/- symbols are a useful option for two important purposes: the need for an initial prioritisation of uncertainties, and to inform probability judgements in the characterisation of overall uncertainty (see SO Section 14).