Key concepts
This text is a summary from the Supporting Opinion of the Uncertainty Analysis Guidance SO 5
Well-defined questions and quantities of interest
The purpose of most EFSA scientific assessments is to determine what science can say about a quantity, event, proposition or state of the world that is of interest for decision-makers.
In order to express uncertainty about a question or quantity of interest in a clear and unambiguous way, it is necessary that the question or quantity itself is well-defined, so that it is interpreted in the same way by different people.
This applies both to the uncertainty analysis as a whole and to its parts.
Readers may be surprised that the list of types of questions of interest does not include ‘qualitative’. This is because if a question of interest is well-defined, which it should always be for the reasons discussed above, then it can be treated as a yes/no question.
The questions or quantities of interest in some EFSA assessments refer to things that may seem challenging to define in terms of the result of a hypothetical experiment or study. Examples include the condition or property of being genotoxic, and calculated quantities such as a Margin of Exposure, neither of which can be directly measured or observed. In practice, however, such questions or quantities can be defined by the procedures for determining them, as established in legislation or official guidance, i.e. the data that are required, and the criteria for interpreting those data.
Conditional nature of uncertainty
The uncertainty affecting a scientific assessment is a function of the knowledge that is relevant to the assessment and available to those conducting the assessment, at the time that it is conducted.
Expressions of uncertainty are also conditional on the assessors involved. The task of uncertainty analysis is to express the uncertainty of the assessors regarding the question under assessment, at the time they conduct the assessment: there is no single ‘true’ uncertainty.
The conditional nature of knowledge and uncertainty means it is legitimate, and to be expected, that different experts within a group may give differing judgements of uncertainty for the same assessment question. Some structured approaches to eliciting judgements and characterising uncertainty elicit the judgement of the individual experts, explore the reasons for differing views and provide opportunities for convergence. A similar process occurs in reaching the consensus conclusion that is generally produced by an EFSA Panel.
The conditional nature of knowledge and uncertainty also contributes to cases where different groups of assessors reach diverging opinions on the same issue; again this is relevant information for decision-making. Where differences in opinion arise between EFSA and other EU or Member State bodies, Article 30 of the Food Regulation includes provision for resolving or clarifying them and identifying the uncertainties involved.
Uncertainty and variability
It is important to take account of the distinction between uncertainty and variability, and also how they are related. Uncertainty refers to the state of knowledge, whereas variability refers to actual variation or heterogeneity in the real world. Both can be represented by probability distributions.
Uncertainty may be altered (either reduced or increased) by further research, because it results from limitations in knowledge, whereas variability cannot, because it refers to real differences that will not be altered by obtaining more knowledge. Our knowledge of variability is generally incomplete, so there is uncertainty about variability. In addition, some types of uncertainty are caused by variability. You can learn more about variability on this page.
It is important that assessors distinguish uncertainty and variability because they have different implications for decision-making, informing decisions about whether to invest resources in research aimed at reducing uncertainty or in management options aimed at influencing variability.
How variability and uncertainty for each component of an assessment should be treated depends on whether the assessment question refers to the population or to a particular member of that population, how each component of the assessment contributes to that, and how those contributions are represented in the assessment model. Care is needed to determine when variability and uncertainty should be separated and when they should be combined, as inappropriate treatment may give misleading results.
Dependencies
Variables in a scientific assessment can be interdependent. It is important to take account of dependencies between variables in assessment, because they can have a large effect on the result. This means that different combinations of values must be considered in proportion to their expected frequency, taking account of any dependencies, and excluding unrealistic or impossible combinations.
Sources of uncertainty can also be interdependent, such as where learning more about one question of quantity alters uncertainty about another. Dependencies between sources of uncertainty should be identified and accounted for, as they can greatly influence the overall uncertainty.
Probabilistic calculations or probability bounds methods are preferable to expert judgments for assessing dependencies.
Dependencies also exist in assessments using qualitative methods, and assessors should evaluate their impact on uncertainty.
Models and model uncertainty
All scientific assessments involve some form of model, which may be qualitative or quantitative, and most assessments are based on specialised models relevant to the type of assessment. Many assessments combine models of different kinds.
EFSA uses different types of models, including conceptual models, hazard/exposure ratios, deterministic and probabilistic models, individual-based probabilistic models, statistical models, and logic models. Uncertainties in model structure and inputs need to be considered and quantified e.g. when characterising overall uncertainty. Model uncertainties should be expressed as probability distributions or bounds for the difference between model outputs and the real quantities they represent.
Models are simplifications of the real world, and while some directly address specific scenarios, others may address simplified scenarios or surrogate questions. The extrapolation from simplified models to the desired scenarios should be considered as a model uncertainty. When a simplified model is used repeatedly for different assessments, the model should be tested. This is the reason for “calibration” of standardised assessment procedures.
Evidence, agreement, confidence and weight of evidence
Evidence, weight of evidence, agreement (e.g. between studies or between experts), and confidence are all concepts related to uncertainty. Increasing the amount, quality, consistency, and relevance of evidence or the degree of agreement between experts generally increases confidence and decreases uncertainty.
The relationship between these concepts is complex and variable. Measures of evidence and agreement alone are insufficient as measures of uncertainty because they do not provide information on the range and probability of possible answers or values. Expressing evidence and agreement on qualitative scales can help structure the assessment process and facilitate discussions among experts. Confidence can be used both quantitatively, as in statistical analysis, where it represents a measure of uncertainty in statistical estimates, and qualitatively, as a subjective measure of trust in a conclusion. Weight of evidence involves weighing multiple studies or lines of evidence against each other to assess the balance of evidence for or against different conclusions. Additional considerations, such as the selection of evidence and methods for evaluating and integrating evidence, must be taken into account in uncertainty analysis.
Influence, sensitivity and prioritisation of uncertainties
Influence and sensitivity are terms used to refer to the extent to which plausible changes in the overall structure, parameters and assumptions used in an assessment produce a change in the results.
Analysis of sensitivity and influence can be used to evaluate the overall robustness of the conclusion with respect to choices made in the assessment. It can also be used to prioritise the most important sources of uncertainty for additional or refined analysis or data collection.
The term sensitivity analysis is often used in the context of a quantitative model, e.g. to measure the impact of changes to input values on the output of a mathematical model. Influence analysis is broader and considers changes resulting from uncertainties and choices made in the assessment, including the assessment’s structure and models used.
Conservative assessments
Many areas of EFSA’s work use deterministic assessments that are designed to be ‘conservative’. The word ‘conservative’ is generally used in the sense of being ‘on the safe side’ and can be applied either to the choice of protection goals, and hence to the question for assessment, or to dealing with uncertainty in the assessment itself. Conservative framing of the assessment question can simplify complex conditions by focusing the assessment on a conservative subset, which is protective of the rest.
The term “conservative” can also relate to two concepts: “coverage,” which refers to the probability that the real value is less adverse, and “degree of uncertainty,” which refers to the amount by which the real value might be less adverse and indicates how much the estimate might be reduced with further analysis. Decision-makers may view an assessment as insufficiently conservative if coverage is low or over-conservative if there is a high degree of uncertainty.
Describing an estimate as conservative requires specifying the quantity of interest, management objective, and an acceptable probability threshold. The first two elements involve risk management judgements, whereas the third element is evaluated by assessors. Asserting that an estimate is conservative without specifying the target quantity and probability conflates the roles of decision makers and assessors and is not transparent, because it implies acceptance of some probability of more adverse values without making clear either what is meant by adverse or what the probability is.
Similar considerations apply to qualitative assessments and assessments of categorical questions. As for quantitative assessments, asserting that a categorical assessment is conservative implies both a scientific judgement (what is the probability that the adverse category actually applies) and a value judgement (what probability would justify assigning the adverse category for management purposes).
Deterministic assessments with conservative assumptions are simple and quick to use and provide an important tool for EFSA, provided that the required level of conservatism is defined and that the assessment procedure has been demonstrated to provide it. Calibration of conservatism is crucial when using the same set of conservative assumptions in multiple assessments.
It is not necessary for the assessor to express estimates or their probability as precise values, nor for the decision-maker to express the required level of conservatism precisely. Assessors may provide approximate values or bounds, while decision-makers can set upper limits on conservatism. Probability bounds analysis can be used to calculate a probability bound for assessment outputs by eliciting probability bounds for each input. Increased use of probability bounds analysis is recommended for case-specific assessments and calibrating standardised procedures.
Expert judgement
Assessing uncertainty relies on expert judgement, as does science in general. Judgements are behind choices such as models, assumptions and assessment scenarios. Assessing reliability and relevance of data, as well as extrapolation, requires judgement. When using confidence intervals derived from statistical analysis of data, assessors must consider if it accounts for sources of uncertainty that affects its use in the assessment or whether some adjustment is required. When these various types of choices are made, the assessors implicitly consider the range of alternatives for each choice and how well they represent what is known about the problem in hand: in other words, their uncertainty. Thus, the subjective judgement of uncertainty is fundamental, ubiquitous and unavoidable in scientific assessment.
Expert judgement includes an element of subjectivity because different people have different knowledge and experience. The Scientific Committee emphasises that expert judgement is not guesswork or a substitute for evidence. On the contrary, expert judgement must always be based on reasoned consideration of relevant evidence and expertise, which must be documented transparently, and experts should be knowledgeable or skilled in the topics on which they advise. Well-reasoned judgements are an essential ingredient of good science.
Expert judgment is essential in scientific assessment but can be influenced by cognitive biases and group dynamics. Procedures are in place to manage conflicts of interest and mitigate biases. Formal approaches for expert knowledge elicitation (EKE) address psychological biases and facilitate aggregation of expert judgments. EFSA has published guidance on EKE and recognises the need for streamlined approaches. Selection of experts should represent a wide range of scientific opinions, and consensus should not imply compromise. Differences of opinion and scientific uncertainty should be reflected in the assessment report.
The Scientific Committee stresses that where suitable data provide most of the available information on an issue and are amenable to statistical analysis, this should be used in preference to relying solely on expert judgement. However, as noted above, most data are subject to some limitations in reliability or relevance, and further uncertainties arise in the choice of statistical model; the greater these limitations and uncertainties, the more the results of statistical analysis will need to be interpreted and/or augmented by expert judgement.
Probability
Decision-makers need to know the range and probability of possible answers for questions or quantities they submit for scientific assessment.
There are two major views on using probability as a measure for quantifying uncertainty. The frequentist view restricts probability to variability-based uncertainties and excludes uncertainties caused by knowledge limitations. The subjectivist view allows probability to represent all types of uncertainties, including knowledge limitations. Subjective probability offers comparability and can be applied to well-defined questions. It allows the use of mathematical tools to handle combinations of uncertainties. The guidance encourages the use of subjective probability, except when assessors find it too difficult to quantify uncertainty.
The subjectivist interpretation of probability does not exclude the frequentist interpretation. Frequentist probabilities must be reinterpreted as subjective probabilities to be combined appropriately. Estimates derived from statistical analysis may have additional uncertainties that require expert judgment.
Approximate probabilities, expressed as ranges on subjective probability, can be used when precise values are difficult to provide or when this is sufficient for communicating uncertainty. Approximate probabilities can be computed using the same mathematics as precise probabilities.
There is more information on probability in this tutorial.
Overall uncertainty
The recommendation to quantify uncertainty applies specifically to overall uncertainty, which refers to the assessors’ uncertainty about the assessment conclusion, considering all relevant sources of uncertainty.
Assessors should attempt to express the overall impact of identified uncertainties quantitatively, documenting qualitatively any uncertainties that cannot be quantified.
In cases where qualitative reporting is required, quantitative evaluation of overall uncertainty is still necessary to determine a justified conclusion.
Overall uncertainty should and can not include information about unknown unknowns.
The characterisation of uncertainty is dependent on the assessors, available evidence, time, and resources. Decision-makers should consider these factors when interpreting and using assessment conclusions.
Learn more about overall uncertainty in this section of the tutorial.
Unquantified uncertainties
The term “unquantified uncertainties” refers to uncertainties which the assessors have identified as relevant to their assessment, but are unable to include in their quantitative expression of overall uncertainty.
Assessors should strive to ensure that all questions or quantities in their assessments are well-defined. If they are unable to achieve this, then the uncertainty of those questions or quantities is literally unquantifiable.
However, even when a question or quantity is well-defined, an assessor may sometimes be unable to make a quantitative judgement of the impact of one or more sources of uncertainty affecting it. Sources of uncertainty which impact on the conclusion are not quantified for either reason (inability to define or inability to quantify) are sometimes referred to as ‘deep’ uncertainties. Deep uncertainties often arise in complex or novel problems.
When assessors cannot quantify the impact of some of the identified uncertainties, they should qualitatively describe them and report them alongside the quantitative expression of overall uncertainty. This is because the overall uncertainty will be conditional on assumptions made in the assessment regarding the sources of uncertainty that were not quantified. This has important implications for reporting and decision-making.
It is therefore important to quantify the overall impact of as many as possible of the identified uncertainties, and identify any that cannot be quantified. The most direct way to achieve this is to try to quantify the overall impact of all identified uncertainties, as this will reveal any that cannot be quantified.
Conditionality of assessments
Assessments are conditional on any uncertainties not included in the quantitative assessment of overall uncertainty. This is because the assessment relies on assumptions about those uncertainties, and the assessment’s output applies only if those assumptions are true. All assessments are inherently conditional based on the current state of scientific knowledge, the available information to assessors, and their judgments about the question at hand. Assessments assume that all relevant uncertainties are identified and there are no unknown unknowns.
Additional conditionality arises when identified uncertainties are not considered when characterising the overall uncertainty. The quantitative expression of overall uncertainty becomes conditional on the assumptions made for those unquantified uncertainties.
Decision-making should consider the implications of conditionality, recognising that assessments are based on scientific knowledge, do not account for unknown unknowns, and are influenced by the expertise and resources available.
Assessors must provide a list of identified uncertainties not included in the quantitative assessment, along with descriptions and explanations. They must not use any language that implies a quantitative judgement about the probability of other conditions or their effect on the conclusion (e.g. ‘unlikely’, ‘negligible difference’). If the assessor feels able to use such language, this implies that they are in fact able to make a quantitative judgement. If so, they should express it quantitatively or use words with quantitative definitions for transparency, to avoid ambiguity, and to avoid the risk management connotations that verbal expressions often imply. Assessors should communicate clearly that they cannot assign probabilities or make quantitative judgments about the unquantified uncertainties. However, this information is still valuable for decision-makers, as it clarifies the limitations of science and guides further analysis or research.
Decision-makers should decide how to address these unquantified uncertainties, potentially through further research or precautionary actions. Decision-makers may have the ability to influence some unquantified uncertainties, such as implementing specific practices or policies.