Introduction
This example-application of pedigree analysis focuses on uncertainty in environmental health risk science and policy. Near the city of Antwerp, an intense controversy has developed on the potential health effects of a waste incinerator. In a neighbourhood near the incinerator, an unusual high number of children had congenital defects. Local population and health workers pointed to the incinerator’s (dioxin) emissions as the cause. The incinerator’s management, supported by local authorities, deemed these accusations as ‘irrational, meaning purely hypothetical and not scientifically proven’.
Following years of heated debate, involving citizen’s committees, policymakers (both local and regional) and scientific experts, the conflict evolved to a phase in which all parties realised that a business as usual style will not work any longer. This lead to the establishment of the Flemish Centre of Expertise on Environment and Health (CEEH) and initiatives to renew interactions between science, policy and society (Keune 2003).
Method
Against this background, a workshop was held to explore how NUSAP Pedigree schemes can support and structure deliberations on uncertainties in environmental risk assessment (Craye 2005). The workshop involved experts and actors directly involved and external experts from the Netherlands Environmental Assessment Agency (RIVM) and representatives of stakeholders in environmental health issues. The workshop session here discussed, focused on an epidemiological study that had been used in the sociopolitical discussions on the incinerators’ impact on the environment and local health. The study investigated whether there were increased health risks among children, whose parents lived or had lived in the particular neighbourhood (Aelvoet 1998). The tailored pedigree matrix that was designed to structure the deliberative uncertainty assessment of this study is presented in the table below.
|
Score
|
Problem framing
|
Data – definitions
|
Data – collection
|
Analysis
|
Review
|
|
4
|
Negotiation
|
Negotiation
|
Task Force
|
Established
|
Extended
|
|
3
|
Scientific
|
Science
|
Direct
|
Discussion
|
External
|
|
2
|
Compromise
|
Pragmatic
|
Bureaucratic
|
Competition
|
Independent
|
|
1
|
Inertia
|
Symbolic
|
Indirect
|
Embryonic
|
Internal
|
|
0
|
Controversy
|
Unknown
|
Fiat
|
No info
|
None
|
Pedigree matrix for the epidemiological study
Through analysis of the study reports and interviews with their authors, the main phases in knowledge production to be covered in the pedigree assessment were devised. The choice of these phases reflects the complementarity between more cognitive and more social aspects. A phase related to problem framing was explicitly added. In this way, a discussion was triggered on the ‘status’ of the used problem definition in relation to other disciplinary framings and socio-political perspectives and on the ‘process’ through which the expert framing and other socio-political framings had (not) been matched.
The other phases had already been used in earlier applications of the pedigree scheme (Funtowicz and Ravetz, 1990). ‘Data-definitions’ relates to all those decisions, logically prior to actual enumeration, concerning the establishment of the relevant conceptual objects and the set of operating procedures. It deals with the question how is defined what has to be enumerated and how the way to do the counting is determined. ‘Data – collection’ deals mostly with the more technical aspect of quality of databases in function of what has to be enumerated. Quality has also to be seen as being influenced by the culture of the institute who is charged with the data collection.
‘Analysis’ highlights the reliability of the chosen method of analysis in function of how the problem was approached. It focuses on the scientific discussion about the choice of methods and on possible ‘extra-scientific’ reasons for preference for certain methods. ‘Review’ is related to the process of quality control of the research with particular attention to its interactions with the socio-political context in which the research took place.
The modes for each pedigree criterion referred to how the process to deal with that phase was run. A higher ranked mode had to be interpreted as a better chance to increase the quality of the study along three dimensions that were deemed essential to achieve ‘results fit for function’ (‘a serviceable truth’):
-
scientific robustness, in a perspective of constructive knowledge development ;
-
practical functioning of the science-policy interface to create a social basis for the policy measures and their knowledge base ;
-
normative considerations on a legitimate knowledge support, i.e. taking account of plural perspectives and rendering agency to all actors involved.
The hierarchical ranking of the modes was sometimes subject of discussion, which was not a surprise as ‘quality’ is a generic concept (on which all agree), whose dimensions and concrete meaning (on which there is no a priori consensus) as a standard in a particular context can only be filled in through deliberation. (Craye et al., 2005)
The discussions were shaped as a reasoned, structured debate focusing on underlying assumptions and frame-dependent choices in the different studies. In each session, a discussion leader had to prevent that only technical features of uncertainty would be covered. The protocol followed in the three sessions is elaborated in more detail by Craye et al.(2005).
The aggregation level chosen for the assessment by the panel was the ‘study as a whole’. Given that high aggregation level, it was not feasible to display on a disaggregated level all the types of uncertainty involved. Two illustrative critical aspects were presented for each of the phases. These related to choices that had been made in the framing and the design of the study, and subsequently had been criticized by other experts and relevant actors (e.g. under the phase ‘data-definitions’: ‘how to define the exposed population?’). Also included were other aspects that had not been openly debated in the past but could have led to a more reflexive knowledge development had they been approached with openness (e.g. under the phase ‘data-definition’: ‘who is competent to define a congenital defect ? a family doctor, a parent, a professor in epidemiology, an operator of a database… ?’).
Two experts – the author of the relevant study and an ‘opponent’ or ‘critical judge’ –introduced each topic. Then, the discussion extended to include the views and reactions of the stakeholders, citizens and policymakers in the panel. The session leader and another social scientist had to guarantee an informed and fair debate took place. He also had a list of possible questions in order to (re)focus the discussion if necessary. These model questions were based on insights on the structure of argumentations (Toulmin 1958, van de Graaf 2000), the content of actors’ frames of meaning (Grin 1997) and the different types of scientific debate and controversy when uncertainty is salient (Von Schomberg 1997). In these ways, the protocol enhanced the reflexivity of the process, both in terms of content, i.e. opening up the problem definition and the scope of argumentation, and in terms of process, i.e. placing the participants in new roles and rules of interaction. This setting challenged the traditional division between the scientist as a provider of facts versus policymakers and the public as defenders of values. The questions promoted discussion on the validity of assumptions, which could reveal particular biases in the framing of the risk. They were intended to deliver insight in the deeper debate on plausible hypotheses, distinguishing it from the more factual discussions on the empirical basis and the methodological work. Included were cycles of typical why?-questions, e.g. “What is the right (research) approach to this problem ? …..Why is this the adequate approach (asks for the definition of the (research) problem) ? ….Why do you define the (research) problem in this way (asks for underlying and supporting ‘theories’) ? ….Why do you use these theories in this case (asks for the fundamental features of framing, the preferences and convictions) ?”
Results
The discussion of each pedigree phase was concluded by giving a score. The scoring was a collective exercise of deliberation, which enabled to summarize the main points of discussion, to explain disagreement and to clarify any ambiguity in the pedigree scheme. The resulting pedigree score for epidemiological study was (1-2, 1-2, 2, 2-3, 0). The notation n-m signifies disagreement in the group.
The low score given to the epidemiological study was consistent with its failure to deliver robust insights and to play a relevant role in the policy debate at the time its results were communicated. In short, it was consistent with its quality for use in the socio-political context. Through discussing the critical aspects for each phase in the pedigree scheme and through providing and sharpening argumentation for specific pedigree scores, insight was gained in the reasons why the study was not fit for purpose and in the dynamics of controversy to which the study contributed. Themes discussed through the pedigree assessment included the incongruity of the epidemiological and socio-political framings, the plurality of disciplinary framings, the relation between the scientific problem and the socio-political issue, the value loadings of the research methods and issues of agency in knowledge production and review.
In this sense, the session promoted reflexivity and collective learning. It showed the potential of pedigree assessment to foster a deeper debate on problem framing and institutional aspects of knowledge development, possibly influencing the positions and perspectives held by different actors, their mutual expectations and the future design of policy relevant research.
Whereas the problem definition used in the epidemiological study and the choice of data sources and methods of data collection had been intensively discussed between the research team and the Ministry of Public Health, the resulting framing failed to address the concerns of the local population and was quite meaningless from the perspective of the incinerator’s management.
The reactions on this framing ranged from ‘an inadequate use of epidemiology’ to ‘a complete irrelevance of the epidemiological approach’. The problem definition used, implicitly called into question the existence of the cluster of congenital diseases in the neighbourhood by statistically testing the significance of these diseases’ incidence in the area, compared to the whole Flemish region. Opponents of the study argued a more correct and relevant use of epidemiology would have been to test the relation between these diseases and possible causing factors.
It turned out that during the discussions on the other phases the participants often referred to the frame-dependency of certain choices, thus confirming the crucial importance of problem frames. This suggested that in the still emerging environmental health science, ignorance and indeterminacy are the predominant forms of uncertainty, largely outweighing in importance methodological and technical aspects.
Overall, the session confirmed the centrality of the issue of framing in this kind of environmental health risk assessment. Participants took more time to discuss the framing than any other phase. The non-scientists also felt that their contribution was most relevant with respect to framing and felt less need to intervene in the more ‘technical’ phases dealing with the choices of data sources and of methods. However, they remained very interested and followed with attention the expert discussions on these issues.
The session raised awareness about the complexity of the issues to be studied and the resulting inherent uncertainty and ignorance. As participants learned that choices and assumptions could not be based exclusively on objective science, questions were raised about who is competent and ‘entitled’ to make the necessary choices.
Many participants suggested the approach could be applied in a constructive way, i.e. when policy supporting research is being developed, thus contributing to a negotiated management of environmental health risks. However, others argued that the method still was too science-centered, thereby devaluing the contributions by citizens and other lay knowledge providers. The lessons learnt during the workshop are being used to develop a set of pedigree schemes that can be deployed in distinct processes dealing with framing, research design and extended review. The experience also points to the need to reflect on the integration of these processes and their results in an overall inclusive approach.
Conclusion
The workshop confirmed the centrality of the issue of framing in this kind of environmental health risk assessment. Participants took more time to discuss the framing than any other phase. The non-scientists also felt that their contribution was most relevant with respect to framing and felt less need to intervene in the more technical phases dealing with the choices of data sources and of methods. However, they remained very interested and followed with attention the expert discussions on these issues. Si ilar workshop sessions addressing, respectively, an exposure study and a biomonitoring study showed the same pattern of engagement.
To some participants the workshop was a real eye opener: it raised awareness about the complexity of the issues studied, the inseparability of facts and problem frames by which these facts are conditioned, and the resulting inherent uncertainty in terms of indeterminacy, value loadings, and ignorance. Furthermore, the workshop clearly confirmed our hypothesis that non-scientists can make a valuable contribution to the critical appraisal of policy-relevant knowledge. This conclusion was shared by the scientists who had participated in the workshop. As participants got more and more convinced that in this kind of study few of the choices and assumptions were straightforward, meaning that they could be based exclusively on objective and factual information and insights undisputed among scientists, questions were raised about who is competent and `entitled' to make the necessary choices.
In this sense, the workshop made openings that could lead to an enhanced reflexivity and collective learning. It showed the potential of the pedigree assessment to foster a deeper social debate and a negotiated management of environmental health risks than the mainstream technical approaches to uncertainty assessment are capable of.
In the workshop experiment, pedigree assessment served a double goal: bridging the scientific knowledge development with the different framings of the problem and repositioning the expertise in the policymaking process. Ideally, interactive pedigree assessment should be part of a broader process of deliberative risk governance, where due discussion platforms are foreseen to let actors express their risk experience and their relevant knowledge in their own language, often in a more anecdotal style. Interactive pedigree assessment will even gain in relevance if more efforts are made to increase insight into plural risk framings by letting actors express their views on solutions, own knowledge, convictions and beliefs in settings like focus groups. In the same way, in the kind of pedigree assessment we did, it is somehow presupposed that a state-of-the art synthesis of the relevant science had already been made.
This reveals that the need addressed by the interactive pedigree assessment is not, in the first place, the provision of a forum for participation, nor a scientific review to evaluate evidence. It is intended to broaden the scope of problem frames considered and to remedy the current lack of transparent interaction at the interface between science, policy and society.
Settings for the expression of stakeholders' concerns on the one hand and for the delivery of expert views on the other, have to be complemented by processes of direct negotiation and deliberation to assess the various validity claims. Our experiment supports the potential of pedigree-based tools in this respect.
Full case description
M. Craye, J.P. van der Sluijs and S. Funtowicz (2005), A reflexive approach to dealing with uncertainties in environmental health risk science and policy, International Journal for Risk Assessment and Management, 5 (2), p. 216-236
References
Aelvoet W., Nelen V., Schoeters G., Vanoverloop J., Vlietinck R., Wallijn E. (1998). Risico op gezondheidsschade bij kinderen van de Neerlandwijk te Wilrijk, Vito-onderzoeksrapport 1998/TOX/R/030, Mol, Belgium.
Craye M., Goorden L., Vandenabeele J., Van Gelder S.(2001). Milieu en gezondheid in Vlaanderen: naar een adequate dialoog tussen overheid, bevolking en wetenschap, UFSIA-STEM onderzoeksrapport, Antwerpen, Belgium.
Grin J., Graaf van der H., Hoppe R. (1997). Technology Assessment through Interaction – a guide. Rathenau Instituut, Den Haag, The Netherlands.
Keune H., Goorden L., Mertens R., Loots I. (2003) Communicatie, Interactie en Reflectie over Biomonitoring Nota Opzet Communicatiestrategie, Lokale contacten en Reflectie. Biomonitoringcampagne, UA-Steunpunt Milieu & Gezondheid, Luik Sociaal en Gezondheidseconomisch Onderzoek, Luik, Belgium.
Toulmin S.E. (1958). The uses of argument. Cambridge.
Van de Graaf H., and Hoppe R. (2000). Beleid en politiek, Coutinho, Bussum.
Von Schomberg R. (1997). Argumentatie in de context van een wetenschappelijke controverse, Universiteit Twente WMW-publicatie 27, Eburon, Delft.
You are in ePLANETe >
