From Evidence-Based Making-Policy to Policy Analytics

This paper aims at addressing the problem of what characterises decision-aiding for public policy making problem situations. Under such a perspective it analyses concepts like “public policy”, “deliberation”, “legitimation”, “accountability” and shows the need to expand the concept of rationality which is expected to support the acceptability of a public policy. We then analyse the more recent attempt to construct a rational support for policy making, the “evidence-based policy making” approach. Despite the innovation introduced with this approach, we show that it basically fails to address the deep reasons why supporting the design, implementation and assessment of public policies is such a hard problem. We finally show that we need to move one step ahead, specialising decision-aiding to meet the policy cycle requirements: a need for policy analytics.


INTRODUCTION
3 to receive explanations before accepting decisions, they are not subjects but active parts of democracy and, in this sense, participatory processes became crucial. That's why policy-making now more than ever are expected to be accountable and "evidence-based" rather than based on unsupported opinions difficult to refute. The transparent relation between decision makers and stakeholders becomes fundamental in order to conquer and preserve consensus. Thus, policies became instrument "to exercise power and to shape the world" [24, p. 3].
In 1997 the concept of Evidence-Based Policy-Making (EBPM) has been introduced, in a modern form, by the Blair government [3]. EBPM is an acronym considered to be self-explanatory, but is it? The idea of creating policies on the basis of available knowledge and research on the specific topic is not new and is generally acceptable. However, we need to deepen the concept and its peculiarity in order to understand better what it means exactly. First of all, what is evidence? According to the Oxford English Dictionary, evidence means "available body of facts or information indicating whether a belief or proposition is true or valid". This definition it is enough clear and univocal? And what does it mean this highlight given to evidence? It is sure that evidence counted also before as a variable to decide. In which sense now its contribution is different?
EBPM has been defined as the method or the approach that "helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation" [16]. It is important to point out that the scope of EBPM is to help and "inform the policy process, rather than aiming directly to affect the eventual goals of the policy" [68]. In other terms we can say that EBPM is nothing more from "the integration of experience, judgement and expertise with the best available external evidence from systematic research" [17]. That said, it is considered as evidence all data from past experiences, all information and good practices from literature review. This is certainly important and necessary, but is it also sufficient to make a good policy? It looks reasonable to claim that the central role is played by the policy decision process and not by the decision (policy) itself. Thus, in order to support such a complex decision process we need information and knowledge considering the policy cycle as a whole, able to support accountability requirements.
Davies [16,17] and Gray [25] claim that the introduction of EBPM produces a shift away from opinion-based decision making towards evidence-based decision making. In this new context the decisions are based on the opinions and judgements of experts. This shift is not easy. On the one hand the distinction 1 INTRODUCTION 4 between opinion-based and evidence-based decision making, EBPM is viewed as an a-political, neutral and objective method to decide, but this statement is controversial. In fact, we know that data could be manipulated, that interpretations are subjective and that good practices are strictly linked with a specific framework. Intuitively speaking, values, preferences and at least decisions should remain a political fact.
The aim of this paper is to review the literature about policy making and evidence-based policy making (and related issues), highlight the origins, understand the critics and the controversies, while looking for a new perspective which we will call (and try to sketch) "policy analytics".
Our principal claims are: 1. The policy making process or "policy cycle" is a long term decision process characterised by: -the specific nature of public policies; -the requirements of legitimation, accountability and deliberation; -the existence of multiple public decision processes under the same policy cycle.
2. Supporting the policy cycle cannot be reduced in producing just "evidence" (in terms of data, knowledge, expertise etc.). The analytics which usually support decision making are necessary, but not sufficient in this case. Constructing evidence should be seen as a specific type of decision aiding process and as such should be methodologically well founded.
3. We need a new and more rich concept accounting for all decision aiding activities aiming at supporting the policy cycle: we call this term "policy analytics" and we will briefly introduce some of its main features in this paper.
The paper is organised as follows. We start addressing the meaning of some important terms and concepts (section 2). Next, we briefly present a review about the EBPM state of the art (section 3). After that we discuss criticisms, which involves the policy-making process and the introduction of evidence within it (section 4). We will then introduce the concept on "Policy Analytics" as a new term grouping the activities and the knowledge created supporting policy-making in the whole policy cycle (section 5). A concluding section including future challenges ends the paper.

Public Policy
To start it is important to understand that the concept of public policy (PP) should be enough wide and abstract in order to adapt itself to various applications and contests. For such reasons, over the past 50 years many definitions have been coined to define PPs. Such definitions have different meanings because the authors bring into focus different aspects as process, stakeholders, objects and decision levels [1,21,22,31,36,37]. From this literature we can identify six main characteristics of PPs: • the relations between the different stakeholders (power), • the different institutional levels, • the duration over time, • the use of public resources, • the act of deciding (including deciding to not decide), • the decisions' impacts.
It is important to underline that such characteristics do not always have the same importance. This changes according to the different context and goals.
In our specific context we want to emphasise that every PP is a process that implies a set of decisions in a public context; thus, it is a public decision process. This process is developed over a period of time and involves different decisional levels, each one of them interacting with the others with respect to a set of determined rules. The process and the related interactions are developed in order to solve a problem having characteristic of public issue, or rather a problem in which resources and rationality are public. The concept of "public issue" is not always clear: the issue that the policy will face is an object which conveys a meaning. Naming a public policy is the action of defining a signifier and it implies the legitimation of the signified. Thus, every subject affected by the policy (policymakers, experts, citizens, stakeholders) make-up his own meaning of the policy, legitimated by its name and definition. Under such a perspective, a public decision is a public choice and it implies an allocation of public resources. Even no-action 2 TERMS AND CONCEPTS 6 in a determined field is considered a policy because it implies the public choice of maintaining the same resource-allocation than before. Speaking about public resources, governments/public subjects have to make understandable how and why they use public resources in order to solve problems. The public decision process is requested to be accountable to the public in contrast with the complexity of the entire process. Thus, we need an operational definition able to summarise the characteristics introduced: We consider a public policy as a deliberation of allocation of public resources to a portfolio of actions aiming at achieving a number of objectives settled by the public decision maker, considered as an organisation. Such a deliberation generates multiple meanings for the subjects of the policy.
In the concept of public policy as defined above we want to highlight that a policy has a meaning for the stakeholders affected by the policy itself and for the citizens in general. A policy does not only pursue quantifiable objectives, but it generates a legitimation space, thus producing inclusion and/or exclusion. This is a crucial difference with respect to generic policies of the type a private business will conceive.

Policy-making process
From the 1950s, when the field of policy analysis was born, policy-making was interpreted and considered as a process, that is a sequence of interactive stages or phases. Under such a perspective the policy-making process can be considered as developing into time and space merging actions and intentions, decisions and also not decisions, impacting on the society and on the political system itself. The idea to model the policy-making process (or cycle) in terms of stages was first put forward by Lasswell [39]. He introduced a model of the process divided in seven stages: intelligence, promotion, prescription, invocation, application, termination, and appraisal. This set of stages has been contested and criticised, but the model itself has been successful as a framework for subsequently studying policy science and policy analysis [35]. During the 1960s and 1970s, a number of different process typologies have been developed, used to organise and systemise the growing research (for such typologies see [1,8,9,36,45]). Today the most conventional process to describe a chronology of a policy making is made up by [31,35]: • agenda setting,

7
• policy formulation, • decision making, This kind of policy-making process (as presented by Lasswell and others) has been designed like a problem-solving model, according to the rational model of decision-making developed in organisation theory and public administration [35]. While the behavioural theory of decision-making presented by Simon [59] has pointed out that the real world does not follow such stages, this kind of process still counts as a reference [35]. This origin of the studies on policy-making, as subsequential stages, is our basis for interpreting the policy-making process as a proper decision-making process.
We need to emphasise that a policy cycle is not a usual decision process as any other occurring within organisational action because it does not imply only the public organisation, it is not only an "internal process" to solve. A public decision process involves multiple and different organisation and/or individuals; thus, there is not a single-determined rationality to simply follow. This process is characterised by several rationalities, even conflicting ones, and the process generates a legitimation space in which these rationalities interact.

Public Deliberation, Legitimation and Accountability
A feature which helps to distinguish a public decision process from other decision processes is "public deliberation": the outcome of the decision is a public issue, a public authority must communicate it, the public must know. The publication of an official document which defines and explains the policy is the act that produces the wanted (and unwanted) outcomes and reactions to the decision. The intermediate and the final act explain the motivations and the causes of that policy. Public deliberation is also expected to establish accountability of the public decision process and of the public authority itself and, to some extend, their legitimation to the general public or stake-holders.
Linked to deliberation we find two concepts, recently used in the field of PP: "Legitimation" and "Accountability". Legitimation and accountability are integral parts of the relations established between stakeholders. They are fundamental in the creation, evolution and maintenance of a conceptual social space (also called 8 "action arena" or "interaction space", to know more about see: [48,49]), where stakeholders interact creating relation, goods, service, but above all develop and define their rationality [27].
Speaking about legitimation we refer to authorisation and consensus. The need for legitimisation borns from the relative dimension of power, that makes it frail in terms of collective acknowledgement. Political power does not get identification and legitimation from a transcendent order; thus, the recognition of its value and so the collective acknowledgement becomes an immanent issue, which has been faced and solved in the framework of public policies and within their criteria. On the one hand the legitimacy of the public action and its decisions, comes from the law, which gives to the elected the power to decide with authority and to manage public resources. On the other hand, legitimation (this concept is considered orthogonal to legitimacy, but not exclusive) is given by consensus on the rationality of action and on the decision process itself. What is important and legitimising is rationality.
Which kind of rationality could be legitimising in the field of PP? The concept of rationality is not forthright, neither trivial as it seems. In research, many forms of rationality have been identified (the idea of multiples rationalities has been introduced first by Max Weber [74]), no-one of them is valid or right and no-one is false, but all of them aim at some validity claims [27]. We distinguish three different approaches in establishing such a validity: • economic rationality (homo economicus) [28,30,55]; policies should maximise the utility of a society seen as the aggregation of the consumers within it; • bounded rationality [60]; policies should satisfy some subjectively defined decision maker's requirements for action; • communicative rationality [27]; policies should result as consensual artifacts through four validity dimensions: -truth; -scientific support; -normative rightness; -sincerity Talking about accountability we refer to openness and transparency: public administrations can, should and sometimes must show and justify the reasons that bring to a decision, to some policy or any allocation of public resources: the 2008 EVALSED guide of the European Union [34] defined it as: "Obligation, for the actors participating in the introduction or implementation of a public intervention, to provide political authorities and the general public with information and explanations on the expected and actual results of an intervention, with regard to the sound use of public resources. From a democratic perspective, accountability is an important dimension of evaluation. Public authorities are progressively increasing their requirements for transparency vis-a-vis tax payers, as to the sound use of funds they manage. In this spirit, evaluation should help to explain where public money was spent, what effects it produced and how the spending was justified. Those benefiting from this type of evaluation are political authorities and ultimately citizens.
We want to highlight that the EU definition put stress on the concept of accountability as an unavoidable dimension of evaluation, from a democratic point of view. This statement make us come back to the concept of legitimation and let us understand that these two concepts are integral part of the same matter. The definition also emphasises that evaluation should aim at helping the accountability of public policies: which are the public resources used and how they are managed, what are the effects of the implemented policy, why do the policy-makers choose for one alternative among others.
as the ability to rank all known available alternatives [50] (see also [6,7]). At the beginning the concept of rational decision was central both for the economic dimension of problem solving and the scientific management of enterprisers [71].
During the II World War for the first time this scientific approach has been used in order to support military and intelligence activities. In this context the idea that decision making can be studied in a scientific and rational way arose. From such premises, since the 40's, science and policy started to be studied within the same framework. It was considered possible to use a scientific method in order to improve policy-making. An important figure in this field is Harold Lasswell, committed to the idea of a "policy science democracy" [38,40,43]. During the '60s and '70s the faith in science and rationality grew up constantly. At that moment anything rational was considered achievable, and any problem was considered solvable if it could be managed in a rational and scientific way. This is part of a more wide movement: the "post-positivism", built as an evolution from the "positivism" (see Comte [13,14] and Saint-Simon [18]), which involved the whole Europe in the nineteenth century.
In 1963 James Buchanan and Gordon Tullock organised a conference in which the shared interest was the application of "economic reasoning" (commonly considered as good example of rationality) to collective, political or social decisionmaking. In December 1967 it was publicly adopted the term "public choice" in order to indicate this topic [50].
The public choice approach is related to the theoretical tradition in public administration, formulated by Wilson [77], later criticised by Herbert Simon. Wilson's major thesis was that "the principles of good administration are much the same in any system of government" [50, p. 203], and "Efficiency is attained by perfection in hierarchical ordering of a professionally trained public service" [50, p. 204]. Wilson gave also a strong economic conceptualisation at the term efficiency. He said "the utmost possible efficiency and at the least possible cost of either money or of energy" ( [77, p. 197] cited in [50, p. 204], see also [50,76,77]).
Already since the '40s Herbert Simon [59,60,61,62,63,64,65] strongly criticised the theory implicit in the traditional study of public administration. He said that there is no reason to believe "that perfection in hierarchical ordering would always be the most efficient and organisational arrangement" [50, p. 204], because there is no reasons to believe at one "omniscient and benevolent despot". On the contrary, Simon makes a first distinction between facts and values that are considered in choosing among alternative possibilities. Then, he focus on the construction of a bridge to link theory and empirical studies. Moreover, he defines the criterion of efficiency "as a norm for evaluating alternative administrative action" [59]. He also argued that the "criterion of efficiency dictates that choice alternatives which produce the largest result for the given application of resource" ( [63] cited in [50]). Thus, in order to use this criterion, the administrative's results must be defined and measured.
After the first years, the post-positivist approach has been criticised by several authors (see for instance [5]) and above all by policy analysts. Such critiques "can be positioned in terms of explicit rejection of both technocratic and accommodative images" [20, p. 191].
Our claim is that this focus on "scientific" and "rational" policy making was in general misunderstood because interpreted as a shift of decisional power from politics to science, from elective bodies to experts, from subjective to objective: instrumental rationality was expected to completely govern the policy process. In post-positivist policy making there was the illusion that policy-makers, with their competence, information and set of tools, could solve in an optimal way problems, under given resources and constraints. Under such a perspective "policy problems were technical questions, resolvable by the systematic application of technical expertise" [24, p. 4]. But, "full information is always an illusion" [24, p. 19] and we never really have the complete knowledge to "optimise" our goals. In fact decisions, and thus policies, are undertaken partly under "ignorance" or underestimating the importance of some variables. This is often related to "learning by doing", which will become a foundational concept of the renewed interest towards rational decision. In spite of such criticism as Dryzek [20, p. 191] said: "these dreams may be long dead, and positivism long rejected even by philosophers of natural science, but the terms "positivist" and "postpositivist" still animate disputes in policy fields. And the idea that policy analysis is about control of cause and effect lives on in optimising techniques drawn from welfare economics and elsewhere, and policy evaluation that seeks only to identify the causal impact of policies." Dryzek's sentence puts in evidence that "positivism" and "post-positivism", in spite of all, are still alive. Actually, we believe the promotion of EBPM has been a return to such approaches.

From medicine to social science
Evidence-based policy-making is born on the roots of evidence-based medicine (EBM) and evidence-based practice (EBP). In fact, it is easy to find these roots in the EBPM logic, the kind of analysis and the way to understand problems and solutions. EBM and EBP are based on a simple concept, that is to find the best solution integrating past experience. The practice of EBM needs to integrate individual clinical expertise with the best available external critical evidence from systematic research, in consultation with the patient in order to understand what alternative suit the patient best. In this sense, we can say with Solesbury [67] that EBM and EBP have both an educative and a clinical function. In other words, this kind of evidence is based on a regular assessment through a defined protocol of the evidence coming from all the research. In order to face this need of EBM for systematic up-to-date review, in 1993 the Cochrane Collaboration in the UK has been founded, which deals with the collection of all such information.
Subsequently, considering the good results obtained in medicine using such an approach, the will of politicians to use the same scientific method in order to support public decisions and legitimise the policies building make its appearance. Given the success of the Cochrane Collaboration in the production of a "gold standard", the Campbell Collaboration has been established in 2000, which provides for systematic review on social science in the field of education, crime, justice and social welfare.

EBPM in the UK
In 1994, the Labour party termed itself as "New Labour" in order to announce a new era: "New Labour" was expected to be a party of ideas and ideals but not of outdated ideology. "What counts is what works". The objectives are radical. The means will be "modern" [3]. In these first announcement it is possible to recognise the same roots and philosophy that pervade EBM and EBP. Moreover, in 1997 when the Labour Party won the general elections they decided to open a new season of policies. In order to organise and promote it, they published the Modernising Government White Paper [10], in which they argue that: "government must be willing constantly to re-evaluate what it is doing so as to produce policies that really deal with problems; that are forward-looking and shaped by the evidence rather than a response to 3 EVIDENCE-BASED POLICY-MAKING: STATE OF THE ART 13 short-term pressures; that tackle causes not symptoms; that are measured by results rather than activity; that are flexible and innovative rather than closed and bureaucratic; and that promote compliance rather than avoidance or fraud.To meet people's rising expectations, policy making must also be a process of continuous learning and improvement. (p. 15) better focus on policies that will deliver long-term goals. (p. 16) Government should regard policy making as a continuous, learning process(...)We must make more use of pilot schemes to encourage innovations and test whether they work.(p.17) encourage innovation and share good practice (p.37)" In this document they describe the goals of the new government changing the approach towards public policy. It is clear that this change implies the adaptation of the evidence-based method and logic. We can consider the Government White Paper as the Manifesto of United Kingdom's EBPM, where the EBPM cover the same role of EBM, that is to give accountability at the field of policy. Such accountability is promoted by two main forms of evidence [57]: • the first one refers to results and then to the effectiveness of the work of the government; • the second one refers to the improvement and consequently the knowledge on how well policy works under different circumstances.
In practice, policy processes have been viewed as learning processes that have to be studied, analysed and monitored in order to get new evidence for building future policies. With this new way to understand the policy process there is a shift of goal, from a short term policy founded on ideology and no-scientific knowledge to a long term policy founded on identified causes of the social problems to face. Under such a perspective, any other no-scientific components of the policy process are considered a misappropriation from the "truth/reality" of the problems. In fact, David Blunkett, in his speech in 2000 [4], emphasised that: "This Government has given a clear commitment that we will be guided not by dogma but by an open-minded approach to understanding what works and why. This is central to our agenda for modernising government: using information and knowledge much more effectively and creatively at the heart of policymaking and policy delivery." "What works and why" became the UK slogan for EBPM promotion, and also the following government put emphasis on EBPM, although with some difference. The shift was from policy learning to policy delivery, and thus the need to go away from experimentation and the awareness that what matters most is hard quantitative data. In fact, in the last years EBPM has evolved from a generic attention for any kind of scientific analysis to an higher attention to quantitative and economic analysis [32,52].

Other experiences
After being developed in UK, EBPM expand its influence towards other English speaking countries: US and Australia. In US, the most representative event was the foundation in 2001 of the US Coalition for Evidence Based Policy that aims to increase government effectiveness through the use of rigorous evidence about what works. Evidence is again consciously borrowed by medicine, with the explicit goal of replicate in the social policies the effectiveness that produced many advances in the field of human health. Evidence-based policy making was seen as an instrument of rationality that let the society avoid the waste of expensive but failing social policies. Evidence is thus a resource-rationing tool [44] in the sense that it indicates the right way to face a social problem, making the country more efficient and let it spend only for satisfactory policies.
In Australia there is no formal coalition and no explicit formal willingness to apply EBPM, but the language of evidence spread to many fields of public policy: we can see examples in health and family services, community services, education and immigration. In these fields we can find sentences referred to evidence that implies that EBPM is being actively promoted in a specific way: "research helps to depoliticise educational reform" [19, p. 190]  "My conclusion is that we should forget about the grand theories of sociology and the ideologies of the old politics and pursue an evidence based approach to welfare reform" Here, evidence is considered as an apolitical solution; Smith and Kulynych [66, p. 163] state that "efficiency becomes the primary political value, replacing discussion of justice and interest".
These sentences more than others show that EBPM is considered as a source of truth and as an instrument to bypass political control. Is this exact?

4 Criticism towards EBPM
In the EBPM debate the authors cast doubt on whether introducing evidence in the policy making process is really innovative. If before policy-making could be described as a "swamp" [58] characterised by complexity, uncertainty and ignorance, then EBPM should help to move it towards firm ground in which sound evidence, rather than political ideology or prejudice, could drive policy. The question is whether this confidence in the power of evidence is a step forward or backwards, because EBPM could appear a return to the old time trust in instrumental rationality. In fact Parson [51, p. 44] states that: "EBPM must be understood as a project focused on enhancing the techniques of managing and controlling the policy-making process as opposed to either improving the capacities of social science to influence the practices of democracy".
Sanderson [57, p.1] argues that: "the resurgence of evidence-based policy-making might be seen as a reaffirmation of the modernist project, the enduring legacy of the Enlightenment, involving the improvement of the world through the application of reason ".
Actually, in the UK evidence-based practice the focus was on effectiveness, efficiency and value for money. This experience is characterised by a managerial emphasis [69, p. 19]. Evidence-based policy making, in its effort to implement accountability, is linked to an instrumentalist mood of managerial reforms that have infiltrated public administration practices in many western democracies over the past three decades. Managerial reforms and evidence-based policy can be assimilated by the same technocratic logic, concerned with procedural competence rather than substantive output [44].
In the following we introduce the main issues for which EBPM has been criticised, that is: -the existence of many evidences; -the multiplicity of factors influencing policy making; -the contingent character of evidence.

Many evidences
There is a big amount of typologies of evidence: the most used distinction is between hard/objective and soft/subjective. The first one includes primary quantitative data collected by researchers from experiments, secondary quantitative

CRITICISM TOWARDS EBPM
16 social and epidemiological data collected by government agencies, clinical trails and interview or questionnaire-based social surveys. Other sources of evidence, "typically devalued as "soft" [44, p. 151], are photographs, literary texts, official files, autobiographical material like diaries and letters, the files of a newspaper and ethnographic and particular observer accounts. Davies defines a scheme [17, p. 15] in which he shows that there are seven kinds of evidence originated by scientific research: impact evidence, implementation evidence, descriptive analytical evidence, public attitudes and understanding, statistical modelling, economic evidence, ethical evidence. Moreover, Davies states that in policymaking "privileging any one type of research evidence or research methodology, is generally inappropriate" [17, p. 11]; thus, a balance between social researchers and a general understanding and competence of the full range of research methods is required. Otherwise, Sanderson [57] states the need for developing just impact evidence in order to build policies through long term impact evaluation; he argues also the use of theory based evaluation, considered more correct in order to understand how policies achieve their effects.
Due to different opinions in the debate, in order to avoid misunderstandings in practice, the UK Cabinet Office clarify the meaning of evidence in the White Paper on Modernising Government [10] in which evidence is defined as: "expert knowledge; published research; existing research; stakeholder consultations; previous policy evaluations; the Internet; outcomes from consultations; costings of policy options; output from economic and statistical modelling" From this definition it seems that this conception of evidence privileges "conventional scientific methods", and from that standpoint the UK social policies use a limited range of evidence. The concept of evidence-based policy is assumed to be a rational and scholarly approach [44,57], which aspires to "goes beyond political ideology" (Latham [42], cited in Marston e Watts [44]). Nevertheless, this kind of policy, that evaluates and prioritises the knowledge sources is far from being neutral or objective. In this case, the selection of the "right" evidence is necessarily a limited view of what counts as a valid knowledge. The building of a hierarchy of knowledge means that we can consider some forms of knowledge more related to reality/truth and this is not neutral. Every theory is based on some hypotheses or interpretations of the complex reality and they are not omni-comprehensive. In choosing what counts as the valid knowledge for policies, policymakers implicitly states their interpretation of reality. For instance observing the recommended and adopted evidences in UK experience we can deduce that the interpretation of the

CRITICISM TOWARDS EBPM
17 reality could be intended as post-positivist, in the sense that the stress is on the relationship cause and effect. This claim is supported by the importance given at the concept of effectiveness and efficiency. These two concept became in the UK policy evaluation often the first, if not the only, qualification needed at one policy in order to be implemented [33].
If the discussion up to this point highlighted the problem that around us there are many evidences (statistics, surveys, polls etc.) which is difficult to choose and/or prioritise, we need to focus on another aspect often neglected or underestimated when talking about evidence. The problem are the multiple interpretations the same "evidence" may carry. This is all the more important since evidence is expected to be used within a policy cycle where multiple stakeholders with multiple concerns are involved and who are going naturally to interpret the evidence differently. To make things more complicated such multiple interpretations can be influenced on how evidence is technically produced. In order to better explain our standpoint we present two examples.

Example 1: Air Quality
Consider the case of Air Quality (see [6]). "Evidence" about Air Quality (in France) is expected to be provided by the ATMO index. This index takes into account four pollutants, measured (does not matter here how) on a scale from 0 to 10 and chooses the max among them (the worst). This way to construct reflects the approximate knowledge we have about the impact on health of these four pollutants: any among them is supposed to be equally unhealthy. However, consider three consecutive measures: the first one at time t 1 is the present situation (at a certain location), while t 2 and t 3 refer to the situation observed after two policies have been implemented (using the same budget). The situation is summarised in Table 1. Should we consider the ATMO index as "evidence" then the policy conducting to situation t 3 should be considered as better than the policy conducting to situa-

CRITICISM TOWARDS EBPM
18 tion t 2 . Indeed for the ATMO index the quality of the air did not improve from t 1 to t 2 , while it did from t 1 to t 3 . Obviously this is counter-intuitive! One could claim that the disaggregate information should be used as "evidence", but then are we sure that each of the four measures does not suffer the same type of problem the ATMO index presents? We are not going to discuss here what should be the more appropriate way to measure Air Quality. What we want to emphasise is that the ATMO index can be used as "evidence" about whether an observed situation is "healthy", but cannot be used as "evidence" about the effectiveness of Air Quality improvement policies. In other terms this index (as any other) allows multiple interpretations which are more or less suitable to the type of assessment we are interested to perform. Such interpretations are strongly related, among others, to how the index has been (technically) established.

Example 2: Statistics about Poverty
Some may claim that row statistics reporting "facts" should be considered as "evidence". But then consider the following fact: 95% of rural households in country XXXX do not have tap water available. What should be allowed to infer from that? Perhaps that connecting rural households to fresh water distribution is a natural priority, requiring an appropriate policy (and relative investments).
Surprisingly, if we ask the household owners what they think about that, we discover that this is not a priority for them ... They claim that they do not see the problem. They fetch water from the near water pools. Ok! Here is the problem. Typically water is fetched by women, while the household owners are men ... Certainly men do not see the problem. What happens if we ask the women? Surprisingly the women also claim that there is no problem ... A more thorough investigation reveals that going for water is one of the rare occasions they have to go out of home! Despite fetching water is a hard task, it pays because it allows to have some (little) social life! The story, which is a simplified version of the real one, tells us that row statistics do not reveal automatically any truth. Facts need to be interpreted in order to be used for any decision process and such an interpretation is related to subjective values, constraints, customs, history, social norms etc.. The example tells us that "evidence" does not exist independently from the decision process for which it is expected to be used. Although "facts" exist, choosing the "facts that matter" is a subjective option and interpreting these "facts" is another subjective option.
Summarising the two examples we can claim that looking for evidence while considering how to solve a problem is certainly a sound attitude and certainly

19
preferable to a purely intuitive approach. However, contrary to the dominant idea that "evidence" should guide policy making, it seems that it is the policy that should guide us in looking for appropriate "evidence". Actually we should consider questions of the type: • Who needs this evidence?
• Why (s)he needs this evidence?
• What is the purpose?
• Who other is affected by such evidence and how?
• What resources do we commit and what do we expect?
It turns out that such questions are practically the same we need to answer when trying to model a decision aiding process (see [70]). Under such a perspective we can still consider that we can follow a scientific approach in aiding policy makers involved in a policy cycle, but without evacuating subjectivity, political priorities, values, culture; placing them instead at the centre of the methodology to be used.
The problem in policymaking is not whether there is enough relevant information, but how to consider, construct and interpret it: "the danger is not that one uses no evidence at all, but that one uses simply the most readily available" [53].

Policy-making as a result of many factors
"Evidence" is not the only determinant of policy-making, but it is just one of several factors that influence policy-makers in choosing and determining policies. It is sure that EBPM is a breaking point with respect to the traditional approach that identifies in power, people and politics the only policy-making factors [51]. However, it is clear that we have to overcome the "naïve" concept of Evidence-Based Policy Making, where research replaces policy, and experts/technicians replace the politicians. Policies are complex "objects" and the policy-making process is influenced by several relevant factors. Davies [17] indicated the following ones: • Experience, Expertise and Judgement.
Policy making implies several stakeholders each carrying different types

CRITICISM TOWARDS EBPM
20 of knowledge such as ground experience (of local groups, citizens, economic actors), expertise (of technical staff, scientists, experts) and judgements (public opinion, elected bodies, committees). Such knowledge is expected to be integrated in the policy making process [46]. This could add significance when the existing information is imperfect or non-existent [26].
Establishing a policy mobilises material and immaterial resources (knowledge, authority, capital, land etc.) and results in allocating resources aimed at implementing a plan of actions. Both such resources are bounded (and scarce). The result is a quest for efficiency both as far as the policy making process and its outcomes are concerned. This "economic" aspect of the policy making process is perhaps the most studied in terms of supporting methodologies and practices [11,12,33,47].
Values are the essence of policy making. They induce preferences, priorities, judgements and justify actions. They have several different origins: ideology, culture, religion, beliefs, knowledge, discussion etc.. It is unlike that any policy making process can be legitimated without making reference to some set of values. However, it should be noted that values evolve over time in unexpected directions (consider the cases of the value of the environment in the last 50 years, the value of women rights in the last 150 years or the value of individual freedom in the last 250 years).
• Habit and Tradition. Political institutions have their own organisational inertia. The policy making process is characterised by procedures and patterns often rooted in culture and history, but nevertheless constraining the potential outcomes. Several times such constraints appear under form of fundamental laws (such as constitutions), but equally likely they can appear as socially constructed legitimation processes and outcomes.
• Lobbyists, Pressure Groups and Consultants. Any policy making process mobilises groups of pressure, informal or organised lobbyists as well as the opinion of experts. Such stakeholders are not always visible and have a less systematic influence. However, they play a key role in the process allowing specific concerns, stakes and interests to find their way in the discussion.
• Pragmatics and Contingencies. Policy making, agendas and decisions are influenced by unanticipated contingencies and "emergency" procedures which do not necessary fit with rational policy making. Policies are expected to take into account long term uncertainties as well as the aspirations of the future generations. This can be in contradiction with a contingent, short term view of policy making [56,54].
The above list of factors, which are pragmatically considered when conceiving or evaluating a policy, shows that "evidence" needs to be declined in terms of knowledge produced within a decision aiding process and not as objective information revealing the truth.

Evidence as a contingent knowledge
Young et al. [78] identify five models in which the relation between research and policy can be shaped and defined, five ways through which inputs of knowledge are managed by the policy cycle: • the knowledge-driven model, • the problem-solving model, • the interactive model, • the political/tactical model, • the enlightenment model.
These models are used in order to understand how evidence is thought to shape or to inform policy in order to explore the assumptions underlying evidencebased policy making. The first two models are the extremes forms, they differ for the direction of influence of the relationship: in the first (knowledge-driven model) research leads policy in a sort of scientific inevitability, while in the second (problem-solving model) research priorities follow policy issues. In the interactive model there is no position of influence, and the relationship is characterised as mutual, subtle and complex. The political/tactical model sees the research priorities as settled by the political agenda, in which studies are used in order to support a political position. In the enlightenment model, otherwise, benefits of research

22
are indirect because they contribute to the comprehension of the context in which the policies will act.
These five models are steps in a ladder between the power of authority and the power of expertise. "Emphasising the role of power and authority at the expense of knowledge and expertise in public affairs seems cynical; emphasising the latter at the expense of the former seems naïve" [67, p. 9]. In the experience of United Kingdom, we can not recognise one of these as a dominant model [75, p. 25]. Sanderson [57, p. 5] defines a set of variables that are implied in the definition of model: "the nature of knowledge and evidence; the way in which social systems and policies work; the ways in which evaluation can provide the evidence needed; the basis upon which evaluation is applied in improving policy and practice". Under such a perspective the Labour government announcement that a new era in which policy will be shaped by evidence, thereby implying that "the era of ideologically driven politics is over" [46, p. 3] is controversial. It is neither neutral, nor uncontested; evidence is a fundamentally ambiguous term.
The way through which EBPM has been perceived and practiced reveals an idea of policy making based on a "cause-effect" principle. To simplify, social outcomes are seen as the result of how certain mechanisms work within a certain social context: once we know the mechanisms and the context we can foresee the consequences. This approach has been criticised from constructivists, for whom the "knowledge of the social world is socially constructed and culturally and historically contingent" [57, p. 6]. Sanderson [57] points out that the research has not the role of producing objectivity, or solutions for policy-makers concluding that constructivism needs to be reconciled with "practical requirements".

Discussion
Let's summarise our claims.
• Policies have a twofold impact: -they deliberate an allocation of resources aiming at pursuing some objectives (not always measurable though); -they generate a legitimation space, thus producing inclusion (or exclusion), inviting stakeholders to enter within (leave) it.
• Policy making is a long term decision making process owing specific characteristics: -it implies participation "de facto" (due to the legitimation associate to any

DISCUSSION
23 policy); -there exist moments (at least one) of public deliberation; -it is expected to be accountable, not only for the implied stakeholders, but for the citizens in general; -it is guided by the search of legitimation, both for the policy itself and the policy makers.
• Policy making should be viewed as a "policy cycle", from the perception of a problem situation, to the design of policies, their legitimation, their implementation, their monitoring, their assessment etc.. Under such a perspective a policy cycle: -needs knowledge aiming at supporting the processes within it; -produces knowledge used both within the cycle and beyond it.
• Within a policy cycle several decision problems arise such as: Which aspects of the problem should be considered as more important? What information is relevant and should be used? Who are the principal stakeholders? Who else is affected by the situation and the possible policies? Which resources allocate and where, how and when? What matters in terms of potential consequences?
Decisions (of any type) result from combining factual information with subjective values, opinions and likelihoods. For this reason decisions are synonymous to responsibility. Under such a perspective there is nothing like an "objective decision". Decisions are taken by somebody and reflect his/her standpoint within a decision process. The consequence is that there will never exist an "objectively defined policy". Policies will always reflect what subjectively matters for those implied in the policy cycle.
• What could be considered as a legitimated source for values, opinions and likelihoods to be considered by the policy makers in presence of multiple stakeholders and multiple scenarios? The market? A referendum? A focus group? A public debate? A poll? Whatever we adopt we should remember that: -it is a subjective choice to privilege any of source of knowledge; -there exist different forms of participation (see [15]), the latter being orthogonal to the efficiency of the decision process (more participation implies less efficiency and viceversa); -the validity of any legitimation claim is a social construction, resulting from argumentation about facts, norms, values, sincerity and relevance.

24
The survey about the Evidence-Based Policy Making movement highlights a number of issues: -there has been a legitimated demand for allowing scientific knowledge, facts, statistics and other information sources to acquire a status within the policy cycle; -despite opposite declarations, there has been a clear trend in considering such "evidence" as the driving force in the process of designing policies, thus claiming for such evidence a status of "objective knowledge"; -such a trend contradicts the nature of the policy cycle both from a substantial point of view (knowledge is not objective, it is functional to some purpose) and from a procedural point of view (there exist many evidences with different possible interpretations, arbitrary used by the stakeholders within the policy cycle); -arguing about policies needs legitimated knowledge; it also produces knowledge which on its turn needs to become legitimated.
What all that tell us as decision analysts? We denote with this term the professionals/practitioners who are invited to enter the policy cycle as "experts" or as "technical staff" with the explicit or implicit role of producing the decision support knowledge within the cycle. We can summarise the type of demand decision analysts receive, in terms of producing information and knowledge aiming at aiding the stakeholders, the policy makers or the citizens to: -better understand the stakes in play; -better understand the potential consequences of potential actions; -better foresee potential unexpected/unwanted outcomes; -better justify, explain, argue about options, decisions and strategies; -design/construct/conceive new options behind the existing ones; -improve participation, inclusion and ultimately democracy.
In doing so decision analysts need to use existing information (facts, science, ground knowledge, best practices etc.), need to model constructively values, opinions and likelihoods for the stakeholders and need to do so in a meaningful, operational and legitimated way. We denote this set of skills under the term of "policy analytics": analytics (creating knowledge out of existing knowledge) aiming at supporting those specific type of decision processes occurring within a policy cycle.

Conclusion
Why designing, implementing and assessing public policies is so complicated? Why aiding to design, implement and assess public policies is so different with respect to other decision aiding skills used when the clients are business oriented and the problem situation does not concern public issued?
The aim of this paper was twofold. On the one hand we tried to understand why public policies represent a specific challenge for decision aiding. On the other hand we analysed the so-called "evidence-based policy making" literature since it represents the most recent attempt to focus on the relation between the policy making process and the technical, scientific, expert, analytical support that such a process demands. An attempt (important to note) originating from the clients' side (the policy makers) and not from the suppliers' side (the analysts or experts). The standpoint of our analysis has been clearly decision analytic. We do not underestimate the sociological or political dimension of these questions. We rather focus on the the challenges policy making offers to our discipline and profession.
The analysis of the so-called policy cycle shows that policy making is a decision making process with precise characteristics: long time horizon, de-facto participative nature, deliberative, accountable and legitimation driven. This is related to the specific nature of public policies which besides being deliberations of resource allocations create a legitimation space for stakeholders and citizens. If decision aiding is a process generating knowledge (possibly in an analytic form, but not only) to be used in a decision process, then it is clear that the knowledge required in order to support policy making processes needs to address such peculiar characteristics (for instance addressing the problem of legitimate knowledge or of legitimated argumentation).
Under such a perspective our paper shows that the evidence-based policy making approach, although originating from a legitimated demand, failed to address such challenges. The reason is basically the (naive) idea that evidence exists independently from policies and that once identified could "objectively" drive the policy cycle. Unfortunately this is not the case. However, the demand for using analytic information in order to support policy making remains valid, but needs to be addressed differently. This new frame (briefly presented in the paper) we call "policy analytics".