A Systems Approach to Establishing Scientific Integrity in Evidence Based Policy Making

A Systems Approach to Establishing Scientific Integrity in

Evidence Based Policy Making

Prof. Dr. Wijnand J. SwartCentre for Plant Health Management (CePHMa)

Faculty of Natural and Agricultural SciencesUniversity of the Free State

Bloemfontein South Africa

[email protected]

EVIDENCE in Policy Context • Sound, credible and robust evidence, whether :

– quantitative– qualitative– statistical– economic– attitudinal / behavioural– anecdotal– social– opinion based – or review based .....

......is an essential and necessary part of the enabling environment for formulating policies that are coherent and effective in terms of their outcomes.

Use GOOD data & information….

Source: DEFRA, UK

Use POOR data & information….

…and use it POORLY…

There are policies that:…and use it WELL…

The Policy Making Process• Techniques, analyses and judgements used to evaluate and

formulate data and information into knowledge / evidence for making effective policies are critical.

What is ‘Sound Evidence?’• Concept of “sound and credible evidence”

is very complex.

• Dependant on inter alia:– types– sources – generating techniques– context – ‘understanding’

...... of data / information / knowledge

DATA

KNOWLEDGE

INFORMATION

WISDOM

CO

NTE

XT

IND

EPEN

DEN

CE

UNDERSTANDING

understanding relations

understanding patterns

understanding principles

• Data are facts (e.g. numbers, names, symbols) and have little value in themselves.

• Information relates to description, definition, or perspective (what, who, when, where).

• Knowledge comprises strategy, practice, method, or approach (how).

• Wisdom embodies principle, insight, morals, or archetypes (why).

• Absolute Truth ?

TRUTH?

‘Sound Evidence’: Context & Understanding

Knowledge vs Science • Science is organized knowledge. Herbert Spencer

• Science: a knowledge of principles and causes.(Webster's Revised Unabridged Dictionary)

• Science is a way of thinking much more than it is a body of facts. Carl Sagan

• “Good Science” could be defined as those practices which contribute most to advances in understanding.

Sindermann: The Joy of Science 1985.

• Science is to see what everyone else has seen but think what no one else has thought. Albert Szent-Gyorgyi

Science and Policy Making• Creative and innovative contributions of

scientists and policy makers and the trust engendered in them by the public, to whom they are accountable is of paramount importance.

• Follows therefore that fostering an environment that promotes research or scientific integrity is an integral part of that accountability and the pursuit of new knowledge.

‘Scientific Integrity’• “Integrity” by definition:

– “honesty” – “a state of being entire or whole”

• Perspectives of ‘scientific integrity’ :

1. Ethical issues relating to misconduct (fraud) or manipulation, suppression, or distortion of facts.

2. Striving towards “wholeness” or excellence in the search for knowledge.

• Essential for maintaining scientific excellence and for keeping the public’s trust.

• Research integrity characterizes :

b. Institutional integrity: creating an environment that promotes responsible conduct and

high levels of integrity embracing standards of excellence, trustworthiness, and

lawfulness

c. Individual integrity: Scientist’s commitment to intellectual honesty and personal responsibility and is an aspect of moral character and experience. A good scientist must:

communicate well obtain research grants excel in teaching and mentoring engage in ethical decision making use knowledge ‘wisely’ to plan and execute research.

Research Integrity

Institutional Integrity:Politics vs Science

• Politicization of science as old as science itself e.g Galileo's theory that the Earth revolves around the sun perceived as a challenge to the authority of the Catholic church.

• Political interference threatens the integrity of government science and policy making all over the world.

• Manipulation, suppression, and distortion of government science misinforms public and leads to poor policy decisions.

• Especially rife in developing countries, e.g. the assertion of the South African president Thabo Mbeki that AIDS is not caused by HIV flew in the face of decades of research and threatened to undermine proper treatment of the disease.

Bush Administration's Misuse of Science • On February 18, 2004, 62 pre-eminent

scientists AND researchers released a statement titled Restoring Scientific Integrity in Policy Making in the USA.

• Scientists charged the Bush administration with widespread and unprecedented “manipulation of the process through which science enters into its decisions.”

• Scientists accused Bush administration of :

1. Epidemic altering and concealing of scientific information by senior officials in various federal agencies.

2. Active censorship of scientific information that the administration considered threatening to its own philosophies

3. Restriction of the ability of government-supported scientists to freely communicate scientific ideas related to "sensitive" issues .

Integrity of Research Institutions• The organizational structure and processes that typify the

mission and activities of a research institution can either promote or detract from the responsible conduct of research.

• These process are in part determined by the external environment and are influenced by the dynamics between and among organizational members.

• Any element or part of an organization can be viewed as a system in and of itself.

• External conditions influence the inputs into an organization, affect the reception of outputs from an organization’s activities, and directly affect an organization’s internal operations.

Source: National Academy of Sciences - http://www.nap.edu

Open systems model of internal environmental elements of a research organization showing: Inputs / resources for organizational functions Structures and processes that define an organization’s operation Outputs / outcomes of activities carried out by individual scientists, research

groups or teams, and other research-related programs.

• Interrelatedness between research organizations and the various external influences that have an impact on integrity in research.

• Systems and subsystems of the external-task environment are embedded within the general sociocultural, political, and economic environment.

• Relationships also exist between and among the elements within the external environment.

Source: National Academy of Sciences - http://www.nap.edu

Holistic Knowledge / Evidence• Rather than focus on the ethical or moral aspects of scientific

integrity, focus here is on the process of generating data and information and integrating it into sound knowledge (sound evidence) for decision-making.

• “Integrity” by definition:– “honesty” – “a state of being entire or whole”

• “Integrate” by definition:– “to combine parts into a whole”

“A whole.. is more than the sum of its parts.”

Jan Christian Smuts, Holism and Evolution

1926

• A collection of data is not information. • A collection of information is not knowledge. • A collection of knowledge is not wisdom.

• Information, knowledge, and wisdom are more than simply collections

• Each concept represents more than the sum of its parts and has a synergy of its own.

‘Sound Evidence’: A Holistic View

A “WHOLE” AS A SYSTEM• “ A system is defined as a set of interacting units with

relationships among them. The properties (or behaviour) of the system as a whole emerge out of the interaction of the components comprising the system.”

• The interactions of the parts become more relevant to understanding the system than understanding the parts.

• This definition of a system implies something beyond cause and effect.

CO

MP

LEX

ITY

NUMBER OF SUB-SYSTEMS

particleatom

moleculecell

organperson

communitystate

nationworldsolar system

galaxyuniverse

• In truth only one system, the “Universe"

• All systems are sub-systems of a larger system. ?

The Ultimate System

Systems Thinking and Policy

• Science is a way of thinking much more than it is a body of facts. Carl Sagan

• “Systems thinking” offers a conceptual framework or model for ‘thinking differently’.

• Systems thinking has permeated many scientific

fields including: education, business management, human development, sociology, psychology, agriculture, ecology and biology, earth sciences.

• Adopting a systemic perspective on solving policy problems therefore appear to offer a useful way of correcting these deficiencies.

• In 1960s, a ‘hard’ (quantitative) systems approach was touted as the policy science.

• However, hopes not realized for variety of reasons; its comprehensive modelling too information-intensive and mathematical.

• The ‘soft’ (qualitative) systems approach of systems thinking has increasingly been used since the 1990’s as a paradigm in policy planning and implementation.

• Soft systems methods stress the self-organizing and adaptive capacities of appropriately designed systems.

Hard vs Soft Systems

Soft systems methods

• Subjective (interpretive) philosophy• Systems + sociological theory base• Flexible methodology• Organizational problem-solving

focus• Creative / intuitive• Analyst is facilitator• Participative• Organizational learning outcomes• Several ambiguous outcomes

Hard systems methods

• Objective philosophy• Computer science + systems theory• Rigid method• Data, process, database technical

focus• Scientifically analytical• Analyst is expert• Analyst dominated• Computer design outcomes• One ‘correct’ solution

Hard vs Soft Systems

A hard systems view of a farming system; a biological network suitable for mathematical solution.

A soft systems view of a farming system; an arena for gaining experience and increased understanding.

Source: Robinson, B. 2003. 11th Australian Agronomy Conference)

Agro-ecosystem

Soft Systems Methodology (SSM)• Can be used both for general problem solving and in

the management of change.

• Used in the analysis of complex situations where there are divergent views about the definition of the problem — "soft problems" or policy options (e.g. How to improve health services delivery; How to manage disaster planning).

• At the heart of SSM is a comparison between the world as it is, and some models of the world as it might be. 

• Out of this comparison comes a better understanding of the world ("research"), and some ideas for improvement ("action").

SSM for Problem Solving

• Differences between models and reality become the basis for planning and policy making process.

reality

conceptual models

understanding and

improvement

• ‘Classic form’ of SSM consists of seven steps:– Problem unstructured by researchers – Problem situation expressed to capture “rich picture” – Create root definitions of relevant systems (i.e. social, political &

environmental) – Making and testing conceptual models based upon world views – Comparing conceptual models with reality – Identifying feasible and desirable changes – Acting to improve the problem situation

Multi-agent systems (MAS)• Policy increasingly has to address topics that have to do with

disequilibrium, dynamics, and locality.

• The overwhelming complexity of biophysical and socio-economic constraints that increasingly characterize rural areas in developing countries necessitates the development of more sophisticated tools to support policy making in these areas.

• Multi-agent systems (MAS) are a relatively new field in computer science that have been proposed as a modelling approach for establishing even higher levels of scientific integrity in the generation and evaluation of evidence for making policies.

• Analogous to artificial intelligence.

• Multi-agent models might be the preferred choice when heterogeneity and interactions of agents and environments are significant and policy responses cannot be aggregated linearly.

• MAS can thus complement bio-economic simulation models which cannot fully capture the heterogeneity in biophysical and socio-economic constraints and the interactions between them.

• There are several policy questions in the context of agricultural development of rural areas where MAS simulations may generate useful information for decision making on public investments in R&D and the targeting of policy interventions.

• Examples of such policy questions:– Should funds be spent on crop breeding for stress resistance

or in research for improved crop management?– Should micro-finance be promoted or should agricultural

inputs be subsidized?

MAS for Policy Making

• “An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through effectors”

• Agents may be persons, farms, markets, computer programmes or anything that is reactive, autonomous, and goal-oriented.

• Agents may have the ability to communicate with other agents, learning, mobility, and flexibility. May even have personality and show emotions!

AGENT

ENVIRONMENTINPUTOUTPUT

Agents

SYSTEM

SENSOREFFECTOR

Agent Flexibility• An intelligent agent is capable of flexible autonomous action.

• FLEXIBLE:– Reactive: A reactive system is one that maintains an

ongoing interaction with its environment, and responds to changes that occur in it (in time for the response to be useful)

– Pro-active: Generating and attempting to achieve goals; not driven solely by events; taking the initiative i.e. goal directed behavior; recognizing opportunities

– Social: Ability to interact with other agents via some kind of agent-communication language, and perhaps even cooperate with others.

Identifying a problem

“Improving” problem

MANAGEMENT

ANALYSISUnderstanding the

problem

Definition of the system

Maintaining the system

MANIPULATION

ANALYSIS

Understanding the system

REACTIVE PROACTIVE

POLICY POLICY

PROBLEM SOLVING

BIBLIOGRAPHY1. Balmann, A. 2000. Modeling land use with multi-agent systems: perspectives

for the analysis of agricultural policies.2. Berger, T. and Ringler, C. 2002. Trade-offs, efficiency gains and technical

change – Modeling water management and land use within a multiple-agent framework, Quarterly Journal of International Agriculture 41:119–144.

3. Berger, T., Schreinemachers, p., and Woelcke, J., 2006. Multi-agent simulation for the targeting of development policies in less-favored areas. Agricultural Syatems 88:28-43.

4. Checkland, P.  1981 Systems thinking, systems practice.  Chichester: Wiley.  5. Checkland, P., and Holwell, S. 1998  Information, systems, and information

systems: making sense of the field.  Chichester, UK: Wiley.   6. Checkland, P.  and Scholes, J.  1991 Soft systems methodology in action. 

Chichester: Wiley. 7. Harrison MI. 1994. Diagnosing Organizations: Methods, Models, and

Processes, 2nd ed. Thousand Oaks, CA: Sage.8. Schreinemachers, P. Berger, T. and Aune, J.B., 2007. Simulating soil fertility

and poverty dynamics in Uganda: A bio-economic multi-agent systems approach. Ecological Economics 64:387-401

9. Union of Concerned Scientists. 2004. Scientific integrity in policy making: An investigation into the Bush administration's misuse of science. Cambridge (Massachusetts): Union of Concerned Scientists; 49 pp.

10. Woolridge, M. 2002. An Introduction to Multiagent Systems by John Wiley & Sons (Chichester, England).

THANK YOU!