Laboratory Experiments in Economics: Coming of Age

Laboratory Experiments in Economics: Coming of Age

Shyam Sunder, Yale UniversityBarcelona LeeX Experimental Economics

Summer School in Macroeconomics

Universitat Pompeu Fabra

Barcelona, June 13, 2014

04/21/23 Design of Experiments 2

A Discipline Grows Up• Since Chamberlin reported the results of his classroom

experiment in 1948, the acceptability, recognition, role, and methods of this sub-discipline have evolved

• Unlike 1970s and 80s, when editors of economics journals routinely rejected experimental papers as a deviant curiosity, a recent issue of AER has more papers using experimental method than any other

• It is clear that the experimental method has grown beyond its “childhood” phase, is no longer “outside the tent”

• Being inside the tent brings responsibilities of “adulthood” for the sub-discipline?

Responsibilities• Identifying core concerns of the discipline on substantive, not just

methodological grounds

• Contribution to core concerns of a discipline

• Constructive interchange with sister methodologies of the discipline

• Balance between advancement of method and substantive knowledge of real phenomena

• Five special concerns

– Time

– Institutions

– Properties of institutions

– Is the experimenter a part of the game; subject expectations

– Lab results as the final word, and main source of research questions

– Risky curves

04/21/23 Design of Experiments 3

Core concerns of the discipline: substantive, not just methodological

• Disciplines get sterile when methods take the front seat, obscuring its classic or newly-identified substantive questions

• While methodological development is necessary part of a healthy discipline, the dominant concern must still be with a better understanding of our world we live in

• What proportion of the effort of the discipline goes into research about questions about the world (external references), and questions about research itself (internal references)?

• A simple test: try explaining your research question (and results) to your parents, and assess if they appreciate your contributions to human civilization

04/21/23 Design of Experiments 4

Contribution to core concerns of a discipline

• Where do we look for questions to address?– On the street, news, and observation of the world– Questions arising in the classroom (by students as well as in our own

minds) that we cannot answer to our satisfaction– Unresolved (perhaps abandoned) puzzles of the discipline– Incremental variations on recent publications– Proving your advisor or academic god-parent right

• Identifying core concerns of economics that could not be addressed without experiments

• What are the core concerns of economics and sister disciplines such as psychology? Do we need to distinguish among them? Does distinction mean un/willingness to learn from others?

04/21/23 Design of Experiments 5

Constructive interchange with sister methodologies

• Contribution of experimental method will also depend on how well we are able to take advantage of constructive interchange with sister methodologies of economics

• For example, economic theory and mathematical modeling

• What can be a constructive relationship between theory and experiments?

04/21/23 Design of Experiments 6

04/21/23 Design of Experiments 7

Assumptions

• Purpose of building models is to gain a better understanding of some real phenomena of interest

• Real phenomena are complex (perhaps infinitely detailed), rarely possible to understand/characterize them completely

• Theory identifies one or a few critical variables to gain a satisfactory (not perfect) understanding of the phenomenon of interest

• Theories are neither wrong nor right; some are more helpful than others in gaining insight

• Compare theories on basis of their help in understanding of the real phenomena

Infinite Detail: Fractalshttp://www.google.com/search?q=fractals&hl=en&client=safari&rls=en&prmd=imvns&tbm=isch&tbo=u&source=univ&sa=X&ei=jNDWT8LMIIzE8QPJlsCUAw&ved=0CG0QsAQ&biw=119

9&bih=600

04/21/23 Design of Experiments 8

04/21/23 Design of Experiments 9

Nature of Theory

• Essence of theory is its simplicity• Simplification by abstraction from details of real

phenomena• Assumptions perform the function of discarding

the mass of detail• Key assumptions and assumptions of convenience• Lack of correspondence between assumptions of

convenience and reality is the essence of a theory, and not a defect of theory (no assumptions, no theory)

04/21/23 Design of Experiments 10

Empirical Test of Theory

• Theory is to real phenomena what a drawing or stick figure is to human body, or map to earth surface

• Correspondence is crude, but captures some essential feature(s)

• Model identifies some tautologies which are necessarily true when assumptions hold (unless there are errors in derivation)

• What does it mean to empirically test a theory?

04/21/23 Design of Experiments 11

Single Theory Experiments

• Only one interesting theory is available for the phenomenon of interest

• “Test” is an assessment of robustness of the theory to deviations from assumptions of convenience

• If data are gathered from an environment that corresponds exactly to the assumptions of the theory, we should expect no deviations (if we do observe deviations, either the theory has error or the correspondence is missing)

• Empirical test is a costly method of discovering errors of derivation

04/21/23 Design of Experiments 12

Creating Theory in Lab

• Exact correspondence to theory in the field or lab is not easy

• Even if we could, little could be learned from it except about presence of errors of derivation or correspondence

• Error in derivation or lab environment or data collection

• Little useful scientific inference is possible from such mutual lack of correspondence

04/21/23 Design of Experiments 13

Scientific Value of A Single-theory Empirical Test

• Assessment of how robustly the predictions of the theory correspond to data as the environment of data becomes less similar to the convenience assumptions of the theory

• See Figure 1. – A is not robust, – B is highly robust, and – C lies in between.

• Under all three cases, the model is literally true (when all its assumptions hold). However, as the environment deviates from the strict assumptions, A’s predictive power declines sharply

04/21/23 Design of Experiments 14

Figure 1.

Single Theory Experiment

Pe

rce

nt

Co

rre

spo

nd

en

ce b

etw

ee

n t

he

Da

ta a

nd

Th

eo

ry

Pre

dic

tion

Distance between the Model and the Data Environments

A

C

B

100%

0

0

04/21/23 Design of Experiments 15

How Do We Identify Key the Assumptions?

• Distinguishing between model and theory• Model is a (“stick figure”) logical structure; a theory uses

the model to suggest some statements about the real phenomena of interest

• Think of the real phenomena that motivates the model and the theory

• Ask which assumptions of the model are intended to limit the real environments sought to be understood

• Number of states, preferences, probability distributions tend to be assumptions of convenience

04/21/23 Design of Experiments 16

Design of Robustness Experiments

• Conduct a series of experiments, all holding the key assumptions, and progressively relaxing the convenience assumptions (e.g., the number of states)

• Conduct a series of experiments progressively increasing the number of alternative choices available to subjects (increasing the number of possible outcomes)

• If model predictions are supported by data when more alternatives are available, result is more robust; e.g.,Vernon Smith (1962) paper

04/21/23 Design of Experiments 17

Fig. 3: Single Theory ExperimentSmith (1962)

04/21/23 Design of Experiments 18

Figure 2: Multi-Theory Experiments

Theory 1

X

Theory 2

Y

Theory 3

Z

Predictions

Results W

04/21/23 Design of Experiments 19

Fig. 4: Multi-Theory Experiment, Plott and Sunder (1982)

1. Time

• Most economic models include time dimension (usually denoted by symbol t

• Few models specify what t represents in real terms—seconds, hours, days, years, or generations

• Presumably, such theories are so general that they holds for all interpretations of the time interval in real units

• Lab experiments could be a way of finding the appropriate interpretations of time in specific theories, in case they exist, and thus make a significant contribution of economic theory

04/21/23 Design of Experiments 20

2. Institutions• Experimental methods have highlighted the importance of

economic institutions, their properties, and their evolution over time

• However, study of institutions in lab presents a special challenge

• Most individual decisions involve choice of a point on a function, but institutions being functions themselves, examination of their evolution calls for choices from a set of functions

• Choice on a function and of a function call for very different cognitive skills, experience, and time, and are difficult to study in the few hours of a typical session

04/21/23 Design of Experiments 21

3. Properties of Institutions

• Experiments have been employed to identify the properties of institutions

• Real life institutions have great deal of detail, and thus can be simplified for laboratory use in thousands of ways

• When we try to use experiments to identify institutional properties, how do we choose which implementation of the institution in the lab is appropriate?

04/21/23 Design of Experiments 22

4. Experimenter as a part of the game

• What are the boundaries of the game we hypothesize the subjects to be playing?

• What do we know about the expectations subjects bring to the lab? What, if any, control can we exercise on their expectations

• Is experimenter inside that boundary or outside?• How do we keep ourselves outside the boundary?• Is it enough to tell them so?

04/21/23 Design of Experiments 23

5. Lab results as the final word?

• When can we stop with the lab results, convinced that we have a good understanding of the phenomenon of interest?

• When do we need to follow up the lab results with data from the field?

04/21/23 Design of Experiments 24

6. Risky Curves• D. Bernoulli (1738) ---Von Neumann Morgenstern (1943): curved

utility (Bernoulli) functions to understand choice under risk combined with dispersion of outcomes as risk

• This idea (EUT) is widely accepted in the field; theorists devise new parameterized curves (e.g., CPT); experimenters devise protocols to elicit data and estimate the parameters

• Meager empirical harvest: little stability in parameters outside the fitted context; power to predict out of sample poor-to-nonexistent; no convincing victories over naïve alternatives; surprisingly little insight into phenomena outside the lab (insurance, security, labor, forex markets, gambling, business cycles, etc.)

– Very quick reviews (research through 1960; measuring individual risk preferences; aggregate level evidence from the field)

• Raise doubts; not sure of way forward, some possibilities– Alternative meanings/measures of risk– Looking for explanatory power in decision makers’ opportunity sets, real

options, and net pay-offs, instead of in unobserved curved Bernoulli functions– Current work in evolution, learning theory, neuroeconomics, and physiology

Empirical Failure of EU 25

04/21/23 Design of Experiments 26

Fundamental Principle of Research Designs (after Einstein)

• Research design should be as simple as possible, but no simpler.

04/21/23 Design of Experiments 27

Research Question

• What question do you wish to answer with your research?

• A question is one sentence with a question mark at the end (?).

• It should be a question whose answer you would like to know, but do not know

• After asking your friends, if you are the only one who does not know, think again, unless you have reasons to disagree with them

• What might the possible answers to the question?

• How could one distinguish what is a better answer?

Macro-Examples

• Robert Lucas and Edward Prescott

• What is the question?

• Why Experiment?

• What is essential?

• What is not essential?

• Robustness check

04/21/23 Design of Experiments 28