8/10/2019 Moral Responsibility Free Will - Florian Cova
1/35
Free will meta-analysis 1
RUNNING HEAD: FREE WILL META-ANALYSIS
Moral Responsibility and Free Will: A Meta-Analysis
Adam FeltzMichigan Technological University
And
Florian CovaSwiss Centre for Affective Sciences, University of Geneva
Word Count: 7,717 (excluding notes, abstract, and references)
Address Correspondence to:
Adam Feltz1400 Townsend DriveDepartment of Cognitive and Learning SciencesMichigan Technological UniversityHoughton, MI [email protected]
8/10/2019 Moral Responsibility Free Will - Florian Cova
2/35
Free will meta-analysis 2
Abstract
Fundamental beliefs about free will and moral responsibility are often thought to shape our
ability to have healthy relationships with others and ourselves. Emotional reactions have also
been shown to have an important and pervasive impact on judgments and behaviors. Recent
research suggests that emotional reactions play a prominent role in judgments about free will,
influencing judgments about determinisms relation to free will and moral responsibility.
However, the extent to which affect influences these judgments is unclear. We conducted a meta-
analysis to estimate the impact of affect. Our meta-analysis indicates that beliefs in free will are
largely robust to emotional reactions.
8/10/2019 Moral Responsibility Free Will - Florian Cova
3/35
Free will meta-analysis 3
Many philosophers and psychologists hold that at least a minimal belief in free will is required
for us to have healthy relationships with others and ourselves. Free will may be necessary for
autonomy, creativity, desert, reactive attitudes, dignity, love, and friendship (Kane, 1996).
However, recent advances in psychology and neuroscience may pose some threats to a belief in
free will. This research suggests that many people appear to be unaware of some of the
neurological antecedents of their behavior (Bargh & Ferguson, 2000; Wegner & Wheatley, 1999;
Wegner, 2002; Wegner & Bargh, 1998; Libet, 1985). One worry is that if these results were to
become widely assimilated, then a belief in free will would be diminished and the desirable
behaviors associated with a belief in free will would also disappear or be dramatically reduced.
For example, in the absence of belief in free will, we may have difficulty maintaining
meaningful relationships with others and interpersonal conflicts may become more common
(Kane, 1996). Empirical research supports these worries to some extent suggesting that beliefs infree will are linked to judgments about punishment (Rakos, Laurene, Skala, & Slane, 2008;
Carey & Paulhus, 2013). Moreover, belief in free will has been argued to be an important factor
for many commonly desirable behaviors such as refraining from cheating, self-control, and job
performance (Vohs and Schooler, 2008; Baumeister et al, 2007; Baumeister, Masicampo, &
DeWall, 2009; Stillman et al, 2010) and has been shown to influence some of the neurological
antecedents of behavior mentioned above (Rigoni et al., 2011). For these reasons, some take it
that belief in free will is so important and engrained that if we were to find out that people really
are not free or morally responsible, we should leave people to their mistaken beliefs (Smilansky,
2002). To disabuse people of their mistaken belief would create a world where nobody has any
of these things.
However, the extent to which advances in neuroscience and psychology call free will into
question or impact everyday conceptions of free will is still an open question (Roskies, 2006;
Mele, 2006, 2013). Belief in free will may be so engrained that it will be incredibly hard to
dislodge even in the face of extraordinary threats (Feltz, 2013; Feltz & Millan, in press). For
example, it appears as if many of the troubling findings from neuroscience have already been
assimilated in portions of the population. But this assimilation has not led to a reduction in
beliefs in the dualistic nature of humans or free will (OConnor & Joffe, 2013). Thus, one
possibility is simply that our ordinary understanding of free will is such that it can easily
accommodate the findings of neuroscience, rather than being at odds with them.
8/10/2019 Moral Responsibility Free Will - Florian Cova
4/35
Free will meta-analysis 4
Indeed, why should people be worried by the findings of neuroscience? One answer is
that they seem to promote a deterministic view of human behavior. Determinism has been
traditionally considered as a threat to human freedom and moral responsibility. Determinism is
the thesis that whatever happens, including human behavior, is entirely caused by previous
events and the laws of nature (Mele, 2006). It means that whenever one acts, that action is
completely the product of the laws of nature and events that took place earlier in ones life, and
those events are in turn completely the product of earlier events, eventually reaching events that
happened long before the person who acted was born. However, it is not clear that determinism
prevents free will and moral responsibility, and philosophers have divergent opinions on that
matter. Compatibilists hold that free will and moral responsibility are compatible with
determinism. Incompatibilists hold that if determinism is true, then we cannot have free will or
be morally responsible for our actions. Thus, if people hold a compatibilist view of free will, itcould be that recent findings of neurosciences do not threaten at all their view of themselves as
free and morally responsible agents.
In the past years, theorists have increasingly made use of empirical methods to study
laypeoples conceptions and intuitions about free will and moral responsibility with findings that
may appear somewhat contradictory. Sometimes people seem to have compatibilist intuitions
and sometimes they appear to have incompatibilist intuitions. 1 For example, when participants
are asked the abstract question if somebody can be free and morally responsible in a
deterministic world, most people respond no. However, if participants are asked if a concretely
described person (e.g., John murdered his wife and children so he could be with his lover) can be
free and morally responsible in a deterministic world, most people respond yes. To resolve this
apparent contradiction, some theorists have proposed that peoples fundamental judgments about
free will and moral responsibility tend to be influenced by negative emotional reactions (Nichols
& Knobe, 2007). In this paper, we survey the results of 30 published and unpublished studies and
submit them to a meta-analysis in order to estimate the extent to which purported negative
emotional reactions influence judgments about the freedom and moral responsibility of agents
living in a deterministic universe. We conclude that negative emotional reactions have some
1 Intuition is a term of art (see Feltz and Bishop, 2010). Here, we consider the intuition that p as an immediate judgment that p .
8/10/2019 Moral Responsibility Free Will - Florian Cova
5/35
Free will meta-analysis 5
impact on judgments about free will and moral responsibility, but this effect is not large enough
to play the theoretical role theorists have attributed to it.
Free will and affective reactions
Existing research concerning intuitions about determinisms relation to free will and moral
responsibility can be divided into two broad categories. 2 A few works investigate intuitions
about particular claims or cases that help inform whether free will and moral responsibility are
compatible with determinism (e.g., the Principle of Alternative Possibilities or manipulation
cases) (Miller & Feltz, 2011; Sripada, 2012; Feltz, 2013; Cova, forthcoming). However, the
majority of existing studies try to determine directly whether laypeoples conceptions of free will
and moral responsibility are prima facie compatible with determinism. Some of these latter
studies involve investigating folk concepts of free will and moral responsibility (Monroe &Malle, 2010, Stillman et al., 2011). Others address folk intuitions about the compatibility
question the question of whether free will and moral responsibility are compatible with
determinism (Sommers, 2010; Kane, 1996). In this paper, we focus on the studies that address
the compatibility question.
Conflicting answers to the compatibility question: the abstract/concrete asymmetry
Do people have the intuitions that an agent living in a deterministic universe can be free and
morally responsible (and therefore are natural compatibilists), or do they consider that
determinism precludes this agents free will and moral responsibility (and therefore are natural
incompatibilists)? Initial investigations concerning the compatibility question seemed to favor
the conclusion that people are mostly natural compatibilists (Feltz et al, 2009). For example,
Nahmias and his colleagues (2005, 2006) gave participants vignettes describing agents living in
deterministic universes and performing particular actions (such as robbing a bank). They then
asked participants whether these agents had free will and were morally responsible for their
actions. In three different studies, a majority of participants answered that these agents had free
will and were morally responsible for their actions. Given these results, it would seem tempting
2 Work using the empirical methods of the behavioral sciences to explore philosophically relevant beliefs sometimesis called experimental philosophy. For an overview of experimental philosophy, see Feltz (2009) and Cova(2011a).
8/10/2019 Moral Responsibility Free Will - Florian Cova
6/35
Free will meta-analysis 6
to conclude that participants tended to think that free will and moral responsibility are
compatible with determinism.
However, things are not that simple. Nichols and Knobe (2007) designed an experiment
in which participants were introduced to the following description of universe A:
Imagine a universe (Universe A) in which everything that happens is completely caused by
whatever happened before it. This is true from the very beginning of the universe, so what
happened in the beginning of the universe caused what happened next, and so on right up until
the present. For example one day John decided to have French Fries at lunch. Like everything
else, this decision was completely caused by what happened before it. So, if everything in this
universe was exactly the same up until John made his decision, then it had to happen that John
would decide to have French Fries.
Participants were divided into two conditions. After reading the description, participants in the
concrete condition received the following additional paragraph and question:
In Universe A, a man named Bill has become attracted to his secretary, and he decides that
the only way to be with her is to kill his wife and 3 children. He knows that it is impossible to
escape from his house in the event of a fire. Before he leaves on a business trip, he sets up a
device in his basement that burns down the house and kills his family. Is Billy fully morally responsible for killing his wife and children?
In this case, most participants (72%) answered that Billy was fully morally responsible for
killing his wife and children. These results are perfectly consistent with the hypothesis that most
laypeople are natural compatibilists. However, participants in the abstract condition did not
receive any additional paragraph but only the following question:
In Universe A, is it possible for a person to be morally responsible for their actions?
In this condition, most participants (86%) answered that it was not possible for this person to be
fully morally responsible. This pattern of responses conflicts with participants answers in the
concrete condition . Lets call this the abstract/concrete asymmetry. The abstract/concrete
8/10/2019 Moral Responsibility Free Will - Florian Cova
7/35
Free will meta-analysis 7
asymmetry suggests that participants answers in the abstract and concrete conditions are not
based on the same psychological processes (Sinnott-Armstrong, 2008; Weigel, 2011). However,
there is wide disagreement about what those processes are, and which of the two answers (if
any) should be considered as revealing participants true conception of free will.
Competing accounts of the abstract/concrete asymmetry
One influential explanation of the abstract/concrete asymmetry is Nichols and Knobe (2007)s
affective performance error model . According to Nichols and Knobe (2007), the key difference
between the abstract and the concrete condition is the amount of emotional reaction generated by
the vignettes. The concrete condition depicting a horrendous murder can be thought as more
upsetting than the abstract condition. For Nichols and Knobe, this is what explains the different
intuitions between the abstract and the concrete conditions. While people tend to think that freewill and moral responsibility are incompatible with determinism (hence the answers in the
abstract condition), strong emotional responses can bias participants to attribute moral
responsibility in the concrete condition. Thus, results indicating that participants are natural
compatibilists would be performance errors from participants. Since compatibilist intuitions
result from an error, they could not be used to infer what the folk truly think about the
relationship between moral responsibility and determinism.
Though influential, Nichols and Knobes affective performance error model is not the
only available account of the abstract/concrete asymmetry. Another account simply relies on the
possibility that the abstract and the concrete conditions lead to different understandings and
interpretations of human agency in Universe A . Nahmias and Murray (2010) hold that most
people are natural compatibilists, and that they have the intuition that agents living in
deterministic universes can be free and morally responsible for their actions. However, they
stress that determinism should be distinguished from bypassing. Bypassing occurs when
agents mental states do not play a role in the production of those agents actions, so that agents
will end up acting the way they do whether they want it or not. Clearly, one can think free will
and moral responsibility to be compatible with determinism while thinking that they are
incompatible with bypassing. Determinism does not entail that an agents mental states are
bypassed or irrelevant to the production of the action. Based on this distinction, Nahmias and
Murray (2010) argue that Nichols and Knobes depiction of Universe A (in which things had to
8/10/2019 Moral Responsibility Free Will - Florian Cova
8/35
Free will meta-analysis 8
happen the way they did) could lead participants to understand that agents mental states are
bypassed. This would explain why participants judge agents not to be morally responsible for
their actions in the abstract condition in spite of their natural tendency to consider free will and
moral responsibility to be compatible with determinism. However, in the concrete condition, it is
explicitly stated that the agent acts the way he does on the basis of his desires (e.g. he wants to
be with his secretary) and beliefs (e.g. he knows that it is impossible to escape from his house in
the event of a fire), which would lead participants to revise their interpretation and to understand
that agents in Universe A, though determined to act the way they do, are not bypassed. This
would explain why the agent is judged more morally responsible in the concrete condition.
Empirical evidence supports this hypothesis, suggesting that participants are indeed more likely
to consider that agents mental states are bypassed in the abstract condition than in the concrete
condition (Murray & Nahmias, forthcoming; but see Rose & Nichols, in press).Finally, a third account focuses on another particular feature of the concrete condition:
the fact that a norm is broken . According to the NBAR hypothesis (where NBAR stands for
Norm Broken, Agent Responsible; see Mandelbaum & Ripley, 2012), people are natural
incompatibilists. This natural tendency accounts for participants answers in the abstract
condition. However, people also have the unconscious belief that whenever a norm is broken, an
agent is responsible for breaking the norm. In the concrete condition where a norm is broken,
this unconscious belief counters our natural tendency to judge free will and moral responsibility
to be incompatible with determinism, leading to an increase in judgments that the agent is
morally responsible.
Testing the affective performance error model: the high/low affect asymmetry
Thus, there are many possible accounts of the abstract/concrete asymmetry. What reasons do
Nichols and Knobe give us to prefer the affective performance error model? According to them,
their theory makes the following prediction: the same difference found between the abstract and
the concrete conditions can be found between two concrete conditions as long as affect is varied
in a similar way. To test for this hypothesis, they ran a second study in which participants
received the description of Universes A and B. Then participants received only one of the
following two pairs of sentences:
8/10/2019 Moral Responsibility Free Will - Florian Cova
9/35
Free will meta-analysis 9
Low Affect condition:
As he has done many times in the past, Mark arranges to cheat on his taxes. Is it
possible that Mark is fully morally responsible for cheating on his taxes?
High Affect condition:
As he has done many times in the past, Bill stalks and rapes a stranger. Is it possible
that Bill is fully morally responsible for raping the stranger?
For each condition, half of the participants were told that Mark (or Bill) lived in deterministic
Universe A, while the other half were told that he lived in indeterministic Universe B. Results
(presented in Table 1) suggest that Nichols and Knobe were right. More participants considered
the agent responsible in the high affect condition than in the low affect condition when theaction was set in deterministic Universe A.
--- Insert table 1 here ---
Lets call this peculiar pattern of responses the high/low-affect asymmetry . The high/low-
affect asymmetry seems to provide support for Nichols and Knobes account of the
abstract/concrete asymmetry. Indeed, it seems that the difference between the high and low
affect cases cannot be accounted by the fact that the first makes it clearer that the agent acts on
the basis of his own desires and beliefs: in both cases, there are no direct references to the
agents mental states. Nor does it seem that the difference can be explained by the fact that a
norm is broken in the first case and not in the second: in both cases, it is clear that a norm has
been broken. Rather, it seems that the only difference between the high and low affect cases is
that the first features a much more gruesome and indignation-arousing violation than the second.
The existence of this high/low-affect asymmetry thus lends important support to Nichols and
Knobes affect-based account of the abstract/concrete asymmetry. The affect-based account has
led many to agree that emotional reactions could bias peoples judgment about free will and
moral responsibility and to speculate about what that means for our ordinary conception of free
will (e.g. Nelkin, 2007; Vargas, 2009).
Replications in experimental philosophy of free will: the trouble with the high/low affect
asymmetry
8/10/2019 Moral Responsibility Free Will - Florian Cova
10/35
Free will meta-analysis 10
In a recent paper criticizing the methodological shortcomings of many empirical investigations
of folk intuitions about philosophically relevant topics, Woolfolk insisted on the fact that
replicability of research findings is a key to the establishment of scientifically sound inquiry
(2013, p. 84). There have been recent worries that most surprising results in psychology, and
particularly in social psychology, might not pass the test of replication, and these have led to a
demand for more replications (e.g., Young, 2012). 3 For these reasons, scientific responsibility
counsels that before speculating on what could cause the effects we described, we should make
sure that these effects are robust.
To what extent have the results we described been replicated? In the previous section, we
have described and distinguished two different phenomena: the abstract/concrete asymmetry ,
and the high/low-affect asymmetry . The abstract/concrete asymmetry (i.e., participants ascribe
less moral responsibility to agents living in a deterministic universe when the question is askedabstractly rather than concretely) has been widely reproduced. It has been reproduced using
different descriptions of the deterministic universe (Cova et al. 2012), different concrete
vignettes (Nahmias et al., 2007; Murray & Nahmias, forthcoming), cross-culturally (Sarkissian
et al, 2010), and even when the agent is forced to act by a particular neurological condition (De
Brigard et al., 2009).
The abstract/concrete asymmetry seems a robust result that deserves an explanation.
However, the same cannot be said of the high/low-affect asymmetry. So far, the dramatic
difference Nichols and Knobe found between the low-affect and the high-affect cases has not
been properly replicated in a published study. The only published paper to directly attempt a
replication failed twice and found both times that participants gave mostly incompatibilist
answers in both cases (Feltz et al., 2009). A similar effect has been found by Cova and his
colleagues (2012), but instead of comparing Nichols and Knobes low-affect and high-affect
cases, they compared the low-affect case to Nichols and Knobes concrete case, which differs in
many respect from the low- and high-affect cases (for example, the concrete condition puts more
emphasis on the agents desires and the role they play in the production of his action, which may
3 For example, Scaife and Webber (2013) found puzzling results about folk intuitions about intentional action, butfurther studies repeatedly failed to replicate these results (Cova, in press). See also Sayedsayamdost (in prep) andChristian Motts replication page: http://pantheon.yale.edu/~jk762/xphipage/Experimental%20Philosophy-Replications.html, and the Psych File Drawer project: http://psychfiledrawer.org.
8/10/2019 Moral Responsibility Free Will - Florian Cova
11/35
Free will meta-analysis 11
explain the difference between the two cases). Thus, it is not clear that the high/low-affect
asymmetry is robust or real (i.e., early findings may reflect Type I error).
This lack of replication is all the more worrying because there are reasons to doubt that
the impact of affective reactions can explain why most participants consider agents morally
responsible for their actions in the concrete cases. First, Nahmias and his colleagues used
concrete vignettes involving neutral actions such as going jogging (Nahmias et al., 2006) or
positive actions such as giving money to charity (Nahmias et al., 2006, 2007), and still found
that most participants judged the agent morally responsible for his actions. Second, in a recent
study, Cova et al. (2012) gave various concrete cases to patients suffering from a behavioral
variant of frontotemporal dementia, a neurodegenerative disease accompanied by a deficit in
emotional responses. Contrary to what the affective performance error model would have
predicted given their lack of emotional reactions, these patients were no more incompatibilistthan control participants and gave mostly compatibilist answers.
However, there are also reasons to think that affective reactions do have an influence on
judgments about free will and moral responsibility. First, one source of evidence comes from a
series of studies suggesting that extraverts are more likely than introverts to judge an agent
living in a deterministic universe is morally responsible for his actions (Feltz & Cokely, 2009;
Schulz et al., 2011; Feltz, Perez, & Harris, 2012; Feltz & Millan, in press). A possible
explanation for this phenomenon is that extraverts are less likely to regulate their own emotions,
and thus are more susceptible to be influenced by the affective content of vignettes. Results from
neuroscience support this to some extent suggesting that some individuals are more likely to
regulate emotional reactions than others (Ochsner & Gross, 2005; see also Smillie, 2013).
Second, Feltz and his colleagues (2012) found that the affective content of a vignette could
influence the type of explanation participants give for the agents behavior. Participants faced
with a high-affect vignette were more likely to explain the agents behavior in terms of the
agents decision than participants reading a low-affect vignette. Given that the same study found
that participants explaining the agents behavior in terms of decisions were also more likely to
perceive him as free and morally responsible, this suggests that the affective content of a
vignette can indeed influence participants judgments by favoring one kind of explanation for
agents behaviors over another. Third, other evidence comes from psychological studies
8/10/2019 Moral Responsibility Free Will - Florian Cova
12/35
Free will meta-analysis 12
showing that inducing anger in participants can increase their propensity to punish and ascribe
moral responsibility to agents (Keltner et al., 1993; Tetlock et al., 1998).
Consequently, it is not clear whether negative affective reactions have an impact on
compatibilist or incompatibilist judgments. Even if affect does influence judgments, it is still
unclear the extent to which the abstract/concrete asymmetry can be explained by this impact. To
help address these issues, and determine whether the high/low-affect asymmetry is a genuine
and replicable effect, a meta-analysis was conducted.
Meta-Analysis
Search Criteria
We used the following criteria for including studies in the meta-analysis: (1) the study included a
description of determinism. This narrowed the group of many possible studies becausedeterminism, as philosophers understand it, is a precise, technical concept. For example, this
criterion excluded a number of studies that used scenarios suggesting, but not explicitly
describing, determinism. It also excluded a number of studies where researchers inferred
compatibilist and incompatibilist judgments absent a description of determinism. (2) The study
manipulated the emotional content of the scenarios. Some studies only had high or low affect
scenarios. Because we were interested in the effect of affect, any study that did not manipulate
the emotional content of the scenario was excluded.
Search for Studies
The effect of affect was first identified by Nichols and Knobe (2007). Because they identified the
effect of interest, we first used a Google Scholar Cited Reference search to find all papers that
referenced Nichols and Knobe (2007) for possible inclusion in the meta-analysis. This method
returned 220 results. Computer based database searches were also conducted. Databases included
in the search were PsychInfo and Philosophers Index. Keywords determinism, free will, and
emotion were used in all database searches. This method returned a total of 3,254 results. Pro
Quest Psychology Journals, PsychArticles, Sage Full Text, Science Direct, Web of Science, and
Wiley Online databases were also searched returning no new papers that were not identified in
the PsychInfo and Philosophers Index search. We also conducted a search for unpublished
studies by posting calls for unpublished studies on discipline specific blogs. Additionally, we
8/10/2019 Moral Responsibility Free Will - Florian Cova
13/35
Free will meta-analysis 13
contacted individuals who had conducted previous studies to see if they also had any
unpublished studies. Finally, we emailed relevant research groups. References of each paper that
met the inclusion criteria were searched for possible inclusion in the meta-analysis. The search
started in late 2012 and concluded in late 2013.
Results were then examined to determine if they met the two inclusion criteria. The
abstracts of the papers were read first. Then, if the abstract indicated that the two inclusion
criteria were likely to be met, the entire paper was read. Both authors agreed on which studies to
include. There were no disagreements about what studies were to be included. No results that did
not reference the original Nichols and Knobe paper met the inclusion criteria. Eleven published
studies met both inclusion criteria. We also included a number of unpublished studies ( K = 19).
--- Insert table 2 here ---
Variables in Studies
Table 2 lists and includes a brief description of the 30 studies. There were a number of
differences between the 30 studies. First, two different experimental designs were usedeither a
within-subjects design where participants received both high affect and low affect scenarios or a
between-subjects design where participants received only one scenario. Second, studies gathered
either categorical (yes/no) or continuous (Likert scale) data. Third, the number of questions
asked varied. Studies that gathered categorical data only asked one question (e.g., Is it possible
for Bill to be fully morally responsible for cheating on his taxes?). Studies that gathered
continuous data typically asked more than one question. However, since participants answers to
multiple questions often had strong internal consistency, often a composite score was reported
(the mean of the responses). We used only composite scores from the studies that reported
continuous data in the meta-analyses. Fourth, there were three distinct types of scenarios used,
but all studies but one used the original Nichols and Knobe (2007), the Nahmias, Coates, and
Kvaran (2007) scenarios, or a close variation of either. The remaining study used a variation of a
Nahmias, Morris, Nadelhoffer, & Turner (2006) scenario. Finally, the action in the high affect
case varied. In one high affect scenario, a man is described as stalking and raping a stranger. In
the other variation, a man is described as falling in love with his secretary and the only way to be
with her is to kill his wife and children, and he does it.
8/10/2019 Moral Responsibility Free Will - Florian Cova
14/35
Free will meta-analysis 14
Results
To give a sense of the overall data, all effect sizes were converted to a form comparable to the
standardized mean difference and are included in the funnel plot in Figure 1. This overall
analysis needs to be interpreted with caution because it is not necessarily permissible to combine
different types of data from different kinds of experimental designs in the same meta-analysis
(see below). A visual inspection of the funnel plot indicated that there was no publication bias.
This was likely because a relatively large number of unpublished studies were included. For all
meta-analyses, we used the methods described in Lipsey & Wilson (2001). For the overall meta-
analysis, within-subjects (proportion gain) and between subjects (odds-ratios) effect sizes for
dichotomous data were converted to a form ( logit d ) similar to the standardized mean difference
for comparison. Because larger sample sizes tend to be more representative of the population, we
weighted the effect size of each study by the inverse variance weight. 4 This method allowsstudies with a larger sample size more statistical importance than studies with smaller sample
sizes in the meta-analysis. For the overall meta-analysis, a random effect model was used
because homogeneity was rejected ( Q (29) = 49.78, p = .009) and revealed a small, statistically
significant standardized mean difference .15 (95% CI: 0.08, 0.22), Z = 4.12, p < .001. The test
for homogeneity also suggested that there were important differences between types of studies.
Because the studies differed in empirically and conceptually important ways, subsequent
analyses were performed.
--- Insert figure 1 here ---
It is controversial whether combining data from different experimental designs (e.g.,
between-subjects and within-subjects) is legitimate. 5 However, we performed a meta-analysis
4 The inverse variance weight is the inverse of the squared standard error value (Lipsey & Wilson, 2001, p. 36).For example, the inverse variance weight for the standardized mean gain is 1/(Standard error of the standardizedmean difference).5 There is no clear answer when combining data from different kinds of designs is permissible. For example, Lipseyand Wilson unequivocally state the standardized mean gain effect size statistic is different from the standardized
mean difference effect size statistic Comparison of the previous effect size statistics with thosethe standardizedmean difference should make it evident that they cannot be expected to yield comparable values. It follows thatthese two effect size statistics should not be mixed in the same meta-analysis (2001, p. 45). Others are lessconcerned with the differences, at least in practice (e.g., Eagly, Makhijani, & Klonsky, 1992). Others opt for a moremoderate position stating that in some instances data from different designs can be compared, but only if someconditions are met. These conditions include (1) putting effect sizes meaningfully in the same metric, (2) the designsdo not generate relevantly different biases, and (3) the designs estimate the effects with an acceptably similar
precision (Morris & DeShon, 2002). The current studies pretty clearly fail condition 2 since the within-subjectsdesign is likely to generate a bias toward consistency, especially in the short time-frame in which participants wereasked to respond. It is questionable whether the current studies satisfy conditions 1 and 2. Morris and DeShon
8/10/2019 Moral Responsibility Free Will - Florian Cova
15/35
Free will meta-analysis 15
combining each type of data from both types of experimental designs (i.e., within-subjects
(studies 1-11) and between-subjects (studies 12-30). Each of these meta-analyses used the data
from overall meta-analysis above. The meta-analysis combining all categorical data (studies 1-
11) indicated that homogeneity should be rejected Q (10) = 19.53, p = .03. A random effect
model indicated that the overall effect size was small .1 (95% CI: -.02, .22), and not significant Z
= 1.64, p = .1. However, caution must be taken in interpreting the result from this meta-analysis
of categorical data. The rejection of homogeneity suggested that there was variability in the
effect sizes greater than could be expected with subject-level sampling error. This variability
could in principle be accounted for. One obvious difference in the effect sizes was the different
experimental designs. An analogue of the analysis of variance (ANOVA) was performed using
the experimental design as the moderator variable to test whether differences were a function of
experimental design (Lipsey & Wilson, 2001). They were ( Q B (1) = 9.1, p = .003). Therefore,there are good conceptual and empirical reasons not to include all categorical data in the same
meta-analysis.
For these reasons, two separate meta-analyses were performed for the experiments that
gathered categorical data. Within-subjects categorical data studies (studies 1-4) were analyzed as
one group, and between subjects, categorical data studies (studies 5-11) were meta-analyzed as a
separate group. Inverse variance weights were used in the meta-analyses. For within-subjects,
categorical data studies, a fixed effect model of the proportion gain effect size was used for the
meta-analysis because homogeneity could not be rejected ( Q (3) = 0.04, p = .99). The mean
effect size (proportion gain) was .002 (95% CI: -0.44, 0.44) and was not statistically significant Z
= .009, p = .99. For between subjects, categorical data studies, the odds-ratios effect sizes were
converted to their natural logarithm. A fixed effect model was used because homogeneity could
not be rejected ( Q (6) = 5.6, p = 0.47). The mean effect size (odds-ratio) was small 1.70 (95%
CI: .97, 2.61) but was statistically significant Z = 2.4, p = .02.
--- Insert figure 2 here ---
--- Insert table 3 here ---
---Insert table 4 here ---
(2002) also offer an empirical method to determine the acceptability of combining data across different designs. Ifthe analogue of the ANOVA that uses the type of design a moderator is significant, then those data should not becombined into the same meta-analysis.
8/10/2019 Moral Responsibility Free Will - Florian Cova
16/35
Free will meta-analysis 16
An overall meta-analysis was also conducted combining all continuous data from each
experimental design (studies 12-30). This meta-analysis indicated that homogeneity should be
rejected Q (18) = 32.59, p = .02. A random effect model indicated that the overall effect size was
small .15 (95% CI: .07, .22) and significant Z = 3.62, p < .01. The rejection of homogeneity
again suggested this variability could in principle be accounted for by the differences in
experimental designs. The analogue of the ANOVA indicated that the experimental design was a
factor in the difference Q B (1) = 6.81, p = .001. Again, there were good conceptual and empirical
reasons not to include all continuous data in the same meta-analysis.
Within-subjects, continuous data studies (studies 12-16) and between-subjects,
continuous data studies (studies 17-30) were analyzed as two separate groups. Means and
standard deviations are reported in Tables 5 and 6 (see Figure 3 for a Forest Plot). Standardized
mean differences (between-subjects, continuous data studies) and standardized gain scores(within-subjects, continuous data studies) and inverse variance weights were calculated. For
within-subjects, continuous data studies, a random effect model was used because homogeneity
was rejected Q (4) = 10.32, p = .04. Given the low number of studies analyzed and the absence
of a priori predictions about the source of the heterogeneity, we assumed that the variability
beyond subject-level sampling was random. Hence, we did not attempt to identify the source of
the heterogeneity. However, it is possible that there is some identifiable source that can account
for the heterogeneity. A random effect model suggested that the mean effect size (standardized
mean gain) was small .08 (95% CI: -.02, .17) and not statistically significant Z = 1.52, p = .13.
For between-subjects, continuous data studies, a fixed effect model was used because
homogeneity could not be rejected ( Q (13) = 15.46, p = 0.28). The mean effect size (standardized
mean difference) was small 0.22 (95% CI: .12, .32) but was statistically significant Z = 4.47, p
% & ! ( ) &
!*"+,"-,()&, .//&0* !()& "+, 123 45
8/10/2019 Moral Responsibility Free Will - Florian Cova
35/35
Free will meta-analysis 35
Figure 2: Forest Plot, Effect Sizes Converted to be Comparable to Cohens d