DIT2: Devising and Testing a Revised Instrument of …dnarvaez/documents/DIT2pub.pdfDIT2: Devising and Testing a Revised ... Psychology aimed at improving the measurement of ... Department

Journal of Educational Psychology1999, Vol. 91, No. 4,644-659

Copyright 1999 by the American Psychological Association, Inc.0022-0663/99/$3.00

DIT2: Devising and Testing a Revised Instrument of Moral Judgment

James R. Rest and Darcia NarvaezUniversity of Minnesota, Twin Cities Campus

Stephen J. ThomaUniversity of Alabama, and University of Minnesota,

Twin Cities Campus

Muriel J. BebeauUniversity of Minnesota, Twin Cities Campus

The Denning Issues Test, Version 2 (DIT2), updates dilemmas and items, shortens the originalDefining Issues Test (DIT1) of moral judgment, and purges fewer participants for doubtfulresponse reliability. DIT1 has been used for over 25 years. DIT2 makes 3 changes: indilemmas and items, in the algorithm of indexing, and in the method of detecting unreliableparticipants. With all 3 changes, DIT2 is an improvement over DIT1. The validity criteria forDIT2 are (a) significant age and educational differences among 9th graders, high schoolgraduates, college seniors, and students in graduate and professional schools; (b) prediction ofviews on public policy issues (e.g., abortion, religion in schools, rights of homosexuals,women's roles); (c) internal reliability; and (d) correlation with DIT1. However, the increasedpower of DIT2 over DIT1 is primarily due to the new methods of analysis (a new index calledN2, new checks) rather than to changes in dilemmas, items, or instructions. Although DIT2presents updated dilemmas and smoother wording in a shorter test (practical improvements),the improvements in analyses account for the validity improvements.

The Defining Issues Test, Version 2 (DIT2), is a revisionof the original Defining Issues Test (DIT1), which was firstpublished in 1974. DIT2 updates the dilemmas and items,shortens the test, and has clearer instructions. This is thethird in a series of articles in the Journal of EducationalPsychology aimed at improving the measurement of moraljudgment (Rest, Thoma, & Edwards, 1997; Rest, Thoma,Narvaez, & Bebeau, 1997). Rest, Thoma, and Edwards(1997) proposed an operational definition of constructvalidity (seven criteria) that could be used to evaluatevarious measurement devices of moral judgment. Rest,Thoma, Narvaez, et al. (1997) reported that a new way ofindexing DIT data, the N2 index, had superior performanceon the seven criteria in contrast to the traditional P index,

James R. Rest, Department of Educational Psychology andCenter for the Study of Ethical Development, University ofMinnesota, Twin Cities Campus; Darcia Narvaez, College ofEducation and Human Development and Center for the Study ofEthical Development, University of Minnesota, Twin Cities Cam-pus; Stephen J. Thoma, Department of Human Development,University of Alabama, and Center for the Study of EthicalDevelopment, University of Minnesota, Twin Cities Campus;Muriel J. Bebeau, Department of Preventive Science and Center forthe Study of Ethical Development, University of Minnesota, TwinCities Campus. James R. Rest died in July 1999.

We thank Lee Fertig, Irene Getz, Carol Koskela, ChristyanMitchell, and Nanci Turner Shults for help in data collection. Wealso thank the Bloomington School District and the Moral Cogni-tion Research Group at the University of Minnesota.

Correspondence concerning this article should be addressed toDarcia Narvaez, Department of Educational Psychology, Univer-sity of Minnesota, 206 Burton Hall, 178 Pillsbury Drive Southeast,Minneapolis, Minnesota 55455. Electronic mail may be sent [email protected].

which has been used for over 25 years. (See the Rest,Thoma, Narvaez, et al., 1997, article for further discussionof N2.) This article reports on a revised version (newdilemmas and items) of the DIT1—the DIT2—with morestreamlined instructions and shorter length. Also, this articledescribes a new approach to detecting bogus data ("newchecks").

While we were reexamining aspects of the DIT1, we alsoreconsidered our methods of checking for participant reliabil-ity. That is, given a multiple-choice test that can be groupadministered—often under conditions of anonymity—someparticipants might fill out the DIT1 answer sheet withoutregard to test instructions, and some participants might givebogus data. The participant reliability checks are methodsfor detecting bogus data. For the past decades, we have useda procedure called "standard checks" to check for bogusdata. In sum, DIT2 uses new checks instead of standardchecks and uses revised items and dilemmas as well as N2.With these three changes, we wanted to see whether theresearch dividends would increase by creating alternatives toDIT1, P index, and standard checks.

However, we had an important question to considerbefore getting into the matter of updating: Why wouldanyone want a DIT score, either updated or not? Two issuesare at the heart of the matter. First, is Kohlberg's approach soflawed that research ought to start anew? Second, can amultiple-choice test like the DIT (as opposed to interviewdata) yield useful information?

The Kohlbergian Approach

The DIT is derived from Kohlberg's (1976, 1984) ap-proach to morality. In the past decades, many challenges tothis approach have been made. Critics raise both philosophi-

644

DIT2 645

cal and psychological objections. In a recent book (Rest,Narvaez, Bebeau, & Thoma, 1999), the criticisms andchallenges to Kohlberg's theory are reviewed and analyzed.In contrast to those who find Kohlberg's theory so faulty thatthey propose discarding it, we have found that continuingwith many of Kohlberg's starting points has generatednumerous findings in DIT research.

To appreciate how Kohlberg's basic ideas illuminateimportant aspects of morality, first consider a distinctionbetween macromorality and micromorality. Just as in thefield of economics, macro and micro distinguish differentphenomena and different levels of abstraction in analysis,we use the terms to distinguish different phenomena andlevels of analysis in morality. Macromorality concerns theformal structure of society, that is, its institutions, rolestructure, and laws. The following are the central questionsof macromorality: Is this a fair institution (or role structureor general practice)? Is society organized in a way thatdifferent ethnic, religious, and subcultural groups can coop-erate in it and should support it? Should I drop out of acorrupt society? On the other hand, micromorality focuseson the particular, face-to-face relationships of people ineveryday life. The following questions are central to micro-morality: Is this a good relationship? Is this a virtuousperson? Both micro- and macromorality are concerned withestablishing relationships and cooperation among people.However, micromorality relates people through personalrelationships, whereas macromorality relates people throughrules, role systems, and formal institutions. In macromoral-ity, the virtues of acting impartially and abiding by general-izable principles are praised (for how else could strangersand competitors be organized in a societal system ofcooperation?). In micromorality, the virtues of unswervingloyalty, dedicated care, and partiality are praised, becausepersonal relationships depend on mutual caring and specialregard. In our view, Kohlberg's theory is more pertinent tomacromorality than to micromorality (for further discussionof macro- and micromorality, see Rest et al., 1999). Some ofKohlberg's critics fault his approach for not illuminating"everyday" morality (in the sense of micromorality; seeKillen & Hart, 1995). However, it remains to be seen howwell other approaches accomplish this.

The issues of macromorality are real and important,regardless of the relative contributions of a Kohlbergian ornon-Kohlbergian approach to issues of micromorality. Re-garding the importance of macromorality issues, considerMarty and Appleby's (e.g., 1991) six-volume series oncurrent ideological clashes in the world. Marty and Applebytalked about the world's major disputes since the cessationof the Cold War. Formerly, the Soviet Union and Marxism/communism seemed to be the greatest threats to democra-cies. However, Marty and Appleby characterized the majorideological clash today as between fundamentalism andmodernism; others describe the clash in ideology as the"culture war" between orthodoxy and progressivism (Hunter,1991) or religious nationalism versus the secular state(Juergensmeyer, 1993). These clashes in ideology lead "tosectarian strife and violent ethnic particularisms, to skir-mishes spilling over into border disputes, civil wars, and

battles of secession" (Marty & Appleby, 1993, p. 1).Understanding how people come to hold opinions aboutmacromoral issues is now no less important, urgent, and realthan the study of micromoral issues. It is premature to saywhat approach best illuminates micromorality. However, weclaim that a Kohlbergian approach illuminates macromoral-ity issues (see Table 4.9 in Rest et al., 1999).

DIT1 research follows Kohlberg's approach in four basicways. It (a) emphasizes cognition (in particular, the forma-tion of concepts of how it is possible to organize cooperationamong people on a society-wide scope); (b) promotes theself-construction of basic epistemological categories (e.g.,reciprocity, rights, duty, justice, social order); (c) portrayschange over time in terms of cognitive development (i.e., itis possible to talk of "advance" in which "higher is better");and (d) characterizes the developmental change of adoles-cents and young adults in terms of a shift from conventionalto postconventional moral thinking. However, we call ourapproach a "neo-Kohlbergian approach" (i.e., it is based onthese starting points, but we have made some modificationsin theory and method).

One major difference is our approach to assessment.Instead of Kohlberg's interview, which asks participants tosolve dilemmas and explain their choices, the DIT1 uses amultiple-choice recognition task that asks participants to rateand rank a standard set of items. Some people are moreaccustomed to interview data and question whether datafrom multiple-choice tests are sufficiently nuanced to ad-dress the subtleties of morality research. Some researchersregard a multiple-choice test as a poor way to study morality,compared with the richness of explanation data from inter-views. Therefore, the prior question concerning whether toupdate the DIT1 needs attention first. These challenges raisecomplex issues that are addressed in a recent book (Rest etal., 1999). Within the short span of an article, we canindicate only the general direction that we take.

The DIT Approach

A common assumption in the field of morality, and onewith which we disagree, is that reliable information aboutthe cognitive processes that underlie moral behavior isobtained only by interviewing people. The interview methodasks a person to explain his or her choices. The moraljudgment interview has been assumed to provide a clearwindow into the moral mind. In his scoring system (Colby etal., 1987), Kohlberg gave privileged status to interview data.At one point, Kohlberg (1976) referred to scoring interviewsas "relatively error-free" and "theoretically the most validmethod of scoring" (p. 47). According to this view, thepsychologist's job is to create the conditions in which theparticipant is candid, ask relevant and clarifying questions,and then classify and report what the participant said. Then,in the psychologist's reports, the participant's theories abouthis or her own inner process are quoted to support thepsychologist's theories of how the mind works.

However, consider some strange outcomes of interviewmaterial. When Kohlberg reported interviews, the partici-pants talked like philosopher John Rawls (Kohlberg, Boyd,

646 REST, NARVAEZ, THOMA, AND BEBEAU

& LeVine, 1990); when Gilligan reported interviews, theparticipants talked like gender feminists (Gilligan, 1982);and when Youniss and Yates (in press) reported interviews,the participants said that they don't reason or deliberate at allabout their moral actions. This unreliability in explanationdata exists because people do not have direct access to theircognitive operations. Perhaps people do not know how theirminds work any more than they know how their immuniza-tion or digestive systems work. Perhaps asking a person toexplain his or her moral judgments is likely to get back whatthey have understood current psychological theorists to besaying. Then, when psychologists selectively quote theparticipants' explanations that agree with their own views,such evidence is vulnerable to the charge of being circular.Thus, interview data need more than face validity.

Contrary to assuming the face validity of interviews,researchers in cognitive science and social cognition con-tend that self-reported explanations of one's own cognitiveprocesses have severe limitations (e.g., Nisbett & Wilson,1977; Uleman & Bargh, 1989). People can report on theproducts of cognition but cannot report so well on the mentaloperations they used to arrive at the product. We believe thatpeople's minds work in ways they do not understand and inways that they can't explain. We believe that one of thereasons that there is so little evidence for postconventionalthinking in Kohlberg's studies (e.g., Snarey, 1985) is thatinterviewing people does not credit their tacit knowledge.There is now a greater regard for the importance of implicitprocesses and tacit knowledge in human decision making.Tacit knowledge is outside the awareness of the cognizer(e.g., Bargh, 1989; Holyoak, 1994) and beyond his or herability to articulate verbally. For example, consider theinability of a 3-year-old to explain the grammatical rulesused to encode and decode utterances in his or her nativelanguage. The lack of ability to state grammatical rules doesnot indicate what children know about language. Similarly, alack of introspective access has been documented in a widerange of phenomena, including attribution studies (e.g.,Lewicki, 1986), word recognition (Tulving, Schacter, &Stark, 1982), conceptual priming (Schacter, 1996), andexpertise (Ericsson & Smith, 1991). This research calls intoquestion the privileged place of interview data over recogni-tion data (as in the DIT1). We believe that any data-gathering method needs to build a case for its validity andusefulness.

Note that the issue here is not whether Kohlberg distin-guished normative ethics from meta-ethics. Rather, our pointis that Kohlberg regarded explanation data from interviewsas directly revealing the cognitive operations by whichmoral judgments are made. We are denying that people haveaccess to the operations or inner processes by which theymake moral decisions. We are denying that the royal roadinto the moral mind is through explanation data given ininterviews. The upshot of all of this is extensive (see moredetailed discussion in Rest et al., 1999). It not only meansthat multiple-choice data may have something of value tocontribute to moral judgment research, but it also results indrawing the distinction between content and structure at adifferent place than Kohlberg did. All content is not purged

from structure in assessment, the highest development inmoral judgment is not denned in terms of a particular moralphilosopher (i.e., John Rawls), and the concept of develop-ment is redefined so that development is not tied to thestaircase metaphor.

We grant that the DIT started out in the 1970s as a "quickand dirty" method for assessing Kohlberg's stages. How-ever, as time has passed and as data on the DIT1 hasaccumulated, different theories about human cognition haveevolved (e.g., Taylor & Crocker, 1981). In keeping withthese changes, we have reconceptualized our view of theDIT1 (see Rest et al., 1999, chapter 6). Now, we regard theDIT1 as a device for activating moral schemas (to the extentthat a person has developed them) and for assessing them interms of importance judgments. The DIT1 has dilemmas andstandard items; the participant's task is to rate and rank theitems in terms of their moral importance. As the participantencounters an item that both makes sense and also taps intothe participant's preferred schema, that item is judged ashighly important. Alternatively, when the participant encoun-ters an item that either doesn't make sense or seemssimplistic and unconvincing, he or she gives it a low ratingand passes over it. The items of the DIT1 balance bottom-up,data-driven processing (stating just enough of a line ofargument to activate a schema) with top-down, schema-driven processing (stating a line of argument in such a waythat the participant has to fill in the meaning from schemasalready in his or her head). In the DIT1, we are interested inknowing which schemas the participant brings to the task.We assume that those are the schemas that structure andguide the participant's thinking in decision making beyondthe test.

Validity of the DIT1

Arguing that there are problems with interview data doesnot automatically argue for the validity of the DIT1. Rather,the DIT1 must make a case for validity on its own. Validityof the DIT1 has been assessed in terms of seven criteria.Rest, Thoma, and Edwards (1997) described the sevencriteria for operationalizing construct validity. A recent book(Rest et al., 1999) cited over 400 published articles that morefully document the validity claims. The validity criteriabriefly are as follows:

1. Differentiation of various age and education groups.Studies have shown that 30% to 50% of the variance of DITscores is attributable to the level of education in heteroge-neous samples.

2. Longitudinal gains. A 10-year longitudinal study showedsignificant gains of men and women and of college attendersand noncollege participants from diverse backgrounds. Areview of a dozen studies of freshman to senior collegestudents (n = 755) showed effect sizes of .80 (large gains).Of all the variables, DIT1 gains have been one of the mostdramatic longitudinal gains in college (Pascarella & Teren-zini, 1991).

3. DIT1 scores are significantly related to cognitivecapacity measures of moral comprehension (r = .60s),recall and reconstruction of postconventional moral argu-

DIT2 647

ments (Narvaez, 1998), to Kohlberg's moral judgmentinterview measure, and (to a lesser degree) to other cognitivedevelopmental measures.

4. DIT1 scores are sensitive to moral education interven-tions. One review of over 50 intervention studies reported aneffect size for dilemma discussion interventions to be .41(moderate gains), whereas the effect size for comparisongroups was only .09 (small gains).

5. DIT1 scores are significantly linked to many prosocialbehaviors and to desired professional decision making. Onereview reported that 32 of 47 measures were statisticallysignificant (see also Rest & Narvaez, 1994, for recentdiscussions of professional decision making).

6. DIT1 scores are significantly linked to political atti-tudes and political choices. In a review of several dozencorrelates with political attitude, DIT1 scores typicallycorrelated in the range, r = .40 to .60. When combined inmultiple regression with measures of cultural ideology, thecombination predicted up to two thirds of the variance (Rs inthe .80s) of controversial public policy issues such asabortion, religion in the public school, women's roles, rightsof the accused, rights of homosexuals, and free-speechissues. Such issues are among the most hotly debated issuesof our time, and DIT1 scores are a major predictor to thesereal-life issues of macromorality.

7. Reliability. Cronbach's alpha is in the upper .70s/low.80s. Test—retest is about the same.

A specification of validity criteria tells us which studies todo to test a construct and what results should be found inthose studies. Operational definitions enable us to examinethe special usefulness of information from a measure. Wewant to know how the construct is different from othertheoretically related constructs. Accordingly, DIT1 scoresshow discriminant validity from verbal ability and generalintelligence and from conservative and liberal politicalattitudes. That is, the information in a DIT1 score predictsthe seven validity criteria above and beyond that accountedfor by verbal ability and general intelligence or politicalattitude (Thoma, Narvaez, Rest, & Derryberry, in press).Further, the DIT1 is equally valid for men and women (Restet al., 1999). In sum, mere is no other variable or constructthat accounts as well for the combination of the sevenvalidity findings than the construct of moral judgment. Thepersuasiveness of the validity data comes from the combina-tion of criteria that many independent researchers havefound, not just from one finding with one criterion.

Why a Revised DIT?

Because we wanted to maintain comparability in studies,DIT1 went unchanged while we went through a full cycle ofstudies. It took much longer to go through a full cycle thanwe originally anticipated; the DIT1 was frozen for over 25years.

There are several issues about DIT1 that DIT2 seeks toaddress (and this moves us to the specific purposes of thepresent article):

1. Some of the dilemmas in DIT1 are dated, and some ofthe items needs new language (e.g., in DIT1, Kohlberg's

well-known dilemma about "Heinz and the drug" is used,the Vietnam War is talked about in one dilemma as if it is acurrent event, and, in one of the items, the term Orientalswas used to refer to Asian Americans). While updatingdilemmas and items, we rewrote the instructions to clarifythem, and we shortened the test from six stories to fivestories when we found that one dilemma in DIT1 was notcontributing as much to validity as were the other dilemmas(Rest, Narvaez, Mitchell, & Thoma, 1998b).

2. DIT2 takes advantage of a recently discovered way tocalculate a developmental score (the N2 index; Rest, Thoma,Narvaez, et al., 1997). (Because issues of indexing arediscussed at length in this recent publication, that discussionis not repeated here.)

3. There is the ever-present problem in group-adminis-tered, multiple-choice tests (that are also often anonymous)that participants might give bogus data. The challenge,therefore, is to develop methods for detecting bogus data sothat we can purge the questionnaires that have bogus data. InDIT1, there are several checks for participant reliability; theusefulness of having some sort of check for participantreliability has been described (Rest, Thoma, & Edwards,1997). Nevertheless, with DIT2, we reconsidered our particu-lar method of checking for participant reliability, especiallybecause such a large percentage (typically over 10%) ofsamples using DIT1 are discarded for questionable partici-pant reliability. (Maybe in our zeal to detect bogus data, wethrew out too many participants.)

To prepare the new dilemmas and items of DIT2, we firstdiscussed various versions amongst ourselves. Then weasked members of an advanced graduate seminar on moral-ity research at the University of Minnesota to take thereformulated DIT2 and to make comments. Then we dis-cussed the dilemmas, items, and instructions again. Giventhat DIT1 has been unchanged for over 25 years and the factthat the Kohlberg group labored for decades over the scoringsystem of the moral judgment interview (Colby et al., 1987),changing the DIT might seem to be a big undertaking.However, the process was surprisingly straightforward andswift (and the results were positive). We conclude there isnothing sacred or special about the original Kohlbergdilemmas or the DIT1 dilemmas that cannot be reproducedin new materials. After freezing the DIT1 for years, we nowencourage experimentation in new dilemmas and newformats. To encourage this experimentation, the new scoringguides and computer scoring from the Center for the Studyof Ethical Development provide special aids to assist in thedevelopment of new dilemmas and new indexes (see Rest &Narvaez, 1998; Rest, Narvaez, Mitchell, & Thoma, 1998a).

DIT2 parallels DIT1 in construction:1. Paragraph-length hypothetical dilemmas are used, each

followed by 12 issues (or questions that someone deliberat-ing on the dilemma might consider) representing differentstages or schemas. The participant's task, a recognition task,is to rate and rank the items in terms of their importance.

2. The "fragment strategy" is used whereby each item isshort and cryptic, presenting only enough verbiage toconvey a line of thinking, not to present a full oration


defending one action choice or another (see Rest et al., 1999;Rest, Thoma, & Edwards, 1997).

3. Dilemmas and items on DIT2 closely parallel the moralissues and ideas presented in DIT1; however, the circum-stances in the dilemmas and wording are changed, and theorder of items is changed.

4. We presume that the underlying structure of moraljudgment assessed by the DIT consists of three developmen-tal schemas: personal interest, maintaining norms, andpostconventional (Rest et al., 1999). See the Appendix for asample story from DIT2.

Validating DIT2

How does one determine whether a new version of theDIT is working? We administered both DIT1 and DIT2 tothe same participants, balancing the order of presentation.We included students at several age and education levels(from ninth-grade to graduate and professional schoolstudents). We wanted to pick criteria for this preliminaryvalidation on which DIT1 was particularly strong, thinkingthat DIT2 would have to be at least as strong on thesecriteria. We used four criteria for initial validity:

1. Discrimination of age and education groups. This isour chief check on the presumption that our measure ofmoral judgment is measuring cognitive advance—a keyassumption of any cognitive developmental measure.

2. Prediction of opinions on controversial public policyissues. As discussed in Rest et al. (1999), one of the mostimportant payoffs of the moral judgment construct is itsability to illuminate how people think about the macromoralissues of society. The DIT predicts how people think aboutthe morality of abortion, religion in public schools, and so on(matters dealing with the macro-issues of social justice, thatis, how it is possible to organize cooperation on a society-wide basis, going beyond face-to-face relationships). Thesignificant correlation between the DIT and various mea-sures of political attitude has long been noted (see the reviewof over 30 correlations in Rest et al., 1999). A secondary goalof this study was to replicate a study by Narvaez, Getz,Thoma, and Rest (1999) by (a) using the specific measure ofpolitical attitude—the Attitudes Toward Human Rights Inven-tory (ATHRI; Getz, 1985); (b) testing whether D1T scores reduceto political identity or religious fundamentalism or to a commonfactor of liberalism or conservatism; and (c) testing whetheror not the combination of DIT scores with cultural ideology(e.g., political identity and religious fundamentalism) morepowerfully predicts controversial public policy issues thanany one of these measures alone. Replicating the Narvaez etal. (1999) findings (both with DIT1 and D1T2 in a new study) isthe first direct replication of these findings beyond the originalstudy, on which we base our interpretation that moral judgmentinteracts with cultural ideology in parallel—not serially—inproducing moral thinking about macromoral issues. More gener-ally, we have taken the position that an important payoff ofmoral judgment research is to illuminate people's opinionsabout controversial public policy issues, and thus it is importantto show that this interpretation is not based on only one study.

3. High correlations between DIT1 and DIT2. Of coursethis is important when comparing two tests purported tomeasure the same thing.

4. Adequate internal reliability in DTT2. This was the finalcriterion for determining the adequacy of DIT2.

We present our findings in four parts. Part 1 compares theperformance of DIT2 (including the changes in dilemmasand items, in indexing, and in participant reliability checks)with DIT1, focusing on the four validity criteria mentionedpreviously. The central questions here are whether updating,clarifying, and shortening DIT2, and purging fewer partici-pants for questionable reliability (practical improvements)can be done without sacrificing validity, and whetherimprovements in constructing a new index (N2) and newmethods of detecting bogus data (new checks) are effective.In Part 2, we seek to isolate the effects of each of the threechanges. What are the particular effects of changing thedilemma and item stimuli, the method of indexing, and themethod of checking for participant reliability? In Part 3, weshift our focus to consider in some detail the problem ofbogus data and methods for detecting unreliable partici-pants. (New checks turns out to be the most uniquemethodological feature discussed in this article.) Finally, inPart 4, we further examine a replication with DIT2 of theNarvaez et al. (1999) study that concerns the discriminabil-ity of the DIT1 from political attitudes and examines theparticular usefulness of the DIT2 in predicting opinionsabout public policy issues (seeking replication of the theoreti-cal claim that moral judgment's most important payoff is theprediction of opinions about controversial public policyissues).

Method

Participants

The overall goal in constituting this sample was to have a mix ofparticipants at various age and educational levels. We soughtparticipants from four educational levels: students who were in theninth grade, students who had recently graduated from high schooland were enrolled for only a few weeks as freshmen in college,students who were college seniors, and students in graduate orprofessional school programs beyond the baccalaureate degree.These four levels of education have been used in studies of the DITsince 1974 (Rest, Cooper, Coder, Masanz, & Anderson, 1974). Atotal of 200 participants from these four age and educational levelsturned in completions of all the major parts of the questionnairepackage. Note that both the least advanced and the most advancedgroups were from the upper Midwest, whereas the two middlegroups were from the South. Thus, correlations with educationcould not be explained as regional differences.

Ninth-grade students. Two classrooms of ninth graders (» = 47)were asked to participate. The students attended a school that waslocated in a middle-class suburb of the Twin Cities metropolitanarea. Testing took place over two class periods of a life skills class.

Senior high graduates, new freshmen. Students (n = 35) froma university in the southeastern United States were offered extracredit in several psychology classes for participation. Freshmanstudents had recently graduated from high school and had been atthe university for only a few weeks.

College seniors. Students (n = 65) from a university in thesoutheastern United States were offered extra credit in several

DIT2 649

psychology classes. College seniors were students who werefinishing their last year as undergraduates.

Graduate school and professional school students. Partici-pants in this category (n = 53) consisted of 37 students in adentistry program at a state university in the upper Midwest (at theend of their professional school program), 13 students at a private,moderately conservative seminary in the upper Midwest, and 3students in a doctoral program in moral philosophy (we wereunsuccessful in our attempts to recruit more moral philosophystudents). Participants who took the tests on their own time werepaid.

Instruments

The choice of instruments followed from the goals of the study,which are (a) to compare DIT1 with DIT2 and (b) to replicate theNarvaez et al. (1999) study.

Moral judgment: DIT1-P.1 The DIT (Rest et al., 1999) is apaper-and-pencil test of moral judgment. DIT1 presents six dilem-mas: (a) "Heinz and the drug" (whether Heinz ought to steal a drugfor his wife who is dying of cancer, after Heinz has attempted to getthe drug in other ways); (b) "escaped prisoner" (whether aneighbor ought to report an escaped prisoner who has led anexemplary life after escaping from prison); (c) "newspaper"(whether a principal of a high school ought to stop publication of astudent newspaper that has stirred complaints from the communityfor its political ideas); (d) "doctor" (whether a doctor should givemedicine that may kill a terminal patient who is in pain and whorequests the medicine); (e) "webster" (whether a manager ought tohire a minority member who is disfavored by the store's clientele);and (f) "students" (whether students should protest the VietnamWar). Each dilemma is followed by a list of 12 considerations inresolving the dilemma, each of which represent different types ofmoral thinking. Items are rated and ranked for importance by theparticipant. For over 25 years, the most widely used index of theDIT1 has been the P score, representing the percentage ofpostconventional reasoning preferred by the respondent. Althoughthe stages of moral thinking reflected on the DIT were inspired byKohlberg's (1976) initial work, the DIT is not tied to a particularmoral philosopher (as Kohlberg's is tied to Rawls, 1971). Kohl-berg's stages are redefined in terms of three schemas (personalinterests, maintaining norms, and postconventional).

DFT2-N2. The revised test consists of five dilemmas: (a)"famine" (A father contemplates stealing food for his starvingfamily from the warehouse of a rich man hoarding food—comparable to the Heinz dilemma in DIT1); (b) "reporter" (Anewspaper reporter must decide whether to report a damaging storyabout a political candidate—comparable to the prisoner dilemma inDIT1); (c) "school board" (A school board chair must decidewhether to hold a contentious and dangerous open meeting—comparable to the newspaper dilemma in DIT1; (d) "cancer" (Adoctor must decide whether to give an overdose of a painkiller to afrail patient—comparable to the doctor dilemma in DIT1; and (e)"demonstration" (College students demonstrate against U.S. for-eign policy—comparable to the students dilemma in DIT1). Thevalidity of DIT2 is unknown because this is the first study to use it.The N2 index takes into account preference for postconventionalschemas and rejection of less sophisticated schemas, using bothranking and rating data. Its rationale is discussed in Rest, Thoma,and Edwards (1997).

Opinions about public policy issues. As in the Narvaez et al.(1999) study, the ATHRI, constructed by Getz (1985), asksparticipants to agree or disagree (on a 5-point scale) with state-ments about controversial public policy issues such as abortion,euthanasia, homosexual rights, due process rights of the accused,

free speech, women's roles, and the role of religion in publicschools. The ATHRI poses issues suggested by the AmericanConstitution's Bill of Rights, similar to the large-scale studies ofAmerican attitudes about civil liberties by McClosky and Brill(1983). The ATHRI contains 40 items, 10 of which are platitudi-nous, "apple pie" statements of a general nature with whicheveryone tends to agree. Here are two examples of the platitudi-nous, noncontroversial items: "Freedom of speech should be abasic human right" and "Our nation should work toward libertyand justice for all." In contrast, 30 items are controversial, specificapplications of human rights. Two examples are "Books should bebanned if they are written by people who have been involved inun-American activities" and "Laws should be passed to regulatethe activities of religious cults that have come here from Asia."During initial validation, a pro-rights group (from an organizationthat had a reputation for backing civil liberties) and a selective-about-rights group (from a group with a reputation for backing rights ofcertain groups selectively) were enrolled for a pilot study (n = 101)with 112 controversial items (Getz, 1985). Thirty of the items thatshowed the strongest divergence between groups were selected forthe final version of the questionnaire, along with 10 items thatexpressed platitudes with which there was no disagreement (seeGetz, 1985, for further details on the pilot study). Therefore, withthe ATHRI, we have a total of 40 human rights issues that arerelated to civil libertarian issues.

In the study by Narvaez et al. (1999), scores ranged from 40 to200. These high scores represent advocacy of civil liberties.Although the items of the ATHRI represent many different issuesand contexts, they strongly cohere (Cronbach's alpha was .93).Narvaez et al. (1999) reported significant bivariate correlations ofDIT1 with ATHRI (rs in the .60s). Also, when measures of politicalidentity and religious fundamentalism were combined in multipleregression with the DIT to predict ATHRI, the R was in the range of.7 to .8, accounting for as much as two thirds of the variance.Further, each of the independent variables had unique predictability(as well as shared variance). Thus, each independent variable wasnot reduced to a single common factor of liberalism or conserva-tism. The present study was intended to replicate those findingsusing a different sample, with both DIT1 and DIT2.

Religious ideology. To measure religious fundamentalism, wechose Brown and Lowe's (1951) Inventory of Religious Belief,following Getz (1985) and Narvaez et al. (1999). It is a 15-itemmeasure that uses a 5-point, Likert-type scale. Its items differenti-ate between those who believe and those who reject the literalnessof Christian tenets. It includes items such as "I believe the Bible isthe inspired Word of God" (a positively keyed item); "The Bible isfull of errors, misconceptions, and contradictions" (a negativelykeyed item); "I believe Jesus was born of a virgin"; and "I believein the personal, visible return of Christ to earth." Scores on theBrown-Lowe inventory range from 15 to 75. High scores indicatestrong literal Christian belief. Criterion group validity is goodbetween more and less fundamentalistic church groups (Brown &Lowe, 1951; Getz, 1984; Narvaez et al., 1999). Test-retestreliability has been reported in the upper .70s. Spearman-Brownreliability has been found in the upper .80s (Brown & Lowe, 1951).In Narvaez et al. (1999), Cronbach's alpha was .95 for the entire

1 Operationalized variables used in statistical analysis are printedas an abbreviated name in capital letters (e.g., DIT1-P, FUNDA).Theoretical constructs are printed in the usual manner (e.g., moraljudgment, religious fundamentalism). In the case of DIT variables,the version is designated by DIT1 or DIT2, and the index used isdesignated after the hyphen (e.g., DIT1-P, the original DIT usingthe P index; or DIT2-N2, the new DIT using the N2 index).


Table 1Participants Groups and Demographics

Average age PercentGroup Number (SD) women

Ninth gradeHigh school graduates/college

freshmenCollege seniorsGraduate/professional school

Total

47 14.64 (0.53) 34

35 18.51 (2.03) 7765 21.55(3.11) 7753 29.06 (5.90) 45

200 21.4(6.39) 58.5

group of 158 participants. This scale taps religious fundamentalismand is labeled FUND A.

Political identity: Liberalism and conservatism. Participantswere asked to identify their political identity on a 5-point politicalconservatism scale, ranging from 1 (liberal) to 5 (conservative).This method of measuring liberalism and conservatism replicatesthe Narvaez et al. (1999) study and is the variable of contention inthe challenge to the DIT1 by Emler, Resnick, and Malone (1983).This variable will be referred to as POLCON (political conserva-tism), with high scores being conservative.

Demographics. Age of participants was given in years. Partici-pants were also asked to state their gender, but because there wereno significant differences on any of the DIT scores for genderscores for both males and females were collapsed for analysis.Education was measured in terms of the four levels of education(1 = ninth grade, 2 = college freshman, 3 = college senior,4 = graduate or professional school student). Participants wereasked whether they were Christians. Participants were also askedwhether they were citizens of the United States (virtually all,98.3%) and whether English was their first language (virtually all,97%). Some participant demographics are shown in Table 1.

Procedure

The order of materials was randomly varied (for all but thedentistry students), with DIT1 coming first for half of the partici-pants and DIT2 coming first for the other half. There were nosignificant differences in terms of order for any of the majorvariables (P and N2 indexes on DIT1, P and N2 on DIT2 or onATHRI, FUNDA, and POLCON). Because the 37 dentistry stu-dents had already taken the DIT1 as part of their regular curriculumrequirements, we sought volunteers to take the remaining packageof questionnaires, and the order was not varied.

For the high school participants, time in two class sessions wasused to take the questionnaires; for the remaining participants, thequestionnaire package was handed out and the participants filledout the questionnaires on their own time.

All Minnesota participants (and parents of the ninth graders)signed consent forms in accordance with the procedures of theUniversity of Minnesota Human Participants Committee. Partici-pants from the southeastern university were recruited in compli-ance with that institution's human participant requirements.

Results and Discussion

In Part 1, DIT1-P is compared with DIT2-N2. How doesthe new revision of the DIT stack up against the traditionalDIT, which has been used for over 25 years and reported inhundreds of studies? The key question is whether, afterdecades of research, we have developed a better instrument.

Then in Part 2, we examine the particular effects of each ofthe three changes in DIT2: (a) using the original wording ofdilemmas and items versus the revised dilemmas and items,(b) using the P index versus using the new N2 index, and (c)using the standard participant reliability checks versus usingnew checks.

Parti

Participant reliability. The DIT contains checks on thereliability of a participant's responses. DIT1 uses a differentmethod for detecting participant unreliability than the DIT2(discussed in detail in Part 3). From the total sample of 200participants, 154 survived the reliability checks of thestandard procedure for DIT1 (77%), whereas 192 survivedthe new reliability checks of DIT2 (96%). Given that in thisstudy the same participants took both DIT1 and DIT2, weconclude that DIT2 purges fewer participants for suspectedunreliability than does DIT1. The difference in proportion ofparticipants purged between the new procedure and thestandard procedure is significant (z = 5.56, p < .0001).

Criterion 1. We expect a developmental measure ofmoral judgment to increase as age and education increases.Table 2 presents the means and standard deviations ofDIT1-P and DIT2-N2 for each of the four educational levels.An analysis of variance (ANOVA) with DIT1-P grouped byfour levels of education produces F(3, 153) = 41.1, p <.0001; an ANOVA with DIT2-N2 produces F(3, 191) =58.9; p < .0001. Table 3 presents age and educational trenddata in terms of correlations of the moral judgment indexeswith educational level (four levels) and with chronologicalage (14-53). Although there might be doubts about the strictlinearity of education level (and therefore the use of level ofeducation as a linear variable in correlations), we assumethat deviations of the educational-level variable from strictlinearity affects both DIT1 and DIT2 equally, thus notbiasing the comparison between DIT measures. The correla-tional analysis shows stronger educational trends withDIT2-N2 than with DIT1-P, although this amount of differ-ence may not make much practical difference. In sum, thepractical advantages of DIT2 (i.e., being shorter, moreup-to-date, and purging fewer participants) are not at the

Table 2Means and Standard Deviations ofDITl-P and DIT2-N2by Four Education Levels

DIT1-P(n = 154)

DIT2-N2(n = 192)

Education level M SD M SD

1. Ninth grade 23.0 10.0 20.5 9.72. College freshmen 28.7 11.5 30.6 14.43. College seniors 33.7 14.1 40.4 13.64. Graduate/professional school 53.9 13.1 53.3 11.5

Note. In DIT1-N2, for comparison purposes, the N2 index isadjusted so that the mean (37.85) and standard deviation (17.19)are equal to those of the P index. DIT1 = Defining Issues Test(original version); DIT2 = Defining Issues Test, Version 2; P = Pindex; N2 = N2 index.

DIT2 651

Table 3Correlations ofDIT With Education

Measure

DIT1-P indexDIT2-N2 index

Educationlevel (l^t)

.62

.69

and Age

Chronologicalage

.52

.56

Note. All correlations of DIT with age and education level aresignificant, p < .0001. The correlation of DIT2-N2 with educationlevel is significantly higher, f(151) = 6.72, p < .001, than thecorrelation of DIT1-P with educational level. The correlation ofDIT1-P with age is not significantly different, t{\5\) = 1.67, ns,from the correlation of DIT1-N2 with age. (Calculation of differ-ences between correlations follows Howell, 1987, pp. 244ff, inwhich the correlations are first transformed to Fisher's r.) DIT1 =Defining Issues Test (original version); DIT2 = Defining IssuesTest, Version 2; P = P index; N2 = N2 index.

cost of poorer validity on Criterion 1. In fact, the opposite istrue.

Criterion 2. We expect a measure of moral judgment tobe related to views on public policy issues such as abortion,free speech, rights of homosexuals, religion in publicschools, women's roles, and so on.

In Table 4, we show both the old and new DIT correlatedwith ATHRI and also the partial correlations with ATHRIcontrolled for FUNDA and POLCON. We show partialcorrelations because previous studies (Rest, 1986) haveshown that both religious fundamentalism and politicalconservatism and liberalism were significantly correlatedwith the DIT. Therefore, the partial correlation attempts tocontrol the shared variance with political or religiousconservatism of the DIT with ATHRI, estimating the relationof moral judgment to public policy issues after controllingfor religious and political conservatism. Again, despite thepractical advantages of DIT2-N2 over DIT1-P, the newversion does not suffer any weaker trends on Criterion 2. Infact, in the partial correlation with ATHRI, DIT2-N2 has asignificant advantage over DIT1-P.

Criterion 3. We expect a measure of moral judgment tohave adequate reliability as measured by Cronbach's alpha.

Table 4Correlations and Partial Correlations of MoralJudgment With ATHRI

ATHRI (controlling forMeasure ATHRI FUNDA and POLCON)

DIT1-PDIT2-N2

.48

.50.40.51

Note. All correlations of the DIT with ATHRI are significant, p <.001. The correlation of DIT1-P with ATHRI is not significantlylower, f(151) = .99, ns, from the correlation of DIT1-N2 withATHRI. The correlation of DIT1-P with ATHRI, partialing out forFUNDA and POLCON, is significantly lower, f(149) = 4.43,p < .001, than the corresponding partial correlation of DIT2-N2.ATHRI = Attitudes Toward Human Rights Inventory (Getz, 1985);FUNDA = religious fundamentalism; POLCON = political iden-tity as conservative; DIT1 = Defining Issues Test (originalversion); DIT2 = Defining Issues Test, Version 2; P = P index;N2 = N2 index.

Because we use ranking data in the P index and as part of theN2 index, we cannot use the individual items as the unit ofinternal consistency. Ranks are ipsative; that is, if one item isranked in first place, then no other item of a story can be infirst place. Therefore, the unit of internal reliability is on thestory level, not the item level. Cronbach's alpha for DIT1-Pover the six stories (n = 154) is .76. For the DIT2-N2, it is.81 (n = 192). Although these levels of Cronbach's alpha arenot outstandingly high, we regard them as adequate.

It is interesting to note that Cronbach's alpha for DIT1 's 6stories plus DIT2's 5 stories (for a total of 11 units) is .90.We might speculate that this finding (i.e., 5 or 6 stories havemodest reliability, but 11 stories have high Cronbach'salpha) indicates that the five or six stories of DIT1 and DIT2each tap some different subdomains within morality. Al-though the DIT1-P and DIT2-N2 cohere enough, there isnevertheless some diversity in what each story taps. Whenwe add the 6-story DIT1 to the 5-story DIT2, the 11 storiesshow higher internal consistency because the 11 stories havemore overlap and are more redundant than the smallersamples of the 5 or 6 stories. Paradoxically, however, a scorebased on the 11 stories contains essentially the sameinformation (although somewhat redundantly) as the scorefrom 5 stories (with less redundancy). This can be seen bycomparing the correlations of the validity criteria from the5-story DIT2-N2 with the 11-story DIT1 + DIT2: For the5-story DIT2-N2, the correlation with education is .69,whereas the correlation with the 11-story DIT is .73. Thecorrelation of the 6-story DIT2-N2 with ATHRI is .50,whereas the correlation of the 11-story DIT1 + DIT2 is .52.By using all 11 stories (virtually doubling the test), the gainin Cronbach's alpha is 8 points, whereas the gain in thecorrelations with validity criteria is only 2 to 4 points.(Hence, we conclude that on the basis of 5 stories, DIT2-N2contains virtually the same information as a moral judgmentvariable that is based on 11 stories with high Cronbach'salpha.)

Criterion 4. We expect DIT1 to be significantly corre-lated with DIT2. This criterion is different from the previousthree criteria in that it does not contrast DIT1 with DIT2,but, rather, examines their overlap. The correlation ofDIT1-P with DIT2-N2 is .71 (using the standard participantreliability checks; n = 154). The correlation of DIT1-N2with DIT2-N2 is .79 (using the N2 index and new checks;n = 178).

With Guilford's (1954, p. 400) correction for attenuationresulting from the less-than-perfect reliability of two mea-sures, the upward bound estimate for the correlation be-tween the two "true" scores is .95 to .99 (depending on thesample used for reliability estimates and the method ofindexing). Hence, the DIT1 and DIT2 are correlated witheach other about as much as their reliabilities allow. DIT1 iscorrelated with DIT2 about as much as previous studies havereported for the test-retest of DIT1 with itself (Rest, 1979, p.239).

In sum, DIT2-N2 is shorter, more streamlined, moreupdated, and purges fewer participants than DIT1-P, and(with N2 and new checks) it has somewhat better validitycharacteristics. According to this study, if either measxnehas


the validity advantage, it seems to lie with DIT2 in additionto its practical advantages.

Part 2

What effects are unique to the new dilemmas and itemsand what effects are a result of the new analyses (N2 andnew checks)? What if the new analyses are computed on theold DIT (i.e., the data from the 6-story DIT1)? Would therebe any advantage in doing so (without using the newdilemmas and items)?

In Table 5, the top row repeats the correlations ofDIT2-N2 with the validity criteria already given in Tables 2and 3 and in the discussion of Cronbach's alpha in Criterion3; the bottom row repeats the correlations of DITl-P withthe validity criteria (also given previously). Rows 1 and 4are provided for easy comparison with rows 2 and 3. Thesecond row (the most important row in Table 5) shows howthe validity criteria are affected by using the old DIT (Heinzand the drug, etc.) with the new index (N2) and the newparticipant reliability checks. (In other words, row 2 uses theold DIT, including Heinz and the drug, but adopts the newdata analyses of N2 and new checks.) The special interest inrow 2 is whether there seems to be any advantage toreanalyzing DIT1 with N2 and new checks. The third rowshows how the correlations are affected by using the newDIT with the old P index and the old standard reliabilitychecks.

First, note the sample sizes. The new participant reliabil-ity checks allow more participants in the sample to becleared for analysis (96% for DIT2; 92% for DIT1) than dothe standard reliability checks (77% for DIT-P). The differ-ence between 92% and 77% is statistically significant(z = 3.98, p < .001, n = 200), and the difference between96% and 92% is statistically significant (z = 1.86, p = .05,n = 200). In other words, the new analyses (N2, newchecks) retain significantly more participants on both DIT1and DIT2 than do the old standard analyses, and with newchecks, DIT2 retains slightly more participants than doesDIT1. Although new checks retains more participants than

does standard checks on both DITs, DIT1 lost nine moreparticipants than did DIT2 using new checks.

Second, note that using the new analyses (N2 and newchecks) makes more of a difference in the validity criteriathan using new dilemmas (DIT2). In other words, the oldDIT (6-story DIT1)—for all its datedness and awkwardwording—seems to produce trends as strong as the new DIT(5-story DIT2) with updated wording when the new analyses(N2 and new checks) are used. The particular advantages ofDIT2 seem mostly to be that it is shorter and retains slightlymore participants (nine more than DIT1), not that thechanges in dilemmas or wording produce stronger validitytrends. Perhaps the datedness and awkward wording ofDIT1 put off some participants and undermined motivationto perform the task, but in the current study, this seemed toaffect only 5% of the participants. When most participantsperform the task of DIT1, the validity trends are as strong asthe updated, shorter version. In both cases, however, the newanalyses with N2 and new checks are preferable to theanalyses used for over 25 years for DIT1.

The third row in Table 6 shows that it is not a good idea touse DIT2 without the N2 index and new checks. From theperspective of this study, the only disadvantage of N2 andnew checks is that they are too labor intensive for handscoring (the original DIT1 could be hand scored). It takesseveral hours of hand computation per participant to performthe routines of N2 and new checks. Only a computer shouldbe put through the amount of calculation necessary toproduce N2 and new checks.

One might wonder whether the DIT's relation to ATHRI is"piggybacking" on a third variable, education. After all,other research (e.g., McClosky & Brill, 1983) has showncorrelations between public policy issues and education.Therefore, partialling out education, the partial correlationof DIT1 with ATHRI is .30 (n = 180, p < .001), and thepartial correlation of DIT2 with ATHRI is .28 (n = 195,p < .001). Again, partialling out education, the partialcorrelation of DIT1 with DIT2 is .62 (n = 178, p < .001).Therefore, there is no indication that education can accountfor the predictability of the DIT.

Table 5Correlations of DIT Measures With the Validity Criteria With and Without New Indexand New Participant Reliability Checks

Measure

DIT2-N2DIT1-N2

DIT2-PDITl-P

EDb ATHRP ATHRI/partiald

With new index and new participant reliability checks

192 .69 .50 .51183 .68 .54 .52

With old P index and standard participant reliability checks

154 .62 .55 .42154 .62 .48 .40

Cronbach's a

00 00

.74

.76

Note. ED = educational level; ATHRI = Attitudes Toward Human Rights Inventory (Getz, 1985);DIT1 = Defining Issues Test (original version); DIT2 = Defining Issues Test, Version 2.aSample retained after participant reliability checks. bCorrelation with educational level (4levels). cBivariate correlation with ATHRI. dPartial correlation with ATHRI, controlling forreligious fundamentalism (FUNDA) and political identity as conservative (POLCON).

DIT2 653

Table 6Multiple Regressions of Moral Judgment, Political Identity,and Religious Fundamentalism (Independent Variables)Predicting Controversial Public PolicyIssues (Dependent Variable)

Variable B

Equation 1, predicting to ATHRI including DIT1-P, n = 154,multiple R = .56,<f/ = 151

DIT1-PPOLCONFUNDA

0.34-3.78-0.18

.38- .23- .15

5.3***- 3 . 2 * *- 2 . 0 *

Equation 2, predicting to ATHRI including DIT2-N2, n = 194,multiple R = .58, df= 191

DIT2-N2POLCONFUNDA

0.28-3.85-0.17

.48- .23- .14

8.0***- 3 . 7 * *- 2 . 2 *

Note. POLCON and FUNDA are negatively related because highscores are more conservative, in contrast to DIT and ATHRI scores,which run in the opposite direction. ATHRI = Attitudes TowardsHuman Rights Inventory. DIT1-P = Defining Issues Test (originalversion), using P index; POLCON = political identity as conserva-tive; FUNDA = religious fundamentalism; DIT2-N2 = DefiningIssues Test, Version 2, using N2 index.*p < .05. **/?< .01. ***p < .001.

For completeness of analysis, additional tables of thevalidity criteria were also computed to separate the effect ofindexing from methods of detecting participant reliability(i.e., using the P index with new checks, and using N2 withstandard checks). The results were generally intermediatebetween rows 1 and 4. So nothing of special interest wasfound here. The general conclusion is that, for the strongestvalidity trends, the researcher might use either DIT1 orDIT2, but should use both new analyses together. Thepractical advantages of DIT2 (i.e., it is somewhat shorter,less dated, and likely to retain slightly more participants) iswhat recommends it over DIT1. We were expecting that thenew dilemmas and wording of DIT2 would make a contribu-tion to greater validity (in addition to using N2 and newchecks), but we were surprised that DIT1 seems to workabout as well (when used in conjunction with N2 and newchecks).

Part 3

Recall that DIT2 involves three changes from DIT1: (a)changes in dilemmas and items (discussed earlier); (b)changes in indexing (discussed in detail elsewhere; Rest,Thoma, Narvaez, et al., 1997); and (c) changes in participantreliability checks, which is addressed in this section.

The problems of bogus data. One inevitable problemwith a group-administered, multiple-choice test is thatparticipants might put check marks down on the question-naire without reading the items or following instructions, orthey might proceed with a test-taking set that is alien to theinstructions. How do researchers determine whether theparticipants' responses reflect moral thinking (as the moral

judgment construct purports) or are bogus? There are fourproblem responses that give bogus data:

1. Random responding. The participant may fill in thebubbles on an answer sheet, but the marks may not haveanything to do with his or her moral cognition. For instance,we have seen answer sheets on which participants filled inthe answer bubbles to form Christmas trees and othergeometric designs. We doubt that such responses accuratelymeasure the construct of moral judgment.

2. Missing data. The participant may not be sufficientlymotivated to take the test and may leave out large sections ofanswers, or just quit.

3. Alien test-taking sets. The participant may choose itemsnot on the basis of their meaning, but on the basis ofcomplex syntax, special wording, or the seemingly loftysound of the words. In this case, the scores do not reflect themoral judgment construct but instead reflect a preference forcomplex style or verbiage.

4. Nondiscrimination of items. The participant may putdown the same response for all items, failing to discriminateamong the items (e.g., putting down 3s for all ratings andranks). Rest, Narvaez, Mitchell, and Thoma (1998a) showedthat for a very large sample (n > 58,000), participants showconsiderable variation in rating and ranking DIT items;therefore, some variation is expected.

If a participant is suspected of any one of these fourresponse problems, we know of no way to salvage or correctthe protocol. Instead, the entire protocol is discarded fromanalysis. In general, previous research has shown thatpurging the protocols of participants who manifest any ofthese four problems results in clearer data trends (Rest,Thoma, & Edwards, 1997), presumably because error vari-ance has been minimized.

Standard checks. In the standard checks procedure usedwith DIT1, four checks are used to detect the likelihood ofeach of the four problems.

1. The problem of random responding. As a guard againstrandom checking, a participant's ratings are checked forconsistency with the participant's rankings. For example, if aparticipant chose Item 10 as the top rank (most importantitem), then no other item should be rated higher in impor-tance than Item 10. Further, with this example, if a partici-pant chose Item 8 as second most important rank, then onlyItem 10 should be rated higher. Our general approach is tocount each violation of a pattern of rank-rate consistency asan inconsistency. Thus, with regard to the first problem(random responding), rate-rank inconsistencies are assumedto indicate random checking. Theoretically, the perfectlyconsistent participant will have no rank-rate inconsistencies.In reality, however, we can expect some inconsistency, evenamong serious, well-motivated participants. Participantssometimes change their minds after being exposed to avariety of issues. So the question becomes, how muchinconsistency should researchers tolerate as the innocentshifting of item evaluations, and how much inconsistency istoo much, reflecting random responding? Where do we drawthe line? In the standard procedure, participants who havemore than eight inconsistencies on a dilemma (counting only


the top two ranks) are considered to have too muchinconsistency as are participants who have inconsistencieson more than two dilemmas. Participants exceeding thesecutoff points are eliminated from the sample. It turns out,over our 25-year experience, that this rate-rank consistencycheck (more than the other three participant reliabilitychecks, described later) accounts for the bulk of purgedparticipants.

2. The problem of missing data. Occasional missing dataare tolerated in standard checks. For example, if someoneomits an occasional rating or ranking, we do not purge theentire protocol. Instead, we readjust scores to make up forthe missing data, in effect calculating readjusted scores toreflect the response patterns in the rest of the protocol andadjusting scores so that every participant's data are on thesame scale. However, too much missing data may reflect ageneral lack of motivation to take the task. In this case, wecannot have confidence in any responses. Again, the ques-tion is how much is tolerable and how much is grounds forpurging the entire protocol? In the standard checks proce-dure, if a participant leaves out two whole stories (forinstance, a participant is asked to complete six stories butonly completes four), then that is regarded as too much. Theproblem in interpreting such a protocol is not that we couldnot readjust a score based on four stories to be on the samescale as six stories; rather, the problem is that we aresuspicious of the motivation of the participant to do the workof the DIT in the four stories (it is possible that even in thefour stories, reliable data were not given).

3. The problem of alien test-taking sets. Participants whochoose items for their pretentiousness or lofty sound are notfollowing instructions to choose items based on theirmeanings. As a check on this alien test-taking set, we havedistributed five meaningless items (M-items) throughout theDIT that may be attractive for their complex syntax or "highsounding" verbiage, but do not mean anything. If a partici-pant ranks too many of these M-items too highly, we assumean alien test-taking set and purge the whole protocol. In thestandard procedure, a score of 8 or more (weighting ranks by4 for top rank, by 3 for second rank, etc.) on the M-itemsinvalidates the protocol.

4. The problem of nondiscrimination. Participants who donot discriminate answers (e.g., those who check i s for allitems) are not complying with our instructions to makediscriminations. Because nondiscriminating participants willnot be picked up in the rate-rank check, a special check hasbeen devised for nondiscrimination. In the standard proce-dure, no more than one story can have more than eight itemsrated the same.

New checks. The new checks procedure recognizes thesame four problems in participant reliability, but deals withthem in ways different from the standard checks procedure.To investigate the consequences of different methods andcutoff points, we concocted a set of protocols that deliber-ately epitomized one or more of the violations we sought todetect. Some of the deliberately bogus data were based on arandom number table (to simulate random responding).Other bogus protocols were based on filling in the answerbubbles to form graphic designs (e.g., the Christmas tree

design). In general, the objective was to have protocols thatwe knew were bogus data and to see whether our reliabilitychecks would pick up all these bad protocols, but would passthrough a high percentage of actual data. We also wanted tosee if the validity trend was still robust with new cutoffscores. We were especially interested in comparing datatrends of the new checks with the old standard checks.

1. The problem of random responding. Because in thestandard checks, the largest numbers of participants arepurged for unreliability based on the rate-rank consistencycheck, we paid the most attention to this procedure. Todetect participants who are randomly checking, this is ournew procedure: We look at a participant's ranks, weight thetop rank as 4, the second most important as 3, the third as 2,and the fourth as 1 (same weights as in deriving the P score).Then we look at the item's rating. If there is an item differentfrom the one in the first rank that is rated more highly thanthe item in the top rank, then that is one occurrence ofinconsistency and is multiplied by 4. All other inconsisten-cies with the top rank are also multiplied by 4. Then we lookat the item ranked as second most important. There shouldnot be any item rated more highly than the second-rankeditem except the item ranked in the top rank. The occurrencesof exceptions to this expectation are counted and weightedby 3, and so on for the third- and fourth-ranked items (theviolations are counted and weighted by 2, for third rank, orby 1, for fourth rank). The weighted inconsistencies for eachstory and across stories are summed. The summed weightedrank-rate inconsistencies across five stories can range from0 to 600. Through trial and error, we arrived at cutoff points.We wanted a stringent enough threshold point to prevent anyof the deliberately bogus data from getting through, but notso low a threshold to make the validity trends suffer. Thus,we arrived at the cutoff point of how much is too much byempirical trial and error. It turns out that if the sum ofrate-rank inconsistencies is more than 200, then that is toomuch, and the protocol is invalidated (purged from thesample). If the sum is under the 200 mark, it is regarded asinnocent confusion, and we tolerate that much inconsistencyby not purging the entire protocol.

2. The problem of missing data. Occasional missing dataare tolerated by DIT2. Using the trial-and-error proceduredescribed earlier, we arrived at cutoff values. If the partici-pant leaves out more than three ratings on any of two stories,the protocol is invalidated. If the participant leaves out morethan six ranks, the protocol is also invalidated.

3. The problem of alien test-taking sets. Participants whopick items for style rather than for meaning are not followingour instructions. In the new checks procedure, we also useM-items to detect this problem. The protocols of participantswhose weighted ranks on the M-items total more than 10 areinvalidated (more lax than the cutoff of 8 on standardchecks).

4. The problem of nondiscrimination. In new checks,participants who rate 11 items the same on a story areconsidered as not discriminating; if the participant fails todiscriminate on two stories or more, the protocol is invali-dated. Nondiscrimination by rates or ranks is grounds forpurging the protocol.

D1T2 655

As mentioned earlier, the new checks purged 8 partici-pants from the sample of 200 (or 4%), whereas the standardchecks purged 46 participants (or 23%) from the sample. Ingeneral, the new checks are less stringent that the standardchecks. Paradoxically, the data in this study suggest that theless stringent method (new checks) produces stronger trendsthan does the more stringent method (standard checks). Howcan this be? One might expect the opposite—that makingsure of participant reliability (the more stringent method)would produce stronger validity trends than a more laxmethod for checking for participant reliability. The key tothis paradox lies in the fact that standard checks purgesproportionately more of the youngest group of ninth graders(58% of the ninth graders were purged by the standardchecks) than for the oldest group (only 8% were purged inthe graduate and professional school subsample). In con-trast, new checks purged only 11 % of the ninth graders (and1 participant from the graduate school subsample). Thedifference between 58% and 11% is significant, z = 4.80,p < .0001, n = 47. One might speculate that with standardchecks, the disproportionate purging of the youngest partici-pants from the sample in effect changes the distribution ofscores in the total sample, making the sample more homoge-neous, attenuating the spectrum of scores, and resulting inslightly weaker correlations and validity trends for standardchecks. In other words, the new checks have strongervalidity trends because they retain more of the lower scoresfrom the youngest participants and thus retain a wider rangeof scores (by which the correlations increase).

Because the cutoff values for the reliability checks areempirically derived, it remains to be seen whether they areoptimum for other samples. The experiences of otherresearchers is the most important consideration here. Tofacilitate experimentation with different cutoff values for thechecks, the scoring service of the Center for the Study ofEthical Development provides a set of variables that can bemanipulated for each sample (Rest & Narvaez, 1998; Rest,Narvaez, Mitchell, & Thoma, 1998a).

Part 4

Recall that Criterion 2 of validity in this study deals withthe correlation of the DIT with ATHRI. The correlation ofmoral judgment with political attitudes has been noted forsome time (typically rs in the .4 to .6 range; see Rest et al.,1999, for a review of several dozen correlations over 25years). Emler et al. (1983) interpreted this pattern ofcorrelations, contending that the DIT is really liberalism-conservatism masquerading as developmental capacity. Theystated that

Moral reasoning and political attitude are by and large one andthe same thing.... We believe that individual differences inmoral reasoning among adults—and in particular those corre-sponding to the conventional-principled distinction—are inter-pretable as variations on a dimension of political-moralideology and not as variations on a cognitive-developmentaldimension, (pp. 1073-1075)

In contrast, our view (Narvaez et al., 1999) is that moraljudgment, political identity (identifying oneself as a liberal

or conservative), and religious fundamentalism are relatedbut also distinct constructs. The variables carry uniqueinformation, and they cannot all be reduced to a commonfactor of liberalism-conservatism. (See Thoma et al., inpress, and Rest et al., 1999, for discussion of the Emler et al.,1983, studies.) In support of the uniqueness of each con-struct or variable (moral judgment, religious fundamental-ism, and political identity as liberal or conservative), Nar-vaez et al. reported a multiple regression having the ATHRIas the dependent variable and having the DIT, FUNDA, andPOLCON as independent variables. Multiple regressionanalysis permits estimation of the unique contribution ofeach independent variable by examining the standardizedbeta weights. Narvaez et al. (Study 1) examined two churchcongregations and found that the beta weights for each of thethree independent variables were each significant in theirown right, indicating that each contributes distinct informa-tion to ATHRI. This finding for church samples wasreplicated in a student sample (Narvaez et al., 1999,Study 2). Now we wish to determine whether the findingsreplicate with both DIT1 and DIT2. Because we place somuch importance on the DIT's unique contribution tounderstanding opinions about controversial public policyissues (the macrolevels of morality), we wanted to havemore than just the Narvaez et al. studies to confirm ourinterpretation.

Because we wanted to replicate the Narvaez et al. (1999)study, we used the Brown and Lowe (1951) instrument asthe measure of fundamentalism. However, there is a problemin that the Brown and Lowe instrument is a measure ofChristian fundamentalism, and the sample from this studycould have included Orthodox Jews, Orthodox Muslims, orothers who would have a low score on Christian fundamen-talism but nevertheless be very orthodox in a non-Christianway. Checking the total sample, it turned out that theoverwhelming proportion (90%) indicated they consideredthemselves to be Christian. Only 20 participants indicatedthat they were non-Christian. However, leaving these partici-pants out of the analysis made little difference in the relationof FUNDA to DIT, ATHRI, or POLCON. Correlations ofFUNDA with DIT2-N2, ATHRI, and POLCON, includingthe non-Christians, were —.10, —.25, and .28, respectively;excluding the non-Christians, the correlations were —.13,— .26, and .21, respectively. Because including or excludingthe non-Christians made little difference where FUNDA wasconcerned (Criterion 2), we left the non-Christians in thesample for the sake of maximizing the sample size on theother three criteria.

The multiple regressions in Table 6 on this samplereplicate with DIT1-P and with DIT2-N2 in the Narvaez etal. (1999) studies: (a) Both studies used the same dependentvariables, ATHRI (controversial public policy issues), andindependent variables (FUNDA, POLCON, and DIT); (b)each independent variable (DIT, POLCON, and FUNDA)has significantly unique predictability to ATHRI; (c) moraljudgment has higher standardized beta weights than doesPOLCON or FUNDA; (d) when all three independentvariables are combined, the combination predicts powerfully


Table 7Predictability to ATHRI From Multiple Regression Beta Weights From Present StudyWith Multiple Regression Weights From Narvaez et al. (1999), Study 1

Measure

ATHRI (n = 154)

ATHRI (TI = 192)

ORTHO"(Study 1)

- .54

ORTHCF(Study 1)

- .55

Multiple Rb

(present study with DIT1-P)

.56

Multiple Rd

(present study with DIT2-N2).58

Note. DIT, POLCON, and FUNDA are components that go into ORTHO and multiple R (fromTable 5). All correlations are significant,/? < .001. DIT1-P = Defining Issues Test (original version),using P index; DIT2-N2 = Defining Issues Test, Version 2, using N2 index; ATHRI = AttitudesToward Human Rights Inventory (Getz, 1985)."Orthodoxy combination variable formed by combining DIT1-P, POLCON, and FUNDA accordingto weights of multiple regression in Study 1 of Narvaez et al. (1999). bMultiple regression fromTable 5, Equation 1. cSame combination of variables based on weights of Study 1, but withDIT2-N2 for moral judgment component. ""Multiple regression from Table 5, Equation 2.

to ATHRI (with DIT1-P, R = .56; with DIT1-N2, multipleR = .58; with the 11-story DIT1 + 2 - N2, R = .63).

A stronger test of Narvaez et al. (1999) is to use the samebeta weights (nonstandardized) as those in Study 1 forcombining DIT, POLCON, and FUNDA. Using beta weightsfrom the original multiple regression (in Narvaez et al.,1999, Study 1) produces a variable called ORTHO (represent-ing the construct, orthodoxy-progressivism, as discussed byHunter, 1991). ORTHO provides a stronger replication ofthe Narvaez et al. (1999) study than a new multipleregression on this new sample because in combining thethree variables from the beta weights of the original sample,we are not capitalizing on sample-specific chance factors (asdo the multiple regressions in Table 6). ORTHO is, in effect,a transfer of the original relations of the independentvariables from Narvaez et al. (1999, Study 1) to the presentstudy in predicting ATHRI. Table 7 shows that the correla-tion of ATHRI with ORTHO is only two or three pointsweaker than the Rs run in Table 6 on the specific newsamples of the present study. In other words, the betaweights derived from the multiple regression for ORTHOfrom Study 1 in Narvaez et al. (1999) generalizes well to thepresent study.

One might be concerned with the problem of multicol-linearity on the multiple regression results. As Howell(1982, pp. 500ff) noted, a problem can exist in interpretation

of multiple regression results when the independent vari-ables are intercorrelated; the problem is that the beta weightsare unstable from sample to sample.

In the present sample, the independent variables aresignificantly intercorrelated; however, the extent of thecorrelation raises to only .28, far short of the problem causedwhen the correlations among independent variables ap-proach + 1.00 or -1.00. Furthermore, Howell (1987) sug-gested that the relative importance of each independentvariable is indicated by the t statistic (indicated in Table 6and all the multiple regression tables). It can be seen in ourtables that relative importance is the same relative order asthat of beta weights. Hence, what is said about the primaryimportance of DIT scores as one of the independentvariables still stands in view of the f-test results.

Hence, one of the findings of the present study is thereplication that the DIT is of first importance amongindependent variables (has higher beta weights and higherf-test scores). In all four replications (two in Narvaez et al.,1999, and both DIT1 and DIT2 results in the present study),this was the stable result; therefore, there does not seem to bea multicollinear problem in the stability of these resultsregarding beta weights.

It is true that the Rs in Narvaez et al. (1999) weregenerally in the range of .7 to .8. In the present study, Rswere in the .5 to .6 range. The difference in R may be due to

Table 8Differences Between Students in Present Sample and Studentsin Narvaez et al. (1999), Sample 2

Variable

DIT1-PPOLCONFUNDAATHRI

Present sample(n = 154)

M

37.863.16

58.13145.93

SD

17.190.92

12.8115.18

Narvaez et alStudy 2 (n

M

48.582.85

55.48159.16

• (1999),= 62)

SD

15.130.94

14.7817.36

Difference(t test, df= 214)

4.53****2.21*1.245.28****

Note. DIT1-P = Defining Issues Test (original version), using P index; POLCON = politicalidentity as conservative; FUNDA = religious fundamentalism; ATHRI = Attitudes Toward HumanRights Inventory (Getz, 1985).*p<.05. ****/><.0001.

DIT2 657

the peculiarities of the samples. As shown in Table 8, thestudent sample in the present study is generally moreconservative, that is, lower in moral judgment, lower onadvocacy for human rights, and more politically conserva-tive than the student sample of Narvaez et al. (1999, Study2). Future research may clarify whether views on publicpolicy issues (ATHRI) are better predicted in more liberalgroups (Narvaez et al., 1999, Study 2) than in moreconservative groups (present study).

Conclusions

The four parts of this article indicate the followingconclusions:

1. After 25 years of research using DIT1 -P, there may nowbe a better DIT that is shorter, more updated, purges fewerparticipants, and has significantly better validity characteristics.

2. To the extent that DIT2 shows an improvement invalidity trends over DIT1 in this study, the increase in validityseems to be attributable to the new ways of analyzing data (inindexing and in checking participant reliability) and not to thenew dilemmas or new wording. We were expecting signifi-cant gains in validity for new dilemmas and wording (DIT1vs. DIT2), for N2 versus P, and for new checks versusstandard checks. Instead, we found significant improve-ments for the new analyses (N2 and new checks) but not forDIT1 over DIT2. Still, the practical advantages of DIT2 (i.e.,it is shorter and updated and thus purges slightly fewerparticipants) recommend experimentation.

3. The reason that new checks show stronger trends on thevalidity criteria seems to be because they retain a widerrange of scores, resulting in a fuller distribution of scores.

4. The present study supports the particular interpretationof Narvaez et al. (1999) regarding the combination andinteraction of moral judgment with cultural ideology in theformation of opinions on public policy issues. DIT2 seemsto operate in a way similar to DIT1 when used to predictattitudes toward public policy issues. More generally, thissupports our view that Kohlbergian theories of morality aremore useful in describing macromorality than micromorality.

Despite the long tradition in using the same dilemmas inKohlbergian (1976, 1984) research (e.g., Heinz and thedrug), this study suggests that there is nothing exceptional ormagical about the DITl's dilemmas and items, or about theclassic Kohlberg dilemmas. It is possible to update, shorten,and revise the DIT without sacrificing validity. This shouldbe encouraging for experimentation with new dilemmas anditems. For instance, profession-specific dilemmas may bedevised for a profession (e.g., for dentists, accountants, orteachers) in the hope of accounting better for profession-specific behavior.

This study reconfirms several basic findings about themoral judgment construct. First, the developmental, age andeducation trends are reconfirmed with DIT2 (i.e., moraljudgment scores increase as age and education increases).Second, moral judgment scores are highly related to viewson controversial public policy issues, as assessed by theATHRI. Further, in multiple regression, moral judgmentalong with political identity and religious fundamentalismpredict the ATHRI scores in combination more strongly than

each independent variable alone, but each does not reduce tothe other. This is consistent with the view expressed inNarvaez et al. (1999) about the relation of moral judgment tocultural ideology. Third, the new index, N2, as reported inRest, Thoma, Narvaez, et al. (1997) shows advantages overthe traditional ways of performing these calculations. Al-though there seems to be some gain in the power of trendsusing these new forms of analysis (N2 and new checks), thecomputations have become so labor intensive that handscoring is no longer an option with N2 or new checks. Tothese replications, we add that DIT1 is highly correlatedwith DIT2 (r = .79) and that the 11 stories of DIT1 plusDIT2 show a very high degree of internal consistency(Cronbach's alpha = .90).

What are the practical implications of the present study?The findings encourage researchers to substitute DIT2 forDIT1. However, because this is only the first study withDIT2, with 200 participants—and because hundreds ofstudies have used the DIT1, involving about half a millionparticipants—the older version must be regarded as the moreestablished entity. Researchers for new projects must decidewhether an updated, shorter, and slightly more powerfulDIT2 with a short track record is preferable to the dated,longer, but better established DIT1. In any case, whetherusing DIT1 or DIT2, the new analyses (with N2 and newchecks) should be employed. (Users of DIT1 can sendpreviously scored data for rescoring, free of charge to theCenter for the Study of Ethical Development.)

The most meaningful verdict on DIT2 must come fromindependent researchers beyond the site of development(Center for the Study of Ethical Development). The general-izability of DIT2, N2, and new checks must come from otherresearchers who may or may not find these innovations useful.

References

Bargh, J. (1989). Conditional automaticity: Varieties of automaticinfluence in social perception and cognition. In J. Uleman & J.Bargh (Eds.), Unintended thought (pp. 3-51). New York:Guilford Press.

Brown, D. G., & Lowe, W. L. (1951). Religious beliefs andpersonality characteristics of college students. Journal of SocialPsychology, 33, 103-129.

Colby, A., Kohlberg, L., Speicher, B., Hewer, A., Candee, D.,Gibbs, J., & Power, C. (1987). The measurement of moraljudgment (Vols. 1-2). New York: Cambridge University Press.

Emler, N., Resnick, S., & Malone, B. (1983). The relationshipbetween moral reasoning and political orientation. Journal ofPersonality and Social Psychology, 45, 1073-1080.

Ericsson, K. A., & Smith, J. (1991). Toward a general theory ofexpertise. New York: Cambridge University Press.

Getz, I. (1984). The relation of moral reasoning and religion: Areview of the literature. Counseling and Values, 28, 94-116.

Getz, I. (1985). The relation of moral and religious ideology tohuman rights. Unpublished doctoral dissertation, University ofMinnesota.

Gilligan, C. (1982). In a different voice. Cambridge, MA: HarvardUniversity Press.

Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.


Holyoak, K. J. (1994). Symbolic connectionism: Toward third-generation theories of expertise. In K. A. Ericsson & J. Smith(Eds.), Toward a general theory of expertise (pp. 301-336). NewYork: Cambridge University Press.

Howell, D. C. (1987). Statistical methods for psychology (2nd ed.).Boston: Duxbury.

Hunter, J. D. (1991). Culture wars: The struggle to define America.New York: Basic Books.

Juergensmeyer, M. (1993). The new cold war? Berkeley, CA:University of California Press.

Killen, M., & Hart, D. (Eds.) (1995). Morality in everyday life.New York: Cambridge University Press.

Kohlberg, L. (1976). Moral stages and moralization: The cognitivedevelopmental approach. In T. Lickona (Ed.), Moral developmentand behavior (pp. 31-53). New York: Holt, Rinehart, & Winston.

Kohlberg, L. (1984). Essays on moral development: The nature andvalidity of moral stages, Vol. 2. San Francisco: Harper & Row.

Kohlberg, L., Boyd, D. R., & Levine, C. (1990). The return ofStage 6: Its principle and moral point of view. In T. Wren (Ed.),The moral domain: Essays in the ongoing discussion betweenphilosophy and the social sciences (pp. 1151-1181). Cambridge,MA: The MIT Press.

Lewicki, P. (1986). Non-conscious social information processing.New York: Academic Press.

Marty, M. E., & Appleby, R. S. (1991). Fundamentalism observed.Chicago: University of Chicago Press.

Marty, M. E., & Appleby, R. S. (1993). Fundamentalism and thestate. Chicago: University of Chicago Press.

McClosky, H., & Brill, A. (1983). Dimensions of tolerance: WhatAmericans believe about civil liberties. New York: Sage.

Narvaez, D. (1998). The influence of moral schemas on thereconstruction of moral narratives in eighth graders and collegestudents. Journal of Educational Psychology, 90, 13-24.

Narvaez, D., Getz, I., Thoma, S. J., & Rest, J. (1999). Individualmoral judgment and cultural ideologies. Developmental Psychol-ogy, 35, 478-488.

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know:Verbal reports on mental processes. Psychological Review, 84,231-259.

Pascarella, E. T., & Terenzini, P. (1991). How college affectsstudents: Findings and insights from twenty years of research.San Francisco: Jossey-Bass.

Rawls, J. A. (1971). A theory of justice. Cambridge, MA: HarvardUniversity Press.

Rest, J. (1979). Development in judging moral issues. Minneapolis:University of Minnesota Press.

Rest, J. (1986). Moral development: Advances in research andtheory. New York: Praeger.

Rest, J., Cooper, D., Coder, R., Masanz, J., & Anderson, D. (1974).Judging the important issues in moral dilemmas—an objectivetest of development Developmental Psychology, 10,491-501.

Rest, J., & Narvaez, D. (Eds.). (1994). Moral development in theprofessions: Psychology and applied ethics. Hillsdale, NJ:Erlbaum.

Rest, J., & Narvaez, D. (1998). Guide for DIT-2. Unpublishedmanuscript. (Available from Center for Study of Ethical Devel-opment, University of Minnesota, 206 Burton Hall, 178 Pills-bury Dr., Minneapolis, MN 55455.)

Rest, J., Narvaez, D., Bebeau, M. J., & Thoma, S. J. (1999).Postconventional moral thinking: A neo-Kohlbergian approach.Mahwah, NJ: Erlbaum.

Rest, J., Narvaez, D., Mitchell, C , & Thoma, S. J. (1998a).Exploring moral judgment: A technical manual for the DefiningIssues Test. Unpublished manuscript. (Available from the Centerfor the Study of Ethical Development, University of Minnesota,206 Burton Hall, 178 Pillbury Dr., Minneapolis, MN 55455.)

Rest, J., Narvaez, D., Mitchell, C , & Thoma, S. J. (1998b). HowTest Length Affects Validity and Reliability of the Defining IssuesTest. Manuscript submitted for publication.

Rest, J., Thoma, S. J., & Edwards, L. (1997). Designing andvalidating a measure of moral judgment: Stage preference andstage consistency approaches. Journal of Educational Psychol-ogy, 89, 5-28.

Rest, J., Thoma, S. J., Narvaez, D., & Bebeau, M. J. (1997).Alchemy and beyond: Indexing the Denning Issues Test. Journalof Educational Psychology, 89,498-507.

Schacter, D. L. (1996). Searching for memory. New York: BasicBooks.

Snarey, J. (1985). The cross-cultural universality of social-moraldevelopment. Psychological Bulletin, 97, 202-232.

Taylor, S. E., & Crocker, J. (1981). Schematic bases of socialinformation processing. In E. T. Higgins, C. P. Herman, & M. P.Zanna (Eds.), Social cognition: The Ontario Symposium (Vol. 1,pp. 89-134). Hillsdale, NJ: Erlbaum.

Thoma, S., Narvaez, D., Rest, J., & Derryberry, P. (in press). Thedistinctiveness of moral judgment. Educational PsychologyReview.

Tulving, E., Schacter, D. L., & Stark, H. A. (1982). Priming effectsin word-fragment completion are independent of recognitionmemory. Journal of Experimental Psychology; Learning,Memory, and Cognition, 8, 336-342.

Uleman, J. S., & Bargh, J. A. (1989). Unintended thought. NewYork: Guilford Press.

Youniss, J., & Yates, M. (in press). Youth service and moralidentity: A case for everyday morality. Educational PsychologyReview.

Appendix

Sample Story From DIT2: The Famine

The small village in northern India has experienced shortages of food before,but this year's famine is worse than ever. Some families are even trying to feedthemselves by making soup from tree bark. Mustaq Singh's family is nearstarvation. He has heard that a rich man in his village has supplies of food storedaway and is hoarding food while its price goes higher so that he can sell the foodlater at a huge profit Mustaq is desperate and thinks about stealing some foodfrom the rich man's warehouse. The small amount of food that he needs for hisfamily probably wouldn't even be missed.

What should Mustaq Singh do? Do you favor the action of taking thefood? (Check one)

in 2O 3D 4D 5D 6D 7D

Strongly Favor Slightly Neutral Slightly Disfavor Stronglyfavor favor disfavor disfavor

Rate the following issues in terms of importance (1 = great, 2 = much.

DIT2 659

3 = some, 4 = little, 5 = no). Please put a number from 1 to 5 alongsideevery item.

1. H} Is Mustaq Singh courageous enough to risk getting caught forstealing?

2. EH Isn't it only natural for a loving father to care so much for hisfamily that he would steal?

3. CII Shouldn't the community's laws be upheld?

4. EH Does Mustaq Singh know a good recipe for preparing soup fromtree bark?

5. ED Does the rich man have any legal right to store food when otherpeople are starving?

6. El Is the motive of Mustaq Singh to steal for himself or to steal forhis family?

7. ED What values are going to be the basis for social cooperation?

8. ED Is the epitome of eating reconcilable with the culpability ofstealing?

9. ED Does the rich man deserve to be robbed for being so greedy?

10. ED Isn't private property an institution to enable the rich to exploitthe poor?

11. ED Would stealing bring about more total good for everybodyconcerned or not?

12. ED Are laws getting in the way of the most basic claim of anymember of a society?

Which of these 12 issues is the 1st most important? (write in the number

of the item) | |

Which of these 12 issues is the 2nd most important? I I

Which of these 12 issues is the 3rd most important? | |

Which of these 12 issues is the 4th most important? I I

Note. An information package can be obtained from the Center for theStudy of Ethical Development, University of Minnesota, 206 Burton Hall,178 Pillsbury Drive Southeast, Minneapolis, Minnesota 55455. Electronicmail may be sent to [email protected], or call (612) 624-0876.

Received November 10, 1998Revision received February 16, 1999

Accepted February 16, 1999 •

AMERICAN PSYCHOLOGICAL ASSOCIATION

SUBSCRIPTION CLAIMS INFORMATION Today's Date:_

We provide this form to assist members, institutions, and nonmember individuals with any subscription problems. With theappropriate information we can begin a resolution. If you use the services of an agent, please do NOT duplicate claims throughthem and directly to us. PLEASE PRINT CLEARLY AND IN INK IF POSSIBLE.

PRINT FULL NAME OR KEY NAME OF INSTITUTION

ADDRESS

MEMBERORCTJSTOMERNUMBER(MAYBEFOUNDONANYPASTISSUELABEL)

DATE YOUR ORDER WAS MAILED (OR PHONED)

PREPAID CHECK ___CHAROECHECK/CARD CLEARED DATE:_

CITY STATECOUNTRY

YOUR NAME AND PHONE NUMBER

TITLE

(If possible, send & copy, front and back, of your cancelled check to help us in our researchof your claim.)

ISSUES: MISSING DAMAGED

VOLUME OR YEAR NUMBER OR MONTH

Thank you. Once a claim is received and resolved, delivery of replacement issues routinely takes 4-6 weeks.

— — — ^ — — . (TO BE FILLED OUT BY APA STAFF) — — — — — - — — —

DATE RECEIVED:.ACTION TAKEN: _STAFF NAME:

DATE OF ACTION: _INV.NO.&DATE:LABEL NO. & DATE:_

Send this form to APA Subscription Claims, 750 First Street, NE, Washington, DC 20002-4242

PLEASE DO NOT REMOVE. A PHOTOCOPY MAY BE USED.

DIT2: Devising and Testing a Revised Instrument of …dnarvaez/documents/DIT2pub.pdfDIT2: Devising and Testing a Revised ... Psychology aimed at improving the measurement of ... Department

Documents