RR-86-43-Powers.pdf

7/28/2019 RR-86-43-Powers.pdf

1/61

ANALYTICAL REASONINGINVOLVED IN GRADUATE

PERCEPTIONS OF FACULTY IN

SKILLSSTUDY:

SIX FIELDS

Donald E. PowersMary K. Enright

GRE Board Professional Report GREB No. 83-23PETS Research Report 86-43

December 1986

This report presents the findings of aresearch project funded by and carriedout under the auspices of the GraduateRecord Examinations Board.

7/28/2019 RR-86-43-Powers.pdf

2/61

Analytical Reason ing Skills Involved in Graduate Study:Perceptions of Faculty in Six Fields

Donald E. PowersMary K. Enright

GRE Board Professional Report GREB NO. 83-23P

December 1986

Copyright, 0 1986 by Educ ational Testing Service. All rights reserved.

7/28/2019 RR-86-43-Powers.pdf

3/61

AcknowledgmentsThe authors wish to thank the members f the GREResearch Committeefor helpful suggestions; Laurie Barnett fr>r programming he analyses;Neal Kingston, William Ward, and Cheryl Wild for helpful reviews of the

report.

7/28/2019 RR-86-43-Powers.pdf

4/61

7/28/2019 RR-86-43-Powers.pdf

5/61

Analytical Reasoning Skills Involved in Graduate Study:Perceptions of Faculty in Six Fields

Despite the complexity of humancognitive abilities, standardizedadmissions tests have tended to focus almost exclusively on themeasurement f broadly applicable verbal and quantitatice aptitudes.One criticism of such omnibusverbal and quantitative ability measuresis that they provide only limited descriptions of students' academicstrengths and weaknesses, and that they do not therefore adequatelyreflect test takers' differential development n other importantcognitive areas.In 1974 the GREBoard approved a plan to restructure the GREAptitude Test in order to allow examinees to demonstrate a broaderrange of academic skills (Altman, Carlson, & Donlon, 1975). A surveyof constituents revealed that, of several possible new areas ofmeasurement e.g., abstract reasoning, scientific thinking, and studyskills), graduate faculty, administrators, and students were mostreceptive to assessing analytical or abstract reasoning skills(Miller & wild, 1979). Developmental activities then followed and,after careful psychometric study of several alternative analyticalitem types, four distinct kinds of items were selected for the newanalytical section of the GREAptitude Test, which was introducedoperationally in the 1977-78 testing year. Graduate institutions werecautioned against using the scores from the new analytical sectionuntil further evidence could be generated on the validity of the newmeasure. Subsequently, the administration of the new measure to largenumbersof examineesunder operational conditions enabled the furthercollection of information about the new measure.Some esearch strongly suggested the promise of the analytical

section: it appeared to measure an ability that was distinguishablefrom the verbal and quantitative abilities measuredby the test(Powers & Swinton, 1981), and the score derived from it was related tosuccessful performance in graduate school (Wilson, 1982).Unfortunately, however, further research suggested serious problemswith the two item types (analysis of explanations and logicaldiagrams) that comprised the bulk of the analytical section.Performance on these item types was shown o be extremely susceptibleto special test preparation (Swinton & Powers, 1983; Powers & Swinton,1984) and to within-test practice (Swinton, wild, & Wallmark, 1983).Consequently, in 1981 the two problematic item types were deleted fromthe test, and additional numbersof analytical reasoning and logicalreasoning items, which constituted a very small part of the originalanalytical measure, were inserted.

The most recent research on the General Test (Stricker & Rock,1985; Wilson, 1984) has given us somereason to question both theconvergent and the discriminant validity of the two remaining itemtypes. Specifically, the two currently used analytical item typescorrelate more highly with other verbal items or with otherquantitati ve items thanreviewing the psychologreasoning, Du an, PoweriSthey do with each other. Moreover, aftercal and educational resea ch literature onI? and Swinton (in press) concluded that the

7/28/2019 RR-86-43-Powers.pdf

6/61

-2-two currently used GREanalytical item types reflect only a limitedportion of the reasoning skills that are required of graduatestudents. The most notable omission is the assessmentof inductivereasoning skills, i.e., reasoning from incomplete knowledge, where thepurpose is to learn new subject matter, to develop hypotheses, or tointegrate previously learned materials into a more useful andcomprehensivebody of information. Thus, it seemed, the analyticalability measure of the GPEGeneral Test might be improved throughfurther effort.

The objective of the study reported here was to generateinformation that might guide the developmentof future versions of theGREanalytical measure. More specifically, the intention was to gaina better understanding of what reasoning (or analytical) skills areinvolved in successful academic performance at the graduate level, andto determine the relative importance of these skills or abilities bothwithin and across academic disciplines. It was thought that thisinformation might be especially useful for developing additionalanalytical item types.

As mentioned earlier, the initial version of the GREanalyticalability measurewas developed after a survey had suggested theimportance of abstract reasoning to success in graduate education.This survey, however, was not designed to provide any detailedinformation on the importance of specific analytical skills, as wasthe intention here.

MethodQuestionnaire Development

Initially, 30 department chairs (in English, education,engineering, chemistry, computer science, or psychology) werecontacted in 30 graduate institutions, and asked to identify threefaculty members n their departments who would be willing to providetheir insights into the analytical or reasoning skills that are mostcritical for successful performance in graduate school. These 30institutions were chosen from the GREDirectory of Graduate Programsin such a way as to ensure somedegree of geographical representation.All of these departments require or recommendhat applicants submitGPEGeneral Test scores; it was felt that these departments might bemore interested than nonrequiring departments in efforts to improvethe GREGeneral Test.At this preliminary stage, faculty memberswere informed of thepurpose of the project and asked to give, in an open-ended ashion,examples of:(a) the analytical, reasoning, or thinking skills theyperceived as most important for successful graduatestudy in their fields (e.g., identifying assumptionson which an argument is based), particularly as theseskills differentiate successful from marginal students

7/28/2019 RR-86-43-Powers.pdf

7/61

-3-(b) specific critical incidents related to thinking or reasoningthat caused them to either raise or lower their estimationof a student's analytical ability (e.g., failing to qualifya conclusion as appropriate)(c) particular reasoning or thinking "flaws" they have observedin their students (e.g., using the conclusion as the premiseof an argument).Useable responseswere obtained from 33 faculty members,whosuggested a total of 138 important reasoning or thinking skills, 86critical incidents, and 75 reasoning "flaws." Someof these responseswere duplicates. Several other respondents did not specify discreteskills or errors but chose rather to send helpful discursive repliesto our inquiry. All responseswere condensedand edited, andgenerally evaluated with respect to whether they should be included inthe larger, more structured questionnaire that was planned. Someresponses constituted usable questionnaire items essentially as stated

by respondents (e.g., "the ability to break complexproblems intosimpler components"). Other responseswere revised or eliminatedbecause they were too general (e.g., "the ability to thinkindependently"), and others because they were too specific or appliedonly to a particular field (e.g., "the ability to resolve intoenthymemic orm any argumentative work" or "the ability to takeecological validity into account").The structured questionnaire was constructed on the basis of thispreliminary survey, on a review of relevant literature (Duran, Powers,& Swinton, in press) and on a numberof additional books or texts onreasoning (e.g., Campbell, 1974; Fischer, 1970; Johnson& Blair, 1983;Kahane, 1984, Nosich, 1982; Salmon, 1984; Striven, 1976; Toulmin,

Rieke, & Janik, 1984; Wason& Johnson-Laird (1972); and Weddle, 1978).Several other articles, e.g., a seminal work by Ennis (1962) and alist of skills by Arons (1979), proved especially useful. Variousissues of CT News, published by the Critical Thinking Project atCalifornia State University at Sacramento, were also perused.Previous work on critical incidents in graduate student performance(Reilly, 1974a, 1974b) was also consulted, and several of theincidents related to critical facility were included in the presentstudy. Finally, the list generated by Tucker (1985), who gathered theimpressions of ETS test development staff, philosophers, and cognitivepsychologists, also proved to be a valuable resource.The final questionnaire (see Appendix A) was structured to

include questions about the importance and frequency of variousreasoning skills, of commonly bserved errors in reasoning, and ofspecific incidents that may have led faculty to adjust their estimatesof students' analytical abilities. Questions were grouped underseveral headings, mainly to give respondents somesense of theirprogress in responding to the rather lengthy questionnaire.

7/28/2019 RR-86-43-Powers.pdf

8/61

-4-The Sample

Six academic fields (English, education, psychology, chemistry,computer science, and engineering) were included in the final survey.These fields were thought to represent the variety of fields ofgraduate study and the variation in the kinds of reasoning abilitiesinvolved in graduate education. Using the data tapes of the HigherEducation General Information Survey (HEGIS), nonoverlapping samplesof 64 graduate institutions with doctoral programswere drawn for eachof the six graduate fields. A randomsampling procedure was used suchthat eight institutions from each of the eight HEGISgeographicregions were selected for each field. This sampling was greatlyfacilitated by the work of Oltman (1982). The admission requirementsof these institutions were determined from the Directory of GraduatePrograms (GRE/CGS, 983), and only those that either required orrecommended REGeneral Test scores were included in the sample. Inthis manner, 40 institutions were selected for the final sample foreach field. In addition, one institution with a relatively largeproportion of Black students and one with a relatively largepercentage of Hispanic students were included in the samples for eachfield, thus raising the total numberof institutions to 42 per field.Letters were then sent to departmental chairpersons, who were asked tonominate two faculty memberswho would be willing to complete thequestionnaire. Respondentswere paid $25 for their participation.Data Analysis

Means and standard deviations were calculated for each questionby academic field of study, and analyses of variance were run for eachquestion to assess differences among he six fields. The variousratings were correlated within questionnaire sections. For example,within the section on reasoning-skills, the ratings of frequency-andimportance were correlated; within the section on reasoning errors,the ratings of frequency and seriousness were correlated. -

Finally, within each section (and for each kind of rating), thedata were factor analyzed to effect somereduction in the large numberof questions. A principal axis factoring, with squared multiplecorrelations as the initial estimates of communalities, was used todetermine the numberof factors to be retained for each section,according to both the magnitude of the eigenvalues and the breaks intheir size. (Our inclination was to err on the side of retaining toomany factors at this exploratory stage.) Various numbersof factorswere then rotated according to the varimax criterion. Although otheroblique rotations could have been used also, it was felt that-thedetection of uncorrelated factors would best serve the objectives offurther test development.

7/28/2019 RR-86-43-Powers.pdf

9/61

-5 -Results

The SampleA total of 165 chairpersons (65% of those contacted) nominated a

total of 297 faculty mem bers, of whom 255 (86%) returned usablequestionnaires. The response rates across fields were genera llycomparable.Full professors co nstituted a slight majority of the respond ingsample (51%) ; associate professors made up the next largest proportion(34%). About 13% were assistant professors, and the remaining smallproportion were deans, associate deans, or lecturers.

Item-level ResultsTables 1 -3 sho w the mean ratings by discipline for each questionincluded in the survey instrument. The numbers in the total column

are the grand means for all disciplines. Numbers under eachdiscipline represent for each item the deviations from these means .The F tests in the right-most column indicate whether the means aresignificantly different am ong the six disciplines. Because theaverage ratings, over all respondents, for frequ enc y of use andimportan ce for success correlated .99, only the importan ce ratingsare presented for reasoning skills. Likewise, only the seriousnessratings are presented for reasoning errors, since their correlationwith frequen cy ratings was .98, and, for critical incidents, only theaverage effect ratings are presented, since their correlation withfrequency ratings was .94.Tables 1-3 show a substantial num ber of significant differences

among disciplines with respect to the importance placed on variousreasoning skills (Table l), the seriousness with which they regardparticular kinds of reasoning errors (Table 2), a nd the impact thatvarious critical incidents have on the estimation of studentsanalytical abilities (Table 3). Table 4, show ing only the veryhighest rated skills and most critical errors and incidents, gives aflavor of the differences amo ng these six disciplines. For exam ple,chemistry faculty placed a high premium on being able to generatehypotheses , que stions, or experim ents, to draw sound inferences fromobservations, and to analyze and evaluate previous research. Engl i shfaculty, on the other hand, saw greater importan ce in skills involvingargum entation-- being able to understand, evaluate, analyze, elaborate,recognize, and support aspects of an argument.Faculty in the six disciplines also appeared to have quitedifferent views as to the num bers of skills that were important intheir respective disciplines. The num bers of reasoning skills thatreceived average ratings of 4.0 or higher varied markedly bydiscipline as follows: 23 for chem istry, 5 for com puter science, 27for education, 22 for engineering, 29 for English, and 26 forpsychology. These differences may have arisen, for example, from our

7/28/2019 RR-86-43-Powers.pdf

10/61

7/28/2019 RR-86-43-Powers.pdf

11/61

-7-For the section on reasoning skills, only the importance ratings wereanalyzed because they were so highly correlated with frequencyratings. Becau se frequency ratings were slightly less correlated withratings of seriousness and criticality in the other two sections, theytoo were analyzed for the questionnaire sections on reasoning errorsand critical incidents.

The reader should bear in mind that the factors resulting fromthis analysis should not be construed as representing dimensions ofanalytical ability, but rather only as reflecting the dimensions thatunderlie faculty perceptions of analytical abilities. Thesedimensions merely reflect the extent to which graduate faculty tendedto rate certain skills as about equally important (or equallyunimportant ) , not the degree to which these dimensions representfactors of the min d. Thus, the results presented below are intendedto provide a parsimonious representation of faculty perceptions ratherthan a basis for postulating distinct analytical abilities.Reaso ning skills. For the ratings of importance of reasoningskills, the largest eigenvalues were 16.4, 3.9, 2.4, 1.6, and 1.1, an dthe application of a scree test (Cattell, 1966) suggested theappropriateness of a five-factor solution, which was then rotatedaccording to the varimax criterion (Kaiser, 1958). The five-factorvarimax rotation accounted for 80% of the comm on ariance. The factorloadings and comm unalities are given in Appen dix B. Table 7summ arizes the variables that were most instrumental in defining eachfactor.Factor I, which accounted for about a third of the com monvariance, was characterized by highest loadings, generally, fromskills involving argume nts. Thus, Factor I seems to involve a kind of

critical thinking related to argume ntation.Factor II accounted for about 29% of the com mon ariance, and wasdefined primarily by variables related to the drawing of conclusions,e.g. I generating valid explanations, supporting conclusions withsufficient data, and drawing sound inferences from observations. Theconclusion-oriented skills that define this second critical thinkingfactor would seem to be of a more active or productive nature,involving the construction of inferences or conclusions, rather thanevaluating the soundness of argum ents or inferences, as is the casefor Factor I.Factors III-V each accounted for a somew hat smaller proportion of

comm on ariance (10% - 15%) than did Factors I and II. Factor III isbest defined by skills related to defining and setting up problems oranalyzing their comp onents as a prelude to solving them. Factor IV isbest characterized by inductive reasoning skills, i.e., the drawing ofconclusions that have some evidential support, but not enough toindicate logical necessity. Factor V is somewha t difficult to define,but, by virtue of its two highest loadings, it seems to reflect anability to generate alternatives.

7/28/2019 RR-86-43-Powers.pdf

12/61

-8-Reasoning errors. For the ratings of seriousness of reasoningerrors, the largest eigenvalues were 6.5 and 1.1, and the two factorsaccounted for 96% of the comm on ariance. (Frequency ratings werealso factor analyzed and are presented in Appendix B. Because theresults were so similar to the analysis of seriousness ratings, they

are not discussed here.) As show n in Table 8, Factor I, whichexplained about 52% of the comm on ariance, was characterized byloadings from errors involved in the evaluation of evidence, e.g.,offering irrelevant evidence to support a point. Factor II, on theother h and, seemed to involve more formal logical errors, particularlyas related to reasoning with more statistically oriented material--forexam ple, failing to take accoun t of a base rate, failing to recogn izedifferences between populations and samples, and confusin g correlationwith causation.Critical incidents. As for reasoning errors, only ratings ofthe effects of critical incidents, not their frequencies, were factoranalyzed. This analysis yielded eigenvalues of 8.2, 1.6, and 0.9 for

the largest factors, and this three-factor solution accoun ted for 89%of the comm on ariance. Table 9 summarizes the results, and thecomplete set of loadings is given in Appendix B.Factor I, explaining about 38% of the conanon ariance, was bestdefined by highest loadings from such incidents as accepting/supporting arguments based more on emotional appeal than evidence,offering nonconstructive or unsound criticism of other studentspresentations, and confusin g anecd ote and/or opinion with hard data.This factor appears to involve critical facility.Factor II, accounting for about 34% of the comm on ariance,appears to involve the ability to consider or to generate

alternatives, being defined primarily by high loadings from suchincidents as accepting conclusion s without critically evaluating them,being able to criticize but unable to suggest better alternatives,being unable to integrate ideas from various sources, and being unableto generate hypotheses.Factor III, defined by such incidents as applying a formula orrule without sufficient justification, and searching for a com plicatedsolution whe n a simpler one is obvious, is difficult to interpret.One possible characterization migh t be a kind of rationality orcritical facility that is some times referred to as practical judgm entor perhaps com mon sense. IScales based on reasoning skill factors. Table 10 gives theaverage scores for each discipline on scales composed of thequestionnaire items that best defined the reasoning skill factorsdiscussed earlier. As is clear, there are substantial differencesamo ng disciplines on these scales. Skills involved in analyzing/evaluating arguments (Scale 1) were rated as extremely important inEnglish (m = 4.53), quite important in education (m = 3.83) andpsychology (m = 3.73), and some what less im portant in the other th reedisciplines, particularly com puter science (m = 2.97).

7/28/2019 RR-86-43-Powers.pdf

13/61

-9-Critical thinking skills involved in developing or otherwisedealing with conclusions (Scale 2) were viewed as very important(means greater than 4.0) in all disciplines except com puter science.Abilities involved in analyzing and defining problem s (Scale 3)

were rated as extremely important in computer science (m = 4.05) andengineering (m = 4.00), but less important in other disciplines,especially English (m = 2.70).The inductive reasoning skills reflected on Scale 4 were rated asmod erately im portant on each of the six disciplines. The skillscom posing Scale 5, generating alternatives/hypotheses, were rated veryhigh in psychology (m = 4.21) and in education (m = 3.93), and assom ewhat less important in other disciplines, particularly com puterscience (m = 2.90).

Other Com ments rom RespondentsA number of general comm entswere made about the study--somepositive and some negative. The study was described a lternately asvery well don e and intere sting , but also, by one responden t, as acom plete waste of time. Mos t of the com men tswere positive,however, and many pertained more specifically to the kinds ofquestions that w ere asked. The consensus seemed to be that thequestionnaire was not easy to com plete. More over, faculty in theseveral disciplines som etimes had different ideas as to what kinds ofquestions would have been appropriate. For example, one Englishfaculty mem ber noted the lack of questions on the use of language incritical writing, and a computer science faculty mem ber observed thatquestions on abilities involved in formulating proofs, whic h are vitalto success in com puter science, were only partially covered in the

questionnaire. An education faculty mem ber noted that the survey dida better job of assessing skills associated with hypothesis-testingthan with o ther research skills.Along these same lines, a number of other respondents alsobelieved that the questions w ere more relevant to other disciplinesthan to theirs. Several com puter science professors, for example,characterized the questions as oriented more toward argument thanproblem solving, in which they had greater interest. An engineeringprofessor said that som e of the questions were more pertinent toeducational research than to scientific or technical research, and oneEnglish faculty found that questions seem ed geared to the hardsciences. Finally, som e noted ambigu ities or redundan cies, or

lamented that the questions w ere too fine. Even with the sedifficulties, however, most of the comm ents about questions w erepos itive: Item s seem espe cially we ll cho sen, ques tions areappropriate, ques tions we re quite thor oug h, a good set ofquestions, topics cov ered are critica l, and you r lists are righ ton target. The majority of comments, therefore, suggested that thequestionnaire was pitched at about the right level and includedappropriate kinds of reasoning skills.

7/28/2019 RR-86-43-Powers.pdf

14/61

-lO-A number of commentswere made about the relationship betweensubject matter and analytical skills, e.g., that success ful problemsolving is predicated on having specific know ledge in a field. Oneresponde nt believed that the questionnaire dow nplayed the impo rtanceof con text effe cts in favo r of strict reas onin g ability, and

another noted that the meas urem ent of analytical abilities is quitediscipline specific. Another commentedon the difficulty of mea suringanalytical ability without regard to the amo unt of know ledgeavailable.Several faculty comm entedon the developm ent of analytical skillsin graduate school and on the differential importan ce of these skillsat various stages of graduate edu cation. one responde nt said, Irated entering behavior o r behavior across the entire program(courses, internships, dissertation). If I were to rate thedissertation experienc e alone, the ratings would have been muchhigher. Man y noted that by the end of their program s, skills wouldbe expected to increase and various reasoning errors could be expected

to occur less frequently: Entering students are mo re likely to mak ethese errors and graduates to make far few er. Ano ther said, In som esense, the essence of graduate tra in ing is analyt ical ski lls. Theseare skills which students acquire. Flhen they enter they make most ofthe mistakes you men tioned. If they can learn, they leave theprogram . Ano ther said, Im more concerned about the presence ofthese behaviors after my course than before it. One simply does notharshly judge a beginning student who makes an error, but one could bevery critical of a student about to finish a Ph.D. thesis....

DiscussionSummary

Som e 255 graduate faculty in six fields of study (chem istry,computer science, education, engineering, English, and psychology) (a)rated the imp ortance for success in their program s of a wide varietyof reasoning skills, (b) judged the seriousness of various reasoningerrors, and (c) indicated the degree to which a variety of criticalincidents affected their estimations of students analyticalabilities.skillskillFacul

Faculty memb ers were able to di scriminate among the variouss they were asked to consider, although the vast majority ofs were seen, on average, as at least mod erately important.ty also made distinctions with respect to both the seriousne ss ofdifferent kinds of reasoning errors and-the effects of variouscritical incidents. Individually, such general skills as easoningor problem solving in situations in which all the needed informationis not know n were viewed as extrem ely importan t in all disciplines.Sucfieaso ning errors and critical incidents as accepting the centralassum ptions in an argum ent without questioning them,-being unable tointegrate and synthesize ideas from various sources, and beingunable to generate hypothese s indepen dently were judged to be veryserious in all disciplines.

7/28/2019 RR-86-43-Powers.pdf

15/61

-ll-More typically, however, disc iplines varied quite markedly wirespect to faculty perceptions of the importance, seriousness, orimpact of various reasoning skills , errors, or incidents.expected, for example, As mighwhereas "knowing he rules of formal logic"rated as one of the most important skills in computer science, thi

skill was rated as quite unimportant in all other disciplines.

th.t be

WEISS

Several relatively interpretable dimensions were found tounderlie faculty perceptions of the substantial numberof reasoningskills, errors, and incidents that were rated in the study. Skillsinvolved in (a) the analysis and evaluation of arguments, (b) thedrawing of inferences and the developmentof conclusions,, (c) thedefinition and analysis of problems, (d) the ability to reasoninductively, and (e) the generating of alternative explanations orhypotheses formed five distinct dimensions, which were perceived asdifferentially important for success in each of the six disciplinesincluded in the study. Analysis and evaluation of argumentswasjudged to be most important in English, defining and analyzingproblems most important in computer science and engineering, andgenerating alternatives most important in psychology and education.Inductive reasoning skills were judged to be about equally importantin all disciplines, and drawing inferences/developing conclusions wasrated as relatively important in all disciplines except computerscience.Implications

In providing some nformation on faculty perceptions of theinvolvement of various reasoning skills in their disciplines, thestudy has, we hope, implications for developing future versions of theGREanalytical ability measure. Converting this information tooperational test items will represent a significant step, however, andit is not crystal clear at this stage exactly how helpful theseresults may be eventually. Nonetheless, the findings do seem ocontain several useful bits of information:

1. Among he specific reasoning skills perceived as the mostimportant were several, e.g., "deducing new information froma set of relationships" and "understanding, evaluating, andanalyzing arguments," that seemwell represented in the twoitem types (analytical reasoning and logical reasoning)currently included in the analytical section of the GeneralTest. This suggests that these item types should continue toplay a role in future editions of the GREGeneral Test.2. Someskills that are not measuredby the current version ofthe analytical measurewere rated as very important."Reasoningor problem solving in situations in which all theneeded information is not known"was among he skills ratedas most important in each discipline, but currentlyunmeasured,at least in any explicit manner, by theanalytical measure. In this regard, however, the previous

7/28/2019 RR-86-43-Powers.pdf

16/61

-12-GRE-sponsored work of Ward, Carlson, and Woisetschlage r(1983) is noteworthy. These investigators studiedill-structu red prob lems , i . e. , problems that do not provideall the information necessary to solve the problem, a nd notedthe resem blance of these problems to one variant of thelogical reasoning item type used in the analytical measu re.They concluded that there w as no indication that ill-structured problems measu re different aspects of analyticalability than do we ll-struct ured proble ms, and there forethat ill-structu red prob lems cou ld not be expe cted toextend the range of cognitive skills already measu red by theGR E General Test. They did note, howev er, that theill-structu red item type cou ld be used to increa se thevariety of items types in the test. The findings of thecurrent study suggest that the inclusion of this item typewould probably m eet with faculty approval in most fields ofstudy.

3. With respect to their perceived importance, skills involvingthe generation of hypotheses/alternatives/explanations tendedto cluster together, and the inability to generate hypothesesindependen tly was one of the incidents rated consistently ashaving a substantial effect on faculty perceptions ofstudents analyt ical abil it ies.A number of years ago the GRE Board sponsored a seriesof studies (Frederiksen & Ward, 1978; Ward, Frederiksen, &Carlson, 1978; Ward & Frederiksen, 1977; Fred eriksen & Ward,1975) that explored the developm ent and validation of testsof scientific thinking, including one especially promisingitem type called Form ulating Hyp othes es, wh ich requ ired

exam inees to generate hypotheses. Although the researchsuggested that this item type c omplem ented the GRE verbal andquantitative measu res in predicting success in graduateschool, the work was discontinued, largely because ofproblems in scoring items that require examinee s toconstruct, not merely choose, a correct response. Carlsonand Ward (1986) have proposed to renew work on theForm ulating H ypotheses item type in light of recentadvances in evaluating questions that involve constructedresponses. The results of the faculty survey reported herewould appear to support this renewal.4. Som e of the highly important skills that are currently

well represented in the analytical measu re are viewed as moreimportant for success in some disciplines than in others.For example, understanding, analyzing, and evaluatingargum ents was seen as more important in En glish than incom puter science. How ever, some skills seen as highlyimportant in som e disciplines but not in others may not be aswell represented currently. For examp le, breaking downcom plex problems into simpler ones was perceived as

7/28/2019 RR-86-43-Powers.pdf

17/61

5.

6.

-13-extremely important in computer science and engineering butnot at all important in English. This would suggest,perhaps, the need to balance the inclusion of itemsreflecting particular skills, so that skills thought to beimportant (or unimpo rtant) in particular disciplines areneither over- nor underrepresented.The several dimensions that app ear to underlie clusters ofreasoning skills may provide an appropriate way to extend thecurrent test specifications for the analytical measu re,especially if new item types are developed to represent someof these dimensions.The reasoning skills that were rated as very important, andconsistently so, across disciplines point to a potentialcom mon ore of skills that c ould be appropriately included inan all-purpose measu re like the GR E General Test. Otherskills judged to be very im portant in only a few disciplinesmight best be considered for extending the measurem ent ofreasoning skills in the GR E Subject T ests. Faculty commentsabout the difficulty in separating reasoning from subjectmatter knowledg e would seem to suppo rt this strategy.

LimitationsAny study of this nature is necessarily limited in severalrespects. First of all, the survey approach used here is but one ofseveral that can be used to inform decisions about extending themeasu rement of analytical abilities. Tuckers (1985) results provideuseful information from different perspectives--those of cognitivepsychologists and philosophers. Other approaches that might also beinformative include the methods of cognitive psycholog y, which could

be used not only to supplemen t but also to extend the s urvey resu ltsreported here. These methods would seem especially appropriatebecause they relate more directly to actual skills and abilities thanto perceptions.Second, the diversity that characterizes graduate educationrenders the results of this study incomplete. Some clues have beengained as to similarities and differences amo ng a limited sample ofgraduate fields. How ever, the substantial differences found amon gfields are a source of concern, since we cannot be certain whethe r ornot some other sample of fields might exhibit even greater variation.Finally, as several survey respondents pointed out, many of the

reasoning skills about which we asked are expected to, and do, im proveas the result of graduate study. In som e sense these skills mayrepresent comp etencies that differ from, say, the verbal skillsmeasured by the GEN E eneral Test in the respect that these an alyticalskills may develop mu ch more rapidly. A question of interest, then,is how to accom moda te the measurem ent of these skills in the contextof graduate admissions testing, which currently focuses on thepredictive effectiveness of abilities that are presum ed to developslowly over a significant period of time.

7/28/2019 RR-86-43-Powers.pdf

18/61

-14-Future Directions

The study suggested several possible future directions. Becauseof the substantial variation amon g fields, one possibility wouldinvolve extending the survey to include additional fields of graduatestudy. Som e refinements could now be made on the basis of pastexperience. For example, ratings of the frequenc y with w hich skillsare used, as well as the frequencies of errors and critical incidents,could probably be omitted without mu ch loss o f informa tion. OII theother hand, it would seem desirable to add categories allowing ratingsof the differential importance of various reasoning skills atdifferent stages of graduate education, ranging from entry level todissertation writing.

Finally, based on the reasoning skills identified as mostimportant, criterion tasks might be developed against w hich thevalidity of the current GR E analytical meas ure could be gauged. Thisstrategy would make especially good sense for those important skillsthat may not be measu rable in an operational test like the GRE GeneralTest, but which might correlate highly with the abilities now measu redby the test. One specific possibility would be the developm ent ofrating forms, which could be used by faculty to rate the analyticalabilities of their students. These ratings could then be used as acriterion against which GR E analytical scores could be judged.

7/28/2019 RR-86-43-Powers.pdf

19/61

-15-References

Altman, R. A., Carlson, A. B., & Donlon, T. F. (1975). Developing aplan for the renewal of the GREnorming test: A report on, Iinvestigations znto the possible shortening of the GRE-VandGRE-Qand the creation of a modular aptitude test (GREReportGREBNo. 74-3). Princeton, NJ: Educational Testing Service.

Arons, A. B. (1979). Some houghts on reasoning capacities implicitlyexpected of college students. In J. Lochheadand J. Clement(Eds.), Cognitive process instruction. Philadelphia, PA: TheFranklin Institute Press.Campbell, S. K. (1974). Flaws and fallacies in statistical thinking.EnglewoodCliffs, NJ: Prentice-Hall, Inc.Carlson, S. B., & Ward, W. C. (1986). A new look at formulatinghypotheses items (proposal to the GREResearch Committee).Princeton, NJ: Educational Testing Service.Cattell, R. B. (1966). The scree test for the numberof factors.Multivariate Behavioral Research, 1, 245-276.-Duran, R. P., Powers, D. E., & Swinton, S. S. (in press). Constructvalidity of the GREanalytical test (Draft final report to theGREResearch Committee, January 1984). Princeton, NJ:Educational Testing Service.EMiS, R. H. (1962). A concept of critical thinking: A proposed basisfor research in the teaching and evaluation of critical thinkingability. Harvard Educational Review, 32, 81-111.Fischer, D. H. (1970). Historians' fallacies: Toward a logic ofhistorical thought. New York: Harper & Row, Publishers.Frederiksen, N., & Ward, W. C. (1975). Developmentof measures for thestudy of creativity (GREBoard Professional Report GREBNo.72-2P; also ETS RB No. 75-18). Princeton, NJ: EducationalTesting Service.Frederiksen, N., & Ward, W. C. (1978). Measures for the study ofcreativity in scientific problem-solving. Applied PsychologicalMeasurement, 2, l-24.-Graduate Record Examinations Board and Council of Graduate Schools in

the United States. (1983). Directory of graduate programs: 1984and 1985 (4 volumes). Princeton, NJ: Educational TestingService.Johnson, R. H., & Blair, J. A. (1983). Logical self defense (2nd ed.).Canada: McGraw-Hill Ryerson Limited.

7/28/2019 RR-86-43-Powers.pdf

20/61

7/28/2019 RR-86-43-Powers.pdf

21/61

-17-Toulmin, S., Rieke, R., & Janik, A. (1984). An introduction toreasoning (2nd ed.). New York: Macmillan Publishing Company,Inc.Tuck r, C. (1985). Delineation of reasoning processes important to theconstruct validity of the analytical test (GREReport submitted

to the GREResearch Committee). Princeton, NJ: EducationalTesting Service.Ward, W. C., Carlson, S. B., & Woisetschlaeger, E. (1983). Ill-structured problems as multiple-choice items (GREBoardProfessional Report GREBNo. 81-18P). Princeton, NJ:Educational Testing Service.Ward. W. C., & Frederiksen, N. (1977). A study of the predictivevalidity of the tests of scientific thinking (GREBoardProfessional Report GREBNo. 74-6P; also ETS RB No. 77-6).Princeton, NJ: Educational Testing Service.Ward, W. C., Frederiksen, N., & Carlson, S. (1978). Construct validityof free-response and machine-storable tests of scientificthinking (GREBoard Professional Report GREBNo. 74-8P; also ETSRB No. 78-15). Princeton, NJ: Educational Testing Service.Wason, P. C., & Johnson-Laird, P. N. (1972). Psychologyof reasoning:Structure and content. Cambridge, MA: Harvard Universtiy Press.Weddle, P. (1978). Argument: A guide to critical thinking. New York:McGraw-Hill Book Company.Wils on, K. M. (1982). A study of the validity of the restructured GREAptitude Test for predicting first-year performance in graduate

(GREBoard Research Report GREBNo. 78-6R). Princeton, NJ:Testing Service.Wilson, K. M. (1984). The relationship of GREGeneral Test item-typepart scores to undergraduate grades (GREBoard ProfessionalReport GREBNo. 81-22~). Princeton, NJ: Educational TestingService.

7/28/2019 RR-86-43-Powers.pdf

22/61

Table 1Mean Ratings of Importance of Reasoning Skills by Discip lines

Variable

ComputerTota I Chem i stry SC ience Education Engineering Engi ish Psychology F(N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46) -

Reasoning Skills--General

Deriving principles from facts or cases 4.02 -.21 -.04 -.09 -.23 .23

Reasoning when facts are known explicitly 3.63 .31 .55 -.06 .44 -.59

.35 3.80**

-.65 10.81***

Reasoning when all information is not known 4.24 -.I1 -.15 -. 10 .29 -.06

Reasoning when inconsis tencies are present 4.03 .07 -.59 -.03 -.Ol .22

.13 1.60

.34 5.92***

Knowing rules of formal logic 2.76 -.46 1.03 -.21 .lO -.31 -.15 9.62

Applying principles to a different specialty 3.75 -.05 -.08 -.13 .23 -.ll .14 1.32

Supporting or refuting a given position 3.52 -.23 -.62 .47 .47 .23 7.74***

Analyzing and evaluating previous research .13

Incorporating isolated data into a framework .38

Monitoring progress in solving problems

4.33

3.37

3.71 0.20

-.66

-.40

.I5

.05

-.34

.02

-.09

.I2

.21 .25 7.63***

-.16 .04 .23

.29 -.24 -. 12

3.19

1.71

Deriving principles that can be applied toother cases 3.68 .16 -.06 .09

Recognizing the probabil istic nature of events 3.24 -.63

Understand i ng and eva I uat ing arguments 4.05

0.11

-.40

.oo

.03

-.56

Deducing information from relationships 3.86 -.16

.31

.09

.02

.44

0.79

9.49

-.38

.16

.ll -.18

-.40 .70

.70 .14

-.07 .Ol

10.51***

0.60

Knowing what evidence will support ahwotheses or thesis, - . 4.19 .32 -.75 .14 -.26 .31 .22 10.35

7/28/2019 RR-86-43-Powers.pdf

23/61

Table 1 (Continued)

Discipline

Variable

ComputerTota I Chem 1stry SCI ence Ed ucat ion Engineering Engl ish Psychology +(N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46)

Specific Skills--Problem Definition

Identifying central issues or hypotheses 4.34 -.13 -.34 .16 .Ol .09 .22 3.20

Recognizing sides of an issue 3.58 -.15 -.75 .39 -,38 .62 .26 10.59

Generating alternative hypotheses 3.87 .05 -. 50 .16 -.24 .13 .39 4.80***

Breaking down complex problems 4.06 .05 .57 -.13 -.85 -. 17 14.34***

Identifying variables in a problem 3.88 .36 .02 .16 -.59 -.47

Identifying approaches to solving problems 3.98 -.25 -.05 -.20 .oo

7.65**L

3.03 LoI

Recognizing similarities between problems 3.83 -.25 .03 -.06 .13 1.16

Developing operational definitions 3.66 .36 .02 .46 5.49***

Setting up formal models 3.36 -. 93 13.49

Recognizing the historical context 2.97

-.66

-. 12

-.70

.59

-.74

.43

-.02

-.07

-.22

.41

.54

.51

.07

.17

-. 11

.73

-.55 1.36 26.65***

Specific Skills--Constructing Theses orArguments

Supporting assertions with details 3.92 -.08 -.60 .13 -.22 .74

Making explicit components In a chain ofreason i ng

Distinguishing relevant and irrelevantinformation

-.04

.22

.04

-.09

.11

11.22**t

3.57 -.17 .06 -. 10 .Ol .29 1.10

4.20 .05 -.45 .09 -. 10 .30 3.78**

7/28/2019 RR-86-43-Powers.pdf

24/61

Table 1 (Continued)

Discipline

Variable

ComputerTota I Chemistry Science Education Engineering Engl ish Psychology E(N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46)

Perceiving relationships among observations 4.10 .15 -.70 .03 .04

Drawing distinctions between similar ideas 3.67 -.48 -.30 .19 -.13

Using analogies appropriately 3.36 -.38 -.26 .12 .18

Elaboratlng an argument 3.80 -.37 -. 57 .30

Drawing sound inferences 4.25 .32 -.74 .25

Producing a consistent argument 4.12 -.54

Synthesizing different positions

Translating graphs or symbolic statements

3.10

3.55

.15

-.34

.40

-.26

.45

.lO

.33

.02

-.43

l 03-.05

-.lO

.50

Specific Skills--Analyzing Arguments

Recogn I zi ng the central argument 4.04 .12 -.57 -.02 -. 11Identifying assumptions 3.98 -.17 -.12 -.03 .06

Testing the validity of an argument 3.57 -.09 .27 -.23 .02

Finding alternative explanations 3.82 .16 -.56 .28 -. 16

Recognizing supporting points 3.64 .04 -. 52 .Ol

Detecting fallacies and contradictions 3.92 .08 -.36 .06

Distinguishlng major points in an argument 3.28 -.12 -.52 -. 14

-.20

-.22

.02

.15 .33

.56 .16

.53 -.18

1.02 .04

-.04 .18

.35 -.Ol

.22 .16

-1.44 .07

.57 .oo

.47 -.20

-.09 .13

-.27 .55

.61 .06

.22 .21

.63 .11

7.74***

6,67***

3.93**

18.59

8.84

4.67 I2. go* 7

18.17

7.13***

3.14***

1.19

7.89***

6.84

2.18

5.84

7/28/2019 RR-86-43-Powers.pdf

25/61

Table 1 (Continued)

Discipline

Variable

ComputerTota I Chemi stry Sc ience Education Engineering Engl ish Psychology F(N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46) -

Being sensitive to different evidence 3.31 -.34 -. 96 .71Recognizing shifts in meaning of an argument 2.90 -.90 -.69 .52

Judging whether a thesis has been supported 3.86 -.08 -.81 .24

Determining whether conclusions are supported 4.25

Distinguishing among factors 3.45

.lO

-. 20

-. 79

-.35

.27

.lO

-.42 .05 .95 18.86***

-.55 1.48 .14 33.50***

-. 32 .73 .25 12.76***

.05 .11 .27 8.65***

.31 -.24 .38 3.31**

Specific Skills-- Drawing Conclusions

Generating vat id explanations 4.16 .18 -.76 .27 .07 .lO .14 8.26

Supporting conclusions 4.24 .15 -.77 .31 .02 .22 .07 8.92***

Recognizing implications of an interpretation 4.07 -.07 -.72 .26 -. 28 .59 .23 14.82***

Qualifying conclusions 3.99 .06 -.73 .13 -.15 .40 .29 8.90

Generating new questions or experiments 3.88 .53 -.58 .07 .21 -.61 .38 10.08

Considering alternative conclusions 3.85 .24 .25 -. 16 .41 6.33***

Comparing conclusions 3.91 .28

-. 59

-.47

-. 77

.04

.14

.18

-. 14

-.09

.07

.05 3.25**

Revising views for new observations 3.84 .30 .07 .18 6.38

Note. Entries under individual discipl ines are deviations from the grand means given under ltTotaI.ll Standard deviations fortotals range from .78 to 1.31, and 33 of 56 are between .9 and 1.1.p < .Ol*yp < .05p < .OOl-

7/28/2019 RR-86-43-Powers.pdf

26/61

Table 2

Mean Ratings of Seriousness of Reasoning Errors by Discip lines

Disci pl ine

Variable

ComputerTota I Chemistry SC ience Ed ucat i on Eng ineer i ng Engl ish Psychology f(N=i55) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46)

Error

Faillng to recognize differences betweensample and population 2.92 -.59 -.55 .64 .41 -.82 .91 16.90***

Failing to account for base rate 2.64 -.18 -.62 .36 .14 -.03 1.34 24.59***

Offering irrelevant statements 3.05 -.46 -.66 .58 -.53 .41 .66 8.27***

Failing to recognize a concealed conclusion 3.34 -.42 -.45 .34 -.24 .45 .32 6.31**

Confusing coincidence with causation 3.60 -.47 -.44 .49 -.22 -.20 .83 9.51***

Using an ambiguous term 3.02 -.66 -.62 .18 -.38 .93 .54 15.13

Arguing to accept an unproven conclusion 3.20 -.20 -.07 .lO

.09

-.21

-.12

-.26 .65 3.96**

Accepting assumptions with questioning 3.96 -.23 -.22 .24 .24 2.14

Fai I ing to evaluate the credib il ity of asource 3.74 -.23 -.63 .35 -.09

Offering irrelevant evidence 3.48 -.40 -.53 .27 -.18

.37 .23 5.55

.61 .23 7.49***

Making general izations from insufficientev i d ence 3.86 -.24 -.42 -.14

inappropriately interpreting text 3.49 -.52 -.61

.31

.39 -.28

.30 .20 3.97**

.62 .40 11.22***

7/28/2019 RR-86-43-Powers.pdf

27/61

Table 2 (Continued)

Discipline

Tota IComputer

Chemistry Sc fence Education Engineering Engl ish Psychology _F_Variable (N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46)

Allowing anecdotal information 3.23 -. 64 -. 33 .72 .Ol -.51 .74 10.32***

Basing conclusions on partial data 3.57 -.08 -.36 .09 .oo .16 .19 1.50

Fail ing to recognize simi larities inanalogies 3.06 -.33 -.42 .28 .03 .17 .28 3.46

Note. Entries under individual disciplines are deviations from the grand means given under Total.~ Standard deviations fortotals range from .98 to 1.48, and 8 of 15 are between 1.0 and 1.2.*p < .Ol-**p < .05***p < .OOl-

7/28/2019 RR-86-43-Powers.pdf

28/61

Table 3

Mean Ratings of Effects of Critical Incidents by Dlscipl lnes

Discipline

Variable

ComputerTota 1 Chem i stry Science Ed ucat i on Engineering Engl(N=255) (N=37) (N=43) (N=42) (N=43) (N=44)

Incidents

Making irrel evant remarks 3.23 -.23 -.46 -.33

Offering nonconstructive criticism 2.79 -.22 -.40

.26

.35 -.23

Submitting a nonresponsive paper 3.67 .17 -.32 -.05 -.23

Accepting an argument on emot iona I appeal 3.34 -.23 .49 -.46Conf usi ng anecdote with hard data 3.24 -.24

-.77

-.45 .28 -. 14

Accepting uncritically conclusions ofauthorities 3.46 -.42 -.23

Being unable to integrate ideas 3.96

-.ll

-.15 -.19

.15

-.05 -.lO

Failing to recognize that evidence cansupport conclusions 3.39 -.12 -.39 .11

-.17

-.22

.05

-.29

.Ol

-.18

Using inconsistent statements 3.33 .12 -.38

Argulng for the obvious 2.76 -.09 -.16

-.oo

-. 14

.12

.14

.09

Unable to perceive analogy breakdown 2.98 -.03 -.35

Falling to draw conclusions about a pattern 3.41

3.44

.18 -.18

Making implausible assumptions .23 -.42

7/28/2019 RR-86-43-Powers.pdf

29/61

Table 3 (Continued)

Discipline

computerTota I Chemistry Science Ed ucat ion Engineering Engl ish Psychology F-Variable (N=255) (N=37) (N=43) (N=42) (N=43) (N=44) (N=46)

Applying rules without justification 3.38 .70 .11 -.33 .72 -1.06 -.14 16.24

Relying solely on narrative 3.68 -.12 -.ll .I3 -.25 .50 -.16 2.99

Preferring complex to simple explanations 3.23 .07 -.ll -.06 -.23 .I6 .a1Unable to see a pattern 3.76 .13 -.32 .02

.39

.47

-.ia

-.12

-.28

.oa

.30

.I7

-.05 .23.ia

-.oo 1.61Fai I ing to qual ify conclusions 3.44 .07 -.56 -.17 -.Ol .2a 4.99

Writing a one-sided essay 3.01 -.30 -.60 -.28 .47 25 7.46Searching for a complicated solution 3.28 .02 .16 .23 -.19 -.04 1.11Ignoring detai Is 3.74 .20 -.37 -.04 .26 .082 .23

Unwilling to respond in an unknown situation 3.28 .56 .06

Unable to generate hypotheses 3.94 .14

-.ll

.31

-.27

Unable to suggest a I ternat i ves 3.43

Unable to suggest tests of hypotheses 3.52

-.15

-.28

.20 a.35 -.19 3.76

-.06 -.12 .23

-.04

-.19

-.oo .oo

-.64 .63

1.54

.90

8.15

Note. Entries under individua l discip l ines are deviations from the grand means given under Total . Standard deviations fortotals range from .93 to 1.27, and 15 of 25 are between 1 O and 1.2.p < .Ol**p < .05**,, < ,001

7/28/2019 RR-86-43-Powers.pdf

30/61

Table 4Reasoning Skills, Errors, and Incidents Rated as

Most Importan t or Critical by Disciplines

Discipline Skills Errors/Incidents

Chemistry Drawing sound inferences fromobservations (4.57)

Critically analyzing andevaluating previous research orreports in a field (4.46)Generating new questions orexperiments to extend or supportthe interpretation of data (4.42)

ComputerScience Breaking down complex problemsor situations into simpler ones(4.62)

Reasoning or solving problems insituations in which all factsunderlying a problem situation areknown (4.19)

Education Supporting conclusions with sufficientdata or information (4.55)

Determining whether the conclusionsdrawn are logically consistent with,and adequately supported b y, thedata (4.52)

Applying a formula, algorithm, orother rule without sufficientjustification (4.08)Being unable to generatehypotheses independently (4.08)

Being unable to integrate andsynthesize ideas from varioussources (3.77)

Making generalizations on thebasis of insufficient evidence(4.17)Confusing coincidence and/orcorrelation with causation (4.10)Failing to evaluate the credibilityor reliability of a source ortext (4.10)

7/28/2019 RR-86-43-Powers.pdf

31/61

Table 4 (Continued)

Discipline Skills Errors/Incidents

Clearly identifying central issuesand problems to be investigated orhypoth eses to be tested (4.50)Drawing sound inferences fromobservations (4.50)

Engineering Breaking down complex problems orsituations into simpler ones (4.60)Reasoning or solving problems insituations in which all the neededinformation is not known (4.53)

English

Identifying all the variables involvedin a problem (4.40)

Elaborating an argument and developingits implications (4.82)Understanding, analyzing, andevaluating arguments (4.75)

Supporting general assertions withdetails (4.66)Recognizing the central argument orthesis in a work (4.61)

Applying a formula, algorithm, orother rule without sufficientjustification (4.09)

Being unable to integrate andsynthesize ideas from varioussources (4.27)Accepting the central assumptionsin an argument without questioningthem (4.20)Relying so lely o n narrative ordescription in papers and reportswhen analysis is approp riate (4.18)

7/28/2019 RR-86-43-Powers.pdf

32/61

Table 4 (Continued)

Discipline Skills Errors/Incidents

Psychology Critically analyzing and evaluatingprevious research or reports in afield (4.58)Confusing coincidence and/orcorrelation with causation (4.43)

Clearly identifying central issuesand problems to be investigated orhypotheses to be tested (4.57)

Accepting the central assumptionsin an argument without questioningthem (4.20)

Determining whether the conclusionsdrawn are logically consistent with,and adequately supported by,the data (4.52)

Note. Num bers in parenthes es are the average ratings for each skill, error, or incident foreach discipline.

7/28/2019 RR-86-43-Powers.pdf

33/61

-29-Table 5

Reasoning Skills Rated Consistently As At LeastModerately Important

Skill MeanRatingReas oning or solving problems in situations inwhich all the needed information is not knownDetecting fallacies and logical contradictionsin argumentsDeducing new information from a set ofrelationshipsReco gnizing structural similarities between onetype of problem or theory and anotherTaking well-known principles and ideas from onearea and applying them to a different specialtyMonitoring ones own progress in solving problemsDeriving from the study of single cases structuralfeatures or functional principles that can beapplied to other casesMak ing explicit all relevant com ponents in a chainof logical reasoningTesting the validity of an argument by searchingfor counterexamples

4.24

3.92

3.863.83

3.753.71

3.68

3.57

3.57

Note. Moderately important is defined as having an averagecrating over all disciplines of 3.5 or greater. There wereno significant differences amo ng disciplines with respectto the average importance of these skills.

7/28/2019 RR-86-43-Powers.pdf

34/61

-3o-Table 6

Errors o r Incidents Rated Consisten tly as at LeastModerately Serious or Critical

Error/Incident MeanRatingAccepting the central assumptions in anargument without questioning them 3.96Being unable to integrate and synthesizeideas from various sources 3.96Being unable to generate hypotheses independently 3.94Being unable to see a pattern in results or togeneralize when appropriate 3.76Ignoring details that contradict an expected ordesired result 3.74Subm itting a paper that failed to address theassigned issues 3.67Basing conclusions on analysis of only part ofa text or data set 3.57Note. Mod erately c ritical is defined as having an averagecrating over all disciplines of 3.5 or greater. There wereno significant differences amo ng disciplines with respectto the average ratings of seriousness or criticality.

7/28/2019 RR-86-43-Powers.pdf

35/61

-31-Table 7

Summaryof Variables Defining Factors Und erlyingRatings of Imp ortance of Reasoning Skills

Factor Variables Loading Highest on FactorsI Recognizing shifts in the meaning of a wordin the course of an argument

Elaborating an argument and developing itsimplicationsRecognizing supporting points in an argumentReco gnizing the historical context of a problemUnderstanding, analyzing, and evaluating argumentsJudging whether a thesis has been adequatelysupportedRecognizing the central argument or thesis in a work

II Gen erating valid explanations to accoun t forobservationsSuppo rting conclusion s with sufficient data orinformationDrawing sound inferences from observationsDetermining whether the conclusions drawn arelogically consistent with, and adequatelysupported by, the dataComp aring conclusions w ith w hat is already knownCons idering alternative conclusion s

III Setting up formal models for problems underconsiderationBreaking down complex problems or situations intosimpler onesTranslating graphs or sym bolic statements intowords and vice versaIdentifying all the variables involved in a problemKnow ing the rules of formal logic

(73)

(72)(70)(67)(64)(64)(63)

(72)

(6%(69)

w3)(67)(5%

(69)(59)

(57)(56)(53)

7/28/2019 RR-86-43-Powers.pdf

36/61

-32-Table 7 (Continued)

Factor Variables Loadincr Hichest on FactorsIV Recognizing structural similarities between onetype of problem or theory and another (61)

Drawing distinctions between similar but notidentical ideas (53)Synthe sizing two different positions into a thirdposition (44)Deriving from the study of single cases structuralfeatures or functional principles that can beapplied to other cases (41)

V Finding alternative explanations for observations (53)Gen erating alternative hypothes es (46)

Note.given. Only the variables that best characterize the factors areLoadings are given in parentheses for the factor on whichthe variables loading was most predominant.

7/28/2019 RR-86-43-Powers.pdf

37/61

-33-Table 8

Summaryof Variables Defining Factors UnderlyingRatings of Seriousness of Reasoning Errors

Factor Variables Loadina Hiuhes t on FactorsI Offering irrelevant evidence to support a point

Mak ing generalizations on the basis of insufficentevidenceFailing to evaluate the credibility or reliabilityof a source or textAccepting the central assumptions in an argumentwithout questioning them

II Failing to take into accoun t the base rate for aphenomenon n a populationFailing to recognize differences between a sampleand a population of interestOffering irrelevant statements about a personscharacter or circumstances to oppose his/herconclusionConfusing coincidence and/or correlation withcausation

(75)

(72)

(71)

(70)

(76)

(68)

63 )

(61)Note.given.the va

Only the variables that best characterLoadings are given in parentheses forriables loading was most predominant.ize the factors areth e factor on which

7/28/2019 RR-86-43-Powers.pdf

38/61

7/28/2019 RR-86-43-Powers.pdf

39/61

-35-Table 10

Means of Scales Based on Items that Best D efinedFactors by Disciplines

ScaleDiscipline 1 2 3 4 5Chem istry 3.34 4.29 3.57 3.38 3.43Com puter Science 2.97 3.37 4.05 3.58 2.90Education 3.83 4.29 3.44 3.73 3.93Engineering 3.25 4.11 4.00 3.61 3.44English 4.53 4.10 2.70 3.87 3.46Psychology 3.73 4.26 3.37 3.78 4.21Total 3.61 4.07 3".52 3.66 3.56

Key. Scales are unwe ighted comp osites of the following variables:Scale 1 = Recog nizing shifts in the meaning of a word in the cou rse ofan argument

Scale 2 =

Elaborating an argum ent and developing its implicationsRecognizing supporting points in an argumentRecog nizing the historical context of a problemJudging whether a thesis has been adequately supportedRecog nizing the central argum ent or thesis in a workUnderstand ing, analyzing, and evaluating argum entsGenerating valid explanations to account for observationsDrawing sound inferences from observationsSuppo rting conclusions with sufficient data or informationDeterm ining whethe r the conclusions d rawn are logically

Scale 3 =

Scale 4 =

Scale 5 =

consistent with, and adequately supported by, the dataComparing conclusions with what is already knownConside ring alternative conclusionsRevising a previously held view to account for new observationsSetting up formal mode ls for problems under considerationBreaking d own comp lex problems or situations into simpler onesTranslating graphs or symbolic statements into w ords and viceversaIdentifying all the variables involved in a problemKnow ing the rules of form al log icReco gnizing structural similarities between one type of problemor theory and anotherDraw ing distinctions between similar but not identical ideasDeriving from the study of single cases structural features orfunctional principles that can be applied to other casesSynthesizing two different positions into a third positionDeriving general or abstract principles from disparate facts orcasesFinding alternative explanations for observationsBeing sensitive to the strength of different types of evidence(correlational, causal, testimony)Gene rating alternative hypothesesReco gnizing the probabilistic nature of most events

Note. Mean s have been divided by the num ber of items on each scale.

7/28/2019 RR-86-43-Powers.pdf

40/61

Appendix AQuestionnaire

7/28/2019 RR-86-43-Powers.pdf

41/61

I. REASONING SKILLS

The followlng are checkllsts of sune general and specific reasonlng skillsIn your graduate program. Please rate these skills wfth respect to:

( 1) frequency-how often do students 1n your graduate program need to use

that may be Important to success

this ski I I?

(2) Importance for success-to what extent does thfs skfll dffferentfate between marglnal and successful studentsin your progran?

For ftlmportance for success,t8 please refer to these descrlptlons:

1. lhls skill Js not relevant to my ffeld of teaching2. There Is little or no difference between marglnal and successful students with respect to this skill3. There Is a moderately important difference between margfnal and successful students wfth respect to this skill4. There Is a very Important difference between marglnal and successful students with respect to thls skrll5. There 1s a crItIcally Important difference betneen margfnal and successful students wlth respect to thfs sklll

A. Reasonfng Sk1 I Is--General

Derlvlng general or abstract principles frundlsparate facts or cases. . . . . . . . . . .

Reasoning or problan solvlng fn sltuatfons Jnwhich all facts underlylng a problem solutionare known expllcltly. . . . . . . . . . . . .Reasoning or problem solvfng In sltuatlons Inwhich all the needed InformatIon is not known

Reasonlng when lnconsistencfes are present inthe fnformatlon . . . . . . . . . . . . . . .

Knowfng the rules of formal logic . . . . . .

Taklng wel I known prlncfples and Ideas fran onearea and applying them to a different specialty

Being able to both support and refute a givenposftlon. . . . . . . . . . . . . . . . . . . .

CYltfcally analyzing and evaluating prevlousresearch or reports In a field. . . . . . . . .

lnccrporatlng Isolated Instances or data Into apreexlstlng fr wk.............

. .

. .

. .

. .

. .

. .

. .

. .

. .Mftorlng ones own progress Jn solving problems .

Frequency ImDortance for Success/d f f erence

Never/ Vet-YHardly Ever Frequent I y

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

rbtRelevant Wet-ate Q-ltlcal

1

1

1

1

1

1

1

1

1

1

Llttle

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

Important

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

7/28/2019 RR-86-43-Powers.pdf

42/61

7/28/2019 RR-86-43-Powers.pdf

43/61

c. Speclflc Sklsts--&struct~ng Thesesor Argunents

Supporting general assertlons with details. . . . .

bklng expllcft all relevant ccmponents In a chainof logIca reasoning. . . . . . . . . . . . . . . .

D!stfngufshlng between relevant and Irrelevantlnfmt!on . . . . . . . . . . . . . . . . . . . .

Percelvlng relatfonships among observations . . . .

Drawing dlstlnctlons betwean slmllar but notldentlcal Ideas . . . . . . . . . . . . . . . . . .

Using analogles appropriately . . . . . . . . . . .

Elaborating an argument and developfng ItsImpI IcatIons. . . . . . . . . . . . . . . . . . . .

D-awing sound Inferences frcm observations. . . . .

Producing an argument that 1s Internally conslstent.

Syntheslzlng two dlfferent posStfons Into a thlrdposItJon. . . . . . . . . . . . . . . . . . . . . .

Translating graphs or symbolic statements Intowords and vice versa. . . . . . . . . . . . . . . .

Other (specIfyI

D. Speclffc Skills-Analyzing Arguments

Recognlzlng the central argument or thesis Inawork...................... ,

Identlfylng both stated and unstated assumptions.Testing the va1ldH-y of an argunent by seat-chlngfor counterexamp 18s . . . . . . . . . . . . . . .Flndlng alternatlve explanations for observations

Recognlzlng suppcrtlng po!nts In an argument. . .

Detecting fal lacfes and logical contradlctlcnsIn arguments. . . . . . . . . . . . . . . . . . .

Freauencv Importance for Success/d I f ference

New-/ VWYI-Wdly Ever Frequently

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

r+kYtRelevant Met-ate CrItIcal

1

1

1

1

1

1

1

1

1

1

1

1

Lfttle

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

Important

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

7/28/2019 RR-86-43-Powers.pdf

44/61

Dlst1ngulshlng major and minor points In anargument. . . . . . . . . . . . . . . . . . . . . .

Eielng sensltlve to the strength of djfferent typesof evidence (correlational, causal, testlmony). . .

Recognlzlng shifts In the meaning of a word ln thecourse of an argument . . . . . . . . . . . . . . .

Judging whether a thesls has been adequate1 ysupported . . . . . . . . . . . . . . . . . . . . .

Determlnlng whether the conclusions drawn areloglcally consistent wlth, and adequatelysuppos ed by, the data. . . . . . . . . . . . . . .

Dlstlngulshlng among contt-lbutlng, necessary andsufflclent factors. . . . . . . . . . . . . . . . .

Other (specIfyI

E. Speclflc Skills-Bawlng Conclusions

Generatlng valld explanations to account forobservations. . . . . . . . . . . . . . . . . . . .Supportlng conclusions with sufflclent data orInformatIon . . . . . . . . . . . . . . . . . . . .F&cognlzlng the lmpl lcatlons or consequences of anlnterpt-etatlon. . . . . . . . . . . . . . . . . . .

@al lfylng conclusions appropriately and recognlzlngways in which conclusions could be challenged . . .

GeneratIng new questions or experiments to extendCN- support the lnterpretatlon of data . . . . . . .

Oonslderlng alternative concIuslons . . . . . . . .

Cornpat-lng concIuslons with what is already known. .

Revising a previously held view to account for newobservations. . . . . . . . . . . . . . . . . . . .

Other (specify)

Freauencv Importance for Success/d 1 f erence

#3ver/Hardly Ever

veryFrequent I y

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

NotFzleevant Wderate cf-ltlcal

Little Important

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

1 2

3

3

3

3

3

3

4

4

4

4

4

4

5

5

5

5

5

5

7/28/2019 RR-86-43-Powers.pdf

45/61

-5-

I I. REASONINGEMmsPlease indicate which of the following kinds of reasoning errors you have observed in your graduate students and thefrequency with which you have noted these errors.

Error

Failing to recognize differences between a smple and apopulation of interest. . . . . . . . . . . . . . . . . . . . . .

Failing to take into account the base rate for a phenanenonin a population . . . . . . . . . . . . . . . . . . . . . . . . .

Offering irrelevant statements about a persons character orcircumstances to oppose his/her conclusion. . . . . . . . . . . .

Fail ing to recognize when a conclusion is concealed within apremise . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Confusing coincidence and/or correlation with causation . . . . .

Using an ambiguous term to shift senses in the sane argument. . .

Arguing that a conclusion should be accepted (rejected)because it has not been disproved (proved). . . . . . ,.....

Accepting the central assunptions in an argunent withoutquestioning them. . . . . . . . . . . . . . . . . . . . . . . . .Fail ing to evaluateor text. . . . . .

the credibility or reliability of a source. . . . . . . . . . . . . . . . . . . . . . .

Offering irrelevant evidence to support a point . . . . . . . . .

Making general izations on the basis of insufficient evidence. . .

Reading into a text ones own views or inappropriatelyinterpreting text or data on the basis of ones own experience. .

Al lowing anecdotal information to override more extensivestatistical data. . . . . . . . . . . . . . . . . . . . . . . . .Basing ccnclusions on analysis of only part of a text or dataset...............................

Failing to recognize relevant similarities and dissimilaritieswhen using analogies to argue . . . . . . . . . . . . . . . . . .

Other (specify)

Other (specify)

Frequency

Never/Hard Iy Ever

V'3t-YFrequent

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

!Ser ousness

IW verySeriOUS !ht-iOUS

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

7/28/2019 RR-86-43-Powers.pdf

46/61

III. CRITICAL IICIDENTS

l-bw frequently have you observed the followfng klnds of Incidents or tendencies In your students?these had on your estlmatlon of the analytlcal ablllty of your students?

Incidents

Repeatedly maklng Irrelevant remarks during class dlscusslons . .

Offerlng crltlclsm of other studentst presentations that wasnot constructive or wel I founded. . . . . . . . . . . . . . . . .

Submlttlng a paper that falled to address the asslgned lssues . .

Accepting or supportlng an argunent based more on emotionalappeal than on evidence . . . . . . . . . . . . . . . . . . . . .ConfusIng anecdote and/or opln Ion wl th hard data. . . . . . . .

Accepting the concIuslons of recognized authorltles wlthoutcrltlcally evaluating them. . . . . . . . . . . . . . . . . . . .Being unable to Integrate and synthesize Ideas from varloussources.............................

Falling to recognize that evidence can suppcrt more than oneccKIcluslon. . . . . . . . . . . . . . . . . . . . . . . . . . . .

Uslng Inconslstent statements to support the same posltlon. . . .

Arguing In support of the obvlous . . . . . . . . . . . . . . . .

Eelng unable to perceive when an analogy breaks down. . . . . . .

Reoognlzlng a pattern but fal I Ing to draw any conclusions aboutlt or attemptIng to explain It. . . . . . . . . . . . . . . . . .

MakIng lmplauslble assumptions. . . . . . . . . . . . . . . . . .Applylng a formula, algorlthm, or other rule wlthout sufflclentjustlflcatlon . . . . . . . . . . . . . . . . . . . . . . . . . .

Relylng solely on narratlve or descrlptlon In papers and reportswhen analysis 1s appropriate. . . . . . . . . . . . . . . . . . .

Referrlng ccmplex or far-fetched explanations over simpler ones.

Being unable to see a pattern In results or to generalize whenappropriate . . . . . . . . . . . . . . . . . . . . . . . . . . .

Falllng to recognize that a deflnlte conclusion cannot bereached (or falllng to qualify a concIuslon approprlately). . . .

New-/ VWY bb Q-eatHardly Ever Frequent Ef feet Ef feet

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

what effect have

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

4

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

5

7/28/2019 RR-86-43-Powers.pdf

47/61

7/28/2019 RR-86-43-Powers.pdf

48/61

Appendix BFactor Analyses

7/28/2019 RR-86-43-Powers.pdf

49/61

Appendix B.lVarimax Factor Loadings for Ratings ofImportance of Reasoning Skills

7/28/2019 RR-86-43-Powers.pdf

50/61

Appendix B.lVarimax Factor Loadings for Ratings of Importance of Reasoning Skills

FactorI 11 III IV v Com mun ality

general o r abstract principlesfrom disparate facts or casesor solving problems in situationsin which all facts underlying a problemsolution are know n explicitlyor solving problems in situationsin which all the needed information isnot known

48

36 28

27

38 35 31when inconsistencies are present inthe information

the rules of formal logic34 32 38 41

53 38well-known principles and ideas fromone area and applying them to a differentspecialty

e to both support and refute a givenposition 51analyzing and evaluating previousresearch or reports in a field 40 45

isolated instances o r data intoa preexisting frameworkng one's own progress in solvingproblems

from the study of single casesstructural features or functional principlesthat can be applied to other casesnizing the probab ilistic nature of mo stevents

analyzing and evaluatingarguments 64

36 20

information from a set ofrelationships 33 33what kind of evidence will supportor refute a hypothe sis or thesis 53 43

41

16

31 40

30 46

23

21

32 25

49

32

52

7/28/2019 RR-86-43-Powers.pdf

51/61

Appendix B.l (Continued)

FactorVariable I 11 III IV v Com mu nalitvSnecific Skills--Problem DefinitionClearly identifying central issues andproblems to be investigated or hypotheses

to be tested 32 30Recognizing two or more sides of an issue 58Generating alternative hypotheses 32Breaking down complex problems or situationsinto simpler onesIdentifying all the variables involved in aproblemIdentifying more than one approach to solvinga problemRecog nizing structural similarities betwe enone type of problem or theory and anotherDeveloping operational (or very precise)definitions of conceptsSetting up formal models for problems underconsiderationRecognizing the historical context of aproblemSpecific Skills --Constru cting These sor Arguments

30

67

59 44

32 56 43

31 36 39

48

69

Supporting general assertions with details 57 40 52Making explicit all relevant compon entsin a chain o f logical reasoningDistinguishing between relevant andirrelevant informationPerceiving relationships among observations

40 40

36 5158

Drawing distinctions between similar but notidentical ideas 46Using analogies appropriately 35

61 47

37 52

53

39

2436 5446 39

33

53

55

42

46

54

29

7/28/2019 RR-86-43-Powers.pdf

52/61

Appendix B.l (Continued)

FactorVariable I 11 III IV v Com mun alitvElaborating an argument and developing itsimplications 72Drawing sound inferences from observationsProdu cing an argum ent that is internallyconsistent 51Synthesizing two different positions into athird position 32Translating graphs or symbolic statementsinto words and vice versaSpecific Skills--Analyzing Argum entsRecogn izing the central argum ent or thesisin a work 63 34Identifying both stated and unstatedassumptions 52Testing the validity of an argum ent by

searching for counterexamples 36Finding alternative explanations fo robservationsRecognizing supporting points in an argument 70Detecting fallacies and logical contradictionsin arguments 58 32Distinguishing major and minor points in anargument 53Being sensitive to the strength of differenttypes of evidence (correlational, causal,testimony) 41 35 42 52Recognizing shifts in the meaning of a wordin the course of an argument 73 65Judging w hether a thesis has been adequatelysupported 64 41 61

6469 56

43 49

44 34

57 36

3833

44 33

53

53

47

46

5364

54

43

7/28/2019 RR-86-43-Powers.pdf

53/61

Appendix B.l (Continued)

FactorVariable I II III IV v CommunalitDetermining whether the conclusions drawnare logically consistent with, and-adequately supported by, the dataDistinguishing among contributing, necessary,and sufficient factorsSpecific Skills--Drawing ConclusionsGenerating valid explanations to accountfor observationsSupporting conclusions with sufficient dataor informationRecognizing the implications or consequencesof an interpretationQualifying conclusions appropriately andrecognizing ways in which conclusionscould be challenged 48 56Generating new questions or experimentsextend or support the interpretationof data

to

Considering alternative conclusions57 37 5259 37 60

67 56

58 49

7.37 3.84 3.21 2.51 25.34

Com paring conclusions with what is alreadyknownRevising a previously held view to accountfor new observationsCommon ariance

36

31

47

8.41

68

43 35 34

72

69

45

64

49

59

58

54

57

Note. Loadings less than .30 have been omitted, as have all decimal p oints.

7/28/2019 RR-86-43-Powers.pdf

54/61

Appendix B. 2Varimax Factor Loadings for Ratings of FrequencySeriousness of Reasoning Errors and

7/28/2019 RR-86-43-Powers.pdf

55/61

Appendix B.2Varimax Factor Loadings for Ratings ofFrequency of Reasoning Errors

ErrorFactor

I IT. Comm unalityFailing to recognize differences between asample and a population of interestFailing to take into accou nt the base ratefor a phenom enon in a populationOffering irrelevant stateme nts abou t aperson's character or circumstancesto oppose his/her conclusionFailing to recognize when a conclusion isconcealed within a premiseConfusing coincidence and/or correlationwith causationUsing an ambiguous term to shift sensesin the same argumentArguing that a conclusion should be accepted(rejected) because it has not been

disproved (proved)Accepting the central assumptions in anargument without questioning themFailing to evaluate the credibility orreliability of a source or textOffering irrelevant evidence to supp orta pointMaking generalizations on the basis ofinsufficient evidenceReading into a text one's own views orinappro priately interpreting text ordata on the basis of ones own experienceAllowing anecd otal information to overridemo re extensive statistical dataBasing conclusions on analysis of only partof a text or data setFailing to recogn ize relevant similaritiesand dissimilarities when using analogies

to argue

48

51

30

40

40

53 38

65 47

37 34 25

63 40

63 44

67 47

63 46

70

42

58

74 56

75 58

53

30

39

43

55

46

42

50 40 42CommonVariance 4.06 2.60 6.66Note. Loadings less than .30 have been omitted, as have all decimal points.

7/28/2019 RR-86-43-Powers.pdf

56/61

Appendix B.2Varimax Factor Loadings for Ratings ofSeriousness of Reasoning Errors

ErrorFactor

I II CommunalityFailing to recognize differences between asample and a population of interestFailing to take into accoun t the base ratefor a phenomenon in a populationOffering irrelevant statemen ts about aperson's character or circumstan cesto oppose his/her conclusionFailing to recognize when a conclusion isconcealed within a premiseConfus ing coincidence and/or correlationwith causationUsing an ambiguous term to shift sensesin the same argumentArguing that a conclusion should be accepted(rejected) because it has not beendisproved (proved)Accepting the central assumptions in anargument without questioning themFailing to evaluate the credibility orreliability of a source or textOffering irrelevant evidence to suppo rta pointMaking generalizations on the basis ofinsufficient evidenceReading into a text one's own views orinapprop riately interpreting text ordata on the basis of one's own experienceAllowing ane cdotal inform ation to overridemo re extensive statistical dataBasing conclusions on analysis of only partof a text or data setFailing to recognize relevant similaritiesand dissimilarities when using analogies

to argue

30 63 48

40 56 47

38 61 51

50 48 48

35 60 48

70 53

71 56

75 61

72 56

65

43

55

68 48

76 59

33

56

53

50

39

46 44 40

7/28/2019 RR-86-43-Powers.pdf

57/61

Appendix B.3Varimax Factor Loadings of Freque ncy andEffects of Critical Incidents

7/28/2019 RR-86-43-Powers.pdf

58/61

Appendix B. 3Varimax Factor Loadings of Effects of Critical Incidents

FactorIncident I I.1 III CommunalityRepeatedly making irrelevant remarks during classdiscussions 60 36Offering criticism of other studen ts' presentation sthat was not constructive or well founded 65 44Subm itting a paper that failed to address theassigned issues 62 42Accepting or supporting an argument based more on

emotional appeal than on evidence 72Confusing anecdote and/or o pinion w ith "hard data" 65Accepting the conclusions of recognized authoritieswithou t critically evaluating themBeing unable to integrate and synthesize ideas fromvarious sourcesFailing to recognize that evidence can support morethan one conclusionUsing inconsistent statements to support the same

positionArguing in support of the obvious

56 40 5138 39 37

Being unable to perceive when an analogy breaksdown 49 38 47Recognizing a pattern but failing to draw anyconclusions about it or attempting toexpla