Argumentative writing and academic achievement: A longitudinal study

Learning and Individual Differences 28 (2013) 204–211

Contents lists available at ScienceDirect

Learning and Individual Differences

j ourna l homepage: www.e lsev ie r .com/ locate / l ind i f

Argumentative writing and academic achievement: A longitudinal study

David D. Preiss a,⁎, Juan Carlos Castillo a, Elena L. Grigorenko b, Jorge Manzi a

a Escuela de Psicología, MIDE UC Measurement Center, Pontificia Universidad Católica de Chile, Chileb Child Study Center, Department of Epidemiology and Public Health, Department of Psychology, Yale University, USA

⁎ Corresponding author at: Escuela de Psicología, PonChile, Av. Vicuña Mackenna - 4860. Macul, Santiago3544605; fax: +56 2 3544844.

E-mail addresses: [email protected], daviddpreiss@g1 The GRE (Graduate Record Examinations) is a stan

ministered by the Educational Testing Service (ETS). It hsoning, quantitative reasoning and analytical writingadmission requirement for graduate or business schoosome other English speaking countries. The SAT is a stanreading, mathematical reasoning, and writing skills admand used for college placement in the USA. The ACT isbased standardized test administered by ACT Inc., whthe SAT. It has subtests in English, Mathematics, Readwell as an optional writing test.

1041-6080/$ – see front matter © 2013 Elsevier Inc. Allhttp://dx.doi.org/10.1016/j.lindif.2012.12.013

a b s t r a c t

Article history:Received 15 March 2012Received in revised form 31 October 2012Accepted 22 December 2012

Keywords:WritingWriting assessmentUniversity admissionsHigh-stakes testingAcademic achievement

Capitalizing on the implementation of a writing assessment initiative implemented at a major Chilean uni-versity, we test how predictive writing is of subsequent academic achievement. First, using a multilevel an-alytic approach (n=2597), the study shows that, after controlling for socio-demographic variables and theuniversity admission tests, writing skills significantly predict first-year university grades. Second, using infor-mation about the performance of students during their first eight semesters in the university (n=1616), alongitudinal hierarchical analysis showed that writing remains a significant predictor of university gradesover time, also after controlling socio-demographic variables and university admissions tests. Moreover, lan-guage skills retain or improve their predictive role over time, whereas mathematics skills seem to decrease intheir importance. Our results show that writing, and the cognitive skills involved in writing, play a criticalrole in advanced stages of academic training, consequently offering additional support for the considerationof this ability for university admission purposes.

© 2013 Elsevier Inc. All rights reserved.

1. Introduction

Currently there is a growing interest in the inclusion of alternativeassessments to the conventional ones used in university admissionprocesses (Atkinson & Geiser, 2009; Kyllonen, 2012; Manzi, Flotts, &Preiss, 2012; Stemler, 2012; Sternberg, 2004, 2010). In this context,writing assessment has taken a prominent place as an additionalhigh stakes measure. Part of this interest originates in a concern forthe quality of writing high-school graduates and university levelstudents are capable of producing (Kellogg & Raulerson, 2007;Kellogg & Whiteford, 2009; Lee & Stankov, 2012; Manzi et al., 2012;The National Commission on Writing in America's schools andcolleges, 2003). Thus, several North American university admissiontests such as the GRE, SAT and ACT1 incorporated writing measures(Jeffery, 2009; Norris, Oppler, Kuang, Day, & Adams, 2004). The ana-lytical part of the GRE was transformed in a writing assessment that

tificia Universidad Católica de7820436, Chile. Tel.: +56 2

mail.com (D.D. Preiss).dardized test created and ad-as three subtests in verbal rea-. The GRE is required as anl in the United States and indardized assessment of criticalinistered by the College Boarda curriculum- and standards-ich works as an alternative toing, and Science Reasoning as

rights reserved.

evaluates both critical thinking and analytical writing skills; the SAThas a section assessing writing by means of a short essay; the ACThas an optional section assessing students' level of understanding ofthe conventions of standard written English as well as their abilityto produce text.

In Chile, where this study was carried out, high stakes writing as-sessment has never been implemented. Yet a growing awareness ofentry-level university students' deficits in writing has promptedhigher education institutions to develop writing assessment initia-tives, although they do not play a role in admissions yet. Specifically,in 2003, the Pontificia Universidad Católica de Chile developed a testof argumentative writing: The Writing Communication Test (WCT)(Manzi et al., 2012). The WCT differs from the abovementionedAmerican examinations in two critical dimensions. On the one hand,it is a test that is taken once the student has been accepted to his orher undergraduate programs —.i.e., the test does not play a role inadmission decisions. Yet, it is still a high-stakes test since taking andpassing the test is a graduation requirement. Indeed, the scoreobtained in the test is used as a criterion to determine whether thestudent must or must not attend additional classes to improve hisor her writing communication skills (hereafter, WCS). On the otherhand, the test uses an analytic rubric that assigns independent scoresfor several dimensions of the essay. In the study reported in thispaper, we evaluate how predictive WCT scores are of subsequentacademic achievement during students' undergraduate education.

Large-scale assessment of WCS has many educational advantages.First, it has an impact on teaching and learning, as educational sys-tems are more likely to teach what is finally assessed (Grigorenko,Jarvin, Nui, & Preiss, 2008; Sternberg, 2010). Second, it informs public

205D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

policy and decision-making processes targeting the development ofwriting within the educational system. Moreover, writing assessmenthas a broad appeal because it has some attributes that make it distin-guishable from other assessment tools (O'Neill, Moore, & Huot, 2009;Powers, Fowles, & Willard, 1994). In fact, writing tests are commonlylabeled as direct writing assessment (hereafter, DWA) since the skillsthat are the target of measurement are assessed directly. In contrastto multiple-choice tests, which measure latent constructs, DWAdoes not measure a latent ability. Provided that measurement stan-dards are established precisely, writing abilities can be assessed in astraightforward way.

DWA involves relevant assessment challenges, which includethose related to the generation of writing prompts, the definition ofthe construct, and the rating process (Manzi et al., 2012). Becausetopic knowledge affects text quality (McCutchen, Teske, & Bankston,2008), a bad choice of a thematic prompt may bias the measurementprocess by giving a relative advantage to a group more versed on thetopic, independently of its writing abilities. In addition to differenttopics, when generating the writing prompt, test designers may optbetween different genres: the narrative, descriptive, argumentative,and expository genres are those most commonly mentioned. Theoption for a specific genre has an impact on writing performanceand the scoring process (Beck & Jeffery, 2007; Kellogg & Whiteford,2012; Lee & Stankov, 2012). Writing argumentative and expositorytext is more cognitively demanding than writing narrative anddescriptive text (Weigle, 2002). Because of its relevance in academicdiscourse, the argumentative genre has been one of the most favoredgenres in writing assessment. Additionally, benchmarks to assess thewritten products must be aligned to the genre demanded by the testprompts. This is not always the case. Beck and Jeffery (2007) assessedthe genre demands related to writing examinations existing in thethree most populated American states and found that the bench-marks used were not aligned to the type of genre stimulated by theprompts and, consequently, the examinations faced several validityissues. Last but not least, there are challenges related to the definitionof the construct, which materialize at the moment of setting up therubrics used to assess the writing samples.

The composition literature distinguishes three types of ratingscales: primary trait scales, holistic scales and analytic scales (Lee &Stankov, 2012; Weigle, 2002). The option for any of these proceduresinvolves an implicit definition of what quality writing is. Primary traitassessment involves the identification of one or more primary traitsrelevant for a specific writing task and related to its purpose, assign-ment and audience. Holistic assessment is based on the overallimpression the rater has about the written product. This impressionis based on a scoring rubric that is complemented by benchmarks.This scoring strategy guides the writing assessment made for theNational Assessment of Educational Progress (Lee & Stankov, 2012).Although holistic scoring is a practical option, it does not allow diag-nosing strengths and weaknesses in writing (Weigle, 2002). More de-tailed information about writing is provided by analytical assessment.It involves assessing different features relevant for good writing.Some analytical scales weight these attributes, so certain attributesare considered more relevant (e.g., the global organization of thetext) and have a larger weight in the final score than others (e.g., or-thography). Besides its utility in providing a more detailed profile ofstudents' writing, analytical scoring is more instrumental in ratertraining as it provides inexperienced raters with more specific guide-lines about the assessed construct (Weigle, 2002).

The use of writing measures has allowed researchers to startassessing how predictive writing is of subsequent academic success.Recent studies show that the ability to produce good argumentativetext is the best predictor of academic success during the first year ofthe university (Geiser & Studley, 2002; Kobrin, Patterson, Shaw,Mattern, & Barbuti, 2008). Specifically, a study which explored discrep-ant SAT Critical Reading and Writing Scores found, after controlling for

student characteristics and prior academic performance, that studentswhohad relatively higherwriting scores, as comparedwith their criticalreading scores, obtained higher grades in their first year of college andin their first-year English course (Shaw, Mattern, & Patterson, 2011).In Chile, the only evidence available concerning the predictive value ofwriting assessments is related to the test described here. Specifically,for the 2008 round of assessment,WCTperformancewas positively cor-related to academic achievement in most undergraduate programs atthe Pontificia Universidad Catolica de Chile with an average correlationof .12 (Manzi et al., 2012). Studies assessing how predictive writing isbeyond the first year at the university are scarce. Here, capitalizing onthe implementation of the Chilean assessment, we intend to addressthat issue by assessing how predictive WCT scores are of academicachievement during the students' subsequent eight semesters of uni-versity study.

2. Methods

2.1. Sample

Data from thePontificiaUniversidadCatólica deChile's 2007 freshmencohort of students were analyzed in this study. Students entering theuniversity graduated from three different types of high schools: some ofthem graduated from schools entirely funded by the State (publicschools), some others graduated from schools that are privatelymanagedbut receive public funding and, in many cases, charge the parents anadditional fee (voucher schools), and some others from entirely privateschools. In Chile, the school of origin is a proxy of the socioeconomic back-ground of the families: students frommore affluent families go to privateinstitutions whereas those from disadvantaged backgrounds go to publicinstitutions (Drago&Paredes, 2011; Elacqua, Schneider, & Buckley, 2006).Although voucher schools as a whole recruit a relatively diverse pool ofstudents, there are significant socioeconomic gaps between voucherschools as they select their students depending upon their educationaland financial goals (Mizala & Torche, 2012). A large part of the studentsparticipating in the study were undergoing professional training as inthe Chilean university system the majority of the students enroll inprofessional programs directly, that is, professional training is part ofthe core of their undergraduate curriculum. Table 3 summarizes the back-ground information for the sample.

The students were automatically registered to take the WCT whenenrolling for their courses and were recruited by their respectiveacademic units. Students were expected to take the test during theirfirst year, although they were given the opportunity of taking thetest two more times during their studies. From the 3760 studentsenrolled in 2007, 2879 (76.56%) took the test during their first year.Missing data were handled in the analysis by means of listwise dele-tion. For the cross sectional multilevel estimation, which was madeon all the students having data available at the end of their firstyear of studies, the final sample was 2597. For the longitudinal esti-mation, which was made on all the students enrolled in programsthat had at least 15 students who had completed their fourth yearof studies by the second term of 2010, the number of participantswas 1616.

2.2. Measures and procedures

2.2.1. The Written Communication TestTheWCT presented the students with three topics and asked them

to produce a 2-page essay on a theme of their preference, amongthree possible alternatives. These themes were related to issues ofgeneral interest, excluding themes related to specific disciplines toavoid possible biases. The topics were presented as an opinion thatcould be challenged or defended by the student. An example of theproposed themes is the following: Some people think that freedom ofspeech, or of the press, is an essential value that is severally damaged

Table 1Distribution of the discrepancy levels in the Writing Communication Test final scoresbetween raters.

Discrepancy Percentage of cases Accumulated percentage

0 to 0.5 points 65.47% 65.47%0.6 to 1.0 points 28.98% 94.46%1.1 to 1.5 points 4.74% 99.19%More than 1.5 points 0.82% 100.00%

Data are based in double rated essays only (N=866).

206 D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

when the rights to privacy of those affected by the dissemination of infor-mation to the public are accorded a higher importance than freedom ofspeech.

The specific guidelines students received to write the essay indicat-ed the different dimensions of writing that were going to be taken intoaccount in the evaluation process. These aspects were the developmentof an argument line that includes a thesis, arguments that sustain it anda conclusion. Additionally, students were required to include at leastone alternative point of view or counterargument. In addition to callstudents' attention to formal aspects such as orthography, use of ac-cents and punctuation, as well as vocabulary use, the guidelines explic-itly asked the students to organize the exposition in such a way that areader could follow a clear and coherent argument line.

An analytic rubric was used for grading the essays. The rubricdistinguished five performance levels in the different dimensionsthat were object of assessment. Some of the assessed dimensionswere related to formal aspects of written discourse whereas otherswere related to content, quality and coherence of argumentative rea-soning. Specifically, the rubric considered the following dimensions(the first five are related to the formal aspects of written discourse):

• Orthography: good use of Spanish basic orthographic rules (literalspelling, punctuation and accents).

• Vocabulary: amplitude, precision and appropriateness to academic use.• Text structure: presence of introduction, development and conclusion.• Textual cohesion: adequate use of phrase grammar and connectors.• Use of paragraphs: sentence use and inclusion of main ideas in eachone of the paragraphs.

• Argument quality: global coherence of arguments, that is, variety andquality of arguments as they relate to a thesis.

• Thesis: presence of an explicit thesis about the topic.• Counterarguing: Coherence of arguments based on presentation of oneor more counterarguments.

• Global assessment: general assessment of content and text quality bythe rater, who gives his or her overall appreciation of the text, aftergrading all the other dimensions.

In each scoring dimension, five performance levels were distin-guished: insufficient performance, limited performance, acceptableperformance, good performance, and outstanding performance. Toillustrate these performance levels, we present the rubric used forthe dimension on use of paragraphs:

• Insufficient performance. The text does not present clearly identifiableparagraphs. Or: Most of the paragraphs are not well implemented asthey reiterate the same idea of the previous paragraph or advancemore than one core idea.

• Limited performance. The text presents clearly identifiable paragraphs;however,more than oneparagraph can either reiterate the same idea ofthe previous paragraph or advance more than one core idea.

• Acceptable performance. The text presents clearly identifiableparagraphs; however, one paragraph can either reiterate the sameidea of the previous paragraph or advance more than one core idea.

• Good performance: The text presents clearly identifiable paragraphs;each paragraph identifies a core idea that is different from the coreidea advanced in the previous paragraph.

• Outstanding performance: The text presents clearly identifiable par-agraphs; each paragraph identifies a core idea that is different fromthe core idea advanced in the previous paragraph. Additionally,there is evidence of progression from one paragraph to the next.

A professional team including 26 specialists in Spanish languagetrained in the use of the rubric rated the essays. These raters were or-ganized in three groups and each one of them was in charge of a su-pervisor whose main task was to maintain the raters calibrated. Useof benchmarks helped to calibrate raters by illustrating the way rubricshad to be applied. At the beginning of the rating process, a number ofessays were scored by all raters to check the level of consistency and

consensus they reach. After this stage, one rater scored all essays. Inorder to assess the level of agreement between raters, 20% of the essayswere assigned to a second rater. In all those cases involving doublerating, the final score was the average between raters. As shown inTable 1, 95% of discrepancies were below 1 point, which wasestablished as the threshold for agreement/disagreement. In a smallproportion of essays (32 cases), the discrepancy between ratersexceeded that threshold. In those cases the supervisor of the ratingprocess proceeded to rate the essay and his or her score was consid-ered as the final score. The final score for each essay was computedas the mean for each one of the nine dimensions assessed in the rubric.

2.2.2. PSUThe PSU is the Chilean university entrance examination, mandatory

for all applicants interested in admission to universities receiving publicfunding. The examination includes a Language and Communication aswell as a Mathematics test. The PSU in Language and Communicationmeasures three competencies: the competency to extract information,the competency to interpret explicit and implicit information and thecompetency to assess explicit and implicit information. The topicsconsidered include knowledge of Spanish language, knowledge of liter-ature, and knowledge of mass media and communication. The PSU inMathematics measures three sets of cognitive skills: recognition offacts, terms and procedures; understanding of information in a mathe-matical context, application of mathematical knowledge to known andunknown situations, and analyzing, synthesizing and evaluating math-ematical relations in problem solving. The topics considered includehigh-school level number and proportionality, algebra and functions,geometry and probability and statistics (DEMRE, 2012).

2.3. Data analysis

For reference, tables with descriptive statistics and intercorrela-tions of the eight WCT dimensions and the raters' global assessmentare included in the Appendices. The raters' global assessment hadsignificant positive correlations (pb0.01) with all the WCT dimensions.The highest correlation coefficients observed for the raters' global assess-ment are those it has with the two dimensions related with argumenta-tive writing (Argument quality, r=.59; and Counterarguing, r=.49).The smallest correlations observed for the raters' global assessment arethose it has with the two dimensions related with the formal aspects ofwriting (Orthography, r=.20; andVocabulary, r=.16). The intercorrela-tions between theWCT dimensions are all significant but for Vocabulary,which has non-significant correlations with Text structure, Argumentquality, Thesis and Counterarguing. For the WCT dimensions, thesmallest significant correlation is that between Orthography and Thesis(r=0.05) and the highest is that between Argument quality and Thesis(r=0.45). Most of the other correlations are in the .1–.3 range

The WCT average scores were merged with a database providingbackground information of the students, their scores in the nationaluniversity admission test (PSU), their high school grades as well astheir GPAs for each academic semester until 2010. Thus, each studenthad information for eight GPAs (two per year from 2007 to 2010).Chilean GPAs range from 1.0 to 7.0, 7.0 being the highest grade. De-scriptive statistics for these variables are summarized in Table 2.

Table 3Background variables.

Values N

Parents' highesteducational level

0. Less than university education 6511. University education 2102

Sex 0. Male 14041. Female 1475

School administration 0. Public 3901. Prívate 18202. Voucher private 656

PSU average

Academic area 0. Social sciences 429 687.171. Engineering 732 734.192. Education 202 623.213. Sciences 628 681.264. Art 193 692.885. Health 228 725.706. Law and humanities 467 691.83

Note: The numbers of subjects in each variable were obtained with pairwise deletionbetween the independent variables and WCT score.

Table 2Written Communication Test (WCT) and students' academic achievement indicators.

Variablename

Description Min Max Mean SD

WCT Written Communication Testscore

1.00 4.89 3.29 0.48

PSU Language University admission languagetest score

483.00 850.00 692.00 62.94

PSU Math University admissionmathematics test score

428.00 850.00 704.00 72.71

PSU Average score in PSU languageand PSU mathematics

520.00 838.00 697.00 53.19

NEM High school grades convertedinto admission scores

435.00 826.00 694.00 53.19

207D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

The association of the independent variables with the GPAs wasestimated in a multilevel modeling framework. The academic pro-grams were defined as level two units, since it was plausible thatthey might have an influence in the variations of the GPAs and, there-fore, violate the ordinary regression models' independence assump-tion. The variables corresponding to the individual backgroundinformation (level one) were sex and parents' educational level, aswell as the administrative dependency of the high school (School Ad-ministration), which as mentioned above in Chile is a proxy ofsocio-economic status. As level two variables we included the aca-demic programs' average PSU score, which was a proxy for selectivityof the programs, and the domain of academic program, which weclassified in seven different groups, as described in Table 3.

3. Results

We first present a cross-sectional analysis focused on the first yearGPAs. After, we analyze the whole range of GPAs for the 2007 cohortfrom 2007 to 2010 in a longitudinal modeling framework. Table 4shows the correlations between the individual level variables to beconsidered in the explanatory models. WCT scores have a significantpositive but moderate association with GPAs, although they are aswell moderately associated with other predictors such as PSU(Language) scores, High School Grades (NEM), Parents' Education andSex, which could reduce its influence when considered simultaneously.

3.1. Multilevel estimation

Next, using a multilevel modeling framework, we assess whetherthe WCT score is related to GPA after controlling for other academicpredictors and socio-demographic variables. Furthermore, we accountfor possible differences in GPA according to academic programs, sinceit is plausible that the distribution of grades is not uniform acrossthem. Thus, we present the results of a series of multilevel modelsfor the 2007 GPAs: student data are the first level and the academicprogram variables are the second level. The intra-class correlationcoefficient obtained for the model without predictors indicates thepercentage of variance of the GPAs associated with the academic pro-grams. For the 2007 GPAs, this coefficient reaches 43%, supporting theassumption that there are considerable differences among academicprograms in terms of their grade distribution.

Table 5 shows the result of the estimation of the fixed and randomeffect component for several models for the 2007 GPAs. The modelsdiffer in the number and type of predictors included. Model I usesonly WCT as a predictor, revealing a significant relationship betweenWCT and GPAs. The following models test the effect of WCT on GPAsafter controlling for other academic and socio-demographic variables.In Model II we control for PSU Language, High School Grades, Parents'Education, School Administration and Sex. PSU Language and HighSchool Grades have significant effects, and we observe that withthese variables in the model, the parameter associated with the

WCT is lower than in Model I but still significant. Interestingly, SchoolAdministration, a variable with clear socioeconomic implications inthe context of the Chilean school system, does not show significanteffects in Model II. In Model III we were also interested in the interac-tions of WCT and PSU Language admission scores with Sex, but theseinteractions were not significant. Model IV added level two predic-tors, namely the domain of the academic program and the average ac-ademic programs' admission test score as a proxy for selectivity. Theresults revealed that, in addition to the previous effects, academicprograms show significant differences in their grades: education pro-grams have the best GPAs and engineering has the worst (when com-pared with Social Sciences as a reference category). Consideredtogether, level two predictors account for almost half of the betweenacademic programs' variance (R2=49%). WCT, PSU Language scoreand High School Grades were significant in all four models. SchoolAdministration is significant in Models III and IV (with students com-ing from voucher schools showing lower GPAs than their publicschool classmates).

Models V, VI and VII follow the same structure of models II, III andIV, but now PSU Mathematics replaces PSU Language as an admissionscore predictor. Results are very similar for the main variables. WCT,PSU Mathematics and High School Grades are significant predictorsin all cases. School Administration is now significant in all cases(with students coming from voucher schools showing lower GPAsthan their public school classmates). The other interesting differenceis that in model VI, where interactions are tested, a significant Sex byPSUMathematics interaction was found, which shows that PSUMath-ematics underpredicts females' achievement in GPAs. In sum, the re-sults of cross sectional multilevel modeling reveals that WCT is asignificant predictor of first year grades regardless of the variablescontrolled in the analyses. Next, we will test whether the WCT re-mains significant when we look into GPAs beyond first year.

3.2. Longitudinal analysis

In this part of the analyses we focused our attention on the influ-ence of WCT across time, testing whether WCT remains a significantpredictor as students advance in their undergraduate programs. Inthis analysis, we used the GPAs at each semester as the dependentvariables (8 points were considered, using data from the first semes-ter of 2007 through the second semester of 2010). For this longitudi-nal analysis we also applied a multilevel perspective (Singer &Willett, 2003), whereby the GPAs constitute the first level units thatare nested in individuals (level two) that at the same time are nestedin academic programs (level three). The cases selected for the

Table 4Correlation matrix of grade point average with admission scores and sociodemographic variables.

1 2 3 4 5 6 7 8

1. GPA –

2. WCT 0.13⁎⁎ –

3. PSU (average) 0.09⁎⁎ 0.17⁎⁎ –

4. PSU-Language 0.18⁎⁎ 0.28⁎⁎ 0.74⁎⁎ –

5. PSU-Math −0.03 0.01 0.79⁎⁎ 0.22⁎⁎ –

6. NEM 0.20⁎⁎ 0.14⁎⁎ 0.43⁎⁎ 0.35⁎⁎ 0.37⁎⁎ –

7. Parents' ed. 0.11⁎⁎,a

0.12⁎⁎,a

0.44⁎⁎,a

0.35⁎⁎,a

0.35⁎⁎a

0.20⁎⁎,a

–

8. Sex (female) 0.22⁎⁎,a

0.12⁎⁎,a −0.29⁎⁎

,a −0.01a −0.41⁎⁎,a

0.08⁎⁎,a

0.03b –

N=2877.a Point-biserial correlations.b Tetrachoric correlations.⁎ pb0.05.

⁎⁎ pb0.01.

208 D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

analysis were academic programs that had at least 15 students whohad completed the second term of 2010 (1616 students belongingto 26 academic programs).

Table 5Multilevel regression models of grade point average (GPA) 2007 on individual and academ

I II III

Fixed effectsLevel 1 — indiv.WCT 0.16⁎⁎ 0.08⁎⁎ 0.0

(6.42) (3.24) (2.3PSU Language 0.00⁎⁎ 0.0

(8.31) (6.6PSU Math.

NEM 0.00⁎⁎ 0.0(15.17) (15.1

Parents' highest educ. level 0.05 0.0(1.60) (1.6

School administration(ref: public)

Private 0.04 0.0(1.12) (1.7

Voucher private −0.07 −0.1(−1.95) (2.8

Sex (female) −0.03 −0.0(−1.08) (0.1

Sex×PSU Lang. −0.0(0.0

Sex×PSU Math.

Sex×WCT −0.0(0.0

(Intercept) 4.45⁎⁎ 1.28⁎⁎ 1.2(36.68) (5.90) (5.2

Level 2 — ac. prog.PSU average

Academic area(ref: social sciences)

Engineering

Education

Sciences

Art

Health

Law and humanities

Random effectsVariance between 0.25 0.25 0.2Variance within 0.34 0.30 0.3Log likelihood −2338 −2168 −2169

Maximum likelihood estimation, unstandardized coefficients, t values in parentheses. Intra⁎ pb0.05.

⁎⁎ pb0.01.

Table 6 presents the results of the estimation. In the analyses wefollowed a similar plan as the one used with the cross-sectionalcase. Starting with WCT as the only predictor in Model I, we added

ic programs' variables.

IV V VI VII

8⁎ 0.08⁎⁎ 0.12⁎⁎ 0.14⁎⁎ 0.12⁎⁎

3) (3.23) (5.27) (4.52) (5.27)0⁎⁎ 0.00⁎⁎

4) (8.39)0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎

(13.62) (13.77) (13.82)0⁎⁎ 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎

4) (15.28) (15.59) (15.68) (15.75)5 0.05 0.04 0.04 0.050) (1.64) (1.48) (1.49) (1.51)

7 0.07 0.03 0.02 0.035) (1.78) (0.70) (0.68) (0.70)1⁎⁎ −0.11⁎⁎ −0.08⁎ −0.08⁎ −0.08⁎

5) (2.90) (2.10) (2.19) (2.14)3 −0.06⁎ −0.01 1.07⁎⁎ −0.010) (2.28) (0.21) (3.90) (0.32)09)

−0.00⁎⁎

(3.97)0 −0.043) (0.92)7⁎⁎ 2.06 −0.08 −0.62⁎ 2.415) (1.43) (0.31) (2.14) (1.55)

−0.00 −0.00(0.57) (1.58)

−0.51 −0.75⁎⁎

(1.94) (2.65)1.00⁎⁎ 0.88⁎⁎

(3.19) (2.63)−0.16 −0.39(0.83) (1.86)0.04 0.03(0.18) (0.12)0.39 0.24(1.49) (0.84)0.02 0.09(0.07) (0.39)

5 0.12 0.34 0.33 0.140 0.30 0.28 0.28 0.28

−2158 −2114 −2105 −2100

class correlation of null model: 0.43. N level 1=2597, N level 2=31.

Table 6Longitudinal multilevel regression models.

I II III IV V VI VII

Fixed effectsLevel 1 — timeSemester −0.07⁎⁎ 0.06⁎⁎

(2.94) (2.72)(Intercept) 4.78⁎⁎ 2.15⁎⁎ 1.90⁎⁎ 2.19⁎⁎ 2.10⁎⁎ 2.11⁎⁎ 1.93⁎⁎

(49.71) (11.92) (3.38) (3.96) (9.92) (3.63) (3.39)Level 2 — indiv.WCT 0.16⁎⁎ 0.09⁎⁎ 0.09⁎⁎ 0.07⁎⁎ 0.12⁎⁎ 0.12⁎⁎ 0.08⁎⁎

(7.31) (4.49) (4.46) (2.78) (6.01) (5.99) (3.18)PSU Language 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎

(6.90) (6.94) (3.60)PSU Math. 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎

(5.26) (5.63) (7.09)NEM 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎ 0.00⁎⁎

(15.28) (15.58) (16.09) (15.43) (15.76) (16.34)School admin. (ref: pub.)

Private 0.07⁎ 0.07⁎ 0.06 0.06 0.06 0.05(2.15) (2.11) (1.90) (1.94) (1.87) (1.57)

Voucher private −0.04 −0.05 −0.06 −0.04 −0.04 −0.05(1.29) (1.39) (1.70) (1.07) (1.14) (1.43)

Parents highest educ. 0.06⁎ 0.06⁎ 0.06⁎ 0.06⁎ 0.06⁎ 0.06⁎

(2.12) (2.19) (2.19) (2.15) (2.17) (2.12)Sex −0.01 −0.02 −0.03 0.01 0.01 −0.01

(0.45) (0.70) (1.42) (0.51) (0.33) (0.37)Lev. 3 — acad. pr.PSU average 0.00 0.00 −0.00 −0.00

(0.45) (0.21) (0.03) (0.51)Academic area (ref: social sciences)

Engineering −0.53⁎⁎ −0.50⁎⁎ −0.64⁎⁎ −0.62⁎⁎

(5.20) (5.04) (6.08) (6.05)Education 0.61⁎⁎ 0.68⁎⁎ 0.55⁎⁎ 0.62⁎⁎

(5.07) (5.78) (4.40) (5.13)Sciences −0.25⁎⁎ −0.21⁎ −0.34⁎⁎ −0.32⁎⁎

(2.74) (2.44) (3.71) (3.52)Art −0.01 0.00 −0.02 −0.01

(0.07) (0.04) (0.17) (0.06)Health −0.06 0.00 −0.12 −0.06

(0.53) (0.01) (1.13) (0.61)Law and humanities 0.08 0.09 0.12 0.14

(0.89) (1.04) (1.26) (1.48)InteractionsWCT×semester 0.01 0.01⁎⁎

(1.15) (2.64)PSU Lang.×semester 0.00⁎⁎

(4.40)PSU Math.×semester −0.00⁎⁎

(2.76)Random effects

Var between lev 3 0.09 0.08 0.01 0.01 0.10 0.02 0.01Var between lev 2 0.15 0.12 0.12 0.00 0.12 0.12 0.00Var within 0.16 0.16 0.16 0.16 0.16 0.16 0.15Sem. random slope 0.00 0.00Cov. slope intercept −0.01 −0.01Log likelihood −8214 −8056 −8036 −7202 −8066 −8044 −7212

Maximum likelihood estimation, unstandardized coefficients, t values in parenthesesIntraclass correlation of nullmodel: 0.38 (level 2), 0.23 (level 3). N level 1=12,808; N level 2=1616;N level 3=26.⁎ pb0.05.

⁎⁎ pb0.01.

209D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

the same individual and academic program level variables in the fol-lowing models. In model I, we observed that WCT was a significantpredictor of GPAs, as it was after one year of university studies. InModel II we controlled for PSU Language, High School Grades,Parents' Education, School Administration and Sex. PSU Languageand High School Grades presented a highly significant effect. At thesame time, WCT remained significant. Unlike the cross-sectional anal-ysis focused on the first year GPAs, the longitudinal analysis showthat having a parent with university education does make a differencein student performance. This effect remains significant throughoutthe models. The inclusion of the academic programs in Model III indi-cates that university grades are significantly different among academ-ic areas, with engineering having the lowest grades, and education

the highest ones. In Model IV we were interested in modeling theeffect of time by estimating whether the predictive capacity of WCT,PSU Language and High School Grades changes as students advancein their academic programs. Thus, a time variable was introduced:the semester, coded from 1 (first semester 2007) to 8 (second semes-ter 2010). Further, the slope of the time variable was allowed to vary(random slopes) and an interaction term with time was included foreach one of the main predictors: WCT, PSU Language and High SchoolGrades. The results of model IV indicate that WCT did not interactwith the semesters, indicating that the significant role of WCT doesnot change over time. However, PSU Language shows a significantpositive interaction with semester, indicating that this test increasesits predictive capacity over time. Models V, VI and VII replicate the

Variable Mean Std. dev. Min Max

1. Orthography 2.39 1.17 1 52. Vocabulary 2.77 0.90 1 53. Text structure 4.05 0.73 1 54. Textual Cohesion 3.66 0.87 1 55. Use of paragraphs 3.36 0.79 1 56. Argument quality 3.47 0.82 1 57. Thesis 4.02 0.86 1 58. Counterarguing 3.03 1.15 1 59. Global assessment 2.89 0.74 1 5

N=2,877.

1 2 3 4 5 6 7 8

1. Orthography –

2. Vocabulary 0.20** –

3. Text structure 0.07** 0.03 –

4. Textual cohesion 0.25** 0.22** 0.15** –

5. Use of paragraphs 0.14** 0.14** 0.26** 0.20** –

6. Argument quality 0.10** 0.01 0.36** 0.16** 0.16** –

7. Thesis 0.05** 0.00 0.42** 0.10** 0.13** 0.45** –

8. Counterarguing 0.10** 0.03 0.26** 0.09** 0.12** 0.35** 0.33** –

9. Global assessment 0.20** 0.16** 0.42** 0.25** 0.27** 0.59** 0.42** 0.49**

N=2,877, * pb0.05, ** pb0.01.

210 D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

structure of models II, III and IV using now PSU Mathematics insteadof PSU Language. The effects are similar in magnitude and directionwhen compared with the models with PSU Language. The maindifference appears in model VII when testing interactions, becausewe now found that WCT significantly interacts with time, showingthat the role of WCT on grades improves over time. In contrast, theinteraction of PSU Mathematics with time indicates that the predic-tive validity of this test decreases as students advance in theirprograms.

In summary, the longitudinal analyses indicate that WCT remainsa significant predictor of university grades over time, even after con-trolling for a number of individual and academic level variables.Moreover, the pattern of interactions with time is relevant. Compar-ing models IV and VII, we observed that WCT did not change its pre-dictive role when coupled with PSU Language, but it did improve thatrole when combined with PSU Mathematics. This seems to indicatethat language skills (as measured by PSU Language and WCT) retainor improve their predictive role over time, whereas mathematicsskills seem to decrease in their importance over time. This observa-tion probably reflects the increasing importance of language skills(reading and writing) in the advanced semesters of the academicprograms.

4. Discussion and conclusions

This paper presents empirical evidence regarding the importanceof writing skills for academic success in university level studies.First, this study shows that writing skills significantly predict firstyear university grades. Moreover, writing remains a significant pre-dictor even after controlling for socioeconomic background variables,which in Chile are strongly correlated with educational outcome vari-ables (Garcia-Huidobro, 2000; Manzi & Preiss, in press). When theanalysis added the factors currently used for university admissionpurposes in Chile (standardized scores in a mandatory university en-trance test and high school grades; i.e. PSU scores), writing was still asignificant predictor of first year university grades. This evidence isconsistent with the predictive power of writing in the US context(Kobrin et al., 2008). Second, this study generated new informationwith regard to the importance of writing over the course of universitytraining. Writing remains a significant predictor of university gradeslongitudinally, after controlling for the scores in the university entryexaminations, high school grades, and background variables such asthe different undergraduate programs, the type of school of origin,and the students' parents' education. Specifically, the longitudinalhierarchical analysis showed that writing remains a significantpredictor beyond the first year, and even more when its impact iscomplemented with the scores in the university entry examinationin mathematics instead of the scores in the university entry examina-tion in language. That is, as students progress in their undergraduateprograms, their language abilities gain in predictive capacity in con-trast with their mathematical abilities. The differentials in predictionbetween these two sets of academic skills require further investiga-tion. Of particular relevance is whether these differences are specificto Chile or characteristic of the relevance that writing acquires atmost advanced levels of university training.

The study has a number of limitations. One of them relates to sam-ple attrition. As noted in the description of the sample, not all thestudents had data for the eight semesters of the study. This mayhave been caused for a number of reasons such as student desertion,temporal suspension of studies, or students shifting to other pro-grams inside or outside the university. We believed that because ofthe lack of studies assessing the impact of writing developmentallyat the university, it was first necessary to understand its impact onstudents progressing normally in their programs. Consequently, wefocused our models on those students progressing in their undergrad-uate programs in a timely fashion. Future models should take into

consideration the way interruptions in the studies impact this devel-opment and address the issue of the impact of writing on students'failure. In order to do so, alternative models such as survival analysisshould be adopted. Last but not least, the data presented here werecollected from only one cohort of students studying at only one highlyselective university. So the degree of generalizability of these findingscould be limited. The study should be replicated in a number of newcohorts and in different institutions recruiting a more diverse sampleof students. Before concluding, it is worth noting that perhaps theWCT works best under precisely the conditions and stakes in whichit is used, that is, as a condition of graduating and a diagnostic forreceiving support. Changing the measure to a high-stakes entryrequirement will require a new empirical study of the utility and pre-diction of the measure because of its impact on the selection processof the students.

In spite of these limitations, we believe that this work expands theliterature on argumentative writing and educational assessment.Based on the results summarized above, we believe that this studypresents empirical evidence favoring the use of writing measures asa graduation requirement. In addition, although these results cannotbe directly generalized to a situation where writing is used as ahigh-stakes entry requirement, we believe that our results showthat writing, and the higher order cognitive abilities involved in effec-tive writing, play a critical role in advanced stages of academic train-ing, consequently offering additional support for the consideration ofthis ability for university admission purposes.

Acknowledgments

This study was supported by Grant 1100580 from FONDECYT-CONICYT. The authors would like to thank the Pontificia UniversidadCatolica de Chile's Vicerrectoría Académica, which provided academicsupport to this study.

Appendix 1. Descriptive statistics of WCT dimensions

Appendix 2. Correlation matrix among WCT dimensions

211D.D. Preiss et al. / Learning and Individual Differences 28 (2013) 204–211

References

Atkinson, R. C., & Geiser, S. (2009). Reflections on a century of college admissions tests.Educational Researcher, 38, 665–676.

Beck, S. W., & Jeffery, J. V. (2007). Genres of high-stakes writing assessments and theconstruct of writing competence. Assessing Writing, 12, 60–79.

DEMRE (2012). Temario de Pruebas de Selección Universitaria-Procesos de Admisión 2012[Contents of Tests for University Admissions—Admission Process 2012]. RetrievedJanuary 30th, 2011, from http://www.demre.cl/temario.htm

Drago, J. L., & Paredes, R. D. (2011). La brecha de calidad en la educación chilena [Theeducational gap in Chilean education]. Revista CEPAL, 104, 167–180.

Elacqua, G., Schneider, M., & Buckley, J. (2006). School choice in Chile: Is it class or theclassroom? Journal of Policy Analysis and Management, 25, 577–601.

Garcia-Huidobro, J. E. (2000). Educational policies and equity in Chile. In F. Reimers(Ed.), Unequal schools, unequal chances: the challenges to equal opportunity in theAmericas (pp. 160–181). Cambridge, MA: Harvard University Press.

Geiser, S., & Studley, R. (2002). UC and the SAT: Predictive validity and differential impactof the SAT I and SAT II at the University of California. Educational Assessment, 8, 1–26.

Grigorenko, E. L., Jarvin, L., Nui, W., & Preiss, D. D. (2008). Is there a standard for stan-dardized testing? Four sketches of the applicability (or lack thereof) of standardizedtesting in different educational systems. Extending intelligence: Enhancement andnew constructs.. Mahwah, NJ: Lawrence Erlbaum Associates.

Jeffery, J. V. (2009). Constructs of writing proficiency in US state and national writingassessments: Exploring variability. Assessing Writing, 14, 3–24.

Kellogg, R. T., & Raulerson, B. A., III (2007). Improving the writing skills of collegestudents. Psychonomic Bulletin & Review, 14, 237–242.

Kellogg, R. T., & Whiteford, A. P. (2009). Training advanced writing skills: The case fordeliberate practice. Educational Psychologist, 44, 250–266.

Kellogg, R. T., & Whiteford, A. P. (2012). The development of writing expertise.In E. L. Grigorenko, E. Mambrino, & D. D. Preiss (Eds.),Writing: Amosaic of perspectives(pp. 109–124). New York: Psychology Press.

Kobrin, J. L., Patterson, B. F., Shaw, E. J., Mattern, K. D., & Barbuti, S. M. (2008). Validity ofthe SAT for predicting first-year college grade point average. New York: The CollegeBoard.

Kyllonen, P. C. (2012). The importance of higher education and the role of noncognitiveattributes in college success. Pensamiento Educativo. Revista de Investigación EducacionalLatinoamericana, 49, 84–100.

Lee, J., & Stankov, L. (2012). Large-scale online writing assessments: New approachesadopted in the National Assessment of Educational Progress. In E. L. Grigorenko, E.

Mambrino, & D. D. Preiss (Eds.), Writing: A mosaic of new perspectives (pp. 371–384).New York: Psychology Press.

Manzi, J., Flotts, P., & Preiss, D. D. (2012). Design of a college-level test of writtencommunication. Theorethical and methodological challenges. In E. L. Grigorenko, E.Mambrino, & D. D. Preiss (Eds.), Writing. A Mosaic of perspectives (pp. 385–400).New York: Psychology Press.

Manzi, J., & Preiss, D. D. (in press). Educational assessment and educational achievementin South America. In J. A. C. Hattie & E. M. E.M. Anderman (Eds.), The InternationalHandbook of Student Achievement. New York: Routledge.

McCutchen, D., Teske, P., & Bankston, C. (2008). Writing and cognition: Implications ofthe cognitive architecture for learning to write and writing to learn. In C. Bazerman(Ed.), Handbook of research on writing. History, society, school, individual, text(pp. 451–470). New York: Lawrence Erlbaum Associates.

Mizala, A., & Torche, F. (2012). Bringing the schools back in: The stratification of educa-tional achievement in the Chilean voucher system. International Journal of EducationalDevelopment, 32, 132–144.

Norris, D., Oppler, S., Kuang, D., Day, R., & Adams, K. (2004). The College Board SATwriting validation research study: An assessment of the predictive and incrementalvalidity. Washington, DC: American Institutes for Research.

O'Neill, P., Moore, C., & Huot, B. (2009). A guide to college writing assessment. Logan, UT:Utah State University Press.

Powers, D. E., Fowles, M. E., & Willard, A. E. (1994). Direct assessment, direct valida-tion? An example from the assessment of writing. Educational Assessment, 2, 89.

Shaw, E. J., Mattern, K. D., & Patterson, B. F. (2011). Discrepant SAT critical reading andwriting scores: Implications for college performance. Educational Assessment, 16,145–163.

Singer, J., & Willett, J. (2003). Applied longitudinal data analysis: Modeling change andevent occurrence. New York: Oxford University Press.

Stemler, S. E. (2012). What should university admissions tests predict? EducationalPsychologist, 47, 5–17.

Sternberg, R. J. (2004). Theory-based university admissions testing for a new millenni-um. Educational Psychologist, 39, 185–198.

Sternberg, R. J. (2010). College admissions for the 21st Century. Cambridge, MA: HarvardUniversity Press.

TheNational Commission onWriting in America's schools and colleges (2003). The neglectedR: The need for a writing revolution. New York: The College Board.

Weigle, S. C. (2002). Assessing writing. New York: Cambridge University Press.