Meta-Analysis: Writing with Computers 1992–2002

www.intasc.org

Technology and AssessmentStudy Collaborative

Amie Goldberg, Michael Russell, & Abigail CookTechnology and Assessment Study CollaborativeBoston College332 Campion HallChestnut Hill, MA 02467

Meta-Analysis:Writing with Computers 1992–2002

Meta Analysis: Writing with Computers 1992–2002

Amie Goldberg, Michael Russell, & Abigail CookTechnology and Assessment Study CollaborativeBoston CollegeReleased December 2002

Michael K. Russell, Project Director/Boston College

Copyright © 2002 Technology and Assessment Study Collaborative, Boston College

Supported under the Field Initiated Study Grant Program, PR/Award Number R305T010065, as administered by the Office of Educational Research and Improvement, U.S. Department of Education.

The finding and opinions expressed in this report do not reflect the positions or policies of the Office of Educational Research and Improvement, or the U.S. Department of Education.

Meta Analysis: Writing with Computers 1992–2002

Amie Goldberg, Michael Russell, & Abigail CookTechnology and Assessment Study CollaborativeBoston College

Released December 2002

Introduction

Over the past two decades, the presence of computers in schools has increased rapidly. While schools had 1 computer for every 125 students in 1983, they had 1 for every 9 students in 1995, 1 for every 6 students in 1998, and 1 for every 4.2 students in 2001 (Glennan & Melmed, 1996; Market Data Retrieval, 1999, 2001). Today, some states, such as South Dakota, report a student to computer ratio of 2:1 (Bennett, 2002).

As the availability of computers in schools has increased, so too has their use. A national survey of teachers indicates that in 1998, 50 percent of K–12 teachers had students use word processors, 36 percent had them use CD ROMS, and 29 percent had them use the world wide web (Becker, 1999). More recent national data indicates that 75% of elementary school-aged students and 85% of middle and high school-aged students use a computer in school (U.S Department of Commerce, 2002). Today, the most common educational use of computers by students is for word pro-cessing (Becker, 1999; inTASC, 2002)

Given the increased presence of computers in schools and use of computers by students, there is an increasing demand for evidence that the use of computers impacts student learning. Most recently, the No Child Left Behind (NCLB) Act repeatedly calls for “scientifically” and “research-based evidence” that programs have a posi-tive impact on student learning. Specific to technology, sections 2402 of the NCLB describes the purposes and goals of the Enhancing Education Through Technology Act as supporting “the rigorous evaluation of programs funded under this part, par-ticularly regarding the impact of such programs on student academic achievement” and encouraging “the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by State educational agen-cies and local educational agencies” (NCLB, 2001, Section 2402). Given that students

Meta-Analysis: Writing with Computers 1992–2002 4

use word processing in school more than any other computer application, it is logical to ask: do computers have a positive effect on students’ writing process and quality of writing they produce?

The study presented here responds directly to the call for research-based evi-dence that the use of word processors has a positive impact on student writing. As is described more fully below, the study presented here employs meta-analytic tech-niques, commonly used in fields of medicine and economics, to integrate the findings of studies conducted between 1992–2002. This research synthesis allows educators, administrators, policymakers, and others to more fully capitalize on the most recent findings regarding the impact of word processing on students’ writing.

Word Processing and Student Writing

Over the past two decades, more than 200 studies have examined the impact of word processing on student writing. Over half of these studies, however, were con-ducted prior to presence and wide-scale use of current menu-driven word processors. In addition, these early studies focused on students who were generally less accus-tomed to working with computer technologies compared to students today.

Regardless of these obstacles, syntheses of early research provide some evidence of positive effects. For example, important findings emerged from Cochran-Smith’s (1991) qualitative literature review on word processing and writing in elementary classrooms. Among them, Cochran-Smith found that students of all ages had positive attitudes toward word processing and were able to master keyboarding strategies for use in age-appropriate writing activities. Cochran-Smith also found that students who used word processors spent a greater amount of time writing and produced slightly longer, neater, and more technically error-free texts than with paper and pencil. How-ever, this review of the literature also indicated that word-processing, in and of itself, generally did not impact the overall quality of student writing.

Other early research, however, such as Bangert-Drowns’ (1993) quantitative meta-analysis of 28 individual studies spanning elementary through post-secondary school levels, indicates that word processing contributed to a modest but consistent improvement in the quality of students’ writing: approximately two-thirds of the 28 studies’ results were in favor of the word processor over handwritten text.

In general, the research on word processors and student writing conducted during the 1980’s and early 1990’s suggests many ways in which writing on computers may help students produce better work. Although much of this research was performed before large numbers of computers were present in schools, formal studies report that when students write on computer they tend to produce more text and make more revisions (Vacc, 1987; Dauite, 1986). Studies that compare student work produced on computer with work produced on paper find that for some groups of students, writ-ing on computer also had a positive effect on the quality of student writing (Owston, 1991; Hannafin & Dalton; 1987). This positive effect is strongest for students with learning disabilities, early elementary-aged students and college-aged students (Phoe-nix & Hannan, 1984; Sitko & Crealock, 1986; Hass & Hayes, 1986). Additionally, when applied to meet curricular goals, education technology provides alternative approaches to sustaining student interest, developing student knowledge and skill, and provides supplementary materials that teachers can use to extend student learn-


ing. Although earlier research syntheses reveal just modest trends, individual studies of that era have shown that writing with a computer can increase the amount of writ-ing students perform, the extent to which students edit their writing (Dauite, 1986; Etchinson, 1989; Vacc, 1987), which, in turn, leads to higher quality writing (Han-nafin & Dalton, 1987; Kerchner & Kistinger, 1984; Williamson & Pence, 1989).

Throughout the 1990’s, however, technology has, and continues to, develop at an astonishing pace. Word processing technologies, are easier to use and no longer the classroom novelty they once were. A new generation of studies that examine the impact of word processing on writing fills today’s journals. In response to improve-ments in word processing and students comfort with technology, the study presented here builds on Cochran-Smith’s (1991) and Bangert-Drowns’ (1993) work by inte-grating research conducted since 1991 that has focused on the impact of word proces-sors on the quantity and quality of student writing.

The study presented here differs in two ways from the two previous meta-analysis described above. First, while Cochran-Smith’s (1991) study was qualitative in nature and Bangert-Drowns’ (1993) employed a quantitative meta-analytic technique, this study aims to combine quantitative and qualitative methods in order to provide a richer, more encompassing view of all data available for the time period under study.

Secondly, the quantitative component provides an expanded scope on student and learning environment level variables in relation to writing performance. These supplemental analyses include factors such as: students’ grade level, keyboarding skills, school setting (urban, suburban, rural), etc.

The specific research questions addressed in this study are:

• Does word processing impact K–12 student writing? And, if so, in what ways (i.e., is quality and/or quantity of student writing impacted)?

• Does the impact of word processing on student writing vary according to other factors, such as student-level characteristics (as described above)?


Methodology

Meta-analytic procedures refer to a set of statistical techniques used to system-atically review and synthesize independent studies within a specific area of research. Gene Glass first proposed such methods and coined the term “meta-analysis” in 1976. “Meta-analysis refers to the analysis of analyses … it …refer[s] to the statistical analy-sis of a large collection of results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature (Glass, p. 3).” The meta-analytic portion of the study was con-ducted using procedures set forth by Lipsey and Wilson (2001) and Hedges and Olkin (1985). The methodology followed five phases:

• identification of relevant studies,

• determination for inclusion,

• coding,

• effect size extraction and calculation, and

• data analyses.

Each of these phases is described separately below.

Identification of Relevant Studies

The search for relevant studies was as exhaustive as possible. Methods used to find studies that focused on word processing included:

• Searching online databases such as ERIC, Educational Abstracts, PsychLit, and Dissertation Abstracts,

• Searching websites known to reference or contain research related to edu-cational technology such as the US Department of Education, technology and educational research organizations.

• Searching scholarly e-journals that may not be indexed

• Employing general search engines (e.g., Google) in keyword searches for additional manuscripts that either had not yet been catalogued in ERIC or were currently under refereed journal review (yet posted on the researcher’s own webpage), and

• Directly inquiring with researchers known to be active in studying educa-tional technology about relevant work via email.

To maximize the pool of studies for consideration, search strategies varied slightly depending on the structure of the source, and included a variety of combinations of terms in each search. Search terms included different forms (i.e. computerized and computer; word process and word processing, etc.) of such words as: computer, writing, word processing, pencil-and-paper, and handwritten.

If, based on the article’s abstract/description, relevancy to the present study could not be determined, it was collected for possible inclusion. The resulting collection included 99 articles (see Appendix C).


Determination for Inclusion

The inclusion criteria for the meta-analysis were stringent. Each study had to:

• be a quantitative study, conducted between the years of 1992-2002, in which results were reported in a way that would allow an effect size calcu-lation, have employed a research design which allowed for either a measure of word-processing’s impact on writing over time, OR a direct comparison between paper-and-pencil writing and computerized writing,

• have ‘quality of student writing’ and/or ‘quantity of student writing’ and/or ‘revision of student writing’ as its outcome measure(s),

• not specifically focus on the effects of grammar and spell-checkers or heav-ily multimedia-enhanced word processing software,

• not examine differences in writing within the context of a test administra-tion (i.e, focused on the mode of test administration rather than the mode of learning), and

• focus on students in grades K–12.

Independently, two researchers read all collected studies to determine eligibil-ity for inclusion based on above criteria. Any discrepancies between researchers were discussed and resolved. In total, 26 studies met all inclusion criteria. An additional 35 studies/articles were on target regarding the topic, but were either qualitative, insuf-ficient in reporting quantitative data (to enable effect size extraction), or were concep-tual or commentary papers that focused on how word processors could be used for instruction. These studies were set aside for separate analysis. The research focus of the remaining 38 articles did not match the purposes of this study. Figure 1 illustrates the results of the literature search and criteria screening, and Figure 2 depicts the studies included in the meta-analysis classified by their measured outcomes.

Figure 1: Computers and Writing 1992–2002: Articles Collected in Literature Search by Type (N=99)

17%(n=17)

13%(n=13)

5%(n=5)

38%(n=38)

26%(n=26)

Focus not applicable to meta-analysis

Included in meta-analysis

Ineligible: Commentary/instructionalpractices articles

Ineligible: quantitative studies notincluding sufficient statistical data

Qualitative studies


Figure 2: Studies Included in Meta-Analysis by Outcomes Measured

31%(n=8)

9

8

7

6

5

4

3

2

1

0Quality Quantity RevisionQuality &

QuantityQuality &Revision

Quality,Quantity, &Revision

Quantity &Revision

23%(n=6)

19%(n=5)

12%(n=3)

8%(n=2)

4%(n=1)

4%(n=1)

Coding of Study Features and Outcome Measures

Study features were coded to aid examination of methodological and substantive characteristics that may contribute to variations in results among studies.

Based on a review of the literature, a coding framework was constructed to encom-pass salient features of each study. According to this coding scheme, two researchers independently coded each study. Afterwards, coding was discussed between research-ers on a study-by-study basis. Coding discrepancies, which occurred infrequently, were discussed and resolved by consulting the original research study.

The final coding frame encompasses seven categories of study descriptors includ-ing: publication type, research methodology, student characteristics, technology related factors, writing environment factors, instructional support facators, and out-come measures.

Appendix A contains a full description of the variables and all thirty-three levels included in the coding framework.

After all studies were coded, a variable representing “methodological quality” (Shadish & Haddock, 1994; Moher & Olkin, 1995) was derived from a subset of the coded variables. For each study, methodological quality was based on a sixteen point scale. This scale was based on the following formula:

• one point was assigned for each dichotomous variable coded as “yes” in the “Research Methodology” category,

• one point for studies obtained from refereed journals (“Publication Type”),

• a maximum of three points for the “Intervention time/Duration of study” and “Sample Size” variables,


• heterogeneity of the sample’s gender and race/ethnicity were each awarded one point (“Student Characteristics”), and

• mention of at least one demographic descriptor for the study’s sample (i.e., gender, race, geographic setting (rural, urban, suburban).

Finally, there was some ambiguity in study reporting which sometimes made coding study features a challenge. Where the presence or absence of a feature could not be reasonably detected (explicitly or by implication), an additional code, “no information available,” was employed.

The codes assigned to each study along with all data used to calculate effect sizes are presented in a datafile available from inTASC.

Extracting and Calculating Effect Sizes

The meta-analytic portion of the data analysis requires the calculation of effect sizes. Conceptually, an effect size represents the standardized difference between two groups on a given measure. Mathematically, it is the mean difference between groups expressed in standard deviation units. In this study, for example, effect sizes were calculated taking the mean performance difference between computerized and paper-and-pencil groups and dividing it by a pooled standard deviation. Generally speaking, effect sizes between .2 and .5 standard deviation units are considered small, those between .5 and .8 standard deviation units are medium, and effect sizes .8 or greater are considered large.

In order to decrease the probability of falsely concluding that word processing has an effect on student writing (i.e., committing a Type I error), the unit of analysis is an “independent study finding.” For each of the three outcome measures, an indepen-dent effect size was calculated. For studies that reported more than one measure for a particular outcome for the same sample (i.e., “writing quality” was often measured in more than one way per study; mechanics, content, organization, etc. were frequently encountered sub-domains), overall means and standard deviations across these mea-sures were calculated and used to calculate a single effect size. In this way, the assump-tion of independence was preserved and inflated Type I error rates were controlled for, yet no study findings were ignored.

At the outset of the study, we had hoped to base the calculation of effect sizes using gain scores (the difference between scores on post-test and pre-test measures). Unfortunately, a considerable number of studies either lacked a pre-post design, or failed to report pre-test data. This precluded the most compelling perspective from being meta-analyzed: comparing gain scores between paper-and-pencil and computer writing groups.

In order to maximize the number of studies included in the analysis, the few pre- and post-test designs were analyzed only in terms of post-test data. This enabled results from the pre/post studies to be analyzed with post-only design data. For all three outcomes (i.e., quantity of writing, quality of writing, and revisions), the stan-dardized mean difference effect size statistic was employed. Since it has been docu-mented that this effect size index tends to be upwardly biased when based on small sample sizes, Hedges’ (1981) correction was applied.

Effect sizes from data in the form of t- and F-statistics, frequencies, and p-values were computed via formulas provided by Lipsey & Wilson (2001).


Adjusting for Bias and Applying Inverse Variance Weights

Following standard meta-analytic procedures, an inverse variance weight was applied to each effect size. Essentially, this procedure weights each effect size by the inverse of its sampling variance in order to give more weight to findings based on larger sampling sizes. Thus, all inferential statistical analyses were conducted on weighted effect sizes.

Outlier analyses of the sampling weights and effect sizes were also performed. No outliers were identified for effect sizes. However, for the “Quantity of writing” analy-ses, two sampling weights were more than two standard deviations from the mean sampling weight. Following a procedure originally employed by Lipsey (1992), the inverse variance weights in this study were adjusted so that they did not over-weight the effects found in these two studies.1

Data Analysis

Three types of data analyses were performed. First, using the effect size extracted from each study, an overall effect size across studies was calculated and tested for statis-tical significance. Second, analyses were performed to investigate the potential effects of publication bias. Finally, to investigate the extent to which study features moder-ated the effect on outcome measures, regression analyses were performed. Below, we describe the methods used to explore publication bias and moderating effects.

Publication Bias

Publication bias analyses were performed via Forest plots, funnel plots, and the fail-safe N analysis. Forest plots were used to visually convey the contribution of each study to its meta-analysis, by plotting study effect sizes and corresponding confidence interval bars in a single display. Funnel plots, another widely-used technique for detecting publication bias, were also employed. These plots graphically investigate possible gaps among the studies’ findings by simply plotting effect sizes against sample sizes. Finally, a fail-safe n analysis (Orwin, 1983) was conducted for each meta-analy-sis. This analysis addresses the “file-drawer” problem in meta-analytic research and provides an estimate of the number of insignificant, unpublished studies that would have to exist in order to render a statistically significant meta-analytic finding insig-nificant.

Significance and Homogeneity Analysis

For each meta analysis, an independent set of effect sizes were extracted, weighted and then aggregated. Prior to exploring the extent to which other factors, such as grade level or publication type, influence the effect sizes, a test for homogeneity was conducted. In essence, the test of homogeneity examines whether the group of effect sizes are part of the same population of effect sizes and thus are not influenced by any other variable. As Table 1 indicates, the effect sizes included in the quantity and quality meta analyses are heterogenous. For this reason, additional analyses were con-ducted in an attempt to identify other factors that may influence the study findings. Due to the small number of studies that included measures of revisions, a formal test for homogeneity was not possible.


Table 1: Results of Tests for Homogeneity

Quantity of Writing (n=14)

Min ES Max ES Weighted SD Homogeneity (Q) df P

-1.617 11.971 4.913 4120.6571 13 .0001

Quality of Writing (n=15)

Min ES Max ES Weighted SD Homogeneity (Q) df P

-2.897 30.117 10.801 24396.9961 14 .0001

Moderator Variable Multiple Regression Models

To explore factors that may influence the effect of word processing on the quan-tity and/or quality of student writing, regression analyses were conducted in which the coded study features were independent variables. These analyses were limited by two conditions. First, these analyses could only include study features that were reported by most researchers. Second, for each study feature included in the regression analyses, there had to be variation among studies. For several study features, all studies received the same code and thus did not vary. These two conditions severely limited moderator analyses.

For each outcome variable, frequencies of study feature variables were examined. After suitable independent variables were identified, variables with more than two levels were recoded into dummy variables. These variables were then categorized into groups by theme. For example, variables such as “presence of control group,” “length of intervention,” “type of publication,” and “conversion of handwritten student work to word processed format” fell under the theme labeled “Study’s Methodological Quality.” Variables such as: “technical assistance provided to students”, “student par-ticipation in peer editing,” “students receive teacher feedback,” were included in the “Student Support” theme.

Ideally, for each outcome, each themed group of variables would be entered as a single block and themed groups would be entered stepwise into a single regression model. However, this was not statistically possible due to the small number of effect sizes. Instead, each themed group of variables was entered as a single block of indepen-dent variables and each theme was analyzed in separate regression models.

Summary of Findings

In this section, we present a summary of the findings. Readers who are familiar with meta analytic techniques or who desire a more technical presentation of the find-ings are encouraged to read Appendix B.

The analyses focuses on three outcome variables commonly reported by stud-ies that examine the impact of word processors on student writing. These variables include: Quantity of Writing, Quality of Writing, and Revision Behavior. Below, find-ings for each of these variables are presented separately.


Quantity of Writing

Fourteen studies included sufficient information to calculate effect sizes that compare the quantity of writing, as measured by word count, between computer and paper-and-pencil groups.

Figure 3 depicts the effect sizes and the 95% confidence interval for all 14 studies sorted by publication year. The fifteenth entry depicts the mean weighted effect size across all fourteen studies, along with the 95% confidence interval.

Figure 3: Forest Plot of Quantity of Writing Meta-Analysis

Effect Size

Effect Size

Lower 95% Confidence lnterval Upper 95% Confidence lnterval

Author

Owston, et al.

D'Odorico &Zammuner

Snyder

Peterson

Hagler

Olson

Jones

Brigman

Wolfe, et al.

Nichols

Dybahl, et al.

Godsey

Padgett

Barrera, et al.

mean

1992

1993

1993

1993

1993

1994

1994

1994

1996

1996

1997

2000

2000

2001

PublicationYear

GrandN*

136

51

51

36

76

14

20

12

60

60

41

44

32

36

669

AdjustedEffectSize

0.00

0.56

0.78

1.31

0.47

0.02

0.48

1.23

-0.05

0.87

-0.14

1.31

0.52

0.21

0.541

Lower95%CI

-0.34

0.00

0.21

0.59

0.01

-1.03

-0.41

0.00

-0.41

0.34

-0.77

0.66

-0.18

-0.44

0.380

Upper95%CI

0.34

1.12

1.35

2.03

0.93

1.07

1.37

2.46

0.3

1.4

0.48

1.96

1.23

0.87

0.702

0 1 2-1-2

0 1 2-1-2*Grand N = npaper + ncomputer

Figure 3 indicates that four of the fourteen studies had effect sizes that were approximately zero or negative, but which did not differ significantly from zero. Figure 1 also shows that four of the fourteen studies had positive effect sizes that


differed significantly from zero. In addition, the mean weighted effect size across all fourteen studies is .50, which differs significantly from zero. Thus, across the fourteen studies, the meta analysis indicates that students who write with word processors tend to produce longer passages than students who write with paper-and-pencil.

Recognizing that our search for studies may have missed some studies that have not been published, a “fail-safe N” analysis (Orwin, 1983) was conducted to estimate the number of studies needed to report no effect to nullify the mean adjusted effect size. This analysis indicates that in order to reverse the effect size found, there would need to be 24 unpublished studies that found no effect. Given that only 14 studies that fit the selection criteria were found and that only four of these had non-positive effect sizes, it seems highly unlikely that an additional 24 studies that found non-posi-tive effects exist. This suggests that our meta-analytic findings are robust to publica-tion bias.

As described above, regression analyses were performed to explore factors that may influence the effect of word processing on the quantity of student writing. These analyses indicated that student supports (i.e., keyboard training, technical assistance, teacher feedback, and peer editing) were not significant factors affecting the quantity of student writing. Similarly, student characteristics (i.e., keyboard experience prior to the study, student achievement level, school setting, and grade level) also were not significant factors affecting the quantity of student writing, although grade level did approach statistical significance. Finally, the study characteristics (i.e., publication type, presence of control group, pre-post design, length of study) were not related to the effect of word processing on the quantity of student writing.

Recognizing that studies that lasted for less than six weeks may not provide enough time for the use of word processors to impact student writing, a separate set of regression analyses were performed for the sub-set of studies that lasted more than six weeks. For this sub-set of studies, a significant relationship between school level and effect size was found. On average, effect sizes were larger for studies that focused on middle and high school students as compared to elementary students. All other factors were remained insignificant.

In short, the meta analysis of studies that focused on the effect of word process-ing on the quantity of student writing found a positive overall effect that was about one-half standard deviation. This effect tended to be larger for middle and high school students than for elementary students

Quality of Writing

Fifteen studies included sufficient information to calculate effect sizes that com-pare the quality of writing between computer and paper-and-pencil groups.

Figure 4 depicts the effect sizes and the 95% confidence interval for all 15 studies sorted by publication year. The sixteenth entry depicts the mean weighted effect size across all fifteen studies, along with the 95% confidence interval.

Figure 4 indicates that four of the fifteen studies had effect sizes that were approxi-mately zero or negative, but which did not differ significantly from zero. Since the power in meta-analysis is the aggregation of findings across many studies, it is not unusual to find a subset of studies that contradict the overall trend of findings. In this case, a qualitative examination did not reveal any systematic differences among these


studies’ features as compared with those studies reporting positive effect sizes. Figure 4 also shows that the eleven remaining studies had positive effect sizes and that seven of these effect sizes differed significantly from zero. In addition, the mean adjusted effect size across all fifteen studies is .41, which differs significantly from zero. Accord-ing to Cohen’s criteria for effect sizes, this is considered a small to moderate effect. Thus, across the fifteen studies, the meta analysis indicates that students who write with word processors tend to produce higher quality passages than students who write with paper-and-pencil.

Figure 4: Forest Plot of Quality of Writing Meta-Analysis

Effect Size

Effect Size

Lower 95% Confidence lnterval Upper 95% Confidence lnterval

AuthorPublication

YearGrandN*

AdjustedEffectSize

Lower95%CI

Upper95%CI 0 1 2-1-2

0 1 2-1-2

Owston, et al. 1992 136 0.38 0.04 0.72

Hagler 1993 38 0.96 0.49 1.44

Jones 1994 20 1.25 0.29 2.21

Jackiewicz 1995 58 0.62 0.09 1.15

Keetley 1995 23 0.20 -0.62 1.02

Lam &Pennington 1995 34 0.25 -0.42 0.93

Nichols 1996 60 0.01 -0.5 0.52

Lichetenstein 1996 32 0.77 0.05 1.49

Wolfe,et al. 1996 120 -0.06 -0.42 0.3

Breese, et al. 1996 44 0.83 0.21 1.44

Langone et al. 1996 12 0.43 -0.71 1.58

Jones &Pellegrino 1996 20 -0.61 -1.5 0.29

Lerew 1997 150 0.88 0.55 1.22

Dybdhal, et al. 1997 41 -0.20 -0.83 0.42

Head 2000 50 0.43 -0.13 0.99

mean 838 0.410 0.340 0.481

*Grand N = npaper + ncomputer


Recognizing that our search for studies may have missed some studies that have never been published, the “fail-safe N” analysis was again conducted. This analysis indicates that in order to reverse the effect size found, there would have to be 16 unpublished studies that found no effect. Given that only 15 studies that fit the selec-tion criteria were found and that only four of these had non-positive effect sizes, it seems highly unlikely that an additional 16 studies that found non-positive effects exist.

As described above, regression analyses were performed to explore factors that may influence the effect of word processing on the quality of student writing. These analyses indicated that student supports (i.e., keyboard training, technical assistance, teacher feedback, and peer editing) were not significant factors affecting the quality of student writing. Similarly, the study characteristics (i.e., type of publication, employ-ment of random assignment, employment of pre-post design, single vs. multiple class-room sampling, length of study, etc.) were not related to the effect of word processing on the quality of student writing. However, when examining student characteristics (i.e., keyboard experience prior to the study, student achievement level, school setting, and grade level), a statistically significant relationship was detected between grade level and quality of writing: as school level increased, the magnitude of the effect size increased.

Recognizing that studies that lasted for less than six weeks may not provide enough time for the use of word processors to impact student writing, a separate set of regression analyses were performed for the sub-set of studies that lasted more than six weeks. For this sub-set of studies, no significant relationships were found. This suggests that the relationship between school level and quality of writing occurred regardless of the length of study.

In short, the meta analysis of studies that focused on the effect of word processing on the quality of student writing found a positive overall effect that was about four tenths of a standard deviation. As with the effect for quantity, this effect tended to be larger for middle and high school students than for elementary students.

Revision Behavior

Only six of the thirty studies that met the criteria for inclusion in this study included measures related to revisions. Of these six studies, half were published in ref-ereed journals, half took place in elementary schools, and only one employed a sample size greater than thirty.

Because of the small sample size (only 6) coupled with the reporting of multiple measures of revisions, which could not be combined into a single measure for each study, it was not possible to calculate an average effect size. Nonetheless, these six stud-ies all report that students made more changes to their writing between drafts when word processors were used as compared to paper-and-pencil. In both studies focused on revision and quality of writing, revisions made by students using word processors resulted in higher quality writing than did students revising their work with paper and pencils. It should also be noted that one study found that students writing with paper-and-pencil produced more substantive revisions than did students who used word processors.


In short, given the small number of studies that compared revisions made on paper with revisions made with word processors coupled with the multiple methods used to measure revisions, it is difficult to estimate the effect of computer use on stu-dent revisions.

Qualitative Analysis of Excluded Studies

In total, 65 articles published between 1992 and 2002 that focused on the effects of computers on student writing were found during our search (see Appendix C). Of these, 30 met our criteria for inclusion in the quantitative portion of the meta-analyses. In many cases, the studies that were excluded contained information about the effect of computers on student writing, but did not report sufficient statistics to calculate effect sizes. In several cases, the excluded studies did not focus on the three variables of interest – quantity of writing, quality of writing, and revisions – but instead provide information about the effect of computers on other aspects of student writing. And in still other cases, excluded studies employed qualitative methods to explore a variety of ways in which computers may impact student writing.

In this section, we summarize the findings across the excluded studies. We do this both to supplement the findings of our quantitative meta analysis and to check that our criteria for inclusion in the meta analysis did not systematically bias our analysis. It is important to remember that the studies summarized below were selected as pos-sible candidates for inclusion in our meta analyses and were selected because it was believed they included information about the effect of computers on the quantity of writing, quality of writing, and the amount of revisions made while writing. Thus, the sample of studies summarized here is not representative of all studies that focus on computers and writing.

Writing as a Social Process

Several of the excluded studies examined how interactions among students were effected when students wrote with computers. These studies describe in rich detail the social interactions between students as they engage in the writing process. In general, these studies found that when students use computers to produce writing, the writing process becomes more collaborative and includes more peer-editing and peer-medi-ated work (Snyder, 1994; Baker & Kinzer, 1998; Butler & Cox, 1992). As an exam-ple, Snyder (1994) describes changes in classroom-talk when students use computers rather than paper-and-pencil. In Snyder’s study, teacher-to-student communications were predominant in the “pens classroom” while student-to-student interactions occurred more frequently in the “computers classroom.” In addition, Snyder describes how the teacher’s role shifted from activity leader in the “pens classroom” to that of facilitator and “proof-reader” in the “computers classroom.” Snyder attributed this change to students’ increased motivation, engagement and independence when writ-ing with computers.


Writing as an Iterative Process

One study focused specifically on how the writing process changed when students wrote on computers versus on paper (Baker & Kinzer, 1998). This study found that when students wrote on paper, the writing process was more linear such that students generally brainstormed, outlined their ideas, wrote a draft, then revised the draft, produced a second draft, and then proof read the draft before producing the final ver-sion. When students produced writing on computers, however, the process of produc-ing and revising text was more integrated such that students would begin recording ideas and would modify their ideas before completing an entire draft. Students also appeared more willing to abandon ideas in mid-stream to pursue a new idea. In this way, the process of revision tended to begin earlier in the writing process and often was performed as new ideas were being recorded. Rather than waiting until an entire draft of text was produced before beginning the revision process, students appeared to criti-cally examine and edit their text as ideas flowed from their mind to written form.

Computers Motivate Students

A few of the excluded studies noted that computers seemed to motivate students, especially reluctant writers. In her case study of two third grade “reluctant writers,” Yackanicz’s (2000) found that these students were more willing to engage and sustain in writing activities when they used the computer. As a result, these students wrote more often, for longer periods of time, and produced more writing when they used a computer instead of paper-and-pencil.

Keyboarding and Computers

One study that focused on a range of middle school students found that it tended to take students longer periods of time to produce writing on computers as compared to on paper (Jackowski-Bartol, 2001). Although no formal measures of keyboarding skills were recorded, Jackowski-Bartol attributed this difference to a lack of keyboard-ing skills and inferred that as students keyboarding skills improve, the amount of time required to produce writing on computers would decrease.

Effects on Student Writing

Several of the excluded studies examined the effect of computers on various aspects of students’ final written products. In an examination of writing produced by high school students who participated in a computer technology infusion product, Allison (1999) reported improvement in students literacy skills, attitudes toward writ-ing, and an increase in the number of students who demonstrated high-order thinking skills in their writing. In a three-week study of 66 sixth graders who were randomly assigned to write on computer or paper, Grejda and Hannafin (1992) found that the quality of student writing was comparable, but students who used word-processors introduced fewer new errors when revising their text as compared to students who re-wrote their work on paper.

In a three year study that examined the effect of computers on student writing, Owston and Wideman (1997) compared changes in the quantity and quality of writ-ing of students attending a school in which there was one computer for every fifteen students versus a school in which there was one computer for every three students.


After three years, Owston and Wideman found that the quality of writing improved at a faster rate in the high access school and that the mean length of composition was three times longer in the high access school. The researchers, however, acknowledged that their findings do not take into account differences between teachers or the demo-graphics of the students. Nonetheless, the researchers state that these variables did not appear to explain the superior writing produced by students in the high access school.

Not all studies, however, report positive effects of computers on student writing. In a three year study in which seventy-two students wrote on computer and paper, Shaw, Nauman and Burson (1994) report that the length and quality of writing pro-duced on paper was higher than writing produced on computer. This finding occurred even though students who wrote on computer had received keyboarding instruction. The authors described writing produced on computer as “stilted” and less creative.

Discussion

Responding directly to calls for systematic, research-based evidence of the effects of computers and student learning, this study employed meta-analytic techniques more commonly used in medicine and economics to summarize findings across mul-tiple studies. Although a large number of studies initially identified for inclusion in the meta-analysis had to be eliminated either because they were qualitative in nature or because they failed to report statistics required to calculate effect sizes, this study indicates that instructional uses of computers for writing are having a positive impact on student writing, both in terms of quantity and quality.

In addition, the findings reported in the excluded studies are consistent with both the findings of our quantitative meta analyses and many of the findings presented in Cochran-Smiths (1991) and Bangert-Downs (1993) summaries of research conducted prior to 1992. In general, research over the past two decades consistently finds that when students write on computers, writing becomes a more social process in which students share their work with each other. When using computers, students also tend to make revisions while producing, rather than after producing, text. Between initial and final drafts, students also tend to make more revision when they write with com-puters. In most cases, students also tend to produce longer passages when writing on computers.

Early research consistently found large effects of computer-based writing on the length of passages and less consistently reported small effects on the quality of student writing. In contrast, although our meta-analyses of research conducted since 1992 found a larger overall effect size for the quantity of writing produced on computer, the relationship between computers and quality of writing appears to have strengthened considerably. When aggregated across all studies, the mean effect size indicated that on average students who develop their writing skills while using a computer produce written work that is .4 standard deviations higher in quality than those students who learn to write on paper. On average, the effect of writing with computers on both the quality and quantity of writing was larger for middle and high school students than for elementary school students.


For educational leaders questioning whether computers should be used to help students develop writing skills, the results of our meta-analyses suggest that on aver-age students who use computers when learning to write produce written work that is about .4 standard deviations better than students who develop writing skills on paper. While teachers undoubtedly play an important role in helping students develop their writing skills, the analyses presented here suggest that when students write with computers, they engage in the revising of their work throughout the writing process, more frequently share and receive feedback from their peers, and benefit from teacher input earlier in the writing process. Thus, while there is clearly a need for systematic and high quality research on computers and student learning, those studies that met the rigorous criteria for inclusion in our meta-analyses suggest that computers are a valuable tool for helping students develop writing skills.

Endnotes 1 The smallest grand sample size among the fourteen studies measuring “quantity of writing” was 12, while the largest

grand sample size was 136. This variation in sample size resulted in a mean inverse variance weight of 12.30 (SD = 8.75), and a range from 2.52 through 31.03. The two largest weights were slightly greater than two standard deviations above the mean in value, and therefore were winsorized down to the value of two standard deviations above the mean,

29.80.


ReferencesAlbertson, L. R. & Billingsley, F. F. (1997). Improving Young Writers’ Planning and

Reviewing Skills while Story-Writing. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, March 24–28, 1997.

Allen, G. & Thompson, A. (1994). Analysis of the Effect of Networking on Computer-Assisted Collaborative Writing in a Fifth Grade Classroom. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA, April 4–8, 1994.

Allison, B. (1999). Facilitating Student Achievement in Writing through the Deft Employment of Computer Technology. Unpublished Master’s Action Research Project, Saint Xavier University/IRI.

Baker E. & Kinzer, C.K. (1998). Effects of technology on process writing: Are they all good? National Reading Conference Yearbook, 47, 428–440.

Bangert-Drowns, R. L. (1993). The word processor as an instructional tool: A meta-analysis of word processing in writing instruction. Review of Educational Research, 63(1), 69–93.

Barrera, M. T., III, Rule, A. C., & Diemart, A. (2001). The effect of writing with computers versus handwriting on the writing achievement of first-graders. Information Technology in Childhood Education Annual, 28.

Becker, H. J. (1999). Internet Use by Teachers: Conditions of Professional Use and Teacher-Directed Student Use. Irvine, CA: Center for Research on Information Technology and Organizations.

Bennett, R.E. (2002). Inexorable and inevitable: The continuing story of technology and assessment. Journal of Technology, Learning and Assessment, 1(1). Retrieved November 1, 2002 from http://www.bc.edu/research/intasc/jtla/journal/v1n1.shtml

Biesenbach-Lucas, S. & Weasenforth, D. (2001). E-mail and word processing in the esl classroom: How the medium affects the message. Language Learning & Technology, 5(1), 135–165.

Bogard, E. A. (1998). The Effects of Computer-Mediated Writing on the Quality and Quantity of Foreign Language Composing. Unpublished Doctoral Dissertation, University of South Florida.

Borthwick, A. G. (1993). Effects of Keyboarding/Typewriting on the Language Arts Skills of Elementary School Students. Paper presented at the Annual Meeting of the American Educational Research Association, Atlanta, GA, April 12–16, 1993.

Bowman, M. (1999). Children, word processors and genre. Scottish Educational Review, 31(1), 66–83.

Bracey, G. (1992). Creative writing: Do word processors help or hurt students’ creativity? Electronic Learning, 12(3), 12.

Breese, C., Jackson, A., & Price, P. (1996). Promise in impermanence: Children writing with unlimited access to word processors. Early Child Development and Care, 118, 67–91.

http://www.bc.edu/research/intasc/jtla/journal/v1n1.shtml

http://www.bc.edu/research/intasc/jtla/journal/v1n1.shtml


Brigman, D. J. P. (1995). The Effects of Writing with Pencil and Paper Compared with Writing with a Computer and a Word Processing Program on First-Grade Students Who Have Difficulty Writing. Unpublished Doctoral Dissertation, The Johns Hopkins University.

Bruce, B. & Peyton, J. K. (1992). A Situated Evaluation of Computer Networking to Teach Writing. Technical Report No. 565. Urbana, IL: Center for the Study of Reading.

Bucci, S. M. (1996). Seven at-Risk Fifth-Graders and Factors That May Influence the Quality of Their Written Text: From Paper and Pencil to Word Processed Drafts (Writing). Unpublished Doctoral Dissertation, University of Pittsburgh.

Burley, H. (1994). Postsecondary Novice and Better Than Novice Writers: Effects of Word Processing and a Very Special Computer Assisted Writing Lab. Paper presented at the Southwestern Educational Research Association, San Antonio, TX, January 27–29, 1994.

Butler, S. & Cox, B. (1992). Diskovery: writing with a computer in grade one: A study in collaboration. Language Arts, 69, 633–640.

Casey, J. M. (1992). Writing to Read in the Classroom: A Literature-Based Writing Literacy Environment. Simi Star Project Report. (ED 367 952).

Creskey, M. N. (1992). The Perspective of hindsight. International Journal of Instructional Media Vol. 19(1), 79–84.

Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61, 107–155.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences. (1st ed.) Hillsadale, NJ: Erlbaum.

Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20 (May), p. 141–59.

DeFoe, M. C. (2000). Using Directed Writing Strategies to Teach Students Writing Skills in Middle Grades Language Arts. Unpublished Ed.D. Practicum II Report, Nova Southeastern University.

D’Odorico, L. & Zammuner, V. L. (1993). The influence of using a word processor on children’s story writing. European Journal of Psychology of Education, 8(1), 51–64.

Dodson, L. E. (2000). The Effects of Using Technology to Enhance Student Ability to Read, Organize and Write Informative Text. Unpublished Master’s Thesis, University of Manitoba.

Dybdahl, C. S., Shaw, D. G., & Blahous, E. (1997). The impact of the computer on writing: No simple answers. Computers in the Schools, 13(3), 41–53.

Ediger, M. (1996). Middle School Pupil Writing and the Word Processor (ED393101).

Escobedo, T. H. & Allen, M. (1996). Preschoolers’ Emergent Writing at the Computer. Paper Presented at the Annual Meeting of the American Educational Research Association, New Tork, NY, April 8–12, 1996.

Etchison, C. (1989). Word processing: A helpful tool for basic writers. Computers and Composition, 6(2), 33–43.


Fan, H.-L. & Orey, M. (2001). Multimedia in the classroom: its effect on student writing ability. Journal of Research on Technology in Education, 33(5).

Fletcher, D. C. (2001). Second graders decide when to use electronic editing tools. Information Technology in Childhood Education Annual, 155–74.

Freitas, C. V. & Ramos, A. (1998). Using Technologies and Cooperative Work to Improve, Oral, Writing, and Thinking Skills: Voices from Experience. Proceedings of the Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT).

Gaddis, B., Napierkowski, H., Guzman, N., & Muth, R. (2000). A Comparison of Collaborative Learning and Audience Awareness in Two Computer-Mediated Writing Environments. Paper presented at the Annual Proceedings of Selected Research and Development Papers Presented at the National Convention of the Association for Educational Communications and Technology, 23rd, Denver, CO, October 25–28, 2000.

Gallick-Jackson, S. A. (1997). Improving Narrative Writing Skills, Composition Skills, and Related Attitudes among Second Grade Students by Integrating Word Processing, Graphic Organizers, and Art into a Process Approach to Writing. Unpublished M.S. Practicum Project, Nova Southeastern University.

Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.

Glass, G. V., McGaw, B., & Smith, M.L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: Sage Publications.

Glennan, T. K. & Melmed, A. (1996). Fostering the use of educational technology: Elements of a national strategy. Santa Monica, CA: RAND.

Godsey, S. B. (2000). The Effects of Using Microsoft Word[R] on Journal Word Counts in the High School English Classroom. Unpublished Master of Arts Action Research Project, Johnson Bible College.

Greenleaf, C. (1994). Technological indeterminacy. The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, Vol. 11(1), 85–130.

Grejda, G. F. & Hannafin, M. J. (1992). Effects of word processing on sixth graders’ holistic writing and revisions. Journal of Educational Research, 85(3), 144–149.

Hagler, W. J. (1993). The Effects of the Word Processor on the Revision Behaviors of Sixth-Grade Students. Unpublished Doctoral dissertation, Auburn University.

Hannafin, M. J. & Dalton, D. W. (1987). The effects of word processing on written composition. The Journal of Educational Research, 80 (July/Aug.) p. 338–42.

Haas, C. & Hayes, J. R. (1986). Pen and paper versus the machine: Writers composing in hard-copy and computer conditions (CDC Technical Report No. 16). Pittsburgh, PA: Carnegie-Mellon University, Communication Design Center.

Hartley, J. (1993). Writing, thinking, and computers. British Journal of Educational Technology, 24(1), pp. 22–31.


Head, B. B. (2000). Revision Instruction and Quality of Writing by Eighth-grade Students Using Paper and Pencil or Word Processing. Unpublished Doctoral disertation, Oakland University.

Hedges, L. (1981). Distributed theory for Glass’s estimator of effect size and related estimators, Journal of Educational Statistics, 6, 107–128.

Hedges, L. & Olkin, I. (1985). Statistical Methods for Meta-Analysis. Orlando, FL: Academic Press.

Hood, L. M. (1994). Effects of Computer Correspondence on Student Writing (Technical Report). Curry School of Education: University of Virginia.

Hydrick, C. J. (1993). The Interaction of a Nine-Year-Old with a Word Processor. Unpublished Doctoral dissertation, Arizona State University.

Jackiewicz, G. (1995). The Effect of Computer Based Instruction on Writing at the Elementary Level. Unpublished Master of Arts Thesis, Kean College of New Jersey.

Jackowski-Bartol, T. R. (2001). The Impact of Word Processing on Middle School Students. Unpublished Master’s Thesis, Chestnut Hill College.

Jankowski, L. (1998). Educational computing: Why use a computer for writing? Learning and Leading with Technology, 25(6), 30–33.

Jones, I. (1994a). The effect of a word processor on the written composition of second-grade pupils. Computers in the Schools, 11(2), 43–54.

Jones, I. (1994b). First-Grade Students’ Computer Supported Writing: Metacognitive and Linguistic Effects. Unpublished Doctoral Dissertation, University of Georgia.

Jones, I. & Pellegrini A. D. (1996). The effects of social relationships, writing media, and microgenetic development on first-grade students’ written narratives. American Educational Research Journal, 33(3), pp. 691–718.

Joram, E., Wooddruff, E., Bryson, M. & Lindsay, P. (1992). The effects of revising with a word processor on written composition. Research in the Teaching of English, 26(2), 167–193.

Keetley, E. D. (1995). Comparison of First Grade Computer Assisted and Handwritten Process Story Writing. Unpublished Master’s Thesis, Johnson and Wales University.

Kehagia, O. & Cox, M. (1997). Revision changes when using word processors in an english as a foreign language context. Computer Assisted Language Learning, 10(3), 239–253.

Kerchner, L. B. & Kistinger, B. J. (1984). Language processing/word processing: Written expression, computers, and learning disabled students. Learning Disability Quarterly, 7(4), 329–335.

Kumpulainen, K. (1994). Collaborative writing with computers and children’s talk: A cross-cultural study. Computers and Composition, 11(3), 263–273.

Kumpulainen, K. (1996). The nature of peer interaction in the social context created by the use of word processors. Learning and Instruction, 6(3), 243–261.

Kurth, R. J. (1987). Using word processing to enhance revision strategies during student writing activities. Educational Technology, 27(1), 13–19.


Lam, F. S. & Pennington, M. C. (1995). The computer vs. the pen: a comparative study of word processing in a Hong Kong secondary classroom. Computer Assisted Language Learning, 8(1), 75–92.

Langone, J., Levine, B., Clees, T.J., Malone, M., & Koorland, M. (1996). The differential effects of a typing tutor and microcomputer-based word processing on the writing samples of elementary students with behavior disorders. Journal of Research on Computing in Education, 29(2), 141–158.

Lerew, E. L. (1997). The Use of Computers to Improve Writing Skills among Low-Achieving Hispanic Students. Unpublished Doctoral dissertation, University of La Verne.

Lewis, R. B. (1998). Enhancing the Writing Skills of Students with Learning Disabilities through Technology: An Investigation of the Effects of Text Entry Tools, Editing Tools, and Speech Synthesis. Final Report. San Diego, CA: Department of Special Education, San Diego State University.

Lewis, R. B., Ashton, T. M., Haapa, B., Kieley, C. L., & Fielden, C. (1999). Improving the Writing Skills of Students with Learning Disabilities: Are Word Processors with Spelling and Grammar Checkers Useful? Learning Disabilities: A Multidisciplinary Journal, 9(3), 87–98.

Lichtenstein, N. (1996). The Effect of Word Processing on Writing Achievement. Unpublished Master’s Project, Kean College of New Jersey.

Lipsey, M. W. W. & Wilson, D.B. (2001). Practical Meta-Analysis (Vol. 49). Thousand Oaks, CA: Sage Publications.

Lipsey, M.W. (1992) ‘Juvenile delinquency treatment: a meta-analytic inquiry into the variability of effects.’ In T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, T.A. Louis and F. Mosteller (Eds) Meta-analysis for explanation. New York: Russell Sage Foundation.

Lohr, L., Ross, S. M., & Morrison, G. R. (1996). Using a hypertext environment for teaching process writing: An Evaluation study of three student groups. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for Educational Communications and Technology (18th, Indianapolis, IN, 1996).

Lomangino, A. G., Nicholson, J., & Sulzby, E. (1999). The Nature of Children’s Interactions while Composing Together on Computers. CIERA Report. Ann Arbor, MI: Center for the Improvement of Early Reading Achievement.

Lowther, D. L., Ross, S. M., Morrison, G. R. (2001). Evaluation of a laptop program: Successes and recommendations. Paper presented at the National Educational Computing Conference Proceedings (22nd, Chicago, IL, June 25–27, 2001).

Lund, D. M. & Hildreth, D. (1997). Integrating the computer into language arts in a fifth grade classroom: A Developing instructional model. Paper presented at the annual meeting of the National Reading Conference (47th, Scottsdale, AZ, December 3–6, 1997).

MacArthur, C.A., Graham, S., & Schwartz, S.S. (1994). Peers + word processing + strategies = a powerful combination for revising student writing. TEACHING Exceptional Children, 27(1), 24–29.


MacArthur, C. A. (1996). Using technology to enhance the writing processes of students with learning disabilities. Journal of Learning Disabilities, 29(4), 344–354.

McBee, D. (1994). The Effect of technology on emergent writing. Research project, Juneau, AK: University of Alaska Southeast.

McMillan, K. & Honey, M. (1993). Year One of Project Pulse: Pupils Using Laptops in Science and English. A Final Report. Technical Report No. 26. New York, NY: Center for Technology in Education.

Market Data Retrieval. (1999). Technology in Education 1999. (A report issued by Market Data Retrieval). Shelton, CN: Market Data Retrieval.

Mead, A. D. & Drasgow, (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114:3, 449–58.

Mehdi, S. N. (1994). Word Processing and the EFL Writing Classroom: A Case Study of Arabic Speakers. Unpublished Doctoral dissertation, Boston University.

Moeller, B., Bell, L. & Reich, C. (1993). Using networked technology to create new writing environments for deaf students: Teachers’ strategies and student outcomes. Paper presented at the meeting of the Society for Research in Child Development in New Orleans, LA, March 25–28, 1993.

Moher, D. & Olkin, I. (1995). Meta-analysis of randomized controlled trials: A concern for standards. Journal of the American Medical Association, 274, 1962–1963.

Mott, M.S. & Halpin, R. (1999). A Comparative Approach to the Examination of the Technical Qualities of a Pen and Paper Writing Assessment for Elementary Students’ Hypermedia-Created Products. Paper presented at the Annual Meeting of the American Educational Research Association, Montreal, Quebec, Canada, April 19–23, 1999.

Mott, M. S., Sumrall, W. J., & Hodges, M. L. (1997). Process and Computer-Based Elementary Writing Curriculum: A Review of Methods and Assessments. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, Memphis, TN, November 12–14, 1997.

Mott, M. S. & Klomes, J. M. (2001). The synthesis of writing workshop and hypermedia-authoring: Grades 1–4. Early Childhood Research & Practice, 3(2).

Moxley, R. A., Warash, B., Coffman, G., Brinton, K., & Concannon, K. R. (1994). Computer writing development in a prekindergarten class of 4 year olds. Journal of Computing in Childhood Education, 5(2), 211–229.

Moxley, R. A., Warash, B., Coffman, G., Brinton, K., & Concannon, K. R. (1997). Writing development using computers in a class of three-year-olds. Journal of Computing in Childhood Education, 8(2), 133–164.

Nichols, L. M. (1996). Pencil and paper versus word processing: a comparative study of creative writing in the elementary school. Journal of Research on Computing in Education, 29(2), 159–166.

Olson, K. A. (1994). Writing and Revising with Pencils and with Computers: An Analysis of the Processes and Products of Seven First-Grade Children. Unpublished Doctoral dissertation, University of Michigan.


Orwin, R. G. (1983). A fail safe N for effect size in meta-analysis. Journal for Educational Statistics, 8, 157–159.

Osborne, P. (1999). Pilot study to investigate the performance of dyslexic students in written assessments. Innovations in Education and Training International; 36(2), 155–60.

Owston, R. D. (1991). Effects of word processing on student writing in a high-computer-access environment (Technical Report 91–3). North York, Ontario: York University, Centre for the Study of Computers in Education.

Owston, R. D., Murphy, S., & Wideman H. H. (1992). The effects of word processing on students’ writing quality and revision strategies. Research in the Teaching of English, 26(3), 249–276.

Owston, R. D. & Wideman, H. H. (1997). Word processors and children’s writing in a high-computer-access setting. Journal of Research on Computing in Education, 30(2), 202–220.

Padgett, A. L. (2000). Journal Writing in the Elementary School: Word Processor vs. Paper and Pencil. Unpublished Master of Arts Research Paper, Johnson Bible College.

Peacock, M. & Beard, R. (1997). “Almost an Invincible Repugnance?”: Word processors and pupil writers. Educational Review, 49(3), 283–294.

Pennington, M. C. (1993). Exploring the potential of word processing for non-native writers. Computers and the Humanities, 27(3), 149–163.

Peterson, S. E. (1993). A comparison of student revisions when composing with pen and paper versus word-processing. Computers in the Schools, 9(4), 55–69.

Philips, D. (1995). Study 6: Using the Word Processor To Develop Skills of Written Expression. Final Report. Evaluation of Exploratory Studies in Educational Computing. Wellington, NZ: New Zealand Council for Educational Research.

Phoenix, J. & Hannan, E. (1984). Word processing in the grade 1 classroom. Language Arts, 61(8), 804–812.

Pisapia, J. R., Knutson, K., & Coukos, E. D. (1999). The Impact of computers on student performance and teacher behavior. Paper presented at the annual meeting of the Florida Educational Research Association (44th, Deerfield Beach, FL, November 10–12, 1999).

Pohl, V. & Groome, D. (1994). Happy hands: Pre-keyboarding facility for emergent writers. The Computing Teacher, 21(5), 16–18.

Priest, N. B. (1995). Developing writing skills of fourth grade students using a variety of intervention strategies. M.S. Practicum, Nova University.

Reed, W. M. (1996). Assessing the impact of computer-based writing instruction. Journal of Research on Computing in Education, 28(4), 418–437.

Robertson, I., Calder, J., Fung, P., Jones, A., O’Shea, T., & Lambrechts, G. (1996). Pupils, teachers, & palmtop computers. Journal of Computer Assisted Learning, 12(4), 194–204.

Roussey, J. Y., Farioli, F., & Piolat, A. (1992). Effects of social regulation and computer assistance on the monitoring of writing. European Journal of Psychology of Education; 7(4), 295–309.


Roblyer, M. D. (1997). Technology and the oops! effect: Finding a bias against word processing. Learning and Leading with Technology, 24(7), 14–16.

Seawel, L., Smaldino, S.E., Steele, J.L., & Lewis, J.Y. (1994). A descriptive study comparing computer-based word processing and handwriting on attitudes and performance of third and fourth grade students involved in a program based on a process approach to writing. Journal of Computing in Childhood Education, 5(1), 43–59.

Shadish, W.R. & Haddock, C.K. (1994). Combining Estimates of Effect Size. In H.M. Cooper & L.V. Hedges (Eds.), The Handbook of Research Synthesis (pp. 261–281). New York: Russell Sage Foundation.

Shadish, W.R., Cook, T. D., & Campbell, D. T. (2001). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin Company.

Shaw, E.L., Nauman, A.K., & Burson, D. (1994). Comparison of spontaneous and word processed compositions in elementary classrooms: A three-year study. Journal of Computing in Childhood Education, 5(3), 319–327.

Sitko, M.C. & Crealock, C. M. (1986, June). A longitudinal study of the efficacy of computer technology for improving the writing skills of mildly handicapped adolescents. Paper presented at at the Invitational Research Symposium on Special Education Technology, Washington, DC.

Snyder, I. (1993a). The impact of computers on students’ writing: A comparative study of the effects of pens and word processors on writing context, process and product. Australian Journal of Education, 37(1), 5–25.

Snyder, I. (1993b). Writing with word processors: A research overview. Educational Research, 35(1), 49–68.

Snyder, I. (1994). Writing with word processors: The computer’s influence on the classroom context. Journal of Curriculum Studies, 26(2), 143–162.

Technology and Assessment Collaborative (inTASC; in press). Introduction to the USEIT Study (Research Report No. 1). Chestnut Hill, MA: Author.

U.S. Department of Education. (2002). No Child Left Behind Act of 2001. Retrieved May 29, 2002, from http://www.ed.gov/legislation/ESEA02/

U.S. Department of Commerce. (2002). A Nation Online: How Americans are expanding their use of the internet. Washington, DC: Author. Retrieved November 1, 2002, from http://www.ntia.doc.gov/ntiahome/dn/nationonline_020502.htm.

Vacc, N. N. (1987). Word processor versus handwriting: A comparative study of writing samples produced by mildly mentally handicapped students. Exceptional Children, 54(2), 156–165.

Waldman, H. (1995). The Effects of a Multimedia Literacy Tool on First Grade Reading and Writing Achievement. Unpublished Doctoral dissertation, The University of San Francisco.

Walker, C. L. (1997). Computers, revision, and the developmental writer: A case study of on-screen versus off-screen revision processes. Research and Teaching in Developmental Education, 14(1), 23–34.

http://www.ed.gov/legislation/ESEA02/

http://www.ntia.doc.gov/ntiahome/dn/nationonline_020502.htm

http://www.ntia.doc.gov/ntiahome/dn/nationonline_020502.htm


Williamson, M. L. & Pence, P. (1989). Word processing and student writers. In B. K. Briten & S. M. Glynn (Eds.), Computer Writing Environments: Theory, Research, and Design (pp. 96–127). Hillsdale, NJ: Lawrence Erlbaum & Associates.

Wolf, F. M. (1986). Meta-Analysis: Quantitative Methods for Research Synthesis. Thousand Oaks, CA: Sage Publications.

Wolfe, E. W., Bolton, S., Feltovich, B., & Bangert, A.W. (1996). A study of word processing experience and its effects on student essay writing. Journal of Educational Computing Research, 14(3), 269–283.

Yackanicz, L. (2000). Reluctant Writers and Writing-Prompt Software. Unpublished Master’s thesis, Chestnut Hill College.

Zammuner, V. L. (1995). Individual and cooperative computer-writing and revising: Who gets the best results? Learning and Instruction, 5(2), 101–124.

Zhang, Y., Brooks, D.W., Frields, T., & Redelfs, M. (1995). Quality of writing by elementary students with learning disabilities. Journal of Research on Computing in Education, 27(4), 483–499.

Zoni, S. J. (1992). Improving Process Writing Skills of Seventh Grade At-Risk Students by Increasing Interest through the Use of the Microcomputer, Word Processing Software, and Telecommunications Technology. Unpublished Practicum Paper, Nova University.


Appendix A

1) Publication Type (one variable)

• Refereed journal article

• Conference presentation

• Manuscript under journal review

• Doctoral dissertation

• Master’s thesis

• Research organization study/technical report

2) Research Methodology (eight variables, unless otherwise indicated, dichotomous: yes/no)

• random assignment of students

• direct comparison to paper/handwritten writing

• presence of pre- and post-test

• standardized/controlled writing conditions

• intervention time/duration of study − less than six weeks− between six weeks and one semester− more than one semester

• sample size − thirty or less− between thirty-one and one-hundred− more than one-hundred

• In the case of handwritten samples: were they converted to computerized form to ensure blindness of scorers/raters?

• other indicators of sound design (i.e., treatment vs. control groups, coun-terbalanced design, absence of confounding variables, etc.)

3) Student Characteristics (six variables)

• Grade level − elementary− middle− secondary

• Gender description − heterogeneous− homogeneous

• Race/ethnic description − heterogeneous− homogeneous


• School-setting − Rural− Suburban− Urban

• Type of students − Mainstream− SPED/At-risk− Gifted− ESL/ESOL

• Writing ability of students − Low− Average− High

4) Technology-related Factors (seven variables)

• Type of hardware used

• Type of software used

• Description of students’ prior keyboarding skills − no mention− minimal− adequate− advanced

• Description of students’ prior word-processing skills − no mention− minimal− adequate− advanced

• Keyboarding training provided as part of study (yes/no)

• Word processing training provided as part of study (yes/no)

• Technological assistance provided to students during study (yes/no)

5) Writing Environment Factors (two variables)

• Writing within Language Arts/English discipline?

• Type of student writing − Collaborative − Individual

6) Instructional Factors (six variables; all dichotomous: yes/no)

• Did students receive writing instruction during the intervention period?

• Receipt of teacher-feedback/editing

• Receipt of peer-feedback/editing

• Were students allowed to revise without any kind of feedback

• Internet or distance editors

• Did students make use of spell-checkers?


7) Outcome Measures (three variables)

• Quantity of writing − Number of words− Number of t-units− Number of sentences

• Quality of writing − Holistic, judgmental (no rubric)− Mechanics, rubric− Grammar, rubric− Style, rubric

• Revision of writing− Number of revisions− Nature of revisions


Appendix B: Results

Computers and Writing: Quantity

Fourteen independent effect sizes were extracted from fourteen studies that com-pared quantity of writing, as measured by word count, between computer and paper-and-pencil groups. Below we present descriptive highlights of the fourteen studies followed by an analyses of effect sizes, and regression analyses that explore moderating variables.

Descriptive Highlights

As detailed in Table B1, 64% of the studies (n = 9) were published in refereed journals, 14.3% (n = 2) employed random assignment, and more than half (n = 8) sampled from multiple classrooms. For 57% of the studies (n = 8), the research dura-tion lasted between six weeks and one semester, and 86% (n =12) utilized standard-ized writing tasks across groups. In 43% (n = 6) of the studies, students were provided with keyboarding training. Individual writing (as opposed to collaborative writing) was the focus in all fourteen studies, and peer editing, teacher feedback, and technical assistance were available to students in 21% (n = 3) of the studies. It was inconclusive whether or not teacher feedback and/or technical assistance were study features in n =5 and n=9 studies, respectively.

With respect to student demographics, only three studies (21%) provided suf-ficient information that indicated that the sample was gender diverse and four studies (29%) indicated that they had racially/ethnically-diverse student samples. Over half of the studies did not provide sufficient information about the participating students to classify their gender or racial/ethnic diversity. All but two studies (n =12) focused on mainstream education samples, and half (n =7) of the studies were conducted with elementary school students. Finally, two studies occurred in rural, three in urban, and four in suburban settings, while the three studies lack any geographic information.


Table B1: Characteristics of Studies Included in Quantity of Writing Meta-analysis

Study Characteristics n of studies (%)

Refereed journal article

Doctoral dissertation

Master’s thesis

Publication type 9 (64.3%) 3 (21.4%) 2(14.3%)

Yes No No information

Random assignment 2 (14.3%) 12 (85.7%) —

Pre-Post design 6 (42.9%) 8 (57.1%) —

Standardized writing sample 12 (85.7%) 2 (14.3%) —

Keyboarding training included in study 6 (42.9%) 6 (42.9%) 2 (14.3%)

Peer-editing 3 (21.4%) 11 (78.6%) —

Handwritten samples converted to WP format

3 (21.4%) 11 (78.6%) —

Technical assistance provided to students 3 (21.4%) 2 (14.3%) 9 (64.3%)

Teacher’s feedback on provided to students 3 (21.4%) 6 (42.9%) 5 (35.7%)

Sample Characteristics n of studies (%)


Sample described demographically 6 (42.9%) 8 (57.1%) —

Gender-diverse 4 (28.6%) 1 (7.1%) 9 (64.3%)

Racially/Ethnically diverse 3 (21.4%) 1 (7.1%) 10 (71.4%)

Prior keyboarding skill 7 (50%) 1 (7.1%) 6 (42.9%)

Single Multiple No information

Sampling – school-level 11 (78.6%) 3 (21.4%) —

Sampling – classroom-level 6 (42.9%) 8 (57.1%) —

Less than six weeks

Between six weeks and one semester

One semester or longer

Length of study 6 (42.9%) 6 (42.9%) 2 (14.3%)

Elementary Middle High

Grade level 7 (50%) 3 (21.4%) 4 (28.6%)

High Average Low MixedNo Infor-mation

Student sample ability level 3 (21.4%) 2 (14.3%) 1 (7.1%) 4 (28.6%) 4 (28.6%)

Rural UrbanSub-urban Mixed

No Infor-mation

School setting 2 (14.3%) 3 (21.4%) 4 (28.6%) 2 (14.3%) 3 (21.4%)


Publication Bias: Funnel Plot

The funnel plot depicted in Figure B1 shows that nearly two-thirds of effect size findings are approximately .50 or greater. Smaller-sized studies demonstrated a wide range of effect sizes, from virtually no effect at all through upwards of 1.2 units, as do the five largest studies (those with sample sizes greater than 50; ranging from -.05 to .87). Striking from the funnel plot, however, is the dearth of studies that employed a sample size greater than 50; exactly half of the studies had sample sizes that were 30 or fewer.

Figure B1: Funnel Plot for Quantitative Meta-Analysis

Actual Sample Size

80706050403020100

1.4

1.2

1.0

.8

.6

.4

.2

0.0

-.2

Ad

just

ed E

ffec

t Si

ze

Weighted Effect Sizes and Homogeneity Analysis

The overall effect of computers, as compared with paper-and-pencil, on quan-tity of student writing, based on twelve independent effect sizes, extracted from twelve studies, resulted in a mean effect size of .501. The weighted mean effect size, d

+ = 4.5226, with a 95% confidence interval ranging from .1.8187 through 7.2265.

Individual weighted effect sizes ranged from –1.62 through 11.97. The homogeneity analysis resulted in Q

t = 4120.6571, df = 13, p < .0001. This significant Q

t statistic indi-

cates that the fourteen effect sizes comprising this analysis do not come from the same population and that there may be moderating variables that impact the magnitude and/or direction of the effect sizes


In order to identify which, if any, of the coded study features have a significant moderating effect on the relationship between computers and quantity of writing, regression analyses were performed.

Weighted Effect Sizes and Regression Analysis

To examine the extent to which effect sizes were moderated by various study fea-tures, a mixed model approach was employed. This approach assumes that some of the variance in the effect sizes is systematic, and thus can be modeled, while another portion of the variance in the effect sizes is random and, therefore, cannot be modeled (for a full discussion of mixed vs. fixed effects modeling in the context of meta-analy-ses, see Hedges & Olkin, 1985).

The first step in the analysis was to dummy code all of the categorical variables. Dichotomous variables were left as is, variables with three or more levels were trans-formed into a series of k-1 (where k is the number of levels in the original variable) dichotomous variables. In the process of creating these variables, categories within the “school level” and “student ability” variable were collapsed. The dummy variable for “school level” was created to compare studies conducted in elementary schools with those conducted in middle and high school combined. Student ability was trans-formed to contrast the “average” and “mixed” groups with the remaining groups (low, high, no information provided).

To identify those variables with sufficient variance between levels required by the matrix algebra used in regression analysis, frequencies of each dummy variable were examined. In general, if each level of a given variable had a frequency of three or more, then it could be successfully entered in the regression analysis.

Of the coded study features, eight variables met the criterion for sufficient vari-ance. These eight variables were grouped into two themes:

• “student support,” which included: keyboard training, technical assistance, teacher feedback, and peer editing

• “student sample characteristics” which included: keyboard experience prior to study, student ability, school setting, and school level.

Additionally, a regression analysis that focused on the “study’s methodological quality” was conducted. For this analysis, methodological quality was calculated by summing assigned points across the 12 variables related to study quality, for a total possible score of 16. As presented below, the aggregate methodological quality rating was not a significant predictor. To explore whether individual aspects of study quality moderated the reported effects, the following study features were entered separately into a regression model: type of publication, presence of control group, presence of pre-post design, length of study, multiple vs. single classrooms and multiple vs. single schools. Additionally, publication year was dummy coded and included along with these methodological variables. To dummy code publication year, studies were divided into two groups: those published between 1992–1995, inclusive, and those published after 1995.


Study Features with Moderating Effects on Quantity of Writing

Regression Model: Student Support in Writing

Table B2 shows the results of the “student support” multiple regression model. In total, the four predictors, entered in a single block, accounted for 6% of the vari-ance. None of the individual predictors were significant. These results suggest that the various types of support provided to students during the course of each study did not systematically affect the amount of writing students produced.

Table B2: Regression Analysis of Student Support Variables on Weighted Effect Sizes of Quantity of Writing

B SE -95% CI +95% CI Z P Beta

Constant 3.7541 2.4246 -.9980 8.5062 1.5484 .1215 .0000

Keyboard training 1.9465 3.5734 -5.0574 8.9503 .5447 .5860 .2133

Teacher feedback -.6918 6.3335 -13.1053 11.7218 -.1092 .9130 -.0629

Peer editing -.5389 5.2453 -10.8198 9.7419 -.1027 .9182 -.0490

Technical assistance .9379 5.0348 -8.9302 10.8061 .1863 .8522 .0853

QM

= .4615, p = .9771

Mean ES R-Square N

4.5250 .0598 14.0000

Regression Model: Student Sample Characteristics

Table B3 presents the results of the “student sample characteristics” multiple regression model. Although the five variables collectively accounted for over 52% of the variance, no variables were found to be significant predictors. One variable, school level, approached significance. As the variable was dummy coded, the large, positive Beta indicates that studies employing middle and high school student samples tended to demonstrate greater effect sizes than did those studies employing elementary school samples. These results suggest that the characteristics of students participating in each study was not systematically related to the amount of writing students produced.

Table B3: Regression Analysis of Student Sample Characteristics Variables on Weighted Effect Sizes of Quantity Of Writing


Constant 1.0722 3.3559 -5.5055 7.6498 .3195 .7494 .0000

Prior keyboarding skills

-.1672 3.5419 -7.1094 6.7750 -.0472 .9623 -.0184

Geographic setting -1.1249 3.7626 -8.4996 6.2499 -.2990 .7650 -.1235

School level 6.9368 3.7644 -.4414 14.3150 1.8427 .0654 .7616

QM

= 3.5892, p = .3094

Mean ES R-Square N

4.3386 .5264 9.0000


Regression Model: Study Methodology

Table B4 presents the results of the “study methodology” multiple regression model. Although the five variables collectively accounted for 33% of the variance, none of the predictors were statistically significant. These results suggest that the vari-ous features of the studies were not systematically related to the amount of writing students produced.

Table B4: Regression Analysis of Study Methodology Variables on Weighted Effect Sizes of Quantity of Writing


Constant 2.2700 4.1183 -5.8018 10.3418 .5512 .5815 .0000

Publication year .8193 3.7754 -6.5804 8.2190 .2170 .8282 .0898

Type of publication -3.1169 4.3803 -11.7023 5.4686 -.7116 .4767 -.3307

Control group design 3.2659 4.7969 -6.1360 12.6678 .6808 .4960 .3617

Pre-post design 2.0379 5.2493 -8.2506 12.3265 .3882 .6978 .2234

Length of study 1.9022 3.9179 -5.7769 9.5813 .4855 .6273 .2086

School level -.5315 5.3005 -10.9205 9.8574 -.1003 .9201 -.0483

Single or multiple classes

1.2208 4.2170 -7.0445 9.4861 .2895 .7722 .1338

QM

= 18.9594, p < .0003

Mean ES R-Square N

4.5248 .3294 14.0000

Sensitivity Analysis

When heterogeneity among effect sizes are found in a meta-analysis, the “robust-ness” of the main findings can be examined through sensitivity analyses (Lipsey & Wilson, 2001). The sensitivity analysis explores ways in which the main findings are either consistent or inconsistent in response to varying the ways in which the data have been aggregated or included in the overall meta-analysis. For example, to provide a sense of how sensitive the main findings are across subgroups (say of school level), sensitivity analyses focus on a particular level of a variable.

A key variable of interest in this analysis is length of study. It can be reasonably argued that in studies of short duration (i.e., six weeks or less) measuring the impact of using computers on students’ writing is different than measuring computers’ impact on writing over a longer period of time. Studies conducted under longer time periods can result in students who are more adept at keyboarding, are more comfortable with features of word processing programs, and have sufficient time to adapt their writing strategies to exploit features of word processors.

Considering this, a sensitivity analysis was conducted which focused only on those studies for which the length of intervention was greater than six weeks. This selection procedure eliminated six of the fourteen studies from the analysis.


Sensitivity Analysis: Longer Study Duration and Student Support Multiple Regression Model

Due to insufficient variance on the other student support variables, the sen-sitivity analyses focused on peer editing and keyboard training, only. The resulting model (R 2 = .038) consisting of these two independent variables was statistically insig-nificant. These statistics indicate that there is no relationship between the weighted effect sizes of quantity of writing and these variables, regardless of length of study.

Sensitivity Analysis: Longer Study Duration and Student Sample Characteristics Multiple Regression Model

As shown in Table B5, the overall student sample characteristics multiple regres-sion model, (excluding student academic ability) was significant (Q

M = 18.9594,

p = .0003). One variable, school level, was significantly related to the weighted effect sizes. For those studies that lasted for more than six weeks, the significant, positive beta weight for school level indicates that there were larger effect sizes for quantity of writing that favored computers over paper and pencil for studies that occurred in middle and high school as opposed to elementary school.

Table B5: Sensitivity Analysis: Regression Analysis of Student Sample Characteristics Variables on Weighted Effect Sizes of Quantity of Writing for Studies Lasting More Than Six Weeks


Constant .1617 2.2657 -4.2791 4.6025 .0714 .9431 .0000

Prior keyboarding Experience

.8231 2.9326 -4.9248 6.5710 .2807 .7790 .0796

Geographic setting .5385 2.6193 -4.5954 5.6724 .2056 .8371 .0521

School level 9.3045 2.9334 3.5550 15.0540 3.1719 .0015 .9020

QM

=18.9594, p<.0003

Mean ES R-Square N

4.0151 .9167 6.0000

Sensitivity Analysis: Longer Study Duration and Study Methodology Multiple Regression Model

The overall “study methodology” model was significant, QM

= 33.1016, p < .0001. This model revealed that five study features had significant relationships with the quantity of student writing weighted effect sizes. Curiously, the large negative beta associated with “year of publication” ( p < .0004) indicates that the findings of those studies conducted between 1992 and 1995, inclusive, had effect sizes more strongly in favor of computers than did the later studies. Also, effect sizes were significantly larger in studies that were published in refereed journals ( p < .006), incorporated a control group ( p < .0005), sampled from more than one classroom ( p < .02) and, in those that did not employ a pre- post-test design ( p < .006).


Table B6: Sensitivity Analysis: Regression Analysis of Study Methodology Variables on Weighted Effect Sizes of Quantity of Writing for Studies Lasting More Than Six Weeks


Constant 1.5660 1.5189 -1.4110 4.5429 1.0310 .3025 .0000

Publication year -10.6699 2.9550 -16.4618 -4.8781 -3.6108 .0003 -1.1902

Type of publication 10.9856 3.9274 3.2879 18.6833 2.7972 .0052 1.2170

Control group design 14.0606 3.9426 6.3330 21.7882 3.5663 .0004 1.5576

Pre-post test design -24.2968 5.7386 -35.5444 -13.0492 -4.2339 .0000 -2.7913

School level -2.4444 2.9519 -8.2301 3.3414 -.8281 .4076 -.1873

Multiple v. single classes

6.9256 2.9727 1.0991 12.7522 2.3297 .0198 .7956

QM

=33.1016, p=.0001

Mean ES R-Square N

4.2706 .9707 8.0000

Computers and Writing: Quality

Fifteen independent effect sizes were extracted from fifteen studies comparing quality of writing between computer and paper-and-pencil groups. Two additional studies focused specifically on the quality of writing produced on computer, but did not include paper-and-pencil comparison groups. As a result, these two studies were not included in the meta analyses. Gallick (1997) reported a large positive, but statisti-cally insignificant effect size (d = 1.18, 95% CI range: –0.78 through 0.2, n = 8) in her single-group, pre-post test designed study, and Hood (1994), in a study with the same design, also reported a large positive, yet statistically insignificant effect size (d = 1.14, 95% CI range: -0.36 through 0.39, n = 14).

Another study, conducted by Snyder (1993), included paper-and-pencil com-parison groups, but did not provide enough statistical data for inclusion in the data analyses. This study reported no mean differences between computerized and paper and pencil groups (n = 51), but variance estimates were not provided and could not be calculated based on the reported statistics.

Below we present descriptive highlights of the fourteen studies followed by an analyses of effect sizes and regression analyses that explore moderating variables.

Descriptive Highlights

As detailed in Table B7, 60% (n = 9) of the included studies that focused on the quality of writing were published in refereed journals. Sixty percent of the studies also employed samples drawn from multiple classrooms, 20% (n = 3) employed random assignment, and for 60% (n = 9) the research duration lasted between six weeks and one semester. Thirteen of the fifteen studies (87%) utilized standardized writing tasks across groups, and in 40% (n = 6) of the studies, students were provided with key-boarding training. Individual writing (as opposed to collaborative writing) was the focus in all fifteen studies. Peer editing was a component in three (20%) of the stud-ies, teacher feedback on writing was present in four of the studies (27%), and techni-cal assistance was available to students in 27% (n = 4) of the studies. It was unclear


whether or not teacher feedback and/or technical assistance were study features in n = 4 and n = 9 studies, respectively.

Table B7: Characteristics of Studies Included in Quality of Writing Meta-analysis

Study Characteristics n of studies (%)

Refereed journal article

Doctoral dissertation

Master’s thesis

Publication type 9 (60.0%) 3 (20.0%) 3(20.0%)


Random assignment 3 (20.0%) 12 (80.0%) —

Pre-Post design 9 (60.0%) 6 (40.0%) —

Standardized writing sample 13 (86.7%) 2 (13.3%) —

Keyboarding training included in study 6 (40.0%) 7 (46.7%) 2 (13.3%)

Peer-editing 3 (20.0%) 12 (80.0%) —

Handwritten samples converted to WP format

3 (20.0%) 12 (80.0%) —

Technical assistance provided to students 5 (33.3%) 1 (6.7%) –

Teacher’s feedback on provided to students 4 (26.7%) 7 (46.7%) 9 (60.0%)

Sample Characteristics n of studies (%)


Sample described demographically 6 (42.9%) 8 (57.1%) —

Gender-diverse 5 (33.3%) 1 (6.7%) 9 (60.0%)

Racially/Ethnically diverse 3 (20.0%) 3 (20.0%) 9 (60.0%)

Prior keyboarding skill 7 (46.7%) 2 (13.3%) 6 (40.0%)

Single Multiple No information

Sampling – school-level 14 (83.3%) 1 (6.7%) —

Sampling – classroom-level 6 (40.0%) 8 (60.0%) —

Less than six weeks

Between six weeks and one semester

One semester or longer

Length of study 6 (40.0%) 8 (53.3%) 1 (6.7%)

Elementary Middle High

Grade level 7 (46.7%) 5 (33.3%) 3 (20.0%)

High Average Low MixedNo Infor-mation

Student sample ability level 4 (28.6%) 2 (13.3%) 1 (6.7%) 3 (20.0%) 5 (33.3%)

Rural UrbanSub-urban Mixed

No Infor-mation

School setting 2 (13.3%) 3 (20.0%) 6 (40.0%) 1 (6.7%) 3 (20.0%)


In terms of student demographics, only five studies (33%) had samples that were documented as gender-diverse, three studies (20%) reported racially/ethnically-diverse samples, while 60% (n = 9) of the studies did not describe the gender or race/ethnic characteristics of the sample. All but two studies (87%) focused on mainstream educa-tion samples. Forty-seven percent (n = 7) of the studies were conducted with elemen-tary school students, 33% (n = 5) were situated in middle schools, and the remaining 20% (n = 3) were conducted in high schools.

Geographically speaking, the studies were distributed across rural (n = 2), urban (n = 3), suburban (n = 6), and mixed (n = 1) settings; three studies lacked any geo-graphic description.

In short, the demographic descriptions of the studies included in this meta-analy-sis did not appear to differ considerably from those studies included in the ‘quantity of writing’ meta-analysis.

Publication Bias: Funnel Plot

The funnel plot depicted in Figure B2 shows that studies reporting positive effect sizes are distributed across small-to mid range sample sizes. To a lesser extent, the same is observed for negative and near-zero effect sizes. The variability of effect size in rela-tion to sample size appears more balanced as compared with that seen in the funnel plot for “writing quantity”.

Figure B2: Funnel Plot of Quality of Writing Meta-analysis

Actual Sample Size

160140120100806040200

1.5

1.0

.5

0.0

-.5

-1.0

Ad

just

ed E

ffec

t Si

ze


Weighted Effect Sizes and Homogeneity Analysis

The overall effect of computers, as compared with paper-and-pencil, on quality of student writing, based on fifteen independent effect sizes, extracted from twelve studies, resulted in a mean effect size of .501. Inverse variance weighting of each effect size resulted in a weighted mean effect size, d

+ = 6.030, with a 95% confidence

interval that ranged from .1369 through .7024. The homogeneity analysis resulted in Q

t = 24396.996, df = 14, p < .0001. This significant Q

t statistic indicates that the fifteen

effect sizes included in this analysis do not come from the same population.

To investigate possible moderating variables, regression analyses on the weighted effect sizes and study features followed.

Weighted Effect Sizes and Regression Analysis

As with the meta-analysis for quantity of writing, a mixed effects model was used to explore the extent to which study features moderated the effect of computers on the quality of student writing. For these analyses, study features were dummy coded and examined to assure that sufficient variance existed.

The coded study features that were appropriate for regression analyses were grouped in the following way:

• “student support” which included keyboard training in the study, peer editing, teacher feedback provided, and technical assistance provided

• “student sample characteristics” which included prior keyboarding ability, academic ability, school level, and geographic location

• “study methodology” which included type of publication, random assign-ment, pre-post test design, multiple classes, time of intervention, conver-sion of handwritten writing samples to computer, and year of study publi-cation.

The regression analysis employing the “study’s methodological quality” variable was not a significant predictor of effect size, and was again disaggregated.

Study Features With Moderating Effects on Quality of Writing

Regression Model: Student Support in Writing

Table B8 shows the results of the “student support” multiple regression model. In total, the four predictors entered as a single block accounted for nearly 16% of the variance. Neither the model as a whole or any of the variables were significant. This finding is consistent with the meta-analysis on “quantity of writing.”


Table B8: Regression Analysis of Student Support Variables on Weighted Effect Sizes of Quality of Writing


Constant 4.0030 4.6310 -5.0738 13.0797 .8644 .3874 .0000

Keyboard training 6.2855 7.1000 -7.6304 20.2014 .8853 .3760 .3680

Peer editing 14.4270 14.0833 -13.1762 42.0303 1.0244 .3056 .6896

Teacher feedback 4.7685 8.7855 -12.4512 21.9881 .5428 .5873 .2520

Technical assistance -13.9250 12.8080 -39.0287 11.1786 -1.0872 .2769 -.7845

QM

= 1.4139, p = .8418

Mean ES R-Square N

6.0322 .1559 15.0000

Regression Model: Student Sample Characteristics

Table B9 shows the results of the “student sample characteristics” multiple regres-sion model. Together, the five variables account for 46% of the variance in the ‘quality of writing’ weighted effect sizes, but the model as a whole is statistically insignificant. However, the beta weight for school level was a statistically significant predictor of “quality of writing,” ( p < .03). Greater effect sizes in favor of computer vs. paper-and-pencil writing were found among the studies that took place in middle and high schools as compared with those studies conducted in elementary schools. This is con-sistent with the findings in the sensitivity analysis for “quantity of writing.”

Table B9: Regression Analysis of Student Sample Characteristics Variables on Weighted Effect Sizes of Quality of Writing


Constant 1.4858 5.1358 -8.5804 11.5520 .2893 .7723 .0000

Prior keyboarding -10.0868 7.3984 -24.5877 4.4142 -1.3634 .1728 -.5959

School level 14.1769 6.5765 1.2869 27.0669 2.1557 .0311 .8375

Geographic setting 7.1541 6.5776 -5.7380 20.0461 1.0876 .2768 .4227

QM

= 4.7012, p = .1950

Mean ES R-Square N

8.5397 .4635 11.0000


Regression Model: Study Methodology

Table B10 presents the result of the “study methodology” regression model. Although the model accounts for 43% of the variance, both the model and beta weights for each individual variable were not statistically significant. These results are consistent with the findings of the “quantity of writing” analysis.

Table B10: Regression Analysis of Study Methodology Variables on Weighted Effect Sizes of Quality of Writing


Constant .7500 13.5884 -25.8833 27.3833 .0552 .9560 .0000

Type of publication -4.5507 9.2025 -22.5876 13.4862 -.4945 .6209 -.2664

Random assignment 9.4191 8.3572 -6.9610 25.7991 1.1271 .2597 .4502

Pre-Post design 3.7035 9.1364 -14.2040 21.6109 .4054 .6852 .2168

Single vs. multiple classroom

4.6590 6.5263 -8.1325 17.4505 .7139 .4753 .2727

Length of study 3.7963 7.5144 -10.9319 18.5244 .5052 .6134 .2222

Handwriting converted to WP format

2.4445 8.5858 -14.3837 19.2727 .2847 .7759 .1169

Publication year -1.4965 7.1140 -15.4398 12.4469 -.2104 .8334 -.0876

QM

= 4.6671, p = .7005

Mean ES R-Square N

6.0328 .4325 15.0000

Sensitivity Analysis

The sensitivity analysis focused on those studies for which the length of inter-vention was greater than six weeks. This selection procedure eliminated four of the thirteen studies from the analysis.

Sensitivity Analyses: Longer Study Duration and Student Support, Student Sample Characteristics, and Study Methodology Multiple Regression Models

The multiple regression models were run on the nine studies that had interven-tions spanning more than six weeks. The peer editing and handwriting conversion variables were excluded due to insufficient variance. The student support multiple regression yielded an insignificant model, Q

M = 1.342, df = 3, p = .72. The student

sample characteristics and study methodology multiple regression models were also found to be statistically insignificant (Q

M = 1.963, df = 3, p = .58, and (Q

M = 4.471,

df = 5, p = .484, respectively). This lack of predictive power indicates that the effect sizes included in this study are not significantly moderated by these variables, regard-less of study duration time.


Appendix C

Article Collected Determination Regarding Inclusion

Albertson, L. R., & Billingsley, F. F. (1997) Focus on reviewing skills

Allen, G., & Thompson, A. (1994) Focus on effect of audience on quality

Allison, B. (1999) WP small part larger intervention, insufficient quantitative data reported

Baker E. & Kinzer, C.K. (1998) Qualitative

Barrera, M. T., et al. (2001) Included in meta-analysis

Biesenbach-Lucas, S. & Weasenforth, D. (2001) ESL focus

Bogard, E. A. (1998) Design and outcomes not aligned

Borthwick, A. G. (1993) Review of research

Bowman, M. (1999) Verbal/collaborative/Vygotsky focus

Bracey, G. (1992) Not an empirical study

Breese, C., et al. (1996) Included in meta-analysis

Brigman, D. J. P. (1995) Included in meta-analysis

Bruce, B. & Peyton, J. K. (1992) WP small part of larger intervention, case study

Bucci, S. M. (1996) Not paper vs. computer design: outcome measure was revision of paper written draft on WP after conference

Burley, H. (1994) WP small part of larger intervention, outcomes not aligned

Butler, S. & Cox, B. (1992) Qualitative

Casey, J. M. (1992) WP part of larger intervention, insufficient quantitative data reported

Creskey, M. N. (1992) Quantitative data not reported

DeFoe, M. C. (2000) WP part of larger intervention, insufficient quantitative data reported

D’Odorico, L., & Zammuner, V. L. (1993) Included in meta-analysis

Dodson, L. E. (2000) WP part of larger intervention, outcomes not aligned

Dybdahl, C. S., et al. (1997) Included in meta-analysis

Ediger, M. (1996) Lack of quantitative data

Escobedo, T. H. & Allen, M. (1996) Emergent writing focus

Fan, H.L. & Orey, M. (2001) Multimedia focus

Fletcher, D. C. (2001) Case study on editing process

Freitas, C. V. & Ramos, A. (1998) Qualitative/Insufficient Quantitative data reported

Gaddis, B., et al. (2000) Focus on effect of collaboration and audience needs

Gallick-Jackson, S. A. (1997) WP small part of larger intervention, outcomes not same focus

Godsey, S. B. (2000) Included in meta-analysis

Greenleaf, C. (1994) No paper/pencil group

Grejda, G. F. & Hannafin, M. J. (1992) Included in meta-analysis

Hagler, W. J. (1993) Included in meta-analysis

Hartley, J. (1993) Not empirical study

Head, B. B. (2000) Included in meta-analysis

Hood, L. M. (1994) Insufficient quantitative data, computer group only


Hydrick, C. J. (1993) Case study, no quantitative data on outcome measures

Jackiewicz, G. (1995) Included in meta-analysis

Jackowski-Bartol, T. R. (2001) Qualitative

Jankowski, L. (1998) Not empirical study

Jones, I. (1994a) Same data already included in meta-analysis

Jones, I. (1994b) Included in meta-analysis

Jones, I & Pellegrini, A.D. (1996) Included in meta-analysis

Joram, E., & et al. (1992) Included in meta-analysis

Keetley, E. D. (1995) Included in meta-analysis

Kehagia, O. & Cox, M. (1997) Design and outcomes not aligned

Kumpulainen, K. (1994) Social focus of collaborative writing

Kumpulainen, K. (1996) Social focus of collaborative writing

Lam, F. S. & Pennington, M. C. (1995) Included in meta-analysis

Langone, J., et al. (1996) Included in meta-analysis

Lerew, E. L. (1997) Included in meta-analysis

Lewis, P. (1997) Software feature focus

Lewis, R. B. (1998) Software feature focus

Lewis, R. B., et al. (1999) Software feature focus

Lichtenstein, N. (1996) Included in meta analysis

Lohr, L., et al. (1996) Multimedia focus

Lomangino, A. G., et al. (1999) Social focus of collaborative writing

Lowther, D. L., et al. (2001) No data collected on writing quality/quantity

Lund, D. M. & Hildreth, D. (1997) Case study/Multimedia

MacArthur, C., et al. (1994) Not an empirical study

MacArthur, C. A. (1996) Not an empirical study

McBee, D. (1994) Emergent writing focus

McMillan, K. & Honey, M. (1993) Insufficient quantitative data reported

Mehdi, S. N. (1994) Design and outcomes not aligned

Moeller, B., et al. (1993) Email focus

Mott, M.S. & Halpin, R. (1999) Multimedia and writing focus

Mott, M. S., et al. (1997) Not an empirical study

Mott, M. S. & Klomes, J. M. (2001) Multimedia writing focus

Moxley, R. A., et al. (1994) Emergent writing focus

Moxley, R. A., et al. (1997) Emergent writing focus

Nichols, L. M. (1996) Included in meta-analysis

Olson, K. A. (1994) Included in meta-analysis

Osborne, P. (1999) Impact of WP measured by coursework/exams

Owston, R. D., et al. (1992) Included in meta-analysis

Owston, R. D. & Wideman, H. H. (1997) No paper/pencil group, focus low vs. high computer access settings

Padgett, A. L. (2000) Included in meta-analysis

Peacock, M. & Beard, R. (1997) Described subgroup of 1980–1992 studies


Pennington, M. C. (1993) Included in meta-analysis

Peterson, S. E. (1993) Included in meta-analysis

Philips, D. (1995) Data collected were prior to decade of interest

Pisapia, J. R., et al. (1999) Outcome measures: test scores

Pohl, V. & Groome, D. (1994) Not empirical study

Priest, N. B. (1995) Outcome measures: WP small part of larger intervention; goal attainment

Reed, W. M. (1996) Review of research

Robertson, S. I., et al. (1996) Outcome measures: technology knowledge and use

Roussey, J. Y., et al. (1992) Individual vs. dyad/Vygotsky focus

Roblyer, M. D. (1997) Not an empirical study

Seawel, L., et al. (1994) Included in meta-analysis

Shaw, E. L., et al. (1994) Insufficient Quantitative data reported

Snyder, I. (1993a) Included in meta-analysis

Snyder, I. (1993b) Not empirical study

Snyder, I. (1994) Same data collection as that included in meta-analysis, qualitative view

Waldman, H. (1995) Multimedia focus

Walker, C. L. (1997) Revision focus

Wolfe, E. W., et al. (1996) Included in meta-analysis

Yackanicz, L. (2000) Qualitative

Zammuner, V. L. (1995) Research focus: individual vs. collaborative writing

Zhang, Y., et al. (1995) Hypercard focus

Zoni, S. J. (1992) WP small part of larger intervention, insufficient quantitative data reported

Meta-Analysis: Writing with Computers 1992–2002

Documents