Csuf pilot to improve writing

CSUF Pilot to Improve Student Writing with

ETS Criterion®, Spring 2010

As compiled by Institutional Research & Assessment and Planning, CSALT, Jennifer Ivie, Henry Delcore,

and ISWI Assessment Team

Sponsored by CSUF ISWI Pilot Program and CSALT Presentation Developed by Professor Kim Morin, December 13,2010

1Copyright © December 2010

Fresno State Student, Stephanie Canales, Discusses ETS Criterion

• http://www.youtube.com/watch?v=pP6feJotLVM

Copyright © December 2010 2

http://www.youtube.com/watch?v=pP6feJotLVM

http://www.youtube.com/watch?v=pP6feJotLVM

How This All Began

• I first sought a computer essay scoring program out of desperation!


I Believed In Giving Writing Assignments To Students…

“Writing helps with - • Making personal

connections• Problem solving• Reasoning• Communication

skills.”


But Then I Kept Asking…

“Where Is the Time to

Grade All of Those

Essays?”


So I Discovered…

Computer Essay Scoring ProgramsI narrowed my search to three of the biggest programs available at the college level… and tried out ETS Criterion®!


How It Works

• Students submit essays electronically• Computer program scores the students’

work based on sample essays previously scored by human raters

• “e-rater” scoring system (analysis of more general writing skills)• “Hybrid Feature Technology” combines syntactic variety and

discourse with analysis of semantic structure


Why I Chose To Pilot ETS Criterion

• Combines Diagnostic with Holistic feedback • Flexibility

– Instructor can create a topic– Instructor can easily provide personalized feedback

• Simple Registration Process – Students are able to purchase Access cards through Bookstore

• No need for preliminary essay submissions• Reasonable Cost

– $13.00 per student per semester– Same fee regardless of number of classes


Positive Aspects

• Easy to use

• Immediate feedback

• Consistency in scoring

• Errors are highlighted but not corrected

• First and last submission are available for comparison


Positive Aspects

• Students often do a number of re-writes trying to improve their score

• Students tend to get frustrated with the computer - not the instructor

• Students discover that good writing means more than correcting grammatical errors!


Take A Tour

ETS CRITERION Online Tour

http://www.ets.org/Media/Products/Criterion/tour2/critloader.html


http://www.ets.org/Media/Products/Criterion/tour2/critloader.html

Campus-Wide InitiativeBy The Numbers...

Over the course of 2 semestersSpring 2010• 134 Instructors used Criterion® • 197 Classes • 5,073 students Fall 2010• 68 Instructors used Criterion® • 173 Classes • 3,756 students


CSUF DemographicsDiverse student population with a significant number of underprepared and first generation students, including many involved in the agricultural and farming economies in the region.


Assessments for Spring 2010..

– Analyzed trends from Automatically Generated Data within Criterion®

– Collected Student and Faculty Surveys (Analysis by Jennifer Ivie)

– Faculty Learning Community Focus Groups provided qualitative feedback


Assessments for Spring 2010..

4. College Learning Assessment (CLA) comparison for students with & without Criterion® (April, 2010)

5. Faculty Scoring Panel compared 1st

and last papers submitted by students on Criterion®


Assessments for Spring 2011..1. Studies with controlled variables –

same course, same instructor, randomly selected group uses Criterion®, control group does not

2. Continue collecting back end data from Criterion®

3. Faculty and student surveys, focus groups

4. CLA5. Other measures?


Automatically-Generated Data• Students who made two submissions to ETS

Criterion had higher average holistic scores than students who only made one submission.

• Students with lower initial holistic scores were able to raise their scores, with subsequent submissions, to the same level as those whose initial holistic scores were higher.


Collegiate Learning Assessment– A significant difference was found

between Criterion users and Criterion non-users on the CLA performance task.

– No significant differences were found between the two groups on any of the analytic writing task scores.


Faculty Scoring Panel• Overall, student writing improved in classes

that used ETS Criterion.

• The greatest improvements from first to last submission were made in classes in which the instructors engaged students in substantial discussions of writing and how to use Criterion.


Student Survey HighlightsStudents felt that Criterion

– …helped with writing mechanics.– ESL students were more likely than English

as a First Language students to respond that Criterion® helped them improve their writing.

– ESL students often preferred Criterion feedback to instructor feedback.


ESL Student ResponseESL students submitted their documents more times than non-ESL students, and were more likely to agree with these statements: (Pearson r’s, all at p<.05 or less):

- I like the kind of feedback Criterion gives. - I spent more time improving my writing with Criterion. - I was able to apply Criterion’s feedback to other writing assignments. - I wish I could use Criterion for other classes.- I feel like the feedback from Criterion has made me a better writer.


Overall Student Response - Positive

What most students liked about Criterion®: • Available on-line 24 hours a day • It checked grammar and spelling • It allowed them to correct mechanical and grammatical errors • It gave immediate feedback. • The website was easy to navigate. • The grading was fair


Student Comments• I like the instant feedback. I also like that it goes

into depth about why you didn't get the max score or why you did well. It is also easy to use just copy and paste!

• I love the fact that I can revise my work for a better score. It is very helpful and I can see my growth as a writer even from this second essay.

• When I submit my paper, I look forward to seeing my score. Usually I score a 5 but I push myself to go and correct my errors until I score 6. I really like this program...

• I like the instant feedback. I also like that it goes into depth about why you didn't get the max score or why you did well. It is also easy to use just copy and paste!

• I love the fact that I can revise my work for a better score. It is very helpful and I can see my growth as a writer even from this second essay.

• When I submit my paper, I look forward to seeing my score. Usually I score a 5 but I push myself to go and correct my errors until I score 6. I really like this program...


Student Sample

• First Essay - Went from Level 4 - 6

• 22 Attempts

Student Comments- Summaries of Challenges

•Some students felt –They were already strong writers, so they felt like they did not need the program.–The comments should be more specific–Frustrated that Criterion identified technical terms or citations as errors.


Overall Faculty Response (n=66 out of 124)

•Creating assignments in Criterion® was easy.•Students found Criterion® useful.•Criterion® helped their students become better writers.•Grading was easier after papers had been submitted to Criterion®.•Students spent more time improving their writing assignments when using Criterion®.


Faculty Response

However, many faculty also felt that:Criterion® didn’t do as much as they hoped it would.They would be able to use the program more effectively with more experience and training

In fact, over 100 faculty members continued with the pilot program for Round 2 in Fall 2010.


Effective Uses of Criterion®

Data from Spring 2010 indicates the following practices as more effective:1. Multiple assignments 2. Multiple revisions over time3. Students correct “fixable errors” as identified by

Criterion®4. Early orientation for students so they practice

using the program and gain a better understanding of how it works

5. Students are given ample time between first and final submissions to make revisions


Effective Uses of Criterion® (cont) • Instructor uses the tools in Criterion® to give

additional feedback. • Students are motivated to use the Criterion®

program as part of the revision process• Criterion® is used as an editing tool to help students

think critically about their writing• Students and faculty understand that the computer

program will make mistakes; it is one part of the editing process

• Scoring is set at the appropriate level (College- 1st year or College 2nd year)


Less Effective Uses of Criterion®

oA single paper at the end of the semester

oSingle submissions – no revision o Instructor gives little or no training or

feedback regarding Criterion® oTechnical writing, business formats,

field specific terminologyoUsing ETS Criterion as "Extra Credit"


ConclusionsWe are still evaluating the effectiveness of Criterion®, but from what we’ve seen so far, under certain circumstances it appears to be useful in improving student writing. The program appears to be most effective with multiple assignments that require multiple submissions over time. Faculty need to be made aware of key features of Criterion® and what they do, then carefully select which features to use. To use the program effectively, students must understand how to access and interpret feedback.


Fresno State Student, Katy Michelle Lewis,

Discusses ETS Criterion

• http://www.youtube.com/watch?v=GTQLF7AujSQ


http://www.youtube.com/watch?v=GTQLF7AujSQ

http://www.youtube.com/watch?v=GTQLF7AujSQ

Fresno State Student, Nick Mullins, Discusses ETS Criterion

• http://www.youtube.com/watch?v=TRbYAVqC6tg


http://www.youtube.com/watch?v=TRbYAVqC6tg

http://www.youtube.com/watch?v=TRbYAVqC6tg

ISWI Is Broadly Endorsed & Supported By:

Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)Professional Development Subcommittee (subcommittee of Senate’s Personnel Committee)Chair of the General Education CommitteeOffice of Undergraduate StudiesDivision of Graduate StudiesInstitutional Research & Assessment Planning Director leading faculty and ISWI Assessment TeamProvost’s Office as implemented by Associate Provost Ellen Junn through the Center for the Scholarly Advancement of Learning and Teaching (CSALT)


Acknowledgements

Bill Covino, ProvostEllen Junn, Associate ProvostChristina Leimer, Director, IRAP ISWI Assessment Team:Kim Morin, Faculty, Theatre ArtsJohn Beynon, Faculty, English Henry Delcore, Faculty, AnthropologyBradley Hufft, Faculty, MusicJennifer Ivie, Faculty, PsychologyEileen Walsh, Faculty, HistoryAsao Inoue, Faculty, EnglishJohn Farrell, Faculty, HistoryDimitri Rogulkin, IRAP, StaffDiane Robinson, ISWI, CSALT, Staff


Csuf pilot to improve writing

Documents

students copyright

criterion copyright

instructor students

criterion faculty

generation students

language students

score students

criterion users