Testing Writing for the E8 Standards

Testing Writing for the E8 Standards

Technical Report 2011

Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller

Testing Writing for the E8 Standards

Technical Report 2011

Otmar GassnerClaudia MewaldRainer BrockFiona LackenbauerKlaus Siller

Bundesinstitut für Bildungsforschung, Innovation & Entwicklung des österreichischen Schulwesens Alpenstraße 121 / 5020 Salzburg

www.bifie.at

Testing Writing for the E8 Standards. Technical Report 2011.BIFIE Salzburg (Hrsg.), Salzburg 2011

Der Text sowie die Aufgabenbeispiele können für Zwecke des Unterrichts in österreichischen Schulen sowie von den Pädagogischen Hochschulen und Universitäten im Bereich der Lehrer aus-, Lehrerfort- und Lehrerweiterbildung in dem für die jeweilige Lehrveranstaltung erforderlichen Umfang von der Homepage (www.bifie.at) heruntergeladen, kopiert und ver-breitet werden. Ebenso ist die Vervielfältigung der Texte und Aufgabenbeispiele auf einem anderen Träger als Papier (z. B. im Rahmen von Power-Point Präsentationen) für Zwecke des Unterrichts gestattet.

Autorinnen und Autoren:

Otmar Gassner Claudia Mewald Rainer Brock Fiona Lackenbauer Klaus Siller

Contents

3 EmbeddingtheE8WritingTestinaNationalandInternationalContext

3 ThePlaceofWritinginAustrianLowerSecondarySchools

4 ValidityAspectswithregardtotheE8WritingTestConstruct

5 TestTakerCharacteristics6 CognitiveValidity6 WritingTheoryinBrief7 CognitiveProcessingintheE8WritingTest9 ContextValidity9 Setting:Task10 Setting:AdministrationofE8WritingTests11 LinguisticDemands:TaskInputandOutput12 ScoringValidity13 CriteriaandRatingScale15 RaterCharacteristics15 RatingProcess15 RatingConditions16 RaterTraining18 PostExamAdjustments18 ReportingResults19 ConsequentialValidity

20 E8WritingTestSpecificationsVersion03(July2011)

20 1.PurposeoftheTest20 2.DescriptionofTestTakers20 3.TestLevel20 4.TestConstructwithE8ConstructSpace23 5.StructureoftheTest23 6.TimeAllocation23 7.ItemFormats23 8.LanguageLevelforInstructionsandPrompts23 9.AssessmentwithWritingRatingScale28 10.PromptsandPerformanceSampleswithJustifications

37 ScaleInterpretations

37 ScaleInterpretation–TaskAchievement39 ScaleInterpretation–CoherenceandCohesion41 ScaleInterpretation–Grammar43 ScaleInterpretation–Vocabulary

45 Literature

48 Appendix

48 PromptInterpretation:LongPrompt50 PromptInterpretation:ShortPrompt53 InventoryofFunctions,NotionsandCommunicativeTasks

3Testing Writing for the E 8 Standards

EmbeddingtheE8WritingTestinaNationalandInternationalContext

ThePlaceofWritinginAustrianLowerSecondarySchools

Thereseemstobesomeagreementthatspeakingandlisteningaretheskillsmostneededwhentryingtosucceedinaforeignlanguageenvironmentandthatbeingable to read is next in priority. This leaves writing as the skill least necessary forsurvival.Nevertheless,writingistrainedfromyearoneofsecondaryeducationonaregularbasis.Insomecoursebooksitstartsoffwithmodelparagraphsthatareper-sonalisedbythelearnersandleadsontoopenwriting,mostlybasedonthecontentofthecoursebookunitinprogress.Itgoeswithoutsayingthatlowerabilitylearnersaregivenmoreguidance,withsomeofthemhardlyeverattemptinganopenwritingtask.

Inpost-beginnerclassestheimportanceattributedtowritingincreases.Itseemstobeawide-spreadbeliefamongteachersofEnglishthatwhenwritingskillsareassessed,otherdimensionsoflanguagecompetencelikevocabularyandgrammarknowledgecanbeassessedautomaticallyatthesametime.Therefore,thewritinggradegoesalongwaytowardstheoverallEnglishgradeforthatparticularstudent.

Whereasthisbeliefmightberesponsibleforthehighregardteachershaveforwrit-ing, the awareness of the complexity of assessment procedures for writing is stilllimited.Thereisnoperceivedneedforsharedstandardlevels,thereisnoagreementonhowwritingshouldbetested,markedandweightedinrelationtotheotherskills(reading, listening, speaking),1 there are a great number of idiosyncratic markingschemesinplace(evenwithinoneschool),andthereisnoagreementonanythinglikepassmarksorcutscoresforgrading.

In this situation there is roomforconstructivewashback in thecourseof the in-troductionofE8Standards.ItishopedthatthewaythetestsareconstructedandassessedwillimpactonthewaywritingistaughtandassessedinAustrianschools.

AlthoughmuchofwhathasbeensaidabovewasformulatedforthefirsteditionofthisTechnicalReportin2008,itisstillrelevantandwecancertainlyseesignificantsignsofchange.Aprogrammetotrainfourhundredwritingratersisinplaceandspreads expertise across the country; test specifications and a number of pilotingphaseshave led tovisible adaptations in thecoursebooksused; 'train the trainerprogrammes'onhowtoassesswrittenperformancesfunctionasstartingpointsforschool-basedprofessionaldevelopment.Finally, thereorganisationofacentralisedapproachtotheassessmentofwrittenperformancesatE12level(Matura)hascon-tributedalottoraisingawarenessofthecomplexityofassessingwrittenscripts.

1 ThislackofagreementisnoticeabledespiteaclearstatementintheAustriancurriculumaboutallfourskillstobetaughtandtrainedequallyintheclassroom;unfortunatelythecurriculumdoesnotsayanythingonweight-ingintests.(seeLehrplanderHauptschule.2008,p.2)

4 Testing Writing for the E 8 Standards

ValidityAspectswithregardtotheE8WritingTestConstruct

Shaw&Weir (2007)havedesignedacleargraphic to illustrate their“frameworkforconceptualisingwritingtestperformance”(seefigure1below).It takesall therelevantparametersintoaccountandcanserveastheblueprintforthedescriptionoftheE8WritingTestsandthetheoreticalframeworkonwhichtheyarebased.Withinthisframeworkthefocusofthediscussionwillbeonthefollowingaspects:testtakercharacteristics,cognitivevalidity,contextvalidity,scoringvalidity,andconsequentialvalidity.

Figure 1: Adapted from Shaw & Weir 2007, 4

6

Test-taker Characteristics

Cognitive Validity

Context Validity

Setting: Task

Response format Purpose Knowledge of criteria Weighting Text length Time constraints Writer-reader relationship

Setting: Administration

Physical conditions Uniformity of administration Security

Linguistic demands: (Task input and output)

Lexical resources Structural resources Discourse mode Functional resources Content knowledge

Response

Scoring Validity

Score

Cognitive Processes

Macro-planning Organisation Micro-planning Translation Monitoring Revising

Consequential Validity

Physical/Physiological Psychological Experiential

Rating

Criteria/rating scale Rater characteristics Rating process Rating conditions Rater training Post-exam adjustment Grading and awarding

Washback on individuals in classrooms Impact on institutions and society Avoidance of test bias


TestTakerCharacteristics

It isobvious that test taker characteristicshave an influenceon theway a task isprocessedanda text iswritten.Three categorieshavebeen identifiedasphysical/physiological,psychologicalandexperientialcharacteristics(Shaw&Weir2007,5).

As regards the first category, any provisions made for schooling can be consideredsufficientfortheE8testsituationasalltesttakersarepupilsinthelastformoflowersecondaryschoolsinAustria.Toputitsimply,anypupilwhoisfitenoughtoattendEnglishclassesatanAustriansecondaryschoolandtobeassessedisfittotaketheE8WritingTest.

Psychological factors, however, are almost impossible to control. Most critical ismotivationasE8Standardsisalow-stakesexamthathasnoinfluencewhatsoeveron the individual test takers’marksoron their school career.Wecanexpect lowachieverstobemoreaffectedbylackofmotivation.Forthisreason,testresultsmightnotfullyrepresenttheactuallanguagecompetenceofthesestudents,buttheymightappeartobeatasignificantlylowerlevelbecauseafairnumberfromthisgroupoftesttakersmaychoosenottoshowwhattheycandoinEnglish.Aslongasthetesthasnopracticalimplicationsfortheindividualtesttaker,itwillbedifficulttogene-raterealinterestandmotivationinthosethatdisplaya'could-not-care-less'attitude.

In2013 theE8WritingTestwill be administerednation-wide for thefirst time.Thishasalreadyhadsomeimpactonteacherattitudeandmightalsohaveapositiveinfluenceonlearnermotivation.Preferredlearningstylesandpersonalitytraitsareotherfactorsthatarerelevant,butcannotbecateredforinthegiventestsituation.

Thethirdgroupoffactorsareexperientialcharacteristicsreferringtofamiliaritywiththetestformat.Whereasthetesttakersareallnewtothisparticulartypeoftesting,they shouldgenerallybe familiarwith the typeofpromptsused in theE8Writ-ingTest.Asdetails fromthe test specificationsconfirm(seepp.20–36),promptsusedarebasedontheBIST-Verordnung (Anlage zur Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungsstandards im Schulwesen – BGBl.II Nr.1/2009 v. 2.1.2009), theCEFR(Common European Framework of Reference for Languages: Learning, Teaching, Assessment) and theAustrian curriculum (LehrplanderHauptschule2008undLehrplanderAHS2006).

LearnerswhohaveonlydonetasksthatareheavilyscaffoldedwillfindtheE8promptschallenging.ThosewhohaveneverfacedopenwritingtasksintheirlearninghistorycannotbeexpectedtoperformwellintheE8WritingTestsorininternationaltests.Wewouldconsideritimportantwashbackifcoursebookauthorsand,consequently,alsoteachersweretorethinktheissuesinvolvedandalsoattemptunscaffoldedwrit-ingtaskswithALLpupils.AfterfouryearsofEnglishatsecondaryschoolandsome(very limited)writing atprimary level amounting tomore than500 lessons, anystudentshouldbeabletodoataskliketheonebelowsuccessfully:


You have come back from a one-week stay with a host family in Cambridge. At home you remember that you left your mobile phone in your room in Cambridge. Write a short email to your host family.

�Tell them where you are now.�Tell them about your mobile.�Ask them for the mobile.�Tell them how you liked your stay.

Figure 2: BIFIE Item Archive (http://www.bifie.at/freigegebene-items)

2011isthefirstyearwithanewgenerationofcoursebooksavailableforAustrianschoolstochoosefrom.Whatwasformulatedaboveasexpectedwashbackin2008hasmaterialised:ThenewcoursebooksincludewritingtasksthataregearedtotheE8WritingSpecificationswithanumberofthemextremelyclosetoactualE8Writ-ingPrompts.Eventhetimeconstraintsandthespecificationsregardinglengthhavebeen takenonboard.Another salient feature is the attempt to actually teach thestudentsaboutusingparagraphswhenproducing(longer)texts.

CognitiveValidity

“ThecognitivevalidityofaWriting task isameasureofhowclosely it representsthecognitiveprocessing involved inwritingcontextsbeyondthetest itself, i.e. inperformingthetaskinreallife”(Shaw&Weir2007,34).Whereasitisnotoriouslydifficulttodescribethecognitiveprocessesastheyarenotdirectlyaccessible,itseemsimportanttodescribeageneralwritingmodelthataccountsforwritinginareal-lifecontextaswellas inanexamsituation.However,onedifferenceshouldbenotedat the outset, namely that there is no time-constraint in most real-life situationswhereasintheE8testingsituationtime,topic,genre,andlengthofoutputarepre-determined.Thismightimposelimitationsontheplanningphaseaswellasonthewritingandrevisionphases.

WritingTheoryinBrief

Inthegivencontext,onlysketchyreferencesshallbemadetovarioussourcesthatpresentanddiscussthewritingprocessandmodelsofL1andL2writingindetail.AccordingtoGrabeandKaplan(1996,230–232),theplanningphase,whichtheycall“goalsetting”,involvesthesefivefactors:

�� anassessmentofthecontext�� apreliminaryrepresentationofthewritingproduct�� anevaluationofpossibleproblemsintaskexecution�� aninitialconsiderationofthegenrerequired�� anorganisationalplan

ShawandWeir(2007,37)makeapointofemphasisingtheadvantagesofamorepsycholinguisticallyorientedmodelofwritingover theGrabe andKaplanmodeland refer toField (2004)andKellogg (1994,1996). Interested readersmaywishto consult the detailed discussion there. The Field model (Field 2004, 329–331)involves

�� macro-planning�� organisation�� micro-planning


�� translation�� execution�� monitoring�� editingandrevising

AreferencetoScardamaliaandBereiter(1987)isessentialhereastheyhavedescribedtwodifferentstrategiesusedbyskilledandlessskilledwritersintheplanningphase: knowledge telling and knowledge transformation.

Inknowledgetelling,novicewritersplanverylittle,andfocusongeneratingcontentfromwithinrememberedlinguisticresourcesinlinewiththetask,topic,orgenre.Knowledgetransformingbytheskilledwriterentailsaheightenedawarenessofproblemsasandwhentheyarise–whetherintheareasofideas,planningandorganisation(content),orinthoseofgoalsandreadership(rhetoric)[…](Shaw&Weir2007,43).

Whereas thisholds true for allwriting,L2writingposes additional cognitivede-mandsonthewritersasField(2005)argues.Attentiondirectedtowardslinguisticaspectslikelexicalretrieval,spelling,andsentencestructurescanimpedethefluencyofwritingandthecapacitytoorganiseandstructurethetextasawhole.Someideasmighthave to be abandoned in the executionphase on the grounds of languageconstraintsandlimitations.

CognitiveProcessingintheE8WritingTest

IntheE8contextwesuggestusingamodifiedGrabe/Kaplan-Fieldmodeltoillustratethewritingprocess,whichwillclearlybebasedonknowledge tellingandthushasaverybriefplanningphasemainlyconsistingofconsideringrelevantcontentpoints.

Thismodelincludesthefollowingphases:

�� assessmentofthecontext(whowritesaboutwhattowhomandwhy?)�� characteristicfeaturesofthegenrerequired�� preliminaryrepresentationofthewritingproduct�� selectionofcontentpoints�� evaluationofpossibleproblemsintaskexecution�� micro-planningatparagraphandsentencelevel�� translation�� monitoring�� revising

Figure3isagraphicrepresentationofthemodifiedGrabe/Kaplan-Fieldmodelhigh-lightingthethreemainsteps.

Figure 3: Modified Grabe/Kaplan-Field Model

Planning

ContextGenre

Preliminary representationContent

Problems

Writing

Micro-planningTranslationMonitoring

Editingand

Revising


IntheE8testsituation,theplanningphaseisclearlynotelaborateorextensive.Afteranassessmentofthecontext,whichincludesidentifyingthetopic,thesituationofthewriter,thetextpurposeandtheaddressee,mosttesttakerswillmovestraighttotheconsiderationofthegenrerequiredanddevelopa“preliminaryrepresentationofthewritingproduct”.Thenthebulletpointswillpre-structurethecontentelementstobeincluded.Anorganisationalplanisnotnecessaryasthetasksarefairlyshortandguidedbycontentpointswithlittleopportunityfordeviation.Especiallywiththeshorttask(40–70words)planningeffortswillbereducedtothebareminimumandberestrictedtothedecisiononwhichcontentpointstoelaborateandhowtoproceedinthatdirection.

Thewritingismorecloselyguidedthaninreallifeasanumberofcontentpointsaregivenintheprompt.Thismakesthewritingprocesssomewhateasierthaninreallife,butontheotherhand, it seemsunavoidable ifwewant toensure inter-raterreliability for thedimensionof taskachievement. Inadditionto this,providingacontentschemaforcandidatesatthislevelisnecessarybecausethecognitiveloadforsimultaneousactivitiesonamacroandmicrolevelwouldbetoogreatandthetasktoodemanding.

Ithasbecomeclearfromthepresentdiscussionthatmacro-planningandorganisa-tionplaynoroleinthegivenwritingcontextandthattheproductdeliveredwillbefirmlysetintheareaofknowledge telling.

Themicro-planningphase,thenextstepofthewritingprocess,mightbethepointwherepossibleproblemsintaskexecutionwillbeidentifiedbeforetheactualwritingbegins.Theproblemswillbecontent-relatedandhavetodowithknowledgeoftheworldandwhat(abstract)ideastousewiththecontentpointsgiven;theymightalsobeconnectedtotheattempttorecalltherequirementsofthegenreinquestionandwiththelanguagenecessarytoexpresstheideas.

Thisstageofidentifyinglanguageresourcesandtheirlimitationsisonlyafractionaway from actually putting pen to paper and undoubtedly is a central aspect ofmicro-planningfocusing

onthepartofthetextthatisabouttobeproduced.Here,theplanningtakesplaceonatleasttwolevels:thegoaloftheparagraph,itselfalignedwiththeoverallgoalofthewritingactivity;withintheparagraph,theimmediateneedtostructureanupcomingsentenceintermsofinformation(Shaw&Weir2007,39).

Micro-planningmergeswiththe translationphasewherepreviouslyabstract ideasonlyaccessibletothewriterhim/herselfaretranslatedintothepublicspacedefinedbyasharedlanguage.IncontrasttoShaw&WeirandField,weseemicro-planningandtranslationastwostagesthatareinterlinkedasthewritermightoscillatebetweentheoneandtheotheratsentenceleveloratparagraphlevel(Shaw&Weir2007,39–40).

Itisinthetranslationstagethatlanguagedecisionshavetobemadeandplanningdecisionshavetobeimplemented.Theactualproductionoftexttakesplaceundertheconstraintsofcontentschemata,genrerestrictionsandthelimitationsoflinguisticresourcesathandinL2.Whathasbeencalled“avoidancebehaviour”(e.g.avoidinglexisorstructuresthatseemunsafe)and“achievementbehaviour”(e.g.usingsimplerstructures,paraphrasing)byField(2004,66–67)needstobetakencareof intheassessmentphase,asdoestheabilitytoproducecoherentandcohesivetexts.

Thenextstepismonitoringalthoughthisisnotnecessarilysequentialandmightbeoscillatingwithphasesoftranslation.“Atabasiclevelmonitoringinvolveschecking


themechanicalaccuracyofspelling,punctuationandsyntax”(Shaw&Weir2007,41).AtE8levelthis iswhatcanbeexpected, ifnotinthelowestsegmentoftesttakers.Inaddition,betterwriterswillalsocheckbackoncontentandgenrerequire-ments.Thesemonitoringactivitieswillleadtoeditingandrevisingifsomepartsofthetexthavebeenfoundunsatisfactory.Thismightinvolveadding,deletingormod-ifyingacontentpoint,addingcohesivedevices,replacingpoorwordsandphraseswithbetterones,orsimplycorrectingmistakesinspellingandstructure.

IntheE8context,writingiscertainlybasedontheknowledge-telling model (Scar-damalia&Bereiter1987);Hyland’ssummaryofthemodelepitomisesE8writingperformances:

A knowledge-telling model addressesthefactthatnovicewritersplanlessoftenthanexperts,reviselessoftenandlessextensively,andareprimarilyconcernedwithgeneratingcontent from their internal resources.Theirmaingoal is simply to tellwhat theycanrememberbasedontheassignment,thetopic,orthegenre(Hyland2002,28).

ContextValidity

Tests shouldbe as close aspossible to authentic real-life situations.Writing is anactivitythatisnormallyperformedbyindividualsatatimesetasideforit.Writershave a purpose and an audience; they have the freedom to interrupt the writingprocessandresumeitatatimeoftheirchoice,especiallyforeditingandrevising;andtheycannormallyusedictionariesandotherresources.Inthegiventestsetting,someconstraintswillbeoperative,butunavoidable.

Shaw&Weir2007(64–142)discussanumberofaspectsofcontextvalidityrelatedtothreeareas:

�� Setting:Task�� Setting:Administration�� LinguisticDemands:TaskInputandOutput

ThesepointswillstructurethediscussionofcontextvalidityoftheE8WritingTests.

Setting:Task

Theaspectstobediscussedhereareresponseformat,purpose,knowledgeofcriteria,weighting, text length, timeconstraints, andwriter-reader-relationship. In theE8WritingTestsauthenticityisoneofthemostprominentaimsofpromptconstruc-tion.However,incontrasttoreal-lifewritingthereisnoprovisionfortheuseofanyresourcematerialssuchasdictionaries.

ThewritingtasksaretargetedatpupilsofAustrianschoolsinyear8andnormallyagedfourteen.Thetasksaredesignedtoappealtothisagegroupandtoelicitscriptsthat show what test takers can do within the framework defined in the BIST-Ver-ordnung.Thedomainsandgenreshavebeencarefullyselectedfromthisframework,whichisbasedontheCEFR,andhavebeenfilteredfurtheronthebasisoftheAustriancurriculum.

Astheresponseformatmaywellplayasignificantroleintestperformance(Aldersonetal.1995),thedecisionhasbeentakentoincludetwoformatsintheE8WritingTest.Thereisashorttask(40–70words)andalongtask(120–180words),whichareassessedseparately.Bothareopenwritingtasks.Goodwritershaveabetterchancetoshowtheirbestinthelongtask,whichisbasedonaB1descriptor,takenfromthe


BIST-Verordnung.Lowerachieversareexpectedtodobetterintheshorttask,whichislimitedinscope,morecloselyguidedandbasedonanA2descriptor.However,both good andweakerwriters are expected to address both tasks as they arenotsupposedtochooseonlyoneofthetasks.

Instructions,deliveredbothorallyandinwritingtothetesttakersbeforetheactualtestbyatestadministrator,andrubrics thatgowitheachtaskpresentcandidateswithinformationregardingtextlength(seeabove)andtimeconstraints.Forcom-pletingbothtasksthetesttakershave30minutesofwritingtimeplus5minutesforeditingandrevisinginall.After35minutesthereissometimeforwordcountbythecandidates.Theactualpromptscontextualisethetaskbydefiningthewriter-reader-relationship,statingpurposeandgenre,andgivingcontentpointstobeincludedinthetext.Theshorttaskcontains3–4contentpoints,thelongone5–8.

Informationonthescoringcriteriausedandtheirweighting,includingtheratingscaleused,scaleinterpretationsandbenchmarkedsamplescripts,ispublishedinthisreport(seepp.12–14,23–27,28–36,37–44).Furthermore,sampleprompts,therating-scaleandbenchmarkedtextsarepubliclyavailableontheBIFIEwebsite2.

Setting:AdministrationofE8WritingTest

In its present form, thewriting testwasfirstpilotedon a sampleof ca. 800 testtakersin2007andin2009abaselinestudywascarriedout.Consequentlydetailedinformationonthe“pilotphase”between2006and2008andonthebaselinetestsin2009werepublishedinaTechnicalReport(Breit&Schreiner2010).Startingin2013,theE8WritingTestswillbesetnationwideeverythreeyearsandallAustrianschoolchildreningrade8willbetested.OnlySENpupils,i.e.thosewithspecialeducationalneeds,willbeexemptedfromdoingthetests.

Inordertoensurereliabletestresults,thecircumstancesunderwhichtheE8WritingTest takesplacemustbe similar.The stepsdiscussed inmoredetailhere concernphysical conditions,uniformityof administration, and test security,basedon theideasbyShaw&Weir(2007).

As thevenuesof theE8WritingTestareclassrooms inAustrianschools,physicaltestconditionsareofverysimilarstandardsandtesttakersshouldfindappropriateconditionsfortakingthetest.

In order to grant the uniformity of administration, the test must be conductedaccordingtostandardisedinstructionsbytrainedtestadministrators.Anextensivetestadministrator’smanualisprovidedduringthetestadministratortraining.Themanualincludesinformationonthebackgroundofthetest,checklistsandToDo’sbothforthepreparation,theactualsettingofthetest(e.g.startingtheexam,com-pletingdifferentlists,standardisedverbalinstructionsforthetestadministratoretc.),andtheconclusionoftheexamination.

Inanationwideexamtherearesomeadministrativeconstraints:apoliticaldecisionhasbeentakenregulatingtestadministrationintheyearstocome:in90%oftheclassestheE8Testswillbeadministeredbytheteachersoftheschool(internaltestadministration).Inafurther3%oftheclassesthetestswillalsobeadministeredinternally, but there will be external quality monitors to assure the correct andstandardisedadministrationofthetests.7%oftheclasseswillbetestedexternally.Alltestadministrators,bothinternalandexternalones,aretrainedtoadministerthe2 http://www.bifie.at/freigegebene-items[24June,2011]


testsaccordingtoagreedstandardisedprocedures.However,itiswithintheresponsi-bilityof the schools’head teachers to take careof a correct and standardised testadministration,asthisistheonlywaytogetreliablefeedbackregardingtheperfor-manceoftheirpupilsandtoplanlocalmeasuresofqualitydevelopment.

Thepromptsused in futurewritingtestshaveallbeenwrittenbytheprospectiveraters,moderated, edited, and screenedby theBIFIEWritingTrainerTeam,pre-testedandstored in the itemarchive.Thetestbookletsaredesignedby the samegroupincooperationwiththepsychometricdepartmentatBIFIE.Theactualdistri-butionofalltestpaperstotheschoolsishandledcentrallybyBIFIESalzburg.

MoredetailedinformationontheadministrationoftheE8TestswillbepublishedinaTechnicalReportafterthefirstnationwidetestingin2013.

LinguisticDemands:TaskInputandOutput

IntheAustrianteachercommunitythecommunicativeapproachtolanguagelearn-ing(Canale&Swain1980asanimportantprecursoroftheBachman1990modelofcommunicativelanguageability)iswidelyaccepted,anditisalsosetdowninwritinginthenationalcurriculum.Asthelearningtasksaremodelledonreal-lifecontexts,thelearningenvironmentaimstomirrorreallifeascloselyaspossible.ExamssetintheAustriancontextneedtosharethesepremisesandtoreflecttheminthetasksset.

Shaw&Weir(2007,91),Alderson(2004,13)andotherscomplainthattheCEFRremainsvagueandwithholdsdetailswhenitcomestostructuresorvocabulary,usingtermslike“simple”inthedescriptors.Whilethisistrue,readingtheCEFRexten-sivelyratherthanfocusingonlyonthesectionscontainingthescalesproveshelpful.Inchapter3,thedevelopmentofthecommonreferencelevelsisexplainedanditismadeclearthattheyprogressinaverycoherentwayfrom“thelowestlevelofgener-ativelanguageuse”(CEFR2001,33)tosocialfunctionsand“descriptorsongettingoutandabout”(CEFR2001,34)basedonWaystage (A2)andasimplifiedversionofsometransactionallanguagefrom“’TheThresholdLevel’foradultslivingabroad”(CEFR2001,34).A2+doesnotsomuchincreasetherangeoftopics,butfocuseson“moreactiveparticipationinconversation”and“significantlymore[onthe]abilitytosustainmonologues”.B1reflectstheThreshold Levelandinvolves“theabilitytomaintaininteractionandgetacrosswhatyouwantto,inarangeofcontexts”aswellas“theabilitytocopeflexiblywithproblemsineverydaylife”(CEFR2001,34).B1+makesincreaseddemandsonthequantitiesofinformationtobehandled.

As this is theway the levelshavebeen constructed (i.e. fromWaystage toA2), itseems legitimate to move from A2 specifications back to Waystage. And here wehaveavocabularylistandalistofstructuresconsideredcharacteristicofthatlevel.AsUCLEShavealsoorientedthemselvesonvocabulary lists fromtheCouncilofEuropePublications(LexicalInventoryinWaystage,1980,45–62;andinThreshold, 1979,85–115),itcanbeconsideredausefulshortcuttopickupthevocabularylistspublishedonthewebforKET(A2)andPET(B1),especiallyas thesehavebeenupdatedonthebasisoffrequencyandusagedatafromlanguagecorpora.Generally,“[the] language specifications of KET are the same as those set out inWaystage1990”(KETHandbook2007,1).

Toresumethediscussionofthevaguenessofdescriptorsusingwordslike“simple”,“basic”or“sufficient”,itmaysufficetosaythatthisvaguenessneedstobecontextu-alised.IftheA2descriptoronGrammaticalAccuracyreads“Usessomesimplestruc-turescorrectly,butstillsystematicallymakesbasicmistakes”(CEFR2001,112),wecanexpectlearnerstousetherangeofstructureslistedinthe Structural Inventory of


Waystage (63–83)ortheKET Handbook(8–9)withseverelyrestrictedaccuracy.Inthissense,evenvaguetermslike“simple”arereasonablywell-definedsothatpromptwritersandratersknowwhattolookfor.

TheE8writingpromptsdonotrestricttesttakersintheiruseofspecificlexicalorstructuralresources,butgivethemtheopportunitytodemonstratetheirlinguisticabilitieswithinthetaskset.Theextentoftheirsuccessindoingsoisassessedaccord-ingtothegradeddescriptorsintheassessmentscale.

Whathas tobenoted,however, is thebasicorientationof theCEFRtowardsanadultlearnerandadominanceoftouristaspectsoflanguagelearning.ThisiswhytheAustrianE8Standardshavealso integrated the specifications setdown in theAustriancurriculumandadaptedtheCEFRdescriptorstotheagegroupofthetestpopulation.Thismainly reflects the selectionofdomains and transactional situa-tions.Ithasnoinfluenceonthestructuresincluded,thoughithassomeinfluenceonthewordlist.Generally,theschoolbooksusedinAustriatakethisintoaccount.AsthetestisexplicitlybasedontheAustriancurriculum,thelinguisticdemandsofthetestarefairforalltesttakers.

The writing prompts used often specify particular language functions to be per-formed,e.g.“invite…,apologise…,askfor…,giveadvice…”.Alistofthesefunc-tionshasbeenmadeavailabletotheteacherspreparingthetesttakerssothattheycanbeexpectedtobeawareofthem(seepp.53–54intheappendix).

Severalresearchpapershaveobservedaninteractionorevenaninterdependenceofcontentknowledgeontheonehand,andwritingperformanceandtestscoresontheother(Read1990,Papajohn1999,Weir2005).Provisionsforthishavebeenmadeby restricting topics toareas that can safelybeassumed tobe familiar to the testtakersastheyaresetdownintheAustriancurriculumandmusthavebeenincludedintheirEnglishlessons.However,thisstillleavesthefactthatsometesttakersmightfeel indisposed todealwith aparticular topic for anumberof reasons, themostcommonprobablybeinglackofmotivationandinterest.

Moredetailedinformationondiscoursemode(i.e.texttypes),functionalresources(i.e.intention/purpose),andcontentknowledge(i.e.topicarea)canbefoundinthetablesonpp.21–22representingtheE8ConstructSpace.

ScoringValidity

Scoringvalidityisconcernedwithalltheaspectsofthetestingprocessthatcanimpactonthereliabilityof test scores. […It] iscriterialbecause ifwecannotdependontheratingofexamscriptsitmatterslittlethatthetaskswedeveloparepotentiallyvalidintermsofbothcognitiveandcontextualparameters.Faultycriteriaorscales,unsuitableratersorprocedures,lackoftrainingandstandardisation,poororvariableconditionsforrating, inadequateprovision forpost examstatistical adjustment, andunsystematicorill-conceivedproceduresforgradingandawardingcanallleadtoareductioninscoringvalidityandtotheriskofconstructirrelevantvariance(Shaw&Weir2007,143–144).

Inthissectionweexamineeachoftherelevantparametersinsomedetail:criteriaandratingscale,ratercharacteristics,ratertraining,ratingprocess,ratingconditions,postexamadjustments,andgrading.


CriteriaandRatingScale

Beforetheactualconstructionoftheratingscale,informationonexistingscaleswascollectedandtheusefulnessofthescalesintheframeworkofE8Testingwasana-lysed:Jacobsetal.scoringprofile1981(Weigle2002,116);TEEPattributewritingscales,Weir1990(Weigle2002,117);FCEScoringrubric1997(Weigle2002,152);TOEFLwritingscoringguide2000(Tankó2005,125);IELTSbands2002(Weigle2002, 159); Analytic writing scale developed by the Hungarian School-LeavingEnglishExaminationReformProject2005(Tankó2005,127).

LumleyreportsfindingsfromWeigle1994,whousedananalyticscaletohave30compositionsassessedbynoviceandexpertraters.Weiglefocusedonnoviceraters,whichisrelevanttotheE8situationinAustriawherearatingcultureisonlyjustevolving.

Shefoundthatraterreliabilityincreasedasaresultoftraining,andthattheimprovedagreementwastheresultofratersgainingbetterconsensualunderstandingofthetermsandlevelsrepresentedinthescale.Shefoundevidencethattraininghelpedclarificationoftheratingcriteria(Lumley2005,44).

Thissupportstheviewofthetestingteamthatinthegivencontextananalyticscalewouldbepreferabletoaholisticscale.ThisviewisalsosupportedbyWeigle2002,whomentionsseveraladvantagesofanalyticoverholisticscoring:

�� Itismoreusefulinratertrainingasinexperiencedraterscanmoreeasilyunderstandandapplythecriteriainseparatescales.�� Itisparticularlyusefulforsecond-languagelearners,whoaremorelikelytoshowa

markedorunevenprofile.�� Ascoringschemeinwhichmultiplescoresaregiventoeachscripttendstoimprove

reliability(Weigle2002,120).

AnotherreasonforrulingoutaholisticapproachwasthefactthatratingproceduresforscriptswithintheAustrianschoolsystemarenotregulated,showgreatvarietyandaretoalargeextentholistic,evenimpressionistic.Asassessmentproceduresforwriting inAustrian schools cannotbe taken as abasis for adisciplined approachtowardsratingscripts,breakingwiththistraditionseemedtobestguaranteeafreshapproachtoassessment.

Taking the general background of Austrian traditions in assessing writing intoaccountandinspiredbytheHungarianscale(Tankó2005,127),thedecisionwastaken to design an analytic scale measuring four dimensions:Task Achievement,CoherenceandCohesion,Grammar,andVocabulary.Whereasthreeofthesefourdimensionshave a strong recognitionvalue forAustrian teachers,Coherence andCohesion might appear unusual and reflects the high importance given to thisdimensionbytheCEFR.Thesefourdimensionspromisedtoyieldenoughdetailforaconstructivefeedbackprofileonindividualtesttakerperformance,informationforinstructionaswellasinformativedataforsystemmonitoring.

The assessment scale was constructed bearing in mind the fact that the overallmajorityofperformancescouldbeexpectedtobearoundA2/B1.ThismeantthatA2andB1descriptorsneededtobeincludedwhileanythingatandaboveB2couldbeneglected.WeareawareofthefactthatthiskindofscalecannotmeasureB2orC1performancesandwehavesettledforstatingthatperformancesabovetheupperendofthedescriptorsintheE8Scalearecalled“aboveA2”forshorttasksand“aboveB1”forlongtasks.But,generally,theapplicabilityofaparticulardescriptordoesnot


automaticallysignalthatascriptisatagivenCEFRlevel.Firstly,bandsconsistofmorethanonedescriptor,andsecondly,linkingwrittenperformancestotheCEFRisacomplexprocedurethatisbeyondthescopeofthisreportandwillbediscussedinaseparatepublication.

Thesecondconsiderationinscaleconstructionwasthecognitiveloadthatraterscanmanageintheratingprocess.Thedecisiontousefourdimensionsisalsoinagree-mentwiththeCEFRrecommendationtoreducethenumberofpossiblecategoriesto“afeasiblenumber”as“morethan4or5categoriesstartstocausecognitiveover-load”(CEFR2001,193).Wetakeitthatthiswarningalsoappliestothenumberofbandsanddescriptors that raters canhandle, sowehaveopted for fourbandssuppliedwithdescriptorsandthreeemptybandsinbetween,makingitaseven-bandscaleplusazeroband.

Atthatpointinscaleconstructionthescalesconsistedofthreecolumns:Thefirstbeingadeflateddescriptorforeachofthefourbands,thesecondbeingextendedandcontainingmoredetail,andthethirdquotingtherelatedCEFRdescriptor.AnimportantdecisionintheprocessofscaleconstructionwastheremovaloftheCEFRlevelsattheendoftheCEFRdescriptorsand,inasecondstep,theremovaloftheCEFR descriptors altogether. This was the logical step to take when some ratersawardedband7toascriptandarguedthatthescriptwasaB2performance.How-ever,suchanargumentisinadmissibleasthepromptsusedinthetestarewrittenonthebasisofA2orB1descriptorsandresponsestothesepromptssimplycannotrepresentperformancesaboveA2orB1respectivelyasonebasicfactoristhescopeofaperformancetogetherwiththegivenlimitationsofdomainsandgenres.SoevenwhentheCEFRB2descriptorforGrammaticalAccuracy“Showsarelativelyhighdegreeofgrammaticalcontrol.Doesnotmakemistakeswhichleadtomisunderstand-ing.”(CEFR2001,114)describestheperformancewell,itdoesnotmeanthatitisB2,butthattheA2/B1taskhasbeencarriedoutverywellandthatthe(grammar)performanceisaverygoodA2orB1performancerespectively.

Inanotherstep,thescalewascondensedtoonepagewithanextendedscaleeachforTaskAchievementShortandTaskAchievementLong.Asthisdeflatedscalemightnotcarryenoughinformationfortheratersatthebeginningoftheirtraining,scaleinterpretationshavebeenprovided(seepp.37–44).Thescalesthemselveshavebeenfine-tunedinthetrainingprocessinanongoingdialoguewiththeraters.Itfollowsfromthis that thescalesarewhathasbeencalledassessor-oriented(Weigle2002,122;CEFR2001,38).

Thewritingscriptsareassessedonfourdimensions:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary.WhereasthelastthreearebasedontheCEFR and the Austrian BIST-Verordnung, the CEFR does not contain anythingexplicitonTaskAchievement.ThedescriptorsofthescalesonOverall Written Pro-duction and Overall Written Interaction mainly refer to linguistic and pragmaticaspects(Can write a series of simple phrases and sentences linked with simple connectors …), whereasthesubscalesonlymakereferencestotexttypes,domainsandcontentaspects(Can write personal letters describing experiences …). Thesedescriptorscannotbeoperationalisedinassessmentterms.

Inourview,however,thecontentaspectofwritingiscentralandlargelyresponsiblefortheoverallqualityofascript.Nevertheless,theratersdonotgiveanoverallgradeforwriting,butallfourdimensionsareratedseparatelyandarereportedasaprofile,whichmoreoftenthannot isunevenormarked.For the feedbackprocedure,anoverallwritingscorewiththefourdimensionsoftheshortandthelongperformanceassessmentisgiven,basedonanequalweightingofalldimensions.


RaterCharacteristics

Ithasbeenreportedthat“SubjectspecialistsandlanguagetrainedEFLteachersdem-onstrateatendencytoemployratinginstrumentsdifferently”(Elder1992,inShaw&Weir2007,169).InthisrespectthepresentsituationinAustriaisuncomplicatedasallratersareteachersofEnglishwhoteachinlowersecondaryschools.SomeofthesearenativespeakersnowlivingandworkinginAustria,somehaveauniversitybackground,otherswereeducatedatUniversityCollegesofTeacherEducation.

Althoughtheratersgo througha specific training that familiarises themwith theratingscalesandtheratingprocedures,differencesintheirexperientialbackgroundandintheirprofessionaltraininganddevelopmentmayleadtodifferingassessmentsofscripts.Inordertomakeratersawareofthisandtostartaprocessofself-reflection,allratersgetdetailedfeedbackontheirratingbehaviouratseveralpointsinthetrain-ingandparticularlyafterthelasttrainingsessionandaftertheadministrationofawritingtest.Theyareinformedabouttheirinter-raterreliabilityandraterseverity.Eventually,harshnessandleniencyofratersistakencareofthroughRaschmodell-ing.

RatingProcess

Milanovicetal.(1996)identifiedanumberofapproachesraterstakeintheprocessofratingascript.Inourtrainingsessionswegenerallyadviseagainstthe“readthrough”andthe“provisionalmarkapproach”,bothofwhicharebasedononereadingofthescript.Ratersareencouragedtoadopta“principledtwoscan/readapproach”totheprocesswithafocusonTaskAchievementandCoherenceandCohesioninthefirstreadingandonGrammarandVocabularyinthesecond.Thelengthofthescriptsseemstosupportthisapproach.

WeareawareofgroupeffectsonraterreliabilityasdescribedbyShaw&Weir(2007,174–175)andhavemadeanefforttousethemtoouradvantageinthestandard-isationmeetingsatthebeginningofthetrainingsessionsandtheratingsession.Inaddition to theprocedures recommended for standardisationmeetings (Alderson,Clapham&Wall1995,112–113)aconsiderableamountoftimeisspentonthedetailedinterpretationoftheprompts(seeappendix,pp.48–52)andanopendiscus-sionofanyquestionsthatmightberaisedbytheraterstakingintoconsiderationthatallratershavealsobeeninvolvedinthewritingofpromptsandtheirpiloting.Anadditionalsetoftenbenchmarks,gainedinanextensivebenchmarkingconferencewithtenbenchmarkers,playsavitalroleinthestandardisationsessions.

RatingConditions

In2013thewholeE8populationofsome90,000pupilswillbetested.InJune,allraters,whohavebeentrainedatdifferent intervals since2006,will takepart inaone-daystandardisationmeetingasdescribedonp.18intrainingphase6,followedupwithaone-dayratingsession.

TherewillberegionalstandardisationmeetingsforallraterswhomarkscriptsfromtheE8WritingTests.Inthesesessionsratersareupdatedon,forexample,anychangesregardingtheassessmentscaleused.Thentheywillcontinuewiththeon-siteratingsession,inwhichtheywillundergosupervisedratingwiththenewtestpromptsandactualscriptsofthetest.ThescriptshavebeencarefullycompiledinratingbookletsbytheBIFIESalzburgpsychometricdepartment.ThiswillprovideBIFIEwiththerelevantdataneededfortestanalysisandfeedback.


Itwilltakehalfaday’sworktodealwitheachpromptandgiveratersenoughtimeforon-site ratingandclarificationof ratingproblemsbasedon theparticular testprompts.Theremainingunratedscripts,approximatelythreequartersintotal,willberatedoff-sitewithinsixtoeightweeksattheraters’convenience.

RaterTraining

AccordingtoAlderson,Clapham&Wall,ratertrainingisoneofthemostessentialaspectsinanefforttoobtainreliableassessments(1995,105).Lumleyrefersto„agrowingbodyofworkthatshowstheeffectivenessofthetrainingprocessinallowingraterstodevelopadequatereliabilityoragreementintheuseofindividualscalesinrelationtospecifiedtesttasks“(2005,62).

ThishasbeentakenveryseriouslybytheBIFIEWritingTrainerTeam,whohavedevelopeda seven-months trainingprogramme for raters starting inOctoberandpreparingtheraters for themockratingsession inApril/May.Thisprogrammeisdescribedinsomedetailbelow.

RECRUITMENTIn the recruitment phase teachers in Austrian lower secondary schools are ap-proachedtobecomewritingraters.Asthetesttakerscomefromthetwodifferenttypesoflowersecondaryschools,theGeneralSecondarySchool(HauptschuleandNeueMittelschule)andtheAcademicSecondarySchool(AHS),carehasbeentakento ensure intakeof raters fromall threeof these school types.While recruitmentwasoriginallycarriedoutbyBIFIESalzburguntil2009,theadministrationoftherecruitmentprocesshassincebeenoutsourcedtotheregionalUniversityCollegesofTeacherEducation.

TRAINING PHASE 1: OCTOBER (1 DAY; FACE-TO-FACE SESSION)AstheCEFRisthemostrelevantbackgrounddocumentfortheE8Standards,thestar-tingpointofthefirsttrainingsessionisThe Common European Framework ingeneralandtheOverall Writing Scales for Production and Interactioninparticular.Thefamil-iarisationwiththeCEFRisimplementedonthebasisoftherecommendationsmadeintheManual on Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF) (2003, 70–77),includ-ingsortingtasks.Itismadeclearatthispointthatoneaspectofwritingisrelatedtocommunicativeactivitiesandstrategies,anotheronetolinguisticcompetences.

TheTestSpecificationsarepresentedanddiscussedindetail:domains,genres,constructspace,promptformat.Inthisphasethereisafocusonpromptproduction.Thereare Guidelines for prompt writers thatprovideassistanceintheprocessofpromptwriting.

Eachprospectiveratertakesonthetaskofwritingonelongoroneshortpromptinatrainingtandemduringtheweeksfollowingthefirsttrainingsession.Allpromptsaresentintothetrainerteamformoderation.Oncethepromptshavebeenscreened,theyarepilotedbythepromptwriters.Behindthisprocedurethereisthebeliefthatratersneedtoknowaboutthequalitiesofpromptsandwhatelementstheymustcontain.Thispreparesthemforbetterinteractionwiththetestpromptsintheactualratingphase.

Withregardtodifferentialvalidityitisimportanttostatethateveryprecautionistakenattheearlystageofpromptwritingtoavoidtestbias.Varyingculturalback-groundsandknowledgeof theworldhavebeentakenintoaccountaswellas thegivenvarietyofcognitivecharacteristics,mothertongue,ethnicity,andgender.


AfteranintroductiontotheAustrianE8Standardstheratingscaleispresentedandexplainedonthebasisofwrittenscaleinterpretations(seepp.37–44).Unfoldingthesevenbandswithfourofthemdefinedandworkingthroughthefourdimensionstakestime.Theprocedureadoptedistolookatthesevenbandsofonedimension,whichleadstosometheoreticalunderstandingofthescales,butconnectionstoactualscriptsarestilltenuous.Sotheratersgettwoscriptseachforindividualmarkingonthefirstdimension(taskachievement).Theydiscusstheirassessmentinsmallgroupsandthetrainerdisclosesthebenchmarkedassessmentsandarguesthecase.Thispro-cedureisrepeatedfortheotherthreedimensions.

In the secondphase of the trainingworkshop theparticipants get sets of bench-markedscriptswhichtheyrateonallfourdimensions.Benchmarkjustificationsarepresentedindetailanddiscussedinplenarytoensurethattheratersunderstandthedescriptorsoftheratingscaleandcanhandletheratingagreementsthathavebeenformulatedforanumberofspecial ratingproblems(e.g.downgradingforbelow-lengthtexts).Afterdiscussionsandargumentationofthejudgementsforthebench-marked scripts theparticipantshavea rough ideaof thecomplexityof the ratingprocessandtheeffortittakestoarriveatreliablejudgements.Theratingsheetsfilledinbytheparticipantsprovideafirstsetofdatathathelpstomonitorintra-raterreliability.

TRAINING PHASE 2: OCTOBER – DECEMBERThesecondtrainingphaseisanopenonewithafairlyloosestructure.Allpartici-pantsfirstwritetheirpromptsandgetthembackfromthetestingteamasscreenedprompts(insomecasesthepromptsarereturnedtothewritersforrepair).Asafirstmeasuretowardsqualityassurancethepromptwritersproducearesponsetotheirownprompt.Thisshouldmakepromptwritersawareofthemoreobviousflawstheirpromptsmighthave.Thepromptsarethenpilotedinoneoftheirclassessothatallparticipantshavearound20scriptsbasedontheirprompt.

TRAINING PHASE 3: JANUARY – MARCHOncethepromptwritingandpilotingisfinished,theonlineratingphasestartsinJanuarywiththetrainerteamsendingoutscriptstotheratersforindividualratingoneachofthefourdimensions.Theratershaveaboutfourweekstodothisandsendintheirratings.Whenallscoringsheetshavebeensubmittedtothetestingteam,the benchmarks are sent out to the raters. In February/March the raters practisetheirratingskillsontheirownpilotscripts.Theyselecttwoscriptstobefedintothetrainingprocess.Theyratethesescriptsandwritejustificationsfortheirratings.Asecondonlineratingphasehelpstostandardisetheraters,whoareencouragedtomakefinaladjustmentstotheirscoresandjudgementsbeforesendingthemtothetrainertogetherwiththedigitalisedscripts.ThetrainergoesthroughthesescriptsandselectsinterestingsamplesfortheupcomingtrainingworkshopinApril/May.

TRAINING PHASE 4: APRIL/MAY (1 ½ DAY; FACE-TO-FACE SESSION)TrainingPhase4startswithadiscussionofopenquestionsfromprevioustrainingphases.Thenthereare twostandardisation sessionswith recycled scriptsandnewbenchmarksandthetimeisspentratingscriptsanddiscussingparticularproblemsarisingintheprocess.

Afterthesetwostandardisationsessionsthefirstpromptwritertandempresentstheirpromptandthewholegrouprates2–4scripts.Experiencefrompreviousratingses-sionshasshownthat,astheratershavetohandleanumberofdifferentpromptsinthisphase,theyneedmoreguidanceintheanalysisofthepromptsandthereforearealsoprovidedwithpromptinterpretations.Thepromptwritersthendisclosetheirjudgementsanddefendtheirscoresinadiscussionwiththewholegroupmonitored


bythetrainer.Thisprocedureisrepeatedsothatthemajorityoftheratershavethechancetodiscusstheirscriptsandtheirjudgementswiththewholegroup.Inter-raterreliabilityandintra-raterreliabilityaremonitoredandpertinentdataoneachrateriscollectedsystematically.

TRAINING PHASE 5: MOCK RATING SESSION: IMMEDIATELY AFTER TRAINING PHASE 4 (1 ½ DAY; FACE-TO-FACE SESSION)Thesecondimportantpartofthismeetingistheanalysisandinterpretationofnewprompts thatwerepilotedona representative sampleofprospectiveAustrian testtakersandwhichmightactuallybeusedinafuturetest.Ratersaregivendetailedinformationandareinvitedtodiscussanyissuesthatarestillunclear.

Thentheactualratingbegins.Theratersreceivebookletsofpilotedscriptswhichwerewritteninresponsetoashortoralongprompt.Thereisaratingplanwithoverlapformultiplerating.Afterabouttenscriptshavebeenrated,theratersmeetwiththesupervisortodiscussanycriticalissuesthatmayhavecomeupduringtherating.Thentheyproceedtoratetheotherscriptsofthatbooklet,whichinvolvessomefreetimemanagementfortheraters.Thisprocedureisthenrepeatedfortheotherbooklets.

Thescoringsheetsfilledinbytheparticipantsprovideasetofdatafortheanalysisofratingbehaviour.Thedataareusedtogiveextensivefeedbacktoallratersontheirinter-raterreliabilityandraterseverity.

TRAINING PHASE 6: UPDATE SESSION IN THE YEAR OF THE ACTUAL TEST (1 DAY; FACE-TO-FACE SESSION)ThereareregionalstandardisationmeetingsforallraterswhomarkscriptsfromtheE8WritingTests.Inthesesessionsratersareupdatedon,forexample,anychangesregardingtheassessmentscaleused.Thentimeisspentontheanalysisandinterpre-tationofthepromptsusedintheactualtest,andbenchmarkedscriptsbasedonthesepromptsarerated.

After theupdate and the standardisation session, theon-site rating session starts,followingthesameprocedureasdescribedintrainingphase5above.

PostExamAdjustments

Although considerable efforts are taken in the training programme to minimisediscrepancies inraterbehaviour,theratingsareadjustedforanyremainingdiffer-ences in rater severity by means of multifaceted Rasch analysis after the scriptshavebeenmarked.Thisbecomespossiblebyhavingacertainproportionofscriptsmarkedbytworaters(doublerating)andanotherproportionofthescriptsbyallraters(multiplerating)sothatraterbehaviourcanbeassessedintermsofmodelfitaswellasseverity.

ReportingResults

ThepurposeoftheE8StandardsisgivingfeedbackonthewritingcompetenceofAustrian pupils in grade 8. The aim, therefore, is system monitoring rather thancertificationorselectionatthelevelofindividualtesttakers.Cutscoreswillbeestab-lished,however,toenableindividualfeedbacktotesttakersandshowwhethertheobjectivesinthenationalcurriculumhavebeenmetasregardswritingcompetenceingeneralandallfourdimensionsinparticular.Consequently,whilethetestresultsarelinkedtotheCEFR,criticalcutscoresonwhichtobaseselectiondecisionsneednotbeestablishedbythetestconstructors.Itishopedthatbyprovidingresultstoindividualteachersandschoolsthisfeedbackwillinstigateaqualitativedevelopment


thatwill radiatebeyond regionsand spread throughout thewhole school system.Thewayfeedbackontheresultsisgiventotesttakersandotherstakeholdersisbeingdevelopedatthemoment.Incompliancewithpoliticalrequirements,onlythetesttakersthemselveswillhaveaccesstotheirindividualresultsthroughacodetheywillbegivenwhensittingtheexam.Teachersandschoolprincipalswillreceiveaggre-gateddataforthegrouprelevanttothem(class,school)viaaninternetplatform.Educationalauthoritieswillreceiveaggregatedreports.

Theinformationthatresults fromthewritingtest isreportedonthefourdimen-sionsoftheWritingScale(TaskAchievement,CoherenceandCohesion,Grammar,Vocabulary).The results for eachdimensionare reportedona scale from0 to7,whichenablesreferencetotheCEFRuptoB1.Ratingsareadjustedfordifferencesin rater severity and taskdifficulty bymeans ofmultifacetedRasch analysis.Theresultsarethereforecomparableacrossalltesttakersregardlessofwhichraterratedtheperformanceandwhatparticularprompttheperformanceisbasedon.Theproc-essofstandardsettingandCEFR-linkingwillbedescribedinmoredetailinatech-nicalreportaftertheactualtestin2013.

ConsequentialValidity

Shaw&Weir(2007,218)taketheterm‘consequentialvalidity’fromMessik1989andinterpretitinthelightofrecentliteraturetoincludewashback(influencesonteaching, teachers, learning, curriculumandmaterials) and impact (influencesonthecommunityat large).TheE8Standardscanbeenvisagedasan instrumenttoinitiatechangesinthedirectionofpositiveorbeneficialwashback.

Inandaround2008newcoursebooksforteachingEnglishtothetargetgroupwerelaunchedandanumberof themclaimtobe informedby theCEFRand theE8Standards.ThismeansthattextbookwritersarewellawareoftheE8StandardsTestsandareadaptingtheirmaterialstowardsthem.

Therequirementsforthewritingtestareclearlylaiddowninthisreportanddemon-stratewhatkindsofwritingourlearnersareexpectedtodeliver.

Theexpectationsofthetestdesignersformulatedin2008havebeenlargelyfulfilled3.Three years later a greatnumberofwriting tasks in thenew coursebooksused inAustrianschoolshavechangedinthedirectionindicatedintheTechnicalReport4of2008.Thereismuchlessscaffoldedwriting;thetasksarerealisticandauthentic;texttyperequirements,variationintexttypes,textlengthandtimeconstraintsareallinlinewiththepresenttestspecifications.Somecoursebooksalsoemphasisetheuseofparagraphsinwriting,givehintsonhowtowritegoodparagraphs,andprovidecorre-spondingexercises.ThismeansthatAustriantesttakerswhosittheE8Testsafter2011willbefamiliarwiththetestformat,theparticularrequirements,andtheinstructions.

3 Firstedition,2008,p.19:„Itishopedthatthiswillleadtolessscaffoldedwriting,thusenhancinglearnerem-powerment.TheemphasisgiventocoherenceandcohesionintheCEFRandtheE8Standardsmightalsofocusteacherattentiononthisareaandentailimprovements.“


E8WritingTestSpecificationsVersion03(July2011)

TheguidingdocumentsforthedevelopmentofthewritingtestspecificationsfortheE8WritingTestaretheAustriancurriculum(AHS2006;APS2008),theBIST-Ver-ordnung (BGBl.II Nr.1/2009 v. 2.1.2009)andtheCEFR (CouncilofEurope,2001).Thefirsttwodocuments listwritingcompetencesatdifferentproficiencylevels intermsoftheCEFR.

1.PurposeoftheTest

Themainpurposeofthewritingtestistoidentifystrengthsandweaknessesintesttakerswritingcompetenceandtousethis informationbothforthe improvementofclassroomproceduresandforsystemmonitoring.Whatismore,individualanddetailed test results are reported to the test takers,which isof interest to the testtakersthemselvesandtheirparents.

2.DescriptionofTestTakers

The test takers are Austrian pupils in the two different types of lower secondaryschools,theGeneralSecondarySchool(HauptschuleandNeueMittelschule),andtheAcademicSecondarySchool (AHS), towards theendofgrade8(8.Schulstufe). Pupils from all three ability groups in APS will be tested. The majority of testtakerswillbeaged14.SENpupils,i.e.thosewithspecialeducationalneeds,willbeexemptedfromdoingthetests.

3.TestLevel

ThedifficultylevelofthetestissupposedtoencompasslevelsA2toB1intheCEFR.

4.TestConstructwithE8ConstructSpace

The tables on pp. 21–22 summarise the construct space relevant for item designwithalistoftheprompttypesusedtotestthewritingcompetencesasspecifiedinthe BIST-Verordnung, targetedatlevelsA2,A2+,andB1oftheCEFR.Thetasksattheselevelsaskfor(mostly)concretecontent.ThereforetopicsarerestrictedtoareasthatcansafelybeassumedtobefamiliartothetesttakersastheyaresetdownintheAustriancurriculumandmusthavebeenincludedintheirEnglishlessons.

More specifically, the tasks display various text types and writing intentions/pur-poses.Tables1and2provideanoverviewof the rangeof text types andwritingintentions/purposesfortheproficiencylevelstested.Fortheactualconstructionofwritingitemspromptwritersaregivenspecialpromptdesignspecifications,whichclearlylistwhatkindofprompt–intermsofprompttype,level,BIST-Descriptor,topicarea,andtexttype–thepromptwritersaresupposedtocreate.


Pro

mp

t

Typ

e

CE

FR

Leve

l

CE

FR

Des

crip

tor

Des

krip

tor

aus

BIS

T-V

O:

Sch

üler

/inn

enk

önn

en…

Top

icA

rea

Text

Typ

esIn

tent

ion/

Pur

po

se

Pri

mar

y

Aud

ienc

e

LongPrompt

B1

¡

Can

writ

eac

coun

tso

f

ex

perie

nces

,des

crib

ing

fe

elin

gsa

ndre

actio

nsin

si

mpl

eco

nnec

ted

text

.

¡

Can

writ

ea

desc

riptio

nof

an

eve

nt,a

rece

nttr

ip–

real

or

imag

ined

.

¡

Can

nar

rate

as

tory

.

¡

Can

writ

epe

rson

alle

tter

s

desc

ribin

gex

perie

nces

,

fe

elin

gsa

nde

vent

sin

som

e

de

tail.

¡

Erfa

hrun

gsbe

richt

e

sc

hrei

ben,

ind

enen

Gef

ühle

un

dR

eakt

ione

nin

ein

em

ei

nfac

hen,

zus

amm

en-

hä

ngen

den

Text

wie

der-

ge

gebe

nw

erde

n

¡

eine

Bes

chre

ibun

gei

nes

re

alen

ode

rfik

tiven

Ere

ig-

ni

sses

,z. B

.ein

erR

eise

,

ve

rfass

en

¡

eine

Ges

chic

hte

erzä

hlen

¡

ausf

ührli

cher

eK

arte

n,

pe

rsön

liche

Brie

feu

ndE

-Mai

ls

sc

hrei

ben

und

darin

auc

hüb

er

E

reig

niss

e,E

rfahr

unge

nun

d

G

efüh

leb

eric

hten

¡

Fam

ilieu

ndF

reun

de

¡

Woh

nen

und

Um

gebu

ng

¡

Ess

enu

ndT

rinke

n

¡

Kle

idun

g

¡

Kör

per

und

Ges

undh

eit

¡

Jahr

es-

und

Tage

sabl

auf

¡

Fest

eun

dFe

iern

¡

Kin

dhei

tund

Erw

achs

enw

erde

n

¡

Sch

ule

und

Arb

eits

wel

t

¡

Hob

bys

und

Inte

ress

en

¡

Um

gang

mit

Gel

d

¡

Erle

bnis

seu

ndF

anta

siew

elt

¡

Ged

anke

n,E

mpfi

ndun

gen

und

G

efüh

le

¡

Ein

stel

lung

enu

ndW

erte

¡

Um

wel

tund

Ges

ells

chaf

t

¡

Kul

tur,

Med

ien

und

Lite

ratu

r

¡

Inte

rkul

ture

lleu

ndla

ndes

kund

liche

A

spek

te

¡

(Fic

tiona

l)B

iogr

aphi

es

¡

Sim

ple

(tech

nica

l)de

scrip

tions

¡

Dia

rye

ntrie

s

¡

Dire

ctio

nsa

ndin

stru

ctio

ns

¡

Em

ails

¡

Lette

rs(p

erso

nal,

advi

ce,a

pplic

atio

n)

¡

Mag

azin

ear

ticle

s

¡

Nar

rativ

ere

port

s

¡

Leng

thy

post

card

s

¡

Sta

tem

ents

ofp

erso

nalv

iew

san

d

op

inio

ns

¡

Sto

ries

(cre

ate

ane

ndin

g;g

iven

an

en

ding

–c

reat

ea

stor

y;u

sea

vi

sual

impu

lse

toc

reat

ea

stor

y;

pe

rson

al)

¡

Toc

onve

y

em

otio

ns,

fe

elin

gs

¡

Toin

form

¡

Toc

onvi

nce,

pe

rsua

de

¡

Toe

nter

tain

,

pl

ease

¡

Tok

eep

into

uch

¡

Tod

escr

ibe

¡

Tog

ive

di

rect

ions

and

in

stru

ctio

ns

¡

Sel

f

¡

Oth

ers

E8

Co

nstr

uct

Sp

ace

Tabl

e 1: C

onstr

uct S

pace

for L

ong

Prom

pts


Pro

mp

t

Typ

e

CE

FR

Leve

l

CE

FR

Des

crip

tor

Des

krip

tor

aus

BIS

T-V

O:

Sch

üler

/inn

enk

önn

en…

Top

icA

rea

Text

Typ

esIn

tent

ion/

Pur

po

se

Pri

mar

y

Aud

ienc

e

ShortPrompt

A2+

¡

Can

writ

eab

oute

very

day

as

pect

sof

his

/her

en

viro

nmen

t,e.

g.p

eopl

e,

pl

aces

,ajo

bor

stu

dy

ex

perie

nce

inli

nked

s

ente

nces

.

¡

Can

writ

eve

rys

hort

,bas

ic

de

scrip

tions

ofe

vent

s,p

ast

ac

tiviti

esa

ndp

erso

nal

ex

perie

nces

.

¡

inF

orm

ver

bund

ener

Sät

ze

et

was

übe

rda

sal

ltägl

iche

U

mfe

lds

chre

iben

,wie

z. B

.

üb

erF

amilie

,and

ere

M

ensc

hen,

Ort

e,S

chul

e

¡

Fam

ilieu

ndF

reun

de

¡

Woh

nen

und

Um

gebu

ng

¡

Ess

enu

ndT

rinke

n

¡

Kle

idun

g

¡

Kör

per

und

Ges

undh

eit

¡

Jahr

es-

und

Tage

sabl

auf

¡

Fest

eun

dFe

iern

¡

Kin

dhei

tund

Erw

achs

enw

erde

n

¡

Sch

ule

und

Arb

eits

wel

t

¡

Hob

bys

und

Inte

ress

en

¡

Um

gang

mit

Gel

d

¡

Erle

bnis

seu

ndF

anta

siew

elt

¡

Ged

anke

n,E

mpfi

ndun

gen

und

G

efüh

le

¡

Ein

stel

lung

enu

ndW

erte

¡

Um

wel

tund

Ges

ells

chaf

t

¡

Kul

tur,

Med

ien

und

Lite

ratu

r

¡

Inte

rkul

ture

lleu

ndla

ndes

kund

liche

A

spek

te

¡

(Fic

tiona

l)B

iogr

aphi

es

¡

Sim

ple

(tech

nica

l)de

scrip

tions

¡

Dia

rye

ntrie

s

¡

Dire

ctio

nsa

ndin

stru

ctio

ns

¡

Em

ails

¡

Lette

rs(p

erso

nal,

advi

ce,a

pplic

atio

n)

¡

Mag

azin

ear

ticle

s

¡

Not

es

¡

Not

ices

¡

Pos

tcar

ds

¡

Sta

tem

ents

ofp

erso

nalv

iew

san

d

op

inio

ns

¡

Toc

onve

y

em

otio

ns,

fe

elin

gs

¡

Toin

form

¡

Toc

onvi

nce,

pe

rsua

de

¡

Toe

nter

tain

,

pl

ease

¡

Tok

eep

into

uch

¡

Tod

escr

ibe

¡

Tog

ive

di

rect

ions

and

in

stru

ctio

ns

¡

Sel

f

¡

Oth

ers

A2

¡

Can

writ

esh

ort,

sim

ple

fo

rmul

aic

note

sre

latin

gto

m

atte

rsin

are

aso

f

im

med

iate

nee

d.

¡

Can

writ

esh

ort,

sim

ple

im

agin

ary

biog

raph

ies

and

si

mpl

epo

ems

abou

tpeo

ple.

¡

kurz

e,e

infa

che

Not

izen

und

M

ittei

lung

ens

chre

iben

,die

si

cha

ufu

nmitt

elba

re

B

edür

fnis

seb

ezie

hen

¡

einf

ache

Tex

tez

. B.z

u

B

ildim

puls

eno

der

S

chlü

ssel

wör

tern

(key

wor

ds)

sc

hrei

ben

¡

kurz

e,e

infa

che

Bio

grafi

enu

nd

a

nder

eei

nfac

hefi

ktio

nale

Te

xte

schr

eibe

n

E8

Co

nstr

uct

Sp

ace

Tabl

e 2: C

onstr

uct S

pace

for S

hort

Pro

mpt

s


5.StructureoftheTest

Thetest contains two sections.Section1 consistsof a shortwriting taskwithanexpectedresponseof40to70words.Section2consistsofalongwritingtaskwithanexpectedresponseof120to180words.

ThetwotaskswillbeassessedseparatelyonthebasisofthefourdimensionsoftheWritingRatingScale.

6.TimeAllocation

Totaltestingtimeavailable:45minutes.Timeforadministrationatthebeginning(handingouttestbooklets):5minutes.Time for administration at the end (word count and collecting test booklets): 5minutes.Workingtime:35minutes.Theshorttaskshouldtakeabout10minutes,thelongtaskabout20,with5minutesforrevision.

7.ItemFormats

Thecandidates’scriptswillbehandwrittenonthepagesprovidedinthetestbook-let.Thewritingtaskisguidedbypromptsthatensurethatthecandidatesproduceenoughlanguagethatmakesreliableandvalidassessmentpossible.

Thepromptsmaycontainblackandwhitepicturesordrawings.TheyneedtobeappropriatefortheageandatalanguagelevelnohigherthanA2.Inputtextsshouldbe authentic, if at allpossible, andas longasnecessary to contextualise the task.Ideally,theyshouldnotbelongerthan50words(excludingcontentpoints).

Thepromptsthataredevelopedaretobefreeofstereotypes.Theyoffertheoppor-tunity towrite fromexperience,butaredesignednot to intrudeon the students’personalfeelings.

Promptsneedtostatethereasonforwriting,theintendedaudienceandtherequiredtext type.Theworking time available and thenumber ofwords for the requiredlengthofthetextswillbeindicatedintheinstructions.

8.LanguageLevelforInstructionsandPrompts

All relevant instructions andprompts are inEnglishwith additional informationgivenbythetestadministrator inGerman.However,theymustbeformulatedinlanguage that is well within reach of the candidates’ expected language level andtherefore easily understandable for all test takers.Test takers must not be put ata disadvantagebecause theyhavedifficultyunderstanding the instructions or theprompts.ThereadingcompetenceexpectedisCEFRlevelA2.

9.AssessmentwithWritingRatingScale

InE8testingthemostsignificantcompetencesneededforwritingareidentifiedforassessmentpurposes,thefirstandforemostofwhichisthecommunicativecompe-tencedemonstratedinanappropriateresponsetothetask.Inpracticaltermsthismeansthatallexpectedcontentpointsofthepromptaretobeclearlyandmeaning-fullymentionedbythetesttakers.Forthehigherbands,elaborationofsomecontentpointsisrequired.Thesecondistheabilitytoproducefluenttextbyusingadequatedevicestocreatecoherenceandcohesiononsentence,paragraphandtextlevel.The

Tabl

e 2: C

onstr

uct S

pace

for S

hort

Pro

mpt

s


thirdisagoodknowledgeofarangeofgrammaticalstructuresandtheabilitytousethemaccurately,andthefourthisthechoiceofvocabularythathasacertainrange,isaccurateandrelevanttothecontent.

Thereforethetestisdesignedtoelicitlanguagesamplesthatallowthecandidatestobeassessedinfourareas:TaskAchievement,CoherenceandCohesion,Grammar,andVocabulary. The two tasks are assessed separately by trained raters, using ananalytic rating scalebasedonthese fourdimensions.Multiple-ratinganddouble-ratingofasufficientlylargesampleofscriptsensurereliability.Differencesinraterseverityareadjustedforintheprocessofmulti-facetedRaschanalysis.

Pages25–27includethefourdimensionsoftheanalyticratingscaleforwritingandmoredetailedscalesforTaskAchievement.


TaskAchievement CoherenceandCohesion Grammar Vocabulary

7

¡ completetask achievement

¡ cohesiononboth sentenceandparagraph levelusingalimited numberofcohesivedevices¡ clearandcoherenttext

¡ goodrangeofstructures¡ relativelyhighdegreeof grammaticalcontroland fewinaccuracieswhichdo notimpaircommunication¡ messageclear

¡ goodrangeofvocabulary communicatingclearideas¡ generallyaccurate vocabulary¡ formulationssometimes variedtoavoidrepetition

6

5

¡ goodtask achievement

¡ goodsentencelevel cohesionasalinear sequenceonasimple level¡ someparagraphlevel coherenceandcohesion¡ fairlyclearandcoherent text

¡ generallysufficientrange ofstructures¡ occasionalinaccuracies whichcanimpair communication¡ messageclear

¡ sufficientrangeof vocabularycommunicating clearideas¡ occasionallyinaccurate vocabulary¡ majorerrorspossible whenexpressingmore complexideas

4

3

¡ sufficienttask achievement

¡ somesimplesentence levelcohesionusing simpleconnectorslike ‘and’,‘but’and‘because’¡ frequentlackof coherenceandcohesion onparagraphlevel¡ textoftenlacksclarityand coherence

¡ limitedrangeofsimple structures¡ frequentlyinaccuratewith basicmistakes,generally withoutcausingbreak- downofcommunication¡ messageusuallyclear

¡ limitedrangeof vocabularymostly communicatingclear ideas¡ frequentlyinaccurate vocabularycontrollinga narrowlexicalrepertoire¡ tendencytousephrases fromtheprompt

2

1

¡ sometask achievement

¡ basiclinearconnectors (‘and’,‘then’)onwordor wordgrouplevel¡ textnotcoherent

¡ extremelylimitedrangeof simplestructures¡ limitedcontrolcausing frequentbreakdownof communication¡ messageseldomclear

¡ extremelylimitedrangeof vocabularycommunicating fewclearideas¡ mostlyinaccurate vocabularyfrequently causingbreakdownof communication¡ severalchunksliftedfrom theprompt

0¡ notask achievement

¡ notenoughassessable language

• notenoughassessable language

¡ notenoughassessable language

WritingRatingScale(July2011)


Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/ inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngrading byoneband(twobandsifrequirementsarenotmetatall).

Textlength:120–180words

Below-lengthanswers • Anythingbelow110wordswillbepenalised(downgradingbyoneband).• Fewerthan80words-downgradingbytwobands(ifthegeneralscoreisband3or above;otherwise,downgradebyoneband).• 50–79words:Assessmentisconfinedtobands1and2.• Answerscontainingfewerthan50wordsreceive0.

Over-lengthanswers• Morethan180words:thewholeanswerisassessed.

ExtendedScales

7¡ completetaskachievementwith

¡ allcontentpointsmentioned&threeormoreofthemelaborated

6

5¡ goodtaskachievementwith

¡ 85%ofthecontentpointsmentioned&twoorthreeelaborated¡ orallcontentpointsmentionedandoneortwoelaborated

4

3¡ sufficienttaskachievementwith

¡ 65%ofthecontentpointsmentioned&oneortwoelaborated¡ orallcontentpointsmentionedwithoutelaboration

2

1¡ sometaskachievementwith

¡ 50%ofthecontentpointsmentioned&noelaboration

0 ¡ notaskachievement

TaskAchievementLongTasks


Texttyperequirements:• Itisexpectedthattexttyperequirementsaremet.Iftheyarenotmet(missing/ inappropriatesalutationorclosingformula;inappropriateregister),thereisdowngrading byoneband(twobandsifrequirementsarenotmetatall).

Textlength:40–70words

Below-lengthanswers• Anythingbelow30wordswillbepenalised(downgradingbyoneband).• Fewerthan30words:Assessmentisconfinedtobands1and2.• Answerscontainingfewerthan20wordsreceive0.

Over-lengthanswers• Morethan80words:thewholeanswerisassessed.

ExtendedScales

7¡ completetaskachievementwith

¡ allcontentpointsmentioned&oneortwoofthemelaborated

6

5¡ goodtaskachievementwith

¡ allcontentpointsmentioned&possiblyoneelaborated

4

3¡ sufficienttaskachievementwith

¡ 70%ofthecontentpointsmentioned&noelaboration¡ or50%ofthecontentpointsmentioned&oneelaborated

2

1¡ sometaskachievementwith

¡ 50%ofthecontentpointsmentioned&noelaboration

0 ¡ notaskachievement

TaskAchievementShortTasks


10.PromptsandPerformanceSampleswithJustifications

Thetaskprototypesbelowaretakenfromthe2007E8writingtest.ItisimportantthatthetasksarestructuredandcontainanumberofcontentpointssothatTaskAchievementcanbemeasured.4

TheinstructionsbelowhavebeenreducedtoaminimumbecausemostthingsareannouncedbythetestadministratorinGermanbeforethetesttakersopentheirtestbooklets.TheinstructionsinEnglisharetoensurethattheykeepthemainpointsinmind,butalsothatlearnerswithafirstlanguageotherthanGermanhavethesamefairchancetodothetask.

10.1LongTask

Instructions

LongPromptfromE8WritingTest2007

10.1.1Script1

4 MorepromptsareavailableontheBIFIEwebsite:http://www.bifie.at/freigegebene-items

Read the instructions carefully and then write your text on the next page.Time: 20 minutesText: 120–180 wordsUse paragraphs.In your text, try not to use language from the task below.

You have just moved to another town/village. Write a letter to your American/English friend in which you tell him/her about your new situation.

Inform him/her about ● your new place of living. ● the reason for moving.Describe ● the town/village you’re living in now (buildings, people,…). ● your new home.Tell him/her about ● the first days of your ‘new life’ (new school, teachers,…). ● how you feel about your new situation.

Dear Bill, how are you? Now I’m living in Vienna. That is in Austria. It is a very big city with nice people. There is also a fun fair called “Prater”. My parents got divorce and so I’m living here with my mother. There are wonderful buildings in this city like the animal park “Schönbrunn” and many castles. I like the river “Donau” very much, because I often go swimming there. My new house is very big and next to it is a forrest. I like that. The first day of my “new life” was not so good. When I came into my class most of the pupils laughed at me but the teacher was nice. I hope you will write back.

Yours, Raphael

(124 words)


Justifications

TaskAchievement 5

Thetextmeets the text type requirements, iswithin theword limit set, andusesan informal register suitable fora letter toa friend.Thecandidateworkshiswaythroughthecontentpoints,onlyjusttouchingonthelastone.HowhefeelsneedstobeinferredfromstatementslikeThe first day of my ‘new life’ was not so good, most of the pupils laughed at me, and the teacher was nice.Alltheothercontentpointsarementioned.Whereas content point 3 shows good elaboration, the elaboration ofcontentpoints1and4islesssuccessful.ThepassageNow I’m living in Vienna. That is in Austria. canbe taken asmentioning contentpoint1 and the following twosentences(It is a very big city with nice people. There is also a fun fair called “Prater”.) canbeseenasanattemptatelaboration.

Withcontentpoint4elaborationisjustasthin.Thestatement My new house is very bigmentionscontentpoint4,butdoesverylittleinthewayofdescribingthenewhome(likeforinstancehowmanyrooms,whathis/herroomlookslike,etc.)andthesentenceadded(next to it is a forrest. I like that.)isnotreallydescriptive.Sothewritercannotbegivenmuchcreditforthisattemptatelaboration.

Thismeetsthedescriptorintheratingscaleforband5:“goodtaskachievementwithallcontentpointsmentionedandoneortwoelaborated.“

CoherenceandCohesion 4

Fromtheverystarttheideasdonotconnectwell.Theintroductoryquestionhow are you?islefthangingtobefollowedbyNow I’m living in Vienna. ThethreeshortsentencesthatfollowaddsomeinformationaboutVienna.Butthenextidea(My parents got divorce)meetsthereaderunpreparedandthereisnolinktosmooththetransition. At this point it becomes clear that the three sentences about Viennashouldactuallyhavebeenmovedtocontentpoint3.Thiswaythefirstunitofthetextcomprisingcontentpoints1and2wouldhaveflowedbetter.Textorganisationisbasedonthesequenceofthecontentpointsprovidedwiththewriter’shandprac-ticallyinvisible.

Thetextiscertainlycohesiveatsentencelevelbuthardlyatparagraphlevel.Movingfromoneideatotheother(tellingaboutthenewplace–thereasonformoving–describingthenewtown)maybeimpliedbytheorderofthecontentpoints,buttheabruptwaythishasbeendoneshowsthewriter’slimitations.

However,thetextis“fairlyclearandcoherent”.Sentencelevelcohesionisgoodandsomebasicconnectorsareusedtodelivera“linearsequence[ofpoints]onasimplelevel”.Thereissomeparagraphlevelcoherence,buttherearenotransitionsorlink-ingdevicesbetweenthevariousideaspresented.Whileband5couldbeconsideredfor thisperformance, the lackofparagraphorganisation leadstodowngradingbyonebandtoband4.

Grammar 5

Atfirstsightthestructuresusedaregenerallysimple,manyofthesentencesareveryshort.Therearetwoidenticalcasesofthepresentprogressive(I’m living),fourusesofis, there is and there are,fourcasesofthepresentsimple(I like [2x], I often go, I hope), threesimplepasttenseforms(was, came, laughed)andonefuture(will write).


Althoughallofthesetenseformsarebasic,theyseemsufficienttoexpressthewriter’sideasandtheyhavebeenusedcorrectly.Moreover,thewriterusescomplexsentencescorrectly,whichhintsatmorecomplexlanguagecompetences.

Thereareactuallytwosubordinateclauses (because, when) andathirdwiththefinalsubordinatorthatomittedin I hope [THAT] you will write back. Moreover, ifonetakesacloselookattheothersentences,wefindgoodpost-modificationofnounphrases(big city WITH…, fun fair CALLED…, buildings…LIKE…).Eventheco-ordinateclauseshaveatwisttothem…and SO…, and NEXT TO IT….Sothereisdefinitelysufficientcomplexitytojustifyband5.

Vocabulary 5

Thecandidatehasasufficientrangeofvocabularytoexpresshimself.Ontheonehand,thereisratherunexpectedvocabularyused(nearly)correctly(got divorce, fun fair, laugh at)asevidenceofagoodrangeofvocabulary,ontheother,thereisalsorepetitionofverysimpleexpressionssuchasnice, big and like,andtherearesomeoccasional inaccuracies (got divorce, forrest). The majority of words belong to themost frequently used basic English vocabulary, but the ideas communicated arealwaysclear.

Agoodrangeofvocabularycanonlyrarelybeseen,asmostpartsofthetextdisplayasufficientrange.Whatisdefinitelylackingistheabilitytovaryformulationstoavoidrepetition(band7),sothetextisband5.

10.1.2Script2

Justifications

TaskAchievement 4

Thetextmentionsallcontentpointswiththeexceptionofcontentpoint2,wherereasonsformovingshouldbestated(=85%).Thereissomeelaborationofcontentpoint4bydescribingthesizeofthegardenandcontentpoint5bysupplyingthereaderwithsomeadditionalinformationabouthisnewfriends(they speak very good english)andhisachievementsatschool(lotsofEnglishlessonsandhimdoingwellinEnglish),althoughthisonlyimplicitlyreferstothecontentpoint“thefirstdaysofyour'newlife'”.Contentpoint6hasbeenliftedfromthepromptaddingthewordgood.

Ascontentpoints1and6haveonlybeendealtwithinanextremelybasicwaybymerelymentioningSalzburgand liftingaphrase fromthepromptandtheelabo-rationofcontentpoints4and5isratherweak,adowngradetoband4isthecon-sequence.

Hi Steven!In Salzburg it is very cool and I’m living in a small flat with 5 rooms two bedrooms a kitchen a livingroom and a bathroom. Our garden is not so big, but big enough for us. The building very beautiful and it give no skyscrapers and it is very hot. I always go in the garden and I lie in the sun. The people are very funny and they accept that I speak english. I have got two new friends and they speak very good english. The teachers are very good and we have a lot of english and I’m the best one, but in Deutsch I’m very bad. I feel very good with my new situation and I wish all my old friends and the teachers a good luck for the next time and I hope you always wish me a good next time.Yours Olav! (149 words)



Textorganisationisquitelowwithsimpleadditionasthedominatingstructuringprinciple.Thesimplest connectorandoccurswithundue frequency,proving thatideasaremostlystrungtogetherwithoutexpressinglogicalrelations.Apartfromthesecondsentence (Our garden is not so big, but big enough for us)thetextdoesnotreadwellbecausesomesentencesthatfolloweachotherhavelittleornoconnectionatcontentlevel,sothetextlackscoherence.Somechunksoflanguagethathavelittleincommonareoftenjoinedinonesentence,e.g.The building very beautiful and it give no skyscrapers and it is very hot.

Therefore,thistextischaracterisedbyanoticeablelackofclarityandcoherenceandsomeratherbasicsentencelevelcohesion.Thiswouldpointtowardsaweakband3.Astherearenoparagraphs,thetexthastobedowngradedtoband2.

Grammar 3

Thewriteruses aquite limited rangeof simple structures correctly, repeating thesamebasicpatternwith little variation.Heonlyusespresent tense structures, al-thoughcontentpoints2and5would invite theuseofpast tense.Therefore, therangeofstructurescannotbeconsideredsufficientforthepurposesofthetask.EvenwithinthenarrowframeofpresenttensesentencestructuresthereisinappropriateuseofthecontinuousforminI’m living in a small flat. Thesimplemessageisusuallyclear,althoughanerrorsuchasit give no skyscrapers causesbreakdownofcommuni-cation.Similarly,thephrasethe building very beautiful leavesusundecidedwhetheritshouldrefertothehousewherethewriterlivesorthebuildingsinSalzburg.Allthiswouldsuggestaveryweakband3,which,however,issupportedbytherelativelyhighdegreeofcorrectness.

Vocabulary 4

Thetextshowssuccessfulcontrolofalimitedrangeofvocabulary,withsomegoodphrasesstickingoutsuchasbig enough for us, skyscrapers, orthey accept that I speak english. Thesimplevocabularyusedinthefirstpartofthetextcommunicatesmostlyclear ideas,but in the last sentence thewriter seems tobeattempting toomuch,leavingthesafearea,andthisresultsinseveralbreakdowns(I wish them a good luck; L1:for the next time; I hope you always wish me a good next time)demonstratingthelimitations,asdoestheuseofDeutschforthesubjectGerman.Thenarrowlexicalrepertoireandalsothetendencytoliftphrasesfromtheprompt(I feel very good with my new situation)wouldindicateaband3,buttheoccasionalneatexpressionandthefactthatsevereproblemsappearonlywhentryingtoexpressamorecomplextrainofthought(toleratedatband5)justifyaweakband4.


10.2 Short Task

Instructions

Short prompt from E8 Writing Test 2007

10.2.1 Script 3

Justifications

Task Achievement 7

The register and the layout are clear indications that this text is an email. The salutation and closing formulas are most appropriate and for these reasons text type requirements are perfectly met. All content points have been mentioned and there is elaboration of content point 1 (It was so great!, I liked the games we played, the food was excellent) and content point 3 as the candidate makes enquiries about an up coming event in the near future (I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too?), so we have complete task achievement – band 7.

Coherence and Cohesion 7

The text admirably incorporates qualities of spoken English (Let’s talk about, Oh and, I nearly forgot, Okay I have to), which one would expect in an informal email, and which make it flow well. A number of cohesive devices are used to connect groups of sentences together very well, such as lexical cohesion (party-it-party), conjunctions (because-and), backward and forward referencing (we played-tell your mum that), and there is evidence of good linear sequencing of points making it a clear and coherent text, pointing it towards band 7. However, there are two abrupt changes in the linear sequence of the text What are you going to do and I nearly forgot, but as paragraphs are not expected in short texts, it remains a band 7.

Your friend’s birthday party was a few days ago. Write an email to tell him/her that you liked the party.

• Tell him/her why you liked the party.• Tell him/her what you liked best.• Ask your friend when you are going to meet again.• Suggest something for the next weekend.

Dear Daisy, how are you? Let’s talk about your party. It was so great! I liked the party best, because Lukas was there. And I liked the games we played. Oh and tell you mum, that the food was excellent! What are you going to do on Sunday? Maybe we can go to cinema or swimming. Tell me please if you have time. I nearly forgot it: Tanja’s birthday party is in two weeks, she invited me, are you invited too? Okay I have to help my mum with dinner.Love you big kissYours,Aida (95 words)

Read the instructions carefully and then write your text on the next page.Time: 10 minutesText: 40–70 words In your text, try not to use language from the task below.


Grammar 7

Thereisarelativelyhighdegreeofgrammaticalcontrolwithonlyoneslightslipinaccuracy.However,theomissionofthedefinitearticleinthephrasego to cinema doesnotimpaircommunicationandthemessagethroughoutthetextisclear,suggestingband7.Thecandidate’sgooduseofthepresent,past,going-to-future,Saxongeni-tive,andasubordinateclauseproveshe/sheisabletoaddressallthelanguagefunc-tionsincludedintheprompt:toinform,toask(how are you?, What are you going to do on Sunday? are you invited too?),andtosuggest (Maybe we can go to cinema).Thisindicatesthegoodrangeofstructuresthecandidateisabletouseaccurately,thussupportingastrongband7.TheverycasualsentenceOkay I have to help my mum with dinnerwithnear-nativeomissionofthedefinitearticle (with dinner),isfurtherevidencethatclearlypointstoaband7.

Vocabulary 7

The vocabulary elicited by the prompt points towards band 7. Not only does itcontainagoodvarietyofappropriateandaccuratecontentwordsexcellent, invite, dinner, but also many collocations that are equally appropriate and accurate andlenda certainnaturalness to the text (the food was excellent; what...going to do on Sunday; have time; in two weeks; I nearly forgot; love you big kiss).Anotherindicatorforband7isthecandidate’schoiceofwordswhichenablehertogethermessageacrossveryclearlythroughoutthetext.Someofthephrases(Let’s talk about; Oh and tell; I nearly forgot; Okay I have to)indicateacertainairof'chattiness'tothetext.Furthermore,theuseofLove you big kiss, asanalternativeoradditiontothecommonclosing lineYours, which exemplifies how the candidate canvary formulations toavoidrepetition,isadditionalevidencethatthisisaband7.

10.2.2Script4

Justifications

TaskAchievement 4

Thetextclearlyfollowstheorganisationofthecontentpoints(liked the party because it was cool/liked the girls and waterpipes best, do such a party tomorrow again?, go to the city …).Although thefirst contentpoint ishandled verybriefly and contentpoint3featuresasanindirectrequestonecanstillsaythatallcontentpointshavebeenmentioned,whichhintstowardsband5.Bands6or7cannotbetakenintoconsiderationbecausethereisnoelaboration.Althoughcontentpoint4consistsoftwosentences,theintroductoryquestiontocontentpoint4isnothingelsethanarepetitionofthequestionincontentpoint2(Can we do such a party again…?, … we can do it again, or?)andtheactualcontentpoint4iscoveredbythefollowingsuggestion(I have a better idea …).

Astheclosingformulaismissing,texttyperequirementsareonlypartlymet(saluta-tion:Hey Kevin!!). Thisleadstodowngradingbyonebandtoband4.

Hey Kevin!!! I’ve liked your party because it was very cool. What I liked most were all the nice girls and all the nice waterpipes. Can we do such a party again tomorrow? It would be very nice. But please, buy more waterpipes, and more grass! And next week-end, we can do it again, or? I have a better idea, we can go to the city and chillout at a concert.(65 words)



Thefirstpartofthetext(sentences1–5)showsgoodsentencelevelcohesionthroughbackwardandforwardreferencing,thematicfrontingandlexicalrepetition(party- it; what I liked most; such a party; it would be ...; waterpipes).Thispart,thoughnotmarked as such in the layout, even shows some good paragraph level coherencewithinthenarrowscopeofashorttext.Theconcludingpart(And next weekend....)providesmorebackreferencinginitforparty,andthefollowingsuggestionisthematicallylinkedtothepreviousquestionthroughI have a better idea.

Therefore,wecanascertain“goodsentence level cohesion”and“somegoodpara-graphlevelcohesion”,whichjustifiesband5.

Grammar 5

Thetextclearlyshowsasufficientrangeofstructurestofulfilthetask,whichrequiresthedescriptionofapastevent,thequestionforafutureevent,andasuggestion.Thepasteventiscorrectlydescribed(was very cool; what I liked most were),thequestioniscorrectlyphrasedandfollowedbytwostatementsmakingcorrectuseofasubjunc-tive(would)andapoliteimperative(But please, buy more…). Thetextalsofeaturesacomparative,thecorrectuseofamodalverbform,andasubordinateclause.

Thetextisgenerallyveryaccurate;themistakesindescribingapastevent(I’ve liked) andtheL1interferenceintheuseofor?insteadofatagquestiondonotimpaircom-municationanditisclearwhatthewriterwantstosay.Hence,band5isappropriate.

Vocabulary 4

Thecandidateshowssufficientlexicalrangetofulfilthetaskandcommunicatehisideas,whichwouldhinttowardsband5.However,indoingsohemakesuseofverysimplevocabulary: very cool, all the nice girls, go to the city, concert, was/were, buy, more, againetc.Theonlywordsstickingoutare waterpipes, grass, such a and to chill out.Althoughthewordswaterandpipeshavebeencombinedandaresemantically‘new’,thewordsthemselvesarestillextremelybasic,asis grass.

Thisleavesuswithas littleas to chill out and such aastheonly lexical itemsthatwouldgobeyondalimitedrange.

Ontheotherhand,vocabularyisaccuratewithto do a partyastheonlyincorrectuse,whichdoesnotcreateanymisunderstanding.Takingintoconsiderationthatthewriterdoesnottakeanylexicalrisksandtheitemsusedaresimple,limitedinrangeandrepetitiveadowngradingfromband5toband4isappropriate.

10.2.3Script5

My friend have at 7.5. birthday. The birthday-party was very good. We had a lot of fun on the party. We play playstation and we went play football but the best was that we are ate pizza. We go at the weekend to a football match.(46 words)


Justifications

TaskAchievement 05

Thecandidatetriestoworkhis/herwaythrough(mostof )thecontentpoints.Con-tentpoint1ismentionedasthestudentwritesthatthe party was very goodandthattheyhad a lot of fun. Thenhe/shementionsafewthingsthattheydidattheparty.Itcanbeassumedthat theseactivitiesare thereasonswhyhe/she likedtheparty.Contentpoint2isshortbutok(the best was that we are ate pizza). Contentpoint3ismissingcompletely,andcontentpoint4isnotrecognisableasasuggestion,thereforenotmentionedcorrectly (We go at the weekend to a football match).

Sothereissometaskachievementwith50%ofthecontentpointsmentionedbutnoelaboration,whichpointstowardsband1.

However, thetextdoesnotmeettexttyperequirementsatall.Thecandidatewasaskedtowriteanemailtoafriend.Thefirstthingthatisstrikinghereisthatthereisnosalutationandnoclosingformula.Allinall,thistextreadslikeareportratherthananemailmessage.Asaconsequence, the texthas tobedowngradedby twobands,whichmeansitistobeplacedatband0.


Atfirstsightcoherenceandcohesioninthistextseemtoberatherpoor.However,thereissomesimplesentencelevelcohesionconnectingtheideasofcontentpoints1and2(friend’sbirthday–thebirthdayparty–funattheparty–activitiesattheparty). Wefindoneinstanceofacohesivedefinitearticle(theparty)andthelogicalconnectionbetweenWe had a lot of fun on the party. We play playstation and [...] isactuallyfairlyobvious.Moreover,inonesentencesomebasicconnectorsareusedfairlysuccessfully (and inordertolinktwomainclauses;correctuseofbut+that-clause).

Thelastsentencebreaksoffthecoherencethatwasthereanddisruptstheminimalqualityofflowthatthefirstpartofthetexthas.That is why coherence and cohesion is clearly better than band 1, but not goodenoughforband3,hence(aweak)band2.

Grammar 1

Thetextshowsanextremelylimitedrangeofsimplestructures.Mostsentencesfol-lowaverybasicsubject-predicate-objectpattern.

Thetaskbasicallyrequiresthedescriptionofapastevent,thequestionforafutureevent,andasuggestion,soaltogetherrathersimplestructures.However,evensomesimplestructuresareusedincorrectly (we went play football; we are ate pizza). Whatismore,thereisnoquestion,nophrasethatindicatesasuggestion,andeventheverybasicverbformstoindicateapresent,pastorfuturetimeaspectarequiteoftenusedincorrectly (my friend have; we play playstationfordescribingapastevent;We go at the weekend to a football match).

5 AsbandzeroforTaskAchievementhasnotbeenreacheddirectly,butafterdowngrading,theotherdimensionsareassessedwithoutanyrestrictions.Script6demonstratesthatwhenTaskAchievementisaclearzerofromtheoutset,theotherdimensionsarenotratedasthereisnotenoughassessablelanguage.


Inaddition,thereareexamplesofthemisuseofprepositionsinverybasicphrases(on the party; at 7.5.),andwrongwordorderinverybasicsentencestructures (My friend have at 7.5. birthday; We go at the weekend to a football match).

Althoughthereisnobreakdownofcommunicationandthemessageisusuallyclear,theextremelylimitedrangeandincorrectuseofsimplestructuresmakethistextaperformanceatband1.

Vocabulary 1

Thetext showsanextremely limited rangeofvocabulary,usingverybasicwords,e.g.friend, birthday, birthday party(mentionedintheprompt),a lot of fun, football match, pizza, andformsof have, be, play and go.

Althoughthecandidateisabletocommunicatehis/herverysimpleideassuccessfullyand(mostly)accurately,theextremelylimitedrangeofvocabularyoverrulesaccuracyandthismakestheperformanceband1.

10.2.4Script6

Justifications

TaskAchievement 0

Thereobviouslyisnotaskachievementasthecandidatehasmerelycopiedthegiveninputtextandthecontentpointsinsteadofdoingwhattheysaid.Theonlyadditionmadebythewriterisminimalandbearsnorelationtothetaskset.

Onthegroundsoftaskachievementbeingbandzero,theotherthreedimensionsarenotassessableaswecouldonlyassessthelanguageusedinthetaskdescription,butnotanyofthecandidate’scompetences.


Noenoughassessablelanguageintermsofcoherenceandcohesion.

Grammar 0

Toolittleindependentlyproducedlanguagetoallowassessment.

Vocabulary 0

Toolittleindependentlyproducedlanguagetoallowassessment.

My friend’s birthday party was a few days ago. Write an email to tell ihm that you liked the party. Tell him why you liked the party. Tell him what you liked best. Ask your friend when you are going to meet again.Suggest something for the next weekend.My best friend’s. Name von my best friend’s is …(58 words)


ScaleInterpretations

Intheratertrainingcoursesoverthepastthreeyears ithasbecomeclearthattheratingscaleasitstandsisbynomeansself-explanatoryandreadyforgeneraluse.Therefore,somecommentsonhowtoreadandinterpretthescaleareaddedhere.

ScaleInterpretation–TaskAchievement

The scale on Task Achievement has no direct correlation with the CEFR andassessesthecontentcomponentsofatextandtexttyperequirements.Thewaythedescriptorsareformulatedleavesroomfordifferinginterpretations,whichwillleadtodivergingassessments.Inordertoreachtheaimsofthetrainingcourseandthestandardisationmeetings,acommonunderstandingofwhatalltheelementsinthescalesmeanandhowtheyrelate toeachother is required.To improve inter-raterreliabilityweneedtoclarifythekeyterms.

CONTENTThefirst issuethatseemssimpleenoughistodecidewhetheracontentpointhasbeenmentioned ina scriptornot.Whereas this isquite straightforward inmostcases, there is room for confusion when, for instance, some key words from thepromptappearinthetext,butthelanguagearoundthemdoesnotmakemuchsense.Asitsaysinthetestspecificationsthat“allexpectedcontentpointsofthepromptaretobeclearly and meaningfullymentioned”(p.23),suchacontentpointwouldnotbeconsideredasbeingmentioned.

WhatratertraineeshavefoundmostchallengingandconfusingwhenassessingTaskAchievementisdistinguishingbetweenmentioningacontentpointandelaboratingit.Withreferencetothelongpromptonp.28thefollowingexample,takenfromoneofthescriptsusedinthetrainingsessions,canservetodemonstratewhat“men-tioningacontentpoint”means:We live now in New York near the Central Park, we moved because my mum had not found a job.Inthissentence,contentpoints1and2arementioned.Thetextgoeson:But in New York my mum has a good job. Thissentenceextendspoint2alittle,butasitislittlemorethanareformulationoftheprevioussentence,itcannotbeseenaselaboration.

Elaborationofthispointcouldhavebeensomethinglikethis:In New York she works as a secretary in a bank on the 35th floor of a high building in Manhattan and is quite happy. Or:In New York she sells pancakes in the streets, and she is happy.Goodelabo-rationtheninvolvestheintroductionofanewidea,arealextensionofwhathasbeensaidbefore.

Lesssuccessfulelaborationissomethingyourecognisewhenyouseeit.Lookatthisexample–Contentpoint5ofthelongpromptonp.28:“Tellhim/heraboutthefirstdaysofyour‘newlife’(newschool,teachers,…)”: I have new friends but you are forever my best friend. The new school is very big. The teachers are sometimes unfriendly. And the school colleagues are not polite.

Thefirstsentencementionsnewandoldfriends.Thenitmovesabruptlytothecuesfromthepromptandaddssomesimplewordstoeach.Finally,thereisanewsen-tencebasedonthesamepattern.–Thereissomeelaborationhere,nodoubt,butitisnotverygood.Thiswillbereflectedintheassessment.

Whereas on one level we can assess Task Achievement quantitatively by simplycountingthecontentpointsmentionedandelaboratedrespectively,thediscussion


abovemakesitclearthat,inadditiontothis,thereisalsoaqualitativecomponenttobeconsidered.ThefirstquestionisHow many?, butthesecondisHow good?.

TEXTS AT BAND ZEROInthefollowingcasesthereisnoassessmentinanyofthefourdimensions:

�� Textsthatdonotdealwiththegiventopicandthecontentpointslisted.�� Textsthatareextremelyrude,sexist,racist,orpropagatingviolence.�� Textsthatshowanattemptatdealingwiththetopicbutdonotcontainenoughassessable language, i.e. fewer than50words in long texts and fewer than20wordsinshorttexts.

Textsthatareplacedatbandzeroduetodowngrading(texttyperequirementsand/ortextlength)areassessedintheotherthreedimensions.

TEXT TYPE REQUIREMENTSAnothertermintheTaskAchievementscale thathas invitedfrequentdiscussionsin the training sessions is text type requirements. The issue, however, is greatlysimplifiedbythecontextofuse,whichistestedatE8level.Thissimplymeansthatinthegiventestsituationthereislittleroomforstylisticvariationonthepartofthetest takers. Inmostcasesan informalregister is theonlyonetheyhaveaccess to,andtesttakersarenotexpectedtointroducestylisticdifferencesrelatedtoparticulartexttypeslikemagazinearticles,reports,diaries,letters,oremails.Testtakerswillbeusingamoreorlessinformalstyleinalltheirtextsandtheonlythingtheyneedtoknowishowtoopenandclosealetteroranemail.Inaratermeetingithasbeendecidedtoconsideremailsasslightlymoreinformalthanletters,butstillrequiringsomekindofsalutationandclosingformula.

Inpracticalterms–andinthecontextofE8testing–thismeansthatmeetingoftexttyperequirementsisconsideredagivenrequirementsothetesttakersdonotgetanybonusforit.Itisonlyincaseofproblemsinthisareathatwetakethisintoconsid-erationandreactbydowngradingonebandortwobandsrespectively.Thefollowingguidelineshavebeendiscussedandagreedoninpastratermeetings:

�� Salutation AND/OR closing formula missing or wrong – downgrade by oneband.�� Seriousregister/styleproblems–downgradebyoneband.

TEXT LENGTHTextlengthisarelatedissue,whichhasbeensetdowninsufficientdetailasafoot-noteintheTaskAchievementscales.Themainpointisthatover-lengthtextsarenotpenalisedwhereastextsthataresignificantlybelowtherequestednumberofwords(110/80and30respectively)aredowngraded.Thisisbasedontheassumptionthatawriterwhoonlydeliverstwothirdsorlessofthelengthrequiredwillhaveseriousproblemstoproduceasubstantialtext.

Thebaselinetestingof2009hasshownthattextswithintherangeof80–109wordsrarelygetaprovisionalscorethatishigherthanband4,thosewithintherangeof50–79wordsgetnohigher scores thanband3.At these lowperformance levels,however,aparticularproblemhasarisen.If,forinstance,a“longtext”ofonly78wordsisprovisionallyplacedatband2,itwillbedowngradedbytwobandsfortextlength,sothefinalscorewouldbezero.Inacaselikethis(withthewordcountsoclose)ithasbeendecidedtodowngradeonlybyonebandfortextlengthsoastorecogniseoveralltaskachievementbyplacementinthelowestband.


ScaleInterpretation–CoherenceandCohesion

Coherenceisaqualitycriterionthatreferstothelogicalarrangementofideasandargumentswithinatext.Inawell-written–thatis“coherent”–textthewritersuc-cessfullyarrangeshis/hersentencestoachieveapurpose,e.g.toreflectthechrono-logicalsequenceofeventsortodevelopaconvincinglineofargument.Acoherenttextmakesiteasyforthereadertofollowthewriter’strainofthoughtsothatthereisnoneedtostopandrereadinordertoestablishmeaningasideasandargumentsflowsmoothlyandlogically.Alesscoherenttext,however,impairsreadabilityandappearsjumpy.

Thetermcohesionrelatestotherelationshipsbetweenelementsofatext,whichisthewaywords,wordgroupsandindividualsentencesarelinked.Thereareseveralwaysinwhichcohesioncanbeestablished.Simplesentencescanbeconnectedbyusinglinkingwordssuchasand, but or because. Forexample,thesequenceMy holi-day was a disaster. It rained almost every day. canbereformulatedasMy holiday was a disaster because it rained almost every day. AnothersolutionwouldbekeepingthetwosentencesbutlinkingthembysayingIt rained almost every day. Therefore my holiday was a disaster. SomesuchcohesivedevicesthatwemayexpectwriterstouseatlevelA2/B1are:

Addition and,or,alsoTime when,after,before,Result so,thereforeContrast ontheonehand–ontheotherhand,althoughReason because,asExemplification forexampleSequence first,then,next,finally

Wewill,however,havetobearinmindthatatextcanbecoherentevenifveryfewofthesecohesivedevicesareusedandthat,ontheotherhand,thefrequentuseofcohesivedevicesdoesnotnecessarilyturnanincoherenttextintoacoherentone.

Othertechniquestomakeatextappearcohesivearereferencesbytheuseofpersonalpronouns,possessives,demonstratives,andcomparatives.Ataverysimplelevel,inthetwosentencesMy best friend is Michael. He is in the same class as I. cohesionisrealisedbyusingthepersonalpronoun ‘he’insteadofrepeatingthenameMichael. Similarly,inapassagesuchas My sister has the big room in the house. Mine is a lot smaller. thepossessiveminereferstotheroomintheprevioussentence,thuslinkingthetwosentences.Demonstrativescanservethesamepurpose.In I got a new camera for my birthday. That was my best present ever. thewordthatreferstothecamerathewritergot,therebylinkingthetwosentencessuccessfully.

Sentencescanalsobeconnectedbysubstitutingoneormorewordsinasentence.InWe have a lot of field trips in our school. The nicest one was to Schönbrunn Zoo. thewriterhasreplacedfield tripsbyoneintheimmediatelyfollowingsentence.In Girls are better at English. Everybody thinks so. thewordsorepresentsthewholeideathatgirlsaredoingbetteratlanguages.Aparticularlysuccessfulwayofestablishingcohe-sioninatextistheuseoflexicalchainsasexemplifiedinthefollowingtextpassage:

When I think of clothing I would say that T-shirts with crazy designs like dots, squares, skulls are definitely in. All my friends are wearing that and they think it’s the latest craze! This year wearing the ‘right’ shoes like ‘Converse’ or ‘Vans’ is very important. Everybody loves to wear them because it’s a must-have!


Inthefirstsentencethewriterusesthephrase definitely intodescribecertainkindsoffashionableT-shirts.Inthefollowingsentencethisideaiscontinuedbyusingthephrasethe latest craze. Attheendoftheparagraphthisconceptofafashionproductis reformulated as it’s a must-have. This establishes a lexical chain that binds thesentencestogetherandestablishesasmoothflowofideasintheparagraph.

IntheCEFR,coherenceandcohesion isanaspectof thepragmaticcompetencesof a language user. The discourse competences relevant for writing that are dealtwith in theCEFRare“flexibility tocircumstances“, “thematicdevelopment“and“coherenceandcohesion“.AsthelatteristhequalitycriterionthatisofparticularrelevancewithintherangeofA2/earlyB1writers,itistheonethatisrepresentedintheE8AssessmentScale.IntheE8ScaleforcoherenceandcohesiontheCEFR(seep.125)movesfromtheverybasicA1skillofbeingabletolinkwordswithlinearconnectorssuchasandorthentoA2,whichmeansalsosuccessfullyusingconnectorsthatexpressreason(because)andcontrast(but)tolinkwordsorwordgroups.A2+includestheabilitytousethesemostfrequentconnectorstodescribesomethingasa“listofpoints“,whereasonelevelfurtherupatB1theloosely“connectedlistofpointshasbecomeafullyconnectedlinearsequenceofpoints”.

OurE8AssessmentScale forwriting includes the aspects of both coherence andcohesion.Regardingcoherenceweexpectatexttobeessentiallyclearinitsmessageandcoherentatbands5to7,butacceptsomeamountofvaguenessandambiguityinband5.Band3 texts are characterisedby frequently incoherent text elements,noticeablyimpairingclarityandreadability,whileband1textsarenotcoherentatallandconsistofmostlydisconnectedchunksoflanguage.Insuchband1textsweonlyfindthemostbasic linearconnectorssuchasandorthenascohesivedevicesonwordgrouplevel,whileband3textsshouldalreadyshowsimplesentencelevelcohesionwithawiderrangeofconnectors.Aband3writerisabletolinksentencessuccessfullyusing simple connectors,butusually fails toproduce longer stretchesofconnectedlanguageatparagraphlevel,makingatextappearasachoppylistofpointsratherthanalongerconnectedsequence.Fromband5upwecandemandthislongerconnectedsequenceofsentences,withthewriterbeingabletolinksen-tencesintoclearparagraphs.Atband5wewanttoseethisabilityreflectedinatleastsomepartsofthetexts,whileatband7thewholeofthetextshouldreflectgoodsentenceaswellasparagraphlevelcohesion.Atthetopband7anexpertwriterwillprobablynotonlymanage to link sentences smoothly and logically toproduce acoherentparagraph,butmightalsoalreadyestablishlinksfromparagraphtopara-graph.Needlesstosay,thedegreeinwhichhandlingsuchissuesofcoherenceandcohesioncanbemasteredbyatesttakeralsodependsonthecomplexityoftheideasputforwardandmayhavetobetakenintoaccountbytherater.Themorecomplexandunexpectedideasthereareinatext,themorewehavetoacceptsomejumpinessinthewaytheyarepresented.

Thecontentpointsinthepromptwillalreadysuggestaparagraphorganisationtothewriter,butitisfinallythedecisionofthetesttakerhowhe/shechoosestoorgan-isehis/hertext.Theabilitytostructureatextofaround150wordsintomeaningfulparagraphs is an important skill that we expect test takers to demonstrate in theE8WritingTest. In long textswe expectparagraphs fromband3up, anda lackofindentationorvisualmarkingofparagraphswillresultindowngradingthetextbyoneband.Asequenceofindividualsentencesmarkedasaparagraphcannotbeacceptedassuccessfulparagraphingifthesentencesarearrangedinahaphazardandrandomway showingno connectionwhatsoever.The same applies toparagraphsconsistingofonesentenceonly.Inshorttextscoherenceandcohesionisgenerallymoredifficulttodemonstrate.Paragraphsarenotmandatoryand,ifusedtogoodeffect,couldbeconsideredareasonforupgradingthetext.


ScaleInterpretation–Grammar

TheScaleforGrammarcomprisesdescriptorsforrange,control,andtheclarityofthemessage.Therefore,theratersevaluatethetesttakers’abilitytomakeuseofarangeofgrammaticalstructures,theleveloftheiraccuracyaswellastheirimpactonthemessage.Thefocusisongrammaticalformsthatcreatemeaningandthatarereasonablycorrecttoaccomplishsuccessfulcommunication.

ShorttasksaredesignedtobeA2tasksandtherangeofgrammaticalstructuresthatis likely tobeelicited in such tasks comprises structures typicallymasteredatA2level.6LongtasksshouldhavethepotentialtoproduceB1languageand,asaconse-quence,alsogrammaticalstructuresrepresentativeofB1level.7

THE CONCEPT OF GRAMMATICAL RANGEGrammaticalrangereferstothevarietyofgrammaticalstructuresfoundinaperfor-mance.Rangecansurfaceinthevarietyofgrammaticalforms(verbmodification,tense,aspect,comparativeforms,superlativeforms)andthecomplexityofsentences(mainclauses,subordinateclauses,conditionalorrelativeclauses)usedinatext.

IntheE8context,grammaticalrangemustbeseeninrelationtothetask.Wecannotexpectthetesttakerstousestructuresthatarenotmeaningfullyelicitedbythetask.Sincethewritingpromptsfocusexclusivelyonfamiliartopicsandhavetocaterforallabilitylevels,theyareasstraightforwardintheirset-upaspossible.Thisdoesnotautomaticallysuggestthattheresponsecannotbemorecomplexthanthestimulus.Evenifataskissimpleinnature,weexpectdifferentiationingrammaticalformsorclausetypes,suchasconditionalorrelativeclauses.

Verbs,forexample,canbemodified,markaspect,anddeterminevarioustypesofsen-tencefunctionsuchasstatement,question,negation,command,andexclamation.Moreover, they can be used in their active or passive forms, and test takers maychoosetousedirectorindirectspeech.

Inadditiontothespecificationsoftheprompt,whichwilltrytoelicitcertaingram-maticalstructuresfortaskfulfilment,thetimeallotmentandtheexpectednumberofwordswillalsohaveanimpactonrange.Thatis,shorttasksarelikelytoprovidefeweropportunitiestoshowgrammaticalrangethanlongtasks.

RANGE VERSUS ACCURACYInamistakesandcorrectiondriventraditionofteaching,theuseofgrammaticallychallenginglanguagecanbecomeaproblemforalearneriferrorsoccur.NotsoinE8testing.ItisE8testingpolicythatrangeoverrulesaccuracyinthesensethatrichgrammaticalrangethroughrisktakingisencouraged,whileminorinaccuraciesthatdonotimpairmeaningplayareducedrole.Themorevariedthegrammaticalrange,thehighertheband.Risktaking,whichresultsinrichstructuresbutreducedcon-trol,canevenbeareasonforupgradingatext.

Globalerrors, i.e.errorsthat interferewiththecomprehensibilityofthetext,willcausedowngradingortheplacementofatextatalowband.Localerrorswhichdonothindercommunicationwillnotautomaticallyleadtodowngradingunlesstheirfrequencyimpairsthemessageorthereadabilityofthetext.

6 ForaninventoryofgrammaticalareasatA2levelseeKETHandbook,8–9.Availableat:http://www.cambridgeesol.org/assets/pdf/resources/teacher/ket_handbook.pdf[24June,2011]

7 ForaninventoryofgrammaticalareasatB1levelseePETHandbook,7–8.Availableat:http://www.cambridgeesol.org/assets/pdf/resources/teacher/pet_handbook.pdf[24June,2011]


Testtakersareencouragedtomakeuseoftheirfullpotentialandthemorecreativethestructuralfeaturestheyshow,thebetter.Nevertheless,theuseofvariationshouldnotbeexaggeratedeither.Thetaskssuggestcertainscenarioswhichrequirespecialstructuralsolutions.Theseshouldproduceauthenticandnaturalvariationbutnotartificialtexts.

ASSESSMENT OF PERFORMANCES Theplacementofaperformanceatacertainbandreflectstherangeofgrammaticalstructuresandtheleveloftheircorrectnesswithinameaningfullyandsuccessfullyaccomplishedcommunicativetask.

Band7 texts feature goodgrammatical rangewhich createsmeaning andnaturallanguagewithintheframeworkofthetask.Thewritervariesthegrammaticalstruc-turesthepromptelicitsandmayoccasionallygobeyondtheobviousandexpected.However,anyenhancementshouldnotmakethetextsoundunnaturalorresultinexaggerationofgrammaticalstructures(rangeforthesakeofrange).Inadditiontogood rangea relativelyhighdegreeofgrammatical control is expected inband7texts.Afewinaccuraciescanoccurbuttheywillnotimpaircommunication.

Band 5 texts show sufficient range of grammatical structures. Sufficient range isachieved,ifthewritermakesenoughuseoftheprompt’sstructuralfeaturestomakethe required communication successful and if the grammatical forms used createappropriatemeaning.Occasionalinaccuracieswhichcanimpaircommunicationcanbetolerated.

Band3textsfeaturealimitedrangeofsimplegrammaticalstructures.Thismeansthatthegrammaticalstructuresarejustenoughtoachievesuccessfulcommunication.Mostlytheyareverysimple,repetitive,andhardlyvaried.Grammaticalstructuresinband3textscanbefrequentlyinaccurateandmayshowbasicmistakes.Generally,thesemistakesdonotcausebreakdownofcommunication.

Band1texts featureanextremely limitedrangeof simple structures.Thisusuallyforces the writer to compromise the message regarding meaning, content, andnaturalnessoflanguage.Extremelylimitedrangeresultsinstructuresthatarerepet-itive and invery simple subject-predicate-object sentencepatterns.The structuresusedhardlygobeyondthelearntrepertoireofbeginners.Inadditiontostructuralrestrictions,band1textsshowlimitedcontrolwhichfrequentlycausesbreakdownofcommunication.


ScaleInterpretation–Vocabulary

When we assess vocabulary we are looking at content words (nouns, full verbs,adjectives,adverbs),collocationsandchunksoflanguagethatawriterusestoper-formawrittencommunicativetask.Whatweneedtoassessislexiscreatingmeaningthat isreasonablycorrect toaccomplishsuccessfulcommunication.Similar tothegrammar scale, the scale for vocabulary also comprises descriptors for range andcontrol.

THE CONCEPT OF LEXICAL RANGERangereferstothebreadthofvocabularyacandidateusesinawrittentext.IntheE8context,rangemustbeinterpretedinrelationtothepromptasraterscanassessonly the vocabulary actually elicited by the prompt. The time allotment and theexpectedtextlengthhaveanimpactonrange.Shorttasksarelikelytoprovidefeweropportunities todemonstrate vocabulary range than long tasks.Asmentioned inthepreviouschapter,shorttaskshavebeendesignedtobeA2tasksandlongtaskshavebeendesignedtobeB1tasks.ForthesereasonstherangeoflexicalitemsthatwecanexpectinshorttasksarewordsandphrasestypicallymasteredatA2level8,forlongtaskswecanexpectafairamountofwordsandphrasestypicallymasteredatB1level9.

Even if a task is simple innaturewemay expectdifferentiationwithin choiceofwords.Forexample,ifataskasksforanarrativedescriptionaboutthefirstfewdaysatanewschool,thetextswillprimarilycontainwordsrelatedtoschool,teachers,subjects,newfriendsetc.,which,however,canbevariedandmodified.Althoughthepromptlanguageisassimpleaspossible,writersmaywellproducearesponsethatexceedsthepromptstimulus.

RANGE VERSUS ACCURACYItisnotenoughforacandidatetousealargenumberofdifferentwordsinatexttoachieveahighbandinassessment.Thewordsacandidatechoosesmustberelevantandappropriatetothetopicandusedinawaythatthecandidateisabletoconveyhis/herideasmeaningfully.Atopwriteramongourtesttakerswillusevocabularythatisgenerallyaccurateenoughtoformulateevenamorecomplexideawithclarity.Test takerswhostay inabsolutely safe languageareas (e.g. languagepickedup inyearsoneandtwo)andavoidtakinganyriskhavelessevidenceofmistakes.How-ever,itisE8policytoencourageourcandidatestoventureoutoftheirsafelanguagezonebyrewardingrisktakingtocommunicatesuccessfully.

Textsthatshowagoodrangeofvocabularyatband7containagoodselectionofcontentwords andphrases thatdemonstrate that the candidate is able to expresshim/herselfclearlyandpreciselyandoccasionallycanevenvaryformulationssoasnottoappearrepetitive.Wemaywellexpectoneortheotherexpressiontostickoutandexceedwhatwetypicallyexpectfromtesttakersatthislevel.

Band5textscontainasufficientrangeofmostlyhigh-frequencywordsthatagainmeettheneedtocommunicateclearideasandaregenerallyusedaccurately.Theremaybesomeoccasionalmistakes,particularlywhenthecandidateistryingtocom-municateamorecomplexidea.

Inaband3textweexpectthelexicalrangetobelimited,containingonlyarathernarrowrepertoireofhigh-frequencywords,butstillthesimpleideasthatarecom-8 Availableat:https://www.teachers.cambridgeesol.org/ts/digitalAssets/113295_ket_vocablist09.pdf[24June,2011].9 Availableat:https://www.teachers.cambridgeesol.org/ts/digitalAssets/113298_pet_vocablist09.pdf[24June,2011].


municatedaremostlyunderstandableevenifthereisacertainamountofinaccuratevocabulary.Withband3candidateswearelikelytodetectexamplesofliftingphrasesfromtheprompttocompensatefortheirlexicallimitations.

Finally,inaband1textawriterwithextremelylimitedlexicalcompetenceinEng-lishwilldemonstratethisbyincludingonlyafewveryhigh-frequencycontentwordswhicharemoreoftenthannotinaccurateandinappropriate.Wecommonlyexpectband1writerstocompensatefortheirlackinlexicalrangebyheavilyliftingdirectlyfromthepromptandby interspersing their textwithL1words inorder to ‘keepgoing’,thushavingthe‘knockoneffect’offrequentlycausingbreakdownincom-munication.

Thenatureofsomepromptsmakesitalmostimpossibletoavoidliftingandratersmusttakecarenottofallintothetrapofautomaticallyplacingatextatband4orbelowbecausethereisevidenceofpromptlifting.Agoodwriterdoesnotjust‘copyandpaste’wordsorphrases,butcanadaptthemandincorporatethemsuccessfullyintothetexttoaccomplishthecommunicativetask.Thisisaskillthatneedstobeacknowledgedpositively.

Anaspectof lexicalaccuracythatratersneedtoaddress is spelling.It iscommonpractice amongst teachers to take marks off for incorrect spelling. However, theemphasisoncommunicatingmeaningsuccessfully iscentral to theE8context.Atext containing many spelling mistakes, in particular those mistakes that changethewholemeaningofaword,isverylikelytodisturbthereaderandcauseabreak-downofcommunication.Ratersneedtoassesstheextentofbreakdownandrateasisnecessary.However,asweencourageourwriterstotakerisks,slight‘slipsofthehand’andminorerrorsinspellingthatdonotchangethemeaningoftheword(e.g. pleas, tomorow)shouldnotbepenalised.Intheenditisthelexicalrangethatismoreimportantthanaccuracyandatextmightmeritoneofthehigherbandsdespitetheinaccuracies.


Literature

Alderson,J.C.2004.Washbackinlanguagetesting.Researchcontextsandmethods.In:Cheng,L.,Watanabe,Y.&Curtis,A.(Eds.).Context and Method in Washback Research: The influence of language testing on teaching and learning. Mahwah, NJ:LawrenceErlbaum.

Alderson,J.C.,Clapham,C.&Wall,D.1995.Language Test Construction and Eva-luation. Cambridge:UniversityPress.

Bachman,L.F.1990.Fundamental Considerations in Language Testing.Oxford:Uni-versityPress.

BIST-Verordnung.2009.SeeVerordnung der Bundesministerin.

Breit,S.&Schreiner,C.(Eds.)2010.Bildungsstandards: Baseline 2009 (8. Schulstufe). Technischer Bericht. Salzburg:BIFIE.Availableasdownloadfromhttp://www.bifie.at/buch/1056[14.April,2011]

Brock,R.&Haslinger,U.(Eds.)2011.Bildungsstandards für Fremdsprachen (En glisch) 8. Schulstufe. Praxishandbuch(Neubearbeitung2011).ÖSZPraxisreihe4.BifieWien.

Canale,M.&Swain,M.1980.TheoreticalBasesofCommunicativeApproachestoSecondLanguageTeachingandTesting.In:Applied Linguistics 1(1),1–47.

CouncilofEurope(Ed.).2001.Common European Framework of Reference for Lan-guages: Learning, Teaching, Assessment.Cambridge:UniversityPress.

Ek,J.A.van[CouncilofEurope].1979.The Threshold Level for modern language learning in schools. Harlow:Longman.

Ek,J.A.van&Alexander,L.G.1980.Waystage English. Anintermediaryobjec-tivebelowThresholdLevel inaEuropeanunit/credit systemofmodern languagelearningbyadults.Preparedfor theCouncilofEurope inassociationwithM.A.Fitzpatrick.Oxford:PergamonPress.

Field,J.2004.Psycholinguistics. The Key Concepts.London:Routledge.

Field, J. 2005. Second language writing: a language problem or a writing problem? Paper presented at IATEFL Research SIG ‘Writing Revisited’ conference, Cam-bridge,25–27February2005.

Gassner,O.,Mewald,C.&Sigott,G.2007.Testing Reading. Specifications for the E8 Standards Reading Tests. LTC Technical Report 2. Wien:bm:ukk.Availableasdown-loadfrom:http://www.bifie.at/publist-07-05-14[24June,2011]

Grabe,W.&Kaplan,R.B.1996.Theory and Practice of Writing. An Applied Linguistic Perspective.London:Longman.

Hyland,K.2002.Teaching and Researching Writing. London:Longman.

Kellogg,R.T.1994.The Psychology of Writing. NewYork:OxfordUniversityPress.


Kellogg,R.T.1996.Amodelofworkingmemoryinwriting.In:Levy,C.M.&Rands-dell,S.(Eds.)The Science of Writing. Mahwah,NJ:LawrenceErlbaum,57–72.

KET Handbook. CambridgeUCLES.2009.Availableasdownloadfromhttp://www.cambridgeesol.org/assets/pdf/resources/teacher/ket_handbook.pdf[14April,2011]

KET Vocabulary List. Cambridge UCLES. 2006. Available as download fromhttps://www.teachers.cambridgeesol.org/ts/digitalAssets/113295_ket_vocablist09.pdf[14April,2011]

Lehrplan der Hauptschule: LebendeFremdsprache(alleSprachen)2008.Availableasdownloadfromhttp://www.bmukk.gv.at/medienpool/16682/bgbl_nr_ii_210_2008.pdf[14April,2011]

Lehrplan der AHS:LebendeFremdsprache(alleSprachen)2006.Availableasdown-loadfromhttp://www.bmukk.gv.at/medienpool/782/ahs8.pdf[14April,2011]

Lumley,T.2005.Assessing Second Language Writing. The Rater’s Perspective. Frank-furt:PeterLang.

Papajohn,D.1999.Theeffectoftopicvariationinperformancetesting:thecaseofthechemistryTEACHtestforinternationalteachingassistants.In:Language Testing 16(1),52–81.

PET Handbook.CambridgeUCLES,2009.Availableasdownloadfromhttp://www.cambridgeesol.org/assets/pdf/resources/teacher/pet_handbook.pdf[14April,2011]

PET Vocabulary List. CambridgeUCLES.2009.Availableasdownloadfromhttps://www.teachers.cambridgeesol.org/ts/digitalAssets/113298_pet_vocablist09.pdf[14April,2011]

Read,J.1990.ProvidingRelevantContentinanEAPWritingTest.In:English for Specific Purposes 9(2),109–21.

Scardamalia, M. & Bereiter, C. 1987. Knowledge telling and knowledge trans-forminginwrittencomposition.In:Rosenberg,S.(Ed.)Advances in Applied Psycho-linguistics, Vol. 2: Reading, writing, and language learning. Cambridge UniversityPress,142–175.

Shaw,S.D.&Weir,C.J.2007.Examining Writing. Research and practice in assessing second language writing.Cambridge:UniversityPress.

Sigott,G.,Gassner,O.,Mewald,C.&Siller,K.2007.E8 Standardstests. Entwick-lung der Tests für die rezeptiven Fertigkeiten: Überblick. LTC Technical Report 1. Lan-guageTesting Centre, Alpen-Adria-Universität Klagenfurt. Available as downloadfromhttp://www.bifie.at/publist-07-02-19[24June,2011]

Tankó,G.2005.Into Europe. The Writing Handbook.Budapest:TelekiLászlóFoun-dation.

Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungsstan-dards im Schulwesen.BGBl.IINr.1/2009v.2.1.2009.Availableasdownloadfromhttp://www.bmukk.gv.at/schulen/recht/erk/vo_bildungsstandards.xml [14. April,2011]


Weigle,S.1994.EffectsofTrainingonRatersofESLCompositions.In:Language Testing 11(1),41–69.

Weigle,S.C.2002. Assessing Writing.Cambridge:UniversityPress.

Weir, C. J. 2005. Language Testing and Validation. An Evidence-Based Approach. Basingstoke:PalgraveMacmillan.


Appendix

PromptInterpretation:LongPrompt

TaskAchievement

Content

�� Althoughthefirstcontentpointcanbedealtwithverybriefly(newplace),thesecondpoint(reasonsformoving)allowsforsomeelaboration.�� Thenextthreepoints(describetown/village,newhome,andfirstdays)definitelyallowforelaborationandweexpectsomewhatdetaileddescriptions(points3,4,5).�� Thelastpointaboutfeelingsismoredifficultbecausethisrequiressomecreativethinking,whichiswhywedonotexpectmuchelaboration.Ifthereiselaborationhere,itneedstobespeciallyrecognised.

DegreesofTaskAchievement

Inthislongtext,allcontentpointsneedtobementionedforcomplete task achieve-mentandthreeelaborated.Elaborationinthelastpointaboutfeelingscannotbeexpectedasthisisconsideredtobedifficulttoachieveatthelevelofthetesttakers.

Good task achievementcanbeawardedevenifonecontentpointisnotdealtwith.‘85%’meansthatfivecontentpointsmustbementioned.Dependingonthequalityofelaboration,wewouldexpectatleasttwopointstobeelaboratedforgoodtaskachievement.Asecondoptionforband5istohaveallcontentpointsmentionedandoneortwoelaborated.

Forsufficient task achievementwewouldexpectfourcontentpointstobemen-tionedandoneortwotobeelaborated,dependingonthequalityofelaboration.

Asecondoptionforband3istohaveallcontentpointsmentioned,butnoelabo-ration.

Some task achievement wouldbegivenifthemessagewasclearenoughtoconveytheinformationthatthepersonhasmovedtoanotherplaceandsomebriefinforma-tionabouttwoothercontentpointstoreachthe50%.

You have just moved to another town/village. Write a letter to your American/English friend in which you tell him/her about your new situation.

Inform him/her about ● your new place of living. (1) ● the reason for moving. (2)Describe ● the town/village you’re living in now (buildings, people,…). (3) ● your new home. (4)Tell him/her about ● the first days of your ‘new life’ (new school, teachers,…). (5) ● how you feel about your new situation. (6)


TextTypeRequirements

�� Thetext type isclearlymarkedasapersonal letter,which iswhyweexpectaninformalopeningandclosingformulalikeDear Jim/Hi Sue andYours/Love etc.�� Theregisterneedstobeinformal.�� Texttyperequirementsarenot met ifthereisnoopeningand/orclosingformulaANDiftheregisterisnotappropriate(e.g.rudelanguageormismatchofinfor-malsituationandformallanguage).Thiswillleadtodowngrading.�� Missingorincorrectopeningand/orclosingformulawillleadtodowngradingbyoneband.�� Inappropriateregister/tonealonewillleadtodowngradingbyoneortwobandsdependingonthequalityofthetext.Thesameappliestoanimpoliteoroffensivetone.

CoherenceandCohesion

Inalongtextwedefinitelyexpectparagraphing.Sincethecontentpointsaregroupedintothreeunits,thetesttakersareprovidedwithsufficienthintsastohowthetextcouldbestructuredintoparagraphs.Alackofparagraphorganisationinbands3to7leadstodowngradingbyoneband.

�� Testtakersneedtouseparagraphsandsomecohesivedevicestoreachband7.Thetextmustbeclearandcoherentatsentence,paragraphandtextlevel.�� Forband5,cohesionshouldbeachievedat leastat sentence level,and longerstretchesshouldshowsomeparagraphlevelcoherenceandcohesion.Forexample,thiscouldbedemonstratedinthedescriptionofthenewplaceand/orhome.�� Forband3,wewouldatleastexpectthetesttakerstousesimpleconnectors(e.g.because, but, and, then)whengivingreasonsformovingorforthedescriptionofplacesand/orthefirstdaysatschool.Sowecanexpectsentencelevelcohesion,butproblemsatparagraphlevel.Thetextmightnotbefullycoherent.�� Forband1,itisenoughifthesentencesarelinkedwithverybasicconnectors.Wedonotexpectparagraphingortextualcoherenceforband1.�� Iftheletteriswritteninchunksratherthansentences,wesuggestthatthereisnotenoughassessablelanguageforcoherenceandcohesion.

Grammar

Inthis longtext,writersareexpectedto informaboutthemovetoanotherplacemakinguseofpastverbforms.Thereasonformovingcaneitherbelinkedwiththisinformationandalsorealisedinthepast,orconnectedtotheinformationaboutthenewplaceandthereforefeaturepresenttenseverbforms.Thedescriptionofthenewplace,thenewhomeandthenewlifewillmostprobablyelicitpresenttenseverbforms.Verygoodsolutionsmightincludecomparisonsbetweenthenewandtheoldplace,thenewandtheoldsituation.However,thesecannotbeconsideredcompul-sorybecausetheyarenotexplicitlyrequiredbythetask.

Vocabulary

Theinstructionsandcontentpointsshouldenticethewritertousealotoffamiliarcontentwordstodescribethenewsituation.Wewouldexpectthewritertomentionthenameofacountry,city, townorvillageandusecontentwordstodescribe it,thebuildings,people,neighbours, andpossibly the countryside there.Thewriterhastheopportunitytodescribehis/herhomeandwordsrelatingtotypesofhouses,householdrooms,furniture,andperhapsagarden,willmostlikelybeinthetext.Wewouldcertainlyexpectthewritertousearangeofadjectivessuchasqualifiers


todescribeappearance,condition,and/orshape;alsoquantifierstodescribesize,inordertocomparethepresentsituationtotheformersituation.Thewritershouldgiveareasonformovingandhe/shewillprobablyusevocabularyrelatingtoapasteventinthefamily:aparentgetting/losingajob,startinganewlife,movingbacktoaformerplaceofresidence,orachangeinthefamilystructureduetoadivorce.

Thewritershouldalsowriteabouthis/hernewlifeandwecanexpectsomevocab-ularyaboutschool, teachers,classmates,subjects,classroom,and/orhomeworktoappear inthetext.Itmaywellbethatthewriterchoosesto ignorewritingaboutschoolandinterprets“thefirstdaysofyour'newlife'“byreferringtooutofschoolactivitiessuchasbuying/movingfurniture,meetingandplayingwithnewfriends,exploringthenewareaandotherfree-timeactivities.Onceagain,wewouldexpectthesentencestocontainamixtureofqualifiersandquantifiers.

Finally,wewouldexpectsomeofthesentencesaboutthenewsituationtoincludeadjectives to describe the positive, ambivalent and/or negative feelings he/she isexperiencinginthenewsituation.

PromptInterpretation:ShortPrompt

TaskAchievement

Content

�� Point1:Thewritersneedtogiveoneorseveralreasonsforlikingtheparty.Itisnotenoughtosaythattheylikedthepartybutthisshouldincludegivingreasons.�� Point 2: This can be done in a more or less elaborate way depending on thewriter’schoicesandabilities.�� Point3:Thisissimpleenoughandwillbedealtwithverybriefly.Mostprobablyitwillimmediatelyleadontopoint4.�� Point4isexpectedtobethemostchallengingoneasitrequiressomethinkingandaminimalamountofcreativity.Writersneedtoincludeatleastonesugges-tionreferringtothefollowingweekend.

DegreesofTaskAchievement

Withshorttasks,therelevantdescriptorneedstobeinterpretedverystrictly.Infact,withfourcontentpointsrequireditmakeslittlesensetotolerateamissingone.ItshouldbetakenforgrantedthatALLfourcontentpointsmustbedealtwithforcomplete or good task achievement. This would normally mean a text length ofaround60words.

�� Itwillbethedegreeofelaborationthatdistinguishesbetweencompleteandgoodtaskachievement.

Your friend’s birthday party was a few days ago. Write an email to tell him/her that you liked the party.

• Tell him/her why you liked the party. (1)• Tell him/her what you liked best. (2)• Ask your friend when you are going to meet again. (3)• Suggest something for the next weekend. (4)


�� Withonecontentpointmissing,goodtaskachievementisnolongerpossible,butband4is.�� Ifonecontentpointismissing,wecanawardband3,i.e.sufficienttaskachieve-ment(justover70%).Ifthereissomeelaboration,itisband4.�� Iftwocontentpointsaremissing,wecannotexpectmuchelaborationeitheranditseemstopointtowardsband1.�� If,however,thereisgoodelaborationofonecontentpoint,itisband3.Ifelabo-rationisweak,itisband2.

TextTypeRequirements

�� WecanexpectaninformalopeningandclosingformulalikeDear Jim/Hi Sue and Yours/Loveetc.�� Forband7 theremustbe anappropriateopeningandclosing formula and theregistermustbeinformalanddemonstratethefamiliarityindicatedbyyour friend.�� Problemswiththebeginningorendingofthetextorminorproblemswiththeregisterwouldleadtodowngradingbyoneband.�� Whentheproblemsbecomemoreobvious,taskachievementwilldropbytwobands.�� Text type requirements are not met when there is no opening and/or closingformulaANDwhentheregisterisnotappropriate.(Downgradebytwobandsiftheproblemswithregisterareveryserious).

CoherenceandCohesion

�� Wedonotexpectanyparagraphingwithshorttasksalthoughgoodwriterswilluseparagraphingtostructuretheirshorttextsaswell.Sosomecreditshouldbegivenforsuccessfulparagraphing.�� Demonstratingthisdimensioninshorttextsisratherdifficult,butwecanexpectthewriterstodealwiththefourcontentpointsintheordergiven.Implicitandexplicitlinking(connectors)isnoteasytoplaceinthiskindoftextandshouldbespeciallyacknowledgedwhenitisusedsuccessfully.Contentpoints1and2sug-gesttheuseofbecauseand/orand.

Grammar

Thisshort taskasks for feedbackaboutapastevent.Thedescriptionofwhatwasliked andwhatwas likedbest opensuppossibilities for comparisonsor even thesuperlative. However, any other solution that implies the information about thelikedandbestlikedactivityatthepartyneedstobeacknowledged.Moreover,thetaskrequiresaquestionforameetinginthefutureandasuggestionforanactivityforthenextweekend.Whilethequestionisverylikelytoelicitafutureverbform(mostprobablythegoing tofutureasgivenintheprompt),thesuggestioncanalsoberealisedinthepresentormakinguseofmodalverbforms.Thelattercouldalsocomeacrossasanotherquestionfollowedbyasuggestionorastatement (What about next weekend? My brother has a birthday party. Would you like to come?)

Vocabulary

We expect the writer to use adjectives such as great, cool, funny and/or to refertonounssuchas the games, music, food, lights, friends that were at the party intheresponsetothefirstcontentpoint.Thewriterwillprobablyrephrasethesecondcon-tentpointandaddsimilarvocabularytostateclearlywhyhe/shelikedthepartybest.Theresponsetothethirdcontentpointwillmostlikelycontainoneofthefollowing:When can I see you again? When are we going to meet again? What are you doing at the


weekend? Thefinalpartofthetextwillmostlikelybeacombinationofasuggestionandaquestioncontainingwordsrelatedtocommonteenageleisureactivitiessuchasmeeting friends, going shopping/to a party, and/orplaying/watching football etc. Asthelastcontentpointreferstotheweekend,wecanexpectthewritertoincludewordssuchasweekend, Friday, Saturday, Sunday, morning, afternoon, evening and/ornight.


InventoryofFunctions,NotionsandCommunicativeTasks1

ConveyingEmotions,Feelings

�� criticisingandcomplaining�� expressingneedsandwants�� expressingpreferences,likesanddislikes�� makingandrespondingtoapologiesandexcuses�� payingcompliments�� sympathising�� talkingaboutfeelings

Informing,Asking

�� askingandansweringquestionsaboutpersonalpossessions�� askingandtellingpeoplethetime,dayand/ordate�� askingforandgivinginformationaboutroutinesandhabits�� askingforandgivingpersonaldetails:(full)name,age,address,namesofrelativesandfriends,occupation,etc.�� askingforandgivingsimpleinformationaboutplaces�� completingformsgivingpersonaldetails�� expressing(in)abilityinthepresentandinthepast�� makingpredictions�� talkingabout(im)probabilityand(im)possibility�� talkingaboutfood�� talkingaboutfutureorimaginarysituations�� talkingaboutfutureplansorintentions�� talkingaboutone’shealth�� talkingabouttheweather�� talkingaboutpasteventsandstatesinthepast,recentactivitiesandcompletedactions�� talkingaboutwhatpeoplearedoingatthemoment�� writingdiariesgivinginformationabouteverydayactivities

Convincing,Persuading,ExpressingOpinions

�� askingandgiving/refusingpermissiontodosomething�� drawingsimpleconclusionsandmakingrecommendations�� expressingagreementanddisagreement,andcontradictingpeople�� expressingdegreesofcertaintyanddoubt�� expressingobligationandlackofobligation�� expressingopinionsandmakingchoices�� expressingpurpose,causeandresult,andgivingreasons�� givingadvice�� givingwarningsandprohibitions�� persuadingandasking/tellingpeopletodosomething

1 Adaptedfrom PET Handbook 2009andKET Handbook2009.Thelistisnotexhaustive,butservestoillustrateaspectsoftheConstructSpace.


Entertaining,Pleasing

�� talkingaboutpasteventsandstatesinthepast,recentactivitiesandcompletedactions�� talkingaboutwhatpeoplearedoingatthemoment�� understandingandproducingsimplenarratives

KeepinginTouch

�� expressingandrespondingtothanks�� givingandrespondingtoinvitations�� makingandgranting/refusingsimplerequests�� makingandrespondingtooffersandsuggestions�� writingletters/emailsgivinginformationabouteverydayactivities�� writingletters/emailsgivingpersonaldetails

Describing

�� buyingandsellingthings(costs,measurementsandamounts)�� describingeducationandskills�� describingpeople(personalappearance,qualities)�� describingsimpleprocesses�� identifyinganddescribingaccommodation(houses,flats,rooms,furniture,etc.)�� makingcomparisonsandexpressingdegreesofdifference

GivingDirectionsandInstructions

�� askingforandgivingtravelinformation�� askingthewayandgivingdirections�� followingandgivingsimpleinstructions�� identifyinganddescribingsimpleobjects(shape,size,weight,colour,purposeoruse,etc.)�� talkingabouthowtooperatethings

Testing Writing for the E8 Standards

Documents