Assessing L2 writing performance: Reconsidering writing prompts as genre-based tasks Heidi Byrnes German Department Georgetown University [email protected].

Assessing L2 writing performance:Assessing L2 writing performance:Reconsidering writing prompts Reconsidering writing prompts

as genre-based tasksas genre-based tasksHeidi ByrnesHeidi Byrnes

German DepartmentGerman DepartmentGeorgetown UniversityGeorgetown University

[email protected]@georgetown.edu

TBLT 2007TBLT 2007University of Hawai’i at ManoaUniversity of Hawai’i at Manoa

Sept. 21, 2007Sept. 21, 2007GUGD GUGD curriculum web page: curriculum web page:

http://www3.georgetown.edu/departments/german/programs/curriculum/index.htmlhttp://www3.georgetown.edu/departments/german/programs/curriculum/index.html

Underlying AssumptionsUnderlying Assumptions

L2 writing prompts can/shouldL2 writing prompts can/should

- be reconsidered as “tasks”, particularly as - be reconsidered as “tasks”, particularly as genre-based tasksgenre-based tasks- be linked to quality of performance- be linked to quality of performance- - facilitate the assessmentfacilitate the assessment of L2 writing performance of L2 writing performance

L2 writing in genre-based tasks canL2 writing in genre-based tasks can- - Provide a favorable environment for integrating “language” and Provide a favorable environment for integrating “language” and “content”“content”

Assessment of L2 writing performance shouldAssessment of L2 writing performance should- - Consider language knowledge and content knowledge in an Consider language knowledge and content knowledge in an integrated fashionintegrated fashion

““Tasks” in L2 education (1)Tasks” in L2 education (1)

An educational “activityAn educational “activity”” With a particular objective, appropriate content, specified With a particular objective, appropriate content, specified

work procedure, range of outcomes, necessitating the use work procedure, range of outcomes, necessitating the use of language of language (Breen 1997, Van den Branden 2006)(Breen 1997, Van den Branden 2006)

In the target language, focused on meaning rather than In the target language, focused on meaning rather than form (form (Nunan 1989)Nunan 1989)

Meaning-oriented language use, resembling language use Meaning-oriented language use, resembling language use in the real world, often occasioned by a problem, involving in the real world, often occasioned by a problem, involving a workplan toward achieving a certain outcome (a workplan toward achieving a certain outcome (Ellis 2003, Ellis 2003, Skehan 1998)Skehan 1998)

““Tasks” in L2 education (2)Tasks” in L2 education (2)

Users of tasksUsers of tasks

Teachers and learners, curriculum developers, Teachers and learners, curriculum developers, researchers, assessment specialists researchers, assessment specialists (Bygate et al. 2001)(Bygate et al. 2001)

Uses of tasks, as a way ofUses of tasks, as a way of– Specifying learning goalsSpecifying learning goals– Designing and organizing educational activities in support of Designing and organizing educational activities in support of

learninglearning– Ascertaining ability to use language (assessment)Ascertaining ability to use language (assessment)– Exploring implications of findings for curriculum and instructionExploring implications of findings for curriculum and instruction

““Tasks”: Tasks”: Relating “language” and “content/meaningRelating “language” and “content/meaning””

Language AND contentLanguage AND content Language knowledge AND content/background Language knowledge AND content/background

knowledgeknowledge Focus on form AND focus on meaningFocus on form AND focus on meaning Ability to use/perform AND ability to developAbility to use/perform AND ability to develop

Additive conceptualization of taskAdditive conceptualization of taskNo conceptual reach into content/meaning in No conceptual reach into content/meaning in

social contextsocial context

Needed: New conception of “language”Needed: New conception of “language”

““Our approach to the teaching of any phenomenon Our approach to the teaching of any phenomenon depends critically on our conception of this depends critically on our conception of this phenomenon. phenomenon.

Unless we can base language teaching and learning Unless we can base language teaching and learning on a richly revealing comprehensive account of on a richly revealing comprehensive account of what kind of phenomenon language is, we are not what kind of phenomenon language is, we are not in a position to answer the many questions that in a position to answer the many questions that arise in educational contexts. arise in educational contexts.

Similarly, the value and success of any approach Similarly, the value and success of any approach designed to support second or foreign language designed to support second or foreign language teaching … will depend critically on the conception teaching … will depend critically on the conception of language that informs it.” (of language that informs it.” (Matthiessen 2006: 31)Matthiessen 2006: 31)

Probing two constructs of “task” (1)Probing two constructs of “task” (1)

Language Language analyzed asanalyzed as

Formal system of rulesFormal system of rules Fundamentally functional Fundamentally functional social semiotic systemsocial semiotic system

Meaning Meaning making making

Expressing things;Expressing things;

Focus on personal Focus on personal meanings;meanings;

““Negotiation of meaning”.Negotiation of meaning”.

Construing/creating that Construing/creating that which is being which is being expressed;expressed;

Developing a voice Developing a voice within socially situated, within socially situated, socially preferred forms socially preferred forms of language use.of language use.

Semantics and Semantics and lexicogrammarlexicogrammar

InventoriesInventories Strategic resourcesStrategic resources

Probing two constructs of “task” (2)Probing two constructs of “task” (2)Context and Context and content content

Given “out Given “out there”there”

Socially constructed;Socially constructed;Language as primary semiotic Language as primary semiotic system;system;““Context of culture” + “context Context of culture” + “context of situation” participating in of situation” participating in the construal of meaningthe construal of meaning

Language learning Language learning asas

Gains in Gains in accuracy, accuracy, fluency, fluency, complexity of complexity of processing in all processing in all modalitiesmodalities

Learning how to mean;Learning how to mean;Building up ‘meaning potential’ Building up ‘meaning potential’ in a range of registers and text in a range of registers and text types/genrestypes/genres

Focus of attentionFocus of attention Sentence-level Sentence-level processingprocessing

Oral/written texts as Oral/written texts as environment arising from and environment arising from and construing social structures construing social structures and meaningsand meanings

Probing two constructs of “task” (3)Probing two constructs of “task” (3)

Processing Processing orientationorientation

Syntagmatic syntactic Syntagmatic syntactic rules orientation rules orientation

Paradigmatic meaning Paradigmatic meaning and choice orientationand choice orientation

““Knowledge” Knowledge” imagined and imagined and valued as valued as

Cognition;Cognition;““Cognitively Cognitively demanding;”demanding;”““Higher-order thinking”Higher-order thinking”

Language-based Language-based shaping of world;shaping of world;Expressed in texts fromExpressed in texts from2 semiotic perspectives2 semiotic perspectives congruent semiosiscongruent semiosis non-congruent non-congruent (“metaphorical”) semiosis(“metaphorical”) semiosis

Integration Integration throughthrough

Psycholinguistic Psycholinguistic mechanisms focused on mechanisms focused on individual language userindividual language user

Semiotic processes Semiotic processes focused on construing focused on construing and shaping the and shaping the physical and social physical and social world = meaning world = meaning makingmaking

Needed: New understanding and Needed: New understanding and descriptions of languagedescriptions of language

The challenge is to relate sociocultural context and the “content” that is The challenge is to relate sociocultural context and the “content” that is likely to be enacted to its particular wording in language, that is, we likely to be enacted to its particular wording in language, that is, we need rich accounts of semantics and lexicogrammar. need rich accounts of semantics and lexicogrammar.

““On the one hand, we need semantics as an ‘interface’ between On the one hand, we need semantics as an ‘interface’ between context and lexicogrammar … a fully fledged semantics of text. … context and lexicogrammar … a fully fledged semantics of text. … Language learners need to learn semantics as a Language learners need to learn semantics as a strategicstrategic resource … resource … a resource for transforming what is not meaning into meaning, a resource for transforming what is not meaning into meaning, construing their experience of the world as meaning and enacting construing their experience of the world as meaning and enacting social roles and relations as meaning; and this will provide them with social roles and relations as meaning; and this will provide them with the ‘bridge’ to lexicogrammar.the ‘bridge’ to lexicogrammar.

On the other hand, both semantics and lexicogrammar need to be On the other hand, both semantics and lexicogrammar need to be learned as learned as resourcesresources rather than as rather than as inventoriesinventories. .

Needed …Needed …

... To meet the challenge of relating language to context, ... To meet the challenge of relating language to context, we also need to have a systematic and comprehensive we also need to have a systematic and comprehensive account of context … Like language, context is a semiotic account of context … Like language, context is a semiotic system, but it is a different kind of semiotic system … system, but it is a different kind of semiotic system … language is a language is a denotativedenotative semiotic system (that is, a semiotic system (that is, a semiotic system that has its own expression plane), semiotic system that has its own expression plane), whereas context is a whereas context is a connotativeconnotative semiotic system (that is, semiotic system (that is, a semiotic system that has other semiotic systems as its a semiotic system that has other semiotic systems as its expression plane). Context is realized by language, and by expression plane). Context is realized by language, and by other denotative semiotic systems such as gesture and other denotative semiotic systems such as gesture and facial expressions; and also by non-semiotic, social facial expressions; and also by non-semiotic, social systems.” systems.” (Matthiessen 2006, 37-38)(Matthiessen 2006, 37-38)

SFL approach to relating SFL approach to relating meaning/content and language meaning/content and language The process of meaning making is multifunctional in all The process of meaning making is multifunctional in all

languageslanguages

Three metafunctional modes of meaningThree metafunctional modes of meaning

– ConstruingConstruing our experience in the world around us and inside us as our experience in the world around us and inside us as

meaning = meaning = ideational metafunctionideational metafunction (Meaning as “representation”: (Meaning as “representation”: participants, processes, circumstances)participants, processes, circumstances)

– EnactingEnacting our social roles and relations as meaning = our social roles and relations as meaning = interpersonal metafunctioninterpersonal metafunction (Meaning as “exchange”)(Meaning as “exchange”)

– PresentingPresenting the meaning construed and enacted as a flow of the meaning construed and enacted as a flow of information = information = textual metafunctiontextual metafunction (meaning as “message/text”: (meaning as “message/text”: “coherence, cohesion”)“coherence, cohesion”)

SFL approach (ctd.)SFL approach (ctd.)

These resonate with three These resonate with three contextual variablescontextual variables

- - field:field: “what”“what” is being talked about (social activity) is being talked about (social activity)

(participants, processes, circumstances)(participants, processes, circumstances)

- - tenor:tenor: “how” “how” the participants relate to each other inthe participants relate to each other in

relation to the activity relation to the activity (mood, modality, appraisal) (mood, modality, appraisal)

- - mode:mode: “how” “how” the oral/written text is structuredthe oral/written text is structured(theme and information structure; logical relationships in (theme and information structure; logical relationships in

clause complexes; grammatical metaphor)clause complexes; grammatical metaphor)

A trinocular conception of A trinocular conception of meaning/contextmeaning/context

Language comprisesLanguage comprises Ideational resources Ideational resources naturalize realitynaturalize reality

“ “Field knowledge” Field knowledge” enables participation in domestic, recreational, enables participation in domestic, recreational, academic, public, and professional activitiesacademic, public, and professional activities

Interpersonal resourcesInterpersonal resources negotiate social relationsnegotiate social relations ” ”Interpersonal knowledge” Interpersonal knowledge” enables a way of valuing and assessing enables a way of valuing and assessing these activities and enacting power and solidarity in relation to shared these activities and enacting power and solidarity in relation to shared values values

Textual resourcesTextual resources manage information flow manage information flow “ “Textual knowledge” Textual knowledge” enables ways of phasing the ideational and enables ways of phasing the ideational and interpersonal meanings into textures that are responsive/sensitive to interpersonal meanings into textures that are responsive/sensitive to the communicative demands of oral and written discourse.the communicative demands of oral and written discourse.

Metafunction in relation to language, Metafunction in relation to language, register, and genreregister, and genre

textual

interpersonal

ideational

field

tenor

mode

genregenre

register

language

Register and genre in SFLRegister and genre in SFL

Register – a “constellation”Register – a “constellation”A constellation of lexicogrammatical features that realizes a particular situational A constellation of lexicogrammatical features that realizes a particular situational

context, characterized in terms of ranges of field, tenor, and mode values in the context, characterized in terms of ranges of field, tenor, and mode values in the contexts within which they are locatedcontexts within which they are located

Genre – a “text type”Genre – a “text type” Recurrent configuration of meaningsRecurrent configuration of meanings Culture as a system of genresCulture as a system of genres Staged, goal-oriented social process, realized in textStaged, goal-oriented social process, realized in text

StagedStaged:: more than one phase of meaning to work through the genre more than one phase of meaning to work through the genre Goal-orientedGoal-oriented:: unfolding phases toward meeting expectations unfolding phases toward meeting expectations SocialSocial:: genres occur in social settings and are interactively undertaken genres occur in social settings and are interactively undertaken

The “teachability” of genresThe “teachability” of genres

Big ChallengesBig Challenges

What constitutes good L2 writing?What constitutes good L2 writing?

How can we link content knowledge and language How can we link content knowledge and language knowledge while setting some limits to the assessment of knowledge while setting some limits to the assessment of “content”?“content”?

How does instructed L2 writing develop? How does instructed L2 writing develop?

Does it/should it figure into the assessment of L2 writing Does it/should it figure into the assessment of L2 writing performance?performance?

Writing development in the GUGD: Writing development in the GUGD: a “case study”a “case study”

Writing tasks as genre-based writing tasksWriting tasks as genre-based writing tasks

Embedded in a four year integrated content oriented and task-based curriculumEmbedded in a four year integrated content oriented and task-based curriculum

Learning goal of advanced multiple literacies in several languagesLearning goal of advanced multiple literacies in several languages

Explicit pedagogy for writing Explicit pedagogy for writing

Writing development integrated into L2 development in all modalitiesWriting development integrated into L2 development in all modalities

Assessment of writing development in curricular contextAssessment of writing development in curricular context

Example: Level III writing task, within 5-level curriculum. Course focus “German Example: Level III writing task, within 5-level curriculum. Course focus “German stories, German histories” (Post WWII – present). Unit: fall of the wall.stories, German histories” (Post WWII – present). Unit: fall of the wall.

Foci for genre-based writing tasksFoci for genre-based writing tasks

Task Appropriateness:Task Appropriateness: Understood in termsUnderstood in terms of of– Generic movesGeneric moves

– Positioning of writerPositioning of writer toward an imagined reader toward an imagined reader audienceaudience

Content:Content: Understood in terms ofUnderstood in terms of– Information Information provided within each of the generic movesprovided within each of the generic moves

Language FocusLanguage Focus: : at 3 strata of language systemat 3 strata of language system– Discourse, sentence, lexicogrammaticalDiscourse, sentence, lexicogrammatical system system

Move to handout forMove to handout for End of curricular level expected writing performance profile: Levels I – End of curricular level expected writing performance profile: Levels I –

IV of the five-level curriculumIV of the five-level curriculum Specification of language foci at 3 strata of language system: Level IIISpecification of language foci at 3 strata of language system: Level III Genre “(Political) appeal” = Genre “(Political) appeal” = AufrufAufruf.. Text used in class to exemplify the Text used in class to exemplify the

genre. genre. Analysis of the text’s generic movesAnalysis of the text’s generic moves Lexicogrammatical analysis of the moves “problem statement” and Lexicogrammatical analysis of the moves “problem statement” and

“appeal for action” in terms of field, tenor, and mode.“appeal for action” in terms of field, tenor, and mode. General guidelines for the creation of genre-based tasksGeneral guidelines for the creation of genre-based tasks AufrufAufruf task sheet, in terms of task appropriateness (genre), content, task sheet, in terms of task appropriateness (genre), content,

and language focusand language focus Feedback sheet for the writing taskFeedback sheet for the writing task Guidelines for the assessment of writing performancesGuidelines for the assessment of writing performances

Linking content/context and language in Linking content/context and language in genre-based writing tasks: Pedagogical genre-based writing tasks: Pedagogical considerations considerations Purposeful texts approximating real-life writing in terms of topics, Purposeful texts approximating real-life writing in terms of topics,

possible relationships between writer and audience, purposes, textual possible relationships between writer and audience, purposes, textual demands in terms of level of L2 abilitydemands in terms of level of L2 ability

Focus on creating meaning (content/knowledge) in a textual world Focus on creating meaning (content/knowledge) in a textual world through and in and with language “forms” at all strata of the system through and in and with language “forms” at all strata of the system that that should be should be chosen for their meaning-making capacity chosen for their meaning-making capacity in relation to in relation to a genre.a genre.

Culturally authentic language use within the genre as culturally Culturally authentic language use within the genre as culturally validated forms of texts and registers with their preferred bundling of validated forms of texts and registers with their preferred bundling of lexicogrammatical featureslexicogrammatical features

Development of a competent non-native voice within stable, yet flexible Development of a competent non-native voice within stable, yet flexible generic structuresgeneric structures

Pedagogical considerations (ctd.)Pedagogical considerations (ctd.)

““Conceptual space” through metalanguage for a focus on content: Conceptual space” through metalanguage for a focus on content: manifestations of “field” in the various genre movesmanifestations of “field” in the various genre moves

Motivated development from more narrative genres, which privilege Motivated development from more narrative genres, which privilege congruent semiosis, to more abstract, essayistic, public genres, which congruent semiosis, to more abstract, essayistic, public genres, which privilege noncongruent (metaphorical) semiosis privilege noncongruent (metaphorical) semiosis long-term trajectory long-term trajectory toward the secondary discourses of the academytoward the secondary discourses of the academy (see e.g., Byrnes & Sprang, (see e.g., Byrnes & Sprang, Christie, Macken-Horarik, Martin, Rothery, Schleppegrell)Christie, Macken-Horarik, Martin, Rothery, Schleppegrell)

““Plausibility” for preferred lexicogrammatical choices – Pawley and Plausibility” for preferred lexicogrammatical choices – Pawley and Syder’s “native-like selection”, particularly at the phrasal and clausal Syder’s “native-like selection”, particularly at the phrasal and clausal level, increasingly more important for more advanced learners. level, increasingly more important for more advanced learners.

Link between L1 and L2 literacies; between learning a language and Link between L1 and L2 literacies; between learning a language and learning in other disciplinary areas.learning in other disciplinary areas.

Implications for assessmentImplications for assessment

The previous categories and pedagogical approaches to L2 The previous categories and pedagogical approaches to L2 writing provide ways of considering writing development as writing provide ways of considering writing development as well as writing qualitywell as writing quality

Assessment can be imagined at the intersection of a number Assessment can be imagined at the intersection of a number of the constructs SFL makes available: of the constructs SFL makes available:

What metafunctions are handled and how well?What metafunctions are handled and how well?

What major subaspects of these metafunctions are What major subaspects of these metafunctions are handled and how well?handled and how well?

Implications for assessment (ctd.)Implications for assessment (ctd.)

What language strata are competently used?What language strata are competently used?from word to phrase, to clause, to clause complex, to text, from word to phrase, to clause, to clause complex, to text, --- linking into cultural context, and from cultural context --- linking into cultural context, and from cultural context back to language back to language

What is the breadth and depth of registerial and generic What is the breadth and depth of registerial and generic resources available for use?resources available for use?

What is the user’s location on the cline between the What is the user’s location on the cline between the resources of the system and instantiation of those resources of the system and instantiation of those resources in a text?resources in a text?

Assessment (ctd.)Assessment (ctd.)

What kind of knowledge is being created: congruent – What kind of knowledge is being created: congruent – noncongruent forms of semiosis?noncongruent forms of semiosis?

What kinds of logical relationships are established and What kinds of logical relationships are established and how?how?In simplex clauses? In complex clauses? What forms of In simplex clauses? In complex clauses? What forms of grammatical are deployed? For what functions? grammatical are deployed? For what functions?

How is the phrase level expanded? Expansion of clausal How is the phrase level expanded? Expansion of clausal and phrasal capacityand phrasal capacity

How is information structure textured? themes, How is information structure textured? themes, hyperthemes; different forms of coherencehyperthemes; different forms of coherence

Assessment (ctd.)Assessment (ctd.)

… … and many more!and many more!

In each case, language use is linked to meaning, In each case, language use is linked to meaning, content, context.content, context.

BUT … BUT …

There are also necessary limits set to what is There are also necessary limits set to what is being assessedbeing assessed

Benefits of assessment for Benefits of assessment for teachers and learnersteachers and learners Predicts systematically the likely linguistic features Predicts systematically the likely linguistic features

that must be in focusthat must be in focus

Spells out the criteria for successful writingSpells out the criteria for successful writing

Helps teachers formalize and be more objective Helps teachers formalize and be more objective about their intuitive judgments about writing qualityabout their intuitive judgments about writing quality

Is formative and diagnostic with high potential for Is formative and diagnostic with high potential for washback into educational processeswashback into educational processes

Benefits (ctd.)Benefits (ctd.)

Clarifies the demands associated with a particular Clarifies the demands associated with a particular writing taskwriting task

Therefore has a high likelihood of enhancing L2 Therefore has a high likelihood of enhancing L2 writing developmentwriting development

Provides a language for reflecting on writingProvides a language for reflecting on writing

Links the goals of L2 writing with those in other Links the goals of L2 writing with those in other subject matter areassubject matter areas

The challenges for this approachThe challenges for this approach

Requires reconceptualizing language as a social semiotic Requires reconceptualizing language as a social semiotic and language development in terms of users’ ability to and language development in terms of users’ ability to deploy generically appropriate resources for meaning deploy generically appropriate resources for meaning making making

Requires much higher levels of awareness (and Requires much higher levels of awareness (and responsibility) by practitioners for knowing how responsibility) by practitioners for knowing how genres/texts work, that is, how meanings are realized in genres/texts work, that is, how meanings are realized in languagelanguage

Requires similar awareness on the part of the testing Requires similar awareness on the part of the testing community and a willingness to reconsider a number of community and a willingness to reconsider a number of constructs that have been central to assessment/testingconstructs that have been central to assessment/testing

How to take up the challenge …How to take up the challenge …

Until we know more about instructed L2 writing development, Until we know more about instructed L2 writing development, perhaps inquiry is best located within an educational perhaps inquiry is best located within an educational context context

- With an explicit curricular trajectoryWith an explicit curricular trajectory- That integrates all aspects of educational work: curriculum, That integrates all aspects of educational work: curriculum,

materials development, pedagogy, assessmentmaterials development, pedagogy, assessment- Uses to advantage the specificity that assessment Uses to advantage the specificity that assessment

demands regarding what constitutes competent L2 writing demands regarding what constitutes competent L2 writing while also informing assessment on the nature of L2 while also informing assessment on the nature of L2 development in writing,development in writing,

- And makes appropriate adjustments to improve And makes appropriate adjustments to improve educational outcomes.educational outcomes.

… … and gain large benefits and gain large benefits

It constitutesIt constitutes

Socially responsive workSocially responsive work– Outcomes of our programs: what is the value added on Outcomes of our programs: what is the value added on

the basis of our educative workthe basis of our educative work Socially responsible work Socially responsible work

– Development of sophisticated, multiple literacies in a Development of sophisticated, multiple literacies in a multilingual and multicultural world multilingual and multicultural world

Enhances L2 learning Enhances L2 learning – Links content and language learning, thereby providing Links content and language learning, thereby providing

perhaps the most deep-seated form of motivation for perhaps the most deep-seated form of motivation for learning languages.learning languages.