1 Using ALA-Score software to score essays and concept maps Wednesday June 22, 2005 Dr. Roy Clariana Penn State University email: [email protected]@psu.edu.

1

Using ALA-Score software to score essays and concept maps

Wednesday June 22, 2005

Dr. Roy Clariana

Penn State University

email: [email protected]

Web: www.personal.psu.edu/rbc4

"First we build the tools, then they build us!" -- Marshall McLuhan

2

goals Your take away:

• Some understanding of how/why it works

• Some ideas that you could implement on Monday morning in your classroom (or in you research)

• Hands-on using ALA-Score software to score student essays

• Hands-on using CMAP tools concept map software

Quick interest survey (teacher, researcher, grade level, subject, writing?, concept maps?)

3

Roy

H.S. math and science teacher (4 years) E.S. technology teacher (5 years) Ed. Consultant in CO, WY, UT (5 years) College professor at Penn State (8

years) www.personal.psu.edu/rbc4 The grand kids live in Denver and we

visit our cabin in summers here

4

Important note

Concept maps and essays are proven powerful generative instructional approaches!

But the main focus of this presentation is assessment rather than instruction, both ongoing in class (formative) and at the end of units of instruction (summative)

5

The “structure” of knowledge

Most tests measure declarative knowledge (e.g., knowing who, what when, where, why and how, etc.)

But essays and concept maps can measure the organization or structure of knowledge

essaysinterviews

tests

lungsoxygenate

blood

CO2artery

pulmonary

atriumventricle

veinlungs

oxygenate

blood

CO2artery

pulmonary

atriumventricle

vein

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

observations

6

Eliciting structural knowledge

Vygotsky (in Luria, 1979); Miller (1969) card-sorting approaches

Deese’s (1965) ideas on the structure of association in language and thought

Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis

Recent neural network representations (e.g., Elman, 1995)

7

Deese, free recall data

moth? – ___________

yellow? – ___________

bug? – ___________

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

8

Deese, free recall data (p.56)m

oth

inse

ct

win

g

bird

fly yello

w

flow

er

bug

coco

on

colo

r

blue

bees

sum

mer

suns

hine

gard

en

sky

natu

re

sprin

g

butt

erfly

moth 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15insect 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12wing 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13bird 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12fly 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11yellow 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5flower 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6bug 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4cocoon 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22color 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0blue 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2bees 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7summer 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0sunshine 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4garden 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2sky 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0nature 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3spring 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2butterfly 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

9-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on

2

moth

insect

wing

birdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

nature

spring

butterfl

Euclidean distance model

Derived Stimulus ConfigurationSimple visual representation from Multi-Dimensional Scaling of this Deese data

Novices will not have a developed knowledge structure

10

concept map example

Bahr (2004) using concept maps to teach English to German students

11

Side trip Wordnet: http://wordnet.princeton.edu/

http://wordnet.princeton.edu/cgi-bin/webwn

What is the Visual Thesaurus? – The Visual Thesaurus offers a unique visual display of the English language. Looking up a word creates an interactive visual map with your word in the center of the display, connected to related words and meanings.

For example, type “bird” in at: http://www.visualthesaurus.com/trialover.jsp

12

Tools to support concept mapping

Yellow stickies!! Pencil and paper may be best for your classroom

Software – PowerPoint is pretty good Inspiration is good but expensive CMAP tool is free, but your tech person

will have to agree to support it At least 22 other tools are available,

some free some not (see next slide)

13

Additional concept map automatic scoring approaches

CMap tools (IHMC) that we will use today C-TOOLS – Luckie (PI), University of Michigan NSF

grant available: http://ctools.msu.edu/ctools/index.html TPL-KATS – University of Central Florida (e.g., Hoeft,

Jentsch, Harper, Evans, Bowers, & Salas, 1990). TPL-KATS: concept map: a computerized knowledge assessment tool. Computers in Human Behavior, 19 (6), 653-657.

SEMNET – http://www.semanticresearch.com/about/ CMAT – Arneson & Lagowski, University of Texas,

http://chemed.cm.utexas.edu Plus 22 other non-scoring map tools

14

Some classroom uses of concept mapping

Usually involve individuals working alone, and involve in some way reading or writing text

Some collaborative strategies have been used

Lets look at a few to give you classroom ideas…

15

Using a student concept map to “capture” a text (i.e., text summary, note taking)

Textbook

Text text text text text text text text text text text

text

texttext

concept map notes

student

text

memo

As Homework?

Textbook


Textbook


Summary


16

Using a concept map in place of an outline in the writing process

essay


text

texttext

concept map notes

student

text

memo

Examples?

17

Using a student concept map to capture and organize research on a topic

textText text

text text tex Text text text text

textttext

texttext

concept map notes

student

text

memo

textText text


textt

www

video

Examples?

video

Report


18

Using a team concept map to capture and organize research on a topic

textText text


textttext

texttext

concept map notes

team

text

memo

textText text


textt

www

video

Examples?

video

Report


19

Example of dyad collaboration

text

texttext

text

text

texttext

text

Yergin

concept map artefact

Verbal discussion (taped)

Analyze the discussion

Blah blah blah blah Blah blah

Hannah

Blah blah blah blah Blah blah

Observations:On taskAbstract talk3-propositions/minQuestionAnswerCriticizeConflictElaborationCo-construction

van Boxtel, van der Linden, Roelofs, & Erkens (2002)

Problem: Sometimes unscientific notions are ingrained

Inferred:Active use of prior knowledgeAcknowledged problemsLook for meaningful relationsNegotiation

Shared objects play an important role in negotiation and co-construction

The incredible value of talk!

Note the attentional effects of the artifact

20

summary

Maps into text Text into maps There is an intimate relationship between

the two and both can provide measures of structural knowledge

Have you considered concept maps

21

ALA-Reader

… an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood …

atrium

contract

lungs

cleansed oxygenated

P artery

P valve

Text CMAP

Link array

22

Clariana & Koul

Participants – A group of 24 practicing teachers enrolled in CI 400

Lesson – while researching the topic “the structure and function of the heart” online, students completed concept maps using Inspiration software and later wrote an essay on this topic from their maps.

Clariana, R.B., & Koul, R. (2004). A computer-based approach for translating text into concept map-like representations. In A.J.Canas, J.D.Novak, and F.M.Gonzales, Eds., Concept maps: theory, methodology, technology, vol. 2, in the Proceedings of the First International Conference on Concept Mapping, Pamplona, Spain, Sep 14-17, pp.131-134. http://cmc.ihmc.us/papers/cmc2004-045.pdf

23

Posttests

Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores (

http://www.personal.psu.edu/rbc4/frame.htm)

Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance agreement

with an expert ALA-Reader PFNet link scores (from 1 to 5)

(so far, only looked at essay scores)

24

ALA-Rater PFNet scores

The scores for each text and rater-pair are shown ordered from best to worst.

ALA-Reader scores were moderately related to the combined text score, Pearson r = 0.69, and ranked 5th overall.

ALA

-Rea

der

25

other tools to score Essays

ETS – PEG (Project Essay Grade), e-rater, Criterion and other products… http://www.ets.org/research/erater.html

Walter Kintsch (and Landau) at CU-Boulder – Latent semantic analysis, many uses, i.e., score online training for the Army - http://lsa.colorado.edu/

Vantage Learning essay scoring products - http://www.vantagelearning.com/

ALA-Reader: http://www.personal.psu.edu/rbc4/score.htm

26

other tools to score Essays

http://Knowledge-Technologies.com http://www.ets.org/research/erater.html http://ericae.net/betsy/ http://torrseal.mit.edu/effedtech/ (the MIT physics

homework tutor) http://knowledge-technologies.com/hrw.html (Preparation

for Standardized Writing Tests) http://knowledge-technologies.com/KeysDemo/

Keys2.1D.html (Prentice Hall – Using the IEA to improve textbook learning)

http://knowledge-technologies.com/tradoc.html (U.S. Army TRADOC – Automated Essay Scoring for Officer Leadership training)

27

Comments and Questions

About foundation of the idea and its use Any ideas on how you have or will use

maps?

Next: demo ALA-Reader

28

Demo ALA-Reader Download ALA-Reader.exe from

www.personal.psu.edu/rbc4 Create terms file (can include 2 synonyms) Create 2 expert baseline reference texts

called expert1.txt and expert2.txt (i.e., Instructor, best student)

Use it Files created

• Summary file called report.txt• Multiple *.prx files (PRX folder)• CMAP files

29

DEMO of CMAP Tools

Lets walk through it together, follow me step by step

30

Example of an online collaboration

p.22, Chiu, Huang, & Chang (2000)

text

texttext

text

text

texttext

text

Hannah(lead)

Jari

Yergin

H: WE should …J: Did you see…Y: Yeah, but …Etc.Etc.

concept map artefact

Online chat

Analyzed the chat textAnd the concept map

creates

concept map session lasted 80 minutes. 3 x 12 online groups, communicate by chat, 745 messages were exchanged (avg. of 62 per group).

Only the lead could alter the concept map

The ‘other 2 members used chat to “advise”

Researchers

31

Demonstration – Cmap tools synschronous collaboration

(see the Project handout)

32

Brainstorm, then make the map

Open IHMC Cmap tools Fill in personal information on first use (I’ll tell

you what to type in here) Click Other Places Open brainstorm file Click collaborate icon

if necessary Type in your first name Collaborate

33

Demo 1

IHMC Public Cmaps conv v2 on Jan 22 2004

34

Oulu EDTECH Public

Demo 1

35

Final Questions

Time to play with the software

36

My research interests

Mind map assessment – automatic scoring software tool called ALA-Mapper http://www.personal.psu.edu/rbc4/ala.htm

Essay assessment – automatic scoring software tool called ALA-Reader http://www.personal.psu.edu/rbc4/score.htm

for Latent Semantic Analysis (LSA) see: http://www.personal.psu.edu/rbc4/frame.htm

prototypes

37

Poindexter and Clariana

Participants – 23 undergraduate students in intro EdPsyc course (Penn State Erie)

Food rewards for participation Setup – complete a demographic survey

and how to make a concept map lesson Text based lesson interventions –

instructional text on the “heart” with either proposition specific or relational lesson approach

Poindexter, M. T., & Clariana, R. B. (in press). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press. link to doc file

#1st

38

Treatments Relational condition, participants were required to

“unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content

Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).

39

Posttests

Concept map (use 26 terms provided)• Link-based common scores

• Distance-based common scores

Multiple-choice tests (Dwyer, 1976)• Identification (20)

• Terminology (20)

• Comprehension (20)

40

Means and sd

Treatments Posttests ID TERM COMP Map-prop Map-assoc control 15.1 12.3 7.3 14.1 9.0

(4.4) (4.6) (5.4) (4.6) (3.6)

proposition- 16.3 14.6 13.8 16.5 11.5 specific

(5.6) (5.7) (3.7) (8.3) (3.4)

relational 17.0 12.7 12.4 13.9 10.7 (2.6) (3.5) (3.0) (9.4) (4.6)

Map-link Map-dist

41

Analysis

MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc.

COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance.

Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table).

42

Correlations

ID TERM COMP Prop ID -- TERM 0.71 -- COMP 0.50 0.74 -- Map-prop 0.56 0.77 0.53 -- Map-assoc 0.45 0.69 0.71 0.73 All sig. at p<.05

Compare to Taricani & Clariana

next

Map-link

Map-linkMap-distance

43

Taricani and Clariana – Replication of Poindexter and Clariana

Taricani, E. M. & Clariana, R. B. (in press). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), in press.

TermComp

Link data 0.78 0.54

Distance data 0.48 0.61

44

Compare these two . . .

Poindexter & Clariana TermComp

Link data 0.77 0.53


Taricani & Clariana TermComp

Link data 0.78 0.54


45

Clariana, Koul, & Salehi

Participants – A group of 24 practicing teachers enrolled in CI 400

Lesson intervention – while researching online, completed concept maps in pairs (newsprint & yellow stickies) to describe the structure and function of the heart and then individually wrote essays on this topic from their maps.

Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press.

# 2nd

46

Posttests

Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores (

http://www.personal.psu.edu/rbc4/frame.htm)

Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance

agreement with an expert

47

Correlation matrix

Map Essay LSA LinkMap 1 Essay 0.49 1 LSA 0.31 0.73 1 Link data 0.36 0.76 0.83 1 Distance data 0.60 0.77 0.71 0.82 1

p < .05 shown in boldface type.

Human Computer

Many investigators have noted the close relationship between maps and essays.

1 Using ALA-Score software to score essays and concept maps Wednesday June 22, 2005 Dr. Roy Clariana Penn State University email: [email protected]@psu.edu.

Documents

concept mapswednesday

structure of associations

interactive visual map

text structure

visual thesaurus

concept mappingyellow

deese datanovices

english language