1 A demo of free essay scoring software Levels: 6-12 PS 117 Roy B. Clariana – Penn State U. [email protected] Casper College.

1

A demo of free essay scoring software Levels: 6-12 PS 117

Roy B. Clariana – Penn State [email protected]

http://www.personal.psu.edu/rbc4Casper College January 14th, 2006

2 of 21

Presentation Goals This presentation will demonstrate how

to download and use a software tool called ALA-Reader to score students’ biology essays

To recruit action researchers from this audience

3 of 21

Personal Profile High school math and science teacher in the US

and Kenya (4 years) Energy and environment education curriculum

developer Elementary school technology coordinator (5 years) Technology education consultant for 50 schools in

Colorado, Wyoming, and Utah (3 years) Director of a technology integration project in the

United Kingdom (1 year) Software development project lead – software,

tools, web-based tools (1 year) Penn State – Associate Professor (9th year)

4 of 21

Presentation flow Quick ALA-Reader demo – to see it work Brief look at its genesis and how it works Step-by-step demo – how you set it up

and use it in your classroom Monday morning

Review the data files produced One more run Final questions, Show of hands

5 of 21

Quick ALA-Reader Demo Place student essays, terms file, expert

referent essay, and ALA-Reader software into the same folder (click)

Double click ALA-Reader Click the Mark L button Type in the file name of a student’s

essay Observe scores

6 of 21

Genesis of ALA-Reader We are interested in concept maps, and

developed software called ALA-Mapper to score concept maps

We noticed that teachers associate concept maps and essays. (i.e., concept maps are a good organization tool.)

So we developed ALA-Reader to score essays based on ALA-Mapper

7 of 21

Concept map validity

Concept maps are generally considered to be valid and reliable measures of science content knowledge (Ruiz-Primo, Schultz, Li, Shavelson, CREST in California, Rye and Rubba (2002), Jonassen, Beissner, & Yacci, 1993. . .).

essaysmultiple-choice

tests

lungsoxygenate

blood

CO2artery

pulmonary

atriumventricle

veinlungs

oxygenate

blood

CO2artery

pulmonary

atriumventricle

vein

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

lungs

oxygenateblood

removeCO2

pulmonaryvein

leftatrium

8 of 21

What raw data can a computer “extract” from a Concept map?

Term counts – in open-ended maps, were all of the required terms included

Propositions – a line (link, edge) connecting two terms

Link labels – the labels on the line Associations – geometric distance

between pairs of terms. Small values indicate stronger relationship.

9 of 21

.. map w/ proposition array

Most studies use only link (edge) information, usually called “propositions”.

LA L OX PA PV DOX RVleft atrium -lungs 0 -oxygenate 0 1 -pulmonary artery 0 1 0 -pulmonary vein 1 1 0 0 -deoxygenate 0 1 0 0 0 -right ventricle 0 0 0 1 0 0 -

Proposition Array

lungs

oxygenate deoxygenate

pulmonary artery

pulmonary vein

left atrium

right ventricle

Semantic Map

(n2-n)/2 pair-wise comparisons

10 of 21

.. map w/ association array

Concept Map Association Array

LA L OX PA PV DOX RVleft atrium -lungs 120 -oxygenate 150 36 -pulmonary artery 108 84 120 -pulmonary vein 72 102 114 138 -deoxygenate 156 42 54 86 144 -right ventricle 66 102 138 42 114 120 -

.

.

.

.

.

.

.

lungs

oxygenate deoxygenate

pulmonary vein

left atrium

right ventricle

S-Mapper

pulmonary artery

11 of 21

Convert raw map data into scores Currently, we use a data reduction and

comparison approach called Pathfinder network representation (UNM, Schanveldt, 1990).

PFNets describe the least weighted path to connect the terms in the data proximity array

Scores are established by comparing the participant’s PFNet to a referent (expert) PFNet, and calculating the number of common links (the intersection)

12 of 21

We use the same approach for ALA-Reader

… an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood …

atrium

contract

lungs

cleansed oxygenated

P artery

P valve

Text PFNet

Proposition array

13 of 21

Our papers on concept maps and essays

Add cmc conference paper Koul, R., Clariana, R. B., & Salehi, R. (2005). Comparing

several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273. link

Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 54, 61-78.

Clariana, R.B., Koul, R., & Salehi, R. (2006). The criterion-related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press.

Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press.

14 of 21

Step-by-step1. Create the writing prompt2. Students write essays, when done, they save

it in a specific format3. Teacher creates key terms with synonyms

and saves as terms.txt4. Teacher creates one or two expert referent

essays5. Put all into the same folder and run ALA-

Reader6. Collect scores from the report file

15 of 21

Step 2. Student essay file formatThe Dance of LifeStudent 5

The heart is the major component of the circulatory system.Its function is to circulate blood, which is comprised of both blood cells and plasma, throughout the body.Two types of blood vessels are used to circulate blood.Arteries are used to carry oxygen rich blood throughout the body, while veins are used to carry oxygen deficient blood to the heart.The heart is one of the most complex organs in the body.One of its primary jobs is to oxygenate blood for the body.Here is how it works.As stated earlier, veins carry oxygen deficient blood to the heart.The blood flows into the right atrium, then the right ventricle, and then the pulmonary artery pumps blood to the lungs for oxygen.Once the blood is oxygenated, pulmonary veins carry blood back to the heart, where it flows from the left atrium to the left ventricle.The aorta pumps oxygen rich blood throughout the body, then the veins carry oxygen deficient blood back to the heart, where the cycle begins again.Quite literally, the heart is “the heart” of the circulatory system.It is amazing to think that such a seemingly simple organ is so complex.

30 max

16 of 21

Step 3. create terms.txt fileaorta aortaaortic_valve aortic_vbody bodycleanse clean carboncontract contracdiastolic diastolendocardium endocarepicardium epicarinferior_vena_cava inferiorleft_atrium left_atrileft_ventricle left_ventlungs lungmembranes membranmitral_valve mitral bicusp atrioventricularmyocardium myocardoxygen oxygenpericardium pericardpulmonary_artery pulmonary_artpulmonary_valve pulmonary_valpulmonary_veins pulmonary_veirelax relaxright_atrium right_atriright_ventricle right_ventsuperior_vena_cava superiorsystolic systoltricuspid tricusp

• 30 terms maximum• Space to separate synonyms• _ to show spaces30 max

fullpattern

17 of 21

Step 4. create expert1.txt fileHeart EssayExpert 1

Blood flows from the extremities towards the inferior vena cava and superior vena cava.Blood enters the right atrium from the inferior vena cava and superior vena cava.Flow passes from the right atrium through the tricuspid valve to the right ventricle.From the right ventricle, blood flows oneway through the pulmonary valve onto the pulmonary

artery.The blood flows through the pulmonary artery to the lungs where it is cleansed and oxygenated.Blood flows from the lungs back to the heart through the pulmonary veins.Blood then enters the left atrium from the pulmonary vein.Flow passes from the left atrium through the mitral valve into the left ventricle.Blood then flows out of the heart through the aortic valve into the aorta.The blood in the aorta flows to all parts of the body.

contract and systolicdiastolic and relax

membranesendocardium and myocardiumepicardium and pericardium

30 max

18 of 21

Step 5. Run ALA-Reader Place student essays, terms file, expert

referent essay, and ALA-Reader software into the same folder (click)

Double click ALA-Reader Click the Mark L button Type in the file name of a student’s

essay Observe scores

19 of 21

Step 6. Use scores See report.txt or L_report.txt

20 of 21

ALA-Reader future Current beta version forms proposition

array based on sentence-level and also linear associations

Add other ways to parse the text, e.g., pure linear, reverse linear, double-chunk, etc.

Add a graphing utility to the tool Download from my web site at

www.personal.psu.edu/rbc4

21 of 21

Questions Questions Show of hands… One-on-one after the session, also email

me with any questions, etc.