AAER Conference Nov 04 2010 Symposium on Limitations of Scientific Knowledge in Educational Research: Bias, Non-Significant Findings & Knowledge Representation.

AAER ConferenceNov 04 2010

Symposium onLimitations of Scientific Knowledge in Educational Research: Bias, Non-Significant Findings & Knowledge Representation

Jaya KannanLarry Lutsky

Yasmine AlwanLisa Bauer

Non-Significant Findings and a Research Response in Self-Directed Learning

Analyzing students’ articulation of learning goals:

How Problems in Research Led to Solutions in Teaching

Jaya KannanLarry Lutsky

Yasmine AlwanLisa Bauer

Purpose of the presentation

Share research work on LGsDiscuss rubric developmentExamine what didn’t work and what

was learnedElicit feedback from participants

Brief History of Learning GoalsAndragogy: theories of adult learning

(Knowles)Learning goals individualize learning &

contribute to the development of self-directed learning

The Purpose Centered Model

Relevance of Learning Goals Constructive Action teaching practicesSeeing connections in goal setting in

different contexts

Method Hypothesis

Sample Procedures

Data collection Method of analysis Type of design

Rubric

Evolution of the RubricNeed to design a rubric Method of working – cyclicalEstablishing inter-rater reliability

Scale items Operational definitionsMath and writing – learning contextsExamples for each scale

Evolution of the rubricDisagreements/clarification:Defining SpecificityDefining complexityWhat constitutes measurability?

Should we use a zero scale?

Evolution of the rubric

Resolutions• Defining Specificity

– Break it up into content and syntax

• Defining complexity– Establishing parameters for what is/is not a

multi-layer goal –case by case evaluation

• Should we use a zero scale?– Only for measurability

• What constitutes measurability?– Arrived at keywords

Rubric – early version1 2 3

Specificity Not specific – I want to learn math

Moderately specific – I want to understand mathematical concepts

Highly specific – I want to learn basic algebra. I want to get an “A” in my course.

Complexity Low complexity – I want to learn how to multiply and divide fractions.

Moderate complexity – I want to learn about statistics so I

can analyze the data from my CA

Highly complex – I want to learn algebra so that I can apply the concepts in my course and also apply outside the class room.

Measurability Low measurability – I want to have more confidence in my ability.

Moderate measurability – I want to improve my math skills

High measurability – I want to be able to solve quadratic equations. I

want to be able to calculate the mean and standard deviation. I want

to be able to determine if there is subject-verb agreement

Rubric – final version1= Low 2 3 = High

Specificity –Content

Specificity - Syntax

I want to learn math.I want to understand mathematical concepts.Broad area superset

I want to learn basic algebraLess broad subset

I want to solve algebraic equationsNarrow subset

I want to understand fractionsDescribes an action

To improve my math skills by completing math problemsDescribes an action in detail

To learn sign numbers so I can solve equations correctlyDescribes more than one action with detail

Complexity I want to learn fractions1 layer of goal

I want to learn how to multiply and divide fractions

>=2 layers of goal

I want to learn about statistics so I can analyze data from my CA

>= 2 layers of goal with application

Measurability I want to improve my math skillsAny desirable changeStated demonstration of ability

I want to reduce the numbers of errors I make while solving math equations by the end of the semesterSomething quantifiable

By the end of the semester, I want to reduce the number of errors I make and complete more work

independently. 2 layers - Quantifiable, time frame

Inter-Rater Agreement(N = 4 raters)

94.4%97.2%86.1%

• Percent of Pairs of Raters within One Score Point of Each Other

58.3%58.3%47.2%• Percent of Pairs of Raters

with Exact Agreement

0.470.670.67

• Mean Inter-Rater Difference

MeasurabilityComplexitySpecificityAgreement Measure

Results of data collected

Results of data collected

Overall change in means over semesters Change in first 2 means, divided by # of sessions

Results of data collected

Results of data collected

Flaws/Challenges in MethodologyApplying hypothesis to problematic data

Retroactive analysis Abundant data Edu purposes

Lack of standardization Goals Raters

Measurement Definitions of scale itemsConsensus reached through extensive

discussion

What was learned

Standardization is difficult to achieve in teaching practiceShould it be standardized in the first

place? Limits in student knowledge and skillsActual work vs. projected work

Teaching styles

Follow up: Next Steps

Discussions with specialists improve understanding

(using the rubric)uniformityRevising goal writing formBuilding in measurability

Generating templates of goals

Follow up: Next Steps

Collaboration of LEC and facultyParticipant feedback on rubric &

studyConnection between research and

teaching