Professional Development Activity Log: A New Approach to Design, Measurement, Data Collection, and Analysis AERA Annual Meeting San Diego April 13, 2004.

Professional Development Professional Development Activity Log: A New Approach Activity Log: A New Approach to Design, Measurement, Data to Design, Measurement, Data Collection, and AnalysisCollection, and Analysis

AERA Annual Meeting

San Diego

April 13, 2004

Longitudinal Study to Measure Longitudinal Study to Measure Effects of MSP Professional Effects of MSP Professional Development on Improving Development on Improving Math and Science InstructionMath and Science Instruction

Math and Science Partnership

A collaborative study conducted by: A collaborative study conducted by: Council of Chief State School Officers (CCSSO)

American Institutes for Research (AIR)

Wisconsin Center for Educational Research (WCER)

AuthorsAuthors

Kwang Suk Yoon, AIR Reuben Jacobson, AIR Mike Garet, AIR Bea Birman, AIR Meredith Ludwig, AIR

Research QuestionsResearch Questions

To what extent is the quality of the professional development supported by MSP activities consistent with research-based definitions of quality (e.g., content focus, active learning, coherence, collective participation, and sustained efforts) (Garet et al., 2001)?

What effects do teachers' professional development experiences have on instructional practices and content taught in math and science classes? Are high-quality professional development activities more likely than lower-quality activities to increase the alignment of instructional content with state standards and assessments?

Logic ModelLogic Model

Implementation ofProfessional Development Content Focus

Collective Participation;Active Learning;

Coherence;Sustained Effort

Pre-PD:

Alignment ofInstruction with Content

Standards; Instructional Practice

Teacher Characteristics: Background Variables,Prior PD ExperiencesTarget Class Students:

DiversitySchool Culture:

Trust

Post-PD:

Alignment ofInstruction with Content

Standards; Instructional Practice

Year 1 Year 2 Year 3Year 0

PDAL

Survey of Enacted Curriculum wave 1

Survey of Enacted Curriculum wave 2

ParticipantsParticipants

Four MSP projects were selected for the study. In each project, we are collecting data with teachers mostly in middle schools or middle grades about their professional development in mathematics and science education.

N=472 teachers

Survey of Enacted Curriculum Survey of Enacted Curriculum (SEC)(SEC)

Instructional practice (e.g., instructional time in target class)

Content coverage and alignment: – Instructional time on topics and subtopics– Expectation for students (e.g., memorize facts,

perform procedure, or solve non-routine problems) Past experiences in professional development Teacher characteristics (e.g., gender,

teaching experience)

Why PDAL?Why PDAL?

Gathers accurate, time-sensitive information; Minimizes recall problem with retrospective reports

Collects disaggregate information about specific PD activities – Reduces bias introduced by gross data aggregation

Generates context sensitive questions Is able to tailor technical assistance to teachers

based on their response patterns Allows teachers to review their own logs –

Teachers can reflect on their own PD experiences

Professional Development Activity Log Professional Development Activity Log (PDAL)(PDAL)

Teachers create an ongoing monthly log of any professional learning activity in which they participate

Longitudinal data collected over 15 months Web-based, self-administered log Aligned with SEC (e.g., content coverage) Inclusive approach to professional development

– Includes MSP-sponsored and non-MSP-sponsored activities

– Documents one-time and recurring activities– Captures both formal and informal activities

PDAL Entries PDAL Entries

Name of activity Number of hours spent on each activity and its duration Whether the activity is a one-time or continuous event (e.g.,

recurring over a number of months) Type of activity (e.g., workshop, summer institute, study group) Purpose of activity (e.g., strengthening subject matter knowledge) PD quality features (e.g., active learning, coherence, collective

participation) Content focus (e.g., algebraic concepts: absolute values, use of

variables, etc.) Instructional practice – instructional topics covered in each activity

(e.g., use of calculators, computers, or other educational technology)

Analysis of PDAL DataAnalysis of PDAL Data

Implementation analysis – Patterns of responses to monthly logs – Response rates; sample attrition; extent of missing data

Descriptive analysis– Patterns of teachers’ PD experiences– Correlates of high-quality PD activities– Latent classes of teachers based on their PD

experiences

Impact analysis– Assess the impact of PD on math & science instruction

Table 1: PDAL Data Structure:Table 1: PDAL Data Structure:Disaggregated log data: Teacher by Disaggregated log data: Teacher by

activity by timeactivity by time

Teacher Activity Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 # of logs

Mike AA 1 1B 1 1 1 1 1 5C 1 1

Julie AA 1 1D 1 1 1 1 1 1 6

Reuben EE 1 1 2Kwang EE 1 1 1 3

F 1 1Bea G 1 1

Table 2: PDAL Data Structure:Table 2: PDAL Data Structure:Activity-level data aggregated across Activity-level data aggregated across

teachersteachers

Teacher Activity Jul-03 Aug-03 Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 # of logs

AA 2 0 0 0 0 0 0 0 0 2B 0 0 1 1 0 1 1 0 1 5C 0 0 0 0 0 1 0 0 0 1D 0 1 1 0 1 1 1 1 0 6

EE 1 2 2 0 0 0 0 0 0 5F 0 0 1 0 0 0 0 0 0 1G 0 0 0 0 1 0 0 0 0 1

# of teachers 3 3 5 1 2 3 2 1 1 21

Implementation AnalysisImplementation Analysis

# of Activities (over 9 months)– Range: 1 - 21– Mean: 3.1

# of Logs (over 9 months)– Range: 1 - 31– Mean: 4.9

Sessions– Per session: mean 12 min.– Per teacher over 9 months: mean of 50 min.

83% finished in one session; 13% finished in two sessions.

Implementation AnalysisImplementation Analysis

Timing of log entries:– Weekends are least popular. Tues. - Thurs. are

most popular– Spike at the beginning of each month

User understanding of the instrument– 95% of logs created were fully completed

Communication:– Info Packets– Phone calls, emails, letter reminders– Helpline

Table 3: Duration and Contact Hours of Table 3: Duration and Contact Hours of PD ActivitiesPD Activities

Variable Source N Mean SD Min Max

Duration 1

PDAL: Logs 1121 1.84 1.30 0 4

PDAL: Teachers 229 1.87 1.00 0 4

Contact Hours

PDAL: Logs 1121 15.72 20.98 1 160

PDAL: Teachers 229 18.61 20.91 1 150

Total Contact Hours

PDAL: Teachers 229 76.97 95.17 1 900

Note: 1 How many days: 0=less than a day, 1=one day, 2=2-4 days, 3=a week, 4=entire month.

Table 4: PD Quality Construct Table 4: PD Quality Construct Active LearningActive Learning

Source N Mean SD Min Max

PDAL: Logs 1107 1.08 0.76 0 3

PDAL: Teachers 225 1.15 0.63 0 3

SEC1: Teachers 382 1.26 0.67 0 3

Sample Items:

Observe demonstrations of teaching techniques?

Review student work or score assessments?

Receive coaching or mentoring in the classroom?

Give a lecture or presentation to colleagues?On the scale of 0=never, 1=rarely, 2=sometimes, 3=often

Table 5: PD Quality Construct Table 5: PD Quality Construct CoherenceCoherence

Source N Mean SD Min Max

PDAL: Logs 1067 2.30 0.66 0 3

PDAL: Teachers 224 2.28 0.49 0 3

SEC1: Teachers 359 1.85 0.71 0 3

Sample Items:

Designed to support the school-wide improvement plan adopted by your school?

Consistent with your department or grade level plan to improve teaching?

Consistent with your own goals for your professional development?

Based explicitly on what you had learned in earlier professional development activities?

On the scale of 0=never, 1=rarely, 2=sometimes, 3=often

Table 6: PD Quality Construct Table 6: PD Quality Construct Collective ParticipationCollective Participation

Source N Mean SD Min Max

PDAL: Logs 1105 0.64 0.74 0 2

PDAL: Teachers 225 0.72 0.61 0 2

PDAL: Teachers(Agg) 225 1.21 0.77 0 2

SEC1: Teachers 382 1.18 0.86 0 2

Items:

I participated with most or all of the teachers from my school.

I participated with most or all of the teachers from my department or grade level.

On the scale of 0 - 2: Sum of the two collective participation items

Conclusion:Conclusion:Revisiting the Logic ModelRevisiting the Logic Model

Implementation ofProfessional Development Content Focus

Collective Participation;Active Learning;

Coherence;Sustained Effort

Pre-PD:

Alignment ofInstruction with Content

Standards; Instructional Practice

Teacher Characteristics: Background Variables,Prior PD ExperiencesTarget Class Students:

DiversitySchool Culture:

Trust

Post-PD:

Alignment ofInstruction with Content

Standards; Instructional Practice

Year 1 Year 2 Year 3Year 0

PDAL

Survey of Enacted Curriculum wave 1

Survey of Enacted Curriculum wave 2

Contact InformationContact Information

Kwang Suk Yoon

(202) 403-5358

ksyoon@air.org

Reuben Jacobson

(202) 403-6925

rjacobson@air.org

Visit us in the future

www.pdal.net