Evaluating the effectiveness OF PROFESSIONAL LEARNING

Access Handouts at: http://tinyurl.com/catxtsh

EVALUATING THE effectiveness OFPROFESSIONAL LEARNING

Jacqueline Kennedy Associate Director of Strategic Initiatives

Learning ForwardRaleigh, NC

December 12-13, 2012

Our Agenda

• Evaluating Professional Learning Essentials

#1

• Revisiting the Process of Educator Change

#2 • Delving Deeper into Evaluation Components

#3

Our Outcomes

• Define evaluation in relationship to professional learning;

• Examine the process of teacher change and its impact on learning; and

• Acquire strategies, tools, and resources to assist in evaluating professional learning.

Our Essential Question

• How can evaluating professional learning leverage school, school system, and state improvement effort?

• How will I align professional learning objectives to measurable short, medium and long-term results for educators and students?

• How will I collaborate with others to construct a framework that outlines a detailed plan for evaluation?

• How do I incorporate evaluation into my work and normative practice?

Our Norms

• Share experiences to enrich others.• Ask questions.• Learn by doing.• Set aside any preconceived notions about

evaluating professional learning.• Apply to your own work.

THINK S – A – L – S – A !

Where are we now?

• Independently, read the case study. • At your table discuss components of your

professional learning evaluation.• Chart your process, considering actions, steps, and

information needed to address questions posed. • Identify questions that emerge about evaluating

professional learning. • Be prepared to discuss your work.

Burning Questions

• Select one question shared at your table that might have implications for the whole group.

• Be prepared to share it.• Post all other questions

(one per sticky note) on the chart.

The Standard

Professional learning that increases educator effectiveness and results for all students uses a variety of sources and types of student, educator, and system data to plan, assess, and evaluate professional learning.-Standards for Professional Learning, 2011

Evaluation is:

A systematic, purposeful process of studying, reviewing, and analyzing data gathered from multiple sources in order to make informed decisions about a program. – Killion, 2008

The systematic investigation of merit and worth. – Thomas Guskey, 2000

Link to Student Results

1. Standards-based

professional learning

2. Changes in educator

knowledge, skills, and

dispositions

4. Changes in studentresults

3. Changes in educator practice

4 Key ComponentsSystematic(Established process)

How rigorous is the process?Is it conducted in accordance with standards and guidelines?

Standards(Predetermined criteria)

Does it have merit and/or worth?Does it meet predetermined standards of success?

Audience Who will use the evaluation?For whom is the evaluation being done?

Intended Uses

How will the evaluation be used?What decisions will be made as a result of the evaluation?

Which best describes your experience?A B

Externally driven and designed Internally driven and designed

Summative evaluations only Planning, formative, and summative evaluation

Event-based Program-based Looking for answers/solutions form others

Discovering or creating solutions and alternatives with others

Feared Embraced

Filed/shelved Used

Done as an afterthought Planned from the beginning

Documentation Evaluation

Process-focused Results-driven

Presentation of results Reflective dialogue

Shifting PerspectivesFrom To

Externally driven and designed Internally driven and designed

Summative evaluations only Planning, formative, and summative evaluation

Event-based Program-based Looking for answers/solutions form others

Discovering or creating solutions and alternatives with others

Feared Embraced

Filed/shelved Used

Done as an afterthought Planned from the beginning

Documentation Evaluation

Process-focused Results-driven

Presentation of results Reflective dialogue

Group Think

1. How do identified shifts influence you and your work in North Carolina?

2. What aspect of evaluating professional learning do you find essential?

4 Purposes

1. Determine merit and/or worth – determine overall value of program

2. Assess impact – determine overall effectiveness of a program

3. Identify improvements– determine program strengths and weaknesses

4. Provide accountability – determine evidence of compliance and resources

Worth

The extent to which a program(s) or activity(ies) is/are essential to a school’s, district’s, agency’s, or individual’s mission. Worth is an indication of the program’s or activity’s perceived value to constituents or to a single individual.

– Do participants value this program?

MeritThe value of the program is judged by comparing its performance against established standards of excellence in the profession.• Did the program

achieve its goals and meet its standard(s) of success?

Means goals vs. End goalsMeans End

Provide professional learning (Activities or strategies employed to achieve a result)

e.g. Train 100% of the teachers on differentiation.

Increase student achievement results(Results or behavioral changes expected)

e.g. Student achievement increase by 10% on a reading inventory.

Programs Demonstrating Merit

May or may not assess the impact of the program on student achievement results.

– Will — if the goal is to increase student achievement (results)

– Will not — if the goal is to provide service

Evaluation to Assess Impact

• Requires program goals to be about increasing student achievement results.

• Requires a theory of change to explain how the program will produce intended results.

Program vs. Event Evaluation

A program is a set of purposeful, planned actions and the support system necessary to achieve the identified goals. Effective professional learning programs are ongoing, coherent, and linked to student achievement.

Example

1. A workshop on Common Core standards.2. A workshop on Common Core standards

coupled with planned planned classroom-support, ongoing coaching and feedback, demonstrations, and observations in other classrooms incorporated into a comprehensive system, ongoing plan designed to develop educator knowledge, attitudes, skills, aspirations, and behaviors.

3 Types of Evaluation

1. Planning – before program design to provide information on conditions or needs to address

2. Formative – during implementation to provide information on whether the program is working as designed

3. Summative – after completion to provide information on outcomes or overall impact

Note to Self

• How are key components, purposes, and types essential to understanding evaluation?

• What about evaluating professional learning interests your most?

Black Box Evaluations

Input Output

Actions Results

Glass Box Evaluations

?A simplistic approach to professional learning evaluation that fail to amplify the

underlying theory and operation of the professional learning program.

A comprehensive approach to professional learning evaluation that illuminates how professional learning program components interact to produce results.

Black Box

ProfessionalLearningAction

Student Achievement

Results?

Focus on outputs rather than what

occurs in the program or what is presumed to be causing those outcomes and why.

Student Achievement

ResultsProfessional

Learning

CurriculumDevelopment

Nonacademicfactors

?

Black Box Focus on inputs and fail to shed light on HOW a program’s

activities and resources interact to

produce results.

Student Achievement

Results

ProfessionalLearning

Actions

Glass Box Focus on what occurs and how it occurs

within the program.

Glass Box

Student Achievement

ResultsProfessional

Learning

Coaching/Follow-upInstructional ResourcesImplementation MonitoringStudent Assessment

Focus on illuminating factors contributing to transformation

process.

Black Box vs. Glass Box

What is the difference?

BREAKSee you back in 15 minutes.

Your Evaluation Process

Write 4-5 sentences that describe the evaluation process/steps you currently use as a leader member responsible for assisting others with evaluating professional learning. • Be specific.• Be frank. • Write in complete sentences.

8 Smooth Evaluation Steps

Planning1. Assess

Evaluability

2. Formulate Evaluation Questions

3. Construct Evaluation Framework

Conducting4. Collect Data

5. Organize, Analyze, & Display Data

6. Interpret Data

Reporting7. Disseminate

and Use Findings

8. Evaluate the Evaluation

Deconstructing the Steps

1. Distribute evaluation step cards at your table.2. Independently, read about your assigned step.3. Regroup with others with the same step.

– Describe how the step contributes to evaluation.– Identify concepts embedded in the step. – Recommend strategies for addressing this step

to increase the effectiveness of professional learning evaluations.

Group Discussion Protocol

• Return to your original table group. • In round robin fashion, beginning with step one,

share key points, evidence, and strategies. • Consider how steps are independently and

collectively important.• Be prepared to discuss insights gained with the

whole group.

Note to Self

• Revisit your process. • Based on what you now

know about the eight evaluation steps, which steps were present? Missing?

• What is one actions you will take to address missing steps in your current process?

Summer Institute Academy Vision

Maria Pitre MartinLynne Johnson

LUNCHSee you back in 45 minutes.

Assumptions about Change

• Independently, read the article by Michael Fullan. • With a partner, respond to one or more of the

questions below. • Jot down notes, and identify common themes that

emerged. 1. What aligned with your thinking? Share why.2. What challenged your thinking? Share why.3. Describe something you want to explore. Be specific. 4. What was the most important takeaway for you?

Change is Learning

Concerns-Based Adoption Model : Developed by Bill Rutherford, Gene Hall, Shirley Hord, and Susan Loucks-Horsley

4 Components:1. Stage of Concern – 7 stages of responses2. Levels of Use – eight ranges of intervention use3. Innovation Configuration – described actions4. Change facilitators – leaders of learning

Change Learning Exchange• Distribute numbered cards (#1-4) at your table.• Read and the corresponding article on change.• Prepare a two-minute talk about your article. Use

any of the following to prepare.– Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping

others understand this area of change.

Change Spotlight• Find a partner that read a different article.• Take two minutes each, discuss your article.

Focus on any of the following: – Why your focus area is important.– Implications of the change process.– Ways to facilitate your area of change. – Challenges you anticipate when helping

others understand this area of change. • Listen for the timer to repeat the process.

Note to Self

• What was one benefit of revisiting the change process?

• What new insights did you gain as a result of your reading and discussion with others?

Change Self-Assessment

• Use the self-assessment to determine the level of implementation in your system.

• Note the evidence that informs your score.• Identify your area of strength and area for

improvement. • Be prepared to meet with others about your

results, and evidence.

Strengths

• Determine which area is your strongest.• Meet with colleagues who share that same area

as a strength. • Share the evidence of your strength, i.e., what is

it you are doing in relationship to this area?• Record strategies you receive.

Areas for Improvement

• Determine which area is your greatest area for improvement.

• Meet with colleagues who share that same area for improvement.

• Identify what happens when this area is a gap.• Use strengths collected from others to identify

possible strategies you might consider to address this gap.

Group Reflection1. What does this mean

for your work?2. What do you need to

share and with whom?3. How will you continue

to develop strengths?4. What is one strategy

you will take away to develop an area for improvement?

BREAKSee you back in 15 minutes.

Evaluation of ill-conceived professional learning programs waste limited human and fiscal resources. In these cases, evaluations sanction programs with gaps in their conceptual frameworks by appearing to condone mediocrity. Wasted evaluations rob students and educators of powerful opportunities to learn at high levels. And when evaluations attempt to assess programs with fuzzy goals, faulty conceptual frameworks, or illogical, poorly structured activities, the evaluation may be difficult to design and of questionable integrity.

-Killion, J. (2008). Assessing Impact. p. 33.

Thorough planning facilitates sound

evaluation.

Evaluating Professional LearningPlanning

1. Assess Evaluability

2. Formulate Evaluation Questions

3. Construct Evaluation Framework

Conducting4. Collect Data

5. Organize, Analyze, & Display Data

6. Interpret Data

Reporting7. Disseminate

and Use Findings

8. Evaluate the Evaluation

Step 1: Assess EvaluabilityPlanning Phase

The degree to which a proposed plan for staff development clearly articulates what the staff development program is and how it is expected to produce change.

Four Components of Assessing Evaluability

1. Examine program goals, objectives, standards for success

2. Examine program’s theory of change3. Examine program’s logic model4. Adjust program based on assessment

Clarifying Your Program

• What are the program’s goals?– Goal is desired state.

• Improve physical well-being.– Objectives describe the changes expected.

• Lose 15 pounds in six months.– Activities explain how to achieve the goals and

objectives.• Walk daily.• Eat healthful foods.

Goal, Objective, or Activity?

1. Produce responsible citizens.2. Increase student accuracy in applying the

scientific process in a laboratory setting.3. Provide opportunities for professional learning.4. Create professional learning communities.5. Increase student performance 15% on state

assessment in mathematical problem solving.

SMART Goals: Combine Goals and Objectives

• Specific change• Measurable change• Attainable, realistic outcome• Results driven (student achievement)• Timebound

- Garmston, JSD, Summer 1997, pp. 64-65

Creating Goals & Objectives

• In table groups, work collaboratively to determine program goals and objectives.

• Refer to the Teacher Quality Project simulation to guide table group discussions, and focus applications.

• Jot down any notes on the process.

SHORT-TERMKnowledge

Skills

Teachers gain knowledge and skills.

Increase teachers’ content knowledge.

MEDIUMAttitudes

AspirationsBehaviors

Teachers gain aspirations, attitudes, and behaviors.

Change teachers’ instructional practices.

LONG-TERMIntended Results

Students gain aspirations, attitudes, behaviors, knowledge, skills.

Increase student achievement.

Goals What results occur from professional learning for

individuals, schools, communities …

Setting Standards for Acceptable Performance

• Specifies how good is good enough• Specifies “success” in advance• Provides a benchmark/baseline for

comparison before and after professional learning

Sample Standards for Success

Increased student achievement– 8% overall increase on state assessment– 7% more students in the advanced category and

20% fewer students in the not-proficient category– 25% more underrepresented students in the

proficient category

Types of ChangeWhat is changing?

• Knowledge• Attitude• Skill• Aspiration• Behavior

KASABsKnowledge Conceptual understanding of information,

theories, principles, and research

Attitude Beliefs about the value of particular information or strategies

Skill Strategies and processes to apply knowledge

Aspiration Desires, or internal motivation, to engage in a particular practice

Behavior Consistent application of knowledge and skills

Student Teacher Principal

Knowledge Students know problem-solving strategies.

Teachers know multiple problem-solving strategies and when to use them.

Principals know what to look for in a problem-solving lesson.

Attitude Students enjoy problem solving.

Teachers believe students’ competence in math is important for success both in and out of school.

Principals believe teachers make a difference in students’ academic success.

Skill Students know how to use their problem solving strategies.

Teachers know how to assess students’ problem-solving ability.

Principals know how to observe and give feedback.

Aspiration Students want to participate in higher- level math classes.

Teachers want all students to learn math.

Principals want all teachers and students to succeed.

Behavior Students apply mathematical reasoning to solve problems in and outside of math.

Teachers provide students with daily practice in problem solving.

Principals monitor student problem-solving scores on benchmark tests and identify students who need extra help.

Types of Change

• In table groups, work collaboratively to align types of change (KASAB) to each objective.



2/3 : 5/6 = 4/6 : 5/6 =

4 : 5

GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy.

Sample KASAB WorksheetType of Change ExampleKnowledge Understands rational numbers and content-

specific vocabulary

Attitude Believes that developing students’ understanding of rational numbers contributes to their success in higher-level math, high school graduation, and college and career readiness

Skill Uses higher-order thinking skills to elicit students’ understanding of rational numbers and thinking during problem solving

Aspiration Desires to teach all students to succeed in higher-level math

Behavior Applies effective question skills in math instruction to elicit student thinking during problem solving

• Increase teachers’ content knowledge and precision in content vocabulary related to rational numbers as evident in lesson plans and instructional materials KNOWLEDGE

• Increase teachers’ accuracy and frequency of use of questioning strategies in math instruction by at least two levels as described in the IC maps SKILLS, ATTITUDE, ASPIRATION, & BEHAVIOR

• Increase students’ ability to explain their thinking as they solve rational number problems by two levels as described on the IC maps KNOWLEDGE & SKILLS

• Increase students’ achievement on benchmark assessments on rational number problems KNOWLEDGE & SKILLS

Objectives

Objectives Evaluation Questions EvidenceIncrease teachers’ content knowledge and precise vocabulary in rational numbers as evident in lesson plans and instructional materials KNOWLEDGE

Did teachers’ understanding of rational numbers increase by at least 20% as measured an analysis of lesson plans and instructional materials?

Pre and post-lesson plan and instructional material analysis

Increase teachers’ accuracy and frequency of use of questioning strategies in math instruction by at least two levels as described in the IC maps SKILLS, ATTITUDE, ASPIRATION, & BEHAVIOR

Are teachers demonstrating accurate and frequent use of questioning techniques in their daily instruction as described in the IC Maps?

IC MapWalk through guideTeacher observation guide

GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy

Objectives Evaluation Questions EvidenceIncrease students’ ability to explain their thinking as they solve rational number problems by two levels as described on the IC maps KNOWLEDGE & SKILLS

Are students explaining their thinking as they solve problems involving rational numbers as described in the IC Map for student thinking in math?

IC MapStudent thinking guideStudent observation guide

Increase students’ achievement on benchmark assessments on rational number problems KNOWLEDGE & SKILLS

Did student performance on the six-week benchmark assessments increase by 10% from previous assessment over the school year on rational number problems

Benchmark assessments

GOAL: Improve student achievement in rational numbers through professional development focused on increasing teachers’ content knowledge and content-specific pedagogy

Theory of Change

• Identifies the components/actions of a program (what the program does)

• Specifies the relationship among the components to explain how the change occurs (sequence of actions)

• Delineates the underlying assumptions upon which the program is based

Source: University of Wisconsin Extension Service

Everyday Theory of Change

Headache Get pills Take pills Headache gone; return to work; contribute to economic well-being of community

Theory of Change

• Components/Actions of the program• Sequence of components to show how results

will occur• Assumptions underlying the program

Teachers participate in collaborative learning experiences.

Teachers implement new learning in their instruction.

Student performance increases.

Professional development core features:•Content focus

•Active learning

•Coherence•Duration•Collective participation

Increased teacher knowledge and skills; changes in attitude and beliefs

Change in instruction

Change in student learning

Core Conceptual Framework for Studying the Effects of Professional Development on Teachers and Students Desimone, L. (2009)

Professional Learning

Opportunities

Teacher interpretation and

utilization of available

understandings and skills

Teacher outcomes:•Change in practice•No change in practice

Student outcomes:

•Positive changes•No changes•Negative changes

Students’ interpretation and

utilization of available understandings and

skills

Student Learning

Opportunities

Timperley, H. & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education, 32, pp. 328-369.

Sample Theory of Change

• Review the theory of change. • Generate a list of assumptions depicted in the

theory of change.• State in your own words what a theory of

change is.• Be prepared to share your definition.

Assumptions • Teachers’ understanding of various instructional

strategies provides their use of them. • In order to use new instructional strategies, teachers have

the appropriate material, human, and time resources. • Success builds with an increase in practice and support.• Consistency and accuracy in implementation of new

practices come over time. • Student achievement comes from consistently applied,

research-based instructional practices delivered by teachers who have in-depth understanding of content and content-specific instructional strategies.

Creating Theories of Change

• In table groups, work collaboratively to develop a theory of change and identify guiding assumptions aligned to the simulation.



Partner Chat

• What is the value of having a theory of change for a staff development program?

Logic ModelSpecific kind of action plan of the program specifying the flow of changes (results) that will occur • Inputs/resources• Activities/processes• Initial outcomes• Intermediate outcomes• Results

– Results (Goals)

Inputs Activities IntermediateOutcomes

InitialOutcomes

Sample Logic Model

• Review the sample logic model. • Notice how the inputs, activities, initial

outcomes, and intermediate outcomes align in a LOGICAL way to demonstrate how the program moves toward the intended results.

Creating a Logic ModelAt your table group, create a logic model for at least two actions or components of your professional learning program’s theory of change.

The Power of What We Think

I can only change how I act if I stay aware of my beliefs and assumptions. - Margaret Wheatley

Running alongside our visible and expressed values is a competing set of values we’re unaware of. - Robert Kegan

Begin challenging your own assumptions. Your assumptions are your windows on the world. Scrub them off every once in awhile, or the light won't come in. - Alan Alda

Assumptions About Professional Learning

1. Read each assumption statement. 2. Code each with your level of agreement.3. Identify potential implications for the role of

teacher evaluation.4. Share your thoughts and answer the questions

with others at your table.

Evaluation Assumptions

• The staff development program is data-driven, research-based, and well-defined.

• The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs.

• Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program.

Note to Self

• How does stating one’s beliefs, and unmasking assumptions help reveal barriers to change?

• How do competing values form obstacles to change?

Moving to Step 2 Means …

We believe we have: 1. A planned, coherent,

in-depth professional learning program that is– Feasible.– Clear.– Powerful.– Worth doing.

2. KASABs.3. Goals.4. Standards for success.5. Theory of change.6. Logic model.

Debriefing – Implications for NC

1. Reflect on the learning for the day using the the following three areas to guide your thinking.– What are the benefits?– What are the barriers?– What are the implications?

2. Consider beginning an evaluation framework (see sample) or theory of change for a current professional learning program.

End-of Day Reflections

• How are the ideas presented today CONNECTED to what you already knew?

• What new ideas did you get that EXTENDED or pushed your thinking in new directions?

• What is still CHALLENGING or confusing for you to get your mind around? What questions or wonderings do you now have?

Problems of Practice Exchange

• Consider your current challenges, dilemmas, and frustrations to strengthen professional learning.

• Outline what you believe to be the most critical areas you seek solutions, strategies, and answers.

• Bring your notes with you tomorrow.

Evaluating the effectiveness OF PROFESSIONAL LEARNING

Documents