Innovation in Education Tools and methods for success Session 2: Tools and techniques 8 de abril de 2014 Porto, Portugal. Kelwyn Looi Office of the Chief Education Advisor, Pearson PLC. A measurable impact on improving someone’s life through learning
53
Embed
Innovation in Education: Tools and methods for success (Session 2)
Innovation in Education Tools and methods for success Session 2: Tools and techniques Joint Pearson and ELIG workshop at the Escola Superior de Educação do Porto.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Innovation in Education Tools and methods for success
Session 2: Tools and techniques
8 de abril de 2014
Porto, Portugal.
Kelwyn LooiOffice of the Chief Education Advisor, Pearson PLC.
A measurable impact on improving someone’s life through learning
Agenda
1. An introduction to Efficacy at Pearson
2. Efficacy in Practice
3. What is the measurable impact on learning outcomes (efficacy) of your case?
4. Q&A
2 Sessão 2: Ferramentas e técnicas l April 2014
What we aim to achieve
• An understanding of Efficacy at Pearson, and the evolution of its implementation in the company
• Introducing the Efficacy Framework as a tool to support innovation in education
• Going through some examples of applying the Efficacy Framework to in a generic case
• Applying efficacy to your innovations and identifying where efficacy can be embedded
1
2
3
4
Watch out for this icon – it signals an activity!
3 Sessão 2: Ferramentas e técnicas l April 2014
An Introduction to the session…
This session will look at tools and techniques that can support the development and structuring of an innovation, taking into account the context in which it operates, objectives, expected results, likely indicators of success and the expected impact
Pearson’s Efficacy Framework will be tested as a [e.g. stand-alone] means to support the stakeholders to innovate in TEL / education
Key objectives
For attendees the learnshop would provide the opportunity:
1. To acquaint yourself with the efficacy framework as a tool to engender learning-focused discussions when assessing and evaluating prospective innovations
2. To examine a specific case through the lens of a rigorous and structured framework, providing key takeaways at both the transversal and individual innovation level
4 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy at Pearson
Efficacy in Practice
Exercise
Key takeaways and useful resources
The path to Efficacy: why?
• As the world’s leading learning company, we feel we have a responsibility and an opportunity to help people make progress in their lives through learning
• We have aligned our activities around the principle of Efficacy to achieve this
6 Sessão 2: Ferramentas e técnicas l April 2014
Pearson’s definition of efficacy
efficacy(dictionary definition)• ability to produce the intended
result
efficiency(dictionary definition)• achieve maximum productivity
with minimum wasted effort
7
Put simply…
it’s all about products that improve results and measurable outcomes for learners.
Efficacy (Pearson’s definition)• a measureable impact on improving someone’s
life through learning
Sessão 2: Ferramentas e técnicas l April 2014
Why now?
• There is a shared understanding that high-quality education drives personal, economic and societal growth
• Governments, individuals, employers and institutions recognise the need to deliver high-quality learning
• New technology makes it increasingly possible to see what works and what doesn’t in helping learners to achieve their goals
8 Sessão 2: Ferramentas e técnicas l April 2014
Strategic accomplishments
• Launched publicly, generating commentary from media and influential education leaders around the world
• Made efficacy and learner outcomes central to Pearson’s global education strategy and organizational design
Product and service accomplishments
• Rigorous process in place for conducting efficacy reviews with a variety of depth
• 123 efficacy reviews completed to date
Progress to December 2013
In the past year, we’ve taken major steps towards putting efficacy at the heart of Pearson’s work
9 Sessão 2: Ferramentas e técnicas l April 2014
Our previous business map…
•Inputs
•Inputs
•Products
Existing
Offerings
New
NewExisting Pearson Impact
10 Sessão 2: Ferramentas e técnicas l April 2014
Our new business map…
Existing
• Inputs• Products
• Outcomes• Services
Offerings
New
Program Solutions
New digital content
Offerings
New
Existing
Offerings
New
NewExisting Pearson Impact
11 Sessão 2: Ferramentas e técnicas l April 2014
Our commitment to reporting our impact on learner outcomes
“Pearson’s purpose is to help people make progress in their lives through learning. So, we better be sure that we can demonstrate [our products deliver a] measurable impact on learning outcomes.
“[By 2018] we will, in a rigorous and externally audited way…[publicly] report on [how our products and services improve] learner outcomes.”
John Fallon Nov 15, 2013
How are we helping millions to improve their lives through learning?
12 Sessão 2: Ferramentas e técnicas l April 2014
The response to the external communication of efficacy has been largely positive
“That’s an enormous commitment and if they really follow through, it gives me some confidence that this is not merely a marketing ploy.”
Feedback from education leaders
“Pearson’s brightest possible future is not as a company that designs educationally effective products, but as one that facilitates conversation and research about efficacy within the broader academic community.”
“At face value, it’s a bold move because it moves the content of education from being critically reviewed by experts to an empirical review,…. It’s the transition from, 'This is a great book’ to ‘This material really works.’ ”
To report audited learning outcomes measures and targets alongside financial accounts, covering the whole business by 2018.
Our commitment
Daniel Willingham Michael Feldstein
13 Sessão 2: Ferramentas e técnicas l April 2014
The Efficacy Framework has already been examined as a tool to support innovation
• At Online Educa Berlin last year (OEB), Pearson partnered with the European Learning Industry Group (ELIG), to deliver an interactive learnshop
• This involved applying the efficacy framework to selected case studies that demonstrated innovations to support technology-enhanced learning (TEL)
• Opening up the conversation to external input is vital to the success of efficacy
Excerpt from Online Educa Berlin
14 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy goals: does what it says on the labelA 3-step process
1.
1. Ensure our products and services deliver the best learner outcomes: Efficacy Reviews
2. Build comprehensive and rigorous evidence to support our claims: Efficacy Studies / Selling
3. Continually learn from learners to innovate and improve our products and services: Efficacy Analytics
2b. Efficacy Sellinguse evidence to sell & support best customer experience
Our path to efficacy: three complementary activities for improving learner outcomes
16 Sessão 2: Ferramentas e técnicas l April 2014
An Efficacy Framework: predicting the likelihood of impacting learner outcomes
Four criteria:
1. What learner outcomes are we trying to achieve?
2. What evidence do we have to believe it is possible to achieve these outcomes?
3. What plans are in place to build and deploy a solution that will impact these outcomes?
4. What capacity exists to achieve these outcomes?
17 Sessão 2: Ferramentas e técnicas l April 2014
The Efficacy FrameworkCriteria area Rating Rationale summary
• Action plan
• Governance
• Monitoring and reporting
• Internal capacity and culture
• User capacity and culture
• Stakeholder relationships
Outcomes
• Intended outcomes
• Overall design
• Value for money
• Comprehensiveness of evidence
• Quality of evidence
• Application of evidence
Evidence
Planning and implementation
Capacity to deliver
Efficacy
KeyGreen: Requires small number of minor actions. Amber/green: Requires some actions (some urgent and some-non urgent).Amber/red: Requires large number of urgent actions.Red: Highly problematic requiring substantial number of urgent actions.
An Efficacy Framework: an explanation of ratings
Good – requires slight refinement, but on track
Mixed – some aspects require attention, some solid
Problematic – requires substantial attention, some require urgent rectification
Off-track – requires urgent action and problem solving
Ratings are not grades on performance
Ratings prompt discussions that lead to actions
Ratings prioritise and suggest timeline
19 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy Framework: Evidence - Why should anyone believe us?
Framework Section
Evidence
Comprehensiveness of evidence
Ask yourself ∙ How well do we understand what our user needs and values?
∙ Is the vision for the innovation supported by research (internal or
external)?
∙ Are we leveraging proven approaches from other innovations?
Quality of evidence
Ask yourself ∙ How rigorous is our evidence?
Is the rigour appropriate for the innovation?
∙ How recent and relevant is the research?
Application of evidence
Ask yourself ∙ How evidence-based is the innovation design?
∙ Has the design been tested?
∙ Does evidence demonstrate that the innovation can be replicated
globally?
20 Sessão 2: Ferramentas e técnicas l April 2014
Rating Rationale
Comprehensiveness of evidence
?Project 1
• Strong set of academic evidence and expertise underpinning the innovation.• Comprehensive teacher focus groups across markets and regions, focusing on the right questions (usability,
price point etc); some concerns about scope of potential customers canvassed (see below). • Traditional competitors are tracked, but not non-traditional competitors who offer tests.
Quality of evidence
?Project 2
• There is a good mix of quantitative and qualitative evidence as well as unbiased samples from the survey. Nonetheless, a large amount of the available evidence is based on the previous innovation (before Sept 2012 enhancement) and, therefore, is not fully applicable to the current innovation. Also, there is little documented evidence coming from the students as to date there has been more focus on teacher rather than student outcomes.
Effective use of evidence
?Project 3
• The external evidence that has been collated is not known or accessible to all members of the team. There may be additional evidence within the business that could be exploited. The use of [this capability] in innovation design is not yet articulated and it is essential that any major decisions should be underpinned by research. Pilot innovations must be timed so information can feed back into the design [of this capability)
Warm-up (5 mins): Rate the evidence for Projects 1-3
Exercise: Evidence
• Good -- requires small number of minor actions• Mixed – requires some actions (some urgent and some non-urgent)• Problematic -- requires large number of urgent actions• Off track – Highly problematic requiring substantial number of urgent actions
Key
21 Sessão 2: Ferramentas e técnicas l April 2014
Rating Rationale
Comprehensiveness of evidence
Project 1
• Strong set of academic evidence and expertise underpinning the innovation.• Comprehensive teacher focus groups across markets and regions, focusing on the right questions (usability,
price point etc); some concerns about scope of potential customers canvassed (see below). • Traditional competitors are tracked, but not non-traditional competitors who offer tests.
Quality of evidence
Project 2
• There is a good mix of quantitative and qualitative evidence as well as unbiased samples from the survey. Nonetheless, a large amount of the available evidence is based on the previous innovation (before Sept 2012 enhancement) and, therefore, is not fully applicable to the current innovation. Also, there is little documented evidence coming from the students as to date there has been more focus on teacher rather than student outcomes.
Application of evidence
Project 3
• The external evidence that has been collated is not known or accessible to all members of the team. There may be additional evidence within the business that could be exploited. The use of [this capability] in innovation design is not yet articulated and it is essential that any major decisions should be underpinned by research. Pilot innovations must be timed so information can feed back into the design [of this capability)
How did you do?
• Good -- requires small number of minor actions• Mixed – requires some actions (some urgent and some non-urgent)• Problematic -- requires large number of urgent actions• Off track – Highly problematic requiring substantial number of urgent actions
Key
Solutions: Evidence
22 Sessão 2: Ferramentas e técnicas l April 2014
Framework area Initial review 3- month estimate
6-month estimate
Comment
Outcomes
Intended outcomes After 6 months, outcomes and metrics will be clear and will influence design. Value for money intelligence will be drawn from pilots.
Overall design
Value for money
Strength of evidence base
Comprehensiveness of evidence After 6 months, the plan to develop the forward evidence base will be finalised and initiated. Quality of evidence
Application of evidence
Quality of planning and implementation
Action plan After 6 months, long-term plans and reporting structures will be in place and governance agreed. Reporting will be at an early stage.
Governance
Monitoring and reporting
Capacity to deliver
Pearson capacity and culture After 6 months, Capacity issues will be clear, pilots delivered and lessons learned and applied. Stakeholder relationships plans will be launched and gathering feedback.
Customer capacity and culture
Stakeholder relationships
An Efficacy Framework: driving improvement
23 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy Studies: holistic, long-term studies with specific learners, teachers, and institutions
24 Sessão 2: Ferramentas e técnicas l April 2014
MyEnglishLab Efficacy study
• MyEnglishLab makes a difference
• The Efficacy study illustrated real-world situations in which MEL is positively impacting classrooms
• Studies uncover proven best practices from customers who are USING MEL in their own classroom
This means a long term customer!
25 Sessão 2: Ferramentas e técnicas l April 2014
Identify commonlearner difficulties
Personalise learning(by L1)
Optimise learning by L1
Research learner behaviours that lead to success (machine learning)
Improve learner engagement(activity design)
Predict learners who will failfor early intervention(predictive algorithms)
Efficacy Analytics and insights from Big Data
26 Sessão 2: Ferramentas e técnicas l April 2014
both students have the same net score
0 100 200 300 400 500 600
-25
-20
-15
-10
-5
0
5
10
Net S
core
Responses Submitted by Student
0 100 200 300 400 500 6000
50
100
150
200
250
300
Student 57Fractal D = 1.60
Net S
core
Responses Submitted by Student
fractal alert: alert teacher and learner to intervene
responses submitted over course
student who will succeed - smoothFractal = 1.60
student who will fail or not complete – noiseyFractal = 1.94
Identifying learners at risk: ignoring assessment and taking a (random) walk
Patent awarded 2013
27 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy at Pearson
Efficacy in Practice
Exercise
Key takeaways and useful resources
User stories: our end goals for efficacy
“As a…, I want to…, so that I can…”
- as a learner, I want to…
EXAMPLE: “As a potential learner of an English Language course, I want
evidence of how the course will help me improve my English so that I can make an informed decision about which provider to choose.”
Fill out the table. Many answers are possible.
29 Sessão 2: Ferramentas e técnicas l April 2014
As a…. I want to… so that I can…
As a student
As a parent
As a teacher
As a school
As a … (you)
Activity 1: User Stories – for an ELT course(10 min)
30 Sessão 2: Ferramentas e técnicas l April 2014
As a…. I want to… so that I can…
As a student
I want to know the course method works for real students like me
so that I can reach my exam (educational) goals
As a parent
I want to know the course is improving my child’s English
so that I can feel comfortable my child is getting the right level of education
As a teacher
I want to use data to understand my students’ preparedness
so that our classes are productive
As a school
I want to know that this is the most appropriate course on the market
so that I can best serve my students
As a … (you)
I want to demonstrate to potential clients that the ELT course method works
so that I can address their learning and financial concerns
Activity 1: User Stories – for an ELT course(10 min)
31 Sessão 2: Ferramentas e técnicas l April 2014
How can efficacy be applied to your work?
What outcomes are you trying to achieve?
• Set clear efficacy goals• Give your people the incentive to focus on outcomes
What’s the evidence?
• Develop innovations underpinned by research• Build and use effective data systems
What’s the plan?
• Make delivering outcomes a core part of your strategy• Take an open approach• Employ iterative and agile processes
What’s the capacity to deliver?
• Talk to your users and understand their students’ needs• Train your students, teachers or others to use your innovation effectively• Shape the debate with influential stakeholders
32 Sessão 2: Ferramentas e técnicas l April 2014
Global Efficacy Strategy
Efficacy in Practice
Exercise
Key takeaways and useful resources
1 hour
• Think through the four areas that we talk about when we measure efficacy – outcomes, evidence, planning, and capacity
• Consider the question:
What is the measurable impact on learning outcomes (efficacy) of the case?
• We will go through Outcomes and Capacity today
You can apply the Efficacy Framework to support innovation in education…
34 Sessão 2: Ferramentas e técnicas l April 2014
Exercise
• Spend some time reading the case that you have in front of you (15 mins)
• Rate for the Outcomes part of the framework and discuss rationale in the group (20 mins)
• Rate for the Capacity part of the framework and discuss rationale in the group (15 mins)
• Discussion on the results and the usefulness of the exercise (10 mins)
1
2
3
4
35 Sessão 2: Ferramentas e técnicas l April 2014
Efficacy at Pearson
Efficacy in Practice
Exercise
Key takeaways and useful resources
On November 15th, Pearson launched a dedicated website:
http://efficacy.pearson.com
outlining the company’s focus on efficacy and commitment to put the learner at the heart of the global strategy.
An interactive version of the efficacy framework also features on the website.
In small groups of 2-3, evaluate the following matrices aligned to your innovations, and discuss where and how efficacy can be embedded?
• What are the advantages of embedding efficacy into the common innovation design processes?
• What are the challenges of embedding efficacy into the common innovation design processes?
• What questions do you have about the integration of the two?
Self-evaluation exercise:
48 Sessão 2: Ferramentas e técnicas l April 2014
The Efficacy Framework
Criteria area Rating Rationale Summary Actions
• Action plan
• Governance
• Monitoring and reporting
• Internal capacity and culture
• User capacity and culture
• Stakeholder relationships
Outcomes
• Intended outcomes
• Overall design
• Value for money
• Comprehensiveness of evidence
• Quality of evidence
• Application of evidence
Evidence
Planning and implementation
Capacity to deliver
49 Sessão 2: Ferramentas e técnicas l April 2014
Self-evaluation: Outcomes
Criteria area Rating Rationale Summary Actions
Outcomes
Does the design of the innovation fit their culture in a way that will eventually impact student achievement?
Can the user achieve the same goals by investing in alternative innovation for lesser investment?
What are we trying to achieve?
What is the vision for the outcomes the user of the innovation wants to see?
What is the end impact on learning from the innovation?
Intended Outcomes
Overall Design
Value for Money
50 Sessão 2: Ferramentas e técnicas l April 2014
Self-evaluation: Evidence
Criteria area Rating Rationale Summary Actions
Evidence
What is different about the proposed innovation than an existing or competing innovation?
Do we have a case study of another innovation with similar characteristics to highlight the evidence of our innovation?
Why do we believe we can achieve?
How consistent is the evidence and is it quantitatively validated?
Comprehensiveness of Evidence
Quality of Evidence
Application of Evidence
51 Sessão 2: Ferramentas e técnicas l April 2014
Self-evaluation: Planning
Criteria area Rating Rationale Summary Actions
Planning
Who are the individuals responsible for guiding, monitoring and revising implementation once adopted?
What routines or protocols will be used to gather feedback, ensure quality of implementation, and report on success as well as areas of need?
How would we achieve it?
Who is responsible for running point on the implementation, creating the roll-out plan (including timeline and metrics for success)?
Action Plan
Governance
Monitoring and reporting
52 Sessão 2: Ferramentas e técnicas l April 2014
Self-evaluation: Capacity
Criteria area Rating Rationale Summary Actions
Capacity
What have we communicated to the user of the innovation regarding what resources are needed and what has the user agreed to allocate?
What are the potential risks, the commonly agreed upon plans to mitigate, and our mutual willingness and trust to re-evaluate what actions are necessary to achieve the desire outcomes?
Do we have the capacity to deliver?
Do I have the right people, resources, and teams to fit and fully support the innovation?