Making a difference: How can we demonstrate the impact of learning technology research?

Making a difference: How can we demonstrate the impact of learning technology research?

Professor Jane Seale

Overview

• Provide an overview of the different ways that funders such as ESRC and HEFCE define and conceptualise “impact” and “users of research”;

• Offer my personal interpretations, with examples, of how learning technology research might demonstrate impact;

• Facilitate debate and reflection regarding how well placed current UK learning technology research is to meet the challenges of demonstrating impact.

Defining impact

• Broadly speaking: – Where research has a transformative and beneficial

effect on people, groups, organisations, communities – Evidence for this impact goes beyond citations in

academic journals• Impact on “real-world” users- through user

engagement

Separating impact of e-learning from the impact of e-learning research

• We “know” that e-learning can be transformative

• How can we demonstrate the impact of research that reveals this transformation?

E-learning has had an impact on pedagogy- it is motivational, teachers have been transformed

What claims for impact can the e-learning research community make?

• Conceptual– Diana Laurillard (2002) Rethinking university teaching:

a conversational framework• 4350 Google Scholar citations

• Practical– Gilly Salmon (2003) E-moderating: The key to

teaching and learning online• 2676 Google Scholar citations

• Can “we” prove that government, universities, lecturers have transformed the way we teach using technologies as a result of these publications (through changes in policies, standards, practice, systems, resources etc.) ? – How??

What claims for impact can I personally make? A personal reflection (1)

• Jane K Seale, Alan J Cann (2000) Reflection on-line or off-line: the role of learning technologies in encouraging students to reflect: Computers & Education, 34, 3-4, 309-320

• Empirical paper: one of my papers submitted to RAE 2001 as part of Kings College submission– Comparison of two individual teaching practices

• Dissemination activity: 2 conferences, Alan's’ personal web page, 2nd related journal paper

• Impact ??– CAL99: positive reaction regarding “honesty” in admitting

failure– CTI Biology Virtual Conference 1997- voted best paper– My most cited paper: 74 Google citations (all academic journals

and conference papers)

A personal reflection (2)• Phipps, L., Sutherland, A and Seale, J (eds). Access All

Areas: disability, technology and learning. ALT/JISC/TechDis. (2002)– An edited pamphlet aimed at practitioners- review/description

of accessibility law and implications for practice– Not submitted to RAE 2008 by Southampton University

• Dissemination (impact)activity: freely available online through TechDis , ILT, ALT

• Impact??– 6th most cited publication in Google Scholar (mixture of

academic and policy related citations)• Equality Challenge Unit (2006) Disability legislation: practical guidelines for

academics..• Welsh Assembly Government (2005) Disability Issues for Post-16 Learning

Provision. • Cited in many new lecturer programme reading lists

Personal reflections (3)• LEXDIS- JISC funded project 2008• Students and academic staff as

partners in the project- user engagement

• Dissemination (impact) activity:– Project website, 3 journal articles

(empirical)

2 newsletter articles, 1 book chapter, 5 conference papers, workshops, desk-side 1-to1

• Impact??– One whole page spread in TES– Students still contributing to the

website 4 years after end of project

– Winner of 2009 IMS Learning Impact Award

Which of these three projects had the most “impact”?

• I personally value each of these 3 pieces of work for different reasons in relation to their impact on academia, policy, & students

• Link between impact and quality potentially complex

• Only the last project had anything like a concrete strategy for user engagement at the beginning of the project

1. How do funders such as ESRC and HEFCE define and conceptualise “impact”?

Impact and Excellence

• Both HEFCE and ESRC assert that you can’t have impact without excellence– Excellent research underpins excellent

impact

Impact is about showcasing and exploitation

• HEFCE: The REF will– Showcase the success of UK research in

contributing to the economy and society– Encourage more effective dissemination,

application and exploitation of research

Impact is about justification of value and worth:• ESRC:

– “In recent years, the government has placed increasing emphasis on the need for evidence of economic and social returns from its investment in research.

– By ensuring that ESRC-funded research makes the biggest possible impact on policy and practice, and improving how we measure and capture this, we are better able to support the case for research funding.

– Impact helps to demonstrate that social science is important – that it is worth investing in and worth using.”

Looking forwards and looking backwards

• ESRC : when bidding: accounting for future impact– ESRC/RCUK: applicants are asked to give an

account of the anticipated future effects of their research on potential non-academic users

• HEFCE REF: accounting for past impact

HEFCE (REF) definitions

• For the purposes of the REF impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

• Having an effect on: – the activity, attitudes, awareness, behaviour, capacity,

opportunity performance, policy, practice, process or understanding

– of an audience, beneficiary, community, constituency, organisation or individuals

– in any geographic location, whether locally, regionally, nationally or internationally

• Impact on academic knowledge excluded• Impacts on students, teaching or other activities within submitting

HEI excluded

REF impact assessment: case studies

• Provision of “case studies” in which starting point is 2* paper(s) and the HEI provides a case for: – Impact of the research described in the paper(s)– What the HEI has done to facilitate impact of the

research

• 20% of overall assessment score• 2 case studies per 15 FTE (6 for 45+ FTE)• Jan 1st 1993- 31st December 2013

Impact Criteria (drawn from Panel C interpretation)

• Reach: understood in terms of the extent and diversity of the communities, environments, individuals, organisations or any other beneficiaries that have benefitted or have been affected– Reach will not be assessed in purely geographic terms, nor in

terms of absolute numbers of beneficiaries, but rather in terms of the extent to which the potential number or group of beneficiaries have been affected

• Significance: The degree to which the impact has enriched, influenced, informed or changed policies, opportunities, perspectives or practices of communities, individuals or organisations

• Panels B & C will assess the two criteria on a holistic basis

Type Examples

Impacts on creativity, culture and society

Informed public or political debateImproved welfare, equality, social justice, Production of cultural artefacts (films, TV)

Economic, commercial, organisational impacts

Development of new or improved products or processesDeveloping alternative economic modelsChanged approach to management of resources resulting in improved delivery

Impacts on environments Changed resource management practices

Health and welfare impacts (incl. protection or advocacy of rights or interests)

Influence on CPD/improved training standardsImproved provision or access to servicesImproved health and welfare outcomes (must include educational outcomes!)

Impacts on practitioners and professional services

Development of resources to enhance professional practiceUse of research findings to define best practice or in conduct of practiceInfluence on professional standards, guidance or trainingPractitioner debate has been informed or stimulated by research findings

Impacts on public policy, law and services

Influencing the work of NGO or commercial organisationsPolicy debate stimulated by research evidenceImproved public understanding of social issues/challenging conventional wisdom

What counts as evidence?

• Qualitative or quantitative for Panel C• For Panel B- wherever possible

quantitative should be included

Example indicators of impact

• Citations– Citation in a public discussion, consultation document or judgement– Citation by journalists, broadcasters or social media– Citation by international bodies such as UNESCO, IMF– Evidence of citation in policy, regulatory, strategy, practice documents

• Measures– Number of visitors, audience, participants– Measures of improved inclusion, welfare or equality– Service satisfaction measures– Outcome measures– Quantitative data relating to cost-effectiveness or organisational

performance

Examples of impact indicators

• Debate– Evidence of debate among practitioners leading to

developments in attitudes or behaviours– Public debate in the media– Parliamentary or other democratic debate

• Use– Use in scrutiny or audit processes such as Select Committees– Incorporation in training or CPD material

• Reviews and documentation– Media reviews– Independent documentary evidence of influence on

guidelines, legislation, regulation, policy or standards– Documented change to professional standards or behaviour

Evidencing impact

• Each case study must explain how the research led to or contributed to the impact and include appropriate sources of information external to the HEI to corroborate these claims

RCUK/ESRC definitions of impact

• “Research Councils UK (RCUK) defines research impact as 'the demonstrable contribution that excellent research makes to society and the economy'.

• Research impact embraces all the diverse ways that research-related skills benefit individuals, organisations and nations. These include:– fostering global economic performance, and specifically the economic

competitiveness of the United Kingdom– increasing the effectiveness of public services and policy– enhancing quality of life, health and creative output.

• A key aspect of this definition of research impact is that impact must be demonstrable. It is not enough just to focus on activities and outputs that promote research impact, such as staging a conference or publishing a report. – You must be able to provide evidence of research impact, for example, 

that it has been taken up and used by policy makers, and  practitioners, has led to improvements in services or business.”

ESRC Types of research impact

• Academic impact is the demonstrable contribution that excellent social and economic research makes to scientific advances, across and within disciplines, including significant advances in understanding, method, theory and application.

• Economic and societal impact is the demonstrable contribution that excellent social and economic research makes to society and the economy, of benefit to individuals, organisations and nations.

•  The impact of social science research can be categorised as:– Instrumental: influencing the development of policy, practice or service

provision, shaping legislation, altering behaviour– Conceptual: contributing to the understanding of policy issues,

reframing debates– Capacity building: through technical and personal skill development.

Planning research impact

• “To plan impact effectively you need to:– identify your key stakeholders, for example, other

researchers; public sector; business/industry– Identify how they will benefit from your research – types of

impact might include: improving social welfare/public services; influencing public policy; contributing to operational/organisational change

– Identify how you will ensure they have the opportunity to benefit, for example through organising public events; conferences; interaction with the media.

– For practical guidance on planning research impact, see the information on developing an impact strategy”

http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-toolkit/

ESRC Impact strategy: objectives

• “What are the likely outcomes of this research? • Who will benefit from this research? • How will they benefit from this research? • How can you involve potential beneficiaries in

this research? • How will you know if it has made a difference?”

ESRC Impact strategy: Messages

• “Clear, succinct messages that summarise your research.• Over-arching messages that can be used while it is underway, or

specific messages that relate to particular parts of the project.• Using different formats

– a media release – a report – a research briefing – a newspaper article – a website page.

• Think in advance about stories, case studies and 'packages' of information that will bring your project to life for key audiences.

• Creating a brand: distinct identity– your project name – a 'strapline' or one-line description of the project – a logo or visual mark, – the application of your brand across a range of materials– Ensure your brand reflects the overall values and objectives of your

research and your impact plan.”

ESRC Impact strategy: targeting audiences• “It is vital to know who your key audiences are.

– Think beyond the obvious organisations and individuals.

• Prioritising your audiences– Since time and money are limited, it is useful to

rank each of your potential audiences and user groups according to their importance and influence relative to your strategy.

– A good question to ask is: if you had half the money and half the time to spend on your strategy, which of your potential audiences would you focus on?”

ESRC guidance on tools and channels

• Branding• Events• Interactive media: websites, social media• Media relations• Networking• Public affairs: influencing policy makers• Publications• Public engagement: shape research agenda, steer project, share

results, raise awareness

Evaluating impact

Evaluating economic and capacity building impact probably quite straight forward. For example, for capacity building could apply this ESRC advice:• “Quantitative data– provide measures, for example of how many people

attended the event and what they thought of it. • Qualitative data – seek to illuminate individual experiences and provide

additional subjective context to the evaluation. These data explore the participants’ experience in more depth than quantitative data.

• Observational data – exploring how people participate in an event can be illuminating. Did they participate in all events/activities? Were some aspects more popular than others? How did people interact with your website or display? A clear idea of what you are looking for is important when structuring observational data. 

• But, is this measuring user engagement and can this be legitimately used as a proxy for impact”

Evaluating impact

• Evaluating impact of conceptual and related impacts harder and ESRC is addressing this

• It is commissioning case studies on methods for capturing the more diffuse types of social science impact– e.g. mapping conceptual developments (such as changes

in thinking, debate, culture and direction)– ‘people-flow’ impacts across the researcher/user interface

Implications for e-learning research

• If we want to defend or protect e-learning research then we have to demonstrate its public value

2. How can learning technology research demonstrate impact?

TLRP TEL as an example

• TEL and impact• “The TEL programme is keenly aware of the need to demonstrate

impact beyond academia. A number of projects have already begun to demonstrate how their TEL funded research could have a practical, positive impact on such areas as:– relieving teacher’s time and paperwork constraints– addressing difficult subjects in schools in different ways – providing better opportunities for disadvantaged people to

develop skills through interacting with technology– showing how technology can be used in learning environments

to save time and money whilst providing improved learner outcomes.

Innovations in translation

• http://tel.ioe.ac.uk/category/impact– Exhibitions– Prizes– Practitioner workshops– Media articles– Journalist informed tweets– Videos– Press releases– Breakfast conference with MP’s

Haptel as a case in point

• Clear messages about: • Cost effectiveness• Evidence that students do

just as well with haptics• User engagement- clinical

dentists on the team• Vision for implications for

future

Haptel as a case in point

• Participated in an evening of demonstrations of the latest medical technology for the general public at the Science Museum

3. Facilitate debate and reflection regarding how well placed current UK learning technology research is to meet the challenges of demonstrating impact.

An impact SWOT analysis for e-learning research

SWOT Examples

Strengths We have a detailed knowledge and insight into one key user group- practitioners.

Weaknesses Poor relationship with government-policy naivetyTendency to follow technological trends rather than set them

Opportunities Methodologies suited to user-engagementKnowledge transfer

Threats The culture of the lone e-learning “champion” researching own practiceThe memory of the failed UKEU- the elephant in the room

Strengths

• We have a detailed knowledge and insight into one key user group- practitioners– We inhabit their world because more often than not we are “one

of them”– Ideally placed to evidence impact on practice and inform

practitioner debates, practice, standards…(but we have to think beyond our own institutions to do this)

Impacts on practitioners and professional services

Development of resources to enhance professional practiceUse of research findings to define best practice or in conduct of practiceInfluence on professional standards, guidance or trainingPractitioner debate has been informed or stimulated by research findings

Weakness

• Very little e-learning research(ers) speak to government– What e-learning report/papers have been cited in key

government education policy documents and debate?

– Web science on the other had/have the ear of government

– TEL/ALT working hard to cement a good relationship with BIS on our behalf• But we have to engage too…..• To do this we have to re-conceptualise who we think our

end-users are

Weakness

• Tendency to follow technological trends rather than set them:– We apply the tools around us to educational contexts

(e.g. web 2.0 tools) whilst there is some innovation in this (applying existing tools to new situations) but couldn’t we be doing so much more?

• Naïve and ill-prepared in our approach to commercialisation of technological developments

Opportunities: methodologies suited to user engagement

• Channels of engagement- e.g. social media• Process of engagement- participatory action

research, partnerships with industry

Methodologically, impact requires us to step out of our comfort zones and make “contact with strangers”

• Brewer (2010) Impact requires the discipline to reconsider its own ‘strangers’, those with whom research relationships came to appear unusual and odd […] more obviously, policy-makers and civil society groups [..]

• Impact has accelerated contact with ‘strangers’ and is disquieting for this reason, but the long-established tradition of action research, participatory forms of research, or research sensitive to respondents’ sense of research fatigue, are forms that enhance people’s participation in the design and conduct of the research precisely in order to increase its impact

Opportunities: Knowledge transfer

• Can our work translate to related fields?• Examples:

– Work based training- internships (current political hot potato with introduction of 9K fees)

– VLE design (why shouldn’t we shape what Blackboard does for once?)

– Distance learning design and delivery for military personnel on duty

Threats: burned fingers and long memories

• UKEU and long memories– A company and website set up by HEFCE that

promoted online degrees from UK universities. UKeU delivered courses over a learning environment developed by Sun Microsystems UK.

– In 2004 it was announced that the project should be wound up as it was a failure having cost £50 m and recruited only 900 students

– E-learning a victim of its own “hype”

Threats: Lone Ranger syndrome

• Brewer (2010) • Impact disturbed the comfortable rules of the

game by which research reputations and resources were established – Marks the demise of the lone researcher model of

social research in the 21st century. – Promotes engagement with real world problems

Threats: Lone Ranger Syndrome

• We need more than ever to develop research teams that possess the wide range of skills required to conduct research AND plan, implement and evaluate an impact plan– Researching own practice, in own institution

through personal interest and enthusiasm, with limited skill-set no longer “cuts it”

Thank you, questions, comments, reflections, observations