Impact assessment framework Natasha Innocent Senior Policy Adviser Learning and Skills MLA.

Impact assessment framework

Natasha Innocent Senior Policy Adviser Learning and Skills MLA

Background

How libraries support learning • Strong roots into communities • Expertise in working in partnership with a

range of other partners • Critical role with digital participation• Support the wider benefits of learning • Provide choice and flexibility • Strong links to Ageing Society

What does the framework do?

Helps libraries describe the impact of their learning activities on individuals and communities

A common language to talk to policy makers and funders about the role of libraries in supporting learning outcomes

A methodology to support libraries to plan, evaluate and develop learning activity

Inspiring Learning

This framework is based on Inspiring Learning commissioned by MLA and launched in 2003

Inspiring learning is based on a broad and inclusive definition of learning

Launched 5 generic learning outcomes

3 more recent generic social outcomes

www.inspiringlearningforall.gov.uk

Generic learning outcomes

Knowledge and understanding

Increase in skills

Attitudes and values

Enjoyment, inspiration, creativity

Activity, behaviour and progression

Learning activities in libraries include any combination of the above

develop your story

select SMART indictors

decide on data collection methods

analyse your results

present and use your findings

Develop your story

• NEEDS why you are developing this

learning activity and who it is for? • INPUTS AND ACTIVITIES what

resources and activities do you need to develop to deliver this service

• OUTPUTS AND OUTCOMES what you expect you will achieve

Selecting SMART indicators

Indicators are used to measure, simplify and articulate your story

Indicators can be descriptive, performance or efficiency based

Any good indicator should be SMART – simple to implement, measurable, action-focused, relevant and time-bound

ENTITLE output indicators

Number of people who have participated in library activities designed to improve their ICT skills

Number of people who have participated in library activities designed to increase their interest in reading and/or improve their reading abilities

ENTITLE outcome indicators

These are specific changes in the attitude, behaviours, knowledge, skills and enjoyment of users as a result of library services

ENTITLE has agreed that the key generic outcomes relating to library ICT and reading activity are knowledge and understanding, skills and attitudes

How to collect your data

Review the data you already collect to see to see if you already collect evidence of generic learning outcome indicators

This will help you to decide on whether to adapt a data collection method you already use or develop a new method of data collection

Data you may already collect

Data source Collected

when

Does it give evidence of learning?

Comments book

Daily Yes

Annual library survey

Yearly Maybe

Evaluation forms

Post project Yes

Letters from users

Daily Maybe

Quantitative research

Requires a large number of respondents to ensure any sample you take is representative and averages meaningful.

Usually collected by questionnaires with multiple choice questions or by structured interviews with a series of closed or multiple choice questions

Qualitative research

Useful for understanding something in depth Involves talking at greater length to a smaller group of people Qualitative information is gathered by face

to face or telephone interviews, focus group discussions, comments cards or books, open ended questions on questionnaires

Ethics

It is important that library staff consider ethical or legal issues when undertaking research. Laws vary from country to country but there are shared principles on

Consent

Interviewing children

Confidentiality

Data Protection

Analyse your results

Analysis of the data you collect will reveal if the ‘story’ you mapped out at the beginning actually happened. That means you can test your hypothesis

The approach to analyse will vary according to the type of tool you used and the type of data you collected – whether it is quantitative or qualitative

Things to think about when analysing data

In interpreting results you need to decide if 64% is a high or low resultThe meaning and significance of the results – you may need to group the data into categories depending on who the programme is targeted atTemplate 2 provides examples of how to link outputs and outcomes to an indicator, data collection methods and results

Present and use your findings

Highlight your key messages in an executive summary

Describe the purpose of the learning activity that you have evaluated

Describe your chosen methodology

Think about how you could effectively present your findings

Conclusions – relate to your original story

Going further

Make your evidence more robust by • Examining the long-term impact of the

activity by re-surveying people to check if there has been any further progress

• Share your findings with other libraries locally and nationally to establish a benchmark

• Compare the results for users and non-users and identify if any significant differences

Advocacy

Make the most of the evidence youcollect

Advocacy is speaking out to win influence, gain support, recognition, partners and funding Combine your outcomes, research, policy and statistical evidence to create a powerful argument for the role your library plays in supporting lifelong learning.

For further information contact

Natasha Innocent

Senior Policy Officer: Learning and

Skills

MLA Council

[email protected]