WP Activity Evaluation Toolkit: Activity Evaluation Questionnaire/file/1... · 2013-01-24 · Widening Participation Research and Evaluation Unit 5 Using the WP Activity Evaluation

Widening Participation

Research and Evaluation

Unit

1

Widening Participation Research and Evaluation Unit

WP Activity Evaluation Toolkit:

Activity Evaluation Questionnaire



Unit

2

Contents

Background to the WP Activity Evaluation Process…………………………………………………….p.3

Using the WP Activity Evaluation Questionnaire…………………………………………………………p.5

Appendix 1: Completing the Evaluation Questionnaire Request Form………………………….p.9

Appendix 2: For Participants: Evaluation Questionnaire FAQs……………………………………..p.12

Appendix 3: Sample Evaluation Questionnaire Request Form……………………………………..p.13

Appendix 4: Sample Questionnaire…………………………………………………………………………..p.15



Unit

3

Background to the Evaluation Process

The Widening Participation Research and Evaluation Unit (WPREU) is developing a tool kit to

enable the consistent monitoring and evaluation of Widening Participation (WP) related

activities taking place across the University.

During the current phase of evaluation (May 2012 – July 2013), we are aiming to evaluate between 60-70% of sustained activities (2 or more activities with the same groups of

participants) and 30-40% of „one off‟ or standalone activities that meet the following definitions:

they are intended, at least in part, to reach a group or cohort that is under-

represented in Higher Education

they are intended to achieve WP related objectives (raising aspiration, raising

awareness, providing relevant advice and guidance or developing relevant skills and

knowledge)

they are intended for pupils in school years 9-13.

Phase 2 of the evaluation plan (July 2013-14) will broaden the scope of evaluation to include

pupils at all stages from Primary School onwards and a focus on WP student retention and

support.

Why Are We Evaluating Our WP Activities?

Although effective evaluation of activities often takes place at a local level, we do not yet have a

structured and comprehensive way for the University to monitor or evaluate the impact of its

WP activities, and consequently it is difficult for us to identify and share good practice. We are

also required to report on the impact of our activities and our evaluation plan in the

University‟s Access Agreement and WP Strategic Assessment. External agencies such as HEFCE (Higher Education Funding Council for England) and OFFA (Office For Fair Access) are

increasingly focusing on the importance of monitoring and evaluating WP activities, not least

to demonstrate value for money.

Driven by this convergence of needs, the WPREU was established in May 2012. One of the

responsibilities of the Unit is to lead on a comprehensive evaluation strategy and to provide

practitioners with a toolkit for use in evaluating their activities.

This evaluation toolkit is designed to enable the University to:

a) monitor the demographic profile of participant cohorts to support effective targeting b) measure and evaluate the impact of an activity on its participants and to gather

evidence about their experience of the activity c) compare different types of WP activity in order to identify the most effective

techniques for achieving particular WP objectives and to share good practice.



Unit

4

The Evaluation Process

This evaluation process is conducted using a mix of methods;

Delivered by WPREU:

participant focus groups and interviews

participant email surveys

teacher interviews

parental surveys

Delivered by practitioners:

evaluation questionnaires

WP activities conducted in collaboration or partnership with the University‟s central Outreach Team will usually be evaluated by the Outreach staff running or supporting the event.

Where an activity is being delivered by practitioners based in faculties or academic

departments, we recommend that, where appropriate, the standard evaluation

questionnaire is used to evaluate the activities.

For more information about the WP activity evaluation plan or to request evaluation

questionnaires for your activity, please visit the WPREU Evaluation webpage at:

www.shef.ac.uk/lets/strategy/wp

If you do not feel that the standard evaluation questionnaire is not appropriate for your

activity or you would like to discuss alternative forms of evaluation, please contact Julian

Crockford ([email protected] ).

http://www.shef.ac.uk/lets/strategy/wp

mailto:[email protected]



Unit

5

Using the WP Activity Evaluation Questionnaire

The WP activity evaluation questionnaire is designed:

a) to ask a consistent series of questions about a WP focused activity. This allows a wide

range of activities taking place across the University to be compared

b) to allow a degree of customisation, so that questionnaires can be tailored to specific

activities

Using the evaluation questionnaire involves four main stages:

Activity WPREU Resource Location 1 Contact schools to inform them about

the evaluation and data collection requirements.

We recommend that this is done during the organisation of the activity.

Standard letter to school contacts

www.shef.ac.uk/lets/strategy/wp/questionnaires

2 Request customised evaluation questionnaires for your WP activity from the WPREU

Evaluation Questionnaire Request Form (For Staff)

WP Activity

Evaluation Toolkit – Information For Staff (this document)

Sample Evaluation Questionnaire


3 Give activity participants information about completing the questionnaire

Evaluation Questionnaire Information Sheet (For Participants) and/or Evaluation Questionnaire Information Sheet (For Participants)


4 Return completed questionnaires to the WPREU using the envelope provided. WPREU will process the forms and provide you with an evaluation report. This will include:

a) statistical demographic breakdown of the participant cohort

b) statistical break down of participant responses

c) collation of their free text comments

Sample evaluation reports




Unit

6

1 – Contact schools to inform them about the research

Because the questionnaire will be gathering personal information from participants, it is

important that their schools are aware of the research project and our commitment to data

protection. Expert advice about data protection has confirmed that young people are

considered able to consent to the use of their personal data.

We need to contact schools prior to the activity (preferably during the organisation stage) to

provide an outline of the evaluation research, to reassure them of our commitment to Data

Protection and to give them a point of contact if they have any questions or concerns about

the research.

A standard letter for this purpose is available from:


We recommend that this is given to all schools participating in the activity as early as possible

in the planning process of the event. For annually repeated events, we strongly recommend

that you send a copy of the letter at the beginning of each cycle.

2 – Requesting questionnaires for your activity

Please use the Evaluation Questionnaire Request Form (For Staff) to request pre-printed

questionnaires for your activity. This will give WPREU staff the necessary information to log

your activity and provide you with a set of custom questionnaires for your activity. This form

allows you to customise the questionnaire to your activity (e.g. by defining the key

developmental objectives of your activity). Each form is also uniquely coded for each

participant.

The Evaluation Questionnaire Request Form is available from


This form is available as a word document that can be sent back to WPREU by email (to

[email protected] ) or internal post (WPREU, LETS, 5 Favell Road).

If you would like more information about completing this form, please see Appendix 1 -

Completing the Evaluation Questionnaire Request Form (p.9).

Timescales:

We aim to provide your questionnaires within 3 days of receiving your request form. If we

return these in the internal mail this can take 2-3 working days. Although we will try to

produce your questionnaire as quickly as possible, we recommend that you request forms at

least two weeks in advance of your event. If you need forms for an event that is less than a

week away, please contact Sarah Bows ([email protected] ext. 21309) or Julian

Crockford ([email protected] ext. 22475).

http://www.shef.ac.uk/lets/strategy/wp/questionnaires







Unit

7

3 – Prepare activity participants for completing the questionnaire

We recommend that you leave 10-15 minutes at the end of your activity, if possible, for

participants to complete and return the questionnaire. For sustained activities, where we

already know who the participants are and do not have to collect demographic information,

we recommend leaving 7-10 minutes for the questionnaire.

It is important that participants have relevant information about the questionnaire and the

implications of completing it. We have provided a powerpoint slide that can be incorporated

into any presentation you are using or a fact sheet to handout with the questionnaire. Both of

these are available from the WPREU webpages at


If participants raise any questions about the questionnaire or about the evaluation process

that you are not comfortable answering, please refer them to the Widening Participation

Research and Evaluation Unit at [email protected]

The key points that participants need to know are:

They should fill in the questionnaire in by colouring in the blobs that correspond to

their response or by filling in the text box

If they do not think a particular question applies to the activity they have undertaken

they should just use „not sure‟ They should not feel obliged to answer any question that they are uncomfortable with

If they do not want to complete the questionnaire, they do not have to do so, and

should just strike a line through the front and hand it back

They should be honest. The activity leader will put the questionnaires into an

envelope that will then be sealed. Although activity leaders will get a report showing

the responses, they will not know who said what. The only people who will be able to

see participant‟s personal data will be a small number of researchers in the WP Research and Evaluation Unit

They can ask for their personal information to be removed from the database if they

wish

Please do not read the questionnaires, but return them directly to the WPREU in a sealed

envelope. We would like to reassure participants that their responses will remain

anonymous. You will be given all the responses as part of an evaluation report.

4 – Return Completed Questionnaires to the WPREU

WPREU will aim to process your questionnaires and provide your evaluation report within

two weeks of receiving your completed questionnaires.





Unit

8

What Happens Next?

When you have returned your completed questionnaires, the WPREU team will process them

using optical mark reading software (OMR). The data will be added to our evaluation

database and a report produced and sent back to you by email. This report will give you;

a demographic breakdown of the participants for your activity

a breakdown of their responses to the questionnaire questions

a report showing all their written responses.

We will aim to process your data and return your report within 2 weeks of receiving your

questionnaires.



Unit

9

Appendix 1 – Completing the Evaluation Questionnaire Request Form

We ask for your contact details and for information about your activity:

The information you include about the title and date of your activity will appear on the

questionnaire header:

As it appears on questionnaire:



Unit

10

If you are running a „subject based‟ activity, the subject(s) you have indicated will appear on a special section of the form (which does not appear for non-subject based activities).


Skill Development Objectives

We ask you to identify the key outcomes of your activity, in terms of its intended impact on

participants, so that these can be evaluated. These can be chosen from a standard set of

activity objectives. Using these will allow us to compare the impact of different activities that

share the same objectives.

The skill development objectives will appear in Section 3 of the questionnaire, which asks

respondents to indicated how much they feel that they have improved in a particular area as

a result of the activity. Please note that “Motivation towards school work” and “Self- confidence” are standard objectives that are automatically included on all questionnaires.



Unit

11


If you find that you have specific objectives that are not covered by the categories included

on the form, you can add up to two customised skills development objectives.


Due to the limitations of space, for subject based activities you can only include 3 skills

development objectives on the form, or 5 for sustained or generic activities.

Resource Implications

We ask you to include some background information about the activity. This will help us to

build a picture of the investment in WP activities across the University and to bring more

context into the evaluation process, however, if you are not able to easily provide this

information please do not delay in returning your form.



Unit

12

Appendix 2 – For Participants: Evaluation Questionnaire FAQs

Why are you asking activity participants to fill in a questionnaire?

The University of Sheffield is evaluating its Widening Participation (WP) and Outreach

activities to gather evidence about what works and to make sure that they are reaching the

people they are intended for.

We also want to find out what kind of activities work best and have the most impact on the

people who take part in them. This will help us share good practice and make sure that we

are concentrating our efforts where they have most impact.

The outcomes from the evaluation process will help the University decide which activities to

concentrate on and, ultimately, how it can best achieve its aim of increasing the

opportunities for people from under-represented groups to benefit from study at Sheffield

or at other universities.

What will happen to the personal information participants provide?

The University of Sheffield takes data protection very seriously. Information about

participants will be held on a secure drive on a database that will be used only by a very small

number of researchers.

The only times we will use this personal information are:

1 – to contact participants again to ask them to take part in some email surveys (but only if

they have given us permission to do so)

2 – to identify participants if they apply to Sheffield in the future. If this is the case we will use

this information for tracking and monitoring purposes only – it will not have any impact on

the application (although participants are free to contact the Admissions Tutor to tell them

they have participated in one of the University‟s WP programmes or to indicate this on their UCAS form)

3 – to see if participants‟ opinions change if they are taking part in a number of activities at the University.

If the information above does not answer your question, or you would like more information

about any of the above, please contact the Widening Participation Research and Evaluation

Unit at [email protected]




Unit

Activity Evaluation Questionnaire Request - SAMPLE

Name: Mr W.P. Practitioner

Faculty: Science

Department: Animal Plant Sciences

Date activity is due to take place: 12.12.2012

Title or name of activity: Biology Lab Visit

Activity type: (please select most relevant category)

[ ] Part of sustained Programme (repeated contact with specific

Participants)

[ ] Subject based (the predominant focus of the activity is on a specific

subject or subjects – the activity is not part of a sustained programme)

[ ] Generic (general one-off or ad-hoc events that are not predominantly focused

on a particular subject, subjects or discipline)

For „subject based‟ activities, please indicate the subject or subjects that the activity will cover:

Biology

Expected number of participants: 29

Please give school year the activity is intended for (eg year 11)

Year 11

Key Skills Development Objectives The evaluation form will automatically ask applicants to evaluate their improvement in the areas of motivation towards their school / college work and their self-confidence. There is room to ask additional questions based on the specific objectives of this particular activities. Please note that for „subject based‟ activities there is only room to ask about an additional 3 skill development objectives. For „sustained‟ and „generic‟ activities there is room to ask about an additional 5.

Personal skills (Please tick those that apply)

[ ] Working with others / Teamwork [ ] Communication skills [ ] Time management skills [ ] Making the right study / subject / progression choices [ ] Thinking about or planning my career

[ ] Working to achieve my full potential Academic skills

[ ] Presentation skills [ ] Library skills [ ] Revision skills [ ] Essay writing skills [ ] Research skills [ ] Maths skills [ ] Science skills Application Skills

[ ] Applying Through UCAS [ ] Writing a UCAS personal statement Skills specific to this activity If the specific objectives of your activity are not included above, please add up to two

customised objectives below. These will appear on the form as: “Please tell us how much you feel you have improved in the following areas as a result of this activity” -

[ ] Saftety in the lab

[ ] Learning about different types of Biology

Questions continued overleaf, please turn over…

If at all possible, please answer the following questions. If you are not able to provide the information at this time – or you have already provided it to the Outreach Team, please send the form back without this information. We may contact you about this later. How is this activity funded (if other, please also indicate source(s) of funding)

OFFA / Other

If possible, please indicate the estimated cost of this activity (not including staff costs)

If possible, please indicate roughly how many hours of permanent staff time it has taken to plan and deliver this activity

If possible, please indicate roughly how many hours of temporary staff time it has taken to plan and deliver this activity

*DA12SUS12154/141112*

*151112111313-0001* 151112111313-0001 DA12SUS12154/141112 3

O

Office Use

FOR INFORMATION ONLY - DO NOT USE FOR YOUR ACTIVITY <Activity Title> Evaluation Form <Activity Date> How to Fill In This Questionnaire

There are two types of question on this questionnaire.

Some questions have a box for you to write in: Please give us your opinion…..

Other questions ask you to colour in a blob with a pen or pencil to indicate your answer:

Yes No

Eg. To agree Yes No

O O O

Personal Information

Your first name: Please chose the option below which is closest to how you would describe your

Your surname / family name:

Date of birth: (dd/mm/yyyy)

Male Female Your gender:

O O

Home postcode:

Please give us your unique learner number (ULN) if you have it:

Your email address: We would like to contact you after the activity to ask you to take part in a survey. Everyone who takes part will be put into a draw for a £50 voucher. Please colour in this blob, if you would NOT like us to contact you in future:

Name of your

ethnic background:

White O Gypsy or Traveller O Black or Black British - Caribbean O Black or Black British - African O Other Black Background O Asian or Asian British - Indian O Asian or Asian British - Pakistani O Asian or Asian British - Bangladeshi O Chinese O Other Asian background O Mixed - White and Black Caribbean O Mixed - White and Black African O

Mixed - White and Asian O school or college:

School/college Year:

9 10 11 12 13 Other

O O O O O O Yes No

Other Mixed background O Arab O Other Ethnic background O

Have any of your close family studied at university before?

Parent or carer? O O

Brother or sister? O O

Not known O Prefer not to say O

What does your father / male carer do as their main job (please put unemployed or retired if they’re not working):

What does your mother / female carer do as their main job (please put unemployed or retired if they’re not working):

Only

Section 1 – About your future plans

YES NO Not Sure

Before this activity… O O O

…were you planning to go to university? O O O

After this activity… O O O

…can you see yourself studying at university in the future? O O O

…can you see yourself studying at Sheffield University in the future? O O O

…has your understanding about university increased? O O O

Section 2 – About this activity

How much would you say this activity has… A Lot Average Not At All Not

Sure 1 2 3 4 5 6 7

…made you feel confident about ‘fitting in’ at

university?

O

O

O

O

O

O

O

O

…made you feel confident that you can get the grades to get into university?

O

O

O

O

O

O

O

O

Data Protection: We will store the data you provide securely and in accordance with the Data Protection Act. We will use this data internall y to monitor and evaluate our activities. Although, we may supply

statistical data to bodies such as the Higher Education Funding Council (HEFCE) or the Office For Fair Access (OFFA), the data will be anoymised to ensure you cannot be identified. For more information about the questions we are asking and how we will use the data, please see www.shef.ac.uk/lets/WPREU

http://www.shef.ac.uk/lets/WPREU

*DA12SUS12154/141112*

*151112111313-0001* 151112111313-0001 DA12SUS12154/141112 3

A Lot Average Not At All

1 2 3 4 5 6 7

O O O O O O O

O O O O O O O O O

How much has this activity increased your A Lot Average Not At All Not

knowledge about: 1 2 3 4 5 6 7 Sure

The costs of studying? O O O O O O O O

Bursaries and financial support? O O O O O O O O

Social life and activities? O O O O O O O O

The chances of getting a job after university? O O O O O O O O

Please tell us about any issues that have an impact on how you feel about applying to university:

What is most important to you when

choosing a subject to study? (Please only

I’m interested in

the subject

I’m good at the

subject

It leads to a well-

paid job

It leads to an

interesting job

choose one) O O O O

Yes No Not

Sure

Have you had the chance to study or learn about Geography before? O O O

After this activity do you feel you would want to study Geography at University? O O O

Please tell us how this activity has made you feel about Geography?

Section 3 – Has this activity helped you? Please tell us how much you feel you have improved in the following areas as a result of

A Lot Average Not At All No

Not

this activity: 1 2 3 4 5 6 7 Change Sure

Motivation towards school /college work O O O O O O O O O

Self-confidence O O O O O O O O O

Making the right study/subject /progression choices

Thinking about or planning my career O O O O O O O O O

Working to achieve my full Potential O O O O O O O O O

Please tell us about any other skills or knowledge you feel you have improved as a result of this activity:

What was the most enjoyable thing

about the activity?

What was the least enjoyable thing

about the activity?

Is there anything you feel we could

improve for future activities?

How much would you recommend

this activity to your friends?

Not Sure

O

Is there anything else you’d like to

tell us about the activity or how you

feel about applying to university?

WP Activity Evaluation Toolkit: Activity Evaluation Questionnaire/file/1... · 2013-01-24 · Widening Participation Research and Evaluation Unit 5 Using the WP Activity Evaluation

Documents