Introduction to Participant Observation as a Data Collection Method in Program Evaluation

1© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

See the PDE booklet, Collecting evaluation data: Direct observationshttp://learningstore.uwex.edu/pdf/G3658-05.pdf

Using observation to collect evaluation data

2© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Checking in…

– Who has used observation as a data collection method?

– What are your experiences with it?

– Who is hoping to use it – How?

3© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Participant observation: What it is

4© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Quantitative vs. qualitative methods

Quantitative Qualitative

Survey Observation

Tests Interview

Questionnaire Focus group

5© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Observation…

Involves all 5 senses: sight, hearing, smell, touch, and taste

observation

includes more than just “seeing”

6© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Use in program evaluation

• When you want direct information• When you are trying to understand an

ongoing behavior, process, unfolding situation, or event

• When there is physical evidence, products, or outcomes that can be readily seen

• When written or other data collection methods seem inappropriate

7© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Observations

Advantages– Provides direct

information; not dependent upon someone’s response

– Unobtrusive– See things in

natural context– Flexible;

discovery oriented

Disadvantages– Observer’s presence

may create artificial situation

– Potential for bias– Time consuming– Requires diligence,

preparation– Challenging to collect

data while participating

8© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Observation – Ethical issues

• Unobtrusiveness is its greatest strength; also potential for abuse in invasion of privacy

• You can venture into places and gather data almost anywhere so questions re. what is ethical– Overt vs. covert

• Remember our Human Subjects Protection guidelines– Consent form for participating in an

observational study Consider cultural appropriateness of using observation

9© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

What are the implications for using observation as an evaluation data collection method?

10© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Participant observation: How to do it?

11© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

What to observe• People (individuals, groups,

communities)– Characteristics – Interactions– Behaviors– Reactions

• Physical settings• Environmental features• Products/physical artifacts

Observing what does not happen may be as important as observing what does happen

12© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Types of observation

Structured UnstructuredLooking for vs. Looking at

Sometimes we have something specific we want to observe – leadership skills; level of participation; etc. We use a structured, preset guide of what to observe or a checklist.

Sometimes we want to see what is naturally occurring or exists without predetermined ideas. We use have an open-ended approach to observation and record all that we observe

13© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Practice: Structured/unstructured observations

Imagine you are sitting in a room where ten youth are sitting at computers learning about Web 2.0 applications.1) If you want to assess to what extent

students are interested and learning, what specifically would you look (listen) for?

2) If you aren’t sure what specifically indicates student interest or learning and you want to see what is going on during the demonstration, how would you proceed?

14© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Example – Observing participation in an after school program

• Who you will observe: youth attending the program

• What you will observe: – Age, gender– Length of time student stays in the

program– Involvement in activities: which activities

• Level of involvement – Interactions with other youth; with staff

• When you will observe: all hours the program is open for one week each month during 2007

15© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Recording your observations

It is not good enough to just observe, you need to systematically record your observations. You might use:

– Observation guide– Recording sheet– Checklist– Field note– Picture– Combination of the above

16© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Sample Observation GuidesGuide for structured observations Guide for unstructured observations

17© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Structured observation guide used for pre and post program evaluation

18© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Field Notebook

Notebooks to record and collect observations as they occur

19© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Who does the observations?

• You – program staff• Participants - Youth• Parents• Teachers• Volunteers • Other stakeholders• Colleagues

20© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Training – preparation/orientation may be necessary

– To learn what to look for – To learn how to record observations– To practice– To ensure that observations across

sites are consistent: observers use the same methods, rate an observation in same way

21© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

How well do you observe?

22© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

23© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Becoming a skilled observer includes…

• Learning to pay attention, see what there is to see, and hear what there is to hear;

• Practice in writing descriptively;• Acquiring discipline in recording notes;• Knowing how to separate detail from trivia;• Using rigorous methods to validate and

triangulate observations;• Reporting the strengths and limitations of

one’s own perspective» M.Q. Patton, 2002. Qualitative Research and

Evaluation Methods. Sage, pg 260

24© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Data analysis and interpretation

Qualitative data = qualitative data analysis– Standard content analysis

• Get to know your data• Focus the analysis• Categorize information• Identify patterns and connections• Interpret – bring it all together

PDE booklet: Analyzing Qualitative Data http://learningstore.uwex.edu/Assets/pdfs/G3658-12.pdf

25© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Practice your observation skills everyday in everyway!

And, add observation

to your data collection toolbox.

26© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation

Steps in planning for observation• Determine who/what will be observed.• Determine aspects that will be observed

(characteristics, attributes, behaviors, etc.).

• Determine where and when observations will be made.

• Develop the observation guide• Pilot test the observation guide• Train the observers and have them

practice.• Conduct the observations • Analyze and interpret the collected

information.• Write up and use your findings.