Dr Geoff Setswe, Dr Mark Ottenweller & Roger Dickinson SAMEA Evaluation in Action conference

Methods for the evaluation of the Abstinence and Faithfulness among Youth

(ABY) interventions in South Africa

Dr Geoff Setswe, Dr Mark Ottenweller & Roger Dickinson

SAMEA Evaluation in Action conference

30 March 2007

2

Structure of the presentation Introduction to the ABY project

ABY interventions at Hope Worldwide

Evaluation methods Survey methods Qualitative methods – Focus Group Discussions

Some baseline demographic results

Recommendations

3

Introduction

The goal of the ABY project is to enhance local responses to prevent HIV infection through encouraging abstinence, faithfulness and avoidance of unhealthy sexual behaviors among youth.

ABY interventions are implemented by Hope Worldwide in selected sites in 5 cities in South Africa.

Project is funded by PEPFAR/USAID over five years (2005-2009) in 5 African countries (SA, Botswana, Kenya, Nigeria and Ivory Coast).

We present methods used for the evaluation of ABY interventions in South Africa.

4

ABY interventions at Hope Worldwide Abstinence interventions encourage

unmarried individuals to abstain or delay sex as the best way to prevent exposure to HIV and STIs. Targeted at primary school learners aged 10-14 years.

Be faithful interventions encourage individuals to practice fidelity in marriage and in other sexual relationships as a critical way to reduce the risk of exposure to HIV. Targeted at youth aged 15-24 years.

Both interventions could be part of ABC or DRC programs

5

ABY interventions at Hope Worldwide Interventions are designed and implemented by

local people and adapted to fit local conditions;

Implemented in age-appropriate and culturally relevant groups, in targeted areas. Include life skills programs with components of Men As Partners (MAP) and VCT for youth;

For 10 -14 and 15 – 19 year-olds, interventions are school and/or church-based. For youth aged 20-24, interventions are church or community-based.

Parents and adult church members have interventions specifically targeted to them.

6

MethodsABY interventions are evaluated using mixed methodologies - quasi-experimental and qualitative designs with 3 strategies:1. Pre-intervention or baseline surveys for learners and

youth; just before the interventions were implemented in 2005/6.

2. Post-intervention surveys for learners and youth (who participated in the interventions); when the interventions have been completed in 2008/9.

3. Focus Group Discussions (FGDs) for parents and church members who participated in the interventions. FGDs will be conducted once – during the middle of the project, in 2007.

7

Pre- and Post intervention surveysOne-group, Pretest/Posttest design

A single group i.e. learners and youth, is studied before and after the ABY intervention. No control group.

Layout O1 X O2

O1 = Baseline or pre-intervention survey in 2005/6

X = ABY intervention in 2006/7 to selected participants

O2 = Post-test survey in 2008/2009

8

Sampling and sample size estimation

City No. of sites

No. of sampled sites

No. ofevaluationpoints

No. of participants

Johannesburg 8 2 3 x 2 = 6 60 x 6 = 360

Cape Town 4 2 3 x 2 = 6 60 x 6 = 360

Durban 3 2 3 x 2 = 6 60 x 6 = 360

Mthatha 3 2 3 x 2 = 6 60 x 6 = 360

Port Elizabeth 2 1 3 x 1 = 3 60 x 3 = 180

Total 20 11 27 points 1,620

9

Sample size estimation

11 sites were randomly selected from 20 intervention sites. Then 27 evaluation points were identified.

The evaluation points comprised: primary schools (for learners aged 10-14 years), high schools (for learners aged 15-19 years) sites for out-of-school youth (aged 20-24 years).

Data were collected from 60 randomly selected learners or youth at each of the 27 evaluation points = 1,620.

Grades 11 and 12 were not evaluated as they cannot be followed up in post-intervention surveys

Data were collected by trained fieldworkers.

10

Survey questionnaires

Two types of questionnaires were used:

1. for learners aged 10-14 years and 2. for learners/youth aged 15-24 years.

Key questions were on:

Awareness/knowledge of HIV/AIDS and attitudes towards PLWHA

Views on dating, relationships, sexuality and pregnancy Views on abstinence and faithfulness The role of the media, role models and workshops in

encouraging abstinence and faithfulness The role of peer-group pressure and violence on abstinence

and faithfulness.

11

56.5%(740)

43.5%(570)

n = 1310. Response rate = 81%

Some baseline demographic results Gender distribution of respondents

12

Some baseline demographic results Distribution of respondents according to cities

20%

33.1%

13.1%

25.6%

8.2%

13

38.5%(505)

46.6%(610)

12.5%(164)2.4%

(31)

n = 1310

Some baseline demographic results

14

Qualitative methods - FGDs

15 FGDs planned – 3 in each of the 5 cities: 5 FGDs - adults who have participated in the ABY interventions

provided through FBOs in the 5 cities 5 FGDs - adults or parents of children at ABY intervention

schools in the five cities. 5 FGDs - persons aged 25 years and above from the general

population in each of the 5 cities.

FGDs will be used to gather in-depth information on parents and church members' opinions and views on abstinence and faithfulness in relationships.

Each FGD will have 8 - 12 participants who will be asked to provide verbal consent to participate and permission to record the interviews.

The training for facilitators of FGDs has been scheduled and the FGD guide is ready.

15

Schedule of ABY FGDsCity Sites Target group

Cape Town To be confirmed (TBC) FBO adults

Parents at schools

25 years + general population

Durban TBC FBO adults

Parents at schools

25 years + general population

Johannesburg TBC FBO adults

Parents at schools

25 years + general population

Mthatha TBC FBO adults

Parents at schools

25 years + general population

Port Elizabeth TBC FBO adults

Parents at schools

25 years + general population

16

Recommendations To ensure credibility:

evaluation should be integrated into the initial planning of the project;

all evaluation preparations and training should be done before the interventions start.

To ensure effective evaluation of ABY interventions, it is essential to: use various designs and strategies in different target

groups; involve team members in the design of the

evaluation; train team members on the methods of evaluation,

data capturing and the importance of data quality and management.

17

Acknowledgements

ABY intervention and evaluation teams at Hope Worldwide South Africa

Learners, youth, parents and adults who participated in the evaluations and FGDs

Funding received from USAID/PEPFAR