The use of ICTs in programme evaluation (Tearfund HIV programme) Gamos May 2012

The use of ICTs in programme evaluation

(Tearfund HIV programme)

Gamos

May 2012

Digital Survey Method

KAP Surveys

• Research

• Assessments

• Evaluation

• Accountability – finding out what works – and proving it!

• Two purposes of KAP surveys:• To understand what’s going on (formative learning)

• To show changes that have taken place over duration of programme

• Key features of methodology:• Large, randomly selected sample

• Internationally tested and accepted indicators

• Statistical tests for reliability

Impact Surveys (evaluation)

Current Status Knowledge/ Attitude Behaviours Outcomes

Individual Descriptors

HIV Knowledge VCT Poverty Descriptors

Household Descriptors

Stigma reactions PMTCT Change Questions

Religious Stigma Poverty Descriptors Religious Involvement OVC

Comment on Religious Inst

Religious Care

Sexual Activity

Experience of Support for the sick, dead

and orphans

Experience of Support for the sick, dead

and orphans

Intentions

Attitudes towards sex

Discuss HIV

Engaged in AIDS Activities

KAP Structure

Example KAP – baseline and endline KAP surveys for Tearfund’s HIV programme in Ethiopia and Malawi.

KAP Relationships

KAP Findings

AIDS activitiesantenatal caresexual behaviourtalked about AIDStestingknowledgestigmaattitudes to sexwomens rightschangessupport

Positive change in indicators linked to exposure to HIV information through Church partners

Survey process (paper)

RecruitmentDesign survey

Conduct interviews

Data entry

Training

Field preparation

s

Submit data Analysis & reporting

Technology can assist with data entry:•Devices for data capture (CAPI software) – eliminates need for data transcription

•Optical reading technologies – eliminate human errors in data entry activity;

•Computer assisted data entry – reduces human errors

Review of ICTs for surveys

• Early experience with PDAs was not encouraging• Netbooks require PC skills (that are rare)• Mobile phones – good for data collection but

screen too small for CAPI.

• NOW tablets! Low cost, big screen, apps do not need computing skills, mobile phone skills are universal.

ICTs for surveys – data capture

• Optical reading technologies – too expensive; better to give budget to partners.

• Computer assisted data entry – Gamos created bespoke software for each project.

ICTs for surveys – data transcription

Survey process - DigitalForms

RecruitmentDesign survey

Conduct interviews

Data entry

Training

Field preparation

s

Submit data Analysis & reporting

Eliminates data entry

• Tearfund & Gamos decided to invest in digital technology – HIV IMPACT programme in Malawi

• Key decisions:– Operating system: Android v. iOS (Apple)

– Bespoke software v. Freeware (Android)

– Devices as capital (asset) or expenditure (disposable)

– Automatic or manual management of data files

DigitalForms

Cost savings:– Data entry software – eliminate need for computer assisted data entry;

– Data entry – no need to recruit data entry staff or to appoint supervisors.

– Printing costs – no need to copy paper questionnaires.

Additional costs:– Hardware – handheld devices

– Writing software to run on the tablets

Overall – no cost premium

DigitalForms

• Modules:– Household & respondent (poverty & social)

– Children & pregnancy (contraception, ANC, delivery, PNC)

– HIV/AIDS awareness (exposure to information, knowledge, attitudes)

– Testing & counselling (as part of ANC and other)

– Response of churches (activities, support, attitudes to PLWH)

– HIV status (consent and disclosure)

• Loops - household membership roster; child birth history

DigitalForms – the product

• The kit: for enumerators:– Tablet– Mains charger– Micro SD card– Backup battery (not all kits)– Carrying case

• For supervisors:– Car battery charger – enables 2 devices to be

charged up from the car battery;– USB microSD card reader – for copying files

onto a central laptop.

DigitalForms – the product

• Training workshop – nothing to report; brief introduction to tap & swipe, then people got into it – seems quite intuitive.

• Challenges from field experience:

– enumerators want the ability to edit records – Gamos created checking software to help supervisors check (but not edit) submitted files;

– Questions need to be numbered so that teams can discuss issues / edits;

– field reports mentioned tablets running out of power (problems with backup batteries?).

Experience to date

• Completion rates – only 1 question was ‘forced’, otherwise non-completion rates were typically around 2%.

• Downloading of updated versions – workshop used beta version and demonstrated downloading software; teams downloaded release version successfully (1 error).

• Android update - 4.30 on Friday afternoon (release date) Asus upgraded to Android 4.0.3 – panic! Fortunately, rollout did not extend to Malawi.

• Clean data!

Experience to date

• Complete user review process• Automatic data uploading (to server)• More comprehensive data validation• Develop more modules• Cheaper devices

www.gamos.org

Next steps…

Supplementary slides

Baseline – endline analysis• Difference-in-differences method

• Takes account of errors:•Beneficiaries may be different from others (selection bias);

•Other changes may take place over time

Beneficiaries (treatment)

Non-beneficiaries (control)

Before after

Exposure to HIV/AIDS messaging

•Church based:•Information – heard public talks (e.g. sermons), seen or heard information at a religious institution relating to HIV;

•Participation in AIDS related activities organised by churches;

•Personal contact – talked about AIDS or multiple sexual partnerships with church members or leaders.

•Others:•Broadcast media – radio, TV

•Health services – antenatal care

•Health services - HIV tests (not antenatal care)

Definition of links (impact?)

Exposed group (treatment)

No change (or negative)

Positive change

Non-exposed group (control)

No change (or negative)

- Link flagged

Positive change

- -

Value of approach

Generated data on:•Access to information on HIV/AIDS•Responses of churches - HIV activities, stigma within church, promote testing, policies•HIV knowledge – protection, PPTCT•Stigmatising attitudes•Attitudes towards sex, women’s sexual rights, •Access to ANC – male involvement•Testing•Sexual behaviour – age at debut, MCPs, condoms•Outcomes – changes in behaviour, care and support, intentions