Going Digital: Use of Technology in Monitoring and Evaluation Urmy Shukla & Sree Sen CLEAR South Asia at J-PAL South Asia at IFMR Second Development Marketplace 2013 Convening Workshop April 24 th , 2014
Going Digital: Use of Technology in
Monitoring and Evaluation
Urmy Shukla & Sree Sen
CLEAR South Asia at J-PAL South Asia at IFMR
Second Development Marketplace 2013 Convening Workshop
April 24th, 2014
Presentation Overview
Introduction
M&E Data – Challenges and Scope for Using Technology
Using Digital (Mobile) Tools for M&E
When to Use Digital Data Collection
Types of Mobile Data collection – Examples
Abdul Latif Jameel Poverty Action Lab (J-PAL)
Research affiliate network of over 92 professors in 34 universities; 491 evaluations in 56 countries
across 7 global offices
J-PAL South Asia: A Brief History
J-PAL South Asia office set up in
2007 at the Institute for Financial
Management and Research,
Chennai
Over 90 completed and ongoing
projects across South Asia:
education, finance & microfinance,
environment & energy, health,
political economy & governance,
labor markets and agriculture
Scientific Directors: Esther Duflo
(MIT), Iqbal Dhaliwal (Ex-IAS –
1996 batch)
CLEAR Initiative
• Global initiative coordinated by the
World Bank Independent
Evaluation Group
The CLEAR South Asia Regional
Centre has been hosted by J-PAL
South Asia at IFMR since May 2011
Established a partnership with the
Centre for Economic Research
(CERP) in Pakistan and works
closely with Innovations for
Poverty Action (IPA) in Bangladesh
CLEAR: Regional Centers for Learning on Evaluation and Results
CLEAR’s Strategic Vision:
• Building evaluation skills of
practitioners, program
implementers, and donors
• Strengthening M&E systems
and practices for strategic
clients
• Building a culture of evidence
based decision-making and
strengthening demand for
evaluations
CLEAR South Asia Mandate
CLEAR South Asia strategy includes
a focus on the use of cutting-edge
technology for effective collection
and utilization of M&E
In collaboration with Fieldata.Org,
CLEAR South Asia aims to promote
the use of M&E technology, such as
mobile phones and PDAs, for more
real-time aggregation and effective
utilization of M&E.
Context – E-society
Emphasis on the use of
technology for more effective
collection and utilization of M&E
The fundamental objective of all e-Society
initiatives is to make Information,
Communication & Technology (ICT)
more inclusive – i.e. ALL members of
society, especially those that are socially
and economically marginalized, should gain
access to and benefit from the knowledge,
power and opportunities brought about
by new ICTs
About Fieldata.org
Fieldata.Org is a mobile-&-web portal for NGOs to raise funds, by offering
real-time monitoring, and objective data for evaluation.
Mission is to improve transparency & decision-making in development
organizations and government agencies, by empowering them with
technology-tools for better monitoring, sharing, and application of data.
Want organizations and donors to objectively answer:
• Which projects should resources be allocated?
• How well do projects utilize these resources?
Programme Theory – Log Frame
Inputs/Program Activities
Outputs Intermediate
outcomes Goal
What we do as
a part of the
program -
deliver, teach,
offer loans, etc.
What are the
resources used
–funds, staff,
equipment,
curriculum, all
materials.
Tangible
products or
services
produced as a
result of the
activities -
usually can be
counted.
Short-term
behavioral
changes that
result from the
outputs -
preventive
health habits,
usage of
tablets.
Long-term
changes that
result from
outcomes – the
result of the
program.
Reporting: On expenditures, activities, coverage (targets)
Monitoring : Of use of inputs/activities and process –
Random spot checks of intermediate outcomes: Annual surveys
Evaluation: To assess long-term outcomes and
impact through studies
Reporting
• On Inputs and Outputs (Achievement of Targets)
Monitoring
• Of Processes and Implementation (Doing things right)
Evaluation
• Of Outcomes and Impact (Doing the right thing)
Management and Decision Making (MIS)
• Using relevant and timely information for decision making (reporting and
monitoring for mid term correction; evaluation for planning and scale up)
ALL OF THE ABOVE DEPEND ON THE AVAILABILITY OF RELIABLE, ACCURATE
AND TIMELY DATA
M&E Framework – Use of Data
Problems in Data Collection and Management
What do the following mean?
Data reliability (will we get the same data, when collected again?)
Data validity (Are we measuring what we say we are measuring?)
Data integrity (Is the data free of manipulation?)
Data accuracy/precision (Is the data measuring the “indicator” accurately?)
Data timeliness (Are you getting the data in time?)
Data security/confidentiality (Loss of data / loss of privacy)
Can technology help us address the above issues? How?
Digital Data Collection
Device Data Software Data Transmission Data Visualization
Mobile Devices
Open Data Kit,
Other softwares
Real time Dashboards,
Reports, summary
Tables, Graphs
Laptops,
Computers
Visual Basic, Java,
SQL, etc.
Online / Offline Excel/.csv files,
customized MIS
reports and
dashboards
GPS Arch GIS Real time Maps, Coordinates
Smart Cards and
hand-held devices
Software Online / Offline Summary tables /
customized reports
Biometric devices
(online)
Software Online / Offline
Summary tables /
customized reports
Why use Mobile Technology in Evaluations
Improving transparency & accountability in development organizations and
government agencies, though technology-enabled M&E for better
monitoring, sharing, and application of data.
Enabling organizations, donors and citizens to use M&E data for real-time
decision-making, better implementation and delivery of projects and
services
Why Mobile Data Collection?
Real-time data from the point of collection
Built-in logical flow and validation checks improves data quality
Ability to collect new types of data – Location (GIS), media (pictures,
audio)
Cost effective over time- involves one-time hardware costs and ongoing
maintenance. No paper, printing costs
Easy to manage and analyze large amounts of data
Reduces intermediate levels of data transmission
Why Digital Technology for Routine Monitoring
Paper Reports
Delay in reporting
Multiple levels of reporting
Information flow is one way
and not actionable
Bulky hard copies of reports
Errors in entry, needs
additional scrutiny
Using Mobile Phones
Almost instantaneous
reporting
Direct reporting
Information flow is both ways
and actionable
Web-enabled reports
Field-level checks on quality
incorporated
Why Digital Technology for Evaluations / Periodic Monitoring
Paper Surveys
Printing, tracking forms is tedious
for large surveys, changes costly
Many errors in data collection,
entry, manual scrutiny
Effective monitoring of data quality
is complicated and laborious
Requires additional hardware
devices for non-text data such as
gps, pictures, audio etc.
Using Mobile Phones
Deployed remotely, tracked in real
time, changes easy on the field
Limited errors due to built in logic
flows and validation, Single entry
Real time tracking, time/date/GPS
features improve monitoring
Single device for gps, audio,
pictures. Easy to integrate, can be
used for verification
Limited flexibility on field: Standardized formats for data collection; upfront
development of software tools upfront
Local language compatibility: Being developed for all languages
Typing may be slower: learning may take time (scribbling on paper is faster)
Lack of connectivity: on field limits real-time transmission of data
Potential Challenges with Digital Data Collection
Data Quality & Real-time Data for Monitoring Purposes
Improving:
• Data quality
• Speed
• Transparency
• Accessibility
• Flexibility
Quick & Easy Set up:
Mobile-based Monitoring and Evaluation in Action:
MFI agents entering weekly loan repayments
for instant tabulation
Community health workers feeding
back information on beneficiaries for
automatic identification of high risk
cases
Auditors collecting survey, observational,
photographic and GIS data on
infrastructure in slums.
Use of Mobile-based Technology in M&E
REPORTING/MIS
Routine (Real-time) reporting
• Weekly loan repayment information of MFI clients reporting by field staff
MONITORING
Ongoing program monitoring
• Beneficiary information collected and sent by health workers (for disease surveillance, delivery of benefits etc.)
Occasional (Surprise) checks
• Spot checks by supervisors to monitor attendance and performance of staff
EVALUATION
Survey data
• Household survey data to assess impact of <<xx>> program
Use of Mobile Technology as key intervention in Programs
Information/Messages
• SMS reminders to beneficiaries about important health activities
(treatment compliance)
Implementation Tools
• Videos and online teaching aids used in schools for regular or remedial
education programs
Biometric, GIS information
• Record beneficiary information for easier tracking and follow up.
Complementarities with other programs by same organization
Case Study: Delhi Voters Project
Project description and objectives
• An RCT evaluating whether providing information to government
officials and slum dwellers can lead to higher accountability and
thereby improved service delivery.
Interventions evaluated:
• The effect of providing information on spending and quality of public
services delivered by elected officials during election sensitive periods
Role of mobile-based data collection
The intervention
• Field-based audits of public services in slums using mobile-based
technology
• Surveys and observations
• Photographs
• GIS indicators
• Send report cards to elected representatives highlighting the quality
and access to public services in their area
The Implementation
GARBAGE FACILITIES
1) Dumpsters 2) Bins
3) Informal Points
TOILETS
Open Public Toilet
Complex
Public Urinals
Started by auditing …
Going a step further with Mobile Based GIS software
Accessibility to public services
• How far is the nearest
toilet?
• How far is the nearest
formal point of
disposing garbage?
Spread Analysis
• Does cleanliness of a
toilet affect household
health?
MOBILE GIS !
When to Use Digital Technology?
Mobile Phones / Tablets
(Smart phones)
SMS/IVRS based (Simple
phones)
GPS / Biometric Devices
Smart (Biometric) Cards
For periodic surveys, mapping,
supervision, electronic databases
Routine monitoring for delivery of key
services
Mapping and data verification at a micro
level (HH, beneficiary)
Direct payment to beneficiaries upon
verification, create digital databases
Initial one-time cost of devices
Additional costs for
maintenance (batteries,
replacement etc)
Ongoing data-plan costs, and
service-provider costs
Real-time access to data to
monitor quality and progress
Environmentally friendly
Mobile
No one-time ‘hardware’ cost
Ongoing costs of printing,
transporting and storing paper
questionnaires
Data-entry operations take
significant time and resources –
training
Longer time-frame before data
is available for analysis
Paper
Cost Effectiveness