Internet and Program Internet and Program Evaluation Evaluation The Non Existent The Non Existent Relationship Relationship Benoît Gauthier Réseau Circum inc. / Circum Network Inc. September 26, 2000 Version 1
Dec 30, 2015
Internet and Program EvaluationInternet and Program EvaluationThe Non Existent RelationshipThe Non Existent Relationship
Benoît Gauthier
Réseau Circum inc. / Circum Network Inc.September 26, 2000
Version 1
2© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
OutlineOutline
What non existent relationship? Internet program delivery
• how does it work?
• how is it influenced by program evaluation?
• potential influence
Program evaluation• how does it work?
• how is it influenced by the Internet?
• potential influence
Why the gap?
3© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Who's this guy?Who's this guy?
Political science, research methods, public administration
1983: social marketing research, Justice Canada 1984-1986: program evaluation, CMHC 1986-1989: program evaluation, DOC 1989-1996: mainly program evaluation, Ekos Research 1996-: social and organisational research as well as
program evaluation, Circum Network
4© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
What non existent relationship?What non existent relationship?
In ternetas a m eans o f
com m unication
P rogram evaluation
modifies / improves
In ternetprogram delivery
modifies / improves1
2
5© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Internet program deliveryInternet program deliveryHow does it work?How does it work?
Inputs
ActivitiesDeveloping
Outputs (tangible)
Immediate impacts
Long term impacts
Web siteInterface with other
delivery m ech.Usage reports
H um an resources Techno logy Other resources
S erveM anage serversM anage telecomM anage loadsEnsure security
R eceive and dealwith user input
Com plete transactionsAnswer e-m ail
Generate statisticsS um m arise feedbackP roduce reports
Access to contentTarget population
reach
S eek user feedback
Expediency o ftransaction
P rogram -specificim pacts
Target populationwelfare
Increased info andknowledge
ActivitiesDelivering
P lan content anddesign
Review docum entationInterview, discussReview literatureDraw plans
B uild content anddesign
Review best practicesDevelopTest
Update content anddesign
Review feedbackReview best practicesDevelopTest
R eport
User satis faction
R educed costs
6© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Influence of program evaluationInfluence of program evaluationon Internet program deliveryon Internet program delivery
Inputs
ActivitiesDeveloping
Outputs (tangible)
Immediate impacts
Long term impacts
Web siteInterface with other
delivery m ech.Usage reports
H um an resources Techno logy Other resources
S erveM anage serversM anage telecomM anage loadsEnsure security
R eceive and dealwith user input
Com plete transactionsAnswer e-m ail
Generate statisticsS um m arise feedbackP roduce reports
Access to contentTarget population
reach
S eek user feedback
Expediency o ftransaction
P rogram -specificim pacts
Target populationwelfare
Increased info andknowledge
ActivitiesDelivering
P lan content anddesign
Review docum entationInterview, discussReview literatureDraw plans
B uild content anddesign
Review best practicesDevelopTest
Update content anddesign
Review feedbackReview best practicesDevelopTest
R eport
User satis faction
R educed costs
Usabilty
Quality of co
ntent
Currency of co
ntent
User satisfa
ction
measurement
7© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Potential influence of PE on Potential influence of PE on Internet program deliveryInternet program delivery
Inputs
ActivitiesDeveloping
Outputs (tangible)
Immediate impacts
Long term impacts
Web siteInterface with other
delivery m ech.Usage reports
H um an resources Techno logy Other resources
S erveM anage serversM anage telecomM anage loadsEnsure security
R eceive and dealwith user input
Com plete transactionsAnswer e-m ail
Generate statisticsS um m arise feedbackP roduce reports
Access to contentTarget population
reach
S eek user feedback
Expediency o ftransaction
P rogram -specificim pacts
Target populationwelfare
Increased info andknowledge
ActivitiesDelivering
P lan content anddesign
Review docum entationInterview, discussReview literatureDraw plans
B uild content anddesign
Review best practicesDevelopTest
Update content anddesign
Review feedbackReview best practicesDevelopTest
R eport
User satis faction
R educed costs
UsabiltyQuality
of content,
exhaustivity
Currency of co
ntent
User satisfa
ction
measurement
Technical
performanceTurn around tim
e,
quality Performance
indicators
Proportion
reached, prop.
beyond reachContent se
rved,
extent of
duplication, ease of
access
Integrity of
transaction,
comparative speed
Knowledge gains...
Costs in
comparison to
alternative delivery
8© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Program evaluationProgram evaluationHow does it work?How does it work?
Inputs
ActivitiesPlanning
Outputs (tangible)
Immediateimpacts
Long term impacts
ActivitiesImplementing
P rogram s P rogram EvaluatorsM eans o f
com m unicationOther resources
Describe programlogic and context
Identify evaluationissues
P lan evaluationm ethodo logy
Review literatureInterview, discussAnalyse, strategize
Review literatureApply social sc ienceUse creativity
Collect in form ation Analyze inform ationCom m unicate
inform ation
W rite "reports"M ake presentationsD iscuss resultsP ublish findings
Assess optionsP lan
Review docum entationInterview, discussReview literatureAnalyse, m odel
Interview individualsInterview groupsS urveyObserveRecycle secondary data
P rocess qualitative infoP rocess quantitative infoM odel, test hypothesesReconcile
Data setsP rogram profile and
evaluation planR eports and
presentations
Linkages b/wknowledge m akers
and users
K nowledgedissem ination
Im proved decis ionsIm proved programdesign and delivery
B etter perform ingprogram s
A better soc iety
9© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Influence of the Internet on Influence of the Internet on program evaluationprogram evaluation
Inputs
ActivitiesPlanning
Outputs (tangible)
Immediateimpacts
Long term impacts
ActivitiesImplementing
P rogram s P rogram EvaluatorsM eans o f
com m unicationOther resources
Describe programlogic and context
Identify evaluationissues
P lan evaluationm ethodo logy
Review literatureInterview, discussAnalyse, strategize
Review literatureApply social scienceUse creativity
Collect in form ation Analyze inform ationCom m unicate
inform ation
Write "reports"M ake presentationsD iscuss resultsP ublish findings
Assess optionsP lan
Review docum entationInterview, discussReview literatureAnalyse, m odel
Interview individualsInterview groupsS urveyObserveRecycle secondary data
P rocess qualitative infoP rocess quantitative infoM odel, test hypothesesReconcile
Data setsP rogram profile and
evaluation planR eports and
presentations
Linkages b/wknowledge m akers
and users
K nowledgedissem ination
Im proved decis ionsIm proved programdesign and delivery
B etter perform ingprogram s
A better soc iety
Internet
Web surveys, on-
line focus groupsSpeedy, fr
ee
distribution
Increased speed
Increased speed
Increased speed
10© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Potential influence of the Internet Potential influence of the Internet on program evaluationon program evaluation
Inputs
ActivitiesPlanning
Outputs (tangible)
Immediateimpacts
Long term impacts
ActivitiesImplementing
P rogram s P rogram EvaluatorsM eans o f
com m unicationOther resources
Describe programlogic and context
Identify evaluationissues
P lan evaluationm ethodo logy
Review literatureInterview, discussAnalyse, strategize
Review literatureApply social scienceUse creativity
Collect in form ation Analyze inform ationCom m unicate
inform ation
Write "reports"M ake presentationsD iscuss resultsP ublish findings
Assess optionsP lan
Review docum entationInterview, discussReview literatureAnalyse, m odel
Interview individualsInterview groupsS urveyObserveRecycle secondary data
P rocess qualitative infoP rocess quantitative infoM odel, test hypothesesReconcile
Data setsP rogram profile and
evaluation planR eports and
presentations
Linkages b/wknowledge m akers
and users
K nowledgedissem ination
Im proved decis ionsIm proved programdesign and delivery
B etter perform ingprogram s
A better soc iety
Web surveys, on-
line focus groups,
real-time data
Increased speed,
two-way exchanges
Increased speed,
two-way exchanges
Increased speed, two-way
exchanges, more effective
&
efficient methods
Distributed
applications,
interactive multi-
party analysis
Speedy, free
distribution,
hyperdocument
model
Increased speed,
depth and reach
On-going dialog
Participation, buy-
in Participation, buy-
in
11© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Why the gap?Why the gap?
Impact of the program evaluation on IPD• experimental character of IPD
• cost of implementing rigourous research methods
• "low" cost of IPD compared to alternatives
Impact of the Internet on program evaluation• resistance to true participation (political cost)
• lack of openness to evidence from others
• knowledge silo model vs. knowledge addition model
12© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
ReferencesReferences
Simon Roy, Evaluating Public Web Sites, http://www.ggi.ca/E/WhatsNew.html
This presentation at http://circum.com (original texts)
13© Benoît Gauthier & Réseau Circum inc. / Circum Network Inc.
Where to find meWhere to find me
Benoît Gauthier
Réseau Circum inc. / Circum Network Inc.
819.770.2423, t/f 819.770.5196
http://circum.com