Integrating Gender into Project-level Evaluation

June2017

IntegratingGenderintoProject-levelEvaluation

ECGReferenceDocument–Appendixes

ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes ii

AcknowledgmentsThemethodologicalguidelinesaretheproductofacollectiveworkofanECGTaskForcecomposedofthefollowinginstitutionsandmembers,ledbytheAfricanDevelopmentBank(AfDB),andsupportedbyMichaelJ.Bamberger,ExpertConsultant.

TaskForcemembers

IndependentDevelopmentEvaluation(IDEV)AfricanDevelopmentBankGroup(AfDB)

SamerHachem,DivisionManagerSohnaNgum,Consultant(OfficeoftheSpecialEnvoyonGender)JessicaHarris,Consultant(OfficeoftheSpecialEnvoyonGender)

IndependentEvaluationDepartment(IED)AsianDevelopmentBank(AsDB)

FarzanaAhmed,LeadEvaluationSpecialistHyunSon,PrincipalEvaluationSpecialist

EvaluationDepartment(EvD)EuropeanBankforReconstructionandDevelopment(EBRD)

ShireenEl-Wahab,PrincipalEvaluationManagerBeatrizPerez-Timermans,Principal,EvaluationManager

OperationsEvaluation(EV)EuropeanInvestmentBank(EIB)

EmmanuelPondard,EvaluationSpecialist

IndependentEvaluationOffice(IEO)GlobalEnvironmentFacility(GEF)

GeetaBatra,Deputy-DirectorAnnaViggh,SeniorEvaluationOfficer

IndependentEvaluationOffice(IEO)IFAD

CatrinaPerch,EvaluationSpecialistMarkKeating,EvaluationOfficer

IndependentEvaluationOffice(IEO)UNWomen

ShravantiReddy,EvaluationSpecialistSabrinaEvangelista,EvaluationSpecialist

IndependentEvaluationGroup(IEG)WorldBankGroup(WBG)

GiselaGarcia,EvaluationOfficerElenaBardasi,SeniorEconomist

ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes iii

TableofcontentsAcknowledgments..........................................................................................................................................................ii

Tableofcontents...........................................................................................................................................................iii

Appendix1:Therationaleforintegratinggenderintoprojectandprogramevaluations...................4

Appendix2:ThespecialchallengesofgenderevaluationwithintheIEOcontext...................................6

Appendix3:OverviewofGREEvaluationDesigns..............................................................................................81.ThemainevaluationdesignsthatcanbeusedforstandardGRE.................................................................................................................82.Morein-depthmethodsforspecialGREevaluations.......................................................................................................................................10

Appendix4:Toolsforcollectingdataforgenderevaluations......................................................................121.Mixedandmulti-methodapproaches.....................................................................................................................................................................122.Secondarydatasources................................................................................................................................................................................................133.Theorybasedevaluation(TheoryofChange).....................................................................................................................................................134.Reconstructingbaselinedata.....................................................................................................................................................................................135.Surveys.................................................................................................................................................................................................................................146.Qualitativemethods.......................................................................................................................................................................................................147.Broaderapplicationsofqualitativemethodsusedinspecial,in-depthGREs.......................................................................................15

Appendix5:AdatacollectionplanningmatrixforaGRE:ApplicationtoahypotheticalVillageDevelopmentProjectinCentralAsia.....................................................................................................................21

Appendix6:ExampleofaGREdesignmatrix:EvaluatingahypotheticalvillagedevelopmentprojectwithdefinedgenderobjectivesinCentralAsia...................................................................................23

1.Theproject’sgenderobjectives.................................................................................................................................................................................233.Theproject’sgenderresponsiveinterventions..................................................................................................................................................244.Definitionofthegenderquestionstobeaddressed.........................................................................................................................................255.Initialproposalfortheevaluationmethodology...............................................................................................................................................26

Appendix7:ExampleofaGREdesignmatrix:EvaluatingahypotheticalpublictransportprojectinEurasiawithdefinedgenderobjectives................................................................................................................33

1.Frameworkoftheevaluationreport.......................................................................................................................................................................332.Theproject’sgenderobjectives.................................................................................................................................................................................333.Theproject’sgenderresponsiveinterventions..................................................................................................................................................344.Definitionofthegenderquestionstobeaddressedintheevaluation.....................................................................................................365.Initialproposalforthegenderevaluationmethodology...............................................................................................................................38

Appendix8:Linkstothegenderindexes.............................................................................................................48

Appendix9:Strengtheningthedisseminationanduseofgender-evaluationfindings.......................491.Theunderutilizationofevaluations........................................................................................................................................................................492.StrategiesforpromotingtheutilizationofGREevaluations........................................................................................................................494.Buildinggenderindicatorsandfindingsintokeyagencyreports.............................................................................................................50

ECGReferenceDocument–IntegratingGenderintoProject-levelEvaluation–Appendixes

4

4

Appendix 1: The rationale for integrating gender into project and program evaluations Thefollowingparagraphs(adaptedfromBamberger,2013)presentarationalethatcanbeadaptedbydifferentagenciestostresstheimportanceofincorporatingagenderfocusintoagencyevaluationprograms.Inthiscase,theexamplesaretakenfromcountry-levelimpactsbutsectorexamplescanalsobeincluded.ForamorecomprehensivereviewseetheOverviewofthe2012WorldDevelopmentReportonGenderandDevelopment.

WhyisitimportanttoincorporategenderintoM&E?

Ineverysociety,therearerulesgoverningappropriatebehaviorformenandwomenandgirlsandboys,inthehome,thecommunity,thelabormarket,schools,andinpolitics.Someoftheserulesareregulatedbysocialcustoms,othersbylawsortheoperationofthelabormarket.Sometimestheformsofcontrolaresubtle,whileothersmaybeenforcedbylegalsanctionsorthethreatofviolence.Whilesomesectorsofsocietymaybelievetheserulestobebasedon“natural”differencesbetweenmenandwomen,therulesare,infact,sociallyconstructedandvaryfromonesocietytoanotherandovertime.However,despitedifferencesacrosssocieties,ineverycountrythathasbeenstudied,theserulesplacewomenatadisadvantagewithrespecttokeydimensionsofdevelopment.

Thepersistenceofsignificantgenderinequalitiesinallregionsnegatesfundamentalhumanrightsandtheexpansionofhumanfreedoms.Inaddition,genderinequalitiesareseriousbarrierstotheachievementofdevelopmentobjectives(BoxA1-1).

InGenderEqualityandDevelopment(WorldBank,2012),itisarguedthatpromotinggenderequitycanmakeamajorcontributiontodevelopment:first,byfullyutilizingthecapacitiesofbothwomenandmen;second,throughimproveddevelopmentoutcomesforthenextgeneration;andthird,bymakinginstitutionsmorerepresentative.Genderequitywillopenthedoorstomorepolicychoicesandinstitutionswillbecomemorerepresentative.


5

5

Inlightofthiscompellingevidence,manygovernmentsandinternationaldevelopmentagencieshaveprioritizedgenderequalityasoneoftheirtopdevelopmentobjectives.Achievinggenderequalityrequiresintegratinggenderintoallaspectsofprogramming,budgeting,implementation,monitoring,andevaluation.Manyorganizationshaveoperationalizedtheirgenderequalitystrategiesthroughgenderactionplans(GAP)thatstressthecriticalroleofappropriategendertoolsfordatacollectionandanalysisformonitoringandevaluation(M&E).WhileaGAPprovidesausefulframeworkforanintegratedapproachtogenderequality,itisnotessentialandmanyagenciesbeginbybuildinggenderintotheirexistingM&Esystems.TheymaythendevelopabroadergenderframeworkaftergainingexperiencewithgenderM&E.

Source:M.Bamberger(2013),EngenderingM&EPREMSeriesonNutsandBoltsofM&Esystems.NoteNo.27.WorldBank.

BoxA1-1:EstimatedEconomicCostsofGenderInequality:SomeexamplesfromAfrica,theMiddleEastandAsia.

• IntheMiddleEastandNorthAfrica,ifwomen’slaborforceparticipationhadincreasedinthe1990satthesamerateaswomen’seducation,theaveragehouseholdincomewouldhavebeen25percenthigher.

• Tanzaniacouldincreasegrowthbyonepercentbyremovingbarrierstowomenentrepreneurs.• IfIndiaincreaseditsratiooffemaletomaleworkersbytenpercent,grossdomesticproduct

wouldincreasebyeightpercent.• TotalagriculturaloutputinSub-SaharanAfricacouldincreasebysixto20percentifwomen’s

accesstoagriculturalinputswereequaltomen’s.• AsiaislosingbetweenUS$42billionandUS$47billionperyearduetowomen’slimitedaccessto

employmentopportunities.• AsiaislosingbetweenUS$16billionandUS$30billionperyearasaresultofgirls’limitedaccess

toeducation.

Source:DFID(2008);UNESCAP(2007).


6

6

Appendix 2: The special challenges of gender evaluation within the IEO context [ThisisanexpandedversionofthediscussioninPartI-ChapterOne]

ThechallengesandopportunitiesresultingfromtheIEOmandate

TheReferenceDocumentrecognizesthatIEOsoperatewithinamandatethatdefinesthescopeoftheirevaluations,thetime-frameswithinwhichtheyoperateandtheevaluationmethodologiesthattheyuse.Giventheseparameters,theIEOapproachtoevaluationingeneral,andGREinparticular,isdifferentfromtheevaluationscenariosandmethodologiesdiscussedinmostevaluationtextbooks.IEOswereestablishedtoensuretheirindependencesothattheycanprovideanobjectiveassessmenttotheBoardofDirectorsandotherkeystakeholdersontheextenttowhichprojectsandotherinterventionshaveachievedtheirobjectives,whethertheyhaveusedtheirresourcesinthemostefficientway,andwhethertheyhavecompliedwiththeadministrativeproceduresandcodesofconductdefinedbytheirrespectiveagencies.

ThismandateprovidesbothuniqueopportunitiesandchallengesforconductingGREs.OpportunitiesarisebecauseIEOsreportdirectlytotheBoardofDirectors,meaningthereareinstitutionallydefinedmechanismsforthedisseminationanduseoftheevaluation,andprocedurestoensuretheobjectivityandindependenceoftheevaluations.ChallengesarisebecauseIEOevaluationsareconductedex-postafterprojectshaveclosed,sotypicallytheycannotinfluencethekindsofbaselineorimplementationdatacollectedontheprojectsbeingevaluated.Consequently,itisnotalwayspossibletoapplymostofthepre-test:post-testexperimentalandquasi-experimentalevaluationdesignsadvocatedbyevaluationtextbooks.ThesechallengesareparticularlyimportantforGREwheretheprocessesofwomen’sempowermentareregulatedthroughsocialmechanismsandprocessesofbehavioralchange,whichideallyshouldbeobservedovertimeratherthanassessedthroughrecallattheendoftheprocess.However,thisreferencedocumentreferstosomeofthenewinformationtechnologies(smartphonesandbigdata)thatofferthepossibilitytoreconstructbaselineandlongitudinaldatafromexistingdatasets,suchastwitterandothersocialmedia,satellitetimeseries,phonerecordsandelectronicfinancialtransactiondatasuchasATMs.Thesesourcesopen-upthepossibilitytobroadentherangeofevaluationmethodologies.

Specialchallengesfacinggenderevaluation

Inadditiontochallengesthatalldevelopmentevaluationsface,GREsfaceanumberoftheirownspecialchallenges:

a. Relevance:Manyagencystaffarenotconvincedthatgenderissuesarerelevantinallsectors.Forexample,somestaffworkingininfrastructure,financeortrademayarguethattheirsectorsare“genderneutral”andthatbothmenandwomenhavethesameneedsandwillbenefit(orbeaffectednegatively)equally.

b. Costandtime:GREfrequentlyinvolveadditionalcostsasmoredatahastobecollected,anddatamaybemoreexpensiveortime-consumingtocollect.Giventhedemandsonevaluationresources,theseconsiderationscanbeaseriousconstraint.

c. Dataisnotavailable:Whereanagencyhasnotcollectgenderdata,thislackofdataisgivenasthereasonfornotincludinggenderinanevaluation,creatinga‘chicken-and-egg’situation.Asgenderissuescannotbeanalyzed,itissometimesarguedthatwedonotknowifgenderissuesarerelevant–sodataisnotcollected.GREdatacanbemoredifficulttocollect,soitmaybehardtomakethecaseforitscollectionwithoutevidenceastoitsrelevance.

d. GRErequirestheuseofnewandunfamiliarmethodologies:Someresearcherswhohaveestablishedtheirprofessionalreputationsbyconductingcertainkindsofevaluation(e.g.,


7

7

collectionandanalysisofconventionalquantitativedata)maybereluctanttoincorporatenewkindsofdataandmethodswithwhichtheyarenotfamiliar.Forsomeresearchers,GREcanalsoseemthreateningasitmayquestionthevalidityofsomeofthetraditionalevaluationmethodsi.Somestaff,particularlythosetrainedineconomicsandquantitativemethods,maynotconsiderthemorequalitativelyorientedGREasbeing“professional”evaluations.

e. Theremaybeconcernsabouttheperceivedpoliticalideologyoffeministresearchers.Althoughthiswillfrequentlynotbethecase,insomecountriesfeministsareperceivedashavingapoliticalagendathatmaybedisruptiveorthatmaydivertevaluationsawayfromwhatareperceivedtobetheirintendedpurposefortheagency.


8

8

Appendix 3: Overview of GRE Evaluation Designs

1.EvaluationdesignsusedforstandardGREsThefollowingarethemostcommonevaluationdesignsusedforstandardGREs:

A.Descriptivepost-projectfieldvisits.ThisisprobablythemostwidelyusedGRE.DeskresearchandpersonalorphoneinterviewswithkeyinformantsarecombinedwithcountryvisitsbyIEOstaffand/orconsultants.Rapidprojectvisitsandindividualorgroupinterviewswithcommunitygroups,arecombinedwithmeetingswithimplementingagenciesandlocalgovernmentofficials,civilsocietyandotherkeyinformantinterviews.Focusgroupdiscussions(FGDs)withcommunitygroups,civilsocietyorimplementingagenciesmayalsobeincluded.

B.Quasi-experimentaldesigns.Whentimeandresourcespermitasurveymaybeconducted.Thismayuseapre-test:post-testcomparisonwithasampleoftheprojectpopulationbutwherepossibleacomparisongroupshouldalsobeincluded.GiventhatIEOevaluationsareconductedretrospectively,baseline(pre-test)datawillnormallyusetechniqueforreconstructingbaselinedata.

C.Theorybasedevaluation.Atheorybasedevaluation,suchasatheoryofchange(TOC),canprovideausefulframeworkforthedesignandimplementationofanevaluation.Frequently,thestandardGREwillincludearelativelysimpleformofTOCwhilespecial,in-depthevaluationsmayuseamoreelaborateform.Consequently,notalloftheelementsdescribedherewillbeincorporatedintoeverystandardGRE.

ATOCcanbeparticularlyusefulforGREasitcanhelpidentifythemanysubtleanddifficult-to-measurefactorsthatcanconstrainsuccessfuloutcomesofgenderinterventions.Manyofthesefactorsarenotnormallyaddressedinconventionalevaluations,sotheTOCcanalsoserveasachecklisttoensurethatimportantquestionsandindicatorsarenotoverlooked.Ideally,aTOCforaGREshouldincludethefollowingelements:

a. Problemdiagnostic:Descriptionoftheproblembeingaddressedandsomeofitscauses.Wherepossiblethisshouldincludeahistoricalanalysistoidentifyhowpastexperienceswithinterventionsinthisareamayaffectattitudestothepresentproject.

b. ProjectIntervention:Identificationofthegender-responsivecomponentsandinterventions.c. Implementationprocesses:Theprocessesthroughwhichthedifferentcomponents/serviceswill

bedelivered.d. Outputs:Thegender-responsiveoutputsthateachcomponentisexpectedtoproduce.e. Outcomes:Theintendedgender-responsiveoutcomesandthecombinationofoutputsthatare

expectedtocontributetoeachoutcome.Thereisrarelyaone-to-onerelationshipbetweenasingleoutputandaparticularoutcome,soitisimportanttoidentifythedifferentoutputsandexternalfactorsthatcancontributetoeachone.

f. Impactsorgoals:Someagenciesbreak-downimpactsintoshort,mediumandlong-termwhileothersdistinguishbetweenimpacts(thatareclearlylinkedtotheproject)andbroaderdevelopmentgoalstowhichprojectimpactsareonlyonecontributingfactor.

Inadditiontotheaboveelements,thatareusuallypresentedinalinearfigurewitheachelementdirectlylinkedtothenextlevelup,thereareotherimportantelementsthatshouldbeaddressedintheGRE.

• Contextualfactors(e.g.economic,legal,organizational,political,cultural,climaticandhistoricalfactorsthatcanaffecthowtheprojectisdesigned,implementedandthegender-responsiveoutputs,outcomesitproduces.Thisshouldbelinkedtofactorsaffectingparticularoutcomeand


9

9

impacts.Thesefactorsarecloselylinkedtothechallengesofaddressingcomplexityintheevaluationdesign(Bamberger,VaessenandRaimondo,2016;FunnellandRogers,2011;Patton2011).

• TheTOCmustbefalsifiable.Atheorymustbetestable(otherwiseitisjustastatementofbelieforhope),whichmeansthatitmustbepossibletostatethatithasnotworkedorthatthereisnocredibleevidencetoshowthatitdidwork.ThisrequiresmorerigorinthearticulationoftheTOCthanisusuallythecase.RequirementsofarobustTOCinclude:

o Atimelineoverwhichoutputs,outcomesandimpactsaretobeachieved.AcriticismofmanyTOCapproachesisthattheydonotincludethistimeline,sothatifexpectedoutcomeshavenotbeenachieveddefendersoftheprojectcanalwaysarguethat“ourtheoryofchangeisvalid,itisjustthatmoretimeisneededtoproduceresults.”Ideally,thereshouldbearelativelylongtimelineforGREasmanyoutcomes,particularlythoseinvolvingbehavioralororganizationalchange,evolveslowlyovertime.Also,theprocessesoftransformativechangeareusuallynotlinear,andtheremaybebacklashandresistancewhichisrecognizedbyfeministresearchersinthesaying“twostepsforward–onestepback”.

o Clearlydefinedandmeasurableindicators.

o Asetofalternative(rival)hypothesesthatcouldexplainhowtheexpectedoutcomescouldhavebeenachieved.Soeveniftheexpectedoutcomesareachieved,itisnotpossibletoconcludethatthesechangeswereduetotheprojectinterventionunlesstherivalhypotheseshavebeentestedandfoundnottobecredible.

g. Emergence:AcriticismofmostTOCsisthattheyarestaticandimplicitlyassumethattheenvironmentinwhichtheprogramoperateswillnotchangeoverthelife-timeoftheproject.However,thisstaticassumptionisrarelytrue.Governmentschange,thelocalandnationaleconomychange,andothercomplementaryorcompetingprogramsarelaunched.Equallyimportantisthefactthatthenatureoftheprojectandhowitoperateswillalsochange.Realistevaluation(Pawson,2013)hasshownthatprojectschangeinresponsetointeractionswithaffectedcommunitiesandotherstakeholders.Servicesanddeliverystylesthatbeneficiarieslikewillcontinueandbestrengthened,whilethosethatpeopledonotlikewilloftenchangeorwilldieoutifno-oneusesthem.Furthermore,programswilloftenevolveinnewdirectionssothattherewilloftenbeimportantunanticipated(positiveandnegative)outcomes.TheTOCmusthavetheflexibletoadapttotheemergentenvironment.Emergencepresentsmanychallengesfortheevaluator.Inadditiontothemethodologicalchallenges,thereisthefactthatmanyprojectswillbeheldaccountableonthebasisofthe(almostalways)relativelystaticresultsframework.Whilethisframeworkcanrevisetheoriginalnumericaltargets,mostresultsframeworksdonothavetheflexibilitytoincorporatecompletelynewandunplannedoutcomesorimpacts,andimplementingagenciesarefrequentlynotassessedontheirflexibilitytoadapttochangingcircumstances.

D.Objectives-basedevaluation(results-basedmanagement).Manyagenciesincorporateintotheirprojectdesignsaresultsframeworkthatdefinesasetofoutput,andoutcomeindicatorsthatdefinetheintendedresults.Usuallytheseincludebaselinemeasures,initialintendedtargetstobeachievedoveragivenperiodoftimeandtheactualvaluesachievedii.Theresultsframeworkisagoodstartingpointfortheevaluation,bothbecauseitprovidesapreciseandcomprehensivedefinitionofintendedresults,andbecauseprojectsarerequiredtocollectinformationonallofthesefactors.

However,thepracticallimitationforGREisthatmostprojectshaveonlyafewobjectivesthataredisaggregatedbysex,andevenfewerspecificgenderobjectives.So,whiletheresultsframework


10

10

providesausefulstartingpoint,itwillalmostalwayshavetobecomplementedbyothersourcesofinformation.Insomecaseitmaybepossibletoreconstructapproximategenderobjectivesby,forexample,askingstaffhowimportanttheinclusionofwomenincommitteesandleadershippositionsorinaccesstotrainingandprojectbenefitswasintheprojectdesign?Importancecanberatedas“veryimportant”,“quiteimportant”,“notimportant”and“notconsidered”.ManyIEOsincludetheseratingsinthegenderflags(seeChapter2,Section2.4)thatareusedtorateprojectdesignandimplementationintermsofhowwellgenderissuesareaddressed.Women’sactualparticipationoraccesstobenefitsindifferentactivitiescanthenbeassessedintheevaluation.Ofcourse,incaseswheretheprojecthadgender-specificobjectives,amorepreciseassessmentcanbemade.

E.Casestudies.Casestudydesignsarebasedontheselectionofcases(households,communities,schoolsandsoforth)thatarebroadlyrepresentativeofthetotalprojectpopulation.Casescanbedescriptive(illustratingthedifferenttypes),illustrative(describingthedifferenttypologiesfoundinsurveysorotherpartsoftheevaluation)oranalytic(seekingtoidentifyorexplaindifferentbehavior,processesofchangeoroutcomes).Casestudiesareaverypowerfulformofevaluationastheycandigdeeperandhelpexplainthelivedexperiencesofdifferentsectorsofthetargetpopulation.Forgenderevaluation,casescanoftenuncoverandhelpunderstandsubtleprocessesthataredifficulttocaptureininterviews.Thelatersectiononspecial,in-depthevaluationswilldiscussthegrowinguseofQualitativeComparativeAnalysis(QCA)casestudies.RobertYin(2003,2004,2012)providesausefulintroductiontocasestudyresearch,whileByrneandRagin(2009)provideacomprehensivereviewofcasestudymethods.However,noneofthesetextsfocusdirectlyongender.

F.Qualitativemethods.AlmostallGREincorporatequalitative(QUAL)methodsintotheevaluationdesign.Thesemethodsincludeunstructuredandsemi-structuredindividualinterviews,groupinterviewsanddiscussions(includingfocusgroups),andobservation(participantandnon-participant).Therearealsoawiderangeofparticipatorygroupdiscussiontechniques(PRA)thatincludemanymethodsdesignedforgroupswithlowliteracy.Onesetofveryusefultechniquesusesocialmappingtohelpunderstandthesocial,economicandpowerstructureofacommunity,andanothersetoftechniquestraceshistoricaltimelinesandthemajoreventsinthehistoryofthecommunity.

2.Morein-depthmethodsforspecialGREevaluationsParticularprojectsorthemesaresometimesselectedformorein-depthevaluation.Thefollowingaresomeofthewiderrangeofevaluationtoolsthatitmaybepossibletouseforthesestudieswhenmoretimeandresourcesareavailableandwherethemandatemaybebroader.Asindicatedearlier,thereisnoclearlinebetweenstandardandin-depthevaluations.

Theorybasedevaluation.In-depthevaluationsareoftenabletousesomeofthemoreadvancedapproachestotheory-basedevaluation,including:

a. Contributionanalysis

Experimentaldesignsarerarelypossible,particularlyforretrospectiveevaluations.Contributionanalysis(Mayne,2011)recognizesthatitisrarelypossibletoassessthedirecteffectofaparticularinterventionasmostprojectsareimplementedincontextswherethereareotheragencies,otherprojects,newgovernmentpoliciesandexternalfactors,allofwhichcancontributetoobservedchangesintheareawhereaprojectisoperating.Consequently,thepurposeistodefineandassessthemostplausible“contributionstory”.Theanalysisusuallyinvolvessixsteps:

• Definehowtheprojectisintendedtocontributetoasetofoutcomes(thecauseandeffectrelationship).

• Layouttheprogramtheory(developthe“programstory”).• Gatherallofthesupportingevidence.• Assessandchallengethecontributionclaim(identifyandtestrivalhypotheses).


11

11

• Findadditionalevidence.• Strengthenthecontributionclaim.

WhilemanyTOCsonlydescribeandassesstheproject’stheoryofchange,contributionanalysisarguesthatitisessentialtoidentifyandtestplausiblerivalhypotheses.Italsostressestheimportanceofcontinuallyseekingnewevidencetosupportorchallengetheevolvingtheory.

b.Outcomeharvesting

Theapproachissimilartocontributionanalysis,butwhiletheformerisoftenimplementedatthestartofaprojectandcontinuesthroughoutitsimplementation,outcomeharvestingisusedattheendofaprojectandisconsequentlyveryrelevantforIEOevaluations(WilsonGrau,2012).Outcomeharvestingcollectsstoriesfromprojectbeneficiariesandotherstakeholdersonchangesthathaveoccurredoverthelifeoftheproject.Itisquitecommonforhundredsofstoriestobecollected.Theyareorganizedintogroupsandtheircredibility,andtheirlinkstotheproject,arethenassessedbytheresearchteam.Manyevaluatorsfindthisapproachusefulbecausethestories(potentialoutcomes)aregeneratedbythestakeholdersthemselvesandnotbytheresearchteam.Similarapproachestocontributionanalysisareusedtotestthecredibilityoftheprogramstory(theory).

c.Realistevaluation

Realistevaluation(Pawson,2013)believesthattheexperimentaldesignquestion“Didtheinterventionworkinthiscontext,withthispopulationandatthispointintime?”istoonarrowtobeofmuchpracticalvalue.Insteadrealistevaluationasks:“Whatworks,forwhom,inwhatrespects,towhatextent,inwhatcontexts,andhow?”Thesequestionsareaddressedbydefining“generativemechanisms”thatexplain“how”theoutcomeswerecausedandtheinfluenceofcontext.Thebasicmodelisdefinedas:C(context),M(mechanisms)O(outcomes).

Theuniquefeatureoftheapproachisthebeliefthatthe“reasoning”oftheactorsinresponsetotheresourcesoropportunitiesprovidedbytheprojectiswhatcausestheoutcomes.Consequently,theprocessofunderstandingthisreasoningandthefactorsthataffectitimpliesaverydifferentresearchapproachfromconventionalevaluationdesignsthatassumetheoutcomesarecausedbytheprojectinterventions.TheapproachispotentiallyveryvaluableforGREaswomen’sreasoninginresponsetoprojectopportunitiesisinfluencedbyawiderangeofsocial,economic,cultural,political,legalandhistoricalfactors–whichiswhyprojectoutcomescanbeverydifferentindifferentcommunitiesandcontexts.TheapproachisalsoconsistentwiththeGREapproachofconsideringwomen’sagency.

d.Broaderapplicationsofqualitativemethods.

Thenatureofmostqualitativemethodsisthattheyrelyonbuildingtrustwiththecommunitiesbeingstudied,andconsequentlytheirapplicationideallyrequiresaconsiderableinvestmentoftime.So,althoughqualitativemethodsarewidelyusedinstandardevaluations,timeconstraintslimittheirfullapplication.ThisisdetailedinAppendix4below.

C

M________O


12

12

Appendix 4: Tools for collecting data for gender evaluations ThisAppendixdescribesthemostwidelyuseddatacollectionmethodsthatcanbeusedfortheGREevaluationdesignsdescribedinChapter3andAppendix3.Manyofthedatacollectionmethodsareusedinseveraloftheevaluations,andTable4.1oftheGenderNoteliststhemethodsmostcommonlyusedforeachdesign.ManyofthedatacollectionmethodscaneitherbeusedinarelativelysimplewayforstandardGREorinmoredepthforspecialevaluations.Thefollowingdiscussiondoesnottrytodistinguishbetweenstandardandmorein-depthdatacollectionapplicationsasthereisnoclear-cutlinebetweenthetwo.

1.Mixedandmulti-methodapproachesAlmostallIEOevaluationsrequireanassessmentofbothquantitative(howmuch?howmany?whoisincludedandexcluded?)andqualitativedimensions(understandingtheimplementationprocesses,thelivedexperiencesofdifferentgroups,andmechanismsofsocialcontrol).Consequently,allGREshouldincorporateamixed/multi-methodapproachiii.QUANTandQUALmethodsareoftencombinedinasomewhatadhocway(e.g.conductingnot-very-wellselectedfocusgroupsattheendofasurvey,orcommissioningafewcasestudiesthatareconductedwithverylittlecoordinationwithaQUANTsurvey).However,mixedmethodevaluationshouldbeconsideredasanintegratedevaluationstrategythatcancombineQUANTandQUALapproachesatallstagesoftheevaluation(Bamberger,RughandMabry,2012;Bamberger,2016).

ThereareseveraladvantagesofamixedmethodapproachforGREevaluations.First,triangulation(comparingindependentestimatesofakeyindicator)canincreasethereliabilityandvalidityofevaluationfindings.Second,combiningdifferentmethodsmayincreasethevalidityoffindingswhenworkingunderbudgetconstraintsiv.Third,mixedmethodshelpdescribeandunderstandinteractionsamongdifferentorganizationsandactorsandtoobserveprocessesofbehavioralchangetocomplementandhelpinterprettheQUANTfindings.

Thereareseveralguidelinestokeepinmindwhenusingmixedmethods:

• Mixedmethodsofteninvolveprofessionalsfromdifferentsocialsciencedisciplineswhohavedifferentwaysofworking.Consequently,itisimportanttoallowmoretimeforteam-buildingandplanninginordertointegratethedifferentapproachesandtofullybenefitfromthewiderrangeoffindingsandanalyticalmethods.TeambuildingisparticularlyimportantforGREasgenderandfeministresearchersoftenbringapproachesthatareunfamiliartomanyQUANTresearchers,whichmayinitiallycreatesomeresistanceorquestioningastowhethersomeoftheQUALmethodsreallymeet“professional”researchstandards.

• Thereshouldbeclosecoordinationwithrespecttodesignandapplicationofalldatacollectionmethods.Thisisessentialtopermittriangulationsothatonemethodcanbeusedtovalidatedatacollectedusingadifferentmethod.Thisisonlypossibleifallinstrumentsarecollectingcomparabledata.Thesituationoftenariseswhereestimatesofhouseholdincomeobtainedfromasurveyareinconsistentwithestimatesobtainedfromin-depthQUALinterviews.Oftenthequestionsareaskedindifferentwayssothatitisnotpossibletodeterminewhetherthedifferencesareduetothewaythequestionwasaskedorwhetherinfactonemethodiscollectingmorereliableinformationv.

• Thetimingofthecollectionofdifferentkindsofdatamustbecoordinatedsothattheinitialfindingsfromonemethodcanbeusedtocorrectanyissueswithanothermethod.Oftendiscrepanciesareonlyfoundlateintheresearchwhenitistoolatetomakeanycorrections.


13

13

2.SecondarydatasourcesAlmostallevaluationsbeginwithareviewofavailablesecondarysources.Thesecaninclude:

a. Projectdocuments.Whilemanyoftheseareincludedinprojectfilesandeasilyaccessible,oftenprojectstaffhavetheirownfileswithadditionalusefulreports.Forolderprojects,someofthesedocumentsmaynothavebeendigitalizedandhardertolocate.

b. Reportsfromotheragencies.Again,itmaybeusefultocontactagencystafftorequestdocumentsfromtheirpersonalfiles.

c. Systematicreviewsprovideasynthesisofresearchandevaluationfindings,thatmeetcertainmethodologicalstandards,fromallstudiesconductedonaparticulartypeofintervention(e.g.off-gridruralelectrification,villagewatersupply).Theseprovideausefulreferencepointtodeterminethemaximumimpactsimilarinterventionshaveachieved.Itisimportanttobeawarethatsystematicreviewsoftenhavequiteselectivescreeningcriteria,suchasonlyincludingrandomizedcontroltrials,soinmanyareasofgenderresearchitmaybethecasethatthevastmajorityofstudieswereexcludedbecausetheyusedqualitativemethods.

3.Theorybasedevaluation(theoryofchange)Theorybasedevaluationsusedamixedmethodsapproachfordatacollectionastheydrawonallavailablesourcesofdata.Forexample,contributionanalysisseeksatvariouspointsintheanalysistoidentifyanynewsourcesofdatathatmayberelevanttomakingthe“projectstory”morecredible,oralternatively,seekingevidencethatcouldchallengethecredibilityoftheprojecttheory.Thiswillofteninvolveseekingoutsourcesofdatathatconventionalevaluationswouldnotnormallyuse.

Therearetwokindsoftheory-basedevaluationthathavetheirownsourcesofdataanddatacollectionmethods:

a. Outcomeharvesting(Wilson-GrauandBritt,2012):Outcomeharvestingcollectsstoriesfromprojectbeneficiariesandotherstakeholdersonchangesthathaveoccurredoverthelifeoftheproject.Itisquitecommonforhundredsofstoriestobecollected.Theyarethenorganizedintogroupsandtheircredibility;theirlinkstotheprojectarethenassessedbytheresearchteam.Manyevaluatorsfindthisapproachusefulbecausethestories(potentialoutcomes)aregeneratedbythestakeholdersthemselvesandnotbytheresearchteam.Similarapproachestocontributionanalysisareusedtotestthecredibilityoftheprogramstory(theory).

b. Realistevaluation(Pawson,2013)seeksinformationtoanswerthequestions:“Whatworks,forwhom,inwhatrespects,towhatextent,inwhatcontexts,andhow?”ThisrequiresacreativeuseofmixedmethodsoftenwithastrongrelianceonQUALmethodstounderstanddifficult-to-measureconcepts,suchasbehavioralchangeandhowmechanismsofsocialcontrolinfluencedecisionsandactionsofindividualsandgroups.Realistevaluationalsoexamineshowthecontextaffectsprogramoutcomesandconsequentlydrawsonsomeofthekindsofdatausedincomplexityevaluationandsystemsanalysis.Processanalysisisalsousedtounderstandhowthegenerativemechanismsoperatevi.Thesearethecontextualfactorsthatcausepeopletoreasoninacertainwayaboutaprojectandwhichresultsintheiractinginaspecificwayinresponsetotheproject.

4.ReconstructingbaselinedataAsalmostallIEOevaluationsareconductedretrospectivelyaftertheprojecthasbeencompleted,itisnotpossibletoconductabaselinestudytocollectdataforapre-test:post-testcomparison.However,thereareanumberofwaysthatbaselinedatacanbe“reconstructed”:


14

14

a. Usingdatacollectedbytheprojectforprojectselectionanddesign.Forsomeprojects,suchaslow-costhousingormicro-credit,applicantshavetocompletequestionnaireswhichmayincludequitedetailedsocio-economicdata(Bamberger,RughandMabry,2012,Chapter5).

b. Usesecondarysurveydata.c. Conductingretrospectivesurveyswhererespondentsareaskedtorecalltheirsituationatthe

timetheprojectbegan.d. Keyinformantinterviews.e. PRAandotherqualitativemethods.f. GISdataandsatelliteimages.

5.Surveys.Forsomelargerevaluations,itmaybepossibletoconductsurveys.Theseincludearangeofapproaches:short,rapidsurveys,unstructuredorsemi-structuredinterviewsorlarge-scalestructuredsamplesurveys.Experienceshowsthatiftheevaluatorshavelocalcounterpartsitisoftenpossibletoconductashortsurveyofseveralhundredhouseholdsinonetotwoweeksandatamodestcost.Sometimesinterviewscanbeconductedbylocalteachersorstudentnurses,orinsomecasesevenhigh-schoolstudents.

IntegratinggenderintostandardQUANTsurveyswilloftenrequirecarefulcoordination.Inmanycasescollectingtherequiredsex-disaggregateddatamaybemoredifficultthanitmightseem.Whenhouseholdinterviewsareonlyconductedwiththe“headofhousehold”,inmanyculturesthemajoritywillbemen.Oftenthemanmaynotknowabouttheactivitiesoffemalemembers(forexample,howmanyhourstheyspendeachdayonhouseholdchoresandcollectingwaterandfuel?)Studieshaveshownthatmenoftenunderestimatethetimeburdenofsuchactivitiesontheirwivesanddaughters.Inmanyculturesmenalsohavelittleinformationontheirchildren’seducationbutareunwillingtoacknowledgethisfact.Anumberofstudieshavealsofoundthatmendonotmentiontheseveralhourstheirwivesspendeachdaycollectingwaterorfuelasaproblemvii.Forallofthesereasonsitisimportanttoplanhowreliablesex-disaggregateddatacanbecollected.Insomecases,itmightrequireamaleandfemaleinterviewertoworktogetherasateam,oritmaymeanarrangingafollow-upinterviewwithoneormorefemalemembersofthehousehold.

Observationisanimportanttooltocomplementsurveys.Forexample,whenaskedwhomakesmajordecisionsonhouseholdpurchasesandchildren’seducation,thewifewilloftensayitisherhusband.However,oncetheevaluatorhasgainedtheconfidenceofthewifeandisinvitedintothehouse,shewilloftenobservethatinfactthewifeisactivelyinvolvedinthesedecisions.

6.Qualitativemethods.Thereareawiderangeofqualitativetechniquesthatevaluatorscandrawon.Manyofthesehavetheadvantageofbeingrelativelyeconomicalandhavetheflexibilitytoadapttolocalconditions.Theyarealsousefulfortheanalysisoftheprocessofprojectimplementation,relationshipsamongdifferentagencies,andforunderstandingprocessesofbehavioralchange.HowardWhiteusedtheterm“ethnographiceconomics”torefertothecollectionofinsightsonhouseholdorcommunitydynamicsfromconversationswithstaff,communitymembersandcasualobservationthatcanprovidebackgroundforunderstandingfactorsthatinfluencebehaviorandprojectoutcomesthattendtogetmissedbyconventionalsurveysandprojectvisitsviii.Someofthemostwidelyusedtechniquesinclude:

a. Keyinformantinterviews.b. Focusgroups.c. Observationchecklists(forexample,forstudyinghowwomenparticipateinmeetings

andotherprojectactivities).


15

15

d. In-depthinterviews.e. PRAandotherparticipatorygroupconsultationmethods.f. Diaries:participantsorprojectstaffareaskedtokeepdetailedrecordsoftheiractivities,

suchastimeuseduringatypicalday,useofdifferentsourcesoffuelandpower,expenditures.

g. Photographsandartefacts(suchasornamentsandreligiousrelics,furnitureandhouseholdpossessions,photographsandgraffiti.

h. Audioandvideorecordings.

7.Broaderapplicationsofqualitativemethodsusedinspecial,in-depthGREsThefollowingillustratesomeofthewaysthatapplicationofqualitativetoolscanbeimprovedwhentimeandresourcespermit.

Initialdiagnosticstudies

Forcommunitybasedprojects,itisveryusefultoassignaresearchertospendseveralweeksinthecommunityinordertohelpunderstandthedailylives,problemsandconcernsandattitudestotheproposedinterventions(PillowandMayo,2012;Clarke,2012;Salmen,1987).Thegoalistobeabletoobserveandtoexperiencecommunityliferatherthantoconductinterviews.Itcanalsobehelpfultounderstandthelanguageandconceptsusedtothinkaboutanddescribekeyconceptssuchaswellnessandsickness,povertyandvulnerability,publicagenciesandtheservicestheyprovide,andpeople’saspirationsandfears.Theresearcherwillalsoexplorelocalhistoryandhowthisaffectsattitudestotheproposedproject.

Itisimportanttonotethatinmanycasesthestudycanbeconductedbyalocalresearcher(withappropriateguidance)sothestudydoesnotnecessarilyhavetobeveryexpensive.

Keyinformantsandinformalpanelstudies

Theluxuryofmoretimemakesitpossibletobetterselectkeyinformantsandtobuildconfidencewiththem.Manyofthemostvaluableinformantsprovenottobethepeopletheevaluatormetduringthetypicalonetotwodayvisittoacommunity.

Iftheevaluatorisabletomakeseveralvisitstothecommunityoveraperiodoftime,itisoftenusefultodevelopanetworkofinformantswhocanbevisitedperiodicallytoreportontheactualprogressoftheprojectontheground,andwhatpeoplearesayingaboutit.Forexample,peopleoftenhavedifferentperceptionsandexpectationsthatwhattheprojectbelievestheyareexpecting.

Participatoryplanningandgroupconsultationtechniques

Participatorymethodscanbeusedbothtoinvolvecommunitiesintheplanning,design,analysisanddisseminationofsurveysorasaparticipatoryevaluationtooltoobtaincommunityperspectivesonpriorityissues,thesocialstructureofthecommunityandthechangingcontributionmadebytheprojectovertime.

TheWorldBankSocialObservatoryprojectinIndiahasmadeextensiveuseofparticipatorytechniquesintheirresearchprojects.Forexample,villagewomendesignedasurveyinstrumenttoidentifyfamilyneedsinpoorcommunitiesinIndia.ThesurveywasthenadministeredtoalmostonemillionpeopleandtheresultswereanalyzedincooperationbetweenthewomenandtheSocialObservatoryteam.Thewomenthendesignedandimplementedparticipatorydisseminationstrategies,usingpictures,marchesandtalks(WorldBank,SocialObservatoryblog).

Withrespecttoparticipatorygroupconsultationmethodsforevaluation,thereisawidearsenaloftoolsandtechniquesthatbaseplanningandevaluationstudiesonfeedbackobtainedthroughgroup


16

16

consultations.Participatoryruralappraisal(PRA)isonewidelyusedsetofapproaches(LykesandHerschberg,2012;Kumar,2002).Thesetechniqueselicitperceptionsofthecommunityanditssurroundings,powerstructures,constraintsonwomenandothervulnerablegroups,andahistoricalperspectiveontheevolutionofthecommunity.Whiletherearerisksofbias,ofcooptionoftheprocessbyasmallgroupofmorepowerfulpeople,orinterpretationsimposed(intentionallyorinadvertently)bytheresearchers,theseapproachesareaveryvaluabletool.However,theyrequireconsiderabletimefortheirproperuseinordertopreparethegroup,gainanunderstandingofthecontext,conductthemeetings(whichcanoftentakethreetofives)andworkwiththecommunitytointerpretthefindings.Thesetechniqueshavebeenusedextensivelyingenderresearch(Cornwall,2003,2008;Kumar,2002).

Story-tellingandsense-making

Inrecentyearstherehasbeenanincreasedinterestinstory-telling.Oftenbeneficiariesareaskedtotellastoryaboutsignificantrecenteventsinthecommunityorwhattheyhaveexperiencedorheardabouttheproject.Oftenthestoriesareanalyzedusing“Sense-makingsoftware”.Thisapproachcanprovideadifferentperspectivebycapturingandanalyzingtheperspectiveofthecommunityratherthanbyaskingthemtorespondtoquestionsdevelopedbytheevaluator.DevaultandGross(2012)provideanoverviewoffeministapproachestoqualitativeinterviewing,listeningandstory-telling.

Anotherapproachtostory-tellingistomakeaudioorvideorecordingswhichcanthenbeanalyzedusingsomeofthenewdataanalyticssoftware.

Case-basedmethods.

Casestudieshavealwaysplayedanimportantroleinprogramevaluation.Theyprovidevaluablewaystoexplainquantitativefindings(particularlyunexpectedfindings),andtoprovidein-depthinsightsintolivedexperiences,projectimplementationprocessesandbehavioralchange.Anumberofdifferentcasestudyapproaches,allofwhichareusefulforGREcanbeidentified:

a. Exploratoryanddescriptivecasestudies.b. Illustrativecasestudies:Theseareoftenusedasafollow-uptoaquantitativesurvey.For

example,surveyanalysismaycreateatypologyofoutcomesandcasescanbepreparedtoillustrateorexplaineachtype.

c. Analytical:thepastdecadehasseenanincreasinguseofQualitativeComparativeAnalysis(QCA).Theunitofanalysisisacasewhichcanbeaslargeasacountryorassmallasanindividual.Casescanalsobehouseholdsororganizations.Thismethodidentifiestheconfigurationoffactors(forexample,householdattributesorcommunitycharacteristics,suchasaccesstoinfrastructure)foreachcase,thatarepresentwhentheoutcomeispresent.QCAisconsideredausefulwaytoprovideanapproximateestimateofcausalitywhenexperimentaldesignsarenotpossible.QCAalsohastheadvantagethatitcanbeusedwithsmallsamples(50orlesscases),anditisalsousefulfortheanalysisofcomplexprogramsasitidentifiesconfigurations(combinations)offactorsthatareassociatedwithanoutcome,ratherthananalyzingasinglefactor(asisthecasewithexperimentaldesigns).ForanoverviewofQCAmethodologiesseeByrneandRagin(eds)2009.UNWomen(2014)illustrateshowQCAwasusedtoevaluatetheimpactsofUNWomen’scountrystrategiesforpromotingwomen’empowerment.

ExperimentalandQuasi-experimentaldesigns.ItisnormallynotpossibletoconductanexperimentaldesignforIEOevaluationsastheyareconductedretrospectively.Yetitmayoccasionallybepossibletousethefindingsofexperimentaldesignsconductedbyotherpartsoftheorganization.Forexample,theWorldBank’sAfricaGenderInnovationLabconductsRCTsandquasi-experimentaldesignstotestinnovativeapproachesforthedesignandimplementationofgenderprojects(AfricaGenderInnovationLab,2016).

However,itmaybepossibletouseaquasi-experimentaldesign(QED).WhilethesedonothavethestatisticalrigorofRCTswithrespecttointernaldesignvalidity,QEDsoftenprovideuseful


17

17

approximationsforcausalanalysis,andsomewouldarguethatawell-designedQEDmaybeabletoaddressexternalvalidityissuesbetterthananRCT.ThefollowingareexamplesofQEDsthatcouldbeconsideredforGREwhenresourcesandtimepermit:

a. Post-testcomparisondesign.Forexample,manyevaluationsofthegenderimpactofmicrocreditprogramsusecross-sectionalstudieswherewomenwhousedavillagebankarematchedwiththosewhodidnot,wherepossibleusingpropensityscorematching.Thechallengewiththesedesignsishowtoaddressinitialdifferencesbetweenthetwogroups(beforethevillagebankbegantooperate)thatmightexplainsomeofthedifferencesinoutcomeindicators(expenditureonfood,educationandotherhouseholdessentials};investmentinhousing;profitsgeneratedbysmallbusinesses;women’sroleinhouseholddecision-making.Forexample,thewomenwhotookoutloansmighthavehadmoresmallbusinessexperience,ortheymightcomefromfamiliesthatweremoresupportiveofwomenrunningasmallbusiness.Post-testevaluationsfinditdifficulttocontrolfortheseinitialdifferences.However,ifamixedmethodsdesignisusedtherearemanyqualitativetechniquesthatcanexplorepossibleinitialdifferencesandhowtheycouldaffectoutcomes.

b. Naturalexperimentsandpipelinedesigns.Sometimesaprojectorpolicyisintendedtoreachallofthepopulationofadistrict,provinceorthewholecountry.However,duetoadministrativeproblems,budgetcutsorproblemssuchasflooding,somesectorsofthetargetpopulationmaynotbereachedormaybesubjecttolongdelays.Inthesecases,theoutcomesforthegroupsthatreceivedtheintendedservicescanbecomparedwiththosethatdidnot,thusprovidinganapproximateestimateofprojectoutcomes.Whilethisapproachisuseful,andquitewidelyused,itmustbeinterpretedwithcareastheremaybesystematicdifferencesbetweenbeneficiariesandnon-beneficiaries.Forexample,administrativeproblemsmaybemorelikelytoaffectpoorerormoreremoteareas.

Asimilarlogiccanbeusedincaseswhereprojectsareimplementedinphasesoveraperiodoftime.Anexamplewouldbetheinstallationofwaterandsanitationinalargeurbanslum.Theprojectwillusuallybeimplementedinphases(startingatoneendofthecommunityandmovingtotheother)overaperiodofyears.Anotherexampleistheconstructionofaroadwhichagaincantakeseveralyears.ThesectorsofthecommunitythatwillnotreceivethewaterandsanitationuntilYear2or3canbeusedasacomparisongroupforthePhase1areasthatwillreceivetheservicesinYear1.Assumingthatthecharacteristicsofhouseholdsineachphaseoftheprojectaresimilar(whichisnotalwaysthecase),thenabaselinesurveycanbeconductedinbothareasatthestartofYear1andthenrepeatedattheendofYear1.Comparingchangesinthetwogroupscanprovideanapproximateestimateofprojectimpacts.Theanalysiscanberefinedtocomparethechangesforwomenandmen.

c. Reconstructingbaselinedata.Asdiscussedearlier,itmaybepossibletofindbaselinedatasothatapre-test:post-testcomparisondesigncanbeused.

Bamberger,RughandMabry(2012)Chapter11andAppendixFreview,withexamples,allofthemostcommonexperimentalandquasi-experimentaldesigns.

Systemsandcomplexityscience-basedapproaches(WilliamsandHummelbrunner,2011,reviewthemainsystemsanalysisapproaches).ThereareanumberofnewevaluationapproachesdevelopedbycomplexitysciencethatcouldbeappliedinGRE.Alloftheseidentifythemainstakeholderoractors,thelinkagesbetweenthem,howinformationflows,leadershippatternsandhowdecisionsaremade.Systemsmapsandmodelsidentifythelinkagesbetweenorganizationsthatfacilitateorconstraindesiredprocessesofchange.Whilesomeoftheapproaches,suchassociometricanalysisrequirethecollectionofconsiderableamountsofsurveydata,othertechniquesarevisualizationtoolstohelpconceptualizethenatureofthesystemwithinwhichaparticularprogramoperates.Manyoftheapproachescanbelinkedtoatheoryofchange.Systemsapproachesareparticularlyvaluableforgender


18

18

analysisastheyhelpunderstandthecomplexityofthesocialsystemwithinwhichaprojectinterventionisseekingtopromotesocialchange.Theseareusuallytoolstomakeclients(aswellasevaluatorsthemselves)awarethatoutcomesareinfluencedbymanymorefactorsthanareusuallytakenintoconsiderationinprogramdesignandevaluation.Thefollowingaresomeofthemostwidelyusedapproaches,allofwhicharedescribedwithexamplesinWilliamsandHummelbrunner(2011).

a. Systemsmapping:Thiscoversanumberofapproachesthatprovideavisualrepresentationofasystem.Systemsmappinghelpsidentifythedifferentpartsofasystemandthelinkagesbetweenthedifferentpartsthatarelikelytochange.Systemsmapscanalsoidentifypositiveandnegativelinkagesbetweendifferentpartsofthesystemandthestrengthofthelinkages.Theapproachiscloselylinkedtoatheoryofchangeandprovidesatoolforexamininginmoredetailthedifferentpartsofthesystemthataffectedtheintendedprocessesofchange.Asystemsmapcouldalsobeusedtomodelalloftheinstitutionsandprocessesthatcomprisethesystemofsocialcontrolthatconstrainsprocessesofwomen’sempowerment.

b. Socialnetworkanalysis.Theapproachisusefulformodellingstakeholderrelationshipsanddescribinghowinformationflowsthroughthesystemanddecisionsaremade.Theapproachisusuallybasedonsurveysorobservationsthatcalculateinteractionsbetweendifferentindividualsorgroupsandthestructureofpoweranddecision-makingwithinagrouporamongdifferentgroups.

c. Systemdynamics:Thisdealwithinterconnectednessanddynamicrelationshipsamongdifferentpartsofaprogramsystem.Theapproachdrawsonengineeringandmanagementandinvolvesdevelopingafiguretorepresentstocksandflowvariables.Theapproachcanbeusedinevaluationtocapturehowcomplexsystemsareaffectedby,andrespondto,developmentinterventions.Forexample,WilliamsandHummelbrunner(2011)usedSystemsDynamicstoevaluatetheeffectsofmicroloanprogramstargetingsexworkersinWestAfricaaspartofalargerprogramtocombatHIV/AIDS.Theanalysiswasabletoidentifyandhelpexplainviolentswingsinthepopularityofthemicroloanprogramwithsexworkers.

d. Criticalsystemsheuristics:Thisapproachfocusesonunderstandingthefactorsthatdeterminewhatisconsideredtoliewithinthesystembeingstudiedthatiswhatgetsevaluated?Thisisprincipallyanethicaldecisionbasedonavaluejudgment.ThisisimportantforGREbecauseinmanycasesgenderissuesarenotconsideredtoliewithinthe(project)systemsogenderisnotincludedintheevaluation.

Conceptmapping(KaneandTrochim,2007).Conceptmappingisatechniquethatusesinterviewswithstakeholdersorexpertstoobtainanapproximateestimateofprogrameffectiveness,outcomesorimpacts.Agroupofexpertsorstakeholdersareaskedtolistthecharacteristicsofasuccessfulprogram(e.g.topromotewomen’sempowerment).Thelisteditemsaresortedintogroups(manuallyorbycomputer)andtheseareorganizedintoasetofdimensions.Thesameordifferentgroupisthenaskedtorate(usuallyona1-5scale)actualprogramsonthesedimensions.Theratingscaneitherbeproducedatonepointintime(aswouldusuallybethecaseforanIEOevaluation,ortheratingscanbeproducedatthestartandendofaprojecttomeasurechange.Conceptmappinghasseveraladvantages.First,thedimensionsonwhichaprogramisevaluatedaredefinedbystakeholders(orexperts)andnotbytheevaluationteam.Thisisparticularlyimportantforevaluatingmulti-dimensional,anddifficulttodefine,conceptssuchasempowerment.Second,softwareisavailabletoconducttheconceptdevelopmentandevaluationonlinesoitcanbemuchmoreeconomicalandcaninvolveawiderrangeofstakeholdersorraters.Theprocesscanalsobecarriedoutrelativelyquickly.UsingnewinformationtechnologyforGRE(Bamberger,2017).Thepastfewyearshaveseenarapiddevelopmentofexcitingnewsourcesfordatacollectionandanalysisthatareopeningupnewapproachestoprogramevaluationthatitwouldhavebeendifficulttohaveimaginedeventenyearsago.ThesearebasedonICTs(smartphones,internetandotherportabledevicesthatcanbeusedinthefieldandbyeventhepooresthouseholdslivinginremoteregions),andonbigdata(generatedfromsatellite


19

19

images,fromtwitterandothersocialmedia,electronictransfers,remotesensorsandtheinternetofthings).Thesearecomplementedbypowerfulnewdataanalyticstoolsthatcananalyzevastquantitiesofdatafarbeyondthecapacityofofficecomputers.Mostoftheseinvolvenewkindsofdata,muchofitcollectedforacompletelydifferentpurpose(suchasanATMtransactionorasocialmediadiscussion),aredistinctfromthekindsofinformationcurrentlyusebyevaluators.Infact,evaluatorshavebeenmuchslowertoadoptnewinformationtechnologiesthanmarketresearchers,medicalresearchers,programplanners,socialmarketers,andagenciesworkinginemergencyrelief.

Despiteevaluatorshavingbeenmuchslowerontheuptake,therearenowexamplesfromallsectorsandregionsontheimmensepotentialofnewinformationtechnologyfordevelopmentevaluation.ManyofthenewtechnologiesareofparticularinteresttoIEOs,includingforGREs,astheycanovercomemanyoftheconstraintsdiscussedearlier.Someofthepotentialapplicationsandadvantagesinclude:

a. Muchoftheinformationisveryfastandcheaptocollectandanalyzeasitisderivedfromexistingdatasourcesandtheevaluationdoesnothavetopayforcollection.

b. Economicalaccessmeansthatevaluationsnolongerhavetorelyonrelativelysmallsamples,butdatacanoftencoverthetotalpopulation.Thismakesitpossibletoconductkindsofdatadisaggregationthatwerenotpreviouslypossible,asthesmallsamplesizedidnotprovidesufficientnumbersfordisaggregateddataanalysis.

c. Easeofdataaccessalsomakesitpossibletoincorporatemanycontextualfactorswhichwerepreviouslyinaccessibleortoextensivetouse.Thispermitsplacingprogramsintheirbroadercontexts,whichisveryimportantforunderstandingthemultiplefactorsthataffectorconstrainsocialchange.

d. Longitudinaldatasetsarestartingtobecomeavailablethatofferthepotentialtoovercomethemajorconstraintonretrospectiveevaluations,namelythelackofbaselinedata.Examplesoflongitudinaldatasetsinclude:ananalysisofchangingattitudesandinformationonmajorsocialissuesasreflectedontwitter(whichnowcoveranumberofyears);datastreamsfromsatelliteimagesthatcapturepopulationmovements,indicatorsofpovertyandeconomicgrowth(suchasvehiculartraffic,typesofhouseconstruction,nocturnallightemissionsinpoorcommunities,areasundercultivation).

e. Dataanalyticsmakesitpossibletoconstructanintegrateddataplatformthatbringstogethermanydifferentsourcesofdatausingacommonmetric.Thismakesitpossibletoidentifypatternsofrelationshipsbetweendifferentdatasetsthatwerepreviouslydifficulttodetect.

f. Real-timedatafeedbackmakesitpossibletodetectchangesandtrendseveninpost-projectdata

Portfolioanalysis.PortfolioanalysiswasdescribedinSection2.4B.Dataiscollectedfromthefollowingsources:

• Policy,planningandprojectdocumentsfromcountryprograms.Documentsarereviewedtodeterminewhetherandhowgenderissuesareaddressed.Sometimestheassessmentsimplyindicateswhetherthereisareferencetogender,butinothercasesthereisaratingofwhethergenderwasacentralpriorityorhowthoroughlyitwasaddressed.

• Keyinformantinterviewsmaybeconductedbyphone,e-mailorinpersontoobtainopinionsonhowgenderwasapproachedindifferentpartsoftheprogram.

• Focusgroupinterviewsmaybeusedinasmallnumberofcountriestodigdeeper.• Afewcountriesmaybeselectedtoconductfieldstudiesthatmayincludeprojectsurveys,site

visitsandinterviewswithdifferentstakeholdergroups.Asmallnumberofrepresentativeprojectsmayalsobeselectedformorein-depthanalysis.Forexample,theWorldBankassessmentoftheimplementationoftheirglobalgenderpolicy(WorldBank,2009)beganwiththeassessmentofhowwellgenderwasincorporatedintopoliciesandprojectsin93countrieswheretheBankhadactiveprograms.Ratingscaleswereusedand1,153projectswereanalyzed.Basedontheseratings,12countrieswereselectedformorein-depthanalysis(e.g.stakeholder


20

20

interviews),ofwhichthreewereselectedforfocusgroupinterviewsandtwoforintensivecountrystudies.

Genderflags.Manyagencieshavedevelopedchecklistsforassessingwhetherandhowwellgenderissueshavebeenaddressedincountryprogramframeworks,sectorprogramsorprojectsix.Theindicatorseitherusea“Yes/No”formatorratehowwelltheissuewasaddressed.Box2.3inChapter2illustratesthequestionsincludedintheWorldBankIEGGenderFlagcountryevaluationtemplate.Checklistscaneitherbeusedforself-assessmentbyoperationsstafforforexternalassessmentbyIEOsorconsultants.

Datacollectionforcasestudies.DescriptivecasestudiesmainlyuseQUALmethodstodescribethecontextinwhichtheprojectoperates.Illustrativecasestudies,ontheotherhandnormallyusedmixedmethodsdatacollectionassurveysareanalyzedtoidentifyatypologyofsubgroupsforexample,farmers,villagebankmembers,mothersusingachildcareclinic.CasesarethenselectedwithineachtypologyandarestudiedusingQUALmethods.

QualitativeComparativeAnalysis(QCA)casestudiesuseaspecializeddatacollectionmethod(Byrne,2009;ByrneandRagin,2009).Amatrixisconstructedwhereeachcaseisarowandeachcolumnrepresentseitheranattributeofthecase,orofthecontextinwhichitislocated,ortheoutcomevariable.InthesimplestformofQCA,allattributesandoutcomesaredichotomized.Forexample,QCAmightbeusedtoassesshowgirlsenrolmentinsecondaryschool(theoutcome)isaffectedby:(i)whetherthemotherhascompletedsecondaryschool;(ii)whetherthegirlliveslessthan1,000metresfromtheschool;and(iii)whethertheschoolhassatisfactorytoilets(asdefinedbytheMinistryofEducation)forgirls.Thecolumnswouldbedefinedasfollows:

• Column1:Casenumber.• Column2:Motherhascompletedsecondaryschool[Yes=1,No=0].• Column3:Girlliveslessthan1,000metersfromtheschool[Yes=1,No=0].• Column4:Theschoolhassatisfactory(asdefinedbytheMinistryofEducation)toiletforgirls

[Yes=1,No=0].• Column5[outcomevariable]:Thegirlisenrolledinsecondaryschool[Yes=1,No=0].

Thedatamaybecollectedinaspecialsurveyoritmaybeavailablefromprevioussurveysorrecords.Oftenthevariablesareselectedtoreducethetimeandcostofdatacollection.


21

21

Appendix 5: A data collection planning matrix for a GRE: Application to a hypothetical Village Development Project in Central Asia Note:Thisisahypotheticalexampletoillustratetheuseofthedatacollectionplanningmatrix.Thisexampleonlyillustratesafewofthepossibleindicatorsanddatacollectionmethodsthatcouldbeconsidered.ThismatrixcouldbeusedinthedesignofallthreecasestudiesdiscussedintheReferenceDocument.

• • Evaluationdesign • Indicators • Datasources• Genderquestion1:Didtheprojectaddresswomen’sneeds?• 1.Didtheprojectstaffbelieve

thatwomen’sneedswereaddressed?

• a.Deskreviewcombinedwithquantitativeorqualitativesurveys.

• b.Ifpossiblebaselinedatawillbereconstructedtopermitapre-test:post-testdesign.

• a.Didtheprojectstaffbelievewomen’sneedswereaddressed?

• b.Whichneedswereaddressed?

• Addressingbothquestions• i.Interviewswithprojectstaff.• ii.Monitoringandotherproject

reports.• 2.Didwomenbelievethat

theirneedswereaddressed?• a.Theproportionofwomenwhosay

projects:• Respondeddirectlytotheir

needs.• Respondedsomewhattotheir

needs.• Didnotrespondtotheirneeds.

• i.Samplesurveywithwomen.• ii.Focusgroups.

• Notesonthefeasibilityoftheproposeddatacollectionmethods:• 1.Reviewmonitoringandotherprojectreportstocheckthekindsofinformationincludedonprojectsperceivedtosatisfywomen’sneeds.• 2.Aretheoriginalprojectstaffstillavailabletobeinterviewed?• 3.Willtimeandresourcepermittheapplicationofasurvey?• 4.Whathasbeentheexperienceinterviewingwomen?Isthisfeasible?Dosurveysorfocusgroupstendtoworkbetterwithwomen?• Genderquestion2:Howdidtheprojectaffectwomen’sempowerment?• 1.Participationinproject-

relatedcommunityorganizations.

• a.Deskreviewscombinedwithquantitativeorqualitativesurveys.

• b.Reconstructingbaselinedataifpossible.

a.Numberofwomenandmenparticipatingindifferentprojectorganizationsforeachyearoftheproject.b.Regularityofattendance.

• a-i.Projectmonitoringreports.• a-ii.Interviewswithproject

staff.• a-iii.Projectprofiles.• • b-i.Monitoringreports.


22

22

• b-ii.Attendancerecords• b-iii.Visitstomeetingstocheck

records• 2.Participationindecision-

makingincommunityorganizations.

• a.Numberofwomenandmenonleadershipcommitteesinprojectorganizations.

• b.Levelofparticipationofwomencomparedtomenmeasuredby:

i. Frequencyofspeaking.ii. Numberofwomen’sideas

approved.

• a-i.Projectmonitoringreports.• b-i.Observationchecklistsused

incommitteemeetings.• b-ii.Audioandvideorecordings

(ifpermitted).

• 3.Impactsonwomen’sstatusinthehousehold• a.Mobility. • • Placesvisited:(adaptlisttocontext):

[1pointforeach]• a.Market.• b.Clinic.• c.Movies.• d.Outsidethevillage.

• Coveringallitems:• iRapidsamplesurveywith

women.• ii.Keyinformants.• iii.Focusgroups.

• b.Abilitytomakesmallpurchases.

• Thingspurchased(adaptlist)[1pointforeach]

• a.Itemsfordailyuse.• b.Itemsforself.• c.Treatsforchildren.

• Coveringallitems.• i.Observationduringvisitsto

families.• ii.Rapidsamplesurveywith

women.• iii.Requestingagroupof

womentocompleteanexpenditurediary.

• c.Involvementinmajorhouseholddecisions.

• (Adaptlist)[pointstobedecided].Maygivedifferentscoresfordecisionsmadeonownormadewithhusband.

• a.Purchaseofgoatorsmallanimal• b.Householdrepair.• c.Leasingland.• d.Purchaseoflandorequipmentfor

farmorbusiness.

•

• Notesonthefeasibilityofdatacollectionmethods:• 1-4Asforquestion1.• 5.Isthereanyexperiencewiththeapplicationofmultiplechoicequestions?Howwelldotheywork?


23

23

Appendix 6: Example of a GRE design matrix: Evaluating a hypothetical village development project with defined gender objectives in Central Asia.

1.Theproject’sgenderobjectivesItwillbenecessarytoconfirmwhatthestatedgenderobjectivesoftheprojectare,andwhetherthereareadditionalgenderobjectivesthatareimplicitbutnotspecificallystatedwhichcouldbeincludedintheassessment.BoxA6-1identifiesthegenderobjectivesthatmightbeincluded.Intheboxthesearedividedintooutputsandoutcomes(followingthecategoriesusedinmostTOCs)butthedistinctionbetweenthetwocategoriesisnotveryclear.Forexample,participationinagroupisconsideredanoutput,whereasassumingapositionofleadershipisconsideredanoutcome.Theprimaryoutcomesareconsideredastheproject’sdefinedgenderobjectives.Whilesomeoftheseareclearlystatedintheprojectdocument,othersareinferredaspossiblegenderoutcomesthatmightbeproducedandwhichgobeyondthedefinedprojectobjectives.Thesecouldbeclassifiedasprimary(defined)genderobjectivesandsecondaryoutcomes(otherpotentialeffectsonwomenintheproject)andtertiarygendereffects(onwomeninothercommunities).So,inthedesignoftheprojectevaluationitwillbeimportanttomakethisdistinctionandalsotoagreewithmanagementwhichsecondaryoutcomesshouldbeassessed.Itwillbeimportantforpoliticalreasonstoclarifythatmanagersarenotbeingassessedonwhethersecondaryandtertiarygenderoutcomesareachieved,butthatthesearebeingreviewedtobetterunderstandthebroaderpotentialoutcomesthatfutureCommunityDrivenDevelopmentprojectscan(andcannot)beexpectedtoachieve.


24

24

3.Theproject’sgenderresponsiveinterventionsTheprojecthasanumberofcomponents/interventionsthatarespecificallyintendedtopromotetheequalparticipationofwomenandtopromotetheireconomicorsocialwell-being,andanumberofothersthathavethepotentialtoadvancethestatusofwomen(seeBoxA6-2).

BoxA6-1:Possiblegenderobjectivestobeincludedintheassessment

Outputs

a. Women’sactiveparticipationinthedifferentprojectcommittees.b. Women’sactiveparticipationintheprivategroupenterprises.c. Women’sequalaccesstoprojectservicesandbenefits.

Outcomes

Primarygenderoutcomes(definedintheprojectdesign)

a. Organizationandleadership• Womenassumeleadershippositionsindifferentcommitteesandgroups.• Involvementindecisionsontheselectionofprojects.• Infrastructureprojectsareselectedthatdirectlybenefitwomen.

b. Accessto,andcontrolofeconomicandproductiveresources• Involvementindecisionsoncontrolofprojectresources.

c. Equalaccessto,andcontrolof,resourcesforprivategroupenterprises.Secondaryoutcomes(effectsonwomenintheprojectthatarenotidentifiedintheprojectdesign)andtertiaryoutcomes(effectsonwomeninothercommunities)

a. Women’spositionstrengthenedwithinthehousehold• Women’srolestrengthenedincontrolofhouseholdresources.• Womenenjoygreaterindependence(forexamplegeographicalmobility).

b. Economic• Increasedemploymentopportunities.• Increasedincomeandearningsopportunities.

c. Social• Increasedaccesstoeducationforwomenandgirls.• Greaterpersonalsecurity(forexample,reduceddomesticviolenceandsexual

harassmentoutsidethehome).


25

25

4.DefinitionofthegenderquestionstobeaddressedBoxA6-3identifiesaninitiallistofgender-relatedquestionsrelatingtofivedimensions:

a. Historyandcontextandhowtheseaffecttheimplementationandlikelygenderoutcomesoftheproject.

b. Relevanceoftheprojectdesigntotheachievementofimportantgenderobjectives.c. Efficiencyofprojectimplementationwithrespecttotheachievementofgenderobjectives.d. Achievementofgenderobjectives(efficacy).Objectivesaredefinedintermsofoutputsand

outcomes.e. Sustainabilityandresilience.WhiletheAppraisalReportdiscussessustainability,itdoesnot

discussresilience–whichisaconceptthathasbeenintroducedsincethisprojectwaslaunched,butwhichisnowfrequentlydiscussedtogetherwithsustainability.

BoxA6-2:Projectcomponentsandinterventionsdesignedtopromotegenderequality,andtheeconomicandsocialwell-beingofwomen

1. Socialinclusionstrategya. Socialmobilization:

i. Womenaredefinedasoneoftheprioritygroupstoreceivetrainingandotherkindsofsupporttoensuretheirparticipationinprojectactivities.

ii. Annualmeetingswillbeheldwithallpartneragenciestoreviewprogressinachievingtheparticipationofwomenandothertargetgroups.

iii. Effortswillbemadetorecruitwomenassocialmobilizers.b. Villageprofiles:

i. Willfocusonissuesofexclusion,includingexclusionofwomenandwillexaminesocial,political,legal,economicandotherfactorsthatcauseorsustainexclusion.

ii. Asocialcapitalanalysiswillexaminedifferencesinsocialcapitalforwomenandmen.

c. Villagemeetings:i. Itisrequiredthatatleasttwoofthesixcommitteemembersshouldbe

women.ii. Aseriesofstructuredmeetingsareplannedandtheparticipationof

womenandmencanbemonitored.d. Preparationofgroupsandstatementsofneeds:effortswillbemadetoensurethat

womenmemberswillhavetheopportunitytostatetheirneeds.2. Identificationofprioritysocialinfrastructure

a. Itwillbeimportanttomonitor/assesswhetherwomen’sprioritiesarereflectedintheselectionofinfrastructure[note:thePADdoesnotstatewhetherthisisapriorityandhowitwillbeachieved].


26

26

5.Initialproposalfortheevaluationmethodology5.1. TableA6-1presentsinitialideasfortheevaluationdesign.Thisonlyprovidesaframeworkthat

willneedtobeelaboratedonceagreementhasbeenreachonthegeneralapproach.Thetableincludesthreecolumns:

• ThequestionscoveringeachofthefivedimensionsoftheevaluationthatarelistedinBox4.• Theindicatorsusedtoaddresseachquestion.• Thedatacollectionmethods.

BoxA6-3:Gender-relatedquestionstobeaddressed

1.Historyandcontext

a. Howwaswomen’sparticipationandaccesstobenefitsaffectedbythetransitionfromtheSovietera?

b. Howdidhighunemploymentaffectwomen’sparticipationandaccesstoprogrambenefits?2.Relevance

a. Howdoestheprogramexpecttostrengthenwomen’sempowermentandensurewomenhaveaccesstoprogrambenefits?

b. Aretheinterventionsrelevanttotheachievementoftheseobjectives?c. Howrelevantwereinclusion,socialmobilizationandprovisionofinfrastructureto

achievingobjectives?d. Howrelevantwassmallscaleprivategroupenterprises?e. Howrelevantarethegenderobjectivesandtheirimplementationsstrategiestothe

achievementofoverallprojectobjectives?3.Efficiency[notallagenciesincludeefficiencyintheevaluation]

a. Werewomenconsultedandinvolvedinprojectdesign?b. Didtheselectedprojectsreflecttheprioritiesofwomen?c. Wereprojectimplementationstrategiesconducivetotheparticipationofwomen?d. Howdidwomen’sparticipationindesignandimplementationaffectoverallproject

outcomes?e. Theeffectivenessofthesocialmobilizationstrategy.

4.Achievementofgenderobjectives(efficacy)a. Outputs

i. Women’sparticipationinprojectleadershipandmanagementii. Women’sroleindecision-making

b. PrimaryOutcomesi. Projecteffectsonwomen’sempowerment.ii. Assessingsocialandeconomicoutcomesforwomen.iii. Didprojectsrespondtowomen’sneedsandpriorities?

c.Secondaryandtertiaryoutcomesi. Weretherepositiveand/ornegativeunintendedprojectoutcomesforwomen?ii. Whateffectsdidtheprojecthaveonwomen’empowerment?

5.Sustainabilityandresiliencea. Whatevidenceistherethatthedifferentgenderoutcomeswillbesustainable?b. Doindividualwomen,andwomen’sgroupsdevelopstrategiestoadapttostressandshocks

anddotheylearnfromexperienceandimprovehowtheyadapt?


27

27

Theframeworkidentifiesthefollowingpossibledatacollectionmethods.Theapplicationofeachofthesedatacollectiontoolswillbeexplainedoncethereisaninitialagreementontherangeoffeasibleandappropriatemethodsforthisevaluation.Itisrecognizedthatnotallofthelistedmethodswillbeusedinthisevaluation.Theinitiallistofdatacollectionandanalyticalmethodsincludes:1. ConstructingagenderTOCthatcanhelpidentifykeyevaluationquestions,indicatorsandthe

pathwaysthroughwhichitisexpectedthatgenderoutcomeswillbeachieved.2. Constructingadatacollectionplanningmatrix.Appendix5presentsanexampleofadatacollection

planningmatrixthatcouldbeusedinthedesignofthepresentevaluation.Thematrixlistsallofthekeygender(andother)questions,theproposedevaluationdesigns,theindicatorsandthedatacollectionmethods.Importantly,italsoassessesthefeasibilityofcollectingtheproposedinformationwithinthebudget,time,methodologicalandorganizationalconstraintswithinwhichtheevaluationwillbeconducted.

3. Secondarysources:a. TheAppraisalReport.b. TheCompletionReport.c. Projectmonitoringreports.d. Projectprofiles.e. Governmentreports.f. Partnerreports.g. Civilsocietyreports.

4. Consultations:a. Expertandkeyinformants.b. Focusgroups.

5. Surveysandquestionnaires:a. Rapid,shortsamplesurvey.b. Ratingscalestobecompletedbybeneficiaries,projectstafforpartners.

6. Casestudiesa. Descriptivecasestudies.b. Analyticalcasestudies(QCA).

7. Qualitativefieldwork:a. Projectvisitsb. Informalinterviews.c. In-depthinterviews.d. Observation.

8. Socialmediaanalysis(ifthisisfeasible):a. Socialmediaanalytics(forexample,analysisoftwitterandothersocialmedia).

9. Pipelinedesignstoconstructacounterfactual.5.2.Evaluatingprojectoutcomespromotingwomen’sempowerment.Theprojectreferstosocialmobilizationasamechanismforstrengtheningtheparticipationofwomenincommunityorganizationspromotedbytheprojectandinthemanagementoftheseorganizations.Italsopromotestheparticipationofwomenintheprivategroupenterprises.Althoughtheprojectdocumentsdonotreferdirectlytogenderempowerment,theseobjectivescouldbeconsideredimportantelementsinwhatmanyagencieswouldcalltheeconomicandsocialempowermentofwomen.Chapter8oftheReferenceDocumentdiscusseshowtheseobjectivescouldbeincorporatedintoagenderempowermentframeworkandillustrateshowanempowermentframeworkcouldbedevelopmenttoassesstheeffectsoftheprojectonpromotingthedifferentdimensionsofwomen’sempowerment.


28

28

• TableA6-1:Keygender-responsivequestions,indicatorsanddatacollectionmethods:

PARTI:SOMEGENERALAPPROACHESTHATCANBEUSEDTHRUGHOUTTHEEVALUATION

1.Gendertheoryofchange:Developingagendertheoryofchangethatisusedtoidentifythekeyevaluationquestionsandtodefinetheprocessesthroughwhichoutputsandoutcomesaretobeachieved.Thisalsodefineskeyassumptionstobetested.

• 2.Resultsframework(ifithasbeenusedintheproject):Thisdefinestheintendedgenderobjectivestobeassessed.

• 3.Descriptivecasestudies:Usuallyarelativelysmallnumberofcasestudiesthatarebroadlyrepresentativeofthemainprojectscenariosandwhichareusedtoillustratehowtheprojectsevolvedandthelivedexperienceoftheprojectpopulations.Casestudiescanbelongitudinal(conductedoverarelativelylongperiodoftime),or(asisusuallythecasewithex-postevaluations)conductedatonepointintime,andrelyingextensivelyonrecall.

• 4.Analyticalcasestudies:TheseusetechniquessuchasQCA(qualitativecaseanalysis)toidentifythenecessaryandsufficientconditionsforprojectoutcomestooccur,andalsothenecessaryandsufficientconditionswhereprojectoutcomesdoNOToccur.Normallyatleast30casesarerequired.WhileQCAarerarely(ifever)usedbyIEG,theycouldbeapotentiallypowerfulanalyticaltoolthatpermitsattributionanalysistobeused.

• 5.Counterfactualanalysis:Thiscomparesprovincesorregionswheretheprojecthasbeenimplementedwithregionswheretheprojecthasyetbeenimplemented.Manyprojectsarerolled-outinphasessothatapipelinedesigncanbeusedwhereregionswheretheprojecthasnotyetbeenimplementedarecomparedwithareaswhereithasbeenimplemented.Sometimesthedesignisbasedonplannedphasing-inofdifferentregionswhileinmanycasesitisbasedonnaturalexperimentswheredelaysarecausedbyunplannedcircumstances,suchasdelaysinfundingoradministrativeproblems.Whiletherearemethodologicallimitationsintheuseofpipelinedesigns,theyofferausefultoolforassessingattributionwhereexperimentalandquasi-experimentaldesignsarenotfeasible.

•

• PARTII:DESIGNSFORASSESSINGEACHDIMENSIONOFTHEEVALUATION

•

Dimension/Question Indicators Datacollectionmethods

• 1.HISTORYANDCONTEXT

1. Howwaswomen’sparticipationandaccesstobenefitsaffectedbythetransitionfromtheSovietera?

• Opinionsfromexpertsandkeyinformantsontheeffectsofhighunemployment.

a. Expertandkeyinformantinterviews.

2. Howdidhighunemploymentaffectwomen’seffectiveparticipation?

• Opinionsfromexpertsandkeyinformantsontheeffectsofhighunemployment.

a. ExpertandKeyinformantinterviews.b. AppraisalReportandICR.

c. • 2.RELEVANCE:Therelevanceoftheprojectconceptanddesignforpromotingwomen’sempowermentandsocialandeconomicbenefits

1. Howdoestheprogramexpectto a. DevelopaTheoryofChange(TOC)thatidentifiesthe a. ThedraftTOCwouldbedevelopedby


29

29

strengthenwomen’sempowermentandensurewomenhaveequalaccesstoservices?

intendedgenderoutputsandoutcomesandtheprocesses/stepsthroughwhichthesearetobeachieved.Someoftheoutputsandoutcomesmayinclude:

• Outputs.

a. Womenparticipatinginthedifferentcommittees.b. Womenparticipatingintheprivategroupenterprises.c. Women’saccesstobenefitsandservicesprovided

throughtheproject.Outcomes.

a. Womeninleadershippositionsinthedifferentgroups.b. Womenowningorgainingmanagementpositionsinthe

privategroupenterprises.c. Women’saccessto,andcontrolofeconomicand

productiveresourcesatthelevelofthehousehold,communityandenterprise.

d. Women’sandgirl’sincreasedaccesstoeducationalopportunities.

e. Women’sincreasedgeographicalmobility.

consultantsonthebasisofprojectdocumentsandinterviews.Feedbackwouldbeobtainedfromprojectstaff,beneficiariesandcivilsociety.

b. ICR.c. Interviewswithprojectstaffandpartner

implementingagencies.d. ICRAnnexes11and12.

2. Howrelevantaretheseintendedoutputsandoutcomesforwomen’sempowerment?

a. Women’sopinionsontherelevanceofeachoutputandoutcome(seepoint1above)totheirlivesandtotheirfeelingofempowerment[***Note:anappropriateandunderstandabletermfor“empowerment”mustbeidentified].

b. Theopinionofkeyinformantsandwomen’sorganizationsontherelevanceoftheoutputsandoutcomesfordifferentgroupsofwomen.

a. Reviewtheoryofchange.b. Focusgroupswithbeneficiaries,project

staffandcivilsociety.c. Anappropriatelocallyunderstood

translationof“empowerment”wouldbedevelopedthroughin-depthinterviewsandpossiblydiscussiongroupswithbeneficiaries.

d. Oneapproachusedfortheevaluationoftheimpactsofmicrocreditonwomen’sempowermentinBangladeshwastomeetwithlocalwomeningroupsandtoworkwiththemtoidentifywhatforthemwouldbekeydimensionsofempowerment(forexample,beingabletotraveloutsidethefamilycompound,tobeinvolvedindecisionsonpurchaseofschooluniforms,beingabletowalkthroughricepaddieswithoutbeing


30

30

requiredtostepintothewatertoallowmentopassonthenarrowpathway).Eachitemwasthenputintoascaleandwomenwereaskedtolocatetheirpresentpositiononeachdimension1.Theadvantageofthisapproachisthatthewomenthemselvesdefinedthedimensionsthatwereimportanttothem,andtheadvances(usuallyverymodest)thattheythoughtwerepossible.

3. Howrelevantaretheinclusionstrategy,socialmobilizationandprovidingessentialinfrastructure,towomen’sempowerment?

a. Women’sopinionsontherelevanceofinclusion,socialmobilizationandprovisionofinfrastructurefortheirlivesandtheirfeelingofempowerment.

b. Theopinionofkeyinformantsandwomen’sorganizationsontherelevanceoftheseprojectcomponentsforthedifferentgroupsofwomen.

a. Expertandkeyinformantinterviews.b. Interviewswithprojectstaff.c. InformationfromthePADandICR.d. Focusgroupsandin-depthinterviews.e. Observationofprojectactivities(for

example,dosocialmobilizationtrainingactivitieshelpwomentoparticipatemoreactively?

f. Audioandvideorecordingsofgroupactivities.

4. Howrelevantaresmallscaleprivategroupenterprisesforwomen’sempowerment?

a. Women’sopinionsontherelevanceofprivategroupenterprisesfortheirlivesandtheirfeelingofempowerment.

b. Theopinionsofkeyinformantsandwomen’sorganizationsontherelevanceoftheseenterprisesfordifferentgroupsofwomen.

a. Asforpoint3.b. Reviewofgrouprecords(forexample,

loansizesforwomenandmen).

5. Howrelevantwerethegenderobjectivesandtheirimplementationfortheachievementofoverallprojectobjectives?

a. Opinionsofprojectstaffontherelevanceofthegenderobjectivesfortheoverallachievementofprojectobjectives.

b. Opinionsofkeyinformantandwomen’sorganizations.

a. Asforpoint3.b. Reviewoftheoryofchangetocheckon

criticalassumptions.

• 3.EFFICIENCY:Gender-responsivenessofprojectimplementation

1. Werewomenconsultedandinvolvedinprojectdesign?

a. Proportionofwomenondifferentplanninggroups.b. Proportionofwomenwhowereawareofthe

proposedprojects.c. Proportionofwomenwhosaytheywereconsulted.d. Proportionofwomenwhowereinvolvedinproject

1Hashemi,SchulerandRiley(1996)“Ruralcreditprogramsandwomen’sempowermentinBangladesh,”WorldDevelopment24(4):635-53.


31

31

implementation. Anappropriatecombinationofthefollowingtoolsandtechniqueswillbeusedtoaddresseachofthesequestions.

Projectmonitoringreports.

a. Projectprofiles.b. Focusgroups.c. Interviewswithprojectstaff.d. Socialmediaanalysis(ifappropriate).e. Rapidsurveys(iffeasible).f. ICR.

2. Didtheprojectsselectedreflecttheprioritiesofwomen?

i. Proportionofwomenwhosayprojects:o Respondeddirectlytotheirneedso Respondedsomewhattotheirneedso Didnotrespondtotheirneeds

3. Wasthewaytheprojectwasimplementedconducivetotheparticipationofwomen?

a. Werethereguidelinesonhowtoinvolvewomeninprojects?

b. Howactivelywerewomeninvolved?4. Howdidtheparticipationof

womeninprojectdesignandimplementationaffecttheachievementoftheoverallprojectobjectives?

a. Howactivelywerewomeninvolvedinprojectdesignandimplementation?

b. Werethereanydifferencesinhowwellprojectsachievedtheirobjectivedependingonhowactivelywomenwereinvolved?

5. Theeffectivenessofthemobilizationstrategy.

a. Weretheredifferencesinhoweffectivelymobilizationstrategiesinvolvedwomenandmen?

b. Whatwerewomen’viewsontheeffectivenessofthemobilizationstrategies?

• 4.OUTPUTSANDOUTCOMES:Achievementofgenderresponsiveprojectobjectives{efficacy]

A.OUTPUTS: • •

1. Women’sparticipationinleadershipandmanagementofprojects.

a. Theproportionofprojectorganizationleadersandmanagerswhoarewomen.

a. Projectmonitoringreports.b. Interviewswithprojectstaff.c. Projectprofiles.d. Audioandvideorecordings.e. Observation.f. Expertandkeyinformantinterviews.

2. Women’sroleinprojectdecision-making.

g. Whatisthelevelofcontributionofwomentoprojectdecisionmaking:

o Womenplayamajorrole.o Womenplaysomerole.o Womenplayverylittlerole.

B.OUTCOMES:Assessinggenderresponsiveoutcomes.

• •

1. Projecteffectsonwomen’sempowerment.

a. Participationincommunityorganizations.b. Participationindecision-makingincommunity

organizations.c. Didtheselectionofvillageprojectsreflectwomen’s

needsandpriorities?d. Impactsonwomen’sstatusinthehousehold:

• Accesstoandcontroloverresources.• Participationindecision-making.

a. Theoryofchange.b. Monitoringreports.c. ICR.d. Expertsandkeyinformants.e. Casestudies.f. Analysisofsocialmedia.g. Audioandvideorecordings.h. Focusgroups.


32

32

e. Geographicalmobility.

f. f.Strengtheningsocialcapital.

i. Observation.j. Rapidsurvey.k. Projectprofiles.l. Self-reportedratingscales.m. Harvard-typetimeuseandaccesstoand

controlofresourceschecklists.

2. Assessingsocialandeconomicoutcomesforwomen.

a. Girl’sandwomen’seducation.b. Personalsecurity.c. Geographicalmobility.d. Strengtheningsocialnetworksandsocialcapital.e. Accesstoinformationabouttheoutsideworldand

aboutthecommunity.f. Reducingtimeburdens.

3. Doprojectsaddresswomen’sneedsandpriorities[***Note:thismaybeincludedunderefficiency].

a.Proportionofwomenwhosayprojects:

• Respondeddirectlytotheirneeds.• Respondedsomewhattotheirneeds.• Didnotrespondtotheirneeds

4. Identifyingunintendedoutcomes. a. Didtheprojectshaveanyunintended(unanticipated)positiveoutcomesforwomen?

b. Didtheprojectshaveanyunintended(unanticipated)negativeoutcomesforwomen?

• SUSTAINABIITYANDRESILIENCE:

1. Whatevidenceistherethatdifferentgenderoutcomeswillbesustainable?

a. Haveprojectscontinuedtodeliverthesamelevelofservicesoverthelifeoftheproject(up-tototimeoftheevaluation?)

a. Sustainabilitychecklists.b. Casestudies.c. Observation.

2. Doindividualwomen,andwomen’sgroupsdevelopstrategiestoadapttostressandshocksanddotheylearnfromexperienceandimprovehowtheyadapt?

a. Whatlessonshavebeenlearnedwithrespecttowaystoanticipateandadapttostressesandshocks?


33

33

Appendix 7: Example of a GRE design matrix: Evaluating a hypothetical public transport project in Eurasia with defined gender objectives.

1.FrameworkoftheevaluationreportItisproposedthattheframeworkoftheevaluationshouldfollowthestructureofatypicalIndependentEvaluationGroupprojectassessmentreport.Thegenderassessmentwillbebuiltintotherespectivesectionsoftheoverallevaluation.TheproposedoutlinepresentedinBoxA7-1alsosuggestssomeadditionalsectionsthatmightbeincludedinapublictransportproject.

2.Theproject’sgenderobjectives.2.1. PrimaryobjectivesTheProjectAppraisalDocumentidentifiesseveralobjectivesrelatingtopromotingequalityofopportunityintheprocessesofrecruitment,trainingandcareeradvancementwithinthepublictransportcompany.Thesewillbedefinedasthedirectgenderobjectives.Followingthepracticeofmosttheoriesofchange,theseobjectivesaredividedintooutputsandoutcomes(seeBoxA7-2).However,thereisextensiveevidencethatwomenandmenhavedifferenttravelandtransportneeds.Furthermore,manyprojectstendtofocusmoreonmen’stransportneeds(gettingquicklyfromhometo

BoxA7-1:ProposedoutlineforthegenderassessmentcomponentofthePublicTransportProjectinEurasia.

[Possibleadditionalsectionsnotnormallyincludedinthistypeofevaluationareindicatedby***].1. Historyandcontext:

a. ***Theproject’sgenderobjectivesandtheprojectdesignthroughwhichtheobjectiveswillbeachieved.Thismightincludeboththeexplicitgenderobjectivesandsomesecondarygenderoutcomes[objectives]thatmightbeachieved.

b. TheGenderTheoryofChange.TheIEOwilldecideifthisisrequired.2. Relevance:Therelevanceoftheprojectconceptanddesignforpromotingwomen’s

empowermentandaccesstotheprojectsocialandeconomicbenefits.3. ***Efficiency:Thegender-responsivenessofprojectdesignandimplementation.4. Achievementoftheprojectgenderobjectives[efficacy].

a. Outputs.b. Outcomes:primary(projectgenderobjectives),secondary(otherpotentialbenefitsfor

womenparticipantsnotincludedinprojectdesign),andtertiary(effectsonotherwomennotinvolvedintheproject).

5. ***Sustainabilityandresilience:Thelikelihoodthatgenderoutputsandoutcomeswillbesustainedandthebuscompanyandotherstakeholderswillhavetheresiliencetolearnfromexperienceandtoadapttotheevolvingcontextwithinwhichtheprojectoperates.

6. Lessonslearned:howtodesignfutureprojectstostrengthenwomen’sempowermentandgenderequality.


34

34

work)andtooverlookwomen’smorecomplexneedstocombinetraveltoworkwithtakingchildrentoschoolandthedoctorandshopping(WorldBank,2010)2.Thesemulti-chainingneedsareignoredbymanytransportprojects,resultinginthelossofmanypotentialwomenpassengerswhohavetorelyoninformaltransportservices.Therearealsoissuesofsecurityasmanystudieshavefoundthathighproportionsofwomenhaveexperiencedsexualharassmentonpublictransport.TheAsianDevelopmentBank(AsDB)GenderandTransportToolkitreportsthattherearegenderdifferencesintravelpatterns,useoftransportmodes,timeuseandtimepoverty,accesstoresourcesfortravel,mobilityandsafetyandthatthereareanumberofgenderbarriersanddifferencesinbenefits3.Asanobjectiveoftheprojectistoprovidehighqualityservicetopassengers,BoxA7-2identifiesanumberofsecondaryandtertiaryoutcomesthatcouldbeconsideredforinclusion.Ifanyoftheseoutcomesareincludedintheevaluation,itwillbeimportanttoclarifythatprojectmanagersandstaffarenotbeingassessedonwhethersecondaryoutcomesareachievedasthesewerenotincludedintheprojectobjectives.ThereasonfortheirinclusionistohelptheEuropeanBankforReconstructionandDevelopment(EBRD)learnlessonstoimprovethedesignandgender-responsivenessoffutureprojects.2.2. DefiningsecondaryandtertiaryoutcomesChapter9discussesinmoredetailhowsecondaryandtertiaryoutcomescouldbedefinedandmeasuredforthisproject.Theproposedapproachisstrategicallyhelpfulbecauseitshowsthatmanyoftheseoutcomesarepotentiallypositiveandcouldsignificantlyincreasetheestimatedprojectbenefits(ratesofreturn).Thisisimportantbecausesecondaryandtertiaryoutcomesareoftendiscussedintermsof“unintendedoutcomes”wherethefocushasusuallybeenonthenegativeoutcomes,suchasincreasesindomesticviolenceorincreaseddemandsonwomen’stime.

3.Theproject’sgenderresponsiveinterventionsTheprojecthasseveralcomponents/interventionsthatarespecificallyintendedtopromotetheequalparticipationofwomeninrecruitment,trainingandjobadvancementwithinthepublictransportcompany(seeBoxA7-3).ThereisalsoareferencetoensuringtheproposedspecificationsforthebusessatisfyEUrequirementsforenvironmentalandsocialimpact.Thiscouldbeinterpretedtoincludestrategiestopreventsexualharassment(acommonproblemonpublictransport)andpossiblyalsotoconsidersafetyandconvenienceconsiderationswithrespecttothelocationofbusstops.Oneimportantsetofissuesthatdonotseemtobeaddressedconcernsthedifferenttransportandtravelneedsofwomenandmen(discussedintheprevioussection).However,asthereisnoreferencetotheseissuesintheprojectdocument,thesequestionscanpresumablynotbeaddressedwhenassessingprojectinterventions.

2WorldBank(2010).Mainstreaminggenderinroadtransport.Chapter2Gendertrippatternsandmobilityconstraints.3ADB(2013)Gendertoolkit:Transport.Maximizingthebenefitsofimprovedmobilityforall.


35

35

BoxA7-2:Identifyinggenderobjectives:Directprojectgenderoutputsandoutcomes,andsecondaryandtertiarygenderoutcomes.

GenderOutputs

DirectGenderOutputs

a. Equalityof:o Recruitmentopportunities.o Careerprogression.o Accesstotraining.

b. Increasethenumberofwomenhired.c. Workshopwithstakeholderstodisseminatelessonslearnedonhowtopromotegender

equality.d. EnsurebusesmeetEuropeanUnionstandards,includingongenderconsiderations(suchas

securityandcomfort).

Secondarygenderoutputs[notdefinedintheprojectdesign]

a. Routeplanningandservicefrequencytakesintoconsiderationwomen’sspecifictransportneeds.

b. Serviceplanningtakesintoconsiderationgender-relatedsecurityandsafety.Forexample:o Driversandconductorsaretrainedhowtoaddresssexualharassmentonthebuses.o Busstopsarewelllitarenotlocatedclosetobarsorotherareaswithgreaterriskfor

women.o Additionalsecurityisprovidedforwomentravellingtoandfromworkatnight.o Driversandstaffaretrainedtorespectwomenpedestrians[studiesinanumberof

countrieshavefoundthatdriversdonotslowdowntoallowwomenpedestrianstosafelycrosstheroad,andsometimesdriverswillnotcompletelystopsothatwomenhavetogetoffwhenthebusisstillmoving].

c. Numberofwomenpassengerssteadilyincreases.

PrimaryGenderOutcomes(programobjectives)

Directgenderoutcomes

a. Morewomenarehired.b. Morewomenreceivetraining.c. Morewomenarepromoted.d. Women’sjobstabilityincreases.e. Morewomenbecomedriversandmechanics.f. Gendergoodpracticelessonsaredisseminatedto,andadoptedbyothertransport

companies.Secondarygenderoutcomes

a. Women’ssafetyimprovescomparedtootherbuscompaniesandformsoftransport.b. Genderresponsiverouteplanningcontributestoreducingwomen’stimeburden.c. Women’incomeincreases(duetogreateraccesstojobopportunitiesthroughmore

convenienttransport].d. Children’shealthimprovesasgender-responsivetransportplanningmakesiteasierfor

workingmotherstotakechildrentothedoctor.e. Children’sschoolattendanceimprovesasitiseasierforworkingmotherstotakechildrento

school.


36

36

4.DefinitionofthegenderquestionstobeaddressedintheevaluationBoxA7-4identifiesaninitiallistofpossiblegender-relatedquestionsrelatingtodimensionslistedinTableA7-1.Thesecoverfivedimensions:

a. Historyandcontextandhowtheseaffecttheimplementationandlikelygenderoutcomesoftheproject.Thisincludesinformationonearlierprogramstoincorporatewomenintothetransportsectorandbroadereffortstobroadenwomen’saccesstolabormarkets.Relevantlegislationandgovernmentpolicieswillalsobedescribed.

b. Relevanceoftheprojectdesigntotheachievementofimportantgenderobjectives.c. Efficiencyofprojectimplementationwithrespecttotheachievementofgenderobjectives.d. Achievementofgenderobjectives(outcomes/efficacy).Objectivesaredefinedintermsof

outputsandoutcomes.e. Sustainabilityandresilience.Thereisnodirectreferencetosustainabilityorresiliencein

theprojectdocument.However,theseareimportantconsiderationsasmanyinitiativestopromotewomen’seconomicequalitybeginwellbutoftenencounterorganizational,cultural,politicalandeconomicchallengesthatreducetheirlonger-termimpact.So,adecisionmustbemadeastowhetheritisappropriatetoincludethisdimensionintheevaluation.

BoxA7-3Projectcomponentsandinterventionsdesignedtopromotewomen’saccesstoemploymentopportunitiesandattentiontowomen’sconcernsinthedesignofbuses(andpossiblybusstops).

1.Interventionstopromotewomen’saccesstoemploymentopportunitiesinthebuscompany.

a. Promotinggender-responsiverecruitmentpolicies.b. Puttinginplacecareeradvancementpoliciesforwomen.c. Trainingforwomencandidatestopromotecareeradvancementandincreasing

thenumberofwomenemployedatthecompany.2.Gender-sensitiveservicedesign.

a. EnsuringbusdesignrespondstoEUrequirementsonsocialandenvironmentalimpacts.

b. Thisrequirementmightalsoincludelocatingbusstopsandtheirdesign(forexample,adequatelighting)totakeintoconsiderationwomen’ssafetyandconvenience.

c. Note:theredoesseemtobeanyreferencetoplanningtransportroutestorespondtowomen’smulti-chainingneedstocombinetraveltoworkwiththeneedtotakechildrentoschoolandtothedoctorandtogoshopping.

3.Disseminatinglessonsonhowtomaketransportprogramsmoregender-responsive.Workshopspromotingknowledgesharingacrossthecountryandthewiderregion.


37

37

BoxA7-4:Gender-relatedquestionstobeaddressedintheevaluation

1.Historyandcontexta. Havetherebeenearlierinitiativestopromotewomen’semploymentinthetransport

sector?Whatweretheresults?b. Havetherebeeninitiativesinothersectorsandwhatweretheresults?c. Isthereanylegislationorgovernmentpoliciesaffectingwomen’slabormarketaccess?

Whathavebeentheeffectsofthesemeasures?d. Howdidtheseearlierexperiencesaffectthedesignofthepresentproject?

2.Relevancea. Howdoestheprogramexpecttostrengthenwomen’saccesstoemploymentandjob

advancementinthebuscompany?b. Aretheinterventionsrelevanttotheachievementofgenderequalityinthebuscompany

andperhapsmorewidely?c. Howrelevantarethegenderobjectivesandtheirimplementationsstrategiestothe

achievementofoverallprojectobjectives?3.Efficiency

a. Werewomenconsultedandinvolvedinprojectdesign?b. Dotheselectedinterventionreflecttheprioritiesofwomen?c. Arethereanyadditionaldesignorimplementationelementsthatshouldhavebeen

included?d. Wereprojectimplementationstrategiesconducivetotheparticipationofwomen?

4.Achievementofgenderobjectives(efficacy)a. Outputs:

i. Increasednumberofwomenrecruited.ii. Increasednumberofwomentrained.iii. Rateofwomen’sadvancementincreases.iv. Consultationmechanismswithwomenareputinplace.v. Busandservicedesigntakesgenderissuesintoconsideration.vi. Genderissuesareincludedinworkshopstodisseminatelessonsfromtheproject.

b. Primarygenderoutcomes.i. Increaseinproportionofwomenemployedbythebuscompany.Proportion

increaseoverthelifeoftheproject.ii. Increaseinproportionofwomeninnon-administrativepositions(drivers,

mechanics).iii. Increaseinwomen’searnings.iv. Increaseinproportionofwomeninmanagerialpositions.

c.Secondarygenderoutcomes.

i. Improvedwork-lifebalance.ii. Enhancedwomen’sempowerment(seeChapter9).

d.Tertiarygenderoutcomes

i. Improvedcomfortandsafetyforwomenpassengers.ii. Timesaving.iii. Increasedaccesstourbanservicesandentertainment.

5.Sustainabilityandresiliencea. Whatevidenceistherethatwomen’sadvancesinthecompanyarelikelytobesustained?b. Havewomeninthecompanylearnedcopingmechanismsforworkingandadvancingina

male-dominatedworkenvironment?


38

38

5.Initialproposalforthegenderevaluationmethodology.5.1.TableA7-1presentsinitialideasfortheevaluationdesign.Thisprovidesaninitialframeworkthatwillneedtoberefinedonceagreementhasbeenreachedontheevaluationapproach.Thetableincludesthreecolumns:

• ThequestionscoveringeachofthefivedimensionsoftheevaluationlistedinBoxA7-4.• Theindicatorsusedtoaddresseachquestion.• Thedatacollectionmethods.

Theframeworkidentifiesarangeofpossibledatacollectionmethodsfromwhichanappropriatesetwillbeselectedforeachstageoftheevaluation.AllofthesemethodsaredescribedintheECGReferenceDocument.Itislikelythatnotallofthelistedmethodswillbeusedinthepresentevaluation.Theinitiallistofdatacollectionandanalyticalmethodsincludes:1. Constructingagendertheoryofchange[TOC]thatcanhelpidentifykeyevaluationquestions,

indicatorsandthepathwaysthroughwhichitisexpectedthatgenderoutcomeswillbeachieved.NotallgenderevaluationsuseaTOCsoadecisionwillbeneededastowhetheritisappropriateforthepresentevaluation.

2. Constructingadatacollectionplanningmatrix.Appendix5presentsanexampleofadatacollectionplanningmatrixthatcouldbeusedinthedesignofthepresentevaluation.Thematrixlistsallofthekeygender(andother)questions,theproposedevaluationdesigns,theindicatorsandthedatacollectionmethods.Importantly,italsoassessesthefeasibilityofcollectingtheproposedinformationwithinthebudget,time,methodologicalandorganizationalconstraintswithinwhichtheevaluationwillbeconducted.

3. Secondarysources.a. Theprojectdocument.b. Projectmonitoringandprogressreports.c. Governmentreports.d. Partnerreports.e. Civilsocietyreports.f. Academicresearchandpublications.g. Systematicreviews.

4. Consultations.a. Expertandkeyinformants.b. Civilsocietyorganizations.c. Otherfundingagencies.d. Focusgroups.

5. Surveys,ratingscalesandchecklists.a. Rapid,shortsamplesurvey.b. Trafficandpassengersurveys.c. Travelsafetyaudits.d. Ratingscalestobecompletedbybeneficiaries,projectstafforpartners.e. Checklists.

6. Casestudies.a. Descriptivecasestudies.b. Analyticalcasestudies(QCA).

7. Qualitativefieldwork.a. Projectvisits.b. Informalinterviews.c. In-depthinterviews.d. Observation.e. Participantobservation.


39

39

f. Traveldiaries.8. Socialmediaanalysis(ifthisisfeasible).

a. Socialmediaanalytics(forexample,analysisoftwitterandothersocialmedia).b. Internetsurveys.

9. Pipelinedesignstoconstructacounterfactual.5.2.MeasuringsecondaryandtertiaryoutcomesChapter9discussesinmoredetailhowsecondaryandtertiaryoutcomescanbedefinedandmeasured.


40

40

• TableA7-1:Keygender-responsivequestions,indicatorsanddatacollectionmethods

•

• PARTI:SOMEGENERALAPPROACHESTHATCANBEUSEDTHRUGHOUTTHEEVALUATION

1.GenderTOC:DevelopingagenderTOCthatisusedtoidentifythekeyevaluationquestionsandtodefinetheprocessesthroughwhichoutputsandoutcomesaretobeachieved.Thisalsodefineskeyassumptionstobetested.

• 2.Resultsframework(ifithasbeenusedintheproject):Thisdefinestheintendedgenderobjectivestobeassessed.

• 3.Descriptivecasestudies:Usuallyarelativelysmallnumberofcasestudiesthatarebroadlyrepresentativeofthemainprojectscenariosandwhichareusedtoillustratehowtheprojectsevolvedandthelivedexperienceoftheprojectpopulations.Casestudiescanbelongitudinal(conductedover

arelativelylongperiodoftime),or(asisusuallythecasewithex-postevaluations)conductedatonepointintime,andrelyingextensivelyonrecall.

• 4.Analyticalcasestudies:TheseusetechniquessuchasQCA(qualitativecaseanalysis)toidentifythenecessaryandsufficientconditionsforprojectoutcomestooccur,andalsothenecessaryandsufficientconditionswhereprojectoutcomesdoNOToccur.Normallyatleast30casesarerequired.

WhileQCAarerarely(ifever)usedbyIEOs,theycouldbeapotentiallypowerfulanalyticaltoolthatpermitsattributionanalysistobeused.

• 5.Counterfactualanalysis:Thiscanbeusedatthenationallevelforprogramsintendedtocoverthewholecountry,orattheprojectlevel.FortheevaluationoftheTransportProject,counterfactualwouldprobablycomparetheprojectwithotherbuscompanies(assumingthereanyother

companiesthatareofasimilarsizewithcommoncharacteristics).Theanalysiswouldrequireabeforeandaftercomparison.

•

• PARTII:DESIGNSFORASSESSINGEACHDIMENSIONOFTHEEVALUATION

•

• Dimension/Question • Indicators • Datacollectionmethods

• [appropriatetoolswillbeselectedforeachphaseoftheevaluationfromthislistof

options]

• 1.HISTORYANDCONTEXT

a. Havetherebeenearlier

initiativestopromotewomen’s

employmentinthetransport

sector?Whatweretheresults?

a. Transportsectorprojectswithgender

components/objectives.

b. Genderoutcomes.

c. Transportprojectsthathadunintendedgender

outcomes.

a. Expertandkeyinformantsinterviews.

b. Governmentreports.

c. Civilsocietyreports.

d. Donorreports.

e. Academicresearch.

b. Howdidtheseearlier

experiencesaffectthedesignof

thepresentproject?

a. Referencesintheprojectdocumentsonprevious

genderinitiatives.


41

41

c. Havetherebeeninitiativesin

othersectorsandwhatwerethe

results?

a. Othersectorprojectswithgenderemployment

components.

b. Projectgenderoutcomes.

d. Isthereanylegislationor

governmentpoliciesaffecting

women’slabormarketaccess?

Whathavebeentheeffectsof

thesemeasures?

•

a. Legislationconcerningwomen’slabormarketaccess.

b. Opinionsonpolicyoutcomes.

e. Arethereanyimportant

economic,politicalorcultural

contextualfactorsthathave

affectedhowthegender

componentsweredesignedor

implementedorthataffected

theiroutcomes?

a. Reportsandindicatorsoncontextualfactorsaffecting

women’slabormarketaccess.•

• 2.RELEVANCE:Therelevanceoftheprojectconceptanddesignforpromotingwomen’sempowermentandsocialandeconomicbenefits

a.Howdoestheprojectexpectto

promotewomen’saccessto

employmentandtoimprovethe

qualityoftransportservicesfor

women?

a. DevelopaTOCthatidentifiestheintendedgender

outputsandoutcomesandtheprocesses/steps

throughwhichthesearetobeachieved.Someofthe

outputsandoutcomesmayinclude:

i. DirectGenderOutputs[seeSection4A].

ii. Secondarygenderoutputs[seeSection4B].

iii. Directgenderoutcomes[seeSection4C].

iv. Secondarygenderoutputs[seeSection4D.]

a. ThedraftTOCwouldbedeveloped

byconsultantsonthebasisofproject

documentsandinterviews.Feedback

wouldbeobtainedfromprojectstaff,

beneficiariesandcivilsociety.

b. Projectdocument.

c. Interviewswithprojectstaffand

partnerimplementingagencies.

d. Expertsandkeyinformants.

e. Householdincomeandexpenditure

surveys.

f. Passengerandtransportsurveys.

g. Socialmediaanalysis(twitter,and

suchlike)iffeasible.

h. Participantobservation(observer

travellingonbuses).


42

42

b.Howrelevantaretheseintended

outputsandoutcomesforwomen

workersandpassengers?

a. Women’sopinionsontherelevanceofeachoutputand

outcome(seepoint1above)totheirlives[***Note:an

appropriateandunderstandabletermfor

“empowerment”mustbeidentified].

b. Theopinionofkeyinformantsandwomen’s

organizationsontherelevanceoftheoutputsand

outcomesfordifferentgroupsofwomen.

a. Reviewtheoryofchange.

b. Focusgroupswithbeneficiaries,project

staffandcivilsociety.

c. Expertandkeyinformantinterviews.

d. Interviewswithprojectstaff.

e. Informationfromtheprojectdocument.

f. Audioandvideorecordingsofproject

activities.

g. Projectmonitoringandprogressreports.

h. ApplicationofOECD-typeratingscale.

c.Howrelevantwerethegender

objectivesandtheirimplementation

fortheachievementofoverallproject

objectives?

a. Opinionsofprojectstaffontherelevanceofthegender

objectivesfortheoverallachievementofproject

objectives.

b. Opinionsofkeyinformantandwomen’sorganizations.


43

43

• 3.EFFICIENCY:Gender-responsivenessofprojectimplementation

a.Werewomenconsultedand

involvedinprojectdesign?a.Werestakeholdersconsultedonprojectdesign?

b.Werethereprovisionstoinvolvewomeninthe

consultations?

c.Whichgroupsofwomenwereconsulted?

-Actualorpotentialemployees

-Civilsociety

f.Howmanywomenwereactivelyinvolvedthe

consultations?

•

a. Projectmonitoringreports

b. Projectprofiles

c. Focusgroups

d. Interviewswithprojectstaff

e. Socialmediaanalysis(ifappropriate)

f. Rapidsurveys(iffeasible)

g. Observation

b.Didtheprojectsselectedreflect

theprioritiesofwomen

Proportionofwomenwhosayprojects:

o Respondeddirectlytotheirneeds.

o Respondedsomewhattotheirneeds.

o Didnotrespondtotheirneeds.

c.Wasthewaytheprojectwas

implementedconducivetothe

participationofwomen?

a. Werethereguidelinesonhowtoinvolve

womeninprojects?

b. Howactivelywerewomeninvolved?

d.Howdidtheparticipationof

womeninprojectdesignand

implementationaffectthe

achievementoftheoverallproject

objectives?

a. Howactivelywerewomeninvolvedinproject

designandimplementation?

b. Werethereanychangesindesigninresponse

tofeedbackfromwomen?

c. Howdidthesechangesaffecttheoverall

efficiencyofthedifferentprojectcomponents?

• 4.OUTPUTSANDOUTCOMES(IMPACTS):Achievementofgenderresponsiveprojectobjectives

A.DIRECTGENDER-RESPONSIVEOUTPUTS:

a.DirectGenderOutputs

i. Increasednumberofwomenrecruited.

ii. Increasednumberofwomentrained.

iii. Rateofwomen’sadvancementincreases.

iv. Consultationmechanismswithwomenareput

inplace.

v. Busandservicedesigntakesgenderissuesinto

consideration.

vi. Genderissuesareincludedinworkshopsto

disseminatelessonsfromtheproject.

a. Theprojectdocument.

b. Interviewswithprojectstaffandpartnerimplementingagencies.

c. Expertsandkeyinformants.d. Householdincomeandexpenditure

surveys.

e. Passengerandtransportsurveys.f. Socialmediaanalysis(twitterandsuch

like)iffeasible.

g. Participantobservation(theresearcherspendstimetravellingonbusesto


44

44

• observesexualharassmentandother

issuesaffectingwomenpassengersand

drivers/conductors.

h. Participantobservation(observertravellingonbuses).

i. Focusgroupswithbeneficiaries,

projectstaffandcivilsociety.

j. Audioandvideorecordingsofproject

activities.

k. Projectmonitoringandprogress

reports.

l. ApplicationofOECD-typeratingscale.

•

B.SECONDARYGENDER-RESPONSIVEOUTPUTS

b.Secondarygenderoutputs.

i. Routeplanningandservicefrequencytakes

intoconsiderationwomen’sspecifictransport

needs.

ii. Serviceplanningtakesintoconsideration

gender-relatedsecurityandsafety.For

example:

o Driversandconductorsaretrainedhowto

addresssexualharassmentonthebuses.

o Busstopsarewelllightedandarenot

locatedclosetobarsorotherareaswith

greaterriskforwomen.

o Additionalsecurityisprovidedforwomen

travellingtoandfromworkatnight.

o Driversandstaffaretrainedtorespect

womenpedestrians[studiesinanumberof

countrieshavefoundthatdriversdonot

slowdowntoallowwomenpedestriansto

safelycrosstheroad,andsometimes

driverswillnotcompletelystopsothat

womenhavetogetoffwhenthebusisstill

moving].

•


45

45

iii. Numberofwomenpassengerssteadily

increases.

C.DIRECTGENDERRESPONSIVEOUTCOMES

c.Directgenderoutcomes

i. Increaseinproportionofwomenemployedby

thebuscompany.Proportionincreaseoverthe

lifeoftheproject.

ii. Increaseinproportionofwomeninnon-

administrativepositions(drivers,mechanics).

iii. Increaseinwomen’searnings.

iv. Increaseinproportionofwomeninmanagerial

positions.

v. Hasthework/livebalanceimprovedfor

womenworkers?D.SECONDARYGENDERRESPONSIVEOUTCOMES

d.Secondarygenderoutcomes

i. Women’ssafetyimprovescomparedtoother

buscompaniesandformsoftransport.

ii. Genderresponsiverouteplanningcontributes

toreducingwomen’stimeburden.

iii. Women’incomeincreases(duetogreater

accesstojobopportunitiesthroughmore

convenienttransport.

iv. Children’shealthimprovesasgender-

responsivetransportplanningmakesiteasier

forworkingmotherstotakechildrentothe

doctor.

v. Children’sschoolattendanceimprovesasitis

easierforworkingmotherstotakechildrento

school.

•

3. Women’sparticipationin

leadershipandmanagementof

projects

a. Theproportionofprojectorganizationleadersand

managerswhoarewomen

a. Projectmonitoringreports.

b. Interviewswithprojectstaff.

c. Projectprofiles.

d. Audioandvideorecordings.

e. Observation.

f. Expertandkeyinformantinterviews.

4. Women’sroleinproject

decision-making

a.Whatisthelevelofcontributionofwomentoproject

decisionmaking:

i. Womenplayamajorrole.


46

46

ii. Womenplaysomerole.

iii. Womenplayverylittlerole.

B.OUTCOMES:Assessinggenderresponsiveoutcomes.

• •

5. Projecteffectsonwomen’s

empowerment

a. Participationincommunityorganizations.

b. Participationindecision-makingincommunity

organizations

c. Impactsonwomen’sstatusinthehousehold.

d. Accesstoandcontroloverresources.

e. Participationindecision-making.

f. Geographicalmobility.

g. Strengtheningsocialcapital.

a. Theoryofchange.

b. Monitoringreports.

c. Projectdocument.

d. Expertsandkeyinformants.

e. Casestudies.

f. Analysisofsocialmedia.

g. Audioandvideorecordings.

h. Focusgroups.

i. Observation.

j. Rapidsurveys.

k. Projectprofiles.

l. Self-reportedratingscales.

m. Harvard-typetimeuseandaccesstoandcontrolofresources

checklists.

6. Assessingsocialandeconomic

outcomesforwomen

i. Girl’sandwomen’seducation.

ii. Personalsecurity.

iii. Geographicalmobility.

iv. Strengtheningsocialnetworksandsocial

capital.

v. Accesstoinformationabouttheoutsideworld

andaboutthecommunity.

vi. Reducingtimeburdens.

7. Doprojectsaddresswomen’s

needsandpriorities[***Note:

thismaybeincludedunder

efficiency].

a.Proportionofwomenwhosayprojects:

i. Respondeddirectlytotheirneeds.

ii. Respondedsomewhattotheirneeds.

iii. Didnotrespondtotheirneeds.

8. Identifyingunintended

outcomes.

a. Didtheprojectshaveanyunintended

(unanticipated)positiveoutcomesforwomen?

b. Didtheprojectshaveanyunintended

(unanticipated)negativeoutcomesforwomen?


47

47

• SUSTAINABIITYANDRESILIENCE:

3. Whatevidenceistherethat

women’sadvancesinthe

companyarelikelytobe

sustained?

a. Didtheproportionofwomenrecruitedincrease,

anddidtheincreasecontinuethroughoutthelifeof

theproject?

b. Didtheproportionofwomenwhoreceived

advancements(promotion)increaseanddidthese

increasescontinuethroughoutthelifeofthe

project?

c. Didthenumber/proportionofwomenreceiving

trainingincrease,anddidtheseincreasescontinue

throughoutthelifeoftheproject?

d. Opinionsofdifferentstakeholdergroupsonthe

likelihoodthattheimprovementsinpointsa-cwill

besustained.

a. Sustainabilitychecklists.

b. Casestudies.

c. Observation.

d. Focusgroups.

e. Individualinterviews.

4. Havewomeninthecompany

learnedcopingmechanismsfor

workingandadvancingina

male-dominatedwork

environment?

a. Whatwerethechallengesfacingwomenworkersto

sustaintheirprogress?

b. Whatweretheopinionsofotherstakeholder

groups?

c. Didwomenreportthattheyhadlearnedanycoping

mechanismstodealwithproblemsidentifiedin

pointb?


48

48

Appendix 8: Links to the gender indexes

1.AfricaGenderEqualityIndex

https://www.afdb.org/en/topics-and-sectors/topics/quality-assurance-results/gender-equality-index/

2.SDGIndicators

https://sustainabledevelopment.un.org/content/documents/21252030percent20Agendapercent20forpercent20Sustainablepercent20Develop

mentpercent20web.pdf

3.TheGender-RelatedDevelopmentIndex(GDI)

http://hdr.undp.org/en/content/gender-development-index-gdi

4.GenderEmpowermentMeasure(GEM)

https://en.wikipedia.org/wiki/Gender_Empowerment_Measure

5.SocialWatchGenderEquityIndex(GGI)

http://www.socialwatch.org/taxonomy/term/527

6.WorldEconomicForumGenderGapIndex(GGI)

https://www.weforum.org/reports/the-global-gender-gap-report-2016

7.AfricaWomen’sProgressScorecard

http://www.uneca.org/sites/default/files/PublicationFiles/agdi_2011_eng_fin.pdf

8.ThematicIndicators

FAOhttp://www.fao.org/fileadmin/templates/ess/ess_test_folder/Workshops_Events/AFCAS_19/AFCAS_05_7_2_b.pdf

UNESCOhttp://uis.unesco.org/en/glossary-term/gender-parity-index-gpi


49

49

Appendix 9: Strengthening the dissemination and use of gender-evaluation findings.

1.TheunderutilizationofevaluationsThereisextensiveevidencethatevenmethodologicallysoundevaluationsarefrequentlyunder-utilized

(BoxA9.1).MostofthesefindingsareequallyapplicabletoGREbutthereareadditionalfactorsthatcan

alsoconstraintheutilizationofgender-responsivefindings.First,manyprojectsdonothavespecific

genderobjectives,orgenderobjectivesaredefinedverynarrowly.Inbothcases,findingsrelatingto

potentiallyimportantgender-relatedsecondaryandtertiaryoutcomesmaynotbeconsideredrelevantif

theywerenotincludedintheresultsframework.ThecasestudiespresentedinPartIIprovidemultiple

examplesofexcludedgenderissues.Second,projectstaffmayberesistanttoacceptingthefindingsof

genderanalysiswhichtheyfeelunfairlycriticizethemfornothavingaddressedgenderissuesthatwere

notincludedintheprojectdesign.Third,thewayinwhichGREfindingsarepresentedisoftennotlinked

directlytodevelopmentobjectives,butaddressesbroaderconcernssuchaswomen’sempowermentor

humanrights.Fourth,GREoftenneedtomakeastrongercaseforthevalueaddedofthetimeand

resourcesinvestedinaddressinggenderissues.Fifth,IEOsmaynothaveadisseminationstrategythat

ensuresthefindingsareaccessibletothewiderrangeofstakeholders(forexample,civilsocietyand

women’sorganizations)thatarethepotentialadvocatesandimplementersoftheevaluationfindings.

Finally,stakeholdersoftenarguethattheGREfindingsarenotbasedonafullunderstandingofthelocal

contextandthatrecommendationsarenotrealisticwithinthelocalpoliticalandculturalscenario.

2.StrategiesforpromotingtheutilizationofGREevaluationsGiventhenatureoftheIEOmandate,notalloftheutilizationstrategiesdiscussedintheliteratureare

applicabletoIEOs.Inparticular,therequirementforindependenceandthefactthatevaluationsare

conductedex-postmeansthatmanyoftheutilization-focusedevaluationstrategies(Patton,2008)that

BoxA9.1Reasonswhyevaluationfindingsareoftenunderutilized

• Badtiming:thereportistoolatetocontributetopolicydecisionsoritcomestooearlybefore

agenciesarefocusingontheseissues.

• Lackofflexibilitytorespondtotheinformationneedsofstakeholders.

• Wrongquestionsareaskedandfindingsnotconsideredusefulorrelevant.

• Evaluationsaretooexpensiveandmaketoomanydemandsonagencystaffandresources.

• Simplisticanswersaregiventocomplexissues.

• Evaluatorsdonotunderstandthecomplexitiesofthelocalcontext.

• Localexpertsarenotconsulted/involvedintheevaluation.

• Findingsarenotpresentedinawaythatiseasilyaccessibletodifferentstakeholders.

Source:Bamberger,SegoneandTateossian(2016)


50

50

requireregularinteractionwithimplementingagenciesarenotdirectlyapplicable.Thefollowingare

someofthestrategiesforpromotingGREutilizationapplicabletoIEOs:

a. Developadisseminationstrategythataddressesalloftheissuesidentifiedintheprevious

section,includingthespecificgender-relatedconstraintsandopportunities.

b. Ensurefindingsandrecommendationsarealignedwithprojectdevelopmentobjectives.Many

GREfindingsaddressbroaderissuessomeprojectstaffmaynotfindrelevantorcannot

implement.Thispresentsastrategicchallengeasmanyoftheseissuesrelatingtoempowerment,

humanrightsandinclusionarecriticaltodevelopment.Findingsmustbeintegratedwitha

capacity-buildingstrategythatcanhelpagenciesreviewandbroadentheirapproachestosocial

aswellaseconomicdevelopment.

c. Relatedtothepreviouspoint,utilizationofGREevaluationfindingscanonlybeachievedifthey

arebasedonafullunderstandingofthelocalpolitical,economicandsocialcontext.Changesin

deeplyengrainedsystemsofsocialcontrolrequirethestrategicidentificationofpotential

interventionpointsthatarerealisticwithineachlocalcontext.

d. Using“carrots”(incentives),“sticks”(sanctionsandpunishments)and“sermons”(showof

supportfromrespectedfigures)toencourageutilization(MacKay,2007).Anexampleofan

incentivemightbetheavailabilityofagenderfundthatcouldbedrawnontosupport

implementationofsomeoftherecommendations.

e. IEOsmayneedtodevelopinnovativecommunicationmechanisms(suchasuseofsocialmedia,

smartphones,collaborationwithcivilsociety,shorterpublicationstargetedtoparticular

audiences).

f. Ensuringthatagreedactionsongenderfindingsareincludedinthemanagementactionplans

thatmostIEOsusetomonitorimplementationofthemanagementagreementsand

commitmentsonevaluationrecommendations.

3.DevelopinganorganizationallearningstrategyIEOsshouldcoordinatewithagencycapacitydevelopmentunitstostrengthenstaffunderstandingof

genderissuesandoutcomesandhowtheyshouldbeaddressedinthedesignoffutureprojects

(programsandpolicies).GREreportsprovidevaluableteachingmaterialastheyillustratehowagency

genderpoliciesandstrategiesactuallyoperateinthefield.Thiscanalsohelpbuildawarenessofthe

importanceofbuildingthecollectionofdataongenderindictorsintoprojectM&Eandmanagement

systems.

4.BuildinggenderindicatorsandfindingsintokeyagencyreportsIEOshouldworktoensurethatgenderindicatorsareincorporatedintokeyagencyreportsandpolicy

documentsaswellasintomostIEGproducts.Chapter2describedthedifferentkindsofindicatorsthat

couldbemainstreamed(seeSection2.7).Thefollowingaresomeoftheindicatorsandevidencethat

couldbeincorporatedintodifferentagencyreports.

BoxA9.2UsefulreferencesonutilizationofGREevaluations

UNWomenIndependentEvaluationOffice(2015)HowtoManageGender-ResponsiveEvaluation:EvaluationHandbook.Chapter7Useandfollow-up.

Bamberger,M,M.Segone&S.Reddy(2014).Nationalpoliciesforsustainableandequitabledevelopment:Howtointegrategenderequalityandsocialequityinnationalevaluationpoliciesandsystems.EvalPartners,UNWomenandIOCE.

KarkaraN(undated).“Advocatingforevaluation:Atoolkittodevelopadvocacy

strategiestostrengthenanenablingenvironmentforevaluation”.


51

51

a. Disaggregatingstandardsocio-economicindicatorsbysex.

b. Presentinggenderchecklists.

c. IncorporatinggenderintotheOECD/DACstandardratingscales.

d. Shortillustrativecasestudies.


52

52

iManyevaluationsrelyondatacollectedatthelevelofthehousehold.GREevaluatorsarguethathouseholdleveldata

ignoresimportantdifferencesinhowfoodandotherresourcesaredistributedamonghouseholdmembers.Disaggregated

analysisiscriticalforGREaswomenandgirlsoftenreceiveasmallershareoffoodsothatmalnutritionratescanbe

higher.

iiForexample,CaseStudyNo.1(theVillageInvestmentProject)includesoutputindicatorsforeachoftheProject

Components(capacitybuildingandempowerment,villageinvestmentsandmanagement).Forthevillageinvestments

component,tensetsofoutputindicatorsaremeasuredforcoresectorsincluding:peopleattheprojectlevelwith

improvedwatersupply,constructionofwatersupplypoints,directfemaleandmaleprojectbeneficiaries.

iiiMixedmethodsareusuallyunderstoodtocombinequantitativeandqualitativemethodsfromtwodifferentsocial

sciencedisciplineswhilemulti-methodapproachesinvolvecombiningdifferentresearchandevaluationmethodsfrom

withinthesamediscipline.However,thedistinctionisnotveryclearandsomeoffersprefertocombinethetwo

approaches.

ivOftenmostoftheevaluationbudgetmightbeinvestedinaquantitativesurveysoastoachievemaximumstatistical

power.Anotheroptionmightbetoreducethesamplesizeandtoinvestsomeoftheresourcesinqualitativemethodssuch

asfocusgroups,keyinformantinterviewsandsoforthCombiningtheseindependentestimatesmightproducemore

usefulandmeaningfulresults.However,increasingvalidityrequiresthatQUALdataisselectedtoensureitis

representativeandcanbecomparedwiththeQUANTfindings,

vAnexampleofacommondifferenceiswhenaQUANTsurveyisaskingaboutchangesinincomesincethestartofthe

projectwhileQUALinterviewsarefocusingmoreonfeelingsofeconomicsecurityandvulnerabilitytoeconomiccrises.

Sometimessurveysfindthatincomehasgoneupbutthatmanypeoplefeelmorevulnerableandinsecure.Itoftentakes

theresearcherssometimethatthesequestionsareexploringdifferentissues.vi BetterEvaluation(http://www.betterevaluation.org/en/approach/realist_evaluation)definesagenerativemechanismsasfollows:Strictlyspeaking,theterm‘generativemechanism’referstotheunderlyingsocialorpsychological

driversthat‘cause’thereasoningofactors.Forexample,aparentingskillsprogrammayhaveachieveddifferent

outcomesforfathersandmothers.Themechanismgeneratingdifferent‘reasoning’bymothersandfathersmayrelateto

dominantsocialnormsabouttherolesandresponsibilitiesofmothersandfathers.Additionalmechanismsmaybe

situatedinpsychological,socialorotherspheres.Contextmatters:first,itinfluences‘reasoning’and,secondly,generative

mechanismscanonlyworkifthecircumstancesareright.Goingbacktoourexample,theremaydifferentsocialbeliefs

abouttherolesandresponsibilitiesofmothersandfathersindifferentcultures,whichmayaffecthowparentsrespondto

theparentingprogram.Whetherparentscanputtheirnewlearningintopracticewilldependonarangeoffactors–

perhapsthetimetheyhaveavailable,theirownbeliefsaboutparenting,ortheirmentalhealth.Finally,thecontextmay

providealternativeexplanationsoftheobservedoutcomes,andtheseneedtobetakenintoaccountduringtheanalysis.

viiAtypicalresponseisthatwomenactuallyenjoythistraveltime(andcarryingheavyburdensontheirhead)time

becausetheysingandchatwiththeirfriendsastheywalk.

viiiAfrequentlycitedexampleconcernstheroleofthemother-in-lawonchildnutrition.Themother-in-lawisfrequently

nottargetedinnutritioneducationprograms,butshecanpreventherdaughter-in-lawfromapplyingthelessonlearned

innutritioneducationprograms.

ixAnumberoforganizationssuchasUNWomen,USAID,DIFDandCAREinternationalhavedevelopedchecklistsforstaff

toassesshowwellgenderissueshavebeenaddressedinthedesignandimplementationoftheirprograms.Forexample,

DFIDstaffareaskedquestionssuchas:“Havewecountedallwomenandgirls?”,“Havebothwomenandmenbeen

consulted?”,“Haveweinvestedequallyinwomenandmen?”,“Dowomenandgirlsreceiveafairshare(ofprogram

resources?)”.