Page 1
IOB
Evaluation policy and guidelines for evaluations
IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | P
Evaluation policy and guidelines for evaluations
Page 2
tyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüssstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTeneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP
vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKmiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoProntoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
aramariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PdisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTrpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTngkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHannshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoutiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkoazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYhanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKabuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingP
vivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKrobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPshingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascussinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsselanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskoubatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTtBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsiagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWtáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKiegMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
allinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
Page 3
Evaluation policy and guidelinesfor evaluations
Page 5
3
Evaluation policy and guidelines for evaluations
Contents
1 Introduction 4
2 Short history 6
3 Basic principles of IOB’s evaluation policy 10
4 Definitions and criteria for evaluations 14
4.1 Whatisevaluation? 15
4.2 Typesofevaluation 16
4.3 Evaluationcriteria 17
4.3.1 Efficiency 17
4.3.2 Effectiveness 18
4.3.3 Impact 19
4.3.4 Relevance 19
4.3.5 Sustainability 20
4.3.6 Policyconsistency 21
5 Guiding principles and ethical standards 22
6 The evaluation process 26
6.1 Programming 27
6.2 Preliminarystudy 29
6.3 TermsofReference 30
6.4 Deployingresearchers 32
6.5 Cooperatingwithresearchersinpartnercountries 32
6.6 Methodology 33
6.7 Reporting 37
7 Quality control 40
7.1 Internalpeerreview 41
7.2 Referencegroups 42
7.3 Departmentalmeeting 42
7.4 Advisorypanel 43
7.5 Self-evaluation 43
8 Communication and feedback 44
Page 7
5
Evaluation policy and guidelines for evaluations
ThisdocumentdescribestheevaluationpolicyandguidelinesforevaluationsofthePolicyandOperationsEvaluationDepartment(IOB)oftheDutchMinistryofForeignAffairs.ItplacestheevaluationpolicyinthecontextoftheOrder on PeriodicEvaluation and Policy Information(knownasRPE2006)1,thegovernment-wideframeworkfortheevaluationofgovernmentpolicyingeneral.ItalsoreflectsrecentdecisionsregardingthefunctionofevaluationattheMinistryofForeignAffairs,andtheroleIOBplayswithinit.
IOB’sevaluationpolicyandtheguidelinesmustbeseeninthecontextofachangedenvironment.AnincreasinglycriticaloutsideworlddemandsaclearerinsightintotheresultsoftheNetherlands’foreignpolicy.Thereisgrowingpressureforministerstoaccountnotonlyfortheuseofpublicfunds,butevenmoresofortheresultsachieved.Thisalsoappliestothevariousareasofforeignpolicy.Ifevaluationsaretocontributetothis,theymustbeindependent,public,transparentandaccessible.
Inpublishingtheseguidelines,IOBwishestoprovidetransparencytothirdparties:colleaguesattheMinistryofForeignAffairs,theDutchparliament,organisationsandinstitutionsaffectedbyIOBevaluations,usersoftheevaluationresults,andinterestedcitizens.Ineachevaluationreport,IOBaccountsforthespecificevaluation;thisdocumentclearlypresentsthegeneralprinciplesandguidelinesIOBappliesinternallyinconductingevaluations.
TheguidelinesoutlinedinthisdocumentapplyinprincipletoallevaluationsconductedbyIOB,unlessexplicitlystatedotherwise.ItshouldbeborneinmindthattheNetherlandshasalongtraditionofevaluationinthefieldofdevelopmentcooperation,whichisexpressedthroughinternationalagreements,fora,cooperation,etc.andwhichimpactsonIOB’swork.Thatisnotthecaseforotherareasofforeignpolicy.Becauseitsmandatealsoextendstotheseotherareasofpolicy,IOBisinanexceptionalposition,especiallyinternationally.
TheevaluationpolicyandguidelinesreflecttheopinionsandperspectivesthatunderpinIOB’swork.Inthatsense,thisdocumentdescribestheidealsituationforIOB.Realitywillsometimesdeviatefromthisidealinsomeways.Therewillalwaysbeacertaintensionbetweentheoryandpractice.However,IOBaimstoworkasmuchaspossibleaccordingtotheseguidelines.Itisimportanttorememberthatneitherpolicynorguidelinesarecarvedinstone.Thismustbeseenasagrowthdocument,inthemostliteralsense.Newinsightsmayleadtoitbeingadaptedinthefuture.IOBwillthusperiodicallyreviewitspolicyandtheguidelinesand,ifnecessary,revisethem.Thelatestupdatedversionofthisdocumentwillbeavailableathttp://www.minbuza.nl.
1 Order on Periodic Evaluation and Policy Information 2006, Government Gazette, 26 April 2006, no. 83.
Page 9
7
Evaluation policy and guidelines for evaluations
Inthefirsthalfofthe1970sthebudgetfordevelopmentcooperationincreasedrapidly.Sorapidlythatconcernswerevoicedfromvariousquarters,andespeciallyinthemedia,thatthiswashavinganadverseeffectonthequalityofaid.Asaconsequenceofthisnegativepublicity,theMinisterforDevelopmentCooperationfelttheneedforanindependentopinionondevelopmentpolicy,autonomousofthepolicydepartmentsinvolved.On1July1977theOperationsReviewUnit(IOV)wassetupbyministerialorder.ItwasmadeclearfromthebeginningthatIOVwouldfocusonbilateralcooperation,bothgovernmentalandnon-governmental(insofarasthelatterwasgovernment-funded),butwouldinprinciplerestrictitsactivitiestotheNetherlands’contributiontojointinterventions.Onthebasisofthisrestriction,theministertookthepositionthatIOVcouldconductinspectionswithoutthepermissionoftherecipientcountryororganisation.IOV’sworkprogrammewasdrawnuptogetherwiththepolicydepartmentsandsubmittedtotheministerforapproval.
OutsidetheworkprogrammetheministerandtheministrymanagementwerealsoauthorisedtocommissionIOVtoconductinspectionsonanadhocbasis.However,thisoccurredonlysporadically.FromthestartIOVfocusedonindividualprojects,submittingconfidentialreportstotheminister.Thereportswereconfidentialbecausetheyreferredtoidentifiableeventsandindividuals.
From1979theHouseofRepresentativesoftheDutchparliamentcalledeachyearforIOV’sreportstobemadeaspublicaspossible.Eventually,itwasagreedthatgeneralreportswouldbesenttoparliament,togetherwithsummariesofprojectinspections.ThefirstsummarisedreportthatIOVsenttotheHousewasthe1984GeneralEvaluationofDutchBilateralDevelopmentCooperation.
Inthe1980s,IOV’sareaofworkshiftedfromtheinspectionofindividualprojectstoevaluationsofsectors,policythemesandcountryprogrammes.DuringverbalconsultationswiththeHouseofRepresentativesin1987theministeragreedthatallIOVreportswouldbefullypublicandwouldbesenttoparliament,togetherwithapolicyletterfromtheministercontaininghisorherresponsetothereport.ThereportswereusedactivelyindebatesbymembersofboththeHouseofRepresentativesandtheSenate.Universitiesandotherinstitutionsinthepublicandprivatesectorsalsoshowedalivelyinterestinthereportsoncetheybecameaccessible.
In1996afar-reachingreassessmentofDutchforeignpolicytookplace.Itwaswidelyacknowledgedthatdevelopmentcooperation,foreignpolicyandinternationaltradecouldnolongerbeseenasindependentpolicyareas.Foreignpolicywas‘decompart-mentalised’andthevariousareasofinternationalactivitywereintegrated.Forthepurposeofthereview,IOVpublishedanewsynthesisstudy,summarisingtheresultsoftheevaluationreportsoftheprecedingdecade.Duringthereorganisationthataccompaniedthepolicyreassessment,itwaslogicaltomodifyIOV’smandate.Whereas,beforethereview,IOVwasonlyresponsibleforevaluationsinthefieldofdevelopmentcooperation,itsareaofoperationswasnowextendedtoincludethe
Page 10
8
2 Short history
wholeofforeignpolicy.Thisimpliedthat,inthefuture,theUnitcouldaddressitselftoallsubjectsthatfellundertheHomogenousBudgetforInternationalCooperation(HGIS).ThechangeinIOV’smandateledtoarevisionofitspositionintheorganisationalstructureoftheministry.ItwouldnolongerbeaccountabletotheDirector-GeneralforInternationalCooperation(DGIS)butdirectlytotheSecretary-General.TheUnitwasalsogivenanewname:inthefutureitwouldbeknownasthePolicyandOperationsEvaluationDepartment(IOB).
InrecentyearsIOB’spositionandrolehasonceagainbeenthesubjectofpoliticaldebateanddiscussionwithintheministry.Manyoftheissuesdiscussedinthe1980swereonceagainontheagenda,includingtheindependenceofevaluation,thepublicnatureofthereports,therelationshipbetweenexternalaccountabilityandinternallearning.Thisledin2008totheMinisterforDevelopmentCooperationreconfirmingtoparliamenttheindependenceofIOBandthepublicnatureofitsreports.However,IOB’sprogrammingauthoritywasrestrictedtodevelopmentcooperation.2Intheotherfieldsofforeignpolicy,primaryresponsibilitynowlieswiththeDirector-Generalconcerned,whocanrequestIOBtoconductevaluations.
2 Letter from the Ministers of Foreign Affairs and for Development Cooperation of 16 October 2007 on implementation of the Boekestijn/Gill’ard motion regarding the Policy and Operations Evaluation Department (IOB) (31200V, no. 9). Report from the Permanent Committee on Foreign Affairs on IOB of 29 May 2008, approved on 25 June 2008.
Page 11
9
Evaluation policy and guidelines for evaluations
Page 12
3Basic principles of IOB’s evaluation policy
Page 13
11
Evaluation policy and guidelines for evaluations
ThePolicyandOperationsEvaluationDepartmentistheevaluationdepartmentoftheMinistryofForeignAffairs.Institutionally,IOBisdirectlyaccountabletotheSecretary-General.Itformspartoftheministry’sAuditCommittee.TogetherwiththeAuditDepartmentandtheInspectionandEvaluationUnit(ISB),IOBformsthecluster‘InControl’.
Internationally,IOBisamemberoftheOECD/DACnetworkofevaluationservices.ItalsotakespartinconsultationsbetweentheevaluationservicesoftheEuropeanUnion,andofthecountriesunitedintheNordic+group.IOBcontributestotheactivitiesofanumberofotherformsofinternationalcooperation,includingtheconsultationcommitteeofGerman-speakingevaluationservices(DACH),theNetworkofNetworksforImpactEvaluation(NONIE)andtheInternationalInitiativeonImpactEvaluation(3IE).
IOB’smandateisasfollows:
The Policy and Operations Evaluation Department meets the need for evaluation of policy and its implementation with regard to all policy areas that are financed by the homogenous budget for international cooperation (HGIS). IOB evaluates the implementation of the Netherlands’ foreign policy, giving particular attention to its relevance, efficiency and effectiveness, the sustainability of the results achieved and the consistency and coherence of the policy.
Inthelightofthismandate,IOBhasthefollowingtasks:
1)toconducthighqualityevaluationsforlearningandaccountabilitypurposes;2)toensurebalancedevaluationprogramminginthefieldofdevelopmentcooperation
andtoadviseonprogramminginotherareasofforeignpolicy;3)tocontributetoimprovementofthequalityofevaluationsconductedby,oronbehalf
of,policydepartments,institutionssubsidisedbytheministry,andmultilateralinstitutionsbydrawingupgenerallyapplicablequalityrequirementsandguidelinesforconductingevaluations,advisingonthedesignandimplementationofindividualevaluations,andconductingsystematicassessmentsoftheseevaluationsorpeerreviewsofotherevaluationservices;
4)tostrengthencapacitytoconductevaluationsinpartnercountriesbysupportingeducationandtrainingandparticipatinginjointevaluations;
5)topromotethedistributionandcommunicationofevaluationresults.
Re1:IOB’smaintaskistoconducthigh-qualityevaluationswithintheframeworkoftheMinistryofForeignAffairs’evaluationprogramme.TheseevaluationscanrelatetoallareasofforeignpolicythatfallundertheHGIS.Inthiscontext,evaluationcomprisescriticalandthoroughinvestigationofapolicyorprogrammeandexplainingtheresults.IOBevaluationsdonotstandinisolationbutareinprinciplepartofthepolicycycle,inwhichplanning,monitoringandevaluationareseenasanintegratedwhole.
Page 14
12
3 Basic principles of IOB’s evaluation policy
Evaluationisthereforeanessentialcomponentofthe‘knowledgefunction’withintheministry.Knowledgeontheimplementationandimpactofpolicysupportstwoconcretefunctions:
• itprovidesthebasisforaccountingtoparliamentandsocietyatlargeforpolicyandexpenditure(accountabilityfunction);
• itgeneratesinformationforthepurposesofimprovingpolicyformulationandimplementation(learningfunction).
Re2:AsstatedintheRPE2006systematicevaluationmustbeafixedcomponentofeverypolicyprocess.Eachpolicyareamustbeperiodicallyevaluatedatthelevelofoperationalobjectives.TheRPE2006thusformsthebasisofevaluationprogramming.Theprogrammingisalsodeterminedbyspecificrequestsfromparliament,ministersandstatesecretaries,andbudgetholders.WithintheMinistryofForeignAffairs,IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperationandadvisesonprogramminginotherareasofforeignpolicy.
Re3:IOBisnottheonlyunitattheMinistryofForeignAffairsconcernedwithevaluation.Policyadvisorsalsohavetodealwithevaluations,asthecontactforevaluationprogrammingwithintheirpolicydepartment,asa‘client’commissioninganevaluation,sometimesas(joint)implementer,orasauseroftheresults.Inadditiontoconductingevaluations,IOBthereforehasanadvisoryfunction.ThisentailsformulatingqualitycriteriaandguidelinesforimplementingevaluationsandadvisingontheTermsofReferenceforevaluationsthatIOBdoesnotconductitself.Duringtheannualsettlementoftheprogramme,agreementsaremaderegardingtheextenttowhichIOBwillplayanadvisoryroleinevaluationsconductedbyoronbehalfofministrydepartments.IOBalsoadvisesonthedesignoftheministry’sevaluationfunction,theevaluationsystemandinstruments,andcoordinationwithplanningandmonitoring.InadditionIOBperformsqualitytestsonprogrammeevaluationsconductedbycofinancednon-governmentalorganisations(MFS),withtheaimofstrengtheningtheirevaluationcapacity.
ForthemultilateralchannelIOBisinvolvedininternationalreviewsoftheevaluationservicesofrelevantinstitutions.
Re4:Strengtheningevaluationcapacityinpartnercountriesisanimportantthemeindevelop-mentcooperation.Independentevaluationcapacityinpartnercountriesisindispensableiftheaimofestablishingasystembasedonmutualaccountabilityistobeachievedinreality.IOBseesitasitstasktocontributetothisbysupportinginternationalinitiativesaimedatstrengtheningevaluationcapacity.Inaddition,wherepossibleandsafeguardingitsowncriteriaforindependenceandquality,IOBworkstogetherwithresearchersfrompartnercountriesinconductingevaluations.
Page 15
13
Evaluation policy and guidelines for evaluations
Ad5:Evaluationsgenerateknowledgeonthebasisofexperiencefromthepastregardingwhatworksandwhatdoesnot,andespeciallywhy.IOBconsidersitimportantthattheresultsofevaluationsaredistributedaswidelyaspossible.Toachievethis,itdiscussesthefindingsinstaffmeetings,withministersandstatesecretaries,andwithothernationalandinternationalpartiesinvolvedintheevaluations.Inaddition,wherepossibleandwhereitdoesnotconflictwithitsownresponsibilities,IOBwilloffersupportinpreparingpolicyresponsesanddescriptionsofevaluationfindingsintheMinistryofForeignAffairs’AnnualReports.
Page 16
4Definitions and criteria for evaluations
Page 17
15
Evaluation policy and guidelines for evaluations
4.1 What is evaluation?
Thereisawidevarietyofdefinitionsoftheconceptofevaluation.Inthe1980sand1990s,theOECD/DACworkedonasharedframeworkofconceptsforevaluationpracticesinthefieldofdevelopmentcooperation.TheresultingdefinitionofevaluationservesasabasisforIOB’swork.
‘Anevaluationisanassessment,assystematicandobjectiveaspossible,ofanongoingorcompletedproject,programmeorpolicy,itsdesign,implementationandresults.Theaimistodeterminetherelevanceandfulfilmentofobjectives,developmentalefficiency,effectiveness,impactandsustainability.Anevaluationshouldprovideinformationthatiscredibleanduseful,enablingtheincorporationoflessonslearnedintothedecision-makingprocessofbothrecipientsanddonors.’3
TheOECD/DACdefinitionreflectsanunderstandingofevaluationthatisbroadlysupportedintheinternationalcommunityofevaluators.Thedefinitionisintendedforevaluationsinthefieldofdevelopmentcooperation.SinceIOB’sworkextendsfurtherthandevelopmentcooperation,conceptslike‘developmental’and‘recipientsanddonors’willnotapplytoallIOBevaluations.
Evaluationisdifferenttoreviewsandmonitoring.TheOECD/DACGlossarydescribesareviewasanassessmentoftheperformanceofanintervention,periodicallyoronanadhocbasis.ForIOBthemaindifferencebetweenareviewandanevaluationisthatareviewisconsideredamanagementinstrumentforoperationalmonitoringoftheimplementationofanactivity,andthereforefocusesonoperationalaspectsandrecordingresults,whileevaluationisamorein-depthassessmentoftheaims,implementationandresultsofanintervention,withthequestionofattributionasitsmainfocus.
Monitoringisaninstrumentofinternalmanagement.Itaims,onthebasisofafixedsetofindicators,toprovidearegularinsightintotheprogressmadeinimplementingactivities.Monitoringallowsabudgetholderortheimplementerofaninterventiontostayuptodateoninputssuppliedandactivitiesperformed(outputs).Monitoringshowswhetherintendedactivitieshaveactuallybeencarriedout.Good,systematicmonitoringisofgreatimportanceforevaluations,asitprovidesasignificantpartofthedataonwhichevaluationisbased.
3 DAC principles for Aid Evaluation, OECD/DAC, 1991 (the definition was also included in the DAC Glossary
of Key Terms in Evaluation and Results-based Management in 2002).
Page 18
16
4 Definitions and criteria for evaluations
4.2 Types of evaluation
TheOrderonPeriodicEvaluationandPolicyInformation2006(RPE2006)prescribesasystemofevaluationbasedontheso-calledoperationalobjectives,OD.Thesystementailsperiodicalevaluationoftheseoperationalobjectives.
TheRPE2006distinguishesbetweenpolicyreviewsandex-postimpactstudies.Bothconceptsdeviatefromtheevaluationterminologyusedinternationally,butaretheequivalentofthemorecommonterms‘policyevaluation’and‘impactevaluation’.Thestartingpointisthatpolicyreviews,whicharelinkedtotheoperationalobjectives,areasfaraspossiblefedbyimpactstudies.Inprincipletheyresemblesynthesisstudies.IOBgivesprioritytothesetypesofevaluation,where–especiallyinthefieldofdevelopmentcooperation–policyreviewsareprecededbyaseriesofex-postimpactstudies(impactevaluations).
OnthebasisofthesystemprescribedintheRPE2006,IOBimpactevaluationswillgenerallyfocusoninterventions(rangingfromprogrammestovariousformsofbudgetsupport).Policyevaluations(orreviews)addressthemesorsectors.Inaddition,integratedforeignpolicycanalsogiveriseto‘decompartmentalised’evaluationsofcountryorregionalpolicies.
Besidesthese–usuallyextensive–evaluations,IOBcanalsoconductshortinternalevaluationsthatarelimitedinnatureandnecessarytomonitortheprogressofprogrammes.
PartoftheevaluationsinIOB’sprogrammingfocusessolelyonDutchprogrammesandactivities.ItislogicalfortheseevaluationstobeconductedbyIOB.However,itisbecomingincreasinglycommontoworktogetherwithinternationalpartnersinthefieldofdevelopmentcooperation.Projectsbyindividualdonorsaregraduallybeingreplacedbyjointfinancingofgovernmentpolicyinpartnercountries.Itisthenalsologicalthatdonorsgivegreaterpreferencetothejointprogrammingandimplementationofevaluations.Incaseswheredonorsworktogether,forexampleinprovidingjointsupportforsectorpolicyorotherfar-reachingformsofjointfinancing(suchasbasket-funding),IOBevaluationswillinprinciplebeconductedincooperationwithotherdonors,aslongasIOB’scriteriaforindependenceandqualityaremet.Thenatureofthecooperationcanvaryfromactualparticipationbyseveraldonorsinanevaluation,agreeingonadivisionoftaskswithinasharedTermsofReference,toagreementonthedivisionoflabourbetweendifferentthemes,sectorsorcountries.
Page 19
17
Evaluation policy and guidelines for evaluations
4.3 Evaluation criteria
Evaluationcallsforatailor-madeapproach.Thatdoesnotmean,however,ignoringqualitycriteriaorabandoningconsistencyinapproach.Inallevaluations,theevaluationcriteriamustbeappliedinanunambiguousand,aboveall,transparentway.Here,inparticular,standardisationisparamounttoenableevaluationsandtheirresultstobecompared.TheevaluationcriteriaappliedbyIOBarelargelyidenticaltothosedrawnupbytheOECD/DAC.TheOECD/DACdefinitionofevaluationspecifiesfivecriteriaforevaluationofdevelopmentinterventions:efficiency,effectiveness,impact,relevanceandsustainability.Theevaluationcriteriarefertodifferentresultlevelsfoundedonthelogicalframeworkorresultchainsbasedonit.Together,thesecriteriaprovideaframeworkwithinwhichresearcherscanpresenttheirfindingsinanorderedandcoherentway.Theextenttowhichequalattentioncanorwillbepaidtoallcriteriawillvaryperevaluation.Furthermore,additionalcriteriacanbeapplied:evaluationof‘decompartmentalised’policy,forexample,mayrequirethatIOBdevotesexplicitattentiontopolicyconsistency.
4.3.1 Efficiency Efficiencymeasureshoweconomicallyresources(inputs)andthewaytheyareappliedareconvertedtodirectresults.Theconceptthereforeillustratestherelationshipbetweeninputandoutputandreferstoalevelofresultsthattheimplementercancheck.
Inevaluatingefficiency,boththequantityandqualityofinputsandoutputsareassessed.Themosteconomicalorcost-effectiveinputisnotalwaysthemostappropriateandtherecanbesignificanttradeoffsbetweenthequantityandqualityofoutputs.Assessingefficiencyalsocallsforcomparisonwithalternativeapproaches,withwhichthesameoutputscanbeachieved.Thisisbynomeansalwayseasybecausethereareoftennopreviouslyformulatedbenchmarksandtheymighthavetobedevelopedonthebasisofpracticalexperience.Thatismorelikelyinthecaseof,forexample,interventionsinvolvingconstructionactivitiesthanofthoseaimingtobringaboutsocialorinstitutionalchange.
Thefollowingelementsshouldbetakenintoaccountwhenevaluatingefficiency:
• Haveappropriateinputsbeendeployedatthelowestpossiblecost?• Haveactivitiesbeenconductedinanuncomplicatedmanner?• Havedecisionsbeenmadeattherightlevelandhasbureaucracybeenavoided
asfaraspossible?• Haveoverheadcostsbeenkepttoaminimum?• Hasduplicationbeenavoided?• Haveconflictsduringimplementationbeenpreventedorsolved?• Haveoutputsbeenachievedwithintheplannedperiodandbudget?
Page 20
18
4 Definitions and criteria for evaluations
4.3.2 Effectiveness Effectivenessrelatestotheextenttowhichthedirectresultsofinterventions(output)contributetothesustainableachievementofpolicyobjectives(outcome).Policyobjectivesaretheobjectivesthatinterventionsareintendedtoachieveandshould,inprinciple,beformulatedintheprojectdocument.Insomecasespolicyobjectivescanbeidenticaltotheoperationalobjectivesspecifiedinthebudgetorspecificcomponentsofthem.
Aninterventionisconsideredeffectiveifitsoutputshavemadeademonstrablecontributiontoachievementoftheintervention’sintendedobjectives.Efficiencyandeffectivenessrefertotwosuccessivelevelsintheresultschain.Apositiveassessmentatoneleveldoesnotautomaticallyimplyapositiveassessmentattheother.Aninterventioncanbeefficientwithouthavinganeffect,andaneffectiveinterventioncanbeinefficientlyimplemented.
Theeffectsofaninterventionarebeyondthedirectcontroloftheimplementer.Interventionsgenerateoutputswhichareassumedtogeneratethedesiredeffects.Evaluatingeffectivenessaimstoestablishthiscausality.Thecoreproblemisthatitwouldbeidealtoknowwhatwouldhavehappenediftheinterventionhadnotoccurred(thecounterfactual).Onlythencanobservedchangesbeattributedtotheinterventionandisitpossibletospeakoftheneteffects.Inotherwords,thereisaproblemofattribution.Aslongasotherfactorshaveavisibleimpactontheobservedchanges,theproblemisrelativelysimpletosolve.Itismoredifficultwheninfluentialfactorsarenotimmediatelyvisible(the‘unobservables’).Researcherscannotknowwhattheydonotknow.Evaluationstrytofindasolutiontothisproblem(see6.6).
Inpractice–especiallyinthecaseoflargeandcomplexinterventions,butalsowithinterventionsthataimtobringabouteffectsthataredifficultorimpossibletoquantify–evaluationsoftenmakeuseofplausibilityargumentation.Itisthenimportantthattheargumentistransparentandcredible.
Inessence,threestepscanbedistinguishedintheevaluationofeffectiveness:
• Themeasurementofchangesintheeffectvariablesincomparisontothesituationatthestart(baseline);
• Attributionoftheobservedchangestotheintervention;• Assessmentofchangesobservedandattributedtotheinterventionintermsofthe
objectives.
Page 21
19
Evaluation policy and guidelines for evaluations
4.3.3 Impact Thebroadconsensusinthecommunityofevaluatorsabouttheconceptsofefficiencyandeffectivenessislessevidentwhenitcomestoimpactandrelevance.
TheOECD/DACdefinesimpactasallsignificanteffectsproducedbyadevelopmentintervention,directlyorindirectly,intendedorunintended,ontheultimatestakeholdersandthirdparties.Whileeffectivenessrefersonlytotheintendedeffectsofanactivity,inthisterminologyimpactreferstoallpossibleeconomic,social,political,technicalandecologicaleffectsatlocal,regionalornationallevelthathaveadirectorindirecteffectonthetargetgrouporotherparties.
Inthestrictercontextofthelogicalframework,theconceptofimpactusuallyreferstothehighestlevelofresults.Thequestionisthentowhatextent,withtheaidofoutputs,theeffectsachieved(outcomes)havecontributedtobroader,morefar-reachingobjectives.Thesecanbeoperationalobjectivesorobjectivesatahigherlevel,suchasmoregeneralpovertyreduction.
Intheinternationalliteratureonimpactevaluations,theconceptofimpactisusedfortheneteffectsattributedtotheintervention,i.e.atthelevelofeffectiveness.
Oneinterpretationofimpactisnotbydefinitionbetterthananother.Itisimportantthatconceptsaremadeexplicitinevaluationreports.Intheinterestofmutualcomparisonitisdesirableinex-postimpactstudiestousetheinterpretationofimpactevaluationsmostcommonlyappliedintheinternationalliterature.InthisrespectIOBfollowstheinstructionsintheRPE2006.
4.3.4 Relevance Theconceptofrelevanceisalsosubjecttodifferentinterpretations.TheOECD/DACdefinesrelevanceastheextenttowhichtheobjectivesofadevelopmentinterventionareconsistentwithbeneficiaries’requirements,countryneeds,globalprioritiesandpartners’anddonors’policies.Thatmakesrelevanceacontainerconcept,withinwhichnochoicesaremade.Thisdefinitionisunsatisfactoryforseveralreasons.First,theassessmentofwhetheraninterventionisalogicalconsequenceoforiscompatiblewithpolicyisinessenceanexantequestionthatshouldbepartoftheapprovalprocedure.Second,donorpolicy,thepartner’spolicyandtheneedsofthetargetgroupdonotnecessarilyallperfectlycoincide.Thatwouldimplythataninterventioncanberelevantandirrelevantatthesametime.Third,policyprioritiescanchangeovertime,suchthataspecificinterventionisrelevantatonemomentandnotthenext,orviceversa.Lastly,thisdefinitionimpliesthattherelevanceofthepolicyitselfisnotupfordiscussion.Inotherwords,itexcludesproblemswithpolicyitself.
Inthelogicalframework,relevancereferstotheextenttowhichtheeffectsofinterventionsmakeasustainablecontributiontoachievingtheultimateobjective(theimpact).Aninterventionhasbeenvaluable,orrelevant,totheextentthatithas
Page 22
20
4 Definitions and criteria for evaluations
generatedeffectsthatbringachievementoftheultimatedevelopmentobjectivecloser.IOBwillapplytheconceptofrelevanceinitsevaluationsinthisrespect.
4.3.5 Sustainability Sinceanactivitycanhardlybeconsideredeffectiveiftheeffectithasachievedisnotlasting,sustainabilityisinfactanaspectofeffectiveness.4Atthesametimetheconceptofsustainabilitycomprisesavarietyofdimensionsthatarerelevantforassessinganintervention.Inthedonorcommunitythesustainabilityofinterventionsislinkedtosevenfactors,whichcanbeusedtoanalyseinterventions.Whichfactorsorcombinationsoffactorsareusefulforassessingsustainabilitywilldependonthenatureoftheintervention.
• Theinvolvementoftherecipientisusuallyrecognisedasafactorthataffectsthesuccessofanintervention.Involvementisexpressedthroughagreementonobjectivesandtheprovisionofhumanandmaterialresources;
• Whendeployingtechnologyitisimportanttotakeaccountofthefinancialandinstitutionalcapacitiesoftherecipientandthedegreetowhichtheconditionsformaintenanceandrenewalareguaranteed;
• Interventionscanhaveaneffectontheenvironmentintheshortorlongterm.Whileshort-termeffectsareusuallyenvisagedandarepreventedormitigated,itisespeciallyimportanttoincorporatelong-termenvironmentaleffectsintheassessment,astheycanneutralisepositivechangesorhaveotherharmfulconsequences;
• Thedegreetowhichchangesaresustainabledependsverystronglyonsocialandculturalfactors,whichareoftennotdirectlyvisible.Studyingperceptionscanprovidebetterinsightintothisdimensionofsustainability;
• Institutionalandcapacitydevelopmentplayanimportantroleindeterminingsustainability.Whenassessinginterventionstheattentiondevotedtotheseaspectswillhelpdeterminetheextenttowhichsustainabilityisachieved;
• Financialandeconomicsustainabilityreferstothedegreetowhichmeasureshavebeentakentoguaranteethatactivitiescancontinueandcompletedworkscanbemaintainedinthefuture;
• Interventionsalwaystakeplaceinawidernatural,political,economicandsocio-culturalenvironment,whichcannotbeinfluencedinanywayatallbutwhichitselfhasastronginfluenceoneffectivenessandsustainability.Changesinpoliticalstability,economiccrises,ornaturaldisasterscanhaveadecisiveeffectonthesustainabilityofinterventions.
4 In the context of humanitarian aid this is, of course, different. It usually entails short-term,
punctual interventions, the effects of which it is hoped will only be necessary for a brief period.
Page 23
21
Evaluation policy and guidelines for evaluations
4.3.6 Policy consistency Thisconceptreferstotheabsenceofinherentcontradictionsbetweendifferentpolicyareas.ConsistencyisparticularlyimportantforIOBintheevaluationofdecompartmentalisedpolicy.Developmentobjectivescanconflictwithinternationaltradeinterestsorforeignpolicyconsiderations.Thatiswhyitisimportant,particularlyincaseswhereobjectiveshavetobeachievedinthecontextofdecompartmentalisedpolicy,todeterminetheextenttowhichinterventionsperformedfromdifferentperspectivesarecoherentwithanddonotobstructeachother.Theaimistohelpachieveintendedobjectiveswithouttheeffectsofoneinterventionneutralisingthoseofanother.
Page 24
5Guiding principles and ethical standards
Page 25
23
Evaluation policy and guidelines for evaluations
IOB’sguidingprinciplesreflectthecorevaluesonwhichitsevaluationsarefounded.Theyareinterrelatedandmutuallyreinforcingand,assuch,theymustbeseenasprovidinganoverallframework,withwhichallaspectsofIOB’sevaluationworkmustbeconsistent.Theprinciplesdeterminewhatcanbeexpectedfromaninspectorandotherresearchersinvolvedinanevaluation,fromevaluationprocessesandproducts,andfromthemethodology.TheyareconsistentwiththeOECD/DAC’sPrinciplesforEvaluationofDevelopmentAssistance(1991)andEvaluationStandards(draft2009).
AllIOBevaluationsmustbeconsistentwiththefollowingguidingprinciples:
Independence Thejudgementsoftheresearchermustnotbeinfluencedbyexternalpressureoraconflictofinterests.Organisationsorinstitutionsthatarecommissionedtoperformanevaluationandmembersofevaluationteamsmaynothavebeenpreviouslyinvolvedintheinterventionsthataretobeevaluatedorinformulationofthepolicyonwhichtheyarebased.
Objectivity Theevaluationmustbebasedonverifiablefacts.Theresearchermustmakeeveryefforttoensurethatthedataonwhichtheevaluationisbaseddoesnotcontainanyinaccura-cies.Thepresentationoffactsshouldbeclearlyandrecognisablydistinguishedfromjudgments.
Transparency Theevaluationmustbeconductedinawaythatcanbefollowedclearlybystakeholdersandthirdparties.Thequestionstobeaddressed,thedatabase,theresearchmethods,findingsandconclusionsmustbepresentedinthereportinaclearandaccessibleway.
Validity TheevaluationmustmeasurewhattheTermsofReferencespecifiesasrequiringmeasurement,andmustdothisinawaythatthereadercanunderstand.
Reliability Theresultsoftheevaluationshouldbestable,i.e.thatarepeatoftheevaluationwillproducethesameresultsandconclusions.
PartnershipInthecaseofevaluationsinthefieldofdevelopmentcooperation,wherepossibleandinsofarasitdoesnotconflictwithotherprinciples,partnersfromtherecipientcountrieswillbeinvolvedintheimplementationofevaluations.Everyeffortwillbemadetocooperatewithindependentresearchersfromthepartnercountries.
Page 26
24
5 Guiding principles and ethical standards
Feasibility Evaluationsareonlyconductedifthedatabaseisaccessibleandtheassociatedmethodologyisavailable.
Correctness Theevaluationmustnotcauseharmtoindividuals.
Cost effectiveness Theevaluationmustbeconductedatthelowestpossibleprice,takingaccountoftheotherguidingprinciples.
Usability Theevaluationmustresultinconclusionsandpointsforattentionthatarevaluablefortheuseroftheevaluationbysuggestingactionsthatliewithintheuser’scapacity.
Evaluationisaformofresearchthatentailsajudgment.Itdiffersfrompureacademicresearchinthatitalwaysresultsinanopinionontheresultsofconcretehumanactions.Researchersthereforehaveagreatresponsibilitytowardsstakeholdersinvolvedintheevaluation.ThisappliesequallytoIOBinspectorsandtoexternalresearcherscontractedbythem.Researcherscanthereforebeexpectedtoadheretointernationallyacceptedethicalstandardsforevaluations.
ThefollowingethicalstandardsapplytoIOB’swork5:
Respect and understanding for other cultures Manyevaluationsareconductedinadifferentculturalenvironmenttothatoftheresearchers.Theycanthereforebeexpectedtocomplywithlocalculturalcodesduringinterviewsandotherformsofdatacollection.
Universal values Someculturalcustomsandpracticesmaybeincompatiblewithfundamentalrightsandfreedoms.Althoughresearchersshouldinprincipleshowrespectforothercultures,theymustnotignoretheeffectofcertainculturalvaluesongenderrelationsorminoritiesandotherspecificgroups.InsuchcasestheyshouldactinaccordancewiththeUniversalDeclarationofHumanRightsandtheassociatedinternationalagreements.
5 The standards are largely based on the evaluation policies of the Danish International
Development Agency (DANIDA) and the Canadian International Development Agency (CIDA).
Page 27
25
Evaluation policy and guidelines for evaluations
Gender Evaluationsshouldbecarriedouttakingaccountofthedifferentrolesperformedbymenandwomenandtheaimofachievinggenderequality.Inpolicyareaswheregenderdifferencesareadimensionoreffectofthepolicypursued,theyshouldbetakenintoaccountinthedesignandimplementationoftheevaluation.
Anonymity and confidentiality Researchersareexpectedtorespectrequestsfrominformerstotreattheinformationtheyprovidewithconfidentialityandtotakestepstoensurethatconfidentialinforma-tioncannotbetracedbacktothesource.
Responsibility Inmanycountriesacriticalstanceregardingthegovernmentcanhavesevereconse-quencesforresearchersfromthecountryitself.Inevaluationsinvolvinglocalresear-chersitisimportantthatsuchrisksberecognisedintimeandstepstakentoavoidthem,forexample,bymakingclearagreementsaboutwhoisresponsibleforthecontentofthereport.
Omissions Researchersmayobtaininformationthatisnotdirectlyimportantinansweringthequestionsaddressedintheevaluation,butisneverthelessrelevantinthelightoftheaccountabilityandlearningfunctionsoftheevaluation.Insuchcases,theclient(inthelastinstance,theIOBDirector)willbeconsultedtodeterminewhetherthisinformationshallbeincludedinthereportandinwhatform.
Assessment of individuals Evaluationsarenotassessmentsofindividuals.Yetevaluationreportscanincludefindingsonleadershipandthemanagementorotherqualitiesofimplementersthatcomeclosetopersonalassessments.Researchersshouldthenmakeeveryefforttopresentthefactsofeasilytraceablecasesinanabstractform.Ifindoubt,theyshouldconsultwiththeIOBDirector.
Evidence of malpractice Evaluationsarenotinvestigationsintolegitimacy.Neverthelessitispossiblethatmalpracticescometolightduringevaluations.InthatcasetheresearchershouldsubmitthecasetotheIOBDirector.Ifthereissufficientreasontodoso,theAuditDepartment(ACD)willbeinformed.
Page 28
6The evaluation process
Page 29
27
Evaluation policy and guidelines for evaluations
6.1 Programming
Thequalityofevaluationbenefitsfromacarefulandtransparentdecision-makingprocess.Thisprocessstartswiththeprogrammingofevaluations.IOBhasfinalresponsibilityforprogrammingevaluationsinthefieldofdevelopmentcooperation.ThatmeansthattheIOBDirectorapprovestheprogrammingafterithasbeendiscussedwiththeFinancialandEconomicAffairsDepartment(FEZ)andtheotherpolicydepartmentsinvolved.Withrespecttotheotherareasofforeignpolicy,IOBadvisesontheprogramming,whichisapprovedinthemeetingoftheSecretary-GeneralandtheDirectors-General.IOBappliesthebasicprincipleslaiddownintheRPE2006forsystematisationoftheprogramming.Theevaluationprogrammingdistinguishesbetweenthreetypesofevaluation:policyreviews(policyevaluations),ex-postimpactstudies(impactevaluations)andotherevaluations.Policyreviewsandex-postimpactstudiesfallundertheRPE2006.TheRPEprescribesthateachpolicyareamustbeperiodicallysubjectedtoapolicyrevieworpolicyevaluationatthelevelofoperationalobjectives.Apolicyreviewshouldmakeasmuchuseaspossibleofex-postimpactstudies.PolicyreviewsandimpactstudieslargelydeterminetheIOB’sresearchagenda.
InadditiontotheRPE2006therecanbeotherreasonsforincludingevaluationsintheprogramming.
First,therearepoliticalconsiderations.Atcertaintimespolicyissuescanbethesubjectofpoliticalorsocialdebate.Itisthenimportanttofeedthedebatewithinsightsobtainedthroughthoroughandindependentevaluation.
Policyrelevancecanalsoplayarole.Thisisdeterminedbytheprioritygiventoanissueandthemomentinwhichtheevaluationiscarriedout.Ifitisclearwhenresultsarerequiredfortheformulationofnewprogrammesorpoliciesandanevaluationcanbeconductedwithinthatperiodwithoutlossofquality,IOBensuresthattheevaluationisprogrammedandimplementedintime,irrespectiveofthescaleorcomplexityofthesubject.
Inthefieldofdevelopmentcooperationthereisinternationalpressuretoimprovecoordinationbetweentheevaluationprogrammingofindividualdonorsandtoconductjointevaluationsasfaraspossible.Jointevaluationsincludethosecarriedouttogetherwithevaluationservicesintherecipientcountries,multidonorevaluations,andvariouscombinationsofthetwo.Workingtogetherwithevaluationservicesinrecipientcountrieshelpsstrengthentheirevaluationcapacityandimprovemutualaccountability.ThechoiceofwhichjointevaluationsIOBparticipatesinwilldependoninitiativesdevelopedduringinternationalconsultations,opportunitiestoenterintostrategicallianceswithlike-mindedevaluationservices,theextenttowhichIOB’scriteriaforindependenceandqualityareguaranteed,andwhethertheycanbefittedintoIOB’sexistingprogramming.
Page 30
28
6 The evaluation process
IOBalsoconsidersitimportanttoallowspaceinitsprogrammingforadhocrequests.ThesemayberequestsfromwithintheministryforIOBtoconductshortevaluationsofalimitednaturetodeterminetheprogressofprogrammesortosupportthedecision-makingprocess.Inaddition,ministers,statesecretariesorparliamentcanrequestIOBtoconductspecificevaluations.IOBmakeseveryefforttocomplywiththeserequests.
Theevaluationprogramminginthefieldofdevelopmentcooperationrelatesnotonlytobilateralcooperationbutalsotothenon-governmentalandmultilateralchannels.Inthecontextofthecofinancingsystem(MFS)IOBconductssystematicassessmentsofprogrammeevaluationscarriedoutundertheresponsibilityoftheco-financednon-governmentalorganisationsthemselves.Inaddition,IOBselectsanumberofthemesorcountriestoconductitsownevaluations.Todate,IOBhasbeensomewhatreservedinprogrammingevaluationsofmultilateralcooperation.Togetherwithotherdonors,IOBhasencouragedmultilateralinstitutionstoimprovethequalityoftheirownevaluationsthroughpeerreviews.Inthefuture,wherepossible,IOBwillparticipateinjointdonorinitiativesaimedatamoresystematicevaluationofmultilateraldevelopmentcooperation.
ThedecompartmentalisationofforeignpolicyandthesubsequentbroadeningofIOB’smandatemeanthatIOBevaluatesnotonlyinterventionsinthefieldofdevelopmentcooperation,butalsoactivitiesinotherareasofforeignpolicy.Itisnotunusualforotherforeignpolicyinstrumentstobedeployedtoaddressaspecifictheme,alongsidedevelopmentcooperation.Thiscanbegoodreasontoconducta‘decompartmentalised’evaluation.Inmanycases,onlytheMinistryofForeignAffairsisinvolvedinsuchanevaluation,butnotalways.Therearealsoissuesofforeignpolicyinwhichotherministriesplayaprominentrole.ForIOBthatisreasontoexplorethepossibilitiesforconductingevaluationsjointlywithotherministries.Insuchcases,andifIOB’sconditionsregardingindependenceandqualityaremet,theseevaluationswillbeincludedintheprogramming.
Page 31
29
Evaluation policy and guidelines for evaluations
6.2 Preliminary study
Onceasubjectforevaluationhasbeenincludedintheprogrammingapreliminarystudycanbeinitiated.Especiallyinthisinitialphase,anumberofimportantdecisionshavetobemadethatwillhavefar-reachingsignificanceforthefurtherprogressoftheevaluation.Theinspectorwillhavetobecomefamiliarwiththepolicyareatobeevaluatedandtheavailablesourcesofinformation,andformsomeideaofthequestionstobeaddressedintheevaluationandthemostadequateapproach.Withlarge,complexevaluationstheinspectorcandrawupastart-upreport.Thereportdescribesthereasonfortheevaluationandtheobjective.Itcanalsogiveabriefdescriptionofthepolicyareatobeevaluatedandformulatecorequestionsforthepreliminarystudy.Itcansuggestwhichtypeofevaluationwouldbemostappropriateandhowitmightbeorganised(theexpectedduration,whetherfieldmissionsareconsiderednecessary,andwhenandhowexternalconsultantsmightbeneeded).Lastly,thestart-upreportcanofferaninsightintotheexpectedproductsoftheevaluationandthebudget.
Start-upreportsarediscussedwiththeIOBmanagementandtheinternalpeerreviewersdesignatedfortheevaluation(see7.1).Afterdecisionshavebeenmadeontheabovementionedmainpoints,thepreliminarystudywillmoveontodraftingdefinitiveTermsofReferencefortheactualevaluation.
Thepreliminarystudycomprisesanumberofactivities:
Reconstruction of policy theory and intervention logic Theextenttowhichasubjectcanbeevaluatedisimprovedbyclearlyformulatedpolicyprinciplesandprogrammeobjectives,adescriptionoftheactivitiesundertaken,andalogicalrelationshipbetweenpolicy,objectivesandactivities.Practiceshowsthatthesecriteriaarebynomeansalwaysmet.Generallyspeaking,thisappliestobothdevelopmentcooperationandotherareasofforeignpolicy.Especiallyatthelevelofoperationalobjectives,policyisoftenbroad-rangingandnotveryspecificallyformulated,aimsandeffectsarefrequentlyunclearandthelogicofinterventionsisnotsufficientlyexplicit.Insuchcases,onthebasisofthepreliminarystudy,inspectorsthemselveswilldevelopapolicytheoryandareconstructedinterventionlogicbasedonit.Theywillthenpresentthetheoryandthelogictothepolicymakertodeterminewhethertheyareinlinewithwhatthelatterhadinmind.
Defining the evaluation objectPolicythemesandsectorsareoftenbroadlyformulatedandcoverawidevarietyofactivities.Thisappliesinparticularattheleveloftheoperationalobjectivesspecifiedintheministry‘sbudgetdocument.Thepreliminarystudyshouldthereforeestablishthelimitsoftheevaluationobject–alsointhecaseofwide-rangingevaluations–suchthatithasaclearfocusandthusremainsfeasible.Thisoffersabetterguaranteethatthefocusoftheevaluationremainsclearandthatonlythosequestionsareformulatedthat
Page 32
30
6 The evaluation process
canreasonablybeexpectedtobegivenavalidanswer.Establishingthelimitsofanevaluationrelatestothetimedimension,thegeographicaldimensionandthenatureoftheactivities.Itisbasedonconversationswithrelevantinformants(stakeholders,externalexperts)andonliteratureandfilestudies.
Identifying result indicators Animportantcomponentofreconstructingpolicytheoryisformulatingaresultschain,distinguishingvariouslevelsofresults.Measurableresultindicatorsareidentifiedforeachoftheresultlevels,toenableperformancetobeassessedatthatlevel.
Identifying data sources Identifyingthescaleandqualityofexistingsourcesofinformation(files,literature,informants,statisticaldatabases)andtheopportunitiesforgatheringsupplementaryinformationdeterminetheextenttowhichthesubjectcanbeevaluatedandhowtheevaluationquestionsareformulated.
Formulating the research questions TheresearchquestionsincludedintheTermsofReferencearedeterminedbytheultimateaimoftheevaluation.Theyrelatetotheself-evidentneedforinformationonresultsachievedatthelevelofoutputsandeffects.Thisinformationisofcoursevaluableforboththeaccountabilityandlearningfunctionofanevaluation.Inaddition,supplementaryquestionsrelatingtoaspecificlearningneedcanbeformulated.
Determining the method of evaluation Thechoiceofresearchmethodsdependsonalargenumberoffactors.Itwillbeinfluencedbythenatureofthesubject,thetimeandbudgetavailablefortheevaluation,datathatisalreadyavailableandneedstobecollected,andthenatureoftheresearchquestions.
6.3 Terms of Reference
TheTermsofReferenceestablishtheconditionsunderwhichtheevaluationwillbeconductedandformtheresultofthepreliminarystudy.TheToRmustcontainatleastthefollowingcomponents:
• thereasonandjustificationfortheevaluation:ashortexplanationoftherelevanceoftheproposedevaluation;
• the(external)aimoftheevaluation;• thecorequestions:formulationofthecentralresearchquestionandthesecondary
questionsderivedfromit;• thedemarcationandscopeoftheevaluation:explanationofhowthefieldof
Page 33
31
Evaluation policy and guidelines for evaluations
researchwillbedemarcatedintermsoftheme,geographicallocationandtime;• approachandchoiceofmethodology,includingfocusonsubsidiarystudies:
descriptionofthedesignoftheevaluation,theindicators,informationsourcesandresearchtechniques;
• representativeness:explanationofhowandtheextenttowhichrepresentativenessplaysaroleintheevaluation;
• organisation:explanationoftheorganisationoftheevaluation;deploymentofexperts,organisationofworkshops,missions,etc.;
• products:indicationoftheexpectedsubsidiaryproducts,includingsubsidiarystudiesoradatabase.Alsoanindicationofwhetherthereportwillbepublishedorwhethertheevaluationisinternal;
• timepath:thetimepathfortheevaluation;• budget:aspecifiedoutlineofthecosts.
IftheTermsofReferenceareforapolicyreviewthequestionsprescribedbytheRPE2006mustalsobetakenintoaccount.Policyreviewsmustaddressthefollowingquestions:
• Whatproblemgaverisetothepolicy?Doestheproblemstillexist?Whatisthecauseoftheproblem?
• Whydoesthegovernmentconsidertacklingtheproblemasitsresponsibility?Whyistheresponsibilityatcentralgovernmentlevel(andnotatlocalorEUlevel)?Whatformdoesthatresponsibilitytakeandwhy?
• Whatobjectivehasthegovernmentformulatedtosolvetheproblem?• Whatinstrumentsarebeingdeployed?Howcoherentarethey?Isthereoverlap?
Whatisknownabouttheimplementationofthepolicyandtheefficiencyofoperationalmanagement?
• Whatistheeffectoftheinstrumentsontheformulatedobjectives(solutiontotheproblem)?Doinstrumentsinotherpolicyareasalsohavesignificanteffectontheformulatedobjectives?Whatarethemainpositiveandnegativeside-effects?
• Howarethebudgetsdetermined?Onwhatbasis?
ThedraftTermsofReferencearenotonlyputontheagendaoftheIOBdepartmentalmeetingbutshouldalsobediscussedinadvancewithdirectstakeholdersandexternalexperts(preferablyintheformofareferencegroup,see7.2),andwiththeinternalpeerreviewers.Thedegreetowhichandthewayinwhichcommentsaredealtwithmustbecleartoallparties.TheTermsofReferenceforanevaluationareapprovedbytheIOBDirector;ifthesubjectoftheevaluationliesoutsidedevelopmentcooperation,theToRwillbeapprovedbytheDirector-Generalconcerned.
TheTermsofReferenceformthebasisonwhichabudgetwillbereservedfortheevaluation.Italsoprovidesaframeworkforcontractingexternalresearchers.ByapprovingtheTermsofReference,theIOBDirectoralsoapprovesthebudgetincludedintheToR.
Page 34
32
6 The evaluation process
6.4 Deploying researchers
TheIOBinspectorbearsfullresponsibilityforsettingup,conductingandcompletingtheevaluation.Inmostcases,theinspectorwilldeployresearcherstoconducttheevaluation.Themomentatwhichtheywillbedeployedwillvary,aswilltheextenttowhichtheyoperateautonomously.
Researcherscanberesearchassistantsorconsultantsfromresearchinstitutions,universitiesorconsultantagencies.
Researchassistantsareusuallyrecentgraduateswhosupporttheinspectorinconductingtheevaluation.Theactivitiestheyperformvaryfromfilestudies,assistancewithfieldmissionsorinterviewinginformantstodraftingpartsofthefinalreport.Thewayinwhichtheyaredeployedwilldependontheirexperienceandcanchangeovertime.Researchassistantsarefullmembersoftheevaluationteam.
WiththemajorityofevaluationsIOBcontractsexternalconsultants.6Twocriteriaaredecisive:theexpertiseofexternalexpertsmustbecomplementarytothatoftheinspector,andtheymusthaveampleresearchexperience,preferablyinthecontextofevaluations.Variousmodalitiesarepossiblefordeployingexperiencedexperts.Oneormoreexpertscanbedeployedasadirectcounterpartfortheinspectorandadivisionoftasksagreedbetweenthem.Anotheralternativeisforwholeorpartoftheevaluationtobecontractedouttoaresearcheroraresearchbureau.Thewayinwhichexternalexpertiseisdeployeddependsonthenatureoftheservicesrequiredandassociatedcosts.Inthecaseofjointevaluationswithotherdonors,thewholeevaluationisgenerallycontractedout.
6.5 Cooperation with researchers in partner countries
InconductingevaluationsIOBaims,wherepossible,toworktogetherwithresearchersfrompartnercountries.Thathelpstobuildindependentresearchcapacityinthesecountries.Furthermore,researchersfrompartnercountriescanbeinvaluableastheyareoftenwellinformedonlocaldevelopmentsandrelations.However,notallevaluationsareofcoursesuitedtosuchcooperation.Thepossibilityofmakinguseofresearchersfrompartnercountriesdependsinthefirstplaceonthesubjectoftheevaluation.Somesubjectsdonotwarrantsuchachoice,forexample,becausetheevaluationwilladdressonlyDutchactivitiesorinterventionsofaglobalnature.Wherethedeploymentofexpertsisalogicaloption(forexample,evaluationofsectorpolicyinaspecificcountry)IOBactivelyexploresthepossibilities.
6 Contracting out and entering into contractual relationships are determined by the procedures laid
down in the Operational Procedures Manual (HBBZ). They will not be considered further in this document.
Page 35
33
Evaluation policy and guidelines for evaluations
6.6 Methodology
Evaluationisaformofresearchthatentailsajudgment.Thecustomarymethodologyislargelyderivedfromthesocialandeconomicsciences.TheTermsofReferencegiveageneraloutlineofthemethodologytobeappliedintheevaluation.Whentheevaluationistobecontractedouttoexternalresearchers,theyareusuallyaskedtoelaboratethemethodologytheywilluseinaninceptionreport.
InaccordancewiththeOECD/DACEvaluationQualityStandards,IOBrequiresthatasoundmethodologyisappliedandthatitisaccountedforinthefinalreport.Themethodologicalchoicesmadeinanevaluationaredeterminedbytheextenttowhichtheyleadtothecollectionofreliabledataandprovideabasisforreachingvalidandreliablejudgments.Themethodsandtechniquesusedcanvaryperevaluation.
Thevalidityofanevaluationisdeterminedbytheextenttowhichthedesignoftheevaluation,themethodofdatacollectionandtheanalysisofthedataleadtoreplicableanswerstothequestionsposedintheTermsofReference.Validityrelatesto:
• thestatementoftheproblem(theclaritywithwhichtheproblemisstated,theformulationoftheresearchquestionsandthedefinitionoftheevaluationcriteria);
• theevaluationobject(thedefinition,operationandlimitationoftheevaluationobjectandsituatingitinthepolicyandinstitutionalcontext);
• thepolicytheory(thepresentationorreconstructionoftheinterventionlogicandresultlevels,andtheoperationalisationofresultmeasurementonthebasisofindicators);
• theanalysis(datasources,datacollectionandtheprocessingandargumentationofconclusionswithfindings).
Thereliabilityofanevaluationisdeterminedbytheextenttowhichtheresultsarestable,i.e.thatrepeatingtheevaluationwillproducethesameresults.Reliabilityisrelatedto:
• theresearchmethods(specificationofandaccountingfortheresearchmethodsused,verificationofdataandtriangulation);
• thescope(representativenessofthesampleandselectionofcasestudiesandspecificationofthelimitsoftheevaluation);
• independence(ofthesourcematerialandoftheresearchersinrelationtostakeholders);
• theprogressoftheevaluationandqualitycontrol(reportingontheprogressoftheevaluationandorganisedqualitycontrolthroughinternalandexternalmonitoring).
ToguaranteethevalidityandreliabilityofevaluationsasmuchaspossibleIOBchoosesacombinationofresearchmethods(‘mixedmethods’).Thiscombinationrelatestothedesignoftheevaluationandtoqualitativeandquantitativemethodsandtechniquesofdatacollectionandanalysis.
Page 36
34
6 The evaluation process
The design of the evaluationThemajorityofIOBevaluationsaredesignedaccordingtotheguidelinesfor‘theory-based’evaluation.Thatislogicalbecausetheaimistoassesstheefficiencyandeffectivenessofpolicyimplementation.Thatrequirestakingpolicytheoryasastartingpoint.Themodeloftheory-basedevaluationisverysuitedtopolicyreviews.Theory-basedevaluationsfocusonadetaileddescriptionofwhathashappenedinthepastandwhy,andanalysingthedemonstrablecausesofthesedevelopments.Todothatinaresponsiblemanner,itisimportantinthefirstinstancetoreconstructthepolicytheory.Theevaluationthenassessesthevalidityofthepolicytheory.Themosttriedandtestedmethodofidentifyingaimsandinterventionstrategiesisthroughthelogicalframeworkortheoftenmoredetailedresultchain.Thismodeloffersaframeworkforsystematicdatacollection,identifyingindicatorsforsystematicassessment,anddeterminingtherelationshipsbetweenthedifferentresultlevels.
Analternativeforthisevaluationdesignistotakechangesinrealityasastartingpointratherthanthepolicytheory.Supportersof‘realistevaluation’advocatethiskindofapproach,whichfirstdevotesattentiontothecontextinwhichchangesoccurandthemechanismsinvolved.Onlyinthesecondinstancedoesitattempttoattributechangestospecificactivities.
Inrecentyearstherehasbeenincreasingemphasisinternationallyontheneedtoprovidestrongerevidenceoftheeffectivenessofgovernmentpolicy.Commonevaluationpractice,whichgenerallyprovidesplausibleexplanationsoftherelationshipbetweeninterventionsandtheireffects,isnolongerconsideredadequate.Thereisaneedtoidentifytheneteffectsofinterventions,ifpossibleinquantifiableterms.Thatmakesitnecessarytoconductimpactevaluations.ThedesireforharderimpactstudiesisalsoexpressedintheRPE2006,whichexplicitlydistinguishesex-postimpactstudiesasaseparatecategoryinadditiontopolicyreviews.InitsevaluationprogrammingIOBthereforeincludestheimplementationofaseriesofimpactevaluationsinappropriatepolicysectorsandthemes.Inpractice,however,theoptionsarenotunlimited.‘Randomisedcontroltrials’(RCTs),inwhicha‘treatmentgroup’iscomparedwitha‘controlgroup’andwhichineffectreproducealaboratorysituation,arenotapplicableinthegreatmajorityofcases.ThemostcommonalternativeforRCTsisaquasi-experimentalresearchdesign.AlthoughthiscloselyresemblesafullyexperimentalRCT,itrecognisesthattheresearcherdoesnothavefullcontroloverthecompositionofthetreatmentgroupandthecontextinwhichthemeasurementsaremade.Theselimitationsareaccountedforintheevaluationreport.
ForthesakeofcompletenessitshouldbemadeclearthatIOB’simpactevaluationsarealsodesignedonthebasisofatheoryofhowtheinterventionstobeevaluatedwork.Themostimportantdifferenceisthatthistheoryisnotexclusivelybasedonareconstructionofpolicy,butislargelyfuelledbyexistingscientificinsights.
Page 37
35
Evaluation policy and guidelines for evaluations
Qualitative and quantitative methods of data collection Variousmethodshavebeendevelopedtomeasuretheeffectivenessofinterventions.Generallyspeakingadistinctioncanbemadebetweenqualitativeandquantitativeresearchmethods.Qualitativeresearchmethodsfocusmoreonprocessesandemphasisethebehaviourofactors.Theyareflexible,oftenbecausetheyhaveaneclecticapproach,andcanplacefindingsintheirculturalandpoliticalcontext.Aqualitativeapproachcangiveaplausibleexplanationofwhyacertaininterventionhasorhasnothadaneffectinagivencontext.Qualitativemethodsplayanimportantroleindetermininginternalvalidity.Itishowevernotalwayspossibletogeneralisethefindingsofqualitativeresearch.Quantitativeresearch,ontheotherhand,explicitlyaimstomakegeneralstatements(externalvalidity).Statisticaltechniquesmakeitpossibletoisolatetheeffectsofaninterventioninacomplexenvironment.Thismethodis,however,verydata-sensitive.Itrequiresalargequantityofreliableinformationtoallowtheformationofsamplepopulations.
Quantitativeandqualitativetechniquesarenotinterchangeable,butcomplementary.Quantitativetechniquescanprovideastatisticalbasisforamorequalitativeapproach,whileaqualitativeanalysecanexplainrelationshipsestablishedbyquantitativeresearch.Withoutatheoreticalframeworkandqualitativeresearchintothemechanismsbehindassumedrelationships,statisticalanalysishasnosubstance.InitsevaluationsIOBappliesbothquantitativeandqualitativetechniques.
Methods of data collection Thevalidityandreliabilityofanevaluationstandsorfallswiththereliabilityofthedatacollection.Wherethedataiscollecteddependsstronglyonthesubjectoftheevaluation.Whendevelopmentinterventionsarebeingevaluatedsomeofthedatawillusuallybecollectedinthepartnercountriesconcerned.Iftheyareavailable,existingdatabasescanbeused.
Themostcommonformsofdatacollectionare:
• File and literature studiesToobtainaclearpictureofthefieldofstudyitisnecessarytobecomefamiliarwiththecompleterangeofwrittensourcesavailable.Thisincludesinformationrelatingtointerventions,morepolicy-relateddocumentationandacademicliterature.Itrequiresworkingthroughthearchivesingreatdetail,preferablyusingasystematiclistofpointsforattention.Oneimportantpointforattentioninafilestudyistoassessthereliabilityoftheinformation.
• Interviews with key informants Generallyadistinctionismadebetweenstructured,semi-structuredandnon-
structuredinterviewmethods.Whichmethodisapplieddependsontheroleoftheinterviewintheresearchandthenatureoftheinformationthatisbeingsought.Especiallyintheinitialorientationphaseinterviewswillgenerallybeopentoallowasmanydifferentopinionsandviewpointstobecollectedfromtheinterviewees.
Page 38
36
6 The evaluation process
• Group interviews Variouskindsofgroupinterviewsareusedinevaluations.Oneofthemost
commonformsisthefocusgroupdiscussion,inwhichtheinteractionbetweentheintervieweescanalsobeavaluablesourceofinformation.
• Direct measurement Alotofquantifiableinformationcanbeacquiredbysimplerecordingand
counting.Especiallyatthelevelofoutputs,thisisthemostlogicalapproach.Directmeasurementisparticularlyusefulifdataisavailablefromabaselinestudy.
• Direct observation Observationmethodscanbelimitedtothephysicaloutputsofanactivity,butcan
alsorelatetocertainprocesses.Observationtechniques(withorwithoutthedeploymentofaudiovisualaids)havealwaysplayedanimportantroleinfieldresearch.Itshouldbeaddedthattheycanbeverytime-intensive.Evaluationsgenerallymakeuseofaquick(moreorlessstructured)tourofphysicaloutputs,a‘transect’,and/orattendingoneormoremeetingsofthetargetgroupsconcerned.
• Surveys Surveysaimtocollectalargequantityofstandardiseddatafromalargepopulation.
Suchdataisverysuitableforstatisticalanalysis.Surveysmakeuseofpreviouslydrawnupquestionnaireswhicharepreferablytestedfirstandwhichcanbeinwrittenorverbalform.Theyareessentialindeterminingthestartingsituation(baseline)andareindispensableinimpactevaluations.Becausesurveyscanbetime-consumingandcostly,impactevaluationsoftenusetheresultsofpreviousquestionnairesormodulesaddedtopendingsurveys,sothattheycanservemultiplepurposes.
• Case studies Casestudiesarein-depthstudiesofasmallnumberofspecificinterventions.They
offerthepossibilityofconsideringtheperceptionsofstakeholdersorreconstructinghistories.Casestudiesareavaluablesupplementtobroadersurveys,becausetheyprovideanopportunitytoexamineandexplainrelationshipsthatbecomeapparentfromstatisticalanalysisingreaterdetail.Ifanevaluationisbasedexclusivelyonaseriesofcasestudies,itisessentialtodeterminetowhatextenttheselectedcasesarerepresentative.
Page 39
37
Evaluation policy and guidelines for evaluations
6.7 Reporting
ThefinalreportofanevaluationisapprovedbytheIOBDirector,aftershe/heissatisfiedthatthecommentsofthereferencegroup,internalpeerreviewersandtheIOBDepartmentalMeetinghavebeenincorporatedadequately.IOBthensendsthereporttotheministerorministersconcernedtobeforwardedtoparliament.Onrequest,IOBcanalsoprovidetheministerwithadditionalinformationonthefindingsoftheevaluation.Ifanevaluationisaddressingapoliticallyverysensitiveissue,IOBwillinformtheministerconcernedingoodtimeontheprogressoftheevaluationand,asfaraspossible,theexpectedoutcomes.
Inprinciple,allIOBevaluationreportsareaccessibletothepublic.Thisrulecanonlybedeviatedfrominexceptionalcases.Anevaluationmight,forexample,beveryconfidentialinnatureormayonlyserveaninternallearningfunction.Thedecisionnottopubliciseanevaluationreportmustalwaysbetakenbeforetheevaluationstarts.
Thereportshouldtakeaccountofthefactthatthefindingsoftheevaluationaretobepresentedtodifferentstakeholderswithvariousinterests.ThebasicprincipleisthatIOBreportsaredesignedtoaccounttobothparliamentandsociety,andtofulfilalearningfunction.Thatsetshighstandardsforthereporting.Parliamentbenefitsfromashortandconcisepresentationofthemainfindingsandconclusions.Accountingfortheresultsandthedeploymentofthepublicfundsrequiredtoachievethemhaspriority.Policydepartmentswillbeinterestedintheextenttowhichthereportprovidesinformationthatoffersopportunitiestoadjustpolicyformulationorimplementation.Inallcasesitisimportanttoaimforatransparentpresentationoffindings,analysisandassessment.
IOBevaluationsidentifyanumberofburningissuesthat,inthelightoftheevaluationfindings,requireattention.IOBthereforeavoidsmakingconcreterecommendations,puttingitselfinthepositionofthepolicymakerandtherebyofabutcherinspectingthequalityofhisownmeat.Furthermore,IOBworksaccordingtotheOECD/DACdefinitionofevaluation,whichrefersonlyto‘lessonslearnt’,andtheguidelinesforpolicyreviewintheRPE2006.
IntheinterestoftransparencytheTermsofReferenceshouldalwaysbeattachedtotheevaluationreportasanannex.Theresearchquestionsshouldbesystematicallyaddressedinthereport,alsoifsomeofthemcannotbeanswered.Ifnecessary,additionalinformationshouldbeprovidedintheannexesontheresearchmethodsemployedandanyproblemsthatariseduringimplementationoftheevaluation.
IftherearefundamentalandsubstantiateddifferencesofopinionbetweenIOBandthestakeholdersregardingthemainconclusionsofthereport,theseshouldbeexplicitlymentionedinthereport.ThedecisiontothiswillbetakenbytheIOBDirector.
Page 40
38
6 The evaluation process
Manyevaluationreportsareasynthesisofsubsidiarystudies.Especiallywherethesesubsidiarystudiescanalsobereadindependentlyitisimportanttomakethemaccessibleforpolicydepartmentsorotherstakeholders.Theyoftencontaindetailedinformationthatisnotincludedinthemainreport.TheevaluationreportsarepublishedbothinprintformandontheMinistryofForeignAffairswebsite.Subsidiarystudiesarealwaysavailableonthewebsiteandareinsomecasesalsopublishedinprintform.Inthecaseofveryextensiveevaluations,intheinterestsofaccessibility,IOBproducesasummarisedversionofthereport.BoththesummaryandthefulltextaresubmittedforinclusioninOECD/DACnetworkforevaluationservices’evaluationinventory.Completedimpactevaluationsarealsoenteredinthedatabasesetupby3IEandNONIEforevaluationsofthiskind.
Page 41
39
Evaluation policy and guidelines for evaluations
Page 43
41
Evaluation policy and guidelines for evaluations
TheIOBinspectorisresponsibleforconductingindependentevaluations.Thequalityoftheworkcarriedoutis,however,aconcernforIOBasawhole.TheIOBmanagementhasitsownresponsibilityinthiscontext,asitapprovestheevaluationreports.Toprepareforthisdecision-makingprocess,IOBhasaqualitycontrolsystemcomprisingfiveinstruments:internalpeerreviewers,referencegroups,thedepartmentalmeeting,theadvisorypanelandtheself-evaluation.
7.1 Internal peer review
Qualitycontrolbenefitsfromintensivecooperationandasystematicintercollegialmonitoring.IOBthereforehasasystemthatensuresthatinspectorscanrelyonsubstantivesupportfromcolleagueswhenconductingevaluations.Start-upreports,TermsofReference,subsidiarystudiesandthedraftfinalreportarediscussedwithtwointernalpeerreviewers,fellowinspectors,beforebeingsubmittedtothedepartmentalmeeting.Theinspectorchooses,inconsultationwiththeIOBDirector,whichcolleaguesaretoactasinternalpeerreviewers.TheDirectorappointsinternalpeerreviewersperevaluation.Theyareselectedandappointedonthebasisofthefollowingcriteria:
• coachingoftheinspector;• internalreadershipasalearningfunctionforrecentlyappointedinspectors;• evaluationexperience;• affinitywiththeevaluationsubjectandexpertiseinthethemeconcerned;• avoidingconflictsofinterest;• anequaldistributionoftheworkloadwithinIOB.
Notallcriteriaareequallyimportantandaspecificcriterioncanbedecisive.
Thetaskoftheinternalpeerrevieweristogiveadvicetotheinspectors.Thisadviceisnotbindingbutinspectorsdohavetoindicatewhattheydowithit.LastingandfundamentaldifferencesofopinionmustbesolvedinconsultationwiththeIOBDirector.
Inspectorsmustincludethecommentsandadviceofinternalpeerreviewersandashortreportoftheirdiscussionswiththereadersinthefileontheevaluation.
Theinternalpeerreviewersarenotpartofthereferencegroupfortheevaluation.
AstheIOBDirectorisresponsibleforapprovingevaluationproductshe/shecannotbeaninternalpeerreviewer.
ThedecisionwhethertomakeuseofinternalpeerreviewintheeventofjointevaluationsbetweenIOBandtheevaluationservicesofotherdonorsislefttotheIOBinspectorresponsiblefortheevaluation,inconsultationwiththeIOBDirector.
Page 44
42
7 Quality control
7.2 Reference groups
AsecondformofqualitycontroloverIOBevaluationsisreferencegroups.Thesegroups,whicharecreatedforeachevaluation,arepartlyexternal,consistingofrepresentativesofthepolicydepartmentsinvolvedintheevaluation,organisationsand/orlocalbodies,andexternalexperts.AreferencegroupischairedbytheIOBmanagementandmeetsatkeymomentsduringtheprocess(e.g.todiscussthedraftToR,partialreportsandthedraftfinalreport).Theinspectorisresponsibleforreportingonreferencegroupdiscussionsand/orprovidingsummariesofwrittencommentsbyreferencegroupmembers.
Forevaluationsthatentailintensivecooperationwithresearchersinaspecificpartnercountry,asoundingboardgroupissetupinthecountryconcerned.Theformthegrouptakeswillvaryfromcasetocase.
ForjointevaluationswithotherdonorsaDutchreferencegroupisusuallynotsetup.Thereisajointsteeringgroupcomprisingrepresentativesoftheparticipatingdonors.Inthecaseoflarge-scale,complexevaluationsasmallermanagementteamissetupfromwithinthesteeringgrouptoruntheevaluationonadailybasis.Thesteeringgroupisthenprimarilyconcernedwithqualitycontrol.Inaddition,withtheseevaluations,theresearchbureauconductingtheevaluationisexpectedtoarrangeitsownqualitycontrol.Thisisoneoftheaspectstakenintoaccountwhenassessingtenders.
7.3 Departmental meeting
TheIOBdepartmentalmeetingplaysapartinqualitycontrolattwomomentsduringtheimplementationofanevaluation:whendiscussingtheTermsofReference(orsometimestheinceptionreport)andduringdiscussionofthedraftfinalreport.TheToRarealwaysdiscussedatafulldepartmentalmeeting.BeforethismeetingitmustbecleartotheinspectorwhichcolleagueshavesubmittedcommentsontheTermsofReferenceandthesubstanceofthecomments,ingeneralterms,sothathe/shecanprepareforthediscussion.Draftfinalreportsarethendiscussedonceagaininaseparatefinalmeetingbetweentheinspectorandtheinternalpeerreviewers,chairedbytheIOBDirector,beforetheDirectorapprovesthereport.Thisfinalmeeting,whichisopentoallIOBstaffmembers,hastwopurposes:itofferstheopportunityforafinalassessmentofthequalityoftheproduct,andhelpscolleaguestolearnfromeachothers’experiences.ItalsoensuresthatthefinalproductofanindividualinspectorbecomesaproductofIOBasawhole.
Page 45
43
Evaluation policy and guidelines for evaluations
7.4 Advisory panel
ToincreasetheusabilityanduseofevaluationsinthefieldofdevelopmentcooperationtheMinisterforDevelopmentCooperationmadeapledgetoparliamenttoestablishanAdvisoryPanelofprominentexternalexperts.Thepanel,withitssecretariatatIOB,willplayaroleinqualitycontrolbymonitoringtheusabilityofevaluationsandofferingadviceonrequestoronitsowninitiative.Itwillbecomeoperationalinthecourseof2009.
7.5 Self-evaluation
Everyevaluationconcludeswithacriticalreviewofhowitproceeded,withtheaimofimprovingtheapproachtofutureevaluations.Theseself-evaluationsarewrittenbytheinspector,sometimestogetherwitharesearchassistantandtheinternalpeerreviewers.TheTermsofReferencefortheevaluationdeterminetheformatoftheself-evaluation.BysystematicallygoingthroughthevariouscomponentsoftheToR,theself-evaluationprovidesaclearinsightintothesuccessfulandlesssuccessfulelementsoftheevaluation.Itshouldinanycaseaddress:(1)theextenttowhichtheresearchassignmentcouldbeimplemented,(2)anydifferencesbetweentheplannedtimepathandtheactualdurationoftheevaluation,and(3)differencesbetweenestimatedandactualcosts.Self-evaluationsarediscussedinternallyandperiodicallyatthedepartmentalmeeting.
Page 46
8Communication andfeedback
Page 47
45
Evaluation policy and guidelines for evaluations
Communicationandfeedbackrelatetothefeedbackofevaluationresults,withaviewtothetransparencyoftheevaluationprocess,theaccountabilityfunctionandtheintegrationoflessonslearnedinfuturepolicyformulationandimplementation.Feedbackisnotlimitedtothephasefollowingcompletionoftheevaluation.Consultationwithandfeedbacktostakeholderstakeplacethroughoutthewholeperiodofimplementation:duringthepreliminarystudy,inrelationtotheTermsofReference,andduringdiscussionofsubsidiarystudiesandthedraftfinalreport.Wherenecessary,consultationandfeedbackisalsoprovidedtothepartnercountries.Whilefeedbackduringtheevaluationisimportantatimplementationlevel,aftertheevaluationhasbeencompleteditismoreimportantatpoliticalandmoregenerallevel.
Theextenttowhichcompletedevaluationsfulfilthelearningfunctiondependsonanumberoffactors.Inthefirstplace,itisamatteroftheusabilityoftheevaluationitself.Thisisdeterminedby:
• theclaritywithwhichtheexternalaim(i.e.outsidetheevaluationitself ),for whichtheevaluationresultscanbeused,isspecified;• theclarityandcompletenesswithwhichtheessenceoftheevaluationandespecially
itsmainfindingsarepresentedintheevaluationreportandthesummary;• thecompletenesswithwhichallresearchquestionsareansweredbythe
conclusions;• theextenttowhichtheresultsoftheevaluationsuggestactionsthatliewithin thecapacityoftherecipientofthereport.
Inthesecondplacelearningfromevaluationsdependsontheextenttowhichtherecipientsandusersarereceptivetothelessonsdrawnfromthestudies.Thisisrelatedtotheinstitutionalcontextandtheorganisationalculture.IOBhasnoinfluenceonthisanditisnotIOB’sresponsibility.Itisthedirectresponsibilityofthemanagementoftheorganisation(theministryorotherorganisationsthatreceiveIOBreports).
IOBdoeshoweverhavearesponsibilitytocommunicateonevaluationreports.Ascommunicationwithpolicydepartmentsiscrucialtoincreasethelearningeffect,itiscrucialthatevaluationsarediscussedintherelevantstaffmeetings.Withregardtoexternalcommunication,IOBdrawsupacommunicationplanforeachevaluation,inconsultationwiththeministry’sInformationandCommunicationDepartment.IOBincreasestheaccessibilityofevaluationresultsandensurestheirwiderdistributionbypreparingmediapresentations,givinglecturesandinterviews,writingarticles,andorganisingconferencesandseminars.
IOBevaluationsarealsoaninstrumentforrenderingaccountabilitytoparliament.WithinthreemonthsofthereportbeingapprovedbytheIOBDirector,theministersendsittoparliament,usuallytogetherwithapolicyreaction.Thetextofthereaction,whichisusuallydraftedbythepolicydepartmentconcerned,containsindicationson
Page 48
46
8 Communication and feedback
howtheresultsoftheevaluationaretobeacteduponinthelightoffuturepolicy.Inthisway,accountingtoparliamentalsocontributestothelearningprocess.
Inthecaseofevaluationsinthefieldofdevelopmentcooperationthereportsarealsopublishedinthepartnercountries,offeredtothegovernmentandmadewidelyknownthroughtheorganisationofmeetings.WithaviewtomutualaccountabilityIOBinvestssystematicallyinsuchfeedbacktothepartnercountries.
Page 49
47
Evaluation policy and guidelines for evaluations
297 2004 Over solidariteit en professionalisering
Evaluatie van Gemeentelijke Internationale Samenwerking (1997-2001).
isbn 90-5328-341-2297 2004 On solidarity and
professionalisation Evaluation of Municipal International Co-operation (1997-2001). isbn 90-5328-342-0 298 2004 Onderzoek naar de kwaliteit van in
2002 afgeronde decentrale evaluaties
Eindrapport. isbn 90-5328-344-7299 2005 Een uitgebreid Europabeleid Evaluatie van het Nederlands beleid
inzake de toetreding van Midden-Europese landen tot de Europese Unie 1997-2003
isbn 90-5328-347-1300 2005 Aid for Trade? An Evaluation of Trade-Related
Technical Assistance isbn 90-5328-349-8301 2006 Van Projecthulp naar Sectorsteun Evaluatie van de sectorale
benadering 1998-2005 isbn 90-5328-351-x301 2006 From Project Aid towards Sector
Support An evaluation of the sector-wide
approach in Dutch bilateral aid 1998–2005.
isbn 90-5146-000-7302 2006 Evaluatie van het Nederlandse
mensenrechtenbeleid in de externe betrekkingen
isbn 90-5328-352-8
303 2006 Dutch Humanitarian Assistance An Evaluation isbn 90-5328-352-8304 2007 Evaluatie van de vernieuwing van
het Nederlandse onderzoeksbeleid 1992-2005
isbn 978-90-5328-353-0304 2007 Evaluation of the Netherlands'
Research Policy 1992-2005 (Summary)
isbn 978-90-5328-353-0305 2007 Impact Evaluation: Water Supply
and Sanitation Programmes Shinyanga Region, Tanzania 1990-2006
isbn 978-90-5328-354-7306 2007 Chatting and Playing Chess with
Policymakers Influencing policy via the Dutch
Co-Financing Programme isbn 978-90-5328-355-4307 2008 Beleidsdoorlichting seksuele en
reproductieve gezondheid en rechten en hiv/aids 2004-2006
isbn 978-90-5328-358-5 308 2008 Het Nederlandse Afrikabeleid
1998-2006 Evaluatie van de bilaterale
samenwerking isbn 978-90-5328-359-2308 2008 Het Nederlandse Afrikabeleid
1998-2006 Evaluatie van de bilaterale
samenwerking (Samenvatting) isbn 978-90-5328-359-2308 2008 The Netherlands’ Africa Policy
1998-2006 Evaluation of the bilateral cooperation (Summary) ISBN 978-90-5328-359-2
Evaluation studies published by the Policy and Operations Evaluation Department (IOB) 2004-2009 Evaluation studies published before 2004 can be found on the IOB website at www.minbuza.nl/iob
Page 50
48
Evaluation studies published by the Policy and Operations Evaluation Department (iob) 2003-2008
309 2008 Het Vakbondsmede-financieringsprogramma
Een evaluatie van steun gericht op versterking van vakbonden en vakbonds- en arbeidsrechten
isbn 978-90-5328-357-8309 2008 The Netherlands Trade Union
Co-Financing Programme An evaluation of support for trade
unions and trade union and labour rights (Summary)
isbn 978-90-5328-357-8309 2008 El Programa de Cofinanciamiento
Sindical Una evaluación del apoyo orientado
al fortalecimiento de sindicatos y de derechos sindicales y laborales (Resumen)
isbn 978-90-5328-357-8310 2008 Clean and sustainable? An evaluation of the contribution of
the Clean Development Mechanism to sustainable development in host countries
isbn 978-90-5328-356-1311 2008 Impact Evaluation: Primary
Education in Uganda isbn 978-90-5328-361-5312 2008 Impact Evaluation: Primary
Education in Zambia isbn 978-90-5328-360-8313 2008 Xplore-programma isbn 978-90-5328-362-2314 2008 Primus inter pares Een evaluatie van het Nederlands
EU-voorzitterschap 2004 isbn 978-90-5328-364-6315 2008 Impact Evaluation: Support to Rural
Water Supply and Sanitation in Dhamar and Hodeidah Governora-tes, Republic of Yemen
isbn 978-90-5328-364-6316 2008 Be our guests Beleidsdoorlichting Nederland als
gastland van internationale organisaties en Samenvatting
isbn 978-90-5328-370-7316 2008 Be our guests Policy review on hosting international
organisations in the Netherlands and Summary
isbn 978-90-5328-371-4
316 2008 Be our guests Examen de la politique meneé par les
Pays-Bas en tant qu'État hôte des organisations internationales (Sommaire)
isbn 978-90-5328-372-1317 2008 Sectorsteun in milieu en water isbn 978-90-5328-369-1318 2008 Samenwerking met Clingendael Evaluatie van de subsidieovereen-
komst tussen de ministeries van Buitenlandse Zaken en Defensie en
Instituut Clingendael isbn 978-90-5328-365-7319 2008 Meer dan een dak Evaluatie van het Nederlands beleid
voor stedelijke armoedebestrijding isbn 978-90-5328-365-3320 2008 Het tropisch regenwoud in het
OS-beleid 1999-2005 isbn 978-90-5328-374-8321 2009 Maatgesneden Monotoring 'Het
verhaal achter de cijfers' Beperkte beleidsdoorlichting
Medefinancieringsstelsel 2007-2010 isbn 978-90-5328-374-5322 2009 Draagvlakonderzoek Evalueerbaarheid en resultaten isbn 978-90-5328-375-2323 2009 Preparing the ground for a safer
world Evaluation of the Dutch efforts to control landmines and explosive remnants of war 1996-2006
isbn 978-90-5328-377-6
Page 51
Evaluation policy and guidelines for evaluations
49
Page 53
51
Evaluatiebeleid en richtlijnen voor evaluaties
Page 54
52 Publishedby:
MinistryofForeignAffairsPOBox20061|2500EBTheHague|TheNetherlandswww.minbuza.nl
Layout:Eindeloos,TheHaguePrinter:OBT,TheHague
©MinistryofForeignAffairs|October2009|Ordercode:BZDR6662/E
Page 55
ratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe
nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa
aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum
houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMia
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
dIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MpalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisVelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgH
openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauW
restBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/K
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntweenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKa
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBa
asChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKiazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSan
alle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
furt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaMan
ekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
adHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhou
openhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusW
BernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWtislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIst
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouPetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAthe
iroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeR
okioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroenouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
MumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
aDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
enenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaP
obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo
elsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiaortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonodIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTjnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavanna
aoundéZagrebHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisV
elgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewY
auloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWrestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStad
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwe
enosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsP
eheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBaasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKi
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSanalle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetB
alaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoP
ashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitK
obeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCa
HanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonoadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideo
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotono
JubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioT
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouopenhagenKualaLumpurLaPazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskou
ortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHel
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.PetersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonW
tislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstazLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouP
etersburgStockholmStraatsburgSydneyTallinnTbilisiTeheranTelAvivTiranaTokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheiroCanberraCaracasChicagoColomboCotonouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampala
azLimaLissabonLjubljanaLondenLosAngelesLuandaLusakaLuxemburgMadridManaguaManillaMaputoMexicoMiamiMilaanMontevideoMontrealMoskouMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRokioTorontoTripoliTunisValle�aVancouverVaticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroe
nouDakarDamascusDarEsSalaamDenHaagDhakaDohaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKMumbaiMünchenMuscatNairobiNewDelhiNewYorkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSa
aticaanstadVilniusWarschauWashingtonWellingtonWenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBaDublinDüsseldorfFrankfurt/MainGenèveGuangzhouGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaK
orkNicosiaOsaka/KobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPenenYaoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujum
houGuatemalaHamburgHanoiHarareHavannaHelsinkiHoChiMinhStadHongkongIslamabadIstanboelJakartaJubaKaapstadKaboelKampalaKhartoemKievKigaliKinshasaKoeweitKopenhagenKualaLumpurLaPobeOsloO�awaOuagadougouParamariboParijsPekingPortofSpainPraagPretoriaPristinaQuitoRabatRamallahRigaRioDeJaneiroRiyadRomeRomeSanJoséSana’aSantiagoSantoDomingoSaoPauloSarajevoSeoulShanghaiSingaporeSkopjeSo�aSt.P
aoundéZagrebAbuDhabiAbujaAccraAddisAbebaAlgiersAlmatyAmmanAnkaraAntwerpenAsmaraAstanaAtheneBagdadBamakoBangkokBarcelonaBeiroetBelgradoBerlijnBernBoedapestBoekarestBogotáBrasiliaBratislavaBrusselBuenosAiresBujumburaCairoCanberraCaracasChicagoColombo
Page 56
IOB
Evaluation policy and guidelines for evaluations
Published by:
Ministry of Foreign Affairs PO Box 20061 | 2500 EB The Hague | The Netherlandswww.minbuza.nl© Ministry of Foreign Affairs| October 2009 | Order code: BZDR6662/E
olicy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department | IOB | October 2009 | Policy and Operations Evaluation Department
This document describes the evaluation policy and guidelines for evaluations of the Policy and Operations Evaluation Department (IOB) of the Dutch Ministry of Foreign Affairs. In publishing these guidelines, IOB wishes to provide transparency to third parties: colleagues at the Ministry of Foreign Affairs, the Dutch parliament, organisations and institutions affected by IOB evaluations, users of the evaluation results, and interested citizens. IOB will periodically review its policy and the guidelines and, if necessary, revise them.