IMPACT EVALUATION OF THE TANZANIA WOMEN’S VIRTUAL BUSINESS INCUBATOR Elena Bardasi, World Bank, PREMGE Alaka Holla, World Bank, AFTPM 1 Dakar February.

IMPACT EVALUATION OF THE TANZANIA WOMEN’S VIRTUAL BUSINESS INCUBATORElena Bardasi, World Bank, PREMGEAlaka Holla, World Bank, AFTPM

1

DakarFebruary 1-4, 2010

Outline

1. Project backgroundi. Objectivesii. Activitiesiii. Features

2. Impact evaluationi. Learning objectives and rationaleii. Set upiii. Implementation issues

2

Promotion of Women’s Entrepreneurship:

Tanzania Virtual Business Incubator Intervention to support women’s entrepreneurs

through delivery of training and BDS Why entrepreneurship?

In Tanzania: 2.7 million enterprises producing 30% of GDP

98% are microenterprises (<5 employees, <US$5,000) The large majority of Tanzanian in the LF are informal

Why women? 80% of microentrepreneurs in Tanzania are women Women have higher constraints than men, and specific

constraints

Choice of a VIRTUAL incubator (‘without walls’)

3

Tanzania Virtual Business Incubator:Objectives

Support the growth of women-owned businesses through delivery of BDS to strengthen their skills (financial literacy, market outreach, PD&D, etc.)

Through supporting women’s entrepreneurship the project aims to: Increase women’s social and economic empowerment Improve women’s well-being as well as their

household Improve children’s outcomes

Target is the micro/very small entrepreneurs but who wants her business to grow

4

Tanzania Virtual Business Incubator:

Components1. Development of a Virtual Business

Incubator2. Impact evaluation and monitoring

framework Impact evaluation is a ‘structural’ part of the

project Impact evaluation is very different from M&E.

Both IE and a strong M&E framework are needed to evaluate the project and measure/understand its impact

3. Communication and dissemination

5

Tanzania Virtual Business Incubator:

Component #1: Activities Pilot project in Dar es Salaam Delivery of training and BDS to 500 women, but

750 are ‘targeted’ (250 are in the control group) AIDOS model

Virtual incubator (3rd generation/incubator w/out walls)

Tailor-made portfolio of resources and support services

Attention to improving product quality and design Market-oriented focus Development of a network of mentors

6

Tanzania Virtual Business Incubator:

The training packageOrientation Business health check; SWOT-analysis of the

business

Entrepreneurship and Business Management Training

Entrepreneurship and enterprising organization; low cost promotional activities; understanding your market; effective marketing; costing, pricing, and calculating your profit; book-keeping; managing cash, inventory and credit; customer care; sources of finance; business plan; HR management; branding and patenting; time management; leadership training; communication and negotiation

Technical Training

Production skills, packaging and labeling, quality management, maintenance, customer care, product pricing, hazard analysis and critical control points, compliance, traceability and bar code

Specialized Technical Assistance and Individualized Counseling, Coaching, and Mentoring

Business counseling; linkage to consultants, advisors, project programmers; marketing assistance and linkage to markets; individualized support from coaches; link with mentors and participation in networking forums; finance facilitation

7

Tanzania Virtual Business Incubator:

Features of the project BDS delivery is at the core of the project ‘Hybrid’ project: PREM – PSD Project born with its impact evaluation Implemented by an NGO. Team includes: PRMGE; AFTPM;

AIDOS (Italian NGO, Technical advisor); Tanzania Gatsby Trust (local implementing partner working in consortium with IMED, SIDO-WED, Kwanza collection); ETC based in Dar

It is a pilot, but sustainability and capacity building are central

Learning from IE useful for both research and implementation (scaling up, adjustments, new areas)

Cost: about US$1.2 million (of which about US$150,000 for the impact evaluation)

8

Promotion of Women’s Entrepreneurship:

Tanzania Virtual Business Incubator

Mpango wa kukuza ujasiriamali na biashara kwa wanawake

9

BIG: Program to make entrepreneurship and women

enterprises grow

Learning objectives of the IE To what extent does business training affect enterprise

outcomes of female entrepreneurs in Dar es Salaam? Main indicator: sales revenue

What kind of program works (doesn’t work)? Business training Business training + individualized attention from

coaches/mentors

Does the training lead to any unintended consequences (good or bad)? Debt Depression Domestic violence Human capital investments in children More female decision-making within the home

10

Why can’t we just do regular monitoring? Regardless of change in revenue (+ or -) wouldn’t

know if program helped anyone

Simple before and after comparison of beneficiaries

Gives program impact + whatever else happened between start & end of program What if there was a lighting strike in the middle of the

program that burned down the major market place of beneficiaries?

What if cheap imports flood the market? What if a female MP starts promoting beneficiaries’ products?

Cannot disentangle these effects from program impact

Could underestimate or overestimate true impact of the program.

11

Why can’t we just do regular monitoring? Comparing people who received program to

people who did not receive the program

Gives program effect + whatever is different between participants and non-participants.

Usually there is a reason people choose to join or not join a training program, and we usually can’t observe this ex-ante. Could be people who are very motivated join the program

will overestimate program impact But could be people who won’t be successful without

help , who just suffered an enterprise-related failure will underestimate program impact

Impossible to disentangle these unobservable characteristics of participants from program impact

12

Why can’t we just do regular monitoring? Baseline + endline surveys

Can at many outcomes of interest Knowing about spillovers in the household

More accurate calculations of cost-benefit ratios Can think about necessary complementary services

for scale-up Example: If depression or debt increases, can think

about pairing program with counseling or financial literacy training

Experimentation Can learn about effectiveness of 2 different

program variants relative to same baseline

13

Basic set up1. Local implementation partner

Advertises program [ongoing] Screens applicants down to 750 entrepreneurs [March]

2. Baseline survey [April-May]

3. Random assignment of 750 women to 3 groups [May] Treatment 1: Traditional business training [250] Treatment 2: Treatment 1 + coaching & mentoring [250] Comparison [250]

4. Program implementation [May]

5. Endline survey 1 year after program start [June-July 2011]

14

Implementation issues

1. Program eligibility2. Where does the number 750 come

from?3. Baseline survey4. Treatment assignment5. Simultaneous treatment6. Date for endline survey

15

Program eligibility

Might not be able to say anything about entire population female entrepreneurs in Dar but need to be clear about population of interest Ideally entrepreneurs comparable to targeted beneficiaries in a

scaled-up version of the program Target those with highest potential to benefit cannot say that we

expect same results during scale-up Target those with very limited potential no effect of program in

impact evaluation no rationale for scale-up

TZ VBI limited to Entrepreneurs established for at least 1 year Certain sectors with growth potential identified in market study Entrepreneurs willing to pay upfront commitment fee

Estimated program impact most relevant for women who meet these criteria

16

Why 750 women?

Answer: Power calculations for sales + capacity of local implementation partner

Power calculations from TZ Enterprise Survey (2006) 10% increase: 1,079 in each group 2,158 total treated 15% increase: 480 in each group 960 total treated 20% increase: 270 in each group 540 total treated 25% increase: 173 in each group 346 total treated 30% increase: 120 in each group 240 total treated

An impact of a 15% increase in sales cannot be distinguished from zero impact with confidence OK because this is an expensive intervention; not interested

knowing about small effects

Caveat: Limited observations on female small business owners

17

What do we tell these 750 women? Before narrowing women down to 750

Need to say All of you cannot participate in the training program this year. We don’t have the capacity this year but we can train you next year. For this year’s slots, there will be a lottery.

Some women will say Then forget it! and then drop out

Treatment assignment occurs on remaining set of women

To keep all 750 women engaged All given an incubator “ID” card that has no

meaning outside of the program

18

Baseline survey

Occurring before assignment of treatment Disappointment or enthusiasm about

treatment status won’t affect responses or behaviors

But risk that women think responses could affect probability of treatment

Can use information in baseline (sector, neighborhood) to stratify treatment

Roughly 6 weeks before program start

19

Baseline survey

Survey firm completely separate from local implementation agency Pros

Survey firm has no incentive to find an impact Local implementation agency does not have

access to completely private information on trainees

Cons Makes locating women more difficult No in-house capacity for continuing the impact

evaluation or starting new ones

20

What can we learn from the baseline survey itself?

Descriptive data on characteristics we know little about (e.g. debt, inventories, suppliers)

Strata

Can also experiment with data collection methods to see how these affect responses (especially to sensitive questions) Type of non-monetary compensation for survey

participation (personal versus business) Order of instruments (household and business) Very cheap or costless Need to stratify on these “treatment arms” when

assigning real treatment

21

Assignment of treatment

Need final list of eligible women from local implementation partner

Use computer program to randomly assign them to 3 groups, stratified on Sector Geography Baseline experimental group

Send list with group assignment back to local implementation partner.

Local implementation partner sends out notices to women only telling them whether or not they have been accepted into the program (not specific treatment arm)

22

Treatment

Treatment will be roughly simultaneous That is, no phase-in Need to find many trainers, coaches, and

mentors

All groups get basic treatment first

Then 250 subset gets more intensive variant afterwards

23

Tanzania VBI Impact Evaluation

Treatment group500

All applicants > 750

All applicants > 750

Comparison group250

Eligible applicants750

Eligible applicants750

Basic treatment250

Basic treatment250

Basic treatment + mentoring

250

Basic treatment + mentoring

250

Baseline surveyAssign applicants to geographical and sectoral

strata

Assign people to strata again

24

Endline survey

Approximately 1 year after program start

Why? Simple answer: Project cycle

Not clear ex ante how long it would take for effects to materialize

Ideally, a second endline 3+ years later

25

What will we learn with this set-up? Relative to the status quo, what is the impact of

business training on enterprise and household outcomes of female entrepreneurs in certain sectors in Dar es Salaam? Status quo includes other training programs. Cannot

prevent comparison group from getting their own training.

Probably not powered enough to compare treatment variants unless impact of individualized services really large

Do survey design features (like compensation or order of instruments) affect responses to sensitive questions (debt, depression, violence, transactional sex, etc)?

26

Do we still need monitoring? Impact evaluation does not fill all of a project’s

monitoring needs

Most importantly, it typically does not track process If mentoring does not work, is this b/c mentoring really

doesn’t work or b/c mentors never met with the beneficiaries?

Did trainers show up in the classroom, did beneficiaries show up?

Implementation agency and field coordinator needs to systematically and quantitatively track this kind of information.

27

Broader monitoring framework

Includes variables difficult to collect in baseline/endline survey (for example, qualitative variables) e.g. increase in the quality of product, product differentiation,

woman’s assertiveness, etc.) Includes variables capturing

process/implementation e.g. number of women reached at each stage; number of visits

by coaches; attendance to training sessions; quality of trainers, etc.

Includes variables assessing increased capacity of the local partner, team performance in dissemination, etc. e.g. number of papers and policy notes; partnership and

collaborations stimulated by the project, etc. Uses a variety of tools

e.g. coach logs, evaluation forms, focus groups, etc.

28