ESF AND THE ROMA IN SLOVAKIA Results from a study on the territorial distribution of Roma-relevant ESF projects (2007 – 13) and their impact on Roma communities.

ESF AND THE ROMA IN SLOVAKIAResults from a study on the territorial distribution of Roma-relevant ESF projects (2007 – 13) and their impact on Roma communities in Slovakia

Jakob Hurrle, Charles University Prague; Andrey Ivanov, UNDP BRC

1. Context

Focusing on results…

• Few important questions:– How to define “result”? How to measure it?– What kind of data is needed? Where to get it?

• Most of all, how to move from monitoring the aggregate status of the population to project-level outcome monitoring and evaluation– To assess effectiveness of interventions– To assess the efficiency of funding (ESF and other)

• Those questions are part of the Roma pilot project: tools and methods for evaluation and data collection funded by DG Regional Policy

The M&E chain

Input Output Outcome Impact

Intermediate Final

Financial, physical

resources

Goods and services

produced by inputs

(classrooms built,

textbooks provided)

Access to, use of, and

satisfaction with services (enrolment, repetition,

dropout rates)

Effect on dimension

of well-being

(literacy)

Example: Employment generation project

• Inputs: Number of training courses, expenditures for re-qualification, number of presentations, number of attendees

• Outputs: Number of people who passed re-qualification course

• Outcome: Number of former unemployed who found jobs• Impact: Registered increase of HH incomes, change in

poverty rates Plus• Sustainability: duration of the job • Positive externalities: Reduced drop-our rates of children

at risk, reduced societal fragmentation

“Intervention” A B C Intervention seen by A is not what is seen by C

From theory to practice: ESF

• How the structural funds ‘perform’ in reality in regards Roma in the case of Slovakia?

• Does the system and the existing procedures allow following the M&E chain?

• If not, what is missing?

• What can be improved for the next programming period?

Assumptions behind the study

• Improving the situation of Roma requires a combination of1. Political will

2. Resources

3. Conceptual framework of intervention

4. Implementation procedures – the ‘nitty-gritty’ of the project cycle

• Usually we focus on (1) and (2), less often on (3) and almost never on (4)

A number of specific questions…

• Does the territorial distribution of the projects match the distribution of the Roma population?

• What share of Slovakia´s Roma population was reached? • To what extend were the invested resources indeed

relevant for the needs of the members of marginalized Roma communities?

• What are the experiences of project owners and what impact did the projects have on beneficiaries?

• Are the project results sustainable?

All this fits into the desired “outcome level M&E”

2. Methodology

2.1 Data analysis 2.2 Case study

Prešov Region

Combination of

two approaches:

2.1 Data analysis

Three data sources:

• Key data from all „Roma relevant“ projects• Sample of 298 Roma-relevant projects

created on basis of analysis of project documentation

• Data set of Atlas of Roma Communities (2004)

Data analysis - key questions:

• What does „Roma-relevant“ mean? Method: Analysis of project documents

• Which types of projects are „Roma-relevant“?Method: Analysis of project documentation in sample

• Which communities where targeted? Method: Matching of project data and data from Atlas of Roma Communities (2004)

2.2 Case study Prešov Region

• Focus on training and employment projects in a structurally disadvantaged region with a high share of Roma

• 8 projects visited

• Attempt to create a sample that represents variety of project types (different types of project owners, ethnic projects / mixed groups of beneficaries, urban / rural, open and closed settings)

3. „Roma-relevant“ projects in the Slovak ESF programme

National projects

71 %

Out of these Roma-relevant: 8,7 %

Calls for proposals

29 %

Out of these Roma-

relevant: 13,4 %

1 011 506 634 € contractedOut of this „Roma relevant“: 185 142 009 € (18 %)

„Roma-relevant“ projects within the five ESF priority axes

Axis 1: Supporting Employment Growth Roma-relevant: 22 %

Axis 2: Social inclusionRoma-relevance: 86 %

Axis 3: Bratislava-region programme Roma-relevance: 35 %

Axis 4: Building capacities and improving the quality of the public administration

No Roma-relevant projects

Axis 5: Technical support

Priority axis 1: Employment

• 80 projects of total value 32,840,855 €, average size 410,511 €

• Project owners tend to be commercial companies• Roma are often just one target group among many

(e.g. 1 working place)• Unclear how many targeted but even a project

with 1 Roma “ticked” as MRC relevant• Focus often on investments into existing staff –

Roma are disadvantaged due to their exclusion from formal labour market

Priority axis 2: Social inclusion. 2.1 and 2.2 574 (570) of total value 113,870,162 (48,891,967) €

Action 2.1: Social field work (69% or 59 %)• 491 (488) projects of total value: 78,951,068

(28,972,874) ۥ Ethnically defined beneficaries, relative small

average budgets sizes 160,796 (59 371) €)

Action 2.2: Employment , training (31% or 41%)• 83 (82) projects of total value 34,919,094

(19,919,094) €; Very various groups of beneficaries. Average budget size: 420,712 (242 916) €

Territorial aspects

a) Does the territorial distribution of the projects match the distribution of the Roma population?

99,2 – 62,2 % 61,9 – 35,6 % 35,4 – 21,6 % 21,4 – 10,7 % 10,4 – 4,3 % 4,3 – 0,01 %

Territorial distribution of costs

Examples:

Region Number of Roma

Number of beneficaries

Cost per

beneficiary

Share of Roma

reachedTrenčín 4 325

(0,72 %)

8 433 282 € 195 %

Nitra 25 437

(3,59%)

5193 602 € 20,42 %

Prešov 85 697

(10,77 %)

61 343 185 € 71,58 %

Košice 89 364

11,61 %

34 568 178 € 38,68 %

Slovakia total 284 566

(5.29%)

148 637 199 € 52 %

Features of targeted municipalities (villages only)

All municipalities in Roma Atlas (2004)

ESF-targeted municipalities

(all priorities)

ESF-targeted municipalities

(action 2.1)

ESF-targeted municipalities

(action 2.2)

Average size 1137 1467 1449 1536

Share of Roma

17,2 % 28,9 % 31,6 16,4 %

Unemployment rate (2004)

21 % 24 % 26 % 17 %

ESF projects in smaller-than-average villages (sample-based)

Number of municipalities included in Atlas of Roma communities

Percentage of municipalities reached by Roma-projects

Percentage of municipalities reached (priority 2.1)

Percentage of municipalities reached (priority 2.2)

All villages in Atlas (2004)

981 20 % 16 % 3 %

Less than 300 inhabitants

150 11 % 9 % 3 %

300 – 700 inhabitants

287 15 % 14 % 1 %

700 – 1135 inhabitants

193 22 % 19 % 3 %

Level of segregation and unemployment

Working index: 1.Location of settlement2.Type of settlement 3.Distance of the settlement4.Physical barrier5.Land ownership6.Access to electricity7.Public light8.Garbage collection in Roma settlement

Locations are on a rank between 0 (no segregation) to 15 (extreme level of segregation and underdevelopment)

Segregation and underdevelopment of targeted municipalities

Region All municipalities

All targeted Action 2.1 Action 2.2

Trenčín 1,01 1,75 2,0 0,0

Nitra 1,05 1,60 1,66 1,44

Prešov 2,39 2,51 2,89 1,59

Košice 2,87 2,84 2,95 2,98

Slovakia 2,14 2,37 2,53 1,90

Major findings

• A large number of applications is based on almost identical texts suggesting massive copy/paste

• The data base doesn’t allow even basic outcome evaluation (outcomes impossible to detect)

• Complexity of administrative procedures makes grant owners dependent on (expensive) consultancy services; various forms don’t generate meaningful data

• Even though Roma are highly overrepresented among the unemployed, a tiny fraction of projects under priority axis “Supporting Employment Growth” were labeled (on questionable grounds) as “MRC relevant”

• Information on the actual ethnic composition of projects´ target groups is missing, which makes it impossible to determine how many Roma are reached

The broader context

• The current setting does not allow robust outcome evaluation of ESF projects– Number of Roma beneficiaries is vague– Outcomes are not clearly defined and cannot be quantified

• Cost-benefit analysis is impossible (only costs/inputs and accounted for)

• The system is heavily skewed towards “social work” that is supposed to provide basic social assistance but does little to lift people out of poverty

Recommendations

• Simple changes:– Modify the application forms so that they generate meaningful

data– Applications should be encouraging “focus on outcomes and

allow for M&E

• Integrate the project level information in processable data management system (currently information is in pdf files)

• Modify the TSP workers functions and TORs so that they are part of a local level data generation system for generating project outputs and outcome related data