Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad.

Evaluating Media Assistance Programs:

What We Have Done and What We Have Learned

Lee B. Becker & Tudor Vlad

2

Background• University of Georgia

• Grady College of Journalism & Mass Communication

• James M. Cox Jr. Center for International Mass Communication Training and Research– Conducts training programs– Focuses on evaluation

Athens and the university in 1840 as depicted in the painting by George Cooke.

3

My Research Interests

• Media effects– Opinion and learning

• Organizational behavior and message construction– Characteristics of workers, including

educational credentials

• Sociology of education and labor markets– Needs assessment– Evaluation of impact

4

Evaluation of the Knight International Press Fellowship

Program

1. Interviewed 531 persons in 8 European and 3 Latin American Countries

2. At 31 people in each of the 11 countries

3. Field work completed in 1999.

5

Basic Distinction

• Assessing and monitoring program process.

• Measuring and monitoring program outcomes.

6

Locus of Impact

• On journalists– Attitudes– Behaviors

• On organizations

• On media system

• On society overall

7

Problems with Self-Reports

• Timing is important– Immediately after training is too soon– Duration of effect is variable

• Likely to overestimate effect– To satisfy the program provider– To justify own investment of time and effort

• Lacks reference– Most participants in training on projectile to change

• Participant may not know the answer

8

Journalists in Philippines

• Overestimated the amount of coverage they gave to the issues behind the conflict.

• Underestimated the amount of coverage of the Manila media to issues other than the conflict.

9

Measurement Solutions• Go from the general to the specific

– Now I want to start with a very general question. Please tell me some of the things that you learned from your participation in the XXX Fellowship that you consider at this point to have been most important to your work as a journalist. Of course, if you didn’t learn anything just tell me that.

– OK. Now I’m going to ask about some specifics. You may already have touched on some of these, but I want to make sure I cover everything. Did your participation in the XXX Fellowship help you develop your skills in findings sources for stories you have reported since you returned?

• Use what we can a “jab and probe” questioning technique.– Are there specific sources that you came into contact with

through the XX Fellowship that you still use today? • IF YES: Who are some of those?

10

Validation Measures• Interviews others who also can attest to

attitudes, behaviors and changes in both– Colleagues– Supervisors

• Examples– Since you completed the XXX Fellowship, have you

held any “brown bag” or other information discussions about your experiences in the program with other journalists?

– As far as you know, has (NAME OF FELLOW) made any efforts to share her/his experiences at the Fellowship with others at your organization since she/he returned?

11

Design Solutions

• Treat timing as a variable– We have compared short-term programs

conducted across time• Timing matters—effect several years• But topic matters too and can offset lag

• Develop a “control” group– True experimental design isn’t often possible– Imperfect “control” better than none

• Develop before and after design

12

Some Control Examples

• Control group made up of matched respondents– Successful applicants for subsequent ongoing

programs– Those who work in similar settings

• Can match based on lists • Can get respondent to help create a match

• Control groups made up of matched organizations

• Best control is the group itself– Before and after has limitation of systemic change

13

Post-Employment Training Common Part of Media Assistance

• When evaluating journalism training component of media assistance, keep in mind– Measurement problems– Design problems

• Use tactics to overcome them

14

Central Concerns of Media Assistance

• Concern with Media Freedom or media independence

• Concern with media performance

• Concern with the connection between Media Freedom and media performance

• Concern with the relationship between media performance and democracy

15

Mapping Media Assistance

• Effort by Monroe Price and colleagues

• Our purpose was to expand that

• Shifted to narrower short-term goal

• Rough estimate: $1 billion year spent on media assistance each year

• Ultimately, valuable to understand media assistance this way

16

Two Actor Groups

• Media assistance community– Donors– Media assistance providers– Targets of media assistance– Media monitors

• Academic community– Political scientists– Communication scientists

17

Academic Community

• Political scientists– Extensive literature on democratization– Extensive debate on meaning of democracy

• Communication scientists– Extensive literature on media freedom– Extensive debate on meaning of media freedom

• Relatively little empirical testing of contribution of media freedom to democracy or democratization

18

Different Perspectives of Advocates of Media Assistance

and Political and Communication Scientists

• Former believe media assistance leads to democratization– Role of the research is to document this

• Scientists think all of these questions are open to examination

19

Media Assistance

Trained, Skilled, Motivated Journalists

Media Organizations that Facilitate and Distribute the Work of Journalists

Independent Media System

Information Needed for Functioning of Democracy, Open Economy

Institutions of Civil Society

Functioning Democracy, Economy

Civil Society Assistance

?

Model of Impact of Media Assistance

20

Media Assistance








?


21

Does Media Assistance Lead to Trained, Skilled, Motivated

Journalists?

– Yes, probably• Evaluation is spotty

– Process evaluation rather than impact evaluation

• Often rely on self-reports of impact• Control groups are rarely used• Evaluations often not independent of funder,

media assistance provider

22

Media Assistance








?


23

Does Media Assistance Lead to Facilitative Media Organizations?

• Maybe– Most evidence is indirect, based on

observations of participants– Designs are inadequate– Many examples of failed investments– Problems of sustainability paramount

24

Media Assistance








?


25

Does Media Assistance Produce Independent Media Systems?

• Some limited concrete evidence it does

• Comparative study by Steven Finkel and colleagues for USAID– Found that USAID Media investments across

165 countries led to gains in media freedom– Used Freedom House indicators

• Considerable controversy over what is meant by media freedom

26

Media Assistance








?


27

Do Independent Media Produce Information Needed for

Democracy?• Almost no evidence either way

– No real explication of what information is needed for democracy

28

Media Assistance








?


29

Do Independent Media Produce Civil Society Institutions?

• No one knows

• Mixed evidence of the effectiveness of civil society assistance as well

30

Media Assistance








?


31

Do Independent Media Foster Democratization?

• Surprisingly little empirical evidence has been gathered– No understanding of what conditions might be

placed on this relationship

32

Six Hypotheses1. The media-supremacist position, which holds that media

freedom and independence produce democracy.2. The democracy-supremacist position, which holds that

democratic reform determines and produces media freedom and independence.

3. The media-freedoms-are-an-element-of-democracy position, which argues that media freedoms are a part of democracy and, as such, have no causal force leading to democracy.

4. The null-effect position, which holds that there is no relationship between media freedom and democracy.

5. A media-freedom-hinders-democracy argument.6. A democratization-hinders-media-freedoms stance.

33

Media Reform

Pretransition PrimaryTransition

SecondaryStage

Mature Stage

Dem

ocra

tizat

ion Pretransition X

Transition X

Consolidation X

MatureDemocracy X

Media Reform

Pretransition PrimaryTransition

SecondaryStage

Mature Stage

Dem

ocra

tizat

ion Pretransition X

Transition X

Consolidation X

MatureDemocracy X

Rozumilowicz Stages of Media Reform Linked to Stage Theory of Democracy

34

Year of Press Freedom

Political Rights +1

Political Liberty +1

Political Rights -1

Political Liberty -1

1981 0.79 0.82 0.79 0.81

1982 0.80 0.84 0.82 0.85

1983 0.79 0.82 0.81 0.85

1984 0.80 0.84 0.82 0.87

1985 0.82 0.83 0.82 0.87

1986 0.79 0.83 0.82 0.87

1987 0.77 0.81 0.81 0.86

1988 0.84 0.80 0.83 0.86

1989 0.82 0.77 0.93 0.89

1990 0.83 0.77 0.94 0.88

1991 0.81 0.77 0.93 0.91

1992 0.82 0.84 0.90 0.88

1993 0.81 0.81 0.89 0.85

1994 0.86 0.88 0.88 0.90

1995 0.88 0.91 0.88 0.91

1996 0.88 0.90 0.90 0.91

1997 0.88 0.90 0.90 0.92

1998 0.89 0.91 0.91 0.92

1999 0.89 0.91 0.90 0.93

2000 0.89 0.90 0.91 0.93

2001 0.91 0.92 0.90 0.93

2002 0.94 0.94 0.93 0.94

2003 0.93 0.94 0.94 0.95

2004 0.93 0.94

Med

ia D

emoc

racy

Li

nk

35

What We Know About Media Freedom Measures?

• Strong evidence of reliability across time• Strong evidence of reliability across measures• Some evidence of validity

– Criterion• FH measures reflected changes in Warsaw Pact countries

– Construct• IREX MSI related in predicted way in analysis of impact of

hypercompetition on press performance

36

37

Professional journalism and GDP per media outlets

0

0,5

1

1,5

2

2,5

3

020406080100120140160180

GDP in USD millions per media outlet

Pro

fess

iona

l jou

rnal

ism

Uzbekistan

Kosovo

Kazakhstan

Georgia

Croatia

BulgariaBosnia &

Herzegovina

Belarus

Azerbaijan

ArmeniaAlbania

Russia

Romania

Moldova

Macedonia

Kyrgyzstan

UkraineTajikistan

Serbia

Relationship between Market Competition and Press Performance

38

Limitations of Media Freedom Measures

• Focus on structure of media system

• Limited evidence of performance

• Limited notion of media independence– Focus on independence from government– Do not recognize pressures of commercial

dependence

• Largely ignore audience

• Largely ignore “needs” of democracy

39

Additional Monitoring

• Develop checklist of needs of democracy– A democratic state is a state in which all citizens

have access to information about how the state operates.

– A democratic state is one in which citizens have the ability to communicate to each other.

– A democratic state is one in which conflicts are managed without resort to violence.

– A democratic state is one in which representations of members of the state are presented in a way that foster appreciation and understanding.

40

Relationship between Press Freedom and Confidence in Media

41

Media Assistance Trained, Skilled,

Motivated Journalists



Information Needed for Functioning of Democracy

Institutions of

Civil Society

Functioning Democracy


?


42

Evaluating Media Assistance Programs: What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad.

Documents

answer slide

society overall slide

becker tudor vlad slide

program provider

media assistance programs

monitoring program process

monitoring program outcomes

media system