Publishing in Public Health

Publishing Roundtable:Publishing results ofprogram evaluations

Peter Winch, Houkje Ross, Jim Ricca

CORE Fall Membership

Meeting 2008

Some recent history

Session on publishing at 2008 CORE Spring Meeting in Atlanta: David Marsh & myself

Elluminate session “Documenting and Testing Innovations”, 15 Aug 2008

Today: Publishing Roundtable, focus on how to publish program evaluations

“Emergent” topic for today’s roundtable

CORE members frequently have exciting results from final project evaluations– Eagerness to disseminate the results

beyond the narrow audience for the final report

– Final results may represent many years of effort, want to demonstrate what has been achieved

Can final project evaluations be turned into peer-reviewed articles?

Challenges to publishing results of final evaluations

1. Estimation of effort required2. Identification of key messages3. Selection of journal4. Presentation of quantitative data5. Presentation of qualitative data

Challenges to publishing evaluations

1. Estimation of effort required Implementing the project, carrying out

the final evaluation, and writing the final evaluation report take much work

When people aim to publish a paper out of a final evaluation, there is often an assumption that most of the work is already done

Also difficult to decide who will do the work, as project staff and evaluation team may quickly move on to other things

Don’t Underestimate Level Of Effort

Remember What’s Involved in Article Development – Data and information collection– Field staff and others involved in the

project need to be available to answer any questions

– Dedicated person to write up the information

– Review team

Challenges to publishing evaluations

2. Identification of key messages

Final evaluation reports tend to be comprehensive, deal with all aspects of the project

One reason they are comprehensive is need to be accountable to donor for how funds were spent

No article can be comprehensive, need to strategically select a few key messages, and present data to support messages

Challenges to publishing evaluations

2. Identification of key messages

One approach is “abstract first”– Write ~ 300 word abstract prior to

working on the article– Good to start with a structured

abstract– Show abstract to some people not

involved in the project – Do they get excited about the article?

– Then outline the article, only including content directly related to the abstract

Abstract example #1

Belachew T, Nekatibeb H. Assessment of outpatient therapeutic programme for severe acute malnutrition in three regions of Ethiopia. East African Medical Journal. 2007; 84(12): 577-588.

This is a structured abstract, and the structure is indicated by the labels.

A bit long: 358 words.

OBJECTIVE: To document the experiences and lessons for rolling out of the OTP service at the wider scale with the aim of assessing the strengths and weaknesses of the project and suggest recommendations for future programming. DESIGN: Qualitative methods of data collection including focus group discussion, observation and in-depth interview of key informants were employed to get relevant data. Review of health facility, reports and programme documents were done to capture further- information. SETTING: Out Patient Programme (OTP) pilot programme implemented by CONCERN/ VALID in three administrative regions of Ethiopia namely: South Nations and Nationalities Peoples Regions (SPNNR), Addis Ababa and Oromia regions. A total of thirteen health centres which had started OTP service from the three regions were included in the study. SUBJECTS: Thirty six key informants and 30 focus group discussants were involved in the study conducted from 16th to 25th November 2006. RESULTS: Out Patient Programme (OTP) has enhanced community's understanding of malnutrition as a health problem through an excellent entry point it created for behaviour change communication (BCC) on optimal infant and young child feeding (IYCF). It has also enhanced utilisation of the existing equipments of the respective health services to promote nutrition and increased mental satisfaction of the providers who observed rapid recovery of malnourished children taking the plumpy nut. It also resulted in increased awareness of the community about malnutrition and its treatment, which resulted in increased need-based demand for the OTP and self-referral of children to health facilities. Shift in the thinking of the providers on the fact that malnutrition can be treated without admitting the child and reduction in the burden of malnutrition and associated mortality are other positive findings of the study. CONCLUSION: While it was observed that the programme was very effective in treating case of severe acute malnutrition and is highly acceptable by planners, health care providers and beneficiaries, there were different operational issues that needed to be strengthened. The irregularity and incompleteness of supply availability, high attrition of trained human power, inadequate supportive supervision especially from local ministry of health, inadequate community mobilisations are some of the shortcomings identified. Based of these findings recommendations were forwarded.

Abstract example #2

Sibley L, Buffington ST, Tedessa L Sr, McNatt K. Home-Based Life Saving Skills in Ethiopia: an update on the second phase of field testing. Journal of Midwifery and Womens Health. 2006;51(4): 284-291.

Not a structured abstract, but has a structure.

Much shorter: 195 words.

Home-Based Life Saving Skills (HBLSS) was integrated over 3 years into a district-level child survival project coordinated through the Ministry of Health and Save the Children Foundation/US in Liben Woreda, Guji Zone, Oromia Region, southern Ethiopia.During late 2004, the second phase of the program was reviewed for performance, home-based management, learning transfer, and program coverage. The immediate posttraining performance score for HBLSS guides for "First Actions" was 87% (a 78% increase over the pretraining baseline) and 79% at 1 year (a 9% decrease from the immediate posttraining score). The home-based management score of women attended by HBLSS guides for "First Actions" was 89%, compared to 32% for women assisted by other unskilled attendants. HBLSS guides teach women and families in the community as they were taught, by using pictorial Take Action Cards, role-play and demonstration, and a variety of venues. Estimates of HBLSS coverage suggest that HBLSS guides attended 24% to 26% of births, and 54% of women giving birth were exposed to HBLSS training. The HBLSS field tests demonstrate a promising program that increases access to basic care for poor, underserved, rural populations who carry the greatest burden of maternal and neonatal mortality.

Challenges to publishing evaluations

3. Selection of journal The best known journals rarely

publish articles based on program evaluations: Lancet, British Medical Journal, New England Journal of Medicine, American Journal of Public Health

They see their niche as publishing results of randomized trials

Challenges to publishing evaluations

3. Selection of journal Journals you might consider:

– Health Policy and Planning– Implementation Science– Tropical Medicine and International Health– Journals on maternal and child health

• International Journal of Gynecology and Obstetrics

• Journal of Midwifery and Women’s Health

• Journal of Tropical Paediatrics• Annals of Tropical Paediatrics

– Local/regional journals

Challenges to publishing evaluations

4. Presentation of quantitative data Focus on impact on mortality and

morbidity tends to cause problems with the reviewers, unless there was a strong study design: comparison group, randomization of who got the intervention, pre and post intervention measurements

Claims you are making should match the level of evidence

Although you are most excited about the mortality impact, stressing it may lead to rejection of your manuscript

What is your level of evidence?

Higher Separate

implementation & evaluation teams/staff

Prospective study Have comparison group Randomization of who

gets intervention Many communities/

units randomized Sample size calculation

based on hypothesis Quality control of data

collection & entry

Lower Implementation team/

staff also evaluate Retrospective or cross-

sectional study No randomization No comparison area at

all Only one intervention

and only one comparison area or district

No sample size calculation

Routine data, no or limited quality control

Level of evidence

Higher Lower

Causal attribution of change to prg.

Yes (or a qualified yes)

No - doesn’t control for other changes

Statements in your paper about the links between the observed changes and the intervention or program

“The intervention was effective in increasing X”“The intervention had a significant impact”“The program produced a large drop in mortality”“A triumph for humanity”“Two thumbs up”

“We observed higher coverage in the intervention area”“The intervention was associated with increases in X and Y”“We recorded fewer deaths in the intervention area, which may be due to X, Y or Z…..”

Challenges to publishing evaluations

5. Presentation of qualitative data

Key data may be qualitative Presenting quotes can use up much of

your word count Sometimes quotes can be presented

in tables Need to find other program evaluation

papers to see how they present qualitative data

Finding your focus

“The difficulty of literature is not to write, but to write what you mean.”

--Robert Louis Stevenson

YOUR PAPER NEEDS A

FOCUS

Consider Your Purpose & Audience

Who do you want to share the information with? – Other CORE members?– Outside audiences?– A specific publication?

For what purposes?– Sharing effective strategies – Publicity– Knowledge sharing among colleagues

Some Things to Focus On…

• New method( ie. No one has tried this before, you did, and you have promising results)

• A particular aspect of your program or project that worked well and achieved a measurable result

• A particular aspect of your program that changed something (like policy or behavior) in some way

Too many good ideas? Finding A Topic

Take your time in developing a topicReview what you knowBrainstorm—

–Allow yourself and/or your team time to consider the ideas

–Look for ideas that are manageable and doable

Try Clustering

Changing

Cultural Norms

Behavior

Change Method

s

Grandmother

SolidarityCircles

Training and

Supervision of

Volunteers

M & E

Revitalized Health

Huts

DOTS

Bamako

Initiative 1987

TB Services at Community

Level?

Cultural

Views of

disease What are

its component

s?

Senegal

Health Syste

m

Where are the Information Gaps?

After you cluster, your idea may become clear

Additional information gathering may be needed – Interviews with project staff– Requests for M&E data– Review of other literature to make

comparisons

What do You Mean?

Details are important. You don’t want your reader to have to ask, “what does that mean?”

It’s not enough to say: This project was innovative.

You must tell the reader what you mean when you say innovative.

Examples for discussion

1. Feedback from Lancet on manuscript estimated lives saved from NGO programs

2. Evaluation of CCF programs in Senegal

3. Evaluating quality in PEPFAR programs: URC

CSHGP Lives Saved Article Case Study

Submitted to The LancetMarch 2008

Publishing RoundtableCORE Fall MeetingOctober 2, 2008

CSHGP Lives Saved article: Background

Preliminary work had been done over the last 2 ½ years, continually refining the initial analysis

Various stages of the work had been presented at CORE x 3, USAID x 2, APHA x 2, and GHC; also in last three CSHGP Results Reports

There was a desire for a peer-reviewed publication to increase the credibility and visibility of the work done by PVOs generally and CSHGP grantees specifically

CSHGP Lives Saved article: Key Issues

What would be the main theme?– The method itself?– The level of impact of projects? (If so,

how to present results of “small” projects in a meaningful way?)

– The cost-effectiveness? (If so, what measure and what benchmark?)

Lancet wants RCTs. Was there a control that we could retrospectively try to use?

CSHGP Lives Saved article: Abstract

Methods

The IMPACT tool developed by the Child Health Epidemiology Reference Group (CHERG) was used to convert project-generated outcome data into estimates of impact on reducing child mortality. Regional (if available) or national trend in under-five mortality was subtracted from the modeled project impact. The additional mortality decline was attributed to the project. Project cost data were also analyzed. The additional costs of the project-associated community-oriented activities were divided by the incremental impact estimated using the IMPACT tool to give the incremental cost effectiveness. Contextual analysis for explanatory and confounding factors was done through a review of project documentation and interviews with project staff.

 Findings

These integrated community-oriented projects intervened on an average of 8·6 CHERG life-saving indicators (range 4 to 12). In the aggregate, they achieved statistically significant positive change on an average of 6·3 of these indicators (range 1 to 12). The interventions most responsible for the estimated mortality decline were preventive and delivered in community settings, although not exclusively so. These outcomes were modeled to result in a median estimated annual decline in under-five mortality of 5·6% (range 0·3 to 10·9%). Seventeen of the 22 projects are estimated to have an impact exceeding the comparison regional/national mortality trend, generating an estimated additional median annual mortality decline of 3·0% (range 0·0 to 9·1%). Among the top tercile of projects, the median estimated cost per project-attributable Disability Adjusted Life Year was $65 (range $22 to $119). A variety of specific community mobilization approaches were used by the projects in the top tercile, but all were strongly community-oriented, built on locally relevant community structures that could deliver a variety of interventions simultaneously, combined with behavior change strategies that focused heavily on peer support.

CSHGP Lives Saved article: Discussion

Discussion (pg. 20)

As concerted efforts are made to accelerate the achievement of MDG 4 with a focus on equity and health for all, it is essential to include effective and cost-effective community-oriented, integrated approaches as part of national packages. Although this analysis points to the importance of strong community mobilization approaches to child mortality reduction in underserved populations, further systematic study of these approaches is necessary.23,24 These data and lessons from practice contribute to the national and global evidence base that is critical to revitalizing a focus on primary health care, particularly at the district level. Such analysis generates lessons that can be used by global health partners to identify innovative and effective approaches that improve coverage, especially for vulnerable populations. Project- and program-level information from the CSHGP-funded NGO projects can be used as a starting point for the development of systematic studies, using more rigorous research designs, to develop and evaluate a mix of cost-effective delivery approaches. A shared agenda with NGOs that uses their contribution to program learning and improved monitoring will significantly contribute to further strengthening the partnerships necessary for achievement of MDG 4.

Selected Reviewer Comments

Reviewer #2Would it not be relevant to consider maternal mortality as an outcome as well as under 5?

Reviewer #3The contribution of this analysis is the general finding that NGOs can contribute to going to scale, and in theory can provide a testing ground for innovative delivery strategies.  Unfortunately, the use of pre-post designs with no comparison group for monitoring the effects of the NGO programs is too weak to support an analysis leading to any more definitive conclusions.

Reviewer #4: p. 20, para 3:  "Such analysis generates lessons…"  Unfortunately this analysis does not generate many lessons.  It does indicate that NGOs can reach significant populations of children, which is important and overlooked, and that simple adequacy evaluations suggest that NGO activities are associated with increases in coverage.  That's the only lesson that is supported by the analysis, except for the important methodological point that NGO projects need to have stronger evaluation designs so that their important contributions can be documented and used to improve program and project design.

Breastfeeding23.2%

ITN26.6%

IPT0.3%

Prompt antimalarial treatment

8.2%

Nutritional Interventions

3.1%

Vitamin A supplementation

7.8%

Measles vaccine

1.1%

Handwashing / POU / Safe feces disposal

6.1%

Clean home delivery (trained

attendant)2.5%

Skilled birth attendance

1.6%

Maternal tetanus toxoid1.9%

Antenatal care0.9% ORT

10.4%

Antibiotics for pneumonia

6.4%

Figure 3: Proportion of all estimated lives saved by intervention

Reviewer #4: I have some problems with Figure 3. The figure as it stands ONLY applies to these specific programmes and the specific areas and populations they covered. It is incorrect to interpret this as showing which interventions save the most lives, since not all interventions were included in all the programmes.

Discussion Points

Was the choice of journal appropriate? Why or why not?

What do we think about the reviewer comments? How would we address them?