Methods in Public Health Services and Systems Research

io

Methods in Public HealthServices and Systems Research

A Systematic Review

Jenine K. Harris, PhD, Kate E. Beatty, MPH, Colleen Barbero, MPPA, Alex F. Howard, MPH,Robin A. Cheskin, BA, Robert M. Shapiro II, MALS, Glen P. Mays, PhD, MPH

Context: PublicHealth Services and SystemsResearch (PHSSR) is concernedwith evaluating the organiza-tion, fınancing, anddeliveryofpublichealth services and their impactonpublichealth.The strengthof thecurrent PHSSR evidence is somewhat dependent on themethods used to examine the fıeld. Methodsused in PHSSR articles, reports, and other documents were reviewed to assess their methodologicstrengths and challenges in light of PHSSR goals.

Evidence acquisition: A total of 364 documents from the PHSSR librarymet the inclusion criteriaas empirical and based in the U.S. After additional exclusions, 327 of these were analyzed.

Evidence synthesis: A detailed codebook was used to classify articles in terms of (1) study design;(2) sampling; (3) instrumentation; (4) data collection; (5) data analysis; and (6) study validity.Inter-coder reliability was assessed for the codebook; once it was found reliable, the availableempirical documents were coded.

Conclusions: Although there has been a dramatic increase in the amount of published PHSSRrecently,methods used remain primarily cross-sectional and descriptive.Moreover, although appro-priate for exploratory and foundational work in a new fıeld, these approaches are limiting progresstoward somePHSSR goals. Recommendations are given to advance and strengthen themethods usedin PHSSR to better meet the goals and challenges facing the fıeld.(Am J Prev Med 2012;42(5S1):S42–S57) © 2012 American Journal of Preventive Medicine

sbto

i2bs

Context

Public health services and systems research(PHSSR) is a multidisciplinary fıeld of study con-cerned with evaluating the “organization, fınanc-

ng, and delivery of public health services and the impactf these services on public health.”1,2 PHSSR brings to-

gether a wide variety of research areas and theoretic andmethodologic traditions.1,3–5

Historically, the development of PHSSR has been slow,with progress not always apparent; however, develop-ment has begun to speed up over the last few decades. In1988, the influential IOM Future of Public Health report6

From the George Warren Brown School of Social Work (Harris, Barbero,Cheskin), Washington University in St. Louis, the School of Public Health(Beatty), Saint Louis University, St. Louis, Missouri; the College of PublicHealth (Howard), the Medical Center Library (Shapiro), University ofKentucky, Lexington, Kentucky; and the University of Arkansas for Medi-cal Sciences (Mays), Little Rock, Arkansas

Address correspondence to: Jenine K. Harris, PhD, Assistant Professor,George Warren Brown School of Social Work, Campus Box 1196, Wash-ington University in St. Louis, St. Louis MO 63130. E-mail: [email protected].

0749-3797/$36.00doi: 10.1016/j.amepre.2012.01.028

S42 Am J Prev Med 2012;42(5S1):S42–S57 © 2012 Amer

called for the development of the evidence base inPHSSR. In 1990, the DHHS decennial publication,Healthy People, set the goal that by 2000 at least 90%of thepopulation would be served by a public health depart-ment that effectively carries out the IOM’s core func-tions.2,7 In direct response, the CDC and the National As-ociation of County and City Health Offıcials (NACCHO)egan to research strategies for guidelines and self-assessmentools to measure how well public health agencies carriedut core functions.2

Between 2001 and 2009, the U.S. government investedmore than $10 billion in new funds to support public healthactivities.2 In the midst of these 10 years of public healthnvestment, CDC introduced its fırst PHSSR agenda (in003) and the Robert Wood Johnson Foundation (RWJF)egan to support efforts to convene representatives fromtates working on PHSSR (in 2006).2 In 2007, the RWJFawarded the University of Kentucky Research Foundationmore than $2.8 million for “Creating a resource center forpublic health systems and services research.”During this time of growth, six overarching goals for

PHSSR were defıned: (1) determine how public health

ican Journal of Preventive Medicine • Published by Elsevier Inc.

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S43

M

agency structure affects performance; (2) defıne and quan-tify dimensions of public health systems, including interor-ganizational relationships; (3) explore the relationship be-tween performance and health outcomes; (4) defıne thecharacteristics of high-performing local, state, and federalpublic health agencies; (5) explore the relationship betweensocial determinants of health and system performance; and(6) evaluate the costs of achieving and maintaining accept-able/optimal levels of performance.8

Meeting each of the six goals requires differentmethodologic strategies. The fırst, third, and fıfthgoals, for example, describe causal relationships. Thesegoals require research designs and analytic strategiesthat allow for hypothesis testing relating outcomes toexplanatory variables. In contrast, the second andfourth goals could be addressed using primarily de-scriptive statistics and qualitative information. Finally,the sixth goal might benefıt from an economic analyticstrategy such as cost-effectiveness or cost–benefıtanalysis. Despite the focus and investment in PHSSR,as recently as 2009 the fıeld was still considered under-developed.2,6 The adequate development of PHSSR is,in part, dependent on the methods used to understandthe fıeld. The goal of the current review is to examinethe research designs and analytic strategies used inPHSSR over the past 3 decades.

Evidence AcquisitionData SourceThe data sourcewas the PHSSR library developed by theUniversityof Kentucky Center for Public Health Services and Systems Re-search (CPHSSR). The PHSSR library incorporates a subset ofworks from the NLM’s Health Services and Sciences ResearchResources database (HSSRR)3; literature from other relevant data-bases; and grey literature primarily from theNewYorkAcademy ofMedicine’s Grey Literature Report. Beginning in 2006, a team oflibrarians and researchers at CPHSSR regularly search thesesources and identify literature for inclusion/exclusion; the searchmethods and inclusion/exclusion criteria used to develop thePHSSR library have been detailed in Scutchfıeld et al.3 As of Octo-ber 2010, when documents to date were collected for this study, thelibrary included 781 documents.

Study Eligibility CriteriaBecause this review focused on PHSSR methods, original em-pirical studies where information was derived from data (quan-titative or qualitative) were sought, rather than reviews, theo-retic, or editorial pieces. Abstracts or summaries for all but fourof the 781 documents were obtained (n�777; 99.5%) and eachwas coded for inclusion or exclusion by two coders indepen-dently. A document was included if it was empirical and basedin the U.S. Sixteen percent (n�125) required resolution by athird coder. A total of 364 (46.8%) articles met the inclusion

ay 2012

The coding process included two steps: (1) codebook develop-ment and (2) coding of articles.

Codebook Development and ReliabilityThe codebook was developed based on three sources: (1) a review9

of methods in research from the top journal in the fıeld of commu-nity psychology; (2) a paper from AcademyHealth10 summarizing66 methods-focused articles in PHSSR; and (3) discussion amongthe teammembers. Five PHSSR articles were selected from the dataand coded by all six coders to test the initial version of the code-book. Following in-depth discussions of the coding for these fıvearticles, the codebookwas refıned. The fınal codebook included fıvesections: (1) general characteristics; (2) data collection and analyticstrategy; (3) sampling; (4) research design; and (5) instrumenta-tion and validity (Appendix A, available online at www.ajpmonline.org).Before coding all articles, reliability testing was conducted to

ensure consistent classifıcation. Using a random start, a systematicsample of 23 articles (7% of the 333 articles) was coded for reliabil-ity. Percentage agreement among the six trained coders rangedfrom 66% to 95% across the 23 articles, with a mean percentageagreement of 80%. To account for the proportion of agreement thatmay have happened by chance, a modifıed form of the kappastatistic was calculated to account for multiple coders andmultipleitems.11

The intraclass correlation coeffıcient (ICC) across all 23 articleswas good, bordering on excellent (ICC�0.73) and ranged from0.53 to 0.92 for each article.11 Eleven articles had excellent agreementamong coders (ICC�0.75), whereas 12 had good agreement (0.4 �

ICC � 0.75). The overall kappa for the data set was 0.51. Accordingto Landis and Koch,12 this kappa represents a moderate amount ofagreement among coders beyond what would have happened bychance. Given acceptable reliability, each of the 333 articles wascoded individually by one of the six coders. Of the 333 articles, sixwere subsequently excluded for not fıtting the inclusion criteria ofbeing original empirical studies, leaving a sample size of 327 fordata analysis (41.9% of the original 781 articles; 89.8% of thosecoded for inclusion). Figure 1 shows this process.

The distribution of these 327 PHSSR articles showed increases inpublication volume that appear to coincide with the release of theinfluential papers (e.g., Future of Public Health) and the PHSSRfunding and organizational efforts described in the background

781 PHSSR documents in the library 4 documents not found

777 abstracts/summaries found

364 included data analysis and were U.S.-based

6 did not include data analysis327 included data analysis

333 full documents obtained31 not available

413 excluded386 did not include data analysis27 were international

Figure 1. Flow chart showing article inclusion into thesystematic review of public health services and systemsresearch (PHSSR)

section above (Figure 2).

cg

urrfvqcwm

e

(pii

vt

n

pdmtv

st

fpap7o

SSR

S44 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Data AnalysisDescriptive statistics and graphswere used to examine thedata. Data quality checkingwas conducted during theearly stages of analysis when-ever unexpected values wereobtained. For example, twoquantitative studies13,14 wereoded as having used focusroups to collect data.Because focus groups aresed most often in qualitativeesearch, these studies wereeviewed. In these two cases,ocus groups were used toote on topics and recorduantitative results, so theategorization of these studiesas not a coding or data entry error. A fewminor corrections wereade where data were improperly coded or recorded.

Evidence SynthesisThe following sections describe the use of research de-signs and sampling frames, data collection and analysismethods, sample size, power, and validity in 327 empiri-cal PHSSR studies.

Public Health Services and Systems ResearchStudy DesignsThemost commonly used quantitative research design inPHSSRwas cross-sectional, whereas case studies were themost frequently used qualitative design. Two-hundredsixty documents (79.5%)were journal articles; 48 (14.7%)were government reports; 18 (5.5%) were nongovern-ment reports; and one document (0.3%) was a book. Ofthese 327 documents, 224 (68.5%) used quantitative re-search designs; 45 (13.8%) used qualitative designs; and58 (17.7%) used a mixed methods research designs. Amajority of the 282 quantitative or mixed methods stud-ies were cross-sectional (n�228; 80.9%), followedby lon-gitudinal (n�29; 10.3%); case–control (n�9; 3.2%); quasi-xperimental (n�7; 2.5%); and experimental (n�7; 2.5%)(Figure 3).Of the228cross-sectionaldesigns, 59 (25.9%)were repeated

cross-sectional studies (cross-sectional data collected at twoormore time points). Themajority (n�46; 52.9%) of quali-tative or mixed studies (n�87) used a case-study approachFigure 4), and 53 (60.8%) collected data at a single timeeriod. Seventy-three (83.9%) of qualitative or mixed stud-es used primary data. Fifty-eight (17.7%) of the 327 studiesn this sample used amixedmethods research design.Mixedmethods approaches have been praised for pro-

iding complementary information and criticized for at-

Figure 2. Increase in the puand systems research (PH

empting to draw conclusions about a single phenome-

on from two irreconcilable perspectives.15,16 In thecontext of PHSSR, mixed methods approaches have beenused for many purposes. For example, Wheeler’s 2007 arti-cle17 examined the influence of a new smokefree hospitalolicy on consumer behavior through focus groups, in-epth interviews, and a cross-sectional survey. Anotherixedmethods studyexamined the roles and funding struc-

ures of local health departments through in-depth inter-iews, administrative data, and annual reports.18

Public Health Services and Systems ResearchSampling StrategiesNonprobability sampleswere themost frequently employedtype of sample in PHSSR. Of the 291 studies reporting onsampling frame, probability samples were used by 68(23.4%) studies and 69 (23.7%) used population samples,whereas nonprobability samples were found in 154 (52.9%)of the included studies. Thirty-eight (11.6%) studies did notreport what sort of sampling frame was used. The samplingstrategy varied by study design (Figure 5).Although often used in studies with nonprobability

ampling, inferential statistics are based on the assump-ion of a known probability for each observation.19 Non-probability samples donot have this quality.20 To accountor a nonprobability sample, researchers can conductower analysis to determine a sample size that wouldpproximate the results expected with a probability sam-le.19,20 Of the 132 nonprobability quantitative studies,9 (59.8%) used one or more inferential statistical meth-ds; fıve of these (6.3%) discussed power calculations.

Public Health Services and Systems ResearchData Collection StrategiesThe most commonly used data collection strategy inPHSSR was written surveys or questionnaires developed

ation of empirical and non-empirical public health services) documents over time

blic

for the purposes of the study. Data were collected using

www.ajpmonline.org

aws

iffipmte

p(ca

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S45

M

many strategies; the most frequent method was writtensurvey or questionnaire (n�145; 44.3%). Researchers of-ten created their own surveys, tailored to answering theirspecifıc research questions. For example, Abarca and col-leagues21 surveyed Florida county health departments onn annual basis to assess community capacity, using aeb-based survey developed as part of the Comprehen-ive Assessment, Strategic Success initiative.The next most commonly used data collectionmethod

nvolved extracting administrative data (n�77; 23.5%)rom existing databases. Administrative data were de-ıned as computerized records gathered for some admin-strative purpose (e.g., birth records, death records, hos-ital discharge fıles). The least common data collectionethods were focus groups (n�20; 6.1%) and observa-

ions (n�20; 6.1%). Distribution of data collection strat-gies is shown in Table 1.Public health services and systems research studies usedrimary data only (n�200; 61.2%); secondary data onlyn�116; 35.5%); or both (n�11; 3.4%). Primary data areollectedspecifıcally for the reportedstudy,whereas second-ry data are not collected specifıcally for the study at hand.

Figure 3. Quantitative research designs used in the quan-titative (n�224) and mixed methods (n�58) studies

Figure 4. Qualitative designs used in the qualitative

ay 2012

Public Health Services and Systems ResearchData CharacteristicsA majority of PHSSR studies collected quantitative data atthe individual level at a single point in time. Most PHSSRstudies collectedquantitativedata (n�224; 68.5%), followedby mixed (n�58; 17.7%). Strictly qualitative data were col-lected in only 45 (13.8%) of the studies.The majority of studies collected data on individuals

(n�198; 60.6%); Figure 6). The second most commonlevel of data collection was local health department(n�62; 19.0%) followed closely by community groups(n�48; 14.7%).More than one third of the studies (n�21;33.9%) of local health departments used secondary data,which were likely to have come from the NACCHO Pro-fıle Study of Local Health Departments.22

Surveys of local health departments have been conductedby NACCHO on a regular basis (1990, 1992, 1996, 2005,2008, and 2010), and it makes the data available for publichealthprofessionals andothers. Findings fromstudies usingNACCHO data often address the PHSSR goals related tohealth system structure and performance. For example, an

Figure 5. Sampling strategy and type of research design(n�284)

Table 1. Data collection method used in 327 PHSSRstudies, n (%)

Qualitative QuantitativeMixedmethods

Administrative data 4 (8) 61 (25) 12 (13)

Focus group 10 (20) 2 (1) 8 (9)

Interview 28 (56) 65 (27) 30 (32)

Observation 3 (6) 10 (4) 7 (8)

Survey 5 (10) 104 (43) 36 (39)

Total 50 (100) 242 (100) 93 (100)

PHSSR, public health services and systems research

m1samwnt(2

nmsl

l

S46 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

early study23 using theNACCHOdata examined coreoccu-pations in local health departments and found that numer-ous vacancies across the country resulting in reduced LHDeffectiveness in termsof responding tourgenthealth threats.Another NACCHO study24 found that LHD performancewas higher in LHDs that include full-time leadership anddiverse funding sources.Information related to poverty, education, racial, and

ethnic composition can all be found at the communitylevel using census data. The other fıve levels of datacollection—state, program or project, state health depart-ent, country, and region—were each used in fewer than0%of studies (n�25).Most studies (n�281; 85.9%) used aingle level ofdata,whereas46 studies (14.1%)collecteddatat two or more levels. Of the 46 studies collecting data atultiple levels, 24 (52.1%) were quantitative; four (8.7%)ere qualitative; and 18 (39.1%) were mixed methods. Fi-ally, most studies (n�210; 64.2%) collected data at oneime point. Data were collected at two time points by 12.5%n�41) of the studies and at three or more time points by3.2% (n�76) of the studies.

Power and Sample Size in Public HealthServices and Systems Research StudiesAlthoughmost studies reported sample size and responserate, few reported on power. Of the included studies,87.5% of the studies (n�286) reported the sample size.More than one third (37.0%; n�121) of the studies re-ported a response rate. Of these studies, the mean re-sponse rate was 75.6%. The lowest response rate reportedwas 3.8%. The highest response rate reported was 100%,reported by 12 studies.Power is defıned as the probability of notmaking a type

Figure 6. Percentage of studies collecting data at differentevels (n�327)

II error (1–�), or the probability of detecting an existing

onzero effect size. The power of a statistical test is deter-ined by three factors: number of observations (sampleize); the size of the effect in the population; and the alphaevel (�). Power increases with a larger effect size, largersample size, and more lenient alpha level.25 Of the 282quantitative or mixed studies, 5.7% (n�16) mentionedpower. Of those studies that mentioned power, 37.5%(n�6) reported adequate power.

Reliability and Validity of Measures in PublicHealth Services and Systems ResearchNearly one third of PHSSR studies reporteddevelopingnewinstruments; however, few reported testing the instrumentsor using validated or reliable existing instruments. Instru-ment quality directly affects data quality and study results.Instrument reliability is one statistical measure of how re-producible the data are from a given survey instrument.26

Instrument validity refers to the extent an instrument mea-sures what it intends to measure (p. 33). Although some-times appropriate in qualitative research, reliability and va-lidity testing as conceptualized here are primarily applicableto quantitative studies.Of the coded studies, 32.7% (n�107) reported devel-

oping an instrument; of these studies, 38.3% (n�41) re-ported validity and reliability testing of their newly devel-oped instrument. Fifteen studies (4.6%) reported testingthe validity or reliability of a new instrument, and 10.7%of studies (n�35) reported using validated and reliableinstruments. More than half of studies did not report oninstrumentation (n�170; 52.0%). This may, in part, bedue towidespread use of secondary data from sources liketheNACCHOProfıle Study, which has not been tested forreliability or validity.

Data Analysis in Public Health Services andSystems ResearchNearly all studies collecting quantitative data presenteddescriptive statistics, and more than half also conductedinferential statistics, whereas many of the qualitativestudies used thematic or content analysis strategies. De-scriptive statistics were reported for 262 of the 282(92.9%) quantitative studies. Overall, 60.2% (n�170) ofquantitative studies used inferential methods.Standard inferential methods including chi-square,

t test, correlation, ANOVA, and regressionwere used in amajority of inferential studies (n�130, 76.5%), whereasinferential methods associated with measurement (e.g.,factor analysis) were used in 28 of the 170 inferentialstudies (16.5%). Systemsmethods (e.g., network analysis;n�3; 1.1%); mapping or spatial analysis (e.g., use of GIS;n�5; 1.8%); and multilevel modeling (n�4; 1.4%) wereused infrequently. Of the quantitative and mixed studies

www.ajpmonline.org

mnrripros

trbtbnp

lta

desabOsr(cs

aclaP

m

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S47

M

15.2%) or using a longitudinal study design (n�28;9.9%), one study used structural equation modeling ormultilevel modeling.Of the 103 studies using qualitative or mixedmethods,

the twomost common qualitative data analysis strategieswere thematic analysis with 51.5% (n�53) and contentanalysis with 46.6% (n�48). Thematic analysis identifıesemerging themes, or patterns found in information. Forexample, Beitsch and colleagues27 used thematic analysisfor a study on establishing a national voluntary publichealth accreditation program; they reported commonthemes from a review of state applications to the Multi-State LearningCollaborative on Performance andCapac-ity Assessment or Accreditation of Public Health Depart-ments. Few qualitative studies included diagrams (n�7;6.8%) or typologies (n�4; 3.9%).

Validity in Public Health Services andSystems Research StudiesFew PHSSR studies discussed external or internal valid-ity. External validitymeasures howwell a study translatesto others outside the study population, whereas internalvalidity measures whether a study provides accurate un-biased estimates of the phenomena it purports to mea-sure. Generalizability is the main characteristic of exter-nal validity. A study that is generalizable has attempted toreduce sources of error variance to obtain results that gobeyond the study sample and apply to a larger group orpopulation.28–30 Of the 282 quantitative or mixed meth-ods studies, 53 (18.8%) reported on generalizability offındings. Six of the 53 (11.3%) reported that fındings weregeneralizable, whereas 47 (88.7%) reported that fındingswere limited or not generalizable beyond the study.Representativeness refers to how well a sample repre-

sents the population and is another component of exter-nal validity.30 Sixty-six (23.4%) of the quantitative orixed methods studies reported on the representative-ess of their sample. Twenty-two (32.4%) of these studieseported being representative,whereas 44 studies (64.7%)eported limited or no representativeness. Of studies us-ng nonprobability samples (n�170), which are morerone to selection bias, only 22.9% of studies (n�39)eported on the representativeness of the sample, andnly 8.8% (n�15) reported that their studies were repre-entative of the population.There are many biases that threaten internal validity;

his review identifıed three of the most common: self-eport bias, nonresponse bias, and recall bias. Self-reportias is the propensity for research participants to respondo researchers in ways that are desirable.31 Nonresponseias occurs when those who are part of the sample but doot participate are systematically different from partici-

ay 2012

exposures or events and is affected by characteristics ofthe event and respondent.33

Of the 282 quantitative andmixedmethods studies, 31(11.0%) reported self-report bias; fıve (1.8%) reportedrecall bias; and 23 (8.2%) reported nonresponse bias.Non-experimental research designs are most susceptibleto threats to internal validity given their lack of controlover study conditions and lack of ability to assess causeand effect.34 These three types of bias were primarilydiscussed in the CDC’s Morbidity and Mortality WeeklyReports (MMWRs) included in this review. EachMMWRtypically included a standard limitations section that dis-cussed representativeness, generalizability, and the po-tential for bias in the Editorial Note following themain text of the study.35,36 A few non-MMWR articlessuch as the one by Avery and colleagues37 also addressedimitations. However, aside from the MMWR, attentiono problems with internal validity was uncommon in therticles reviewed.The distribution of threats to internal validity by studyesign (cross-sectional, case–control, longitudinal, quasi-xperimental, experimental) was examined. Of the 31tudies reporting self-report bias, 29 were cross-sectionalnd two were case–control. Of the fıve reporting recallias, four were cross-sectional and one was case–control.f the 23 reporting nonresponse bias, all were cross-ectional. Themajority of threats to internal validity wereeported in cross-sectional studies, with 48 of the 22821.1%) cross-sectional studies reporting any of the threeommon threats to internal validity. Three case–controltudies reported threats to internal validity.

LimitationsAswithmany systematic reviews, the data source is one ofthe primary limitations for the current study. There aretwo possible weaknesses of the library produced by theCPHSSR. The fırst is that theremay be relevant items thathave not been included. This limitation speaks to a largerchallenge in PHSSR, which is the lack of a consistent termthat represents PHSSR in large databases. CPHSSR staffhas advocated for the inclusion of a “public health ser-vices and systems research” medical subject heading(MeSH) term, yet no such term exists to date. This, inpart, may be due to the lack of consistency in terms usedto describe the fıeld.5 The second weakness is that therere items included in the library that are not PHSSR. Thishallenge is related to the purpose of the library; theibrary is meant to serve the needs of PHSSR researchersnd therefore contains some items that may not beHSSR but that may be useful for PHSSR.Additionally,overall intercoder reliabilitywas foundtobeoderate; however, good or excellent reliability scores were

pCSmretspM

S48 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

although the codebookwas extensive, it was likely not com-prehensive. For example, collecting additional data thatwould allow formore specifıc conclusions, such as identify-ing which studies were appropriate for conducting powerand effect size calculations, would strengthen the fındings.In addition, there may be important characteristics of themethods utilized that were not identifıed.

DiscussionIn the past few years, the PHSSR library has experiencedrapid growth, including a substantial increase in the vol-ume of empirical PHSSR being disseminated throughreports and journal articles (Figure 2). The empiricalPHSSR examined here varied from surveillance studiesof influenza and other infectious and chronic dis-eases,36,38–40 to quantitative and qualitative studies ex-amining how public health agencies, workforce, and ed-ucational programs rate according to standards,41–47 toresearch on how public policy influences health and howPHSSR can influence policy.48–50Although the volumeofresearch seems to be catching up to the need for evidencein PHSSR, questions remain about the quality of themethods being used and therefore the quality of evidencebeing accumulated.The studies examined in this review demonstrated wide-

spread use of individual-level data, cross-sectional designs,and nonprobability samples, and limited use of inferentialandcomplex statistics.These characteristicspresentbarrierstomeeting several of the PHSSR research goals.8 For exam-le, although the surveillance studies (mostly from theDC) were based on Behavioral Risk Factor Surveillanceurvey data, which is collected using a probability sample,ostof these studiespresentedonlydescriptive informationather than taking advantage of the ability to make infer-nces based on a large national probability sample. In addi-ion,manyof the studies examiningpublichealth standards,uch as the 10 Essential Public Health Services, also usedrimarily descriptive statistics,with a fewexceptions such asays and colleagues’43 use of factor analysis to identify

dimensions of performance in local public health systems.Finally, the limited use of non–individual level data pointedto a disconnect between PHSSR goals, which primarily fo-cused on the public health system, andPHSSRdata sources.Equally importantly, these same study characteristics

may limit PHSSR in providing much needed externallyvalid, generalizable information to the fıeld. For example,much of the work on public policy and public health inPHSSR was qualitative and therefore not generalizable. Ithas been recommended that more PHSSR policy studiesattempt to link public policy to measurable quantitative

generalizable information on health outcomes associatedwith particular policies.50

Additional challenges to the quality of the evidencemight be found in the lack of validity and reliabilitystudies examining the many new instruments being de-veloped across the fıeld, and the lack of power analysesreported, whichmay hinder the contributions of negativestudy results. Therefore, the following recommendationsare designed to increase the ability of PHSSR to meetmany of its goals and challenges:

1. Increase the use of systems-level data, study designs,sampling frames, and analytic strategies that can betteranswer the complex questions facing PHSSR. Researchquestions in PHSSR that focus on causal relationships,such as the relationship between system structure andperformance, would benefıt from additional use ofstudy designs, sampling frames, and analytic strategiesthat are representative, generalizable, and can capturechange over time. Some of these qualities can be ad-dressed by simply taking advantage of the unique op-portunities that exist in PHSSR to conduct naturalexperiments and use creative quasi-experimental de-signs (e.g., regression-discontinuity) as public healthsystems grow and change.In addition, if PHSSR adopts study designs and sam-

pling frames that collect representative information,and uses inferential statistics where appropriate, thenumber of studies that meet the criteria for externalvalidity will increase. External validity in PHSSR stud-ies may be especially important when trying to buildsuccessful public health systems. For example, identi-fying characteristics of successful local or state healthdepartments is most useful when this information canbe generalized to the larger population of health de-partments. Finally, efforts are currently underway tocollect, harmonize, and validate PHSSR-specifıc largenational data sets (www.publichealthsystems.org/cphssr/MembershipResources/1411/SynopticAnalysis),which may aid in improving the availability and use ofPHSSR data that goes beyond the individual level.

2. Report study power where appropriate when negativeresults are found.Understanding whether the relation-ship examined in a study with negative results is notsignifıcant, or whether the study just did not haveenough power, will increase the strength of the evi-dence in PHSSR. Studies examining important ques-tions but having insuffıcient power might be revisited.Conversely, those studies with adequate power that didnot fınd a signifıcant effect could provide useful infor-mation about what does not work, and, depending onthe strength and accumulation of evidence, may not

www.ajpmonline.org

arwPt

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S49

M

3. Increase the testing of validity and reliability of existing(e.g., NACCHO Profıle Study) and new PHSSR instru-ments, alongwith increasing the use of already validatedinstruments. Like calculating and reporting study power,testing and reporting on the validity and reliability ofinstruments will strengthen the evidence and allowPHSSR to become more effıcient through the accumula-tion and use of consistent and valid measures.

Although this study is the largest systematic review ofPHSSR methods to date, many of these recommenda-tions are not new for PHSSR. These suggestions echosome of the fındings from prior work in 2009 by Scutch-fıeld and colleagues3 and the recent AcademyHealth pa-per “ANeedsAssessment forData andMethods in PublicHealth Systems Research.”3,10 The consistent messagecross these three papers seems clear: It is time for PHSSResearchers to purposefully adopt research strategies thatill improvemethodologic strength and sophistication ofHSSR in order to best answer the big questions facinghe fıeld.

Publication of this article was supported by a grant from theRobert Wood Johnson Foundation.The project teamwould like to thankMargaretHower for her

help in the initial codebook development andKari Lindberg forassisting with data entry.No fınancial disclosures were reported by the authors of this

paper.

References1. Mays GP, Halverson PK, Scutchfıeld FD. Behind the curve? What we

know and need to learn from public health systems research. J PublicHealth Manag Pract 2003;9(3):179–82.

2. Scutchfıeld FD,MaysGP, LurieN. Applying health services research topublic health practice: an emerging priority. Health Serv Res 2009;44(5Pt 2):1775–87.

3. Scutchfıeld FD, Lawhorn N, Ingram R, Pérez DJ, Brewer R, BhandariM. Public health systems and services research: dataset development,dissemination, and use. Public Health Rep 2009;124(3):372–7.

4. Savoia E, Massin-Short SB, Rodday AM, Aaron LA, Higdon MA,Stoto MA. Public health systems research in emergency prepared-ness: a review of the literature. Am J Prev Med 2009;37(2):150–6.

5. Harris JK, Beatty KE, Lecy JD, Cyr JM, Shapiro RM II. Mapping themultidisciplinary fıeld of public health services and systems research.Am J Prev Med 2011;41(1):105–11.

6. IOM. Committee for the Study of the Future of Public Health editor.The future of public health.WashingtonDC: National Academy Press,1998.

7. DHHS. Healthy People 2000. www.healthypeople.gov/publications/.8. LenawayD,Halverson P, Sotnikov S, TilsonH, Corso L,MillingtonW.

Public health systems research: setting a national agenda. Am J PublicHealth 2006;96(3):410–3.

9. Luke DA. Getting the big picture in community science: methods that

ay 2012

10. Holve E, Papa K, Arnold S, Ix M. A needs assessment for data andmethods in public health systems research. AcademyHealth 2010.

11. Fleiss JL. Measuring nominal scale agreement among many raters.Psychol Bull 1971;76(5):378–97.

12. Landis JR, Koch GG. The measurement of observer agreement forcategorical data. Biometrics 1977;33(1):159–74.

13. Baird JR, Carlson KJ. National Public Health Performance Stan-dards assessment: fırst steps in strengthening North Dakota’s publichealth system. J Public Health Manag Pract 2005;11(5):422–7.

14. Scutchfıeld FD, Beaulieu J, Ireson C, Buege A. Public health competen-cies required by managed care organizations. J Public Health ManagPract 2002;8(5):22–9.

15. Razum O, Gerharus A. Methodological triangulation in public healthresearch—advancement or mirage? [Editorial]. Trop Med Int Health1999;4(4):243–4.

16. Sale JEM, Lohfeld LH, Brazil K. Revisiting the quantitative-qualitativedebate: implications for mixed-methods research. Qual Quant 2002;36(1):43–53.

17. Wheeler JG, Pulley L, Felix HC, et al. Impact of a smoke-free hospitalcampus policy on employee and consumer behavior. PublicHealthRep2007;122(6):744–52.

18. Potter MA, Fitzpatrick T. State funding for local public health: obser-vations from six case studies. J Public Health Manag Pract 2007;13(2):163–8.

19. LoBiondo-Wood G, Haber J. Nursing research: methods and criticalappraisal for evidence-based practice. 7th ed. St. Louis MO: Mosby/Elsevier, 2010.

20. Babbie ER. The practice of social research. 12th ed. Belmont CA:Wadsworth Cengage, 2010.

21. Abarca C, Grigg CM, Steele JA, Osgood L, Keating H. Building andmeasuring infrastructure and capacity for community health assess-ment and health improvement planning in Florida. J Public HealthManag Pract 2009;15(1):54–8.

22. National Association of County and City Health Offıcials. 2005 Na-tional Profıle of Local Health Departments. 2006.

23. Gerzoff RB, Brown CK, Baker EL. Full-time employees of U.S. localhealth departments, 1992–1993. J Public Health Manag Pract1999;5(3):1–9.

24. Handler AS, Turnock BJ. Local health department effectiveness inaddressing the core functions of public health: essential ingredients. JPublic Health Policy 1996;17(4):460–83.

25. Hallahan M, Rosenthal R. Statistical power: concepts, procedures, andapplications. Behav Res Ther 1996;34(5-6):489–99.

26. LitwinMS.How tomeasure survey reliability and validity. BeverlyHillsCA: Sage Publications, 1995.

27. Beitsch LM, Mays G, Corso L, Chang C, Brewer R. States gatheringmomentum: promising strategies for accreditation and assessmentactivities in multistate learning collaborative applicant states. J PublicHealth Manag Pract 2007;13(4):364–73.

28. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. The dependabilityof behavioral measurements: theory of generalizability for scores andprofıles. New York: Wiley, 1972.

29. StreinerDL,NormanRN.Healthmeasurement scales: a practical guideto their development and use. 3rd ed. Oxford, UK: Oxford UniversityPress, 2003.

30. Rea L, Parker R. Selecting a representative sample. 3rd ed. San Fran-cisco CA: Jossey-Bass, 2005:114–27.

31. Donaldson SI, Grant-Valone EJ. Understanding self-report bias inorganizational behavior research. J Business Psychol 2002;17(2):245–60.

32. Aschengrau A, Seage GR III. Bias. Sudbury MA: Jones and Bartlett,2003;251–79.

33. Coughlin SS. Recall bias in epidemiologic studies. J Clin Epidemiol

S50 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

34. Szklo M, Nieto FJ. Basic study designs in analytical epidemiology.Sudbury MA: Jones and Bartlett, 2007.

35. CDC. Influenza vaccination coverage among children aged 6months–18 years—eight immunization information system sentinelsites, U.S., 2008–09 influenza season.MMWRMorbMortalWkly Rep2009;58(38):1059–62.

36. CDC. State-specifıc prevalence of obesity among adults—U.S., 2007.MMWRMorb Mortal Wkly Rep 2008;57(28):765–8.

37. Avery GH,Wholey DR, Christianson JB. Physician evaluations of caremanagement practices in Medicaid programs. Am J Manag Care2005;11(3):156–64.

38. CDC. Update: influenza activity—U.S., September 28, 2008-April 4,2009, and composition of the 2009–10 influenza vaccine. MMWRMorb Mortal Wkly Rep 2009;58(14):369–74.

39. CDC. State-specifıc influenza vaccination coverage among adults—U.S., 2006–07 influenza season. MMWR Morb Mortal Wkly Rep2008;57(38):1033–9.

40. CDC. Impact of expanded newborn screening—U.S., 2006. MMWRMorb Mortal Wkly Rep 2008;57(37(:1012–5.

41. Honore PA, Schlechte T. State public health agency expenditures:categorizing and comparing to performance levels. J Public HealthManag Pract 2007;13(2):156–62.

42. Joly BM, O’Rourke K, Tilson HH, Leonard JF. Use of national publichealth performance standards to assess Maine’s diabetes system. JPublic Health Manag Pract 2007;13(1):68–71.

43. Mays GP, McHugh MC, Shim K, et al. Identifying dimensions ofperformance in local public health systems: results from the NationalPublicHealth Performance Standards Program. J PublicHealthManagPract 2004;10(3):193–203.

44. Purcell JM. Recruiting the future public health workforce: an analysisof prospect communication among accredited Schools of PublicHealth. J Community Health 2009;34(3):216–21.

45. Scutchfıeld FD, Quimson S,Williams SJ, Hofstetter R. The demand fordoctorally prepared public health personnel in institutions of highereducation. Am J Prev Med 1988;4(5):298–301.

46. CDC. Assessment of epidemiology capacity in State Health Departments—U.S., 2009.MMWRMorbMortalWklyRep 2009;58(49):1373–7.

47. CDC. Status of state electronic disease surveillance systems—U.S.,2007. MMWRMorb Mortal Wkly Rep 2009;58(29):804–7.

48. Anderson LM, Brownson RC, Fullilove MT, et al. Evidence-basedpublic health policy and practice: promises and limits. Am J Prev Med2005;28(5S):226–30.

49. Stachenko S. Challenges and opportunities for surveillance data toinform public health policy on chronic non-communicable diseases:Canadian perspectives. Public Health 2008;122(10):1038–41.

50. Berkowitz B, Ivory J,Morris T. Rural public health: policy and researchopportunities. J Rural Health 2002;18S:186–96.

Appendix: Full List of Studies in the ReviewAbarca C, Grigg CM, Steele JA, Osgood L, Keating H. Building and mea-

suring infrastructure and capacity for community health assessmentand health improvement planning in Florida. J Public Health ManagPract 2009;15(1):54–8.

Ablah E, Tinius AM, Horn L, Williams C, Gebbie KM. Community healthcenters and emergency preparedness: an assessment of competenciesand training needs. J Community Health 2008;33(4):241–7.

Agee B, Gimbel RW. Assessing the legal and ethical preparedness of masterof public health graduates. Am J Public Health 2009;99(8):1505–9.

Ahluwalia IB, Bolen J. Lack of health insurance coverage among working-age adults, evidence from the Behavioral Risk Factor Surveillance Sys-tem, 1993–2006. J Community Health 2008;33(5):293–6.

Aiello AE, Coulborn RM, Perez V, Larson EL. Effect of hand hygiene oninfectious disease risk in the community setting: a meta-analysis. Am J

Alejos A, Weingartner A, Scharff DP, et al. Ensuring the success of localpublic health workforce assessments: using a participatory-based re-search approach with a rural population. Public Health 2008;122(12):1447–55.

Alexander LK, Dail K, Horney JA, et al. Partnering to meet training needs:a communicable-disease continuing education course for publichealth nurses in North Carolina. Public Health Rep 2008;123(S2):36–43.

Asch SM, Stoto M, Mendes M, et al. A review of instruments assessingpublic health preparedness. Public Health Rep 2005;120(5):532–42.

Avery GH, Wholey DR, Christianson JB. Physician evaluations of caremanagement practices in Medicaid programs. Am J Manag Care2005;11(3):156–64.

Axnick NW, KatzM, Schiffer C, JohnsonW, Cross F. Survey of city/countypublic health agencies to determine the development, use, and effect ofprogram performance standards. Am J Public Health 1986;76(6):692–4.

Baird JR, Carlson KJ. National public health performance standards assess-ment: fırst steps in strengthening North Dakota’s public health system.J Public Health Manag Pract 2005;11(5):422–7.

Baker EL. Pilot study of public health workforce competency, agency ca-pacity and performance. University of North Carolina at Chapel HillSchool of Public Health, North Carolina Institute for Public Health.Health Policy and Administration

Baker EL, Blumenstock JS, Jensen J, Morris RD, Moulton AD. Building thelegal foundation for an effective public health system. J LawMed Ethics2002;30(3S):48–51.

Baldwin LM, HollowWB, Casey S, et al. Access to specialty health care forrural American Indians in two states. J Rural Health 2008;24(3):269–78.

Bara D, Mcphillips-Tangum C, Wild EL, Mann MY. Integrating childhealth information systems in public health agencies. J Public HealthManag Pract 2009;15(6):451–8.

Baron S, Sinclair R, Payne-Sturges D, et al. Partnerships for environmentaland occupational justice: contributions to research, capacity and publichealth. Am J Public Health 2009;99(S3):S517–S525.

Basta NE, Edwards SE, Schulte J. Assessing public health department em-ployees’ willingness to report to work during an influenza pandemic. JPublic Health Manag Pract 2009;15(5):375–83.

Bastida E, Brown HS 3rd, Pagan JA. Persistent disparities in the use ofhealth care along the U.S.-Mexico border: an ecological perspective.Am J Public Health 2008;98(11):1987–95.

Baumbach J, Mueller M, Smelser C, Albanese B, Sewell CM. Enhancementof influenza surveillance with aggregate rapid influenza test results:New Mexico, 2003–2007. Am J Public Health 2009;99(S2):S372–S377.

Bazzoli GJ. Public-private collaboration in health and human service deliv-ery: evidence from community partnerships. Milbank Q 1997;75(4):533–61.

Beaulieu J, Scutchfıeld FD. Assessment of validity of the National PublicHealth Performance Standards: the local public health performanceassessment instrument. Public Health Rep 2002;117(1):28–36.

Beaulieu J, Scutchfıeld FD, Kelly AV. Content and criterion validity evalu-ation of National Public Health Performance Standards measurementinstruments. Public Health Rep 2003;118(6):508–17.

Beaulieu JE, Scutchfıeld FD, Kelly AV. Recommendations from testing ofthe National Public Health Performance Standards instruments. J Pub-lic Health Manag Pract 2003;9(3):188–98.

Beckett AB, Scutchfıeld FD, Pfeifle W, Hill R, Ingram RC. The forgotteninstrument: analysis of the National Public Health Performance Stan-dards Program governance instrument. J Public Health Manag Pract2008;14(4):E17–E22.

Befort CA,Orr S, Davis A, Ely A, Steiger K. Perspectives on research amongKansas County health department administrators. J Public Health

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S51

M

Beitsch LM,BrooksRG,GriggM,MenachemiN. Structure and functions ofstate public health agencies. Am J Public Health 2006;96(1):167–72.

Beitsch LM, Grigg M, Menachemi N, Brooks RG. Roles of local publichealth agencies within the state public health system. J Public HealthManag Pract 2006;12(3):232–41.

Beitsch LM,MaysG,Corso L, ChangC, Brewer R. States gatheringmomen-tum: promising strategies for accreditation and assessment activities inmultistate learning collaborative applicant states. J Public HealthManag Pract 2007;13(4):364–73.

Bekemeier B. Credentialing for public health nurses: personallyvalued . . . but not well recognized. Public Health Nurs 2007;24(5):439–48.

Bekemeier B. Nurses’ utilization and perception of the community/publichealth nursing credential. Am J Public Health 2009;99(5):944–9.

Bekemeier B, Riley CM, Padgett SM, Berkowitz B. Making the case: lever-aging resources toward public health system improvement in Turn-ing Point states. J Public Health Manag Pract 2007;13(6):649–54.

Bernet PM. Local public health agency funding: money begets money. JPublic Health Manag Pract 2007;13(2):188–93.

Bernstein AB, Gauthier AK. Defıning competition in markets: why andhow? Health Serv Res 1998;33(5 Pt 2):1421–38.

Birkhead GS, Davies J, Miner K, Lemmings J, Koo D. Developing compe-tencies for applied epidemiology: from process to product. PublicHealth Rep 2008;123(S 1):67–118.

Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in health: perspectives ofa multi-ethnic, multi-racial America. Health Aff (Millwood) 2007;26(5):1437–47.

Blendon RJ, Buhr T, Cassidy EF, et al. Disparities in physician care: experi-ences and perceptions of a multi-ethnic America. Health Aff (Mill-wood) 2008;27(2):507–17.

BoultonML, Lemmings J, BeckAJ. Assessment of epidemiology capacity instate health departments, 2001–2006. J Public Health Manag Pract2009;15(4):328–36.

Brooks RG, Beitsch LM, Street P, Chukmaitov A. Aligning public healthfınancing with essential public health service functions and NationalPublic Health Performance Standards. J Public Health Manag Pract2009;15(4):299–306.

Brosnan CA, Brosnan P, Therrell BL, et al. A comparative cost analysis ofnewborn screening for classic congenital adrenal hyperplasia in Texas.Public Health Rep 1998;113(2):170–8.

Broz D, Levin EC, Mucha AP, et al. Lessons learned from Chicago’s emer-gency response tomass evacuations caused byHurricane Katrina. Am JPublic Health 2009;99(8):1496–504.

Bryant-Stephens T, Kurian C, Guo R, Zhao H. Impact of a householdenvironmental intervention delivered by lay health workers on asthmasymptom control in urban, disadvantaged children with asthma. Am JPublic Health 2009;99(S3):S657–S665.

Buehler JW, Whitney EA, Berkelman RL. Business and public health col-laboration for emergency preparedness in Georgia: a case study. BMCPublic Health 2006;6:285.

Burckhardt CS, Anderson KL. The Quality of Life Scale (QOLS): reliability,validity, and utilization. Health Qual Life Outcomes 2003;1:60.

Burns LR, Lee JA. Hospital purchasing alliances: utilization, services, andperformance. Health Care Manage Rev 2008;33(3):203–15.

Butler J, Tews D, Raevsky C, et al. Accreditation/Performance assessmenton-site reviews in Michigan, Missouri, North Carolina, and Washing-ton: implications for States and an evolving national model. J PublicHealth Manag Pract 2007;13(4):395–403.

Cahn MA, Auston I, Selden CR, et al. The Partners in Information Accessfor the Public Health Workforce: a collaboration to improve and pro-tect the public’s health, 1995–2006. JMed Libr Assoc 2007;95(3):301–9.

Cantreill J, Hung D, FahsMC, Shelley D. Purchasing patterns and smokingbehaviors after a large tobacco tax increase: a study of Chinese Ameri-

ay 2012

CDC. Impact of expanded newborn screening—U.S., 2006. MMWRMorbMortal Wkly Rep 2008;57(37):1012–5.

CDC. National, state, and local area vaccination coverage among childrenaged 19-35 months—U.S., 2007. MMWR Morb Mortal Wkly Rep2008;57(35):961–6.

CDC. State-specifıc influenza vaccination coverage among adults—U.S.,2006–07 influenza season. MMWR Morb Mortal Wkly Rep 2008;57(38):1033–9.

CDC. Use of enhanced surveillance for hepatitis C virus infection to detecta cluster among young injection-drug users—New York, November2004-April 2007. MMWR Morb Mortal Wkly Rep 2008;57(19):517–21.

CDC. State-specifıc prevalence of obesity among adults—U.S., 2007.MMWRMorb Mortal Wkly Rep. 2008 57;28(765–8.

CDC. Subpopulation estimates from the HIV incidence surveillancesystem—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008;57(36):985–9.

CDC. Impact of seasonal influenza-related school closures on families—Southeastern Kentucky, February 2008. MMWR Morb Mortal WklyRep 2009;58(50):1405–9.

CDC. Reduced hospitalizations for acute myocardial infarction after im-plementation of a smoke-free ordinance—City of Pueblo, Colorado,2002–2006. MMWR Morb Mortal Wkly Rep 2009;57(51):1373–7.

CDC. Status of state electronic disease surveillance systems—U.S., 2007.MMWRMorb Mortal Wkly Rep 2009;58((29):804–7.

CDC. 2009 pandemic influenza A (H1N1) virus infections—Chicago, Illi-nois, April-July 2009. MMWR Morb Mortal Wkly Rep 2009;58(33):913–8.

CDC. Adult blood lead epidemiology and surveillance—U.S., 2005–2007.MMWRMorb Mortal Wkly Rep 2009;58(14):365–9.

CDC. Deaths related to 2009 pandemic influenza A (H1N1) among Amer-ican Indian/Alaska Natives—12 states, 2009. MMWR Morb MortalWkly Rep 2009;58(48):1341–4.

CDC. Effectiveness of 2008–09 trivalent influenza vaccine against 2009pandemic influenza A (H1N1)—U.S., May-June 2009. MMWR MorbMortal Wkly Rep 2009;58(44)::1241–5.

CDC. Influenza vaccination coverage among children aged 6–23months—U.S., 2007–08 influenza season. MMWR Morb Mortal Wkly Rep2009;58(38):1063–6.

CDC. Influenza vaccination coverage among children aged 6 months–18years—eight immunization information system sentinel sites, U.S.,2008–09 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(38):1059–62.

CDC. Influenza vaccination coverage among children and adults—U.S.,2008–09 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(39):1091–5.

CDC. National, state, and local area vaccination coverage among adoles-cents aged 13–17 years—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(36):997–1001.

CDC. National, state, and local area vaccination coverage among childrenaged 19-35 months—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(33):921–6.

CDC. Norovirus outbreaks on three college campuses—California, Michi-gan, and Wisconsin, 2008. MMWR Morb Mortal Wkly Rep2009;58(39):1095–100.

CDC. Performance of rapid influenza diagnostic tests during two schooloutbreaks of 2009 pandemic influenza A (H1N1) virus infection—Connecticut, 2009. MMWR Morb Mortal Wkly Rep 2009;58(37):1029–32.

CDC. Preliminary FoodNet Data on the incidence of infection with patho-gens transmitted commonly through food—10 States, 2008. MMWRMorb Mortal Wkly Rep 2009;58(13):333–7.

CDC. Reduction in rotavirus after vaccine introduction—U.S., 2000–2009.

S52 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

CDC. State-specifıc prevalence and trends in adult cigarette smoking—U.S., 1998–2007. MMWRMorb Mortal Wkly Rep 2009;58(9):221–6.

CDC. State-specifıc secondhand smoke exposure and current cigarettesmoking among adults—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(44):1232–5.

CDC. Surveillance for pediatric deaths associated with 2009 pandemicinfluenza A (H1N1) virus infection—U.S., April-August 2009.MMWRMorb Mortal Wkly Rep 2009;58(34):941–7.

CDC. Update: influenza activity—U.S., August 30-October 31, 2009.MMWRMorb Mortal Wkly Rep 2009;58(44):1236–41.

CDC. Update: influenza activity—U.S., April-August 2009. MMWRMorbMortal Wkly Rep 2009;58(36):1009–12.

CDC. Update: influenza activity—U.S., September 28, 2008-April 4, 2009,and composition of the 2009–10 influenza vaccine. MMWR MorbMortal Wkly Rep 2009;58(14):369–74.

CDC. Assessment of epidemiology capacity in State Health Departments—U.S., 2009. MMWRMorb Mortal Wkly Rep 2009;58(49):1373–7.

CDC. Intent to receive influenza A (H1N1) 2009 monovalent and seasonalinfluenza vaccines—two counties, North Carolina, August 2009.MMWRMorb Mortal Wkly Rep 2009;58(50):1401–5.

CDC. Assessment of epidemiology capacity in State Health Departments—U.S., 2009. MMWRMorb Mortal Wkly Rep 2009;58(49):1373–7.

CDC. Impact of expanded newborn screening—U.S., 2006. MMWRMorbMortal Wkly Rep 2008;57(37):1012–5.

CDC. Influenza vaccination coverage among children aged 6 months–18years—eight immunization information system sentinel sites, U.S.,2008–09 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(38):1059–62.

CDC. Influenza vaccination coverage among children aged 6–23months—U.S., 2007–08 influenza season. MMWR Morb Mortal Wkly Rep2009;58(38):1063–6.

CDC. Influenza vaccination coverage among children and adults—U.S.,2008–09 influenza season. MMWR Morb Mortal Wkly Rep 2009;58(39):1091–5.

CDC. Laboratory surveillance for wild and vaccine-derived polioviruses—worldwide, January 2008-June 2009. MMWR Morb Mortal Wkly Rep2009;58(34):950–4.

CDC. National laboratory inventories for wild poliovirus containment—Western Pacifıc region, 2008. MMWR Morb Mortal Wkly Rep2009;58(35):975–8.

CDC. National, state, and local area vaccination coverage among adoles-cents aged 13–17 years—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(36):997–1001.

CDC. National, state, and local area vaccination coverage among childrenaged 19-35 months—U.S., 2007. MMWR Morb Mortal Wkly Rep2008;57(35):961–6.

CDC. National, state, and local area vaccination coverage among childrenaged 19-35 months—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(33):921–6.

CDC. Progress in immunization information systems—U.S., 2008.MMWRMorb Mortal Wkly Rep 2010;59(5):133–5.

CDC. Reduced hospitalizations for acutemyocardial infarction after imple-mentation of a smoke-free ordinance–City of Pueblo, Colorado, 2002–2006. MMWRMorb Mortal Wkly Rep 2009;57(51):1373–7.

CDC. Reduction in rotavirus after vaccine introduction—U.S., 2000–2009.MMWRMorb Mortal Wkly Rep 2009;58(41):1146–9.

CDC. State-specifıc influenza vaccination coverage among adults—U.S.,2006–07 influenza season. MMWR Morb Mortal Wkly Rep 2008;57(38):1033–9.

CDC. State-specifıc prevalence of obesity among adults—U.S., 2007.MMWRMorb Mortal Wkly Rep 2008;57(28):765–8.

CDC. State-specifıc secondhand smoke exposure and current cigarettesmoking among adults—U.S., 2008. MMWR Morb Mortal Wkly Rep2009;58(44):1232–5.

CDC. Status of state electronic disease surveillance systems—U.S., 2007.

CDC. Subpopulation estimates from the HIV incidence surveillancesystem—U.S., 2006. MMWR Morb Mortal Wkly Rep 2008;57(36):985–9.

CDC. Surveillance for pediatric deaths associated with 2009 pandemicinfluenza A (H1N1) virus infection–U.S., April-August 2009. MMWRMorb Mortal Wkly Rep 2009;58(34):941–7.

Chan B, Feldman R, Manning WG. The effects of group size and groupeconomic factors on collaboration: a study of the fınancial performanceof rural hospitals in consortia. Health Serv Res 1999;34(1 Pt 1):9–31.

Chandler T, Qureshi K, Gebbie KM, Morse SS. Teaching emergency pre-paredness to public health workers: use of blended learning in web-based training. Public Health Rep 2008;123(5):676–80.

CheadleA,HsuC, Schwartz PM, et al. Involving local health departments incommunity health partnerships: evaluation results from the part-nership for the public’s health initiative. J Urban Health 2008;85(2):162–77.

Chen LS, Kwok OM, Goodson P. U.S. health educators’ likelihood ofadopting genomic competencies into health promotion. Am J PublicHealth 2008;98(9):1651–7.

Chenoweth D, Estes C, Lee C. The economic cost of environmental factorsamong North Carolina children living in substandard housing. Am JPublic Health 2009;99(S3):S666–S674.

Chesson HW, Harrison P, Scotton CR, Varghese B. Does funding for HIVand sexually transmitted disease prevention matter? Evidence frompanel data. Eval Rev 2005;29(1):3–23.

Chintapalli S, Goodman M, Allen M, et al. Assessment of a commercialsearchable population directory as a means of selecting controls forcase-control studies. Public Health Rep 2009;124(3):378–83.

Cioffı JP, Lichtveld MY, Tilson H. A research agenda for public healthworkforce development. J Public Health Manag Pract 2004;10(3):186–92.

Clark NM, Lachance L, Doctor LJ, et al. Policy and system change andcommunity coalitions: outcomes from allies against asthma. Am J Pub-lic Health 2010;100(5):904–12.

Colon-Ramos U, Atienza AA, Weber D, Taylor M, Uy C, Yaroch A. Prac-ticing what they preach: health behaviors of those who provide healthadvice to extensive social networks. J Health Commun 2009;14(2):119–30.

CosbyAG,BowserDM.Thehealth of theDeltaRegion: a story of increasingdisparities. J Health Hum Serv Adm 2008;31(1):58–71.

Costich JF, Honore PA, Scutchfıeld FD. Public health fınancial manage-ment needs: report of a national survey. J Public Health Manag Pract2009;15(4):307–10.

Costich JF, Scutchfıeld FD. Public health preparedness and response capac-ity inventory validity study. J Public Health Manag Pract 2004;10(3):225–33.

Curry CW, De AK, Ikeda RM, Thacker SB. Health burden and funding atthe Centers for Disease Control and Prevention. Am J Prev Med2006;30(3):269–76.

Dausey DJ, Chandra A, Schaefer AG, et al. Measuring the performance oftelephone-based disease surveillance systems in local health depart-ments. Am J Public Health 2008;98(9):1706–11.

Davis MV, Cannon MM, Corso L, Lenaway D, Baker EL. Incentives toencourage participation in the national public health accreditationmodel: a systematic investigation. Am J Public Health 2009;99(9):1705–11.

Davis MV, Macdonald PD, Cline JS, Baker EL. Evaluation of public healthresponse to hurricanes fınds North Carolina better prepared for publichealth emergencies. Public Health Rep 2007;122(1):17–26.

Declercq E, Caldwell K, Hobbs SH, Guyer B. The changing pattern ofdoctoral education in public health from1985 to 2006 and the challengeof doctoral training for practice and leadership. Am J Public Health2008;98(9):1565–9.

Defriese GH, Hetherington JS, Brooks EF, et al. The program implicationsof administrative relationships between local health departments and

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S53

M

Delpierre C, Lauwers-Cances V, Datta GD, Berkman L, Lang T. Impact ofsocial position on the effect of cardiovascular risk factors on self-ratedhealth. Am J Public Health 2009;99(7):1278–84.

Derose SF, Asch SM, Fielding JE, Schuster MA. Developing quality indica-tors for local health departments—Experience in Los Angeles County.Am J Prev Med 2003;25(4):347–57.

Downey LH, Ireson CL, Scutchfıeld FD. The use of photovoice as a methodof facilitating deliberation. Health Promot Pract 2008;2008/03/07.

Driscoll D, Rojas Smith L, Sotnikov S, et al. An instrument for assessingpublic health system performance: validity in rural settings. J RuralHealth 2006;22(3):254–9.

Duncan AR, Priest PC, Jennings LC, Brunton CR, BakerMG. Screening forinfluenza infection in international airline travelers. Am J PublicHealth2009;99(S2):S360–S362.

Dymova N, Hanumara RC, Enander RT, Gagnon RN. Use of the global teststatistic as a performance measurement in a reanalysis of environmen-tal health data. Am J Public Health 2009;99(10):1739–41.

EdgarM,Mayer JP, Scharff DP. Construct validity of the core competenciesfor public health professionals. J Public Health Manag Pract 2009;15(4):E7–E16.

Edwards JC, Kang J, Silenas R. Promoting regional disaster preparednessamong rural hospitals. J Rural Health 2008;24(3):321–5.

Ellis BH, Bannister WM, Cox JK, et al. Utilization of the propensity scoremethod: an exploratory comparison of proxy-completed to self-completed responses in the Medicare Health Outcomes Survey. HealthQual Life Outcomes 2003;1:47.

Ellison JH. National Public Health Performance Standards: are they ameans of evaluating the local public health system? J Public HealthManag Pract 2005;11(5):433–6.

Eng TR. Prevention of sexually transmitted diseases—a model for over-coming barriers between managed care and public health. Am J PrevMed 1999;16(1):60–9.

Erwin PC. A self-assessment process for accreditation preparedness: apractical example for local health departments. J Public Health ManagPract 2009;15(6):503–8.

Erwin PC, Sheeler L, Lott JM. A shot in the rear, not a shot in the dark:application of a mass clinic framework in a public health emergency.Public Health Rep 2009;124(2):212–6.

Farhang L, Bhatia R, Scully CC, Corburn J, Gaydos M, Malekafzali S.Creating tools for healthy development: case study of San Francisco’sEasternNeighborhoodsCommunityHealth Impact Assessment. J Pub-lic Health Manag Pract 2008;14(3):255–65.

Fielding S, Fayers PM, Mcdonald A, Mcpherson G, Campbell MK. Simpleimputation methods were inadequate for missing not at random(MNAR) quality of life data. Health Qual Life Outcomes 2008;6:57.

Fielding S, Fayers PM, Ramsay CR. Investigating the missing data mecha-nism in quality of life outcomes: a comparison of approaches. HealthQual Life Outcomes 2009;7:57.

Findley SE, Irigoyen M, Sanchez M, et al. Effectiveness of a communitycoalition for improving child vaccination rates in New York City. Am JPublic Health 2008;98(11):1959–62.

Ford EW, Wells R, Bailey B. Sustainable network advantages: a gametheoretic approach to community-based health care coalitions. HealthCare Manag Rev 2004;29(2):159–69.

Foster-Fishman PG, Pierce SJ, Van Egeren LA. Who participates and why:building a process model of citizen participation. Health Educ Behav2009;36(3):550–69.

FoxM, Curriero F, Kulbicki K, Resnick B, Burke T. Evaluating the commu-nity health legacy of WWI chemical weapons testing. J CommunityHealth 2010;35(1):93–103.

Garcia R, Fenwick C. Social science, equal justice, and public health policy:lessons from Los Angeles. J Public Health Policy 2009;30(S1):S26–S32.

George A, Springer C, Haughton B. Retirement intentions of the publichealth nutrition workforce. J Public Health Manag Pract 2009;

ay 2012

Gerzoff RB, Brown CK, Baker EL. Full-time employees of U.S. local healthdepartments, 1992–1993. J Public Health Manag Pract 1999;5(3):1–9.

Gerzoff RB, Gordon RL, Richards TB. Recent changes in local healthdepartment spending. J Public Health Policy 1996;17(2):170–80.

Ginsburg M, Goold SD, Danis M. (De)constructing “basic” benefıts: citi-zens defıne the limits of coverage. Health Aff (Millwood) 2006;25(6):1648–55.

Gizzi C, Klementiev A, Britt J, Cruz-Uribe F. The use of assessment inpromoting secondhand smoke policy in a local health jurisdiction. JPublic Health Manag Pract 2009;15(1):41–6.

Goes JB, Zhan C. The effects of hospital-physician integration strategies onhospital fınancial performance. Health Serv Res 1995;30(4):507–30.

Goldman RH, Cohen AP, Sheahan F. Using seminar blogs to enhancestudent participation and learning in public health school classes. Am JPublic Health 2008;98(9):1658–63.

Goldsmith LJ, Ricketts TC. Proposed changes to designations of medicallyunderserved populations and health professional shortage areas: effectson rural areas. J Rural Health 1999;15(1):44–54.

Goodman RM. A construct for building the capacity of community-basedinitiatives in racial and ethnic communities: a qualitative cross-caseanalysis. J Public Health Manag Pract 2009;15(2):E1–E8.

Goody B. Defıning rural hospital markets. Health Serv Res 1993;28(2):183–200.

Goold SD, Fessler D,Moyer CA. Ameasure of trust in insurers. Health ServRes 2006;41(1):58–78.

Goold SD, Klipp G. Managed care members talk about trust. Soc Sci Med2002;54(6):879–88.

Gordon RL, Baker EL, Roper WL, Omenn GS. Prevention and the reform-ing U.S. health care system: changing roles and responsibilities forpublic health. Annu Rev Public Health 1996;17:489–509.

Gordon RL, Gerzoff RB, Richards TB. Determinants of U.S. local healthdepartment expenditures, 1992 through 1993. Am J Public Health1997;87(1):91–5.

Graham JP, Leibler JH, Price LB. The animal-human interface and infec-tious disease in industrial food animal production: rethinking biosecu-rity and biocontainment. Public Health Rep 2008;123(3):282–99.

Griffın SO, Jones KA, Lockwood S, Mosca NG, Honore PA. Impact ofincreasing Medicaid dental reimbursement and implementing schoolsealant programs on sealant prevalence. J Public Health Manag Pract2007;13(2):202–6.

Gwynn RC, Garg RK, Kerker BD, Frieden TR, Thorpe LE. Contributions ofa local health examination survey to the surveillance of chronic andinfectious diseases in New York City. Am J Public Health 2009;99(1):152–9.

Haffer SC. Using multiple survey vendors to collect health outcomes infor-mation: how accurate are the data? Health Qual Life Outcomes2003;1:6.

Hajat A, Cilenti D, Harrison LM, et al. What predicts local public healthagency performance improvement? A pilot study in North Carolina JPublic Health Manag Pract 2009;15(2):E22–33.

Halverson PK, Mays GP, Kaluzny AD. Working together? Organizationaland market determinants of collaboration between public health andmedical care providers. Am J Public Health 2000;90(12):1913–6.

Handler AS, Turnock BJ. Local health department effectiveness in address-ing the core functions of public health: essential ingredients. J PublicHealth Policy 1996;17(4):460–83.

Hanusaik N, O’loughlin JL, Kishchuk N, Eyles J, Robinson K, Cameron R.Building the backbone for organisational research in public healthsystems: development of measures of organisational capacity forchronic disease prevention. J Epidemiol Community Health 2007;61(8):742–9.

Harden A, Garcia J, Oliver S, et al. Applying systematic review methods tostudies of people’s views: an example from public health research. J

S54 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Harrison KM, Kajese T, Hall HI, Song R. Risk factor redistribution of thenational HIV/AIDS surveillance data: an alternative approach. PublicHealth Rep 2008;123(5):618–27.

Helmkamp JC, Aitken ME, Lawrence BA. ATV and bicycle deaths andassociated costs in the U.S., 2000–2005. Public Health Rep 2009;124(3):409–18.

Hempstead K. The accuracy of a death certifıcate checkbox for diabetes:early results from New Jersey. Public Health Rep 2009;124(5):726–32.

HendryxM,AhernMM.Mortality inAppalachian coalmining regions: thevalue of statistical life lost. Public Health Rep 2009;124(4):541–50.

Hessler BJ, Soper P, Bondy J, Hanes P, Davidson A. Assessing the relation-ship between health information exchanges and public health agencies.J Public Health Manag Pract 2009;15(5):416–24.

Holtgrave DR, Kates J. HIV incidence and CDC’s HIV prevention budget:an exploratory correlational analysis. Am J Prev Med 2007;32(1):63–7.

Honoré PA, Simoes EJ, Jones WJ, Moonesinghe R. Practices in publichealth fınance: an investigation of jurisdiction funding patterns andperformance. J Public Health Manag Pract 2004;10(5):444–50.

Honoré PA, Simoes EJ, Moonesinghe R, Kirbey HC, Renner M. Applyingprinciples for outcomes-based contracting in a public health program. JPublic Health Manag Pract 2004;10(5):451–7.

Honoré PA, Simoes EJ, Moonesinghe R, Wang X, Brown L. Evaluating theecological association of casino industry economic development oncommunity health status: a natural experiment in the Mississippi Deltaregion. J Public Health Manag Pract 2007;13(2):214–22.

Horney JA. Evaluation of the Certifıcate in Community Preparedness andDisaster Management Program at the University of North CarolinaGillings School of Global Public Health. Public Health Rep 2009;124(4):610–6.

Hosler AS. Retail food availability, obesity, and cigarette smoking in ruralcommunities. J Rural Health 2009;25(2):203–10.

Hurley R, Grossman J, Lake T, Casalino L. A longitudinal perspective onhealth plan-provider risk contracting. Health Aff (Millwood) 2002;21(4):144–53.

Hurley RE, Grossman JM, Strunk BC. Medicare contracting risk/Medicarerisk contracting: a life-cycle view from twelve markets. Health Serv Res2003;38(1 Pt 2):395–417.

Hwang J, Mcclintock S, Kachur SP, Slutsker L, Arguin P. Comparison ofnational malaria surveillance system with the national notifıable dis-eases surveillance system in the U.S. J Public Health Manag Pract2009;15(4):345–51.

Jacobson PD. Measuring the value of public health systems. University ofMichigan School of Public Health, Department of HealthManagementand Policy, School of Public Health, 2006.

Jacobson PD, Neumann PJ. A framework to measure the value of publichealth services. Health Serv Res 2009;44(5):1880–96.

Janosky JE, Laird SB, Sun Q. Content and context of a research registry forcommunity-based research. J Community Health 2008;33(4):270–8.

Jewell CJ, Bero LA. ”Developing good taste in evidence”: facilitators of andhindrances to evidence-informed health policymaking in state govern-ment. Milbank Q 2008;86(2):177–208.

JonesM, O’carroll P, Thompson J, D’ambrosio L. Assessing regional publichealth preparedness: a new tool for considering cross-border issues. JPublic Health Manag Pract 2008;14(5):E15–22.

KanarekN, Fitzek B, Su SC, BrowerM, JiaH. County lung cancermortality:a decision treemodel for control and prevention. J PublicHealthManagPract 2008;14(4):E1–9.

KanarekN, Stanley J, Bialek R. Local public health agency performance andcommunity health status. J Public Health Manag Practice 2006;12(6):522–7.

Kazda MJ, Beel ER, Villegas D, Martinez JG, Patel N, Migala W. Method-ological complexities and the use of GIS in conducting a communityneeds assessment of a large U.S. municipality. J Community Health2009;34(3):210–5.

Keane C, Marx J, Ricci E. Local health departments’ mission to the unin-

Kennedy VC. Public health workforce employment in U.S. public andprivate sectors. J Public Health Manag Pract 2009;15(3):E1–8.

Kerby DS, Brand MW, Johnson DL, Ghouri FS. Self-assessment in themeasurement of public health workforce preparedness for bioterrorismor other public health disasters. PublicHealth Rep 2005;120(2):186–91.

Kessel A, Green J, Pinder R, Wilkinson P, Grundy C, Lachowycz K. Multi-disciplinary research in public health: a case study of research on accessto green space. Public Health 2009;123(1):32–8.

Kinner K, Pellegrini C. Expenditures for public health: assessing historicaland prospective trends. Am J Public Health 2009;99(10):1780–91.

Kohr JM, Strack RW, Newton-WardM, Cooke CH. The use of programmeplanning and social marketingmodels by a state public health agency: acase study. Public Health 2008;122(3):300–6.

Kohrs FP, Mainous AG. Is health status related to residence in medicallyunderserved areas? Evidence and implications for policy. J RuralHealth1996;12(3):218–24.

Kramer MR, Hogue CR. Place matters: variation in the black/white verypreterm birth rate across U.S. metropolitan areas, 2002–2004. PublicHealth Rep 2008;123(5):576–85.

Kruger DJ, Shirey L,Morrel-Samuels S, Skorcz S, Brady J. Using a community-based health survey as a tool for informing local health policy. J PublicHealth Manag Pract 2009;15(1):47–53.

Lacar ES, Soto X, Riley WJ. Adolescent obesity in a low-income MexicanAmerican district in South Texas. Arch Pediatr Adolesc Med 2000;154(8):837–40.

Lauer J, Kastner J, Nutsch A. Primary care physicians and pandemic influ-enza: an appraisal of the 1918 experience and an assessment of contem-porary planning. J Public Health Manag Pract 2008;14(4):379–86.

Lee CJ. Does the internet displace health professionals? J Health Commun2008;13(5):450–64.

Leep C, Beitsch LM, Gorenflo G, Solomon J, Brooks RG. Quality improve-ment in local health departments: progress, pitfalls, and potential. JPublic Health Manag Pract 2009;15(6):494–502.

Lempa M, Goodman RM, Rice J, Becker AB. Development of scales mea-suring the capacity of community-based initiatives. Health Educ Behav2008;35(3):298–315.

LeonK,McdonaldMC,Moore B, RustG.Disparities in influenza treatmentamong disabled Medicaid patients in Georgia. Am J Public Health2009;99(S2):S378–S382.

Li F, Harmer P, Cardinal BJ, BosworthM, Johnson-Shelton D. Obesity andthe built environment: does the density of neighborhood fast-foodoutlets matter? Am J Health Promot 2009;23(3):203–9.

Linas BP, Zheng H, Losina E, Walensky RP, Freedberg KA. Assessing theimpact of federal HIV prevention spending on HIV testing and aware-ness. Am J Public Health 2006;96(6):1038–43.

Marshall H, Ryan P, Roberton D, Street J, Watson M. Pandemic influenzaand community preparedness. Am J Public Health 2009;99(S2):S365–S371.

Martin EG, Pollack HA, Paltiel AD. Fact, fıction, and fairness: resourceallocation under the Ryan White CARE Act. Health Aff (Millwood)2006;25(4):1103–12.

Mays GP, Halverson P, Miller CA. Assessing the performance of localpublic health systems: a survey of state health agency efforts. J PublicHealth Manag Pract 1998;4(4):63–78.

Mays GP, Halverson PK, Baker EL, Stevens R, Vann JJ. Availability andperceived effectiveness of public health activities in the nation’s mostpopulous communities. Am J Public Health 2004;94(6):1019–26.

Mays GP, Halverson PK, Stevens R. The contributions of managed careplans to public health practice: evidence from the nation’s largest localhealth departments. Public Health Rep 2001;116:50–67.

Mays GP, Hesketh HA, Ammerman AS, Stockmyer CK, Johnson TL,Bayne-Smith M. Integrating preventive health services within commu-nity health centers: lessons from WISEWOMAN. J Womens Health

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S55

M

Mays GP, Hurley RE, Grossman JM. An empty toolbox? Changes in healthplans’ approaches for managing costs and care. Health Serv Res 2003;38(1 Pt 2):375–93.

Mays GP, Mchugh MC, Shim K, et al. Getting what you pay for: publichealth spending and the performance of essential public health services.J Public Health Manag Pract 2004;10(5):435–43.

Mays GP, Mchugh MC, Shim K, et al. Identifying dimensions of perfor-mance in local public health systems: results from the National PublicHealth Performance Standards Program. J Public Health Manag Pract2004;10(3):193–203.

Mays GP, MchughMC, Shim K, et al. Institutional and economic determi-nants of public health system performance. Am J Public Health2006;96(3):523–31.

Mays GP, Scutchfıeld FD, Bhandari MW, Smith SA. Understanding theorganization of public health delivery systems: an empirical typology.Milbank Q 2010;88(1):81–111.

Mays GP, Smith SA. Geographic variation in public health spending: cor-relates and consequences. Health Serv Res 2009;44(5):1796–817.

Mccann PJC. Agency discretion and public health service delivery. HealthServ Res 2009;44(5):1897–908.

Mccarthy WJ, Mistry R, Lu Y, Patel M, Zheng H, Dietsch B. Density oftobacco retailers near schools: effects on tobacco use among students.Am J Public Health 2009;99(11):2006–13.

Mcclellan CS. Utilizing a national performance standards local publichealth assessment instrument in a community assessment process: theClarendon County Turning Point Initiative. J Public Health ManagPract 2005;11(5):428–32.

Mccue MJ, Mccall N, Hurley RE, Wyttenbach M, White M. Financialperformance and participation in Medicaid and Medi-Cal managedcare. Health Care Financ Rev 2001;23(2):69–81.

Mcdonnell KA, StrobinoDM, BaldwinKM,GrasonH,MisraDP. Compar-ison of FIMR programs with other perinatal systems initiatives. MaternChild Health J 2004;8(4):231–8.

Mcgrath MM, Fullilove RE, Kaufman MR, Wallace R, Fullilove MT. Thelimits of collaboration: a qualitative study of community ethical reviewof environmental health research. Am J Public Health 2009;99(8):1510–4.

Mckinney MM, Morrissey JP, Kaluzny AD. Interorganizational exchangesas performance markers in a community cancer network. Health ServRes 1993;28(4):459–78.

Meit M, Ettaro L, Hamlin BN, Piya B. Rural public health fınancing: impli-cations for community health promotion initiatives. J Public HealthManag Pract 2009;15(3):210–5.

Merrill J, Keeling J, Gebbie K. Toward standardized, comparable publichealth systems data: a taxonomic description of essential public healthwork. Health Serv Res 2009;44(5):1818–41.

Merrill J, Meier BM, Keeling J, Jia H, Gebbie KM. Examination of therelationship between public health statute modernization and localpublic health system performance. J Public Health Manag Pract2009;15(4):292–8.

Miller CA, Moore KS, Richards TB. The impact of critical events of the1980s on core functions for a selected group of local health depart-ments. Public Health Rep 1993;108(6):695–700.

Miller CA,Moore KS, Richards TB,McKaig C. A screening survey to assesslocal public health performance. Public Health Rep 1994;109(5):659–64.

Miller CA, Moore KS, Richards TB, Monk JD. A proposed method forassessing the performance of local public health functions and prac-tices. Am J Public Health 1994;84(11):1743–9.

Miriti MK, Billah K, Weinbaum C, et al. Economic benefıts of hepatitis Bvaccination at sexually transmitted disease clinics in the U.S. PublicHealth Rep 2008;123(4):504–13.

Misra DP, Grason H, Liao M, Strobino DM, Mcdonnell KA, Allston AA.The nationwide evaluation of fetal and infant mortality review (FIMR)

ay 2012

conduct of essential maternal and child health services by FIMRprograms. Matern Child Health J]?2004; 8/sb:volume-nr4):217–29.

Molinari C,Morlock L, Alexander J, Lyles CA.Hospital board effectiveness:relationships between governing board composition and hospital fı-nancial viability. Health Serv Res 1993;28(3):358–77.

Moore J. Studying an ill-defıned workforce: public health workforce re-search. J Public Health Manag Pract 2009;15(6S):S48–S53.

Morrissey J, Calloway M, Johnsen M, Ullman M. Service system perfor-mance and integration: a baseline profıle of theACCESS demonstrationsites: access to community care and effective services and supports.Psychiatr Serv 1997;48(3):374–80.

National Association of County and City Health Offıcials. 2005 nationalprofıle of local health departments. Washington DC: NACCHO, 2006.

Nelson JC, Bittner RC, Bounds L, et al. Compliance with multiple-dosevaccine schedules among older children, adolescents, and adults: resultsfrom a vaccine safety datalink study. Am J Public Health 2009;99 Suppl2:S389–97.

Nelson JC, Rashid H, Galvin VG, Essien JD, Levine LM. Public/privatepartners. Key factors in creating a strategic alliance for communityhealth. Am J Prev Med 1999;16(3S):94–102.

NeumannPJ, JacobsonPD, Palmer JA.Measuring the value of public healthsystems: the disconnect between health economists and public healthpractitioners. Am J Public Health 2008;98(12):2173–80.

Noonan VK, Kopec JA, Noreau L, et al. Comparing the content of partici-pation instruments using the international classifıcation of functioning,disability and health. Health Qual Life Outcomes 2009;7:93.

Odierna DH, Schmidt LA. The effects of failing to include hard-to-reachrespondents in longitudinal surveys. Am J Public Health 2009;99(8):1515–21.

Olden PC, Clement DG. The prevalence of hospital health promotion anddisease prevention services: good news, bad news, and policy implica-tions. Milbank Quarterly 2000;78(1):115–46, iii–iv.

Oliva G, Rienks J, Chavez GF. Evaluating a program to build data capacityfor core public health functions in local maternal child and adolescenthealth programs in California. Matern Child Health J 2007;11(1):1–10.

Orians C, Rose S, Hubbard B, et al. Strengthening the capacity of localhealth agencies through community-based assessment and planning.Public Health Rep 2009;124(6):875–82.

Padget SM, Bekemeier B, Berkowitz B. Collaborative partnerships at thestate level: promoting systems changes in public health infrastructure. JPublic Health Manag Pract 2004;10(3):251–7.

Padgett SM, Bekemeier B, Berkowitz B. Building sustainable public healthsystems change at the state level. J Public Health Manag Pract2005;11(2):109–15.

Pals SL, Murray DM, Alfano CM, Shadish WR, Hannan PJ, Baker WL.Individually randomized group treatment trials: a critical appraisal offrequently used design and analytic approaches. Am J Public Health2008;98(8):1418–24.

Patel AS, Powell TA, Woolard CD. Assessment of applied epidemiologycompetencies among the Virginia Department of Health workforce.Public Health Rep 2008;123(S1):119–27.

Pati S, Danagoulian S. Immigrant children’s reliance on public healthinsurance in the wake of immigration reform. Am J Public Health2008;98(11):2004–10.

Pearcy JN, Keppel KG. Monitoring change in health disparity. J PublicHealth Manag Pract 2008;14(5):481–6.

Pearson J, Windsor R, El-Mohandes A, Perry DC. Evaluation of the imme-diate impact of the Washington, D.C., smoke-free indoor air policy onbar employee environmental tobacco smoke exposure. Public HealthRep 2009;124(S1):134–42.

Peck BM, Asch DA, Goold SD. Measuring patient expectations: does theinstrument affect satisfaction or expectations? Med Care 2001;39(1):100–8.

Pillay R. The skills gap in hospital management in the South African public

S56 Harris et al / Am J Prev Med 2012;42(5S1):S42–S57

Pinto RM. Community perspectives on factors that influence collaborationin public health research. Health Educ Behav 2009;36(5):930–47.

PotterMA, Fitzpatrick T. State funding for local public health: observationsfrom six case studies. J Public Health Manag Pract 2007;13(2):163–8.

Prentice RL, Anderson GL. The women’s health initiative: lessons learned.Annu Rev Public Health 2008;29:131–50.

Purcell JM. Recruiting the future public health workforce: an analysis ofprospect communication among accredited Schools of Public Health.J Community Health 2009;34(3):216–21.

Ransom PE, Wei Y, Stellman SD. Community capacity for cancer preven-tion. J Health Hum Serv Adm 2009;32(1):5–29.

Reed J, Pavletic D, Devlin L, Davis MV, Beitsch LM, Baker EL. Piloting astate health department accreditation model: the North Carolina expe-rience. J Public Health Manag Pract 2009;15(2):85–95.

Resnick BA, Zablotsky J, Janus ER,Maggy B, Burke TA. An examination ofenvironmental public health organizational and workforce confıgura-tions in the Northeast/Mid-Atlantic U.S.: how do we determine if theseconfıgurations impact performance? J Public Health Manag Pract2009;15(6):509–17.

RileyW, Brewer R. Review and analysis of quality improvement techniquesin police departments: application for public health. J Public HealthManag Pract 2009;15(2):139–49.

Riley W, Parsons H, Mccoy K, Burns D, Anderson D, Lee S, Sainfort F.Introducing quality improvement methods into local public healthdepartments: structured evaluation of a statewide pilot project. HealthServ Res 2009;44(5):1863–79.

Rohan AM, Booske BC, Remington PL. Using the Wisconsin CountyHealth Rankings to catalyze community health improvement. J PublicHealth Manag Pract 2009;15(1):24–32.

Rosenfeld LA, Fox CE, Kerr D, et al. Use of computer modeling for emer-gency preparedness functions by local and state health offıcials: a needsassessment. J Public Health Manag Pract 2009;15(2):96–104.

Rosenheck R, Morrissey J, Lam J, et al. Service delivery and community:social capital, service systems integration, and outcomes among home-less persons with severe mental illness. Health Serv Res 2001;36(4):691–710.

Ruth BJ, Sisco S, Wyatt J, Bethke C, Bachman SS, Piper TM. Public healthand social work: training dual professionals for the contemporaryworkplace. Public Health Rep 2008;123(S2):71–7.

Savoia E,Massin-Short SB, RoddayAM,Aaron LA,HigdonMA, StotoMA.Public health systems research in emergency preparedness: a review ofthe literature. Am J Prev Med 2009;37(2):150–6.

Savoia E, TestaMA, Biddinger PD, et al. Assessing public health capabilitiesduring emergency preparedness tabletop exercises: reliability and va-lidity of a measurement tool. Public Health Rep 2009;124(1):138–48.

Schauffler HH, Chapman SA. Health promotion and managed care: Sur-veys of California’s health plans and population. Am J Prev Med1998;14(3):161–7.

SchlesingerM. Paradigms lost: The persisting search for community inU.S.health policy. J Health Politics Policy and Law 1997;22(4):937–92.

Schlesinger M, Gray BH, Gusmano M. A broader vision for managed care,part 3: The scope and determinants of community benefıts. Health Aff2004;23(3):210–21.

Schlitt JJ, Juszczak LJ, Eichner NH. Current status of state policies thatsupport school-based health centers. Public Health Rep 2008;123(6):731–8.

Scutchfıeld FD, Beaulieu J, Ireson C, Buege A. Public health competenciesrequired by managed care organizations. J Public Health Manag Pract2002;8(5):22–9.

Scutchfıeld FD, Beversdof CA, Hiltabiddle SE, Violante T. A survey of statehealth department compliance with the recommendations of the Insti-tute of Medicine report, The Future of Public Health. J Public HealthPolicy 1997;18(1):13–29.

Scutchfıeld FD, Hall L, Ireson CL. The public and public health organiza-tions: issues for community engagement in public health. Health Policy

Scutchfıeld FD, Hiltabiddle SE, Rawding N, Violante T. Compliance withthe recommendations of the Institute ofMedicine report, The Future ofPublic Health: a survey of local health departments. J Public HealthPolicy 1997;18(2):155–66.

Scutchfıeld FD, Knight EA, Kelly AV, Bhandari MW, Vasilescu IP. Localpublic health agency capacity and its relationship to public healthsystem performance. J Public Health Manag Pract 2004;10(3):204–15.

Scutchfıeld FD, Spain C, Pointer DD, Hafey JM. The public health leader-ship institute: leadership training for state and local health offıcers. JPublic Health Policy 1995;16(3):304–23.

Scutchfıeld FD, Zuniga De Nuncio ML, Bush RA, Fainstein SH, LaroccoMA, Anvar N. The presence of total quality management and continu-ous quality improvement processes in California public health clinics. JPublic Health Manag Pract 1997;3(3):57–60.

Seagle HM,Moore JB, Dubose KD. An assessment of the walkability of twoschool neighborhoods in Greenville, North Carolina. J Public HealthManag Pract 2008;14(3):e1–8.

Shebl F, Poppell CE, ZhanM, et al.Measuring health behaviors and landlinetelephones: potential coverage bias in a low-income, rural population.Public Health Rep 2009;124(4):495–502.

Shores KA, West ST. The relationship between built park environmentsand physical activity in four park locations. J PublicHealthManag Pract2008;14(3):e9–16.

Siegel C, Haugland G, Chambers ED. Performance measures and theirbenchmarks for assessing organizational cultural competency in behav-ioral health care service delivery. Admin Policy Ment Health 2003;31(2):141–70.

Skiles MP, Scott KD, Young C. When the cat is out of the bag: a case studyin public health rationing in Oregon during the 2004–2005 influenzavaccine shortage. J Public Health Manag Pract 2008;14(5):464–70.

Smith PJ, Singleton JA. Vaccination coverage estimates for selected coun-ties: achievement of Healthy People 2010 goals and association withindices of access to care, economic conditions, and demographic com-position. Public Health Rep 2008;123(2):155–72.

Spice C, Snyder K. Reviewing self-reported impacts of community healthassessment in local health jurisdictions. J Public Health Manag Pract2009;15(1):18–23.

Spinello EF, Fischbach R. Using a Web-based simulation as a problem-based learning experience: perceived and actual performance of under-graduate public health students. Public Health Rep 2008;123(S2):78–84.

Stoto MA. Regionalization in local public health systems: variation inrationale, implementation, and impact on public health preparedness.Public Health Rep 2008;123(4):441–9.

Stoto MA, Straus SG, Bohn C, Irani P. AWeb-based tool for assessing andimproving the usefulness of community health assessments. J PublicHealth Manag Pract 2009;15(1):10–7.

Strobino DM, Baldwin KM, Grason H, et al. The relation of FIMRprograms and other perinatal systems initiatives with maternal andchild health activities in the community. Matern Child Health J2004;8(4):239–49.

Studnicki J, Gipson LS, Berndt DJ, et al. Special healthcare taxing districts:association with population health status. Am J Prev Med 2007;32(2):116–23.

Subramanyan GS, Yokoe DS, Sharnprapai S, Nardell E, Mccray E, Platt R.Using automated pharmacy records to assess themanagement of tuber-culosis. Emerg Infect Dis 1999;5(6):788–91.

Thacker SB, Brownson RC. Practicing epidemiology: how competent arewe? Public Health Rep 2008;123(S1):4–5.

Thacker SB, Stroup DF, Carande-Kulis V, Marks JS, Roy K, GerberdingJL. Measuring the public’s health. Public Health Rep2006;121(1):14–22.

Thomas JC, Macdonald PD,Wenink E. Ethical decision making in a crisis:a case study of ethics in public health emergencies. J Public Health

www.ajpmonline.org

Harris et al / Am J Prev Med 2012;42(5S1):S42–S57 S57

Toomey TL, Chen V, Forster JL, Van Coevering P, Lenk KM. Do cigaretteprices vary by brand, neighborhood, and store characteristics? PublicHealth Rep 2009;124(4):535–40.

Tremain B, Davis M, Joly B, Edgar M, KushionML, Schmidt R. Evaluationas a critical factor of success in local public health accreditation pro-grams. J Public Health Manag Pract 2007;13(4):404–9.

TrepkaMJ,Martin P,MavundaK, RodriguezD, ZhangG, BrownC. A pilotasthma incidence surveillance system and case defınition: lessonslearned. Public Health Rep 2009;124(2):267–79.

Trust for America’s Health. Shortchanging America’s health 2006: a state-by-state look at how federal public health dollars are spent.WashingtonDC: Trust for America’s Health, 2006.

Tsakos G, Bernabe E, O’brien K, SheihamA, De Oliveira C. Comparison ofthe self-administered and interviewer-administeredmodes of the child-OIDP. Health Qual Life Outcomes 2008;6:40.

Turnock BJ, Handler A, Hall W, Potsic S, Nalluri R, Vaughn EH. Localhealth department effectiveness in addressing the core functions ofpublic health. Public Health Rep 1994;109(5):653–8.

Turnock BJ, Handler AS, Miller CA. Core function-related local publichealth practice effectiveness. J Public Health Manag Pract 1998;4(5):26–32.

Tutor RP, ZarateMA, Loury S. Pesticide exposure surveillance and preven-tion skills of staff in easternNorthCarolina health departments. J PublicHealth Manag Pract 2008;14(3):299–310.

Uscher-Pines L, Farrell CL, Cattani J, et al. A survey of usage protocols ofsyndromic surveillance systems by state public health departments inthe U.S. J Public Health Manag Pract 2009;15(5):432–8.

Van Meijgaard J, Fielding JE, Kominski GF. Assessing and forecastingpopulation health: integrating knowledge and beliefs in a comprehen-sive framework. Public Health Rep 2009;124(6):778–89.

Waters HR, Foldes SS, Alesci NL, Samet J. The economic impact of expo-sure to secondhand smoke in Minnesota. Am J Public Health2009;99(4):754–9.

Weiler P, Boggess J, Eastman E, Pomer B. The implementation of modelstandards in local health departments. American J Public Health1982;72(11):1230–7.

Weiner BJ, Shortell SM, Alexander J. Promoting clinical involvement inhospital quality improvement efforts: the effects of topmanagement,board, and physician leadership. Health Serv Res 1997;32(4):491–510.

Wells R, Lemak CH, D’aunno TA. Factors associated with interorganiza-tional relationships among outpatient drug treatment organizations1990–2000. Health Serv Res 2005;40(5):1356–78.

Wheeler JG, Pulley L, Felix HC, et al. Impact of a smoke-free hospitalcampus policy on employee and consumer behavior. Public Health Rep2007;122(6):744–52.

Whittingham J, Ruiter RA, Zimbile F, Kok G. Experimental pretesting ofpublic health campaigns: a case study. J Health Commun 2008;13(3):216–29.

Wholey DR, Gregg W, Moscovice I. Public health systems: a social net-works perspective. Health Services Res 2009;44(5):1842–62.

Wilczynski NL, Haynes RB, Lavis JN, Ramkissoonsingh R, Arnold-OatleyAE. Optimal search strategies for detecting health services researchstudies in MEDLINE. CMAJ 2004;171(10):1179–85.

Williams JC. State of emergency preparedness of Kentucky’s rural publichealth workforce: assessing its ability to identify community healthproblems. Public Health Rep 2008;123(2):178–88.

Wing C. Effects of written informed consent requirements on HIV testingrates: evidence from a natural experiment. Am J Public Health2009;99(6):1087–92.

Wing S,HortonRA,MuhammadN,GrantGR, TajikM, ThuK. Integrating

May 2012

community health effects of industrial hog operations. Am J PublicHealth 2008;98(8):1390–7.

Wingo PA, Kulkarni A, Borrud LG, Mcdonald JA, Villalobos SA, GreenDC. Health disparities among Mexican American women aged 15-44years: National Health and Nutrition Examination Survey, 1999–2004.Am J Public Health 2009;99(7):1300–7.

Wray RJ, Becker SM, Henderson N, et al. Communicating with the publicabout emerging health threats: lessons from the Pre-Event MessageDevelopment Project. Am J Public Health 2008;98(12):2214–22.

Yawn BP, Fryer GE, Phillips RL, Dovey SM, Lanier D, Green LA. Using theecology model to describe the impact of asthma on patterns of healthcare. BMC Pulm Med 2005;5:7.

Yeh SCJ, Wan TTH, Huang CH, Lo YY. Ambulatory care visits and physi-cian satisfaction: From medical directors’ perspectives. Health CareManag Rev 2007;32(3):236–44.

Yoo BK, Kasajima M, Fiscella K, Bennett NM, Phelps CE, Szilagyi PG.Effects of an ongoing epidemic on the annual influenza vaccination rateand vaccination timing among the Medicare elderly: 2000–2005. Am JPublic Health 2009;99(S2):S383–S388.

Young G, Beekun RI, Ginn GO. Governing board structure, business strat-egy, and performance of acute care hospitals: a contingency perspective.Health Serv Res 1992;27(4):543–64.

Yu SM,Huang ZJ, KoganMD. State-level health care access and use amongchildren in US immigrant families. Am J Public Health 2008;98(11):1996–2003.

Zahner SJ. Memoranda of understanding between Medicaid MCOs andpublic health departments. Manag Care 2001;10(9):47–52.

Zahner SJ. Evaluation of mandated memoranda of understanding betweenlocal health departments and Medicaid managed care plans. J PublicHealth Manag Pract 2002;8(5):11–21.

Zahner SJ. Local public health system partnerships. Public Health Rep2005;120(1):76–83.

Zahner SJ. Partnerships for learning population-based public health nurs-ing: web-delivered continuing education for public health nurse pre-ceptors. Public Health Nurs 2006;23(6):547–54.

Zahner SJ, Corrado SM. Local health department partnerships withfaith-based organizations. J Public Health Manag Pract 2004;10(3):258–65.

Zahner SJ, Gredig QN. Improving public health nursing education: recom-mendations of local public health nurses. Public Health Nurs2005;22(5):445–50.

Zahner SJ, Gredig QN. Public health nursing practice change and recom-mendations for improvement. Public Health Nurs 2005;22(5):422–8.

Zahner SJ, Kaiser B, Kapelke-Dale J. Local partnerships for communityassessment and planning. J Public Health Manag Pract 2005;11(5):460–4.

Zahner SJ, Vandermause R. Local health department performance: compliancewith state statutes and rules. J PublicHealthManagPract 2003;9(1):25–34.

Zaric GS, BrandeauML. Optimal investment in a portfolio of HIV preven-tion programs. Med Decis Making 2001;21(5):391–408.

Zuniga MA, Carrillo-Zuniga G, Seol YH, Fos PJ. Multi-criteria assessmentof county public health capability disparities. J Health Hum Serv Adm2009;32(3):238–58.

Appendix

Supplementary data

Supplementary data associatedwith this article can be found, in the