Implementation of the National Incident Management System in New Jersey – A Mixed Methods Study Len Elisha Clark, DPA CEM June 10, 2010.

Implementation of the Implementation of the National Incident National Incident

Management System Management System in New Jersey –in New Jersey –

A Mixed Methods Study A Mixed Methods Study Len Elisha Clark, DPA CEMLen Elisha Clark, DPA CEM

June 10, 2010 June 10, 2010

Purpose of the Study Purpose of the Study and Research Questionand Research Question

PurposePurpose Rigorous research of local emergency Rigorous research of local emergency

management topics is missingmanagement topics is missing Coordination of response activities is Coordination of response activities is

noted as a flaw in the popular medianoted as a flaw in the popular media Seek to examine relevant public Seek to examine relevant public

administration theories in a “real-world” administration theories in a “real-world” environment environment

Purpose of the Study Purpose of the Study and Research Questionand Research Question

Research QuestionResearch Question What theories of implementation inform What theories of implementation inform

the assessment of the implementation the assessment of the implementation of the National Incident Management of the National Incident Management System by municipalities in New Jersey? System by municipalities in New Jersey?

Academic and TheoreticalAcademic and TheoreticalLiterature Found to be Literature Found to be

Particularly HelpfulParticularly Helpful ““The Garbage Can Model” – Cohen, The Garbage Can Model” – Cohen,

March and OlsenMarch and Olsen ““The Price of Federalism” – PetersonThe Price of Federalism” – Peterson ““Implementation and Public Policy” - Implementation and Public Policy” -

Mazmanian and SabatierMazmanian and Sabatier ““Implementation Theory and Practice” Implementation Theory and Practice”

- Goggin, O’Toole, Lester & Bowman- Goggin, O’Toole, Lester & Bowman

MethodologyMethodology

Case study using mixed methodsCase study using mixed methods Sampling FrameSampling Frame

Population of NJ municipalities (n=566)Population of NJ municipalities (n=566) Unit of AnalysisUnit of Analysis

Class – NJ municipalitiesClass – NJ municipalities

MethodologyMethodology

Purposive SamplingPurposive Sampling NIMSCAST and after-action reportsNIMSCAST and after-action reports Not all municipalities submittedNot all municipalities submitted

MethodologyMethodology

Data AnalysisData Analysis Quantitative – Regression and General Linear Quantitative – Regression and General Linear

ModelModel NIMSCAST – indexed dataNIMSCAST – indexed data

Compliant, Mostly compliant, In progress, Non-compliantCompliant, Mostly compliant, In progress, Non-compliant Funding dollars – per capitaFunding dollars – per capita Qualitative Qualitative

Content AnalysisContent Analysis After-action reportsAfter-action reports NIMSCAST narrativeNIMSCAST narrative Grant funding reports Grant funding reports

ResultsResults

Hypothesis 1 – Hypothesis 1 – Those municipalities which indentified Those municipalities which indentified

fewer local units responsible for the fewer local units responsible for the implementation of NIMS will have implementation of NIMS will have greater success in its implementation.greater success in its implementation.

ResultsResults

Regression Regression N=374N=374 Mean Index value=106, s.d.=24 (d.v.)Mean Index value=106, s.d.=24 (d.v.) Mean # of units = 5, s.d. = 3Mean # of units = 5, s.d. = 3 (i.v.) (i.v.) R Square = 0.001R Square = 0.001 Significance = 0.604 > 0.05Significance = 0.604 > 0.05 Cannot reject null hypothesis Cannot reject null hypothesis

Content analysisContent analysis N=22N=22 No agreement on agencies involvedNo agreement on agencies involved OHSP reports “impossible to ascertain each response OHSP reports “impossible to ascertain each response

agencies [sic] level of readiness…” (2008: 49) agencies [sic] level of readiness…” (2008: 49)

ResultsResults

Hypothesis 2 – Hypothesis 2 – Municipalities within the New York Municipalities within the New York

Metropolitan Statistical Area (MSA) are Metropolitan Statistical Area (MSA) are more likely to implement NIMS than more likely to implement NIMS than within the Philadelphia MSA or those within the Philadelphia MSA or those outside of either MSA.outside of either MSA.

ResultsResults

Analysis Analysis N=376N=376 75% rated as “compliant” or “mostly 75% rated as “compliant” or “mostly

compliant”compliant” Outside either MSA – highest percentage Outside either MSA – highest percentage

“non-compliant” and “complaint”“non-compliant” and “complaint” New York MSA – highest percentage “in New York MSA – highest percentage “in

progress” and “mostly compliant”progress” and “mostly compliant” Significance 0.190 > 0.05 – Chi-Square testSignificance 0.190 > 0.05 – Chi-Square test Cannot reject null hypothesisCannot reject null hypothesis

ResultsResults

Hypothesis 3 – Hypothesis 3 – Municipalities that received federal Municipalities that received federal

Emergency Management Performance Emergency Management Performance Grants (EMPG) have implemented more Grants (EMPG) have implemented more components of NIMS than those which components of NIMS than those which received no amount of funds. received no amount of funds.

EMPG funds evaluated on a “per-capita” EMPG funds evaluated on a “per-capita” basis with a presumptive baseline value basis with a presumptive baseline value of $1.00 per capita for all municipalitiesof $1.00 per capita for all municipalities

ResultsResults

Regression Regression N=376N=376 Mean Index value=106, s.d.=24 (d.v.)Mean Index value=106, s.d.=24 (d.v.) Mean # of EMPG values = $1.06, s.d. = $0.19Mean # of EMPG values = $1.06, s.d. = $0.19

(i.v.)(i.v.) R Square = 0.009R Square = 0.009 Significance = 0.073 > 0.05Significance = 0.073 > 0.05 Cannot reject null hypothesis Cannot reject null hypothesis

Content analysisContent analysis 8.4% of statewide award passed through to 8.4% of statewide award passed through to

municipalitiesmunicipalities

ResultsResults

Hypothesis 4 –Hypothesis 4 – Municipalities in counties which received Municipalities in counties which received

more Presidential Disaster Declarations more Presidential Disaster Declarations from the years 1979 through 2008 tend from the years 1979 through 2008 tend to implement more components of NIMS to implement more components of NIMS than those which received fewer than those which received fewer declarations.declarations.

ResultsResults

Regression Regression N=376N=376 Mean Index value=106, s.d.=24 (d.v.)Mean Index value=106, s.d.=24 (d.v.) Mean # of units = 5, s.d. = 1Mean # of units = 5, s.d. = 1 (i.v.)(i.v.) R Square = 0.014R Square = 0.014 Significance = 0.02 < 0.05Significance = 0.02 < 0.05 Can reject null hypothesis Can reject null hypothesis Y=116 (constant) -2.0x (with a negative Y=116 (constant) -2.0x (with a negative

decrease)decrease)

ResultsResults

Hypothesis 5 – Hypothesis 5 – Using a General Linear Model to Using a General Linear Model to

determine, how, if any way, these determine, how, if any way, these factors (# of agencies involved [fixed factors (# of agencies involved [fixed variable], location within an MSA variable], location within an MSA [covariate], disaster declaration [covariate], disaster declaration experience [covariate]) combine t shape experience [covariate]) combine t shape the implementation of NIMSthe implementation of NIMS

ResultsResults

ModelModel n=374n=374 Philadelphia MSA – highest mean Index Philadelphia MSA – highest mean Index

value=108, s.d.=24 value=108, s.d.=24 Non-MSA – lowest = 103, s.d = 32Non-MSA – lowest = 103, s.d = 32 R Square = 0.031R Square = 0.031 Significance for declarations = 0.013 < 0.05Significance for declarations = 0.013 < 0.05 Cannot reject this covariate Cannot reject this covariate The balance of the factors remained The balance of the factors remained

statistically insignificantstatistically insignificant NY + 3 points, the balance -4 in comparison to NY + 3 points, the balance -4 in comparison to

the each otherthe each other

ResultsResults

Hypothesis 6 –Hypothesis 6 – Municipalities which submitted Municipalities which submitted

“compliant” NIMSCAST reports will “compliant” NIMSCAST reports will identify fewer negative after-action identify fewer negative after-action report outcomes in the ETEAM NIMS-report outcomes in the ETEAM NIMS-related categories than those related categories than those municipalities which received other municipalities which received other compliance ratings. compliance ratings.

ResultsResults

Regression Regression n=74n=74 Mean Index value=110, s.d.=20 (i.v.)Mean Index value=110, s.d.=20 (i.v.) Mean # of shortfalls = 1, s.d. = 1 (d.v.)Mean # of shortfalls = 1, s.d. = 1 (d.v.) R Square = 0.012R Square = 0.012 Significance = 0.346 > 0.05Significance = 0.346 > 0.05 Cannot reject null hypothesis Cannot reject null hypothesis

Content AnalysisContent Analysis Shortfalls (23) – Equipment (3), Facilities (5), Shortfalls (23) – Equipment (3), Facilities (5),

Training (5), Personnel (4), Planning (6)Training (5), Personnel (4), Planning (6)

ResultsResults

Paradigm of Maximum VariationParadigm of Maximum Variation MSA location and Index score key factorsMSA location and Index score key factors NY MSA – Rockaway Township (4) and NY MSA – Rockaway Township (4) and

Harrison (132)Harrison (132) Philadelphia – Springfield (14) and Philadelphia – Springfield (14) and

Pemberton Township (133)Pemberton Township (133) Non-MSA – Franklin (1) and Commercial Non-MSA – Franklin (1) and Commercial

Township (132) Township (132)

ResultsResults

The number/type of involved agencies was The number/type of involved agencies was neither standardized nor significant neither standardized nor significant

Location within the State was not a factorLocation within the State was not a factor EMPG funding did not appear to be a factor EMPG funding did not appear to be a factor Disaster declaration experience appears to Disaster declaration experience appears to

hold as an influencing factor, but counter-hold as an influencing factor, but counter-intuitiveintuitive

In combination, the previous factors remain In combination, the previous factors remain the same the same

Full-scale exercises or actual occurrences were Full-scale exercises or actual occurrences were not conducted and shortfalls not capturednot conducted and shortfalls not captured

DiscussionDiscussion

NJ municipalities may not be NJ municipalities may not be “compliant” with the Federal intent, “compliant” with the Federal intent, but progress is being made.but progress is being made.

Theoretical SignificanceTheoretical Significance Majority of the theoretical assertions were Majority of the theoretical assertions were

not supported by the results.not supported by the results. Can see a positive tie into the Garbage Can see a positive tie into the Garbage

Can Model’s idea of “Organized Can Model’s idea of “Organized Anarchies.” Anarchies.”

DiscussionDiscussion

Methodological SignificanceMethodological Significance Mixed methods capitalized upon Mixed methods capitalized upon

strengths of both quantitative and strengths of both quantitative and qualitative methods qualitative methods

NIMSCAST and ETEAM data proved NIMSCAST and ETEAM data proved easily adaptable to this use, although easily adaptable to this use, although not designed as a statistical dataset.not designed as a statistical dataset.

DiscussionDiscussion

Methodological SignificanceMethodological Significance RecommendationsRecommendations

Conduct a longitudinal study (2005 to present)Conduct a longitudinal study (2005 to present) Include regional or national studiesInclude regional or national studies Examine from a “systems” aspectExamine from a “systems” aspect Investigate the use of Patton’s “Complexity Investigate the use of Patton’s “Complexity

Theory” for non-linear examinations Theory” for non-linear examinations Studies using “mixed methods” is a viable Studies using “mixed methods” is a viable

method method

DiscussionDiscussion

For the Field Of ActionFor the Field Of Action Government should continue to leverage Government should continue to leverage

“Strong Statute” and “Wilsonian” Perspectives “Strong Statute” and “Wilsonian” Perspectives Federal government should refine definition of Federal government should refine definition of

compliance and its metricscompliance and its metrics Consider a multiple tiered approach with Consider a multiple tiered approach with

additional fundingadditional funding Move from “compliance” and develop a Move from “compliance” and develop a

“systems” approach to build capacity – “systems” approach to build capacity – planning, training, exercises, etc.planning, training, exercises, etc.

DiscussionDiscussion

Action-ResearchAction-Research Examine the results in greater detailExamine the results in greater detail

Why did 25% not score well in NIMSCAST?Why did 25% not score well in NIMSCAST? Incorporate units into this processIncorporate units into this process

Identify factors which impede Identify factors which impede implementationimplementation

Some known, others unknownSome known, others unknown Why the absence of municipal submissions Why the absence of municipal submissions Identify clear, concise measures in the Identify clear, concise measures in the

absence of Federal direction absence of Federal direction

DiscussionDiscussion

Develop actionable items – policy and/or Develop actionable items – policy and/or process redesign and an implementation process redesign and an implementation planplan

Collect and evaluate data Collect and evaluate data ReportsReports ExercisesExercises Actual emergenciesActual emergencies

Needed due to constant reworking of Needed due to constant reworking of requirements by Federal government requirements by Federal government

Parting ThoughtsParting Thoughts

NIMS implementation is based on sound NIMS implementation is based on sound public administration theory.public administration theory.

Local agencies have a clear Local agencies have a clear understanding of their importance in understanding of their importance in public safety public safety

Statistical tests challenge existing theoryStatistical tests challenge existing theory New Jersey is making positive strides New Jersey is making positive strides

along the compliance continuum along the compliance continuum