Evidence-Based Management

Evidence-Based ManagementA New Approach to Teaching the Practice of Management

ACPE Annual Meeting, April 26 30, 2013, New York

Tony Kovner, Michiel Bosman, Eric Barends

EBMgt: My Experiences

EBMgt as a Process

Why Don’t Managers use it?

Learning from Medicine

Contributions of Organizational Behavior

Experience in Teaching

The Politics of EBMgt

EBMgmt: Definition

“…the systematic, evidence-informed

practice of management, incorporating

scientific knowledge in the content and

process of making decisions.”

(Rousseau 2012)

EBMgt as a process

Framing the question behind the decision

Finding the sources of information

Assessing the accuracy of information

Assessing the applicability of information

Assessing the actionability of information

Determining if the information is adequate

(Hsu and others 2009)

Why don’t Managers use EBMgt?

Never heard of it. They are using it.

Cost now more probable than benefits in the future

Managers have to be persuaded and trained

The process doesn’t clearly lead to one-best way to intervene

Politics impacts interests

Learning from EB-Medicine

Interventions that achieve positive

predictable results

Hindrances that blocked implementation of

EBM same now for EBMgt

Financial as well as political impacts

Contributions of Organizational Behavior

Hiring Talent-relying on structured interviews

(Rousseau)

Challenging Performance goals impacting on

performance (Latham and Locke)

Managers who set a vision outperform other

managers (Kirkpattrick)

Teaching EBMgt

NYU / Wagner, Capstone course

Projects

Hourly nurse rounding process

Redesigning the nurses’ clinical ladder

Improving the supply distribution process

Identifying causes of emergency department waiting

The Politics of EBMgmt

Evidence is not sufficient to change people’s behavior

The quality of the argument and story-telling by persons presenting the evidence is what persuades stakeholders

How persuade the manager that this is in his own interest?

Michiel Bosman MD MMM FACPEMD, University of Amsterdam

MMM, CMU

Exec PhD (2015), OSU Spears School

Collaborator, Center for EBMgmt

Serial Entrepreneur

“there is a large research-user gap”

“practitioners do not read academic journals”

“the findings of research into what is an effective intervention

are not being translated into actual practice”

“the relevance, quality and applicability of research is

questionable”

“practice is being driven more by fads and fashions than

research”

“many practices are doing more harm than good”

What field is this?

McMaster University Medical School, Canada

Medicine: Founding fathers

David Sackett Gordon Guyatt

How it all started

More than 1 million articles in 40,000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a primary care physician more than 100!)

Problem: too much ‘evidence’

Problem: too much evidence

HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR manager to keep up this means reading 3 to 4 articles every day (for a ‘general’ manager more than 50!)

Problem: too many half truths

BTW: most of the research is seriously flawed or irrelevant for practice. And some claim so is most management advice.

The 5 steps of Evidence-Based Practice

1. Formulate a focused question (Ask)

2. Search for the best available evidence (Acquire)

3. Critically appraise the evidence (Appraise)

4. Integrate the evidence with your professional expertise and apply (Apply)

5. Monitor the outcome (Assess)

1. Incompetent people benefit more from feedback than highly competent people.

2. Task conflict improves work group performance while relational conflict harms it.

3. Encouraging employees to participate in decision making is more effective for improving organizational performance than setting performance goals.

True or false?

How evidence-based are managers?

959 (US) + 626 (Dutch) HR professionals 35 statements, based on an extensive body of

evidence true / false / uncertain

On average: 35% - 57% correct

HR Professionals' beliefs about effective human resource practices: correspondence between research and practice, (Rynes et al, 2002, Sanders et al 2008)

Evidence-Based Practice

1991 Medicine

1998 Education

1999 Social care, public policy

2000 Nursing

2005 Criminal justice

2010 Marketing

2013 Healthcare Management?

Evidence-Based Consulting

Close the research/practice gap

Professor + Consultant

EB Mgmt Workshops

Teach the 5 step EB Practice process

Answer business questions

CAT: Critically Appraised Topic

Closing the research/practice gap

Executive Doctorate in Management

University of Maryland College

Oklahoma State University

Development of tools to measure EB Mgmt Attitudes (EBMAS) and skills/competencies (Fresno)

Teaching Evidence Based Practice to managers

Evidence?

intuition, expertise, personal experience, collective experience,

organizational facts & data, best practices, benchmarking, outcome of scientific research

All managers base their decisions on ‘evidence’

BUT ...

Managers give little or no consideration to the

quality of the evidence they base their decisions on

SO ...

Teach managers how to find and and critically appraise

the evidence before they make their decision

The 5 steps of EBP

1. Formulate a focused question (Ask)

2. Search for the best available evidence (Acquire)

3. Critically appraise the evidence (Appraise)

4. Integrate the evidence with your professional expertise and apply (Apply)

5. Monitor the outcome (Assess)

0. Creating awareness Why do we need it?

Trust me, 20 years of management experience

Errors and Biases of Human Judgment

Seeing order in randomness Mental corner cutting Misinterpretation of incomplete data Halo effect False consensus effect Attribution error Group think Self serving bias Sunk cost fallacy Cognitive dissonance reduction

Confirmation bias Outcome bias Authority bias Small numbers fallacy Recall bias Anchoring bias Inaccurate covariation detection

Errors and Biases of Human Judgment

1. Formulate a focused question

Asking the right question?

Does team-building work?

Does the introduction of self-steering teams work?

Does lean management improve the performance of

our hospital?

Is 360 degree feedback for doctors effective?

What is a ‘team’?

What kind of team?

In what contexts/ settings?

What counts as ‘team-building’?

What does ‘work’ mean?

Focused question?

Does team-building work?

P = Population

I = Intervention or success factor

C = Comparison

O = Outcome

C = Context

Answerable question: PICOC

P = Population

I = Intervention or successfactor

C = Comparison

O = Outcome

C = Context

Focused question: PICOC

Employee productivity?

Patient satisfaction?

Return on investment?

Market share?

Organizational commitment?

2. Finding the best available evidence

Best available experiential evidence

Best available internal evidence

Organizational values and stakeholders’

concerns

Best available external evidence

Evidence-based decision

Evidence-based decision

scientific

research

Scientific databases: management

ABI/INFORM

Business Source Elite

PubMed

PsycINFO

Web of Knowledge

ERIC

3. Critical appraisal of studies

Making sense of evidence

Critical appraisal

Construct validity (lean six sigma = value stream mapping, root cause analysis, goal setting, participative decision making)

Internal validity (does it work?)

External validity (will it also work for my employees / organization?)

Levels of internal validity

Best research design?

Best available

Critical appraisal

Step 4: Turning evidence into practice

Applicable?

organizational characteristics

cultural & political aspects

financial aspects /cost-effectiveness / ROI

priorities

change readiness / resistance to change

implementation capacity

timing

TEST?

CAT: Critically Appraised Topic

CAT-walk

Evidence-based practice:

If doctors can do it managers can do it!