Gap analysis investments in research prof. norman braveman

Braveman BioMed Consultants - "Working with you for excellence in research." 1

Gap Analysis: Guiding Government

Investments In Research Using Data Driven

Scientific Program and Policy Development

Norman S. Braveman, Ph.D.

President

Braveman BioMed Consultants

Elsevier Reed Conference

Accelerate Research Excellence: Evaluate, Benchmark and

Strategize

September 23, 2001

New Delhi, India

Braveman, N.S. What science indicators can tell policymakers. In Margaret O. Meredith, Stephen D. Nelson and Albert H. Teich (Eds.) Science and Technology Yearbook: 1991. Washington, D.C.: American Association for the Advancement of Science, 267-278.

Braveman, N.S. Evaluation at NIH: The policy issues. In Susan L. Sauer (Ed.) R & D in FY 1989: Looking Ahead in an Election Year. Washington, D.C.: American Association for the Advancement of Science, 1988.“Evaluation at NIH: The policy issues”

Policies based on data are better than those that are not

The primary driver or source of data for development of science policy and programs should be science itself

But we need to be cognizant of other drivers and sources of information

AS A GRANT MAKER YOU HAVE AN EXTREMELY IMPORTANT IMPACT ON SCIENCE

© Braveman BioMed Consultants - "Working with you for excellence in research." 2

So what prevents decision making based on science from happening?

Limited and/or uninformative data –> need for additional and new metrics (demographic/bibliometric vs. content)

Lack of access to or difficulty in accessing data –> getting data is not easy and time consuming for first level users

Austere times –> tendency to not develop

Prevailing attitudes about formal planning in science



Prevailing attitudes toward PP&D in science

It‟s not possible or advisable to plan science -> can lock you into unproductive pathways

It‟s important to be flexible and able to change directions quickly to follow leads/new directions

„Peer review‟/publication of findings are the only endpoints that matter in determining future directions

Serendipity or chance historically lead to great findings –> the double helix; clinical applications of iPS


"In the fields of observation, chance favors only the prepared mind." Louis Pasteur

A „prepared mind‟ (in science) is one that has considered alternative paths, even in an informal way, particularly when results don‟t turn out as expected.

“A bad is experiment is one that will provide answers only if the

hypothesis is supported or disproven. A good experiment is one that will provide answers no matter what the outcome of the

experiment.” Anonymous Peer Reviewer

The context

Some sources of support for research in the US

Current interests of grant makers

An approach to Data Driven Scientific Program and Policy Development using PP&D

Logic Analysis

Strategic and Implementation Planning

Gap Analysis

Scientific fingerprinting – a way to collect data for PP&D of science policies and programs

What is scientific fingerprinting

Examples of current use in PP&D


Funding Agency

2011 Agency Budget Request (% for R&D)

National Institutes of Health $32.1 billion (85)

National Science Foundation $7.4 billion (80)

Department of Energy $28.4 billion (18)

NASA $19.0 billion (33)

Department of Defense $708.0 billion (10)

US Department of Agriculture $146.0 billion (2)

Various State Funding Programs (stem cells, tobacco –related diseases)

--



There are approximately 4000 private grant makers in the US

Health Research Alliance: not-for-profit, non-governmental funders of biomedical

research and training from biomedical science to applications that advance health

47 member organizations

HRA member organizations provide over $1.5 billion in research awards each year, funding over 5500 researchers annually


Many of these sources of funding relies on PP&D in order to

stay current in their area of science

be ahead of the curve in planning for future scientific developments

Insure that their resources are used appropriately and in the most effective way possible…particularly in austere times


Scientifically meritorious research

Innovative research that will move the field ahead

Development of the research workforce of the future Basic scientists

Clinician and clinical scientists

Interdisciplinary scientists

Diversity in racial and ethnic backgrounds

Topic of the day Personalized medicine

Translational research

Systems approach to biomedicine

Stem cell research

Disease of the day Obesity

Cancer

Heart disease


The foundation of PP&D is evaluation…

a systematic method for collecting, analyzing, and using data and information to answer questions about projects, policies and scientific programs

„Data „ is not the plural of „anecdote‟. Unknown


Demographic/Quantitative Who is receiving funding

When did they receive it

Where are they located

How much did they receive; how much in total is being spent on the area of science

How many publications/inventions have resulted from the research (return on investment)

How many publications have resulted from grants


Demographic/Categorical

Content Based/Qualitative Captures the content of the research in the portfolio ->

What is the content of the research that is being addressed? What isn‟t being addressed…and why?

Systematic, comprehensive, reliable and valid

Can link a current portfolio to portfolios that exist in other organizations (e.g., funding agencies)

Can add important scientific information in addressing some of the quantitative questions


What are the goals of PP&D for organizations that fund research?

To maximize the impact of funding by advancing science

To determine whether what has been done has met the specific goals and objectives of the organization

To chart the future priorities and directions of the organization

Logic Model Clear statement of mission and goals of the organization

Identification of inputs and outputs

Identification of expectations and measures of success

Strategic Planning Statement of values, mission, priorities and goals

Implementation Planning Timelines, benchmarks, mechanisms, sequence of events

Gap Analysis


A Major Component of PP&D

Answers the questions – Where are we?

Where do we want to be?

Are we making the best use of our resources?

“It‟s the difference between what an organization is doing or has been doing and what the organization should be doing or wants to do…” Source Unknown


Applied from a business/economics environment to organizations

involved in funding research

Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation

What are the inputs in the world of science? -> money, people, facilities/equipment, knowledge

What do we mean by “optimization” or “integration” when we‟re talking about funding of science? -> a unique combination of inputs that yields best performance on outcome measures


Applied from a business/economics environment to organizations

involved in funding research

Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation

Used to compare a program‟s actual performance with its potential performance

How do we determine a funding organizations potential performance?

from strategic plan, implementation plan and logic model

How do we measure a funding organization‟s actual performance?

use outcome measures developed from logic model


Also addresses specific questions about the current state-of-the-science as well as about pathways to scientific advancement

What do we need to do to move ahead?

What are the barriers and where are the opportunities?

Gaps are then defined in terms of the current portfolio of the organization and the near- and long-term state-of-the-science



Background & Goals

Logic Model

Outcome Measures

Data Collection

Expert Panel

•Future of area •Barriers •Possible priorities

Organization Data-Driven Decision Process

Gap Analysis

Current portfolio vs. “ideal” portfolio

Stakeholders

Strategic Plan

Background (stakeholder input; source documents) What is the organization‟s role in the big picture?

What are our present goals?

Where are we going?

How have we gotten to where we are?

What has changed that we need an updated assessment?

Who are the other participants and what is their role?

What is/has been our relationship with them?


Background

Portfolio Analysis (data collection) What has the organization accomplished toward current

goals?

What science have we been funding? What science have we not been funding?

What is/are the scientific content of the same or a similar portfolio(s) of other organizations?

What science areas are covered by applications that have been submitted to our organization?

Of those submitted to our organization, which ones are being funded? Which ones are not being funded…and why?


Background

Data collection

Gap Analysis Compare content of current portfolio with „ideal‟ portfolio

Assess content of current support by all organizations for duplication, overlap and collaboration

Compare the content of current portfolio(s ) with the future of science

Compare content of all submitted applications with program goals and future of science

Compare content of successful applications with program goals and future of science

Compare content of unsuccessful applications with the future of science and program goals


Background

Data collection

Gap Analysis

Expert Panel (thought leaders) Where is the field going in the short-/long-term?

What are the scientific barriers to getting there?

Assessment of current content of programs by all organizations for duplication, overlap and collaboration

Where are the gaps between the content of current portfolios and the future of science…with „ideal‟ portfolio

Recommendations/priorities about where the organization being analyzed can make significant and unique contributions/impacts


These are the drivers of the development of science policies and

funding programs

But they aren‟t the only ones…the final step -> organizational decision making


Background

Data collection

Gap Analysis

Expert Panel

Organizational Decision Making (stakeholders and leadership) What unique scientific contributions do we want to be

known for?

Resources available – people, fiscal

Possibilities for collaboration/leveraging with other organizations

Political considerations

Determine which gaps are going to be addressed

Develop timelines and benchmarks



A representation of a document generated with the Elsevier Fingerprint Engine

A precise abstract representation of a text which allows us to “look into the text” (content) rather than only looking at the meta data (demographics).




NIH Research, Condition, and Disease Categorization Process (RCDC) Introduced in 2009

A computerized process used at the end of each fiscal year to sort and report the amount funded in each of 229 historically reported categories of disease, condition, or research area.

The Florida Biomedical Research Program – scientific program management and development

National Heart, Lung and Blood Institute (NIH) – identifying ideas and tracking progress


Florida Biomedical Research Program:

Integration of PP&D Within An Ongoing Scientific

Program


Strategic Plan P

rogr

am P

lan

nin

g P

rogr

am D

eve

lop

me

nt

Portfolio Analysis

Implementation Plan

Expert Science Panel

Recommendations

Council Review

Logic Model & Gap Analysis • Program Goals & Objectives • Outcome Measures • Gap Analysis

Call Development • Priority science areas • Funding mechanisms • Application configuration

Issue RFA(s)

Annual Report •Grant Outcomes •Success Stories •Program Outcomes

Notice of Award

Summary Statements To Applicants

Grantsmanship Seminars (Answer questions about RFA)

Ap

plic

atio

ns

Applications

Peer Review

Council Review & Funding Recommendation

Gra

nt

Mo

nit

ori

ng

Grants Management &

Reviewer Finder Systems

Grants Management &

Program Review

Progress Reports

Financial Reports

Gap Analysis

Logic Model • Program Goals • Outcome

Measures


Biomed Experts & Reviewer Finder

National Heart, Lung and Blood Institute (NIH)

Identifying ideas and tracking their implementation through

program development



SCIENTIFIC WORKSHOPS

NIH OFFICES

DHHS LEADERSHIP

NATIONAL

ADVISORY

COUNCIL

OR BOARD

EXTRAMURAL

SCIENTISTS

INDUSTRY/ PRIVATE SECTOR

CONGRESS

PROFESSIONAL GROUPS

PATIENT

GROUPS

OTHER FEDERAL AGENCIES

IC Staff

Decision

Process

NIH Early

Notification

System

ISSUE RFA OR PA

IC ADJUSTMENTS

Opportunity

Ad Hoc Advisory

Groups

I

C

L

E

G

I

S

L

A

T

E

D

M

I

S

S

I

O

N




The effective use of data driven PP&D in developing science policy and funding programs is dependent on metrics that reflect the scientific content of the policy or programs as well as demographic information.

The metrics must be:

reliable, valid, and meaningful to the user

accessible, interpretable, and readily analyzable by the user

And the system used to compile and summarize the metrics must be adaptable to a variety of circumstances


Gap analysis investments in research prof. norman braveman

Education