Gap analysis investments in research prof. norman braveman
Post on 08-May-2015
190 Views
Preview:
Transcript
Braveman BioMed Consultants - "Working with you for excellence in research." 1
Gap Analysis: Guiding Government
Investments In Research Using Data Driven
Scientific Program and Policy Development
Norman S. Braveman, Ph.D.
President
Braveman BioMed Consultants
Elsevier Reed Conference
Accelerate Research Excellence: Evaluate, Benchmark and
Strategize
September 23, 2001
New Delhi, India
Braveman, N.S. What science indicators can tell policymakers. In Margaret O. Meredith, Stephen D. Nelson and Albert H. Teich (Eds.) Science and Technology Yearbook: 1991. Washington, D.C.: American Association for the Advancement of Science, 267-278.
Braveman, N.S. Evaluation at NIH: The policy issues. In Susan L. Sauer (Ed.) R & D in FY 1989: Looking Ahead in an Election Year. Washington, D.C.: American Association for the Advancement of Science, 1988.“Evaluation at NIH: The policy issues”
Policies based on data are better than those that are not
The primary driver or source of data for development of science policy and programs should be science itself
But we need to be cognizant of other drivers and sources of information
AS A GRANT MAKER YOU HAVE AN EXTREMELY IMPORTANT IMPACT ON SCIENCE
© Braveman BioMed Consultants - "Working with you for excellence in research." 2
So what prevents decision making based on science from happening?
Limited and/or uninformative data –> need for additional and new metrics (demographic/bibliometric vs. content)
Lack of access to or difficulty in accessing data –> getting data is not easy and time consuming for first level users
Austere times –> tendency to not develop
Prevailing attitudes about formal planning in science
© Braveman BioMed Consultants - "Working with you for excellence in research." 3
© Braveman BioMed Consultants - "Working with you for excellence in research." 4
Prevailing attitudes toward PP&D in science
It‟s not possible or advisable to plan science -> can lock you into unproductive pathways
It‟s important to be flexible and able to change directions quickly to follow leads/new directions
„Peer review‟/publication of findings are the only endpoints that matter in determining future directions
Serendipity or chance historically lead to great findings –> the double helix; clinical applications of iPS
© Braveman BioMed Consultants - "Working with you for excellence in research." 5
"In the fields of observation, chance favors only the prepared mind." Louis Pasteur
A „prepared mind‟ (in science) is one that has considered alternative paths, even in an informal way, particularly when results don‟t turn out as expected.
“A bad is experiment is one that will provide answers only if the
hypothesis is supported or disproven. A good experiment is one that will provide answers no matter what the outcome of the
experiment.” Anonymous Peer Reviewer
The context
Some sources of support for research in the US
Current interests of grant makers
An approach to Data Driven Scientific Program and Policy Development using PP&D
Logic Analysis
Strategic and Implementation Planning
Gap Analysis
Scientific fingerprinting – a way to collect data for PP&D of science policies and programs
What is scientific fingerprinting
Examples of current use in PP&D
© Braveman BioMed Consultants - "Working with you for excellence in research." 6
Funding Agency
2011 Agency Budget Request (% for R&D)
National Institutes of Health $32.1 billion (85)
National Science Foundation $7.4 billion (80)
Department of Energy $28.4 billion (18)
NASA $19.0 billion (33)
Department of Defense $708.0 billion (10)
US Department of Agriculture $146.0 billion (2)
Various State Funding Programs (stem cells, tobacco –related diseases)
--
© Braveman BioMed Consultants - "Working with you for excellence in research." 7
© Braveman BioMed Consultants - "Working with you for excellence in research." 8
There are approximately 4000 private grant makers in the US
Health Research Alliance: not-for-profit, non-governmental funders of biomedical
research and training from biomedical science to applications that advance health
47 member organizations
HRA member organizations provide over $1.5 billion in research awards each year, funding over 5500 researchers annually
© Braveman BioMed Consultants - "Working with you for excellence in research." 9
Many of these sources of funding relies on PP&D in order to
stay current in their area of science
be ahead of the curve in planning for future scientific developments
Insure that their resources are used appropriately and in the most effective way possible…particularly in austere times
© Braveman BioMed Consultants - "Working with you for excellence in research." 10
Scientifically meritorious research
Innovative research that will move the field ahead
Development of the research workforce of the future Basic scientists
Clinician and clinical scientists
Interdisciplinary scientists
Diversity in racial and ethnic backgrounds
Topic of the day Personalized medicine
Translational research
Systems approach to biomedicine
Stem cell research
Disease of the day Obesity
Cancer
Heart disease
© Braveman BioMed Consultants - "Working with you for excellence in research." 11
The foundation of PP&D is evaluation…
a systematic method for collecting, analyzing, and using data and information to answer questions about projects, policies and scientific programs
„Data „ is not the plural of „anecdote‟. Unknown
© Braveman BioMed Consultants - "Working with you for excellence in research." 12
Demographic/Quantitative Who is receiving funding
When did they receive it
Where are they located
How much did they receive; how much in total is being spent on the area of science
How many publications/inventions have resulted from the research (return on investment)
How many publications have resulted from grants
© Braveman BioMed Consultants - "Working with you for excellence in research." 13
Demographic/Categorical
Content Based/Qualitative Captures the content of the research in the portfolio ->
What is the content of the research that is being addressed? What isn‟t being addressed…and why?
Systematic, comprehensive, reliable and valid
Can link a current portfolio to portfolios that exist in other organizations (e.g., funding agencies)
Can add important scientific information in addressing some of the quantitative questions
© Braveman BioMed Consultants - "Working with you for excellence in research." 14
What are the goals of PP&D for organizations that fund research?
To maximize the impact of funding by advancing science
To determine whether what has been done has met the specific goals and objectives of the organization
To chart the future priorities and directions of the organization
Logic Model Clear statement of mission and goals of the organization
Identification of inputs and outputs
Identification of expectations and measures of success
Strategic Planning Statement of values, mission, priorities and goals
Implementation Planning Timelines, benchmarks, mechanisms, sequence of events
Gap Analysis
© Braveman BioMed Consultants - "Working with you for excellence in research." 15
A Major Component of PP&D
Answers the questions – Where are we?
Where do we want to be?
Are we making the best use of our resources?
“It‟s the difference between what an organization is doing or has been doing and what the organization should be doing or wants to do…” Source Unknown
© Braveman BioMed Consultants - "Working with you for excellence in research." 16
Applied from a business/economics environment to organizations
involved in funding research
Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation
What are the inputs in the world of science? -> money, people, facilities/equipment, knowledge
What do we mean by “optimization” or “integration” when we‟re talking about funding of science? -> a unique combination of inputs that yields best performance on outcome measures
© Braveman BioMed Consultants - "Working with you for excellence in research." 17
Applied from a business/economics environment to organizations
involved in funding research
Identifies the gap between the optimized allocation and integration of the inputs, and the current or future level of allocation
Used to compare a program‟s actual performance with its potential performance
How do we determine a funding organizations potential performance?
from strategic plan, implementation plan and logic model
How do we measure a funding organization‟s actual performance?
use outcome measures developed from logic model
© Braveman BioMed Consultants - "Working with you for excellence in research." 18
Also addresses specific questions about the current state-of-the-science as well as about pathways to scientific advancement
What do we need to do to move ahead?
What are the barriers and where are the opportunities?
Gaps are then defined in terms of the current portfolio of the organization and the near- and long-term state-of-the-science
© Braveman BioMed Consultants - "Working with you for excellence in research." 19
© Braveman BioMed Consultants - "Working with you for excellence in research." 20
Background & Goals
Logic Model
Outcome Measures
Data Collection
Expert Panel
•Future of area •Barriers •Possible priorities
Organization Data-Driven Decision Process
Gap Analysis
Current portfolio vs. “ideal” portfolio
Stakeholders
Strategic Plan
Background (stakeholder input; source documents) What is the organization‟s role in the big picture?
What are our present goals?
Where are we going?
How have we gotten to where we are?
What has changed that we need an updated assessment?
Who are the other participants and what is their role?
What is/has been our relationship with them?
© Braveman BioMed Consultants - "Working with you for excellence in research." 21
Background
Portfolio Analysis (data collection) What has the organization accomplished toward current
goals?
What science have we been funding? What science have we not been funding?
What is/are the scientific content of the same or a similar portfolio(s) of other organizations?
What science areas are covered by applications that have been submitted to our organization?
Of those submitted to our organization, which ones are being funded? Which ones are not being funded…and why?
© Braveman BioMed Consultants - "Working with you for excellence in research." 22
Background
Data collection
Gap Analysis Compare content of current portfolio with „ideal‟ portfolio
Assess content of current support by all organizations for duplication, overlap and collaboration
Compare the content of current portfolio(s ) with the future of science
Compare content of all submitted applications with program goals and future of science
Compare content of successful applications with program goals and future of science
Compare content of unsuccessful applications with the future of science and program goals
© Braveman BioMed Consultants - "Working with you for excellence in research." 23
Background
Data collection
Gap Analysis
Expert Panel (thought leaders) Where is the field going in the short-/long-term?
What are the scientific barriers to getting there?
Assessment of current content of programs by all organizations for duplication, overlap and collaboration
Where are the gaps between the content of current portfolios and the future of science…with „ideal‟ portfolio
Recommendations/priorities about where the organization being analyzed can make significant and unique contributions/impacts
© Braveman BioMed Consultants - "Working with you for excellence in research." 24
These are the drivers of the development of science policies and
funding programs
But they aren‟t the only ones…the final step -> organizational decision making
© Braveman BioMed Consultants - "Working with you for excellence in research." 25
Background
Data collection
Gap Analysis
Expert Panel
Organizational Decision Making (stakeholders and leadership) What unique scientific contributions do we want to be
known for?
Resources available – people, fiscal
Possibilities for collaboration/leveraging with other organizations
Political considerations
Determine which gaps are going to be addressed
Develop timelines and benchmarks
© Braveman BioMed Consultants - "Working with you for excellence in research." 26
© Braveman BioMed Consultants - "Working with you for excellence in research." 27
A representation of a document generated with the Elsevier Fingerprint Engine
A precise abstract representation of a text which allows us to “look into the text” (content) rather than only looking at the meta data (demographics).
© Braveman BioMed Consultants - "Working with you for excellence in research." 28
© Braveman BioMed Consultants - "Working with you for excellence in research." 29
© Braveman BioMed Consultants - "Working with you for excellence in research." 30
NIH Research, Condition, and Disease Categorization Process (RCDC) Introduced in 2009
A computerized process used at the end of each fiscal year to sort and report the amount funded in each of 229 historically reported categories of disease, condition, or research area.
The Florida Biomedical Research Program – scientific program management and development
National Heart, Lung and Blood Institute (NIH) – identifying ideas and tracking progress
© Braveman BioMed Consultants - "Working with you for excellence in research." 31
Florida Biomedical Research Program:
Integration of PP&D Within An Ongoing Scientific
Program
© Braveman BioMed Consultants - "Working with you for excellence in research." 32
Strategic Plan P
rogr
am P
lan
nin
g P
rogr
am D
eve
lop
me
nt
Portfolio Analysis
Implementation Plan
Expert Science Panel
Recommendations
Council Review
Logic Model & Gap Analysis • Program Goals & Objectives • Outcome Measures • Gap Analysis
Call Development • Priority science areas • Funding mechanisms • Application configuration
Issue RFA(s)
Annual Report •Grant Outcomes •Success Stories •Program Outcomes
Notice of Award
Summary Statements To Applicants
Grantsmanship Seminars (Answer questions about RFA)
Ap
plic
atio
ns
Applications
Peer Review
Council Review & Funding Recommendation
Gra
nt
Mo
nit
ori
ng
Grants Management &
Reviewer Finder Systems
Grants Management &
Program Review
Progress Reports
Financial Reports
Gap Analysis
Logic Model • Program Goals • Outcome
Measures
© Braveman BioMed Consultants - "Working with you for excellence in research." 33
Biomed Experts & Reviewer Finder
National Heart, Lung and Blood Institute (NIH)
Identifying ideas and tracking their implementation through
program development
© Braveman BioMed Consultants - "Working with you for excellence in research." 34
© Braveman BioMed Consultants - "Working with you for excellence in research." 35
SCIENTIFIC WORKSHOPS
NIH OFFICES
DHHS LEADERSHIP
NATIONAL
ADVISORY
COUNCIL
OR BOARD
EXTRAMURAL
SCIENTISTS
INDUSTRY/ PRIVATE SECTOR
CONGRESS
PROFESSIONAL GROUPS
PATIENT
GROUPS
OTHER FEDERAL AGENCIES
IC Staff
Decision
Process
NIH Early
Notification
System
ISSUE RFA OR PA
IC ADJUSTMENTS
Opportunity
Ad Hoc Advisory
Groups
I
C
L
E
G
I
S
L
A
T
E
D
M
I
S
S
I
O
N
© Braveman BioMed Consultants - "Working with you for excellence in research." 36
© Braveman BioMed Consultants - "Working with you for excellence in research." 37
© Braveman BioMed Consultants - "Working with you for excellence in research." 38
The effective use of data driven PP&D in developing science policy and funding programs is dependent on metrics that reflect the scientific content of the policy or programs as well as demographic information.
The metrics must be:
reliable, valid, and meaningful to the user
accessible, interpretable, and readily analyzable by the user
And the system used to compile and summarize the metrics must be adaptable to a variety of circumstances
© Braveman BioMed Consultants - "Working with you for excellence in research." 39
top related