Slide 1 Brian Zuckerman Presented to COSEPUP Evaluation of the National Science Foundation's Experimental Program to Stimulate Competitive Research (EPSCoR) and Similar Programs in Other Federal Agencies December 17 th , 2012 Results of the Assessment of the Defense Experimental Program to Stimulate Competitive Research (DEPSCoR)
Results of the Assessment of the Defense Experimental Program to Stimulate Competitive Research (DEPSCoR). Brian Zuckerman Presented to - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Slide 1
Brian Zuckerman
Presented to
COSEPUP Evaluation of the National Science
Foundation's Experimental Program to Stimulate
Competitive Research (EPSCoR) and Similar Programs
in Other Federal AgenciesDecember 17th, 2012
Results of the Assessment of theDefense Experimental Program to Stimulate
Competitive Research (DEPSCoR)
Slide 2
Study Origin and Timeline
• FY2008 Defense Authorization Act (Public Law 110-181), Section 241, instructed the Secretary of Defense to utilize a defense Federally Funded Research and Development Center (FFRDC) to carry out an assessment of the DEPSCoR program.
• Institute for Defense Analyses was tasked to conduct study February 2008
• Results briefed to Senate and House Armed Services Committee staff (SASC/HASC) November 2008
• Study results cleared for public distribution January 2009
Slide 3
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 4
Descriptive Statistics
• Between 1993 and 2008, 729 total DEPSCoR awards– 546 individual Principal Investigators (PIs)
» 121 PIs with multiple awards» 42 with three or more» 1 PI with eight awards
– 22% of PIs have won 42% of awards
• 1993-2008 funding of $243 million– Decline after 2000 peak partially reversed in 2008 competition
• 27 states and territories (states) have been eligible for at least one year since program authorized in current form in 1995, plus Missouri (eligible in 1993)
– All eligible states except for the Virgin Islands have won awards
• 19% (5) of eligible states have won 35% of awards– Montana, Alabama, Oklahoma, Nebraska, South Carolina
• 7% (5) of institutions have won 28% of awards– Montana State, U. Nebraska-Lincoln, U. Wyoming, West Virginia U., U.
Arkansas
Slide 5
DEPSCoR Program Objectives
1. To enhance the capabilities of institutions of higher education in eligible states to develop, plan, and execute science and engineering [S&E] research that is competitive under the peer-review systems used for awarding federal research assistance
2. To increase the probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government for science and engineering research
Slide 6
The DEPSCoR State Share of DOD S&E Increased
Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year
Note: Graph includes all states ever involved in DEPSCoR program 1995-2008
Note: The dotted lines represent linear regression models applied to the data
R&D (Actual) R&D (Linear Model) Research (Actual) Research (Linear Model)
Slide 7
DEPSCoR Funding As a Percentage of DOD Funding in DEPSCoR States
• DEPSCoR has declined in importance as a source of funding for eligible states since 2000
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year
Near or Above Threshold Rising Fast Middle Lagging
• DEPSCoR-eligible states fell into four groups:– 6 states Near or above threshold (AL, HI, LA, MS, NM, SC)– 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD)– 6 states Middle (AR, DE, KS, OK, RI, TN)– 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY)
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year
Slide 9
• DEPSCoR share of university S&E R&D funding varies substantially by state
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year
Note: Average calculated as total DEPSCoR funds during eligible years divided by total DOD funds during eligible years
Average DEPSCoR Funding as a Fraction of DOD Funding by State
Slide 10
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 11
Expanded National Research Infrastructure (1)
• Involving new investigators:– Recent (2006-2008) cohorts had about 60% new PIs
– Most Army-funded DEPSCoR awardees (82%) had not previously received funding from Army Research Office (ARO)
– 56% of PIs had been funded by the NSF either previous to or within the same year of their first DEPSCoR award
• Training graduate students and postdoctoral fellows:– ARO and Office of Naval Research (ONR) data suggest that
awards fund about 1 PhD, 1 Master’s degree, 2 postdocs
• Building physical infrastructure: – DEPSCoR awards have supported purchase and maintenance
of equipment but data not collected systematically by services
Slide 12
Expanded National Research Infrastructure (2)
• Leveraging new funding for defense-related research is limited:
– 8% of non-DEPSCoR ARO awardees in DEPSCoR states received a DEPSCoR award before (or in the same year as) their first non-DEPSCoR ARO award
– 4 DEPSCoR awardees (less than 1%) won a DOD Multidisciplinary University Research Initiative (MURI) award after or in the same year as their first DEPSCoR award
» 2 of these investigators received non-DEPSCoR DOD funding before their first DEPSCoR award
• Leveraging other funding:– 63 DEPSCoR awardees (12%) received their first NSF funding
subsequent to their first DEPSCoR funding
Slide 13
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expand national research infrastructure
2. Activities consistent with statute• Activities were found to be consistent with statute
3. Assessment of program elements• Available data on DEPSCoR program activities and outcomes are
insufficient for monitoring and evaluation purposes
4. Assessment of activities of state committees• State committees prioritized proposals that met state infrastructure
development goals and reflected the mission/research needs of DOD• Committee processes varied widely from state to state and limited and
variable data prevented detailed assessment
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 14
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 15
Approach Taken
• Assessment focused on comparing the current state-based formula with an institution-based criterion based on a maximum threshold for DOD S&E research funding
• As directed in the study legislative mandate, particular emphasis was given to supporting defense missions and expanding the nation's defense research infrastructure
– Since the legislative charge for the assessment does not specify how “expanding the nation’s defense research infrastructure” should be interpreted, the assessment considered advantages and disadvantages using a variety of possible interpretations
• Assessment also considered alternative criteria:– Indicators of state-level S&T capacity (SEI, Milken Institute)– Normalization by state population (Census)
• State-based and institution-based criteria can be combined– PI-level criteria are also possible
Slide 16
Effect of Institution Based Funding Threshold
• 77 institutions in 2008 DEPSCoR-eligible states (360 total) received nonzero research funding from DOD in 2005
– 38 Carnegie “Very High” or “High” research institutions in DEPSCoR states
• $5 million threshold would make 269 institutions eligible– Twelve of the 77 institutions in currently eligible jurisdictions (e.g., University of
Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of Nebraska) would become ineligible
• Considering only Carnegie “Very High” or “High” research universities, shift would increase number of eligible universities from 38 to 121
Maximum DOD Funding Threshold
All Universities With Non-Zero DOD Research
Funding in 2005
Eligible for the 2008 Competition
Potentially Eligible for the 2009 Competition
No limit 360 77 114
$10 million 316 75 109
$5 million 269 65 90
$3 million 231 55 76
$1 million 157 36 52
Slide 17
Definition of “Expanding National Research Infrastructure”
• If interpreted as increasing equity in funding among states or achieving state-level infrastructure goals, a state-based formula would be advantageous
– Current state-based formula for eligibility harnesses the state EPSCoR committees to coordinate infrastructure and capacity-building at the state level
– If intent is to increase the competitiveness of historically-underrepresented states, eligibility can easily be determined at a state level
• If interpreted as involving new investigators or institutions in defense-related research, an institution-based formula would be advantageous
– Allows targeting of programmatic resources toward investigators at institutions that have not historically built relationships with DOD
– Approach taken by late 1980s/early 1990s DOD Research Initiation Program
– While a state-based approach includes the flexibility to channel DEPSCoR proposals toward historically underrepresented universities or new investigators within an eligible state, the institution-based approach allows greater flexibility to target underrepresented universities and investigators throughout the entire country
• Could not be determined whether state-based or institution-based approach would elicit more qualified applications to support defense missions
– Larger number of eligible institutions implies more proposals, but quality indeterminate
Slide 18
Effect of State Population Normalization
• Green states were eligible in 2008, red states are “graduates”, blue states were never eligible
• Comparing top chart with bottom shows dramatic difference in order
• DEPSCoR graduates among highest per capita recipients
• Several DEPSCoR-eligible states (e.g., AK, DE, ND, RI, MT, SD) above average in funding per capita
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
State
FY
20
05
Do
D U
niv
ers
ity
S&
E R
&D
Fu
nd
ing
(K
$)
$0.00
$10.00
$20.00
$30.00
$40.00
$50.00
$60.00
$70.00
State
FY
20
05
Do
D U
niv
ers
ity
S&
E R
&D
Fu
nd
ing
Pe
r C
ap
ita
’05 DOD S&E R&D Funding, by State
’05 Per capita DOD S&E R&D Funding, by State
Slide 19
Other Capacity Indicators
• Some DEPSCoR-eligible states (e.g., NH, RI, DE, KS) above average in Milken Institute state S&T index
– Some “graduated” states below average
• Some DEPSCoR-eligible states (e.g., DE, KS, NE, NH, RI, VT) in the top or second quartiles for more than half of Science and Engineering Indicators’ seven R&D output measures
0
10
20
30
40
50
60
70
80
90
Milk
en In
dex,
200
8 Sc
ore
Slide 20
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 21
IDA Recommendations
• DOD should change the current process for review of proposals to focus more heavily on investigators’ future potential to conduct research rather than on their current research capabilities
• DOD program managers should be formally encouraged to serve as mentors and facilitators for DEPSCoR investigators seeking to engage in further defense-related research
• DOD should create data systems that will allow systematic tracking of DEPSCoR activities and outcomes
• Congress should re-examine and consider clarifying ambiguities in the DEPSCoR legislative mandate
• Once the DEPSCoR objectives have been clarified, redesign the program with a strategy for enhancing competitiveness at relevant level (e.g., individual, institution, state)
Slide 22
Some DEPSCOR Legislative Language Is Inconsistent
• Objectives specify that the research institution is the level at which competitiveness is to be enhanced but authorizing legislation also specifies that eligibility for DEPSCoR be determined at the state level
– 2008 change to eligibility criteria leaves ambiguous whether Congress intends DEPSCoR to target competitiveness at the institution or state level
• Objectives specify that goal is to increase probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government but eligibility determined based on DOD funding
– Statute does not discuss whether Congress intends DEPSCoR should focus narrowly on [6.1] research that is relevant to DOD missions and priorities although this is how program is run
Slide 23
Recommendations for Legislative Clarification
• To ensure that the program is implemented in a manner that meets current legislative priorities, Congress should clarify whether
– The program is intended to increase competitiveness for federal research funding in general or for particular types of research funding
– The program is intended to fund primarily basic research, primarily applied research, or a combination
– The primary unit at which competitiveness should be enhanced is the institution, state, or other (e.g., individual investigator)
• Eligibility criteria, funded activities, and other program elements should be structured in accordance with the program’s objectives
Slide 24
Competitiveness can be Enhanced at Different Levels
• DEPSCoR supports individual or small-group research projects and can therefore be understood to primarily target capacity-building at the level of the individual
– Though supporting training and purchase of equipment target capacity-building at level of the institution as well
• DEPSCoR also operates at state level (e.g., involvement of EPSCoR committees, state-based eligibility criteria)
• While it might be argued that institutional competitiveness depends on individual competitiveness and state competitiveness depends on institutional competitiveness, these dependencies are neither straightforward nor self-evident
Slide 25
Structure Program to Enhance at the Desired Organizational Level
• Assessment identified other programs that might serve as potential models for DEPSCoR
» One per state; integrated into state S&T plan; EPSCoR committee involvement; capacity building rather than research
– Centers: NIH IDeA/Centers of Biomedical Research Excellence (COBRE)
» Large-team research, equipment, mentoring; milestones for transition to support by standard NIH mechanisms
– Investigator-level: DOD Research Initiation Program (RIP) or NIH/IDeA Academic Research Enhancement Award (AREA) Program
» Institution-based eligibility criteria; small research awards to individual investigators; build relationships/expertise to allow for transition to support by standard research mechanisms
Linear regression model of DEPSCoR state share of DOD university funding
Slide 29
DEPSCoR Emphasis is on Basic Research
• Research oriented towards developing operationally useful devices or components is not a legislatively-mandated program goal
• DEPSCoR projects mostly fund basic research– Not a statutory requirement– Administered by basic research organizations within Air Force
and Army, plus Office of Naval Research (ONR)» Interviews with DEPSCoR program officers suggest that more
applied research does not perform well in review– DEPSCoR program officers tend to track transitions to 6.2 or
6.3 research, but not further
• Two transitions to operational use were found
Slide 30
Two Transitions to Operational Use Were Found
• Ronald DeVore, University of South Carolina: Wavelet mathematics for image compression for tactical applications
– DeVore and colleagues collaborate with program managers at Naval Air Warfare Center NAWC at China Lake to deliver wavelet-based image processing platform
– Charles Creussere of NAWC implements wavelet-based image processing system for navigation in the Tomahawk Block II program
• Michael Pursley, Clemson University: wireless, mobile, distributed, multimedia communication networks
– Pursley and colleagues working since 1970s with ITT on tactical radio development
– Group used DEPScoR funding to support research that provided better anti-jam communications and greater multiple-access capability
– Research led to the Soldier Level Integrated Communications Environment (SLICE) wide band networking waveforms that have been integrated into the SINCGARS radio