Recent Advances in Performance Measurement of Federal Workforce Development Programs Evaluation and Performance Management of Job Training Programs Organized by European Commission Directorate-General for Employment, Social Affairs and Equal Opportunity and University of Maryland School of Public Policy Randall W. Eberts W.E. Upjohn Institute for Employment Research November 7, 2009
19
Embed
Recent Advances in Performance Measurement of Federal Workforce Development Programs
Recent Advances in Performance Measurement of Federal Workforce Development Programs. Evaluation and Performance Management of Job Training Programs - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Recent Advances in Performance Measurement of Federal
Workforce Development Programs
Evaluation and Performance Management of Job Training Programs
Organized by European Commission Directorate-General for Employment, Social Affairs and Equal Opportunity and University of Maryland School of Public Policy
Randall W. EbertsW.E. Upjohn Institute for Employment Research
November 7, 2009
2
Purpose
• Describes a Department of Labor project conducted by the Upjohn Institute that addresses some of the problems with current workforce system performance targets outlined in previous presentations:– Cream skimming– Inability to account for differences in local labor market conditions
• Accounts for factors outside the control of state and local administrators– By “outside the control,” we mean factors that affect performance
outcomes but are not related to the services and other assistance provided by the programs
– Local labor market conditions (unemployment rates)– Personal characteristics of participants (prior work history, educational
attainment, barriers to employment)• Offers a systematic, objective and transparent framework for:
– Setting targets – Focusing on the value-added of WIA services– Diagnosing WIA performance – Leveling the playing field
3
Framework
• Procedure follows the basic regression-adjusted approach used to adjust JTPA performance targets
• Uses estimates of the effects of unemployment rates and personal characteristics on performance outcomes, based on all WIA exiters in all 50 states
• Estimates are based on the experience of individual participants within their local labor markets
• Procedure allows the aggregation of performance outcomes and factors from the individual to the WIB to the State to the Nation– By using the same weights for each level of jurisdiction, the
differences add up– Thus the targets are consistent across jurisdictions
• Focus on three WIA programs and three common measures
4
Procedure
• Three step process: one for each jurisdictional level: national, state, WIB
• Step One: Adjust the national targets for assumed changes in unemployment rates
• Step Two: Use the national adjusted targets as the departure for setting state performance targets– State and national performance outcomes differ because of
differences in unemployment rates and participant characteristics
• Step Three: Use each state’s adjusted targets as departure for setting targets of WIBs within the state– WIB and state performance outcomes differ because of
differences in unemployment rates and participant characteristics
5
Basic Equation
• Use ordinary least squares regression to relate the performance measures to individual participant characteristics and local labor market conditions as measured by local unemployment rates
Yisq = bo + b1*Xisq + b2Ds + b3Dq + b4Usq + error
Yisq : performance measure Xisq personal characteristics and employment history
Ds : state or WIB dummy Dq : quarter dummy Usq : quarterly unemployment rate by WIB or state
• Unemployment rates are entered in three ways depending on performance measure
• Personal characteristics and employment history are entered as categorical variables with one of the categories omitted from the equation as the reference group
(Percentage change of the performance measure associated with a one percentage point change in the unemployment rate; estimates are statistically significant at the 0.001 (***), 0.01 (**) and 0.05 (*) confidence levels)
The adjusted targets take into account changes in the assumed unemployment rates, whereas GPRA targets remain flat during increases in unemployment rates.
12
Step Two: State Targets• State targets differ from national targets:
– Differences in unemployment rates– Differences in personal characteristics
• Add adjustment to the departure national target rate
A B C D E
WIA Adult
Entered Employment
State A National Difference(A-B)
Effect on EE Adjustment: Weighted Difference
(C * D)
Unemployment rates 12.6% 8.3% 4.3 -1.8 -7.70
High School drop out 10.3 4.6 5.7 -0.049 -0.279
BA degree 7.6 1.8 5.8 +0.022 +0.128
Disabled 6.4 4.9 1.5 -0.096 -0.144
Work experience 39.0 64.0 -25.0 +0.14 -3.50
Adjusted Target 53.3 64.8 Total adjustment(add column E)
-11.5
13
Step Three: WIB Targets• WIB targets differ from state targets:
– Differences in unemployment rates– Differences in personal characteristics
• Add adjustment to departure state target rate
A B C D E
WIA Adult
Entered Employment
WIB A in State A
State A Difference(A-B)
Effect on EE Adjustment: Weighted Difference
(C * D)
Unemployment rates 7.8% 12.6% -4.8 -1.8 +8.64
High School drop out 4.7 10.3 -5.7 -0.049 +0.279
BA degree 9.3 7.6 +1.7 +0.022 +0.037
Disabled 2.3 6.4 -4.1 -0.096 +0.394
Work experience 39.5 39.0 -0.05 +0.14 +0.07
Adjusted Target 62.7 53.3 Total adjustment(add column E)
+9.42
14
Adjustments Add Up • Adjustments add up from WIB to State to Nation
– Based on differences in characteristics– Weights are the same at all levels
State Adjustment ee ret earnings ee ret earnings place att lit
Adult Dislocated Youth
The adjusted targets, and their components, are shown for six states. It should be noted that the direction of the effect of the unemployment rate may be different for retention than for the other two performance measures since retention is estimated as the change in the unemployment. Differences in the changes in the unemployment rate between the state and the nation may be different from the differences in the levels.
16
WIA Adult WIA Dislocated Worker
Mean Minimum Maximum Mean Minimum Maximum
Entered Employment
UR 0.63 -10.4 4.9 0.35 -5.8 2.7
Labor Market -0.12 -3.7 0.03 -0.15 -5.7 1.3
Personal Characteristics
-0.42 -8.8 10.1 -0.07 -5.7 1.6
Retention Rate
UR 0.27 -1.2 1.6 0.27 -1.2 1.6
Labor Market -0.11 -4.0 1.1 -0.10 -4.2 1.6
Personal Characteristics
-0.24 -6.3 7.7 0.25 -2.7 1.4
Earnings
UR 93.5 -1536 724 43.2 -710 335
Labor Market 29.8 -8.4 947 71.5 -20 2284
Personal Characteristics
-210 -2595 1572 642 -1596 2381
Means of the Adjustment Components for WIA Adult and Dislocated Workers
17
State differences in personal characteristics contribute to a difference of as much as 12 percentage points in performance outcomes and the differences have increased in recent years.
Significant Differences in Personal Characteristics Across States and WIBs
-6
-4
-2
0
2
4
6
8
10
2000 2001 2002 2003 2004 2005 2006 2007
Program years
Per
cen
tag
e P
ts
Note: WIA Adult Entered Employment. Participant attributes are weighted by their estimated effect on performance outcomes.
Mean
Minimum
Maximum
18
Summary
• Target adjustment procedure provides a systematic, transparent, and objective way to set national, state, and WIB performance targets for WIA programs
• Adjustment factors, since they are related to factors that are familiar to administrators, can be easily scrutinized to better understand and diagnose programs– Also familiar since state adjustment procedure is similar to the
JTPA method
• National performance targets have already been adopted by USDOL and GAO
• Currently exploring adjustments at the state and WIB levels