Early Lessons from TAACCCT Evaluation by Donna Tonini, Research Specialist, OCCRL National Council for Workforce Education Conference Milwaukee, WI 10/16/2013.

Early Lessons from TAACCCT Evaluation

by Donna Tonini, Research Specialist, OCCRL

National Council for Workforce Education Conference Milwaukee, WI 10/16/2013

Who is OCCRL?

Our Mission: A research office which studies policies, programs, and practices designed to enhance outcomes for diverse youth and adults who seek to transition to and through college to employment.

Examples of our evaluation projects include:•TAACCCT Grant Evaluations•Shifting Gears•Adult Education Bridge Program Evaluations•College and Career Readiness•Adult Career Pathways

What is TAACCCT?• Department of Labor (DOL) Grant: Trade Adjustment

Assistance Community College and Career Training, 10/2011 – 9/2014

• Purpose: to provide funding to community colleges to scale up and improve short-term (< 2 years) educational & career-training programs to prepare students for employment in high-skilled, high-wage jobs.

• Target Population: While the grant supports TAA (Trade Adjustment Assistance) workers (who were retrenched /downsized due to

overseas job outsourcing), the DOL also expects the programs will benefit a wide range of individuals.

• Goal: to enable the higher education community to develop, institute and improve education and training programming which will assist the target population in acquiring the skills, degrees, and credentials to meet the high-skilled and high-wage employment needs mandated by employers.

Understanding Priorities of the Sponsor and the Goals of the Colleges

The DOL’s Round I Priorities for TAACCCT:

•Accelerate progress for low-skilled workers

•Improve retention and achievement rates and/or reduce time to completion

•Build programs that meet industry

•Strengthen online and technology-enabled learning

Introducing the Colleges:H2P: Health Professions Pathways Consortium

• Cincinnati State – Lead (OH)

• Anoka-Ramsey Community College (MN)

• Ashland Community and Technical College (KY)

• City Colleges of Chicago (IL)

• El Centro College (TX)

• Jefferson Community and Technical College (KY)

• Owens Community College (OH)

• Pine Technical College (MN)

• Texarkana College (TX)

TAACCCT Strategies – H2P

• Online assessment and enhanced career guidance

• Contextualized developmental education

• Competency-based core curriculum

• Industry-recognized stackable credentials

• Enhanced retention support

• Training programs for incumbent health professions workers

• Enhanced data and accountability systems

• Galvanize a national movement

Develop Evaluation Plan Based on Priorities & Goals - Three Linked

Studies

Implementation Evaluation of the model, including Programs of Study and core strategies, as articulated through consortium theory of change.

Impact Evaluation of short- and longer-term outcome measures for participants, including a rigorous evaluation (quasi or true experimental design).

Continuous Improvement through regular review of the measures and outcomes, using consortium and college data to support sustainability and scale up.

Implementation

Outcomes

Continuous Improvement

Develop Questions that Address Priorities & Goals

What is the baseline and progress on implementation related to TAACCCT-impacted Programs of Study and strategies?

•Who are your partners and what roles do they play in supporting the implementation of TAACCCT-impacted Programs of Study and strategies? How are student outcomes affected?•What strategies and activities are the consortium and co-grantee colleges implementing, and what is the expected impact on the Programs of Study? How are student outcomes affected?•What modifications and improvements are being made to funded Programs of Study? How are student outcomes affected?•What barriers and supports exist, and how do they impact colleges’ efforts in creating and enhancing Programs of Study and strategies? How are student outcomes affected?

Lessons

• Differing interpretations of DOL requirements

• Clarity of expectations

• Information sharing

• Defining a common set of measurables

• Creating value for the client

  Open, Clear, and Constant Communication!

• Identify key stakeholders: Program leadership Institutional Review (IR) External partners (Employers, Workforce Agencies)

• Involve stakeholders in process: Engage colleges in shared vision Develop logic models

• Leverage promising practices: Communities of Practice (COP) Monthly teleconferences Basecamp Blogs

Sample Logic Model: Implementation of H2P Grant Strategies

(Graphic developed by D. Bragg, 2013)

Develop Data Collection and Analysis Plans to Address Priorities & Goals

List TAACCCT Partners (include employers, community, workforce & other

relevant entities )

New Partnership?

Role Explanation Section

Partner 5

Partner 1

Partner 2

Partner 3

Partner 4

Tools for Collecting Data on Strategies, POS

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Yes

/No

Mon

th

Yea

r

Current Yes 4 2012 No Yes 6 2012 No No No No Yes 6 2012 NoPlanned Yes 4 2013 No Yes 5 2014 Yes 5 2013 No No No No No

TAACCCT Activities

EXAMPLE: Medical Assistance Certificate

Current and

Planned Activities

Is this POS New or

Enhanced?

Program of Study (POS)

Additional course

Instructors

Enhanced/ Upgraded

Curriculum Other

Improved Facilities and/ or

InfrastructureNew

Courses

Professional Development Faculty/ Staff

Internships or Apprentice-

ships

Upgrading Classroom Supplies/

EquipmentNew

Technology

Enhanced

Highlighting Promising Practices

• Curricular Innovations: Core curriculum Contextualized

developmental education Galvanizing a national

movement

• Improved Student Services: Dedicated Retention

Specialist/Career Coach Online prior learning

assessments (PLA)

• Strengthened Career Pathways: Stackable credentials Incumbent worker

training programs

• Enhanced Partnerships: Employers Workforce agencies

Locating the Stories to TellWhat is the baseline and progress on

implementation related to TAACCCT-impacted Programs of Study (POS) and strategies?

POS: The largest number of POS are Associate’s degree programs.

Top 3 (of 8) Strategies at Implementation Level: * Enhanced Retention Support (8 of 9 colleges)* Online Assessment and Enhanced Career Guidance (7 of 9

colleges)* Enhanced Data and Accountability Systems (7 of 9

colleges)

Commonalities:Barrier: Lack of TAA participants (1)Strength: Value added by external partners (employers,

etc.)Area for Improvement: Communication, info-sharing

(inter-college)

Finding Patterns in the Data

How are the consortia, students and employers impacted?

Outcomes:•Enrollments - H2P: 3,835 Credentials - H2P:430

Sample Outcomes from student demographic course-level data:•Completers – Retention – Credit Hours - Employment

•Upcoming Analysis: Multi-level analysis of student level data - degree attainment, wage increases.•Pathways to Results – Continuous Improvement

Annual Performance Report – Table 1

Obstacles and Opportunities For Learning

• Please share your experiences from the field for discussion.

Office of Community College Research and Leadership University of Illinois at Urbana-Champaign 51 Gerty Drive, 129 CRC | Champaign, IL 61820 | Phone: (217) 244-9390 | Email:

[email protected]