© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice.

© 2010 by ACT, Inc. All rights reserved. 1

Creating an Aligned Assessment System to Promote Work Training

Readiness and Career Success

Steve RobbinsVice President, Research

ACT, Inc.

Though designed to meet a wide array of needs, all ACT programs and services have one guiding purpose:

Helping people achieve education and workplace success.


The Challenges

• Create a comprehensive assessment system that predicts success in education and work

• Tailor assessments to critical transition points and context

• Understand the complexity and interplay of cognitive and noncognitive factors across time and jobs


Do noncognitive factors matter?

• What we know from the world of work:– Project A from the military – Training and work outcomes


Combining Personality & Cognitive Ability Tests

Correlations between general cognitive ability and personality tests and measures of job performance in Project A

Cog Pers Both Criteria

.63 .26 .67 Core technical proficiency

.65 .25 .70 General Soldiering Proficiency

.31 .33 .44 Effort and Leadership

.16 .32 .37 Personal Discipline

.20 .37 .42Physical fitness and military bearing

(McHenry, Hough, Toquam, Hanson, & Ashworth, 1990)

Creating opportunity for incremental validity especially as criteria vary


Training and Work Outcomes

• Noncognitive factors are predictive of important job performance criteria (e.g., Rotundo & Sackett, 2002; Casillas et al., 2009)– Task Performance– Organizational Citizenship– Counterproductive Work Behaviors

– Safety • Noncognitive factors meet a variety of employer needs

– Screening– Selection– Training


Validity of Cognitive and Noncognitive Measures for Training and Work Outcomes

Training

OutcomesWork

Performance

General Mental Ability .54 a .62 b

Math .48 a .52 b

Reading .44 a .35 b

Conscientiousness .17 c .22 c

Emotional Stability .10 c .11 c

Notes. a Brown, Le, & Schmidt (2006). b Salgado et al. (2003). c Schmidt et al. (2007) corrected for indirect range restriction.


No Unified Theory of Key Characteristics

• However, the literature suggests that several factors are important:– Motivation, Social Engagement,

and Self-Regulation – Personality characteristics (the Big Five)

• Role of behavior ratings vs. self-report– Different perspectives– Coaching/training – Monitoring progress


Common and Specific Factors

• Are the same characteristics (i.e., common factors) associated with educational and work outcomes?

• Are there unique (i.e., specific) characteristics that contribute to the prediction of educational and work outcomes?

• In particular, how do age and setting impact the outcomes of interest?


Behavioral Development(Motivation,

Social Engagement, & Self-Regulation)

Cognitive Development & Acquisition of Foundational Skills

(Academic Learning & Achievement)

Social Capital

School/

Work

Factors

Integrated Pyramid for Success

CareerDevelopment(Exploration,

Crystallization, Choice & Match)

Allen & Robbins (2010)


The human

side of the pyramid


ACT’s noncognitive programs are part of a comprehensive and integrated set of solutions, which address the three broad areas essential for academic and work success across the lifespan:

Academic achievement & foundational skills Behavior (noncognitive) Career planning (noncognitive)

Set of Solutions


Achievement & Foundational

Skills

Grades

College Work6 7 8 9 1

011

12

EXPLORE® PLAN® ACT® COMPASS®

Comprehensive & Integrated Academic & Work Solutions: Grade 6 - Work

SRI and BMS SRIBehavior

WorkKeys®

Performance, Talent & BMS

ACT Interest InventoryCareer

Planning FitCollege Fit


Creating a Comprehensive Set of Behavior (Self & Other Report) Assessments

• Role of meta-analysis and validity generalization– Robbins et al. (2004). Do psychosocial and study skill

factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

– Robbins et al. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.


Behavior: Self-Report

Domain Middle School High School & College Work*

Motivation (Getting work done)

Academic Discipline Commitment to School Optimism

Academic Discipline Commitment to College Goal Striving General Determination Study Skills Communication Skills

Carefulness Discipline Order

Social Engagement(Getting along)

Family Attitude toward Education

Family Involvement Relationships w/ School

Personnel School Safety Climate

Social Activity Social Connection

Cooperation Goodwill Influence Sociability Striving

Self-Regulation(Keeping your cool)

Managing Feelings Thinking Before Acting Orderly Conduct

Academic Self-Confidence Steadiness

Optimism Stability

Note. *Creativity and Savvy do not fully map onto these domains.


Behavior: Other Report

Domain Middle School & High School Work

Motivation (Getting work done)

Initiative Planning & Organizing Sustained Effort Performance

Initiative Planning & Organizing Persistence Responsibility

Social Engagement(Getting along)

Communication Working with Others

Communication Working with Others

Self-Regulation(Keeping your cool)

Managing Feelings Conduct

Organizational Citizenship Stress Management Following Rules Adaptability


Career Planning

Domain Middle School & High School Work

Interests ACT Interest Inventory DISCOVER Career

Information Decision System

Fit Inventory

Values DISCOVER Career Information Decision System

Fit Inventory

- Neuman et al. (2009). Job congruence, academic achievement, and earnings

- Tracey & Robbins (2006). The interest-major congruence and college success relation: A longitudinal study.


Tailoring Assessments to Critical Transition Points and Context

• The role of noncognitive measures in promoting work training readiness and career success – Talent supply chain – Meeting various needs


Talent Supply Chain

• Noncognitive factors are important at all of the key transitions that lead individuals to the world of work.

Pre-KPrimary

(K-8)

Secondary

(9-12)

Post Secondary (2 & 4-yr)

Work


Noncognitive Measures Meet Various Workforce Needs

Purpose Needs ACT Noncognitive Solutions

Pre-Selection -- Screen people in most cost effective way

-- Find honest/dependable employees

-- Save time in the screening process

WorkKeys Performance

Recruitment -- Identify people who fit the work environment

-- Identify people with skills that match the job

WorkKeys Fit

Selection -- Select employees with skills that best fit the job

-- Save time in selection process

-- Select people in most cost-effective way

-- Find honest/dependable employees

-- Certifying employees

WorkKeys Performance

WorkKeys Talent

WorkKeys Fit


Noncognitive Measures Meet Various Workforce Needs (cont.)

Purpose Needs ACT Noncognitive Solutions

Coaching/

Development

-- Identify other jobs that an employee can fit

-- Develop employees for future company needs

-- Employees identify areas of improvement

WorkKeys Talent

WorkKeys Fit

Behavioral Monitoring Scales

Succession/Leadership Planning

-- Identify candidates for top-level positions

-- Develop employees for future needs of company

-- Retain top performers

WorkKeys Talent

WorkKeys Fit

Training/ Development

-- Identifying work readiness

-- Identify basic workplace skill levels

-- Educating about career planning

-- Job Placement

WorkKeys Talent

WorkKeys Fit

Behavioral Monitoring Scales


Critical Issues

• Self vs. other report

• Changing attitudes, behaviors, and skills

• Complexity of personality & career relationships– Different personality characteristics are important

for different jobs (whether across or within jobs)– Moderation effects of personality on ability– Curvilinear effects of personality on performance


Self vs. Other Report• Connelly & Ones (2010) conducted meta-analyses of

self-other ratings based on 44,000+ individuals across 263 independent samples. – Other ratings are strong predictors of behaviors,

particularly for academic achievement and job performance.

– In some cases, other ratings yielded predictive validities substantially greater than self-ratings.

– When other-ratings are added to self-ratings, results show considerable increases in validity (with gains more pronounced for Conscientiousness, Emotional Stability, and Extraversion).


Changing Attitudes, Behaviors, and Skills

Behavior modeling training (BMT), based on Bandura’s social learning theory, has become a widely used psychologically-based approach to training in work environments.

A recent meta-analysis by Taylor et al. (2005) summarizing 117 studies found substantial effects of BMT on a variety of training outcomes.

Outcome d

Attitudes .56

Behaviors .33

Declarative knowledge .69

Procedural knowledge-skills 1.21


“It was about here, wasn’t it, Ed, when you came on board as sales manager?”

Harvard Business Review. March 2007. p. 90


Complexity of Personality & Career

• Different personality characteristics are important for different jobs (whether across or within fields).– Conscientiousness appears to be important for all

jobs.– Extraversion is relevant to some jobs (e.g., sales)

but not others (e.g., computer programmers).


Complexity of Personality & CareerAcross Fields

• PositionComputer Programmers

• O-NET Job ZoneFour: Considerable Preparation Needed

• Education Most occupations in this zone require a four-year bachelor's degree, but some do not.

http://online.onetcenter.org/link/summary/15-1021.00

• PositionSales Representatives

• O-NET Job ZoneFour: Considerable Preparation Needed

• EducationMost occupations in this zone require a four-year bachelor's degree, but some do not.

http://online.onetcenter.org/link/summary/41-4011.00


Complexity of Personality & Career: Comparing Scores Across Fields

0 10 20 30 40 50 60 70 80

Striving

Stability

Sociability

Savvy

Order

Optimism

Influence

Goodwill

Discipline

Creativity

Cooperation

Carefulness

ComputerProgrammers

SalesRepresentatives

Computer Programmers and Sales Representatives

0 10 20 30 40 50 60 70 80

Striving

Stability

Sociability

Savvy

Order

Optimism

Influence

Goodwill

Discipline

Creativity

Cooperation

Carefulness

Scale Scores


Complexity of Personality & CareerModeration

Postlethwaite et al. (2009) found that conscientiousness was a stronger predictor of safety behavior for individuals with lower levels of cognitive ability.


Complexity of Personality & CareerCurvilinear Relationships

Le et al. (in press) found curvilinear relationships between Emotional Stability and job performance (task, OCB, CWB), as well as between Conscientiousness and task performance.


Questions?

Correspondence regarding this presentation should be addressed to:

Steve RobbinsVice President, Research

ACT, [email protected]


ReferencesAllen, J. & Robbins, S. (2010). Effects of interest–major congruence, motivation, and

academic performance on timely degree attainment. Journal of Counseling Psychology, 57, 23–35.

Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, 87-100.

Casillas, A., Robbins, S. B., McKinniss, T., Postlethwaite, B., & Oh, I.S. (2009). Using narrow facets of integrity to predict safety: A test validation study. International Journal of Selection and Assessment, 17, 119-125.

Connelly, B.S. & Ones, D.S. (2010). An Other Perspective on Personality: Meta-Analytic Integration of Observers’ Accuracy and Predictive Validity. Psychological Bulletin, 136, 1092–1122.

Le, H., Oh, I.-S., Robbins, S. B., Ilies, R., Holland, E., & Westrick, P. (in press). Too Much of a Good Thing: Curvilinear Relationships Between Personality Traits and Job Performance. Journal of Applied Psychology.

McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335-354.

Neumann, G., Olitsky, N., & Robbins, S. (2009). Job congruence, academic achievement, and earnings. Labour Economics, 16, 503-509.

Postlethwaite, B., Robbins, S., Rickerson, J., & McKinniss, T. (2009). The moderation of conscientiousness by cognitive ability when predicting workplace safety behavior. Personality and Individual Differences, 47, 711-716.


References (cont.)Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do

psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

Robbins, S. B., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology, 94, 1163-1184.

Rotundo, M. & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66–80.

Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, 573-605.

Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of Cognitive Ability and Personality in Job Performance and Training Performance: Some Surprising New Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6.

Taylor, Paul J., Russ-Eft, D.F., & Chan, D.W.L. (2005). A Meta-Analytic Review of Behavior Modeling Training. Journal of Applied Psychology, 90, 692-709.

Tracey, T.J.G., & Robbins, S.B. (2006). The interest-major congruence and college success relation: A longitudinal study. Journal of Vocational Behavior, 69, 64-89.

© 2010 by ACT, Inc. All rights reserved. 1 Creating an Aligned Assessment System to Promote Work Training Readiness and Career Success Steve Robbins Vice.

Documents

act programs

work success

research act

work sri

pyramid slide

world of work

jobs slide

work training readiness