Computer Applications in Testing and Assessment James P. Sampson, Jr. Florida State University Copyright 2002 by James P. Sampson, Jr., All Rights Reserved.

Computer Applications in Testing and Assessment

James P. Sampson, Jr.

Florida State University

Copyright 2002 by James P. Sampson, Jr., All Rights Reserved

2

Overview of Topics

• Computer-Assisted vs. Computerized

• Rationales for Computer-Assisted Testing

• Potential Benefits

• Potential Problems

• Future Use

3

Computer-Assisted vs. Computerized

• Both describe use of computer technology in assessment process

• Used with nearly the same frequency

• Using the term computerized may cause problems and foster misunderstanding

4

Computerized

• Implies that computers replace humans previously needed to complete a task

• May lead to practitioner under use or misuse

• An inaccurate depiction of computer technology in testing

5

Computerized

• Under Use– Uninformed rejection of computer

technology

– View computers as performing inappropriate functions

– Believe computers lead to dehumanization

6

Computerized

• Misuse

– View computers as effective in performing functions not intended by developer

– Use of computers to compensate fora lack of professional skill

7

Computer-Assisted

• Computer fulfills a support role

• Improves, not replaces the human element

• More likely to avoid under use or misuse

8

Rationales for CAT

1. Focus on assigning humans and computers to tasks best suited to the capabilities of each

2. Develop previously unavailable techniques to improve the quality of tests and assessments

9

Potential Benefits of CAT

• Overview of topics:

– Enhanced validity and reliability

– Enhanced client integration of assessment data in counseling

– Enhanced staff efficiency andcost-effectiveness

10

Enhanced Validity and Reliability


– Test Administration and Scoring

– Administration of Interviews and Checklists

– Computer-Based Test Interpretation

11

Test Administration and Scoring

• Greater degree of standardization

• More accurate results for people with disabilities

• Availability of adaptive testing

• Collection of ancillary data (e.g., response manner and specific response recording)

12






13

Administration of Interviews and Checklists

• More comprehensive topic coverage

• More consistent topic coverage

• Less evaluative tension from repeated questioning by the practitioner

• Results less affected by time of day

• More honest client response

14






15

Computer-Based Test Interpretation (CBTI)

• Assists in the interpretation of test data

• Provides expanded & consistent knowledge base

• Includes research data andthe clinical experience of other clinicians

• Standardizes computer functioning

• Provides consistency

16

Types of CBTI

• Descriptive interpretations

• Clinician-modeled interpretations (renowned clinician type)

• Clinician-modeled interpretations (statistical model type)

• Clinical actuarial interpretations

17

Descriptive CBTI

• Descriptive sentences related to specific scales

• Scales interpreted independent of each other

• Typically cryptic in nature

18

Clinician-Modeled CBTI (Renowned Clinician Type)

• Replicates the interpretive judgment of a single clinician

• Based on the clinician with the most experience with the test

• Provides consistent results

• Mistakes also consistent

19

Clinician-Modeled CBTI(Statistical Model Type)

• Replicates interpretive judgmentsof a group of practitioners

• Includes a group comprised of experienced clinicians

• Statistically valid model

• Incorporates the best consensus possible among experts

20

Clinical Actuarial CBTI

• Provides narrative descriptions and clinical hypotheses

• Based on clinical research findings for particular score patterns

21

Potential Benefits of CBTI

• More comprehensive and objective

• Not subject to interpreter bias orextraneous circumstances

• Provides rapid data processing

• Ready for use in early counseling sessions

• Helps practitioners organize and systematicallyaccess extensive data bases

• Easily updated when new research is available

22

Potential Benefits of CBTI

• CBTI serves in a consulting role

• Practitioner integrates data from variety of sources to create assessment reports

• Computerized interpretive reports only used as adjuncts, not as the final word

23






24

Enhanced Client Integration of Assessment Data in Counseling

• Computer applications used as one aspect of the therapeutic process

• CBTI reports in clinical settings are typically not given to most clients

• Interpretive reports are typically provided directly to clients in career counseling– provides basic understanding of results – outlines the decision-making process– suggests follow-up resources

25

Enhanced Client Integration of Assessment Data in Counseling

• Generalized test interpretations– video-based technology

26






27

Enhanced Staff Efficiency and Cost-Effectiveness

• Computer-assisted testing is more cost-effective

• Saves testing time

• Improves time utilization of support staff

• Gives practitioners more time to prepare test interpretations, write test reports, and work directly with clients on more complex issues

28

Potential Problems Associated with Computer Applications

• Overview of Topics– Threats to Validity and Reliability

• administration and scoring• test interpretation• ethical issues• ethical standards

– Ineffective Implementation

29

Threats toValidity and Reliability

• Administration and Scoring

– Norms used from paper-and-pencil tests

– Blanket assumptions of equivalency

– Scoring errors

– Perceived inherent credibility - infallibility

– Client misunderstandings of instructions

30


• Test Interpretation– Different reports from different software

developers

– Differences in CBTI reports and conclusion reached by clinical interviewer

– Computer may be perceived addinga level of scientific precision that does not exist

31


• Misuse of CBTI

– Qualified practitioners may misuse CBTI as a result of being overworked

– Unqualified practitioners may use to offsettheir lack of training and experience

– Practitioner dependence on CBTI

32


• Appropriate Use of CBTI– Computer serves as a consulting function– Practitioner has final responsibility for

communicating test results– Qualifications necessary for appropriate use– Prescreening– Not all clients can provide valid item

responses– Follow ethical standards

33

Potential Problems Associated with Computer Applications

• Overview of Topics– Threats to Validity and Reliability

• administration and scoring• test interpretation• ethical issues• ethical standards

– Ineffective Implementation

34

Ineffective Implementation

• Staff Resistance – Lack of staff participation in decision-making – Lack of experience with computers– Poor quality program documentation– time and effort required

• Poor staff support results in reduced acceptance

35

Effective Implementation

• Improves the likelihood of success– Careful planning– Group input in decision making– Reasonable rate of change

• Effective implementation involves interpersonal dynamics more than technology

36

The Future

• Hardware Development– Human voice– Video-based stimuli– Virtual reality

• Possible Problems– High cost– Extraneous auditory and visual data may

distract test taker– Stereotypes

37

The Future

• Software Development

– Expert and decision support systems

– Expert systems will be added to existingCBTI systems

– Expert systems can be used as a powerful training resource

38

Conclusion

• Do computer applications actually improve testing and assessment in counseling?

– Depends more on practitioner attitudes and skills than future advances in technology

– Without effective use, new technology is meaningless

Computer Applications in Testing and Assessment James P. Sampson, Jr. Florida State University Copyright 2002 by James P. Sampson, Jr., All Rights Reserved.

Documents

testing slide

misuse slide

costeffectiveness slide

dehumanization slide

consistency slide

assessments slide

test administration

overview of topics computer