1 Outline 1. Ethics vs. Law 2. Ethical use of tests – issues: Competence Informed Consent Knowledge of Results Confidentiality Test security.

1

Outline

1. Ethics vs. Law2. Ethical use of tests – issues:

Competence Informed Consent Knowledge of Results Confidentiality Test security Divided Loyalties

2

Outline

4. Professional Issues Theoretical issues Test adequacy Actuarial vs. clinical prediction

5. Social Issues Dehumanization Access to benefits of psychological testing

3

Outline

6. Two examples of bad tests Handwriting analysis TOEFL

7. The CPA Code of Ethics (synopsis) Respect for the dignity of persons Responsible caring Integrity in relationships Responsibility to society

4

Ethics vs. Law

Ethics: what one should or should not do, according to principles or norms of conduct

Ethics codes are not produced by democratically-elected legislatures

Enforcement mechanisms are usually informal, may be complex, even unconscious

5

Ethics vs. Law

Law: what one must or must not do, according to legal dictates

Laws are created by democratically-elected legislatures in democracies such as Canada

Laws come with explicit penalties for infractions and a formal enforcement system

6

Ethical use of tests – issues:

1. Competence2. Informed Consent3. Knowledge of Results4. Confidentiality5. Test security6. Divided Loyalties

7

1. Competence

develop competence in assessment concepts and methods

recognize boundaries of your competence

competence is very important if test result affects someone’s life

8

2. Informed Consent

Generally, the client/subject must voluntarily consent

psychologist must inform client about nature and purpose of assessment in understandable language

9Informed consent - exceptions

Sometimes, it is acceptable to test without getting consent – e.g., you have given implied consent to be tested by registering in this course, but I have never sought your explicit consent

Sometimes it is necessary to test without getting consent, even when consent is explicitly refused by person to be tested – e.g., when mandated by law

10

3. Knowledge of Results

must fully disclose test results in understandable language

but do not do so in a way that reveals the content of test items, because that content is proprietary information (e.g., somebody owns it – it’s not yours to reveal)

11Constraints on client access to results PIPEDA Principal

4.9: Upon request, a

person must be informed of the existence, use, and disclosure of his or her personal information and must be given access if requested.

Section 9.3b says that access can be denied if it would mean revealing confidential commercial information (unless latter is severable).

12Constraints on client access to results Ontario Personal

Health Information Act: access rights do not apply to a record that contains raw data from standardized psychological tests or assessments, unless those data are reasonably severable.

CPA and leading Canadian test publishers say that Test Materials are not “personal information” so should not be released.

13

4. Confidentiality

Test results are confidential information

Release of results should only be made to another qualified professional after client’s consent

14

5. Test Security

Test materials must be kept secure

Test items are not revealed except in training programs and when mandated by law, to protect test integrity

Test items are private property

156. Automated Scoring/Interpretation Systems

Psychologist is still responsible for proper interpretation of test results

Psychologists are professionals – they take personal responsibility for their work

16

7. Divided loyalties

Who is the client? The person being

tested, or the institution you work for?

What if these parties have conflicting interests?

How do you maintain test security but also explain an adverse decision?

17

Divided loyalties

Suppose you are asked to test a child who might have disorder X or Y

Suppose X is very common and so the government provides resources to help schools deal with it

Suppose Y is not common and no resources are available in school to deal with it

If your testing suggests Y, will you report Y – or X, so the child gets some sort of help?

18

Professional Issues

1. Theoretical issues2. Test adequacy3. Actuarial vs. clinical prediction

19

1. Theoretical Issues

Are you measuring a stable characteristic of the person being tested?

If so, differences in scores over time reflect measurement error or subject variables such as fatigue

20

1. Theoretical Issues

Are you measuring a stable characteristic of the person being tested?

In fact, we have little evidence that this is true, so what is the value of your test result – will it still be true next year?

Especially problematic for personality tests

21

1. Theoretical issues

Are you measuring a stable characteristic of the person being tested?

Suppose you test someone as part of a hiring process.

If the test result says that person does not have the characteristic you’re looking for, does that mean they could never acquire that characteristic?

22

2. Adequacy of tests

How do we measure test adequacy?

(We’re supposed to be good at measuring things!)

What should go into an assessment of test adequacy?

23

2. Adequacy of tests

So far, in considering the relative merits of various tests, we have asked whether they are psychometrically adequate

We haven’t asked: is the best test available good enough?

Society uses that standard when testing becomes a legal issue

But the answer is not psychometric

243. Actuarial vs. clinical judgment U.S. Supreme

Court Justice Potter Stewart said in a 1964 case that he didn’t know how to define pornography, but he knew it when he saw it.

Is it possible for us to make good judgments on a question when we cannot articulate the basis for the judgment?

253. Actuarial vs. clinical judgment Actuarial judgment

occurs when we feed test scores into statistical formulas to diagnose a psychological condition or predict future performance.

Clinical judgment occurs when we have a trained psychologist interpret test scores to diagnose a psychological condition or predict future performance.

263. Actuarial vs. clinical judgment In actuarial

judgment, we cannot make accurate predictions tailored to individuals

Instead, our conclusion will be the same for every person with a given set of test scores

In clinical judgment, the claim is that you can determine “what caused what” in an individual’s person’s life (Dawes, 1994)

But clinical judgment does not improve with experience

273. Actuarial vs. clinical prediction Meehl’s work in

the 1950s: feed test scores into statistical formula to predict future performance – or have trained psychologist interpret scores

statistical formula does a better job of prediction

still, someone has to be legally responsible – that has to be a trained professional

28

Social issues

1. Dehumanization2. Access to benefits of psychological

testing

29

1. Dehumanization

Does computerized testing and analysis of test results create a danger of minimizing human uniqueness?

humans are very complex – which allows us to be individuals, different from each other

but testing and interpretation generalize

30

1. Dehumanization

Is generalization a problem?

Computer-scored and administered psychological tests might determine important aspects of your life

If you think this is a problem, why do you think so? What evidence do you have that this hurts people?

What alternative approach would you use?

312. Access to psychological testing We’ve considered

possibility of being hurt by tests – but what about possible benefits of tests? Who gets those benefits?

WAIS-III kit costs $775 per person – for tester to buy the kit. Cost must be passed on to someone. Who should that be?

322. Access to psychological testing Psychological

services are not covered by OHIP; Psychiatric services are

Psychiatrists have opposed extending OHIP coverage to psychological testing

Is that just turf protection?

There is rivalry between the professions – but we could make a stronger case for billing OHIP if we used only actuarial judgment

33

Two examples of bad tests

1. Graphology (handwriting analysis)2. TOEFL

34Graphology (handwriting analysis) Used in hiring

decisions in business

Used to make assessments of personality type and to predict on-the-job performance

Simner & Goffin (2003): very little evidence of validity for graphology

studies using non-autobiographical writing are the most useful and show the least evidence of validity.

35Graphology (handwriting analysis) Dean (1992) meta-

analysis of handwriting studies showed an average correlation of .16 between graphological judgment and job performance, using autobiographical writing.

Correlation drops to .09 when using non-autobiographical writing.

36Graphology (handwriting analysis) Simner & Goffin:

both general mental ability testing and structured employment interviews correlate around .50 with job performance. When used together, the two correlate .63 with job performance.

Thus, GMA testing and interviewing are much more likely to lead to sensible hiring decisions.

37Test of English as a Foreign Language

Widely used to select among students who are not native speakers of English for admission to Canadian universities.

TOEFL manual presents a distorted picture of validity studies

Simner (1998) presents CPA’s recommendation – don’t use TOEFL for university admission decisions

38

TOEFL – Criterion Validity

TOEFL manual leaves out studies that show low to moderate correlations between TOEFL scores and teachers’ ratings of English language proficiency (.34 to .55)

TOEFL manual leaves out studies that show low to moderate correlations between TOEFL scores and oral interviews (.30 to .60)

39

TOEFL – Criterion validity

Generally, these studies assess concurrent validity, not more relevant predictive validity

Simner: some criterion validity tests for TOEFL involve the Michigan TELP, very similar to TOEFL – so this is really parallel forms reliability, not validity evidence

40

TOEFL – Construct validity

ETS: Native speakers of English do much better than non-natives on TOEFL.

Simner: in these studies, native speakers are a more highly-selected group.

Non-natives were unselected examinees from a range of backgrounds.

Natives were selected from among entering freshmen

Test anxiety differences

41

TOEFL – Construct validity

Unpublished ETS study: 5000 English speakers had an average score of 590, below the 600 cutoff used for non-natives at many Canadian universities.

If native English speakers can succeed with scores < 600 why can’t non-natives also succeed with scores below 600?

42Test of English as a Foreign Language Simner: Since 1995,

10 Ontario universities have raised their TOEFL cutoff from around 550 to 600. This means that only the top 10% of examinees are accepted, instead of top 30%.

Taylor-Russell tables: r = .20, now 10% accepted, before 30%

Net gain in percentage likely to be successful is 4%, at a cost of rejecting an additional 20%.

43TOEFL: Simner’s recommendations make admissions

decisions on the basis of all relevant information, not just TOEFL scores

do not use rigid cutoff scores

consider the kinds and levels of English proficiency required in different fields and levels of study

consider resources available for improving English language skills

44

CPA Code of Ethics

The following slides show the four basic principles advanced in the CPA Code of Ethics. All of the material on the next 15 slides is quoted directly from that Code.

There is significantly more information on each principle at this web address: http://www.cpa.ca/cpasite/userfiles/Documents/Canadian%20Code%20of%20Ethics%20for%20Psycho.pdf

(Note: scroll down if the page looks blank)

45CPA Principle I: Respect for the Dignity of Persons.

In the course of their work as scientists, practitioners, or scientist-practitioners, psychologists come into contact with many different individuals and groups, including: research participants; clients seeking help with individual, family, organizational, industrial, or community issues; students; trainees; supervisees; employees; business partners; business competitors; colleagues; employers; third party payers; and, the general public…

46CPA Principle I: Respect for the Dignity of Persons.

In these contacts, psychologists accept as fundamental the principle of respect for the dignity of persons; that is, the belief that each person should be treated primarily as a person or an end in him/herself, not as an object or a means to an end…

47CPA Principle II: Responsible Caring. A basic ethical expectation of any discipline is that its activities will benefit members of society or, at least, do no harm. Therefore, psychologists demonstrate an active concern for the welfare of any individual, family, group, or community with whom they relate in their role as psychologists. This concern includes both those directly involved and those indirectly involved in their activities.

48CPA Principle II: Responsible Caring. However, as with Principle I, psychologists’ greatest responsibility is to protect the welfare of those in the most vulnerable position. Normally, persons directly involved in their activities (e.g., research participants, clients, students) are in such a position. Psychologists’ responsibility to those indirectly involved (e.g., employers, third party payers, the general public) normally is secondary.

49CPA Principle III: Integrity in Relationships.

Psychologists are expected to demonstrate the highest integrity in all of their relationships. However, in rare circumstances, values such as openness and straightforwardness might need to be subordinated to the values contained in the Principles of Respect for the Dignity of Persons and Responsible Caring.

50CPA Principle III: Integrity in Relationships. The relationships formed by psychologists in the course of their work embody explicit and implicit mutual expectations of integrity that are vital to the advancement of scientific knowledge and to the maintenance of public confidence in the discipline of psychology. These expectations include: accuracy and honesty; straightforwardness and openness; the maximization of objectivity and minimization of bias; and, avoidance of conflicts of interest. Psychologists have a responsibility to meet these expectations and to encourage reciprocity.

51CPA Principle III: Integrity in Relationships.

In addition to accuracy, honesty, and the obvious prohibitions of fraud or misrepresentation, meeting expectations of integrity is enhanced by self-knowledge and the use of critical analysis.

52CPA Principle IV: Responsibility to Society.

Psychology functions as a discipline within the context of human society. Psychologists, both in their work and as private citizens, have responsibilities to the societies in which they live and work, such as the neighborhood or city, and to the welfare of all human beings in those societies.

53CPA Principle IV: Responsibility to Society.

Two of the legitimate expectations of psychology as a science and a profession are that it will increase knowledge and that it will conduct its affairs in such ways that it will promote the welfare of all human beings.

54CPA Principle IV: Responsibility to Society.

Freedom of enquiry and debate (including scientific and academic freedom) is a foundation of psychological education, science, and practice. In the context of society, the above expectations imply that psychologists will exercise this freedom through the use of activities and methods that are consistent with ethical requirements.

55CPA Principle IV: Responsibility to Society.

The above expectations also imply that psychologists will do whatever they can to ensure that psychological knowledge, when used in the development of social structures and policies, will be used for beneficial purposes, and that the discipline’s own structures and policies will support those beneficial purposes. Within the context of this document, social structures and policies that have beneficial purposes are defined as those that more readily support and reflect respect for the dignity of persons, responsible caring, integrity in relationships, and responsibility to society.

56CPA Principle IV: Responsibility to Society. If psychological knowledge or structures are used

against these purposes, psychologists have an ethical responsibility to try to draw attention to and correct the misuse. Although this is a collective responsibility, those psychologists having direct involvement in the structures of the discipline, in social development, or in the theoretical or research data base that is being used (e.g., through research, expert testimony, or policy advice) have the greatest responsibility to act. Other psychologists must decide for themselves the most appropriate and beneficial use of their time and talents to help meet this collective responsibility.

57CPA Principle IV: Responsibility to Society.

When a person’s welfare appears to conflict with benefits to society, it is often possible to find ways of working for the benefit of society that do not violate respect and responsible caring for the person. However, if this is not possible, the dignity and well-being of a person should not be sacrificed to a vision of the greater good of society, and greater weight must be given to respect and responsible caring for the person.

58

CPA’s Caveat…

Even with the above ordering of the principles, psychologists will be faced with ethical dilemmas that are difficult to resolve. In these circumstances, psychologists are expected to engage in an ethical decision-making process that is explicit enough to bear public scrutiny. In some cases, resolution might be a matter of personal conscience.

59

CPA’s Caveat…

However, decisions of personal conscience are also expected to be the result of a decision-making process that is based on a reasonably coherent set of ethical principles and that can bear public scrutiny. If the psychologist can demonstrate that every reasonable effort was made to apply the ethical principles of this Code and resolution of the conflict has had to depend on the personal conscience of the psychologist, such a psychologist would be deemed to have followed this Code.