The Testing Environment and Threats to Validity Considered

The Testing Environmentand Threats to Validity Considered“. . . the play's the thing wherein I'll catch the conscience of the King.” –Hamlet, Prince of Denmark

Or

Catching your own conscience; reflecting on your testing approaches to improve validity. You don’t want extraneous factors to impact scores.

Topics on Administering & Analyzing TestsWhen using them for important summative assessments

Test administration

Item analysis

Cheating

Fairness – Real and Perceived

Important that students perceive value in the assessments Motivated to do especially well on tests related to

recorded grades, report cards passing or failing decisions

Three attributes of perceived fairness Teach what you test.

Broadly meant; not just what you said in class. Give adequate notice.

Pop quizzes are okay as long as you let students know they are a possibility.

Be fair in grading. Consistent and reasonable. Fairness does not mean easy. Many students appreciate high

standards and a challenging test.

Suggestions, prior to test day, for . . .

Preparing Test Materials

1. Give clear directions – what, if any, supplemental materials are permitted (scratch paper, dictionaries, calculators).

2. Use a simple format – use “white space” and logical page breaks.3. Allow lead time for printing up the exams – don’t race into office

and expect your test to be done right away.4. Spot check – pages complete, in correct order, not inverted.5. Take one test and answer it – non-fatal mistakes you can correct

with oral directions; fatal means re-printing (don’t postpone).6. Worry about security – keep secure; may wish to number copies.7. Comparable forms for multi-section courses – students in later

classes will ask students in earlier sections about the test.

Suggestions, prior to test day, for . . .

Preparing Yourself and the Students

Assuming you have provided excellent content instruction, recommended actions include:

Give adequate notice to the students. This includes the type(s) of items that will be found on the test. Schedule the test so that you have time built-in so that when the test is completed you have sufficient time for scoring of the test. Consider likely circumstances.

Are there any special equipment needs? Is there anything scheduled that could be disruptive (fire drill, pep rally)? Should you pre-guide students as to what to do if there is unanticipated commotion in the room or building? (What to do with test documents.)

Make advance preparation for accommodations.

Suggestions, on the day of the test, for . . .

Immediate Preparation of Students

1. Think about classroom seating arrangement – would rearranging be offensive; how would you rearrange (e.g. every other seat).2. Determine how materials (e.g., test, answer sheet) will be passed out (yourself; passed front to back) - check that everyone has all materials.3. Establish control – students often talk when materials are passed about, decide when to declare conversation time over.4. Direct students to place names on answer sheet – and on test itself if this is important to you for security reasons. This may suggest to student’s that you look at any marks they make on the test itself.5. Review test directions – some students skip the written directions while others agonize too long over them (should be no surprises).6. Provide directions as to what the students do when they finish their test (e.g., turn in, wait quietly, study).7. Give quick words of encouragement – humor may work or may backfire.

Suggestions, once the test starts, for . . .

Managing the Testing Environment

1. Monitor the testing situation – don’t use this time as personal time to work on other classes; circulate around the room, help students stay on task; see if it appears someone is not correctly following the directions (make sure you tell students you will be doing this). When you do sit, sit behind students out of their vision.

2. Answer questions with fairness in mind – don’t give hints to individual students, clarification only. If it appears what you might be telling them is important, consider telling the entire class.

3. Apprise students of remaining time – if timing is important or if there is no clock visible.

Suggestions, once the test finishes, for . . .

Scoring and Returning Test Results

1.Many teachers dislike this part; it is time consuming. The utility of having provided yourself with a previously prepared scoring key and time to score is key for success in this area. I suggest you score exams yourself.2.Return results promptly; most often the sooner the better. Getting results back quickly is good for the mental health of students, and yourself!3.Don’t announce individual scores (although you can publicly acknowledge a student’s outstanding performance); don’t post grades (legal issues). You may make general narrative statements (either positive or negative) . . . I prefer positive (for example, the class did well). 4.Reiterate the scoring system you used. Your scoring approach should not be a surprise to the students but, by restating it, you demonstrate you tested them exactly as you said you would.5.Decide how you want to deal with student questions. Will you do over all items, some items? What kinds of questions will you accept, how long will the questions go on? Will you change grades based on the discussions?

Suggestions, once the test review finishes, for . . .

Retaining Test Results

1. Recover materials as desired – do this in such a way that students do not see the scores of others. Do you want all parts back? Can students keep old tests for studying? If collected, what will you do with them?

2. Record scores – do this after you have gone over test with students. Use grade book (hardcopy or electronic).

3. Make notes - on changes you would like to make in the assessment system when you teach this in the future. Do this now while it is fresh in your mind.

Item Analysis: ExampleNote: the values in the table are percentagesof correct responses, High/Low/Total Group

Classroom Item Analysis, “So What” . . .

Item analysis is important because a poor or bad item may add to the unreliability of the test. Why? Because if your purpose in the test was to distinguish between those students who know the material and those who do not, a bad item fails to work toward this end (e.g., items answered correctly by all, or items that no one answers correctly.)

A rough "rule-of-thumb" is that if the item difficulty is more than .75, it is an easy item; if the difficulty is below .25, it is a difficult item.

It is my feeling that items with difficulties less than 20% or more than 80% (especially if they fail they discriminate) deserve your attention. Such items should either be revised or replaced. Perhaps you have a non-functioning distracter. Perhaps your question misleads the best students. An exception might be at the beginning of a test where easier items (90% or higher) may be desirable for initial confidence building.

Taking item analysis to the next level . . . Item Analysis Programs

If you get into item analysis on a large scale (many students/items) consider: Independent vender computer software “Test Packages” on test construction which often include item analysis features. These may not be good long term since the companies that produced them come and go (no tech support). The “Test Development Software Package” provided by the publisher of the textbook you are using.Test item analysis using the “Microsoft Excel” spreadsheet program.Your school district may have a centralized test scoring service (usually limited to selected response types of exams). This likely means that you would need to require that your students use a bubble sheet because it would be scanned.These computer programs may use more complex calculations for indices (aka, indexes) related to item difficulty or discrimination but the concept will be the same. For example, item discrimination might be presented in terms of a correlation coefficient (with a Greek letter or an unusual sounding name, of course). Remember how to interpret a correlation coefficient? The scale is “-1 to +1” so you would interpret it much like the “D” statistic.

Classroom Item Analysis . . .Examining the quality of individual items; “p” and “D” statistics

When the test is completed, we have two basic questions: Was the difficulty of the item appropriate or was it too hard or too

easy? I can compute the item difficulty index by dividing the number of correct answers by total answers:

p = percent correct The higher the value; the easier the item

If the item was difficult, was this difficulty okay? Does it discriminate between those who know and those who don’t? To answer this we compute the item discrimination index. In this case we look at each item in terms of did the students who do the best on this test tend to get it right more so than the students who did the worst. The book divides the class in half (H & L); I tend to like thirds (H, M, L). Compare the percent correct between H & L groups

D = H% - L% The higher the value, the more the item discriminates . . . notice

it go negative or positive. What’s a “+” or “-” mean?

CheatingGraphic from “A Taxonomy of Text and Exam Fraud” by Dennis Maynes, 2008

Say it ain’t so, Joe . . .

Cheating and Academic Integrity Most teachers would rather not discuss this; perhaps a form of denial. Prevalence

All levels of education Student View “No big deal – everybody does it.”

Practices Plagiarizing, especially Internet-based

Cut and paste without attribution Other projects completed outside class

Assistance from parents, siblings, family friends, fellow students range from incidental to complete authorship

Cheating on exams, tests (crib notes, look at another’s test, pass notes) Let’s look at some cheating examples; a major source was

Gary K. Clabaugh & Edward G. Rozycki, Preventing Cheating and Plagiarism, 2nd Edition (2003) Oreland, PA: NewFoundations Press

Taking note of “crib” notes . . .So called because of how one “cradles” the small sheets.

OLD SCHOOL: Get into the classroom prior to test and write crib notes on the desktop or the

back rest of seat in front. Variation, tape note folder on underside of desktop. Make sure the crib notes can be easily destroyed. Example, chew gum during

exam; if a teacher becomes suspicious eat the piece of paper. Do NOT write on yourself, it is impossible to hide if a teacher questions you.

Print off crib notes in size 6 font about 4” wide by ½” long. Acquire transparent pen (BIC recommended). Roll the paper around the tube of ink and slide it in.

Remove label from a clear beverage, such as Snapple. Put notes on the back of the label, then paste it back in place using transparent glue. During the exam take slow "thoughtful" swigs out of the bottle.

Wear laced up shoes on the day of the exam (with laces remove). Attach your crib notes behind each tongue of the shoe.

NEW SCHOOL: Crib notes on an electronic organizer: a mini tape recorder can be fitted with a

tiny earphone that can be hidden by combing hair over the ear; programmable calculators; use of a cell phone in a lavatory.

Other Cheating Gambits . . .

Signal answers among group. Keep an eye out for the rearrangement of items on desktops, coded body movements, and hand signals.

A student who has postponed a test or who takes it in a later section, enlists a confederate to smuggle an extra copy of the exam out of the exam room.

Fraternity approach. Smuggle a copy of the test from the room at the end of the test, note the answer sequence when the test is reviewed then keep both the test and the answers on file.

Student sits for the exam, pretends to be taking the test, but completes nothing. Then they turn in the blank exam and makes sure the instructor notices it has been returned. When the tests are graded and the student gets a zero because they didn't take the test, he or she complains bitterly that their test has been lost.

When tests are returned students either alter their answers from wrong to right or fill in answers they deliberately left blank. Then they complained that their test was marked incorrectly.

Students fake injury or illness to postpone tests for which they are unprepared. Students show up late, feign surprise and claim that they misunderstood when

the test was to be given.

Cheating Suggestions for Prevention

1. Create positive classroom atmosphere - reasonable standards, sensible workloads, keep content interesting, keep upbeat attitude yourself.

2. Talk to students about cheating – define cheating behavior, identify consequences; immediate/long term. Is cheating rewarded in society?

3. Reduce anxiety and uncertainty - be clear and informative about upcoming tests, what will be tested, how.

4. Be vigilant – consider changing seating, walk among the students during testing.

5. When cheating occurs, confront it – teachers’ disregard of cheating is one reason students say they feel so free to do it.

6. Determine if your school has a policy; support it - if the policy is nonfunctional, then help develop new policy.

Practical Advice

1. Use summary checklists regarding each aspect of test administration to remind you of considerations at each phase you value.

2. Learn to do simple item analysis, use occasionally.

3. Take cheating seriously.