Sample Cognitive Lab Interview Protocol - ETS Home · PDF file3 COGNITIVE LAB INTERVIEW PROTOCOL SEATING INSTRUCTIONS Follow the school’s procedures to escort one student from his

1

Sample Cognitive Interview Protocol

Teresa King & Cara Cahalan Laitusis Educational Testing Service

2008

Please reference as:

King, T.C. & Laitusis, C.C. (2008). Sample Cognitive Interview Protocol. Princeton, NJ:

Educational Testing Service.

2

COGNITIVE LAB INTERVIEW

PROTOCOL

ID Information

Student ID: Place Label Here

Grade/Form: 4A Test Adm Initials: Date: School:

3

COGNITIVE LAB INTERVIEW PROTOCOL SEATING INSTRUCTIONS

Follow the school’s procedures to escort one student from his or her classroom to an empty testing room or corner of a quiet room. Make sure that you and the student are seated so that you face one another directly across the table or desk if possible. Set the interview protocol and the stopwatch in front of you and the closed student test booklet in front of the student. Place the digital recorder on the table near the student.

INTRODUCTION Hello! My name is (YOUR NAME). I work at Educational Testing Service. Thank you so much for your help. Today you will be participating in a special study on reading. Its goal is to find better ways to test reading for all students. We need your help creating a reading test. If at any point you decide you do not want to continue that is your choice and you are free to stop and go back to class. Do you have any questions before I begin the instructions?

Create small talk…ask a couple friendly questions such as the following:

Do you like to read?

If yes, say

Oh, good, then I think you will like what we are doing today.

4

If no, say

Okay, well what is your favorite subject?

Appropriately respond. (Also make a note in the test administrator answer sheet whether student likes to read.) Then move go back to the directions. Say,

You will not be asked to write your name on any of the work you do, and no one in the school will see your answers. The information you provide will be combined with information from other students all over New Jersey. Because the study is so important to schools and students, I want to thank you ahead of time for all the hard work you are about to put into your answers. I know that you will do the best that you can. AUDIO RECORDING/SOUND CHECK We are going to record today’s test so I remember what you say. First we’re going to test out this audio recorder and the microphone.

Show the student how to clip the external microphone to his/her shirt. Make a test recording by telling the student:

When I say GO, I want you to say, “Today is Friday” in a clear voice.

Press the RECORD button on the audio recorder and say “YOU MAY BEGIN” to prompt the student.

5

Be sure to press RECORD before you tell the student to begin so your voice is recorded too. If necessary, remind the student to say the sentence (“Today is Friday.”) Once the student has said the sentence, press PLAY to listen to the recording.

If you cannot hear the student clearly on the recorder, make sure the microphone is plugged into the proper jack on the recorder and repeat the test recording process until you get a clear recording, then continue with the administration:

If you can hear the student clearly, continue with the

administration: Press the RECORD button to begin recording and continue with the administration. Say clearly into the recorder the Student ID number, today’s date, and your initials.

Good job. Now we’ll begin the instructions.

REMINDER: Do not turn off the recorder. It should still be recording.

6

DIRECTIONS

Think aloud hints to administrator

We’re interested in capturing all the mental processes that the student engages while completing this reading comprehension test. To do this your goal is to have the student speak aloud all his or her thoughts while solving the reading comprehension questions and by asking follow up questions for each item and passage. There are several things to keep in mind to ensure the data collected is as complete as possible: • If a student is continually providing short responses or not answering, use "continuers" to encourage the student to be more descriptive. The trick is to get them to continue without putting words in their mouth. Make sure not to ask questions that lead a student’s response. You have to be as objective and non-bias as possible. The way you get them to keep talking is by offering a gentle verbal nudge like this: “What are you thinking now?” “Any other thoughts?” “Tell me how you came to pick that answer.” • Use your best judgment. If a student is responsive but is not easily explaining his or her reasoning, gently probe the student without putting any bias in their response. • Focus on the task at hand, the particular item or passage. Do not try to ask a student a question in general terms. • Ask the student to explain why answer options are not right if s/he does not provide that information. • It’s important to capture the frustration aspect of taking the test (and performing the think aloud exercise) if it seems a student is hung up with something.

7

We are going to do 3 activities today. The first activity will take most of our time and we will first practice how to do it. The second and third activities will be very short. This whole test will take just under 1 hour. Please turn to the page that says:” Practice A.” As a way to learn more, you will be asked to take a reading test in a different way today. What you are going to do is take a reading test and as you come to find your answer rather than thinking your thoughts in your head, you will say all your thoughts out loud. You’ll realize that while answering a reading test sometimes your thoughts are about the facts of the passages or questions, sometimes they are your opinions, and sometimes they are not complete thoughts. That’s okay because all your thoughts are important. I’d like you to say everything you are thinking - as you are thinking it - out loud. What I’d like you to do is take this reading test and answer the questions just like the way you would take a reading test for your class except that I want you to tell me everything you are thinking out loud. During our practice I will read the passage out loud, and then explain the types of questions on the think aloud test. After I give you the basic instructions for the test I will say my thoughts out loud for the first question – as if I were the student taking the test to give you an idea of what it means to think out loud. After I finish my example you will get a turn to practice saying your thoughts on the second question. We will make sure you understand how to think aloud during the practice before we move on to the test. If we need to, you can practice a couple of times. Do you have any questions before we begin the practice?

If yes, answer the questions. If no, continue to the practice.

8

THINK ALOUD PRACTICE A Please turn your page.

Make sure the student is on the page with the Practice A passage, then say:

This passage tells the most common idea of what a blue moon is. I’m going to read the practice passage out loud. Please follow along reading it to yourself as I read aloud. Sometimes – not very often – we get two full moons in one month. That second full moon is called a “blue moon.” No one knows why. Now we say “once in a blue moon” to mean “once in a long time.”

Pause. Then begin the instructions for the questions.

Below the passage are two questions. Question 1 is an unfinished sentence. One of the words below the sentence finishes the sentence correctly. If you are not sure of the answer you can go back to the passage and read again what the passage says. Explain out loud all your thoughts as you come to your answer. When you select to the correct answer fill in the answer circle and then look up to me to let me know you’re done. I will ask you a couple questions about the question you just answered. I’m going to read the practice question out loud. Please follow along reading it to yourself as I read aloud. After I read the question out loud I will think out loud my answer. Okay?

As long as the student understands, continue. If not, answer any questions and then continue. Remember to read the think aloud example as if this is your internal dialogue to yourself as you are thinking. Use your finger to point to the passage when you are referring back to the passage.

9

To be a “blue moon,” the moon must be …. The answer choices are: dark, long, blue, full. After reading the passage, the question, and the answer options I’m pretty sure I know what the answer is, it’s L, full. Now I’ll explain to you how I got to the answer. Follow along in your booklet as I explain how I came to my answer. This is what I’m thinking: To be a “blue moon,” the moon must be dark, must be long, must be blue, must be full. I know right away that it is not long because moons are never long, and I don’t remember reading that word in the passage. Uhh, I will look for the words “blue moon” in the passage. I knew there was going to be a question about blue moons. I’m not good at science, I wish this was a different subject. Okay, I’m going to go back and read in the passage and see if I can find the words blue moon and hopefully that will help me answer the question. “That second full moon is called a “blue moon.” No one knows why. Now we say “once in a blue moon” to mean “once in a long time.” It probably isn’t blue, because that doesn’t make sense to use the same word – to be a blue moon the moon must be blue. Uh-oh, did my stomach just growl, I wish I didn’t rush out of the house this morning without time for breakfast, I’m starving. Okay, here it is, a second full moon is called a “blue moon.” So to be a “blue moon,” the moon must be full. So I pick answer choice “L”, full.

Fill in circle “L” for the answer “full” then look up to student to indicate you are finished thinking aloud.

Let’s review what I did. I thought out loud all the thoughts I had while I was trying to solve the answer. I said everything I was thinking, how I came to my answer, and why I didn’t select the other answer choices. Once or twice my thoughts drifted away from solving the problem, but since that what I was thinking, I said it out loud. Let me give you one more example before you try.

Pause briefly.

10

On the last example I was pretty sure of the answer. Now I’m going to pretend to be another student answering the same question, only now, I’ve read the passage and question and I’m not sure what the answer is. Here is how I would answer this question if I wasn’t sure what the answer was. This is what I’m thinking: To be a “blue moon,” the moon must be dark. Maybe To be a “blue moon,” the moon must be long. Uh, that’s weird, I’m guessing that’s not right. To be a “blue moon,” the moon must be blue. That’s kind of weird too but maybe…. To be a “blue moon,” the moon must be full. Well I know moons can be full so maybe that’s it. I want to do well on this test so I think I should double-check the paragraph to see if I can find the answer because I really don’t know what it is. I’m glad this is a short passage. Don’t rush. Okay…I think it could be dark, blue, or full so I’ll check for each of those answers. Sometimes – not very often – we get two full moons in one month. That second full moon is called a “blue moon.” No one knows why. Now we say “once in a blue moon” to mean “once in a long time.” I think the moon must be blue is a trick so it’s not that one. So it’s either dark or full then, and did they say dark…no…okay I’m not really sure but I am going to say it’s L, full. To be a blue moon the moon must be full.

Fill in circle “L” for the answer “full” then look up to student to indicate you are finished thinking aloud.

Let’s review what I did in this practice. After I finished reading the question I wasn’t sure of the answer. I explained all my thoughts out loud while I was trying to solve the answer. I said everything I was thinking and why I didn’t select the other answer choices. Sometimes my thoughts may have not made sense, but it was because I was working out an idea and I decided it wasn’t a good idea before I finished a sentence so I change what I was thinking. Since that what I was thinking, I said it out loud. Do you understand what it means to think out loud?

If yes, move on, if no say, “let’s move onto the next question and if you need help I will help you”.

11

Let’s move onto the next question. The underline will help you find the words no one knows. This is your turn to practice thinking out loud. Read the question out loud or silently, whatever is easier for you. As soon as you finish reading the questions explain all your thoughts out loud as you come to your answer. If you know the answer, tell me all your thoughts as you come to your answer. If you don’t know the answer, tell me all your thoughts about how you would find answer. Select an answer and mark the circle next to it. Look up to me to let me know when you are done. You’ll notice I will be taking some notes as you talk. I am going to be making a note about whether these are good or bad questions. So you can keep on talking out loud even if you see me writing. One way you can help me in this study is to circle any word you don’t know to let me know it’s a hard word. You may begin.

Allow student to read item and answer choices aloud or silently. Student should express all thoughts out loud. If the student does not begin speaking, indicate that they should be saying their thoughts out loud as they solve the question. Take notes on student’s response on the “Practice A Think Aloud Notes” page as the student is thinking aloud (page 11) If student is silent for 5 seconds use a continuer such as: “What are you thinking now?” “Any other thoughts?” “Tell me how you came to pick that answer.” Once student is complete decide if student answered sufficiently or insufficiently. Provide suggestions for any minor errors. If student answers insufficiently move to practice B. (page 14) If student answers sufficiently, Move to “Practice A item and passage questions”. (page 12) ITEM and PASSAGE QUESTIONS INSTRUCTIONS Read all the questions as open-ended questions. Fill in responses into the appropriate box in your protocol. Take abbreviated notes for the follow up questions. Student responses should be short. If a student starts to talk too much try to stop them short.

12

Passage Practice A

Think Aloud Notes

Is he/she certain of the answer?

Somewhere in between?

Uncertain of the answer?

Other

Does he/she seem frustrated answering this question?


Satisfied of his or her answer?

Other

Is he/she referring to the passage to confirm answer?

If yes, is he/she doing so correctly?

Doing so incorrectly? Other

Did he/she answer before reading the options?

After reading options? After thinking through answer choices?

Other

Did he/she re-read the passage when answering a question?

Re-read part or all of the question?

Re-read par or all of the answer choices?

Other

Is there anything specific that confuses the student? Is there anything that interests the student?

What did the student read first?

2

Question 2 notes

General Passage Notes:

13

Now I’m going to ask you some questions about the questions you just answered. Passage Practice A Passage Practice A Question 2

Yes It was okay No, it was hard

1. Was this an easy question for you?

Yes Maybe/Not Sure

No

2. Do you think you got this question correct?

3. Why or why not?

Yes It was okay/sort of/kind of

No

4. Did you like this question?

5. Why or why not?

Passage Practice A Passage Practice A Questions

Yes, very much

Yes, it was okay

In the middle

No, not really

No, not at all

1. How much did you like reading about this passage?

2. Why?

Very easy

Pretty easy

In the middle

Pretty hard

Very hard

3. Was this an easy or hard passage for you?

4. Why?

14

Sometimes words or ideas in passages or test questions make it easier or harder to understand. Words that make a passage or test question easier to understand could be words you already learned at home, or in class before. Words that are hard to understand could be really long words or words you never saw before. 5. Were there any words or

ideas that stick out that you remember that made reading the passage or answering a question easier for you?

No Yes

6. If yes, what words or ideas?

7. Were there any words or

ideas that stick out that you remember that made reading the passage or answering a question harder for you?

No Yes


Go to page 27 to continue.

15

PRACTICE B Turn to the Practice B passage in your booklet.

Make sure the student is on the practice B passage page, then say:

I will read the practice passage out loud. Please follow along reading it to yourself as I read aloud. Robert Peary and his men were pushing toward the North Pole. One morning, after only seven miles the dog teams abruptly halted. The fierce wind and shifting ice had opened a large crack in their path, leaving a lane of water a quarter mile wide. Unable to cross this “lead,” Peary unhappily ordered the men to make camp. That night a terrific grinding sound shook Peary from his restless sleep. The ice was crushing together closing the lead. Peary hurriedly alerted the others. As the lead narrowed, the men very carefully guided their sledges across floating cakes of ice.


Below the passage are three questions. I will show you how I would think out loud my thoughts as I come to my answer. First, I will read the practice question out loud. Please follow along reading it to yourself as I read aloud.

Remember to read the think aloud example as if this is your internal dialogue to yourself as you are thinking.

Why did Peary and his men stop? Your answer choices are: They had become lost, the wind was fierce, they were exhausted, their way was blocked by water. Here is what I am thinking:

16

What did Peary and his men stop, do I know that? I think it had to do with them sleeping or the water because I remember that from the passage, I don’t think it was that they were lost, I don’t remember that word in the passage at all. I’m going to look back to the middle of the passage for the part that talked about Peary and his men stopping. “The fierce wind and shifting ice had opened a large crack in their path, leaving a lane of water a quarter mile wide….Okay, I don’t think the answer is the wind but they do say the wind. “Unable to cross this “lead,” Peary unhappily ordered the men to make camp. Ohh, so it was the large crack with the water that made the lead. I never read the word “lead” used that way before. I hope this isn’t a trick but I think the answer is that their way was blocked by water. So I will fill in answer option T, their way was blocked by water.

Fill in answer circle “T” and look up to student.

Does this practice help you understand what you have to do?

If no, say “Well, let’s have you try the next question.” Then say, If yes, say,

It’s your turn to practice thinking aloud. You may read the question out loud or silently, whatever is easier for you. Explain out loud all your thoughts as you come to your answer. When you are finished, fill in your answer circle. Look up to me to let me know you are done. Remember, circle any words you don’t understand to let me know they are hard words.

Allow student to read item and answer choices aloud or silently. Student should express all thoughts out loud. If the student does not begin speaking, indicate that they should be saying their thoughts out loud as they solve the question. Take notes on student’s response on the “Practice B Think Aloud Notes” page while student is thinking aloud. (page 18)

17

If student is silent for 5 seconds use a continuer such as: “What are you thinking now?” “Any other thoughts?” “Tell me how you came to pick that answer.” When the student finishes thinking aloud and selecting an answer determine whether the student grasps the task or not. Provide suggestions for any minor errors. If the student now understands how to think aloud say the text below. If the student still does not understand thinking aloud, have the student complete the last item in practice B.

That was great. Now we will move to the next part of the practice. OR Let’s try working on the next question together.

Either Move to the “Practice B item and passage questions” if student has mastered thinking aloud. (page 19) OR If student needs practice - work with student to think aloud the third item of Practice B. Once you and the student have completed the third item, make a decision about whether student has the capability to perform the task or is not able or willing. Go on to Practice C if you believe student has the ability and willingness and will grasp the task with more practice. (page 21) OR

18

Go on to PLAN B if the student does not seem able to understand the think aloud procedure. ITEM and PASSAGE QUESTIONS INSTRUCTIONS Read all the questions as open-ended questions. Fill in responses into the appropriate box in your protocol. Take abbreviated notes for the follow up questions. Student responses should be short. If a student starts to talk too much try to stop them short.

19

Passage Practice B

Think Aloud Notes




Other




Other






Other




Other



2

Question 2 notes




Other




Other






Other




Other



3

Question 3 notes


20

Now I’m going to ask you some questions about the questions you just answered. Passage Practice B Passage Practice B Question 2



Yes Maybe/Not Sure

No


8. Why or why not?


No


10. Why or why not?

Passage Practice B Passage Practice B Questions

Yes, very much

Yes, it was okay

In the middle

No, not really

No, not at all


10. Why?

Very easy

Pretty easy

In the middle

Pretty hard

Very hard


12. Why?

21



No Yes




No Yes



22

PRACTICE C Turn to Practice C passage in your booklet.

Make sure the student is on the practice C page, then say:

I will read the practice passage out loud. Please follow along reading it to yourself as I read aloud. I used to know an old man who could walk by any cornfield and hear the corn singing. “Teach me,” I’d say when we’d passed on by. (I never said a word while he was listening.) “Just tell me how you learned to hear that corn.” And he’d say, “It takes a lot of practice. You can’t be in a hurry.” And I’d say, “I have the time.”


Below the passage are four questions. I will show you how I would think out loud my thoughts as I come to my answer. First, I will read the practice question out loud. Please follow along reading it to yourself as I read aloud.

Remember to read the think aloud example as if this is your internal dialogue to yourself as you are thinking.

What did the girl telling the story ask the old man? Your answer choices are: What singing corn sounded like, why he thought that corn sings, how he learned to hear corn singing, how long it took him to learn to hear corn singing. Here is what I am thinking: The girl, does the girl have a name?…no. What did the girl telling the story ask the old man? I know, I’ll look for a question mark. I have to find what the girl asks and that means that sentence will end with a question mark. I don’t want to read the whole

23

passage again, I’ll just look for question marks. Okay so I’m browsing. No question marks, but I see quotation marks, I guess the girls question or that old man’s response are inside quotation marks. What are the choices again? What singing corn sounded like, why he thought that corn sings, how he learned to hear corn singing or how long it took him to learn to hear corn singing. Okay, it’s all about corn singing. I don’t think corn sings. …“Teach me.”… no that’s not it…”Just tell me how you learned to hear that corn”… That could be it, …“And he’d say”… yes, that must be it because that means the “Just tell me…” was the girl talking. So, I’m going to pick “K” how he learned to hear corn singing.

Fill in answer circle “K” and look up to student.

Now it’s your turn to practice thinking aloud. You may read the question out loud or silently, whatever is easier for you. Explain out loud all your thoughts as you come to your answer. When you are finished, fill in your answer circle. Look up to me to let me know you are done. Remember, circle any words you don’t understand to let me know they are hard words.

Allow student to read item and answer choices aloud or silently. Student should express all thoughts out loud. If the student does not begin speaking, indicate that they should be saying their thoughts out loud as they solve the question. Take notes on student’s response on the “Practice C Think Aloud Notes” page while the student is thinking out loud. (page 24) If student is silent for 5 seconds use a continuer such as: “What are you thinking now?” “Any other thoughts?” “Tell me how you came to pick that answer.” Once student is complete decide if student answered sufficiently or insufficiently. When the student finishes thinking aloud and selecting an answer determine whether the student grasps the task or not.

24

Provide suggestions for any minor errors. If the student now understands how to think aloud say the text below. If the student does not seem able to understand the think aloud procedure, move to the PLAN B section of the protocol

That was great. Now we will move to the next part of the practice. OR Thank you very much for your hard work.

Move to the “Practice C item and passage questions” if student has mastered thinking aloud. (page 25) OR Move to the “PLAN B” section of the protocol. ITEM and PASSAGE QUESTIONS INSTRUCTIONS Read all the questions as open-ended questions. Fill in responses into the appropriate box in your protocol. Take abbreviated notes for the follow up questions. Student responses should be short. If a student starts to talk too much try to stop them short.

25

Passage Practice C

Think Aloud Notes




Other




Other






Other




Other



2

Question 2 notes




Other




Other






Other




Other



3

Question 3 notes


26

Now I’m going to ask you some questions about the questions you just answered. Passage Practice C Passage Practice C Question 2



Yes Maybe/Not Sure

No


13. Why or why not?


No


15. Why or why not?

Passage Practice C Passage Practice C Questions

Yes, very much

Yes, it was okay

In the middle

No, not really

No, not at all


18. Why?

Very easy

Pretty easy

In the middle

Pretty hard

Very hard


20. Why?

27



No Yes




No Yes



28

We are finished the practice. There are more passages and questions in your booklet. Each passage tells what you need to know to answer the questions about it. If you are not sure which answer is right, use your best judgment and pick one. Do you have any questions?

If the student does, answer the questions. If the student does not, take a look at the clock in the room and determine the time that the student should stop the think aloud activity. It should be 10 minutes prior to the end of the session. Write the end time down somewhere for YOU to see, then open the student’s test booklet open to the first passage and say:

When I tell you to begin, you should start with this first passage.

Point to the first passage and each of the questions in the student’s test booklet. Then say,

I’d like you to read the passages and questions out loud or silently, whatever you prefer. When you begin to answer the first question SAY YOUR ALL YOUR THOUGHTS OUT LOUD AS YOU COME TO YOUR ANSWER. Answer all the questions for the passage and then STOP at end of last question for each passage. I’ll then ask you some questions about the questions you just answered, just like we did in the practice. Don’t forget, I will be taking some notes as you talk so I know which are good and bad questions. Keep on talking even if you see me writing. If there is a word you don’t know, let me know it’s a hard word by circling it. We will have roughly 30 minutes to do the passages. Do not feel that you are rushed. I am more interested in what you are thinking than if we finish the test. We will work together as you take the test.

29

If you have any questions during the test, you can ask me. Do you have any questions before we begin?

Answer any questions. Then, say:

All right, you may turn your page and begin with the first passage.

Start stopwatch (if you wish), otherwise, keep your eye on the time and stop the think aloud session when there is 10 minutes left in the session. Follow along as student reads passage and encourage the student to think out loud if s/he does not begin. Allow student to read item and answer choices aloud or silently. Student should express all thoughts out loud. If the student does not begin speaking, indicate that they should be saying their thoughts out loud as they solve the question. Take notes on student’s response on the “Think Aloud Notes” page while the student is thinking out loud for each passage. If student is silent for 5 seconds use a continuer such as: “What are you thinking now?” “Any other thoughts?” “Tell me how you came to pick that answer.” Once student is complete decide if student answered sufficiently or insufficiently. Take notes on student’s response on the “Passage 1 Think Aloud Notes” page. At the end of every passage, make sure to ask the item and passage questions for every item and passage.

30

ITEM and PASSAGE QUESTIONS INSTRUCTIONS Read all the questions as open-ended questions. Fill in responses into the appropriate box in your protocol. Take abbreviated notes for the follow up questions. Student responses should be short. If a student starts to talk too much try to stop them short. Stop student when there are 10 minutes left in the session and turn to page 52. Mark in test administrator protocol the place where the student stopped. If it is in the middle of a student’s verbal response, allow student to finish.

31

Passage 1

Think Aloud Notes




Other




Other






Other




Other



1

Question 1 notes




Other




Other






Other




Other



2

Question 2 notes


32

Passage 1

Think Aloud Notes




Other




Other






Other




Other



3

Question 3 notes




Other




Other






Other




Other



4

Question 4 notes


33

Now I’m going to ask you some questions about the questions you just answered. Passage 1 Question 1



Yes Maybe/Not Sure

No


18. Why or why not?


No


20. Why or why not?

Passage 1 Question 2



Yes Maybe/Not Sure

No


23. Why or why not?


No


25. Why or why not?

34

Passage 1 Passage 1 Question 3 Yes It was okay No, it

was hard 26. Was this an easy question for you? Yes Maybe/Not

Sure No


28. Why or why not?

Yes It was

okay/sort of/kind of

No


30. Why or why not?

Passage 1 Question 4 Yes It was okay No, it


Sure No


33. Why or why not?

Yes It was


No


35. Why or why not?

35

Passage 1 Passage 1 Questions Yes,

very much

Yes, it was okay

In the middle

No, not really

No, not at all


26. Why?

Very

easy Pretty easy

In the middle

Pretty hard

Very hard


28. Why?



No Yes




No Yes


36

Passage 2

Think Aloud Notes




Other




Other






Other




Other



5

Question 5 notes




Other




Other






Other




Other



6

Question 6 notes


37

Passage 2

Think Aloud Notes




Other




Other






Other




Other



7

Question 7 notes




Other




Other






Other




Other



8

Question 8 notes


38

Now I’m going to ask you some questions about the questions you just answered. Passage 2 Passage 2 Question 5



Yes Maybe/Not Sure

No


38. Why or why not?


No


40. Why or why not?




Yes Maybe/Not Sure

No


43. Why or why not?


No


45. Why or why not?

39



Sure No


48. Why or why not?

Yes It was


No


50. Why or why not?



Sure No


53. Why or why not?

Yes It was


No


55. Why or why not?

40


very much

Yes, it was okay

In the middle

No, not really

No, not at all


34. Why?

Very

easy Pretty easy

In the middle

Pretty hard

Very hard


36. Why?



No Yes




No Yes


41

Passage 3

Think Aloud Notes




Other




Other






Other




Other



9

Question 9 notes




Other




Other






Other




Other



10

Question10 notes


42

Passage 3

Think Aloud Notes




Other




Other






Other




Other



11

Question 11 notes




Other




Other






Other




Other



12

Question 12 notes


43




Yes Maybe/Not Sure

No


58. Why or why not?


No


60. Why or why not?




Yes Maybe/Not Sure

No


63. Why or why not?


No


65. Why or why not?

44



Sure No


68. Why or why not?

Yes It was


No


70. Why or why not?



Sure No


73. Why or why not?

Yes It was


No


75. Why or why not?

45


very much

Yes, it was okay

In the middle

No, not really

No, not at all


42. Why?

Very

easy Pretty easy

In the middle

Pretty hard

Very hard


44. Why?



No Yes




No Yes


46

Passage 4

Think Aloud Notes




Other




Other






Other




Other



13

Question 13 notes




Other




Other






Other




Other



14

Question 14 notes


47

Passage 4

Think Aloud Notes




Other




Other






Other




Other



15

Question 15 notes




Other




Other






Other




Other



16

Question 16 notes


48

Passage 4

Think Aloud Notes




Other




Other






Other




Other



17

Question 17 notes




Other




Other






Other




Other



18

Question 18 notes


49




Yes Maybe/Not Sure

No


78. Why or why not?


No


80. Why or why not?




Yes Maybe/Not Sure

No


83. Why or why not?


No


85. Why or why not?

50



Sure No


88. Why or why not?

Yes It was


No


90. Why or why not?



Sure No


93. Why or why not?

Yes It was


No


95. Why or why not?

51



Sure No


98. Why or why not?

Yes It was


No


100. Why or why not?



Sure No


103. Why or why

not?

Yes It was


No


105. Why or why not?

52


very much

Yes, it was okay

In the middle

No, not really

No, not at all


50. Why?

Very

easy Pretty easy

In the middle

Pretty hard

Very hard


52. Why?



No Yes




No Yes


53

Stop. That is the end of this section. Thank you for your thoughtful answers. You are doing a great job. We are going to move onto the second activity. First, let’s stand up and stretch our legs.

Stand up and encourage student to take a 10 second break.

54

ORAL READING FLUENCY In our next activity, you’ll read some passages out loud at your normal reading rate. I’ll be using this stopwatch and taking some notes while you read. If you come to a word you’re not sure of, make your best guess and then keep going. I’ll tell you when to stop reading each passage. Please turn to the page of your booklet that says Reading Passages.

Get the stopwatch ready and be ready to turn the page to score. Make sure the student has turned to the oral reading fluency title page but is not looking at the first passage itself.

Okay, when I tell you to, you’re going to turn the page over and start reading out loud at your normal reading rate. You’ll see that some of the pages have 2 passages. If you finish the first passage before I say stop, go right on with the second. You can pick up the booklet and hold it in your hands if that’s the way you like to read, or you can keep it on the table. Just make sure you are comfortable. Are you ready? Okay, as soon as you see me turn my page over, you may turn the page over and start reading out loud.

55

Start stopwatch as soon as the student begins reading. Record reading errors by making slashes through incorrect or skipped words) on your score sheet for Passage #1 Make sure the student cannot see the score sheet. Supply a word for the student after 5 seconds of silence or sounding out. The following are NOT considered errors. • Words read correctly are scored as correct. • Self-corrected words are counted as correct as long as the final word is the correct word. • Repetitions are counted as correct. • Examples of dialectical speech are counted as correct. • Inserted words are ignored. The following ARE errors and should be marked: • Mispronunciations are counted as errors when it changes the word to a non-word or an incorrect word. • Substitutions are counted as errors. Each word substituted is counted as one error. • Omissions are counted as errors. Each word omitted is counted as one error. • Words read to the student by the examiner after 5 seconds have gone by are counted as errors. After 1 minute has elapsed, circle the final word read by the student and say:

OK, you can stop reading that one. Take a short break; relax for a few seconds. [After a break of about 10 seconds, say:] OK, now you’ll do the same thing with a different passage. When I tell you to, turn the page over and start reading the next passage out loud just like you did before. Are you ready? Okay, as soon as you see me turn my page over, you may turn the page over and start reading out loud.

56


OK, you can stop reading that one. Great job. You have one final passage to read out loud, but let’s take another quick break first. [After a break of about 10 seconds], say: Are you ready? Okay, as soon as you see me turn my page over, you may turn the page over and start reading out loud.

57


OK, you can stop reading that one, and that means you’re done with this exercise! Thanks. Now you’ll do one more activity.

58

STUDENT SURVEY Please turn your page to the page that says Student Survey. Your contributions to this research study are important. We’d like to know what you thought about some of the tests you’ve taken. For this last activity, I’d like you to answer some questions. Please turn your page to see the questions.

As soon the student finds the page, say:

It is your choice, I can read the questions to you, or you may read them to yourself. Please circle the letter of the answer choice that is the closest to how you feel and write in your answer to the open ended questions. If you are not sure what a question means, ask me and I will help you. Would you like me to read the questions to you, or would you like to read them to yourself?

Determine the student’s reading preference. If the student prefers to read the questions to him/herself, say “You may begin now.” If the student prefers the questions read aloud, say:

We will begin now. Question 1. Do you like to read? Your answer choices are: (A) Yes (B) No

59

Please circle your answer. [Pause.] Question 2. Compared to the students in your class, how well do you think you read? Your answer choices are: (A) Above average (B) The same (C) Below average Circle your answer. [Pause.] Question 3. Compared to all the students in your grade in NJ, how well do you think you read? Your answer choices are: (A) Above average (B) The same (C) Below average Circle your answer. [Pause.] Question 4. How often do you read out loud? Your answer choices are: (A) Often (B) Not very often (C) Almost never or never Question 5. Do you think saying your thoughts out loud helped you complete the test in the first activity? Your answer choices are: (A) Yes (B) No Circle your answer. [Pause.] Question 6. Do you think a reading test with passages and multiple choice questions – one that looks like the test from your first activity – is a good way to show your reading skills?

60

Your answer choices are: (A) Yes (B) No Circle your answer. [Pause.] Question 7. Do you think including a passage to read out loud on a reading test – one that looks like the test from your second activity – is a good way to show your reading skills? Your answer choices are: (A) Yes (B) No Circle your answer. [Pause.] Question 8. Do you have any ideas about how to make a better reading test for you? Your answer choices are: (A) Yes (B) No Circle your answer. If you have some ideas, please write them on your survey so we can use your ideas to help make a better reading test. [If students fills in response, wait until student is complete.] Question 9. What kind reading class do you attend? Your answer choices are: (A) Regular (B) Special education (just for reading) (C) Special education (for some classes including reading) (D) Self contained special education classroom Circle your answer. [Pause.] Question 10. Do you have a disability that affects your reading? Your answer choices are:

61

(A) Yes (B) No Circle your answer. [Pause.]

Give the student a couple of minutes to complete their surveys if needed.

THANK YOU! Good job. We’ve finished all the activities. Thanks very much for doing your best. As a thank you, here is a $10 Barnes and Noble Gift Certificate for your hard work today.

Dismiss students according to study and school procedures.

Sample Cognitive Lab Interview Protocol - ETS Home · PDF file3 COGNITIVE LAB INTERVIEW PROTOCOL SEATING INSTRUCTIONS Follow the school’s procedures to escort one student from his

Documents