Clarity and why its important, Liz Norman, ANZCVS Exam Writing Workshop, November 2013

Clarity and why its important

Liz NormanMassey University

Australian and New Zealand College of Veterinary Scientists, Exam Writing Workshop, November 2013

Validity again

• Not a property of a test, but relates to the interpretations or actions made on the basis of test scores

• Eg:

“The award of Fellowship of the College implies that the awardee has sufficient knowledge and experience in a

particular area of veterinary science to entitle him/her to be acknowledged as a specialist or consultant in that area"

Chain of inferences

1. Fellows have sufficient knowledge, skills, attitudes and judgement to be considered specialists

2. The things we can measure in examinations are things that are necessary to be a Fellow

3. A particular examination (all 4 components) is representative of all that we could measure in an examination

4. The examination score is a measure of achievement in the examination

5. The passing score of 70% correctly separates someone with sufficient knowledge, skills, attitude and judgement to be a Fellow from someone without

What we don’t want to measure

• Ability to take tests• Ability to write legibly and fast• Ability to rote learn whole pages of textbooks or review

articles - prewriting• Ability to write down a huge series of unconnected facts

in no particular order• Ability to research examiner's fields of interest and rote

learn impressive aspects of that• Ability to interpret what examiners are thinking

Why clarity is important

• We want the candidate to do the task we envisaged, not something else– Validity – measuring what we intend to measure

• We want to minimise irrelevant difficulty– Validity – measuring what we intend to measure

General issues with question writing

CommunicationThe examination questions are the question setter’s expression of the question setter’s task.

The candidate’s answer represents the candidate’s expression of the candidate’s interpretation of the questions.

The marker evaluates the marker’s interpretation of the candidate’s expression of the candidate’s answer.

The marker uses the marker’s interpretation of the setter’s expression of the setter’s task to evaluate the candidate’s answer.

Modified from Pollitt & Ahmed (1999). A new model of the question answering process. Paper presented at the International Association for Educational Assessment, Bled, Slovenia. Retrieved from http://www.cambridgeassessment.org.uk/images/109651--a-new-model-of-the-question-answering-process.pdf

Decisions on structuring a Q

• Breaking the task down into parts or keeping it as one whole

• Whether to provide strategy guidance – prompts on how to structure their answer

• Amount of sifting or unpacking required• Inclusion of distractors• Technical terms vs everyday language• Highlighting of words or phrases• Mark allocation

Structure

1. Explain the aetiopathogenesis of hepatic lipidosis in cats. (12 marks)

2. The following drugs are commonly used in the management of hepatic lipidosis in cats. For each drug describe its mechanism of action AND any adverse effects or precautions for use.

a) Ursodeoxycholic acid. (3 marks)

b) S-adenosyl methionine (3 marks)

3. Outline the principles of nutritional management and fluid therapy for a cat with idiopathic hepatic lipidosis (12 marks)

Strategy guidance

You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide

Strategy guidance

Briefly outline your interpretation of the laboratory results. Do not just state the abnormalities.

Expectations and stereotypes

Examples:• male animal case• differential diagnoses candidate would consider• expectation of hard questions• expectation that Qs will ask about what

something is rather than what it is not

Contextualising Qs

• Context is good because it brings relevance and authenticity

• Allows assessment of concrete or specific examples not abstract concepts or generalisations

• Allows assessment of applied learning (doing not just knowing)

• All these carry with them a potential for bias.

Real world context

• Advantages– Assessing application to a real problem– “Shows how” rather than “knows”

Miller (1990). The assessment of clinical skills competence performance. Academic Medicine, 65(9), S63-S67

Real world context

• Advantages– Assessing application to a real problem– “Shows how” rather than “knows”

• Disadvantages– More words used– More intended/unintended demand– Familiarity – schemas/sterotypes– Focus may direct candidates to the wrong aspects– Images can be particularly distracting

Guidelines for context

• If you are going to use context use a natural real one

• Contextualise purposefully for what it brings to the task

• Only use images when the Q could not be asked without them

• Don’t decorate!

Unpacking or sifting

• A crate of 12 cans of cola costs $4.20. How much do 7 crates of cola cost?

• A ski pass costs $4.20. How much would it cost for 7 days?

Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy & Practice, 14(2), 201-232

A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Other aspects

• Distractors• Everyday language• Highlighting of words and phrases• Showing the mark allocation

Wording questions

Don’t write questions; write tasks

What is your diagnosis?

State the most likely diagnosis orState the most likely diagnosis and explain your reasoning orDiscuss the differential diagnoses you would consider in this case or…..

Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

Contrast: to find differences or to distinguish between things.

Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

Illustrate: to use a picture, diagram or example to clarify a point.

Specify boundaries of the answer

Speciese.g. “in both dogs and cats…”

Quantities and amountse.g. “Provide 5 reasons why…”

With reference toe.g. “ With reference to the published research from ..”

List the clinical signs of hypothyroidism in dogs.

List the three most common owner-observed clinical signs of hypothyroidism in dogs andexplain how thyroid hormone deficiency leads to each of these signs.

Q parts – the scope

Examples of problems….

Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.

Examples of problems

Outline your approach to confirming the initial clinical diagnosis and a management and prevention plan for this problem. This discussion should include an outline on further observations taken about ….


…list in dot point form: the gross pathological features, the characteristic histopathological changes, and the clinical pathology changes. In your discussion, list one antemortem test/procedure that can be used to aid in the diagnosis …


A veterinarian asks you for assistance in designing a protocol for the delivery of a vaccine for cats in their practice. What factors would you take into consideration in designing this protocol?

Examples of question problems

Are there any clinical features which can help you determine a patient’s prognosis?


Describe and discuss the following:

a) preparedness


State what you believe is your most likely diagnosis.


Discuss commonly found tumours and tumour-like disorders associated with the oral cavity and dental tissues of the horse.


How would you localise the site of the lesion?

Answer provided in the marking scheme:Spinal lesion between T3 and L3


Write notes on

a) considerations in the selection of stockpersons and animal attendants


• Describe and give reasons for your further clinical examination (if any), and recommendations to the owner/trainer for diagnostics and treatment. Explain what you think is really important and why. Explain if you think there is any controversy in treatment plans.


Version 1

Using examples, describe the categories of ways that the productivity and profitability of a livestock enterprise may be compromised by disease.

Version 2

Using examples, describe at least five different ways that the productivity and profitability of a livestock enterprise may be compromised by disease.

Sentence-level clarity

• Simple sentence structures• Grammatically correct• Subject verb object sequence• Remove superfluous words

Question timing and why its important

How long it will take the candidate

• Unstructured tasks – open ended• Need to control the demands of the Q carefully

• Question reading time• Writing speed

Klatt & Klatt (2011). How much is too much reading for medical students? Assigned reading and reading rates at one medical school. Academic Medicine, 86(9), 1079-1083.

Summers & Catarro (2003). Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal, 50(3), 148-157.

Effect of time stress

• Time stress– Increases the use of schemas (sterotyping)– Decreases working memory processing capacity– Decreases the ability to maintain relevant information

and suppress irrelevant information

What we don’t want to measure

• Ability to take tests• Ability to write legibly and fast• Ability to rote learn whole pages of textbooks or review

articles - prewriting• Ability to write down a huge series of unconnected facts

in no particular order• Ability to research examiner's fields of interest and rote

learn impressive aspects of that• Ability to interpret what examiners are thinking

“Effective item writers are trained, not born … “

Downing & Haladyna (2006). Handbook of test development: Lawrence Erlbaum Associates Publishers. p.11