Higher learning skills, good feedback EASiHE Workshop 10 th February 2010
Jan 13, 2016
Higher learning skills, good feedbackEASiHE Workshop
10th February 2010
Introduction
• Welcome
• Introductions
• Overview of the day
2
Competency Modelling
Supporting Higher-Level Outcomes and Better Feedback
Lester GilbertLearning Societies Lab
University of Southampton
Competence
Discriminate learning & teaching
Teaching
Learning
Technology support
The big picture
Competence hierarchy
Prerequisitecom petencies
Com m onenabling
com petenciesIndefin ite
depth
H igher tolower
capab ilities
F ind Problemsolving
Use
Rules
Reca ll
Concepts
D iscrim -inations
Synthesis
Analysis,Evaluation
Application
Com pre-hension
Know ledge
M errill G agneBloom
Competency tree
Prerequisites
A com petency m ayitself be a tree Com m onLinked, but no
im plied sequence
Com prises
A B
C
D
E
F
G
KJ
I
H
Example nursing competencies
Competency No. Capability Subject Matter Content Attitude C00 Use Ethical principles Values C10 Apply Ethical issues Actively C11 Demonstrate Client confidentiality Respectfully C12 Identify Ethical Issues Sensitively C20 Use Professional regulation Actively C21 Evaluate Professional regulation Considerately C22 Identify Limitation in own practice Values C23 Recognize Need for referral Willingly C24 Recall Professional regulations Willingly
Sitthisak, Gilbert, & Davis (2007)
Example competency 3D tree
Sitthisak, Gilbert, & Davis (2007)
Questions, comments, discussion…
Gilbert, L., & Gale, V. (2008).Principles of E-learning Systems Engineering.
Chandos Publishing. ISBN 1-84334-290-1
HLOs (Higher-order Learning Outcomes)Bill Warburton
EASiHE/iSolutions
16
A formative taxonomy
• For FORMATIVE evaluation, Bloom et al. (1971) distinguished a hierarchy of levels of behaviour parallel to the cognitive taxonomy
• Added value to the cognitive taxonomy by making it easier for teachers to identify ‘mal-rules’ – flaws in reasoning
ABILITY TO MAKE APPLICATIONS
ABILITY TO MAKE TRANSLATIONS
SKILL IN USING PROCESSES & PROCEDURES
KNOWLEDGE OF RULES & PRINCIPLES
KNOWLEDGE OF FACTS
KNOWLEDGE OF TERMSIncreasing level of abstraction
17
Testing KNOWLEDGE OF FACTS(Bloom 1956: KNOWLEDGE RECALL)
– Memorise something
– Recognise something
– Recall of information or identity:
• Who, what, when, where, how ...?
• Describe…
(School of Oceanography and Earth Science, University of Southampton)
18
KNOWLEDGE OF RULES & PRINCIPLES
– Memorise and recall a general rule
• What, when, where, how ...? • Describe…
– Describe interrelationships among many items
– Memorise and recall applications of a rule
– Memorise and recall exceptions to a rule
– Does *not* deal with Application of a rule
19
Testing KNOWLEDGE OF RULES & PRINCIPLES(Bloom 1956: COMPREHENSION)
Example:
In order to write a chemical formula, you have to know
a. only the symbols of the elements that are in the compound
b. only the proportions in which the atoms of elements combine
c. both the symbols of the elements that are in the compound and the proportions in which the atoms of elements combine
d. the atomic weight of the elements that form the compound
(Bloom et al. 1971)
20
Skill in using Processes & Procedures
• Steps along the route to Mastery
• The difference between ‘Knowing That’ and ‘Knowhow’?
• Accuracy in use
• Practice makes perfect
• Expression of (justified) confidence
• Not formally represented in Bloom’s Taxonomy of the Cognitive domain but important for assessing Mastery
• Formative drills
21
Skill in using Processes & Procedures Example of process-based item (JISC/REAQ Report,2009)
22
ABILITY TO MAKE TRANSLATIONS• Put idea in own words or use new examples of what is learned
• Transform a term, fact, rule, principle, process or procedure from one form to another
• Take a phenomenon presented in one mode/form and represent it by an equivalent form/mode
• Move from a verbal to a symbolic form
• Determine when a new illustration is appropriate or not
• Move from a concrete to a more abstract form, or from a general to a more specific illustration, and vice versa.
23
Testing ABILITY TO MAKE TRANSLATIONS (Bloom 1956: COMPREHENSION)
(i) Interpret (native Spanish speaker) or
(ii) Translate (second language speaker) parts of speech from English to Spanish
(School of Humanities, University of Southampton)
Example: testing a participant’s ability to
24
ABILITY TO MAKE APPLICATIONS
• Recognize the essentials of the problem
• Use of a rule/principle learned in one context to solve a problem presented in a new or unfamiliar context
• Identify rules/principles/generalisations relevant to a problem
• Use ideas to solve a problem which is different from those previously encountered in the instruction or instructional materials.
• Most complex of the Formative categories - depends on other classifications but requires application of ideas in new situations or problems
25
Testing ABILITY TO MAKE APPLICATIONS (Bloom 1956: APPLICATION)Example: testing a participant’s ability to
(i) solve problems (in this case, using Wien’s Law) …(ii) use facts, rules and principles
(School of Physics and Astronomy, University of Southampton)
26
Testing ANALYTICAL skills • Subdivide something to show how it is put together
• Find the underlying structure of a communication
• Identify motives
• Separate something into component parts: • What are the parts or features of...? • Classify...according to... • Outline/diagram... • How does...compare/contrast with...? • What evidence can you list for...?
26
27
Testing ANALYTICAL skills
GNP per capita 1991 ($ USA)
GNP growth p.c. p.a. ‘80-’91
Pop. growth 80-91
Total employment 1980-85 (%)Agriculture
Industry
Services
A 500 2,5% 1,5% 51 20 29B 1570 5,8% 1,6% 74 8 8S.A. 2560 0,7% 2,5% 17 36 36C 25110 1,7% 0,3% 6 32 321. A=South Korea; B=Kenya; C=Canada 2. A=Sri Lanka; B=Germany;
C=Thailand3. A=Sri Lanka; B=Thailand; C=Sweden* 4. A=Namibia; B=Portugal;
C=Botswana
(CASTLE project, University of Leicester)
Q. Which countries' statistics are being reported in A, B and C?
This example tests a participant’s ability classify data according to specific criteria:
28
Testing SYNTHETIC skills – Create original products in verbal or physical
form
– Combine ideas to form a new whole:
• What would you predict/infer from...? • What ideas can you add to...? • How would you create/design a new...? • What might happen if you combined...? • What solutions would you suggest for...?
28
29
Testing SYNTHETIC skills
(Thinking Skills Admission Tests, University of Cambridge)
A* B C D E
Q. The picture shows a cube that I have made. Which one of the shapes below, if cut out and folded, could make a cube the same as mine?
Example: testing a participant’s ability to predict/infer the 3D appearance of a 2D net. Requires the abstract abilities to accurately reconstruct solids, rotate them about three axes and combine the results with a predicted model.
30
Testing EVALUATION skills – Make value decisions about issues
– Resolve controversies or differences of opinion
– Develop opinions, judgements or decisions:
• Do you agree that...? • What do you think about...? • Place the following in order of priority... • What criteria would you use to assess...? • What are the most important aspects of...?
31
Testing Evaluation skills
a. The assertion and the reason are both correct, and the reason is valid
b. The assertion and the reason are both correct, but the reason is invalid*
c. The assertion is correct but the reason is incorrect
d. The assertion is incorrect but the reason is correct
e. Both the assertion and the reason are incorrect.
(CASTLE toolkit, University of Leicester)
Example: testing a participant’s ability to evaluate the link between cause and effect in terms of predefined criteria
"The United States took part in the Gulf War against Iraq BECAUSE of the lack of civil liberties imposed on the Kurds by Saddam Hussein's regime."
32
Mastering the art of item authoringThree principle challenges for new authors:
• How to reducing the scope for guessing
• How to testing ILOs at higher levels of abstraction
• How to varying the difficulty of items
Effective questions and feedbackVeronica Gale
EASiHE/Independent Consultant
HLO questions• Write questions and feedback after finalising the ILO • Do students have enough information to be able to
answer the question correctly?• Is the question focused on one main learning point?• If there is a score, is it clear to students how the scoring
works?• If there is a ‘correct answer’ is this the only one that can
be considered correct?• Does the question use terminology that is suitable for
this stage in the course?• Have you piloted the assessment?
34
Example question from Bloom
35
FACT: The number of shareholders in most large corporations has increased considerably during the last 30 years.
CONCLUSION: Control of corporations has become more democratic in the last 30 years.A: The fact is good evidence to support the
conclusion
B: The fact is good evidence to disprove the conclusion
C: Neither A nor B applies clearly To help them learn, what do the students who select A need in their feedback?
Effective feedback• Whether the response is correct or not
• If the student is incorrect: an explanation of why, what the correct answer is and why this is correct
• If the student is correct: confirmation of why this is correct
• Recommended action e.g., reference back to the learning materials, remedial tuition, further questions for practice
All to be:• Presented immediately
• With the question, their response and the feedback visible at the same time
36
When there are no ‘right’ and ‘wrong’ answers• Scoring may be inappropriate
• Discuss the merits and drawbacks of the different options
• Give an expert’s opinion on the best choice, with full explanation
• Consider a model answer, range of sample answers or demonstration
• Ask a follow-up question on the rationale for the response
37
Visual feedback
38
Activities
Group discussions
Activity One
40
Review the questions provided and identify:• what works well and • what needs to be improved
Activity Two
41
Drawing on your own experiences and thinking about the discussions in the workshop so far, create a poster listing:• what to do, and • what to avoid when writing e-assessments and feedback for higher learning outcomes.Your readership are academics creating new formative questions.
Lessons learned from project managing high volume e-assessment
David Bacigalupo
EASiHE/ECS
Conclusion and next stepsLester Gilbert
EASiHE/ECS