Graduate Attributes Assessment and Reporting
For 2016 Accreditation
October 2015
Accreditation 2016
• 10 of 11 Engineering programs up for accreditation in 2016
• We “must demonstrate that the graduates of a program possess
the [12] attributes” and “there must be processes in place that
demonstrate that program outcomes are being assessed in the
context of graduate attributes, and that the results are applied to
the further development of the program.”
2
Graduate Attributes
3
Graduate attributes are the qualities, skills and understandings a
university community agrees its students should develop during
their time with the institution and consequently shape the
contribution they are able to make to their profession and
society. They are qualities that also prepare graduates as agents
of social good in an unknown future. (Bowden et al., “Generic Capabilities of Australian
Technology Network University Graduates”, http:/www.clt.uts.edu.au/ATN.grad.cap.project.index.html, 2000)
Graduate Attributes - CEAB “Generic characteristics, specified by the Accreditation Board, expected to be exhibited by graduates of accredited Canadian engineering programs at the time of graduation.”
The graduate attributes broadly define the required outcomes, but they are difficult to measure directly. 4
1. KB: Knowledge base for engineering 2. PA: Problem analysis 3. IN: Investigation 4. DE: Design 5. ET: Use of Engineering tools 6. IT: Individual and teamwork 7. CS: Communication skills 8. PR: Professionalism 9. IE: Impact of engineering on society and the environment 10. EE: Ethics and equity 11. EP: Economics and project management 12. LL: Life-long learning
Indicators
• To measure graduate attributes, all universities must define their own measurable descriptors called “indicators”:
o “Descriptors of what students must do to be considered competent in
the attribute; the measurable and pre-determined standards used to
evaluate learning (i.e. measureable characteristics of attributes or
components of attributes).”
Note: We are required to “measure” indicators, not GAs
5
Complete Process
Defining Graduate Attribute
Indicators
Curriculum Mapping
Collecting Data of Student Learning
Analysis and Interpretation
Data-informed
Curriculum Improvement
6
• Core Graduate Attribute indicators are defined o Some graduate attribute indicators will ultimately be customized for each department
• Pilot projects have been undertaken to evaluate how best to report
results of Graduate Attribute assessment
• Graduate Attribute curriculum maps have been prepared for each
program
• Graduate Attribute indicators must be assessed and reported for
2015-2016 academic year (as a minimum)
Objectives
• Two principal objectives in developing accreditation
procedures for the Engineering programs:
1) Maintain the accreditation of our programs in the new CEAB
accreditation context
2) Do so with as little additional work imposed on instructors
7
Procedure • Quantitatively assess all GAs at the level of “D” (Developed) or “A”
(Applied) in a courses, in all years, for all students, by correlating
performance in “assessment tools” with that in GAs / indicators
• However, to get satisfactory data for all GAs in a given program,
departments must also o use other approaches to evaluate “soft skills” GAs (e.g., IT, CS, PR, IE, EE, EP,
LL) o employ other approaches to evaluate GAs other than just correlating GAs with
graded work
• A special emphasis on obtaining data for the “soft skills” GAs will be
undertaken in FACC 100 and FACC 400 (which are common to all
programs).
8
Procedure (cont.) • The faculty is providing general support to assist departments in
meeting the necessary requirements to obtain, analyze and
present their GA data.
• However, every course, program and department is different, so
the process cannot satisfy all situations
o Adaptation and customization will be required by each department
o CEAB guidelines and expectations are very vague
Although the faculty can provide guidance, professors and departments will sometimes need to use their judgment, as many questions do not have clear answers!
9
Overview Department
Program
General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)
Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means
Excel (analysis is the responsibility of the prof. / dept.)
myCourses (most courses, Faculty does the analysis)
10
Reporting Assessments
myCourses Gradebook Setup
Excel Format
General Approach Use myCourses for the majority of courses to correlate individual student performance in certain GAs with the various assessment tools used by the instructor
• The gradebook setup can be done centrally, if requested by the instructor
• Professors (only) to associate graduate attribute indicators with each graded item
• Professors (or TAs) are responsible for entering data (grades) in myCourses at a level that is sufficiently refined to distinguish between different graduate attribute indicators
• Analysis of the data will be done centrally
12
myCourses Setup Options Two options to setup myCourses gradebook
1) The gradebook setup can be done centrally, if requested by the instructor
o Information is submitted through a general questionnaire
2) The gradebook setup can be done by the instructor or TAs
o One-on-one consultation with professors will be available
o Hands-on TA training will be offered in mid-October
13
14
Pre-Setup Tasks • Define the role of your TA:
o http://kb.mcgill.ca/#tab:homeTab:crumb:8:artId:4131:src:article
15
Pre-Setup Tasks (cont.) • Define the gradebook rules:
Student View Display Options How to treat ungraded items
16
Information for General Setup • The assessment tools for your course • The grade distribution
17
Information for General Setup (cont.) • The grade distribution within each category:
Example:
18
Question#1 Graded out of 20 10% 4.50%Question#2 Graded out of 20 10% 4.50%Question#3 Graded out of 20 10% 4.50%Question#4 Graded out of 60 30% 13.50%Question#5 Graded out of 80 40% 18%
100% 45%
Final exam45%
Distribution based on grade points
• Even distribution
• Distribution by grade point
Assignment#1 Graded out of 20 20% 1.80%Assignment#2 Graded out of 80 20% 1.80%Assignment#3 Graded out of 80 20% 1.80%Assignment#4 Graded out of 50 20% 1.80%Assignment#5 Graded out of 100 20% 1.80%
100% 9%
Even distribution of gradesAssignments
9%
Information for General Setup (cont.)
19
Determine the Graduate Attribute indicators for each component within a graded assessment tool
General Setup
20
Step 1 – Create the assessment tool categories
Step 2 – Create the sub-categories (items) & linking the Graduate Attribute indicators
Graduate attribute indicators
Step 1 – Create Assessment Tools Categories
21
Example
22
Step 2 – Create the Sub-categories (items) & Linking the Graduate Attribute Indicators
23
Example Case1: The grade distribution for the Final Exam is based on the grade points
Case2: The grade for the Assignments category is evenly distributed between the items
24
Final Exam Question 1: • Graded out of 50 • Graduate attribute indicator: KB.7 • KB.7 - Recalls and defines information, first principles
and concepts in specialized engineering science
Question 2: • Graded out of 100 • Graduate attribute indicator: PA.2 • PA.2 - Develops models from first principles to analyze
complex engineering problems
Question 3: • Graded out of 80 • Graduate attribute indicator: PA.4 • PA.4 - Critically evaluates the validity and accuracy of
existing solution methods
Assignments Assignment 1: • Graded out of 10 • Graduate attribute indicator: KB.3 • KB.3 - Recalls and defines information, first principles
and concepts in the natural sciences
Assignment 2: • Graded out of 100 • Graduate attribute indicator: PA.1 • PA.1 - Identifies and formulates complex engineering
problems
25
Assignments Assignment 1: • Graded out of 10 • Graduate attribute indicator: KB.3 Assignment 2: • Graded out of 100 • Graduate attribute indicator: PA.1
Final Exam Question 1: • Graded out of 50 • Graduate attribute indicator: KB.7 Question 2: • Graded out of 100 • Graduate attribute indicator: PA.2 Question 3: • Graded out of 80 • Graduate attribute indicator: PA.4
Gradebook
26
Remarks
• Every course will be different, this is just an example
• Inclusion of question weights is critical to the analysis
• Prof/TA enters grades on myCourses
• Analysis of the data will be done centrally
27
Exceptional Cases - Using Excel
Excel can be used exceptionally • Courses that require more flexibility than myCourses can offer,
e.g. FACC 100, Capstone design courses
o Proper formatting is required
o Professors are responsible for calculation of grades for each
graduate attribute indicator
o Professors are responsible for the analysis of data
28
Excel Setup
• Example for the proper structure o First spreadsheet shows grades for all “assessment tools” and
the associated indicators • Similar structure as in myCourses is expected, e.g.
o Each attribute is summarized in its own individual worksheet • Each indicator is recorded and presented individually • An overall calculation for the graduate attribute is recorded
29
Format – Excel Excel spreadsheet showing grades for all “assessment tools” and the associated indicators
Different indicators are evaluated on 1 assessment
Different indicators are associated with different assessments
Different questions on the exam are associated with different indicators
Each attribute is summarized in its own individual worksheet
30
Excel - Data Calculation
A typical Graduate Attribute
Each indicator is recorded and presented individually An overall calculation
for the graduate attribute is recorded.
31
Excel – Data Analysis
32
Additional approaches • Departments will need to add to the general approach to:
o Acquire sufficient quality data for the “soft skills” graduate
attributes
o Use alternate GA evaluation approaches to satisfy (the unclear)
CEAB requirements regarding types and variety of measurement
tools
o Maria Orjuela-Laverde (Academic Associate, Pedagogical
Coordinator) is available for pedagogical assistance to develop
alternate GA evaluation methods
33
Additional approaches (cont.)
http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx 34
Additional approaches (cont.)
http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx 35
Recommendations • Recommendations to departments:
o Examine your curriculum maps to look for “holes” in the data
o Start by focussing on “Intro to XXX Engineering” course (or its closest
approximation) and capstone design courses to complement your general
approach for GA data acquisition
o List GA indicator codes (e.g. “KB.1,” “PA.2,” etc.) for each question/exam
sheet for all graded items (assignments, labs, exams)
o List learning outcomes on all course outlines (a requirement for the
revised course information sheets!)
o Remember to collect appropriate sample materials for your courses.
36
Overview Department
Program
General approach - infer GAs from assessment tools in courses (i.e. correlating grades with individual GAs)
Alternate approach - ensure good coverage of GAs is obtained - used to evaluate GAs by other means
Excel (analysis is the responsibility of the prof. / dept.)
myCourses (most courses, Faculty does the analysis)
37
Learning Outcomes Course Information Sheet
39 http://www.engineerscanada.ca/sites/default/files/intro_team_chair_presentation_english-sep19.pptx
Course/Lab Learning Outcomes
• “What the student is expected to have learned - These should be analogous to indicators and may have been used for GA assessment purposes. A small number of outcomes (general measurable skills acquired) are expected - not a list of content.”
• “Laboratory learning outcomes - These should be analogous to indicators and may have been used for GA assessment purposes. What are the important specific skills and techniques the student is expected to have acquired from the laboratory experience - not a list of content.”
40
41
Faculty assistance • Setting up gradebook templates in myCourses • Analyzing and processing the data acquired via myCourses
• Providing support (Nasim and Maria) to both professors and TAs • Preparing a survey for students in capstone design courses • Obtaining GA data
o in common FACC courses o via exit survey of graduating students
• Establish a faculty-wide “program improvement” procedure based on GA data
42
Misc. comments
• Some courses not on myCourses
• For a “standard” course, the process has been designed so that the
additional work required by a professor is small (as low as 2 hours
per course)
• Subsequent years will require less work (e.g. myCourses
gradebook template can be imported from previous years)
43
Resources
• Website: (https://www.mcgill.ca/engineering/faculty-staff/accreditation)
o List of indicators
• Video clips o How to setup your gradebook
• Instructions
o How to setup your gradebook
• TA Workshops
o Two sessions were offered in September o Hands-on sessions will be offered in mid-October
44
Thank you!
Keywords for Learning Outcomes
• In–Depth Knowledge • Complex Engineering Problems • First Principles • Research
Questions to consider for these GAs: • Does the course or components of the course:
o address complex engineering problems/open-ended problems? o require the use of in-depth knowledge of mathematics, natural sciences, or
engineering fundamentals? o require the use of in-depth knowledge of discipline-specific engineering? o require the development of solutions from first-principles? o require research for the development of solutions? o require research as an investigative tool?
46
The Affected Graduate attributes: KB: Knowledge Base PA: Problem Analysis IN: Problem Investigation DE: Engineering Design CS: Communication Skills