Program development process at Queen’s University to demonstrate graduate attributes Brian Frank Director (Program Development) Faculty of Engineering.

Program development process at Queen’s University to demonstrate graduate attributes

Brian Frank

Director (Program Development)Faculty of Engineering and Applied Science

Queen's University

1

Focus

“The institution must demonstrate that the graduates of a program possess the attributes under the following headings... There must be processes in place that demonstrate that program outcomes are being assessed in the context of these attributes, and that the results are applied to the further development of the program.”

2

CEAB InstructionsDescribe the processes that are being or are planned to be used. This must include:a) a set of indicators that describe specific abilities expected of

students to demonstrate each attributeb) where attributes are developed and assessed within the

program…c) how the indicators were or will be assessed. This could be

based on assessment tools that include, but are not limited to, reports, oral presentations, …

d) evaluation of the data collected including analysis of student performance relative to program expectations

e) discussion of how the results will be used to further develop the program

f) a description of the ongoing process used by the program to assess and develop the program as described in (a)-(e) above

3Engineering Graduate Attribute Development (EGAD) Project

Approach• Short term objectives (2010-2011):

• Set up a comprehensive process limited to a small number of courses to help programs understand the process

• Use data to help faculty see value in outcomes assessment for program improvement

• Long term: • Comprehensive assessment of all attributes

throughout programs• Evaluate validity of data• Students take responsibility for demonstrating

some attributes

Queen's University timeline Summer 2009: Working groups of faculty, students,

topical experts created specific program-wide indicators (next slide, and in Appendix 3.1A)

Summer 2009: Setup learning management system (Moodle) to manage assessments

Sept 2009-April 2010: Piloted assessment in first year

Sept 2010-April 2011: Piloted assessment in first year, faculty wide second year, and fourth year (common across programs)

April – July 2011: Student surveys and focus groups, curriculum mapping, data analysis

Curriculum planning happening throughout

Why initial emphasis on first year?

• First year is faculty-delivered, core to all students

• Provides opportunity to pilot a process• Help disseminate outcomes

assessment procedures to other instructors

• Long term: assessment process continue in first year program to inform development

Aside: Idealistic course development process


Identify courseobjectives and

content

Create specific outcomes for each

class

Map to experiences(lectures, projects,

labs, etc.)

Identify appropriatetools to assess

(reports, simulation,tests,...)

Student input

Deliver, grade, seek feedback

Analyze and evaluate data

OverallImprovement

Create and Execute a Plan

Program-wide assessment process flow


Defining Purpose and Outcomes

ProgramMapping

Stakeholder input

Identifying and Collecting Data

Analysis andInterpretation

Create a ProgramImprovement Plan

Program & CourseImprovement

Human capital

• Director, Program Development to manage process

• Faculty member from each program• Other experts as appropriate

(economics, information management, etc.)

Currently separate from faculty-wide curriculum development committee

9

Resources/time commitment Creating assessment criteria: 7 committees

of approximately 5 people who each met about 4 times

Mapping criteria to a course and creating rubrics for assessment: ~ 10 hours

Large scale curricular changes: ~10 person committee, most of whom had 1 course relief bought out by dean

Coordination (resource gathering, planning, curricular planning): ~30% of a position

Academic and curricular structure

Dean

Associate Dean (Academic)

Director (Program Development)

NSERC Design Chair

DuPont Canada Chair in Engineering Education

Faculty-wide curriculum committee

Dean’s Retreat Curriculum Review Committee(DRCRC)

Graduate attribute assessment committee

What are indicators?

Engineering Graduate Attribute Development (EGAD) Project 12

Lifelong learningAn ability to identify and address their own educational needs in a changingworld in ways sufficient to maintain their competence and to allow them to

contribute to the advancement of knowledge

Can this be directly measured?

Would multiple assessorsbe consistent?

How meaningful would the assessment be?

Probably not, so more specific measurable indicators are needed.This allows the program to decide what is important

Indicators: examples

Engineering Graduate Attribute Development (EGAD) Project 13

Lifelong learningAn ability to identify and address their own educational needs in a changingworld in ways sufficient to maintain their competence and to allow them to

contribute to the advancement of knowledge

Critically evaluates informationfor authority, currency, and

objectivity when referencingliterature.

Uses information ethically and legally to accomplish a specific purpose

Identify gap in knowledge and develop a plan to address

Graduateattribute

The student:

Describes the types of literature of their field and how it is produced

Indicators

Establishing Indicators

• A well-written indicator includes:• what students will do• the level of complexity at which they will do it• the conditions under which the learning will

be demonstratedEngineering Graduate Attribute Development (EGAD) Project

14

Critically evaluates information for authority, currency, and objectivity in reports.

Content areaLevel of expectation(“describes”, “compares”, “applies”, “creates”, etc.)

context

Graduate attribute

categories

levels

Assessment criteria


Linkage to OCAV UDLEs

Rubric example

Creating defined levels (“scales”) of expectations reduces variability between graders, makes expectations clear to students

threshold target

Sample First year indicators for problem analysis and design


3.02-FY1 Identifies known and unknown information, uncertainties, and biases when presented a complex ill-structured problem

3.02-FY2 Creates process for solving problem including justified approximations and assumptions3.02-FY3 Selects and applies appropriate quantitative model and analysis to solve problems3.02-FY4 Evaluates validity of results and model for error, uncertainty3.03-FY1 Generates ideas and working hypothesis3.03-FY2 Designs investigations involving information and data gathering, analysis, and/or

experimentation3.03-FY3 Synthesizes data and information to reach conclusion3.03-FY4 Appraises the validity of conclusion relative to the degrees of error and limitations of theory

and measurement3.04-FY1 Adapts general design process to design system, component, or process to solve open-

ended complex problem.3.04-FY2 Accurately identifies significance and nature of a complex, open-ended problem3.04-FY3 Identifies customer and user needs3.04-FY4 Gathers and uses information from appropriate sources, including applicable standards,

patents, regulations as appropriate.3.04-FY5 Produces a variety of potential design solutions suited to meet functional specifications3.04-FY6 Performs systematic evaluations of the degree to which several design concept options

meet project criteria3.04-FY7 Compares the design solution against the problem objective

Sample fourth year indicators for Problem analysis and Design


3.02-GY1 Identifies problem, known and unknown information, uncertainties, and biases

3.02-GY2 Creates process for solving problem including justified approximations and assumptions

3.02-GY3 Selects and applies appropriate model and analysis to solve problems

3.02-GY4 Evaluates validity of results and model for error, uncertainty

3.04-GY1 Identify problem and constraints including health and safety risks, applicable standards, economic, environmental, cultural and societal considerations

3.04-GY2 Applies appropriate knowledge, judgement, and design tools, in creating and analyzing conceptual design solutions to select best concept

3.04-GY3 Creates and tests simulations, models, and/or prototypes at various points in design with complexity appropriate to design stage

3.04-GY4 Assesses design performance based on requirements, yield, reliability, and/or safety as appropriate

3.04-GY5 Identifies possibilities for further improvement and conducts design review to evaluate performance of the overall process.




ProgramMapping

Stakeholder input





Curriculum mapping

20

21

22

A knowledge base for e

ngineering

Communication sk

ills

Design

Economics

and project

management

Ethics and equity

Impact

of engineerin

g on socie

ty and the enviro

nment

Individual and te

am work

Investigation

Life-lo

ng learn

ing

Problem analysis

Professi

onalism

Use of engineerin

g tools

0

50

100

150

200

250

Development of Enginering Attributes

Not Taught / Not Assessed

Not Taught / Assessed

Taught / Not Assessed

Taught / Assessed

Student surveys and focus groups

• Provides student input:• implementing attribute assessment in

program• perceptions on where attributes are

developed within the program as complement to curriculum mapping via faculty survey

• perception of importance within program

Questions

• What do you think are priorities within the program?

• What courses contribute to development of attribute {}?

• Which attributes are difficult to demonstrate?

• How would you recommend that attributes be developed?

Self reported demonstration at program entry

Top five Grad Attributes where students reported a rating of 2 or 3 (yes or to a great degree) out of 3 Individual and Team Work 88.73% Communication Skills 78.17% Professionalism 69.02% Problem Analysis 61.26% Investigation 60.56%

Potential for students to perceive little value in learning activities directed toward developing these attributes

First year program supports:

27

Attributes in students’ top five responses Individual and Team Work* 94.97% Knowledge Base in Engineering 93.53% Problem Analysis* 93.53% Professionalism* 85.58% Investigation* 82.48% Design 80.58% Impact of Engineering on Society 80.58% *Identified as a strength coming in to the program

First year program supports

28

Bottom three responses Ethics and Equity 64.03% Economics and Project Management 69.56% Lifelong Learning 73.19%

These three are a significant focus in APSC-100, embedded in various activities.

Attributes perceived to be program priorities

29

Attribute # of students who incl it in top 5

# of students who selected it as #1 priority

Problem Solving* 120 34

Individual & Team Work* 97 17

Knowledge Base 95 51

Communication* 76 6

Professionalism* 47 4

Graduating students: low priority attributes in program

30

Attribute # who included it in bottom 3

# who ranked it lowest priority

Lifelong learning 80 38

Economics and Project

Management

68 25

Ethics and Equity 45 1

Use of Eng Tools 38 15

Impact on Society

37 6

Focus group suggestions• Communicate graduate attributes and draw

attention back to them• What is lifelong learning”?• Professionalism and ethics and equity should be

focused on in upper years




ProgramMapping

Stakeholder input





Assessment in 2010-2011Attribute First year Upper yearsKnowledge base FCI, CalculusProblem analysis Complex problem

solvingCapstones

InvestigationDesign Design project CapstonesEngineering tools Complex problem

solvingCapstones

Communications Design project Communications, Capstones

Individual and teamwork

Design project Capstones

Impact of engineering Design project CapstonesProfessionalism Complex problem

solvingCapstones

Ethics and equity Complex problem solving

Capstones

Economics EconomicsLifelong learning Design project Capstones 33

Analyze and evaluate… Histogram of results by level (did or

did not meet expectations) Histogram of results by student (how

many indicators did each student fall below

Trend over time Triangulation: examination of

correlation between results on multiple assessments of the same indicator data with exam results)

First year:Second year of pilot

Threshold Target 1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding

3.02 - FY1: Identifies known and unknown information, uncertainties, and biases when presented a complex ill-structured problem

Information not identified properly, no information, or information copied from assignment

Some important information or biases not identified, or trivial/incorrect information included

Identifies known and unknown information, uncertainties, and biases

Meets expectations PLUS: Includes information from authoritative sources to inform process, model, and conclusions

3.02 - FY2: Creates process for solving problem including justified approximations and assumptions

No or inadequate process

Process identified misses some important factors; some assumptions left unidentified or unjustified.

Creates justified process for solving problem, suppored by information.

Meets expectations PLUS: Comprehensive process model; comparison with other possible approaches

3.02 - FY3: Selects and applies appropriate quantitative model and analysis to solve problems

No analysis, or model/analysis selected is inappropriate

Model selected; some errors in analysis or inappropriate assumptions

Selects and applies approriate quantitative model and MATLAB analysis to solve problems, using reasonable approximations and assumptions

Meets expectations PLUS: Authoritative research used to defend assumptions and approximations made

3.02 - FY4: Evaluates validity of results and model for error, uncertainty

No evaluation of solution

Superficial evaluation of solution

Evaluates validity of results and model for error, uncertainty

Meets expectations PLUS: Evaluates conclusions and presents potential improvements

Threshold Target 1 - Not Demonstrated 2 - Marginal 3 - Meets Expectations 4 - Outstanding

3.07 - FY3: Summarizes and paraphrases written work accurately with appropriate citations

Insuffi cient content to assess summary of work. Summary misinterprets researched material.

Records information from few resources. Misses significant points of view.

Summarizes and paraphrases written work accurately.

Synthesizes main ideas to construct new concepts. Summarizes the leading thoughts in the field and gives a broader picture of the problem.

FEA

S - 3

.07-

FY3

(Pro

pRep

ortP

M)

FEA

S - 3

.07-

FY3

(Pro

pRep

ortF

S)

FEA

S - 3

.07-

FY3

(Pro

pRep

ortA

A)

FEA

S - 3

.07-

FY3

(T2S

cope

&In

fo)

APSC100 APSC100 APSC100 APSC100

0100200300400

1234

time

Knowledge base: MathematicsCalculus instructor asked questions on exam that specifically targeted 3 indicators for “Knowledge”:

1. “Create mathematical descriptions or expressions to model a real-world problem”

2. “Select and describe appropriate tools to solve mathematical problems that arise from modeling a real-world problem”

3. “Use solution to mathematical problems to inform the real-world problem that gave rise to it”

Indicator 1:

• The student can create and/or select mathematical descriptions or expressions for simple real-world problems involving rates of change and processes of accumulation (overlaps problem analysis)

39

Context: calculatingIntersection of two trajectories

Indicator 2:Students can select and describe appropriate tools to solve the mathematical problems that arise from this analysis

40

Context: differentiationsimilar to high school curriculum

Indicator 2:• Students can select and describe

appropriate tools to solve the mathematical problems that arise from this analysis

41

Context: implicit differentiation, triginverse




ProgramMapping

Stakeholder input

Collecting Data




2010-08 2010-09 2010-11 2011-01 2011-02 2011-04

10

20

30

40

50

2.000

2.200

2.400

2.600

2.800

3.000

3.200

3.400

3.600

3.800

4.000

% Below targetLinear (% Below target)MeanLinear (Mean)

Approximate deliverable date

Perc

ent b

elow

targ

et

Mea

n sc

ore

All first year indicators over time

0 1 2 3 4 5 6-10 11-15

16-20

21-25

26-30

31-35

36-40

41-50

0

50

100

150

200

250

300

350

400

2642

67 65 6245

228

100

4125 10 2 2 0

344

187

7346 38

7 200 0 0 0 0 0 0

Below target Below threshold

Number of indicators

Num

ber o

f stu

dent

s# Students falling below expectations in first year

Graduating year

Graduating year• Starting point: histograms• Very few students falling below

threshold level in capstone courses for most indicators

46

3.02

GY1

3.02

GY2

3.02

GY3

3.02

GY4

3.04

GY1

3.04

GY2

3.04

GY3

3.04

GY4

3.04

GY5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Not demMarginalMeetsOutstanding

Area for improvement in graduating year: technical literature

FEAS - 3.12-FY1

FEAS - 3.12-FY2

FEAS - 3.12-FY5

FEAS - 3.12-FY6

0

10

20

30

40

50

60

1 - Not Demonstrated 2 - Marginal 3 - Meets ExpectationsAttributes

Per

cent

age

(%)

47

3.12-FY1 Uses information effectively, ethically, and legally to accomplish a specific purpose, including clear attribution of Information sources.

3.12-FY2 Identifies a specific learning need or knowledge gap.3.12-FY5 Identifies appropriate technical literature and other information sources to meet a need3.12-FY6 Critically evaluates the procured information for authority, currency, and objectivity.

Data evaluation

• Across multiple capstone courses, students scoring lower on indicators involving:• Evaluating validity of results• Evaluating techniques and tools• Evaluating effectiveness of results• Evaluating information

• Pattern: evaluation

48

Curriculum Mapping: CurriKit

• Curriculum mapping software developed by U Guelph

• Provides information to identify:• the courses which develop each

graduate attribute• what assessment is done and when• which instructional approaches are used




ProgramMapping

Stakeholder input

Collecting Data




First year development • More focus on grader training and calibration• Problem analysis: greater focus on making an

effective argument• Design: the focus on safety and risk assessment• Communications: Weak communicators flagged

and supported in fall semester (EPT). Required resubmission/assistance for falling below threshold

• Lifelong learning: evaluating information• Ethics and equity and professionalism get stronger

emphasis

Development in upper years

• In 2012-2013 all programs will have an open-ended design experience in third year which will apply disciplinary tools and principles, and incorporate professional issues and communications

• In 2013-2014 capstone courses will be revised to strengthen professional skills (safety, role of engineers in protecting public safety, ethics, communications)

Program improvement• Students take more responsibility for

learning and demonstrating attributes• Faculty-wide curriculum development

• Engineering design and practice sequence

• Assessment built into sequence• Multi-disciplinary approach to

developing professional skills in an integrative experience

• Developing leadership through peer mentoring program

53

Process development• Improve common indicators• Develop disciplinary indicators• Triangulation: Indicators measured using

multiple methods or events to assess validity• Attributes measured at multiple times in

students' program with leveled expectations• Satisfy both CEAB and province (OCAV UDLEs

for Queen’s Quality assurance process)

54

End

Moodle

Sample course Moodle page

Moodle online assignments


Assignment upload


Outcome grading for assignments


Outcome grading - popup


Moodle development

Customizing Moodle for our purposes Group upload/grading of assignments Peer evaluation Class response system (“clickers”)

Future collaboration

Program development process at Queen’s University to demonstrate graduate attributes Brian Frank Director (Program Development) Faculty of Engineering.

Documents

year program

program outcomes

program expectationsdiscussion

outcomesbased assessment

comprehensive assessment

piloted assessment

program improvementlong

outcomes assessment