Experiences on Course Development Practices in a Power System Analysis Master Course a.k.a. tales of a journey through the valley of death in providing.

Experiences on Course Development Practices in a Power System Analysis Master Course a.k.a. tales of a journey through the valley of death in providing a high quality learning environment Assoc. Prof. Dr. Ing. Luigi Vanfretti and Maxime Baudette KTH SmarTS Lab [email protected] KTH Scholarship of Teaching and Learning Stockholm, March 12 th 2015.

Acknowledgment There has been a lot of former students involved in this efforts, through teaching assistant duties and beyond, which we cant list here but you know who you are, and we thank you very much! A very special thanks is expressed to Mostafa Farrokhabadi, who was the first student helping with the research work related to this talk until Maxime Baudette got involved.

Outline Motivation Constructive Alignment Theory Implementation: T&L activities, Assessment Activities Evaluation of the implementation: Feedback, R- SPQ-2F, Ranking Algorithm Filtering Feedback: Repertory Grid Introducing Peer Instructions with Clickers

EG2100 in Brief Introductory course on Power System Analysis 6 ECTS (7.5 ECTS before 2014) Given on 2 periods (September to December) About 85 students The course is divided in five important topics: Modern Power Systems with an Introduction to Sustainable Energy Technologies and Smart Grids Fundamental Principles for Power System Analysis AC circuits Electrical Modeling of Generators, Transmission Apparatus and Networks Methods for Analysis and Design of Power Networks in Steady State and Unbalanced Operation Steady State Stability Analysis using the Power-Flow Formulation Methods for Analysis of Power Distribution Systems in Steady State

Motivation The previous course (until 2010) that was replaced by this one had several drawbacks in its pedagogic design and material: The course material was outdated, inconsistent and lacked appeal. No teaching method was used to encourage the learning process The failure rate of the course was very high These drawbacks required a complete overhaul of the course: from preparing new material, to the design of a new implementation using teaching methods. I never teach my pupils, I only attempt to provide the conditions in which they can learn. A. Einstein

How to address a large variety of background? Students come with a very different background, and often lack any exposure to power systems at all: Mandatory for M.Sc. Program in Electrical Power Engineering (TELPM), and M.Sc. Joint Program in Smart Electrical Networks and Systems (SENSE) Elective in other M.Sc. Programs at the school of Electrical Engineering and some Erasmus Mundus M.Sc. Programs Also offered to power and energy professionals within industry collaborations and PhD students (including those from other universities) This variety in the students background makes it challenging to design well aligned and adequate T&L activities and proper assessment methods. Students Background

Redesign of the Course - Goals In 2011 it was decided re-design the course to achieve the following goals: Increase the success rate Largely increase the depth and moderately the coverage of the course content Link the course to the current developments in the industry and academia, including the Smart grid concept Re-Design Approach: Implement the constructive alignment theory (CAT) as the didactic teaching approach of choice for the course Adopt the consensus-oriented decision-making (CODM) model to design the course Develop effective multiple feedback channels to gather the students perception and select relevant feedback

Constructive Alignment Basic Principles (Biggs and Tang) Basic principle: The theachers fundamental task is to get students to engage in learning activities that are likely to result in their achieving the desired outcomes in a reasonably effective manner. What the student does is more important in determining what is learned than what the teacher does. Constructive Alignment Theory: Provides a set of principles These principles can be used to devising Teaching and Learning Activities (T&Ls) The activities can help in achieving the Intended Learning Outcomes (ILOs).

Using Principles of Constructive Alignment Using the principles from constructive alignment: Start by carefully aligning T&LAs and Assessments These activities should to support the students to fulfill the ILOs. The students role (MAJOR): The students use the activities to construct their knowledge and achieve desired outcomes. The teachers role (minor): To design a learning environment that encourages the student to perform the T&Ls that aid the students to construct their knowledge. Course Objectives (Intended Learning Outcomes) Teaching and Learning Activities Assessment What should the students do to demonstrate that they reached the objectives? What work is appropriate for students to do to reach the ILOs? What should the student be able to do as a result of the course?

Variety of Background Affects the proper implementation of CAT Affects students readiness to engage in the Teaching and Learning Activities Diagnostic test results show a low average in the students preparation Fundamental knowledge varies a lot. Question % of correct answers 199 240 390 470 553 684 759 862 969 1027 1132 1284 1369 1453 1577 1690 1767 1863 1967 2074 2144 2268 2351 Total64.8

CODM Design Process (Hartnett) Links the design group to the consensus model. Five 2-hour sessions (10 hrs) Allows to deliver a shared proposal: Important if the design group will also be the implementation group (needed commitment). Closing Framing the Problem Open Discussion Identifying Underlying Concerns Developing Proposals Developing a Preferred Solution Choosing a Direction

Design Rule for CAT Implementation Example of using the consensus based model Step 1 (Framing the problem): design rule used for one of the T&L activities using Cause-and-Effect analysis.

T&L Activities 1: Lectures (22), Invited Lecture (4) 2: Reading Quiz, Conceptual Quiz 3: Homeworks (3) 4: Tests (2) 5: Final Exam Course Objectives (Intended Learning Outcomes) Teaching and Learning Activities Assessment [1], [2] [5] [3],[4]

T&L Activities - Lectures Lectures are intended to develop conceptual understanding:

T&L Activities- Invited Lectures Invited Lectures are intended to develop the interest of students by providing insights from the industry and top researchers from Academia. Sample of Invited Lectures Magnus Danielsson. System Design, R&D, Net Insight AB, Stockholm, Sweden. Lecture on Communication Issues in Smart Grid Applications Svein Harald Olsen, Statnett SF, Oslo, Norway. 2 Lectures on IEC CIM Sonja Berlijn, SVP R&D, Statnett S, Oslo, Norway. Lectures active R&D projects (Lean Line + Voltage upgrading) David Petesch/Yannick Fillon, RTE - National Center for Grid Expertise, Paris La Dfense France Lecture on EMTP and Real-Time Simulation at RTE Prof. Federico Milano, UCD Challenges for power system modelling and simulation

T&L Activities- Daily Reading Quiz, Conceptual Quiz In total, they are worth 10 BONUS points of the final grade. Reading Quiz consists of some very basic questions about the content of the lecture The questions are designed in a way that a student should go through the lecture content before arriving to the lecture! Peer Instruction Implementation Conceptual Quizzes asked about the lecture content during the lecture. The questions addressed different learning levels, focusing on developing deep learning. They sometimes involved a small amount of numerical calculations. The students answered the questions using a remote clicker.

T&L Activities Homeworks For the students to practice the methods and study the concepts covered during the lectures. There are 3 Homeworks (30 points) covering the contents learned in the respective lectures. The homework are aimed to prepare the students for the assessment tasks (tests and final exam). In 2014 computer assisted assignments were introduced. Classroom Home!

Assessment Tests and Final Exam 2 Tests during the semester. Final exam The tests and exam were built in two parts, with theory questions in the first part and problems in the second part. The problems were similar to those solved in the homework. The assessment of the course is done though the different graded activities of the course. All the grades are added to give a total grade on 100 points Course ActivityPoints Daily Quiz10 (Bonus) Homeworks30 Tests30 Final Exam40 Total100 (+10 Bonus points) Total PointsGrade Letter 94 100A 87 93B 81 87C 74 80D 65 73E 60 64Fx 00 59F

Evaluation of the implementation Three Course Evaluations One after each test (end of P1 and end of P2) One after the final exam R-SPQ-2F Evaluation (at the beginning of the course) Two different question types (surface and deep approach) Students are graded using their answers The results will reveal the students approach Interviews using the Repertory Grid Technique A form of structured interview to find out a participants preference on a given topic and the way these preferences are ordered on a rating scale Allows to link qualitative to quantitative feedback.

Qualitative DataInstructions on Form Explanation on KTH Social Quantitative Data

Sample Feedback using Google Forms Quantitative Feedback Qualitative Feedback

Course Evaluations 2011-2014 After 2011, the form was improved for including an introductory part with instructions, explaining the students the grading scale they should use.

Course Evaluations 2012-2014

Course Evaluations - YOY improvements The feedback provided by students that retook the course shows that the changes improved the course. The acceptance of the changes comes after the students realize that it has a positive impact on their learning. Note: very reduced population for statistical analysis Sample Question Is the new course structure (lectures, daily exercises, weekly exercises, and weekly tests) preferable to the previous course structure (lecture, assignments, and final examination)?

Activity No. 1 Design of a course evaluation process Instructions: Discuss in small groups (no larger than 3) the following questions. Using your answers, propose a design to evaluate a sample course. What feedback do you need for your particular course? T&L activities: do they contribute to the students understanding? Do the T&Ls have an impact on the Assessment Activities? In what form do you want it and how much? Quantitative, Qualitative, Both How do you gather it in an efficient way? What tools do you use. How often do you gather it? Number times and type of feedback to be gathered (a) that allows to trace changes, (b) of new information (originating from the changes). How do you use it? How do you show the students that their opinion is considered.

The Revised two-factor Study Process Questionnaire: R-SPQ-2F Allows to determine the students learning approach The process is exposed to some drawbacks!

R-SPQ-2F: A new ranking algorithm Deep Learning Surface Learning 1 st Quarter 2 nd Quarter 3 rd Quarter 4 th Quarter

Grade distribution and its correlation with students learning approach

Filtering Feedback: Repertory Grids The repertory grid method is a person-centered approach that allows the student to develop their own constructs using their own perception of the subject. Repertory grids serve as a feedback mechanism where the students can assess course elements using well-defined constructs and considering different scales. The construction of each grid allows a methodological analysis of the grids to identify constructive feedback to improve the course: a formal quantitative method to select and discard feedback. Good GridBad Grid Good Feedback Further Analysis Bad Feedback Discard

Implementation of repertory grids The elements of the grid were determined by the T&L activities. The constructs were built from interviews with some students A small group of students (about 10%) was carefully selected to represent the different learning approaches Each student was interviewed individually about the different elements (qualitative data collection) The constructs were identified by analyzing all interviews An empty grid was sent to each student who should fill it in

Repertory grid analysis Different software provide methods for analysis of the grids (Rep IV, Idiogrid), such as: Principal Component Analysis (PCA): allows better interpretation of feedback. Cluster Analysis: re-orders elements in a tree, similar objects are placed together. Allows to interpret the feedback. Descriptive analysis (Mean, Mode)

Activity No. 2 Feedback Analysis Instructions: Assume that you have a high volume course (many participants). In this activity, you should analyze the method for feedback analysis presented, and propose alternatives to relate the students to their learning approach and classify which feedback to address in a systematic (reproducible) way. Are there other alternatives (methods) to analyze the course feedback and relate it to the students learning approach? How do you check if your implementation of CAT is working? How can you use the proposed method to filter the different feedback and choose which to address?

Peer Instruction

Peer Instruction and the use of Clickers Reading Quiz and Conceptual Quiz: To implement peer instruction there is very good technology available These are called clickers and are used for gathering students feedback in real-time. Clickers were given to each student and kept by them throughout the course.

A flexible tool for different uses! In the EG2100 course, clickers are used for the following: Unique identifier for each student Attendance registration during Lectures Pre-requisite test at the beginning of the course (not graded) In-class questions: Tests on the mandatory pre-readings (5 questions) Questions during the lecture (~5 questions) Rewarded by bonus points

In-class interactions Tests on the mandatory pre-readings (5 questions) Questions about the content of the pre-readings Regular questions, all done in a row Questions during the lecture (~5 questions) Questions about the subject being taught Regular questions, often one by one If average results are obtained, students are invited to discuss with their neighbors and answer the question once again (peer instruction) If bad results were obtained, the lecturer re-explained the material.

Preparation of Teaching Materials for Peer Instruction Material has to be prepared to introduce the peer instruction as a change of activity. The lectures need to be prepared in 10-15 min modules Questions need to be prepared to assess deep learning (or surface when needed) Instead of a review at the end of the lecture, peer instruction allows for review of deep learning concepts during the lecture. This has a cost: Your time! Think if you can afford it before doing it!

Pros/Cons Pros: Greatly appreciated by the students: Higher attendance and better preparation of the students Better in-class attention Helps the teacher to get direct feedback (e.g. unclear notions) Flexible ! Cons: Additional Data Management Technical issues may happen in class (crash, bad reception, empty battery, etc.) Different lecturing style (might be hard to accept by students in early phases)

Students feedback about the clickers The feedback received is very positive and encouraging for continuing this peer-instruction implementation The mandatory pre- readings helped your understanding of the course material. It was helpful to discuss your answers with your classmates (peer instruction) The daily quizzes helped you keeping up to date with the course materials. It was fun to use the clickers. It helped you be more attentive in class

Activity No. 3 Peer Instruction Method Using Clickers Instructions: Assume that you have already your teaching prepared from previous course deliveries. In this activity, you should analyze how to modify the course materials to adapt it to the peer instruction method using clickers. Discuss the following questions, provide a proposal on how to implement peer instruction and an example slide for your particular course? At what learning depth level do you intend to use peer instruction? When using the peer instruction method, what factors do you need to consider so that you can use clickers to obtain quick feedback of the students understanding of a particular topic? How do you transform (timing, amount of material and content) your course material to adapt it to the peer instruction method?

Highlights and Words of Caution A wider variety of stakeholders can participate in CODM process this would allow for a very good course design addressing different points of view! CAT is a good vehicle to enhance the students learning. Its implementation has to be considered carefully. Implementing CAT is extremely expensive DO NOT BELIEVE the propaganda that says the contrary! R-SPQ-2F new ranking algorithm efficiently classifies the students learning approach depth. Useful for sampling feedback from students with different learning approaches. The two main challenges for the course designers are: 1)To continuously modify the course so the surface approach students move toward deep approach. 2)To do carry out (1) when there is absolutely no benefit to the teaching staff (a lot of work that brings no reward)

Discussion (1/2) Continuity and Effort Continuity: The teaching staff changes from year to year depending on the availability of PhD students (Luigi is always involved and the only one to provide continuity) Course planning based on consensus requires continuity, in order to focus on improvements instead of challenging the whole structure Effort: Workload and Time The T&L activities require a large workload for the teaching staff to prepare and correct the assignments (very expensive and not properly funded) Course planning based on consensus requires time and involvement from the teaching staff The feedback process requires a lot of time for gathering and analysis of the results Integration of clickers in the lectures requires a lot of time (very expensive!) in order to provide a good experience for the students (questions should be well organized and well formulated)

Discussion (2/2) Systemic factors: brace yourself to a large inertia and resistance to change Students and other faculty will oppose different didactic approaches to those that they are used too you will face a large resistance to change. Not having a single exam to be able to pass the course has been criticized and brought up to the program coordinator year after year This attitude is hard to change in the 4 th year. However, we have shown that the acceptance of the students improves over time when they are able to reap the benefits of a good learning environment. You will waste your time with complaints from the student union, program coordinators, department head just for trying to do things differently (I would say BETTER!) There is no means to enforce student feedback at KTH, this weakens statistical analysis and the amount of samples. Changes in rules or flexibility is needed if KTH wants to take education seriously. No pain, No gain? More like lots of pain, and no gain! The one most important thing we have learned is that there is no reward for good teaching in KTH do it at your own risk and for your own self development. The only benefit you could obtain, by investing even more time, is to try to generate some publications for your teaching portfolio (in case you want to find somewhere else to teach with quality).

Even though this has been a wonderful adventure in teaching and learning, all good things must come to an end. Luigi cant afford to continue to subsidize the delivery of the course by using time from his PhD students (which translates to use the funds from research projects!) to deliver the course. A loss of 290 000 SEK was generated in 2014. A good learning environment comes at a cost: we do not get paid enough to deliver a high quality learning environment. Luigi will not longer be the responsible for this course nor involved in it, and with that goes the 4 years of experience But at least we enjoyed the walk through the valley of death, and we have learned a lot about how to deliver a high quality learning environment! Epilogue A tale just like that of the one ring

Thank you! It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them. Steve Jobs Questions? [email protected] Good luck with the rest of your day!

References [1] L. Vanfretti and M. Farrokhabadi, Consensus-Based Course Design and Implementation of Constructive Alignment Theory in a Power System Analysis Course, European Journal of Engineering Education, 2014. https://dx.doi.org/10.1080/03043797.2014.944101 [2] L. Vanfretti and M. Farrokhabadi, Evaluating Constructive Alignment Theory Implementation in a Power System Analysis Course through Repertory Grids, IEEE Transactions on Education, vol. 56, no. 4, pp. 443 452, Nov. 2013. DOI: http://dx.doi.org/10.1109/TE.2013.2255876 [3] L. Vanfretti and Federico Milano, Facilitating Constructive Alignment in Power System Engineering Education using Free and Open Source Software, IEEE Transactions on Education, vol. 55, no.3, August 2012, pp. 309-318. DOI: http://dx.doi.org/10.1109/TE.2011.2172211 [4] S. Taylor, Managing postgraduate research degrees. In H. Fry, S. Ketteridge and S. Marshall (eds) The Effective Academic: A Handbook for Enhanced Academic Practice. London: Kogan Page. [5] S. Taylor and N. Beasley. A handbook for doctoral supervisors. Routledge, New York, 2005.

Course Objectives (Intended Learning Outcomes) After completing the course, the students should be able to: Using first principles derive the basic concepts and methods used for power system analysis. Describe the methods used for modeling and simulation of different power system phenomena, and Explain how computer methods affect the modeling process Construct mathematical models for computing the steady state performance, and basic unbalanced performance of power systems. Describe the use of the Unified Markup Language for data modeling of power system components, and interpret the Common Information Model and its relation with the power system models used in class. Describe, derive, and compare different models of the most common equipment used in power network models. Using different methods, to compute, analyze, and reflect on the performance of a power system under steady state and unbalanced operation, for transmission and distribution networks Apply a power system and general purpose software tools to analyze the performance of a power system, and evaluate it for different operating conditions Contrast different software development methods that are used in the creation of power system analysis software Using first principles derive the basic concepts and methods used for power system analysis. To construct data models and mathematical models for computing the steady state performance, and basic unbalanced performance of power systems. To derive, describe and compare different models of the most common equipment used in power network models. Using different methods, to compute, analyze, and reflect on the performance of a power system under steady state and unbalanced operation To describe basic characteristics of renewable and distributed energy sources, as well as monitoring, analysis and control technologies used in Smart Grids.

Evaluation Diagnostic test at the beginning of the course (non graded) Raise student awareness about their individual level of required background In-class questions are rewarded with a very flexible bonus point system Technical issues require to have a flexible system Any evaluation activity requires to maintain a database of students

Experiences on Course Development Practices in a Power System Analysis Master Course a.k.a. tales of a journey through the valley of death in providing.

Documents

course material

course goals

previous course

design of power networks

students background

modern power systems

brief introductory course

course development practices