PROVOST’S LEARNING INNOVATIONS GRANTS 2015 APPLICATION · teaching and learning in terms of its potential to positively impact student outcomes and the ... Simon Buckingham Shum

PROVOST’S LEARNING INNOVATIONS GRANTS

2015 APPLICATION I. INSTRUCTIONS

Complete this form in its entirety and email it to [email protected] no later than January 26, 2015. Please note to save and rename this document substituting your name (in place of “NAME”) in the file name.

Ask your Department Head complete to complete the Department Head certification on page 12 and send a digitally-signed or printed, signed, and scanned copy with this application.

If you have any questions about completing this application, please email them to [email protected] or call Michael Starenko at 585-475-5035.

II. APPLICANT INFORMATION

Name: Rob Garrick Email: [email protected] Phone: 5-4288

College: CAST Department: MMET

Faculty rank: (full-time faculty only): Assoc. Professor

Department head name: S. Ramkumar

Proposed project name: Learning Analytic Analysis of Flipped Classroom Video Usage

Total funds requested: (requests of $1,000 to $5,000 will be considered): $4590

Administrative use: ❏ Focus Grant ❏ Exploration Grant

III. PLIG TYPES

Exploration Grants (approximately 30% of the funding pool for this cycle)

These grants provide seed funds for faculty to investigate an innovative mode or model of teaching and learning in terms of its potential to positively impact student outcomes and the student experience at RIT. These are funds for "proof of concept" investigations into the development, adaptation, or application of a new or different teaching approach, practice, or procedure.

Focus Grants (approximately 70% of the funding pool for this cycle)

Focus Grants provide funds for faculty to develop, apply, and/or research an innovative mode or model of teaching and learning that directly supports an RIT priority. These are funds for the development, adaptation, application, and/or research into a new or different teaching approach, practice, or procedure in the priority areas of focus.

The two focus areas for this cycle are:

Flipped Classroom

The flipped classroom model (sometimes called the inverted classroom) is one in which traditional in-class activity—the lecture—is delivered outside of class via recorded lectures and other web-based materials. In-class time is used for collaborative project work, small group problem-solving, and other such activities that allow students to engage at a deep level with the content they viewed outside of (and before) class. To learn more about the model, visit the Teaching and Learning Services webpage at rit.edu/tls/course-design/flipped-classroom.

Learning Analytics

According to the 2014 NMC Horizon Report, (http://www.nmc.org/publications/2014-horizon-report-higher-ed), “Learning analytics research uses data analysis to inform decisions made on every tier of the education system, leveraging student data to deliver personalized learning, enable adaptive pedagogies and practices, and identify learning issues in time for them to be solved.”

Simon Buckingham Shum notes: "Micro-level [learner-level] analytics support the tracking and interpretation of process-level data for individual learners. This data is of primary interest to learners themselves, and those responsible for their success, since it can provide the finest level of detail, ideally as rapidly as possible. Researchers are adapting techniques from fields including serious gaming, automated marking, educational data mining, computer-supported collaborative learning, recommender systems, intelligent tutoring systems/adaptive hypermedia, information visualization, computational linguistics and argumentation, and social network analysis." (http://iite.unesco.org/pics/publications/en/files/3214711.pdf).

Smaller sets of data are permitted in this PLIG focus area. The scope of a proposal may be on the order of a single course or a few courses.

For further background about learning analytics, please see: https://net.educause.edu/ir/library/pdf/ELI7059.pdf.

This application is for a: ❏ Focus Grant ❏ Exploration Grant

IV. BUDGET SUMMARY

There is a Microsoft Excel worksheet to calculate your budget

The total shown in this worksheet must match the “Total funds requested” on the first page of this application form.

• The worksheet will automatically calculate the appropriate benefit rate based on the salaries entered.

• This figure must be included in the total award request when any salary dollars are requested. Lastly, please do not override any formulas in the worksheet.

You can download the worksheet at https://www.rit.edu/academicaffairs/plig/index.php.

Note that any equipment or other materials purchased with grant funds are the property of your department and revert to the department after your project is completed.

V. TIMELINE

Please indicate any variances to the planned PLIG schedule and your reasons. If you do not intend to deviate from the schedule, you may leave this section blank.

Task Date Proposed variance and reason

Full project plan submitted Aug. 24, 2015

Preliminary findings submitted Jan. 25, 2016

Summary of final findings submitted Aug. 22, 2016

Final budget accounting submitted Aug. 22, 2016

Faculty Teaching and Learning Commons posting (a summary of findings, examples of teaching designs or materials, etc.) due

On or before Oct. 2, 2016

Participation in Teaching and Learning Services PLIG dissemination event

On or before Nov. 21,

2016

VI. STATEMENT OF UTILITY (two pages maximum)

Using the proposal evaluation criteria provided in Evaluation section of the Call for Proposals document, provide an overview of the project you are proposing, including: • Project objectives • An explanation of the teaching/learning problem(s) it is designed to address • An explanation of the significance of the project to student outcomes and/or the student

experience.

• A brief description of how the project integrates with activity already underway at RIT in the priority area and/or how this approach has been successfully used at RIT already.

The objective of this project is to utilize learner analytics to study what, when, and how students use flipped classroom videos. We would like to explore the following research questions:

• Based on student academic performance (Student GPA), and student demographics (year, SAT/ACT, gender, age, transfer credit, NTID …) Is there a correlation between student academic or demographic characteristics and those students who watch the pre-lecture, sample problem, and demonstration videos?

• When do students watch the videos? (day, time, in relation to class & tests) • How long do they watch the videos (pausing, reviewing) • When in the video do they typically pause the video (is there a common

difficult topic?) • How often do they watch the videos (do they come back to videos before

tests, finals?) • Do they watch the videos in batches or one at a time? • Does watching the videos in a certain way (when, how) predict test

performance? • What videos or topics are viewed more often, or less often (difficult or ‘easy’

topics) • Does video watching habits change over the semester? (watch more or less

as semester progresses) • Does video watching change based on test performance?

In parallel, we would like to interview students in the class over the semester to gather qualitative data on video usage to triangulate with the above research questions. From these interviews was also would like to understand:

• What do students do while watching the videos (ie. take notes, screenshots,…)

• What classroom activities, or assessments encourage or discourage additional learning activities (notes, screenshots, …) while watching videos.

STATEMENT OF UTILITY (continued)

For the MCET-330 Fluid Mechanics and Fluid Power class we have over 100 videos on Mycourses for students to use. These videos include pre-class lecture videos (flipped classroom), sample problem videos (cartoons, whiteboard, pen and paper, and computer solved), and demonstration videos. The videos are organized by lecture topic/class and the analytics of students viewing the videos is captured in Mycourses by individual.

Teaching Problem Addressed

In a flipped classroom design the lecture is external to the class so you do not typically see the lack of understanding on a topic as students are building their own cognitive connections to previous knowledge initially external to the classroom. This project will use the learning analytic captured data on video usage to understand how, who, why, and when students use the pre-lecture, sample problem, and demonstration videos. We know from the end of class focus groups that students use and appreciate the videos (sample student comments below from Fall 20141)

“Student>>>You could also save time in class he's not teaching the class, he's teaching it while your home, basically, come to class ready.

Student>>>Hi, I thought it particularly useful just because like you say you can go back [for tests] and that's usually a big problem that people have in class, you know, the professor will go through too fast for you to follow and usually and if you ask them, "Hey, can we just like work ?" because that's something that some students do, they are always like, "Yeah, no". So you can like stop, go back, look at it, stop go back, look at it within the span of like two minutes until you finally get it and move on and that's really helpful.”

Significance of Project In a flipped classroom design the videos are a new and different resource that students typically have not had in their previous classes or K-12 experience. Therefore, it is important for us as educators to understand which students use this resource, when they use the videos, and how they use the videos. This understanding will allow us to better tailor this resource for their needs and help them understand how other students use video resources to improve learning. The new knowledge from this project will help educators improve their video resources, help students understand how others have used the video resources, and hopefully lead to a larger externally funded survey across multiple flipped classroom instructors.

Project Integration with RIT work Over the past few years RIT’s Innovation Learning Institute (ILI) has encouraged faculty to explore the uses of a flipped classroom design. This projects aims to harvest the stored data from this effort already stored in Mycourses for the MCET-330 Fluid Mechanics and Fluid Power class. We anticipate that this project will inform The ILI/TLS organization, the RIT flipped classroom faculty working group concerning student usage, and RIT students on how other students use flipped classroom resources. A second important output of this project will be to fully explore the video learning analytics available through MyCourses to identify the features of high value and areas for improvement that can be strengthened in MyCourses.

VII. STATEMENT OF CREATIVITY (three paragraphs maximum)

Provide a brief description of how this is a novel approach, or a new application of an existing mode or model of teaching and learning, and/or research about teaching and learning represents an entirely new paradigm. (Please note that special consideration will be given to proposals that demonstrate a new use/application of a model, system, or technology already in use at RIT.)

This project involves both analyzing the learning analytic video data and gathering qualitative data through student interviews to look at the video resources usage in parallel to the in classroom activities. Other studies, such as the large Guo, Kim, Rubin study1 reviewed video engagement in terms of the production style of the video (lecture, “talking head:, digital tablet, PowerPoint). While the production style is important we believe it is also important to understand what in classroom activities (tests, quizzes, homework, projects) synergistically support video resource usage. We also believe it is important to understand how students use video. Do they take notes while watching videos, screen shots,… and what in classroom activities encourage these additional learning activities. Brooks, Epp, Logan, and Greer studied lecture capture1 and found that many students could be classified as “just in time” learners watching the lecture capture videos just before in class assessment activities. This project will extend this finding to determine if this just in time video watching expands to every class in the flipped classroom approach due to the in class activities in each class.

1 Guo, Philip J., Juho Kim, and Rob Rubin. "How video production affects student engagement: An empirical study of mooc videos." Proceedings of the first ACM conference on Learning@ scale conference. ACM, 2014.

1 Brooks, Christopher, et al. "The who, what, when, and why of lecture capture." Proceedings of the 1st International Conference on Learning Analytics and Knowledge. ACM, 2011.

VIII. STATEMENT OF EFFICACY (two pages maximum)

Provide a brief description of the experiment/research design, methodology, and methods of data collection you will use to gauge efficacy.

For the MCET-330 Fluid Mechanics and Fluid Power class I have developed lecture videos for each class, and sample problems for each lecture/topic (figure 1).

Figure 1: Typical Class Video Resources (MCET-330 Week 2, Class #4)

Students should watch the video ahead of class and are encouraged to take notes on the video content. At the beginning of class after an open question period students take a quiz on the video content and then proceed to solve homework problems in cooperative/active learning groups (figure 2).

STATEMENT OF EFFICACY (continued)

Figure 2: Typical Class Cooperative/Active learning Groups (Photo Credit – RIT E. Lamark)

Students report watching the videos but as noted by Gorissen, Van Bruggen, and Jochems student reporting does not always match learning analytic results1 The videos for MCET-330 are housed in MyCourses allowing students to watch the videos on a variety of devices. The MyCourses system captures when a student watches a video, length of time watching the video, and number of times the video is viewed by the student. The initial step of the project will be to review and filter the total dataset (2013-14 and 2014-215 academic years along with Fall 2015-16 AY). This filtering is often referred to as data pre-processing to combine data sources, remove irrelevant data, identify missing entries, identify users and assign anonymous tracking number, and identify outliers. For the analysis we will combine the MyCourses video usage data source, the video characteristics (length, and production type) the grades on specific quizzes, tests source, and student demographic data sources. With this filtered data source we will analyze the interaction of each student with the lecture videos, sample problem videos, and demonstration videos. In parallel we will seek student volunteers who will work with undergraduate researchers to document their use of videos to add depth and triangulation to the analysis.

IX. DISSEMINATION PLAN (optional)

Provide details about the journal, conference, show, other external vehicle with strong potential for dissemination of your results. Include supporting documentation such as preliminary interest or acceptance with your application, if available. (Please note that special consideration will be given to proposals that have a defined opportunity for external dissemination, such as an academic journal or professional conference.)

Teaching and Learning Services will arrange channels for disseminating results within RIT.

This project has a high likelihood of yielding results that can be widely disseminated and that would be of interest to those in the STEM education, Flipped Classroom, and Higher Education fields. This project builds upon the Technology Rich Interactive Learning Environment disseminated products already generated by the proposer. These products include conference papers, and a book chapter.

X. ADDITIONAL CONSIDERATIONS

Please address these questions, if needed. • Will your project require assistance for extensive or unusual media, multimedia, simulation,

and/or software development? If so, please explain?

• All courses offered by RIT must be accessible to students with disabilities, according to

Section 504 of the Rehabilitation Act of 1973 and Title II of the Americans with Disabilities Act of 1990 (rit.edu/studentaffairs/disabilityservices/info). Is your proposed teaching approach accessible to all students, with reasonable accommodation? If not, please explain.

• RIT abides by the Family Educational Rights and Privacy Act of 1974 (FERPA), which

prohibits instructors from making students' identities, course work, and educational records public without their consent (rit.edu/xVzNE). Will any data gathering or sharing for your project raise any FERPA issues? If so, please explain.

No, data is within MyCourses and RIT Institutional Research (student demographics)

NA

No individual student data will be made public or disclosed. Student records will be converted to subject identification numbers

XI. DISSEMINATION AGREEMENT

By completing this grant application, I agree to provide the materials described here, in support of disseminating what is learned from this project to other faculty at RIT.

I also agree to return all/a portion of the funds that I receive for this project to RIT if I fail to complete or provide the materials described here. • Full project plan (including roles and responsibilities, milestone dates, and pertinent project

details)

• Overview of preliminary findings (may include experiment/study design, lessons learned, initial data collection, and/or literature review summary)

• Final project summary (including data collection, lessons learned, implications for further study, and which may be in the form of an article abstract, conference presentation outline, or short report)

• Faculty Teaching & Learning Commons posting (a summary of findings, examples of teaching designs or materials)

• Participation in a faculty dissemination event • Final budget accounting (reconciliation of budget provided with your application and the

actual project expenses)

By submitting this application, I accept this agreement.

XII. DEPARTMENT HEAD CERTIFICATION

I support this PLIG application and budget, and verify that _______Dr. Robert Garrick________ is a full-time faculty member in good standing in my department.

Department head name: S. Manian Ramkumar Date: Feb. 2, 2015

Department head signature:

Phone: 5-6081 Email: [email protected]

PLIG  2015  Budget  WorksheetApplicant's  Name:

Personnel Purpose/JustificationFull  time  Faculty/Staff

123Student

1 Student  worker  to  gather/tabulate  Mycourse  data23

T Personnel  Total

T Benefits  -­‐  Calculated  Automatically

Equipment Purpose/Justification123

T Equipment  Total

Licenses Purpose/Justification123

T Licenses  Total

Travel Purpose/Justification123

T Travel  Total

Other  (Specify) Purpose/Justification1 Student  Interview Interview  some  students23

T Other  Expenses  Total

Total  Award  Request

Garrick

PLIG  2015  Budget  Worksheet

Amount

-­‐$                                          -­‐$                                          -­‐$                                           Number  of  Students 2

Pay  per  hour 12.53,000$                                 Hours  per  week 10-­‐$                                           Weeks 12-­‐$                                           3000

3,000$                                

240$                                        

Amount-­‐$                                          -­‐$                                          -­‐$                                          

-­‐$                                          

Amount-­‐$                                          -­‐$                                          -­‐$                                          

-­‐$                                          

Amount-­‐$                                          -­‐$                                          -­‐$                                          

-­‐$                                          

Amount1,350$                                 Interview  # 9-­‐$                                           Hours  of  interview 2-­‐$                                           Weeks  of  interview 5

Pay  per  hour 151,350$                                 1350

4,590$                    

Garrick