Teacher.tv detailed design, formative and summative evaluation

1

TEACHER.TV DETAILED DESIGN, FORMATIVE AND SUMMATIVE EVALUATION

Fostering creative thinking skills in first-year students of the graphic design program at Cégep du Vieux-Montréal by Émilie René-Véronneau ETEC651 Nadia Naffi Concordia University

2

TABLE OF CONTENTS

3 Rationale for this intervention

4 Objectives

4 Actions to be taken prior to production

4 Defining guidelines

5 Production schedule

6 Homepage

6 Footer

7 Registration process

8 Channel landing page

9 Broadcast front-end

10 Broadcast back-end

11 Conducting the formative and summative evaluations

12 Formative evaluation - Usability and technical review

13 Formative evaluation - Content and editorial review

14 Summative evaluation - Performer engagement

15 Summative evaluation - Individual performance

15 Summative evaluation - Performance transfer

15 Summative evaluation - Organizational performance

16 References

CO

VER

SO

UR

CE

Nor

disk

fam

iljeb

ok (1

876,

190

4 or

192

3)

3

Rationale for this intervention

This intervention aims to establish an innov-ative and multi-institutional platform where students and teachers can gather in an online informal learning setting.

the graphic design classroom

For a long time, graphic design was a trade which was taught through apprenticeships. Nowadays taught in college, classes have a lim-ited attendance of 25 students and most of the time spent teaching consists of giving feedback and coaching on unique projects.

In the current setting, there is very little time for the teacher to show students how they would go about solving a problem or creating something in their daily practice. Moreover, the graphic design program at Cégep du Vieux-Montréal does not include an internship in the field. These internships can be difficult to access and even when found, employers are busy and may not have time to take proper care of the student.

Therefore, the students have limited oppor-tunities to observe expert behaviors which could inspire their own practice.

new opportunities for informal learning The speed of today’s Internet connections enables people to communicate with each other through live online video.

In the past years, the gaming community has seen the rise of video streaming platforms such as Twitch.tv, Livestream and Ustream. These platforms allow people who are playing, work-ing, or experimenting to share and broadcast their experience online, as if they were on television.

As viewers tune in the broadcaster’s chan-nel, they access a chatroom where all can dis-cuss what is happening and ask questions. The broadcaster reads the chatroom every once in a while and can discuss with the viewers. This kind of informal and relaxed setting lends itself to highly situated questions. While it can still take place within educational institutions, informal learning covers learning that takes place outside of the required curricula of for-mal learning. Depending on the intention and awareness of the learner, it can take the form of self-directed learning, incidental learning or socialization. (Schugurensky, 2000)

Form Intentionality Awareness

Self-directed yes yes

Incidental no yes

Socialization no no

The platform supports all three types of informal learning. As a self-directed learner, a viewer may ask the broadcaster a specific question and add a timestamp to the provided answer for future retrieval. Learning incident-ally, the viewer may benefit from someone else’s question, or simply from information provided by the broadcaster. In terms of social-ization, the viewer may simply be interacting with other viewers and broadcasters, becom-ing aware that they have learned something at a later time.

timestamps

Marsick and Watkins (2001) claim that informal learning can be improved through facilitation and increased awareness from the learner. The timestamp feature serves to facilitate and increase that awareness. It also allows the platform administrators to assess the type of content which viewers value most and makes the platform different from other existing alternatives.

Timestamps allow viewers to keep track of moments they value during broadcasts and gather a collection they are able to revisit as needed to inform their own thinking or practices.

SOURCE Adapted from Schugurensky, (2000)

4

Objectives

This intervention caters to all levels of per-formers and mainly addresses the following performance objectives:

performance objectives for students

Given information about creativity and creative thinking as well as proper models of behavior, performers apply techniques to leverage their creative skills.

performance objectives for teachers

Given information about creativity, creative thinking and reflexive teaching, the performers explain their own creative processes by mod-elling creative behaviors to the students, facili-tating the students’ acquisition of key creative personal characteristics.

performance objectives for all performers

Given a positive environment where creative behaviors are valued, performers interact in playful, cooperative and trusting ways to facili-tate creative outcomes and processes.

Actions to be taken prior to production

In order to get this initiative started, funding will need to be secured from the Programme d’aide à la recherché sur l’enseignement et l’ap-prentissage (parea), offered by the Ministère de l’Éducation, Enseignement supérieur et Recherche. If the first iteration is successful, more sponsors will be involved to grow the pro-ject to other institutions.

A committee made of pedagogical advisors and teachers interested in innovative teach-ing practices and technology will be formed to administer the platform.

In order to stay focused on our goal of pass-ing tacit knowledge from expert to novice, only teachers will be allowed to broadcast. Any teacher interested in broadcasting can simply sign up on the site using their institutional email. At the beginning of the school year, teachers who have manifested interest will be met at their institution to receive background information and training for the interven-tion, discuss ideas for presentations, and draw inspiration from each other.

For the success of this intervention, it is cru-cial that all performers believe in the idea and see the value that informal learning can bring to students first.

As additional institutions join the initiative, they will be expected to advertise the platform through their institutional publications and occasional advertisements.

5

Defining guidelines

Because of the innovative nature of this inter-vention, some training content will be required. Concerns are also expected to arise in terms of managing the community, therefore best prac-tices need to be defined.

video and audio production training

A video production expert will be required to create a training which will aim at ensur-ing basic proficiency in shooting video and recording audio when using the platform, as well as establishing best practices for a variety of broadcasting conditions.

terms and privacy guidelines

Interactions between all participants should be respectful. While each broadcaster is allowed to define specific rules, no bullying or harasse-ment should be tolerated.

Although the learning context is informal, broadcasters need to be aware that they are revealing their world to anyone registered to the site and that their privacy is not protected. For example, teachers should not engage in illegal behaviors such as consuming illicit sub-stances or revealing confidential information.

Since this type of intervention is uncommon, a social media expert, as well as a usability expert from Ustream will be asked to define appropriate and unappropriate behaviors based on their experience. These guidelines will be used to define the terms for the platform.

Production schedule

Production of the platform will undergo the following steps. Some of these steps may over-lap in order to speed up the production process. The time from drafting the platform to launch at a first institution is estimated to one year. If the first initiative is successful, other institu-tions will be invited to join and a mobile version will also be produced.

platform developmentœ Review of current wireframes

and production of any missing wireframes with usability expert

œ Integration of wireframes and development of the platform

œ Pilot study, usability and load testing

œ Review of platform according to pilot study and load testing

œ Integration of graphic design to platform

œ Quality control and debugging

œ Official launch

strategy and communicationsœ Branding and graphic design of platform

œ Advertisement strategy for enlisting broadcasters and viewers

œ Identification of potential broadcasters and viewers for early adoption

œ Focus group on branding, graphic design and advertisement strategy

œ Review of branding, graphic design and advertisement strategy according to focus group

œ Advertisement campaign

œ Official launch

6

Home /

Lambert Ouellet Consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam. Tune in!

Julie SigouinLogo sketches

Lambert Ouellet“Air Chamber” chair

Daniel LévesqueExperimenting with Bootstrap

Josée DallaireAsian Bookbinding

Currently live

Upcoming events

Search by institution, teacher or interest...

Luc GirouardCégep du Vieux-MontréalDrone photography

Émilie René-VéronneauCégep du Vieux-MontréalLetterpress printing

Christian BélangerCollège Marie-VictorinHand lettering

François MartinDawson CollegeWeekend Sessions

LOGO

About Help Terms and Privacy FAQ Contact Us

BrowseBrowse

March 3rd 2015 10 pm March 11th 2015, 9 am March 20th 2015, 10 pm April 2nd 2015, 10 am to 12 pm

Homepage

The homepage presents featured channels based on popularity and trends as well as upcoming events. Once the students log into their account, these elements are adapted to better represent the interests of the students as defined in their user profile. Users also have the possibility to search directly for what they are looking for by using the search field.

Users can navigate events and channels through different means: they can pick among the choices suggested by the platform on the homepage, they can search for a specific insti-tution, teacher or topic in the search form, or they can browse through a list organized with the same categories.

As they add events or subscribe to different channels or topics, users get email notifications which bring them directly to the events they want to participate in. Notifications are sent a day in advance, as well as when the broadcast goes live.Users can return to their account set-tings by clicking on their avatar in the top right of the screen.

Footer

The footer contains some of the standard links for websites.

about The about page gives more information on the rationale of this intervention as well as some background information about informal learning.

help The help section contains documentation to guide users through the use of the site’s fea-tures, as well as a few video tutorials to help broadcasters get started.

terms and privacy This page contains the terms of service and privacy policy for the platform.

faq The FAQ (Frequently asked questions) page regroups questions that users may have and that are not answered by the help file or that do not fit its format in one knowledge base. An example could be, "How can I become a moderator?"

contact us The contact form is used to provide means to reach the administrators of the site.

7

� 0:45:31

Logo sketching

February 15th 2015

� 1:01:35

T-shirt printing

February 2th 2015

� 0:20:36

Brainstorming for chair

December 18th 2014

1 3 4 5 ...2

Search...

Home / Your account

LOGO

Name

About you

Your interests

Your channels

Department

Institution

Camille Lehoux

ID 7271646

Nickname Camillou

Graphisme

Cégep du Vieux-Montréal

Upload picture

Manage your subscribed channels

Cool technique!Use for series ofcircus posters

Nice explanation

Funny! Trying to combine things that are opposite

Save changes

Tell us more about you...

Graphic design

PrintingWeb design

Photography

Bookmarked moments

Your Profile

Search by institution, teacher or interest...Browse


Registration process

When first signing up for the service, students pick their institution and department from a dropdown menu and input their student ID. This creates a temporary password that is then sent to their institutional emails. They can log in using their student ID and password to view their profile and change their settings.

profile

Since informal learning should be a fairly comfortable context where students feel free to ask questions, they will be allowed to pick any nickname or avatar they wish to use. Their account remains linked to their student ID in cases where there would be a necessity to iden-tify them.

In order to get personalized recommendations, students can choose among a variety of inter-ests which are linked to the taxonomy used in the site. While they can subscribe and unsub-scribe from specific channel pages, a user’s subscriptions can also be managed from the account page.

timestamps

Timestamps collected during the different broadcasts are stored in the users’ profiles. The event and time of the marking are stored and students can add a short description. A search engine assists the students in finding those moments again..

8

14098

Lambert Ouellet Past events

Home / Cégep du Vieux-Montréal / Lambert Ouellet

LOGO

Subscribed

�

� 2:45:31� 86� graphic design,logo, ideas

Logo sketching

February 20th 2015

� 1:33:31� 117� printing, t-shirtsilkscreen

T-shirt printing

February 4th 2015

� 2:01:25� 15� printing, christmassilkscreen

Christmas cards printing

December 20th 2014

About Lambert OuelletLorem ipsum dolor sit amet, consectetur adipiscing elit. Proin nisi orci, vulputate sed ligula quis, posuere condimentum ipsum. Donec pulvinar fringilla lobortis. Integer non tristique nisl. Morbi vestibulum sed lectus in eleifend. In aliquam consectetur metus, sit amet varius elit sagittis ac. Mauris tincidunt eros in sapien sodales euismod. Proin magna ligula, dapibus ut blandit vitae, lobortis eu lacus. Fusce lobortis consectetur urna sed volutpat. Integer at ex ut orci volutpat dictum id eget libero. Donec sed iaculis est, ac dictum arcu. Nulla sollicitudin felis sit amet ante sodales ultricies. Cras eu urna a nibh venenatis malesuada fringilla non nulla. Pellentesque eget nibh at justo vulputate malesuada in vel velit.

Quisque volutpat, purus sed pretium imperdiet.


OFF-AIR

1 3 4 5 ...2


Logo sketching

February 20th 2015


T-shirt printing

February 4th 2015



December 20th 2014

1 3 4 5 ...2


Logo sketching

February 20th 2015


T-shirt printing

February 4th 2015



December 20th 2014

1 3 4 5 ...2

Browse


Channel landing page

The channel pages are homepages for specific broadcasters. If the broadcaster is on-air, the viewer can tune in directly to the event. If the broadcaster is off-air, an image that represents the channel is shown instead. Broadcasters can enter custom text to describe the objectives of their channels or rules that they want to estab-lish. An archive of past events is found in the sidebar.

9

45:31 14

“Air chamber” chair -- Sunday, March 29th, 2015 Lambert Ouellet

Viewer1: What kind of wood are you using?Viewer8: earlier he said it’s cedar wood Viewer2: are they bike air chambers? Viewer4: @Viewer6 did u study at UQAM?Viewer6: @Viiewer4 I’m doing the BA right nowViewer4: @Viewer6 coolViewer8: are you bringing it to class next week? �Viewer6: but we’re on strike for nowViewer12: Lambert! Long time no see! What areyou up to this evening?


Subscribed

� ��


LOGO

Timestamp

Browse


Broadcast front-end

As the teacher broadcasts, viewers tune in using the front-end interface and ask questions or comment. The teacher scours the chatroom window every now and then to answer ques-tions verbally. Viewers can attempt to answer others’ questions if the broadcaster delays in providing answers. This allows to foster a help-ful environment where everyone can benefit from each others’ knowledge.

As time goes by, the broadcaster may notice that some viewers are regulars and would make good community moderators. When viewers are promoted to moderate a channel, they retain this right unless it is removed from them by the broadcaster. Moderators have the ability to warn or exclude other viewers from the stream.

addressing potential issues

The wrench and flag buttons allow to to warn the broadcaster that audio or video are not working properly so they can modify their set-tings or to signal inappropriate behavior to the platform administrators.

Every flag received by a broadcaster will be reviewed by the administrators of the plat-form. Depending on the seriousness of the

inappropriate behavior, sanctions can range from a warning to being prevented from broadcasting.

Broadcasters can flag users who misbehave in order to deliver a first warning. For every other subsequent flag, the viewer gets locked out of the channel, and then the platform itself, for an increasing amount of time. After a num-ber of warnings, the user gets banned from the platform permanently. A banned account can only be readmitted to the platform on the con-dition that the banned user reaches an agree-ment with the administrators.

adding a timestamp

The add timestamp button allows viewers to create a bookmark of what is happening in the video. When clicked, the user can adjust the time of stamping and add a short description. These collected moments are stored into the users’ profile page.

10

Share

Start broadcast

Choose video source

Choose audio source

45:31

Quality

Choose PIP source

Volume

Edit broadcast title

Share your broadcast with your followers:

Tags for this broadcast

Enter your custom message here.


�

OFF-AIR

ON

OFF

OFF

4:3 16:9

� � � �

Start broadcasting


LOGO

Browse


Start broadcast Private

Choose video source

Choose audio source

45:31

Quality

Choose PIP source

Volume

“Air chamber” chair -- Sunday, March 29th, 2015


�

OFF-AIR

ON

OFF

OFF

4:3 16:9

Start broadcasting


LOGO

Viewer1: What kind of wood are you using?Viewer8: earlier he said it’s cedar wood Viewer2: are they bike air chambers? Viewer4: Viewer6 did u study at UQAM?Viewer6: Viiewer4 I’m doing the BA right nowViewer4: Viewer6 coolViewer8: are you bringing it to class next week? �Viewer6: but we’re on strike for nowViewer12: Lambert! Long time no see! What areyou up to this evening?

Promote to moderator

View user profile

Flag for inappropriate behavior

Browse


Broadcast back-end

In order to share and control their broadcast, teachers have access to the back-end interface. Once they start broadcasting, the chatroom takes the place of sharing options on the right of the screen.

This interface allows them to edit the details for their event and share it with their subscrib-ers and social media.

The broadcaster back-end is used to set up and troubleshoot issues during broadcasting. Before broadcasting, the teacher can fill in the details for the title, as well as relevant tags. Using the settings at the bottom of the screen, the broadcaster chooses what video and audio sources they wish to use as well as the quality and volume.

Depending on the activity undertaken, the teacher may want to use a camera or capture what is happening on their screen and add a picture-in-picture view of themselves.

There are no set ways to interact with view-ers other than following the minimal guidelines set by the platform in terms of ethics and pri-vacy. As such, interaction styles may differ from one broadcaster to the next.

11

Conducting the formative and summative evaluations

The following pages define the formative and summative evaluations for this intervention.

participants

Depending on which aspect of the platform is being evaluated, administrators of the plat-form, teachers from relevant disciplines, as well as previously consulted experts in video and audio production, usability and social media will participate.

Viewers will also be solicited through ques-tionnaires. For usability testing, five viewers of different abilities will test the platform. While this may seem a small number of participants, involving five testers is usually sufficient to uncover 85% of usability issues. (Nielsen, 2009)

evaluation instruments and frequency of evaluation

The evaluation instruments need to offer some flexibility as creativity is a complex phenom-enon. Interviews will mostly be used when consulting with broadcasters, as they are fewer in number, while viewers will fill out question-naires with open-ended questions.

The metrics provided by the platform will also allow to further investigate the behaviors of all performers.

Evaluations will normally be conducted on the yearly basis but multiple evaluations will be conducted when first establishing the platform as we want to ensure that a core of performers create a sustainable online culture.

SO

UR

CE

Nor

disk

fam

iljeb

ok (1

876,

190

4 or

192

3)

Tool Data collected

Sou

rce

When RationaleU

sabi

lity

and

tech

nica

l rev

iew

Que

stio

nnai

re

and

inte

rvie

wQuestionnaire - Likert-scale questions

– Signing up is a user-friendly process

– Broadcasting is a user-friendly process

– The site meets standard guidelines for accessibility

– The navigation is user-friendly

– Links are working

– The site is well-designed on desktop, tablet and mobile iterations

– The site is functional on current browsers

– The site is functional on current browsers

For any item not meeting at least 3 points out of 5, further information will be gathered through interviews in order to explore and resolve usability issues.

Questionnaire - Open-ended questions

– Are there any usability issues with the platform? Explain.

– Are the technical guidelines relevant, user-friendly and complete?

– Which feature of the platform do you find most useful? Explain.

Bro

adca

ster

s, a

ll co

nsul

ting

exp

erts

, an

d a

sam

ple

of f

ive

view

ers

of

dif

fere

nt a

bilit

ies

Before implementation,

after the first delivery and on a yearly basis

– Assessing usability of the intervention

Met

rics

Website - Metrics– Performers are using the features of the site (tags, sharing, timestamps, etc.)

– Amount of problems reported through the system, as well as types of problems

– Amount of problems reported, as well as types of problems

Usa

bilit

y ex

pert

s


after the first delivery and on a yearly basis

– Assessing usability of the intervention

Form

ativ

e Ev

alua

tion

12

Tool Data collected

Sou

rce

When RationaleC

onte

nt a

nd e

dito

rial

rev

iew

Inte

rvie

wInterview - Open ended questions– Are the guidelines for appropriate behaviors and moderations

relevant, and complete?

– Are the guidelines for privacy protection relevant, and complete?

– Are there spelling or grammatical errors in the content?

– Is the level and tone of the content appropriate for college students? Soc

ial m

edia

and

us

abili

ty e

xper

ts

Before implementation

– Asssess the content quality in order to implement the intervention

Inte

rvie

w a

nd m

etri

cs

Using a sample of broadcasts, one yearly broadcast per teacher:

Interview - Open ended questions

– What kind of value do you think this type of online informal learning could bring to the program?

– Are the broadcasters discussing material relevant to the curriculum?

– Are the viewers asking questions relevant to the curriculum?

– Are the interactions respectful and age appropriate?

– Are the questions asked in the chatroom addressed by the broadcaster?

Website - Metrics– Quantity and quality of questions in chatroom

Bro

adca

ster

s

Three, six, and nine months after

implementation, and on a yearly basis

– Asssess the content and interaction quality of the intervention

Inte

rvie

ws

and

met

rics

Interview - Open ended questions

– Are the suggested channels relevant?

– Is the taxonomy relevant and consistent?

– Are some questions asked often enough to move them in the FAQ?

– Should some questions in the FAQ be integrated to the help documentation?

Website - Metrics– Quantity and reasons of flagged behaviors for broadcasters

– Quantity and reasons of flagged behaviors for viewers

– Quantity of clicks on suggested channels

Soc

ial m

edia

and

us

abili

ty e

xper

ts



– Asssess the content and interaction quality of the intervention

Form

ativ

e Ev

alua

tion

13

Tool Data collected

Sou

rce

When RationaleP

erfo

rmer

eng

agem

ent

Inte

rvie

wInterview - Open ended questions– Do you have any concerns with regards to delivering an online event? Explain.

– To what extent do you envision yourself engaging in these events? Explain.

Bro

adca

ster

s


after experiencing some examples

– Assessing perceptions towards the intervention.

Inte

rvie

w Interview - Open ended questions– Have you encountered any non-technical issues during delivery?

– To what extent do you envision yourself engaging in these events again?

Immediately after first delivery

Inte

rvie

w a

nd m

etri

cs Interview - Open ended questions– Do you feel this intervention has had

an impact on your interactions with students? In what way?

Website - Metrics– Quantity and length of events delivered

– Variety and relevance of topics covered

– Chatroom sentiment analysis

– Quantity of events viewed by students

– Quantity of moderators

– Level of activity in chatroom

Yearly maintenance evaluation

Que

stio

nnai

re

Questionnaire - Open ended questions– How often do you envision participating in activities like this?

– What did you enjoy most about this activity?

– What did you enjoy least about this activity?

Vie

wer

s

Before implementation, and immediately

after first delivery

– Assessing perceptions towards the intervention.

Que

stio

nnai

re

Questionnaire - Open ended questions– Which events have you preferred over the past year? Why?

– Do you feel this intervention has had an impact on your interactions with teachers and other students? In what way?

Yearly maintenance evaluation

Sum

mat

ive

Eval

uati

on

14

Tool Data collected

Sou

rce

When RationaleIn

divi

dual

per

form

ance

Que

stio

nnai

re Questionnaire - Open ended questions– What have you learned from the events that you have participated in

which could make you a more creative person?

– Have the events you’ve watched had any impact on your habits, thinking or techniques?

– What are the reasons for which you usually timestamp a video?

Vie

wer

s Three, six, and nine months after


– Assessing results in meeting performance objectives.

Inte

rvie

w

and

met

rics

Interview - Open-ended question– Has the broadcasting of events improved your ability to voice your creative processes?

Website - Metrics– Amount and content of moments timestamped by students for a specific teacher B

road

cast

ers




Per

form

ance

tra

nsfe

r

Met

rics

Website - Metrics– Students describe their timestamped moments in way that could suggest

they will be relevant to their performance and projects.

Bro

adca

ster

s




Org

aniz

atio

nal p

erfo

rman

ce

Met

rics

Attendance - Metrics– Attendance to different exhibits of the graphic design department,

especially the graduation exhibit.

Employment rates - Metrics– Percentage of students hired in graphic design jobs or admitted

to graphic design undergraduate programs pre- and post-intervention.

Admission rates - Metrics– Quantity and quality of new students requesting admission to the program.

Inst

itut

ion

adm

inis

trat

ion

On a yearly basis– Assessing results in meeting

performance objectives.

Sum

mat

ive

Eval

uati

on

15

16

References

Marsick, V. J., Watkins, K. E. (2001). Informal and

incidental learning. New directions for adult and

continuing education, 2001(89), 25¬34.

Nielsen, J. (2000, March 19). Why You Only Need

to Test with 5 Users. Retrieved April 14, 2015, from

http://www.nngroup.com/articles/why-you-only-

need-to-test-with-5-users/

Schugurensky, D. (2000). The forms of informal

learning: Towards a conceptualization of the

field. Retrieved April 14, 2015, from https://tspace.

library.utoronto.ca/bitstream/1807/2733/2/

19formsofinformal.pdf.

17

Teacher.tv detailed design, formative and summative evaluation

Documents