Case Study

The Use of Synchronous Digital Video for Observation of, and Feedback on, Teaching Performance in the Learning and Skills Sector.

A Case Study for the Standards Unit of the Department for Education and Skills: ITT Reform Pilots

This case study narrative is an adjunct to the evaluation report

Martin Dyke, Alan Harding, Sue Lajeunesse

June 2006

Further Information:Dr Martin Dyke [email protected] of Education Tel: 023 8059 5563 University of Southampton Fax: 0238059 3556Highfield Southampton SO17

IntroductionThe University of Southampton worked with Cricklade College, Education and Training Services of the British Army, the Isle of Wight College, and Southampton City College to deliver this project for the DfES Standards Unit. The project used video conferencing systems in partnership colleges and training organisations as a tool to support the remote observation of teaching performance in real time.

The project was aimed at developing strategies to improve and support Initial Teacher Education, facilitate Human Resource Management and help support observation of teaching by subject specialists. The overarching aim was to:

To develop operational policy, procedures and make recommendations for the remote observation, analysis and feedback of teaching performance.

The team worked closely with teachers to develop protocols that helped us utilise this technology in a supportive and developmental way.

The online observation equipment was installed in partnership centres these included a work-based education and training establishment and General Further Education Colleges. This case-study broadly focuses on the partnership colleges for which this description is typical:

...a medium size general further education (FE) college …The college delivers a wide and predominantly vocational curriculum to students aged 16 to 18 and to adults. There were 1,050 students aged 16 to 18 and 4,200 aged over 19 enrolled at the time of inspection in term one, with final over 19 enrolment for the previous year being 9,854. The college delivers work-based learning in engineering, construction, hospitality and to a small number of learners in hair and beauty. The college also delivers technical certificates on behalf of other local work-based learning provider (OFSTED, 2005)

However, it should be noted that the commentary is generic; it draws on the blended experience of using the technology across the range of partnership centres in which it was applied. The case-study therefore is not specific to the experience of one institution.

The TechnologyThe technology consisted of portable Video Conferencing Suites (CODECs) one in the college classroom and the other based at the University. These were run across the Internet Protocol over the JANET network. The portable video conferencing suite consisted of a flight box with a camera and CODEC, a screen, speakers and DVD recorder. A second camera was cabled directly to the CODEC. The equipment is pictured overleaf in Figure 1.

M.Dyke Page 2 12/04/2023

Figure 1

The main technical challenge found with the application of this technology was securing a connection across institutional information systems and fire-walls. The systems and technology were tested in the summer 2005 and test observations were completed successfully across Internet Protocol (IP). Technically, the process worked with high quality audio and video maintained for the duration of an observed lesson. Observers could watch the lesson, manipulate and switch between cameras, enabling them to survey the whole learning environment and communicate effectively with the class and observed tutor.

On return from the summer vacation the systems no longer appeared to work as reliably. Educational organizations tend to complete work on information systems infrastructure and security in the summer vacation; these networks and firewalls, together with different proprietary standards, are said to be the bane of video conferencing (BECTA 2006). Establishing a connection and using the H.323 protocol to negotiate the firewall and network configurations have proved to be the greatest challenge:

H.323 provides the call management element of a video conference. It enables session initiation, call setup, call closing, call routing and error handling. After encoding the audio and video data is transmitted using other standards such as H.263 (video) and G.711 (audio). (BECTA March 2006).

These complex technological issues have impacted on this project. To date, twenty five opportunities for online observations of teaching have been created, of four have been unsuccessful. In each case there was a failure to establish an online connection. A process of trial and error enabled the project to find a combination of equipment and settings that worked. The most reliable connections were established when the organization’s firewall had been bypassed, a process that can meet a high level of

M.Dyke Page 3 12/04/2023

organizational resistance. Later in the project it was discovered that the proprietary version of H.323 on our CODEC was not compatible with the version on the University firewall. It proved essential that both the CODECS and the institutional firewalls were using compatible versions of H.323 protocol. Mike Harwood resolved the technical issues and his account is recorded below.

Mike Harwood - University Data Networking Group

To get around the problem of not knowing what was really going on behind the scenes and through the firewall, I began running test calls to external sites and capturing or logging the TCP/IP traffic for analysis from the firewall. This meant starting a capture or recording of all traffic flowing through the firewall from Martin’s video conference IP address to another external video conference IP address. I took a large number of packet captures and began analyzing them against some captures of successful connections. I built up a detailed understanding of all the steps taken by the video conferencing devices when they try to negotiate, connect, start to send and receive data, video and audio.

I then began to see a similarity in the tests and a common failing point during the initial connection start-up. There are a large number of TCP and UDP network ports required to start a video conference call and they all need to be opened up in the university campus firewall if the traffic is to pass in and out.

The firewall rules that are in place in our firewall are based on 2 main TCP ports, 1719 and 1720. If traffic is seen coming in on those ports it is assumed that this is video conferencing traffic. The 'language' for video conferencing traffic is specified by the H323 protocol and our campus firewall is aware of this protocol and can read into H323 video conferencing data traffic and look for any requests from the video conferencing devices asking to open further TCP or UPD ports (other than the already open 1719 and 1720). So we rely on the firewall to open these extra ports dynamically on request by inspecting the packets as they pass through.

Unfortunately, it turns out that since the firewall upgrade the firewall’s protocol definition has changed and the firewall was no longer properly recognising these requests to open additional firewall ports. As a result the video conferencing calls would just hang whilst trying to connect and eventually time out and fail because they could get no response through the firewall.

To properly confirm this assumption I set up an environment outside of the firewall on the outside of the campus network with Martin’s video conferencing device and was then able to call a number of external sites successfully.

I then proceeded to call around and ask if any support groups knew of similar problems with video conferencing equipment and our type of firewall and eventually one support group confirmed they had seen similar problems. I then called our firewall support group and showed them our firewall logs and captures and they started to look into the problem.

Eventually they suggested we tried installing an upgrade package, a patch, to the firewall as it was supposed to have some fixes and additions to the H323 protocol handling.

Once this was installed I called around and did some further tests with Martin and some of the other members of staff, that had reported problems, and they said that the problems had now been resolved and their video-conferencing devices had started working properly again.

M.Dyke Page 4 12/04/2023

Planning

Initial permissions were sought through a senior manager within the respective organisation. Typically, there was a three month lead in period before the observation hardware and associated equipment could be installed and the testing period undertaken. In that time, permissions were sought and granted, filtering from senior management level to staff on the ground. Broadly speaking, following initial acknowledgement from senior management, the communication between the project team and the college sub-divided into technical support and curriculum teams. Initially, the technicians liaised with the organisation’s firewall support company to open a number of specific ports for the hardware and were happy, following a testing phase, that ‘these holes’ would not compromise the organisation’s technical security.

Once formal permission had been granted for the project, and technical issues addressed, meetings were held with various curriculum teams and participants. The project aims and processes were outlined, participants completed a questionnaire on their attitudes and preferences to online observation and were given an opportunity to ask questions and seek clarification. The observation process and protocol was developed from this consultation with participants; the intention was to apply the technology in a way in which the observed were comfortable. For example, the participants expressed a preference for the observation not to be video recorded. Accordingly, only live observations and feedback in real time with no digital recording we conducted. The project team also noted that video recording of classes was best avoided as it raised more complex legal and ethical considerations than live observation in real time

Arrangement of the observations tended to be directly between the project team and the volunteering teachers. Sufficient notice was given of the online observation to allow the lecturers to seek permission of their students to be involved in the process. Regular updates regarding the progress of online observations was filtered back through the communication network by the project team; where appropriate and with permissions sought, the online observation was treated as an annual observation for the member of staff.

Sue Lajeunesse – University Project Team

“There is no doubt that the logistical arrangements and hardware installation were time consuming and involved a number of staff members to be present at both the host organisation and the test organisations.”

In order to make this communication not surveillance technology you do need to work with the participants sensitively, use the applications in a way that puts them at ease

In order to install the equipment successfully geographical location of the partnership centres was less significant than the quality of the personal relationships established with both teaching and technical staff. A project such as this presents complex organisational and technical challenges that require from, both sides, a degree of patience, goodwill, good humour and mutual support. Being able to communicate regularly and freely with familiar people and organisations proved invaluable. The technical issues were within the broad skill set of those involved; but can be said to have required a predisposition to learn how to apply these skills to new applications and contexts. Where individuals embraced the challenge of developing new skills the

M.Dyke Page 5 12/04/2023

project was successful, where individuals were less comfortable with experimenting with new applications for information technology progress was relatively slow. The technical staff in colleges and partnership centres had the skills to enable them to make the necessary adjustments to their networks and fire-wall infrastructures. The project success relied heavily upon technical staff that were able to commit to and support the developments.

The Observation

Within an observed classroom two cameras and one microphone were installed. The online observers were able to control both cameras and to use what is called ‘Far End Camera Control’. This enabled us, at the University, to switch between cameras to pan and zoom. These functions were found to be very powerful and key to providing valid observation judgements and feedback. Another important feature of this approach was that the online observers were always visible and able to talk to the class and teacher. Far End Camera Control and the visual presence of the online observer throughout were critical features of the observation protocol. A typical configuration is illustrated in Diagram 1 below. It was arranged, for many of the observations, for a trained observer to be present in the classroom to enable comparison of the descriptions and judgement of online observers with that of a face-to-face observer.

Diagram One

OnlineObserver(s)

IP

Network

Remote control of pan and zoom of camerasOption to record

TV

TV

Observer

Online Observation Configuration

Teacher and Class

On the day of the planned observation, an online connection was established some ten to fifteen minutes before the lesson began and the observed teacher received a pre-observation briefing on the process. At the start of the lesson the online observers introduced themselves to the students and explained the purpose and process of the research. The ethical issues, such as extent of the Far End Camera Control and who would be watching, were made explicit and the participants given an opportunity to ask questions. Once the lesson started, the online observers muted their microphone but remained visible to the observed throughout.

M.Dyke Page 6 12/04/2023

Figure 2

For each quartile of the lesson the observers recorded descriptions of the teaching and learning and issues related to the online observation and provided an OfSTED / ALI grade. Each observer provided a judgment against the criteria and an overall professional judgment for the lesson. The observations covered a range of subject areas including Travel and Tourism, IT, Essential Skills and Teacher Training. The profession and modal grades given to the observations correlated with the OFSTED outcomes for the appropriate Inspection areas, where applicable. One out of Five observations was moderated for the project through simultaneous online and in-class observation. Figure 2 gives an observer eye view of the class from the camera at the front of the classroom, in this photo the observer is about to switch cameras and therefore has the camera icons on screen. Figure 3 overleaf provides an image of the equipment from the camera at the back of the classroom.

Feedback

Paul McKillop - Observed Online at Isle of Wight College I was given immediate, constructive criticism by Sue and she was able to indicate the areas of weakness and make suggestions on how the workload can be shifted between tutor and student using active learning. She promised to send some supportive texts on the subject, and did. These materials have been extremely useful on an ongoing basis. The opportunity for a new tutor to draw on such expertise is exceptional.

At the end of each observation the observed teacher and the class were asked questions about the process and their response to online observation. Each teacher was given both written and oral feedback on her/his performance in confidence. The oral teacher feedback was provided either face to face or online, depending on the individual circumstances and context. Providing a confidential setting for feedback proved more difficult than expected, in some cases it was possible to use the same classroom used for observation, once the students had left, for feedback. However, it was difficult for the person delivering the feedback to judge the level of privacy, they needed to confirm the far end volume of their voice and ensure people were not entering the room, out of camera shot, during the post-observation debrief and discussion. In the right conditions observation feedback via video online was as successful as face-to-face for both parties. Quite often the room was booked for use by another class and alternative arrangements were necessary. In these circumstances

M.Dyke Page 7 12/04/2023

instant messenger software, such as MSN or Macromedia Breeze, was used for a web-cast conference.

Figure 3

Overall findings and experiencesArt the outset there was a belief that online observation as a second best alternative to face-to-face observation. It was considered an approach that might work in situations where moderation of teaching practice was required or where managing the supply of tutors and demand of students for observation was difficult; for example in the provision of subject specialist observers. By the end of the project it became apparent that it is a viable alternative means of observation. When blended with face-to-face observation it can enrich the observation and feedback process. Although the observer gets a different perspective when observing online it enables the observer to focus more on the micro level of interaction. Interestingly it was also found that the observation descriptions and grades of the online observer were the same as the classroom observer in 12 of 14 observed lessons and only one grade out on the other two. In this project we found it a valid and reliable means of observing lessons and making judgements about teaching performance.

The online observations were most successful when the online observer knew the institution and the teacher personally, where they have observed the teacher face-to-face and established an effective working relationship. The only negative feedback about the process was received, on the one occasion, when teacher was not known to the online observers.

Once the networking problems were overcome the equipment was relatively easy to setup and use. Observing a lesson and making notes while driving the cameras at a distance does take a little getting use to and requires intense concentration. It was found that observers had to actively and continuously scan the class from different angles in order to make informed judgements. From a health and safety perspective online observers found that more than three hours of watching a screen per day was the maximum they could reasonably manage. Online observation was more intense and tiring than initially expected.

M.Dyke Page 8 12/04/2023

In relation to the placement of cameras it was found that cameras positioned in the corners of classrooms were least intrusive this reduced the extent of the panning necessary to 90°. Further, when the camera was centrally located on a side wall of a classroom observers needed to pan 180°. With such a wide range of panning the students were more likely to be conscious of the camera movement. The ideal camera positions were in corners of room with one camera at the back of the room and one at the front. The monitor was usually placed at the front, with the observer on a screen facing the class. This did not prove to be a significant distraction but the project team’s view is that the position and level of visibility of the observers should be negotiated with the class. Observer presence was, for the team, a key dimension of the technology being used for communication not surveillance. A typical configuration is illustrated in Diagram One.

For all observations the online observer could be seen and was able to communicate with both teacher and students. This was found to be an important element in the process of working with the teachers and not simply observing from afar. Individual observed teachers managed the visual and audio presence of the online observer in a way they found most comfortable.

Sylvia Waghorn – Observed online at Southampton City College

I found the experience of online observation fascinating and felt that the observation itself was no more intrusive than had a person been sitting in the room. The students reported that they soon became comfortable enough with the equipment that they ignored it.

The Future?The use of online observation can provide a less obtrusive mechanism for the observation of teaching. It can help provide for moderation of teaching practice without the need for a number of observers being physically present in the classroom. It can also provide an alternative means to link up trainee teachers with subject specialist observers and advisors. Potentially it has significant implications as an efficient and effective tool to support the improvement of teaching and conduct of practitioner research. There is the scope to broaden the classroom teacher’s community of practice and to facilitate greater externality in quality assurance processes.

Sally Sims – Participant at Cricklade College

Personally, I am very keen see the wide-spread use of this technology. I believe it will extend more equality of opportunity to trainee PCET teachers to access mentoring support (through subject specialist observation feedback) where such specialists are geographically remote.

The project team are keenly aware of the proliferation of surveillance cameras often manipulated over IP, and are seeking to create a framework for the developmental use of this technology in educational settings. The project team is keen to apply these applications as communication not surveillance technology; the project has worked with practitioners, teachers and students to establish acceptable protocols for the use of digital technologies in the observation of teaching performance.

M.Dyke Page 9 12/04/2023

Acknowledgements:

This project has worked closely with members of the University of Southampton’s partnership of colleges including:

Cricklade College Defence Educational and Training Services (Army) Isle of Wight College Southampton City College

The project could not have been successful without the full commitment and involvement of each organisation’s students, teachers, technical staff and senior management teams. Particular thanks go to: John Baglow, Steve Bax, Sue Baxter, Paul Buckland, Dave Crome, Stephen Ellison, Eddy Elms, Elliston Ensor, Sue Flawn Paul Gallop and his IT team, Terry Gaskin, Mike Harwood, Richard Hill, Andy May, Paul Morgan, Paul McKillop, Dean Palmer, Mark Powell, Gillian Quinton, Jon Searle, Sally Sims, Margaret Smith, , Sylvia Waghorn, Adrian Ware, and Paul Wyatt. The research team would like to give their whole hearted thanks to the individuals who participated in this research, with have enjoyed working with you and look forward to working together in the future.

ReferencesBECTA (2006) Tech News March 2006 http://www.becta.org.uk/page_documents/research/technews/mar06.pdf

OFSTED (2005) Inspection Report for Southampton City College Available at: http://www.ofsted.gov.uk/reports/manreports/2466.htm (Accessed 20 June 2006).

M.Dyke Page 10 12/04/2023

http://www.ofsted.gov.uk/reports/manreports/2466.htm

http://www.becta.org.uk/page_documents/research/technews/mar06.pdf

Centre of excellence for research and teaching

The Use of Synchronous Digital Video for Observation of, and Feedback on, Teaching Performance in the Learning and Skills Sector.

A presentation for the Standards Unit of the Department for Education and Skills: ITT Reform Pilots

Martin DykeAlan Harding Sue Lajeunesse

July 2006

University of School of Post Compulsory Southampton Education Education and Training

Project Aims

AimTo develop operational policy and procedures and to make recommendations for the remote observation, analysis and feedback of teaching performance.


Objectives

To develop operational policy and procedures and to make recommendations for the remote observation, analysis and feedback of teaching performance. evaluate the observation of teaching performance and the

experience and preferences of teachers, observers and other stakeholders.

develop operational policy and procedures from the experience and evaluations of participants /stakeholders

evaluate the use of digital technology in the observation, analysis and feedback on teaching performance

Department for Education and Skills Standards Unit

M.Dyke Page 11 12/04/2023


Key Findings: Consultation and Questionnaire

Teachers at ease with the ‘real time’ online observation 75% agreed to participate in project

Majority would also agree to process being recorded 70% agree to recording for moderation purposes 66% agree to recording for research purposes 56% agree to recording for non-commercial teacher education purposes

Majority ambivalent about process Prefer to know observer (33% prefer to 3% prefer not to) Prefer observer knows their college (39% prefer to 3% prefer not) Prefer subject specialist (31% prefer to 3% prefer not)

Small group strongly opposed to process (22%) Two thirds of which convinced they would not change their mind Concerns expressed:

‘impersonal’ ‘confidentiality’ ‘intrusive’ ‘disruptive’ ‘loss of atmosphere’‘uncomfortable with being seen on camera’


OnlineObserver(s)

IP

Network


TV

TV

Observer


Teacher and Class

OnlineObserver(s)

IP

Network


TV

TV

Observer

Teacher and Class



Overview: Observation of Teaching

Partnership colleges and training organisations

Live in real time

As Communication not Surveillance Media

Online Observer Present in Classroom

Far End Camera Control

Over Internet Protocol

Process developed in Consultation with teachers

Video Conference Technology

M.Dyke Page 12 12/04/2023


Methods

Action Research Model

Participatory Aspirations

Consultation and Questionnaire 118 participants to date

Protocol built upon feedback from consultation

Direct Observation Data 25 Observations of teaching to date Used standard UK Quality Assurance criteria and classifications

User Experience Interviews and Questionnaire


Key Findings: Observation data

Four of Twenty-four observations failed

Firewall and Networking Configuration Problems

Twenty-one observations fourteen combined online observation with observer in classroom.

Spearman Rho = 0.77 at 0.01 significance level

Online observer graded closer to the mean

Qualitative Descriptions of lesson similar

A Comparsion of Online Observer and In-class Observer Grades of Lessons

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Observation Number

Ob

se

rva

tio

n G

rad

es

Online Observer In-class Observer


Conditions in which tested

Observers shared cultural capital, close working relationship

Observers’ close relationship with organisations and observed

Adult classes in regular classrooms

Good quality audio and video maintained

Far End Camera Control

Online observer presence in classroom

M.Dyke Page 13 12/04/2023


Key Findings: Observation data

Aperture and Context

Online Observer Presence Communication Not Surveillance

Far End Camera Control Essential

Verification of Doubts Online Observation Intense Conscious deliberate active scanning

Observed quickly forget cameras

Observed positive about experience Observation etiquette


Challenges / Issues

Ethical

Organisational & technical Firewalls and Networks

Intensification of Work?

Surveillance & Power

Semiotics

Aperture Macro and Micro

Need to Constantly Verify Doubt


Next Steps

Video Rich Learning Environments

Synchronous High End Video-communication

Webcast development

New configurations for eLearning

Bringing together diverse communities of practice

For further information contact:

Dr Martin Dyke

School of Education

University of Southampton

Southampton SO17 1BJ UK

[email protected]

0044 23 8059 5563

M.Dyke Page 14 12/04/2023

Case Study

Documents

video conferencing systems

education fe college

workbased education

video conference

video data

cricklade college

education colleges

college classroom