ONLINE TUTORING PILOT SEPTEMBER 1, 2008 - August 31, 2009 FINAL

ONLINE TUTORING PILOTSEPTEMBER 1, 2008 - August 31, 2009

FINAL EVALUATION REPORT

Fiscal Agent: ESC-20

Entity Name: Education Service Center, Region 20

Project Coordinator’s Name: Susanna Garza

Project Coordinator’s Phone #: 210-370-5634

Project Coordinator’s Fax #: 210-370-5755

Project Coordinator’s E-mail Address: [email protected]

Project Coordinagtor’s Address: 1314 Hines Ave.; San Antonio, TX 78208

Date submitted: September 30, 2009

The final evaluation of this grant was prepared by Richard Loeffler, Educational Specialist, and Susanna Garza, Coordinator, Instructional Technology & Media, ESC-20.

GOAL OF THE PILOT

The pilot was established on September 1, 2008 with the goal to help increase student achievement and overcome academic obstacles at a select number of high school campuses through the use of live tutors via an online tutoring resource.

Online Tutoring Pilot Final Evaluation Report 1

mailto:[email protected]

mailto:[email protected]

SUMMARY OF ACTIVITIES

Planning for the Online Tutoring Pilot (OTP) began on September 1, 2008 with the development of a Request for Proposal to identify the highest quality online tutoring service in the market. Three responses from vendors were received on October 9, 2009. Products were evaluated by ESC staff members from three regions using a resource description matrix that rated ease of access, quality of resources, variety of subject and grade level, support in Spanish, cost, and other criteria. To ensure uniform, non-biased reporting, sample Mathematics and Science problems and questions were supplied to all evaluation volunteers. Comments were reviewed by an ESC-20 panel to ensure objectivity in the evaluation process. In addition to ESC volunteer evaluations, ESC-20 staff evaluated proposals submitted by vendors using a matrix. This process helped ensure a fair and objective rating of offerings as well as the best value.

Tutor.com was awarded the contract and became the vendor-partner for this pilot. A total of 3,235 student access codes with an unlimited number of online tutoring sessions were purchased at $100 each.

Meanwhile two groups of participating high schools were identified. Group 1 was comprised of public high schools and open enrollment charter high schools that had a state accountability rating of academically unacceptable for either or both of the 2006-2007 and 2007-2008 school years. Group 2 campuses included all other public high schools and open enrollment charter high schools. Campuses were contacted via electronic discussion lists (TecSIG, DigiKnow) and personal email messages and phone calls, and asked to respond to a Request for Statement of Interest (RFSOI #906). A total of 34 campuses were identified (12 in group 1 and 22 in group 2) based on the quality of their statement of interest, level of commitment and adequate infrastrucure. Schools that scored lower in the statement of interest or submitted the form after the deadline were placed on a waiting list.

Just in Time Online Tutoring was officially launched in mid-November 2008 and the 34 participating campuses were assigned single campus logins. On the first week of January 2009, campus logins were replaced with individual student logins to monitor progress more accurately. The distribution of student logins was based on a ratio of approximately 10% of the student enrollment at each campus based on the October 2007 snapshot.

Online tutoring sessions allowed students to log on anonymously and connect with a live tutor in three minutes or less. On their screen, students could see a whiteboard with interactive tools, a chat window—similar to those available in Yahoo Chat or AIM—where they could communicate with their tutor. Tutors never gave students the answers to any problems, but instead helped students work through the problems and find solutions.

Tutor.com has approximately 2,000 tutors on staff, who go through a rigorous selection and certification process. Many tutors are former classroom teachers in the U.S. and Canada. Sessions expired on August 31, 2009 when the pilot was scheduled to end.

Campuses on the waiting list were added in spring of 2009 as group 3, totaling 43 participating campuses in 15 regions. As a result, the number of student logins at some campuses was redistributed based to usage and needs to accommodate more schools.

Below is the list of participating high school campuses and their number of student logins.


District Name RegionSchool Name Group Slots CityDonna ISD 1 Donna High School 2 155 DonnaMcAllen ISD 1 Lamar Academy 2 16 McAllenPharr-San Juan-Alamo ISD 1 Buell Central High School 3 19 PharrPharr-San Juan-Alamo ISD 1 PSJA High School 3 45 San JuanDayton ISD 4 Dayton High School 2 122 DaytonGalveston ISD 4 Ball High School 1 50 GalvestonHardin ISD 4 Hardin High School 3 40 HardinKlein ISD 4 Vistas High School 2 17 HoustonNorth Houston HS For Business 4

North Houston HS For Business 1 10 Houston

Buffalo ISD 6 Buffalo High School 1 21 BuffaloWillis ISD 6 Willis High School 2 189 WillisFruitvale ISD 7 Fruitvale High School 3 30 FruitvleGilmer ISD 7 Gilmer High School 2 65 GilmerMarshall ISD 7 Marshall High School 2 157 MarshallMineola ISD 7 Mineola High School 3 45 MineolaPalestine ISD 7 Palestine High School 1 102 PalestineAtlanta ISD 8 Atlanta High School 1 62 AtlantaBowie ISD 9 Bowie High School 1 49 BowieQuanah ISD 9 Quanah High School 2 16 QuanahWichita Falls ISD 9 Hirschi Magnet High School 2 71 Wichita FallsCedar Hill ISD 10 Cedar Hill High School 2 149 Cedar HillCommerce ISD 10 Commerce High School 1 54 CommercePrinceton ISD 10 Princeton High School 2 35 PrincetonArgyle ISD 11 Argyle High School 2 20 ArgyleAubrey ISD 11 Aubrey High School 2 40 AubreyLittle Elm ISD 11 Little Elm High School 2 123 Little Elm

Mansfield ISD 11Mansfield Legacy High School 3 150 Mansfield

Goldthwaite ISD 12 Goldthwaite High School 2 17 GoldthwaiteMidway ISD 12 Midway High School 2 178 WacoWest ISD 12 West High School 2 44 WestComal ISD 13 Canyon Lake High School 2 42 FischerDel Valle ISD 13 Del Valle High School 2 191 Del ValleTaylor ISD 13 Taylor High School 1 110 TaylorHighland Park ISD 16 Highland Park High School 2 20 AmarilloBrownfield ISD 17 Brownfield High School 1 10 Brownfield


Midland ISD 18Midland Freshman High School 1 105 Midland

Midland ISD 18 Midland Senior High School 2 195 MidlandEl Paso ISD 19 El Paso High School 3 130 El PasoAlamo Heights ISD 20 Alamo Heights High School 2 80 San AntonioHondo ISD 20 Hondo High School 3 50 HondoLa Pryor ISD 20 La Pryor High School 1 33 La PryorSan Antonio ISD 20 Highlands High School 3 125 San AntonioSomerset ISD 20 Somerset High School 1 25 Somerset

Outreach efforts included the creation of the Just in Time Online Tutor logo; distribution of brochures in English and Spanish outlining the features and benefits of online tutoring and bookmarks; campus visits starting on November 11, 2008; and the creation of a dedicated webpage. At http://homeworkhelptx.net users could find easy access to the resource, a training video and answers to frequently asked questions, and a wiki with additional information and best practices. Schools were contacted on a weekly basis and provided with their campus usage statistics. Additional campus visits took place in the spring semester. Posters to encourage use for TAKS preparation were mailed to all campuses in early spring of 2009. Brochures in English and Spanish, bookmark and poster are included in Appendix 1. Campus visits are documented in Appendix 2.

An advisory committee was established to provide guidance during the pilot implementation. The committee was comprised of eleven members including three campus administrators, three campus facilitators, one TEA administrator, one Texas State Library administrator, the ESC-20 educational specialist, and the program manager. Advisory committee meetings were held in January, March and May 2009 via telephone conference. Although some members were not able to attend all meetings, the insight provided regarding challenges, and the ideas to overcome those challenges were extremely valuable. Minutes from each meeting were recorded, published on the website, and emailed to all participating campuses and advisory committee members.

The K-12 Databases were promoted at every campus visit and brief information was added to the back of the Just in Time Online Tutor brochure. Since both resources were state-funded and supported student achievement, ESC-20 staff took advantage of every opportunity to increase awareness and promote usage.


http://homeworkhelptx.net/

http://homeworkhelptx.net/

USAGE AND ANALYSIS OF DATA

The initial focus of the pilot, as detailed in the grant proposal, was to assist students at participating high school campuses to overcome academic obstacles in the areas of Mathematics and Science. Thanks to the availability of online tutoring for numerous courses, student support expanded to include most core content areas in both English and Spanish.

Before issuing the Tutor.com logins, students and their parents/guardians were asked to sign an agreement to ensure that logins would not be shared.

Between the months of November 2008 and August 2009, a total of 9,018 online tutoring sessions took place. The breakdown of these sessions is provided in the table below. More details are provided on pages 6 through 10 of this report.

Subject Area Number of Sessions Percent of TotalMath – Elementary 175 1.9%Science - Elementary 23 0.3%Math – Mid-Level 75 0.8%Algebra 2,512 27.9%Algebra II 1,571 17.4%Geometry 1,367 15.2%Trigonometry 185 2.1%Calculus 680 7.5%Earth Science 26 0.3%Biology 305 3.4%Chemistry 643 7.1%Physics 489 5.4%English 287 3.2%Essay Writing 155 1.7%Social Studies 525 5.8%Total 9,018 100.0%


Chart 1 and graph 1 show usage by content area between November 2008 and August 2009. The data shows that the highest usage was observed during the months of February through May, with the highest peak in April, which was right before TAKS. The data also shows that most online tutoring sessions took place in the areas of Mathematics, and the highest activity was observed in Algebra (27.9%), Algebra II (17.4%) and Geometry (15.2%) followed by Science where Biology, Chemistry and Physics were the most frequently accessed subjects. Only 31 sessions were requested in Spanish.

Chart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by SubjectChart 1: Monthly Online Tutor Usage by Subject

Nov 0 0 0 5 2 15 1 1 0 0 0 0 0 0 1 25Dec 9 7 0 106 53 117 2 68 0 45 67 29 24 9 36 572Jan 17 5 4 202 139 162 18 58 1 44 104 77 37 22 67 957Feb 37 2 11 462 280 249 21 175 5 46 157 95 53 51 185 1829Mar 44 6 46 211 130 183 51 199 7 20 116 68 41 13 82 1217April 53 2 12 828 574 554 32 123 8 66 91 97 68 30 99 2637May 9 1 0 371 262 59 59 46 3 39 80 116 33 7 49 1134June 2 0 2 213 88 19 1 5 1 35 15 6 30 16 2 435July 2 0 0 111 33 7 0 1 0 8 12 0 1 6 2 183August 2 0 0 3 10 2 0 4 1 2 1 1 0 1 2 29Totals 175 23 75 2512 1571 1367 185 680 26 305 643 489 287 155 525 9018


TOTA

L

SOC

IAL

STU

DIE

S

ESSA

Y W

RIT

ING

ENG

LISH

PHYS

ICS

CH

EMIS

TRY

BIO

LOG

Y

EART

H S

CIE

NC

E

CAL

CU

LUS

TRIG

ON

OM

ETRY

GEO

MET

RY

ALG

EBR

A II

ALG

EBR

A

MAT

H-M

ID L

EVEL

SCIE

NC

E-EL

EM

MAT

H-E

LEM


Chart 2 and graph 2 show the breakdown of sessions by grade level. While the pilot targeted exclusively high schools, the structure of the tutoring service allowed access to all grade levels. Students were able to select a grade level each time they logged into Tutor.com. Some students opted for AP and college-level work, while others opted to work at a lower level as shown in the last group of columns on the right side of the chart and graph. The vast majority of students worked on grades 9 through 12.


Chart 2: Monthly Online Tutor Usage by Grade LevelChart 2: Monthly Online Tutor Usage by Grade LevelChart 2: Monthly Online Tutor Usage by Grade LevelChart 2: Monthly Online Tutor Usage by Grade LevelChart 2: Monthly Online Tutor Usage by Grade LevelChart 2: Monthly Online Tutor Usage by Grade Level9th 10th 11th 12th College Intro OTHER

Nov 4 18 1 2 0 0Dec 70 194 179 99 6 24Jan 111 310 330 163 7 43Feb 456 438 497 360 10 84Mar 256 186 312 332 22 109April 762 695 615 456 19 91May 166 305 365 265 21 14June 209 46 124 22 13 21July 97 34 31 8 0 13August 2 0 23 1 0 1Totals 2133 2226 2477 1708 98 400

Graph 2: Monthly Usage by Grade Level

Chart 3 and graph 3 show the ratio of online tutoring sessions by users per month. It is evident that many students took advantage of this resource on multiple occasions. The average number of sessions per student between November 2008 and August 2009 was 6.7, with the highest at 15 sessions per student. The data also indicates that the average duration of sessions was fairly consistent from December 2008 through August 2009, at 19.11 minutes. The time spent in online tutoring during the nine months of the pilot totaled 161,008 minutes, or more than 2,683 hours of individualized instruction.


Graph 4 shows the frequency of use by month during the nine-month duration of the pilot. There were periods of low usage such as November (the pilot was launched the week before Thanksgiving), December (two-week winter break), January (benchmark testing during the first week back to school at numerous campuses), March (spring break), and summer months. The highest number of online tutoring sessions was observed the week of April 13-17, with a total of 621 sessions. The 597 sessions documented in the fall of 2008 took place when generic campus logins were assigned.

As far as the total number of sessions and location, usage varied significantly from one participating campus to another. Some students took advantage of the resource from home. Usage statistics also varied throughout the day, with 62% of sessions taking place during regular school hours, 28% during weekdays before 8 AM or after 3 PM, and 9% of sessions on weekends. Overall, group 2 schools used online tutoring much more regularly than group 1 schools.

Numerous factors were responsible for the differences in usage. Details are discussed in the Challenges section of this report.


Graph 4: Sessions per Month

TRAINING

Training was first provided to ESC-20 staff by a Tutor.com representative. It immediately became evident that formal training would not be necessary due to the simplicity and intuitiveness of the resource.

On November 11, 2008 a videoconference was scheduled to provide an overview of the resource and the program. Fourteen participating schools were present. Between the week of November 10 and the first week of December 2008, all participating schools in groups 1 and 2 received campus visits. A second visit to these schools and newly added schools was completed in February of 2009. During each visit, informal training was provided to classroom teachers, campus OTP facilitators, administrators, and classroom teachers. In addition, demo sessions were scheduled to model the ease of use and effectiveness of the resource. Verbal feedback for these informal training sessions was very positive. No evaluation forms were distributed or collected.

Formal training for students was unnecessary due to the fact that tools used in the online tutoring resource are tools with which students are already familiar, such as chat, interactive whiteboard, drawing, and the Web. Once students were able to login, they worked independently.

OVERCOMING CHALLENGES

Due to unpredictable circumstances, the pilot faced several challenges during its nine-month implementation. The comments below are based on observations during campus visits; student, teacher and school leaders’ comments; responses from email messages and phone calls; and feedback from advisory committee meetings.

✦ Toward the beginning of the pilot implementation, two campus principals resigned. At one of these campuses the facilitator resigned as well. Both schools decided to drop the pilot.

✦ Even though online tutoring services were available anytime, anywhere, access from school was higher due to the fact that many students had no Internet access at home. During campus visits, students were reminded that they could access the resource from any public library. In addition, schools were encouraged to provide access to the resource before and after school.

✦ Although intense efforts were made to launch Just in Time Online Tutor as quickly as possible in the school year, it wasn’t until January 2008 when most participating schools started taking advantage of the pilot. This resulted from a combination of school holidays (Thanksgiving and winter break) and time needed to identify participating students.

✦ Some schools reported a delay in issuing student logins due to the lack of parent signature of the Just in Time Online Tutor Student Agreement Form. In order to overcome this obstacle, it was immediately decided that student signatures would suffice.


✦ In order to generate awareness and increase usage, ESC-20 staff monitored statistics on a weekly basis, and followed up with campuses via email, phone calls and campus visits. Training on the use of the resource and demo tutoring sessions were offered to all participating campuses.

✦ In order to avoid the possibility of a negative stigma associated with online tutoring and to generate enthusiasm for the resource, facilitators and administrators at participating campuses were encouraged to distribute student logins among high achievers as well as students who were behind with academics.

✦ Introducing online tutoring as a new initiative once the school year had already started, or when credit recovery and TAKS supplemental programs were already in place required intense outreach efforts.

✦ At campuses when online tutoring was used by students taking advanced courses, it was reported that use of the tutoring service was linked to the amount and complexity of classwork, and student expectations.

✦ While most students were able to connect to a live tutor in a few seconds, others had to wait for several minutes and occasionally they are not able to connect at all during the time allowed. This happened more often with some courses (i.e., Chemistry) as well as at the end of the school day. It was suggested that if two or three students were working on the same problem, they should work together at the same workstation to avoid long waits and bandwidth issues.

✦ Some students were disappointed to find out that there was a one-hour limit for each session, so when they needed to extend their tutoring session, they reconnected a few minutes later and started a new one.

SUCCESS STORIES

In spite of the brevity of the Online Tutoring Pilot and the challenges faced, numerous positive results were observed.

✦ Once the students began using online tutoring, they were overwhelmingly positive about both the experience and its benefits. They seemed to especially like the anonymity of the interaction. This was communicated through conversations with students during campus visits as well as comments from end-of-session surveys. Only a small percentage of students completed the surveys, though. The complete list of student comments is included in Appendix 3 of this report.

✦ In February of 2009, a participating campus in Waco was featured in the local news channel, KXXV News Channel 25. The transcript is included in Appendix 4 of this report.

✦ Online tutoring was very helpful when used for credit recovery, preparing for tests (including TAKS), when the teacher was not available, and to accelerate students who needed to be challenged.


✦ The tutoring approach was very beneficial as well. Tutors never provided students with answers to problems. Rather, they helped them use problem solving skills and draw their own conclusions.

✦ Efforts to promote the use of online tutoring at campuses where the pilot implementation was successful included the following: - Issuing of student certificates (most sessions, most minutes, most improved);- Drawings for popcorn, books, iPods and iTunes gift cards; - Mobile labs available after school several days a week exclusively for online tutoring plus transportation home provided;- Targeting specific critical areas, such as essay writing, Math or Science;- Availability of computers in the library specifically dedicated to online tutoring to be used anytime during the school day (students were given a pass with the login information when extra help was needed);- Credit recovery teachers used online tutoring to individualize instruction;- Reminders of the availability of online tutoring during morning announcements;- Sharing lists of online tutoring participating students with all teachers at the campus and at department meetings;- Allowing students to store the login information in their cell phones;- Taping bookmarks to computer workstations at the library or computer lab as constant reminders;- Allowing counselors to refer students for online tutoring;- Promoting the resource not as something that students must use but rather as something they can use to help them improve their grades;- Students who had a positive experience encouraged reluctant users to use the resource;- Integrating online tutoring in a TAKS Academy and summer school ;- Encouraging ESL students to use the online tutoring service available in Spanish; and- Including an information letter to parents explaining the implementation and benefits of online tutoring.

SURVEY RESULTS

Campus administrators and facilitators at participating campuses were asked to complete an online survey on the last week of school. The survey was open for 7 days. Not all campus administrators or facilitators responded to the survey.

As shown in the survey details below, the opinions about the quality of the introduction, ESC-20 support and ease of use were very positive. The rating of the service provided by Tutor.com, the availability of tutors and variety of subjects also reflected an overall positive opinion. A minority of campuses -probably those that took more aggressive steps to promote the resource and implement the pilot- shared that online tutoring assisted their students to improve TAKS performance and end-of-year exams, increase attendance and to reduce discipline referrals.

The overall impression of online tutoring as a valuable resource to support student success was very positive, which parallels the impression perceived during camps visits, student comments and other communications throughout the year. The top concerns were the delay in the implementation of the pilot and the occasional delay in connecting with tutors.





CONCLUSIONS

In spite of the introduction of the online tutoring pilot in late fall, the short period of implementation, and the differences in usage statistics, it was evident that once the resource was made available to students, their response was overwhelmingly positive.

Although all campuses were eager to participate in the pilot and committed to the requirements listed on the Statement of Interest, some were not able to implement online tutoring due to a variety of unforeseen circumstances.

While some adults were unfamiliar with the technology, students were extremely comfortable with the chat and whiteboard features as well as the entire experience. Once students were provided access, they used the resource extensively.

Live online tutoring is expensive. At $100 per student, only a limited number of campuses -and therefore students- could be included in the pilot. The initial data captured in 2008-2009 is encouraging and worth exploring the long term benefits of online tutoring.


CONTINUATION OF THE PILOT

ESC-20 and Tutor.com agreed to transfer the unused student logins to the 2009-2010 school year. It was established that instead of individual student logins, 10,000 online tutoring sessions would be available on a first come, first served basis until May 31, 2010 to support the continuation of the pilot.

All Texas schools were invited to participate in the pilot extension, including schools participating in the pilot during 2008-2009.


ONLINE TUTORING PILOT SEPTEMBER 1, 2008 - August 31, 2009 FINAL

Documents