Flynn_SummativeProject

8/4/2019 Flynn_SummativeProject

1/16

1

Formative Report for Educational ResourcesPrepared for: ThinkHard Design Studies, LLC

Group Members:Daniel Flynn

Tony Hetrick

Jennifer Hornyak

EDTECH 505-4172

April 19, 2011


2/16


3/16

3

Executive Summary

A small team was contracted by ThinkHard Design Studies, LLC to conduct a detailed analysis

of 29 educational websites (WBLEs). In order to analyze each website, a rubric created by

Baya'a, Shehade, & Baya'a (2009) was distributed in electronic form to experts in the field of

education. Both quantitative and qualitative data was collected based on the expert feedback.

The rubric helped to evaluate each website in the following areas: usability, content, educational

value, and vividness.

Overall, the WBLEs were only rated as adequate with the educational value category receiving

poor reviews. However, the contentcategory received relatively high reviews. Of the evaluated

WBLEs, 44% contained an informational instructional style while only 9% used a drill-and-

practice instructional style. To improve the WBLEs, it is recommended that both the usability

and educational value aspects are modified to meet the rubric criteria.


4/16

4

Results and Interpretations

Usability

A majority of the WBLEs that were reviewed contained significant usability issues. The main

issue had to do with navigation, which allows users to tell where they are, where they have been,

and where they need to go next. Also, design and lack of enjoyment contributed to the low

score. The below graph displays the average score of all WBLEs reviewed.

The data showed a strong correlation between page design and navigation. When a site wasevaluated on its design, only 12.5% of the time was there a navigation score that ranked higher

than the design score. This was also evident from the review comments. For example, a

reviewer ofZebra Keys stated that the site is too busy. It has too many advertisements for the

audience to navigate with flow. Another reviewer forKids Biology stated that the ads and

extra links were constantly interfering with the instruction making it nearly impossible to process

the navigation or efficiently gain knowledge from the text.

Other than interfering with the navigation, the design of the WBLEs contributed to other

usability issues. The reviewers stated that the sites were overwhelming by being too busy,

used inappropriate or too small font size, or were lacking contrast between the text and the

background. The lack of good design contributed to making certain sites difficult to use, despite

good content. A reviewer ofESL Go explained that while the site provided good grammar help,

it was poorly designed which led to confusion and a jumbled mess.


5/16

5

While the majority of the sites contained usability comments that listed problems, not all sites

were poorly designed. For instance, a reviewer stated that The Musical Mindwas very clean

and professional looking with no ads and nice use of color.

To improve usability, CARP (Contrast, Alignment, Repetition, and Proximity) design principles

should be followed. These principles are the guidelines that will enable the WBLEs to look

clean, neat, and professional (Williams & Tollett, 2006, p. 113) and the WBLEs will better

communicate their content and they will be more enjoyable (Williams & Tollett, 2006).

Content

A majority of the WBLEs that were reviewed contained significant doubt as to content

authority. For the most part, however, the WBLEs content for relevance, sufficiency and

appropriateness are quite relevant for users and show adequate ratings for the most part. The

following chart shows the averages of the results of the WBLEs.

The results showed that reviewers found most of the websites relevant in terms of content.

However, there is a high doubt on the authority of the resources used in creating the content

of the site. Reviews of some the WBLEs authority (Q4.2) were mentioned in the comment

sections (Q4.6) and they included:


6/16

6

WBLE Comment

Camera Work Tutorial The site is run by experts in their fields, (based on information of

profiles) however, not sites or other resources are mentioned

to back up their claims of information

Franchise Webquest I would have given the authority a better mark if the links to

back up information were working.

Kids Biology There are no sources, so it is unclear where the information is

coming from. Overall, I felt the level of information provided

was not sufficient

There are 43% of sites that have an excellent rating for authority of websites. However, this

leads to an issue of inclusion of sites not based on authority in the portal page since the sites

that dont have proper resources will likely lose credibility with potential users. Most of thereviewers did like the overall content but lacked the confidence in referencing valid sources that

would give an authority for inclusion in the portal. This being said, reviewers believe that the

WBLEs are relevant in their appropriateness with a strong level of difficulty and sustainability.

For the sites that have content that is backed by proper resources, they often score high on

relevance, sufficiency, and appropriateness. Reviewers often comment positively about the

overall content of the site. For example, The Genius of Leonardo da Vinci site, a reviewer

commented that the content may have been slightly excessive, but I'd rather have that than

insufficient. These comments were frequent thought out the WBLE and show a general

concern of how a site should be constructed with proper resources.

Overall, content scored well, except for the amount of sites that lack proper authority (resources,

working links to resources, dependable, etc.) that would bring credibility to the portal. One

possible solution could be to create a second section within the portal for sites that lack proper

resources and label it something other than primary sites.

After analyzing the content for the various age groups, there seems to be a heavy disposition

of sites for 4-6th grade and 7-9th grades, making up 56% of the sites. 16% of the sites were

made for 10th-12th grade and 9% of the sites were for the 1st-3rd grade. Kindergarten ages sitesrepresented 3%, or only 1 WBLE. See the below chart for additional detail.


7/16

7

Educational Value

Overall, reviewers found the WBLEs to have varying degrees of educational value. The rubric

addressed each website in the areas of: instructional style, instructional setting, and instructional

application. While some variance was found in these areas, it would be beneficial to expand the

portal to include WBLEs that represent a wider variety of instructional styles and instructional

settings as well as address a wider range of instructional applications.

Instructional Styles

The WBLEs were evaluated for the following instructional styles: Informational, Drill &

Practice, Tutorial, Simulation, Instructional Game, and Mixed Mode which is a WBLE that

includes a combination of one or more of the above instructional styles. According to the data,

the majority (44%) of the sites represents WBLEs that are informational and contain little or no

student interaction. None of the WBLEs were found to be of the simulation instructional style.

Please see the below chart for additional information.


8/16

8

Instructional Setting

The WBLEs were rated by the expert reviewers in terms of which setting each WBLE would be

most effective. The majority of the sites, 53% or 17 WBLEs, in the list were rated to be most

effective for individuals to use outside of class (at home or after school). 25% of the sites were

found to be appropriate for a whole class instruction using a projector or large display. It would

be beneficial to include a wider variety of WBLEs appropriate for all settings on the portal so

that educators would be able to utilize the portal to a greater extent.

Instructional Application

The instructional application that was determined to be most appropriate was basic instruction

for all learners. The graph below shows that while there are a range of instructional

applications among the WBLEs evaluated, the majority would be most advantageous to be used

in basic instruction. To improve the educational value of the WBLES on the list, there should be

some consideration to including a wider variety of instructional activities.


9/16

9

Vividness

Overall, the WBLEs ranked as adequate in the category of vividness. According to the

rubric, vividness includes sufficient links to other pages as well as updated and current content

and links. Current information and workable links are important for a WBLE to obtain a high

scoring review. Expert reviewers rated the links found in the WBLEs that point to other pages as

adequate with a mean result of 2.06 on a scale of 1 to 3, with 1 being the best ranking possible.

In addition, experts rated the WBLEs in terms of updated content and links as adequate with a

mean result of 2.56. It is recommended that additional sites be added to the portal which includesupdated links that point to appropriate and supplementary content, as well as sites that contain

current content.

Page Comparisons

Upon comparing the sites on the two different pages, they were found to be comparable.

Overall, there was only 2.6% separating the two pages, with page #2 performing slightly better.

The mean result for page #1 was 1.94 and the mean result for page #2 was 1.89.


10/16

10

Site Rankings

Highest Rankings - Most Appropriate Instructional Activities

Among the 10 sites that were rated the highest by the expert reviewers, the most appropriate

instructional activity is basic instruction for all learners. Below is the list of the instructional

values from the top 10 sites:

1. 5 WBLEs - Basic instruction for all learners

2. 3 WBLEs - Enrichment instruction for students who will benefit from more advanced

activities

3. 1 WBLE - Differentiated instruction for students with varying learning styles

4. 1 WBLE - Remedial instruction for students who did not initially grasp the concept

Highest Rankings - Common Elements

Among the 10 sites that were rated the highest by the expert reviewers, their common elements

were from the categories ofusability and content. Below are their rankings in order:

1. Enjoyment (Usability Q3.6): Multimedia and humor that are relevant, clear and

appropriate to the subject matter

2. Readability (Usability (Q3.7): Users can easily view contents of the web pages

3. Relevance (Content Q4.3): Information focuses on the main topic and does not include

irrelevant or marginal data

4. Authority (Content Q4.2): Information relies on authentic organizations and dependable

resources


11/16

11

Lowest Rankings - Common Elements

Among 10 sites that were rated the lowest by the expert reviewers, their common elements were

from the categories ofeducational value and vividness. Below are their rankings in order:

1. Help Tools (Educational Value Q5.8): Help tools to assist users in solving problems

2. Updating (Vividness Q6:3): Designers continuously update the information, content and

links

3. Communication (Educational Value Q5.5): Users are able to interact with designers of

the WBLE and other peers

4. Rubric (Educational Value Q5.7): Activities include rubrics to evaluate user performance

Overall Recommendations

In order to improve the web portal, it is recommended that the data presented in this report

be used to provide additional WBLEs on the portal that would rank highly amongst expertsespecially in the area of educational value and usability. The rubric, provided in the appendix,

should be used in choosing any additional sites. It may also be considered that the lowest ranking

WBLEs be removed from the portal and replaced with higher ranking WBLEs. The chart below

depicts the WBLE in terms of overall ranking of the WBLEs. Sites that should be considered to

be replaced in the portal include:Kids Biology, ESL Go, ZebraKeys, and Free Drawing Lessons.


12/16

12

When choosing new WBLEs for the portal, a specific focus should be placed on the portal

including WBLEs that are strong in the area of content so that educators will be able to use

the WBLEs across the curriculum. Based on the data, an additional recommendation is to add

more websites that are geared towards the Kindergarten grade level. It is also very important the

portal includes websites that address a variety of instructional applications across many different

educational settings. In addition, it would be beneficial to categorize the websites on portal by

grade level, allowing the users to easily access proper content immediately.


13/16

13

References

Baya'a, N., Shehade, H. M., & Baya'a, A. R. (2009). A rubric for evaluating Web-based learning

environments. British Journal of Educational Technology, 40(4), 761763.

Williams, R., & Tollett, J. (2006). The non-designers web book: An easy guide to creating,

designing, and posting your own web site (3rd ed.). Peachpit Press CA.


14/16

14

Appendices

Charts / Graphs

Average Scores among All WBLES


15/16

15

A rubric for evaluating Web-based learning environments (Baya'a,

Shehade, & Baya'a, 2009)

Usability Criterion

Purpose The purpose, benefit and importance of the WBLE should be presented

clearly.

Homepage The homepage should include a well-labeled, clearly defined table of

contents.

Navigation Users need to know where they are, where they have been and where to

go next.

Design Web pages have to be neat, simple and not overstuffed, and background

should not interfere with text.

Enjoyment Multimedia and humor are recommended, but should be relevant, clear,

appropriate to the subject matter, and should not be heavy in a way that

slows down the presentation of the web pages.

Readability Users should be able to read the content of the web pages easily, by

proper use of text, sounds and images.

Content Criterion

Authority Information in the WBLE should rely on authentic organizations and

dependable resources in the field of the presented material.

Accuracy Information should be accurate and based on factual knowledge and

professional thinking.

Relevance The information provided ought to focus on the main topic of the WBLE,

and should not include irrelevant or marginal data.

Sufficiency The amount of information in the WBLE should be sufficient but not

excessive.

Appropriateness The information should be presented in an appropriate method and

suitable level of difficulty to the target users.


16/16

16

Education Value CriterionLearning

activities

WBLE should provide learning activities that expose the users to new

information and encourage them to construct new knowledge and

educational substance.

Activity plan Each learning activity should be accompanied with a clear and suitable

activity plan.

Resources The activities should include well prepared content presented in various

ways, as well as references to additional resources on the Web.

Communication Users should be able to interact with the designer of the WBLE andwith their peers through different communication tools (email, chat and

forums).

Feedback Users performance and outcome of each activity should be tracked and

evaluated through a mechanic or human feedback.

Rubric Each activity ought to include a rubric to evaluate the users

performance.

Help tools WBLE ought to include assistant tools that help the users solve

problems (technical, contextual and didactical) that they might face inthe process of performing the activities.

Vividness Criterion

Links WBLE should direct users to other web pages: in other sites or in the site

itself. These links should always be active, adequate and enriching.

Updating WBLE designers should update the information, content and links of the

WBLE continuously.

Flynn_SummativeProject

Documents