Evaluating Open Educational Resource (OER) Objects

Evaluating Open Educational Resource

(OER) Objects

Using the Achieve OER Evaluation Tool on OER Commons

CC BY Achieve 2013

http://creativecommons.org/licenses/by/3.0/us/

2

Open: Containing an open license - no restrictions on remixing or reusing[Having no enclosing or confining barrier; not restricted to a particular group or category of participants]

Open Educational Resources:[With Webster’s Definitions]

O

E

R

Educational: Used for teaching and learning[Pertaining to the action or process of educating or being educated]

Resource: Object used to support an effort or task[A source of support or aid, especially one that can be readily drawn upon when needed]

3

What are OER?

• OER are teaching, learning, and research resources that contain an open license.

• They provide extraordinary opportunities for educators to freely share knowledge and resources.

• They offer great potential for instructional innovation and networks for sharing best practices.

• There are, literally, millions of OER objects available online.

Open Educational Resources (OER)

4

OER objects may include (but are not limited to):• Images• Applets• Games • Worksheets• Lesson plans• Original-source texts• Assessments• Units• Textbooks • Teacher/learner support materials

o NOTE: Any smaller component of a more complex object, that can exist as a stand-alone, alsoqualifies as an “object” (e.g. a unit in a textbook, a lesson in a unit, or an activity in a lesson).

Open Educational Resources (OER)

5

The Achieve OER Evaluation Rubrics

How and why were the Achieve OER Rubrics created?• Educators using the vast system of sharing need a method for filtering OER

to meet the needs of their students.

• The purpose of the rubrics is to provide a structure for evaluating an online resource in a systematic, purposeful and comprehensive way.

• The rubric criteria are based on Achieve protocols used to assist states in alignment and quality review studies.

• The Achieve OER Evaluation Tool, hosting the rubrics on OERCommons.org, allows a user to identify, evaluate, and sort objects based on the specified essential elements of quality.

6

How do the Achieve OER Rubrics work?

• The rubrics represent an evaluation system for objects found within Open Educational Resources.

• They are hosted as an online evaluation tool on the repository, OERCommons.org, but can also be used independently.

• They are applied to any content area. o NOTE: At this stage only Common Core Standards for English Language Arts/Literacy

and Mathematics are available in the Achieve OER Evaluation Tool. Content standards for History, Social Studies, Sciences, and technical subjects are not yet available.

• Each rubric is applied independently to an object.

• They rate the potential, not the actual, effectiveness of an object.

• They use a five-point scoring system to describe levels of quality.

Applying the Achieve OER Rubrics

7

The Detailed Scoring Descriptors are Specific to Each Rubric

3: Superior

2: Strong

1: Limited

0: Very weak / None

N/A: Rubric not applicable to the object

NOTE: N/A should be used when a particular rubric does not apply to the object being rated. This is not a pejorative score; it simply means it would be inappropriate to apply this rubric to the particular object.

The Rubrics’Common Scoring Scale

8

There are Seven Rubrics on the Achieve OER Evaluation Tool

Rubric I. Degree of Alignment to StandardsRubric II. Quality of Explanation of the Subject MatterRubric III. Utility of Materials Designed to Support TeachingRubric IV. Quality of AssessmentRubric V. Quality of Technological InteractivityRubric VI. Quality of Instructional Tasks and Practice ExercisesRubric VII. Opportunities for Deeper Learning

NOTE: Sometimes one or more of the rubrics may not be relevant to a particular object. In those cases, a rating of N/A is appropriate.

The Rubrics

Using the Achieve OER Evaluation Tool on OERCommons.org

OERCommons.org hosts the rubrics as the Achieve OER Evaluation Tool

10

Achieve OER Evaluation Toolon OERCommons.org

http://www.oercommons.org/

Go to the OERCommons Website to Register and/or Log in:


Register and Log In

11


From the Home screen use the browse options to search for OER:

• Use the Search screen by typing a key word or object title

The OERCommons.org Home Page

12

• Browse All allows you to see a listing of all OER categories available

13


• My OER allows you to go immediately to the group of OER you have saved

14


• Groups allows you to create a group or to view groups created by you or others

15


16

• Contributeallows you to create an OER or submit a resource from a website to OERCommons


• You can also go directly from this screen to the Advanced Search

17


Discover New Resources: OERCommons posts links to the most recent additions to the site.

18


• Help: Find answers to frequently asked questions and get information about Conditions of Use, OERCommons in general, OER objects, Open Author, member tools, etc.

19


• Browse All takes you to this screen where you can further refine your search by selecting a particular subject area, grade level, or materials type.

On this screen you can also see the number of OER available in each category.

Browse the Site for Objects

20

• My OER allows you to see only those OER you have saved.

During your searches you may find an OER that you may want to find easily later. In those cases you can save the objects to My OER and return to your personal list at any time.

21

Your Name

Access Your Saved OER

22

Search the Site Using Groups

Your group search can be filtered in much the same way as an object search.

Search groups by:

Grade level

Subject

Type

Groups allows you to create a group, to view all groups, or to view the groups you have created. Groups allows schools, districts, etc. to more easily share OER.

Contribute allows you to create an OER or submit a resource to OER Commons from the web

23

Contribute a Resource

Create an OER using Open Author

Upload a resource from the web

Refine Your Search for OER

24

Advanced Search allows for more refined searches based on:

• Alignment with a specific CCSS

By using the drop-down menus here educators are able to quickly view all objects that have been aligned to a specific Common Core subject, a grade level, a learning domain, or even a specific standard.


25

Advanced Search allows for more refined searches based on:

• Evaluation with any of the Achieve OER Rubrics

Educators are able to find all objects evaluated using a specific Achieve OER Rubric. This will allow them to quickly find, for example, an object that has been rated for the quality of its assessments or its technological interactivity.


26

Advanced Search also allows for more refined searches based on any or all of the following:

key words (using the search bar) any of six subject areas grade level language material type content source primary user member activity conditions of use

Once you have refined your search options you will be sent to a list of those OER objects meeting your search criteria.

This example shows the first few of 8,709 primary and secondary Humanities objects.


27

You can then change the number of items shown per page

and …

Reorganize your list by title, relevance, rating, date, or the number of visits.


28

You can scroll through the object list manually or by using the page numbers.


29

By clicking on Expand All you can see every Collapse All (Complete Item Description) for the objects on your list.

By clicking on Collapse All you can close all item descriptions to save space.


30

By un-clicking any criteria in the left column you can refine your search even further.


31

Here you see the example list of the first three objects from our refined search.

For each object you can see:

• The title

• The subject

• The grade level

• The collection contributor


32


33

Clicking on the collection title for any object will allow you to see a list of all objects contributed by that author.

This can be particularly helpful for educators who find a particular contributor that seems to fit their needs.

If you click on (Complete Item Description) for any individual object, you will be able to read the full text of the object’s abstract.


34

35

There are two ways of approaching the evaluation of a resource in OERCommons:

1) Holistically evaluating the object with a single 5-star rating, (similar to the rating systems used on Netflix, Yelp and Amazon)

2) Separately evaluating each of the components of quality, as with the Achieve OER Rubrics

Ratings and Actions

36

In the list view for the objects on OERCommons you will see a symbol to the right of the object’s title in the list. It may look like one of these:

Ratings and Actions

This symbol shows five “empty” stars and indicates that the object has not yet been evaluated with either system.

This symbol shows that the object has been given four stars using the holistic 5-star rating system but has not been rated using the Achieve OER Evaluation Tool.

This includes this symbol, , along with the 3-star rating, indicating that the object has been evaluated by at least one rater using both the 5-star rating system and the Achieve OER Evaluation Tool.

On the far right for our example search list we see the present state of the objects in the evaluation cycle.

• This object has been holistically rated with the 5-star system.

Search the Site for OER

37

• This object has been evaluated and rated using both the 5-star rating system and the Achieve OER Evaluation Tool


38

• This object has not been rated at all


39

Conditions of use are also identified here. There are four possible conditions:

Conditions of Use

40

No restrictions on remixing, redistributing or making derivative works.

Remixing, redistributing or making derivatives works comes with some restrictions, including how it is shared

Do not remix or make derivative works

Everything else/All Rights Reserved. US-based educators have certain permissions under Fair Use and the TEACH Act.

For more detailed information click on [Help] on the OER home page, found next to login, and then Conditions of Use.

Two of these are “Remix and Share”

Conditions of Use

41

This one is “Read the Fine Print.”

Conditions of Use

42

Each item in the list offers three to be taken:

From this list screen you can • Save this item to My OER

• Add a tag (keywords, etc.) to this item

• Align this item to the Common Core State Standards for Mathematics or ELA/Literacy

Actions

43

After finding an object of interest, click on the title to move to the launch page for that OER.

Selecting an OER Object

44

From this object’s launch page you can do the following:

Read the abstract

The OER Object’s Launch Page

45


46

See previous users’ holistic ratings


47

• See average ratings from all previous evaluators using the Achieve OER Evaluation Tool.


48

• See a list of standards aligned by previous users


49

• View the resource on its home website.

Clicking here will take you to the site where the object is hosted. This is the only way to fully study the object and align/evaluate it accurately.


50

Provide additional standards to the alignment

Click here if additional standards are needed in the alignment.


51

Leave a comment about this object for yourself or for subsequent users


52

• Evaluate the object

Click here when you are ready to begin rating the object.

53

After selecting an object and clicking on [Evaluate Resource] on its launch page, you will be taken to the object’s website of origin and to the Achieve OER Evaluation Tool.

Achieve OER Evaluation Tool:Guidelines & Reminders

54


You can minimize the Tool here in order to view the object in a full screen.

Clicking on the [x] will close the Tool.

55


On the Tool’s introduction page you will find: • A link to a PDF version of the

rubrics on Achieve’s website

56

After selecting and object and clicking on on its launch page, you will be taken to the Achieve OER Evaluation Tool. On the An explanation of the Tool • A link to Achieve.org an object• A link to the• A link to a video providing

an overview of the rubrics


http://achieve.org/

57

After selecting and object and clicking on on its launch page, you will be taken to the Achieve OER Evaluation Tool. On the opening page, seen at the right, you will find: • A link to A• A link to a the rubrics on

Achieve’s website • An explanation of the

Tool, including guidelines and reminders.

• Guidelines & Reminders about evaluating an object

• A link to the• A link to a video providing an

overview of the rubrics


58

• A link to the Common Core State Standards


http://www.corestandards.org/

http://www.corestandards.org/

59

• A link to a video providing an overview of the rubrics


60

And …• A link back to the OER

Home page can be used at any time during your review process.


61


When you are ready to begin click on

62

Here is a full view of the Evaluation Tool’s start page:

To download a full text version of the rubrics at anytime during the review process click here:


63

The Achieve OER Evaluation Tool rubrics can be used in any order and can be selected by simply clicking on the rubric title.

Remember to click after rating for each rubric to ensure that your work is saved.

You can easily tell if your work was saved by the symbol at the beginning of each rubric title.

The screenshot below shows the first three rubrics on the Tool’s cover page after a user has started rating:

indicates Rubric I was viewed but either not yet rated or the rating was not saved

indicates the rating for Rubric II was saved

o NOTE: Rubric III in this screenshot has not yet been viewed.


64

The Achieve OER Evaluation Tool - Applying Rubric I:

We will begin with Rubric I: Degree of Alignment to Standards

Click here to view an instructional video about Rubric I.

Achieve OER Evaluation Toolon OERCommons.org: Rubric I

65


IMPORTANT: Selecting skip as not applicable indicates an N/A rating, which, for Rubric I, would apply only if there are no CCSS that align to the object and/or there is no intention by the object to align to standards. If previous users have already aligned standards they will need to be rated before moving to other rubrics.

Click on skip as not applicable if Rubric I is does not apply.

66


If alignment to the CCSS is needed click on align resource.

67

The Alignment Process:After you click on align resource you will be taken to an alignment screen to select the curriculum standard for alignment:

First identify the subject area:

NOTE: at any time during the process you can click on [Return to Evaluation] to return to the opening evaluation screen.


68

Next select the grade level:


69

Then the learning domain:


70

And finally the alignment tag:


71

After selecting an alignment tag you can read the full text in the yellow box at the upper right.


72

Click on to show the standard as selected for alignment

Or on to go back to the rubric selection list and/or start the process over.


73

If you click on [Add Selected Tag] you will see your selection’s title and code at the upper left corner of the selection screen.

Now you can search for more standards by selecting a different alignment tag or changing learning domain or even grade level.


74

You can remove a standard you have selected from the alignment by clicking the [x] next to the title.


75

When you are satisfied with your alignment selection(s), click on


76

After you [Return to Evaluation], you will see your selected standards listed in the center of the screen for Rubric I. Click the standard you want to rate and it will be highlighted in orange.

Achieve OER Evaluation ToolRubric I

77

Hover over the rating numbers to see an abbreviated version of the rating descriptors for Rubric I.

Select the most appropriate rating of the for the degree of alignment for each standard identified.


78

Across the bottom of Rubric I you see options to: • Leave a [+ Comment],• Clear rating,• Move to the [Next/Previous

standard] (or you can click on any un-highlighted standard),

• [Save & Go to Next Rubric] (You can also click on any rubric title to leave Rubric I, but your ratings will not be saved.)


79

Using OERCommons.org to evaluate an object for Rubric II: Quality of Explanation of the Subject Matter

Begin applying Rubric II by clicking on the title line for Rubric II or on [Save & Go to Next Rubric] at the end of Rubric I.

Remember that your ratings for Rubric I will need to be saved.

Achieve OER Evaluation ToolRubric II


80

To view an instructional video about Rubric II click here.


81

Hover over the rating numbers to see an abbreviated version of the rating descriptors.

Review the object with your focus on how well it explains the subject matter and select the appropriate rating.


82

Across the bottom of Rubric II you see options to: • Leave a [+ Comment],

• Clear rating,

• [Save & Go the Next Rubric] (You can also click on any rubric title to leave Rubric II, but your ratings will not be saved.)


83

Using OERCommons.org to evaluate an object for Rubric III: Utility of Materials Designed to Support Teaching

Begin by clicking on the title line for Rubric III or on [Save & Go to Next Rubric] at the end of Rubric II.

Remember to save your ratings for all rubrics.

Achieve OER Evaluation ToolRubric III


84

To view an instructional video about Rubric III click here.


85


Review the object again with your focus on the utility of the teacher support materials and select the appropriate rating.


86

Across the bottom of Rubric III you see options to: • Leave a [Comment],

• Clear rating,

• [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric III, but your ratings will not be saved.)


87

Using OERCommons.org to evaluate an object for Rubric IV: Quality of Assessments

Begin by clicking on the title line for Rubric IV or on [Save & Go to Next Rubric] at the end of Rubric III.


Achieve OER Evaluation ToolRubric IV


88

To view an instructional video about Rubric IV click here.


89


Review the object again with your focus on the quality of the object’s assessments and select the appropriate rating.


90

Across the bottom of Rubric IV you see options to: • Leave a [+ Comment],

• Clear rating,

• [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric IV, but your ratings will not be saved.)


91

Using OERCommons.org to evaluate an object for Rubric V: Quality of Technological Interactivity

Begin by clicking on the title line for Rubric V or on [Save & Go to Next Rubric] at the end of Rubric IV.


Achieve OER Evaluation ToolRubric V


92

To view an instructional video about Rubric V click here.


93


Review the object again with your focus on the quality of the interactivity of the object’s technological feature(s) included in the object and select the appropriate rating.


94

Across the bottom of Rubric V you see options to: • Leave a [+ Comment],

• Clear rating,

• [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric V, but your ratings will not be saved.)


95

Using OERCommons.org to evaluate an object for Rubric VI: Quality of Instructional and Practice Exercises

Begin by clicking on the title line for Rubric VI or on [Save & Go to Next Rubric] at the end of Rubric V.


Achieve OER Evaluation ToolRubric VI


96

To view an instructional video about Rubric VI click here.


97


Review the object again with your focus on the quality of its instructional and practice exercises included in the object and select the appropriate rating.


98

Across the bottom of Rubric VI you see options to: • Leave a [+ Comment],

• Clear rating,

• [Save & Go to the Next Rubric] (You can also click on any rubric title to leave Rubric VI, but your ratings will not be saved.)


99

Using OERCommons.org to evaluate an object for Rubric VII: Opportunities for Deeper Learning

Begin by clicking on the title line for Rubric VII or on [Save & Go to Next Rubric] at the end of Rubric VI.

Remember to save your ratings for every rubric.

Achieve OER Evaluation ToolRubric VII


100


Click here to read more about Deeper Learning from the Hewlett Foundation’s website.

101

To view an instructional video about Rubric VII click here.


102


Review the object again with your focus on opportunities for deeper learning and select the appropriate rating.


103

At the bottom of Rubric VII you see options to: • Leave a [+ Comment],• Clear rating


104

Make sure you have rated all the rubrics in the Evaluation Tool. Return to any rubrics by simply clicking on the rubric title.

You can tell if any are not saved by the symbols at the left of the rubric title.

indicates the rubric is not yet rated or your rating is not saved

indicates your rating has been entered and saved


105

When all the rubrics have been rated saved you can view the final results by clicking on [Save & View Results] for Rubric VII or on [View Results]. (Remember to save your results for Rubric VII.)


106

After all seven rubrics are rated and saved on this results view you will see: • your current ratings and • average ratings for all

previous users

Achieve OER Evaluation Tool:Finalize Your Review

107

In the example you see the final results for a fourth grade math object.

When you have rated and are satisfied click on [Finalize OER Review].

If you want to change any ratings you can click on [Go Back & Change Your Scores].

Always FINALIZE your results when you are finished rating all seven rubrics.

Achieve OER Evaluation Tool:Finalize Your Review

Slides developed by Achieve:www.achieve.org1400 16th Street, NW / Suite 510 Washington, DC 20036

Evaluating Open Educational Resource (OER) Objects

Documents

oer definition

oer evaluation tool

filtering oer

oer page

oer evaluation rubricshow

oer rubrics work

oer commons cc

object note