WHITE PAPER Critical Evaluation of Quality Criteria and Quality Instruments in OER Repositories for the Encouragement of Effective Teacher Engagement DR MARINA CONNELL JOHN CONNELL JANUARY 2020
WHITE PAPER
CriticalEvaluationofQualityCriteriaandQualityInstrumentsinOERRepositoriesforthe
EncouragementofEffectiveTeacherEngagement
DRMARINACONNELLJOHNCONNELL
JANUARY2020
1
Table of Contents
ABSTRACT .............................................................................................................. 2KEYWORDS...........................................................................................................................................................2ACKNOWLEDGEMENTS .......................................................................................... 2
INTRODUCTION ...................................................................................................... 3DEFINITIONOFTERMS...................................................................................................................................3PURPOSEOFTHEWHITEPAPER...............................................................................................................4BACKGROUNDANDTHEORETICALAPPROACHOFTHISREVIEW.............................................5METHODOLOGY..................................................................................................................................................6INTENDEDBENEFITSOFTHISREVIEW.................................................................................................7OVERVIEW OF LEARNING RESOURCE QUALITY CRITERIA AND FEEDBACK SYSTEMS 9TAXONOMIESOFQUALITYINRELATIONTOUSINGOER..............................................................9THEIMPORTANCEOFQUALITYTOTHEUPTAKEOFOER.........................................................14EVALUATINGQUALITYINOER................................................................................................................15RANKINGSYSTEMS&COLLECTIVEINTELLIGENCE......................................................................17EXAMPLESOFREPOSITORIESWITHEFFECTIVEQUALITYSYSTEMS...................................18KlasCement.........................................................................................................................................................18Curriki...................................................................................................................................................................20MERLOT...............................................................................................................................................................21FurtherDiscussion..........................................................................................................................................23
THEQUALITYCHALLENGES:ThreeShortCaseStudies................................................................25CaseStudy1:Scientix.....................................................................................................................................25CaseStudy2:GeoGebra.................................................................................................................................27CaseStudy3:Photodentro...........................................................................................................................30
DIFFERENTAPPROACHESTOQUESTIONSOFQUALITYINOER.............................................31CONCLUSION: A CRITICAL EVALUATION OF QUALITY CRITERIA ............................. 32
APPENDIX: ACTIONABLE POINTS & RECOMMENDATIONS .................................... 36
BIBLIOGRAPHY ..................................................................................................... 38
2
ABSTRACT ThispaperoffersashortevaluationofthevarietyofqualitycriteriausedinOpenEducationalResourcesandsomeofthemethodsandpracticesinusetoensurequality.Thepapersurveysandreviewseffectivepracticesandtechnologiesusedinlearningobjectrepositoriestomoreeffectivelyengageeducatorsinprovidingstructuredfeedback.Theaimistoprovideausefulresourcetorepositoryownersandmanagersseekingtodevelopandimproveontheservicestheyalreadyoffertoeducators.AnimportantadditionalaimistosupportteachersinusingdigitallearningresourcesintheclassroomsothatthedeploymentofOERinschoolsinparticularmightgrowandflourishoverthenextfewyears.‘Quality’isbynomeansasimpleconcepttodefineinthecontextofOERrepositoriesandthepaperseekstolayoutanumberofdifferentwaysinwhichtheconceptcanbeanalysedandmadeuseful.Thepaperwasproducedthroughashort-termqualitativeanalysisinwhichweexaminedanumberofexistingOERrepositories,awiderangeofacademicandotherwritingsonthesubject,asmallnumberofinterviewswithkeyexpertsandsomepublishedslidewarerelevanttothesubject.Thepaperoffersasetofpotentialfurtherresearchquestionsaswellasadistillationofanumberof‘actionablepoints’thatOERleadersmightbeabletouseasstartingpointsforongoingdevelopmentworkontheirownqualityassurancesystems.
KEYWORDS Quality, Quality Criteria, Quality Circle, Collective Intelligence, Evaluation, OER, Repository
ACKNOWLEDGEMENTS Anumberofpeoplegaveustheirkindandgenerousassistancewithourtaskinresearchingandwritingthiswhitepaper.WearethankfultoallofthemfortheirhelpinproducingwhatwehopewillproveausefulpapertotheOERcommunityinEuropeandbeyond:DavidMassart,ElenaShulman,HansDeFour,BramFaems,ElinaMegalou,JoeWilson,PaulBacsich,MarthaZellinger,NeilButcher,AndrewMoore,FernandoCampos,NellyBuchser Anydeficienciesinthefinalpaperremaintheresponsibilityoftheauthorsalone.
3
INTRODUCTION TheOpenEducationmovementhasgainedsubstantialtractionsincethetermOpenEducationalResources(OER)wascoinedin2000.However,thereremainsmuchscopeforfurtheradvocacyandpromotionofOpenEducationgenerallyandoftheprinciplesandvaluesthattheconceptembodies.OpenEducationisabroadcanvasthatisabletoaccommodatearangeofunderstandingsoftheterm.Itisalsoatermthatgathersanarrayofdifferentelementsbeneathitsumbrella,ofwhichOERisone,althoughonethatismuchdiscussed.OERsaregenerallystoredinaLearningObjectRepository(LOR).
DEFINITION OF TERMS
Intermsoftheunderstandingof‘open’inOER,thispapertakesasastartingpointthestatementfromDavidWileythat:
“Openisacontinuous,notbinary,construct.”1
Inotherwords,therecanbenofixeddefinitionof‘open’inthecontextofOER.Insteaditshouldbeinterpretedinrelativeterms–isarepository,forexample‘openenough’tobedefinedasofferingOER,orisonerepository‘moreopen’or‘lessopen’thananother?Itshouldbeacknowledged,however,thatmanyofthosewhoworkinthefieldprefertoacceptmorefixedandprescriptivedefinitionsofopenness.‘Open’,inthesenseusedinthisreport,relatestotheextenttowhichusersaregrantedrightstousearepositoryoraresourceinrelationtothe5Rsofopeneducationalresources,namely:
Retain-therighttomake,own,andcontrolcopiesofthecontent(e.g.,download,duplicate,store,andmanage)
Reuse-therighttousethecontentinawiderangeofways(e.g.,inaclass,inastudygroup,onawebsite,inavideo)
Revise-therighttoadapt,adjust,modify,oralterthecontentitself(e.g.,translatethecontentintoanotherlanguage)
Remix-therighttocombinetheoriginalorrevisedcontentwithothermaterialtocreatesomethingnew(e.g.,incorporatethecontentintoamashup)
Redistribute-therighttosharecopiesoftheoriginalcontent,yourrevisions,oryourremixeswithothers(e.g.,giveacopyofthecontenttoafriend)2
1https://opencontent.org/blog/archives/11232http://opencontent.org/definition/
4
Themorethattherightsattachedtoaresourceorarepositorymeetthe5Rs,themoreitcanbejudgedtobe‘open’.The5RsalsoofferareasonablefoundationforanexplorationofqualityinrelationtoOERandopenrepositoriesgenerally.Theoft-accepteddefinitionofOERisthatdevelopedbytheTheWilliamandFloraHewlettFoundation,derivedfromanumberofprecedingdefinitions:
“OpenEducationalResourcesareteaching,learningandresearchmaterialsinanymedium–digitalorotherwise–thatresideinthepublicdomainorhavebeenreleasedunderanopenlicensethatpermitsno-costaccess,use,adaptationandredistributionbyotherswithnoorlimitedrestrictions".3
ThisdefinitionstatesthatOERcanincludebothdigitalandnon-digitalresources.Ashorter,sharperdefinitioncomesfromOECD,butonethatfocusesondigitalresourcesinparticular:
“Digitisedmaterialsofferedfreelyandopenlyforeducators,studentsandself-learnerstouseandre-useforteaching,learningandresearch.”4
Thispaperwilladdressitselfmainlytodigitalresources,althoughtheissuesdealtwithpertainbroadlytonon-digitalresourcesaswell.PURPOSE OF THE WHITE PAPER Intheirpaper,OpenEducationalResourcesPlatformBasedonCollectiveIntelligence,theauthors,DeOliveiraetal5,setoutinsuccincttermstheissuethatliesattheheartofthiswhitepaper.
“AnOERischaracterizedbyitsmetadata….TheOERsarestoredinrepositories.ThepurposeofOERrepositoriesistosupporteducatorsinsearchingforcontentinastructuredway,sharingtheirownresources,reusingexistingmaterialsandcreatingnewresourcesthroughadaptingortranslating,andincollaboratingwithothermembersoftheusercommunitybycommentingupon,reviewing,promotinganddevelopingresources.”
EducatorssearchingforOERbyandlargemakeuseofrepositoriestosearchfor,locate,download,use,re-useandadaptthelearningresourcestheyneedfortheirteaching.Itwouldseemtomakesensethateffectiverepositorieswillmakethisprocessassimpleandaseasy-to-useaspossible.Theyshouldalsoseektogiveteachersmechanismsthatwillallowthemtoprovideprofessionalfeedbackonthevalueandusefulnessoftheresourcesoncetheyhavedeployedthemintheirteaching.Bythismeans,andothers,therepositorywillthenbeabletoofferteacherssomeinformationbywhichtheyareabletopre-determinetosomeextentthequalityoftheresourcestheyselect.
3https://hewlett.org/strategy/open-educational-resources/TheWilliamandFloraHewlettFoundation. Retrieved27March20134http://www.oecd.org/education/ceri/38654317.pdfIschinger,BOECDGivingKnowledgeforFree,p.105https://ieeexplore.ieee.org/document/8537851deOliveira,etalOpenEducationalResourcesPlatformBasedonCollectiveIntelligencep.1
5
Thefocusofthiswhitepaper,therefore,istoundertakeashortevaluationofqualityissuesinrelationtoOER,withafocusontheK12sectorwherethatispossible,andtoexamineanytechniquesthatcanbeidentified,eitherinproposalsbyeducatorsandacademicsorinactualuseinOERrepositories,thatmightsupporteducatorsinintegratingdigitallearningresourcesintotheirteachingpractice.MuchoftheliteratureandresearchintoOpenEducationalPractices(OEP)generally,andintoOERinparticular,hasbeenproducedandcarriedoutfromahighereducation(HE)ortechnical&vocationaleducation&training(TVET)perspective,and,todate,withafewhonourableexceptions,lesssofromtheperspectiveofteachersandstudentsintheschoolssector.Forwhateverreasons,thereseemstohavebeenanidentifiablylowerlevelofengagementwithOpenEducationfromwithintheK12sectorgenerallyovermanyyears.Ofcourse,manyoftheOpenEducationlessonslearntandappliedwithinHEandTVETwillapplyjustasmuchtoOEPinschools,butweshouldbecautiouslyawarethattheirapplicationinK12mightraisesomecontextuallyspecificissuestoo.
BACKGROUND AND THEORETICAL APPROACH OF THIS REVIEW ThiswhitepaperhasbeencommissionedbytheLearningResourceExchange(LRE)SubcommitteeofEuropeanSchoolnet.TheLRESubcommitteeisagroupofEuropeanMinistriesofEducationandotherorganisationsthatpromotesandsupportsthecreation,discovery,exchange,and(re-)useofOERandOEP.TheLRESubcommitteefeelsthatthetimeisrighttosurveyandreviewbestpracticesandtechnologiesused,orproposedforuse,bylearningrepositoriestomoreeffectivelyengageeducatorsinthetermsalreadydescribedabove.Thepaperprovidesareviewandasmallnumberofrecommendationsonqualitycriteria(eitheralreadyinuseorbasedoncurrentpedagogicalresearchliterature)anduser-feedbacksystemsthataremosteffectiveinclosingthegapbetweenOERs,inproposalsorinrepositories,andtheiractualuseinclassrooms.Thepaperattemptstodescribeestablishedgoodpracticesforeducationalrepositorieswithregardtoratingandevaluativesystemsandfeedbackmechanismsbasedonprofessionaljudgementoronwell-foundedqualitycriteria,withafocusonsystemsthatcaptureuserfeedbackwithinOERrepositoriesthemselves.Toachievetheseaims,thepaperwasrequiredtoencompassfivekeydeliverables:
• Abriefoverviewoflearningresourcequalitycriteriaandrankingsystems• Keyexamplesofimplementationsoftheserankingsystemsthatcapture
userfeedbackonOERs• ChallengesfacedbyK12educationalrepositoriesthathaveimplemented
rankingsystemsbasedonuserfeedbackandthesolutionsadoptedbyrepositoriesthataddresschallenges;
6
• Anevaluationoftheeffectivenessofdifferenttypesofqualitycriteriaforstructuringfeedbackandrankingsystemsthatsupportteachersinusingdigitallearningresourcesintheclassroom;
• Acomprehensivebibliographyreferencingrelevantreadingsandstudiesthatcanbeusedinselecting,developingorimplementingqualitycriteriaandrankingsystemsthateffectivelyexploituserfeedbackforevaluatingandusingOERs.
Someofthedeliverablesweremoresuccessfullyachievedthanothers.Inparticular,arathersmallnumberofOERrepositorieswereidentifiedthatseemtohaveimplementedeffectivesystemsforevaluatingqualityorforrankingresourcesbyquality,andfewalsothatsystematicallyinvolvedusersinfeedingbackonthequalityoftheresourcestheyfindordeploy.However,withinthatsmallsample,thereweresomeexamplesofgoodpractice.
METHODOLOGY Giventheshorttimescaleavailabletoproducethispaper,theinformationanddatawereexaminedandprocessedonthebasisofthetype,goalsanddeliverablessetoutabove.Inconductingthereview,weusedthefollowingtechniques:
a) Contentanalysisb) Small-scalesurveyingofkeyrepositoriesandindividualsc) Deskresearchd) Interviewsbyemailandbyaudio
Theresearchisexclusivelyqualitativeinnature,usinganexploratoryapproachtoanexistingrangeofacademicandotherworkinthefield.Afewkeyinsightsweregainedfromextensivereadingofthematerialfoundandfromasubsequentattempttosynthesizeandsummarizethethinkingaroundquestionsofqualityinrelationtolearningresources.Anumberofworkingrepositorieswereidentifiedandasmanyofthemascouldbeidentifiedwithinthetimeavailablewereexamined.Theexaminationwasalsocarriedoutwithintheconstraintssometimesimposedbytherepositoriesthemselveswithregardtoauthenticationandease-of-access.Itwashopedtogainafeelforthoserepositories,ifany,thathavesoughttoestablisheffectivepracticeintheuseofqualitycriteriaandrankingsystems.Averysmallnumberofkeyexpertswereidentified(fromourownknowledgeandfromourreadingoftheliterature)whoitwasfeltcouldofferinsightsbasedontheirgreaterexperienceofthefield,and‘interviews’wereconductedbyemailorbyphone.Witheachpartofourapproach,theaimwastofocusinparticularonthesituationinrelationtoK12,thisdespiteanapparentdearthofmaterialandpracticerelatingtotheschoolssectorinOpenEducation.SomegoodpracticeintheK12sectorwasidentifiedandthishasbeenhighlightedinthepaper.
7
However,itwasnecessarytodependtosomeextentonthinking,writingandpracticeemanatingfromHEandTVETtoo.Theinformationgatheredforthereportrangesacrossabroadspectrumofsources,includingsomewell-known(andsomeless-well-known)academicpapers,educationalblogsandwebsites,socialmediagroups(includingFacebook,Youtubeandothers),whitepapersandavarietyofreports.Throughouttheprocess,theinvestigationsoughttoaddresstherequirementsofteachersinrelationtotheiractualneedsintheclassroomandtoavoidwherepossibleanyemphasisontop-down,centralized,pre-determinedassertionsofqualityforresources.Tothisextent,itishopedthatthelongertermneedsofrealeducators,especiallyintheschoolssector,willbemetbythoserepositoriesthatendeavourtogiveteachersthemselvesinfluenceoverthedeterminationofcontentquality.Theaimistoprovidesomesemblanceofawayforwardforthosewhoseektomakefurtherprogressinthisarea.Thereportalsofocusesontheassumptionthat,nomatterhowmuchqualitycanbeascribedtoanysingleresourceortoanycollectionofresources,whetheropenorotherwise,ifthoseresourcesdonothaveapositiveimpactonstudents’learning,theyareultimatelyofnoeducationalvalue.Thisis,however,anassumptionthatisitselfsomewhatproblematicwhenweconsiderOERinrelationtotheneedsofteachersandtheirteachingpracticeaswellasinrelationtothelearningthatfollowsfromthatteaching.Havingperusedthesubsetoftheidentifiedliterature,itisdifficultnotagree,ingeneral,withtheviewofPaulBacsich,BarryPhillipsandSaraFrankBristowintheirFinalReportonLearnerUseofOnlineEducationalResourcesforLearning,that:
“Theliteratureonlearneruseofonlineeducationalresourcesisveryimmature,withalackofmeta-reviews.Theoverwhelmingmajorityofpublishedstudiesdonotgeneralisebeyondtheirparticularcontextsofstudy.Thereisnoconsistentmethodology.”6
Theirfindingextendsalongwayintotheliteratureonqualitycriteriaandfeedbacksystemsinlearningresources,whethertheyareopenorotherwise,andthatisperhapsevenmorethecasewhenthefocusisonthesequestionsisinrelationtoK12.Ofcourse,thosewhohavealreadycontributedovertheyearstotheliteratureonOERhavedonesoextensivelyandinsightfully.However,thefieldisstillyoungandneedstogrowandmatureoverthenextfewyears,buildingontheverygoodworkalreadydone.INTENDED BENEFITS OF THIS REVIEW Inashortreportsuchasthis,itisdifficulttocaptureanythingmorethananimpressionofthestateofplaywithregardtothequestionsathand.However,it
6https://www.advance-he.ac.uk/knowledge-hub/learner-use-online-educational-resources-learning-luoerl-final-reportBacsich,P;Phillips,B;Bristow,SFLearnerUseofOnlineEducationalResourcesforLearningP.3
8
ishopedthatthethoughtsgatheredherewillbeabletoactasaspurtodeeperandbroaderworkinfuture.AdditionalresearchwillundoubtedlyberequiredtopromoteandexpandontheeffortsthathavealreadybeenmadetoensurethateducatorsarebetterabletoselectandintegratedigitalresourcesintotheirteachingpracticeacrossEuropeandbeyond.Ifthepaperleadstofurtherresearchintoquestionsraisedhere,thatwillberegardedassuccess.
9
OVERVIEW OF LEARNING RESOURCE QUALITY CRITERIA AND FEEDBACK SYSTEMS
TAXONOMIES OF QUALITY IN RELATION TO USING OER
Anyattempttodefinequalityinrelationtoeducationalresources,whetheropenornot,isfraughtwithdifficulty.Qualityisbothsubjectiveandheavilycontext-specific,andattemptingtosetouttheparametersofqualityinrelationtoOERoftenleadstosubjectivelayerbuiltuponsubjectivelayer,witheachlevelofdefinitionprovingdifficultorimpossibletomeasureinanysimpleway.Thereisalsothesimpleassertionthattherecan,ultimately,benoinherentorintrinsicsenseof‘quality’withinaresourceotherthaninaparticularcontextwhereithastoservespecificneedsforeducators,studentsorself-learners.Itmustimpactonthelearninginapositiveway.A‘good’resourceinonecontextmightbea‘poor’resourceinanother–itdependsupontheusetowhichitisputandforwhosebenefit.
AsDavidWileysosuccinctlydescribestheissue:
“…thecoreissueindeterminingthequalityofanyeducationalresourceisthedegreetowhichitsupportslearning.Butconfusingly,that’snotwhatpeoplemeanwhentheysaythatatextbookorothereducationalresourceis“highquality.”….thedegreetowhichitsupportslearningistheonlycharacteristicofaneducationalresourcethatmatters.”7
However,Wiley’scontention,thoughafundamentaltruth,isnotenough.Itistoosimplisticandittakeslittleaccountoftherealitiesofateacher’sneeds.Takingaccountofthebroaderresponsibilitiesrequiredinteachingandlearning,itisimportanttokeepinmindthatteachers,beforeteachingorlearningeventakesplace,havetofindlearningresourcesthatareappropriatetotheirteachingrequirements,thatarereadilysearchableandavailable,andthat,wherepossible,alreadyhavesomeperceivedleveloftrustworthinessbecausetheresourcehaspreviouslybeencuratedorhasprovedusefultoothereducators.SowhileWiley’sviewiscentraltoanyconsiderationofquality,sotooaretheneedsofteachersastheyseekout‘goodquality’learningresourcesinplanningtheirteachingandtheirstudents’learning.Ofcourse,teacherswanttheirlessonsandtheresourcestheydeploytohaveapositiveimpactonstudents’learning,buttheycannotknowwhetherthatwillbethecaseuntilaftertheteachinghasbeencarriedoutandthelearninghasbeenassessed.Beforeallthatcanhappen,teachersneedsomeassuranceabouttheperceivedqualityofaresourceinordertohelpthemselectthebestcontentpossiblefortheirteaching.SoinlookingatqualitywithrespecttoOER,weshouldthinkof‘quality-before-the-fact’aswellas‘quality-after-the-fact’.Neitherissufficient,andbotharenecessaryincontemplatingthedomainofOEP.If,outoftheirteachingpractice,teachersareabletoevaluatethequalityoftheresourcestheyusedinthe
7https://opencontent.org/blog/archives/3821Wiley,DStopSaying“HighQuality”
10
classroom,that,inasense,completesacircleofquality:eachtimethecircleiscompleted,theperceivedqualityofaresourcecanberenderedmoreandmoretrusted.So,afteraresourcehasbeendeployedtheteacherisabletogivefeedbackonwhetherandhowtheresourcewasusefulornot.Aneffectiverepositoryshouldallowtheeducatortocompletethis‘qualitycircle’,fromdiscovery,touse,toprofessionalevaluativefeedback,andtheprocesscanthenberepeatedasmanytimesasnecessary.
A Simple Outline of the Notion of the ‘Quality Circle’ in Relation to OER
Thiscanleadtoarankingoftheresourcesavailablefromarepositoryorothersource,sothatwhenanothereducatorcomesalonglookingforsimilarresources,therepositoryisabletoserveupanorderedlistthatreflectspreviousevaluations.Evenbetterisafunctionthatenablessortingbymorethanonemetric,asisthecaseinsomeoftheexamplesexaminedinthispaper.Thequalityofalearningresourcedependsonthedemandsandrequirementsbothoftheeducatorandofthelearnerratherthanonsomepre-determinedcharacteristicoftheresource:
“Inthissense,qualityassuranceandvalidationcannotcomefromatop-downapproachthatwillperformacentralizedqualitycontrol.Onthecontrary,suchqualitycontrolshouldtakeintoaccounttheend-users,engagethemproactivelyinprovidingfeedbackandsuggestionsforenhancements,supporttheminadaptingthecontentthemselvesandsharingitwiththecommunity.”8
Takingthisapproachfurther,oneofthemostusefuldelineationsofqualityinthiscontexthasbeenprovidedbyCamilleri,EhlersandPawlowskiintheir“State
8http://lre.eun.org/wp/LRE_White_paper_001.pdfChounta,Irene-AngelicaAreviewofthestate-of-artoftheuseofmachine-learningandartificialintelligencebyeducationalportalsandOERrepositoriesp.22
11
oftheArtReviewofQualityIssuesRelatedtoOER”9.Intheirpapertheysetoutthefollowingfeatures,someofwhichareuniversalorgeneralwhileothersaremorespecifictoaparticularinstanceofusebywhichitmightbepossibletocometoanunderstandingofthequalityofalearningresource:
Efficacy–bythisismeantthefitnessforpurposeoftheobject/conceptbeingassessed.WithinthecontextofOER,thismightincludeconceptssuchasease-of-reuseoreducationalvalue.Impact–impactisameasureoftheextenttowhichanobjectorconceptproveseffective.Impactisdependentonthenatureoftheobject/conceptitself,thecontextinwhichitisappliedandtheusetowhichitisputbytheuser.Availability–theconceptofavailabilityisapre-conditionforefficacyandimpacttobeachieved,andthusalsoformspartoftheelementofquality.Inthissense,availabilityincludesconceptssuchastransparencyandease-of-access.Accuracy–accuracyisameasureof(a)precisionand(b)absenceoferrors,ofaparticularprocessesorobject.Excellence–excellencecomparesthequalityofanobjectorconceptto(a)itsquality-potential,i.e.themaximumtheoreticalqualitypotentialitcanreach.
Thefirstofthesefactors–efficacy,orfitnessforpurpose–mightbeviewedasthesinglefactormostrelevanttoteachersseekingeffectiveresourcesfortheirteaching.Whetherthatisthecaseornot,allfivefactorstakentogetherembraceboththequestionofimpactonlearningandtheneedfora‘pre-definition’ofqualitythatworksfortheteacherhopingtocreateinterestinglearningexperiencesforstudents.Intakingforwardthesecharacterizationsofquality,theyproposeaconceptualframeworkbuiltaroundthreesetsofqualityapproaches:
QualityAssuranceofResources–…alifecyclemodeltounderstandthequalityfactorsaffectingindividualresources,includingtheircreation,useandevaluationQualityAssuranceofStrategies/Policies–usingamaturitymodel…theinstitutionaldevelopmentofpolicieswhichgovernandpromotethecreationofOERQualityAssuranceofLearning–…course-specificqualityassurance,includingprocessesofteaching,assessmentandrecognition10
Bytakingthisapproach,theygobeyonddefiningqualityintermssimplyofsomeintrinsiccharacteristicsoftheresources.Forexample,theyoutlinethevaluereinforcedintheresourcesbythefeaturesbuiltintotheprocessofcreatingtheresourcesinthefirstplace.Evenhere,though,theultimatequalityoftheresource,nomatterthecalibreofitsprovenance,canonlybemeasuredagainst
9https://pdfs.semanticscholar.org/475a/50bbc6bb0b0e580aa1470ff88d972b60983d.pdfCamilleri,AnthonyF.;Ehlers,UlfDaniel;Pawlowski,JanStateoftheArtReviewofQualityIssuesrelatedtoOpenEducationalResources10Camilleri,AnthonyF.etalibid.
12
itseffectivenessinfacilitatinglearning.Theynotetoo,inpassing,thatapplyingtheprocessofresource-creationtoOERisnecessarilymorecomplexthanthesameprocessappliedtotraditionally-producedresources,giventhepotentialmultiplicityoffacetsofauthorship,sharing,modification,useandre-useinOER.Intheirpaper“QualityAssuranceintheOpen”AtenasandHavemann11consideredthewiderapproachoflookingatthesharedcharacteristicsofOERrepositoriesratherthanthequalityoftheresourcesthemselves.Theytookasastartingpointtheassumptionthat:
“ThepurposeofOERrepositoriesistosupporteducatorsinsearchingforcontentinastructuredway,sharingtheirownresources,reusingexistingmaterialsandcreatingnewresourcesthroughadaptingortranslating…”12
Onthisbasistheysetoutwhattheyidentifyasthedefiningcharacteristicsofthemosteffectiverepositories,characteristicsthatareeithersocialortechnical,orboth.Whilethetablebelowwouldbedifficulttotranslateintoguidelinesforteachersseekingtoevaluateresources,itdoesserveapotentiallyusefulpurposeforthosewhoaretryingtodevelopeffectiveandfully-functioningrepositories13.
QualityAssurance
Description Themes Characteristics
Featuredresources
Abilityoffeaturingresourcesthatarepotentiallyofhighinterestforteachersbecauseoftheirdesignorcontent.
Search,Share,Collaborate
Social
Userevaluationtools
Toolsfortheresourcestobeevaluatedbyusersaimingtoratearesource.
Collaborate Social
Peerreview Peerreviewaspolicytoreviseandanalyseeachresourcetoensureitsquality.
Collaborate Social
Authorship Analyseiftherepositoriesincludethenameoftheauthor(s)oftheresources.
Search,Reuse Social
Keywords Methodicallydescribetheresourcestofacilitatetheretrievalofthematerialswithincertainspecificsubjectareas
Search Technical
Metadata Introducestandardisedformatsofmetadata(DublinCore-IEEELOM-OAI-PMH14)forinteroperability
Search,Share,Reuse Technical
11https://eprints.bbk.ac.uk/8609/1/30-288-1-PB.pdfAtenas,JavieraandHavemann,Leo(2013).Qualityassuranceintheopen:anevaluationofOERrepositories.INNOQUAL:TheInternationalJournalforInnovationandQualityinLearning12Atenas&Havemann,ibid,p.2413WearegratefultoDavidMassartforofferingclarificationonthisdistinction14OAI-PMHisaprotocolforexchangingmetadata,notametadataformat.See:https://en.wikipedia.org/wiki/Open_Archives_Initiative_Protocol_for_Metadata_Harvesting
13
Multilingualsupport
Designtheinterfaceoftherepositoryinamultilingualwaytowidenthescopeofusersbyallowingthemtoperformsearchofcontentindifferentlanguages.
Search,Share,Reuse,Collaborate
Technical
SocialMediasupport
Introducesocialmediatoolstoenabletheuserstosharetheresourceswithinsocialmediaplatforms.
Search,Share,Reuse,Collaborate
Social,Technical
CreativeCommonsLicences
SpecifythetypeofCreativeCommonsLicenceperresourceorgiveinformationaboutthespecifictypeoflicenceforalltheresources.
Search,reuse,collaborate
Technical
SourceCodeorOriginalFiles
Allowdownloadingtheoriginalfilesorsourcecodeofresourcessotheycanbeadapted.
Reuse,Collaborate Technical
Theimportanceofensuringqualityindigitallearningresources(indeedanylearningresources,nomattertheirformat)cannotbeunderestimated.Camilleri,EhlersandPawlowski,alreadycited,madetheimportantpointthat,althoughopenlearningresourcesaregenerallyavailable,theyare‘notfrequentlyused’.Inthereasonsforwhythismightbeso,theylistthefollowing:
“…[lackof]organisationalsupport,alackofsharingculturewithinorganisations,lackofskills,quality,trustortimeandskillsforadaption.Onlyoneelementisrelatedtotheavailabilityoftechnicaltoolsforsharingandadaptingresources.Notasinglebarrierrelatestothequestionofaccessibilityandavailability.”15
So,lackofquality(including,wepresume,perceivedlackofqualityasaconsequenceofapaucityofpriorevaluations),orevenasimplelackofinformationaboutquality,isadeterminingfactorsintherelativelackofactualuseofdigitallearningresources.Inthecreationofdigitallearningresources,theEuropeanConsortiumforAccreditationsuggests16anumberofqualitycriteriawhichshouldbetakenintoaccountwhendesigningsuchlearningobjectives:
• Internalandexternalstakeholdersshouldbeinvolvedintheprocessofdesigningandrevisinglearningobjectives,forexamplebyparticipatinginmeetings,pedagogicalboards,satisfactionsurveys,evaluationprocedures,etc.
• Learningobjectivesshouldbedescribedinacomprehensibleway.Inparticular,theyshould:
o Bedefinedinclearandconcreteterms(shortandsimplesentences),
15Camilleri,AnthonyF;etal,ibidp.2516Camilleri,AnthonyF.;etalibid.p.17QuotingfromEuropeanConsortiumforAccreditation-http://ecahe.eu/w/images/b/ba/Publication-Learning_Outcomes_in_Quality_Assurance_and_Accreditation.pdf
14
o Focusonwhatstudentsareexpectedtobeabletodemonstrateanddescribeobservableabilitieswhichcanbeassessed.
However,giventhattheperceptionofqualityisjustoneofseveralbarrierstouptakeandusageofOER,anystrategiesundertakentoimprovequalitymustbeseenasjustonefacetofpoliciesdesignedtoincreasetheuseofOERinanyeducationalsector.Questionsoftrust(itselfoneelementofquality,ofcourse),time,skillsandculturemustbedealtwithalongsidequestionsofquality.
THE IMPORTANCE OF QUALITY TO THE UPTAKE OF OER SomestudieshavelookedatwhatteachersneedinordertoencouragethemtouseOERrepositories.Inherpaper,ClassificationandQualityCriteriaforOpenEducationalResourcesintheFieldofForeignLanguageLearning,ZitaKrajcsomadetheself-evidentbutwholly-necessarypointthat:
“…teachersneedhighqualityresources.”17
However,shewentontosay:“Furthermore,thepoolofresourcesmustbebigenoughtooffersomethingformany.Thesearchingmechanismfortheresourcesisalsocrucial;teachersneedefficientwaysforsearchingandfindingmaterials(reachedwithinthreeclicks)fromdifferentcurriculabackgrounds(classificationbysubject,topicsandlevel).”18
Andshecontinues,echoingtheconceptofthe‘qualitycircle’alreadymentioned:
“Classificationassuchiscrucialforthesearchmechanism,butqualityfactorsdeterminethelong-termsuccessfulnessofdigitalresourcesontheusers’side.InthecaseofOERthesefactorsrelatetoinput(qualityofthestructureandpotential),implementation(process)andoutput(outcome)aspects).”19
Shemakestheimportantpointthatensuringqualityintheprocessofcreatinglearningresourcesdoesnotnecessarilyequatetohighimpactontheeventuallearningintendedbyitsuse:
“…[a]focusontheinputqualitycriteriaofOER….hasnotbeenprovenascausalforthelearningeffectyet.Moreresearchisneededtoconfirmandextendthefollowingqualitycriteria.”20
Weispossible,perhaps,toassertwithsomeconfidencethatthisisacausalrelationshipthatisunlikelyevertobeprovenasauniversalrule.
17https://content.sciendo.com/configurable/contentpage/journals$002fjolace$002f4$002f1$002farticle-p48.xmlKrajcso,RClassificationandQualityCriteriaforOpenEducationalResourcesintheFieldofForeignLanguageLearningp.4918Krajcso,Ribid.p.4919Krajcso,Ribid.p.5020Krajcso,Ribidp.50
15
EVALUATING QUALITY IN OER Ingeneral,itwasdifficulttoidentifyasignificantbodyofworkrelatingtospecifictoolsortechniquesthatcanbeused,andmoreimportantlyarealreadybeingused,byeducatorsandotherstoevaluatethequalityoftheresourcestheyuse.Asmallnumberofrepositorieswereidentifiedthathavesuccessfullyimplementedeffectivefeedbackqualitymechanismsandthesewillbeoutlinedonpages18to24below.IntermsofthequestionofqualitybothinthecreationandchoiceofOER(asopposedtothequalityoftheresourceinrelationtoitsimpactonlearning)anumberofattemptswereidentifiedthatsetoutsomeguidelinesthatteacherscoulddeploytoassisttheminusingOERwell.ArelativelyclearandstraightforwardrubricwascreatedbytheUniversitySystemofGeorgia(theUSGiscomposedof26highereducationinstitutionsincludingfourresearchuniversities,fourcomprehensiveuniversities,ninestateuniversitiesandninestatecolleges,aswellastheGeorgiaPublicLibraryService,allintheUSStateofGeorgia).AcomponentofUSGcalledAffordableLearningGeorgia21setoutthefollowingguidelinesforteachers:AFFORDABLELEARNINGGEORGIAOEREvaluationCriteriaClarity,Comprehensibility,andReadability
• Isthecontent,includinganyinstructions,exercises,orsupplementalmaterial,clearandcomprehensibletostudents?
• Isthecontentwell-categorizedintermsoflogic,sequencing,andflow?• Isthecontentconsistentwithitslanguageandkeyterms?
ContentAccuracyandTechnicalAccuracy
• Isthecontentaccuratebasedonbothyourexpertknowledgeandthroughexternalsources?
• Arethereanyfactual,grammatical,ortypographicalerrors?• Istheinterfaceeasytonavigate?Aretherebrokenlinksorobsolete
formats?AdaptabilityandModularity
• Istheresourceinafileformatwhichallowsforadaptations,modifications,rearrangements,andupdates?
• Istheresourceeasilydividedintomodules,orsections,whichcanthenbeusedorrearrangedoutoftheiroriginalorder?
• Isthecontentlicensedinawaywhichallowsforadaptationsandmodifications?
21https://www.affordablelearninggeorgia.org
16
Appropriateness• Isthecontentpresentedatareadinglevelappropriateforhigher
educationstudents?• Howisthecontentusefulforinstructorsorstudents?• Isthecontentitselfappropriateforhighereducation?
Accessibility• Isthecontentaccessibletostudentswithdisabilitiesthroughthe
compatibilityofthird-partyreadingapplications?• IfyouareusingWebresources,doeseachimagehavealternatetext
thatcanberead?• Dovideoshaveaccurateclosed-captioning?• Arestudentsabletoaccessthematerialsinaquick,non-restrictive
manner?
SupplementaryResources• Haveyoureviewedthesesupplementaryresourcesinthesamemanner
astheoriginalOER?Whilethetenorofthisparticularrubricisfairlyspecifictoitsowninstitutionalsetting(forexample,atleastsomeofitsfocusseemstobeonmakingavailablecompletecoursesaswellassingle-useresources-teachersaremorelikelytorequirethelatterthantheformer),itdoesseemtooffersimpleguidelinesthatcouldbeadaptedformanyothercontextswhereteachersareeithercreatingorchoosingOER,orboth.Itshouldbenotedinpassingthat,accordingtoarecentpiecefromLibbyVMorris,USG’sAffordableLearningGeorgiahassavedstudentsmorethan$31montextbookcosts.22Morepertinenttothispaper,Morrisgoesontoraisethequestionofquality,amongstanumberofotherissues,thatshebelievesoughttoinformtheresearchagendaonOEPandOERforthenextfewyearsatleast:
“Thequantityofopenmaterialsislargeandgrowing,andthewidespreadenthusiasmisevident.Nextupistobuildamorerobustresearchagendatobetterunderstandquality,creationandadoptionprocesses,andoutcomes.Whataretheeffectsofopeneducationalresourcesoncoursedevelopmentandimplementation?Onstudentlearning?Whatarethecharacteristicsofcreatorsandearlieradopters?Howdofacultymembersidentify,select,andevaluateopenresourcesforcollege-levelcourses?Howdoinstructor-designedresourcesdifferfromadoptedandadaptedresourcesincoursedesignandeffectonlearning?Willstudents’priorexperiencewithOERaffectmotivationtoindependentlylocateanduseotherfreematerialsforlearning?Howwillthisinfluenceachievementandsuccess?”23
22https://link.springer.com/content/pdf/10.1007/s10755-019-09477-7.pdfMorris,LibbyVContemplatingOpenEducationalResourcesp.2publishedinInnovativeHigherEducation(2019)44:329–33123https://link.springer.com/content/pdf/10.1007/s10755-019-09477-7.pdfMorrisibid.p.3
17
ManyofthesequestionsplaydirectlytoanyconsiderationofqualityinrelationtoOER.RANKING SYSTEMS & COLLECTIVE INTELLIGENCE DeOliveiraetalintheirpaper,alreadycited,OpenEducationalResourcesPlatformBasedonCollectiveIntelligence,describehowcumbersomethetaskcanbeforteacherstosearchfor,locate,downloadanduseOERfromrepositories:
“Theircontentcanbeofmixedquality,withseveralresourceslackingimportantdescriptivemetadatawhich,alliedtotheoftensub-optimalindexingfromtheirsearchengines,resultsintheuserhavingtosortthroughmanyirrelevantcontentwhenqueryingforspecificeducationalresources.Sinceitisofteneasierforuserstosearchforcontentontheirpreferredsearchengine,theserepositoriesarerarelyaccessed.”24
Forthesereasons,anintelligentuserofGoogleoranyothercapablesearchengineisoftenabletoidentifylearningcontentandevenOERmoreeffectivelythanispossiblewiththeinternalsearchcapabilitiesofmanycurrentlyoperatingOERrepositories(althoughwithsomehonourableexceptions,aswillbeseeninthecasestudyonKlasCementbelow).Giventheaddedfactthatmanyrepositoriesareslowtorefreshtheircacheofcontent(usuallyforgood,thoughsometimesbureaucratic,reasons),andthatauniversalsearchenginecanbeusedeffectivelytopinpointrecentlycreatedresourcesfromacrosstheWeb,theneedforaneffectivesystemthatwilltrulyfacilitatethetaskoftheteacherseekingresourcesfromwithinasinglerepositoryisobvious.IfarepositoryistocompetewithGoogleanditspeers,itmustbeabletomatchthemwithinitsownlimiteddomainintermsofeaseofuse,flexibilityandpower,andofcoursetherepositoryitselfneedstobeextensiveenoughtoofferalargeandenticingcacheofresourcesfortheeducatortowishtoreturntoagainandagaintomineforcontent.DeOliveiraetalposittheideathatthiscanmosteffectivelybeachievedthroughthedevelopmentofaplatformthatexploitstheconceptofcollectiveintelligence:
“Theplatform’sgoalistoprovideaninteractivewebplatformwithsocialnetworkfeatures,toeasethepublication,searchandrankingofopeneducationalresourcesthroughacollectiveintelligence.Althoughcollectiveintelligencemaybringtomindtheideaofgroupconsciousness,incomputerscienceitusuallymeansthecombiningofbehavior,preferences,orideasofagroupofpeopletocreatenovelinsights.”25
Theyseektoharnesscollectiveintelligencetocreateasystemthatwillrankcontent(andusers)overtimeandthroughmultipleusesinwhich:
24https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.125https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.1-2
18
“….thehigherthescore,themorereliabletheuserorthecontentis…”26
Collectiveintelligence,whencombinedwithmachinelearning,isalsothemeansbywhichbureaucracycanbereduced,since:
“ThePlatformtrustsinthecommunitytodecidewhichuserisallowedtopublishresources,givinghighrankinguserstheabilitytopublishandinvitenewusersaspublishers.Thenewresourcesgothroughanautomaticfiltering,wheremachinelearningalgorithmstrytofurthermitigatethepresenceofinappropriatecontent.Theresultofthisfilteringisthenvalidatedbyhighrankingusers,toavoidanywronglyclassifiedresource.”27
Thisapproachformedthebasisforthecreationofaprototypeplatform,calledPlataformaMECdeRecursosEducacionaisDigitais28(MECREDforshort),inpartnershipwithBrazil’sMinistryofEducation.Theplatform,asofOctober2018,hadsome30,000publishededucationalresourcesand5000registeredusersaveraging1200downloadspermonth.Whileitwasnotpossibletomakeuseoftheplatformforthepurposesofthisreview,theapproachoutlinedbytheauthorsaboveisonethatcouldperhapsbeencouragedandlearnedfrom.Someconsideration,however,mightbegiventotheexpectationthatcombiningcollectiveintelligenceandmachinelearningcanonlybetrulyachievablewheretherepositoryenjoysacriticalmassofdata,bigenoughforMLtofunctioneffectively.29EXAMPLES OF REPOSITORIES WITH EFFECTIVE QUALITY SYSTEMS Asmallnumberofrepositorieswereidentifiedthat,itisfelt,aremakinggooduseofqualityfeedbackandranking.Wefocusonthreeoftheseherethatofferinterestingexamplestoconsiderinthecontextofthelong-termdevelopmentofOEP,butgivenmoretime,itiscertainthatotherscouldhavebeenidentifiedtoo.
KlasCement KlasCement30,anopenrepositorystartedbyoneteacher,HansDeFour31,inaschoolinFlandersin1998,isnowavaluableandextensiveOERrepositorysupportedbytheMinistryofEducationthatserveseducatorsinFlanders(andintheNetherlands,bydintoftheirsharedlanguage),aswellasacrossEurope.Withmorethan220,000uniqueusers,averyhighpercentageofthemactive,andwithmorethan58,000resourcescurrentlyheldintherepository,KlasCementisamajorplayeramongstOERrepositoriesinEuropeandbeyond.
26https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.227https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.228See:https://plataformaintegrada.mec.gov.br/home29ThankyoutoDavidMassartformakingthispoint30https://www.klascement.net31https://www.klascement.net/profiel/1/YouneedtoberegisteredonKlasCementandloggedintoviewthisprofile
19
Theavailableresourcescovermanydifferenttypesoflearningmaterials,includingarticles,downloadablelearningmaterials,websites,apps/software,interactiveexercises,images,videos,audiomaterials,andothers.Registereduserscanmakeflexibleuseoftheresources,beingabletosearch,view,download,re-useandadaptcontentfortheirteachingandlearningrequirements.Theyarealsoabletocommentoncontent,giveratingstoindividualresourcesandcanprintresourcesaswellassharethembyemailoronTwitter,Facebookandothersocialmediaplatforms.KlasCementisacontinuallydevelopingplatform,onethatseekstostayattheforefrontofOERfunctionality,notforitsownsakebutinordertocontinuetoofferitsuser-bodythemosteffectiveandusefuleducationalrepositorythatitcanintothefuture.ThosewhodirectKlasCement(agroupthatstillincludesitsfounder)useagroupofpart-timeteachersemployedbytheMinistryofEducationtoensurethatallcontent,whetheruser-generatedornot,meetstheirminimumstandardsofqualitybeforeitisapprovedforinclusionintherepository.Thesequalitystandardsinclude,amongstothers:
• ThattheresourcemetadatameetstheIEEE-LOMMetadatastandardtoensurethatitenablesfullinteroperability,coveringtitle,keywords,subjectmatteretc
• Thattheresourcealsoincludesadditionalmetadatatocoversareassuchaspedagogicalcontext,theme,etc.
• Thattheresourceisfullyproof-read,freeofmistakes,trustworthy,curriculum-relatedandpractical
• Thattherearenocopyrightviolations• Thatitoffersanidentifiableusefulnessinaneducationalsetting
Thereisalsoa‘crowd-sourced’componentthattakesaccountofuserreviews,commentsandratings.Forexample,foreachresourceavailablethroughKlasCement,theteacherorusercancommunicateanissueabouttheresourcetomoderators–theycanraiseissuesoffactualerror,problemswithhyperlinks,oranyotherissuethattheplatformshouldtakeaccountof.Tohelpeducators,KlasCementdeliverssearchresultsbasedonamathematicalformulathatincludes:
• Correspondencewiththekeywordinthesearchbar• Scoresgivenbyusers(andthenumberofusersthatgivethescore),• Timeonline• Typeofcontributor(teacher,student,companyetc)
Theendresultisaplatformthatservesitsuserstotheextentthat,withinitsownimmediateregionalandnationalcontext,itencouragesmanyuserstobypasstheubiquitousGoogleandinsteadtogofirsttoKlasCementwhentheyaresearchingforgoodandtrustworthycontentforalessonoracourse.IntheirconstantdesiretoimprovethefunctionalityofKlasCementsothatitbettermeetstheneedsofitsusers,theplatform’smanagershaveindicatedthattheywouldliketobeabletocreateamoredetailedandricher‘collectiveintelligence’thantheyarecurrentlyusing.Forexample,thescoregivenbyateachertoaresourcethathasbeenusedinteachinghasalimitedutilitygiventhefactthatthescoremightbe
20
givenforavarietyofreasons:designoftheresource,culturalcompatibility(anassessmentthat,ofcourse,willvaryfromculturetoculture),factualerrors,andmanyothers.Eachofthese,andanyother,reasons,mightinthemselvesbesignificantbutifafutureuserdoesnotknowwhatthatreasonwas,thenthescoreinitselfhasrestrictedvalue.32KlasCementisanOERrepositorythathasbeenabletogrowandprosperorganicallysinceitshumbleoriginsmorethantwodecadesago,tothepointnowwhereitoffersteachersandstudentsinFlandersandbeyondatrulyvaluableandadmirableplatform.Butevenmorecommendablethanwhathasbeenachievedtodateisthedeterminationofthosewholeadittocontinuetoimprovethequalityofitsofferingstoteachers.Itisarepositorythatdeservestobebetterknownbeyonditsownoriginalboundaries.
Curriki ThefeedbackmechanismsemployedbyCurriki33arealsoausefulreferencepointinthisdiscussion.CurrikiisanAmericaneducationnon-profitwhoseaimistobuildalargeglobalcommunitylibraryofOER,andtheyplacesharingattheheartoftheirmission.Theyaskmembersto:
• Sharewhatyoulearn.• Sharewhatyouknow.• Shareyourcontent.• Together,wecanmakeeducationmoreequitable.
WhenyousearchforcontentonCurriki34theresultinglistofresourcesoffersanumberoffunctionsthatateachercanusetoenhancethesearchandtoevaluateresourcesforothersinthefuture.Therearelinksofferingagreaterlevelofinformationonsubjects,appropriategrade/agelevels,andresource-types,aswellaslinkstoenabletheteachertosharetheresourcebyemail,onFacebook,TwitterandGoogleClassroom,andasaURI(forexample,inablogpost).Userscanflagthecontentasinappropriateandalsonominateforreview.Finally,ofcourse,ausercangivetheresourcearatingbetween1starand5stars,afeaturealreadydiscussedinthesectiononKlasCementabove.Currikiadditionallyratesresourcesaccordingtoitsownrubric,fromNR(notyetrated)upto3.0.Whenauserconductsasearchtheyareabletosorttheresultinglistofresourcesaccordingtomultiplecriteria:bytitle(A-ZorZ-A),bynewestfirstoroldestfirst,bymemberratingandbyCurrikirating.Interestingly,italsoenablesresourcestobesortedby‘standardsaligned’.Indeed,thesearchfunctioncanbeenhancedbysearchingaccordingtowhichevercurriculumstandardsyouwishtobealignedto.ThisismostobviouslywhereCurriki’slimitationsbecomeevidentto
32Indeed,ifascoreoutof5,forinstance,isfraughtwithdifficulty,thenasimplelike/dislikebinaryevaluationofaresourceisevenmoreproblematic,sincethereasonforthechoicecanvaryconsiderably,fromapplicabilitytoaparticulargroupoflearners,toeducationalrichness,togrammaticalcorrectness,topedagogicalprinciples,andsoon.33www.curriki.org34NotethattheCurrikiwebsiteprovedproblematicattimesintheSafaribrowseronAppleMacintosh,problemsthatwerenotapparentonotherbrowsers
21
aninternationalaudience.Despiteitsglobalaspirations,CurrikiisverymuchanestablishedAmericanplatformandsothe(admittedlyverylong)listofstandardsthatresourcescanbealignedtoisalmostexclusivelyAmerican,fromCommonCoretomanyindividualStatestandards.
Finally,asamemberofCurriki,ausercanalsoaddtheresourcetotheirpersonallibraryofcontent.CurrikiisverymuchfocusedontheK12sectorand,fromabriskperusalofthesite,mostofthecontentdoesseemtoofferresourcesofatleastreasonablequalitytoteachersandmanyappeartobeofhighquality.ItsAmericanoriginsmight,ofcourse,limititsusefulnesstoEuropeanandotherinternationalschoolssystems.Nonetheless,itoffersamodelthatotherOERrepositoriescouldlearnfrom.
MERLOT AnotherAmericanrepositoryofmuchlongerstandingthanCurriki,isMERLOT35,onewhichisverywell-knowngenerallyintheOpenEducationmovement.Itoffersaccesstocuratedonlinelearningandsupportmaterialsandcontent-creationtools,andisledbyaninternationalcommunityofeducators,learnersandresearchers.MERLOTwassetupin1997attheCaliforniaStateUniversityCenterforDistributedLearningandhasaHigherEducationfocus.AssuchitsusefulnesstoK12educatorsisnecessarilylimited.However,itdoesappeartoofferagoodmodelinrelationtotheevaluationofqualityfromwhich,aswithCurriki,otherOERrepositoriessitedelsewhereintheworldcouldlearn.
35https://www.merlot.org/merlot/index.htm
22
MERLOTissimilartoCurriki(andpossiblyofferedamodelthatCurrikimightitselfhavelearnedfrom)inthatitoffersusersthecapabilitytoratearesourcefrom1starto5stars.Resultsofacontentsearchcanbesortedbytheirratings,buttheycanalsobesortedbywhetherornotusershavecommentedontheresourceoraddedtheirown‘learningexercises’totheresource.Thesearchfunctionalso,importantly,permitssearchingbyaccessibility,andbyitsinclusioninany‘bookmarkcollections’or‘coursee-portfolios’.A5-starratingsystemsuchasthismighthavethevirtueofsimplicity.However,itcanalsobearguedthatsuchasystemistoosimplistic,asalreadynotedinthesectiononKlasCementabove.Theunderlyingreasonswhyausermightgiveascoreof1outof5,example,candiffersogreatlythatthescoregivenreallyofferslittleofsubstancetosubsequenceusersofthatresources,sincetheywillnotknowwhetherthelowscorewasgivenbecauseofpoordesignoranerroroffactoraculture-clashorwhatever.Ontheotherhand,ifascorecanbeascribedalongsidesomequalitativecommentastothereasonforthescore,thatwouldatleastgosomewaytoincreasingtheutilityofthebarerating.Asacentrallycuratedrepository,thereisa‘top-down’aspecttoMERLOT,butalsoa‘bottom-up’aspect.Allthematerialsarecontributedbymembers(membershipisfree),butmanyofthematerialssubmittedgothroughapeer-reviewprocess,overseenbyanyoneoftwentyeditorialboards.Eacheditorialboardismanagedbyaneditorandassociateeditor,andeachhasateamofpeerreviewers.
“AlloftheseitemshavebeencontributedbytheMERLOTmembercommunity,whohaveeitherauthoredthematerialsthemselves,orwhohavediscoveredthematerials,foundthemuseful,andwishedtosharetheirenthusiasmforthematerialswithothersintheteachingandlearningcommunity.”36
Materialsarereviewedonthebasisof:
• Qualityofcontent• Potentialeffectivenessasateachingtool• Easeofuse
MERLOTevenprovidestrainingforpeerreviewers,althoughallpeerreviewersarerecognisedauthoritiesintheirownfield.Crucially,MERLOTisnotarepositoryinthesensethatitactuallystoresallthecontentwithinitsownstoragespace.Rather,itisametadata-basedrepository.Inotherwords,allofthecontentmadeavailablethroughMERLOTisretainedinitsoriginallocationanditisthemetadataattachedtotheresourcesthatmakethemsearchableinMERLOT.ThecontentismerelyrendereddiscoverableandretrievablethroughMERLOT.Ofcourse,thiscreatestheoverheadofhavingto
36http://info.merlot.org/merlothelp/topic.htm#t=MERLOT_Collection.htm
23
verifyallthelinksonaregularbasis.MERLOTcarriesoutthistaskonamonthlybasisandreportsadeletionrateofapproximately2%onaverage.Thisisamodelthat,ontheonehand,mightappearmoreflexibleandexpandablethanafixedrepositorythatstoresallofitscontentwithinitsownbounds.However,itshouldalsobenotedthat,inaccessingasystemsuchasthis,theteacherrunstheriskofcontentpreviouslyuseddisappearingthenexttimeitissearchedfor,aproblemsthatisunlikelytooccurinarepositoryinwhichresourcesarestoredonafixedsingleserverorsetofservers(suchasisthecasewithKlasCementalreadydescribed).BothCurrikiandMERLOTofferveryeasy-to-usemechanismstouserswhowishtoratethecontenttheydownload,usingasimple5-starsystem.Othersystemsofferuser-feedbacksystemsthataremuchmorecomplex.Oneofthemost
complexfoundduringthereviewwasthatofferedbyOERCommons37inwhichusersfirsthavetoalignthecontenttooneormoreofalargenumberofUS-basedcurriculumstandards,andthenareaskedtoevaluatetheresourcesacrosssixfurthersetsofcriteria.
Further Discussion ItispossibletoconcludethatKlasCement,CurrikiandMERLOTalloffer,tosomeextent,systemsthatworkonthebasisofaformof‘collectiveintelligence’notdissimilartothatoutlinedbyDeOliveiraetalabove.Thesimplefactthatlargenumbersofusersareevaluatingresourceswithinaverylargecacheofcontentbuildsalevelofcommonandsharedknowledgeacrosstheplatformsthatwouldappeartoservethebroadneedsoftheircommunitiesofusersverywellindeed.Itshouldbenoted,however,thatthosedevelopingandmanagingKlasCementfeelthattheycanandwillgofurtherindevelopingtheconceptofcollectiveintelligenceintheirrepository.Largeandeffectiverepositoriessuchasthesealsoenablethe‘qualitycircle’concepttooperate,ensuringsomemeasuresofqualityboth‘beforethefact’–i.e.whentheteacheristryingtolocategoodqualityusablecontentforuseinteachingpractice–and‘afterthefact’–i.e.whentheimpactofanyparticularresourceisassessedandsuchintelligenceisfedbackintothesystem.
37https://www.oercommons.org
OERCommonsUser-FeedbackSystem
24
AcursoryexaminationofvariouscategoriesofresourcesinOERCommonssearchedforrevealsthatmanyoftheresourceswithintherepositoryareasyetunevaluated.Thecomplexityandsomewhatbureaucraticnatureoftheevaluationprocessmighthaveabearingonthis,buttheeffectisthatanyperceptionofcollectiveintelligencedissipatesthroughthesystem’sseeminglackofuse.Theease-of-useofasystemforevaluatingandfeedingbackonaresourceis,itcanbeinferred,asignificantfactorindeterminingwhetherornottheresourcesareevaluatedtoanyusefulextent.Complexandcumbersomesystemsarelesslikelytobeusedsuccessfullyorextensively.Ofcourse,feedbackmechanisms,aswellasbeingtoocomplexorcumbersome,canalsobetoosimplistic.AnexampleofthisthatcanbetakenfrombeyondtheworldofOERrepositoriesisthetoolthatYoutubeprovidestoenableaviewertolikeordislikeavideo,thesimplethumbs-uporthumbs-down,whichineffectworksinsimilarfashiontothe‘Like’facilityinFacebook.UsingsuchatoolinanOERrepositorywouldcertainlymakethetaskoffeedingbacksimpleforeducators.However,asalreadynoted(p.24)thevalueofasimplebinary‘like’or‘dislike’islikelytobeinadequateforthepurposesofrankingresourcesforusersinanyrigorousway.However,arankingsystemthatatleastletsusersseehowmanytimestheresourcehasbeenopened,andmoreimportantly,downloaded,wouldatleastoffersomenotionofthepopularityofaresource.Whetherthatpopularitycanbetranslatedintoanassessmentofqualityisquestionable,ofcourse,sincearesourcecreatedforanichesubjectwillappeartobemuchlesspopularthanamuchmoreprevalentsubject.Asimplerankingby‘numberoftimesdownloaded’thereforeneedstobetreatedwithcaution.BothEuropeanSchoolnet’sownEuropeanSchoolnetAcademy38andtheEuropeanTeacherAcademy39(whichwasdevelopedforSchoolEducationGatewaybyEuropeanSchoolnetonthebasisofitsexperiencewithSchoolnetAcademy)havesimilartools,namelyathumbs-uplinkforeachofthecoursesorwebinarsmaterialslistedintheircatalogues.However,ratherthansimplyregisterauser’s‘like’ofacourseinthecatalogue,SchoolnetAcademytakestheusertoitsFacebookpagesothatheorshecanleaveanapprovingmessageorarecommendationthere(or,wepresume,anoteofdisapproval).Whilebothoftheserepositoriesundoubtedlyoffercontentofahighquality,theapparentlackofuseofthecommentinganduser-feedbacksystemsavailableonthemmustdetractsomewhatfromtheoverallutilityoftheplatforms.OntheTeacherAcademy,forexample,thereareclearlymanyhighlyusefulcoursesforteachersseekingtobuildontheirknowledgeandskills(thesheerbreadthanddepthofqualitythatisevidentacrossallthecoursesonofferisundeniable,andespeciallysosinceteacherscanoftenfinditdifficultorevenimpossibletoobtainstrongandeffectivesupportfortheirprofessionaldevelopmentneedsneedfromanywhereelse).However,ithastobesaidthat,despitetheobviousqualityon
38https://www.europeanschoolnetacademy.eu39https://www.schooleducationgateway.eu/en/pub/teacher_academy.htm
25
offer,ateachercomingtothecatalogueforthefirsttimewillfinditdifficulttogetanypriorfeedbackfromusersonhowusefulthecourseswereandwhy.
THE QUALITY CHALLENGES: Three Short Case Studies Thefocusinthissectionisonthechallengesfacedbyrepositoriesastheyseektoensurethatusershavethebestexperiencepossibleinrelationtothequalityresourcestheylocateandsubsequentlymakeuseof.Threeshortcasestudiesthatprofiledifferentqualityissuesareoffered.Scientixisastronglycuratedsitethatadmitscontenttoitsrepositoryonlyifitmeetsitsownminimumstandardsofquality.WhenuserssearchScientixforcontent,theycanbesurethatleastthecontenttheyidentifyhassomebasiclevelofquality.ThissectionofferssomequalitativeconsiderationsbasedontheexaminationofthesearchfunctioninScientix.GeoGebraisaninteractivemathematicsapplicationthatisbothbrowserandappbasedonmultipleplatformsandthatisaimedatmathematicslearnersfromprimaryschoolthroughtouniversity.Arecentpaperco-authoredbyGeoGebra’sCEOofferssomefascinatinginsightsintohowaplatformmightgoabouttryingtoimprovethequalityofitssearchfunctionsoastoensurethatusersfindtheresourcesthataremostappropriatetotheirneeds.PhotodentroLORistheNationalLearningObjectRepositoryforprimaryandsecondaryeducationinGreece.Itisanextensiveplatformthathostsalargecacheoflearningobjects,eachofthemtaggedwitheducationalmetadatabasedontheIEEE-LOMMetadatastandard.Itisopentoeveryone:students,teachers,parents,andothers.
Case Study 1: Scientix Scientixwascreatedin2010ontheinitiativeoftheEuropeanCommissionandhasbeencoordinatedbyEuropeanSchoolnetsinceitsfoundation.Ithasbecomeaverypopularplatform,onethatoffersaverylargecomplementofmainly(but
notonly)STEMresourcesacross24EUlanguages.AllofitsservicesareaccessibletoanySTEMeducatorsandotherstakeholdersforfree.TheScientixUpdate,201940,describeditsofferingsthus:
• AccesstoinformationonEuropeanscienceeducationprojects• Accesstohigh-qualityresourcesinscienceeducation,withon-demand
translationofsometeachingmaterialforfree
40SCIENTIXUPDATE2019http://files.eun.org/scientix/scx3/publications/Scientix3_Final_Publication_web.pdf
26
• AccesstoScientixnews,enablinguserstokeepinformedaboutthelatestnewsonscienceeducationinEurope
• AScientixeventcalendar,bringingtogetherscienceeducationeventsfromacrossEuropeandbeyond
• Exchangeandsharingtoolssuchasforumsandchats.AcriticalobjectiveofScientixistosupporttheSTEMteachingcommunityacrossEuropetoenableeducatorstofind,downloadandmakeuseoftheverybest,high-qualitylearningresourcesthatcanbefoundinSTEMeducation.Therepositoryitselfisacollectionthatisdividedintoteachingmaterials,STEMreportsandtrainingcourses.ItalsooffersopportunitiestoattendorparticipateinwebinarsandconferencesrelevanttoSTEM.Inadditiontothis,Scientixprovidesarobustplatformfordiscussionandexchangebetweenministriesofeducation,NGOsandotherrelevantagenciesandorganisationsastheydevelopandimplementSTEMpoliciesacrossthecontinent.Importantly,itiscentraltotheScientixvaluesandprinciplesthatteachersthemselveswillplayakeyroleininfluencingstudentperceptionsofSTEM(andother)subjectsandinsupportingstudentstoaimforSTEMopportunitiesinhighereducationandintheircareersintothefuture.TheScientixresourcerepositoryisacuratedcollectionoflearningmaterialscentredonSTEMeducation.ManyoftheseresourcesaretheproductsofprojectsthathavebeenfundedandledbyScientix,andtodatealmost2,500teachingmaterialssitintherepositoryalongsidehundredsofSTEMreportsanddozensofhighqualitytrainingcourses.TherepositoryalsogivesaccesstoarangeofScientixMoodlecourseswhichareself-pacedandnon-tutored.ThecurationprocessiscentraltohowScientixoperates.RegisteredusersontheScientixportalareabletouploadresources,andthesearethenreviewedbytheScientixeditorialteam.Uploadguidelinesareofferedsothatresource-creatorshaveaclearideaforhowtheycanensuretheirresourcesareacceptedintotherepository.AswiththeMERLOTrepository,itisonlythedetailedmetadataoftheresourcethatisuploadedsinceScientixlinkstotheoriginallocationoftheresource(andallsuchlinksareverifiedbeforeinclusion).Inmostcases,thisisthewebsiteoftheprojectthatcreatedtheresourceinthefirstplace.Thoseuploadingresourcesarerequiredtoofferpre-defineddatasuchastypeofresource,appropriateeducationalstageandage-rangeofstudents,andschoolsubjects,amongstmanyothers41.Itshouldalsobenotedthat:
“TheScientixresourcerepositoryisnotopentoprojectscreatedwithoutofficialendorsement(i.e.theymusthaveprivateorpublicfunding,beit
41D.MassartandE.Shulman,LearningResourceExchangeMetadataApplicationProfileversion4.7,EuropeanSchoolnet,2011.OurunderstandingisthatScientixusesavariantoftheLREMetadataApplicationprofile,asdoanumberofEuropeanMinistriesofEducationprovidingresources
27
regional,nationalorEU),self-fundedprojectsorschoolprojects(teacher'sprojects).Resourcesfromsuchprojectswillbeautomaticallyrejected.”42
PerceptionsofqualityinthelearningresourceswithintheScientixrepositoryarealmostentirelybasedonthefactthatallresourcesarecuratedbytheScientixeditorialteamandthatalltheresourceshavebeendesignedtomeettheminimumqualityrequirementssetbytheeditorialteam.Therepositoryplacesalinkwithineachresourcedescriptiondesignedtopermituserstoaddcommentsandaratingfortheresource,but,ontheevidenceofourbrisksurveyoflearningmaterialsoffered,theydonotseemtobeusedmuch.Qualitytherefore,inthisparticularcontext,ispredeterminedbytheproceduresoftherepositoryanddoesnotappeartodependmuchifatallonusers’perceptionsofquality.Itisclearthatplatformoffersabroadrangeofhigh-qualityresourcesintheformofteachingmaterials,reports,trainingcoursesandLREmaterials.Evenacursoryexaminationbyaprofessionaleducatoroftheresourcesonoffershowsthatthematerialsandcoursesavailableoffermuchofvaluetoteachersandlearners.Eachresourceoffersverycompletelevelsofmetadataforfilteredsearching,aswellastheoptionsofwritingcommentsaboutthematerial,ratingthematerialfrom1starto5stars,andreportingproblemswiththecontent.Additionally,Scientixhasmadetheverypositivemovetoofferthesearchfunctionin24Europeancommunityandotherlanguages.Indeed,Scientixalsooffersafreetranslationserviceforanyonewhowantstotranslatearesourceintoanotherlanguage.ThesearchfunctionitselfwithinScientixappearstobehighlyeffective.AsearchformaterialsinScientixoffersfilteringbysubject,byresource-type(forexample,app,case-study,course,experiment,lesson-planetc),byappropriateage-group,andbylanguage.However,onaccessingtherepository,itquicklybecomesclearthatonlyaminorityofresource-usersareactuallymakinguseofthefeedbackmechanismsavailabletothem.ThisisofcoursenotachallengethatisfacedexclusivelybyScientix.Rather,itisachallengefacedbyallOERrepositories.
Case Study 2: GeoGebra Averyeffectiveandwell-designedOERrepositoryintheEuropeancontextisGeoGebra43.ItisbasedinAustriabuthasaninternationalfootprint.Ithaswonastringofawardsanditusesaverylarge,flexibleanddynamicglobalteamtodevelopitsresourcesandtotranslatethoseresourcesintomultiplelanguages.Itisanopenplatformbothinthesensethatanyonewiththeknowledgeandskillstodosocancreateanduploaddynamicresourcestothe
42http://www.scientix.eu/resources/upload-guidelines43https://www.geogebra.org
28
GeoGebrarepository,andinthesensethatitisopentoanyonetomakeuseofforteachingandlearningpurposes.GeoGebradescribesitselfthus:
“GeoGebraisdynamicmathematicssoftwareforalllevelsofeducationthatbringstogethergeometry,algebra,spreadsheets,graphing,statisticsandcalculusinoneeasy-to-usepackage.GeoGebraisarapidlyexpandingcommunityofmillionsofuserslocatedinjustabouteverycountry.GeoGebrahasbecometheleadingproviderofdynamicmathematicssoftware,supportingscience,technology,engineeringandmathematics(STEM)educationandinnovationsinteachingandlearningworldwide.”44
GeoGebraoffersaconvenientandinstructivecasestudyofsomeofthechallengesfacedbyanyOERrepositoryasitseekstodeliveraplatformthatisconsistentlyabletoofferusers‘gooddynamicmaterials’thataccordwiththeirown–thatis,theusers’–ownstandardsofquality.Forthissection,extensiveusehasbeenmadeofahighlyusefularticlewrittenbyMarkusHohenwarter(CEOofGeoGebra)andCristianHedes,entitled:“UsingQualityFactorstoFind“Good”ResourcesontheGeoGebraMaterialsSharingPlatform.”45(ThepaperwasverykindlysharedwithusbyMarthaZellinger,whooffersPartnerSupportforGeoGebra.)HohenwarterandHedesoffertheinsightthatexpertusersofGeoGebrahaveindicatedthattheytendtosearchformaterialswhichhavebeenproducedby‘highqualityauthors’,namelyresourceproducerswhotheyknowhaveproducedgoodqualitymaterialsinthepast.Inresponsetothis,GeoGebratookthestepofaddingtheoptionforuserstofollowcertainauthorsandtoreceivenotificationswhenevertheseauthorsuploadednewcreationstotheplatform.Butanumberofotherqualitymeasuressuggestedby‘expertusersandliterature’werealsoaddedintotheplatform,andHohenwarterandHedesexplainwhatthosefactorsareandhowtheychosetodealwiththeminGeoGebra.UsingthefindingsofKimeswenger201646,theydefinedthreefurtherfactorsthatneededtobetakenintoaccountintheirattempttoimprovethesearchforgoodmaterials.Thesearethematerialfactor,theauthorfactorandtheuserfactor.Thefirstofthese–materialfactor–seekstomeasuretheactivityhappeningaroundacertainmaterial,forexample,howmuchitisbeingused,copiedandfavourited.Theyderivedthefollowingparameterstomeasurethisfactor:
44https://www.geogebra.org/about45http://www.iadisportal.org/digital-library/using-quality-factors-to-find-%C2%93good%C2%94-resources-on-the-geogebra-materials-sharing-platform46Kimeswenger, B., 2016. Addressing Quality Aspects of Dynamic Mathematics Materials. Proceedings of 13th International Congress on Mathematical Education (ICME 13). Hamburg, Germany
29
• featured:materialsthathavebeenfeaturedbytheGeoGebrateamshouldhavehigherpriority
• type:books(collectionsofworksheets)shouldhavehigherprioritythansingleworksheets
• derived:materialswithmanycopiesshouldhavehigherpriority• groups:materialsusedbymanygroupsshouldhavehigherpriority• books:materialsusedinmanybooks(collections)shouldhavehigher
priority• favourites:materialsthathavebeenfavouritedbymanyusersshould
havehigherpriority• viewsperday:materialsthatareviewedmoreoftenshouldhavehigher
priorityThesecond–authorfactor–hastoreflectthe‘expertiseandrelevanceofthematerial’sauthor’.Inotherwords,resourcescreatedbypeoplewithmanyfollowersshouldbegivenhigherprioritythanthosewithfewerfollowers.Theyderivedthefollowingparameters:
• followers:materialsofauthor’swithmanyfollowersshouldhavehigherpriority
• community:materialsofauthorswithGeoGebracommunitybadgesshouldhavehigherpriority
• publicmaterials:materialsofauthorswithmanypublicmaterialsshouldhavehigherpriority
• teacher:materialsofteachersshouldhaveslightlyhigherprioritythanmaterialsofotherusers
Thefinalfactor–userfactor–isareal-timemeasureusedatthepointofsearching,andisdependentupontheuserdoingthesearching.Theparametersthatwerederivedare:
• language:materialsinthesamelanguageastheuser’slanguageshouldhavehigherpriority
• userID:thematerialsofthesearchingusershouldhavehigherprioritythanmaterialsfromotherusers
Bycombiningthesenewfactorswiththeequationspreviouslyusedtodoatext-relevancesearch,usingnormalizationfunctionsandweighingfactorsderivedfromanalysingtopsearchresults,GeoGebrahasbeenable,theybelieve,toimprovethequalityofsearchresults.Thisis,theysay,stillverymucha‘workinprogress’andtheyareworkingtofurtherimprovethequalityofresultsbydelvingfurtherintothedetailofthevariousparametersestablishedforeachoftheadditionalfactors.Theyarealsoawarethattheroutetheyhavetakenmighthavethedetrimentaleffectofmakinggoodqualitymaterialscreatedbynewauthorslessvisible,andmakeittougherfornewauthorstoestablishafootholdintheplatform.
30
Case Study 3: Photodentro
AcriticalcomponentoftheGreeknationalinfrastructureforK12educationisthePhotodentroplatformwhichoffers5OERrepositoriesmakingdigitalcontentavailabletoallschoolsinthecountry.Theplatformactsasacentralaccesspointtoschool-basedlearningresourcesintheGreekeducationsystem.Italsoactsasanationalaggregator,aserviceforharvestingeducationalmetadatafromexternalcollectionsandrepositories.Theplatformnowservesastheparamountdigitalwork-spaceformorethan120,000teachersandmorethan1,000,000studentsinthecountry’sprimaryandsecondaryschools.Greecehasestablisheddigitaleducationalcontentasoneoffivefundamentalpillarsintheeducationsystem,theotherfourbeing:
• In-serviceteachertraining• Thedigitalclassroom(classroomsequippedwithinteractiveteaching
systemsandsupportedbyanation-wideschoolnetwork)• Horizontalsupportactions• Electronicadministrationplatformscomplementthegroupofpillars.47
Atthemetadatalevel,qualityinPhotodentroisassuredthroughadherencetothePhotodentroIEEELOMGRApplicationProfile.However,thetaskofassuringthequalityofthecontentintherepositoriesisachievedbymeansofthePhotodentroQualitySeals,whichis:
“…anewrepositoryofthePhotodentrofamilytosupportaqualityassuranceschemeforitscontentbasedonQualitySeals.AQualitySealcanrepresenteitheraprocess,orasetofevaluationcriteria,orabrandname.TherepositoryhostsandmanagesQualitySeals,stampingrecordsforOERs,whileithasaRegistryofCertifiers.”48
Thereisastrongemphasisacrosstheplatformonmetadataqualitysothatthesearchforandretrievalofcontentisaseffectiveaspossible.TheQualitySealsconceptisavaluableattempttobuildcontinuousmetadatacurationintothe
47Megalou,EandKaklamanis,COpenContent,OERRepositories,InteractiveTextbooks,andadigitalSocialPlatform:TheCaseofGreecep.1https://www.academia.edu/37054610/OPEN_CONTENT_OER_REPOSITORIES_INTERACTIVE_TEXTBOOKS_AND_A_DIGITAL_SOCIAL_PLATFORM_THE_CASE_OF_GREECE48Megalou,EandKaklamanis,Cibidp.2
31
platform,andisonethatshouldofferamodelthatotherrepositoriesmightbenefitfromtoo.QualityinmetadataismaintainedinPhotodentrothroughtheworkofasub-groupofteacherswhohavebeentaskedwithannotatingallcontentwithclearandrobustmetadataandpublishingit.Theyworktoaclearlyspecifiedandstandardizedprocessthatsetsouttheprocedurebywhichcoordinatorsassignresourcestoteamsallthewaythroughtofinalpublication.TheseteamsaresupportedbytrainingcoursesandtrainingmaterialsintheformofFAQs,guidesandmanuals.Beforetheprocessiscompleteforanyindividualresourceorcollectionofresources,afinalcheckisconductedbythecoordinatorofthecollection.TheGreekeducationsystemishighlycentralisedandalltheresourcescontainedinPhotodentroLORaredevelopedby120project-employedfully-qualifiedteachersworkingacross10domain-specificworkgroups,withtheaimofaugmentingandsupplementingthespecifiedGreektextbookswithhigh-qualitydigitalinteractiveresources.Eachgroupisoverseenbyanacademicwithsignificantdomainandpedagogicalexpertisesoastoensurethequalityofthelearningobjects.Thisisaverydifferentmodelfromthatemployedby,forexample,byKlasCement,Currikiandothers,whichuseamuchbroadersystemofuser-generatedcontent.Differentmodelsareneededfordifferentcircumstances,however,andtheGreekmodelinPhotodentrohasmuchtocommenditforacentralisedsystemofeducationsuchasthatwhichappliesinGreece.DIFFERENT APPROACHES TO QUESTIONS OF QUALITY IN OER Thesethreeshortcasestudiesarebynomeansintendedtoofferacomprehensivetakeonhowrepositoriesshouldseektooffertheirusersthehighestqualityoflearningmaterialspossible.Thatcannotbeachievedinashortpapersuchasthis.Howeveritdoesdemonstratethreeapproaches(ofmanypossibleapproaches)thateachoftherepositoriesinquestionhavetakentothisissue.Eachisderivedfromadifferentsetofassumptions,buteachisavalidattempttobuildsomelevelofqualityassuranceintotherepositories.Itisthecaseofcoursethat,whilesomemightseektodevelopuniversalsolutionsorglobalapproachestoqualityinOER,attheendofthedayitisuptoeachrepositorytodeterminehowbestitcanachieveitsobjectivesonbehalfofitsusers,andthatbestfitswiththeveryparticularcontextinwhichitexists.
32
CONCLUSION: A CRITICAL EVALUATION OF QUALITY CRITERIA Qualityliesattheheartofthispaper.Definitionsofqualityinrelationtoeducationalresources,openorotherwise,abound.Ofallthepossibledefinitions,theonethatseemstoresonatemostwiththeneedsofeducatorswhowishtosearchforanduseOERisthatof‘fitnessforpurpose’.Itistheoneaspectofthevariousdefinitionsonofferthatmostreadilybearsscrutinyfromtheperspectiveofateachersearchingfor‘goodquality’resources.Thislinksbacktothefirstofthefive‘qualityfactors’outlinedbyCamillerietalonp.11.Fromtheuser’sperspective,thebestresourcetheycanfindwillbetheonethatbestfulfilstheirteachingpurposeand,theyhope,thelearningpurposeoftheirstudentstoo.Inthissense,qualityisviewedasa‘before-the-event’conceptwithrespecttothe‘QualityCircle’describedinthispaper.Theteacherwithaclearideaofthepurposeofhisorherteachingwillbeabletoevaluateanyresourcestheyfindmoreeffectivelythantheteacherwithonlyavagueideaofwhattheyhopetoachieveinaparticularlessonorcourse.Forexample,agoodprofessionalteacherwillknowwhatknowledgeorskillstheywillbeteaching,whethertheyaretryingtobuildonwhatstudentsalreadyknoworsimplyreinforcingexistingknowledgeorskills,howstimulating,enjoyableandchallengingtheywantthelearningtobe,whatmethodologiestheyintendtoemploy,whatkindofenvironmentforlearningtheyaretryingtocreate,whatkindofassessmentandevaluationtheywilluse,andwhatpersonalqualitiestheyhopetoinstilintheirstudents.Thisisonlyasubsetofallthepotentialpurposesateachermighthaveinmindforalesson,buttheteacherwhohasaclearideaofpurposewillbebetterabletoselectappropriateandrelevantcontent.FromtheperspectiveoftheOERrepository,therefore,themoretheyareabletoprovidemetadata,descriptions,commentandratingsthatgivetheteachertheinformationtheyneedtoconnectaresourcetotheirteachingrequirements,themoreeffectivelytheteacherwillfindcontentthatis‘fitforpurpose’.Assuch,tryingtodeliveracriticalevaluationofqualitycriteriaisnotsomethingthatcanbeachievedinanyuniversalsense:suchevaluationcanonlyreallybecarriedoutonapiece-by-piecebasisbytheteacherastheylocateandusecontent.However,withthisasthekeyyardstickbywhichtomeasurequality,OERscanperhapsmoreeffectivelystructuremetadataandfeedbacksystemstohelpteacherstryingtoassessthe‘fitnessforpurpose’ofresources.Whentheissueunderconsiderationisthatofquality–andparticularlytheissueofhowarepositorydealswithratingthequalityofitsresources-thekeychallengetendstobearoundthebehaviourofitsusers:howcaneducatorsandotherswhoseekaccesstoopenresourceseitherbepersuadedtogivefeedbackonresourcestheyhavealreadyusedorgainsomeawarenessofaresource’squalitybeforetheymakeuseofit?
33
SUGGESTED QUESTIONS FOR FURTHER RESEARCH Fromthereadingandresearchcarriedoutforthisshortreview,itispossibletoidentify,inconclusion,afewkeyissuesthatariseoutofacriticalevaluationoftheevidenceoutlinedinthispaper.Theycanbesummarisedthus:i. IntheEuropeancontextinparticular,therealchallengeofmultiple
languagesexists.Thelanguageofsearchandthelanguageoftheresourcesthemselvesisacriticalqualitativefactortobetakenintoconsiderationbyrepositories.Usersareoftenhavingtosearchforresourcesinasecondoreventhirdlanguagebutitisclearthatsomerepositoriesaremakinggoodprogresstowardsdealingwiththischallenge.ThisisachallengethatisalreadybeingexploredintheEUNProject‘eQNET’49,whichisexaminingthecriteriabywhichsomeresources‘travelwell’acrossnationalandculturalboundarieswhileothersdonot50.
ii. WhileusersmaybedownloadingresourcesfromOERrepositories,
theyarenotalwayscommentingonissuesofusefulnessandqualitywithintherepositoriesthemselves.Atleastonepossiblereasonforthis,andthatmeritssomeinvestigation,isthat,insomecountriesatleast,teachersoftensetuptheirownonlinegroupsandcommunitiesinwhichtoshareideas,resources,lessonplansandsoon.Thiscreatesakindofhinterlandbeyondtheopenrepositoriesandopencommunitiesinwhicheducatorsarecertainlymakinguseofopenresourcesbuttheyaredealingwithqualityissueswithinclosedorexclusivegroups,oftenataregionalornationallevel.Thereislittleresearchtodatetoverifythisassertion,whichisbasedonexperienceandobservation.
iii. Thepotentialcontentionbetweentwounderstandingsofqualityin
relationtolearningresourceshastobedealtwithpragmaticallybyrepositories.Forthebenefitofeducatorsseekingresourcesfortheirteachingpractice,theirprimeconsiderationwillalwaysbefinding‘useful’resources,resourcesthatmatchtheirpedagogicalrequirements.Intermsoflearners,however,thetruequalityofaresourcewillalwaysbeitsultimateimpactonthelearning,adeterminationthatcanonlybemadeafter-the-fact,andoftenwillbeadeterminationthatisapplicableonlyinthelimitedsituationdefinedbytheactuallearningactivityinwhichtheresourcewasused.
iv. Wefeeltheconceptofthe‘QualityCircle’inrelationtoOEPisaconceptworthdevelopingbeyonditsinitialmodestoutlinehere.TheresearchbasefordefinitionsofqualityinrelationtoOERisdividedtosomeextent,therefore,betweenthosewhowouldwishtoacceptWiley’sdefinitionofquality(outlinedonpage11ofthispaper)asparamountandthosewho,inlookingatthedomainfromateacher’sperspective,
49http://eqnet.eun.org/web/guest;jsessionid=E20318079722343915D3DFAE80661EF850http://lreforschools.eun.org/web/guest/travel-well
34
understandtheneedforsomepre-determinationof‘quality’sothat,whenasearchismadeforteachingresources,teacherscanbereasonablysureoffindingsomethingappropriateanduseful.
v. Furtherinvestigationcouldbemadeofthoserepositoriesthatmosteffectivelybuildpracticalandusableuser-feedbacksystemsintotheirplatforms.Theseseemmostlikelytoofferacredibleandtrustedservicetousers.Whetherthesearesimplestarratingsystemsorsomethingmorecomplex,theyseemtoworkbestwhenitistheusersthemselveswhoareabletoplayaroleindeterminingquestionsofqualityinrelationtothelearningresources.
vi. WorkcouldbedonetodeterminehowmanyOERrepositoriesareofferingsuchfunctionalitynowandhowsuccessfultheyareintheirimplementations.ItisunfortunatethatCurrikiinparticularisalmostexclusivelyaimedataUSaudiencedespiteitsstatedglobalaspirations–perhapstheycanbepersuadedtoallowtheirresourcestobemappedtocurriculafrombeyondAmericanshores,althoughwhethertheresourcesaresuitableforanon-Americanaudiencehastobeaskedtoo.
vii. Theconceptof‘collectiveintelligence’usedbyDeOliveiraetalisonethathasmuchtocommendit.SystemssuchasCurrikiandMERLOT,whilenotnecessarilyfollowingalltheconstructsofferedbytheBraziliansystemdescribed,doundoubtedlyofferanapproachtoqualityassurancethatisbuiltaroundthecollectiveviewsexpressedbyusersthroughtheuseofratingsystemsand,asaresult,offeredinthesearchfiltersthatenableuserstorankfindingsaccordingly.
viii. TheadaptationsmadetothesearchfunctionwithinGeoGebraintermsofwhattheycallthematerialfactor,theauthorfactorandtheuserfactorispersuasiveandisperhapssomethingthatotherlargerepositoriescouldtakeonboard.Theyarechangesthatcanonlybeeffectivelyappliedtorepositorieswithverylargecollectionsandaverylargeusercommunity.Again,itmightbeviewedasarealapplicationofameasureofcollectiveintelligencetothesearchtoolwithintheplatform.Thatsuchamendmentsbringtheirownissueswiththem,forexampleintermsofpossibleexcludingnewauthorsfromsearchresults,issomethingthat,asGeoGebraacknowledges,requiresfurtherresearch.
ix. Thecomplexityandlevelsofbureaucracybuiltintoarepositorywillhaveabearingonitseffectiveness.Teachersareunlikelytouseacomplicatedfeedbacksystem.
x. IfarepositoryistocompetewithGoogleanditspeers,itmustbeabletomatchthemwithinitsownlimiteddomainintermsofeaseofuse,flexibilityandpower.Therepositoryneedstobeextensiveenoughtoofferalargeandenticingcacheofresourcesfortheeducatortowishtoreturntoagainandagaintomineforcontent
35
xi. Somefurtherworkmightbedoneonaddingbreadthanddepthtotheconceptof‘efficacy’asitrelatestothequalityofeducationalcontent.Thefitness-for-purposeofaresourceiscriticalinhelpingteacherstofindexactlywhattheyneedforaparticularlessonorcourse.Theperceivedorstatedefficacyofaresourcesmightbethemostappropriatefactortotakeintoaccountinthinkingabout‘quality-before-the-fact’inrelationtoopeneducationresources.
Finally,whilethispaperdidnotventurefarintothequestionofArtificialIntelligenceandMachineLearning,itisreasonabletosuggestthatthisisanareathatwillprovetobeincreasinglyimportantaswemoveforwardtothenextgenerationofOERrepositories.Anyresearchquestionsarisingoutofthisareawillhavetoincludenotonlytechnicalandpedagogicalissuesbutalsoquestionsofethics,fairnessandresponsibilitytoteachers,learnersandresearchersthemselves.QualityisundoubtedlyadifficultconcepttodefineinrelationtoOER.ItisnonethelessadifficultythatmustbetackledifweallwishtoseetheOpenEducationmovementgofromstrengthtostrengthinthefuture.
36
APPENDIX: ACTIONABLE POINTS & RECOMMENDATIONS Alistof‘actionablepoints’havebeenextractedfromthebodyoftheWhitePaperandhavebeencollectedhereasafinal‘aidememoire’forthoserepositoriesthatareseekingtoimprovetheservicetheyoffertheirusers.Attheheartofthelistbelowisaneedalwaystostartfromtheperspectiveofthelearnerandtheteacher.Thesimpletruthusthatanyrepository,nomatterhowbigitmightbe,thatdoesnothaveaworkingconceptionofitsuserbaseandoftheirrealteachingandlearningneedsisunlikelytobeabletoservetheirneedstrulyeffectively.Eachoftheactionablepointsbelowcomeswiththerelevantpagenumberinthepapersothatthepointcanbeplacedinitscontexteasily.
• P.4o Repositoriesshouldalsoseektogiveteachersmechanismsthat
willallowthemtoprovideprofessionalfeedbackonthevalueandusefulnessoftheresourcesoncetheyhavedeployedthemintheirteaching.
• P.9o Thesimplisticnotionthatqualitycanonlybedeterminedbythe
impactofaresourceonlearningisnotsufficientforrepositories.Ittakeslittleaccountoftherealitiesofateacher’sneedsinseekingoutgoodcontent.
• P.9-10o TheconceptofaQualityCirclemightbeconsciouslyimplemented
inrepositoriesinordertodevelopamodelofcollectiveintelligencerelevantandappropriate,butthereisnoone-size-fits-all.
• Culledfromvarioussectionsofthepaper,thisisalistofpotentiallyrelevantfactorsinarepository’squalityassurancesystem
o P.11§ Efficacy§ Impact§ Availability§ Accuracy§ Excellence
o P.12-13§ Featuredresources§ Userevaluationtools§ Peerreview§ Authorship§ Keywords§ Metadata§ Multilingualsupport§ SocialMediasupport§ CreativeCommonsLicences§ SourceCodeorOriginalFiles
o P.15-16
37
§ OEREvaluationCriteriaClarity,Comprehensibility,andReadability
§ ContentAccuracyandTechnicalAccuracy§ AdaptabilityandModularity§ Appropriateness§ Accessibility§ SupplementaryResources
o P.16§ Whataretheeffectsofopeneducationalresourceson
coursedevelopmentandimplementation?§ Onstudentlearning?§ Whatarethecharacteristicsofcreatorsandearlier
adopters?§ Willstudents’priorexperiencewithOERaffectmotivation
toindependentlylocateanduseotherfreematerialsforlearning?
§ Howwillthisinfluenceachievementandsuccess?o P.19
§ Correspondencewiththekeywordinthesearchbar§ Scoresgivenbyusers(andthenumberofusersthatgive
thescore),§ Timeonline§ Typeofcontributor(teacher,student,companyetc)
o P.24§ Arankingsystemthatatleastletsusersseehowmany
timestheresourcehasbeenopened
38
BIBLIOGRAPHY
1. AbderrahimElMhouti,AzeddineNasseh,MohamedErradi,Howtoevaluatethequalityofdigitallearningresources?InternationalJournalofComputerScienceResearchandApplication2013,Vol.03,Issue.03,pp.27-36
2. Anderson,L.W.,Krathwohl,D.R.:Ataxonomyforlearning,teaching,andassessing:arevisionofBloom’staxonomyofeducationalobjectives.Longman,NewYork(2001)
3. Atenas,J.andHavemann,Leo(2013)Qualityassuranceintheopen:anevaluationofOERrepositories.INNOQUAL-InternationalJournalforInnovationandQualityinLearning1(2),pp.22-34.ISSN2294-9763.
4. Atkins,D.,Brown,J.,&Hammond,A.(2007).AReviewoftheOpenEducationalResources(OER)Movement:Achievements,Challenges,andNewOpportunities.CommonwealthofLearning.(2011).
5. Atkins,D.,Brown,J.,&Hammond,A.(2007).AReviewoftheOpenEducationalResources(OER)Movement:Achievements,Challenges,andNewOpportunities.
6. Barker,E.,&Ryan,B.(2003).TheHigherLevelSkillsforIndustryRepository,1–5.Retrievedfromhttp://metadata.cetis.ac.uk/implementations/usage_survey/cs_hlsi.pdf
7. Baumgartner,PImprovingReusabilityofOEREducationalPatternsforContentSharing,Retrievedfrom:https://www.semanticscholar.org/paper/Improving-Reusability-of-OER-Educational-Patterns-Baumgartner/f9af6537db6face09185264cda711597a5100621
8. Bienkowski,M.,&Klo,J.(2014).TheLearningRegistry:ApplyingSocialMetadataforLearningResourceRecommendations.InRecommenderSystemsforTechnologyEnhancedLearning(pp.77-95).SpringerNewYork.
9. Billon,N.,Nistor,A.,Mihai,G.,Grizelj,A.,Myrtsioti,E.,Miklasinska,O.,Pocze,B.&Gras-Velazquez,A.(2019).ScientixUpdate-September2019.EuropeanSchoolnet,Brusselshttp://files.eun.org/scientix/scx3/publications/Scientix3_Final_Publication_web.pdf
10. Brent,I.(2012).ObstaclestocreatingandfindingOpenEducationalResources:thecaseofresearchmethodsinthesocialsciences.MediainEducation,1–17.Retrievedfromhttp://www-jime.open.ac.uk/jime/article/viewArticle/2012-05/html
11. Broadbent,O&McCann,EThinkUp,Developmentofe-learningresources,ISBN:978-1-909327-14-6©RoyalAcademyofEngineering2016
12. Brodahl,C.&Smestad,B.ATaxonomyasaVehicleforLearning.InterdisciplinaryJournalofE-LearningandLearningObjects,(2009).5,111-127.
13. Brodahl,C.&Smestad,B.TheMathWizProject.Design,Technologies14. Brooks,D.&McCormack,M.(2019)HigherEducation’s2019trendwatch
andtop10strategictechnologies.Researchreport.Louisville,CO:ECAR.Roth,D.(2019).
39
15. Browne,T.,Holding,R.,Howell,A.,&Rodway-Dyer,S.(2010).ThechallengesofOERtoAcademicPractice.JournalofInteractiveMediainEducation,1–15.Retrievedfromhttp://jime.open.ac.uk/jime/article/viewArticle/2010-3/html
16. Butcher,NEditedbyAshaKanwar(COL)andStamenkaUvalic´-Trumbic´ABasicGuidetoOpenEducationalResources(OER),(UNESCO)ISBN978-1-894975-41-4
17. Clements,K.I.,&Pawlowski,J.M.(2012).User-orientedqualityforOER:understandingteachers’viewsonre-use,quality,andtrust.JournalofComputerAssistedLearning,28(1),4–14.doi:10.1111/j.1365-2729.2011.00450.x
18. ConsultingLtd,UK:PaulBacsich,withadditionalresearchfromGilesPepler,SaraFrankAdultEducationandOpenEducationalResourcesEuropeanUnion,2015.Retrievedfrom:http://www.europarl.europa.eu/supporting-analyses
19. CorneliaBrodahl,EvaluationofActiveLearningMaterials-Retrievedfrom:https://home.uia.no/cornelib/hjem/conferences/ALICT/ALICT-%20Evaluation%20of%20Active%20Learning%20Materials-.pdf
20. Davis,H.C.,Carr,L.,Hey,J.M.,Howard,Y.,Millard,D.,Morris,D.,&White,S.(2010).Bootstrappingacultureofsharingtofacilitateopeneducationalresources.LearningTechnologies,IEEETransactionson,3(2),96-109.
21. Davis,H.C.,Carr,L.,Hey,J.M.,Howard,Y.,Millard,D.,Morris,D.,&White,S.(2010).Bootstrappingacultureofsharingtofacilitateopeneducationalresources.LearningTechnologies,IEEETransactionson,3(2),96-109.
22. DelosArcos,B.(2014).FlippingwithOER:K12teachers’viewsoftheimpactofopenpracticesonstudents.InProceedingsofOpenCoursewareConsortiumGlobal2014:OpenEducationforaMulticulturalWorld.
23. DeVries,IEvaluatingopeneducationalresources-Lessonslearned,2ndWorldConferenceonEducationalTechnologyResearches–WCETR2012,ThompsonRiversUniversity,900McGillRd.,KamloopsV2C0C8,Canada,Retrievedfrom:https://www.oerknowledgecloud.org/archive/1-s2.0-S1877042813010422-main.pdf
24. Downes,S.(2007).ModelsforSustainableOpenEducationalResources.InterdisciplinaryJournalofKnowledgeandLearningObjects,3,29-44.
25. Downes,S.(2007).ModelsforSustainableOpenEducationalResources.InterdisciplinaryJournalofKnowledgeandLearningObjects,3.Retrievedfromhttp://www.oecd.org/edu/ceri/36781698.pdf
26. Ehlers,U.-D.(2007).QualityLiteracy-CompetenciesforQualityDevelopmentinEducationandE-Learning.EducationalTechnology&Society,Vol.10,No.2.
27. Farrow,Robert;Pitt,Rebecca;delosArcos,Beatriz;Perryman,Leigh-Anne;Weller,MartinandMcAndrew,Patrick(2015).ImpactofOERuseonteachingandlearning:datafromOERResearchHub(2013–2014).BritishJournalofEducationalTechnology,46(5)pp.972–976
28. Friesen,N.(2010).UsingFreeandOpenOnlineResources.Retrievedfromhttp://smallcities.tru.ca/index.php/cura/article/view/22.
29. Fulantelli,G.,Gentile,M.,Taibi,D.,&Allegra,M.(2008).TheOpenLearningObjectmodeltopromoteOpenEducationalResources.Journal
40
OfInteractiveMediaInEducation,1.RetrievedAugust23,2013,fromhttp://jime.open.ac.uk/article/2008-9/344
30. Greenberg,J.(2009).Theoreticalconsiderationsoflifecyclemodeling:ananalysisofthedryadrepositorydemonstratingautomaticmetadatapropagation,inheritance,andvaluesystemadoption.Cataloging&classificationquarterly,47(3-4),380-402.
31. GuidelinesforOpenEducationalResources(OER)inHigherEducation(p.22).Vancouver.Retrievedfromhttp://www.col.org/resources/publications/Pages/default.aspx
32. Hamdan,N.,McKnight,P.,McKnight,K.&Arfstrom,K.M.(2013).AReviewofFlippedLearning.www.flippedlearning.orgaccessedMarch17,2014.
33. Hilton,J.I.,Wiley,D.,Stein,J.,&Johnson,A.(2010).Thefour“R”sofopennessandALMSanalysis:Frameworksforopeneducationalresources.OpenLearning:TheJournalofOpen,Distanceande-Learning,25(1),37-44.
34. Hoel,T;Pawlowski,JMHowwillthedigitaltextbookof2030solveMeno’sparadox,OPENEDUCATION2030.JRC-IPTSCALLFORVISIONPAPERS.PARTII:SCHOOLEDUCATION
35. Holden,C.(2003).Fromlocalchallengestoaglobalcommunity:learningrepositoriesandthegloballearningrepositoriessummit.TheAcademicADLCoLab,0–30.Retrievedfromhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.279&rep=rep1&type=pdf
36. Hylén,J.(2006).OpenEducationalResources:OpportunitiesandChallenges.Paris.Retrievedfromhttp://oecd.org/edu/ceriMIT.(2011).OurHistory.Retrievedfromocw.mit.edu/about/.OpenCoursewareConsortium.(n.d.).AbouttheOpenCoursewareConsortium.RetrievedNovember15,2011,Retrievedfromhttp://www.ocwconsortium.org/en/aboutus/abouttheocwc.
37. Hylén,J.(2006).Openeducationalresources:Opportunitiesandchallenges.ProceedingsofOpenEducation.Retrievedfromhttp://library.oum.edu.my/oumlib/sites/default/files/file_attachments/odl-resources/386010/oeropportunities.pdf
38. Jacobi,R.,&Woert,N.vander.(2012).Trendreport:OpenEducationalResources2012.SpecialInterestGroupOpenEducationalResources.Retrievedfromhttp://www.surf.nl/en/publicaties/Documents/trendrapport%20ER2012_10042012(ENGELSLR).pdf
39. Jung,I.,Sasaki,T.&Latchem,C.A framework for assessing fitness for purpose in open educational resources in IntJEducTechnolHighEduc(2016)13:3.https://doi.org/10.1186/s41239-016-0002-5
40. K.Clements,J.Pawlowski,N.Manouselis(2014)WhyOpenEducationalResourcesRepositoriesFail–ReviewofQualityAssuranceApproaches,EDULEARN14Proceedings,pp.929-939
41. Kanwar,A.,Uvalic-Trumbic,S.&Butcher,N.(2011)Abasicguidetoopeneducationalresources(OER).UNESCO.Retrievedfromhttp://unesdoc.unesco.org/images/0021/002158/215804e.pdf
42. Krajcso,ZClassificationandqualitycriteriaforOpenEducationalResourcesinthefieldofforeignlanguagelearning,Retrievedfrom:
41
https://www.degruyter.com/downloadpdf/j/jolace.2016.4.issue-1/jolace-2016-0004/jolace-2016-0004.pdf
43. Kroop,S;Mikroyannidis,A;Wolpers,MEditors,ResponsiveOpenLearningEnvironments:OutcomesofResearchfromtheROLEProject,SpringerOpenRetrievedfrom:https://link.springer.com/content/pdf/10.1007%2F978-3-319-02399-1.pdf
44. Kurilovas,E,Bireniene,VandSerikoviene,S.”MethodologyforEvaluatingQualityandReusabilityofLearningObjects”TheElectronicJournalofe-LearningVolume9Issue12011,(pp39-51),availableonlineatwww.ejel.org
45. Lund,T.andHojsholt-Poulsen,L.(2010),“BuildingSuccessfulEducationalRepositories–Consolidatedrecommendationsreport”,EdReNeprojectdeliverable,eContentPlus
46. McGreal,R.(Ed.).(2004).Onlineeducationusinglearningobjects.Routledge.
47. Megalou,E&Kaklamanis,COpenContent,OERRepositories,InteractiveTextbooks,andadigitalSocialPlatform:TheCaseofGreeceRetrievedfrom:https://www.academia.edu/37054610/OPEN_CONTENT_OER_REPOSITORIES_INTERACTIVE_TEXTBOOKS_AND_A_DIGITAL_SOCIAL_PLATFORM_THE_CASE_OF_GREECE
48. Massart,DandShulman,ELearningResourceExchangeMetadataApplicationProfileversion4.7,EuropeanSchoolnet,2011.Retrievedfrom:http://lreforschools.eun.org/c/document_library/get_file?p_l_id=10970&folderId=12073&name=DLFE-1.pdf
49. Megalou,E;Gkamas,V,;Papadimitriou,S;Paraskeva,M&Kaklamanis,COpenEducationalPractices:MotivatingTeacherstoUseandReuseOpenEducationalResoursesRetrievedfrom:https://www.academia.edu/30309880/OPEN_EDUCATIONAL_PRACTICES_MOTIVATING_TEACHERS_TO_USE_AND_REUSE_OPEN_EDUCATIONAL_RESOURCES
50. Megalou,E;Gkamas,V,;Papadimitriou,S;Paraskeva,M&Kaklamanis,COpenEducationalPractices:MotivatingTeacherstoUseandReuseOpenEducationalResoursesRetrievedfrom:https://www.academia.edu/30309880/OPEN_EDUCATIONAL_PRACTICES_MOTIVATING_TEACHERS_TO_USE_AND_REUSE_OPEN_EDUCATIONAL_RESOURCES
51. Morris,LVContemplatingOpenEducationalResources,Publishedonline:1August2019SpringerNatureB.V.2019
52. Nesbit,J.C.,Belfer,K.,&Vargo,J.(2002).AConvergentParticipationModelforEvaluationofLearningObjects.CanadianJournalofLearningandTechnology,(3),105-120.RetrievedOctober15,2008,fromhttp://www.cjlt.ca/index.php/cjlt/article/viewArticle/110/103
53. Neven,F.,&Duval,E.(2002).Reusablelearningobjects:asurveyofLOM-basedrepositories.ProceedingsofthetenthACMinternational,0–3.Retrievedfromhttp://dl.acm.org/citation.cfm?id=641067
54. Ochoa,X.,&Duval,E.(2009).Quantitativeanalysisoflearningobjectrepositories.LearningTechnologies,IEEETransactions,1–14.Retrievedfromhttp://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5184802
42
55. Ochoa,XAutomaticevaluationofmetadataqualityindigitalrepositories,Retrievedfrom:https://link.springer.com/article/10.1007/s00799-009-0054-4
56. OECD.(2007).GivingKnowledgeforFree:TheEmergenceofOpenEducationalResources.OECDPublishing.Retrievedfromhttp://www.oecd.org/education/ceri/givingknowledgeforfreetheemergenceofopeneducationalresources.htm
57. ORO(2014).OpenResearchOnline–TheOpenUniversity.http://oro.open.ac.uk
58. Palmer,C.L.,Teffeau,L.C.,&Newton,M.P.(2008).Strategiesforinstitutionalrepositorydevelopment:acasestudyofthreeevolvinginitiatives.LibraryTrends,57(2),142-167.
59. Pawlowski,J.M.,&Hoel,T.(2012).TowardsaGlobalPolicyforOpenEducationalResources:TheParisOERDeclarationanditsImplicationsWhatareOpenEducationalResources?Retrievedfromhttp://monet.informatik.rwthaachen.de/giotto/OpenScout_df6f1252-bfa6-11e1-a668-e13baff9bd23.pdf
60. Pawlowski,J.M.&Clements,K.I(2013)TrustedEducationalNetworksforInternationalizationofOpenEducationalResources,InternationalJournalforInnovationandQualityinLearning(INNOQUAL),Volume1,Issue1.(2013),ISSN:2294-9763
61. Pawlowski,J.M.(2007).Qualitymarke-learning:developingprocess-andproduct-orientedqualityforlearning,educationandtraining,InternationalJournalofLearningTechnology,Vol.3,Issue1(February2007).
62. Pegler,C.(2012).Herzberg,hygieneandthemotivationtoreuse:Towardsathree-factortheorytoexplainmotivationtoshareanduseOER.JournalofInteractiveMediainEducation,1–18.Retrievedfromhttp://www-jime.open.ac.uk/jime/article/viewArticle/2012-04/html
63. Pérez-Mateo,M.,Maina,M.F.,Guitert,M.,&Romero,M.(2011).Learnergeneratedcontent:Qualitycriteriainonlinecollaborativelearning.EuropeanJournalofOpen,DistanceandELearning.
64. Petrides,L.,&Nguyen,L.(2008).Openeducationalresources:inquiringintoauthoruseandreuse.InternationalJournalofTechnologyEnhancedLearning,1,98–117.Retrievedfromhttp://inderscience.metapress.com/index/9428665670616423.pdf
65. Pirkkalainen,H.,Jokinen,J.&Pawlowski,J.(2014)UnderstandingSocialOEREnvironments–aQuantitativeStudyonFactorsInfluencingtheMotivationtoShareandCollaborate.IEEETransactionsonLearning
66. Quinn,C.(1996).PragmaticEvaluation:LessonsfromUsability,http://www.ascilite.org.au/conferences/adelaide96/papers/18.html
67. RajivS.Jhangiani,RobertBiswas-Diener,OPEN:Thephilosophyandpracticesthatarerevolutionizingeducationandscience,(ISBN:9781911529002)
68. Richter,T.,&Ehlers,U.D.(2010).BarriersandMotivatorsforUsingOpenEducationalResourcesinSchools.OpenED,1–12.Retrievedfromhttp://openaccess.uoc.edu/webapps/o2/handle/10609/4868
43
69. Robertson,R.J.(2010).AreOERsjustRe-usableLearningObjectswithanopenlicense?kavubob'smiscellanea.http://kavubob.wordpress.com/2010/09/01/rlosoersopened/
70. Schuwer,R.,Wilson,T.,Valkenberg,W.Van,&Lane,A.(2010).ProductionofOER:aquestforefficiency.UniversitatObertaDeCatalunya,N.OpenUniversity
71. Skovsmose,0.(2001).Landscapesofinvestigation.ZentralblattfürDidaktikderMathematik,3(4),123-132
72. SquiresD.,&Preece,J.(1996).UsabilityandLearning:EvaluatingthePotentialofEducationalSoftware.Computers&Education,27(1),15-22.
73. Stacey,P.(2012).TheEconomicsofOpen.RetrievedMay15,2012,Retrievedfromhttp://http://edtechfrontier.com/.
74. Stenger,M.10Openeducationalresourcesyoushouldknowabout[Blogpost].(2018).Retrievedfromhttps://www.opencolleges.edu.au/informed/features/10-open-educational-resources-know/
75. Taylor,J.C.(2007).OpenCoursewareFutures�:CreatingaParallelUniverseIntroduction�:FromElitetoMassHigherEducationTheOpenEducationalResources(OER)MovementOpenCourseWareConsortium.October,1-9.
76. TechnologiesUNESCO.(2012).2012ParisOERDeclaration.WorldOERCongress,Paris,June2012.
77. TheNorwegianCentreforICTinEducation.(2012).QualityCriteriaforDigitalResources.Støtteskjemaforvurderingavpedagogiskegnethetogtekniskeogformellekravveddigitalelæringsressurser.Retrievedfromhttps://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser
78. Thomas,C.,&McDonald,R.H.(2007).Measuringandcomparingparticipationpatternsindigitalrepositories.D-libMagazine,13(9/10),1082-9873.
79. Tuomi,I.(2006).OpenEducationalResources:WhattheyareandwhydotheymatterReportpreparedfortheOECD.Retrievedfromhttp://www.meaningprocessing.com/personalPages/tuomi/articles/OpenEducationalResources_OECDreport.pdf
80. U.S.GovernmentAccountabilityOffice,ReporttoCongressionalCommittees,Collegetextbooks(GAO-13-368).(2013).Retrievedfromhttps://www.gao.gov/assets/660/655066.pdf
81. UNESCO(2012).2012ParisOERDeclaration.WorldOERCongress,Paris,June2012.Retrievedfromat:http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CI/CI/pdf/Events/Paris%20OER%20%20Declaration_01.pdf
82. UNESCO.(2002).ForumontheImpactofOpenCoursewareforHigherEducationinDevelopingCountriesFinalreportUNESCO.Design,2002(July),1-3.
83. UNESCO.(2002).UNESCOPromotesNewInitiativeforFreeEducationalResourcesontheInternet.
84. UNESCO.(2011).GuidelinesforOpenEducationalResources(OER)inHigherEducation.CommonwealthofLearning.Retrievedfromhttp://www.col.org/PublicationDocuments/Guidelines_OER_HE.pdf
44
85. Wesner,KEuropeaneducation2030–open&free,integrativeandsustainable–avision.KooperativeBerlin,Retrievedfrom:https://blogs.ec.europa.eu/openeducation2030/files/2013/05/Wesner-OE-SE-2030-fin.pdf
86. White,D.,&Warren,N.(2011).OpenEducationalResources�:TheValueofReuseinHigherEducation(p.29).Oxford,UK.Retrievedfromhttp://www.jisc.ac.uk/media/documents/programmes/elearning/oer/OERTheValueOfReuseInHigherEducation.pdf.
87. Wiley,D.(2007).OntheSustainabilityofOpenEducationalResourceInitiativesinHigherEducation:PapercommissionedbytheOECD’sCentreforEducationalResearchandInnovation(CERI).OECD’sCentreforEducationalResearchandInnovation(CERI)(p.21).OECDPublishing.Retrievedfromhttp://www.oecd.org/dataoecd/33/9/38645447.pdf
88. Wiley,D.(2010).ImpedimentstoLearningObjectReuseandOpennessasaPotentialSolution.REVISTABRASILEIRADEINFORMÁTICANAEDUCAÇÃO,17(03),08.Retrievedfromhttp://ceie-sbc.tempsite.ws/pub/index.php/rbie/article/view/1022/1016
89. Windle,R.J.,Wharrad,H.,McCormick,D.,Laverty,H.,&Taylor,M.G.(2010).SharingandreuseinOER:experiencesgainedfromopenreusablelearningobjectsinhealth.JournalofInteractiveMediainEducation,(01),1–18.Retrievedfromhttp://jime.open.ac.uk/jime/article/viewArticle/2010-4/html
45
DrMarinaConnell JohnConnellDirectors,IAmLearnerhttp://iamlearner.net+441501772409+447930204672