David Ackerman, Associate VP Crystal Butler, Research Associate
Jan 18, 2016
David Ackerman, Associate VPCrystal Butler, Research Associate
1 – MAKE IT COUNT
2 – MAIN METHOD
3 – GETTING RESULTSwhat did we find out?
4 – LESSONS LEARNED...and what does it mean?
AGENDA
SAKAI MARKET SHAREMAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
what determines market share?
how do we get good data?
GETTINGRESULTS
1 – COUNT METHOD
2 – RESPONSE RATE
one institution, one vote: enrollment rates are not reflected
since enrollments vary, sample data can't be generalized
commercial LMS providers won't share client lists; open source registries are poorly managed
THE ISSUES
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
GETTINGRESULTS
3 – ABSENT PROVIDER DATA
SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Counting installations rather than users could present a skewed perspective of the changing LMS landscape.
COURSEMENU
MAKE ITCOUNT
THE COUNT
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, including Community Colleges.
COURSEMENU
MAKE ITCOUNT
THE COUNT
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, with Community Colleges removed.
COURSEMENU
MAKE ITCOUNT
THE COUNT
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Four-year+ schools had an average response rate of 28.4%....making it impossible to generalize about student usage across colleges and universities with widely varying enrollment rates.
COURSEMENU
MAKE ITCOUNT
RESPONDENTS
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Commercial vendors do not share their complete customer lists.
COURSEMENU
MAKE ITCOUNT
LMS PROVIDERS
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Open source providers may try to track system installations...
COURSEMENU
MAKE ITCOUNT
LMS PROVIDERS
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
...but have incomplete control over implementation and client downloads, so cannot provide accurate usage statistics.
COURSEMENU
MAKE ITCOUNT
LMS PROVIDERS
SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
As the end users, learners should be tallied to determine market share. By completion of their fourth year in college, the ratio of students who will have used a school's primary LMS approaches 100%.
COURSEMENU
MAKE ITCOUNT
THE USERS
GETTINGRESULTS SAKAI MARKET SHARE
1 – IDENTIFY INSTITUTIONS
2 – DETERMINE A DATA GATHERING METHOD
select schools and get enrollment numbers
how to count 100% of the student population of interest?
3 – ESTABLISH A PROCESS the plan, pitfalls, and (striving for) perfection
THE STEPS
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Schools were selected using the U.S. Department of Education's Institute of Education Sciences database...
COURSEMENU
MAKE ITCOUNT
IPEDS
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
with the following criteria:• 4-year+ not-
for-profit• degree
granting• based in the
U.S.• 1,000 or more
students• have a physical
campus
COURSEMENU
MAKE ITCOUNT
IPEDS
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
We have a lot of work to do: how to determine the LMS in use by over 1,500 schools?
COURSEMENU
MAKE ITCOUNT
IPEDS RESULTS
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Web search.
With a large data set requiring repetitive search operations and intelligent evaluation of results, we had an ideal use case for Amazon's Mechanical Turk.
COURSEMENU
MAKE ITCOUNT
MECHANICAL TURK
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MTurk works best when the instructions are standardized, so we had to establish a web search that would return optimal results. We tested the term 'learning management system' and related terms suggested by Google and other sources...
COURSEMENU
MAKE ITCOUNT
THE SEARCH
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
...but querying specific LMS names returned the best links.
COURSEMENU
MAKE ITCOUNT
THE SEARCH
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
COURSEMENU
MAKE ITCOUNT
THE SEARCH
GETTINGRESULTS
MAINMETHOD
1 – MTurk TESTING
2 – WORKER QUALIFICATION
• first run: 100 items, 2 Master Classification Workers each (200 HITs)
• second run: 50 items, 3 Master Workers each (150 HITs)• third run: 400 items, 3 Master Workers each (1200 HITs)
• goal: create workforce with 95% accuracy or greater
• reality: standards had to be set at 85% accuracy to generate a large enough worker base
SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
COURSEMENU
MAKE ITCOUNT
GETTINGRESULTS
MAINMETHOD
• use 3 workers per item; verify LMS choice for all worker disagreements
• do 5% audit on all worker agreements• compare final LMS list to provider information found on
web
4 – DATA VERIFICATION
• 3 Custom Qualification Workers per item• remaining institution list split in half and run sequentially in
two sets
3 – DATA COLLECTION
THE SEARCH
SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Mechanical Turk provides analytics for completion velocity, average hourly pay rate, and percent HITs submitted.
COURSEMENU
MAKE ITCOUNT
MTurk TESTING
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Of the 49 workers who attempted our HITs, only 10 performed well enough to meet our qualification standards.
COURSEMENU
MAKE ITCOUNT
WORKER QUALIFICATION
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
Audits consistently found a 95% accuracy rate for worker agreements. • Workers:
.95 x 3621 = 3440• In-house
search:1.0 x 1263 = 1263• Accuracy:
4703/4884 = 96.29%
We are confident that our results are not off more than +/- 5%
COURSEMENU
MAKE ITCOUNT
DATA VERIFICATION
GETTINGRESULTS
MAINMETHOD SAKAI MARKET SHARE
1 – OUR FINDINGS
2 – A COMPARISON
LMS market share based on 2010-2011 1-year headcount
how do our results differ from the CCS count?
3 – THE SPREADSHEET our data by LMS
THE RESULTS
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS SAKAI MARKET SHARE
THE NUMBERS
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS
Based on a one-year enrollment of 13,723,024 students.
SAKAI MARKET SHARE
THE COMPARISON
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS
Campus Computing Survey Count by
Institution
New York University Count by Enrollment
SAKAI MARKET SHARE
ENROLLMENTS
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS SAKAI MARKET SHARE
OUR DATA
MAKE ITCOUNT
MAINMETHOD
LESSONSLEARNED
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS SAKAI MARKET SHARE
1 – GROWTH OF CANVAS
2 – SAKAI
6.2% market share since founding in 2008, with many recent migrations
8.3% market share since deployment in 2005, with few recent migrations and many previous Sakai installations having been replaced by other systems
3 – OPEN SOURCE 26.9% of primary LMSs by enrollment are open source, less than the 33.6% reported by the CCS count; Blackboard is often the primary LMS at schools with smaller parallel or test installations of other systems
CONCLUSIONS
MAKE ITCOUNT
MAINMETHOD
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS
LESSONSLEARNED SAKAI MARKET SHARE
1 – PUBLISH PAPER
2 – FIND HOME FOR SPREADSHEET OF DATA
Include analysis as to statistical significance of comparison survey
This will allow schools to correct any errors as well as report a change in LMS
3 – SHARE DATA
FUTURE WORK
MAKE ITCOUNT
MAINMETHOD
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS
LESSONSLEARNED SAKAI MARKET SHARE
QUESTIONS
MAKE ITCOUNT
MAINMETHOD
MAKE ITCOUNT
MAINMETHOD
GETTINGRESULTS
LESSONSLEARNED SAKAI MARKET SHARE