Ulearning evaluation approach

ULearning Skillnet

Overview of Evaluation Approach

Rob Cosgrove & Kieran SwailMonday 8th September 2008

Shannon Business Park

Role & Approach

• Role = monitor, evaluate and report

• Approach = Listen, observe, engage, advise, recommend and review

Scope of evaluation

• 2 reports Interim early 2009, Final early 2010

• Key data & indicators set by Skillnet– Mandatory, Strongly recommended, Optional– Kirkpatrick scale of evaluating training Level 1-3 & Phillips Level 5 see Guidelines handout.– Innovation & Networking = Opportunity to stand out

Phillips ROI ModelThe Phillips ROI methodology is a 10-step process preferred by Skillnets

We need to select suitable course/modules

Consider Impact Measurement Skillnet Training for this initiative and next round.

Proposed ROI Method

• Identify module(s) to target for measurement• Criteria

– Multiple students doing same module in same company failing that – similar companies closely spaced

– preference for larger companies– preference for modules with concrete skill

deliverables – Interview module lecturer for modules chosen for

ROI measurement.– In depth interviews with reps from each provider– In depth interviews/ liaison with trainers in

modules selected for ROI assessment

Progress to date

• Attended Launch in Shannon• Met with Project Co-ordinator and

members of Steering Group in Dublin• Met with Skillnets Liaison Officer in

Dublin• Evaluators met and agreed joint

approach

Methodology / Outputs to Interim Report• Attend Steering Group Meetings Sep, Oct – Dec 08• f2f Consultations with select no of network member

companies Oct – Dec 08• Review Lead Generation Process & General Marketing

Activity • Identify course/module (s) for ROI assessment• Students/Trainees

– Pre Module Start• Subset 10-15 minute telephone interviews• eSurvey or integrate with AUA pre- commencement data

capture– Interview Student Cohort Nov 08– Review End Module feedback sheets

• Draft Report Dec 08 & Interim Report Feb 09

Added Value & Support

• Wiki• Spot Evaluation of an event - pre & post

survey – process and content review Nov/Dec 08?

• Ideas– Calendar of Events Network Activities &

promotional activities– Online Forum?– Analysis of sales funnel – lead generation to sales

conversion

Risk Area Severity Probability Mitigation

ROI may not be easily measurable via Phillips.Degree level ROI less tangibleROI slower to accrueSofter skills harder to measure credible ROI.

Evaluation Moderate High Select modules for ROI assessment with care.

Training days targets may not be hit.

Marketing High High

Network may seem supply led rather than demand led – a sales channel for universities.

Administrative ?? Moderate Ensure network members are the key driver of course content –not providers.

Regional Focus on the west misses key markets – Dublin - is less effective

Marketing High Moderate Retarget marketing activity to Greater Dublin Area

Marketing focus unstructured – not leading to closed sales.

Marketing High Moderate Monitor & Evaluate marketing/sales process

Q How do you attract new network members and trainees?

Trainees• Telephone – cold v warm / • Text Mkting -Marketeers call the

3rd screen• Direct Mail – postcard mkting• Competitions• Demos - “wow” factor• PR / Word of Mouth• Testimonials / Referrals• Web Marketing – Web 2.0

– email, blog, podcasts, social networking, Flickr YouTube, Facebook, Bebo, MySpace RSS feeds..

• Advertising– Radio, newspapers,

supplements, journals. Local business circulars, Magazines, Billboards, Google Adwords,

Network Members•Network - Regular themed events•Branding & Merchandising•Trade shows, Exhibitions•Discussion Forums HR/CPD

www.linkedin.com•Conferences•Market Research•Joint Venture e.g. IRCSET - the Irish Research Council for Science, Engineering and Technology•Journalists Visits – education, science & technology •Sponsorship

Apply W5Who?, What?, When?, Where?, Why?

ULearning Skillnet

ThankyouRob Cosgrove & Kieran SwailMonday 8th September 2008