PSE RINGORANG NERC CIP PROJECT FINAL REPORT · PDF fileFinal Report - PSE Ringorang NERC CIP Tournaments_v0 4_draft-hy 20110830 (2) Page 8 of 72CONTACT: [email protected]

FARALLON RISK GROUP LLC P.O. Box 1672, BELLEVUE, WA 98005 © Copyright FARALLON RISK GROUP Final Report - PSE Ringorang NERC CIP Tournaments_v0 4_draft-hy 20110830 (2) Page 1 of 72CONTACT: [email protected]

PSE RINGORANG™ NERC CIP PROJECT FINAL REPORT

Reviewed

Prepared for

PSE IT

Purpose

Analysis of Ringorang™ as an Effective Learning Retention Tool

Prepared by

Cary E. Sholer

Version 0.4

mailto:[email protected]


Revision History

Revision Date Revision Number

Revision Author Description

August 15, 2011 0.1 C. Sholer First draft of the final report of the PSE Ringorang™ Enterprise project.

August 23, 2011 0.2 C. Sholer Validated data and updated report tables, graphs, observations, and findings.

August 26, 2011 0.3 C. Sholer Rewrite of sections 2 and 3 to include additional findings. Full QA review by Vergence and Farallon.

August 29, 2011 0/4 H. Liu and J. Keen Added section 4 about individual and team competitions; and, added section 5 about baseline and final survey analysis.

Trademarks and Copyrights

Farallon Risk Group™ and the Farallon Risk Group “double-walled” logo are filed with the US Patent and

Trademark Office as trademarks owned by Vergence Entertainment LLC. This document is copyrighted

by Farallon Risk Group LLC (Farallon) and all rights to its content are reserved.

Ringorang™ and the Vergence Entertainment “V” logo are filed with the US Patent and Trademark Office

as trademarks owned by Vergence Entertainment LLC. Dallions® is a registered trademark issued by the

USPTO to Vergence Entertainment LLC.

Acknowledgement and Appreciation

Farallon Risk Group would like to acknowledge and appreciate Marc Fineman, the business owner at PSE

for the Ringorang project, for his outstanding stewardship of the Ringorang™ Enterprise Project and the

NERC CIP Learning Tournaments. Rudiger Wolf, the CIO at PSE is to be commended for setting the bar

ever-higher in our executive review sessions and pushing our team to deliver measurable and valuable

results. Robert Feeney, the CEO and Founder of Vergence Entertainment is to be acknowledges and

appreciated for his personal involvement to support development of custom features that made the

Ringorang™ more fun and more effective. Jay Dysart from JLOOP and his team of developers must be

commended for their fantastic design creativity skills and their herculean development efforts to finish

custom features with great quality. Josh Keen, from Vergence Entertainment must be commended for

his fantastic team coaching and change management skills, which enable the Ringorang Team Captains

to excel in their role. Haiyan Liu from Vergence Entertainment must be acknowledged for her tireless

efforts to manage the application, user accounts, curriculum learning elements and relentless analysis of

the data.



Document Approval Sign-off Sheet

Document Name PSE RINGORANG NERC CIP PROJECT FINAL REPORT

Project Name RINGORANG ENTERPRISE -PHASE 1

Project Phase Phase 1 Deliverable Final Report

Version Number 0.3 Vendor Name Farallon Risk Group

Description of Document

The Final Report prepared in support of analyzing the results of the Ringorang™ Enterprise – Phase 1 project contains data NERC CIP learning tournament results, data analyses, observations, learning retention findings and future learning recommendations.

Reviewer’s Names

______________________ Approver’s Name __________________________

Reviewer’s Initials

______________________ Approver’s Signature

__________________________

Submission Date ______________________ Approval Date __________________________



Table of Contents

Revision History .............................................................................................................................. 2

Trademarks and Copyrights ............................................................................................................ 2

Acknowledgement and Appreciation ............................................................................................. 2

Document Approval Sign-off Sheet ................................................................................................ 3

Table of Contents ............................................................................................................................ 4

List of Tables.................................................................................................................................... 5

List of Figures .................................................................................................................................. 6

Glossary of Terms and Acronyms ................................................................................................... 8

Executive Summary......................................................................................................................... 9

Ringorang™ Learning Retention Data Analysis Findings ............................................................... 10

1.0 Description of Final Report ................................................................................................... 12

1.1 Description of Ringorang™................................................................................................. 13

1.2 Description of Ringorang™ NERC CIP Learning Tournaments................................................ 15

1.3 Organizational Change Management.................................................................................. 17

1.4 Method of Analysis ........................................................................................................... 18

2.0 Is Ringorang™ an Effective Learning Retention Tool? .......................................................... 19

2.1 Does Ringorang™ Impact Learning Retention? ..................................................................... 19

2.1.1 Tournament #1 Performance ..................................................................................................................20 2.1.2 Hypotheses on Learning Performance Differences .............................................................................22

2.1.3 All Tournaments Performance ................................................................................................................23

2.1.4 Summary Performance Data....................................................................................................................26

2.1.5 Tournament Themes .................................................................................................................................28 2.1.6 Leadership, Communications and Competition ....................................................................................30

2.2 Can We Measure Learning Retention by Team and Team Member, and Provide Learning

Diagnostic by Team? .......................................................................................................... 34

2.2.1 Learning Retention Performance by Team Member ...........................................................................35

2.2.2 Learning Retention Performance by Team ...........................................................................................39

2.3 Can We Measure Learning Retention by Department Manager and Team Member, and Provide

Learning Diagnostic by Department? .................................................................................. 41

2.3.1 Learning Retention Performance by Department Manager ..............................................................41

2.3.2 Learning Retention Performance by Department Team Members ..................................................43 2.3.3 Which Leader Impacts Learning Retention More? ..............................................................................48

2.5 Do Participants Perform Better with, or without, Prior Training? .......................................... 51

3.0 Is Ringorang™ an Effective Curriculum Diagnostic Tool....................................................... 53

3.1 Can we create measures and provide quality improvement reports on learning curriculum? .. 53



3.2 Can Ringorang™ Track Learning Retention Results by CLEs and Offer Curriculum Quality

Improvement Suggestions? ................................................................................................ 54

4. How Does Competition Impact Learning Retention Results? ................................................. 57

4.1 Individual Competition ....................................................................................................... 57

4.1.1 Individuals - How Did Learning Retention Change Over a Span of Three Tournaments? ............57

4.1.2 Tournament 1 – Individual Competition Results, Observations and Findings ................................57 4.1.3 Tournament 2 – Individual Competition Results, Observations and Findings ................................59

4.1.4 Tournament 3 – Individual Competition Results, Observations and Findings ................................60

4.2 Team Competition ............................................................................................................. 61

4.2.1 How Does Learning Retention Change Over a Span of Three Tournaments? ...... Error! Bookmark

not defined.

4.2.2 Tournament 1 – Team Competition Results, Observations and Findings .......................................61

4.2.3 Tournament 2 – Team Competition Results, Observations and Findings .......................................62 4.2.4 Tournament 3 – Team Competition Results, Observations and Findings .......................................63

5. Survey Feedback...................................................................................................................... 65

5.1 Description of the Survey Process ....................................................................................... 65

5.1.1 What were the objectives of the surveys?............................................................................................65

5.1.2 What was the method we used to survey participants? ....................................................................65 5.1.3 Why did we have two surveys: baseline and final surveys? ..............................................................65

5.1.4 What Was the Distribution of Participants Completing the Surveys?..............................................65

5.2 Baseline Survey Feedback – Sent After Tournament 1 .......................................................... 65

5.2.1 Baseline Survey - Best Response.............................................................................................................65

5.2.2 Baseline Survey - Worst Response........................................................ Error! Bookmark not defined.

5.2.3 Baseline Survey - Most Undecided or No Opinion Responses......... Error! Bookmark not defined. 5.2.4 Baseline Survey - Subjective Analysis from Survey Feedback.......... Error! Bookmark not defined.

5.3 Final Survey Feedback – Sent After Tournament 3 ............................................................... 67

5.3.1 Final Survey - Best Response ...................................................................................................................67 5.3.2 Final Survey - Worst Response .............................................................. Error! Bookmark not defined.

5.3.3 Final Survey - Most Undecided or No Opinion Responses ............... Error! Bookmark not defined.

5.3.4 Final Survey - Subjective Analysis from Survey Feedback ................ Error! Bookmark not defined.

List of Tables

Table 1 - Glossary of Terms and Acronyms............................................................................................8

Table 2 - NERC CIP Learning Tournament Summary of Results ...............................................................9

Table 3 - Tournament 1 Performance - Points ..................................................................................... 20

Table 4 - Tournament 1 Performance - Correct Answers ...................................................................... 20



Table 5 - Tournament 1 Performance - % Correct Answers................................................................... 21

Table 6 - Tournament 1 Performance – Questions Played .................................................................... 21

Table 7 - NERC CIP Tournaments Performance - Points ........................................................................ 23

Table 8- NERC CIP Tournaments Performance – Correct Answers ......................................................... 24

Table 9 - NERC CIP Tournaments Performance - % Correct Answers ..................................................... 24

Table 10- NERC CIP Tournaments Performance – Questions Played ...................................................... 25

Table 11 - Summary Tournament Performance ................................................................................... 31

Table 12 – Tournament Summary Change Percentages ....................................................................... 33

Table 13 - Legend of Learning Retention Performance......................................................................... 34

Table 14 - Learning Retention Performance by Team Member ............................................................. 35

Table 15 – Top Performance by Team Member by Correct Answers and Dallions®................................. 39

Table 16 - Learning Retention Performance by Teams ......................................................................... 40

Table 17 – Learning Retention Performance by Team Members ........................................................... 41

Table 18 - Department Managers’ Learning Retention Performance Report ......................................... 42

Table 19 - Department Detailed Learning Retention Performance Report ............................................. 43

Table 20 - Highest Retention Performance Sores by Department Team Member ................................... 46

Table 21 - Lowest Retention Performance Sores by Department Team Member .................................... 48

Table 22 - Learning Retention Performance by Teams ......................................................................... 48

Table 23 - Department Managers’ Learning Retention Performance Report ......................................... 49

Table 24 - Curriculum Learning Elements (CLEs) by Topics and Subtopics .............................................. 53

Table 25 – Measuring Responses to CLE Questions.............................................................................. 54

Table 26 - Question Quality Report .................................................................................................... 54

Table 27 - Diagnostic Report on Learning Retention by Topic ............................................................... 54

Table 28 - Diagnostic Report on Learning Retention Results by Subtopic............................................... 55

Table 29 - Ringorang NERC CIP Survey Distribution ............................................................................. 65

List of Figures

Figure 1 - Ringorang™ NERC CIP Tournaments Summary Results ......................................................... 27

Figure 2 - Daily Participation Curve for NERC CIP Tournaments ............................................................ 29



Figure 3- Team Performance by Tournament - Points .......................................................................... 50

Figure 4 - Team Ranking by Tournament ............................................................................................ 51

Figure 5 - NERC CIP Compliance Training vs No Training Distribution.................................................... 53



Glossary of Terms and Acronyms Table 1 - Glossary of Terms and Acronyms

Term or Acronym Definition or Description

CLE Curriculum Learning Element CLEQ Curriculum Learning Element Question

Curriculum

Learning Element

The most discrete learning retention element that can be measured.

Curriculum Learning Element Question

A learning element question is a learning element expressed as a question. For the purposes of this report, a learning element question is unique question asked in one or more of the Ringorang™ learning tournaments.

Impulsive Training The group of users who completed NERC CIP training between the time of the first NERC CIP tournament communication and before the end of the week immediately

following the completion of Tournament #3.

Know (verb) to perceive directly : have direct cognition of

Knowing (noun) Acquaintance with or exposure to

Knowledge (noun) (a) : the fact or condition of knowing something with familiarity gained through experience or association, or (b) : the fact or the act or experience of one that learns

Knowledge Retention

Knowledge Retention is the sum value of knowledge units retained by an individual or group of individuals which can be measured by recall or recognition. For the

purposes of this report, learning retention is the sum number of correct answers recalled or recognized by an individual or group of individuals.

Knowledge Unit

A knowledge unit is the most discrete unit of measure to track and measure knowledge. For the purposes of this report, a Knowledge Unit will be equal to knowing the correct answer to a learning element question.

Learn (verb) (a) : to gain knowledge or understanding of or skil l in by study, instruction, or

experience (b) : memorize

Learning (noun) (a) : the act or experience of one that learns (b) : knowledge or skil l acquired by instruction or study (c) : modification of a behavioral tendency by experience (as exposure to

conditioning) “of having information or of being learned

Learning Element

A learning element is the most discrete unit of measure to track and measure learning. For the purposes of this report, a Learning Element will be fact of knowledge, ability to identify the correct information related to a well -defined task, or behavioral attitude expressed as a preferred acti on while participating in an

event.

No Training The group of users who have no known history of completing NERC CIP training to-date.

Prior Compliance Training

The group of users who completed NERC CIP training prior to the first NERC CIP tournament communication.

Retain to keep in mind or memory

Retention (noun) an ability to retain things in mind; specifically : a preservation of the aftereffects of experience and learning that makes recall or recognition possible



Executive Summary

The purpose of this report is to answer several key questions about the Ringorang™ NERC CIP Learning

Tournaments hosted on the Vergence ASK (Attitude Skills Knowledge) platform. Specifically, we

coordinated three NERC CIP Learning Tournaments at Puget Sound Energy (PSE) with the stated goal of

determining if the Ringorang™ application is an effective learning retention tool.

A summary of our analysis on the Ringorang™ NERC CIP Learning Tournaments is listed below:

Ringorang™ NERC CIP Learning Tournaments are an effective means to motivate the PSE-IT

workforce to interact with NERC CIP curriculum.

Ringorang™ makes education fun by using team play, real-time feedback, quick and easy to

learn curriculum learning elements, awards and prizes.

Tournaments, teams, team captains, community of players, competition, and awards

motivate participants to engage intently with Ringorang™ learning curriculum.

Teams and team captains inspire workforce members to participate.

Augmenting team competition with awards and prizes stimulates greater intensity among

users to engage with the learning curriculum.

Parsing curriculum into learning elements and intelligently scheduling the delivery of

curriculum learning elements (CLEs) enabled users to learn and retain the NERC CIP

compliance curriculum quickly.

Tagging CLEs by NERC CIP topics and subtopics, and scoring CLEs by retention importance

and level of difficulty makes it possible for us to: schedule, deliver, monitor, measure and

improve the quality of NERC CIP CLEs.

Summary results of the Ringorang™ NERC CIP Learning Tournaments are contained in the table below.

Table 2 - NERC CIP Learning Tournament Summary of Results

Summary Performance Tournament 1 Tournament 2 Tournament 3

Days of Play

9

4

5

Invited to Participate

203

205

213

Number of Active Participants

151

111

133

Questions Played

10,530

4,737

7,570

Correct Answers

7,337

3,394

5,653

Dallions™ 1,034,259

517,819

994,308

Average Dallions per Active Participant per Day of Play

761

1,166

1,495




Average Questions Played per Active Participant per Day of Play

7.7

10.7

11.4

Average Correct Answers per Active Participant per Day of Play

5.4

7.6

8.5

% Participants Active in Play 74% 54% 62%

% Correct Answers per Tournament 70% 72% 75%

PSE-IT workforce members were invited to participate in the Ringorang™ NERC CIP Learning

Tournaments.

o Of the 203 PSE participants invited to play in Tournament 1, 151 chose to play,

resulting in 74% of the people invited participating in this tournament.



(Tournament 2 was hosted on a 4-day holiday week.)



56 PSE-IT workforce members impulsively completed NERC CIP Compliance training to

better prepare themselves for the Ringorang™ NERC CIP Learning Tournaments.

In Tournament 3, 15 Teams vied for the desirable “Captains Trophy”, and seven teams

competed intensely all the way to the last day, hoping to seize one of the “Top 5 Team”

awards.

Of the Top 20 Winners in the final tournament, Tournament 3, six users completed NEC CIP

Compliance Training a few months in advance of the Ringorang™ tournaments, 9 completed

NERC CIP Compliance Training impulsively just before a Ringorang™ tournament, and 5

completed no NERC CIP Compliance Training whatsoever.

Ringorang™ Learning Retention Data Analysis Findings

We analyzed tournament data for any significant findings worthy of reporting to project sponsor, PSE,

and to the vendor, Vergence Entertainment. Our list of findings and the page number in which the

findings can be found are listed in the reference table below.

Finding 1 – Ringorang™ as deployed at PSE by Farallon Risk Group and Vergence Entertainment for the

NERC CIP Learning Tournaments appears to be an effective learning retention tool as compared to

existing NERC CIP Training methods. .................................................................................................. 22



Finding 2 - Employees who complete compliance training in advance of participating in Ringorang™

NERC CIP Learning Tournaments may or may not have a significant performance advantage; and,

Farallon Risk Group meticulously analyzed data to determine the answer to this finding.. ..................... 23

Finding 3 - Ringorang™ NERC CIP Learning Tournaments as deployed at PSE by Farallon Risk Group and

Vergence Entertainment appear to be a highly effective learning method with retention levels equivalent

to other proven training methods. ..................................................................................................... 26

Finding 4 - Ringorang™ NERC CIP Learning Tournaments as managed at PSE motivate participants to

impulsively complete NERC CIP Compliance Training out-of-cycle to either prepare for tournament play;

or due to their newly gained confidence and mastery of CLEs, they believe they will pass the NERC CIP

Compliance exam with ease. ............................................................................................................. 26

Finding 5 – The deployment methods used by Farallon Risk Group and Vergence Entertainment at PSE,

generated a direct and positive correlation between the frequency participants are exposed to NERC CIP

CLEs in Ringorang™ NERC CIP Learning Tournaments and the participants’ learning retention level

(Correct Answers) and their learning retention rate (% Correct) Answers. ............................................ 28

Finding 6 – Themes chosen by Farallon Risk Group and approved by PSE IT for the Ringorang™ NERC CIP

Learning Tournaments made a difference in daily participation numbers, and themes that included

competition yielded better participation and commitment. ................................................................. 30

Finding 7 – The level of leadership involvement at PSE in creating, sending and reinforcing tournament

communications appears to make a significant difference in the Ringorang™ NERC CIP Learning

Tournaments as measured by Questions Played. ................................................................................ 34

Finding 8 - Competition as applied by Farallon Risk Group and Vergence Entertainment does appear to

influence the intensity of play in Ringorang™ learning tournaments as measured by the increase in

DALLIONS® earned by participants..................................................................................................... 34

Finding 9 – Learning retention scores appear to influenced by frequency of exposure to the curriculum

learning elements (CLEs) and the number of tournaments played by each participant in the Ringorang™

Learning Tournaments as measured by the number of Correct Answers per Participant per Day of Play

and % Correct Answers per tournament. ............................................................................................ 34

Finding 10 - As deployed at PSE by Farallon Risk Group and Vergence Entertainment, Ringorang™ is a

highly effective learning retention tool as demonstrated by an increase of 57% in Correct Answers and 96%

in DALLIONS® per Participant per Day of Play. .................................................................................... 34

Finding 11 - Learning Commitment and Learning Retention can be measure in Ringorang Learning

Tournaments and measured by team member using % Correct and % Questions Played. ...................... 39

Finding 12 - Using the reports available from Ringorang™ NERC CIP Learning Tournaments, we are able

to provide learning retention diagnostics by team and provide learning retention recommendations by

team members. ................................................................................................................................ 41



Finding 13 - Using the reports available from Ringorang™ NERC CIP Learning Tournaments, we are able

to provide learning retention diagnostics by department managers and provide learning retention

recommendations by team members. ................................................................................................ 48

Finding 14 – At PSE and based on the method of deployment by Farallon Risk Group and the coaching

provided by Vergence Entertainment, Team Captains have a greater influence on learning retention

performance in NERC CIP Learning Tournaments than do Department Managers. ................................ 50

Finding 15 - Farallon Risk Group and Vergence Entertainment team members optimized learning

retention results by using competitive themes, learning teams, and Ringorang Team Captains.............. 51

Finding 16 - Data from the NERC CIP Learning Tournaments, managed by Farallon Risk Group and

Vergence Entertainment, suggests completion of compliance training is not a prerequisite to perform

well in Ringorang™ Learning Tournaments. ........................................................................................ 53

Finding 18 – Using Ringorang™ Learning Retention reports, Farallon Risk Group and Vergence

Entertainment were able to provide meaningful analytics by curriculum topic and subtopics, and we

could identify learning gaps, and offer recommendations to fill learning gaps. ..................................... 57

Finding 19 - Data from Ringorang™ NERC CIP Learning Tournaments, based on the CLEs written by

Vergence Entertainment and deployed by Farallon Risk Group, suggests compliance awareness posters

have a poor to bad learning retention rate. ........................................................................................ 57

1.0 Description of Final Report

The purpose of this report is to answer several key questions about the Ringorang™ NERC CIP Learning

Tournaments hosted on the Vergence ASK (Attitude Skills Knowledge) platform. Specifically, we

coordinated three NERC CIP Learning Tournaments at Puget Sound Energy (PSE) with the stated goal of

determining if the Ringorang™ application is an effective learning retention tool.

To measure whether Ringorang™ is an effective learning retention tool, we designed three tests. These

tests were delivered in the form of learning tournaments. The participants of the test/learning

tournament were employees and contractors working in the Information Technology group at PSE. The

education curriculum was knowledge and procedures related to NERC CIP. To meet federal

requirements as an electric utility company, PSE must certify that their workforce understand federal

standards to protect the integrity, security and availability of the IT infrastructure and data that provides

for operations of the networks connected to the regional power grids. The learning objective of our

project was to train PSE IT workforce on NERC CIP requirements and PSE’s information security best

practices.

We hosted three tournaments, each with a different theme and a set of varying organizational change

management attributes. By varying the theme and organizational change management attributes, we

expected to find the attributes that drive enrollment, participation, focus, learning and retention.



The learning content was broken into learning elements. Each learning element was tagged by its topic

and subtopic. All learning elements were scored on their difficulty of learning. The level of importance

was also noted for all learning elements. Our intent was to measure learning at the most discrete

manner possible so that we could provide and optimize the education of the participants in subsequent

tournaments and provide future guidance on where additional training would benefit the PSE IT

organization.

This report contains summary information, data analyses, observations and conclusions or findings from

the three NERC CIP learning tournaments hosted at PSE. Where data analysis provides defensible

evidence, we provide conclusions; and, where data analysis is inconclusive or the data is insufficient to

draw a conclusion we provide findings worthy of further consideration in the design of future

deployments of Ringorang™ learning tournaments.

1.1 Description of Ringorang™

Ringorang™ is an interactive software product that increases learning retention and makes

learning fun and compelling. Ringorang™ includes hosted, server-based software with multiple

client applications including web-browser, desktop and mobile applications. Ringorang™ is a

product of the Vergence ASK system. A.S.K. stands for Atti tudes, Skills and Knowledge;

Vergence’s ASK platform is predicated on influencing those three factors in human

development through interactive micro-engagements centered around asking questions and

compelling the recipient to cognitively respond. Those micro-contributions, when repeated,

amount to learning and retention in the aggregate. Most importantly, Vergence delivers

Ringorang™ using definitive forms of entertainment, play and rewards to motivate persistence

and make the process of learning enjoyable.

Participants are invited to activate their Ringorang™ user accounts prior to learning

tournaments and again between tournaments. As active Ringorang™ users, they have access to

play in Ringorang™ tournaments where content such as multiple choice questions are delivered

at various times, offering time-delimited engagements with Curriculum Learning Elements

(CLE’s). To incentivize participation in learning tournaments, points are distributed for each

question played. These points tally to reflect total points earned during tournament play. And

participants are able to view their points subsequent to answering questions. The top scoring

individuals and the top scoring teams are listed on Ringorang™ leaderboards, which are always

viewable on the Ringorang™ game website.

Curriculum learning elements are optimized in Ringorang™ game format and uploaded prior to

each tournament. Users can play the Ringorang™ game in two formats, normal play and

lightning round. Normal mode of play allows users to respond to questions arriving at varying

frequencies throughout the day, one question per hour. Each question takes 20 seconds of time

to play. Lightning round mode of play allows participants to interact with questions in quick

succession, answering a string of questions, back-to-back.



When a participant logs into the Ringorang™ game application, they are referred to as “users”.

When a user logs in, the application identifies the user and connects them using a data

connectivity link to the user’s pre-loaded profile screen. The Ringorang™ application displays the

user’s current tally of points, correct answers and percent correct. Team scores are also

displayed on Ringorang™ website. Leaderboards show comparisons regarding points, number of

questions answered correctly, and percent participation.

When users sign into Ringorang™ game, they received system wide notices of upcoming NERC

CIP Ringorang™ tournaments, reminders to play, encouragement and game tips. When users

played in the PSE NERC CIP Tournaments, users engaged with Ringorang™ NERC CIP questions,

which were optimized in short text format. Each Ringorang™ question was mapped to one or

more NERC CIP topic and subtopic. Ringorang™ questions were identified with a level of

importance rating, level of difficulty rating, and a unique identifying number.

Users earn points (DALLIONS®) based on level of participation (i.e., how many questions did the

user answer) and correct responses to questions when playing the Ringorang™ game (i.e., how

many questions did the user get right or wrong). DALLIONS® are aggregated by user. Users can

earn rewards by earning sufficient points to redeem for varying values of rewards.

DALLIONS® are awarded in both positive amounts and negative amounts. DALLIONS® are

calculated based on speed and accuracy. If the participant selects the “correct” answer within

the threshold set for responding to a question, the participant earns positive points. If the

participant selects an “incorrect” answer receive negative points; and, if the partici pant fails to

submit an answer within the allowed time, the participant receives no points. The speed at

which the participant submits answers impacts the amount of points earned or lost. Hence,

participants quickly learn guessing the answer does not pay. If participants do not submit

answers quickly enough within the response time thresholds, they are not penalized. No points

are awarded or subtracted for a non-response to a Ringorang™ question.

In the PSE NERC CIP tournaments, users played for themselves as individuals, and they played as

a member of a team. Teams were formed by random selection of team members, not by

departments. We desired to remove department biases from the tournament play. Users also

had the opportunity to serve as team captains, a role created to help drive participation and

persistence with team members. The objective of team play was to create an environment

where users think beyond themselves, and “find the time” to play.

The Ringorang™ NERC CIP game was designed to meet the following objectives:

be intuitive and easy to play;

design and create NERC CIP curriculum learning elements, optimized in Ringorang™ game

format, to support the NERC CIP compliance objectives;

questions must be easy to understand with quick response answer format;



host three NERC CIP learning tournaments,

display leaderboards, showing percent of participation, number of correct answers, and

points (DALLIONS®);

leaderboards displayed rankings by individual and team participation, percent of correct

answers, and points; and

all participants’ interactions with the Ringorang™ NERC CIP game get logged into the

Ringorang™ Database.

Ringorang™ provided several standard online reports, including:

Question Quality Report,

Daily Participation Report,

Non Participation Report, and

Individual and Group Performance Reports.

Using the Ringorang™ online “Question Quality Report”, the project team was able analyze the

results of N-ET learning tournaments, paying closest attention to NERC CIP learning elements

with the lowest learning retention scores. Then we identify the questions that required rework

based on weakness in the question or its set of answers. For questions we found to be well

written with a good set of answers, yet had the question had low retention scores, we left the

question alone and decided to play these questions more frequently.

With the “Daily Participation Report”, we identified the most loyal participants, and then follow-

up with them to gather feedback on what we did right to make their engagement a positive

experience.

Using the Non Participation Report, we identified registered NERC CIP participants with active

Ringorang™ user accounts who had not yet choose to participate in the current NERC CIP

learning tournament. Then we discreetly communicated with these individuals through their

managers and team captains.

The Individual and Group Performance reports show us how many questions were played,

answered correctly, answered incorrectly, or not answered in time.

1.2 Description of Ringorang™ NERC CIP Learning Tournaments

At PSE, we used Ringorang™ NERC CIP Learning Tournaments to:

augment existing NERC CIP compliance training;

provide an interactive game to increase learning retention through comparative play;



reinforce learning retention by exposing users to NERC CIP curriculum learning elements;

and

improve skills on information security best practices for critical cyber assets;

Ringorang™ NERC CIP tournaments delivers questions at random times each hour. We

configured the PSE NERC CIP Learning Tournaments to run 16 hours, between the hours of 6 AM

and 10 PM to participants. Participants were allowed to interact with up to 16 questions per day.

For participants who prefer to use normal mode of play, they receive one question an hour

during their work day, which means they could potentially receive up to 16 questions per day,

but most only interacted with Ringorang during their work hours, so they interacted with just 8

questions per day. For participants who preferred to play in lightning mode of play, they logged

in and responded to questions not played within the prior 24 hours, which means they were

presented 16 questions in succession, taking less than 5 minutes to play all question. Both the

hours of play for normal mode and the number of hours of replay in Lightning Round are

software settings that can be manually adjusted. In Tournament 3, we set the Lightning Round

hours of replay the last 48 hours of questions, giving participants a chance to interact with up to

32 questions per cycle.

Many participants chose to log into and play lightning round first to catch up on any question

they missed during their off hours, and then they would switch to normal play for the remained

of their work day to maximize their points. And a few would log in during off hours to maximize

their participation scores.

Rewards were offered at the conclusion of each NERC CIP tournament, and these rewards were

used to incentivize participants to enter tournaments and maintain active engagement through

the end of each tournament.

Ringorang™ NERC CIP Learning Tournaments were hosted three times between May 2, 2011 and

June 24, 2011. Each NERC CIP tournament lasted a different number of days, depending on the

design of that tournament. Tournament 1 was an eight-day tournament and it spanned one

weekend, so some users played on the weekend, which enabled them to play 10 days.

Tournament 2 was hosted on the Memorial Day holiday week, and therefore Tournament 2 was

a 4-day tournament, hosted on Tuesday through Friday. Tournament 3 was played on a regular

work week for 5 days, hosted from Monday through Friday.

Participants were chosen by department manages based on who would benefit from being

exposed to more NERC CIP training, and who would likely support testing a new application.

Two thirds of the PSE IT organization was targeted to play. Each participant received more than

one invite to enroll on the Ringorang™ website, so that they could play in the Ringorang™ NERC

CIP Learning Tournaments. Enrollment invites were sent by the Ringorang™ game application

itself. Tournament communications were sent to users to encourage them to participate in

upcoming NERC CIP learning tournaments. We varied the source of tourname nt participation

encouragement communications to test which source would generate the highest level of



participation. The communication sources we tested were: Ringorang™ project manager,

Business Owner, Team Captains, Department Managers and the PSE CIO. The Ringorang™

mobile application was not used at PSE. So field employees who played in the NERC CIP

tournament had to wait until they were back at their PSE office location to play, or they logged

in from home after-work to play.

NERC CIP Curriculum Learning Elements were designed and tagged to:

educate participants on the detailed NERC CIP requirements;

evaluate learning retention of NERC CIP training curriculum by topic and subtopic;

tag curriculum learning elements by level of difficulty and organize the delivery of content

at increasing levels of difficulty in each tournament ;

load a few difficult questions every day to make the NERC CIP curriculum challenging even

to those who had already completed their annual NERC CIP compliance training

requirement;

tag questions by their importance and then compile game content with repeating questions;

schedule repeating questions in different patterns to gauge if retention levels are influenced

by repetition and if so, which repeating pattern is most effective;

one fun questions was added to each day’s set of questions to make the tournament

interesting, and

to monitor and track the quality of each question so that we could refine its content or its

set of potential answers to increase the retention of the correct question.

1.3 Organizational Change Management

We anticipated that organizational resistance to change, resistance to software applications,

and resistance to new learning tools would bias the results of our test to prove or disprove

Ringorang™ as an effective learning retention tool.

To minimize the impact of resistance to change, new software applications, and new learning

tools, we attempted to make the users’ experience with Ringorang, the NERC CIP learning

tournaments, and their interactions with the learning retention tool “fun”.

To overcome organization resistance to change, we used varying degrees of leadership

communications, starting at a low-level in Tournament 1, mid-level in Tournament 2, and senior-

level in Tournament 3. By varying the degrees of leadership sending the tournament

encouragement communications, we expected to see a difference in participation levels.



To overcome resistance to interact with a new software application, we designed the

tournaments to be fun, hoping many users would opt-in to play. We knew from early

discussions with department managers, the attitude of, “I am too busy”, could negate our test

results. The advice given to us from department managers was to find a way to increase the

“what’s in-it for me” ( WIFM) factor. We took this advice to heart, added relevant prizes, hand -

picked team captains, and a provided a Ringorang™ coach to assist the team captains with tips

and encouragement.

1.4 Method of Analysis

We used three forms of analysis: objective data, survey data, and interviews.

The source of the objective data was from the Ringorang™ database, which captured all interactions

with users. The database contains a record for every time the user interacts with the Ringorang™ game

and stores these interactions, including: accepted delivery of the question, played the question, did not

play the question, correctly answered the question, or incorrectly answered the question.

The Ringorang™ database also logs and stores the delivery of all questions, the number of people who

played the question, the number of correct answers, the number of incorrect answers, and the number

of times users timed-out before responding to question.

Two surveys were conducted; both surveys used “Survey Monkey”, an online web-based survey tool.

Surveys were used to gauge the user’s attitude, satisfaction with the tournaments, the quality of

curriculum learning elements, and the user’s likelihood to recommend Ringorang™ as a learning

retention tool. Users were sent emails twice soliciting their help to complete a survey. The first email

was sent at the end of Tournament 1 and second email was sent at the end of Tournament 3. The

Ringorang™ surveys asked for the users’ opinions of:

satisfaction level with the NERC CIP tournament communications,

quality of the NERC CIP curriculum,

the impact of the team captain,

their experience in general using a game to increase their knowledge, and

if the Ringorang™ game helped them learn NERC CIP requirements.

Both informal and informal interviews were conducted during the tournaments to ask what was working

well, and what we could do to improve the experience of users, teams, and team captains. We hosted

two meeting with team captains to solicit their feedback. We video-taped interviews with participants

who had high participation levels, asking them what they liked most about Ringorang, the NERC CIP

tournaments, team play, and the curriculum. We used these interviews to help us understand with more

specificity the survey ratings.

Using these three methods of analysis, we are confident in our conclusions and relatively comfortable

with our findings.



2.0 Is Ringorang™ an Effective Learning Retention Tool?

2.1 Does Ringorang™ Impact Learning Retention?

Three Ringorang™ NERC CIP Learning Tournaments were hosted between May 2, 2011 and June 24,

2011. To help determine the learning effectiveness of Ringorang, we evaluated the performance of each

tournament and any gain (or loss) in knowledge between tournaments.

We used four performance measures in our analysis of Ringorang™ NERC CP Learning Retention.

Dallions®: After the completion of the first tournament, and based on performance as

measured in average DALLIONS® per participant, there appears to be a significant difference

between participants who completed No Training and the groups who completed Prior

Compliance Training and Impulsive Training; yet, there appears to be no significant

difference between groups who completed Prior Compliance Training and Impulsive

Training.

Correct Answers: After the completion of the first tournament, and based on performance

as measured by the average correct answer percent per participant, there appears to be no

significant difference between the groups who completed Prior Compliance Training,

Impulsive Training, and No Training.

Percent Correct: After the completion of first tournament, and based on performance as

measured in the average percent of correct answers per participant, there appears to be no

significant difference between the groups who completed Prior Compliance Training,

Impulsive Training, and No Training.

Questions Played: After the completion of the first tournament, and based on performance

as measured by the average Questions Played per participant, there appears to be a

significant difference between participants who completed No Training and the participants

who completed Prior Compliance Training and Impulsive Training; yet, there appears to be

no significant difference between participants who completed Prior Compliance Training

and Impulsive Training.

To help us analyze the impact of Ringorang™ learning tournaments as an effective method to learn and

retain knowledge, we divided the participating users into three groups, based on when and if they

completed NERC CIP compliance training before or between tournament rounds. We used the following

groupings in our analysis of whether Ringorang™ learning tournaments are an effective learning method.

Prior Compliance Training: The group of users who completed NERC CIP training prior to the

first NERC CIP tournament communication. This group of users completed the PSE NERC CIP

compliance training prior to May 2, 2011.

Impulsive Training: The group of users who completed NERC CIP training between the time

of the first NERC CIP tournament communication and before the end of the week



immediately following the completion of Tournament #3. This group of users completed the

PSE NERC CIP compliance training prior to May 2, 2011 and before June 31, 2011.

No Training: The group of users who have no known history of completing NERC CIP training

to-date. This group of users has no record of completing NERC CIP training prior to June 31,

2011.

2.1.1 Tournament #1 Performance

Table 3 - Tournament 1 Performance - Points

Tournament #1 PERFORMANCE - DALLIONS®

Prior Compliance

Training Impulsive Training

No Training

Group Count 68 48 30

Tota l Points 498,575 349,160 186,525

Average Points 7,332 7,274 6,218

Standard Deviation 6,074 6,101 6,056

Median 6,285 5,931 4,476

Observations - Performance as Measured by Points (DALLIONS®)

In Tournament #1, as expected, the participants who completed Prior Compliance Training

performed best as judged by the average points (DALLIONS®) per participant.

o Prior Compliance Training – Average Points = 7,332 Dallions®

o Impulsive Training – Average Points = 7,274 Dallions®

o No Training – Average Points = 6,218 Dallions®

Participants who completed No Training performed much lower than their peers who

completed training, hence it appears there is a significance difference between participants

who completed Prior Compliance Training and Impulsive Training versus participants who

completed No Training.

The participants who completed No Training either know less NERC CIP facts or they may be

much slower in submitting their responses to Ringorang™ learning game questions than

their peers. DALLIONS® are calculated by correct answer and speed of submitting answer.

Table 4 - Tournament 1 Performance - Correct Answers

Tournament #1 PERFORMANCE - CORRECT ANSWERS

Prior Compliance


No Training


Tota l Correct 3,548 2,431 1,345

Average Correct 52 51 45

Standard Deviation 31 33 33

Median 52 52 45

Observations – Performance as Measured by Correct Answer



In Tournament #1, as expected the participants who completed Prior Compliance Training

had the best scores as judged by the average correct answer per participant.

o Prior Compliance Training – Average Correct Answers = 52

o Impulsive Training – Average Correct Answers = 51

o No Training – Average Correct Answers = 45

There appears to be a significant difference between the participants who completed Prior

Compliance Training and Impulsive Training versus participants who completed No Training

as measured by correct answers.

Table 5 - Tournament 1 Performance - % Correct Answers

Tournament #1 PERFORMANCE - % CORRECT ANSWERS

Prior Compliance


No Training


Tota l Correct 3,548 2,431 1,345

Tota l Played 4,993 3,559 1,940

Average % Correct 71% 68% 69%

Observations – Performance as Measured by Percent Correct Answers

In Tournament #1, as expected the participants who completed Prior Compliance Training

had the best scores as judged by the average percent correct per participant.

o Prior Compliance Training – Average % Correct = 71.1%

o Impulsive Training – Average % Correct = 68.3%

o No Training – Average % Correct = 69.3%

There appears to be no significant difference between the participants who completed Prior

Compliance Training, Impulsive Training and No Training as measured by percent correct

answers.

Table 6 - Tournament 1 Performance – Questions Played

Tournament #1 PERFORMANCE - QUESTIONS PLAYED

Prior Compliance


No Training


Tota l Played 4,993 3,559 1,940

Average Played 73 74 65


Median 79 83 64

Observations – Questions Played

In Tournament #1, as expected the participants who completed Impulsive Training had the

best scores as judged by average number of questions player per participant.



o Prior Compliance Training –Average Questions Played = 73

o Impulsive Training – Average Questions Played = 74

o No Training – Average Questions Played = 65

There appears to be no significant difference between the participants who completed Prior

Compliance Training and Impulsive Training.

Yet there does appear to be a significant performance difference between participants who

completed No Training as compared to participants who complete Prior Compliance

Training and Impulsive Training.

Finding 1 – Ringorang™ as deployed at PSE by Farallon Risk Group and Vergence Entertainment for the NERC CIP Learning Tournaments appears to be an effective learning

retention tool as compared to existing NERC CIP Training methods.

We observed contradictions in our data analysis for Tournament #1, as noted with the

underlined phrases below.

In all four performance measurements, we see no significant difference between the group

of participants who completed Prior Compliance Training and Impulsive Training.

In three out of four performance measurements, the groups of users who completed Prior

Compliance Training and Impulsive Training had a significant difference in performance as

compared to group who completed No Training.

o Average Points: 7332, 7274, 6218

o Average Number Correct: 52, 51, 45

o Questions Played: 73, 74, 65

Yet in one performance measure, the group who completed No Training performed slightly

better than their group who completed Impulsive Training. The percentages are so close

that is appears there is no significant difference between all three groups of users in percent

correct answers.

o Average % Correct: 71%, 68%, 69%

2.1.2 Hypotheses on Learning Performance Differences

Using the performance measurements from Tournament 1, we hypothesize users who complete

Prior Compliance Training and Impulsive Training perform better in Ringorang™ learning

tournaments than users who complete no training. Based on NERC CIP Learning Tournament 1 data,

we formed five hypotheses to prove or disprove in Tournaments 2 and 3.

Participating in two or more Ringorang™ learning tournaments will yield about the same

learning retention performance results for participants who complete Prior Compliance

Training, Impulsive Training and No Training.



Completing Prior Compliance Training before the onset of a Ringorang™ learning tournament

will enable participants to have a significant learning retention performance advantage over

their peers who complete Impulsive Training just before, during or between tournaments, and

a significant performance advantage over participants who complete No Training.

Completing Impulsive Training just prior, during or between Ringorang™ learning tournaments

will enable participants to have a significant learning retention performance advantage over

their peer who completed Prior Compliance Training and over those who choose not to

complete No Training.

Completing No Training, yet participating in three Ringorang™ learning tournaments, will

enable participants to perform at or about the same level of learning retention as their peers

who complete Prior Compliance Training and Impulsive Training.

Finding 2 - Employees who complete compliance training in advance of participating in Ringorang™ NERC CIP Learning Tournaments may or may not have a significant performance advantage; and, Farallon Risk Group meticulously analyzed data to determine the answer to

this finding..

Of these four hypotheses, only one can be true. We will attempt to disprove all four hypotheses

listed above. Our analysis and summary findings are contained in section 2.3 below.

2.1.3 All Tournaments Performance

To analyze the performance differences between the groups of participant who complete Prior

Compliance Training, Impulsive Training, and No Training, we will analyze the performance results of all

three tournaments aggregated together.

Table 7 - NERC CIP Tournaments Performance - Points

All Tournaments PERFORMANCE - DALLIONS®

Prior Compliance


No Training


Tota l Points 1,167,792 906,103 472,492

Average Points 6,524 6,864 6,300

Standard Deviation 5,000 5,112 5,271

Median 5,894 6,139 5,294

Observation – Points (DALLIONS®)

Combining the data from all three NERC CIP tournaments, we calculated the average

number of points earned by participants.

o Prior Compliance Training – Average Points: 6,524 Dallions®

o Impulsive Training – Average Points: 6,864 Dallions®

o No Training – Average Points: 6,300 Dallions®



The population group who completed Impulsive Training outperformed the groups who

completed Prior Compliance Training and No Training by a few hundred points.

While there is some difference in points, there is no significant difference in points between

the three groups.

Table 8- NERC CIP Tournaments Performance – Correct Answers

All Tournaments PERFORMANCE - CORRECT ANSWERS

Prior Compliance

Training

Impulsive

Training

No

Training


Tota l Correct 7,556 5,677 3,130

Average Correct 42 43 42


Median 40 42 44

Observations – Correct Answers

Aggregating the data from all three NERC CIP tournaments, we see the average number of

correct answers is extremely close across all three groups, and the standard deviations are

also very close.

o Prior Compliance Training - Averaged Correct Answers is 42

o Impulsive Training – Average Correct Answers is 43

o No Training – Average Correct Answers is 42

Correct Answers are a measure of learning retention, and in Tournament 3, we see the

learning retention performances are almost identical for three groups of users.

Using performance data from three NERC CIP learning tournaments, we found that there is

no significant difference in the Average Correct Answers between groups who complete

Prior Compliance Training, Impulsive Training and No Training.

Table 9 - NERC CIP Tournaments Performance - % Correct Answers

All Tournaments PERFORMANCE - % CORRECT ANSWERS

Prior Compliance

Training

Impulsive

Training

No

Training


Tota l Correct 7,556 5,677 3,130

Tota l Played 10,392 7,943 4,443

Average % Correct 73% 71% 70%

Observations - % Correct Answers

Participants who completed No Training performed equally well as measured by Average %

Correct Answers as users in the groups who completed Prior Compliance Training and

Impulsive Training.



o Prior Compliance Training – Average % Correct: 73%

o Impulsive Training – Average % Correct: 71%

o No Training – Average % Correct: 70%

Aggregating the data from all three tournaments allows us to see that the average percent

correct for all three groups is very similar, ranging from 70% to 73%.

There appears to be no significant difference between Average % Correct Answers between

groups who complete Prior Compliance Training, Impulsive Training, and No Training.

Table 10- NERC CIP Tournaments Performance – Questions Played

All Tournaments PERFORMANCE - QUESTIONS PLAYED

Prior Compliance


No Training


Tota l Played 10,392 7,943 4,443

Average Played 58 60 59


Median 56 58 61

Observation – Questions Played

Participants who completed No Training performed equally well as measured by Average

Questions Played in the groups who completed Prior Compliance Training and Impulsive

Training.

o Prior Compliance Training – Average Questions Played = 58

o Impulsive Training – Average Questions Played = 60

o No Training – Average Questions Played = 59

Aggregating the data from all three tournaments allows us to see that the average

Questions Played for all three groups is very similar, ranging from 58 to 60.

There appears to be no significant difference between average Questions Played between

groups who complete Prior Compliance Training, Impulsive Training, and No Training.

In all four learning performance metrics there is no significant difference between users

who complete Prior Compliance Training, Impulsive Training, or No Training.

o Average Points (DALLIONS®): 6524, 6864, 6300

o Average Correct Answers: 42, 43, 42

o Average % Correct Answers: 73%, 71%, 70%

Since there is no significant learning performance difference between groups who complete

Prior Compliance Training, Impulsive Training and No Training; we believe: Ringorang™ is a

highly effective learning retention tool.



Finding 3 - Ringorang™ NERC CIP Learning Tournaments as deployed at PSE by Farallon Risk Group and Vergence Entertainment appear to be a highly effective learning method with

retention levels equivalent to other proven training methods.

As we analyzed the learning retention levels these three groups of participants, we

discovered an interesting phenomenon.

o Announcing Ringorang™ learning tournaments created a reaction among 56 of the

203 PSE IT team members who completed NERC CIP Compliance Training out-of-

cycle either to prepare for upcoming tournaments; or, they felt confident they could

pass the NERC CIP Compliance Training program with ease as a result of the learning

retained from being exposed to curriculum learning elements (CLEs) presented in

the NERC CIP Learning Tournaments.

Finding 4 - Ringorang™ NERC CIP Learning Tournaments as managed at PSE motivate participants to impulsively complete NERC CIP Compliance Training out-of-cycle to either prepare for tournament play; or due to their newly gained confidence and mastery of CLEs,

they believe they will pass the NERC CIP Compliance exam with ease.

2.1.4 Summary Performance Data

To evaluate the effectiveness of Ringorang™ to increase learning retention levels of an organization, we

analyzed the summary performance data from three tournaments. The summary data we analyzed

included: participation levels, number of correct answers, % of correct answers, and questions played.

We will attempt to prove or disprove if learning retention levels increased or decreased as we repeated

presentation of curriculum learning elements (CLEs) by the Ringorang™ game player used in the NERC

CIP Learning Tournaments. And if learning retention levels or rates changed as a result of multiple

interactions with the NERC CIP CLEs, which we scheduled to repeat in the NERC CIP Learning

Tournaments, could we identify the influencing factor(s) that lead to significant change in learning

commitment (Questions Played), learning retention levels (Correct Answers) and learning retention rates

(% Correct).

The graph below, Figure 1, shows the summary results of the three NERC CIP Learning Tournaments. In

the graph below, the blue bars represent the number of Questions Played in each tournament by

participants and the red bars represent the number of Correct Answers selected in each tournament.

We should explain participants have only 20 seconds to view, read, understand, and select the correct

answer when responding to a question delivered by the Ringorang™ game. Since participants have just

20 seconds to read, understand and select the correct answer, we believe they do not have time to

research, ask or find the answers; rather through recall and recognition skills, they must quickly choose

the “correct” answers to each CLE question.



Figure 1 - Ringorang™ NERC CIP Tournaments Summary Results

Observations – Summary of Effort and Learning Retention

The data contained in the graph above has been normalized for five-day tournaments. We analyzed the

total Questions Played, Correct Answers, and % Correct.

The number of Questions Played by participants increased in each tournament, reflecting

the participants’ commitment to participate in the Ringorang™ NERC CIP Learning

Tournaments to benefit themselves, their team, their department and their organization.

o Tournament 1 = 5,265 Questions Played, baseline learning commitment

o Tournament 2 = 5,921 Questions Played, an increase of 12.5% above Tournament 1

o Tournament 3 = 7,570 Questions Played, an increase of 28% above Tournament 2

The number of Correct Answers chosen in each tournament reflects the learning retention

level of participants who interact with the NERC CIP curriculum learning elements (CLEs)

presented in the Ringorang™ NERC CIP Learning Tournaments. The number of Correct

Answers increased in each learning tournament. Repeated exposure of NERC CIP CLEs

0

1000

2000

3000

4000

5000

6000

7000

8000

Tournament 1 Tournament 2 Tournament 2

Questions Played 5265 5921 7570

Correct Answers 3669 4243 5653

Ringorang NERC CIP Tournament ResultsQuestions Played and Correct Answers

70% Correct

72% Correct

75% Correct



presented in the Ringorang™ NERC CIP Learning Tournaments increases learning retention

levels of the PSE IT organization.

o Tournament 1 = 3,669 Correct Answers, baseline learning retention level

o Tournament 2 = 4,243 Correct Answers, an increase of 15.6% over Touranment1

o Tournament 3 = 5,653 Correct Answers, an increase of 33% over Tournament 2

The % Correct answers earned in each tournament reflect the organization’s learning

retention rate. In each successive tournament, participants scored higher on % Correct. We

attempted to make the NERC CIP game content more challenging in each tournament to

keep the participants from getting bored. Yet, even though we delivered more challenging

questions, the learning retention rate increased.

o Tournament 1 = 70% Correct, baseline learning retention rate

o Tournament 2 = 72% Correct, an increase of 2 percentage points over Tournament 1

o Tournament 3 = 75% Correct, an increase of 3 percentage points over Tournament 2

Finding 5 – The deployment methods used by Farallon Risk Group and Vergence Entertainment at PSE, generated a direct and positive correlation between the frequency participants are exposed to NERC CIP CLEs in Ringorang™ NERC CIP Learning Tournaments and the participants’ learning retention level (Correct Answers) and their learning retention rate (%

Correct) Answers.

2.1.5 Tournament Themes

We changed the theme in each tournament to gain an understanding of which theme would

yield the best participation results. We normalized the data from each tournament for a 5-day

tournament, and then we plotted the data to see if the participation numbers and the

participation curve varied by theme. What was the theme of each tournament?

Tournament 1

o Theme = “Please Participate”. We used the “% Participation” as the measure to

track performance in Tournament 1. In the Tournament 1 announcement

communications, we did not mention competition. And in our follow up

conversations with PSE IT Department Managers and Ringorang™ Team Captains,

we did not encourage or discourage competition for Tournament 1; however, some

Ringorang™ Team Captains rallied their teams using “competition” as the motivator

to get their team members enthused about committing to participate in

Tournament 1.

Tournament 2

o Theme = “Demonstrate Your Knowledge”. We used the number of “Correct Answers”

In Tournament 2 as our metric to measure performance. We discouraged

competition in follow up conversations with Team Captains, asking them to not



push for points, but rather focus their team members on getting the most possible

correct answers.

Tournament 3

o Theme = “Play to Win“. We openly encouraged competition in Tournament 3. The

Tournament 3 announcement communications highlighted this tournament players

should compete to win. Subsequent to our announcement communications being

sent out, we hosted a Team Captains meeting on day-two of the tournament with

our Executive Sponsor in attendance. The Executive Sponsor encouraged

participation, demonstrate NERC CIP knowledge and he cheered them to try and

win awards and prizes offered in Tournament 3. We also distributed Ringorang™

participation reports on day-thee to PSE IT Department Managers and on day-four

to Ringorang™ Team Captains.

Figure 2 - Daily Participation Curve for NERC CIP Tournaments

Observations – Participation Curve

We analyzed daily participation numbers to determine if one theme and communication

approach produced superior participation results as compared to another theme and

communication approach.

Daily Participation results:

o The participation level in Tournament 1 and Tournament 3 are within 1% of each

other, and the slope of their curves is nearly identical. There appears to be no

significant difference in daily participation numbers using “Please Participate”

versus “Play to Win”.

Daily Participations Curve for Three Tournamentsnormalized for 5-day tournaments

0

20

40

60

80

100

120

140

Vo

lum

e o

f p

art

icip

an

ts

T1 (5/2~5/11)

T2 (5/31~6/3)

T3 (6/20~624)

T1 (5/2~5/11) 121 133 127 124 84

T2 (5/31~6/3) 98 109 103 89 80

T3 (6/20~624) 121 128 132 121 94

Day 1 Day 2 Day 3 Day 4 Day 5



o The level of participation in Tournament 2 was lower than Tournaments 1 and 3,

and the shape of its curve was different on the latter two days.

o The lower daily participation numbers may be explained by the composition of this

work week. Rather than being a standard 5-day work week, Monday was Memorial

Holiday and this was the first work week that parents may be able to take off to

start their family’s summer vacation.

Finding 6 – Themes chosen by Farallon Risk Group and approved by PSE IT for the Ringorang™ NERC CIP Learning Tournaments made a difference in daily participation numbers, and themes

that included competition yielded better participation and commitment.

2.1.6 Leadership, Communications and Competition

In an attempt to measure the difference leadership makes on commitment to participate in the

NERC CIP Learning Tournaments, we analyzed changes in the number of Questions Played. We

varied the level of leadership involvement in authoring tournament communications,

distributing tournament announcements, and sending reinforcing messages to tournament

participants. The variations in levels of leadership involvement in tournament communications

are described below.

In Tournament 1, the Project Manager authored and distributed tournament

announcements to the participants, and the Business Owner sent reinforcement messages

to all managers. Ringorang™ Team Captains were encouraged to host at least one meeting

with their team members. Some Ringorang™ Team Captains not only hosted a meeting, but

also communicated their competition strategy.

In Tournament 2, the Business Owner authored and distributed the tournament

announcements to all participants and department managers. Most Ringorang™ Team

Captains sent reminders to encourage their participants to play each day.

In Tournament 3, the Business Owner and Executive Sponsor collaborated on the authoring

of the tournament announcement; the Business Owner distributed the tournament

announcement to all participants and department managers. The Executive Sponsor sent a

reinforcing memo affirming the value of learning the NERC CIP curriculum, clarifying the

purpose was to help reduce potential, future “self-reports” of out-of-compliance incidents.

On day 3 of the tournament, Department Managers received tournament reports showing

the participation levels of their team members. On day 4, Ringorang™ Team Captains

received tournament reports showing the participation levels of their team members. Both

of these reports were sent with suggestions for the department manager and team captains

to follow-up with the low-participation team members.

To measure the difference leadership communications make, we used data displayed in Figure 1

and Figure 2. We used Tournament 1 as our baseline to measure change; we calculated the

change in participation numbers, number of Questions Played and number of Correct Answers;

and looked for significant changes.



Table 11 - Summary Tournament Performance


Days of Play 9 4 5

Invited to Participate 203 205 213

Number of Active Participants 151 111 133

Questions Played 10,530 4,737 7,570

Correct Answers 7,337 3,394 5,653

Dallions® 1,034,259 517,819 994,308

Average Dallions® Dallions® per Active Participant per Day of Play

761 1,166 1,495

Average Questions Played per Active Participant per Day of Play

7.7 10.7 11.4

Average Correct Answers per Active Participant per Day of Play

5.4 7.6 8.5

% Participants Active in Play 74% 54% 62%

% Correct Answers per Tournament 70% 72% 75%

Observations – Summary Tournament Performance

The Summary Tournament Performance Table shows the length of play for each tournament.

The number of Questions Played varied much due to: the number of days of play for each

tournament, the effort expended by each participant to answer questions each day, and the

number of participants who actively played in each tournament. Correct Answers varied much

also due to the same items: days of play, number of active participants, and effort to answer

questions each day. Total DALLIONS® varied even more, due to same variables as Correct

Answers; and, due to the intensity of play by the participants. Speed is a function in the

calculation of Dallion™ points. To earn more points, participants must respond faster, choosing

the correct answer. When participants choose the wrong answer, they lose points. To earn more

points in a tournament, the participants must be faster at choosing the correct answer.

We invited 203 people to participate in Tournament 1. Several PSE IT employees asked if they

could be invited to play in Tournament 2, and another 8 new employees asked if they could be

invited to play in Tournament 3.

o A big difference between the NERC CIP Learning Tournaments was the number of

days of play.

Tournament 1 had 9 days of play, 8 work week plays plus Sunday.

Tournament 2 was scheduled on Memorial Week. As a result Tournament 2

ran only for 4 days.

Tournament 3 ran for 5 days on a normal work week.

o The number of Invited Participants changed slightly and the number of Active

Participants in each NERC CIP Tournament changed much.

Tournament 1 had 203 Invited Participants and 151 Active Participants.





o DALLIONS® are a function of speed to choose the correct answers. Notice the total

number of DALLIONS® in Tournaments 1 and 3 were very close, even though

Tournament 1 had nine days of play and Tournament 3 had only five days of play.

Tournament 1 = 1,034,259 Dallions®

Tournament 2 = 517,819 Dallions

Tournament 3 = 994,308 Dallions®

o Notice that the Average DALLIONS® per Active Participant per Day of Play varied

greatly between Tournaments 1, 2, and 3. Participants played more intensely in

Tournament 3 and Tournament 1. Tournament 2 did not have much intensity of play.

What did Tournaments 1 and 3 have in common with each other and not in common

with Tournament 2: Repetition of Curriculum Content.

T1 = 761 Average DALLIONS® per Active Participant per Day of Play

T2 = 1,166 Average DALLIONS® per Active Participant per Day of Play.

T3 = 1,495 Average DALLIONS® per Active Participant per Day of Play.

o Which attribute was prevalent in Tournaments 2 and 3, yet not as prevalent in

Tournament 1: Repetition of curriculum learning elements (CLEs).

o What attributes were different in Tournament 3 than in Tournament 2 that could

explain such a large change in Dallions®? Leadership involvement to create, send

and reinforce tournament communications was greater, and the emphasis placed

on team competition by Ringorang™ Team Captains was more focused.

o Commitment can be measured. Time spent on a task or activity reflects

commitment. Time spent interacting with the NERC CIP curriculum reflects a

participant’s commitment to learn the curriculum content. Questions Played is the

measure we chose to analyze participation commitment to learn and retain the

NERC CIP curriculum learning elements.

T1 = 7.7 average Questions Played per participant per day.



o Learning Retention results can be measured by the total number of Correct Answers,

and any change in % Correct Answer. The number of Correct Answers per Active

Participant per Day of Play is the measure we chose to evaluate the learning

retention level of the participants in the NERC CIP Learning Tournaments.

Tournament 1 = 5.4 Correct Answers per Active Participant per Day of Play.





Table 12 – Tournament Summary Change Percentages

Summary Change Percentages Tournament

1 Tournament

2 Tournament

3 Change % in Active Participation per Tournament

baseline -26% -12%

Change % in Questions Played per Active Participant per Day of Play

baseline 38% 47%

Change % in Correct Answers per Active Participant per Day of Play

Baseline 42% 57%

Change in DALLIONS® per Active Participant per Day of Play

Baseline 53% 96%

Observations – Summary Performance Changes

Using Tournament 1 as our baseline, we noted the following changes in performance

percentages:

The average Questions Played per Active Participant per Day of Play increased 38% in

Tournament 2 and 47% in Tournament 3 over Tournament 1.

o Since Questions Played is a measure of learning commitment, then we can see

increasing levels of commitment to participate in the NERC CIP Learning

Tournaments with each successive tournament communications, which can be

explained by commitment to play or familiarity in the Ringorang™ game.

o And since a Ringorang™ NERC CIP practice round was offered before the start of the

tournaments, we discounted the idea that participants were significantly more

comfortable with the Ringorang™ Game in Tournaments 2 and 3. Rather, we

attribute the 38% and 47% change in Questions Played per Active Participant per

Day of Play to leadership communications.

We observed a learning retention change of 42% in Tournament 2 and 57% in Tournament 3,

which is significant. This amount of change can only be explained by the participants’

learning retention ability. Learning retention is the skill and ability to recognize the correct

answer from prior learning and their ability to recall the correct answer to previ ously missed

questions.

o Tournament 2 had a negative change of 26% for Participation, yet this tournament

yielded 38% more Questions Played per Day of Play, 42% more Correct Answers and

53% more Dallions® per Active Participant per Day of Play. In Tournament 2, many

Team Captains took it upon themselves to encourage their team members to print

and study the federal NERC CIP standards. Some Ringorang™ Team Captains sent

emails daily, leveraging “innovative learning” techniques. A few team captains

hosted meetings with the team members to solicit and share learning ideas.



o Tournament 3 had a negative change of 26% for Participation, yet this tournament

yielded 47% more Questions Played per Day of Play, 57% more Correct Answers and

96% more DALLIONS® per Active Participant per Day of Play. We established a few

rules in Tournament 3, like “no sharing of learning content” from the Ringorang™

game. We loaded many new questions and a higher percentage of hard questions to

make this tournament very challenging. Even so, the average number of Correct

Answer jumped significantly. Average Dallions® earned by each participant nearly

doubled from Tournament 1. We believe this is due to the increase in learning and

an increase speed in the participants’ ability to recognize correct answers.

Finding 7 – The level of leadership involvement at PSE in creating, sending and reinforcing tournament communications appears to make a significant difference in the Ringorang™ NERC

CIP Learning Tournaments as measured by Questions Played.

Finding 8 - Competition as applied by Farallon Risk Group and Vergence Entertainment does appear to influence the intensity of play in Ringorang™ learning tournaments as measured by

the increase in DALLIONS® earned by participants.

Finding 9 – Learning retention scores appear to influenced by frequency of exposure to the curriculum learning elements (CLEs) and the number of tournaments played by each participant in the Ringorang™ Learning Tournaments as measured by the number of Correct

Answers per Participant per Day of Play and % Correct of Answers per tournament.

Finding 10 - As deployed at PSE by Farallon Risk Group and Vergence Entertainment, Ringorang™ is a highly effective learning retention tool as demonstrated by an increase of 57%

in Correct Answers and 96% in DALLIONS® per Participant per Day of Play.

2.2 Can We Measure Learning Retention by Team and Team Member,

and Provide Learning Diagnostic by Team?

To compare learning retention performance, we will use the legend below to categorize learning

retention performance levels.

Table 13 - Legend of Learning Retention Performance

Legend of Learning Retention Performance

100% Superb

95% Outstanding

90% Excellent

85% Very Good

80% Good

75% Very Fair

70% Fair

65% Low

60% Very Low

55% Poor

50% Very Poor

35% Bad



2.2.1 Learning Retention Performance by Team Member

Table 14 - Learning Retention Performance by Team Member

User's Initials Team Name Tournament

Dallions® Total

Correct Answer

Total Questions

Played

Total Available Questions for Tournaments

% Correct Answer

% Questions Played

SE SAM T3 6,265 33 33 80 100.0% 41.3%

DA The Gorge T3 690 4 4 80 100.0% 5.0%

CF Gas Works T3 20,109 75 77 80 97.4% 96.3%

TI WECC Warriors T3 16,518 63 65 80 96.9% 81.3%

NM Gas Works T3 19,068 75 80 80 93.8% 100.0%

RS Cyber Dogs T3 16,211 74 79 80 93.7% 98.8%

MG WECC Warriors T3 14,326 65 72 80 90.3% 90.0%

MO WECC Warriors T3 17,599 72 80 80 90.0% 100.0%

GA Blue Screens of Death T3 15,814 72 80 80 90.0% 100.0%

AT SAM T3 16,177 71 79 80 89.9% 98.8%

JS The Dams T3 15,551 68 77 80 88.3% 96.3%

DT Cyber Dogs T3 12,304 69 79 80 87.3% 98.8%

GY Rudy's Rowdies T3 14,316 64 74 80 86.5% 92.5%

FA Nercamania T3 13,967 69 80 80 86.3% 100.0%

PS Seagers CIP'ers T3 9,022 43 50 80 86.0% 62.5%

TB Cyber Dogs T3 10,084 48 56 80 85.7% 70.0%

JJ WECC Warriors T3 3,622 18 21 80 85.7% 26.3%

CM The Dams T3 12,074 59 69 80 85.5% 86.3%

JH Gas Works T3 16,505 68 80 80 85.0% 100.0%

DC EMP T3 13,032 62 73 80 84.9% 91.3%

BB Nercamania T3 13,122 63 75 80 84.0% 93.8%

SP Blue Screens of Death T3 16,215 67 80 80 83.8% 100.0%

RH SAM T3 15,015 67 80 80 83.8% 100.0%

LS Gas Works T3 10,362 61 73 80 83.6% 91.3%

PS WECC Warriors T3 12,814 66 79 80 83.5% 98.8%

KS Mount Adams T3 10,157 49 59 80 83.1% 73.8%

KK WECC Warriors T3 8,659 47 57 80 82.5% 71.3%

GR Blue Screens of Death T3 13,079 65 79 80 82.3% 98.8%

CM Blue Screens of Death T3 10,244 65 79 80 82.3% 98.8%

JM Mount Adams T3 9,939 59 72 80 81.9% 90.0%

KS SAM T3 11,971 54 66 80 81.8% 82.5%

JW The Gorge T3 10,322 58 71 80 81.7% 88.8%

CS Seagers CIP'ers T3 4,608 26 32 80 81.3% 40.0%

MH WECC Warriors T3 4,243 26 32 80 81.3% 40.0%

TW Chief Seattle T3 2,452 13 16 80 81.3% 20.0%




Dallions® Total

Correct Answer

Total Questions

Played


% Correct Answer

% Questions Played

ST Chief Seattle T3 8,209 43 53 80 81.1% 66.3%

MJ EMP T3 13,309 64 79 80 81.0% 98.8%

BJ Rudy's Rowdies T3 8,918 46 57 80 80.7% 71.3%

RF SAM T3 4,076 29 36 80 80.6% 45.0%

MB Seagers CIP'ers T3 6,491 37 46 80 80.4% 57.5%

VC The Dams T3 10,399 52 65 80 80.0% 81.3%

DB Glacier Peak T3 10,019 59 74 80 79.7% 92.5%

BW Chief Seattle T3 11,026 55 69 80 79.7% 86.3%

AM Blue Screens of Death T3 6,556 31 39 80 79.5% 48.8%

SH The Gorge T3 8,718 42 53 80 79.2% 66.3%

DM The Dams T3 12,147 63 80 80 78.8% 100.0%

KF Gas Works T3 10,089 55 70 80 78.6% 87.5%

KL Mount Baker T3 2,267 11 14 80 78.6% 17.5%

SS The Dams T3 9,678 54 69 80 78.3% 86.3%

SR Blue Screens of Death T3 11,101 60 77 80 77.9% 96.3%

LP Mount Adams T3 9,262 51 66 80 77.3% 82.5%

SW Gas Works T3 12,807 61 79 80 77.2% 98.8%

JW SAM T3 8,878 44 57 80 77.2% 71.3%

CM The Gorge T3 8,240 44 57 80 77.2% 71.3%

JW EMP T3 8,722 50 66 80 75.8% 82.5%

CA SAM T3 7,872 50 66 80 75.8% 82.5%

RS The Dams T3 11,055 56 74 80 75.7% 92.5%

KP Seagers CIP'ers T3 9,621 55 73 80 75.3% 91.3%

BH EMP T3 9,028 54 72 80 75.0% 90.0%

DM Chief Seattle T3 5,198 39 52 80 75.0% 65.0%

TG The Dams T3 3,975 24 32 80 75.0% 40.0%

DS Cyber Dogs T3 3,691 24 32 80 75.0% 40.0%

JL Gas Works T3 3,281 24 32 80 75.0% 40.0%

BB Seagers CIP'ers T3 7,826 47 63 80 74.6% 78.8%

JF Cyber Dogs T3 9,381 57 77 80 74.0% 96.3%

RL Seagers CIP'ers T3 9,312 54 73 80 74.0% 91.3%

WJ SAM T3 8,249 54 73 80 74.0% 91.3%

JD Mount Baker T3 8,836 58 79 80 73.4% 98.8%

LF WECC Warriors T3 9,666 55 75 80 73.3% 93.8%

IS Glacier Peak T3 7,821 49 67 80 73.1% 83.8%

NS Rudy's Rowdies T3 8,492 54 74 80 73.0% 92.5%

NS Seagers CIP'ers T3 6,092 40 55 80 72.7% 68.8%

JM Cyber Dogs T3 3,673 24 33 80 72.7% 41.3%

AL Nercamania T3 10,121 53 73 80 72.6% 91.3%

II EMP T3 7,327 50 69 80 72.5% 86.3%

VC Chief Seattle T3 13,037 52 72 80 72.2% 90.0%

PM The Gorge T3 4,034 26 36 80 72.2% 45.0%

JL EMP T3 8,933 57 79 80 72.2% 98.8%

RM Gas Works T3 6,804 46 64 80 71.9% 80.0%




Dallions® Total

Correct Answer

Total Questions

Played


% Correct Answer

% Questions Played

SM Gas Works T3 3,203 23 32 80 71.9% 40.0%

JB Cyber Dogs T3 3,111 23 32 80 71.9% 40.0%

WS Nercamania T3 7,446 48 67 80 71.6% 83.8%

AD Nercamania T3 2,437 25 35 80 71.4% 43.8%

RN Mount Adams T3 8,166 52 73 80 71.2% 91.3%

QL Nercamania T3 8,620 53 75 80 70.7% 93.8%

VP Glacier Peak T3 6,935 48 68 80 70.6% 85.0%

TT Mount Adams T3 2,102 12 17 80 70.6% 21.3%

YL Mount Baker T3 6,760 50 71 80 70.4% 88.8%

TB Nercamania T3 8,888 51 73 80 69.9% 91.3%

CC Chief Seattle T3 3,178 23 33 80 69.7% 41.3%

BS Cyber Dogs T3 3,053 23 33 80 69.7% 41.3%

JL Blue Screens of Death T3 6,254 39 56 80 69.6% 70.0%

MB The Gorge T3 7,496 34 49 80 69.4% 61.3%

BS Nercamania T3 5,891 36 52 80 69.2% 65.0%

CB Glacier Peak T3 2,451 22 32 80 68.8% 40.0%

CS The Dams T3 4,481 30 44 80 68.2% 55.0%

JJ Seattle's Best T3 5,294 46 68 80 67.6% 85.0%

Seattle's Best T3 5294 46 68 80 67.6% 85.0%

SW SAM T3 6,195 50 74 80 67.6% 92.5%

YL Cyber Dogs T3 2,816 25 37 80 67.6% 46.3%

BB Mount Baker T3 7,448 49 73 80 67.1% 91.3%

TS Glacier Peak T3 5,402 49 73 80 67.1% 91.3%

LR Blue Screens of Death T3 8,871 49 74 80 66.2% 92.5%

CH Glacier Peak T3 5,923 49 75 80 65.3% 93.8%

PK The Gorge T3 2,550 22 34 80 64.7% 42.5%

VD SAM T3 5,193 40 62 80 64.5% 77.5%

AH Gas Works T3 5,542 46 72 80 63.9% 90.0%

BB The Dams T3 4,746 46 73 80 63.0% 91.3%

JW Rudy's Rowdies T3 3,892 32 51 80 62.7% 63.8%

BB Seagers CIP'ers T3 2,147 15 24 80 62.5% 30.0%

JP Rudy's Rowdies T3 558 5 8 80 62.5% 10.0%

RC Mount Adams T3 2,934 23 37 80 62.2% 46.3%

SC The Dams T3 2,943 29 47 80 61.7% 58.8%

DM Nercamania T3 2,549 29 48 80 60.4% 60.0%

IH Glacier Peak T3 3,612 35 58 80 60.3% 72.5%

CC Mount Adams T3 1,374 19 32 80 59.4% 40.0%

ZL Glacier Peak T3 2,525 17 29 80 58.6% 36.3%

LL Seagers CIP'ers T3 3,069 39 67 80 58.2% 83.8%

RR WECC Warriors T3 2,404 18 32 80 56.3% 40.0%

SN Seagers CIP'ers T3 1,865 29 52 80 55.8% 65.0%

AJ EMP T3 5,990 37 68 80 54.4% 85.0%

DM Chief Seattle T3 717 13 24 80 54.2% 30.0%

TV Blue Screens of Death T3 613 17 32 80 53.1% 40.0%




Dallions® Total

Correct Answer

Total Questions

Played


% Correct Answer

% Questions Played

BJ Seagers CIP'ers T3 726 29 55 80 52.7% 68.8%

JK Nercamania T3 142 1 2 80 50.0% 2.5%

CS SAM T3 2,626 15 33 80 45.5% 41.3%

JR Mount Adams T3 1,299 27 62 80 43.5% 77.5%

MD The Gorge T3 3,004 26 69 80 37.7% 86.3%

KP Glacier Peak T3 2,821 11 35 80 31.4% 43.8%

EW Gas Works T3 404 3 10 80 30.0% 12.5%

Observations – Learning Retention by Team Member

The table above illustrates learning retention scores in the column labeled “% Correct Answers”.

Participation Commitments is represented in the column labeled “% Questions Played”.

Which five team members had the highest learning retention score?

Using the table above, the team members with the highest NERC CIP learning retention

score in Tournament 3 are:

o SE = 100% Learning Retention

o DA = 100% Learning Retention

o CF = 97.4% Learning Retention

o TI = 96.9% Learning Retention

o NM = 93.8% Learning Retention

The team members with the highest percent of learning commitment in the NERC CIP

Learning Tournament 3:

o SE = 41.3% Learning Commitment

o DA = 5.0% Learning Commitment

o CF = 96.3% Learning Commitment

o TI = 81.3% Learning Commitment

o NM = 100.0% Learning Commitment

Notice that SE and DA had low learning commitment levels, which suggest that % learning

retention along, is not a sufficient measurement of learning retention, so we re -sorted the

team members learning retention table by “Correct Answer Total” and “Dallions® .



Table 15 – Top Performance by Team Member by Correct Answers and Dallions®


Dallions® Total

Correct Answer

Total Questions

Played

Total Available

Questions for Tournaments

% Correct Answers

% Questions

Played

CF Gas Works T3 20,109 75 77 80 97.4% 96.3%

NM Gas Works T3 19,068 75 80 80 93.8% 100.0%

RS Cyber Dogs T3 16,211 74 79 80 93.7% 98.8%

MO WECC Warriors T3 17,599 72 80 80 90.0% 100.0%

GA

Blue Screens of Death T3 15,814 72 80 80 90.0% 100.0%

AT SAM T3 16,177 71 79 80 89.9% 98.8%

FA Nercamania T3 13,967 69 80 80 86.3% 100.0%

DT Cyber Dogs T3 12,304 69 79 80 87.3% 98.8%

JH Gas Works T3 16,505 68 80 80 85.0% 100.0%

JS The Dams T3 15,551 68 77 80 88.3% 96.3%

Observations – Group Learning as Measured by Teams

Using the table above we see a consistent order of performance in the last two columns,

suggesting that sorting learning retention by “Correct Answer Total” and “Dallions®” produces a

more reliable measurement of learning retention.

Using the table above, which team members demonstrated the highest NERC CIP learning

retention?

o CF = 97.4% NERC CIP Learning Retention (% Correct Answers)

o NM = 93.8% NERC CIP Learning Retention (% Correct Answers)

o RS = 90.0% NERC CIP Learning Retention (% Correct Answers)

o MO = 90.0% NERC CIP Learning Retention (% Correct Answers)

o GA = 89.9% NERC CIP Learning Retention (% Correct Answers)

Using the table above, The team with their DALLIONS® in T1 are:

o CF = 96.3% Learning Commitment (% Questions Played)

o NM = 100.0% Learning Commitment (% Questions Played)

o RS = 98.8% Learning Commitment (% Questions Played)

o MO = 100.0% Learning Commitment (% Questions Played)

Learning Retention and Learning Commitment can be measured by team member.

Finding 11 - Learning Commitment and Learning Retention can be measured in Ringorang Learning Tournaments and measured by team member using % Correct and % Questions

Played.

2.2.2 Learning Retention Performance by Team



The table below shows NERC CIP Learning Tournament scores for Tournament 3.

Table 16 - Learning Retention Performance by Teams

Team Name Tournament Dallions®

Total

Correct Answer

Total Questions

Played


% Correct Answer

% Questions

Played

Gas Works T3 108,174 537 669 880 80.3% 76.0%

SAM T3 92,517 507 659 880 76.9% 74.9%

The Dams T3 87,049 481 630 800 76.3% 78.8%

Blue Screens of Death T3 88,747 465 596 720 78.0% 82.8%

WECC Warriors T3 89,851 430 513 720 83.8% 71.3%

Nercamania T3 73,183 428 580 800 73.8% 72.5%

Seagers CIP'ers T3 60,779 414 590 880 70.2% 67.0%

EMP T3 66,341 374 506 560 73.9% 90.4%

Cyber Dogs T3 64,324 367 458 720 80.1% 63.6%

Glacier Peak T3 47,509 339 511 720 66.3% 71.0%

Mount Adams T3 45,233 292 418 640 69.9% 65.3%

The Gorge T3 45,054 256 373 640 68.6% 58.3%

Chief Seattle T3 43,817 238 319 560 74.6% 57.0%

Rudy's Rowdies T3 36,176 201 264 400 76.1% 66.0%

Mount Baker T3 25,311 168 237 320 70.9% 74.1%

Observations

The NERC CIP Learning Team with the best overall performance can be measured by total

correct answers, Dallions®; and learning retention opportunities can be determined by studying

each team’s % Correct Answer and % Questions Played by team members.

The Top 5 performing teams as measured by learning retention are:

o Gas Works = 537 Correct Answers and 108,174 Dallions

o SAM = 507 Correct Answers and 92,517 Dallions

o The Dams = 481 Correct Answers and 87,049

o Blue Screens of Death = 465 Correct Answers and 88,747 Dallions

o WECC Warriors 430 Correct Answers and 89,851 Dallions

The Low 3 performing teams as measured by learning retention are:

o Chief Seattle = 238 Correct Answers and 43,817 Dallions

o Rudy's Rowdies = 201 Correct Answers and 36,176

o Mount Baker = 168 Correct Answers and 25,311 Dallions

Analyzing these three teams’ team members learning retention levels we learn that the greatest

opportunities for the lower performing teams to improve their performance would be to ask the

following team members to sign up for future NERC CIP Learning Tournaments or attend a NERC

CIP Compliance Training class.



Table 17 – Learning Retention Performance by Team Members

Team Name User's Initials Tournament

Dallions® Total

Correct Answer

Total Questions

Played

Total Available


% Correct Answer

% Questions

Played

Chief Seattle TW T3 2,452 13 16 80 81.3% 20.0%

Chief Seattle ST T3 8,209 43 53 80 81.1% 66.3%

Chief Seattle BW T3 11,026 55 69 80 79.7% 86.3%

Chief Seattle DM T3 5,198 39 52 80 75.0% 65.0%

Chief Seattle VC T3 13,037 52 72 80 72.2% 90.0%

Chief Seattle CC T3 3,178 23 33 80 69.7% 41.3%

Chief Seattle DM T3 717 13 24 80 54.2% 30.0%

Chief Seattle T3 43,817 238 319 560 74.6% 57.0%

Rudy's Rowdies GY T3 14,316 64 74 80 86.5% 92.5%

Rudy's Rowdies BJ T3 8,918 46 57 80 80.7% 71.3%

Rudy's Rowdies NS T3 8,492 54 74 80 73.0% 92.5%

Rudy's Rowdies JW T3 3,892 32 51 80 62.7% 63.8%

Rudy's Rowdies JP T3 558 5 8 80 62.5% 10.0%

Rudy's Rowdies T3 36,176 201 264 400 76.1% 66.0%

Mount Baker KL T3 2,267 11 14 80 78.6% 17.5%

Mount Baker JD T3 8,836 58 79 80 73.4% 98.8%

Mount Baker YL T3 6,760 50 71 80 70.4% 88.8%

Mount Baker BB T3 7,448 49 73 80 67.1% 91.3%

Mount Baker T3 25,311 168 237 320 70.9% 74.1%

Observations

Department team members with the following User Initials would benefit from additional NERC CIP

Compliance Training.

Lowest learning retention level:

o DM = 13 Correct Answers, 54.2% Correct Answers and 30% Questions Played

o JP = 5 Correct Answers, 63.5% Correct Answers and 10% Questions Played

o KL = 11 Correct Answers, 78.6% Correct and 17.5% Questions Played

o TW = 13 Correct Answers, 81.3 % Correct Answers and 20% Questions Played

o CC = 23 Correct Answers, 69.7% Correct Answers and 41.3% Questions Played

Finding 12 - Using the reports available from Ringorang™ NERC CIP Learning Tournaments, we are able to provide learning retention diagnostics by team and provide learning retention

recommendations by team members.

2.3 Can We Measure Learning Retention by Department Manager

and Team Member, and Provide Learning Diagnostic by Department?

2.3.1 Learning Retention Performance by Department Manager



Table 18 - Department Managers’ Learning Retention Performance Report

Department Manager

Dallions® Total

Questions

Played Correct Answer

Total Total Questions for Tournament

% Correct Answer

% Questions Played

Manager 19 32,865 222 168 240 75.7% 92.5%

Manager 22 37,820 295 213 320 72.2% 92.2%

Manager 10 29,828 288 202 320 70.1% 90.0%

Manager 9 20,647 142 110 160 77.5% 88.8%

Manager 16 87,843 607 469 720 77.3% 84.3%

Manager 3 85,987 757 543 960 71.7% 78.9%

Manager 17 22,054 123 105 160 85.4% 76.9%

Manager 1 10,157 59 49 80 83.1% 73.8%

Manager 5 21,260 175 133 240 76.0% 72.9%

Manager 21 27,753 224 167 320 74.6% 70.0%

Manager 12 51,548 392 286 560 73.0% 70.0%

Manager 2 10,769 110 76 160 69.1% 68.8%

Manager 11 30,619 271 182 400 67.2% 67.8%

Manager 4 53,555 430 318 640 74.0% 67.2%

Manager 8 52,706 374 286 560 76.5% 66.8%

Manager 13 8,209 53 43 80 81.1% 66.3%

Manager 18 14,431 106 85 160 80.2% 66.3%

Manager 7 43,418 369 275 560 74.5% 65.9%

Manager 20 14,949 179 110 320 61.5% 55.9%

Manager 15 44,657 362 270 720 74.6% 50.3%

Manager 6 12,186 73 60 160 82.2% 45.6%

Manager 14 3,111 32 23 80 71.9% 40.0%

Observations – Department Managers’ Learning Retention Performance

Unlike team performance, we must sort department manager learning tournament results by %

of Questions Played and % Correct Answers to find the highest and lowest performing

departments when measuring learning retention levels. This is due to the uneven size of

departments, some having as few as 1 or 2 employees participating and others having 10 or

more employees participating in the tournaments.

The Top 5 Department Managers with the greatest learning retention levels are:

o Manager 19 = 168 Correct Answers, 32865 Dallions, 75.7% Correct Answers and

92.5% Questions Played







o Manager 9 = 110 Correct Answers, 20647 Dallions, 77.5% Correct Answers and 88.8%

Questions Played



The Low 3 Department Managers who have low NERC CIP learning retention levels are:

o Manager 15 = 270 Correct Answers, 44657 Dallions, 74.6% Correct Answers, and


o Manager 6 = 60 Correct Answers, 12186 Dallions, 82.2% Correct Answers, and 45.6%

Questions Played

o Manager 14 = 32 Correct Answers, 3111 Dallions, 71.9% Correct Answers, and 40.0%

Questions Played

To diagnose the opportunities to improve learning retention at the department level, we need

to view the learning retention scores of employees in each department.

2.3.2 Learning Retention Performance by Department Team Members

The table below provides results from the NERC CIP Learning Tournaments grouped by

department manager and then sorted by User Initials. User Initials represent team members

reporting to the department manager. This report also contains the overall PSE IT Organization’s

learning retention performance scores from the Ringorang™ NERC CIP Learning Tournaments.

Table 19 - Department Detailed Learning Retention Performance Report

User's Initials

Department Manager

Dallions® Total

Questions Played

Correct Answer Total

Total Questions for Tournament

% Correct Answer

% QUESTIONS PLAYED

KS Manager 1 10,157 59 49 80 83.1% 73.8%

Manager 1 10,157 59 49 80 83.1% 73.8%

BB Manager 2 7,826 63 47 80 74.6% 78.8%

SC Manager 2 2,943 47 29 80 61.7% 58.8%

Manager 2 10,769 110 76 160 69.1% 68.8%

RH Manager 3 15,015 80 67 80 83.8% 100.0%

KS Manager 3 11,971 66 54 80 81.8% 82.5%

DB Manager 3 10,019 74 59 80 79.7% 92.5%

LP Manager 3 9,262 66 51 80 77.3% 82.5%

BH Manager 3 9,028 72 54 80 75.0% 90.0%

NS Manager 3 8,492 74 54 80 73.0% 92.5%

SM Manager 3 3,203 32 23 80 71.9% 40.0%

WS Manager 3 7,446 67 48 80 71.6% 83.8%

BS Manager 3 5,891 52 36 80 69.2% 65.0%

LR Manager 3 3,069 67 39 80 58.2% 83.8%

SN Manager 3 1,865 52 29 80 55.8% 65.0%



User's Initials

Department Manager

Dallions® Total

Questions Played



% Correct Answer

% QUESTIONS PLAYED

BJ Manager 3 726 55 29 80 52.7% 68.8%

Manager 3 85,987 757 543 960 71.7% 78.9%

DT Manager 4 12,304 79 69 80 87.3% 98.8%

MJ Manager 4 13,309 79 64 80 81.0% 98.8%

DM Manager 4 5,198 52 39 80 75.0% 65.0%

AL Manager 4 10,121 73 53 80 72.6% 91.3%

YL Manager 4 6,760 71 50 80 70.4% 88.8%

RC Manager 4 2,934 37 23 80 62.2% 46.3%

ZL Manager 4 2,525 29 17 80 58.6% 36.3%

EW Manager 4 404 10 3 80 30.0% 12.5%

Manager 4 53,555 430 318 640 74.0% 67.2%

RF Manager 5 4,076 36 29 80 80.6% 45.0%

CA Manager 5 7,872 66 50 80 75.8% 82.5%

LR Manager 5 9,312 73 54 80 74.0% 91.3%

Manager 5 21,260 175 133 240 76.0% 72.9%

TB Manager 6 10,084 56 48 80 85.7% 70.0%

TT Manager 6 2,102 17 12 80 70.6% 21.3%

Manager 6 12,186 73 60 160 82.2% 45.6%

LS Manager 7 10,362 73 61 80 83.6% 91.3%

MB Manager 7 6,491 46 37 80 80.4% 57.5%

TG Manager 7 3,975 32 24 80 75.0% 40.0%

NS Manager 7 6,092 55 40 80 72.7% 68.8%

JM Manager 7 3,673 33 24 80 72.7% 41.3%

JL Manager 7 8,933 79 57 80 72.2% 98.8%

JW Manager 7 3,892 51 32 80 62.7% 63.8%

Manager 7 43,418 369 275 560 74.5% 65.9%

JS Manager 8 15,551 77 68 80 88.3% 96.3%

GY Manager 8 14,316 74 64 80 86.5% 92.5%

KL Manager 8 2,267 14 11 80 78.6% 17.5%

DS Manager 8 3,691 32 24 80 75.0% 40.0%

TB Manager 8 8,888 73 51 80 69.9% 91.3%

CB Manager 8 2,451 32 22 80 68.8% 40.0%

AH Manager 8 5,542 72 46 80 63.9% 90.0%

Manager 8 52,706 374 286 560 76.5% 66.8%

BW Manager 9 11,026 69 55 80 79.7% 86.3%

KP Manager 9 9,621 73 55 80 75.3% 91.3%

Manager 9 20,647 142 110 160 77.5% 88.8%

JF Manager 10 9,381 77 57 80 74.0% 96.3%

RM Manager 10 6,804 64 46 80 71.9% 80.0%

SW Manager 10 6,195 74 50 80 67.6% 92.5%



User's Initials

Department Manager

Dallions® Total

Questions Played



% Correct Answer

% QUESTIONS PLAYED

BB Manager 10 7,448 73 49 80 67.1% 91.3%

Manager 10 29,828 288 202 320 70.1% 90.0%

AT Manager 11 16,177 79 71 80 89.9% 98.8%

YY Manager 11 2,816 37 25 80 67.6% 46.3%

BD Manager 11 5,193 62 40 80 64.5% 77.5%

IH Manager 11 3,612 58 35 80 60.3% 72.5%

KP Manager 11 2,821 35 11 80 31.4% 43.8%

Manager 11 30,619 271 182 400 67.2% 67.8%

CF Manager 12 20,109 77 75 80 97.4% 96.3%

KF Manager 12 10,089 70 55 80 78.6% 87.5%

JW Manager 12 8,878 57 44 80 77.2% 71.3%

CH Manager 12 5,923 75 49 80 65.3% 93.8%

DM Manager 12 2,549 48 29 80 60.4% 60.0%

CC Manager 12 1,374 32 19 80 59.4% 40.0%

CS Manager 12 2,626 33 15 80 45.5% 41.3%

Manager 12 51,548 392 286 560 73.0% 70.0%

ST Manager 13 8,209 53 43 80 81.1% 66.3%

Manager 13 8,209 53 43 80 81.1% 66.3%

JB Manager 14 3,111 32 23 80 71.9% 40.0%

Manager 14 3,111 32 23 80 71.9% 40.0%

JH Manager 15 16,505 80 68 80 85.0% 100.0%

TW Manager 15 2,452 16 13 80 81.3% 20.0%

JL Manager 15 3,281 32 24 80 75.0% 40.0%

WJ Manager 15 8,249 73 54 80 74.0% 91.3%

IS Manager 15 7,821 67 49 80 73.1% 83.8%

AD Manager 15 2,437 35 25 80 71.4% 43.8%

BS Manager 15 3,053 33 23 80 69.7% 41.3%

DM Manager 15 717 24 13 80 54.2% 30.0%

JK Manager 15 142 2 1 80 50.0% 2.5%

Manager 15 44,657 362 270 720 74.6% 50.3%

NM Manager 16 19,068 80 75 80 93.8% 100.0%

RS Manager 16 16,211 79 74 80 93.7% 98.8%

BB Manager 16 13,122 75 63 80 84.0% 93.8%

JM Manager 16 9,939 72 59 80 81.9% 90.0%

SW Manager 16 12,807 79 61 80 77.2% 98.8%

TS Manager 16 5,402 73 49 80 67.1% 91.3%

BB Manager 16 4,746 73 46 80 63.0% 91.3%

JP Manager 16 558 8 5 80 62.5% 10.0%

AJ Manager 16 5,990 68 37 80 54.4% 85.0%

Manager 16 87,843 607 469 720 77.3% 84.3%



User's Initials

Department Manager

Dallions® Total

Questions Played



% Correct Answer

% QUESTIONS PLAYED

PS Manager 17 9,022 50 43 80 86.0% 62.5%

DC Manager 17 13,032 73 62 80 84.9% 91.3%

Manager 17 22,054 123 105 160 85.4% 76.9%

SE Manager 18 6,265 33 33 80 100.0% 41.3%

RN Manager 18 8,166 73 52 80 71.2% 91.3%

Manager 18 14,431 106 85 160 80.2% 66.3%

MO Manager 19 17,599 80 72 80 90.0% 100.0%

FA Manager 19 13,967 80 69 80 86.3% 100.0%

JR Manager 19 1,299 62 27 80 43.5% 77.5%

Manager 19 32,865 222 168 240 75.7% 92.5%

TD Manager 20 2,438 15 13 80 86.7% 18.8%

WE Manager 20 5,775 53 38 80 71.7% 66.3%

DR Manager 20 4,990 65 42 80 64.6% 81.3%

MC Manager 20 1,746 46 17 80 37.0% 57.5%

Manager 20 14,949 179 110 320 61.5% 55.9%

BJ Manager 21 8,918 57 46 80 80.7% 71.3%

JW Manager 21 8,722 66 50 80 75.8% 82.5%

VP Manager 21 6,935 68 48 80 70.6% 85.0%

CC Manager 21 3,178 33 23 80 69.7% 41.3%

Manager 21 27,753 224 167 320 74.6% 70.0%

JD Manager 22 8,836 79 58 80 73.4% 98.8%

II Manager 22 7,327 69 50 80 72.5% 86.3%

VC Manager 22 13,037 72 52 80 72.2% 90.0%

QL Manager 22 8,620 75 53 80 70.7% 93.8%

Manager 22 37,820 295 213 320 72.2% 92.2%

IT Organization 716,372 5,643 4,173 7,920 74.0% 71.3%

Observations

Using our color coding scheme for learning retention performance, it is quite easy to identify the

learning retention opportunities for improvement.

The following table shows the department team members with the highest NERC CIP learning

retention scores.

Table 20 - Highest Retention Performance Sores by Department Team Member

User's Initials

Department Manager

Dallions® Total

Questions



% Correct Answer

% QUESTIONS

PLAYED

CF Manager 12 20,109 77 75 80 97.4% 96.3%

NM Manager 16 19,068 80 75 80 93.8% 100.0%

RS Manager 16 16,211 79 74 80 93.7% 98.8%

MO Manager 19 17,599 80 72 80 90.0% 100.0%



User's Initials

Department Manager

Dallions® Total

Questions



% Correct Answer

% QUESTIONS

PLAYED

AT Manager 11 16,177 79 71 80 89.9% 98.8%

FA Manager 11 13,967 80 69 80 86.3% 100.0%

DT Manager 4 12,304 79 69 80 87.3% 98.8%

JH Manager 15 16,505 80 68 80 85.0% 100.0%

JS Manager 8 15,551 77 68 80 88.3% 96.3%

RH Manager 3 15,015 80 67 80 83.8% 100.0%

GY Manager 3 14,316 74 64 80 86.5% 92.5%

MJ Manager 3 13,309 79 64 80 81.0% 98.8%

BB Manager 3 13,122 75 63 80 84.0% 93.8%

DC Manager 3 13,032 73 62 80 84.9% 91.3%

SW Manager 3 12,807 79 61 80 77.2% 98.8%

LS Manager 7 10,362 73 61 80 83.6% 91.3%

DB Manager 7 10,019 74 59 80 79.7% 92.5%

JM Manager 7 9,939 72 59 80 81.9% 90.0%

JD Manager 22 8,836 79 58 80 73.4% 98.8%

JF Manager 10 9,381 77 57 80 74.0% 96.3%

JL Manager 10 8,933 79 57 80 72.2% 98.8%

BW Manager 9 11,026 69 55 80 79.7% 86.3%

KF Manager 9 10,089 70 55 80 78.6% 87.5%

KP Manager 9 9,621 73 55 80 75.3% 91.3%

KS Manager 9 11,971 66 54 80 81.8% 82.5%

LR Manager 9 9,312 73 54 80 74.0% 91.3%

BH Manager 9 9,028 72 54 80 75.0% 90.0%

NS Manager 9 8,492 74 54 80 73.0% 92.5%

WJ Manager 9 8,249 73 54 80 74.0% 91.3%

AL Manager 9 10,121 73 53 80 72.6% 91.3%

QL Manager 9 8,620 75 53 80 70.7% 93.8%

VC Manager 9 13,037 72 52 80 72.2% 90.0%

RN Manager 9 8,166 73 52 80 71.2% 91.3%

LP Manager 9 9,262 66 51 80 77.3% 82.5%

TB Manager 9 8,888 73 51 80 69.9% 91.3%

JW Manager 9 8,722 66 50 80 75.8% 82.5%

CA Manager 9 7,872 66 50 80 75.8% 82.5%

II Manager 9 7,327 69 50 80 72.5% 86.3%

YL Manager 9 6,760 71 50 80 70.4% 88.8%

SW Manager 9 6,195 74 50 80 67.6% 92.5%

KS Manager 1 10,157 59 49 80 83.1% 73.8%

IS Manager 1 7,821 67 49 80 73.1% 83.8%

Observations

The individuals listed above have demonstrated high NERC CIP learning retention

performance. They could be exempted from participating in a future NERC CIP Learning

Tournament; or they could be certified as being proficient NERC CIP Compliance Training.

The following table shows the department team members with the lowest NERC CIP learning

retention scores.



Table 21 - Lowest Retention Performance Sores by Department Team Member

User's Initials

Department Manager

Dallions®

Total Questions



% Correct Answer

% QUESTIONS PLAYED

RC Manager 14 2,934 37 23 80 62.2% 46.3%

CB Manager 14 2,451 32 22 80 68.8% 40.0%

CC Manager 14 1,374 32 19 80 59.4% 40.0%

ZL Manager 14 2,525 29 17 80 58.6% 36.3%

MC Manager 14 1,746 46 17 80 37.0% 57.5%

CS Manager 14 2,626 33 15 80 45.5% 41.3%

TW Manager 14 2,452 16 13 80 81.3% 20.0%

TD Manager 20 2,438 15 13 80 86.7% 18.8%

DM Manager 20 717 24 13 80 54.2% 30.0%

TT Manager 20 2,102 17 12 80 70.6% 21.3%

KP Manager 20 2,821 35 11 80 31.4% 43.8%

KL Manager 20 2,267 14 11 80 78.6% 17.5%

JP Manager 20 558 8 5 80 62.5% 10.0%

EW Manager 20 404 10 3 80 30.0% 12.5%

JK Manager 20 142 2 1 80 50.0% 2.5%

Observations

The individuals listed above would benefit from additional exposure to NERC CIP curriculum

either by playing in future NERC CIP Learning Tournaments or attending a NERC CIP

Compliance Training class.

Finding 13 - Using the reports available from Ringorang™ NERC CIP Learning Tournaments, we are able to provide learning retention diagnostics by department managers and provide

learning retention recommendations by team members.

2.3.3 Which Leader Impacts Learning Retention More?

We thought it would be interesting to contrast the performances of Team Captains versus Department

Managers. The two tables below show performances of both. Which table contains the lowest range of

learning retention scores and variations in color?

Table 22 - Learning Retention Performance by Teams

Team Name Dallions® Total

Correct Answer

Total Questions

Played

Total Available


% Correct Answer

% Questions

Played

Gas Works 108,174 537 669 880 80.3% 76.0%

SAM 92,517 507 659 880 76.9% 74.9%

The Dams 87,049 481 630 800 76.3% 78.8%

Blue Screens of Death 88,747 465 596 720 78.0% 82.8%

WECC Warriors 89,851 430 513 720 83.8% 71.3%

Nercamania 73,183 428 580 800 73.8% 72.5%

Seagers CIP'ers 60,779 414 590 880 70.2% 67.0%

EMP 66,341 374 506 560 73.9% 90.4%

Cyber Dogs 64,324 367 458 720 80.1% 63.6%

Glacier Peak 47,509 339 511 720 66.3% 71.0%

Mount Adams 45,233 292 418 640 69.9% 65.3%



Team Name Dallions® Total

Correct Answer

Total Questions

Played

Total Available


% Correct Answer

% Questions

Played

The Gorge 45,054 256 373 640 68.6% 58.3%

Chief Seattle 43,817 238 319 560 74.6% 57.0%

Rudy's Rowdies 36,176 201 264 400 76.1% 66.0%

Mount Baker 25,311 168 237 320 70.9% 74.1%

Table 23 - Department Managers’ Learning Retention Performance Report

Department Manager Dallions® Total

Questions

Played

Correct Answer

Total

Total Questions

for Tournament

% Correct Answer

% QUESTIONS

PLAYED

Manager 19 32,865 222 168 240 75.7% 92.5%

Manager 22 37,820 295 213 320 72.2% 92.2%

Manager 10 29,828 288 202 320 70.1% 90.0%

Manager 9 20,647 142 110 160 77.5% 88.8%

Manager 16 87,843 607 469 720 77.3% 84.3%

Manager 3 85,987 757 543 960 71.7% 78.9%

Manager 17 22,054 123 105 160 85.4% 76.9%

Manager 1 10,157 59 49 80 83.1% 73.8%

Manager 5 21,260 175 133 240 76.0% 72.9%

Manager 21 27,753 224 167 320 74.6% 70.0%

Manager 12 51,548 392 286 560 73.0% 70.0%

Manager 2 10,769 110 76 160 69.1% 68.8%

Manager 11 30,619 271 182 400 67.2% 67.8%

Manager 4 53,555 430 318 640 74.0% 67.2%

Manager 8 52,706 374 286 560 76.5% 66.8%

Manager 13 8,209 53 43 80 81.1% 66.3%

Manager 18 14,431 106 85 160 80.2% 66.3%

Manager 7 43,418 369 275 560 74.5% 65.9%

Manager 20 14,949 179 110 320 61.5% 55.9%

Manager 15 44,657 362 270 720 74.6% 50.3%

Manager 6 12,186 73 60 160 82.2% 45.6%

Manager 14 3,111 32 23 80 71.9% 40.0%

Observations – Team Captains versus Department Managers’ Learning Retention Performance

Clearly there is less range in scores and variations in color in the Team Captains’ Learning

Retention Performance table.

Team Captains have a greater influence on Ringorang™ NERC CIP Learning Tournament

results.



Finding 14 – At PSE and based on the method of deployment by Farallon Risk Group and the coaching provided by Vergence Entertainment, Team Captains have a greater influence on learning retention performance in NERC CIP Learning Tournaments than do Department

Managers.

3.1 Team Comparison

Figure 3- Team Performance by Tournament - Points

Observation

The team performance graph above plots Teams on the x-axes and DALLIONS® on the y-axes.

Team competition causes a higher commitment to participate and greater focus to learn and

retain the curriculum.



Figure 4 - Team Ranking by Tournament

Observation

Each team has either a single triangular bullet, or a vertical line with a triangular bullet. The

triangular bullet represents the team’s ranking for Tournament 3. The vertical line represents

the spread between their best and worst ranking for the three tournaments.

The Gas Works team won all three tournaments, and had the least variation in team ranking

as noted by having no vertical line.

WECC Warriors had the greatest spread between its best and worst team ranking, yet

improved its ranking much by the end of Tournament 3.

Finding 15 - Farallon Risk Group and Vergence Entertainment team members optimized learning retention results by using competitive themes, learning teams, and Ringorang Team

Captains.

2.5 Do Participants Perform Better with, or without, Prior Training?

To measure the effectiveness of completing NERC CIP Compliance Training or not completing NERC CIP

training prior to playing in the Ringorang™ NERC CIP Learning Tournaments, we cross compared the Top

1

2

3

6

5

4

10

8

9

14

7

11

16

15

17

20

1

2

5

3

6

8

4

15

14

7

10

11

9

13

12

20

1

2

4

8

6

5

9

3

7

11

15

12

10

13

14

16

1

3

5

7

9

11

13

15

17

19

21

Team Ranking and Range for Three Tournaments

Rank - T1

Rank - T2

Rank - T3



20 Winners of Tournament 3 against their completion dates of NERC CIP Compliance Training. The table

below shows the completion dates and notes on who did not complete NERC CIP Compliance Training.

Tournament 3 – Top 20 Participants Performance

Individuals’ DALLIONS® Final Results for T3 (June 20th – 24th)

Rank User's Initials DALLIONS® Training

1 FC 20,109 No prior tra ining

2 MN 19,068 4/25/2011 just prior to T1

3 OM 17,599 No prior tra ining

4 IT 16,518 1/18/2011

5 HJ 16,505 1/10/2011

6 SP 16,215 5/13/2011 after T2 and before T3

7 SR 16,211 5/2/2011 at s tart of T1

8 TA 16,177 5/4/2011 during T1

9 AG 15,814 3/3/2011

10 SJ 15,551 4/27/2011 just prior to T1

11 HR 15,015 No prior tra ining

12 GM 14,326 2/8/2011

13 YG 14,316 6/13/2011 after T2 and before T3

14 FD 13,967 4/12/2011

15 JM 13,309 4/29/2011 just prior to T1

16 BB 13,122 No prior tra ining

17 GL 13,079 No prior tra ining

18 CV 13,037 1/19/2011

19 CD 13,032 1/11/2011

20 PP 12,814 6/2/2011 at end of T2

Observations – Top 20 Winners Compliance Training Completion Dates

Seven of the Top 20 winners completed Compliance Training on NERC CIP more than a

month prior to Tournament 1. We refer to this group as “Prior Compliance Training”.

Eight of the Top 20 winners completed NERC CIP Compliance Training just prior or during

tournament play. We refer to this group as “Impulsive Compliance Training”.

Five of the Top 20 winners from Tournament #3 completed No Training prior to or during

the tournaments. We refer to this group as “No Training”.

Of the five users in the Top 20 Winners list who completed No Training on NERC CIP, two

finished in the Top 5. The first place and third place winners completed No Training prior or

during the NERC CIP Learning Tournaments.

The graph below shows the training distribution of Top 20 Winners in Tournament 3.



Figure 5 - NERC CIP Compliance Training vs No Training Distribution

Observations – NERC CIP Compliance Training versus No NERC CIP Compliance Training

There appears to be no significant difference between those participants who complete

Prior Compliance Training, Impulsive Compliance Training, and No Training.

Finding 16 - Data from the NERC CIP Learning Tournaments, managed by Farallon Risk Group and Vergence Entertainment, suggests completion of compliance training is not a prerequisite

to perform well in Ringorang™ Learning Tournaments.

3.0 Is Ringorang™ an Effective Curriculum Diagnostic Tool

3.1 Can we create measures and provide quality improvement

reports on learning curriculum?

We can organize curriculum content by topic and subtopics. We refer to each unique question in the

Ringorang™ game as a curriculum learning element (CLE).

Table 24 - Curriculum Learning Elements (CLEs) by Topics and Subtopics

ID Question Topic Subtopic

1 PSE must document changes to its Cyber Security Policy and submit to the Senior manager responsible for NERC CIP within: CIP 003 R2

2 PSE must update each Personnel Risk Assessment for individuals with NERC CIP access at what frequency? CIP 004 R3

4 Outputs of the cyber vulnerability assessment of Electronic Security Perimeters MUST include: CIP 005 R4

3 Cyber Assets that authorize and/or log access to the Physical Security Perimeters must (____). CIP 006 R2

Prior Compliance

Training 35%

Impulsive Compliance

Training40%

No Training

25%

NERC CIP Compliance Training Distribution

Two of the Top 3 Winners completed No NERC CIP

Training.



ID Question Topic Subtopic

5 At a minimum, PSE is required to review user accounts annually to verify access privileges are in accordance with (____)? CIP 007 R5

Table 25 – Measuring Responses to CLE Questions

Curriculum Learning Elements Are Measured by Each Participant's Response to Each Question

Questions: PSE must document … PSE must update… Cyber Assets that authorize …

User's Initials

AT Incorrect Correct Correct

BM Correct Correct Correct

BW Correct Incorrect Correct

BB Unplayed Unplayed Unplayed

… … … …

And we can we track learning retention results of each CLE by the number of participants who

received the question, answered it, correctly answered it, and who answered incorrectly.

Table 26 - Question Quality Report

ID Question Received Answered Correct %correct Incorrect Not

Answered Level of

Difficulty Tags

1 PSE must document … 75 66 52 79% 14 9 1 CIP 003 R2

2 PSE must update … 75 72 25 35% 47 3 1 CIP 004 R3

3 Cyber Assets that… 77 65 56 86% 9 12 3 CIP 006 R2

4 Outputs of the cyber … 78 69 56 81% 13 9 3 CIP 005 R4

5 At a minimum, PSE… 86 57 31 54% 26 29 5 CIP 007 R5

6 Cyber Assets used... 94 76 58 76% 18 18 5 CIP 006 R3

7 Documentation related… 95 87 84 97% 3 8 3 CIP 008 R2

… … … … … … … … … …

3.2 Can Ringorang™ Track Learning Retention Results by CLEs and

Offer Curriculum Quality Improvement Suggestions?

We carefully analyzed the results of the PSE-IT Ringorang™ NERC CIP Learning Tournaments. The table

below shows the learning retention levels achieved at PSE by NERC CIP topics and subtopics. Learning

retention gaps are shown in orange through red performance scores.

Table 27 - Diagnostic Report on Learning Retention by Topic

Diagnostic Report of Learning Retention Results by Topics

Answered Correct % Correct Performance by Topics

3336 2764 82.9% CIP 002 Topic Total

2773 2289 82.5% CIP 003 Topic Total





3868 2886 74.6% CIP 004 Topic Total

2414 1696 70.3% CIP 005 Topic Total

2404 1608 66.9% CIP 006 Topic Total

3685 2655 72.0% CIP 007 Topic Total

1428 1234 86.4% CIP 008 Topic Total

1307 1130 86.5% CIP 009 Topic Total

499 384 77.0% Awareness & Leadership Messages Topic Total

Observation

CIP 008 and CIP 009 had the highest learning retention scores.

CIP 006, CIP 005 and CIP 007 had the lowest learning retention scores.

Awareness and Leadership Messages scored 77%, but we need more information to

understand this topic.

We need to explore subtopics to really understand quality of learning retention results of

our learning curriculum. We expanded our analysis to include each subtopic. See the results

below.

Table 28 - Diagnostic Report on Learning Retention Results by Subtopic



1523 1253 82.30% CIP 002 R1

875 733 83.80% CIP 002 R2

501 427 85.20% CIP 002 R3

437 351 80.30% CIP 002 R4

3336 2764 82.90% CIP 002 Topic Total

780 683 87.60% CIP 003 R1

254 200 78.70% CIP 003 R2

809 601 74.30% CIP 003 R3

764 674 88.20% CIP 003 R4

166 131 78.90% CIP 003 R5

2773 2289 82.50% CIP 003 Topic Total

548 474 86.50% CIP 004 General Information

835 505 60.50% CIP 004 R1

383 320 83.60% CIP 004 R2

439 245 55.80% CIP 004 R3

1663 1342 80.70% CIP 004 R4

3868 2886 74.60% CIP 004 Topic Total

255 171 67.10% CIP 005 R1

459 360 78.40% CIP 005 R2

366 260 71.00% CIP 005 R3

615 466 75.80% CIP 005 R4

719 439 61.10% CIP 005 R5

2414 1696 70.30% CIP 005 Topic Total






380 316 83.20% CIP 006 R1

107 77 72.00% CIP 006 R3

218 211 96.80% CIP 006 R4

170 156 91.80% CIP 006 R6

315 231 73.30% CIP 006 R7

794 278 35.00% CIP 006 R8

2404 1608 66.90% CIP 006 Topic Total


378 266 70.40% CIP 007 R1

261 160 61.30% CIP 007 R2

206 150 72.80% CIP 007 R3

773 661 85.50% CIP 007 R4

1063 650 61.10% CIP 007 R5

410 338 82.40% CIP 007 R8

173 129 74.60% CIP 007 R9

3685 2655 72.00% CIP 007 Topic Total


333 289 86.80% CIP 008 R1

908 807 88.90% CIP 008 R2

1428 1234 86.40% CIP 008 Topic Total


296 282 95.30% CIP 009 R1

368 310 84.20% CIP 009 R2

216 192 88.90% CIP 009 R3

148 148 100.00% CIP 009 R4

125 92 73.60% CIP 009 R5

1307 1130 86.50% CIP 009 Topic Total

294 261 88.80% Al l Hands Meeting - Leadership Messages

205 123 60.00% NERC CIP Compl iance Awareness Posters

499 384 77.00% Awareness & Leadership Messages Topic Total

Observations

Several subtopics had excellent or higher learning retention results.

o Superb: CIP 009 R4

o Outstanding: CIP 009 R1, CIP 006 R4

o Excellent: CIP 6 R6

Five subtopics had Low to Poor learning retention results.

o Poor: CIP 005 R5, CIP 009

o Very Poor: CIP 004 R3, Compliance Posters

o Terrible: CIP 6 R8



Finding 17 – Using Ringorang™ Learning Retention reports, Farallon Risk Group and Vergence Entertainment were able to provide meaningful analytics by curriculum topic and subtopics,

and we could identify learning gaps, and offer recommendations to fill learning gaps.

Finding 18 - Data from Ringorang™ NERC CIP Learning Tournaments, based on the CLEs written by Vergence Entertainment and deployed by Farallon Risk Group, suggests compliance

awareness posters have a poor to bad learning retention rate.

4. How Does Competition Impact Learning Retention Results?

4.1 Individual Competition

4.1.1 Individuals - How Did Learning Retention Change Over a Span of Three

Tournaments?

Table 30 – change %Correct of all players for All Three Tournamnet

Observations

Knowledge retention increased by: repeating questions based on value to organization;

Optimizing replay of low performing questions.

4.1.2 Tournament 1 – Individual Competition Results, Observations and Findings



Table 31 - Individuals' Participation Final Results for T1

Individuals' Participation Final Results for T1

Rank User's Initials Participation % #Correct

1 MA 100.00% 126

2 SP 100.00% 90

3 FD 99.24% 91

4 HJ 98.47% 105

5 JW 97.71% 89

6 IT 95.42% 110

7 GL 95.42% 108

8 SR 94.66% 89

9 JM 94.66% 88

10 HB 92.37% 83

11 FC 91.60% 101

12 LQ 91.60% 89

13 LK 91.60% 87

14 HR 91.60% 83

15 SJ 90.08% 96

16 DJ 90.08% 80

17 BD 89.31% 106

18 RL 89.31% 85

19 AG 87.79% 104

20 CV 87.79% 68

Observations –Top 20 Winners Participation% and #correct

Two of the 20 winners answered all questions available for tournament 1.

For tournament 1, we had numerous ties on participation. We broke the ties on number of

correct answers chosen by the participants.



Over half of the participants do not work in the Information Security organization, rather for

other departments in IT.


Table 32 - Individuals' Correct Answer Final Results for T2

Individuals' Correct Answer Final Results for T2

Rank User's Initials # Correct Dallions®

1 IT 62 15420

2 MN 62 13925

3 AG 56 11032

4 FC 56 10961

5 GL 56 10760

6 BW 55 12230

7 HJ 54 12559

8 SJ 54 11454

9 MC 53 9548

10 FD 52 8813

11 SR 52 8384

12 TD 52 8195

13 SP 51 9750

14 SC 50 11383

15 KJ 50 8239

16 TA 48 8811

17 KD 47 9736

18 YG 47 8963

19 BC 46 8840

20 BD 46 8746

Observations–Top 20 Winners #correct



For tournament 2, we had numerous ties on #correct. We broke the ties by Total Dallions

chosen by the participants.

The spread in points between the top 20 scoring individuals is less than anticipated;

meaning, the top 20 were competitively trying to win.

We had 6 ties. We broke the ties using the number of points earned, Dallions.


Table 33 - Individuals' Dollions Final Results for T3

Individuals' Dollions Final Results for T3

Rank User's Initials Dallions®

1 FC 20109

2 MN 19068

3 OM 17599

4 IT 16518

5 HJ 16505

6 SP 16215

7 SR 16211

8 TA 16177

9 AG 15814

10 SJ 15551

11 HR 15015

12 GM 14326

13 YG 14316

14 FD 13967

15 JM 13309

16 BB 13122

17 GL 13079

18 CV 13037

19 CD 13032

20 PP 12814



Table 33 - Top scoring individual not on the winning team for T3

Top scoring individual not on the winning team for T3

Rank User's Initials Team Name Dallions®

1 YG Rudy's Rowdies 14316

2 SR Cyber Dogs 14195

3 FD Nercamania 13967

4 JM EMP 13309

5 BB Nercamania 13122

Observations–Top 20 Winners Dallions® and Top scoring individual not on the winning team

To provide more incentive for our employees who maybe not on the not very active team to

work hard.

Finding 20：Using Ringorang™ NERC CIP Tournament leaderboard reports, Farallon Risk

Group and Vergence Entertainment, suggests the participants work in the Information

Security organization not a prerequisite to perform well in Ringorang™ Learning Tournaments.

4.2 Team Competition

4.2.1 Tournament 1 – Team Competition Results, Observations and Findings

Table 34 - T1 - Team Winners for Participation%

T1 - Team Winners for Participation%

Rank Team Name Participation %

1 Gas Works 80.82%

2 Blue Screens of Death 80.34%



3 SAM 73.19%

4 The Dams 67.84%

5 Nercamania 58.59%

6 Cyber Dogs 53.63%

7 Mount Baker 48.95%

8 WECC Warriors 48.76%

9 EMP 47.14%

10 Seagers CIP'ers 46.85%

11 The Gorge 45.80%

12 Mount Rainier 42.65%

13 DJ NERC and the CIP-Tones 42.56%

14 Glacier Peak 41.22%

15 Mount Adams 37.02%

16 Chief Seattle 33.49%

17 Rudy's Rowdies 28.82%

18 Reservoir Dogs 27.10%

19 Snoqualmie Falls 11.93%

Observations –Team leaderboard for participation %

We had a very tight finish up to and including the last hours of the tournament.

The Gas Works team placed also had four of their team members ranked in the top 20 for individual participation winners.

We had two teams vying for the top position. We had four teams vying for 3rd to 6th place.

A few teams just could not get their team members to participate at the same level of the other teams.


Table 35 -T2 - Team Winners for Correct Answer



T2 - Team Winners for Correct Answer

Rank Team name Team #Correct

1 Gas Works 411

2 Blue Screens of Death 309

3 Cyber Dogs 303

4 Seagers CIP'ers 224

5 SAM 217

6 Nercamania 202

7 Glacier Peak 189

8 The Dams 187

9 Chief Seattle 170

10 Mount Baker 148

11 The Gorge 147

12 Rudy's Rowdies 141

13 Mount Adams 112

14 EMP 110

15 WECC Warriors 109

16 Mount Rainier 92

17 Snoqualmie Falls 91

18 DJ NERC and the CIP-Tones 82

19 Reservoir Dogs 70

Observations –Team leaderboard for correct answer

Participants on teams felt a high sense of obligation to complete their Ringorang questions each day from a sense of duty to support their team.


Table 36 - T3 - Team Winners for Dallions®



T3 - Team Winners for Dallions

Rank Team Name Dallions®

1 Gas Works 101286

2 Blue Screens of Death 88274

3 WECC Warriors 87447

4 SAM 85815

5 The Dams 80131

6 Nercamania 70604

7 EMP 68087

8 Cyber Dogs 61508

9 Seagers CIPers 57962

10 Chief Seattle 47140

11 Glacier Peak 45058

12 The Gorge 45054

13 Mount Adams 43131

14 Rudy's Rowdies 36176

15 Mount Baker 25701

Observations –Team leaderboard for Dallions®

We had 15 teams in the last tournament, combined some teams with more inactivity users. Many of the team captains finished in the top 20. We chose captains semi randomly. They were not chosen on their NERC CIP knowledge level, but rather on their ability to collaborate and communicate effectively with their peers.

Finding 21：Team Captains drive participation. Teams create a fun, competitive environment.

Competition drives intensity and interest level. Team competition causes a higher sense of

obligation to know the content and perform well.



5. Survey Feedback

5.1 Description of the Survey Process

PSE IT hosted 3 Ringorang interactive learning NERC CIP games. The purpose was to increase NERC CIP

awareness and to help reduce the number of self-reports. Survey Monkey Surveys were conducted after

the first and third tournaments.

5.1.1 What were the objectives of the surveys?

Helping to make the Ringorang experience enjoyable and “It’s All about Learning!”

5.1.2 What was the method we used to survey participants?

According to the different needs of the survey questionnaire, design use computer programs

way to realized network a questionnaire.

5.1.3 Why did we have two surveys: baseline and final surveys?

Continues to collect customer feedback is more meaningful on different date.

5.1.4 What Was the Distribution of Participants Completing the Surveys?

There were a good number of responses from the participants. The first survey received 19%

responses (38 responses out of 200 participants. The second survey received 18.5% responses

(37 responses out of 200).

Table 37 - Ringorang NERC CIP Survey Distribution

Survey Distribution Baseline Survey

Final Survey

How Many Participants Were Invited to Complete the Survey? 200 200

Number of Participants Who Chose to Complete the Surveys? 38 37

% of Participants Invited versus Completed Surveys?

Change in % Participants Invited versus Completed Surveys? 19% 18.5%

5.2 Baseline Survey Feedback – Sent After Tournament 1

*S1-T1: Survey 1 – Tournament #1; S2-T3: Survey 2 – Tournament #3

5.2.1 Did your team captain make a positive difference on your team’s

participation level during this tournament?

Did your team captain make a positive difference on your team’s participation level during this tournament?



S1-T1 Response Response

Percent Count Yes 73.70% 28

No 26.30% 10

total 100.00% 38

Observations

Positive: 73.7% of the survey participants said their team captain make a positive difference on their team's participation level during tournament 3.

Participants insert comment: o “My captain was encouraging.” o “Participation and encouragement to use the tool as a fun mini training breaks while

retaining the PSE's NERC-CIP retention of Critical Asset security information.” o “Regular encouraging e-mails.”

Analysis

It appears that more captains drove participation.

Many team captains went the extra mails to help their whole team perform well.

5.2.2 How likely are you to recommend Ringorang as a productive learning game

to other departments at PSE?

How likely are you to recommend Ringorang as a productive learning game to other departments at PSE?

S1 - T1

Did your team captain make a positive difference on your team’s

participation level during this tournament?(S1-T1)

Yes, 73.70%

No, 26.30%



Response Response

Percent Count

Definitely will recommend 7.90% 3

Probably will recommend 18.40% 7

Not sure 39.50% 15

Probably will not recommend 7.90% 3

Definitely will not recommend 23.70% 9

Never Used 2.60% 1

total 100.00% 38

Observations

Positive: 26.10% of the survey participants said they will recommend Ringorang as a productive learning game to other departments at PSE.

Analysis

It appears that less positive responses on recommending Ringorang as a productive learning game to other departments at PSE.

5.2.3How likely are you to recommend the Ringorang learning game as a

productive method to learn?

How likely are you to recommend the Ringorang learning game as a productive method to learn?

S1 -T1

Response Response

Percent Count

Definitely will recommend 7.90% 3

Probably will recommend 21.10% 8

Not sure 39.50% 15

Probably will not recommend 5.30% 2

Definitely will not recommend 23.70% 9

Never Used 2.60% 1

total 100.00% 38

Observations

Not sure: 39.50% of the survey participants said they are not sure to recommend the Ringorang learning game as a productive method to learn.

5.3 Final Survey Feedback – Sent After Tournament 3

5.3.1 Do you feel this enhanced your understanding of NERC CIP?

Do you feel this enhanced your understanding of NERC CIP?

S2 -T3

Response Response

Percent Count

Yes 62.20% 23

No 37.80% 14



total 100.00% 37

Observations

Positive: 62.2% of the survey participants said this enhanced their understanding of NERC CIP.

Participants insert comment: o “Only having very limited NERC CIP training before, I feel as though I enhanced my overall

understanding and awareness of NERC CIP and its relevance to PSE.” o “Through repetition of some of the questions, I learned more about NERC/CIP.” o “By reinforcing the regulations and rules.”

Analysis

Ringorang increased learning.

5.3.2Where do you think you gained the most benefit as a result of participating

in the tournament?

Where do you think you gained the most benefit as a result of participating in the tournament?

S2 -T3

Response Response

Percent Count

Improved general awareness of NERC CIP issues 56.80% 21

Reinforcement of past learning 35.10% 13 New learning 35.10% 13

Analysis

Participants think they gained the most benefit as a result of participating in the tournament is

improved general awareness of NERC CIP issues (56.8%).

Do you feel this enhanced your understanding of NERC CIP?

Yes, 62.20%

No, 37.80%



5.3.3 Do you have any recommendations to improve the Ringorang learning

experience?

Do you have any recommendations to improve the Ringorang learning experience?( S2 – T3)

Observations

Participants insert comment: o “Little more time - perhaps to enable looking up the answer versus rushing through or

cheating. o “Instead of having to leave the webpage open all of the time, how about a desktop

widget or something - all I really care about is that little flash thingy. Possibly have it in the windows icon tray and have it pop up a toast message like outlook does.”

o “Have a "learn at your own pace" without timing to see how you do.” Analysis

Participants found it not easy to answer questions within such a short time.

Participants prefer that there is another way to alert players that a question is coming without sound, possibly have it in the windows icon tray and have it pop up a toast message like outlook does.

5.4 Baseline Survey and Final Survey Feedback of the Same

Questions

How likely are you to recommend Ringorang as a productive learning game to other departments at PSE?

Survey 1-Tournament 1 Survey 2-Tournament 3 Change

Response Response Response Response Response Response

Percent Count Percent Count Percent Count Definitely will recommend 8.11% 3 13.51% 5 5.41% 2

Probably will recommend 18.92% 7 21.62% 8 2.70% 1 Not sure 40.54% 15 21.62% 8 -18.92% (7)

Probably will not recommend 8.11% 3 16.22% 6 8.11% 3 Definitely will not recommend 24.32% 9 27.03% 10 2.70% 1

Totals 100.00% 37 100.00% 37 0.00% -

*S1-T1: Survey 1 – Tournament #1; S2-T3: Survey 2 – Tournament #3



Observations

Biggest change was on "Not sure" with 7 less people or 18.92% less people choosing "Not sure" as their response.

Positive recommendations increased by 3 people or 8.11%.

Negative recommendations increased by 4 people or 10.81%.

Analysis

It appears that as people play the Ringorang game, they form a clear opinion on whether they would or would not recommend Ringorang as a learning tool to other departments.

Finding: After three tournaments, more survey respondents have a negative recommendation

(43%) than a positive (35%) recommendation by about 8%.

Overall, how satisfied were you with using an interactive learning game to learn detailed information?

S2-T3

Response Response Percent Count

Very satisfied 10.81% 4 Somewhat satisfied 35.14% 13

Neither satisfied nor dissatisfied 16.22% 6

Somewhat dissatisfied 10.81% 4 Very dissatisfied 27.03% 10

Totals 100.00% 37

How likely are you to recommend Ringorang as a productive learning game to other

departments at PSE?(S1-T1 & S2-T3)

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Definitely will

recommend

Probably will

recommend

Not sure Probably will not

recommend

Definitely will not

recommend

S1 - T1

S2 - T3



Observations

Here is 16.22% reserve the right not to publish comments. Positive: 45.9% of the survey participants said they were somewhat or very satisfied.

Negative: 37.8% of the survey participants said they were somewhat or very dissatisfied.

Analysis

After three tournaments, more survey respondents have a positive satisfied (45.9%) than a negative (37.8%) dissatisfied.

What do you like most about the Ringorang learning game?

S1 - T1 S2 - T3 Change

Response Response Response Response Response Response

Percent Count Percent Count Percent Count

answered question 60.53% 23 72.97% 27 12.45% 4

skipped question 39.47% 15 27.03% 10 -12.45% (5)

total 100.00% 38 100.00% 37

Observations

Comments increased by 4 people or 11.97%.

Participants insert comment: o “Learning.” o “Lightning Round to learn in a concentrated focused way.” o “Easy, quick, 5 minutes a day, immediate feedback.”

Analysis

Raise participants learning power.

Overall, how satisfied were you with using an interactive learning game to learn

detailed information?(S2-T3)

Very satisfied, 10.81%

Somewhat satisfied,

35.14%

Neither satisfied nor

dissatisfied, 16.22%

Somewhat dissatisfied,

10.81%

Very dissatisfied, 27.03%



More participants like to choose to play the Lightning Round. Lightning Round was the preferred mode of play.


PSE RINGORANG NERC CIP PROJECT FINAL REPORT · PDF fileFinal Report - PSE Ringorang NERC CIP Tournaments_v0 4_draft-hy 20110830 (2) Page 8 of 72CONTACT: [email protected]

Documents