Final Action Research Report

Identifying Process Gaps Through Evaluation

Suzie Rose

University of Colorado, Denver

July 26, 2010

Identifying Process Gaps Through Evaluation

Introduction

I am an instructional designer in the training department of Edelweiss Communications.

Throughout the duration of this study, I held a leadership role in an initiative to improve the

extent to which approximately 2,000 existing customer-facing employees stayed abreast of

company changes. To do this, I had to analyze the extent to which employees were currently

staying abreast of changes, possible causes of low performance, and implement possible

solutions for process improvements.

First, it is necessary to define why Edelweiss Communications’ customer-facing

employees must stay abreast of company changes. Their goal is to provide exceptional customer

service experiences at all times. This requires that they have extensive knowledge and skills

about policies, internal and external procedures, products, services, and more. As the company

changes and grows with updates and new initiatives, existing employees must increase their

knowledge and skill sets. To ensure employees have the resources necessary to do this,

Edelweiss Communications distributes weekly updates. Weekly updates outline the details of

company changes and will be discussed in more detail in a subsequent section of this document.

The training department that I am part of is responsible for supporting delivery of the

information within weekly updates. It is responsible for identifying if training is needed and if

so, determining the method in which it is delivered. This study will help identify the extent to

which customer-facing employees stay abreast of company changes so the training department

can assess possible gaps and implement process improvements.

Problem Statement

Upon commencement of this study, Edelweiss Communications did not measure the

extent to which customer-facing employees stayed abreast of company changes. This lack of

measurement placed the training department in a precarious position. It was unable to validate if

the processes used to support the delivery of information in weekly updates was effective or

valuable. Without this information, the training department struggled to show its contributions

to the business organization. This situation held the potential to affect decisions regarding

resources and budget allocation to the training department.

Edelweiss Communications was also unable to identify employees who were not staying

abreast of new information. Employees who do not stay abreast of new company information

need additional coaching to prevent occurrences where they inadvertently provide

misinformation to customers. Customers who receive misinformation are likely to leave a

company which ultimately costs Edelweiss Communications valuable revenue. In fact, a study

by Frederick Reichfeld and W. Earl Sasser Jr., showed that a 5% increase in customer loyalty

can increase profits from 25% up to 85% by extending the average customer relationship lifetime

(1990). This significant difference clearly indicates that customer retention and loyalty is

valuable to any company.

Purpose

The purpose of this study was to help customer-facing employees stay abreast of

company changes. There will be multiple phases of this study. However, this phase of the study

(a) defined the problem, (b) measured the current state of employee knowledge as related to

company changes, (c) conducted a focus group to collect qualitative data from key players, (d)

analyzed the results of a knowledge based assessment to create a baseline of data, (e) compared

and contrasted findings from the focus group, and (f) determined possible causes of low

performance. It laid the foundation for Edelweiss Communications’ ultimate goals of increasing

efficiencies and improving processes; possibly through methods explored in the Literature

Review section of this document.

To put this study into action, I evaluated the extent to which employees were currently

staying abreast of company changes at the time of the study. I then analyzed the data collected

to identify gaps in processes. According to Ron Kurtus, the founder of the School for

Champions, “By organizing or re-organizing your operations and processes, you can make your

business a more effective machine” (2002). Kurtus’ viewpoint echoes that of Edelweiss

Communications and illustrates its aspiration to create an efficient and well-oiled organization.

To achieve this aspiration, Edelweiss Communications chose to utilize Level Two assessments

as defined by Donald L. Kirkpatrick (1959). These assessments served as the source of baseline

data. Throughout this study, a synthesis of Level Two assessments and process improvements

culminated into an exploration of the relationship between evaluation and process improvements

in corporate settings. It helped assess possible gaps in Edelweiss Communications’ processes

and propose improvements.

Intended Audience

This initiative had a variety of audiences. The primary audience for this study was Dr.

Jennifer VanBerschott and fellow students in the Research in Information and Learning

Technologies, ILT 6720 class. The secondary audiences included the management staff of

Edelweiss Communications’ content management and training departments including managers,

training managers, curriculum developers, content developers, project managers, vice presidents,

and presidents of both departments.

Note: Participation was not compensated beyond normal salary and benefits currently received

at the time of the study.

Research Questions

Although future areas of focus and process improvement initiatives will follow this study,

the focus of this portion of the study was identifying gaps in processes and analyzing possible

causes for them. To identify and analyze these items properly, an overarching research question

and guiding questions were developed to help keep this study focused.

Overarching Question

To what extent are customer-facing employees staying abreast of company changes?

To help customer-facing employees stay abreast of company changes, it is important to

establish a baseline of data. Focusing this phase of the study on an attempt to capture baseline

data helps the long-term goal of helping employees stay abreast of company changes.

Guiding Questions

1. What processes are involved in helping customer-facing employees stay abreast of company changes? To what extent are the processes involved?

Processes are a possible cause and distinct antecedent of the extent to which employees

stay abreast of company changes. In many cases, the processes involved have an extensive

amount of influence on the transfer of knowledge. This includes those within company

procedures and those involved in the delivery methods of the information.

2. What are the most effective methods of evaluation to help identify gaps in processes?

For the purposes of this study, Edelweiss Communications used Level Two assessments

as defined by Kirkpatrick’s model of training evaluation. This question aimed to identify if the

method is effective or if another phase of study should be implemented with a different data

collection method.

3. What are possible process improvement methodologies that may improve the state of the

situation?

Since Edelweiss Communications had not yet defined the type of process improvement

methodology it wished to implement, it was decided that research regarding industry

methodologies be explored throughout this phase of study. These findings are found in the

Literature Review section of this document.

Although the overarching question remained the same throughout the duration of the

study, the guiding questions changed. The second guiding question originally read, “What

correlations and patterns exist between employee assessment scores and the weekly update

delivery methods?” Upon further inspection, it was clear that this was an inherent feature of

data analysis and would naturally be answered throughout the analysis process.

Figure 1

Instead, the following question was added, “What are possible process improvement

methodologies that may improve the state of the situation?” Throughout the literature review

conducted during this research study, it was found that many methodologies are currently in

practice to help companies improve processes and efficiencies. Because the ultimate goal of all

phases of this study is to help Edelweiss Communications’ employees stay abreast of company

changes, methodologies on how to do so held a high level of importance. Although Edelweiss

Communications must find its own tailored methodology and further investigation will be done

in subsequent portions of this study, it is important to research alternative methodologies of

process improvements while the foundation of the study is built – in phase one.

Context of the Study

Process All company changes and initiatives funnel through Edelweiss

Communications’ content management department which creates a

report summarizing each week’s updates. This is distributed to

leadership staff at each location within the company. The leadership

staffs then redistribute the information to site supervisors and trainers

who deliver it through team meetings, instructor led training sessions,

or eLearns. The training department supports this process by defining

delivery methods and creating content if applicable. Employees have

scheduled time to participate in team meetings, instructor led

trainings, and eLearns. Participation in these activities is a job

requirement and is enforced by employees’ supervisors. Refer to

figure one for a flow chart of this process.

Demographics

To provide details about Edelweiss Communications’ demographics, a community profile

was necessary. A community profile, as described by Stringer (2007), is a snapshot of the

context in which the investigation is placed. The snapshot of Edelweiss Communications

includes communication methods, locations, company organization, and more. Employees

communicate with customers through phone conversations, emails, instant messages (IMs),

door-to-door interactions, and in-store interactions. Refer to Figure 2 for a visual representation

of this flow. They are located in eight locations within the United States. Some employees work

directly for the corporation; some are contracted through third-party agreements. Employees

include men and women of all age groups and nationalities. The median age is 28. These items

were considered throughout this study to help Edelweiss Communications identify trends and

gaps that may be related to demographics.

Background

Weekly updates have been distributed in the process described for many years. As the

company grows and responsibilities increase, Edelweiss Communications fears that attention to

these updates is dropping in priority for both employees and their supervisors. Retention of

weekly updates is assumed as a job responsibility, and may therefore receive little attention; it

Figure 2

does not seem to have a significant priority in terms of participation or enforcement of

participation. This could result in employees falling out of touch with weekly updates and could

be a cause of possible performance opportunities. By reassessing the priority and accountability

levels of the weekly updates, Edelweiss Communications may be able to stimulate change in this

situation.

Literature Review

A vital component of this study was to research existing information about the

relationship between evaluation and process improvements in corporate settings. Being familiar

with existing studies and literature helps one situate the study into existing knowledge and

hopefully contribute to it (Knopf, 2006). The key component of this familiarity for Edelweiss

Communications was the ability to situate this study into existing knowledge. The literature

review was extremely helpful in researching, identifying, and defining methodologies of process

improvements that are currently in practice in corporate environments. This insight provided a

course of action for a subsequent phase of the study: implementation of process improvements.

This comparison of viewpoints aligns with Stringer’s stance that literature reviews set the stage

for comparisons of viewpoints from existing literature with those one discovers through their

own studies (2007).

Literature Search Questions

Throughout the process of this literature review, articles and studies similar in culture and

context to those of Edelweiss Communications’ were searched for; this helped explore and

substantiate the study’s core concerns. This study explored the relationship between evaluation

and process improvements in corporate settings to help assess possible gaps in Edelweiss

Communications’ processes and propose improvements. Because this study has never before

been conducted within Edelweiss Communications, it was necessary to expand the focus of the

literature review to a broader stage, but also contain it within its core concerns. To do so,

literature search questions were created to break down and emphasize the study’s core concerns:

1. What links and correlations exist between evaluation techniques or methods and process

improvements in corporate settings?

2. What are the most effective methods of evaluation to help identify gaps in processes?

3. What are the most effective methods of implementing process improvements?

Search Procedures

All content searches started online to allow for easy duplication and cross-referencing of

resources. As a student of the University of Colorado Denver, I began my search within the

Auraria Library Database search engine. From there, I navigated to the Article Databases &

Indexes where I selected the “Companies” category. After sorting for databases related to

corporate training, I found a niche in the EBSCO database. In my search, I used the following

combinations of terms: (a) “Process Improvement + Assessment,” (b) “Process Improvement +

Experimental Design,” (c) “Process Improvement + Study,” (d) “Process Improvement +

Identify Gaps,” (e) “Process Improvement + Methodologies,” (f) “Process Improvement +

Lean,” (g) “Process Improvement + Kaizen,” (h) “Six Sigma,” (i) “DMAIC,” (j) “Evaluate +

Kirkpatrick + Knowledge.” I was not familiar with some of these terms at the beginning of the

literature review, but rather discovered them as I performed research inquiries. These include

Six Sigma, DMAIC, Lean, and Kaizen.

Literature Review Findings

Overview

As previously noted, the literature review was helpful in researching, identifying, and

defining methodologies of process improvements that are currently in practice in corporate

environments. Services are bound by time in terms of the processes that lead to the delivery of

an outcome that benefits a customer (Fraser & Fraser, 2009). As competition increases,

customers’ expectations about products and services increase as well. Statistics show that

retaining satisfied and loyal customers is more profitable than acquiring new ones (Pakdil,

Özkök, Dengiz, Kara, Selvi & Kargı, 2009). To retain customers, it is important that companies,

Edelweiss Communications included, evaluate their processes to improve flows and efficiencies

wherever possible. Becoming truly customer-centric is essential for the success of all businesses

(Naumann & Giel, 1995). As the American culture continues to shift its focus from product-

centric to customer-centric, businesses must keep pace with the shift. Therefore, companies look

for ways to improve their organizational efficiencies to meet customer expectations (Jones,

Parast, & Adams, 2010). Although some company efficiencies are not directly customer-facing,

most contribute in a large way to a customer’s experience with a company, service, or product.

A part of customer-centricity can be achieved through process improvements. In fact,

Continuous Process Improvement (CPI) has attracted significant attention from corporate and

industry environments (Li & Rajagopalan, 2008). Improving processes increases company

efficiencies, employee job satisfaction, and the delivery of products and services to customers.

Eliminating errors that result in customer dissatisfaction improves processes (Pakdil et al., 2009).

But just how do companies eliminate errors and improve processes?

Kaizen

The goal of CPIs is to evaluate and improve a current state

(Tarantino, 2009). This ties directly to Edelweiss Communications’

goal. Edelweiss Communications wants to evaluate and improve the

current state of the extent to which employees stay abreast of

company changes. This type of evaluation and improvement takes a

great deal of focus and attention. A successful corporate cultural

movement that helps support CPI is Kaizen. Carter & Carmichael

(2009) explain that Kaizen is the Japanese word for good change (see

Figure 3) and it encompasses a commitment to continual and steady

improvement for management and employees. Weaknesses are actively sought out, documented,

and remedied via dedicated discussion and actions that are part of the normal workday, rather

than being occasional or external.

Ortiz (2006) explains that the philosophy of Kaizen is the foundation and driver for

process improvement initiatives. It helps build a foundation for change that will enable a

company to sustain process improvements for its lifetime. A successful Kaizen program has a

governing committee, communication boards and newsletters, regular meetings, and event

tracking.

The following sections of this review outline the industry leaders in process improvement

methodologies that are possible because of Kaizen: Six Sigma, Lean, and Lean Six Sigma. Any

or all of these methodologies have the potential to help Edelweiss Communications improve the

extent to which employees stay abreast of company changes.

Figure 3

Six Sigma

Six Sigma is a data-driven philosophy used to drive management decisions and actions

across an organization (Caulcutt, 2001). Data is used here as a subjective term and can include

various observable information of both a quantitative and qualitative nature. Motorola generated

the concept in 1981 and pioneers using Six Sigma helped refine and develop it (Bendell, 2005).

As the first US organization to implement Six Sigma, General Electric contributed to broadening

its use across manufacturing, design and transactional areas (Grant, 2008). These companies are

examples of pioneers which implemented the ideas and concepts of the Six Sigma methodology

to establish concrete financial advantages.

Six Sigma uses a structured method to solve business problems (Jones et al., 2010)

through the use of a standard framework known as Define-Measure-Analyze-Improve-Control

[DMAIC] (Gygi, DeCarlo, & Williams, 2005). Refer to Figure 4 for a visual representation of

this flow. Although this is the most recognized framework, and some literature refers to it alone,

Six Sigma uses another framework known as Define-Measure-Analyze-Design-Verify

[DMADV] (Jones et al., 2010) – Refer to Figure 5 for a visual representation of this flow.

Figure 4. DMAIC Framework

Figure 5. DMADV Framework

The DMAIC framework is designed to help eliminate waste and defects to improve on

existing processes, whereas the DMADV framework is designed to develop new products,

services, or tools (Jones et al., 2010). Each framework follows the phases within it in a linear

manner. For detailed definitions of each phase, refer to Appendix A.

Although Six Sigma is the most recognized method of process improvement, it receives

mixed reviews from critics. Motorola and General Electric report significant savings as a result

of their Six Sigma initiatives (Pande, Neuman, & Cavanagh, 2000). Yet others fear it will fail

because of intense business competitiveness (Jones et al., 2010). In fact, one study found that

companies that adopted Six Sigma underperformed in the stock market and achieved only

incremental gains (Bernett & Nentl, 2010). However, of all the literature included in this review,

that is the only reference to less than favourable results. Bernett and Nentl (2010) also contradict

others by stating that in 1998 Motorola’s second quarter profits were minimal and the company

eliminated 15,000 jobs.

Despite this unfavourable review, Six Sigma is generally respected and considered

successful in most corporate environments. Six Sigma is appealing to corporate organizations

because it is built upon a foundation of statistical methods to improve quality, and it can define

success in a quantifiable manner (Tarantino, 2009). This is of great importance to organizations

such as Edelweiss Communications that aim to quantify return on investment and track

improvement rates.

Lean

Another prevalent CPI method is Lean. John Krafcik coined the term ‘Lean’ to describe

the new production techniques at Toyota after World War II (Bendell, 2005). Lean is defined as,

“The systematic pursuit of perfect value through the elimination of waste in all aspects of the

organization’s business processes” (Bendell, 2005). It incorporates five basic principles: (a) the

elimination of waste, (b) the identification of the Value Stream, (c) the achievement of Flow

through the process, (d) pacing by a pull (rather than push) signal, and (e) the continuous pursuit

of perfection (Womack, Jones, & Ross, 1990). Refer to Appendix B for further explanation of

Lean definitions.

Both Six Sigma and Lean are built upon the foundation of Kaizen and business process

improvement (Bendell, 2005). Unlike Six Sigma, Lean focuses on concepts and values as its

structure rather than phases of a process. Another difference is that Lean is heavily centered on

process analysis and flowcharting/mapping techniques, rather than data (Grant, 2008). Like Six

Sigma, Lean focuses on increasing efficiencies and decreasing waste.

Also like Six Sigma, Lean receives mixed reviews from critics. Bernett and Nentl (2010)

report that Lean has been well received by corporate organizations since Womack and Jones

captured organizations’ attention in 1996 with the release of their book, Lean Thinking.

However, other critics claim that Lean has a limited application range; there is general agreement

that it is well suited to companies with similar characteristics to the automotive industry rather

than corporate organizations (Lasa, de Castro, & Laburu, 2009).

However a recent contribution to the Lean model includes Value Stream Mapping (VSM)

which focuses on the analysis and improvement of discontinuous flow lines. It includes five

phases: (1) selection of product family, (2) current state mapping, (3) future state mapping, (4)

definition of working plan, and (5) achievement of working plan (Lasa et al., 2009). This adds a

more concrete plan of implementation to the concept and allows more organizations to

implement the Lean model. By comparing the current state and the identified future state of a

process, organizations are better able to define and achieve a working plan.

Lean Six Sigma

In a quest for a truly holistic approach to business and process improvement, many

corporate organizations attempt to combine Six Sigma and Lean into practice simultaneously.

Lean practices and techniques focus on streamlining processes, whereas Six Sigma practices and

techniques help identify and eliminate root causes of problems (Evans & Lindsay, 2005). An

effective combination of these methods would create an invaluable process improvement

solution for many organizations.

A case study presented by Fraser and Fraser (2009) discusses the successful

implementation of a Lean Six Sigma model by the National Grid on a project concerning

renewal prompts for vehicles. In this case, the organization used both models concurrently and

matched the phases side by side. This is one variation of combining Lean and Six Sigma.

In an attempt to uncover patterns of implementation between Lean Six Sigma, Shah,

Chandrasekaran, and Linderman (2008) conducted a research study using implementation and

performance data from a sample of 2511 industrial plants. They found that implementation of

Lean practices improved the likelihood of implementing Six Sigma and indicated a significant

difference in the performance levels between those who did and those who did not implement

these models. However, there was no report of how the organizations combined the two models.

Since there was not a structured method of how to combine the models, there could be great

variation between these organizations.

Tarantino (2009) supports the combination and explains that combining the two provides

a powerful best practice for process and customer service improvements if done so in the hands

of focused experts. However, the reviews are mixed. Bendall (2005) states, “It would be clearly

desirable if a single process-improvement-based approach was available that effectively

combined the approaches” (p. 975). He continued to explain that most attempts of combination

resulted in a situation where one of the two methods was taken as a dominate, and the other as a

subordinate. He pointed out that this is not a particularly good approach, since both methods

have unique features and benefits that only more effective combinations would retain.

Kirkpatrick

Because Edelweiss Communications began its process improvement initiative by

evaluating its current state using Kirkpatrick’s learning evaluation theory, it is necessary to

include literature regarding this facet as well. The goal of this inclusion is to benchmark

Edelweiss Communications’ choice of evaluation with other organizations. This provides the

opportunity to compare Edelweiss Communications with those considered to be best-in-class.

Benchmarking helps draw comparisons and determine which processes are adaptable to other

organizations (Carter & Carmichael, 2009).

Kirkpatrick’s learning evaluation theory consists of four levels of training evaluation: (a)

reaction, (b) learning, (c) behaviour, and (d) results (1959). Rowden (1998) writes, “These

often-cited levels describe a rubric that is often used to evaluate programs. It is the last two

levels—behaviour change and results—that ROI is concerned with” (p. 32). Level One

evaluation measures employee satisfaction with training. Level Two evaluation assesses the

acquisition and retention of declarative knowledge, procedural knowledge, and attitude changes

(Lupton, Weiss, & Peterson, 1999). Levels Three and Four, behaviour change and results, are

the measurable changes we can see in employees and the benefits a company enjoys from

employees implementing what they have learned (Rowden, 1998). Based upon these definitions,

Kirkpatrick’s model may be effective in helping execute Six Sigma’s Measure phase.

Quality of Literature

Overall, the quality of the literature used in this review was very strong. All articles were

peer reviewed and many were from popular notable publications such as the Journal of Applied

Statistics and International Journal of Production Research. The literature provided excellent

information about what is currently happening in corporate organizations. For each CPI method,

information about its origin, its core structure or framework, and how well it works or is being

received by corporate organizations was available. A broad variety of angles or points of view

were also used in various articles. For example, Grant’s article (2008), provided overviews of

Six Sigma, Lean, and a few less recognized methodologies. Jones et al. (2010) defined and

described Six Sigma and then explored best practices for implementation. Bernett and Nentl

(2010) conducted a survey about continual improvement and explored the findings and opinions

of those surveyed. Nuran and Frasier (2009) as well as Matson and Stauffer (2009) each

explored findings of case studies. This variety allowed for an exceptional sample of articles.

Combining these articles into a whole helped build a multi-faceted view of the existing

relationships between evaluations and process improvement efforts.

However, some intrinsic limitations were present due to the lack of time allotted to

perform the literature review. If more time were available, the opportunity would have been

available to expand data collection methods to include visits to libraries and searches through

tangible periodicals. Although the online search allows for easy duplication of the search

process, it also creates a limitation of data to be collected.

The literature also showed a limit in participants represented. All authors were

journalists or researchers rather than facilitators, business leaders, or other participants that may

have been able to share first-hand experiences with the success or failure of a specific CPI

method. The inclusion of these points of views would have provided a greater diversity to the

study and would have possibly brought about different results.

Yet, the largest limit of this literature review was the lack of evaluation techniques

explored. This is caused by both a lack of time to conduct research on what those techniques

may be, and a lack of founding knowledge about evaluation techniques. This created a situation

in which key search terms were not as strong as they could have been. This could have been

resolved if more time were allotted for the literature review. However, the combination of both

situations (lack of time and knowledge) results in the inability to research evaluation techniques

to a greater extent. In literature reviews that will be conducted in future phases of this research

study, a section devoted to evaluation techniques will be implemented.

Gap in Literature

Most of the literature in this review focused on describing and explaining the concepts,

methodologies, and tools of each CPI method. They also discussed strengths, weaknesses, and

success factors. However, individual company needs, goals, and context were excluded. Further

information on organizational specifics is found in strategy literature which forces business

leaders to complete in-depth analysis and cross referencing between the two areas. Because this

is a timely and arduous task, most business leaders select a methodology for their company based

on suggestions from friends, colleagues, gurus, and practitioner publications (Charlesworth,

2000). This indicates that there is a gap in the literature that would help business leaders assess

their circumstances and match it with an appropriate process improvement methodology. In fact,

Thawesaengskulthai and Tannock note that without an explicit decision support, organizations

may base their selection on subjective judgments. A process improvement methodology selected

through subjective judgements may not provide a company with all the features and benefits it

needs.

The literature also failed to explain how a company would implement a process

improvement methodology. For example, the Define-Measure-Analyze-Improve-Control

framework of Six Sigma provides a detailed breakdown of what to do, but fails to explain how to

do so. Edelweiss Communications’ study is filling this gap by providing an explicit example of

a study.

Literature Review Summary

This literature review explored existing content about how evaluation of training

initiatives helps corporate organizations identify and implement process improvements. Models

and concepts explored include Kaizen, Six Sigma, Lean, and Lean Six Sigma. The literature

review allowed for study of each methodology in terms of origin, structure or framework, how

they work, and the extent of their success.

The review also showed findings on Kirkpatrick’s model of training evaluation in an

attempt to find correlations between Edelweiss Communications’ current method of evaluation

and how that may fit into the context of process improvements. Although no articles combined

these issues within their own context, uniting these articles through this review helped build a

multi-faceted view of the existing relationships between evaluations and process improvement

efforts. The following items break down the literature review as compared to the literature

search questions introduced at the beginning of this section.

1. What links and correlations exist between evaluation techniques or methods and

process improvements in corporate settings?

As noted in the Literature Gap section of this document, the articles failed to explain

exactly how a process improvement methodology is put into action. Because of this gap,

the links and correlations between evaluation techniques or methods and process

improvements are also undocumented. However, the literature on Kirkpatrick’s learning

evaluation theory suggest that Level Two assessments are a successful method for

capturing data regarding the current state of a situation. But ultimately, links and

correlations cannot be established based on the literature review alone.

2. What are the most effective methods of evaluation to help identify gaps in

processes?

As previously noted, methods of evaluation and process improvements were not

identified in the literature. Therefore links between these two concepts were not

explored. However, the literature on Kirkpatrick’s learning evaluation theory suggest

that Level Two assessments are a successful method for capturing data regarding the

current state of a situation.

3. What are the most effective methods of implementing process improvements?

The literature review outlined the most prevalent methods of process improvements

found. This includes Six Sigma, Lean, and Lean Six Sigma. The literature suggests that

all three of these methods are effective, however, a delicately balanced combination of

Six Sigma and Lean seem to hold the most promise as a complete and holistic approach

to process improvements.

Methods

My primary data gathering method followed the protocol of statistical-observational

study. To adhere to the statistical approach of the study, I collected quantitative data from a

specific population (Roberts, 2010). I created and distributed a 30-question electronic

assessment. The questions were based on factual knowledge and molded to comply with Level

Two evaluations as defined by Kirkpatrick. This fulfills the statistical requirements of this type

of study. The analysis of this data focused on the observational aspect of my statistical-

observational study. As defined by Roberts, (2010) “In an observational study, the sample

population being studied is measured, or surveyed, as it is…Data is simply gathered and

correlations are investigated” (p. 1). I did not manipulate any data, I simply collected the

information, restructured it to ensure proper analysis, and identified patterns or correlations.

My secondary data gathering method was conducted in a focus group where I asked

interview questions to collect viewpoints and thoughts from key players in the study. This

exemplifies an action research aspect because it incorporates members of the community to help

identify and solve problems.

The data collected from this initiative contributes to Edelweiss Communications’ ability

to assess and identify gaps in additional areas such as the effectiveness of weekly updates,

training content, and delivery methods. By reassessing the priority and accountability levels of

the weekly updates, Edelweiss Communications is now better able to stimulate change in this

situation.

However, the study may cause a conflict of goals for customer-facing employees. If we

increase the accountability of their responsibilities regarding staying abreast of company

changes, they may become overwhelmed. Because they have restrictions placed upon them in

terms of time management and performance, enforcement of responsibilities (that may not have

been enforced in the past) may hinder their performance in other areas, thus causing frustration

and possible employee attrition.

Participants

The primary participants of this study were 1,918 customer-facing employees employed

by Edelweiss Communications. There was not direct communication with them because their

feedback was collected via a 30-question electronic assessment through a local learning

management system (LMS). It also included two training managers employed by Edelweiss

Communications with whom I conducted a focus group.

I led the project, created the assessment, collected and analyzed data, and compiled this

report. The content management department created the weekly updates mentioned throughout

this study and delivered weekly conference calls to communicate its content to site leaders.

Trainers and supervisors delivered the information in the weekly updates to customer-facing

employees. The training department leaders sponsored the project and communicated data

results and analysis to their superiors and the content management department up to and

including presidents and vice presidents.

Data Gathering

Quantitative

As previously mentioned, my primary data gathering method followed the protocol of

statistical-observational study. To adhere to the statistical approach of the study, I collected

quantitative data from approximately 1,918 existing customer-facing employees. I created and

distributed a 30-question electronic assessment through an LMS. Employees had 30 minutes

scheduled in their shift to complete the assessment. The 30-minute allocation may have been

scheduled anywhere within a two week span. Participation in this assessment is a job

requirement with enforcement from employees’ supervisors.

Software capabilities prevented participants from skipping questions or returning to

previous questions. Participants could however, exit the assessment and re-enter at the location

where they left.

The questions focused on factual information, were directly related to the information

within the weekly updates, and covered information about company changes spanning Quarter

One 2010 (January – March).

Qualitative

I conducted one focus group with two training managers. These training managers are

located in different sites and were selected because they oversee the largest variety of site

locations. I facilitated this over the phone on a group conference call as defined by the training

managers’ schedules for the duration of one hour. I recited disclosures to them before the focus

group began so they could understand the impact of their participation as listed in Appendix C.

I already had an established rapport with these individuals through previous working

relationships and was therefore able to help participants feel that they could say or express what

they really felt or thought (Stringer, 2007). Refer to Appendix D for focus group questions.

Ethical Procedures

For the quantitative aspect of this study, participants were notified that data from the

assessment would be collected for company informational purposes. They did not have the

option to refuse participation because the assessment scores are also used for other purposes such

as individualized coaching. However, they did have the option to exclude their assessment

results from the data used in this study. Any and all data was and will be distributed solely

within Edelweiss Communications on a need-to-know basis only. Data distributed throughout

Edelweiss Communications was provided to leadership staff only and is regulated by internal

records management policies and procedures.

Data distributed to Dr. Jennifer VanBerschott and fellow students in the class was altered

with pseudonyms or placeholders to protect the identity of the participants, the locations of the

sites, the company name and profile, etc. Due to the vast amount of raw data (2,000 participants

x 30 questions = 60,000 lines of data) and the fact that nearly all information within it was

identifying, it was excluded from the final submission for this class. Instead, I used source data,

which is the restructured data that served as my source for analysis.

As previously noted, for the qualitative aspect of this study, I provided full disclosures as

outlined in Appendix C. The disclosures listed (a) purpose of the study, (b) why the members of

the focus group were selected for participation, (c) the members of the study who would be privy

to their responses, (d) their options for participation, and (e) procedures for withdrawal.

Checks for Rigor

To ensure the outcomes of my research remain

balances (Stringer, 2007). Stringer (2007) states,

multiple sources of information are incorporated” (p. 58).

By using both quantitative and qualitat

In the statistical-observational portion of my study, I incorporate

checking (Stringer, 2007). Participants

level of involvement. Customer-

Supervisors can review the results of their own assessment and the results of their direct reports’

assessments. The training and content management departments and above can review all

results. The project lead and sponsor (and above if applicable) in the training department can

review all raw data, analysis, and reports.

To ensure the outcomes of my research remained trustworthy, I incorporated multiple checks and

Stringer (2007) states, “The credibility of a study is enhanced when

multiple sources of information are incorporated” (p. 58). This is the concept of triangulation.

By using both quantitative and qualitative data, I was able to implement this concept.

observational portion of my study, I incorporated the concept of member

checking (Stringer, 2007). Participants were, and are, able to view their scores based on their

-facing employees can view the results of their own assessment.


training and content management departments and above can review all


review all raw data, analysis, and reports. Figure 6 clarifies this concept visually.

Project Lead and Sponsor

Training / Content Mgmnt

Supervisors

Participants

multiple checks and

“The credibility of a study is enhanced when

This is the concept of triangulation.

ive data, I was able to implement this concept.

the concept of member

able to view their scores based on their

facing employees can view the results of their own assessment.


training and content management departments and above can review all


clarifies this concept visually.

Figure 6

I kept all records of raw data, reports, analysis, communications, and focus group notes to

provide and show documentation of the procedures followed during this study to create to fulfill

the concept of Dependability (Stringer, 2007).

Data Analysis Procedures Quantitative

I used Crystal Reports to capture data from all the employees who completed the

assessment. Crystal Reports generated data about (a) employee job titles (supervisor or

subordinate), (b) employee locations, (c) scores, and (d) which questions employees passed or

failed.

By utilizing the functions in Microsoft Excel such as sort, import, and pivot tables, I

restructured the data from Crystal Reports to emphasize the most relevant information. The

restructured data served as my source data for analysis and included (a) average scores of the

assessment (by location and company-wide), (b) pass/fail percentages of the assessment (by

location and company-wide), and (c) pass/fail percentages of each question (by location and

company-wide). Please refer to Appendices E-G for complete details. In addition to this

information, I also collected further data in various ways. I (a) searched archival scheduling

records to determine the delivery method as related to each question, (b) collected the wording of

each question as it appeared in the assessment from the development source files, and (c)

collected the correct answer for each question from the development source files. Items b and c

are directly related to identifiable company information; therefore they are listed in Appendix G

as placeholders.

The analysis of this data focused on the observational aspect of my statistical-

observational study. As previously noted, I did not manipulate any data, I simply collected the

information, restructured it to ensure proper analysis, and identified patterns or correlations. The

data I analyzed included (a) average scores of the assessment (by location and company-wide -

listed in Appendix E), (b) pass/fail percentages of the assessment (by location and company-wide

- listed in Appendix E and F), and (c) pass/fail percentages of each question (by location and

company-wide - listed in Appendix G).

To analyze the data, I identified the aspects of the information that would help my

audience understand the meaning of the data (Stringer, 2007). This falls in line with the method

of analyzing key experiences as defined by Stringer. I focused on events and trends that impact

my stakeholders. I began by categorizing and coding my data to identify events or experiences

that may have some significance. While doing this, I searched for relationships between the

following items: (a) the pass/fail percentage of the assessment and location (see E and F), (c) the

pass/fail percentage of each question and location (see Appendix G), and (d) the pass/fail

percentage of each question and delivery method (see Appendix G). This helped me find themes

in the variety of relationships listed, which in turn, helped me identify elements that compose the

experience for participants of the study.

Qualitative

To identify significant features and elements of my participants’ experiences, I analyzed

my qualitative data through means of categorizing and coding as defined by Stringer (2007). To

do this, I used the verbatim principle: using terms and concepts from the participants themselves

(Stringer, 2007). I then (a) reviewed the collected data, (b) unitized the data, (c) categorized the

data, (d) identified themes, (e) organized a category system, and (d) developed a report

framework (Stringer, 2007).

Schedule

To ensure timely completion and organization throughout the duration of this project, I

implemented and followed the schedule listed in the below table (Table 1 – Schedule).

Table 1 - Schedule Item Task Type Completion

Submit Action Research Proposal - Draft

Milestone June 21, 2010

Revise Action Research Proposal Development June 22, 2010 – June 26, 2010

Submit Action Research Proposal Milestone June 26, 2010

Write Interview Questions Data Collection and Analysis June 21, 2010

Focus Group to collect qualitative data

Data Collection and Analysis June 29, 2010

Run reports to collect quantitative data

Data Collection and Analysis June 28, 2010 – July 2, 2010

Restructure reports/create source data Data Collection and Analysis June 28, 2010 – July 2, 2010

Analyze quantitative data Data Collection and Analysis July 3, 2010 – July 9, 2010

Analyze qualitative data Data Collection and Analysis July 3, 2010 – July 9, 2010

Write Literature Review Development June 27, 2010 – July 4, 2010

Submit Literature Review – Draft Milestone July 5, 2010

Revise Literature Review Development July 6, 2010 – July 10, 2010

Submit Literature Review Milestone July 10, 2010

Write Data Analysis Report Development July 11, 2010 – July 18, 2010

Submit Data Analysis Report – Draft Milestone July 19, 2010

Revise Data Analysis Report Development July 20, 2010 – July 26, 2010

Submit Data Analysis Report Milestone July 26, 2010

Write Final Research Report Development July 20, 2010 – July 25, 2010

Submit Final Research Report - Draft Milestone July 26, 2010

Revise Final Research Report Development July 27, 2010 – July 31, 2010

Submit Final Research Report Milestone July 31, 2020

No changes to the schedule were required throughout the duration of this study.

Findings

Quantitative

Throughout the trends listed in this section, please note that Edelweiss Communications

considers a score of 80% or higher to be a passing score. The following sections citing exceeds

and fail percentages or vernacular refer to this designation.

Notable Exceeds and Fail Trends

Edelweiss Communications expects that supervisors ensure at least 80% of their teams

pass the assessment. Only 81 of 229 supervisors accomplished this goal. In fact, of 229

supervisors, 2 had 100% of their employees fail. Refer to Appendix F for additional details.

Of 229 supervisors, only 3 had a team larger than 10 and 100% of their employees passed

the assessment. They were able to communicate and reinforce the content contained in the

weekly updates despite their high employee count and were therefore recognized for their

abilities to exceed expectations. Refer to Appendix F for additional details.

These results signify that more employees are failing than exceeding and hint that

customer-facing employees may be struggling to stay abreast of company changes.

Notable Center Trends

In general, pass percentages were low. The highest center (Snapdragon) had 80% of its

employees pass. The lowest center (Rhododendron) had 50% of its employees pass. All centers

had a median average of 67% passed. This data is particularly significant in the sense that it

implies that employees are likely to be struggling to stay abreast of company changes. In fact,

further details reveal that Snapdragon had the highest average scores at 86%. Rhododendron had

the lowest average scores at 79%. These numbers reflect totals, which includes both customer-

facing employees and supervisors. It is also important to note that Snapdragon had the highest

percent passed and the highest scores. Rhododendron had the lowest percent passed and the

lowest scores. Refer to Appendix E for additional details.

Question Level Analysis – Below 80%

Of 30 questions, 8 had less than 80% answered correctly. Refer to Appendix H for an

analysis of causes listed in the order of lowest to highest average score. To protect company

confidentiality, the actual questions are not listed, but only the question number and an analysis

of possible reasons for the low score. This data should show the variety of possible causes and

themes between them.

To analyze the reasons these questions received low success rates, I used the analysis

method as defined by Stringer (2007) called Antecedents and Consequences. To complete this

process effectively, Stringer suggests that facilitators or participants identify (a) the core

problem, (b) major antecedents to the problem, (c) significant factors related to the antecedents,

(d) major negative consequences, and (e) other significant consequences. The remaining data in

this section and in Appendix H was analyzed using these methods.

A more global analysis revealed that more than 25% of the questions were answered

incorrectly. This signified that employees were not staying abreast of company changes. It also

indicated that the procedure in which weekly updates is distributed may not be effective.

Major antecedents of this problem were identified through finding patterns in the possible

causes of low scores for each question as listed in Appendix H. This includes employees (a) may

not have used the Intranet properly, (b) may have expected information to be in different

locations, (c) may not have been using the Intranet at all (d) were not receiving the information

as required in team meetings, (e) were not receiving the information in full, (f) were not

receiving it correctly (g) or may have missed some of the questions due to reading

comprehension founded on (a) improperly written questions, (b) convoluted policies and

procedures, or (c) insufficient reading comprehension skills. This analysis indicates that Level

Two evaluation may be an effective method of evaluation to help identify gaps in processes since

processes are prevalent in some of these findings. It provides numerical data which is vital to

stimulating change in a corporate environment.

Qualitative

The results of the focus group provided great insight into the inner workings of Edelweiss

Communications’ processes. My participants were selected because they have the largest

amount of visibility into possible issues. Refer to Appendix D for focus group questions.

Employees may not receive the information

Throughout the focus group, participants clarified the process of distributing information

in the weekly updates. Participant A explained that supervisors attend weekly update calls, and

are then asked to review the information in team meetings. However, supervisors (a) may not

follow the scheduled training times, (b) may not deliver the content correctly, or (c) may not

cover the information at all. She stated, “Our trainers will show up to train during a team

meeting, and no one is there.” This shows that team meetings are not happening at scheduled

times and signifies that they may not be happening at all. As a result, employees are forced to

use the Intranet as their only resource for the information, rather than learning the information in

team meetings. Participant B agreed that supervisors may not cover the information because

there is a lack of consistency in following protocol. She was only 70% confident that the

messages were communicated. This data clearly indicated that the process of releasing

information from the weekly updates was not effective.

Lack of Priority

After explaining that supervisors attend weekly update calls, and are then asked to review

the information in team meetings, Participant A voiced her concern that the weekly calls are little

more than someone reading the documents on the phone. She explained that as a result,

audiences don’t feel a sense of urgency. She added that there are disconnects between the

delivery of the call and the information reaching customer-facing employees. In fact, people

give out codes for weekly update call attendance; so many supervisors may not even listen to the

calls first-hand.

She continued to explain that there is a lift in participation if the director is enforcing the

priority of the weekly updates; but it must continue to be a priority. When the director enforces

it, it becomes a priority. But they key is that this be driven from executive leaders. Participant B

added that it really depends on the priority of the moment. Priorities of the moment make

weekly updates seem like something that can be pushed aside. She agreed that it needs to be a

consistent message from supervisors, managers, and directors.

This sparked a conversation about ownership. Both participants explained that the

training department acts as a messenger for certain information in weekly updates, but even then,

Training does not own the initiative. Participant A stated, “It is important to the corporate

training teams and operational teams, but it lacks priority and accountability on the front line.”

Not only did this portion of the focus group identify gaps in the current process, but it

also led to ideas to improve the situation. Leadership support is a vital component of making any

process work and seemed to be a gap in the existing process.

Delivery Methods

In regard to the methods of training delivery, Participant B stated that instructor led

training (ILT) has a stronger take-away if done by a trainer (rather than a supervisor) because

trainers have the ability to make sure the transfer of knowledge occurs. To illustrate this point,

she stated, “It’s more focused and intense.” Employees know it has an impact and importance

because it was chosen for “training;” in team meetings, the message doesn’t seem as important.

Since one method of delivery not only has a stronger take-away, but it sends an unspoken

message to employees about the importance, urgency, and priority of the content, a possible gap

in the process may be linked to the unspoken connotations that are tied to each method of

delivery.

Suggestions for Improvements

Participant B continued to explain that to improve the extent to which customer-facing

employees stay abreast of company changes, there must be an increase in consistency in

following protocol. Participant A agreed and stated that Edelweiss Communications as a whole

needs to focus on the weekly updates rather than the weekly update call. She recommended

creating a check and balance system in team meeting environments – a sign-off process to help

develop tracking methods for team meetings. She also mentioned the idea of an online survey

where employees could select topics that they did or did not receive training on. Participant B

agreed that there needs to be a consistent message from supervisors, managers, and directors,

“There needs to be accountability, monitoring, and measurement.” Each of these ideas isolate

and identify key components that could contribute to process improvements and are directly

linked to the research question, “What are possible processes that may improve then state of the

situation?” as found in Appendix D.

Reaction to the Assessment

In reaction to the assessment Edelweiss Communications distributed as a method of

evaluation to help identify gaps in processes, Participant A stated that she hasn’t heard a lot of

noise and that may be because she had not received the results or scores yet. Both participants

agreed that an assessment is not the best way to evaluate if customer-facing employees are

staying abreast of company changes or to find gaps in processes. Rather, they stated that we can

get the best snapshot of gaps in the process by conducting focus groups with employees.

Although my participants had the largest amount of visibility into possible issues, their

feedback is only a part of the first cycle of action research. They are considered one of the

primary stakeholders, but employees and their supervisors (secondary stakeholders) can provide

valuable feedback about the causes and effects of employees failing to stay abreast of company

changes as well. And like the training managers I interviewed, employees and their supervisors

can provide valuable insight into possible solutions and the most effective methods of evaluation

to help identify gaps in processes. The next cycle of action research will include focus groups

with them to help further clarify and add information from a larger pool of stakeholders.

Limitations

By using both quantitative and qualitative data, I was able to implement the concept of

triangulation, which states that, “The credibility of a study is enhanced when multiple sources of

information are incorporated” (Stringer, 2007, p. 58). However, complete triangulation is not

fully achieved at this time. It’s imperative to include qualitative analysis that typifies the points

of views of both primary and secondary stakeholders. At this time, I have only accounted for a

sample of primary stakeholders. There are many additional parties with whom I plan to conduct

focus groups such as employees and their supervisors, the content management team that

develops the weekly updates, and trainers. Once these items are complete and taken into

account, the triangulation of this research project will be much stronger.

Comparison of the Research to Literature Review

The literature review that I conducted explored existing content about how evaluation of

training initiatives helps corporate organizations identify and implement process improvements.

Models and concepts explored included Kaizen, Six Sigma, Lean, and Lean Six Sigma. The

literature review allowed for study of each methodology in terms of origin, structure or

framework, how they work, and the extent of their success.

The review also studied findings on Kirkpatrick’s model of training evaluation in an

attempt to find correlations between Edelweiss Communications’ current method of evaluation

and how that may fit into the context of process improvements.

My study takes into account methods of evaluation, whereas many articles within my

literature review did not. However, the method used in the study thus far, Kirkpatrick’s Level

Two assessments, is not sufficient evidence on which to identify gaps in the process properly.

This issue will be resolved by incorporating more qualitative analysis into further cycles of this

study.

My study failed to examine methods in which Edelweiss Communication can improve

the processes that may not be working in its culture. The literature I studied covered methods

such as Kaizen, Six Sigma, Lean, and Lean Six Sigma effectively and provided a valuable

resource for me to tap into once I reach that phase of the research process. This cycle of research

has clearly shown that improvements to Edelweiss Communications’ current processes are

necessary. The following cycles will gather additional data from various points of view to

reinforce this idea and explore methods of improvement. Once data is collected effectively,

Edelweiss Communications will be in a position to select a process improvement method that

meets the needs of Edelweiss Communications effectively.

Summary of Findings

Quantitative data analysis showed that the success rate for over 25% of the questions was

below expectations. This global observation signified that employees were not staying abreast of

company changes. It also indicated that the procedure in which weekly updates is distributed is

not effective enough. Causes include: (a) employees may not have used the Intranet properly,

(b) they may not have received the information as required in team meetings, and (c) reading

comprehension may have influenced the data collected. This analysis indicated that Level Two

analysis may be an effective method of evaluation to help identify gaps in processes since

processes were prevalent in some of these findings. However, the qualitative analysis revealed

that focus groups including all the primary and secondary stakeholders may provide different and

valuable insight into how and why process improvements may be necessary. The feedback

already collected qualitatively reveals a number of promising improvement ideas.

Implications for Practice

Based on these findings, it is evident that Edelweiss Communications must invest

additional time into collecting qualitative data from a more diverse group of stakeholders. It

must also (a) identify particular processes that are the causes of core problems, (b) select a

methodology of process improvement, (c) design a plan to improve the process, and (d)

implement the plan into action.

This initiative will contribute to Edelweiss Communications’ ability to assess and

identify gaps in additional areas such as the effectiveness of weekly updates, training content,

and delivery methods. By reassessing the priority and accountability levels of the weekly

updates, Edelweiss Communications may be able to stimulate improvements.

Conclusion

The quantitative findings of this study clearly indicate that there is a gap present.

However, this data does not reveal the source of the gap. An analysis of questions breaks down

the findings into more tangible possibilities. For example, the three most prevalent possibilities

found in this analysis include (a) improperly written questions, (b) convoluted policies and

procedures, or (c) insufficient reading comprehension skills. These hypotheses of causes are

helpful, but are made more complete once the qualitative data of this study is integrated into the

findings. The qualitative data gathered in this study revealed that employees may not have

received the information as required in team meetings or may not have received it correctly.

However, the results found within this study are applicable to Edelweiss Communications

only. It is possible to replicate the study in another corporate environment, and is recommended

to do so, but both quantitative and qualitative data are unique to Edelweiss Communications’

unique processes, policies, procedures, and culture. However, because of this tailored approach,

the findings are incredible valuable, if only to a small group of individuals.

An additional limitation is the lack of qualitative data. This is in part due to the (a) lack

of time available to complete this phase of the study, (b) lack of departmental involvement in the

study, and (c) lack of resources which made it difficult to identify and locate individuals who

would be able to contribute to the study. Only a small number of primary stakeholders were

interviewed in this phase of the process. Yet, this phase of the study revealed that other groups

must be brought into the process and their thoughts and ideas must be integrated into the study.

Since this is only phase one of the study, there will be ample time to put this into action, and the

data that was collected provided further justification to do so.

Ultimately, this study has proven beneficial for many parties. The information collected

throughout this study will help the training department identify the strengths and weaknesses of

their participation in the weekly update process. It will also help the content management

department identify possible the strengths and weaknesses of their participation and ownership of

the weekly update process. And finally, it will help employees explain possible low scores; it is

unreasonable to expect employees to achieve high scores on an assessment about information

that was not delivered, or not delivered correctly.

This study has helped Edelweiss Communication take its first steps to help accomplish its

ultimate goal of helping employees stay abreast of company changes.

References

Bendell, T. (2005). Structuring business process improvement methodologies. [Editorial]. [Electronic version]. Total Quality Management, Vol. 16, No. 8–9, 969-978.

Bernett, R., & Nentl, N. (2010). Opinions and expectations about continuous improvement programs. [Editorial]. [Electronic version]. The Journal For Quality & Participation, January 2010, 35-38.

Carter, L., & Carmichael , P. (2009). Best practices go beyond benchmarking. [Editorial]. [Electronic version]. Leadership Excellence, November 2009, 16-17.

Caulcutt, R. (2001). Why is Six Sigma so successful? Journal of Applied Statistics, 28, 301 306.

Charlesworth, K., A question of quality? A survey of Performance improvement initiatives, 2000 (British Quality Foundation, Institute of Management: London).

Dictionary.com. (2010). Retrieved June 21, 2010 from Correlation: http://dictionary.reference.com/browse/correlation.

Dictionary.com. (2010). Retrieved June 21, 2010 from Pattern: http://dictionary.reference.com/browse/pattern.

Evans, J.R. and Lindsay, W.M., The management and control of quality, 6th ed., 2005 (South Western College Publishers: Ohio).

Fraser, N., & Fraser, J. (2009). Lean Six Sigma applied to a customer service process within a commercial finance organization. [Editorial]. [Electronic version]. Management Services, Autumn 2009, 25-31.

Grant, V. (2008). Lean Six Sigma. [Editorial]. [Electronic version]. Management Services, Spring 2008, 20-23.

Gygi, C., N. DeCarlo, and B. Williams, Six Sigma for Dummies, Wiley Publishing, Inc. (2005).

Jones, E. C., Parast, M. M., & Adams, S. G. (2010). A framework for effective Six Sigma implementation. [Editorial]. [Electronic version]. Total Quality Management, Vol. 21, No. 4, 415-424.

Knopf, J. W. (2006). Doing a literature review. [Editorial]. [Electronic version]. PS: Political Science & Politics, January 2006, 127-132.

http://dictionary.reference.com/?browse/?correlation

http://dictionary.reference.com/?browse/?pattern

http://dictionary.reference.com/?browse/?pattern

Kirkpatrick, D. L. (1959) Evaluating training programs, 2nd ed., Berrett Koehler, San Francisco.

Kurtus, R. (2002). Steps to improve or grow your business. Retrieved June 21, 2010 from schoolforchampions.com: http://schoolforchampions.com/business/improvement_steps.htm.

Lasa, I. S., De Castro, R., & Laburu, C. O. (2009). Extent of the use of Lean concepts proposed for a value stream mapping application. [Editorial]. [Electronic version]. Production Planning & Control, Vol. 20, No. 1, 82-98.

Lean Terms & Definitions . (2010). Retrieved April 10, 2010 from The Northwest Lean Networks: http://www.nwlean.net/leandefs.htm#7wastes.

Li, G., and Rajagopalan, S. (2008). Process improvement, learning, and real options. [Editorial]. [Electronic version]. Production and Operations Management, Vol. 17, No. 1, 61-74.

Lupton, Robert A., John E. Weiss, and Robin T. Peterson (1999), “Sales Training Evaluation Model (STEM): A Conceptual Framework,” Industrial Marketing Management, 28, 1 (January), 73-86.

Matson, E., and Stauffer, L. (2009). Developing an assessment tool for two organizations using Six Sigma principles. [Editorial]. [Electronic version]. Engineering Management Journal, Vol. 21 No. 4, 7-15.

Naumann, E., and Giel, K. (1995). Customer satisfaction measurement and management. Wisconsin: ASQ Quality Press.

Ortiz, C. (2006). All-out kaizen. [Editorial]. [Electronic version]. Industrial Engineer, April 2006, 30-34.

Pakdil, F. Özkök, O. Dengiz, B. Kara, I. Selvi, N. Kargi, A. (2009). A systematic approach to reduce human and system-related errors causing customer dissatisfaction in a production environment. [Editorial]. [Electronic version].Total Quality Management, Vol. 20, No. 1, 129-137.

Pande, P.S., Neuman, R.P., & Cavanagh, R.R. (2000). The Six Sigma way: How GE, Motorola and other top companies are honing their performance. New York: McGraw-Hill.

Reichfeld, F. and Sasser, W.E. (1990). “Zero Defections: Quality Comes to Service”. Harvard Business Review, September – October, 1-8.

Roberts, D. (2010). Statistical Studies. Retrieved June 21, 2010 from Statistical Studies: http://www.regentsprep.org/regents/math/algtrig/ats1/statsurveylesson.htm.

http://schoolforchampions.com/?business/?improvement_steps.htm

http://www.regentsprep.org/?regents/?math/?algtrig/?ats1/?statsurveylesson.htm

Rowden, R. W. (1998). The return-on-investment from training. [Editorial]. [Electronic version]. Business Forum, Volume 27 Number 1, 31-36.

Shah, R., Chandrasekaran, A., & Linderman, K. (2008). In pursuit of implementation patterns: the context of Lean and Six Sigma. [Editorial]. [Electronic version]. International Journal of Production Research, Vol. 46, No. 23, 6679-6699.

Stringer, E. T. (2007). Action Research. Thousand Oaks, California: Sage Publications.

Tarantino, A. (2009). Risk management for the next generation. [Editorial]. [Electronic version]. Industrial Management , january/february 2009, 14-17.

Thawesaengskulthai, N., & Tannock, J. (2008). A decision aid for selecting improvement methodologies. [Editorial]. [Electronic version]. International Journal of Production Research, Vol. 46, No. 23, 6721-6737.

Toronto. (2010). Retrieved June 21, 2010 from Glossary of Terms: http://www.toronto.ca/lobbying/glossary.htm.

Wayne State University. (2010). Retrieved June 21, 2010 from eLearning Glossary: http://www.eng.wayne.edu/page.php?id=1263.

Wikipedia.org. (2010). Retrieved June 21, 2010 from Customer Experience: http://en.wikipedia.org/wiki/customer_experience.

Wikipedia.org. (2010). Retrieved June 21, 2010 from Instructor-led training: http://en.wikipedia.org/wiki/instructor-led_training.

Wikipedia.org. (2010). Retrieved June 21, 2010 from Procedures: http://en.wikipedia.org/wiki/procedures.

Womack, J.P., Jones D.T., and Ross D., The Machine that Changed the World, Rawson Associates, New York, 1990, 323 PP.

http://www.toronto.ca/?lobbying/?glossary.htm

http://www.toronto.ca/?lobbying/?glossary.htm

http://www.eng.wayne.edu/?page.php?id=1263

http://en.wikipedia.org/?wiki/?customer_experience

http://en.wikipedia.org/?wiki/?instructor-led_training

http://en.wikipedia.org/?wiki/?procedures

http://en.wikipedia.org/?wiki/?procedures

Appendix A – Six Sigma Flows and Definitions

Shared Define During the Define phase, organizations establish project parameters, review and

define goals, as well as identify customer needs and processes/products to be improved.

Measure Measuring consists of evaluating the performance of the core business process and collect data needed for future comparisons.

Analyze The purpose of the Analyze phase is to identify the causes of poor performance and to determine how the process can be improved.

DMAIC DMADV Improve The purpose of the Improve

phase is to address the root causes of poor performance.

Control In the Control phase, organizations ensure that the improved procedures and tools are consistently applied.

Design In the Design phase, organizations design, develop, and implement new procedures or tools.

Verify In the Verify phase, organizations ensure that the new procedures or tools are consistently applied.

(Matson & Stauffer, 2009)

Appendix B – Lean Definitions

Principle Term Definition Elimination of waste Waste Anything a paying customer is not willing to pay

for. 7 categories of waste: • Over-production • Waiting • Transportation • Inventory • Motion • Over-processing • Defects

Identification of the Value Stream

Value Anything a paying customer is willing to pay for

Achievement of Flow through the process

Flow Continuous evaluation of inventories, batch sizes, and work methods & distribution

Pacing by a pull signal Pull Production is restricted to actual demand rather than capacity

Continuous pursuit of perfection

n/a

(NWLEAN Inc, 2010)

Appendix C – Focus Group Disclosures

I’m asking you to participate as a subject in a research project entitled "Weekly Updates

and Process Improvements.” You’ve been selected for this focus group because you oversee the

largest variety of site locations. The purpose of this study is to help customer-facing employees

stay abreast of company changes. By identifying gaps and analyzing possible causes for them,

the training department is equipped to assess the strengths and weaknesses of our current

processes.

If you choose to participate in this focus group your responses will be recorded and

shared only with myself and the project sponsor. If you have additional information that you

would like to share but not record, that is an option. We ask that you are as objective and open

as possible. There are no reasonably foreseeable risks associated with participating in this study.

There may not be direct benefits to you from participating in this study.

Participation is voluntary. You have the right to refuse to be in this study. If you decide

to be in the study and change your mind, you have the right to discontinue participation at any

time. However, any information already used up to the point when you withdraw consent, will

not be removed from the study’s data. Whatever you decide, you will not lose any benefits to

which you are otherwise entitled.

This report will not contain personal identifying information such as your name or

department. Your participation will not be identified and information you provide will be kept

confidential and not reported to others outside the research project in a way that personally

identifies you. You may ask any questions about this project of the researcher or project sponsor

at any time.

Appendix D – Focus Group Questions

1. To what extent do you feel customer-facing employees are currently staying abreast of

company changes that are outlined in weekly updates?

2. How are processes involved in the retention of weekly updates?

3. How would you describe the current situations surrounding how we release weekly

updates?

a. What is working?

b. What is not working?

4. Are things different in different locations?

a. If so, how are they different?

5. Do you see any trends between delivery methods and how well employees are retaining

the information in the weekly updates?

6. What is the current priority of the involvement of employees and supervisors as related to

the weekly updates?

7. Is there consistency among how supervisors reinforce the importance of the weekly

updates?

8. What motivates employees to care about the weekly updates?

9. Are there consequences for employees who do not stay abreast of weekly updates?

10. How did employees receive the assessment we released for Q1?

11. Do you think the assessment is an accurate way to see if employees are retaining

information from the weekly updates

a. What worked?

b. What didn’t?

Appendix E – Average Pass Percentages and Average Scores

Site Type Total

Count

Passed % Passed Ave

Score %

Forsythia Employee 265 184 69% 58%

Supervisor 11 11 100% 96%

Total 276 195 71% 83%

Gloxinia Employee 227 159 70% 82%


Total 241 169 70% 82%

Laburnum Employee 344 196 57% 80%


Total 355 205 58% 80%

Wisteria Employee 101 69 68% 84%


Total 104 72 69% 84%

Whortleberry Employee 181 116 64% 84%


Total 189 124 66% 84%

Rhododendron Employee 197 95 48% 70%


Total 208 103 50% 79%

Snapdragon Employee 227 185 81% 85%


Total 234 192 82% 86%

Valerian Employee 397 284 72% 85%


Total 421 306 73% 85%

Across All Sites 2028 1366 67% 83%

Appendix F - Percent Passed Detail – Listed by Site Location

Site

Manager

Employees Failed

Employees Passed

Grand Total

Percentage Passed

Forsythia Supervisor Name 12 2 14 14%

Supervisor Name 6 5 11 45%
















Supervisor Name 13 13 100%






Forsythia Totals 79 194 273 71%

Gloxinia Supervisor Name 4 4 0%





























Gloxinia Totals 63 154 217 71%

Laburnum Supervisor Name 4 4 0%




















































Laburnum Totals

142 191 333 57%

Wisteria Supervisor Name 1 1 0%














Wisteria Totals 33 69 102 68%

Whortleberry Supervisor Name 9 2 11 18%
























Whortleberry Totals

57 116 173 67%

Rhododendron Supervisor Name 1 1 0%



























Rhododendron Totals

102 103 205 50%

Snapdragon Supervisor Name 4 4 8 50%



























Snapdragon Totals

39 157 196 80%

Valerian Supervisor Name 11 6 17 35%
































Valerian Totals 115 304 419 73%

1918

Appendix G - Question Totals – Listed by Site Location

Question

Totals

Forsythia

Gloxinia

Laburnum

Wisteria

Whortleberry

Rhododendron

Snapdragon

Valerian Delivery

Method

Sampl

e Que

stio

n 1

(and

cor

rect ans

wer)

94%

81%

94

%

98%

98

%

93%

91

%

98%

97

%

Non

e Sa

mpl

e Que

stio

n 2

(and

cor

rect ans

wer)

69%

49%

62

%

75%

50

%

83%

74

%

89%

67

%

Non

e Sa

mpl

e Que

stio

n 3

(and

cor

rect ans

wer)

96%

92%

95

%

95%

98

%

96%

95

%

98%

95

%

Non

e Sa

mpl

e Que

stio

n 4

(and

cor

rect ans

wer)

75%

81%

65

%

69%

75

%

87%

64

%

85%

73

%

Non

e Sa

mpl

e Que

stio

n 5

(and

cor

rect ans

wer)

86%

88%

82

%

81%

86

%

86%

83

%

91%

87

%

Non

e Sa

mpl

e Que

stio

n 6

(and

cor

rect ans

wer)

92%

99%

95

%

88%

83

%

98%

83

%

94%

97

%

Non

e

Sampl

e Que

stio

n 7

(and

cor

rect ans

wer)

81%

77%

78

%

81%

89

%

83%

80

%

85%

78

%

Non

e Sa

mpl

e Que

stio

n 8

(and

cor

rect ans

wer)

70%

76%

72

%

64%

70

%

67%

65

%

82%

69

%

Tea

m M

eetin

g

Sampl

e Que

stio

n 9

(and

cor

rect ans

wer)

93%

96%

93

%

91%

92

%

99%

90

%

96%

90

%

Tea

m M

eetin

g

Sampl

e Que

stio

n 10

(a

nd cor

rect ans

wer)

97%

98%

96

%

94%

98

%

98%

95

%

99%

97

%

Non

e Sa

mpl

e Que

stio

n 11

(a

nd cor

rect ans

wer)

83%

84%

84

%

83%

80

%

83%

79

%

87%

82

%

Non

e Sa

mpl

e Que

stio

n 12

(a

nd cor

rect ans

wer)

86%

84%

91

%

85%

90

%

87%

72

%

92%

88

%

Non

e

Sampl

e Que

stio

n 13

(a

nd cor

rect ans

wer)

77%

74%

77

%

78%

73

%

77%

73

%

89%

77

%

Non

e Sa

mpl

e Que

stio

n 14

(a

nd cor

rect ans

wer)

65%

54%

72

%

61%

65

%

62%

55

%

77%

72

%

Non

e Sa

mpl

e Que

stio

n 15

(a

nd cor

rect ans

wer)

81%

65%

78

%

80%

96

%

69%

87

%

85%

89

%

Non

e Sa

mpl

e Que

stio

n 16

(a

nd cor

rect ans

wer)

95%

98%

96

%

84%

98

%

99%

92

%

93%

97

%

Non

e Sa

mpl

e Que

stio

n 17

(a

nd cor

rect ans

wer)

73%

70%

78

%

67%

71

%

76%

70

%

79%

75

%

Non

e Sa

mpl

e Que

stio

n 18

(a

nd cor

rect ans

wer)

95%

94%

96

%

93%

97

%

92%

98

%

95%

97

%

10 M

inutes

Tea

m M

eetin

g

Sampl

e Que

stio

n 19

(a

nd cor

rect ans

wer)

54%

51%

56

%

58%

55

%

51%

44

%

68%

51

%

10 M

inutes

Tea

m M

eetin

g

Sampl

e Que

stio

n 20

(a

nd cor

rect ans

wer)

92%

92%

93

%

89%

89

%

91%

91

%

98%

90

%

10 M

inutes

Tea

m M

eetin

g

Sampl

e Que

stio

n 21

(a

nd cor

rect ans

wer)

94%

96%

96

%

92%

96

%

94%

89

%

90%

95

%

10 M

inutes

Tea

m M

eetin

g

Sampl

e Que

stio

n 22

(a

nd cor

rect ans

wer)

93%

97%

96

%

86%

95

%

98%

85

%

95%

95

%

Non

e Sa

mpl

e Que

stio

n 23

(a

nd cor

rect ans

wer)

90%

90%

95

%

86%

87

%

86%

90

%

93%

93

%

Non

e Sa

mpl

e Que

stio

n 24

(a

nd cor

rect ans

wer)

76%

80%

75

%

71%

78

%

76%

71

%

80%

79

%

Non

e Sa

mpl

e Que

stio

n 25

(a

nd cor

rect ans

wer)

72%

87%

77

%

62%

64

%

67%

48

%

81%

89

%

10 M

inutes

Tea

m M

eetin

g

Sampl

e Que

stio

n 26

(a

nd cor

rect ans

wer)

85%

96%

93

%

72%

74

%

94%

66

%

91%

96

%

Non

e Sa

mpl

e Que

stio

n 27

(a

nd cor

rect ans

wer)

92%

97%

90

%

84%

94

%

91%

89

%

94%

95

%

Non

e Sa

mpl

e Que

stio

n 28

(a

nd cor

rect ans

wer)

92%

97%

95

%

96%

95

%

93%

97

%

65%

96

%

Intern

al

Cen

ters

Sampl

e Que

stio

n 29

(a

nd cor

rect ans

wer)

91%

94%

93

%

84%

89

%

91%

90

%

93%

91

%

1 ho

ur IL

T

Sampl

e Que

stio

n 30

(a

nd cor

rect ans

wer)

84%

94%

89

%

75%

77

%

86%

72

%

89%

91

%

10 M

inutes

Tea

m M

eetin

g

Total

84%

84%

85

%

81%

83

%

85%

79

%

88%

86

%

Running Head: IDENTIFYING PROCESS GAPS THROUGH EVALUATION 1

Appendix H – Question Breakdown for Low Scores

Question Number

Correct Percentage

Possible Causes of Low Scores

19 54% Upon assessing the Policy and Procedure, this fact was stated clearly. It’s possible that this was a new procedure and was not given enough attention in team meetings. It’s also possible that this is a situation employees do not come across often.

14 65% If employees used Edelweiss’ Intranet for this question, it’s possible that they expected this information to be in the applicable Policy, but it is in the correlated Procedure. The Policy does refer to the Procedure properly and effectively.

2 69% Employees may have read the question incorrectly (“are not” rather than “are”). If employees in different queues have different procedures, this may have caused confusion.

8 70% It's possible that the product name caused confusion because it includes a written numeric value. However, the options available were not numerically close to the product name and would not cause further confusion. Also, only Wisteria and Snapdragon received team meeting time. This was not likely to be an area of focus for other centers.

17 73% There were numerous changes concerning this release with no training time scheduled. The date of the release changed and was announced the following week. The change of this related item may have caused employees to unconsciously add a sense of complication to the question.

25 72% This was a change of policy and was written effectively and accurately in the Policy and Procedure. It's possible this may not have been communicated effectively in team meetings.

4 75% There are four pieces of information in each response. This may have caused confusion. The possible answers to this question were extremely similar and lengthy. It's possible that some employees may have known the correct answer but selected the wrong option due to reading comprehension

24 76% This was not new information. The weekly update explained a deviation from this information for some areas. Therefore, employees may have chosen either of the two other distracters because they assumed the question would be focusing on a change.

IDENTIFYING PROCESS GAPS THROUGH EVALUATION 60

Appendix I – Definitions

Correlation - The simultaneous change in value of two numerically valued variables

(Dictionary.com, 2010)

Customer Experience – The sum of all experiences a customer has with a supplier of goods or

services, over the duration of their relationship with that supplier (Wikipedia.org, 2010)

Customer-Facing Employees – People who are paid for the performance of their duties by

Edelweiss Communications who speak directly to customers (in part from: Toronto, 2010)

eLearning – Internet-enabled learning that encompasses training, education, just-in-time

information, and communication (Wayne State University, 2010)

Instructor Led Training – (ILT) The practice of face-to-face delivery of training and learning

material between an instructor and learners, either individuals or groups (Wikipedia.org, 2010)

Pattern - A combination of qualities, acts, tendencies, etc., forming a consistent or characteristic

arrangement (Dictionary.com, 2010)

Policies – The rules, conventions, and protocols adopted by Edelweiss Communications, of

which it requires employees to adhere

Procedures – A specified series of actions executed in the same manner to always obtain the

same result under the same circumstances (Wikipedia.org, 2010)

Team Meetings – Meetings where employees collaborate in a group to share information

facilitated by employees’ direct supervisors

Final Action Research Report

Documents

development

improperly

studys core

process improvement

content management

weekly update

total quality

ensure proper