Using Open Badges to Certify Practicing Evaluators

1    

Using Open Badges to Certify Practicing Evaluators

Randy Davies

Richard E. West

Daniel L. Randall

Brigham Young University

Full Citation of article published in the American Journal of Evaluation.

Davies, R., Randall, D., & West, R. E. (2015). Using open badges to certify practicing evaluators. American Journal of Evaluation, 36(2), 151-163. (T1; AR=20-25%).

2    

Using Open Badges to Certify Practicing Evaluators

For several decades members of the American Evaluation Association (AEA) have

discussed and disputed the issue of evaluator certification (Becker & Kirkhart, 1981; Worthen,

1972). In 1999 Jones and Worthen correctly predicted that the complexity of the issue would

prolong the dispute for some time. The topic of certification was much debated in 1999

(Altschuld, 1999; Bickman, 1999; Jones & Worthen, 1999; Worthen, 1999), and discussion has

continued sporadically in academic journals and AEA conference sessions (e.g., Dewey et al.,

2008). Clearly not everyone feels the AEA needs an evaluator certification. Efforts by the AEA

and others to increase prominence and use of evaluation seem to have been somewhat successful

(LaVelle & Donaldson, 2010), and many practicing evaluators may feel their work would not be

greatly impacted by a designation of certified evaluator. However, discussion regarding

certification often includes the opinion that an evaluator certification process would be valuable,

and some believe it may become required for practitioners working with government

organizations (Smith et al., 2011). Yet, we do not seem to be any closer to having a formal

certification process endorsed by the AEA than we were a decade ago.

Many in the AEA seem to have concluded that an evaluator certification system may be

overly complicated and controversial (Morris, 2011). However, since 1999 much has happened

in technology and social media innovation that may make a certification system easier to create

and manage. If an evaluator certification for the AEA were to become a reality, several issues

would need to be addressed. First, we would need to come to some agreement on the

combination of knowledge, skills, and experience necessary for an individual to be considered a

qualified evaluator. Second, a method would be required to assess evaluator credentials. Finally,

3    

we would need to implement a means for administering and maintaining the system. In addition,

several questions regarding policy would need to be resolved (e.g., requirements for

recertification, exceptions for experience, and recognition of specialization).

The purpose of this paper is to open a dialogue in the community of evaluators to explore

the possibility, challenges, and benefits of establishing a certification system for AEA evaluators

based on digital open badges. Although in its early stages of development, open badging offers

several potential benefits for overcoming many of the limitations and challenges associated with

traditional certification systems. In this paper we outline the current status of evaluator

certification efforts, including limitations and potential. We then propose a certification system

based on digital open badges that could provide the AEA with a flexible, efficient, yet rigorous

way to certify evaluators.

Certification as a Possibility and a Challenge

Certification Language and Forms

In an article published in the American Journal of Evaluation in 1999, Altschuld

summarized a report submitted to the AEA Board of Directors regarding evaluator certification

which defined various forms of evaluation with their relevant terms including credentialing,

certifying, and licensing. He pointed out that credentialing acknowledges that an individual has

completed training or participated in a field experience or practicum. Certification assesses an

individual’s knowledge, skills, and abilities, verifying that he or she has attained a certain level

of expertise. Licensing, however, grants formal or legal permission to engage in professional

practice for the purpose of earning a livelihood. These distinctions are important and useful.

While most of us would agree that AEA is not likely to pursue a licensing process for a

variety of legal and practical reasons, it may be helpful for evaluators and potential clients to

4    

have a credentialing or certifying mechanism. This process would not hold the AEA as a

professional organization liable for guaranteeing or policing the quality of work being conducted

by a licensed evaluator. But it would be useful to potential clients and other interested

stakeholders (Smith et al., 2011) to have perhaps a registry of practicing evaluators in their area

who have been trained and have in the past demonstrated some level of evaluation skill and

expertise. The problem is how to achieve this goal in an effective and efficient manner for both

those wishing to obtain certification and those (such as AEA) willing to providing it.

Application of a Credentialed Evaluator Designation

One working example of an evaluator credential verification program is the Canadian

Evaluation Society’s (CES) credentialed evaluator (CE) designation for its members. The CES

is careful to point out, for legal and practical reasons, that this designation is not a certification of

evaluation proficiency or a license to practice evaluation: It simply verifies “the holder has

provided evidence of the education and experience required by the CES to be a competent

evaluator” (http://www.evaluationcanada.ca, np).

The Canadian system is a voluntary designation that includes some aspects of

certification in addition to a credential verification process. Individuals wishing to obtain the CE

designation must provide evidence that they have obtained a graduate-level degree or certificate,

although accommodations can be made through a prior learning assessment for individuals who

do not have a graduate-level degree. Candidates for the CE designation must have at least two

years of evaluation-related work experience within the last ten years, with letters of reference

from clients. Candidates must also pass a test requiring a series of narratives (each 150 words or

less) adequately addressing at least 70% of the specified evaluator competencies in each of five

domains. Members of a credentialing board consisting of evaluation experts have authority and

5    

responsibility to award the designation. In addition, credentialed evaluators are required to

complete ongoing training (40 hours every three years) in order to maintain their status.

Basically a credentialed evaluator must show evidence of training and experience then pass a

written assessment. This application and renewal process is implemented though an online

system designed to manage applications and candidate accounts, with a fee to cover

administration costs.

The Canadian CE designation is definitely a commendable effort that seems to be

working; however, it faces several implementation challenges including training confirmation,

assessment validity, experience verification, and program administration. These aspects should

be considered when contemplating a comparable AEA evaluator certification process.

Training concerns. Worthen (1999) recommended that the profession begin evaluator

certification by implementing an accreditation system for institutions that provide evaluation

training, pointing out that the low quality of many formal evaluations is traceable to inadequate

or irrelevant academic preparation (see also Christie, 2003). Similarly, Dewey et al. (2008) found

that many skills desired by clients (e.g., interpersonal and project management skills) do not

match the skills an evaluator acquires in graduate school. In addition, not all evaluation training

programs, largely provided by universities, offer comprehensive training for prospective

evaluators. Programs providing evaluator training show considerable diversity in the number of

required courses and the specific topics covered in these courses (Davies & McKay, 2014).

Similarly, the notion of capacity building as a means of increasing evaluation use encourages

untrained individuals to gain evaluation skills and build evaluation processes into their programs.

As a result, many individuals practice evaluation or gain employment as evaluators with

relatively no formal training. These individuals have gained some skill as evaluators but may not

6    

receive recognition for the training or experience they have obtained as they likely would not

receive a degree in evaluation based solely on the training they received.

At issue for any evaluator certification program that requires evidence of evaluation

training based on a degree or certificate is the variability in quality and breadth of training

evaluators might receive at different institutions. Degrees do not always communicate the

specific competencies their graduates have gained (Matkin, 2012), creating a need to accredit

evaluation degree programs so that training in critical competencies is assured. Of concern for

some who are seeking certification is the fact that many competent individuals have received

evaluation training but never obtained a degree or certificate. Thus a profession interested in

certification must establish clear criteria to designate adequate training, along with directions for

those who do not have a degree from an authorized provider to supply evidence that they have

equivalent skills and experience.

Assessment concerns. Altschuld (1999) pointed out that the primary method for

certifying members of a profession is to administer a test. The purpose of an evaluator

certification assessment is to verify that the individual has the requisite knowledge and

understanding expected of a qualified evaluator. This specific aspect of certification (i.e.,

candidate assessment) can be a major stumbling block for establishing an evaluator certification

process. The costs associated with construction and administration of a single valid summative

assessment that adequately covers the knowledge, ability, and understanding required of an

evaluator is a daunting endeavor.

At issue for any evaluator certification program utilizing a single assessment of evaluator

competencies are validity and efficiency. The validity issue is complicated by the diversity of

contexts in which evaluation is practiced. The evaluation context often demands different skills

7    

of the evaluator. Professions with certification and licensing exams must invest considerable

time, effort, and funds to maintain the integrity and validity of the assessment—often a business

in and of itself. In addition, organizers must consider who will have the authority and

responsibility to make the assessment and whether a single assessment is the best approach given

the diversity and extent of learning expected of a qualified evaluator.

Experience concerns. Training evaluators is important, but training by itself may not

mean an individual should be certified as qualified for this work. Training that includes practical

experience is a good predictor of success (Dewey et al., 2008; Worthen, 1999; Yarbrough et al.,

2011), but the addition of experience does not guarantee an evaluator will consistently complete

satisfactory evaluations for all clients. Many factors must be taken into account. For example, an

individual may have participated as a member of an evaluation team tasked with a small but

important part of a project (e.g., data collection or statistical analysis). Another evaluator might

have extensive experience but limited to evaluations in a specific field utilizing a familiar

evaluation approach. Another might have experience working independently on small-scale

evaluations of short duration. Still another might have worked exclusively as a member of an

evaluation team on large-scale multi-year evaluations.

Any evaluator certification program faces the challenge of assessing the quality and

extent of an individual’s experience. The subjective nature of the assessment must be

recognized, as well as the necessity to differentiate the type and focus of the experience. An

accurate and fair assessment must include not just the amount but the type of experience an

individual acquires. A well-structured evaluator certification would also need to communicate

the breadth and type of experience the evaluator possessed.

8    

Administration and implementation issues. Numerous challenges are associated with

implementing and maintaining an effective certification process. Clearly costs are involved in

establishing and maintaining a system to manage candidates’ applications and reviews, as well as

compile lists of successful applicants. Costs also include the time and effort of individuals

operating the system, including administrators and authorized reviewers. Technology has greatly

enhanced our ability to manage these types of systems, but systems still need to be created and

maintained.

Badges as a Potential Solution to Certification

One possible way to initiate and maintain an evaluator certification system is to adopt an

open badge framework. The open badge concept utilizes a micro-certification strategy with a

top-down design process that breaks up the larger certification problem into flexible, manageable

units (Glover & Larif, 2013). Although few examples of quality open badge certification systems

are available, some online educational communities and universities have utilized open badge

technology to provide individuals with evidence that they possess certain skills, have realized an

accomplishment, or have completed specific training (Young, 2012). In order to judge the

potential of open badges for evaluator certification, we first need to understand how they are

developed, how they are currently deployed, and how they may have been inappropriately

applied as a certification system.

The Concept of Badges

Historically, physical badges were used to represent an individual’s position within a

field, community, or profession as well as the authority an individual might possess. For

example, law enforcement and military badges inform others of the rank and position of

authority an individual holds. Badges are also used to acknowledge an achievement or

9    

acquisition of a skill. Perhaps the most common example is the merit badge system used in many

youth programs (e.g., the boy and girl scouts).

In recent years the digital badge has emerged, with a digital image replacing the physical

badge (Brandon, 2013). These digital badges have been used by social networks to acknowledge

popular venues or establish the reputation of a contributor (Antin & Churchill, 2011; Easley &

Ghosh, 2013). In educational systems, such as Khan Academy, Duolingo, and http://diy.org,

badge systems are used to motivate students and track their learning progress (Abramovich,

Schunn, & Higashi, 2013). Badges have also been successfully adopted by the gaming

community to acknowledge accomplishments within a game and to motivate players (Easley &

Ghosh, 2013).

Digital badges offer many potential advantages over physical badges, such as being easy

to display and search, as well as having the potential to provide more data about a specific

accomplishment including who issued the badge and what specific competences were gained

(Brandon, 2013; Selingo, 2012). However, to date most digital badges exist within the platform

that issued them and are not transferable to different sites on the web (Glover, 2013). Thus these

credentials do little to show others what a person has learned or accomplished.

Part of the problem is that digital badges have been seen as a social networking gimmick

or a motivational strategy within a game, rather than a possible certification solution. As a social

networking tool, digital badges often lack credibility, especially when the assessment system is

based on the recommendations of peers. Many recommenders do not know anything about the

individual who may receive the award, giving the recommendation with the hope that it will be

reciprocated. Badges based on gamification theory (i.e., systems designed to reward learners by

issuing badges and stars representing the degree to which a level of the game has been

10    

successfully completed) have been criticized for not accurately representing actual

accomplishments and for not transferring well to other learning environments (Easley & Ghosh,

2013). This criticism is not surprising when their primary purpose is to increase participation and

effort, not to certify a specific skill base. The challenge for those attempting to create a

professional certification system will be to build on gamification successes while addressing

potential concerns of rigor, validity, and credibility.

More recently however, many have argued that digital badges have the potential to be

much more than gamification or social networking gimmicks, providing instead an efficient

approach to credentialing individual students’ skills and knowledge (Randall, Harrison, & West,

2013), including soft skills (e.g., communication, team work, and other interpersonal skills). To

assist in developing badges for authentic learning assessment, the Mozilla Foundation has

developed an open badging infrastructure that is gaining popularity.

Development of Open Badges

The Mozilla Foundation recognized the value that digital badges could provide learners,

but also saw the need to empower users by letting them own their own data (Goligoski, 2012).

To accomplish this, Mozilla created open badges: digital badges that recipients can share on their

own website or social network profiles. Open badges are issued through the Open Badge

Infrastructure (OBI) and accompanying development community, which, because the OBI is

open source code, can be used by anyone who wants to issue open badges.

Interest from universities or departments within universities is increasing (Glover &

Latif, 2013), and the number of organizations using or developing open badges is increasing as

well. One notable example is Purdue University, which used the Mozilla Open Badge

Infrastructure to create its own badging platform, called Passport (Tally, 2012). Educational

11    

organizations, such as Peer 2 Peer University (P2PU) and Codeschool, are also issuing open

badges. Other notable groups issuing or designing open badges includes NASA, the

Smithsonian Center for Education and Museum Studies, the Smithsonian American Art

Museum, Microsoft, and Disney-Pixar (Mozilla Open Badges, n.d.).

Research into the effectiveness of these open badge implementations is still nascent.

While more research is needed to further understand the potential benefits and challenges of

using open badges, many have recognized that open badges might be a way of legitimizing

informal learning experiences including professional development and adult learning situations

not situated in a specific degree (Goligowski, 2012) or even as a disruptive technology for

reforming higher education through alternative assessment certification for students (Collins &

Pea, 2011). For example, Guzman (2014) of the Seattle Times cited Catalano’s explanation, “The

model of granularity in music purchases has moved us from the album to the song. . . . The

model of granularity in proving skills or expertise is going to move from the certificate or degree

to the badge.” In arguing that badges represent a paradigm shift for higher education, Staton

(2014) further explained, “Higher education . . . is in the midst of dramatic, disruptive change. It

is, to use the language of innovation theorists and practitioners, being unbundled” (np). Whether

or not these bold claims end up being prophetic, clearly the Mozilla open badges allow for a

different kind of credentialing than we have typically done in the past. However, a badge, or any

certificate, can only be as good as the rigor of its implementation, and thus organizations need to

develop strategies and principles for utilizing the technological affordances that the platform

provides for rigorous assessments.

12    

Principles for Rigorously Using Badges

Because the Mozilla OBI allows for open badges to be used for any purpose, they can be

a good solution to those looking for rigorous ways to credential authentic learning experiences

(i.e., informal training through ongoing professional development or work experience). Thus

open badges may provide a good method for credentialing, and possibly certifying, professional

evaluators without limiting the way in which individuals can demonstrate their professional

credentials.

Transparency. One advantage of badges is that the technology allows increased

transparency about the credentialing process. As Guzman (2014) explained, “If resumes are a

bunch of claims, badges are a bunch of evidence” (np). Open badges consist of a digital badge

image with embedded metadata containing information about the badge including the

organization issuing the badge, a brief description of the badge, a URL to the criteria needed to

earn the badge, the date the badge was issued, and (if applicable) the expiration date for the

credential. These metadata could also include a link to a piece of evidence: for example, a

product that was created in order to earn the badge, a letter of reference, or a resume listing an

individual’s specific experiences. Through these metadata, badges make it very clear to potential

employers what a person did to earn the badge, providing greater transparency than many other

methods of certification. More metadata options will likely be available in the future. The

metadata contained in the badge provide the information necessary to help others make sense of

the badge and also to prevent forgery. Open badges move the general certification

acknowledgement provided by an advanced degree to a micro-certification level.

Flexibility. Because open badges are built on open source technology, they can be

flexibly adapted for any situation. For example, an organization wishing to acknowledge a

13    

specific designation (e.g., a credentialed evaluator or even a certified evaluator) might use a top-

down design to outline potential micro-certifications and credentials (i.e., badges) an individual

could obtain. The organization would specify the combination of badges needed to obtain the

designation it offers, which might include various levels (e.g., basic to advanced). Rather than

accrediting specific degree-offering programs, the organization would authorize individuals to

issue badges—recognizing that the badge issuer will award the micro-certification or credential

according to the criteria outlined by the organization. This system allows greater flexibility to

individual candidates. A candidate seeking a designation offered by an organization can then

obtain the required badges from any authorized issuers. After obtaining the required badges, the

candidate can apply to the organization offering the designation to receive the desired credential.

Another flexibility benefit of open badges is that they can be integrated into existing

technological systems and websites through plugins, enabling efficient management of the

certification system. This flexibility allows individuals to take ownership of the badges they

earn. For example, at Brigham Young University (BYU), the Instructional Psychology &

Technology (IP&T) program has implemented EdTec badges into their pre-service instructional

technology courses. These badges were designed to recognize individuals who have completed

training or obtained certain skills, including soft skills. A Wordpress-hosted website houses the

badges; administration and issuing of badges is accomplished either by using Wordpress plugins

built by third parties or by developing proprietary plugins that can be released back to the badges

community (see https://github.com/msedev/badges). Thus both individuals and the organization

can display the badges within the system and outside the system for external access.

Rigor. The rigor of any badging system depends on the rigor of the assessment criteria

and the trust placed on those authorized to issue a badge. Many applications of badging systems

14    

lack credibility because they do not control who is authorized to issue a badge and they do not

clearly specify the criteria required to obtain a badge. To establish a credible certification system

based on open badges, these two issues must be addressed. First, the technology used to manage

the system must be able to control access to the system for those authorized to issue specific

badges, who, of course, must be vetted, trained, and periodically reviewed. Second, clear

expectations must be established for each badge, whether the badge represents a micro-credential

or a micro-certification. The authorized badge issuer may need to administer an exam, gather

evidence, review evidence provided, and possibly keep records of some kind. In some cases the

credibility of the system will require a recertification of the badge, meaning the metadata for the

badge will include an expiration date. This would need to be done if the expertise associated with

the badge can go out of date, requiring the individual to update relevant knowledge, skills, or

abilities.

Management. One obvious advantage of using an open badge template for a certification

system is management. At the organization level a committee might oversee the process, but

much of the work is done by authorized badge issuers, who may also be those who provide

training through a university program or a training institution. This division of labor model (also

known as crowd sourcing) makes the system less cumbersome for those who manage it, and can

reduce the cost in time and money.

Another management advantage of using open badges as a means of certification is that

the structure allows for larger domains to be broken down into smaller skill and knowledge

components. A skill subset can be established based on a set of component skills (component

badges) making up a specific micro-certification unit. For example, a certification system might

organize badges into clusters. Discrete skills within the cluster would be represented by

15    

component badges to be earned by individuals and issued by an authorized person. Those who

master a requisite number of component skills would automatically earn the badge associated

with that cluster of component badges. Micro-certification units could be established with badges

representing basic evaluation skills. Badges representing a skill unit could be automatically

issued when an individual earned the required component badges. For example, a micro-

certification unit representing basic quantitative competency might be automatically granted

when individuals earn a designated number of component badges. This organizational approach

(i.e., top down design) is easier to manage because one individual is not responsible for assessing

and issuing component skill badges he or she is not qualified to assess. Badge issuers can be

authorized to issue specific badges. Individuals can obtain component badges from a variety of

authorized badge issuers. In this way issuing unit badges and ultimately the overall credential

can be automated to a certain extent.

Another attribute of open badges is that each badge can be awarded at various levels of

expertise. For example, an individual might earn a micro-certification badge representing

experience at the basic level by completing a specified number of component badges. As the

candidate gains more experience and earns more component badges, his or her experience badge

can be upgraded to represent competence at an advanced and ultimately at an expert level. Thus

not only can system management be structured to improve efficiency, the certification process

can be designed to be somewhat automated in identifying levels of proficiency.

Open Badges as the Basis of a Certification Process

No solution to AEA’s evaluator certification problem will be perfect and any solution

AEA adopts will include startup issues and maintenance challenges; but utilizing an open badge

format is one that may hold promise. We propose in this paper that using a framework based on

16    

open badges to establish an evaluator certification system for the AEA could help alleviate and

overcome many of the concerns and inadequacies associated with traditional certification

systems. The system we recommend might best be labeled a certification process, as many of the

components would require authorized individuals to assess a candidate’s knowledge and skills.

Obtaining the status of Certified Evaluator might be described as earning a credential.

Recognition of a Certified Evaluator could be issued at various levels (basic to expert)

and would be transparent in clearly communicating the knowledge, skills, and experience of the

individual. This transparency would contribute to the validity and credibility of the credential. A

certification system like the one we propose would be flexible in the ease with which it could

adapt and change. In addition, it would meet the needs of a wide range of prospective candidates:

those who obtain training through traditional evaluation programs and those who gain

competence without obtaining a university degree or certificate. Such flexibility, with some

effort, could also facilitate the grandfathering in of evaluators with prior training and previous

experience. Most important, the system would be fairly efficient, with much of the process being

automated. In the following sections we present what an evaluator certification process based on

open badges might look like and how it might be administered.

In this scenario the AEA would take ownership of the credentialing process, managing

the system by using existing technologies and the AEA’s web presence. The AEA would initially

need to establish a committee to define the credential. This committee could later transition into

a standing certification committee with a rotating membership tasked with maintaining the

system. In this scenario we anticipate the participation of many qualified AEA members to serve

as authorized issuers of badges (or, if you prefer, micro-certification units and component

badges).

17    

Preparatory Phase

Badge identification. In the initial development phase of the system, a committee of

AEA expert evaluators and external stakeholders would need to establish an initial set of badges

representing the knowledge, skills, and experiences a qualified evaluator might obtain. The

process of defining essential competencies could be expedited by work already completed. For

example, the Essential Competencies for Program Evaluators (ECPE) (Stevahn, King, Ghere, &

Minnema, 2005) might serve as a useful framework for establishing some of the requisite

knowledge, skills, and dispositions an evaluator might need to be considered a trained

professional. The badges could then be arranged using a top-down design and organized into

clusters (or units) that represent various aspects of the certification. The criteria for earning each

of the component badges would need to be established. In addition, rules would need to be

created to define how a unit badge would be obtained in each of the essential areas (i.e.,

knowledge, skills, and experience) and ultimately which combination of unit badges (micro-

certifications representing knowledge, skills, and experience) would be needed to obtain the

status of Certified Evaluator. Criteria would need to be established for any competency level

associated with a specific component badge, unit badge, or certification distinction (e.g., what

combination of component badges represents competency at a basic level compared to the

advanced level of the unit badge). In the end, those involved in this developmental phase would

determine the criteria for certifying an individual as a qualified evaluator at the basic, advanced,

and expert levels.

For example, the development committee might identify the need for a component

training badge called evaluation basics. The committee might establish various criteria

associated with earning this badge. One criterion might be that an individual has explored the

18    

question of how evaluation differs from research. In order to earn this badge the individual might

also need to be familiar with basic evaluation terminology, evaluators’ roles in society, AEA’s

Guiding Principles for Evaluators, and the Joint Committee’s Program Evaluation Standards

(AEA, 2004; Yarbrough, Shulha, Hopson, & Caruthers, 2011). To award this badge, someone

authorized to issue it would assess the knowledge of the individual on this content. The badge

might be awarded when the individual completes an introductory evaluation course as part of an

established university program. The badge might be awarded by the issuer to an individual based

on prior training and experience. The candidate might earn the badge by attending a workshop at

AEA’s annual conference or summer institute, or by completing a future online course the AEA

might develop.

This badge might be clustered with other basic knowledge badges to form a training unit.

Other component badges in this unit might include an understanding of AEA’s public statements

(e.g., cultural competence), evaluation history, meta-evaluation strategies, and others. An

individual who has earned a specified number of the component badges in this cluster would be

granted the unit badge. Other unit badges would be outlined in each of the certification areas

representing an individual’s knowledge, skills, and experience. An individual who earned the

minimum number of requisite unit badges from each of the certification areas would be awarded

the status of Certified Evaluator at the basic level. Advanced levels of this status might be

awarded as an individual earns additional badges or upgrades the quality level associated with

specific badges.

When this badge identification phase is completed, a theoretical framework would be

established for identifying qualified evaluators, with rules in place for acknowledging an

individual as a Certified Evaluator. Once this structure is implemented (which would likely take

19    

some time and effort), much of the process could be automated based on the rules for awarding

unit certifications.

Management system. One of the benefits of using the Mozilla OBI or its equivalent is

that it can be integrated into existing technological systems and websites through plugins. The

current AEA website, which has its own user authentication system, might be used to manage a

certification system for the AEA. A section of the website would display individual badges and

set out the hierarchal structure of the certification. Members and interested parties could visit the

site to learn about the evaluator certification process and requirements. The system could also be

used to manage the processes of applying for certification and issuing of badges by authorized

individuals. This would require an ongoing commitment of talent and resources by the AEA.

Once a management system is in place, badges that have been awarded could be

displayed on member profile pages, enabling potential evaluation clients and other interested

parties to search for qualified evaluators working in their city or state. Member profiles might

also be enhanced with a brief bio of the evaluator, information about the specific field or area in

which he or she practices, or a link to a resume.

Issuer authorization. Essential to the success of this system are volunteers who will

serve as badge issuers. Clearly this would constitute a challenging issue for the AEA, but one

that could be managed. Individuals would need to be trained and periodically supervised.

Training for badge issuers could be offered at AEA’s annual conference. Potential badge issuers

from universities and other institutes offering evaluation training are likely to be qualified

already. Others in the industry with evaluation experience would also be likely candidates. Those

volunteering to issue badges would need to be vetted and given enhanced access to the

credentialing system to allow them to issue specific badges.

20    

Several procedural details would need to be worked out. Potentially, the system could be

programmed to notify an authorized individual whenever a request for a badge might be

received; the issuer would assess the request and decide whether to issue the badge. An applicant

who had received training from an authorized badge issuer could stipulate a specific authorized

individual to issue the badge, expediting the review process according to the authorizer’s

knowledge of the candidate’s qualifications. Using an authorized badge issuer system would

require accrediting individual badge issuers but would eliminate the need for accrediting

evaluation programs. It would also remove the need to create and administer a single summative

assessment of an evaluator’s knowledge and skill.

Implementation Phase

Once the certification system has been defined and structured, implementation can begin.

The preparatory phase may require substantial time and attention, but the maintenance and

improvement of the system can be much more manageable than many traditional systems.

AEA certification committee. After the initial development phase, an oversight

committee would be required to maintain the system. Committee members would be responsible

for periodically reviewing the badging structure, updating badge criteria and expectations, and

adding badges as needed. They would also take responsibility for maintaining the system,

training new authorized issuers, arbitrating disputes, and promoting the program to AEA

members and affiliates.

Candidate process. For an individual new to evaluation who wishes to be acknowledged

as a Certified Evaluator, the following process might be typical. The candidate would engage in

training and earn badges associated with that training. Because the badges and the criteria for

earning those badges are communicated clearly on the AEA website, training providers could

21    

align their courses with specific component badges. For example, university programs, the

summer institute, and AEA’s annual conference workshops might identify those badges an

individual could earn by completing a course or workshop offered. The process could be similar

for experienced evaluators with prior training. An individual who has received training or gained

experience would be able to apply for component badges through AEA’s certification system.

Candidates would provide the relevant evidence, which an authorized individual would review

before deciding whether to award the badge. Thus, experienced evaluators could be easily

grandfathered into the new system, although the initial swell of those applying for badges would

require additional time at the beginning of system deployment.

Because the process of earning badges would likely be accomplished over time,

evaluators in training could benefit from the opportunity to be recognized for the knowledge,

skills, and experience they had received. Once candidates had completed the requisite component

badges, they would be awarded unit badges. Once they had completed the requisite number of

unit badges in each of the certification areas, they would be awarded the status of Certified

Evaluator. Initially this certification would be awarded at the basic level. Over time candidates

could upgrade their certification to an advanced or expert level.

The flexibility of this system allows individuals to achieve the status of Certified

Evaluator in a variety of ways. Some might have expert levels of experience as an evaluation

project leader but only basic skills training. Others might have expert knowledge and skills but

only basic experience. This system can also accommodate those seeking the Certified Evaluator

credential based on experience and demonstrated expertise, which might be arranged by special

request to the AEA certification committee or through the regular system. In each case the

22    

individual candidate can be acknowledged as being a Certified Evaluator in a relatively

organized and rigorous manner.

Conclusions

Certainly some evaluation professionals may not feel the AEA needs a certification

system. But for those who see the need, much of the debate and concern over what type of

system the AEA might adopt centers in how to establish and manage a valid certification system

that is efficient, rigorous, and credible. Our past inability to agree on a solution may have been

caused by the weakness of traditional views of certification in failing to provide a practical

vantage point. In the traditional view, certification is understood from a macro summative

perspective. Many traditional certification systems require a rigidly applied summative

certification exam, potentially both costly and cumbersome.

We have proposed in this paper a new technology-driven approach to certification using a

digital open badge perspective that builds the certification from many micro-credentials. The

open badge framework uses a top-down design to structure and understand the certification

process. The overall goal of certification is broken down into manageable pieces. At the lowest

level of the certification process, smaller micro-certifications are completed. These component

elements (badges) make up certification units. Completing certification units forms the basis for

the overall credential.

Utilizing this process, the system is transparent. The criteria for each component or

micro-certification is clearly communicated. When an individual obtains a certification, others

can understand clearly and specifically though the metadata attached to the badge how the

certification was achieved and what knowledge, skills or experience the individual has gained.

An open badge certification system is flexible. Individuals can earn the certification by obtaining

23    

the necessary knowledge, skills, and experiences in varied ways. The top-down design of the

system accommodates easy modification and improvement. This type of system is efficient.

Much of the process can be automated once the rules for certification are established. Built on a

division of labor approach to certification, an open badge system can be less work and less

costly. Authorizing and training badge issuers improves the rigor and credibility of the

assessment.

In summary, any certification process the AEA chooses to adopt will be difficult to

establish and maintain. Certainly the use of an open badge system would not solve all the issues

relevant to the creation of a certification system; however, we hope it will help move the

discussion forward. Consensus is always a concern for evaluators. Given that Open Badges are a

recent innovation, there are certainly many questions that remain about the technological and

organizational issues surrounding badging systems. However, a carefully prepared and

efficiently implemented certification system based on open badges could potentially provide a

transparent, flexible, efficient, rigorous and credible way to certify evaluators. The system can be

easily modified and improved. Although no system can be implemented without time, effort, and

resources, the potential benefits of using an open badge infrastructure make this an enticing

option and viable solution to the AEA’s need for a system of evaluator certification -- one that

warrants further consideration by AEA members.

24    

References

Altschuld, J. W. (1999). Certification of evaluators: Highlights from a report submitted to the

board of directors of the American Evaluation Association. American Journal of

Evaluation, 20, 481-493.

Antin, J., & Churchill, E. (2011). Badges in social media: A social psychology perspective.

Human Factors, 1-4. Retrieved from http://uxscientist.com/public/docs/uxsci_2.pdf

Abramovich, S., Schunn, C., & Higashi, R. (2013). Are badges useful in education? It depends

upon the type of badge and expertise of learner. Education Tech Research Development.

Retrieved from http://www.lrdc.pitt.edu/schunn/research/papers/Abramovich-Schunn-

Higashi.pdf

American Evaluation Association (2004). American Evaluation Association guiding principles

for evaluators. Retrieved from http://www.eval.org/p/cm/ld/fid=51

Bickman, L. (1999). AEA, bold or timid? American Journal of Evaluation, 20(3), 519-520.

Becker, H., & Kirkhart, K. (1981). The standards: Implications for professional licensure and

accreditation. American Journal of Evaluation, 2(2), 153-157.

Christie, C. A. (2003). Understanding evaluation theory and its role in guiding practice: Formal,

folk and otherwise. New Directions for Evaluation, 97, 91-93.

Collins, A. & Pea, R. (2011). The advantages of alternative certifications for students. Education

Week, 31(8), 22-23.

Davies, R. & MacKay, K. (2014). Evaluator Training: Content and Topic Valuation in

University Evaluation Courses. American Journal of Evaluation, online first. DOI:

10.1177/1098214013520066

Dewey, J. D., Montrosse, B. E., Schroter, D. C., Sullins, C. D., & Mattox, J. R., II. (2008).

Evaluator competencies: What’s taught versus what’s sought. American Journal of

Evaluation, 29, 268-287.

25    

Easley, D., & Ghosh, A. (2013). Incentives, gamification, and game theory: An economic

approach to badge design. ACM EC conference. Retrieved from

http://www.arpitaghosh.com/papers/EC-final.pdf

Glover, I., & Latif, F. (2013). Investigating perceptions and potential of Open Badges in formal

higher education. In J. Herrington, A. Couros, & V. Irvine (Eds.), Proceedings of World

Conference on Educational Multimedia, Hypermedia and Telecommunications 2013, (pp.

1398-1402). Chesapeake, VA: AACE.

Guzman, M. (2014). The future of credentials: Will degrees and resumes make room for the

badge? Seattle Times. Retrieved from http://blogs.seattletimes.com/monica-

guzman/2013/04/27/the-future-of-credentials-will-degrees-and-resumes-make-room-for-

the-badge/

Jones, S. C., & Worthen, B. R. (1999). AEA members’ opinions concerning evaluator

certification. American Journal of Evaluation, 20(3), 495–506.

LaVelle, J. M., & Donaldson, S. I. (2010). University-based evaluation training programs in the

United States 1980-2008: An empirical examination. American Journal of Evaluation,

31, 9–23.

Matkin, G. W. (2012). The opening of higher education. Change Magazine. Retrieved from

http://www.changemag.org/Archives/Back%20Issues/2012/May-

June%202012/Higher%20Ed.full.html

Mozzila Open Badges, (n.d.). Who’s currently issuing open badges? Retrieved from

http://openbadges.org/participating-issuers/

Morris, M. (2011). The good, the bad, and the evaluator: 25 years of AJE ethics. American

Journal of Evaluation, 32(1), 134-151.

26    

Randall, D. L., Harrison, J. B., & West, R. E. (2013). Giving Credit Where Credit Is Due:

Designing Open Badges for a Technology Integration Course. TechTrends, 57(6), 88-95.

Selingo, J. (2012). Colleges need some big ideas to drive change from within. Chronicle of

Higher Education, 58(7), A1-A12.

Smith, N. L., Brandon, P. R., Hwalek, M., Kistler, S. J., Labin, S. N., Rugh, J., & Yarnal, L.

(2011). Looking ahead: The future of evaluation. American Journal of Evaluation, 32(4),

565-599.

Staton, M. (2014). The degree is doomed. Harvard Business Review Blog Network. Retrieved

from http://blogs.hbr.org/2014/01/the-degree-is-doomed/

Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies

for program evaluators. American Journal of Evaluation 26(1), 43-59.

Worthen, B. R. (1972). Certification for educational evaluators: Problems and potential. Paper

presented at the annual meeting of the American Educational Research Association,

Chicago, IL.

Worthen, B. R. (1999). Critical challenges confronting certification of evaluators. American

Journal of Evaluation, 20(3), 533–555.

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program

evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand

Oaks, CA: Sage

Young, J. R. (2012). Badges earned online pose challenge to traditional college diplomas.

Chronicle of Higher Education. Retrieve from http://chronicle.com/article/Badges-

Earned-Online-Pose/130241/

27