www.ssoar.info
Reviewing media literacy intervention studies forvalidityPotter, W. James; Thai, Chan L.
Veröffentlichungsversion / Published VersionZeitschriftenartikel / journal article
Empfohlene Zitierung / Suggested Citation:Potter, W. J., & Thai, C. L. (2019). Reviewing media literacy intervention studies for validity. Review of CommunicationResearch, 7, 38-66. https://doi.org/10.12840/ISSN.2255-4165.018
Nutzungsbedingungen:Dieser Text wird unter einer CC BY-NC Lizenz (Namensnennung-Nicht-kommerziell) zur Verfügung gestellt. Nähere Auskünfte zuden CC-Lizenzen finden Sie hier:https://creativecommons.org/licenses/by-nc/4.0/deed.de
Terms of use:This document is made available under a CC BY-NC Licence(Attribution-NonCommercial). For more Information see:https://creativecommons.org/licenses/by-nc/4.0
Diese Version ist zitierbar unter / This version is citable under:https://nbn-resolving.org/urn:nbn:de:0168-ssoar-61188-4
Reviewing Media Literacy Intervention Studies for Validity
OPEN ACCESS Top-Quality SciencePeer ReviewedOpen Peer ReviewedFreely available online
Review of Communication Research2019, Vol.7
doi: 10.12840/ISSN.2255-4165.018ISSN: 2255-4165
W. James PotterUniversity of California at Santa Barbara
This study is an examination of validity in published articles that have provided tests of the effectiveness of media literacy
interventions. We identified 88 published tests of media literacy interventions then analyzed their content using five coding
variables that indicated the degree to which authors of those studies established basic validity. We first conducted a meaning
analysis to identify the definitions that authors of those studies presented for media literacy. Then we used those definitions
to determine the extent to which those authors provided a complete (content validity) and accurate (face validity) operation-
alization in the design of their measures.
Suggested citation: Potter, W. J., & Thai, C. L. (2019). Reviewing Media Literacy Intervention Studies for Validity. Review
of Communication Research, 7, 38-66. doi: 10.12840/ISSN.2255-4165.018
Keywords: Media literacy interventions, validity, meaning analysis, explication, critical analysis
Received: Nov. 19, 2017 Open peer review: Dec. 17 Accepted: May 28, 2018 Prepublished online: Jun. 1 Published:
Jan. 2019
Editor: Giorgio P. De Marchis (Universidad Complutense de Madrid, Spain).
Reviewers: Erica Weintraub Austin (Washington State University), Melissa Tully (University of Iowa).
Abstract
Chan L. ThaiSanta Clara University
• The designs of published studies claiming to test media literacy interventions were analyzed to determine how the con-
ceptual foundation was constructed in each study and whether the study was designed based on those conceptual founda-
tions to determine content and face validity.
• A total of 88 studies were selected after a series of electronic searches of studies that used the term “media literacy
intervention” in their keyword lists, titles, and abstracts.
• A meaning analysis found that 22 studies (25.0%) provided no conceptual foundation for media literacy, and 21 (23.9%)
used an existing definition of media literacy. Despite there being hundreds of definitions for media literacy in the literature,
the authors of the remaining 45 studies (51.1%) presented their own definition for media literacy.
• The assessment of validity found that none of the studies presented a test of media literacy that completely captured the
elements in their definitions of media literacy, so the content validity of this literature was judged as poor.
• The evaluation of face validity uncovered many problems in a lack of correspondence between what authors intended to
measure and what they actually measured. The most prevalent discrepancy was with measures of skills where authors
frequently measured beliefs about study participants’ levels of skills rather than taking measures of actual performance.
• We pose a series of three questions that illuminate the current nature of the media literacy intervention literature as well
as serve to guide future designers of such studies.
Highlights
39
When scholarly fields are brand new, researchers must
make many assumptions about the nature of the phenome-
non they are studying as well as the methods that might be
useful in generating knowledge about that phenomenon.
Then as a field’s research literature grows, scholars benefit
from a larger base of knowledge, which allows them to use
definitions of their key concepts that are more clear, com-
plete, and insightful. And the literature provides more guid-
ance about which research design elements (measures,
Content
I. THE CRITERION OF VALIDITY IN SOCIAL SCIENCE RESEARCH .....................3A. Content Validity ......................................................................................................3B. Face Validity ............................................................................................................4
II. MEANING ANALYSIS OF MEDIA LITERACY ......................................................4Table 3. Hierarchy of Quality in Conceptual Foundation. ............................................6
METHOD .......................................................................................................................7A. Selecting Media Literacy Intervention Studies ..........................................................7B. Coding Variables .....................................................................................................8
Media literacy definitions cited. ...............................................................................8Focal definition of media literacy. ............................................................................8Definitional elements. ..............................................................................................8Content validity. ......................................................................................................9Face validity. ...........................................................................................................9
C. Reliability ..............................................................................................................10RESULTS .....................................................................................................................10
Presentation of Meaning ............................................................................................10Validity ......................................................................................................................10Table 4. Analysis by Components for Media Literacy by Studies. ................................11
V. DISCUSSION ...........................................................................................................12A. Reasonable Standard .............................................................................................12Table 5. Analysis by Domains for Media Literacy by Studies. .....................................12
Conceptual foundation. ..........................................................................................13Table 6. Analysis of Domains of Skills and Knowledge. .............................................13
Measures. ..............................................................................................................14B. Important Questions ..............................................................................................14
Basic research or evaluation study? ........................................................................14VI. CONCLUSION .......................................................................................................18REFERENCES ............................................................................................................18
Table 1. Sampling of Definitions of Media Literacy. ...................................................25Table 2. Definitions for Media Literacy: Components and Domains. ..........................27Skills Focused Components ........................................................................................27
COPYRIGHTS AND REPOSITORIES ........................................................................29
treatments, samples, procedures, and analyses) are faulty and
which are more useful. Therefore, a key indicator of the
value of growth in a scholarly field is validity because valid-
ity reflects the degree to which scholars provide clear defini-
tions of a field’s key concepts as well as the degree to which
scholars use those definitions to guide the design of their
studies and the construction of their measures.
In this study, we conduct a content analysis of published
studies of media literacy interventions in order to generate
Reviewing Media Literacy Intervention Studies for Validity
2019, 7, 38-66
W. James Potter & Chan L. Thai
40 www.rcommunicationr.org
expectations about what a particular concept should (predic-
tive and concurrent) and should not (discriminant) be re-
lated to, while the logical/conceptual type of validity is
established through argumentation and expert judgments.
In our content analysis, we focus attention on what are argu-
ably the most fundamental logical/conceptual types of valid-
ity -- content validity and face validity. These types are
fundamental because if these types cannot be established
first, the testing for other types of validity has little meaning.
A. Content Validity
Content validity focuses attention on the structure of the
meaning of concepts that is revealed when scholars provide
definitions of those concepts. The assessment of content
validity then entails the comparison of elements in that de-
fined structure with the set of items that researchers use to
measure that concept. Content validity is a judgment about
the degree to which the set of measures used in a research
study reflect all the elements laid out in the definition of a
concept. Bausell (1986) illustrates the essence of content
validity with the question, “Do the different components of
the measurement procedure (which are usually items) match
the different constituents of the attribute being measured?”
(p. 156). Those items need to be representative of the entire
concept (Vogt, 2005). Thus, content validity is stronger when
designers of studies have used measures to assess all the ideas
presented in their conceptualization.
Across the many different definitions of media literacy
in the literature, the two most prevalent definitional compo-
nents appear to be skills and knowledge (see Table 2). How-
ever, there are also frequent mentions of other components
such as behaviors, affects, and beliefs. Notice that there is a
considerable variety of ideas within each of these compo-
nents. We use the term “domains” to refer to the different
ideas within a component. For example, within the compo-
nent of skills, some definitions call for the development of
general skills, while other definitions focus on particular
skills needed to access media messages, to interpret the
meaning in those messages, and to produce messages – we
consider these to be different “domains” of skills. Notice how
the definitions presented in Table 2 display a variety both
with components and with domains. Some scholars (e.g.,
Bachmair & Bazalgette, 2007; Hobbs, 1998) have claimed
there has been a convergence towards a consensus definition
where media literacy is defined as “the ability to access,
answers to two questions. First, when authors claim that
their study provides a test of the effectiveness of a media
literacy intervention, what do they mean by “media litera-
cy?” We conducted a meaning analysis to generate an an-
swer to this first question, then used the results of this
meaning analysis to answer the second question: How well
do designers of intervention studies use their conceptualiza-
tions of media literacy to construct their measures?
I. The Criterion of Validity in Social Science Research
Validity is an essential concern for determining the
quality of any social science research project (Brinberg &
McGrath, 1985; Chaffee, 1991; Guilford, 1954; Nunnally,
1967). Validity refers to the correspondence between what
scholars say they are measuring and what they actually
measure (Rust & Golombok, 1989; Vogt, 2005). Williams
(1986) explains, “The question of validity is a question of
‘goodness of fit’ between what the researcher has defined
as a characteristic of a phenomenon and what he or she is
reporting in the language of measurement” (p. 21). More
concretely, validity is concerned with the match between
the meaning of a concept and how it is operationalized in
a research study (Chaffee, 1991).
The task of assessing validity begins with a meaning
analysis to determine how the authors of a study have de-
fined their key concepts (Chaffee, 1991). This task proceeds
by using authors’ meaning as the standard for judging valid-
ity. As Chaffee writes, “Validity should not be equated with
‘truth.’ Disappointing as this might sound, the philosophi-
cal concept of truth is not a usable criterion” (p. 11). Instead,
the criterion for judging validity within a research study is
the meaning expressed by the authors of that study. This
point is especially important in media literacy research
where there are so many different conceptualizations of
what it is. See Table 1 for a sampling of definitions in the
literature, and notice that each definition has a different
configuration of components (e.g., skills, knowledge, be-
liefs). While there is considerable overlap among those
components across definitions, each of those definitions is
unique in terms of the configuration of those components.
There are two general kinds of validity -- the empirical
type and the logical/conceptual type. The empirical type
of validity relies on collecting data to show support for
Reviewing Media Literacy Intervention Studies for Validity
41 2019, 7, 38-66
B. Face Validity
While content validity focuses attention on a concern
about whether designers of media literacy studies have in-
cluded measures for all the components and domains in their
conceptualization of media literacy, face validity focuses
attention on a concern about whether the measures used are
acceptable operationalizations of the components and do-
mains they purport to assess. If authors claim that a par-
ticular measure is an assessment of component X but the
measure is instead an assessment of component Y, then this
non-match signals a barrier to establishing face validity. For
example, let’s say that authors of an intervention study define
media literacy as requiring the skill of critical thinking. In
the methods section the authors say their measure of critical
thinking consists of one item (I am confident about my ability
to think critically about media messages) and a five-point Likert
type set of responses (1 = Strongly Agree, 2 = Agree, 3 =
Neither Agree nor Disagree, 4 = Disagree, 5 = Strongly
Disagree). When we compare this conceptualization to the
measure, we can see a lack of correspondence, because the
measure makes an assessment of respondents’ beliefs about
the level of their skills rather than an assessment of their ac-
tual levels. Instead, if the designers of the study defined media
literacy as a condition where people have a high degree of
confidence in their level of skill on critical thinking (regard-
less of their actual level of skill), then this measure would
correspond to the conceptualization and therefore qualify
as exhibiting adequate face validity.
II. Meaning Analysis of Media Literacy
An analysis of authors’ meaning of their key concepts is
essential as a first step in making judgments about content
and face validity. Once authors’ meanings are found in their
writings, those meanings become the standard for judging
validity of measures as being complete (content validity) and
accurate (face validity).
There are a variety of ways that authors can convey their
meaning for key concepts in their writing about empirical
studies. One common way is for authors to assume that all
readers share the same meaning for a term, so the authors
treat it as a primitive concept and present no definition for
the term in their writings. Chaffee (1991) explains that prim-
itive terms are those where all people share the same meaning
analyze, evaluate and communicate messages in a variety of
forms” (Aufderheide, 1997, p. 79). This definition arose out
of discussions of a group of scholars who convened the “Na-
tional Leadership Conference on Media Literacy” in the
mid-1990s. Such a definition is valuable because it provides
designers of media literacy interventions with a widely
shared conceptual foundation for their studies. However,
other scholars are skeptical of this claim of a consensus
definition for media literacy and point out that the field is
characterized much more by its variety of definitions than
its convergence to a consensus (Potter, 2010; Silverblatt,
Ferry, Finan, 2015; Tyner, 2009).
Our concern with validity in this study is much less on
debating whether there is a consensus definition or not and
much more on assessing how well designers of media litera-
cy intervention studies operationalize their chosen defini-
tions into measures. Thus, we begin our content analysis of
published intervention studies with a meaning analysis to
identify authors’ meanings they present for media literacy.
We do not make judgments about the definitions themselves;
instead, we focus attention on how well the authors use their
selected definitions in the design of their studies. Therefore,
the meaning analysis part of our content analysis is con-
cerned with recording all the components and domains of the
definitions for media literacy that authors present as the
conceptual foundation of their empirical studies.
Making a judgment of content validity involves compar-
ing a study’s set of measures of media literacy against a
criterion. In designing our content analysis study, we did not
select one definition of media literacy as the criterion, be-
cause there is no commonly accepted definition that could
be used as a standard. Instead, we recorded what the authors
of each study presented as their definition of media literacy
and used the authors’ definitions as the standard to evaluate
the degree of validity demonstrated in the construction of
measures for that study. Thus for each study we examined,
we conducted a meaning analysis to identify the components
and domains expressed by the authors’ conceptualization of
media literacy. We then compared this configuration of
components and domains to the authors’ presented set of
measures. The degree of content validity is indicated by the
proportion of matches of the components and domains in
authors’ conceptualizations of media literacy with the com-
ponents and domains in their set of reported measures.
W. James Potter & Chan L. Thai
42 www.rcommunicationr.org
lowest quality because it is based on the assumption that all
scholars share the same meaning for a term. Given that the
literature has grown large enough to exhibit many defini-
tions of “media literacy,” research study designers are pro-
vided with a great deal of guidance; thus, studies that
operate at this level are regarded as faulty.
Researchers who move their work upwards from Level 1
recognize that they need to avoid the assumption that all
readers share the same definition for media literacy, so they
provide a definition they are using as the foundation for their
study. If they simply present a definition, but they do not
provide a source for the definition in the form of a citation
in their review of the literature, then their work is limited to
Level 2. Although their study demonstrates a higher level of
precision compared to Level 1 by clearly communicating
their definition to readers, they provide no scholarly founda-
tion for that definition, so the presented definition appears
arbitrary and is untethered to any previously published schol-
arship on media literacy. Studies at this level leave the read-
er wondering if the presented definition is in fact from the
existing literature or whether the authors have constructed
their own definition. If the latter, readers question why the
authors created a new definition when so many already ex-
ist.
Researchers who move their work upwards from Level 2
present a definition of media literacy and cite a source for it
from the literature. Level 3 research has the advantage of
providing readers with a clear articulation of the authors’
meaning and ties that meaning to a history of thinking
through the citation. However, this option leaves readers
wondering whether there were also other definitions consid-
ered, and if so, why those other meanings were rejected.
Researchers who move their work upwards from Level 3
demonstrate an awareness that there are multiple meanings
for media literacy in the scholarly literature. Level 4 research
studies present a review of the literature that describes more
than one definition of media literacy that concludes with the
authors selecting one of those definitions as the foundation
for their study.
Researchers who move their work upwards from Level 4
go beyond demonstrating an awareness of multiple defini-
tions for media literacy and present their own constructed
definition for media literacy. Level 5 research is superior to
the other four lower level studies because it presents readers
with a definition that appears to be synthesized through a
critical evaluation of existing definitions. This synthesis
so that it “would be foolish to expend a lot of effort on its
definition” (p. 8). The most obvious primitive terms are ar-
ticles (e.g., the, an) and prepositions (e.g., of, by, with) but
they can also be nouns (e.g., person, chair, tree). In contrast
to primitive terms are derived terms where authors recognize
that there is not a common meaning, so authors need to
present a definition to clarify for readers the meaning those
authors are using in their study. This is especially important
with hypothetical constructs (e.g., intelligence, attitude,
anxiety) that might appear to be primitive terms but are
actually technical terms that often vary in meaning across
scholars. When these terms are used in scholarly forums,
there is an obligation for authors to specify the definition
they are using so readers can be clear about how they are
being used. In his classic book Explication, Chaffee (1991)
argues that scholars need to be more careful about avoiding
assumptions that terms are primitive and instead carefully
lay out the meaning for their readers. To guide this task,
Chaffee (1991) presented a multi-step process he called ex-
plication where scholars analyze literatures, make evalua-
tions of the meanings that occur there, and move on to
synthesize their own meanings that clarify trends in the
literature and thereby articulate a clear foundation for their
own studies. Chaffee’s multi-step process of explication sug-
gests that there is a hierarchy of quality in the way authors
of empirical studies can derive and present the meaning they
are using for their focal concepts. We translate Chaffee’s
procedure of explication into a six-level hierarchy that can
be used to track the progress of a field’s progress towards
precision over time (see Table 3). When a scholarly field is
new, researchers can find little guidance in their literature
for constructing a conceptual foundation for their studies
and selecting measures with the highest degree of demon-
strated validity, so their research designs make their contri-
butions at the lower levels on this hierarchy. However, as the
literature grows, research designers are provided with an
increasing amount of guidance, so that they can base their
design decisions less on untested assumptions and more on
trusted patterns of empirical findings. Thus, movement up-
wards on this hierarchy requires researchers to add more
value as scholars by finding useful patterns in the literature
then using those patterns to create conceptual foundations
for their studies that are clearer and more precise.
Level 1 on this hierarchy represents those studies where
authors treat their focal concept - in this case, media literacy -
as a primitive term. This option is regarded as exhibiting the
Reviewing Media Literacy Intervention Studies for Validity
43 2019, 7, 38-66
those elements so that the weakest are rejected, and the
strongest are then assembled into a new configuration of
meaning. While it might appear at first like the generation
of another meaning would add to the definitional clutter,
this is not the case. A good synthesized definition does not
simply add another meaning; instead, it serves to replace
much of the fuzzy definitional work with a definition that
is not only more clear and useful but also builds from the
strengths in the literature (Chaffee, 1991).
When scholars conduct reviews of the literature, their
task is made easier to the extent that the authors of the indi-
vidual studies in that literature have operated at higher levels
on this meaning explication hierarchy. Reviewers of the
literature then can be more confident that they are identify-
ing the actual meanings of those various authors have used
in their studies rather than having to infer those meanings.
Reviewers can then group those studies according to their
identifies the most useful elements in those definitions then
assembles those definitional elements into a coherent defini-
tion that is somehow superior to any of the existing options.
However, the authors have not shown this process of critical
evaluation and synthesis to readers.
At the pinnacle of this hierarchy is Level 6, which dem-
onstrates the full degree of explication suggested by Chaffee.
Authors who publish Level 6 studies move beyond simply
describing the multiple meanings in the scholarly literature
and take the readers through the step-by-step process they
used in critically analyzing the range of meanings in the
literature and how those meanings were synthesized into a
coherent set. Authors operating at Level 6 demonstrate con-
siderable scholarly skill of synthesis in order to analyze
meanings of a focal concept for their component elements,
systematically make judgments about the value of all the
elements in those definitions, then sort through the value of
Table 3. Hierarchy of Quality in Conceptual Foundation. (back to pg. 5; forward to pg. 8)
Level
1 No Definition 22 studies (25.0%)
Authors provide no definition of media literacy; media literacy is assumed to be a primitive term where all
readers share the same meaning.
2 Foundationless Definition 11 studies (12.5%)
Authors present their study’s definition of media literacy with no foundation (i.e., there are no meanings for
media literacy presented in their review of the literature).
3 Selection from Single Definition 14 studies (15.9%)
Authors present their study’s definition of media literacy with a foundation of citing a single meaning for
media literacy in their review of literature; there is no acknowledgment that there are other meanings for
media literacy.
4 Selection from Multiple Definitions 7 studies (8.0%)
Authors present their study’s definition of media literacy with a foundation that cites multiple meanings for
media literacy in their review of literature; they select one of those meanings but do not explain why they
selected that particular meaning for their study.
5 Construction with No Explanation 26 studies (29.5%)
Authors present their study’s definition of media literacy; although they cite multiple meanings for media
literacy in their review of literature, they do not explain how they critically evaluated those multiple definitions
in the construction of their study’s definition.
6 Construction with Explanation 8 studies (9.1%)
Authors present their study’s definition of media literacy with a foundation that cites multiple meanings for
media literacy in their review of literature; they show how they critically analyze those definitions and
construct their own definition from the results of their critical analysis.
W. James Potter & Chan L. Thai
44 www.rcommunicationr.org
analysis, we record the presence of key characteristics of
conceptualizations authors present for media literacy in
those published studies to provide the foundation we use to
make judgments about content validity and face validity.
Specifically, we record the components and domains authors
present in their definitions of media literacy that serve as the
foundation for their studies. In assessing content validity, we
compare the configuration of components and domains in
the conceptual foundations with the set of measures pre-
sented in each study. In assessing face validity, we make
judgments about how well the measures the researchers use
are adequate measures of the components and domains they
say they are measuring.
Method
A. Selecting Media Literacy Intervention Studies
Like Jeong, Cho, and Hwang (2012) did in their meta-
analysis of media literacy intervention studies, we began
with an electronic search of several communication litera-
ture data bases (Communication Abstracts, Psyclit, and
ERIC). But unlike Jeong and colleagues who used many
keyword phrases (media literacy, media literacy intervention,
media literacy curriculum, media literacy program, interven-
tion, advertising, and skepticism), we used the single term
“media literacy & intervention.” Readers may object to our
not using additional search terms that they regard as syn-
onyms for media literacy. However, we could not be sure
which terms are synonyms in widespread use and which are
believed to be synonyms by only a few scholars, so we de-
cided to be conservative for the sake of precision. Admit-
tedly, this resulted in a smaller set of studies than we would
have generated by using more key-word terms, however, by
using one key search term we attempted to avoid an “apples
and oranges” non-equivalency. We argue that the fairest
selection criterion was to rely on authors telling us whether
or not their study was a test of a media literacy intervention
by selecting their study only if they provided the key-word
terms of both “media literacy” and “intervention.”
When we identified a potential study, we searched
through its reference list to identify additional studies that
might have provided tests of media literacy interventions.
We then screened out studies that were not published,
different meanings rather than treating all studies in a single
group and thereby assuming that the authors across all those
studies shared the same meaning.
Grouping studies by meaning is also important when
conducting a meta-analysis, which is a “review that uses a
specific statistical technique for synthesizing the results of
several studies into a single quantitative estimate (i.e., a
summary effect size)” (Petticrew & Roberts, 2006, p. 19).
Thus, when authors plan to conduct a meta-analytical re-
view, a meaning analysis is a crucial first step in order to
ensure that the studies selected for the sample all share the
same meaning for their key concepts. One of the strongest
criticisms of meta-analysis is that it “too often seeks to com-
bine dissimilar studies -- sometimes called the ‘apples and
oranges’ problem” (Petticrew & Roberts, 2006, p. 203). It
appears that Jeong, Cho, and Hwang (2012) were aware of
the use of many synonyms for media literacy when they
selected studies for their meta-analysis of the media literacy
intervention literature. They said they started with four key-
word phrases (media literacy, media literacy intervention,
media literacy curriculum, and media literacy program) to
select studies for their meta-analysis, then broadened their
search criteria. “To include the studies that did not use the
term ‘media literacy,’ we used search terms (e.g., ‘interven-
tion,’ ‘advertising,’ ‘skepticism’)” (p. 466). This procedure
indicates that Jeong and colleagues believed that these seven
terms shared the same meaning and furthermore that the
authors of the 51 studies they selected for their meta-analysis
all shared this meaning.
We believe this body of literature can benefit from a
meaning analysis, and we present one in this manuscript.
We use Chaffee’s (1991) hierarchy to structure our meaning
analysis. Each article in the media literacy intervention lit-
erature is analyzed to determine how authors have pre-
sented their meaning for media literacy. We do not expect
all empirical tests of media literacy to share the same mean-
ing, nor do we expect all studies to be coded at the same
option on the six-level meaning hierarchy. The distribution
of studies across these six options will indicate the degree of
precision authors of these studies exhibit in deriving mean-
ing and presenting it to their readers.
In summary, our content analysis of the empirical media
literacy intervention literature is designed to document the
meanings authors present for “media literacy” then to ana-
lyze the measures those authors have used to operationalize
their conceptualized meanings into measures. In our content
Reviewing Media Literacy Intervention Studies for Validity
45 2019, 7, 38-66
source, it was counted only once.
Focal definition of media literacy.
If authors cited only one definition in their review of the
literature, coders regarded this one definition to be the au-
thors’ conceptualization of media literacy. If the authors
presented more than one definition of media literacy in their
review of the literature, coders then looked for what the
authors claimed to be using as the conceptual foundation for
their study as well as an articulation of their reasoning pro-
cess for their selection and rejections. Coders then catego-
rized this conceptualization on one of the six levels of the
meaning presentation hierarchy developed from Chaffee’s
(1991) criteria (see Table 3).
Definitional elements.
Coders recorded the elements in the definitions used by
authors as their conceptual foundation for media literacy.
These elements took the form of components and domains.
The definitional components were skills, knowledge, behav-
iors, beliefs, attitudes, affects, and other. The domains were
the more specific types of elements within each component.
For example, in the skills component, domains were indi-
vidual skills such as analysis, critical thinking, synthesis,
etc. We did not begin with a list of all possible domains;
instead we recorded what authors regarded as domains.
We defined knowledge as acquired factual information.
Factual information has truth value so that one can assess
whether a fact is accurate or not. In contrast to factual in-
formation is social information which is important but does
not have a factual basis, so this type of information was
coded either as a belief, an attitude, or an affect.
Beliefs have been defined as cognitions about the prob-
ability that an object or event is associated with a given at-
tribute (Fishbein & Ajzen, 1975). Simply stated, a belief is
faith that something is real or is true.
Attitudes were defined as judgments about something.
These evaluative judgments have valence and intensity (Fab-
rigar, MacDonald, & Wegener, 2005). Valance refers to
whether the object of the attitude meets (satisfactory), ex-
ceeds (positive), or falls short of the standard (negative). In-
tensity refers to how far from the standard the object is
perceived to be.
Affect refers to the feelings that people experience. Affect
includes emotions and moods. Behaviors are typically de-
fined as the overt actions of an individual (Albarracin,
because published studies have gone through a scholarly
review process and therefore have an expectation of higher
quality than studies that have not. However, we screened out
studies that did not provide a test of the effectiveness of in-
terventions, because we wanted to see authors report their
measures of media literacy so we could compare the mea-
sures to their meaning in determining content and face valid-
ity.
Our resulting sample was 88 published articles that
claimed to test the effectiveness of a media literacy interven-
tion. All of these studies feature the terms “media literacy”
and “intervention” prominently -- either in title, keyword
list, or abstract. Many of these studies featured these terms
prominently in each of those places as well as many times
in text.
B. Coding Variables
The development of the list of variables, their definitions,
and their codes was a process involving many pilot tests over
several years. When it was finished, the coding itself pro-
gressed smoothly mainly because almost all of the coding
variables were manifest, rather than latent, variables (Hol-
sti, 1969; Krippendorff, 2012). That is, we minimized the
need for coders to make their own inferences of authors’
meaning about media literacy by recording only those mean-
ings expressed by the authors.
The meaning analysis first involved the coding of three
variables: Number of media literacy definitions cited, the
authors’ focal definition of media literacy, and the compo-
nents and domains in those focal definitions. When authors
presented only one definition of media literacy, we used that
meaning as an expression of their conceptual foundation.
When authors presented more than one meaning for media
literacy in their review of the literature, we looked for an
expression by the authors indicating which of those defini-
tions they were using as the conceptual foundation of their
study. This procedure generated data on five variables that
we describe below in more detail.
Media literacy definitions cited.
Coders counted the number of different definitions of
media literacy that authors cited in their introduction, review
of literature, and rationale sections. If a particular definition
was presented more than once, it was counted only once.
Also, if a particular definition was credited to more than one
W. James Potter & Chan L. Thai
46 www.rcommunicationr.org
Zanna, Johnson, & Kumkale, 2005).
Skills were defined as cognitive abilities that can be de-
veloped through training and practice. Wiley (1991) writes
that skills “are abilities to perform tasks. Most such abilities
are acquired, i.e., learned” (p. 78). He says that the valid
measurement of skills requires observation of how people
perform on a task, which is “a goal-oriented activity of de-
terminable duration on which performance can be evaluat-
ed” (p. 105). While the measurement of skills requires the
observation of performance, oftentimes researchers will
measure something (like a belief or attitude) and argue that
this measure is a valid outcome of the application of a skill.
Such a procedure is a shortcut that requires a strong argu-
ment to convince readers that such an outcome could only
have occurred by the use of the skill in question. This is an
especially important argument to make convincingly in
media literacy intervention studies because the purpose of
the intervention is typically to increase the level of a par-
ticular skill or set of skills that the authors regard as the
media literacy part of the intervention. In some studies, the
skill was the outcome variable, that is, the participants who
experienced the media literacy intervention were expected
to exhibit higher scores on the skill measure compared to
participants who did not experience the media literacy inter-
vention. In other studies, the authors exhibited more interest
in another outcome variable, such as an attitude or belief
about some risky behavior; these studies had a two-step
structure where the media literacy intervention was ex-
pected to increase skill levels and those elevated skill levels
were expected to explain changes on the outcome variable.
When authors specified a component, coders also re-
corded whether authors specified domains within that com-
ponent. For example, if authors defined media literacy in
terms of skills, then coders looked for whether authors ar-
ticulated particular skills (such as analysis, evaluation, read-
ing, message production, etc.). If authors defined media
literacy in terms of knowledge, then coders looked for wheth-
er the authors specified types of knowledge (such as how the
industry works, content formulas, etc.).
Content validity.
Coders did not make a global judgment about content
validity but instead used a system of counting components
and domains. First, coders examined authors’ conceptualiza-
tion of media literacy and recorded the components and
domains in those definitions. For example, let’s say that the
authors of a media literacy intervention study defined media
literacy as knowledge about the motives of media companies,
the skills of analysis and evaluation, and attitudes about the
media. In this case, coders would record three components
(knowledge, skills, attitudes) and four domains (one domain
in the knowledge component, two in the skills component,
and one in the attitude component).
Second, coders counted the number of components for
which the authors provided measures. Returning to the ex-
ample, if the authors listed measures that tested their par-
ticipants’ knowledge of the motives of media companies,
their skill of analysis, their skill of evaluation, and their at-
titudes about the media, then coders would list four compo-
nent/domains measured. This would be a match of 100%
and indicate perfect content validity. However, if authors
presented measures for only three of these component/do-
mains, then coders would record those three which would
indicate 75% on content validity.
In a few studies, the authors presented a conceptualiza-
tion of media literacy that had many components and do-
mains but then argued that it was too much for one study to
test all of these components and domains. If those authors
articulated that only a sub-set would be tested in their study,
then we used the sub-set as the criteria for judging content
validity and face validity.
Face validity.
Coders made a judgment about whether or not each
measure of media literacy matched the component-domain
unit that the authors claimed. Thus we did not try to make
judgments about how good the measures were on a contin-
uum; instead, we limited ourselves to counting the number
of matches. For example, if authors expressed a component
of knowledge and a domain of knowledge about media in-
dustries, then we looked for measures testing recall of facts
about the media industry that were taught in the interven-
tion; we did not make a judgment about how complete those
measures were as tests of all aspects of media industries, nor
did we make a judgment about whether those facts were core
or peripheral, nor did we make a judgment about how clear-
ly worded those measures were – all of which would have
required a considerable degree of judgment from coders. We
focused only on fit, that is, did the measure fit the component
and domain that the authors claimed? Therefore, if authors
claimed that a measure such as “How many hours during
an average week do you read business publications about the
Reviewing Media Literacy Intervention Studies for Validity
47 2019, 7, 38-66
authors of 7 studies (8.0%) presented multiple definitions of
media literacy in their reviews and selected one of these
definitions as their foundation.
While the authors of the remaining 34 studies presented
a review of the literature showing multiple meanings of
media literacy and rejected all the definitions preferring to
construct their own definition, the minority of these (8 stud-
ies) displayed a scholarly treatment of that constructed defi-
nition as suggested by Chaffee. That is, in 8 studies the
authors displayed multiple definitions and furthermore crit-
ically analyzed those meanings to show readers how they
constructed their own definitions; in the other 26 studies,
the authors simply described the multiple meanings then
presumably rejected them all by presenting another defini-
tion of their own.
The most popular component to be mentioned in the
conceptual foundations was some sort skill; 59 of the 66
studies mentioned the component of skills in their definition
of media literacy. The next most prevalent component men-
tioned was some knowledge, then there is a drop off in
counts to behaviors, beliefs, and attitudes (see Table 4).
In summary, our meaning analysis revealed a wide dis-
tribution across all six levels of the meaning presentation
hierarchy. Over half the studies were categorized at level 3
or lower.
Validity
In making our judgments about validity, we considered
only the 66 articles that presented a definition for media
literacy; the other 22 articles were ignored because there was
no basis for making a judgment about validity. Also, if the
authors of a particular study presented a definition with
multiple components/domains but then said they were lim-
iting their study to test only a sub-set of these, we used only
the specified sub-set as the standard for validity. But if au-
thors did not say they were limiting their test to a sub-set, we
used the full set of components/domains as a standard for
validity.
The results of our validity analyses are displayed in
Table 4. Studies that presented a definition of media literacy
that included a skill component are presented on the first
line of the table. If the authors defined media literacy in
terms of a knowledge component, it appears as a count on
the second line. If a study’s conceptual foundation called for
more than one component, it was counted more than once
media industries?” was a measure of knowledge, we made
a judgment that this measure did not fit their conceptualiza-
tion because it was a measure of exposure to information,
not a measure of how much knowledge the participants ac-
quired and retained.
C. Reliability
The two authors served as coders on the project. Both
coders coded 55 articles thus creating an overlap of 22 arti-
cles that was used to test for inter-coder reliability. The
percentages of agreement were corrected by Scott’s pi are as
follows: definitions cited, .94; position on meaning hierarchy,
.89; definitional elements, .84; measures of media literacy,
.82; and face validity, .79.
Results
Presentation of Meaning
Across these 88 studies, there was a considerable variety
in the way authors presented their meaning of media litera-
cy (see Table 3). Authors of 22 studies (25%) provided no
definition for media literacy, which indicates that they were
treating it like a primitive concept with the assumption that
all their readers would share the same meaning.
The 66 (75.0%) articles that did present definitions of
media literacy in their review of the literature exhibited a
wide variation. The authors of 11 studies (12.5%) presented
a definition without a citation as to the source of that defini-
tion and no review of the media literacy definitions litera-
ture. Thus, there were 55 studies that presented one or more
definitions of media literacy with citations that indicated the
source of those definitions. Within these 55 studies, the
range was 1 to 7 definitions cited with a median of 2 defini-
tions. It is interesting to note that across these 55 studies, the
most prevalent definition, which was the NAMLE defini-
tion, was mentioned only 7 times (12.7%), which shows that
there is no consensus definition for media literacy, at least
among designers of published media literacy intervention
studies.
Authors of 14 studies (15.9%) presented only one defini-
tion of media literacy in their review of the literature and
used that definition as a foundation for their study, while
W. James Potter & Chan L. Thai
48 www.rcommunicationr.org
much larger than the number in the Definition column is
because many researchers who were attempting to measure
a skill developed items to measure a belief instead. For ex-
ample, several researchers defined media literacy as the
ability to analyze media messages but then, rather than de-
sign a measure to assess their participants’ ability to analyze,
they instead asked their participants to respond to a state-
ment (e.g., I am confident in my ability to analyze media
messages) by choosing a number from a five-point Likert
type scale (e.g., strongly agree, agree, neutral, disagree,
strongly disagree). By doing so, they end up measuring par-
ticipants’ beliefs about their skills rather than measuring par-
ticipants’ level of performance on their skills. Another
disconnect with measuring skill occurred when researchers
asked their participants to respond to a statement (e.g., How
often do you analyze characters when you watch television
shows?) by choosing a number from a five-point Likert type
scale (e.g., always, almost always, often, rarely, never). Here,
they were measuring participants’ self-reported behaviors
rather than the level of performance on the skill.
When we shift attention from components to domains,
we dig deeper into this pattern of disconnects between con-
ceptualizations and operationalizations. Table 5 shows that
the skills component was not only the most prevalent in the
conceptual foundations, but this component displayed the
most domains with 29 studies specifying one particular skill,
and therefore shows up more than once in Table 4.
When we look at the general pattern exhibited in Table
4, we can see that there are considerable discrepancies be-
tween the components authors specified in their definitions
of media literacy compared to the measures they reported.
Some of these discrepancies show up as drop-offs, that is,
authors define media literacy with more components than
they measure. For example, with the component of skill, 59
studies included this component in their definition but only
22 of those studies presented a measure for any type of skill.
This pattern of a drop-off is also seen with the component
of affect.
On other components, the discrepancy is not a drop-off
but an increase, that is, there are higher counts in the mea-
sured column compared to the counts in the conceptual
foundation column. For example, in 11 articles the authors
argued that beliefs were an important component in their
definition of media literacy and yet 24 measured beliefs as
part of media literacy. At first, this might seem like a pattern
of over-measurement where authors establish no basis for
beliefs in their definition but then develop a measure for it.
However, this is not the explanation for the pattern. Instead,
this pattern is explained by authors attempting to measure
one type of component but then designing items that mea-
sure another type of component. With the component of
beliefs, the reason the number in the Measured column is
Table 4. Analysis by Components for Media Literacy by Studies. (back to pg. 10)
Component In Definition Measured Matches
Skills 59 22 12
Knowledge 25 26 9
Behaviors 12 16 3
Beliefs 11 24 7
Attitudes 9 22 7
Affects 5 1 0
Other 6 7 2
n = 66 studies. Numbers in the cells represent studies.
The numbers in the columns do not sum to 66 because almost all studies referred to more than one
component.
Reviewing Media Literacy Intervention Studies for Validity
49 2019, 7, 38-66
pointing out the flaws in the media literacy intervention
literature, we are not advocating perfection as a standard;
instead, we are arguing for a standard that we believe is not
only reasonable but one that is also essential if this literature
is to grow in value. In this Discussion section, we will first
present that standard and show why it is reasonable and es-
sential. Then we will raise some important questions about
the design of media literacy intervention studies – questions
that we will examine in the second half of this Discussion
section.
A. Reasonable Standard
When authors present their study as a test of a media
literacy intervention, we recommend they create a concep-
tual foundation that clearly communicates to the reader what
their meaning is for media literacy, then use this meaning to
guide their selection and/or construction of measures of
media literacy. Thus we argue that in order for a study to be
considered a test of a media literacy intervention it must meet
two minimum criteria. First, authors must present an ar-
ticulation of what they mean by media literacy as a concep-
tual foundation for their study. Second, authors need to
present measures of media literacy that conform to their
13 studies specifying two particular skills, 5 studies specify-
ing three skills, and 9 studies specifying four or more skills.
When we compare the number of mentions of skill domains
in the conceptual foundations with which skills domains
that were measured in those studies, we can see a big drop-
off. This drop-off is also exhibited in the other five compo-
nent areas, but it is not as pronounced as with the drop-off
in the measurement of skills.
When we look for matches within each study, we can see
that the drop-offs are even more pronounced. Table 6 pres-
ents a comparison between how many domains are specified
in the conceptual foundations and how many of those do-
mains were measured. Within the skills component, 30
studies specified one skill domain in the conceptual founda-
tion, but only 5 of those studies provided a measure for that
particular domain, and 13 studies specified two skill do-
mains but only 3 of those studies provided measures for each
of those two.
V. Discussion
We begin this discussion by acknowledging that all re-
search studies have flaws; there are no perfect studies. By
Table 5. Analysis by Domains for Media Literacy by Studies. (back to pg. 11)
# of Mentions in Definition Skills Know Beh Belief Att Affect
One 29 23 9 12 7 8
Two 13 5 5 1 1 0
Three 5 3 1 1 1 0
Four or More 9 6 0 0 0 0
Totals 56 37 15 14 9 8 139
# of Mentions in Definition Skills Know Beh Belief Att Affect
One 12 13 9 10 11 0
Two 6 3 2 3 3 1
Three 1 2 0 3 2 0
Four or More 1 4 1 3 0 0
Totals 20 22 12 19 16 1 105
n = 66 studies. Numbers in the cells represent studies
Note: Know = Knowledge; Beh = Behavior; Att = Attitudes
W. James Potter & Chan L. Thai
50 www.rcommunicationr.org
scholars at this stage in the development of the field. How-
ever, given this rich resource of alternative meanings, au-
thors cannot assume that all their readers will share the same
meaning for the term as their authors do, so we argue that it
is imperative for authors to communicate their preferred
meaning. When authors ignore this task, they fail to provide
a conceptual foundation that would increase the contribution
their study can make to knowledge about media literacy.
Authors can easily meet this standard by simply writing
a sentence in their manuscripts where they provide their
definition of media literacy. Of course, it would be better if
authors also presented a review of the media literacy litera-
ture to demonstrate an awareness of different definitions to
give their work a more scholarly grounding. Moreover, it
would be better still if authors presented a critical analysis
of that literature that would show readers either why they
selected one of those definitions over others or why they
constructed their own definition by synthesizing the best
definitional elements in that literature so that their concep-
tual foundation would have maximum value as a scholarly
foundation for their particular study.
This recommendation that authors clearly articulate their
meaning for media literacy is also essential because without
conceptual foundation.
We argue that this is a reasonable standard because schol-
ars who search for studies about media literacy interventions
expect to learn something about how media literacy can be
taught to others and how that teaching can be most effective.
Therefore if authors label their work as a media literacy in-
tervention study by using the term in the title, abstract,
keyword list, and throughout their article, then they mislead
readers when they do not tell them their meaning for media
literacy.
Conceptual foundation.
This recommendation might seem so obvious that there
would be no need to state it. However, our findings revealed
that 25% of the studies in our sample provided no foundation
for media literacy, which is shocking because this is an easy
criterion to meet.
In making this recommendation that authors of media
literacy intervention studies tell readers what their meaning
of the term is, we are not arguing for the use of any particu-
lar meaning. We believe that the many definitions of media
literacy already displayed in the literature is a strength be-
cause it offers many different choices of meanings for
Table 6. Analysis of Domains of Skills and Knowledge. (back to pg. 12)
Skills Number of Domains Measured
Domains in CF Zero 1 2 3 4 5 6 Percent
1 25 5 16.7%
2 7 3 3 18.8
3 1 4 0 0 0
4 2 0 2 1 0 0
5 2 0 1 0 0 1 25.0
Knowledge
Domains in CF Zero 1 2 3 4 5 6 Percent
1 5 5 50.0%
2 2 2 1 20.0
3 1 0 1 1 33.3
4 0 2 0 0 0 0
5 0 0 0 0 0 1 100.0
6 1 0 1 0 0 0 1 100.0
Note: CF = Conceptual Foundation
Reviewing Media Literacy Intervention Studies for Validity
51 2019, 7, 38-66
in that definition, and actually measured that domain. If
these 40 valid measures were spread out over 40 different
studies, this would indicate that 24 studies (66 – 40) that
provided a conceptual definition of media literacy but did
not measure any of the domains in their definition.
When we add these 24 studies that do not meet the sec-
ond of our criteria to the 22 studies that do not meet the first
criterion, we can see that half the studies in our sample do
not meet both criteria and should therefore not be considered
as part of the media literacy intervention literature despite
the authors of those studies labeling them as such. We are
not arguing that these studies fail to contribute something
valuable to the larger media effects literature. Instead, our
argument is that authors mislead readers and searches of the
literature when they label their studies as tests of media lit-
eracy interventions but then fail to present what they mean
by media literacy and/or fail to present any measures of
media literacy. This is a serious flaw in the media literacy
intervention literature that can be corrected relatively easily
in the future.
B. Important Questions
Our analysis of the current literature on media literacy
interventions raises many questions about why those pub-
lished authors have made the design decisions the way they
did. We conclude this Discussion section with the posing of
three of these questions, which form a sequence of issues that
researchers must deal with when they design any test of a
media literacy intervention. We will show that if authors
confront the issues posed by this sequence of questions rath-
er than ignore them, they will put themselves in a position
to design tests with stronger validity and hence their findings
will contribute more value to the media literacy intervention
literature.
Basic research or evaluation study?
Are researchers more interested in (a) determining the
overall performance of the intervention or (b) isolating one
factor (or a small number of factors) in the intervention to
determine whether it (they) influences the outcome variable?
At first glance, this might seem to be a subtle difference, but
as we will show in this section, the difference presents a fork
in the road for a series of design decisions that would result
in very different types of studies being conducted, and there-
fore very different sets of results being generated (Coffman,
this, there is no basis for readers to judge the validity of
measures.
Using valid measures is an essential requirement of all
social science empirical research (Croucher & Cronn-Mills,
2015; Merrigan & Huston, 2008).
Measures.
The measures of media literacy included in a study can
be used in a variety of ways, such as an outcome variable,
an intervening variable, or an antecedent variable. As an
outcome variable, media literacy measures can be designed
as a post-test administered after an intervention. The expec-
tation then is that this measure would be an assessment of
whether the intervention had an immediate influence on
participants’ levels of media literacy. However, in order to
test this expectation, designers also need to use the measure
in a pre-test. Unless they administer the measure of media
literacy both before and after the intervention, researchers
have no way of telling if the intervention changed partici-
pants’ levels of media literacy. As an intervening variable,
measures of media literacy can be gathered after an interven-
tion then used to predict some outcome variable like the
counter-reading of media messages that advocate risk behav-
iors, such as smoking, illegal drug consumption, or unpro-
tected sex. As an antecedent variable, measures of media
literacy can be administered before a treatment then used as
a predictor of the success of the intervention. Whatever their
position in a research design, media literacy measures are
essential when researchers position their study as a media
literacy intervention study.
In our sample, we found 21 studies (23.9%) that pre-
sented no measures of media literacy. Of these 21 studies, 18
also provided no definition of media literacy while the other
three studies did provide a definition but did not measure
media literacy. Although 66 studies presented measures of
media literacy, most of these studies failed to craft measures
that measured the components/domains they explicitly cited
in their definitions. For example, 59 studies defined media
literacy at least in part through skills but only 22 (37.3%) of
these studies claimed to provide a measure of that skill in
their Methods section, and only 12 of those 22 studies pro-
vided a measure of skill rather than a measure of a belief
about skills. If we look down the “Matches” column in Table
4, we can see that there were 40 measures that had face valid-
ity, that is, where authors presented a conceptual foundation
for media literacy, constructed a measure to assess a domain
W. James Potter & Chan L. Thai
52 www.rcommunicationr.org
Second, researchers focus on only one element in the inter-
vention in an effort to isolate its singular influence on the
outcome variable. In order to do this, researchers try to
control all the factors in the intervention and vary only one
element across different treatments. Third, the intervention
is administered as an experimental treatment to volunteers
who typically come to a laboratory setting where their ex-
perience with the intervention can be highly controlled to
ensure that all participants in a treatment group are given
the same experience. Fourth, the intervention is adminis-
tered by the researchers or their confederates in a controlled
situation so that uniformity is maintained across all partici-
pants and treatment conditions with the exception that one
of the factors of influence is carefully varied across each
treatment condition. Fifth, there are no a priori standards of
success but instead, researchers compare outcome variable
means across the different treatment groups. Sixth, pilot
testing is used to improve the distinctiveness of the factors of
influence that are varied across treatments rather than trying
to increase the value of the treatment to all participants. And
seventh, the ultimate goal of this process is to make claims
about the relative strength of different factors of influence.
Within the published literature of media literacy interven-
tion studies, it appears that all of the studies in our sample
exhibited a basic research design although 37 of those stud-
ies reported a situation that had more matches (on these 7
characteristics) with the evaluation option compared to the
basic research design option. However, none of the 37 were
a complete match on all 7 evaluation elements with none of
the 37 reporting that there was an a priori established stan-
dard for effectiveness. One of the key characteristics of eval-
uation studies is the use of benchmarks that are determined
by the agency before the design of the study (McKenzie &
Smeltzer, 1997). These benchmarks are then used to deter-
mine whether the intervention was successful or not. These
benchmarks are laid out a priori by the funding agency or
by the goals of the curriculum within which the intervention
is being tried. An example of such a benchmark is: After
experiencing the intervention lesson that teaches about me-
dia industries, 70% of all children will provide correct an-
swers on at least 8 items on the 10-item post-test of knowledge
about media industries.
None of the studies in the analysis presented any bench-
marks to be used as criteria for effectiveness. Instead, all 37
studies defaulted to looking for statistical differences across
treatment and control groups. Because none of these
2003; Chen, 2013; Penfield, Baker, Scoble & Wykes, 2014;
Reinking & Alvermann, 2005; Stufflebeam, 2007). The
decision of which path to take can be guided by a consider-
ation of seven characteristics: (a) who designs the interven-
tion, (b) complexity of the intervention, (c) nature of targets
of the intervention, (d) nature of the agents who will deliver
the intervention, (e) standard for judging success of the in-
tervention, (f) pilot testing, and (g) ultimate goal for testing
the intervention.
Evaluation study. Designing an evaluation study is more
useful than designing a basic research study when scholars
are presented with seven characteristics. First, the interven-
tion is designed by a sponsoring agency or people hired or
funded by the sponsoring agency (such as a public school
system, a health agency, a consumer activist group, or a
philanthropic institution). Second, the intervention is typi-
cally a combination of many presentation elements (such as
lectures, print materials, videos, and websites) that involve
targets in many activities (e.g., watching, reading, critiquing,
discussing, and producing). Evaluation researchers assume
that all of these elements work together in a system as the
intervention is delivered in a series of lessons spread out over
time. Third, the intervention is administered to intact groups
in the field (e.g., such as elementary school classrooms)
where random assignment of targets to conditions is impos-
sible and where random assignment of intact groups to con-
ditions is typically limited. Fourth, the people who deliver
the intervention are typically part of the naturalistic environ-
ment (e.g., elementary school teachers, parents) who are
given some training to administer the intervention but who
are not expected to be perfectly matched on all instruc-
tional criteria, which introduces unavoidable differences in
the way the intervention is administered. Fifth, the agency
has created standards for success of the intervention before
its administration. Sixth, there are typically several rounds
of pilot testing to help improve the success of the intervention
where improvements focus on increasing clarity of materials.
And seventh, the ultimate goal of this process is to develop
an instructional package that can be disseminated to other
groups in the hope of overcoming some widespread problem
or trying to make society better in some way.
Basic research study. In contrast, designing a basic research
study is more appropriate to the extent to which the media
literacy intervention meets the following seven characteris-
tics. First, the intervention is designed by the scholars -- not
a sponsoring agency -- who also design and execute the test.
Reviewing Media Literacy Intervention Studies for Validity
53 2019, 7, 38-66
the intervention to empower people to make better decisions
on their own? This distinction reveals a key element in many
conceptualizations of media literacy, which is empowering
individuals to think for themselves and not automatically
accept the meanings presented in many media messages. The
choice here makes a big difference in how the interventions
are designed and what is regarded as a standard to use in
judging the success of the intervention.
Researchers who take the persuasion path of thinking,
design the intervention to convince people to reject one belief
(or behavior) and replace it with another one. In order to
achieve this goal, researchers design an intervention that
contrasts the faulty nature of a belief that participants are
assumed to hold with an alternative belief that the research-
ers regard as better in some way. Success is then measured
by examining how many participants exhibited the research-
er-sanctioned belief following the intervention. Thus, data
analysts look for convergence as evidence of effectiveness,
that is, the extent to which all participants accepted the al-
ternative belief after experiencing the intervention. The more
participants who converge to accepted this belief, the more
effective the intervention is judged to be.
Researchers who take the empowerment path, in con-
trast, design an intervention to help participants learn how
to analyze media messages so they can determine for them-
selves which meanings are faulty or not useful for their own
purposes. Such interventions are designed to show partici-
pants the risks of automatically processing media messages
and show them alternatives to this automatic processing so
that those participants learn to do something that can be
applied in their everyday lives in a way to make them more
in control of how they process meaning and thus to develop
beliefs and behaviors that serve their own needs better.
Therefore, the effectiveness of interventions that are de-
signed to achieve empowerment look for evidence of indi-
viduals thinking for themselves, which suggests divergence
much more than convergence. Thus, a media literacy inter-
vention is successful with a range of opinions exhibited
across participants as long as participants have developed a
higher awareness of the process of developing their opinions
and are able to support those opinions with evidence ar-
ranged in logical arguments. This distinction can be suc-
cinctly stated as the persuasion perspective focuses on giving
people what researchers regard as a better fish, while the
empowerment perspective focuses on teaching people how
to fish better.
evaluation studies compared performance to a standard,
none were able to make meaningful claims about the effec-
tiveness of their tested interventions; instead, they were
limited to making statements about whether there was a
statistical difference in means between the group of partici-
pants who experienced the intervention compared to the
participants who did not (the control group). By largely fol-
lowing many but not all of the conventions of evaluation
studies, these studies were hybrids.
We are not arguing that hybrid designs are always weak-
er than “pure” designs; to the contrary, hybrid studies can
be stronger than traditional designs if scholars synthesize
the strongest elements of each option into a single design
that amplifies strengths and minimizes weaknesses. How-
ever, if the hybrid design arises from designers taking short-
cuts (e.g., it is easier to test for group means than to develop
adequate performance criteria), then the hybrid fails to take
advantage of inherent strengths of a particular method. Such
hybrid designs can still make a contribution to literature, but
the value of that contribution is limited (or enhanced) by the
degree to which researchers reduce the design weaknesses
by increasing strengths. The pattern of design decisions with
evaluation studies that we found indicates that designers
have been eliminating a strength (use of standards for suc-
cess) and replacing it with a weakness (comparison of
means).
We recommend that designers of media literacy interven-
tion studies carefully consider the strengths and weaknesses
of both basic research and evaluation studies. Designers can
begin this task by thinking about the extent to which their
envisioned study meets the 7 characteristics described above.
For example, if their planned study meets more characteris-
tics of an evaluation study than a basic research study, de-
signers should consider whether they can change the
remaining characteristics in their situation to meet the needs
of an evaluation study more completely. One way to do this
would be to create performance criteria as a benchmark of
success. The analysis, then, could avoid comparing group
means and instead compare the performance of individuals
to the standard and report how many participants met the
standard after experiencing each of the tested interventions.
Media literacy as persuasion or empowerment? Do authors
regard the purpose of the intervention as persuading people
to change their beliefs (or behaviors) from something the
authors regard as faulty to something else the authors regard
as desirable? Or instead, do authors regard the purpose of
W. James Potter & Chan L. Thai
54 www.rcommunicationr.org
lated by social cognitive theory (Bandura, 2009) clearly
shows that when people have strong efficacy beliefs, they are
able to learn more and to use that learning. However, we also
argue that there is a difference between people’s beliefs about
their skills and their actual level of performance on those
skills. Our reading of this media literacy intervention litera-
ture has led us to infer that most authors who talk about the
importance of skills to media literacy are referring to people’s
abilities (a) to construct counter-readings of the surface
meanings presented in messages, (b) to analyze those mes-
sages in various contexts, (c) to infer motives of senders, and
even (d) to construct their own messages. The above listed
skills require assessments of individual’s performances in
order to be able to plot those individuals on a range of abili-
ties to perform those tasks. Moreover, skills require perfor-
mance both in the intervention as well as in the measurement
of them. In the athletic realm, basketball coaches do not ask
prospective players: How well do you shoot free throws?
(very good, good, average, below average). Instead, they
observe their performance. Of course, beliefs are also impor-
tant in the sense that if people believe they have good bas-
ketball skills, they are likely to continue playing the game
and work to improve their skills compared to people who do
not hold such positive beliefs. However, it is a person’s ac-
tual performance level more so than their beliefs that reflect
how well the game is played.
While determining the level of basketball players’ free
throw skill through performance is relatively easy, determin-
ing the level of media literacy skills is much more challeng-
ing. Research designers can begin working on this challenge
by using a three-step procedure. First, they need to clarify
as much as possible what the skill is that they want to teach
or improve. Second, they need to think about what the vari-
ous levels of performance are on the skill are, then determine
what observables would indicate performance at each level.
And third, they need to think about the skill as requiring a
sequence of sub-tasks, then design measures to track par-
ticipants through the process of applying that skill in the
completion of each step in the process.
In summary, scholars who present definitions of media
literacy that include a skills component need to think
through the concerns presented in this section. They need
to provide more detail in the form of specifying domains of
skills and be more clear about defining what those skills are.
They also need to articulate their vision about whether they
are dealing with skills or competencies and if skills, then are
In our analysis, we frequently found a disconnect be-
tween conceptual foundations and design decisions on this
point. That is, frequently authors articulated media literacy
by using empowerment-type language, such as attempting
to get participants to think more critically. These researchers
claimed that their media literacy interventions were designed
to teach targets to make counter-readings of messages, resist
the persuasive appeals of messages, and apply their own
standards in formulating their own attitudes rather than
accepting the attitudes and beliefs of product spokespersons,
newscasters, and characters in fictional narratives. However,
then these same authors exhibited design decisions that
formulated a traditional persuasion study where they used
the intervention to persuade participants to change their
beliefs or behaviors.
We are not arguing that empowerment type studies are
inherently superior to persuasion type studies. Instead, we
argue that designers of media literacy intervention studies
need to be clear about which perspective they favor then
design an appropriate study to achieve their intention. There
is room under the broad media literacy umbrella for both
types of studies. However, when researchers argue that their
belief is empowerment, but then design a persuasion study,
they create a hybrid that serves neither purpose well.
Skills as performance or belief? Almost every conceptualiza-
tion of media literacy suggests a skills component. Also, a
high percentage of empirical tests of media literacy claim to
deal with at least one skill. Recall that in our study, we found
that among the 66 published studies that provided a media
literacy conceptual foundation, 59 defined media literacy at
least in part as requiring the development of skills. When
designers of media literacy intervention studies confront the
issue of skills, they must ask themselves whether they regard
skills as performances or as people’s beliefs about abilities?
The way researchers answer this question indicates how
much of a challenge they are willing to undertake when
designing their interventions and measures of media literacy
skills. If researchers regard skills as beliefs about abilities,
the challenge is relatively easy to meet. However, if research-
ers regard skills as performances that involve the application
of thinking processes to complete tasks, then the challenge
is considerably higher. This increase in design complexity
and cost is a likely reason why so few studies in our sample,
treated skills as performances.
We argue that there is value in measuring participants’
beliefs about their skills. The large research literature stimu-
Reviewing Media Literacy Intervention Studies for Validity
55 2019, 7, 38-66
ture generally exhibits a low level of scholarly quality in the
presentation of meaning as well as incomplete and inaccu-
rate follow through on the design of measures. Thus, most
of the studies in this literature have failed to establish a
minimum level of validity.
Our analysis also illuminated some important issues for
designers of media literacy intervention studies to consider
in their designs, including the basic criterion of presenting a
definition for media literacy, whether the study being con-
ducted is truly an evaluation of an intervention or a basic
research study, the purpose of media literacy interventions,
and how skills should be treated. If we wish to increase the
validity of our media literacy intervention studies, we must
clearly articulate what we mean by media literacy and use
those meanings to guide our research design decisions in a
much more systematic manner.
those skills broad or specialized to media literacy. Scholars
who clearly lay out their positions on these issues will be
providing a great deal more guidance to designers of mea-
sures. Then the designers of those measures need to be more
complete and accurate in their operationalizations in order
to achieve validity.
VI. Conclusion
This study has presented an examination of the validity
of 88 published studies of media literacy interventions.
While this literature has been generated by sincere scholars
who have invested a great deal of effort designing interven-
tions trying to help people improve their media literacy, this
analysis revealed that the media literacy intervention litera-
References
References marked with an asterisk indicate studies included in the analysis.
*Abelman, R., & Courtright, J. (1983). Television literacy: Amplifying the cognitive level effects of television’s prosocial fare
through curriculum intervention. Journal of Research and Development in Education, 17(1), 46-57.
Adams, D., & Hamm, M. (2001). Literacy in a multimedia age. Norwood, MA: Christopher-Gordon Publishers.
*Aguaded-Gómez, I., Tirado-Morueta, R., & Hernando-Gómez, Á. (2015). Media competence in adult citizens in Andalusia,
Spain. Information, Communication & Society, 18(6), 659-679.
Albarracin, D., Zanna, M. P., Johnson, B. T., & Kumkale, G. T. (2005). Attitudes: Introduction and scope. In D. Albarracin,
B. T. Johnson, & M. P. Zanna (Eds.). The handbook of attitudes (pp. 3-19) Mahwah, NJ: Erlbaum.
Alvermann, D. E., Moon, J. S., & Hagood, M. C. (1999). Popular culture in the classroom: Teaching and researching critical media
literacy. Newark, DE: International Reading Association.
Anderson, J. A. (1981). Receivership skills: An educational response. In M. Ploghoft & J. A. Anderson (Eds.) Education for
the television age. Springfield, IL: Charles C. Thomas.
Anderson, J. A. (1983). Television literacy and the critical viewer. In J. Bryant & D. R. Anderson (Eds.), Children’s understanding
of television: Research on attention and comprehension (pp. 297–327). New York: Academic Press.
*Ashley, S., Lyden, G., & Fasbinder, D. (2013). Exploring message meaning: A qualitative media literacy study of college
AcknowledgementThe arguments made by the authors are their own and their perspective is not necessarily shared by the reviewers.
W. James Potter & Chan L. Thai
56 www.rcommunicationr.org
freshmen. The Journal of Media Literacy Education. 4(3). Available at: http://digitalcommons.uri.edu/jmle/vol4/iss3/4
Aufderheide, P. (1997). Media literacy: From a report of the National Leadership Conference on Media Literacy. In R. Kubey
(Ed.). Media literacy in the information age: Current perspectives, Information and behavior (Vol. 6) (pp. 79 - 86). New Brunswick,
NJ: Transaction Publishers.
*Austin, E. W., & Johnson, K. K. (1997a). Immediate and delayed effects of media literacy training on third grader’s decision
making for alcohol. Health Communication, 9(4), 323-349.
*Austin, E. W., & Johnson, K. K. (1997b). Effects of general and alcohol-specific media literacy training on children’s decision
making about alcohol. Journal of Health Communication, 2, 17-42.
*Austin, E. W., Chen, Y. Y. Pinkleton, B. E., & Johnson, J. Q. (2006). Benefits and costs of Channel One in a middle school
setting and the role of media-literacy training. Pediatrics, 117(3), e423-e433.
*Austin, E. W., Pinkleton, B. E., & Funabiki, R. P. (2007). The desirability paradox in the effects of media literacy training.
Communication research, 34(5), 483-506.
*Austin, E. W., Pinkleton, B. E., Austin, B. W., & Van de Vord, R. (2012). The relationships of information efficacy and
media literacy skills to knowledge and self-efficacy for health-related decision making. Journal of American college health,
60(8), 548-554.
*Austin, E. W., Pinkleton, B. E., Hust, S. T., & Cohen, M. (2005). Evaluation of an American Legacy Foundation/
Washington State Department of Health media literacy pilot study. Health Communication, 18, 75-95.
*Babad, E., Peer, E., & Hobbs, R. (2012). Media literacy and media bias: Are media literacy students less susceptible to
nonverbal judgment biases? Psychology of Popular Media Culture, 1, 97-107.
Bachmair, B., & Bazalgette, C. (2007). The European Charter for Media Literacy: Meaning and potential. Research in
Comparative and International Education, 2(1), 80-87.
Bandura, A. (2009). Social cognitive theory of mass communication. In J. Bryant & M. B. Oliver (Eds.). Media effects: Advances
in theory and research (3rd ed.) (pp. 94-124). New York: Routledge.
*Banerjee, S. C., & Greene, K. (2006). Analysis versus production: Adolescent cognitive and attitudinal responses to
antismoking interventions. Journal of Communication, 56, 773-794.
*Banerjee, S. C., & Greene, K. (2007). Antismoking initiatives: Effects of analysis versus production media literacy
interventions on smoking-related attitude, norm, and behavioral intention. Health Communication, 22(1), 37-48.
Bausell, R. B. (1986). A practical guide to conducting empirical research. New York: Harper & Row.
*Begoray, D. L., Banister, E. M., Wharf Higgins, J., & Wilmot, R. (2014). Online, tuned in, turned on: Multimedia approaches
to fostering critical media health literacy for adolescents. Asia-Pacific Journal of Health, Sport and Physical Education, 5(3),
267-280.
*Beltramini, R. F., & Bridge, P. D. (2001). Relationship between tobacco advertising and youth smoking: Assessing the
effectiveness of a school-based antismoking intervention program. The Journal of Consumer Affairs, 35, 263-277
*Bickham, D. S., & Slaby, R. G. (2012). Effects of a media literacy program in the US on children’s critical evaluation of
unhealthy media messages about violence, smoking, and food. Journal of Children and Media, 6(2), 255-271.
*Bier, M. C., Schmidt, S. J., Shields, D., Zwarun, L., Sherblom, S., Primack, B., Pulley, C., & Rucker, B. (2011). School-based
smoking prevention with media literacy: A pilot study. Journal of Media Literacy Education, 2, 185-198.
Blanchard, R. O., & Christ, W. G. (1993). Media education and the liberal arts: A blueprint for the new professionalism. Hillsdale,
NJ: L. Erlbaum Associates.
Brinberg, D., & McGrath, J. E. (1985). Validity in the research process. Newbury Park, CA: Sage.
Brown, J. A. (1998). Media literacy perspectives. Journal of Communication, 48(1), 44 – 57
Buckingham, D. (1993). Children talking television: The making of television literacy. Washington, DC: The Falmer Press.
Buckingham, D. (1998). Media education in the UK: Moving beyond protectionism. Journal of Communication, 48(1), 33–43.
*Buijzen, M. (2007). Reducing children’s susceptibility to commercials: Mechanisms of factual and evaluative advertising
interventions. Media Psychology, 9, 411-430.
*Buijzen, M., & Mens, C. (2007). Adult mediation of television advertising effects: A comparison of factual, evaluative, and
Reviewing Media Literacy Intervention Studies for Validity
57 2019, 7, 38-66
combined strategies. Journal of Children and Media, 1(2), 177-191.
*Byrne, S. (2009). Media literacy interventions: What makes them boom or boomerang? Communication Education, 58, 1-14.
Chaffee, S. H. (1991). Explication. Newbury Park, CA: Sage.
Chen, H. L (2013). Evaluation vs. research: What’s the difference. Retrieved April 17, 2018 from https://web.stanford.edu/
group/design_education/wikiupload/2/27/Helen_Evaluation.pdf.
*Chen, Y. C. (2013). The effectiveness of different approaches to media literacy in modifying adolescents’ responses to
alcohol. Journal of health communication, 18(6), 723-739.
*Chen, Y. C. Y., Kaestle, C. E., Estabrooks, P., & Zoellner, J. (2013). US Children’s Acquisition of Tobacco Media Literacy
Skills: A Focus Group Analysis. Journal of Children and Media, 7(4), 409-427.
*Choma, B. L., Foster, M. D., & Radford, E. (2007). Use of objectification theory to examine the effects of a media literacy
intervention on women. Sex Roles, 56, 581-591.
Coffman, J. (2003). Ask the Expert: Michael Scriven on the Differences Between Evaluation and Social Science Research.
The Evaluation Exchange, 9(4). Retrieved January 8, 2012 from http://www.hfrp.org/evaluation/the-evaluation-exchange/
issue-archive/reflecting-on-the-past-and-future-of-evaluation/michael-scriven-on-the-differences-between-evaluation-
and-social-science-research
*Comer, J. S., Furr, J. M., Beidas, R. S., Weiner, C. L., & Kendall, P. C. (2008). Children and terrorism-related news: Training
parents in coping and media literacy. Journal of Consulting and Clinical Psychology, 76, 568-578.
*Coughlin, J. W., & Kalodner, C. (2006). Media literacy as a prevention intervention for college women at low- or high-risk
for eating disorders. Body Image: An International Journal of Research, 3, 35-43.
Croucher, S. M., & Cronn-Mills, D. (2015). Understanding communication research methods: A theoretical and practical approach.
New York: Routledge.
*Duran, R. L., Yousman, B., Walsh, K. M., & Longshore, M. A. (2008). Holistic media education: An assessment of the
effectiveness of a college course in media literacy. Communication quarterly, 56(1), 49-68.
*Eisen, M. (2001). “Intermediate Outcomes from a Life Skills Education Program with a Media Literacy Component.” In
William Crano and Michael Burgoon (Eds.). Mass Media and Drug Prevention: Classic and Contemporary Theories
and Research. The Claremont Symposium on Applied Social Psychology. Lawrence Erlbaum Associates: Mahwah, NJ.
*Evans, A. E., Dave, J., Tanner, A., Duhe, S., Condrasky, M., Wilson, D., Griffin, S., Palmer, M., & Evans, M. (2006).
Changing the home nutrition environment. Effects of a nutrition and media literacy pilot intervention. Family Community
Health, 29, 43-54.
Fabrigar, L. R., MacDonald, T. K. & Wegener, D. T. (2005). The structure of attitudes. In D. Albarracin, B. T. Johnson, &
M. P. Zanna (Eds.), The handbook of attitudes (pp. 79–124). Mahwah, NJ: Erlbaum.
*Feuerstein, M. (1999). Media literacy in support of critical thinking. Journal of Educational Media, 24, 43-54.
*Fingar, K. R., & Jolls, T. (2014). Evaluation of a school-based violence prevention media literacy curriculum. Injury prevention,
20(3), 183-190.
Fishbein, M. & Ajzen, I. (1975). Belief, attitude, intention, and behavior: An introduction to theory and research. Reading, MA:
Addison-Wesley.
*Geraee, N., Kaveh, M. H., Shojaeizadeh, D., & Tabatabaee, H. R. (2015). Impact of media literacy education on knowledge
and behavioral intention of adolescents in dealing with media messages according to stages of change. Journal of advances
in medical education & professionalism, 3(1), 9.
*Goldberg, M. E., Neidermeier, K. E., Bechtel, L. J., & Gorn, G. J. (2006). Heightening adolescent vigilance toward alcohol
advertising to forestall alcohol use. Journal of Public Policy & Marketing, 25, 147-159.
*Gonzales, R., Glik, D., Davoudi, M., & Ang, A. (2004). Media literacy and public health: Integrating theory, research, and
practice for tobacco control. American Behavioral Scientist, 48, 189-201.
Grow, G. (1990, August). Enhancing self-direction in journalism education. Paper presented at the Association for Education in
Journalism and Mass Communication convention, Minneapolis, MN.
Guilford, J. P. (1954). Psychometric methods. New York: McGraw-Hill.
W. James Potter & Chan L. Thai
58 www.rcommunicationr.org
*Halliwell, E., Easun, A., & Harcourt, D. (2011). Body dissatisfaction: Can a short media literacy message reduce negative
media exposure effects amongst adolescent girls? British Journal of Health Psychology, 16, 396-403.
*Harrison, K., & Hefner, V. (2014). Virtually perfect: Image retouching and adolescent body image. Media Psychology, 17(2),
134-153.
*Hindin, T. J., Contento, I. R., & Gussow, J. D. (2004). A media literacy nutrition education curriculum for head start parents
about the effects of television advertising on their children’s food requests. Journal of the American Dietetic Association, 104,
192-198.
Hobbs, R. (1998). The seven great debates in the media literacy movement. Journal of Communication, 48(1), 16-32
Hobbs, R. (2011). The state of media literacy: A response to Potter. Journal of Broadcasting & Electronic Media, 55(3), 419-430.
*Hobbs, R., & Frost, R. (2003). Measuring the acquisition of media literacy skills. Reading Research Quarterly, 38, 330-355.
*Hobbs, R., Donnelly, K., Friesem, J. & Moen, M. (2013, November). Learning to engage: how positive attitudes about the
news, media literacy, and video production contribute to adolescent civic engagement. Educational Media International,
50(4), 231-246.
Holsti, O. R. (1969). Content analysis for the social sciences and humanities. New York: Addison-Wesley.
*Huesmann, L. R., Eron, L. D., Klein, R., Brice, P., & Fischer, P. (1983). Mitigating the imitation of aggressive behaviors
by changing children’s attitudes about media violence. Journal of Personality and Social Psychology, 44, 899-910.
*Irving, L. M., & Berel, S. R. (2001). Comparison of media-literacy programs to strengthen college women’s resistance to
media messages. Psychology of Women Quarterly, 25, 103-111.
*Irving, L. M., DuPen, J., & Berel, S. (1998). A media literacy program for high school females. Eating Disorders, 6, 119-131.
Jeong, S-H., Cho, H., & Hwang, Y. (2012). Media literacy interventions: A meta-analytic review. Journal of Communication,
62(1), 454-472.
*Kaestle, C. E., Chen, Y., Estabrooks, P. A., Zoellner, J., & Bigby, B. (2013). Pilot evaluation of a media literacy program
for tobacco prevention targeting early adolescents shows mixed results. American Journal of Health Promotion, 27(6), 366-
369.
*Kline, S. (2005). Countering children’s sedentary lifestyles: An evaluative study of a media-risk education approach.
Childhood, 12, 239-258
McVey, G. L., & Davis, R. (2002). A program to promote positive body image: A 1-year follow-up evaluation. The Journal of
Early Adolescence, 22, 96-108.
Krippendorff, K. (2012). Content analysis: An introduction to its methodology (3rd ed.). Los Angeles, CA: Sage.
*Kupersmidt, J. B., Scull, T. M., & Austin, E. W. (2010). Media literacy education for elementary school substance use
prevention: study of media detective. Pediatrics, 126(3), 525-531.
*Kupersmidt, J. B., Scull, T. M., & Benson, J. W. (2012). Improving media message interpretation processing skills to promote
healthy decision making about substance use: the effects of the middle school media ready curriculum. Journal of health
communication, 17(5), 546-563.
Levitt, A. M., & Denniston, R. W. (2014). Federal agency efforts to advance media literacy in substance abuse prevention.
Journal of Media Literacy Education, 6(2), 79-86.
Lewis, J., & Jhally, S. (1998). The struggle over media literacy. Journal of Communication, 48(1), 109–120.
Mackey, M. (2002). Literacies across media: Playing the text. London: Routledge.
*Maksl, A., Craft, S., Ashley, S., & Miller, D. (2017). The usefulness of a news media literacy measure in evaluating a news
literacy curriculum. Journalism & Mass Communication Educator, 72(2), 228-241.
*Martens, H., & Hobbs, R. (2015). How media literacy supports civic engagement in a digital age. Atlantic Journal of
Communication, 23(2), 120-137.
Masterman, L. (1985). Teaching the media. London: Comedia.
McKenzie, J. F., & Smeltzer, J. L. (1997). Planning, implementing, and evaluating health promotion programs: A primer. Boston:
Allyn and Bacon.
*McVey, G. L., & Davis, R. (2002). A program to promote positive body image: A 1-year follow-up evaluation. The Journal
Reviewing Media Literacy Intervention Studies for Validity
59 2019, 7, 38-66
of Early Adolescence, 22(1), 96-108.
Media Literacy Project (n.d.) What is media literacy? Retried from https://medialiteracyproject.org/learn/media-literacy
*Melki, J. P. (2015). Guiding Digital and Media Literacy Development in Arab Curricula through Understanding Media
Uses of Arab Youth. Journal of Media Literacy Education, 6(3), 14-28.
Merrigan, G., & Huston, C. L. (2008). Communication research methods. New York: Oxford.
Messaris (1998). Visual aspects of media literacy. Journal of Communication, 48(1), 70 - 80.
Meyrowitz, J. (1998). Multiple media literacies. Journal of Communication, 48(1), 96-108.
*Mihailidis, P. (2009). The First Step Is the Hardest: Finding Connections in Media Literacy Education. Journal of Media
Literacy Education, 1(1), 53-67.
*Moore, J., Dechillo, N., Nicholson, B., Genovese, A., & Sladen, S. (2000). Flashpoint: An innovative media literacy
intervention for high-risk adolescents. Juvenile and Family Court Journal, 51, 23-34.
*Mora, M., Penelo, E., Gutiérrez, T., Espinoza, P., González, M. L., & Raich, R. M. (2015). Assessment of two school-based
programs to prevent universal eating disorders: media literacy and theatre-based methodology in Spanish adolescent
boys and girls. The Scientific World Journal, 2015.
*Naiditch, F. (2013). A media literate approach to developing diversity education. Journal of Media Literacy Education, 5(1), 6.
*Nathanson, A. I., & Yang, M. (2003). The effects of mediation content and form on children’s responses to violent television.
Human Communication Research, 29, 111-134.
Nunnally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.
*Park, S., & Burford, S. (2013). A longitudinal study on the uses of mobile tablet devices and changes in digital media literacy
of young adults. Educational Media International, 50(4), 266-280.
*Parola, A., & Ranieri, M. (2011). The practice of media education: International research on six European countries. Journal
of Media Literacy Education, 3(2), 4.
Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014). Assessment, evaluations, and definitions of research impact:
A review. Research Evaluation, 23(1), 1 21–32.
*Peterson, T. L. (2012). Exploring baseline food-media literacy of adult women. Journal of Media Literacy Education, 4(1), 2.
*Pinkleton, B. E., Austin, E. W., Chen, Y. C. Y., & Cohen, M. (2013). Assessing effects of a media literacy-based intervention
on US adolescents’ responses to and interpretations of sexual media messages. Journal of Children and Media, 7(4), 463-479.
*Pinkleton, B. E., Austin, E. W., Cohen, M., Chen, Y. C. Y., & Fitzgerald, E. (2008). Effects of a peer-led media literacy
curriculum on adolescents’ knowledge and attitudes toward sexual behavior and media portrayals of sex. Health
communication, 23(5), 462-472.
*Pinkleton, B. E., Austin, E. W., Cohen, M., Miller, A. & Fitzgerald, E. (2007). A statewide evaluation of the effectiveness
of media literacy training to prevent tobacco use among adolescents. Health Communication, 21, 23-34.
*Piran, N., Levine, M. P., & Irving, L. M. (2000). GO GIRLS! Media literacy, activism, and advocacy project. Healthy Weight
Journal, Nov/Dec 2000, 89-90.
*Posavac, H. D., Posavac, S. S., & Weigel, R. G. (2001). Reducing the impact of media images on women at risk for body
image disturbance: Three targeted interventions. Journal of Social and Clinical Psychology, 20, 324-340.
Potter, W. J. (2010). The state of media literacy. Journal of Broadcasting & Electronic Media, 54, 675-696.
*Primack, B. A., Douglas, E. L., Land, S. R., Miller, E., & Fine, M. J. (2014). Comparison of Media Literacy and Usual
Education to Prevent Tobacco Use: A Cluster-Randomized Trial. Journal of School Health, 84(2), 106-115.
*Primack, B. A., Sidani, J., Carroll, M. V., Fine, M. J. (2009). Associations between smoking and media literacy in college
students. Journal of Health Communication, 14, 541-555.
*Rabak-Wagenar, J., Eickhoff-Shemek, J., & Kelly-Vance, L. (1998). The effect of media analysis on attitudes and behaviors
regarding body image among college students. Journal of American College Health, 47, 29-35.
*Raich, R. M., Portell, M., Pelaez-Fernandez, M. A. (2010). Evaluation of a school-based programme of universal eating
disorders prevention: Is it more effective in girls at risk? European Eating Disorders Review, 18, 49-57.
*Ramasubramanian, S., & Oliver, M. B. (2007). Activating and suppressing hostile and benevolent racism: Evidence for
W. James Potter & Chan L. Thai
60 www.rcommunicationr.org
comparative media stereotyping. Media Psychology, 9, 623-646.
*Reichert, T., LaTour, M. S., Lambiase, J. J., & Adkins, M. (2007). A test of media literacy effects and sexual objectification
in advertising. Journal of Current Issues and Research in Advertising, 29, 81-92.
Reinking, D., & Alvermann, D. E. (2005). What are evaluation studies, and should they be published in RRQ? Reading
Research Quarterly, 40(2), 142-146
*Richardson, S. M., Paxton, S. J., & Thomson, J. S. (2009). Is BodyThink an efficacious body image and self-esteem program?
A controlled evaluation with adolescents. Body Image, 6, 75-82.
*Ridolfi, D. R., & Vander Wal, J. S. (2008). Eating disorders awareness week: The effectiveness of a one-time body image
dissatisfaction prevention session. Eating Disorders, 16, 428-433.
*Rosenkoetter, L. I., Rosenkoetter, S. E., Ozretich, R. A., & Acock, A. C. (2004). Mitigating the harmful effects of violent
television. Applied Developmental Psychology, 25, 25-47.
Rust, J., & Golombok, S. (1989). Modern psychometrics (2nd ed.). London: Routledge.
*Scharrer, E. (2006). “I noticed more violence:” The effects of a media literacy program on critical attitudes toward media
violence. Journal of Mass Media Ethics, 21, 69-86.
*Scharrer, E. (2009). Measuring the Effects of a Media Literacy Program on Conflict and Violence. Journal of media literacy
education, 1(1), 12-27.
*Scharrer, E., & Wortman Raring, L. (2012). A media literacy curriculum on violence in the United States: studying young
people’s written responses for evidence of learning. Journal of Children and Media, 6(3), 351-366.
*Scull, T. M., & Kupersmidt, J. B. (2011). An evaluation of a media literacy program training workshop for late elementary
school teachers. The journal of media literacy education, 2(3), 199.
*Sekarasih, L., Walsh, K. R., & Scharrer, E. (2015). “Media violence is made to attract and entertain people”: Responses to
Media Literacy Lessons on the Effects of and Institutional Motives Behind Media Violence. Journal of Media Literacy
Education, 6(3), 1-13.
Sholle, D., & Denski, S. (1995). Critical media literacy: Reading, remapping, rewriting. In P. McLaren, R. Hammer, Sholle,
D., & S. S. Reilly (Eds.) Rethinking media literacy: A critical pedagogy of representation (pp. 7 - 31). New York: Peter Lang.
Silverblatt, A. (1995). Media literacy: Keys to interpreting media messages. Westport, CT: Praeger.
Silverblatt, A., & Eliceiri, E. M. E. (1997). Dictionary of media literacy. Westport, CT: Greenwood Press.
Silverblatt, A., Ferry, J., & Finan, B. (2015). Approaches to media literacy: A handbook, (2nd ed.). New York, NY: Routledge.
Speech Communication Association. (1996). Speaking, listening, and media literacy standards for K through 12 education. Annandale,
VA: National Communication Association.
*Sprafkin, J., Watkins, L. T., & Gadow, K. D. (1990). Efficacy of a television literacy curriculum for emotionally disturbed
and learning disabled children. Journal of Applied Developmental Psychology, 11, 225-244.
*Steinke, J., Lapinski, M. K., Crocker, N., Zietsman-Thomas, A., Williams, Y., Evergreen, S. H., & Kuchibhotla, S. (2007).
School-aged children’s perceptions of women in science using the Draw-A-Scientist-Test (DAST). Science Communication,
29, 35-64.
Stufflebeam, D.L. (2007). CIPP Evaluation Model Checklist. Retrieved January 8, 2012 from http://www.wmich.edu/
evalctr/archive_checklists/cippchecklist_mar07.pdf
Tyner, K. (2009). Media literacy: New agendas in communication. New York: NY: Routledge.
Vogt, W. P. (2005) Dictionary of statistics & methodology, (3rd ed.). Thousand Oaks, CA: Sage.
*Vraga, E. K., Tully, M., Akin, H., & Rojas, H. (2012). Modifying perceptions of hostility and credibility of news coverage
of an environmental controversy through media literacy. Journalism, 13(7), 942-959.
*Wade, T. D., Davidson, S., & O’Dea, J. A. (2003). A preliminary controlled evaluation of a school-based media literacy
program and self-esteem program for reducing eating disorder risk factors. International Journal of Eating Disorders, 33(4),
371-383.
*Walsh, K. R., Sekarasih, L., & Scharrer, E. (2014). Mean Girls and Tough Boys: Children’s Meaning Making and Media
Literacy Lessons on Gender and Bullying in the United States. Journal of Children and Media, 8(3), 223-239.
Reviewing Media Literacy Intervention Studies for Validity
61 2019, 7, 38-66
*Walther, B., Hanewinkel, R., & Morgenstern, M. (2014). Effects of a brief school-based media literacy intervention on
digital media use in adolescents: cluster randomized controlled trial. Cyberpsychology, Behavior, and Social Networking, 17(9),
616-623.
*Webb, T., & Martin, K. (2012). Evaluation of a US school-based media literacy violence prevention curriculum on changes
in knowledge and critical thinking among adolescents. Journal of children and media, 6(4), 430-449.
*Webb, T., Martin, K., Afifi, A. A., & Kraus, J. (2010). Media literacy as a violence-prevention strategy: A pilot evaluation.
Health promotion practice, 11(5), 714-722.
Wiley, D. E. (1991). Test validity and invalidity reconsidered. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social
science: A volume in honor of Lee J. Cronbach (pp. 75-107). Hillsdale, NJ: Earlbaum.
*Wilksch, S. M., & Wade, T. D. (2009). Reduction of shape and weight concern in young adolescents: A 30-month controlled
evaluation of a media literacy program. Journal of the American Academy of Child Adolescent Psychiatry, 48, 652-653.
*Wilksch, S. M., Durbridge, M. R., & Wade, T. D. (2008). A preliminary controlled comparison of programs designed to
reduce risk of eating disorders targeting perfectionism and media literacy. Journal of the American Academy of Child Adolescent
Psychiatry, 47, 939-947.
*Wilksch, S. M., Tiggemann, M., & Wade, T. D. (2006). Impact of interactive school-based media literacy lessons for reducing
internalization of media ideals in young adolescent girls and boys. International Journal of Eating Disorders, 39, 385-393.
Williams, F. (1986). Reasoning with statistics: How to read quantitative research (3rd ed.). New York: Holt, Rinehart and Winston.
*Yamamiya, Y., Cash, T. F., Melnyk, S. E., Posavac, H. D., & Posavac, S. S. (2005). Women’s exposure to thin-and-beautiful
media images: body image effects of media-ideal internalization and impact-reduction interventions. Body Image, 2, 74-
80.
Zettl, H. (1998). Contextual media aesthetics as the basis for media literacy. Journal of Communication, 48(1), 81-95.
*Zillmann, D., Callison, C., & Gibson, R. (2009). Quantitative media literacy: Individual differences in dealing with numbers
in the news. Media Psychology, 12(4), 394-416.
62
From Scholars
Adams & Hamm (2001): “the ability to create personal meaning from the visual and verbal symbols we take in every day
from television, advertising, film, and digital media. It is more than inviting students to simply decode information. They
must be critical thinkers who can understand and produce in the media culture swirling around them” (p. 33).
Anderson (1981): the “skillful collection, interpretation, testing and application of information regardless of medium or
presentation for some purposeful action (p. 22).
Naiditch (2013): “the ability to develop and use critical thinking skills (such as sorting through, analyzing, and assessing
information) to interpret media messages and to create meanings out of those messages…by becoming media literate,
people learn to use critical lenses both as consumers of media messages and as producers of their own messages”(p. 339)
Scharrer & Raring (2012): “an inquiry-based process of consideration of multiple layers of a topic in the formation of one’s
own evaluation” (p. 352).
Sholle & Denski (1995): “it must be conceived as a political, social and cultural practice” (p. 17).
Silverblatt & Eliceiri (1997): “a critical-thinking skill that enables audiences to decipher the information that they receive
through the channels of mass communications and empowers them to develop independent judgments about media
content” (p. 48).
From Professional Associations
National Communication Association: A media literate person understands how words, images, and sounds influence the
way meanings are created and shared in contemporary society in ways that are both subtle and profound. A media liter-
ate person is equipped to assign value, worth and meaning to media use and media messages. (http://www.natcom.org/
uploadedFiles/About_NCA/Leadership_and_Governance/Public_Policy_Platform/K-12Standards.pdf)
National Council for the Social Studies: The multimedia age requires new skills for accessing, analyzing, evaluating, creat-
ing, and distributing messages within a digital, global, and democratic society. The acquisition and application of critical
analysis and media production skills are part of what constitutes media literacy. (http://www.socialstudies.org/positions/
medialiteracy)
National Leadership Conference on Media: The ability to access, analyze, evaluate, and communicate messages in a wide
variety of forms literacy. (Aufderheide, 1997).
From Citizen Action Groups
Action Coalition for Media Education: Encourage critical thinking and free expression, examine the corporate media system,
and inspire active participation in society. (http://www.acmecoalition.org/)
Center for Media Literacy: “A framework for accessing, analyzing, evaluating and creating media. The development of
critical thinking and production skills needed to live fully in the 21st century media culture. Also defined as a “new vi-
sion of literacy for the 21st century: the ability to communicate competently in all media forms, print and electronic, as
well as to access, understand, and analyze and evaluate the powerful images, words and sounds that make up our con-
temporary mass media culture.” Also, “Through a four-step ‘inquiry’ process of awareness. . .Analysis. . .Reflection. .
.Action, media literacy helps young people acquire an empowering set of ‘navigational’ skills” which include the ability
to access, analyze, evaluate, and create media.
(http://www.medialit.org/about-cml)
Citizens for Media Literacy: How to think critically about TV and advertising. Special emphasis is placed on the benefits
of telling one’s own stories rather than being pre-occupied with manufactured stories designed to promote the purchase
of products. (http://www.main.nc.us/cml/)
Coalition for Quality Children’s Media (KIDS FIRST!): Teaching kids to become more critical media users and to reduce
the impact of and exposure to violent and biased media. We teach them to recognize programs that are intellectually and
Table 1. Sampling of Definitions of Media Literacy. (back to pg. 3)
W. James Potter & Chan L. Thai
www.rcommunicationr.org
63
creatively stimulating; that break down racial, gender, handicapped and cultural boundaries; and that are produced with
high technical and artistic standards. (http://www.kidsfirst.org/kidsfirst/html/whatcq.htm)
Common Sense Media: The ability to identify, find, evaluate, and use information effectively. From effective search strate-
gies to evaluation techniques, students learn how to evaluate the quality, credibility, and validity of websites, and give
proper credit.
(https://www.commonsensemedia.org/educators/digital-citizenship/information-literacy)
Media Watch: challenge abusive stereotypes and other biased images commonly found in the media. (http://www.medi-
awatch.com/?page_id=32)
National Association for Media Literacy Education: The purpose of media literacy education is to help individuals of all
ages develop the habits of inquiry and skills of expression that they need to be critical thinkers, effective communicators
and active citizens in today’s world. (https://namle.net/publications/core-principles/)
Governmental Groups
Eunice Kennedy Shriver National Institute of Child Health and Human Development (through the Media Smart Program):
having awareness of and critical thinking skills about the media’s role in influencing choices related to nutrition and
physical activity. Young people should learn to question the who, what, why, and how behind words and images in the
media. As a result, they develop critical thinking skills that help them form their own opinions and make their own
choices about the messages they see and hear. (https://www.nichd.nih.gov/msy/Pages/index.aspx)
New Mexico Media Literacy Project (now known as the Media Literacy Project): the ability to critically consume and create
media. Media literate individuals are better able to decipher the complex messages they receive from television, radio,
newspapers, magazines, books, billboards and signs, packaging and marketing materials, video games, and the Internet.
Media literacy skills can help one understand not only the surface content of media messages but the deeper and often
more important meaning beneath the surface. (https://medialiteracyproject.org/learn/media-literacy/)
Office of National Drug Control Policy: “To a) recognize how media messages influence us (e.g. develop a vocabulary to
recognize manipulative techniques, develop skills to protect oneself against messages about drugs or negative lifestyle
choice that are embedded in the media), to b) develop critical thinking (e.g. know that messages are constructed by
people with points of view and commercial interests, uncover value messages inherent in media, evaluate information
for accuracy and reliability), to foster self-esteem (e.g., creatively produce satisfying and constructive messages)”. (Levitt
& Denniston, 2014; https://www.ncjrs.gov/ondcppubs/pdf/strat_pt1.pdf)
2019, 7, 38-66
64
Skills Focused Components
Generic Skills
Skill building (Alliance for a Media Literate America)
Skills necessary for competent participation in communication across various types of electronic audio and visual media”
(Speech Communication Association, 1996, Standard 23)
Skills of Accessing
Ability to access media messages (Media Literacy Project, n.d.)
Ability to access meaning from media messages (Adams & Hamm, 2001; Anderson, 1981; Silverblatt & Eliceiri, 1997)
Skills of Interpretation
Ability to make one’s own interpretations from media messages (Anderson, 1981; Adams & Hamm, 2001; Silverblatt &
Eliceiri, 1997)
Ability to use aesthetic building blocks to create and shape cognitive and affective mental maps (Zettl, 1998)
Ability to analyze media messages (Anderson, 1981; Adams & Hamm, 2001; Brown, 1998)
* Particularly ideological analysis, autobiographical analysis, nonverbal communication analysis, mythic analysis, and
analysis of production techniques (Silverblatt, Ferry, & Finan, 2015)
* Critical thinking about media messages (Adams & Hamm, 2001)
* “a critical-thinking skill that enables audiences to decipher the information that they receive through the channels of
mass communications and empowers them to develop independent judgments about media content” (Silverblatt & Eliceiri,
1997, p. 48).
Skills of Message Production
Ability to communicate effectively by writing well (Brown, 1998)
Ability to produce media messages (Adams & Hamm, 2001; Auferheide, 1997 Hobbs, 1998
Ability to create counter-representations of media messages (Sholle & Denski, 1995)
Knowledge Components
Knowledge of Media Industry
“knowledge about how the mass media function in society. . . Ideally, this knowledge should encompass all aspects of the
workings of the media: their economic foundations, organizational structures, psychological effects, social consequenc-
es, and, above all, their ‘language,’ that is the representational conventions and rhetorical strategies of ads, TV programs,
movies, and other forms of mass media content” (p. 70); “an understanding of the representational conventions through
which the users of media create and share meanings” especially visual representations. (Messaris, 1998, p. 70)
Understanding the process, context, structure, and production values of the media (Silverblatt, 1995)
Knowledge of Media Content
Understanding of media content (understanding of the conduits that hold and send messages), of media grammar (under-
standing of the language or aesthetics of each medium), and of the medium (understanding of the type of setting or en-
vironment) (Meyrowitz, 1998)
Knowledge of Media Effects
Understand the effects of the various types of electronic audio and visual media, including television, radio, the telephone,
the Internet, computers, electronic conferencing, and film, on media consumers.” (Speech Communication Association,
Table 2. Definitions for Media Literacy: Components and Domains. (back to pg. 3)
W. James Potter & Chan L. Thai
www.rcommunicationr.org
Reviewing Media Literacy Intervention Studies for Validity
65 2019, 7, 38-66
1996, Standard 22)
Understanding of how the media distort aspects of reality as they manufacture their messages and how symbol systems
mediate our knowledge of the world (Masterman, 1985)
Learning about “text processing within the broad and complex context of a social, cultural, educational, and commercial
textual ecosphere” (Mackey, 2002, p. 8)
Understanding how media messages shape people’s construction of knowledge of the world and the various social, eco-
nomic, and political position they occupy within it” (Alvermann, Moon, & Hagood, 1999, pp. 1 - 2)
Knowledge about One’s Self
Understanding of one’s place in the world (Blanchard & Christ, 1993; Sholle & Denski, 1995)
Behavioral Components
Generic
“a political, social and cultural practice” (Sholle & Denski, 1995, p. 17)
Empowerment
Becoming empowered citizens and consumers (Blanchard & Christ, 1993; McLaren, Hammer, Sholle, & Reilly, 1995; Sholle
& Denski, 1994)
Moving people from dependency to self direction by being more reflective (Grow, 1990)
“Policing one’s own viewing behaviour – if not by reducing the amount of television they watch, then at least by watching
it in ways which are assumed to minimize its influence” (Buckingham, 1993, p. 21)
Becoming sophisticated citizens rather than sophisticated consumers (Lewis & Jhally, 1998)
Empowering and liberating people to prepare them for democratic citizenship and political awareness” (Masterman, 1985,
p. 15, writing about the Council of Europe Resolution on Education in Media and New Technologies which was ad-
opted by European Ministers of Education).
Activism
Becoming stimulated by social issues that are influenced by the media; these issues are things like violence, materialism,
nutrition, body image, distortion in news reporting, and stereotyping by race, class, gender, and sexual orientation
(Anderson, 1983)
Becoming “active, free participants in the process rather than static, passive, and subservient to the images and values com-
municated in a one-way flow from media sources” (Brown, 1998, p. 47)
Challenging abusive stereotypes and other biased images commonly found in the media (Media Watch)
Social Networking
Creating communities of people who interact in complex social and cultural contexts and use this awareness to decide what
textual positions to accept (Buckingham, 1998)
“primarily something people do; it is an activity, located in the space between thought and text. Literacy does not just reside
in people’s heads as a set of skills to be learned, and it does not just reside on paper, captured as texts to be analysed. Like
all human activity, literacy is essentially social, and it is located in the interaction between people” (Barton & Hamilton,
1998, p. 3; cited in Margaret Mackey, 2002, p. 5-6)
Affective Components
Pay more attention to one’s own affective investment as one consumes the media (Sholle & Denski, 1995)
Ability to appreciate media messages (Adams & Hamm, 2001) especially respected works of literature (Brown, 1998)
66
Copyrights and Repositories
This work is licensed under the Creative Commons Attribution-NonCommercial-3.0 Unported License.
This license allows you to download this work and share it with others as long as you credit the author and the journal. You
cannot use it commercially without the written permission of the author and the journal (Review of Communication Research).
Attribution
You must attribute the work to the author and mention the journal with a full citation, whenever a fragment or the full text
of this paper is being copied, distributed or made accessible publicly by any means.
Commercial use
The licensor permits others to copy, distribute, display, and perform the work for non-commercial purposes only, unless you
get the written permission of the author and the journal.
The above rules are crucial and bound to the general license agreement that you can read at:
http://creativecommons.org/licenses/by-nc/3.0/
Corresponding author
W. James Potter
Department of Communication
4005 Social Sciences & Media Studies
University of California at Santa Barbara 93106-4020
Attached is a list of permanent repositories where you can find the articles published by RCR:
Academia.edu @ http://independent.academia.edu/ReviewofCommunicationResearch
Internet Archive @ http://archive.org (collection “community texts”)
Social Science Open Access Repository, SSOAR @ http://www.ssoar.info/en/home.html
W. James Potter & Chan L. Thai Reviewing Media Literacy Intervention Studies for Validity
Review of Communication Research 2019, 7, 38-66