JVL22(1)_pp.19-28

Journal of Vi$ual Literacy. Spring 2002Volume 22. Number 1. 19-28

The Symbiotics ofSemiotics and Visual Communication

Sandra E. Moriarty, Ph.D.

University of Colorado - BoulderBoulder, Colorado, U.S.A.

Abstract

This article reviewsvisual semiotics, aphilosophical approach thatseeks to interpret messages in terms of their signs and patterns ofsymbolism. Semiotic analysis is particularly relevantfor messagesthat rely on visual communication cues. The theory is that Peirciansemiotics, rather than Saussurian semiology, can provide a centralorganizing perspective for analyzing how the "standsfor" functionin sign systems produces meaning in nonverbal communicationsituations.

Robert Craig's challenge to communication scholars to develop aconsensus on communication theory as a field (Craig, 1999), cansimilarly be seen as a challenge to visual communication scholars.

He makes the argument that there is no unifying theory of communication,nor should there be, but instead, there is a set of assumptions that can unifythe diverse traditions of communication theory. Likewise, visualcommunication has emerged from a diverse set of traditions as, in his words,an "interdisciplinary clearinghouse."

The search for a "field" of visual communication also parallels the workof Umberto Eco who has conducted a similar effort to define the "field" ofsemiotics. In contrast to Craig's premise, Eco's discussion of semioticsbegins with a notion about the importance of having some sense of a centraltheory. He explains, "One cannot do theoretical research without havingthe courage to put forward a theory; and therefore, an elementary model as

20

a guide for subsequent discourse " (1979, p. 7-8) Likewise, in visualcommunication, it would be useful to have some organizing theory.However,like semiotics, the field of visual communication is broad and varied (Moriartyand Kenney, 1995; Moriarty, 1995; Moriarty 1997); and, similar to Craig'sreview of communication theory, it is probably too broad and varied to haveany central or unifying theory.

Eco takes his search a step further in distinguishing between a field (afield of studies, that is, a repertoire of interests that is not as yet completelyunified) and a discipline (an area with its own method and a precise objective).In his analysis of semiotics, Eco says that if semiotics is a field, "then itshould be possible to define semiotics inductively by extrapolating from thefield of studies a series of constant tendencies and therefore a unified model."

On the other hand, "If semiotics is a discipline, then the researcher ought topropose a semiotic model deductively which would serve as a parameter onwhich to base the inclusion or exclusion of the various studies included in thefield of semiotics."

He then suggeststhat we mustkeep in mind the semiotic field as it appearstoday, in all its many and varied forms and in all its disorder. The challengehe sees is to propose a simplified research model, "in order to suggest a unifiedmethod of approach to phenomena which apparently are very different fromeach other, and as yet irreducible." Likewise, Craig says that, "althoughcommunication theory is not yet a coherent field, I believe it can and shouldbecome one." He explains that "A field will emerge to the extent that weincreasingly engage as a communication theorists with socially importantgoals, questions, and controversies that cut across the various disciplinarytraditions.. .."

Both Craig and Eco concur that communicationand semioticsare foundedon sets of theories that need organizing. Likewise, visual communication,with its links to aesthetics and literacy (education), among others, is equallyconfused. Although it is too much to presume that a central, unifying theorymight be developed for visual communication, it might be within the limitsof feasibility to articulate a set of central, organizing theories - the theoriesthat intersect at the point where visual processing departs from verbalprocessing. It is the purpose of this paper to explicate a theory of visualsemiotics as one possible theory that might be included in such a set.

Semiotics: A Theory of SignsSemiotics is the study of signs and signals, sign systems, and sign processes.

Jakobson defines semiotics as "the exchange of any messages whatever andthe system of signs that underlie them." (Sebeok, 1991, p. 60) Fiske adds thenotion of generation of meaning to this definition (1990, p. 42). In other words,

messagesare made of signs and conveyed through sign systems called codes;

Journal o/Visual Literacy, Volume 22, Number 1

meaning is derived only to the degree that the receiver of the messageunderstands the code.

Researchers in semiotics come from varied areas, such as communication,linguistics, anthropology, and marketing, as well as the natural sciences, wheresign systems are studied in such areas as cellular biology and zoology. Thefocus of our concern is on the communication aspects of a sign, and particularlythe communication of nonverbal signs.

What is a Sign? In semiotic theory, a sign is anything that stands forsomething else; a sign stands for an object or concept (Hopes, 1991, p. 141;Eco, 1986, p. 15). The Swiss linguist Ferdinand de Saussure, who is known asthe father of continental semiology, expressed this relationship as between asound or image---called a signifier-and the concept for which it stands-called the signified (Fiske, 1990), p. 43-44). Because of his linguisticbackground, Saussure saw the relationship between these two as arbitrary, inother words, the link between the sign and what it stands for is understood byconvention. This arbitrariness is true in most spoken and written language,however, that may not be so for other types of signs, such as visuals that providecues to stimulate recognition through resemblance, or mimesis, a point wellargued by Paul Messaris in his book Visual Literacy (1994).

C.S. Peirce, who is characterized as the founder of American semiotics,

defines signs as something that stands to somebody for something in somerespect or capacity. That's a tripartite relationship that includes Saussure'ssignifier (the sign) and signified (the object or concept) as well as the Peirce'snotion of an interpretant (the "stands for" element or process). By interpretanthe means the idea contained in the concept as it is decoded, or a subsequentthought to which the sign gives rise (Hoopes, 1991, p. 34).

Because Peirce approaches semiotics as a cognitive philosopher, he is moreinterested in meaning interpretation and modes of cognition, and does not limithis analysis to only arbitrary codes. This is clear in his analysis of three typesof signs: iconic, indexical, and symbolic. Iconic are mimetic representations,that is, they look like what they represent. With an indexical sign, there is aphysical connection - evidence between the signified and the signifier. Forexample, a footprint means someone has walked by and smoke means there isa fire. An indexical sign cues something or is a clue or evidence of something.Only in the third category, the symbolic sign, (a flag, school colors, an abstractlogo or trademark) is the meaning arbitrary.

In other words, Saussure's approach is more useful for analyzing language-based sign systems. Nonverbal systems might be better analyzed using Peirce'sapproach because, besides the arbitrary, it allows for other systems of signifying,such as mimesis and evidence (clues and cues), which are fundamental to visuals.Peirce's semiotics also focuses on the concept of interpretation, which is key tomaking sense of visuals, a point I will elaborate on later in this paper.

Moriarty - The Symbiotics of Semiotics and Visual Communication

---

21

22

What are the Key Issues of the Semiotic Tradition?The biggest problem faced by visual communication scholars is sorting

out the factors and processes that are intrinsic to visual processing andseparating them from language-based processing.

The Problem of Language DominanceOne of the problems in organizing the field of visual communication (if

it can be called a field) is the dominance of verbal language. The languagemetaphor, that is, presuming that visual communication operates in the sameway as verballlingualllexicalcommunication,has led to a subordinatepositionfor visual communication studies because visual communication doesn't

operate in the same way as language. Hence scholars like Barthes (1971)decry "the paradox of a message without a code" and the lack of a meta-language that corresponds to the signifieds. The problem is that scholarsattempt to find parallels to language in the way visuals construct meaning. Inher book, VisualIntelligence, Ann Marie Barry (1997) describes how visualsare processed leading us to the conclusion that the language metaphor maybe inappropriate because the two systems are radically different in their basicoperations.

Privileging language as a primary system through which other systemsare expressed has made it difficult to develop the concepts and, yes, even thelanguage to describe how visual communication works (Moriarty, 1994). AsThomas Sebeok, founder and directorof the Center for Language and SemioticStudies at Indiana University observed in a review of the contributions ofvarious philosophers to semiotics, Bacon, for one, "did not commit the vulgarerror of identifying language with communication." (1991, p. 71)

It is the thesis of this essay that visual communication is equally asimportant as verbal language and that it deserves recognition as an area ofstudy on its own. But this brings us back to the problem stated earlier: thatthere is no recognized discipline-like linguistics and speech communication(rhetoric) in the verbal language area-that focuses on visual communication.There is no field there; no central theories and, therefore, little support fromthe academy and little scholarly firepower.

Symbolic Meaning and PropositionsThe irony is that Messaris, in his book VisualPersuasion (1997), limited

the symbolic function primarily to language and, thus, unfortunately, restrictsthe ability of visuals to function symbolically. He says:

" .. .as far as semantic features are concerned, it is the indexical andiconic properties of visual images that most clearly set them apartfrom language...It is true that some kinds of visual representations(e.g. technical diagrams or maps) are arguably based at least in part

Journal of Visual Literacy, Volume 22, Number 1

on arbitrary conventions, and in that sense, they can be said to entailthe type of semantic relationship that Peirce labeled symbolic.However, symbolic signs are, if anything, even more characteristicof language and other major modes." (p. x)

One would think that art historians who have spent centuries uncoveringthe symbolic meanings in Renaissance paintings would challenge such astatement.

However, there are challenges to this view from his own school. LarryGross and Sol Worth,both scholars in the tradition of the Pennsylvania schoolof visual communication, write of two interpretive strategies for visuals, thesymbolicand thenatural (Gross, 1981, 1985).Inhis article on visual narratives,Gross explains, "symbolic events are those we assume were intended tocommunicate something to us [author's italics]. Further, we assume that theseevents are articulated by their 'author' in accordance with a shared system ofrules of implication and inference."

I believe, however,that Messaris's point about symbolic meaningreflectssomething else, and that is his doubt about the ability of visuals to carrypropositional meaning. This point was made earlier in the work of his mentor,Sol Worth, who proposed that "pictures can't say aint" in an article by thattitle (1982). Messaris states that position very clearly later in the introductionto his book when he says, "what visual communication lacks most cruciallyis a so-called propositional syntax." (p. x)

So the question about the communication of symbolic meaning comesdown to implication and inference and how they relate to propositions. Aproposition is a statement that can be proven or disproven using logical meanssuch as an argument. An inference is the forming of a conclusion frompremises, also an exercise in logic. It should be noted that inferences, and,yes, even propositions are not just language-based concepts. GretchenBarbatsis, for example, has been working to articulate the propositionalmeaning of form in video and film and is developing a body of work todemonstrate how propositions and inferences can be "stated" visually. AndAnn Marie Barry in her book VisualIntelligence, (1997, p. 333) explains,

"how the simplejuxtaposition of one image next to anothercan affectour thinking and feeling in radical ways, and how in rapid successionimages can ultimately form convincing arguments which, whenreceived and believed, can change the values, attitudes and lifestylesof critical masses, and even topple governments."

An example of a propositional visual is an illustration that shows a beeapproaching a photo of flowers. Even in black and white, the meaning of theproposition is still clear: the photo (in this case a photocopy) is so realisticthat it can fool a bee.


- --

23

24

Which Concepts or Metaphors have been Most Productive?Inapplying semiotics to visual communication,one concept is particularly

important, and that is the process of interpretation. The "stands for" elementin Peirce's definition of a sign refers to the way meaning is created throughencoding by the source and decoding by the receiver (reader, in semioticanalysis). That is the model as it is phrased in communication theory,however,visual communication involves other types of meaning beyond those in theintended/perceived schema. I pointed out in the review of the definitions ofsigns, that the definitions are based on different notions of how the processof signification works. To summarize:

Arbitrary signification: works by conventionMimetic signification: works by iconic representationEvidential signification: works by cues and cluesSignaling: works by recognition (hard wired or convention)

Peirce's most important contributionis in the area of interpretation,whichrests on the notion of "a dynamic view of signification as a process" (Deely,1990, p. 23). Since he was more centrally concerned with thinking, ratherthan with communicating, his primary contribution to cognition-and onlyincidentally to communication-was his proposition that "all thinking is theinferential interpretation of signs" (Hoopes, 1991, p. 11). In other words, athought is a product of the signification process, an idea that provides thelink between cognition and communication. He explained that the meaning-making process-finding the signified-is an infinite process of interpretationusing all of the signification techniques. Furthermore, that "to interpret"means to define a relationship in the something-stands-for-something-elseconstruct (Eco, 1986, p. 2, 44). So we have at the base ofPeircian semioticsa thinking process based on inference that results in interpretation.

This process of interpretation involves the consideration of the meaningof a sign in terms of all possible signification possibilities - a sign can be asignal, or an iconic, indexical, or symbolic sign, all at the same time.Interpretation is done by puzzling out the inferences from these multiple levelsof signification. And this type of search for meaning is infinite, in Peirce'sview, because every signifier can be translated into other signifiers andinterpretants through an endless process of inference chaining.

The complexity of this type of inferential interpretation is illustrated inan ad for a Jaguar car that uses the car shot against a background of a highschool building to stand for achievement. In order to make sense of an adusing a visual like this, you have to recognize the car and the fact that itsymboizes accomplishment, and the high school building and the fact that itis cueing a class reunion. Furthermore, through this complex process ofsymbolization you understand that the ad is also intended to engage all theemotional and aspirational context associated with a reunion situation. This

Journal of Visual Literacy, Volume 22, Number 1

----

25

is an example of how the inference chaining process works in visualcommunication.

The reason Peirce's concept of interpretant is so important to us in visualcommunication is that visual meaning is more open for interpretation than isverbal. Contrary to conventional wisdom which suggests that if you can see,you can understand (i.e. the meaning is transparent), the interpretation ofmany visual messages may be more complex and more demanding of thedecoder because of the inferential dimension on which visual interpretationrests.

Visualinterpretation,however,involvesmorethan simple inference.Fromsemiotics we know that what is missing is sometimes as important as what isthere. Eco (1979) suggested that a viewer goes through a process of "syntheticinference" which involves both denotative (realism, representation) andconnotative (associations, attitudes, emotions) processes. Association, inparticular, unlocks this chaining process. In this complex inferential processwhere information is being actively synthesized, an involved audienceextendsand fills in meaning, as well as decodes the meaning (Fry, 1983).

Which Epistemologies are Most Useful?Peircian semiotics is a theory of knowing, rather than a theory of

languaging. This philosophical grounding leads to a theory of visualcommunication based on how we come to know things, rather than how wetransmit knowledge. It can best be explained through the understanding of aconcept Peirce referred to as abduction.

In trying to better understand the complex inferential process used ininterpreting visuals, scholars have tried to articulate the logic behind thisprocess using Peirce's theory of abduction (Moriarty, 1996; Buchler, 1955;Peirce, 1931-35; Hoopes, 1991). In contrast to inductive (reasoning to) anddeductive (reasoning from) logic, abduction is an inferential process thatfashions conjectures based on "clues" that are available or conditions thatare known. As Neiva says in his discussion of semiotics, rather than mimesisor conventional meanings, "the logic of signs is the logic of possibilities"(1999).

In order to accumulate clues, the abductive process begins withobservation, the bits and bytes of perception. It's similar to the way a doctoraccumulates symptoms until he or she arrives at a diagnosis. Peirce describedthe formationof an abductivehypothesisas an "act of insight," the idea coming"like a flash"-the proverbial lightbulb. In a moreformal statement,abductivereasoning assembles the observations and attributesa variety of characteristicsor conditions to a subject (the conjecture process) until a match is made anda conclusion can be stated. Another metaphor for abductive thinking is thesemiotician as a detective, which Eco and Sebeok (1983) presented in theirbook, The Sign afThree.


26

Beyond the linear forms of deductive and inductive logic, abductivereasoning more closely resembles massive parallel processing, one that isnot at all like language processing. The information is gathered, assimilatedall at once, and matched to previous patterns of knowing, which is the wayvisual recognition works. This opens up the question of whether, in responseto Messaris, visual propositions are not impossible, but rather different, fromlanguage-based propositions, based as they are on intuition, rather thaninduction or deduction.

Which Research Questions are Most Relevant?Because of its ability to analyze visual meaning-making as a different

process than verbal meaning making, the visual semiotics approach ishypothesized to be a useful tool to free visual communication from thelimitations of the language metaphor. Hence, additional research and theorydevelopment continues to be needed in isolating those visual factors andprocesses that truly are distinct from verbal language processing.

In particular, the issues that remain surround the role and function ofvisual symbolism and the way propositions and inferences actually work invisual interpretation. Messaris (1994), in his earlier work on visual literacy,identified a need for an understanding of a different set of interpretive skillsthat are more intuitive than conventional. Scholars in visual communicationcontinue to move toward that goal.

To that end, it also would be useful to better understand the differencesin the meaning-making processes of the four types of signification as theyare embedded in visual processing. As referred to earlier, these arehypothesized to include the following:

Arbitrary signification: works by conventionMimetic signification: works by iconic representationEvidential signification: works by cues and cluesSignaling: works by recognition (hard wired or convention)

ConclusionHow is meaning produced, conveyed, and interpreted in messages that

are primarily visual? This question is particularly relevant when the messageis one that relies almost exclusivelyon visual communicationcues. The reasonthere has been little workin this area is largelybecause of a lack of a structuredapproach, or theory, to use in analyzing such processes.

If we agree there is a need for an organizing theory and if we agree thatthere is a field of visual communication developing that is deserving of sucha theory, then this paper proposes that semiotics provides a useful theoreticalfoundation to apply to visual communication because it helps unlock the

Mt\\~l~iitiMM Vi~"Mit\t~~r~tMilU\.


--

27

This analysis relates semiotics theory to visual communication in an effortto better understand how visual communication works. Table 1 summarizes

the key issues raised in this paper.

Table 1The Symbiotics of Semiotics and Visual Communication

References

Barbatsis, G. (1998, June 24-28). Analyzing meaning in form: Domesticcomedy's angle of refraction. Paper presented at the VisualCommunication Conference, Winter Park, CO.

Barry, A. M. (1997). Visual intelligence. Albany: State University of NewYork Press.

Buchler, J. (Ed.). (1955). Philosophical writings of Peirce.NewYork: Dover.Craig, R.T. (May, 1999). Communication Theory as a Field, Communication

Theory 9:2, pp. 119-161.Deely, J. (1990). Basics of semiotics. Bloomington: Indiana University Press.Eco, U. (1979).A theory of semiotics. Bloomington: Indiana University Press.Eco, U. (1986). Semiotics and the philosophy of language. Bloomington:

Indiana University Press.Eco, U. & Sebeok,T.A. (Eds.) (1983).The sign of three.Bloomington: Indiana

University Press.Fiske, J. (1990). Introduction to communication, (2nd. ed.). London:

Routledge.Fry, D. L. & Fry, V.H. (1983). A semiotic model for the study of mass

communication. In M. McLaughlin (Ed.), Communication Yearbook 9(443-462). Beverly Hills: Sage.


Semiotics A theory of signs (visual, as well as verbal)and how they are interpreted

Key issues The problem of language dominance; thechallenge of symbolization and propositions

Most productive concept Interpretation and the role of the interpretant

Most useful epistemologies The philosophy of knowing-abduction andinference-rother than tmnsmitting

Research questions that remain How interpretation-and abduction-work;The differences in the four types of signification

28

Gross, L. (1985). Life vs. art: The interpretation of visual narratives. _Studiesin Visual Communication, 4, 2-11.

Gross, L. (1981). Symbolic strategies. InL. Gross (Ed.), Sol Worth:Studyingvisual communication (pp. 134-147). Philadelphia: University ofPennsylvania.

Hoopes, J. (1991). Peirce on signs. Chapel Hill: The Unviersity of NorthCarolina Press.

McKibbon, B. (1993). The age of missing information. New York: Dutton.Messaris, P. (1997). Visual persuasion: The role of images in advertising.

Thousand Oaks, CA: Sage.Messaris, P. (1994). Visual literacy: Image, mind & reality. Boulder, CO:

Westview Press.

Moriarty, S. (1997). A conceptual map of visual communication. Journal ofVisual Literacy, 17, 9-24.

Moriarty, S. (1996). Abduction: A theory of visual interpretation.Communication Theory, 6, 167-187.

Moriarty, S. (1995). Visual communication theory: A searchfor roots. Paperpresented at the Visual Communication conference, Flagstaff, AZ.

Moriarty, S. & Kenney, K. (1995). Visual communication: A taxonomy andbibliography. Journal of Visual Literacy, 15, 7-156.

Moriarty, S. (1994). Visual communication as a primary system. Journal ofVisual Literacy, 14, 11-21.

Neiva, E. (1999). Redefining the image: Mimesis, convention, and semiotics.Communicatoin Theory 9,75-91.

Peirce, C. S. (1931-1935). Collected papers, Vols. 1-6. (Harshorne & Weiss,Eds.). Cambridge, MA: Harvard University Press.

Sebeok, T.A. (1991). A sign isjust a sign. Bloomington: Indiana UniversityPress.

Worth, S. (1982). Pictures can't say 'ain't.' InS. Thomas (Ed.), Film/culture:Explorations of cinema in its social context (pp. 97-109). Metuchen, NJ:Scarecrow Press.
