-
How Software Designers Interact with Sketchesat the
Whiteboard
Nicolas Mangano, Thomas D. LaToza, Marian Petre, and Andre van
der Hoek,Member, IEEE
AbstractWhiteboard sketches play a crucial role in software
development, helping to support groups of designers in reasoning
about
a software design problem at hand. However, little is known
about these sketches and how they support design in the moment,
particularly in terms of the relationships among sketches,
visual syntactic elements within sketches, and reasoning
activities. To
address this gap, we analyzed 14 hours of design activity by
eight pairs of professional software designers, manually coding
over
4000 events capturing the introduction of visual syntactic
elements into sketches, focus transitions between sketches, and
reasoning
activities. Our findings indicate that sketches serve as a rich
medium for supporting design conversations. Designers often use
general-purpose notations. Designers introduce new syntactic
elements to record aspects of the design, or re-purpose sketches as
the
design develops. Designers constantly shift focus between
sketches, using groups of sketches together that contain
complementary
information. Finally, sketches play an important role in
supporting several types of reasoning activities (mental
simulation, review of
progress, consideration of alternatives). But these activities
often leave no trace and rarely lead to sketch creation. We discuss
the
implications of these and other findings for the practice of
software design at the whiteboard and for the creation of new
electronic
software design sketching tools.
Index TermsInteraction styles, systems analysis and design,
user-centered design
1 INTRODUCTION
SKETCHES play an important role in any design process.They serve
as an extension of a designers own memory[37], help them reason
through complex tasks [54], and sup-port them in picturing and
evolving hypothetical ideas andabstract concepts [17]. Since
sketches are explicit and exter-nalized representations [28], they
assist designers in suchtasks as brainstorming [51], evaluating
ideas [7], and collab-orating with others [8], [58].
Unsurprisingly, sketches play an important role in thesoftware
design process as well. A typical software projectresults in
numerous, diverse sketches [8], [41]. Fig. 1 presentstwo
representative images taken from two software compa-nies: a
whiteboard containing a sketch of a potential solutionto a design
problem (left) and a whiteboard wall containingthe collective
sketches frommany designmeetings (right).
These kinds of whiteboard sketches, and the design pro-cesses
that they support, have seen an increase in interestfrom the
software engineering research community in thepast several years.
Building upon work that looks at soft-ware designers in action more
generally (e.g., [13], [19],[21], [41], [60]), the design work of
software designers at thewhiteboard has been examined from a range
of perspectives,
including idea generation [2], design notations [6], [33],
[42],[56], decision making [59], [60], epistemic uncertainty
[4],and collaboration [32].
While these previous studies make important contribu-tions, our
understanding of how software design sketchesare actually produced,
and used, at the whiteboard is stilllimited. We have a sense of the
kinds of sketches softwaredesigners produce and, at a high level,
the types of activitiesthe sketches support (e.g., understanding
code, idea exter-nalization, design review, comparison of
alternatives). Wedo not know, however, precisely how the sketches
comeabout and evolve, nor how they support designers in navi-gating
their design problems.
To address this gap, we conducted the first study toexamine how
sketches evolve and support reasoning activi-ties on a
moment-to-moment basis. Drawing on data fromdesign sessions from
eight pairs of professional softwaredesigners, we coded 14 hours of
videos for 4238 eventsacross 155 sketches, capturing
moment-to-moment data ofhow designers introduced types of visual
syntactic ele-ments, shifted their focus between sketches, and
usedsketches to support reasoning activities. We then analyzedthis
data to answer several research questions:
Types of sketches. What types of sketches do designerscreate? Do
the sketches created vary in relation tothe approach to design
taken? How syntacticallycomplex are sketches? How do sketches
evolve?
Focus. How long do designers focus on individualsketches? How do
designers make reference tosketches? How do designers shift their
attentionbetween sketches?
Reasoning. How do designers use sketches to under-stand and
advance the state of the design at hand?Which types of reasoning
activities do the sketches
N. Mangano is with Molimur, Mission Viejo, CA 92692.E-mail:
[email protected].
T. D. LaToza and A. van der Hoek are with the Department of
Informatics,Donald Bren School Information and Computer Sciences,
University ofCalifornia, Irvine, CA 92697. E-mail: {tlatoza,
andre}@ ics.uci.edu.
M. Petre is with the Faculty of Mathematics and Computing, The
OpenUniversity, Milton Keynes, MK7 6AA, United Kingdom.E-mail:
[email protected].
Manuscript received 28 Mar. 2014; revised 16 Sept. 2014;
accepted 5 Oct.2014. Date of publication 13 Oct. 2014; date of
current version 4 Feb. 2015.Recommended for acceptance by M.
Dwyer.For information on obtaining reprints of this article, please
send e-mail to:reprints.org, and reference the Digital Object
Identifier below.Digital Object Identifier no.
10.1109/TSE.2014.2362924
IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015 135
0098-5589 2014 IEEE. Personal use is permitted, but
republication/redistribution requires IEEE permission.See
http://www.ieee.org/publications_standards/publications/rights/index.html
for more information.
-
support? Are the outcomes of design discussionsalways recorded
in the sketches supporting thesediscussions?
The result of our analysis is a rich account of the natureof
sketching at the whiteboard, revealing the relationshipbetween
sketching and the design process it supports.While we report on
many results that help shed light onhow software designers work,
several findings stand out.First, we found that, while all
designers created domain,requirements, and systems sketches, the
number of sketchesthey created depended strongly on the approach to
designtaken. Second, we found that designers rapidly shifted
theirfocus over small groups of sketches, with each sketch play-ing
an important and distinct role within the design conver-sation. As
a result, designers rarely worked with only asingle sketch for more
than 30 seconds. Third, we foundthat sketches served an important
role in helping designersdiscuss alternatives, review their
progress, and performmental simulations. Over 80 percent of the
reasoning activi-ties we identified made use of sketches.
The remainder of this paper is organized as follows:Section 2
presents background material on sketching, bothgenerally and as
studied in software development. Section 3introduces our study
design and Section 4 the codingscheme used in our data anlysis.
Section 5 presents ourobservations and results. Sections 6 and 7
discuss ourthreats to validity and findings, respectively, and
section 8provides a brief conclusion.
2 BACKGROUND
Designers at work, with their tendency to sketch, havereceived a
great amount of attention over the years. Earlywork focused on
characterizing the activity of design. Fornon-routine,
ill-structured problems, Simon and Newellcharacterized design as a
process leading to a good enoughsolution to the problem in the
context at hand [37], [48].Some models of design attempted to make
design repeat-able by delineating the solution space in which
alternativesolutions are weighed and explored [49]. Other work
dem-onstrated that software design is a learning activity, andthat
exploring the problem is as important as exploring thesolution
[11]. Later research found that software designersapproach design
using an opportunistic strategy, in whichdesigners work with
designs different levels of abstractionsimultaneously, and dive
selectively into detailed design
when it suits them [21], while showing a relative absence
ofalternatives. Another study found that designers exploresolutions
both vertically, by moving between levels ofabstraction, and
laterally, by utilizing different strategies tosolve the same
problem [17].
Other work has examined the role of sketching in thedesign
process and the types of sketches that designers pro-duce. Ferguson
categorized the sketches that designers pro-duce as thinking,
talking, and communication sketches [15].Suwa found that sketches
serve to help thinking activities byrapidly externalizing thoughts
onto paper [52]. Schondescribed the process of reviewing and
thinking over thesketches that a person has created as having a
reflective con-versation with the material [45]. Suwa et al. found
that suchreflective conversations over the visual details in
onessketch led to unexpected discoveries which stemmed
fromunintended details when drawn [51]. Goldschmidt calledthe
process of reinterpreting sketches beyond their originalintended
depiction as seeing as, and noted that professio-nals aremore adept
at the act of reinterpretation [18].
Software design, in particular, has seen a similar explo-ration
of design and sketching. Several studies have foundthat informal
sketched prototypes receive more feed-back than formal prototypes
[23], [46], [57]. Mynatt demon-strated that office workers can
identify chunks of categorizedcontents in their own writing on the
whiteboard [35]. Dammet al. found that software developers begin
with informalsketches and refine them into more formal notations
[12].Newman et al. observed that website developers switchbetween
levels of abstraction in designingwebsites [38].
Recently, there have been an increasing number of stud-ies
investigating software design. LaToza et al. found thatprofessional
software developers consider whiteboards andpaper the most
effective means of designing [29]. Cherubiniet al. found that
whiteboards frequently serve as themedium of choice for working
through important designdecisions, but also that most drawn
sketches are transientand often discarded [8]. Dekel and Herbsleb
found that theknowledge contained in sketches is highly dependent
onthe designers ability to reconstruct the meaning of sketches,and
that notations such as UML serve as idioms and are notfollowed
strictly [14]. Petre found that designers juxtaposesketches and
deliberately change formalisms to highlightdifferent aspects of a
problem, and use provisionality toallow a dialog with incomplete
ideas [41]. Yatani et al.found that, while there were important
barriers to using
Fig. 1. Two examples of software design sketches on
whiteboards.
136 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
diagrams in a distributed development team, diagrams stillplayed
an important role in ad-hoc meetings, designing,onboarding, project
awareness, and documentation [58].
Recent work examined reasoning and decision-makingqualities used
over sketches. Ball and Christensen demon-strated that mental
simulations allow designers to discussincomplete ideas and reduce
uncertainty [3]. Baker and vander Hoek showed that designers often
examine two ideassimultaneously, and rotate between pairs of ideas
[1].Zannier and Maurer found that designers are more criticalof
designs when evaluating them all at once, and less criticalwhen
evaluating ideas serially [60]. Tang et al. observed thatsoftware
designers often make a decision without justifyingit and move
forward [53]. Christiaans and Almendra notedthat software design is
more complex than product design,because decisions in software must
involve knowledgeof the user interface and structure, and thus
decisions arechained [9].
Using data from three of the eight design sessionsreported in
this paper, attendees of the Studying Profes-sional Software Design
workshop derived a variety of mod-els of the design process [55].
Petre analyzed sketching atthe whiteboard from the perspective of
cognitive dimen-sions of notations and highlighted both the
importance ofgesture in whiteboard sessions and how, as the role of
asketch changed over the course of a design discussion, thetype of
representation evolved [43]. Ossher et al. found thatdesigners used
multiple representations of concerns whichbecame increasingly
detailed and formal through the designconversations [40]. Nakakoji
and Yamamoto found thatdesigners used designing and drawing as
synonyms,used drawings to establish shared understanding, used
avariety of notations, reappropriated sketches for new pur-poses,
redrew sketches, and sometimes did not write downimportant concepts
[36].
Based on these studies of design, many tools have beenproduced
for supporting design. These tools range fromgeneric whiteboard
sketching tools, such as Flatland [34],DENIM [38], Post-Brainstorm
[16], Designers OutPost [26],and Range [25], to tools more closely
tied to softwaredesign, such as SILK [27], Knight [12], SUMLOW
[20],InkKit [10], and Calico [30]. A comprehensive summary
ispresented by Johnson et al. [24]. Based on an extensivereview of
the literature, Moody proposes a set of principlesfor designing
software design notations for cognitive effec-tiveness [33].
Our study builds on this workfocusing specifically onhow
sketching at the whiteboard supports informal soft-ware designby
collecting and analyzing moment-to-moment data on the introduction
of new types of visualsyntactic elements, focus transitions between
sketches, andreasoning activities.
3 METHOD
To investigate software design sketching at the whiteboard,we
conducted an observational study of professional soft-ware
designers. To recruit participants, we approachedalumni of UC
Irvine with a request for access to professionalsoftware designers
who excelled at software design. Thisrequest led to contacts at a
number of different organizations,
which, through discussions and follow-up, resulted inrecruitment
of the eighteen professional software design-ers who participated
in the study. The professionals camefrom seven different
organizations. As design at the white-board often occurs in
collaborative settings involving mul-tiple designers [8], [30],
such as ad-hoc meetings, weorganized participants into pairs. Each
pair belonged tothe same organization (two organizations provided
twopairs of designers).
We administered a uniform design prompt to all pairs,videotaped
each pair at work during their design session,interviewed the
participants afterward, and, once all videoswere collected,
analyzed the whole set of videos. Weexcluded one video from the
analysis due to periodicallyobstructed views and poor audio
quality, yielding a total ofeight pairs that formed the basis of
the results presented here.
The task instructions, coding guide, and raw data fromour study
are publicly available.1
Participants. All those recruited were viewed as expertdesigners
by their peers. Designers expertise ranged acrossa variety of
domains (including health care, documentworkflow, and photo imaging
software) and professionalspecializations (including system
analyst, software architec-ture, UI development, and software
development). Allworked directly with software and reported that
theyemploy whiteboards in their work. Members of each pairwere
familiar with each other, although they did not neces-sarily work
together in their organizations at the time of thestudy. The
professional experience of the participantsranged from 2 years to
26 years. Most designers had a num-ber of years of experience, with
an average of 16.5 years.While we were initially surprised that a
professional withonly 2 years of experience would be viewed as an
expert, asubsequent conversation with his peers confirmed that
hewas. 15 of the participants were male and one was female.Table 1
lists each designers pair, professional experience,and
specialization.
Task. Each pair was provided with a written prompt ask-ing them
to design an educational traffic flow simulationprogram to be used
by a professor in a civil engineeringcourse to teach students
traffic light patterns. The prompt,two pages in length, described a
set of open-ended goalsand requirements, including offering
students the ability to:(1) create visual maps of roads, (2)
specify the behaviorof lights at intersections, (3) simulate
traffic flow, and (4)change parameters of the simulation (e.g.,
traffic density).Pairs were asked to produce a design at the level
of detailnecessary to present to a group of software developerswho
will be tasked with implementing it.
Setting. The study was conducted at participants placesof work,
with a constrained setup that was replicated asclosely as possible
at each site. We asked each pair to workat a whiteboard
approximately 8ft wide by 4ft tall. Partici-pants were asked not to
write on the prompt or other paperin order to capture as much of
their work as possible withthe particular camera setup we used.
All sessions lasted approximately two and a half hours.This time
was divided into four segments:
1. Available online at:
http://sdcl.ics.uci.edu/study-materials-and-data/
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 137
-
1) approximately one hour and fifty minutes for thedesign
activity itself (mean 1 hour and 47 minutes,min 1 hour, max 2 hours
and 4 minutes),
2) a ten minute break,3) ten minutes for a summary, and4) a
twenty minute semi-structured exit interview.At the start of the
first segment, participants were given
the design prompt and asked to work on a design thataddressed
this prompt. After the break, they had tenminutes to briefly recap,
explain, and motivate theirdesign. In the exit interview, we asked
participants to sum-marize how they went about their design, what
their pro-fessional background was, and whether they felt thedesign
process they followed was different from how theynormally proceeded
and, if so, how (see Section 6 for a dis-cussion of these
differences and how they affect the valid-ity of our results).
All sessions were video recorded, with one cameradirected at the
whiteboard and one positioned to capture abroader view of the
participants. Three of the videos areavailable by request.2 Our
analysis focused primarily on thedesign activity portion of the
videos. A total of 14.26 hoursof design at the whiteboard was
studied and dissected.
4 CODING SCHEME
We created three coding schemes focusing on three aspectsof
design at the whiteboard:
1) What types of sketches and visual syntactic elementsdo
software designers use during design at thewhiteboard?
2) How do software designers focus on and transitionamong the
sketches they produce?
3) What reasoning activities do software designers usetheir
sketches to support?
Each coding scheme captured events that happened andidentified
them as actions or activities. Some events are self-contained
actions that happen within a moment (e.g., pro-posing a design
alternative). Others describe activities thatoccur over a period of
time (e.g., a review of progress).Hence activities are delineated
by pairs of events, identify-ing the beginning and end of the
activity. All events arelabeled with information characterizing the
event and withappropriate timestamps.
The coding schemes were developed using an iterative,inductive
approach, comparable to open coding [50]. First,we chose three
sessions that we viewed as representative ofthe eight sessions. We
then examined portions of these ses-sions to determine how to
segment the sketches and activi-ties meaningfully, as well as to
identify the relevantconcepts and categories that emerged. We
operationalizedthis into coding schemes, articulating the coding
criteria indetail. After several iterations, we finalized the
codingschemes. Two authors then independently coded a
fourthsession, yielding a Cohens kappa inter-rater agreement of.7
and percentage agreements of 80.0 percent, 82.4 percent,and 78.6
percent for the sketches, focus fixations and transi-tions, and
reasoning activity categories, respectively. Thisindicates an
acceptable inter-rater reliability in an analysisof 13 percent (1)
of the design sessions. The remaining sevensessions were then coded
by a single author.
Fig. 2 summarizes the coding schemes. In the remainderof this
section, we describe each coding scheme in turn.
4.1 Sketches
Observations of designers have found that, rather thanviewing
whiteboard content as a single entity, people seg-ment content on
the whiteboard into distinct sketches [34].We define a sketch, for
our purposes, as a segment of thewhiteboard containing content:
1) related to a single, central topic (an invariate
[5]),with
2) similar visual features,3) proximity,4) continuity, and5) a
similar initial creation time.Sketches may be primarily textual
(e.g., a list), primarily
pictorial (e.g., a drawing), or contain elements of both (e.g.,a
diagram).
The first coding scheme examined the sketches designerscreated,
identifying events in which sketches were created orerased or new
visual syntactic elements (see below for a pre-cise definition)
were added. To segment whiteboard contentinto sketches, we
performed two passes. In the first pass, weexamined snapshots of
the video at 5-minute intervals,examining if new sketches had been
introduced. Each newsketch that appeared in the snapshots was
numbered fromleft-to-right, top-to-bottom. In the second pass, we
added toand revised our set of sketches for each session by
watchingthe videos, and identifying sketches that shared a
centraltopic in conversation and visual focus by the designers.
Forexample, a sketch consisting of a user interface for
creatingmaps was revised into two different sketches when
weobserved that the designers divided their attention betweenamap
and its input panel (center left of Fig. 3).
TABLE 1Participants
Designer Pair Exp. (years) Professional specialization
1 A 26 UI development2 A 20 Interaction design3 B 15 Software
architecture4 B 11 Software architecture and
UI development5 C 2 Software architecture6 C 24 Software
architecture7 D 21 Project manager, software
architect8 D 26 Software architecture9 E 12 Software developer10
E 10 System analyst11 F 8 Software developer12 F 6 Software
developer13 G 25 Software developer14 G 25 Software developer15 H
23 Software developer16 H 10 Software developer
Pairs C and D belonged to the same organization, as did pairs B
and G.
2. Available online at:
http://www.ics.uci.edu/design-workshop/videos.html
138 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
Sketch type. We labeled each sketch with a type based ontwo
factors:
1) the set of graphical symbols and compositional
rulescharacterizing the sketch, and
2) designers verbal references to the sketch.The first factor
draws heavily on Moodys definition of
visual syntax [33], which he defines as the combination ofvisual
vocabulary (graphical symbols) and visual grammar(compositional
rules). Our usage differs in that our sketchtypes emerged from the
data and from existing sketch typessuch as UML, rather than being
taken from a fixed andabstract set of graphical symbols and
compositional rules. Insituations where the first factor alone left
the type of sketchambiguous, we also considered how the designers
referred tothe sketch. For example, to differentiate sketches with
bothER diagram and class diagram visual syntactic elements,
werelied on how designers themselves referred to the sketch. Insome
situations, sketches evolved in place from one sketchtype to
another. In these cases, we used a combination of key
Fig. 2. An overview of the coding schemes, listing the three
aspects of the design process analyzed and the dimensions (italics)
and possible valuesfor each dimension.
Fig. 3. Whiteboard content was segmented into distinct
sketches.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 139
-
visual syntax elements and how designers referred to thesketch
to determinewhen the sketch type changed.
Sketch domain. Sketches also varied in terms ofwhich aspectof
design eachmodeled (e.g., a list of requirements vs. a list
ofsoftware components). Through our iterative analysis ofsketch
content, we arrived at four domains and used thesedomains to
categorize each sketch. These four domainswere:
1) requirements repeating or extending the requirementsprovided
in the prompt,
2) the architecture or implementation of the softwaresystem,
3) the user interface of the system as the user sees
andinteracts with it, and
4) the problem domain modeling the problem describedin the
prompt.
Visual syntactic element types. We identified each instancein
which a new type of visual syntactic element (VSE) wasfirst added
to a sketch. A visual syntactic element refers toa component of a
sketch that is uniquely identifiedthrough a combination of its
visual elements and itsarrangement in relationship to other
components (i.e.,compositional rules). VSEs included text,
graphical sym-bols (such as boxes or arrows), drawing elements
(such asshading), and drawn objects (such as cars). To
categorizevisual elements of VSEs, we used Bertins six retinal
varia-bles, a set of atomic building blocks for any visual
repre-sentation: shape, size, color, brightness, orientation,
texture [5].For example, a box-and-arrow sketch contains two
VSEsboxes and arrowsdiffering in the shape retinal variable,and a
list title is distinct from a list item because of differ-ences in
arrangement. In coding the types of VSEs, we iter-atively built a
set of VSE types, recording each new VSEtype we identified during
coding. Each VSE type wasrecorded with a picture used to identify
subsequent occur-rences. To reduce the coding required, we did not
recordadditional instances of the use of a VSE within a sketchafter
recording its initial introduction.
4.2 Focus Fixations and Transitions
The second coding scheme examined software designersfocus and
transitions between sketches. At each moment ofthe design sessions,
we tracked each pairs focus point, whichwe coded as a specific
whiteboard sketch, the prompt, ornone. We determined the focus
point based on factors suchas the person speaking, where their gaze
was directed(based on the two recorded camera angles), the location
oftheir pen, and the topic of the active discussion. While
bothdesigners nearly always had the same focus point, in
situa-tions in which they differed, we coded the pairs focus
pointas belonging to the designer who was speaking, or, if nei-ther
was speaking, to the designer who was writing. Weignored all
activity conducted off-camera (which was usu-ally re-reading
portions of the prompt).
In our open coding, we observed that designers oftenreferred to
adjacent sketches in short bursts, but did notmake any changes to
that adjacent sketch. We define amomentary reference as an instance
inwhich a sketch has a con-tinuous focus duration shorter than 3
seconds, during whichno changes were made. We distinguished three
behaviourssignalling designers reference to adjacent sketches:
1) quick glance,2) pointwith a finger, or3) split focus
(pointing and glancing at different
sketches).Intervals with no focus pointwhen the designers
did
not gaze at the whiteboard or prompt for a period of 15 sec-onds
or longerwere identified as inactive. Inactive inter-vals could
contain momentary references; only establishinga new focus point
with a 3 second gaze or edit ended inac-tive intervals. For
example, if the designers spent twominutes talking without using
the whiteboard, pointed at asketch for a brief second, and then
returned to talking, thiswas coded as a single inactive interval
with a momentaryreference to a sketch.
Whenever designers focus shifted between sketches onthe
whiteboard, we noted the origin and destinationsketches, including
their sketch type and domain (focustransitions to or from focus
points other than whiteboardsketches were not labeled). In
addition, we identified andcoded three additional dimensions of the
relationshipbetween the source and destination sketches:
Abstraction. Sketches with an abstraction relationship
rep-resented a concept at differing levels of abstraction, eitherby
providing a more detailed view of the entire sketch (e.g.,a class
diagram with only class names vs. a class diagramwith class names,
methods names, relationships, cardinal-ity) or by providing a more
detailed view of a portion of thesketch (e.g., a list and map,
where the map depicts a con-crete realization of a list
element).
Alternative. Sketches with an alternative relationshipdescribed
different potential solutions. We coded sketchesas alternatives if
the designers explicitly referred to the con-tent in the sketches
as competing solutions and used thesketches to discuss and compare
the solutions at least once.
Point-in-time. Sketches with a point-in-time relationshipwere
visually similar sketches depicting changes over time,phases, or
conditions.
4.3 Reasoning activities
The third coding scheme examined the reasoning
activitiessupported by sketches. We define reasoning activities
asevident cognitive processes (as reflected by speech) with
aspecific purpose or goal. To code reasoning activities,
weprimarily used speech, but we also used gaze and gesturewhen it
helped to confirm or clarify designers intent. Wechose to focus on
coding reasoning activities that:
1) involved lower-level goals over short periods oftime,
2) had well-defined boundaries, and3) could be consistently
distinguished by both coders.Thus, we did not include designing as
a reasoning activ-
ity, as it spanned much of the session and did not
havewell-defined boundaries. This yielded three
reasoningactivities:
Mental simulation. A mental simulation refers to instances
inwhich the designers talk through an execution of a mentalmodel
for the purpose of evaluating a design idea, allowingdesigners to
turn uncertainly into approximate answers byperforming thought
experiments [3]. We build on the defini-tion used by Ball and
Christensen [3] in which the key feature
140 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
is that the talk-through involves a simulation run that
altersthe designers mental representation to a new state. Similar
toOlson et al.s definition of walkthrough [39], we includeinstances
such as simulating the intended user interactingwith the design,
following the movement of data within thesystem, or tracing state
changeswithin the system.
Review of progress. A review of progress includes allmomentswhen
the designers review their design for the pur-pose of taking stock
of what they have done. Our definitionbuilds onwhat Olson et al.
call summary [39], in which thedesigners momentarily take a step
back from the design toconsider the progress they have made, the
goals they haveyet tomeet, or the tasks they have yet to complete
[27].
Suggested design alternative. A suggested design
alternativeincludes instances in which a designer presents a
solutionthat competes with or replaces an existing solution.
Unlikeprior work that examined the structure of design rationalein
detail [39], we did not build a comprehensive designrationale
graph, but rather recorded instances wheredesigners verbally
presented a new design alternative thatcontrasted an existing
design idea. We recorded an eventwhenever an alternative was first
mentioned; when design-ers quickly mentioned alternatives in rapid
succession,these were coded as separate alternatives.
Mental simulation and review of progress were coded asa pair of
events, signifying the beginning and end of theactivity. As we
found that the discussion of distinct alterna-tives often
overlapped and was difficult to distinguish, wetracked only the
moment in which a design alternative wasintroduced, and we coded
these as a single event. For eachinstantaneous event or event
signifying the beginning of anactivity, we noted both the type of
reasoning activity andthe role sketches served in supporting the
activity:
1) No sketches were useddesigners did not gaze at orphysically
reference the whiteboard.
2) Designers gestured over existing sketches withoutdrawing.
3) Designers drew on, annotated, or edited existingsketches.
4) Designers created a new sketch.
5 RESULTS
We first report our results on the sketches that designers
pro-duced.We then report how designers focused on andmovedtheir
attention among sketches. Finally, we report on thereasoning
activities of the designers and how sketchessupported these
activities. Throughout, we report bothquantitiatve data from our
coding scheme and qualitativeobservations explaining this data.
Numbers characterizingthe behavior of designers are reported as an
average acrossall pairs with a standard deviation (e.g., 23 7
percent).
5.1 Sketches
In the following sections, we first examine how designerschoice
of approach to design influenced the types ofsketches they drew. We
then examine the types of sketchesdesigners created and the types
of visual syntactic elementsused within these sketches. Finally, we
examine howdesigners annotated sketches and how sketches
evolvedthrough the design sessions.
5.1.1 Approaches to Design
While most pairs addressed the requirements, system, anduser
interface of the design, pairs tended to focus theirefforts on a
single aspect, using it to drive the design of theother aspects.
This choice of focus led to three approachesto design that we
observed, each of which was reflected inthe time designers spent
with sketches of the correspondingdomains (Fig. 4, Fig. 9):
user interface-driven: pairs A, C, G, and H (user inter-face
sketches: 42 13%),
system-driven: pairs B, D, and F (system sketches: 42 8%),
and
requirements-driven: pair E (requirements sketches:33%).
These patterns were also evident in the quantity ofsketches
designers produced (Table 2). Each pairsapproach to design was
partially influenced by thedesigners job descriptions: the only
pair with a softwareanalyst (E) was the only pair to focus on
requirements,and the only pair with an interaction designer (A)
focusedon user interface design. Pairs with software architectsand
software developers were split between focusing onthe user
interface and the system.
User-interface-driven. User-interface-driven pairs (A, C, G,H)
generally asked themselves: How can users use our sys-tem to
accomplish their goals?, focusing on front-endsketches and
discerning how to get input from the user anddisplay a result.
These pairs relied heavily on user interfacesketches to structure
and organize their work (Fig. 3, Fig. 5a).
Reflecting the traffic simulator problem domain,
theuser-interface-driven pairs made frequent use of maps(30 6% of
session time) to brainstorm user interactionsand support discussion
of the problem. Maps also served asa hub: pairs frequently shifted
attention to a map beforeshifting to a sketch of a different sketch
type.
Lists were the second most common sketch type for
user-interface-driven pairs (19 9% of session time). Lists
helpeddesigners to record potential classes they discovered
whiledesigning the user interface. Lists also served an
importantrole in recording requirements, which the designers
period-ically referenced to verify that they were being met.
Towardthe end of the sessions, three of the UI pairs finally
consid-ered the software architecture in detail, either by
creatingan ER diagram (A and H) or by making lists (C)
describingthe structure of a map class. They stated that they did
thisto satisfy the prompts requirements that they describe the
Fig. 4. The fraction of session time each pair spent examining
the designprompt, working with sketches of each domain, or not
working at thewhiteboard.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 141
-
system in sufficient detail for software developers to
imple-ment it.
System-driven. In contrast to user-interface-driven
pairs,system-driven pairs (B, D, F) focused first on the
systemdesign, designing classes and their interactions. In
general,these pairs asked themselves: How does the systemwork? As
seen in Fig. 5b, these pairs split their timebetween using software
sketches to document and discussthe classes in their system design
(as in the left of Fig. 5b)and using problem sketches to discuss
and understand howtheir system worked (e.g., the map in the center
of Fig. 5b).
All three system-driven pairs created sketches listingnouns and
sketches representing a map with an intersectionsymbol. The left
side of Fig. 5b presents an evolved repre-sentation of two examples
of these sketches, where the listof nouns became the software
architecture alongside theintersection sketch. Beyond this basic
setup, system-drivenpairs proceeded with their own variations. One
pair (B)picked a class from their design, defining it in detail.
Late inthe session, two of the pairs (B, D) drew a second class
dia-gram in order to define classes and their relationships inmuch
greater detail.
While working with system sketches, designers pointedto classes,
marked up elements with annotations andarrows, and explained
classes verbally to their partner. Listswere sometimes used to
document design decisions, suchas explicit instructions on how a
particular componentbehaved (far right of Fig. 5b), or to record
user stories (cen-ter of Fig. 5b). As these designers dug more
deeply into thesystem design, they created sketches allowing them
toexplore the system and problem in more detail, such as linegraphs
and code. These sketches helped them exploreproofs of concept of
their proposed solutions.
Requirements-driven. The requirements-driven pair (E)focused
first on listing requirements, posing the question:What is it that
we need the system to do? They inten-tionally avoided designing the
system through much of the
session, later reporting that we wanted to be clear about
whatwere building, not how were going to build it, only
consider-ing the system and user interface late in the session.
Lists played a central role, helping the pair to directtheir
thinking within the design session, reason aboutparts of the
system, and document design decisions. Afterlooking at the prompt,
they began with a list of require-ments (upper left corner of Fig.
5c), and then broke eachout into separate, more detailed lists of
requirements,maintaining a one-to-one relationship between items
inthe initial list and the title of each new list. They then
laidout a set of use cases (right side of Fig. 5c), creating a
mapto help them generate use cases by enumerating ways inwhich the
user would interact with the map. As they wentthrough the use
cases, they annotated each requirement,identifying how it mapped to
use cases. As they addressedrequirements, they checked off the
corresponding require-ments. At the end of the session, they erased
a large sec-tion of the board to create an ER diagram.
5.1.2 Types of Sketches and Visual Syntactic Elements
Across all sessions, designers used a total of 11 sketch
types(listed in Table 2). Designers often created ad-hoc
sketchtypes reflecting the domain, including the most
frequentsketch type-maps. Designers also created system
diagramswith defined notations (e.g., class diagram, ER
diagram),sketches with ad-hoc notations (e.g., array structure,
GUI),and general-purpose sketches (lists, tables, line
graphs,drawings). Designers created an average of 19.4 (
8.9)sketches per session, each with an average of 4.8 types
ofvisual syntactic elements. Table 3 lists the most frequenttypes
of visual syntactic elements for the most frequenttypes of
sketches.
Despite the traditional focus on software design nota-tions for
describing system structure, most of the sketchesdid not concern
the design of the system. Categorisingsketches by domain, pairs
created an average of 6.5 ( 6.9)
TABLE 2The Sketches Designers Drew
Number of sketches (by pair) Avg.#VSE typesA B C D E F G H
Avg.
(by sketch type)map 3 9 6 3 7 19 5 6 7.3 5.3list 5 3 6 3 15 7 2
5 5.8 4.0table 11 - - - - - 4 1 2.0 3.7GUI 2 1 1 - 2 5 - 2 1.6
4.5ER diagram 1 - - 1 1 - - 3 0.8 6.7class diagram - 2 1 - - - - -
0.4 11.8code - 1 - - - 2 - - 0.4 5.0drawing 1 - - - - - - - 0.1
1.0traffic signal - - - - - - 2 1 0.4 4.8line graph - 1 - - 1 - - -
0.3 4.5array structure - 1 - - - - - - 0.1 3.0
(by domain)problem 3 6 1 3 5 23 5 6 6.6user interface 16 5 6 0 5
3 6 4 5.5system 2 6 5 3 1 9 2 5 4.3requirements 2 1 2 1 15 1 0 3
3.3
Total sketches 23 18 14 7 26 36 13 18 19.4
142 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
problem sketches, 6.4 ( 4.4) user interface sketches, 4.1 (2.6)
system sketches, and 3.6 ( 5.1) requirements sketches.Even when
using system sketches, designers more oftenchose to sketch using
the syntactically simple lists overmore elaborate and complete
notations such as ER diagrams(0.8 1.0 per pair) or class diagrams
(0.4 0.7 per pair). Forexample, designers often simply made lists
of classes andmethods rather than use boxes with a title to denote
classesas in class diagrams.
Reflecting the traffic problem domain, maps were themost common
type of sketch created (7.3 5.4 per pair) andwere used by all
pairs. Designers used maps to reason abouttraffic in two different
ways: understanding how intersec-tions work, and designing user
interfaces. Software-drivenpairs often used problem domain maps to
figure out howcars moved through intersections. They asked
questionssuch as how does a car decide to make a left turn, do
carsinstantaneously move from one intersection to the next, andwhat
component of the system controls a behavior. Toanswer these
questions, designers sometimes went to a verylow level of
abstraction, drawing individual cars in theirdepictions of maps
(Fig. 5b).
In contrast, user-interface-driven pairs instead usedmaps
depicting the user interface. These pairs generally rea-soned about
traffic at the level of traffic flows, consideringscenarios such as
that on Mondays and Wednesdays peoplewent out to lunch, changing
the traffic density. As a result,the maps these designers drew were
often at a higher level(Fig. 3, Fig. 5a).
In supporting these uses, the VSE types designers usedvaried
between road-level views andmap-level views. Road-level views often
used either lines demarcating the edges ofroads (parallel lines33
percent of maps, seven pairs) orintersecting roads (intersection30
percent, five pairs), andrepresented traffic with indicated
individual cars, intersec-tions, and traffic patterns with
rectangles (37 percent, sevenpairs), circles (28 percent, six
pairs), or dotted lines (17 per-cent, five pairs). Map-level views
used lines to represent indi-vidual roads (single line30 percent,
six pairs) or sets ofroads (grid14 percent, five pairs), indicating
flows of trafficwith arrowswith lines (47 percent, seven
pairs).
Lists were the second most common sketch type (5.9 4.1 sketches
per pair) and were used by all pairs. Designersused lists to record
requirements, components, use cases,and component behavior. Most
lists were syntacticallysimple and contained few types of VSEs
(average 3.7 1.6),most commonly first-order elements (92 percent,
eightpairs) and titles (76 percent, eight pairs). Designers
oftendid not expand items in a list: only 36 percent
containedsecond-order elements. Instead, designers created new
lists.Parentheses were sometimes used to express
provisionalinformation, often next to titles or first order
elements.Designers very rarely reordered or rearranged lists.
Mostwere not numerically numbered, and for those that were,the
order served only to make items quickly referenceableor to provide
a count of items. This suggests that designersrarely worked with
lists that encoded meaning in theirorder. Lists often depended
implicitly on other sketches,but designers rarely made these
dependencies explicit; only20 percent contained arrows explicitly
showing relation-ships from elements in the list to other sketches.
Few listsreferenced sketches by name.
Tables were most often used to represent aspects of theuser
interface, typically as a mechanism for inputting data(e.g., Fig.
3). Tables varied in the ways in which designerscreated them. Some
tables were created bottom-up, begin-ning with values in the table,
and only later (if ever) anno-tated with row headers. For example,
when designing trafficintersections, designers wrote down row
values into thetable, and later added the headers for these values.
Other
Fig. 5. Each pair focused their efforts on a specific aspect of
the design.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 143
-
tables were created top-down; designers drew the row head-ers
first and then typically included additional syntactic ele-ments
such as column headers and grid lines.
GUIs were often created after interface maps, helpingdesigners
to determine how the map content might bemanipulated by a user. For
example, designers wouldmimic moving sliders, then imagine how the
map interfacewould update as a result. GUIs most often contained
abounding box (53 percent, five pairs), check box (46 percent,three
pairs), and title (39 percent, three pairs).
Entity-relationship (ER) and class diagrams contained alarger
set of core notations shared by most sketches(Table 3); this likely
reflects ER and class diagrams statusas well-known notations with
conventions on how to drawthem. Relationships between entities were
ubiquitous, anddesigners often explicitly differentiated the type
of relation-ship they denoted. For example, in ER diagrams,
designersused relationships with no head or tail (83 percent,
threepairs), relationships with triangle heads (67 percent,
fourpairs), and solid dots (50 percent, three pairs). Once
made,decisions on relationship type were never changed. Design-ers
also tended to use ER notations (e.g., dots and hollowcircles to
represent containment and referencing), evenwhen working with class
diagrams, which have a different
formal representation defined in the UML (e.g., dashedarrows and
diamond heads).
Designers sometimes explicitly avoided notations theyconsidered
to be too detailed for their purposes. For example,upon noticing
that their partner had created a second-orderitem in an ER diagram,
one designer (E) interrupted, notingthat: we dont have time to go
into that level of detail. Classdiagrams were used by only two
pairs (B, C). To explain therun-time behavior of components,
designers used class dia-grams rather than sequence diagrams. It is
interesting toobserve that this allowed them to explore many
distinct sce-narios instead of just one. However, in the exit
interviews ofthree of the software pairs (B, D, F), they stated
that theywould normally create sequencediagrams as the next
steps.
5.1.3 Annotating Sketches
Designers often annotated their sketches, using visual
syn-tactic elements to guide attention, support
decision-making,record provisional information, or reference other
sketches.As designers moved between sketches, they
sometimesannotated sketches for emphasis or to guide attention.
Forexample, one designer (H) walking through an explanationsaid
especially this and underlined an element. Emphasismarks did not
reflect any observable pattern: designers used
TABLE 3The Most Frequent Sketch Types and Their Most Frequent
VSEs
Each sketch type is listed with an example image (above), the
average number of sketches per pair (n), and the number of pairs
that used the sketch type. Each VSEis listed with the percentage of
sketches that used it and the number of pairs which used it.
144 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
a broad range of VSE types, including an underline, a circle,
astar, a large X, emboldened dots, and vertical bars. Oncemade,
thesemarkswere not subsequently used or referencedbut persisted for
the life of the sketch. Designers sometimesused VSEs to explicitly
refer to other sketches, typically withdeclarative labels
referencing other sketches or with arrows.
Annotations helped to support decision-making. Severalpairs used
check marks on lists, maps, class diagrams and ERdiagrams. One pair
(H) discussed a requirement that involvedmany parts of the system.
To address this issue, they drewquestion marks next to all event
objects, and systematicallyvisited the question marks and placed
either a check mark orlarge X to signify whether the object did or
did not addressa requirement. Another pair (E) checked off
requirements in alist as they verified that theywere addressed.
Designers also annotated sketches to record
provisionalinformation. For example, designers who could not
settleon a name for a list wrote a second name in parenthesesnext
to the official title. This behavior occurred across manypairs (A,
B, C, E, F) and several representation types, espe-cially ER and
class diagrams. Other pairs used slashes inlists to indicate
alternative names.
5.1.4 Evolving Sketches
Designers continually invented new types of visual syntac-tic
elements to capture and externalize concepts they dis-cussed while
designing. Fig. 6 plots the growth of VSEtypes in sketches over
time. For example, one pair (B)wished to determine how users were
interacting with mapsand drew a UI map sketch (right side of Fig.
5b). They firstdrew a box representing the whole area and then drew
linesto represent the user adding streets to the map. They
thenproposed: what if every time two streets intersected
wouldcreate an intersection?, and drew an intersection VSEwhere two
lines met (mark where lines meet in Fig. 3b).Next, they decided
that cars would be generated and enterfrom off the maprepresented
as an arrow VSEand beterminated when they exited the maprepresented
as asolid dot VSE. Overall, the longer designers spent at asketch,
the more VSE types they created (r .62 .36).
As designs progressed and designers had more conceptsto
describe, designers sometimes externalized concepts onthe
whiteboard by borrowing syntactic elements from othernotations. For
example, designers often began sketches as alist, perhaps
reflecting their simplicity. As the discussioncontinued, designers
sometimes borrowed syntactic ele-ments from other types of
sketches. For example, lists thatrepresented software entities
sometimes began to borrowsyntax from class and ER diagrams such as
arrows, cardi-nality, and bounding boxes (Fig. 7a and 7b; Fig. 8a
and 8b).Lists representing requirements documents gained labelson
first order elements or gained arrows to represent depen-dencies
between requirements. As a result, some lists beganto evolve into
sketches of another type (e.g. Figs. 7 and 8).
In some situations, designers chose to evolve theirsketches,
often in situations where aesthetic presentation wasnot a priority.
In other cases, sketches began their transition inplace before
being redrawn (Figs. 7b and 7c; Figs. 8b and 8c).Redrawing sketches
was often triggered by a shift from usinga sketch to help think
through the design to using a sketch todocument the design. In some
cases designers also left the
Fig. 6. The number of VSE types in a sketch as a function of
time. Eachdata point depicts the number of VSE types in an
individual sketch whenit has had focus for the corresponding
time.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 145
-
prior sketch. For example, toward the end of the session,when
designers wished to record more of the design knowl-edge in their
heads, pairs B and H redrew the sketch fromscratch in much greater
detail. As they drew the new sketch,they left the old sketch
intact, allowing them to refer to the oldsketch as they drew the
new one.
Pairs varied greatly in how often they erased existingsketches,
ranging from erasing 0 percent to 81 percent oftheir sketches.
Overall, pairs erased 41 31% of theirsketches. Pairs that created
more sketches erased more, asthey needed the space to create the
additional setkches.
5.2 Focus and Transitions
Throughout the design sessions, pairs shifted their focusamong
sketches on the whiteboard. Fig. 9 depicts the focusperiods of all
sketches. Designers sometimes had a single,core sketch that
dominated their design session. For exam-ple, pair B spent much of
their session interacting with aclass diagram (top red line in Fig.
9b) and pair H interactedextensively with a map (top blue line in
Fig. 9h). But,designers more often worked with groups of
relatedsketches; and even pairs B and H used a wide range of
addi-tional sketches in support of their core sketch.
Designers often rapidly shifted between related sketchesthat
each served a separate purpose in supporting thecurrent design
activity. For example, when enumeratingitems in a list, pairs
shifted their focus to a map to discusshow cars moved, to a table
to understand how different cat-egories of elements in the map
interacted with each other,and finally to a class diagram to record
new aspects of thedesign. Overall, transitions between sketches
were very fre-quent, leading to short focus periods in which pairs
workedwith a single sketch (Figs. 9 and 10). Designers rarely (12
5% of total focus periods) focused for more than 30 secondsat a
single sketch, and they often spent less than 10 secondsat a sketch
(66 9%).
As designers worked with a group of sketches, they oftenshifted
back to a sketch with which they had recentlyworked (Fig. 11). In
some cases, pairs ping-ponged backand forth between a single pair
of sketches (e.g., top twosketches of pairs B and D in Fig. 9).
Average focus periodswere short across all sketch typesthere were
no types ofsketches that designers did not use in close
collaborationwith other sketches (Fig. 9).
To examine the typical size of the groups of sketcheswith which
designers worked, we counted the number ofsketches which received
the designers focus within win-dows of 30 seconds, 1 minute, and 5
minutes (averagedacross all windows in 1 second increments; e.g.,
windows of
0-30, 1-31, etc.). Overall, pairs used an average of 1.5 (
.9)sketches every 30 seconds, 2.0 ( 1.2) sketches every minute,and
3.9 ( 2.0) sketches every 5 minutes (Fig. 12).
Focus transitions were often to sketches of different types(53
17%) or domains (43 16%) and sometimes crossedlevels of abstraction
(27 13%). For example, system-drivenpairs (B, C, E) relied heavily
on problem domain sketches tosupport reasoning about components
within their systemdesign sketches.
All system-driven pairs frequently moved between amap that they
marked up with circles to represent cars tothe equivalent entity in
a system sketch. Often, the design-ers did not write on either
sketch, but instead internalizedthe insights of their discussion.
In other cases, designers didedit sketches. For example, several
pairs (B, C, G) mentallysimulated the movement of cars through
intersections usinga map. They rapidly switched back and forth
between themap and the class diagram to design the data model,
draw-ing relationships between different classes to capture
neces-sary control and data flow. As designers very rarely
drewseparate sketches to represent alternatives or separatepoints
in time (see Sections 5.3.3 and 5.3.1, respectively),designers very
rarely shifted between sketches representingalternatives (1 1%) or
points in time (1 3%).
In 25 percent ( 6%) of the total focus periods, designersquickly
referred to a sketch for less than 3 seconds withoutediting it as a
momentary reference. We categorized eachmomentary reference as a
quick glance, point, or split focus.Quick glances (43 14% of
momentary references) alloweddesigners to gather information or
seek confirmation. Point-ing (37 14% of momentary references)
helped guide theattention of the other designer in the pair to
explain, reviewprogress, or mentally walk through the design. Split
focus(20 10% of momentary references) allowed designers toreason
about how a design might work, using the knowl-edge gained from one
sketch to help identify omissions,mistakes, and inconsistencies.
For example, the designer onthe left in Fig. 13 simultaneously
pointed at a data modeland a map to understand how a car object is
passed betweencomponents as it travels through an intersection.
Mentalsimulations were often mentally intensive, requiring
exten-sive attention. Split focus enabled designers to
externalizethe sketches relevant to the discussion, reducing the
cogni-tive overhead of recalling where sketches were located
andallowing more cognitive resources to be used on the reason-ing
activity at hand. This is consistent with distributed cog-nition
and the concept of computation offloading [44].
5.2.1 Transitions between Levels of Abstraction
Shifting to a sketch at a different level of abstraction
enableddesigners to focus on a particular aspect of the design
by
Fig. 7. Content (a) initially created as a list (b) evolved in
place to a classdiagram, and (c) later was redrawn as a new class
diagram (pair B).
Fig. 8. Content (a) initially created as a list (b) evolved by
borrowing syn-tactic elements, and (c) later was redrawn as a new
ER diagram (pair D).
146 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
Fig. 9. Sketch focus during the design sessions, depicted for
each 10 second interval (focus periods shorter than 10 seconds are
rounded up to 10seconds). The time shown begins when pairs were
first given the design prompt and ends with the end of the design
task. Each horizontal row corre-sponds to a single sketch or
reasoning activity (bottom three rows). Sketches are listed in the
order in which they were created.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 147
-
omitting non-relevant details. For example, six of the
eightpairs shifted at least once from a map of intersecting roadsto
a sketch of a single road, which helped them focus on themechanics
of individual cars while ignoring details of therest of the map.
Designers did not shift between sketches atdifferent levels of
abstraction until their design had becomesufficiently complex.
Designers began shifting betweensketches at different levels of
abstraction, on average, 33 (17) minutes into the session. After
this point, abstractionshifts were frequent, occurring once every
49 ( 25) sec-onds. Overall, 26% ( 13%) of each pairs transitions
shiftedbetween abstraction levels.
The designers frequently shifted to lists in order to dis-cuss
components at a high level of abstraction; lists were thetarget
sketch type in 39% ( 15%) of abstraction transitions.These lists
were often lists of system components or require-ments. All
designers created a list of software classes, allow-ing them to
defer discussion of details until later in thesession and carry on
conversations at a high level of abstrac-tion. As the design
session progressed, some pairs (B, D, F)fleshed out classes with
more detailed designs, oftenexpressed in ER diagrams. But the more
abstract lists con-tinued to play an important role, helping
designers to keeptrack of where they were in the design process by
checkingoff, erasing, and pointing at the lists.
Maps were a frequent target of abstraction transitions aswell
(35 19%), helping to support less abstract discussions.Designers
shifted to map sketches from software and listssketch types in
order to better understand the problemdomain. For example,
designers moved between sketches ofan entire map to a single
intersection, allowing them toexplore traffic configurations at an
intersection without theclutter of a larger map. In turn, focusing
at a lower level of
abstraction led to a large variety of visual syntactic
elements,such as elements for single lanes, many lanes, lights,
trafficconfigurations, and car queues.
5.3 Reasoning Activities
Sketches played an important role in supporting at leastthree
distinct types of reasoning activities: mental simulation,review of
progress, and suggesting design alternatives (Table 4).Pairs
averaged 17.9 ( 8.4) mental simulations, 11.8 ( 4.6)reviews of
progress, and 14.4 ( 7.4) discussions of alterna-tives per session
(Table 5). Pairs varied greatly in the fre-quency of these types of
reasoning activities: mentalsimulation (5 to 30), review of
progress (5 to 17), and discus-sion of alternatives (5 to 27).
Sketches played an importantrole in supporting reasoning
activities: 81% ( 13%) ofreasoning activities used sketches, with
53% ( 8%) of rea-soning activities gesturing over existing sketches
withoutediting, 22% ( 11%) editing an existing sketch, and 6% (5%)
creating a new sketch. In the following sections, weexamine each
reasoning activity in turn.
5.3.1 Mental Simulations
Mental simulations allowed designers to walk through thebehavior
of their design. As in other reasoning activities,mental simulation
was often supported by sketches (83
Fig. 10. Mean focus period length by sketch type (averaged
acrosspairs).
Fig. 11. The number of focus transitions since the sketch last
had focus,as a percentage of total focus transitions (reported by
pair).
Fig. 12. The number of sketches receiving the focus within
windows of30 sec, 1 min, and 5 min.
148 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
17%). Mental simulation sometimes involved edits (30 30%) or a
creating a new sketch specifically to support themental simulation
(9 11%). When designing user interfa-ces, designers sometimes
mimicked using their design bytalking and gesturing over sketches,
using the experience todemonstrate a nonfunctional requirement that
the systemshould satisfy. For example, one designer (A) simulated
auser creating roads by dragging the mouse across the mapand stated
that in her proposed interaction you can createsomething very
quickly without fussing with it. Mentalsimulation helped designers
not only to imagine using theirdesign, but to put themselves in
themindset of the end-user.
When designing with system and problem sketches,mental
simulation enabled designers to expose the gutsof their design in
order for it to be scrutinized. For example,one pair (D) declared
that they were putting [their] footinto the dirty details before
initiating a mental simulation.All system-driven pairs used mental
simulations to explainhow data and control was passed between
software classesor how the state of the model was changed over
time. Men-tal simulation was typically not strictly sequential from
startto finish: designers instead backed up in their
explanations,paused to explain pieces, and abandoned simulations
whenthey became distracted.
Surprisingly, designers almost never supported mentalsimulation
by creating separate sketches representing dif-ferent conditions or
points in time. In the single instancein which this did occur, pair
B reasoned about the systemby drawing two adjacent intersection
sketches labeled Tand T 1. They then simulated the movement of
carsthrough an intersection by shifting between the beforeand after
diagram.
Overall, mental simulations served several purposes:Propose new
design ideas. As designers simulated and
walked through the behavior of aspects of the design with
atemporal component, such as the behavior of end-users orcode, they
sometimes generated design ideas. For instance,while walking
through the end-user experience, designers(A, G, H) created new
interface components on the fly tocontinue the simulation. As
another example, while walkingthrough control or data flow, some
designers (B, D, F) cre-ated the needed class diagram elements in a
similar manner.
Clarify and refine. Mental simulations helped to bringfragmented
assumptions about the system together into acohesive story. For
example, after one designer (G) becamelost in the details of their
design, the other designer clarifiedthe design by mentally
simulating its steps.
Reflect and evaluate. Mental simulations brought outdetails of
the design, allowing them to be evaluated. Forexample, after
stepping through data flow between softwarecomponents, one pair (B)
discovered a potential problem intheir design: well that may end
upwith spaghetti code. . . sowhat about a traverser pattern. In
another case, a designerin pair F changed a design decision and
retreated to a previ-ously-rejected design idea. After performing a
mental simu-lation, he stated: I didnt before, but now I think
[showingindividual cars] is important. When a designer
walkedthrough a simulation, the partner would sometimes jump inand
ask: What about X happening? Mental simulationsoften spurred
partners to propose situations and corner casesthat could lead to
problems in the design.
Reveal implications. Mental simulations helped to
revealimplications of the design that were not immediately
obvi-ous. For example, one pair (D) had determined that carsmust go
through an intersection. But after one designer sim-ulated this
behavior with an explanation of how this occurs,the other designer
asked: But how does the intersectionknow where to send it?,
prompting a new discussion.
5.3.2 Review of Progress
Reviewing progress was a central part of designers activi-ties
at the whiteboard, allowing designers to take stock ofwhere they
were and evaluate the design as it existed. Onaverage, designers
stopped to review progress every 9.3 (2.5) minutes for 49 ( 11)
seconds, and all eight pairsreviewed their progress. When reviewing
progress, design-ers often consulted whiteboard sketches they had
drawn (88 13%). These consultations only occasionally led to
edits(24 19% of instances reviewing progress). Reviews ofprogress
almost always involved reviewing requirementsketches (86 13%).
Designers reviewed lists of
Fig. 13. An example of a split focus between two sketches.
TABLE 4Reasoning Activities
Reasoning Activity Example
Mental simulation So the cop looks at the intersections [and]
says, okay its a green light on these roads. This road has
somenumber of cars in it. If the lights green one tick, [it] means
we pop, and we do that queue and second pic-ture. So from T0 to T1,
thats how weve gone through two.
Review of progress Okay. Maybe we should start talking about the
view. Weve got kind of a rough sketch of the base class ofthe model
and kind of a rough idea of the controller, which is a combination
of this cop with the visitor. . .
Suggestion of designalternative
Or you could say, well the queuelets go back to the model for a
secondso instead of just a straightqueue you could have like a. .
.Well, we have for interior road[s], we have a maximum number of
slots, solets say this is the head.
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 149
-
requirements, which were then cross-referenced with
othersketches to determine if the current design satisfied
therequirements as the designers understood them.
Pairs varied in how they tracked requirements, which inturn
influenced how they reviewed their progress. Somepairs (D, F, G)
did not list requirements. Pairs D and F weresystem-driven pairs
that visited mainly system and problemdomain sketches. Pair D
performed their reviews of prog-ress to check their work over
software components. Pair F,in contrast, performed their reviews of
progress by role-playing the designer and implementer roles; one
designertold the other to: pretend Im the developer, describe
thesystem to me. Pair G, a user-interface-driven pair, usedsketches
primarily to summarize their progress.
Other pairs (A, C, and Hall user-interface-driven
pairs)maintained one or two lists of high-level
requirementsthroughout the design session (e.g., Fig. 5a).
Designersvaried in terms of when the requirements were authored:two
pairs (A, H) created their entire list of requirements ina short
period of time, later using the list to check thatrequirements were
satisfied but making very few edits; theother pair (C) made many
edits and additions to theirrequirements throughout the design
session.
Reviewing progress served several functions:Verify coverage of
requirements. Designers reviewed
requirements one at a time, ensuring that each was satisfiedby
the design by cross-referencing sketches. This sometimesled pairs
to return to deferred issues, spawning new discus-sions to address
them. Designers used check marks andother symbols to check off
completed requirements.
Identify assumptions. Designers reviewed their sketches tocheck
that the design was logically sound, helping touncover assumptions
and portions of the design that wereincomplete. For example, some
pairs realized that carscould travel between multiple
intersections, spawning a dis-cussion to expand the design for
these cases.
Plan what to address next. After reviewing a set ofsketches,
designers sometimes discussed the sketches theyneeded next,
pointing at existing sketches to establish whatthey had missed. For
example, pair B looked over what theyhad created, checked it
against the requirements, remarkedthat they had 30 minutes, and
agreed that they shouldredraw everything to communicate the design
to the devel-opers who would implement it.
5.3.3 Discussion of Alternatives
Designers often discussed alternatives; overall, pairs
dis-cussed an alternative once every 8.7 ( 4.4) minutes.
Discus-sing alternatives included both situations where
alternativeswere considered seriously, and those where straw
manalternatives were used to clarify an existing design idea
with-out any intention of following them through.When unable
tochoose definitively between alternatives, designers often
made a provisional decision, resolving to re-evaluate
thedecision later, as: its more expensive to get stuck on an
idea;[we] need tomove on! (A). For example, six of the eight
pairsconsidered alternative designs for left turn signals.
Severalchose one approach early on or deferred the decision,
revisit-ing the decision later with in-depth discussion when
review-ing progress ormentally simulating the situation.
When considering alternatives, designers often (74 17%)made use
of whiteboard sketches, either by pointing, editing,or creating new
sketches. Discussing alternatives was mostoften supported by simply
pointing or glancing at existingsketches. In 76 percent ( 15%) of
cases, no trace of discussedalternatives remained on the
whiteboard. Only 18 percent (17%) of instances of discussing
alternatives involved editingan existing sketch and only 6 percent
( 5%) involved creat-ing new sketches. Even when creating new
sketches, design-ers rarely made comparisons between them by
switchingbetween alternatives in competing sketches. Instead,
theydiscussed the alternatives. Designers almost never
(twosketches) discussed competing alternatives by explicitly
rep-resenting each alternative in a separate sketch.
While sketches played a crucial role in helping designersto work
through the ramifications of their choices, sketchesoften did not
accurately reflect the current alternative underdiscussion.
Nonetheless, designers discussed them as if theyrepresented the
most up-to-date state of the design. Forexample, one pair (C)
discussed user interactions for creatingroads, sketching
alternatives in which roads were repre-sented by either connected
square blocks or by straight lineswith notches. After deciding on
the latter, they subsequentlygestured at the first sketch, speaking
as if it had lines withnotches representing roads. In some cases,
designers evenpointed and gestured at imaginary sketches. For
example,one designer (H) pointed at areas in a blank square
andexplained a design as if therewere sketched content.
6 THREATS TO VALIDITY
There are three major threats to the validity of our results:the
selection of participants, the choice of the design task,and the
context in which the design task occurred.
First, our selection of participants may threaten the
gen-eralizabiliy of the results if our eight pairs are not
represen-tative of the general population of software designers. It
ispossible that the results might differ for designers with
dif-ferent backgrounds or specializations or who are familiarwith
other design approaches such as model-driven designor agile design.
However, we mitigated this issue by recruit-ing participants with
extensive experience in design froma variety of backgrounds. We
recruited 16 designers fromseven different companies, each with its
own cultureand design methodologies. Participants had a median
of17.5 years of experience as professional designers. Ofcourse, as
the designers were experienced, the results mightnot generalize to
less experienced designers.
The second threat to validity is the choice of design
task.Designers approach tasks based on their existing experi-ence,
including their knowledge in the domain. Individualswith extensive
experience in a domain may already knowwhich design alternatives
are optimal or should be avoided,or use their intuition based on
their experience to design
TABLE 5Number of Reasoning Activities (By Pair)
A B C D E F G H Avg
Men. Sim. 30 20 19 14 5 24 8 23 17.9Rev. Prog. 17 7 8 5 15 12 13
17 11.8Disc. Alt. 22 12 17 5 7 12 13 27 14.4
150 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
more efficiently. In order to place designers on an evenplaying
field, we chose a design task in a domain that mostparticipants
were likely to have encountered in their livesbut were unlikely to
have worked on professionally. Thismay have influenced how the
designers worked. However,a previous study which used the same task
[31] comparedthe overall design process to that of Olson et al.
[37], who inturn compared the process to that used in practice,
andfound a great amount of similarity in practices.
The scope of the design task is also a factor. An
overly-specific task might not allow designers sufficient
creativity,artificially constraining the ways in which designers
mightnaturally engage in a design process. Conversely, an
overly-generic task might lead designers to approaches and
solu-tions so divergent as to be incomparable. To mitigate
thisissue, we asked for a specific set of deliverablesa
verbalsummary of the design and an architecture to provide to ateam
of developersbut allowed the designers completefreedom through the
design session in how they arrived atthose deliverables. At the
same time, we recognize thatour task was still relatively small,
compared to real-worlddesign projects.
The third threat to validity is the context in which thedesign
task was performed, including the time available forthe design
session and the environment in which the designwork occurred.
Design took place amongst pairs of design-ers, with equal
responsibilities and knowledge of the task.In practice, design may
involve groups larger than pairsand may sometimes involve designers
with unequal knowl-edge, leading to participant behavior within
sessions that isless balanced. Participants were instructed to
write exclu-sively on the whiteboard rather than use other media
suchas paper; while designers in practice may design primarilyusing
the whiteboard, this may be supplemented by use ofmediums such as
paper to record and make note of thedesign. Participants were
limited to a single two-hour timespan and had no access to other
people or resources. Inpractice, design typically occurs over
longer periods of timeand involves many different stakeholders,
with whom thedesigners may work and from whom they may obtain
feed-back during the design process. Design in practice may
alsooccur during a wide variety of software lifecycle phases,often
forcing designers to consider the existing design andthe
constraints that it imposes on the design task at hand. Incontrast,
our participants approached a new design task.
Despite these potential issues, the designers generallyfound
their experiences during the study to be represen-tative of their
experiences in real world design. Duringthe interview at the
conclusion of the design session,some designers stated that the
design discussions withinthe session matched those that they have
in their ownmeetings. This sort of discussion is very much what
theengineers do . . . within the team working with
projectmanagement and UI design cross-functionally. However,a few
designers also reported that they adjusted the waythey approached
the design problem due to the limitedamount of time. For example,
one designer stated: itwas, I think, predictable and expected that
we spend a lotof time on early discussion and progressively get
quickerat making decisions and moving along and puttingtogether the
data model at the end.
7 DISCUSSION
Our results reveal that sketches play a crucial role in
sup-porting design conversations. In turn, variations in
theseconversations, both from designers approach to the
designproblem and the evolution of their conversations them-selves,
influence the types of sketches designers draw andthe visual
syntactic elements within these sketches. Wefound that design
conversations span small groups ofsketches, with each sketch
playing a distinct role in support-ing the design conversation.
In the remainder of this section, we first list our mainfindings
and then discuss these findings and their implica-tions for tool
design. Where our findings corroborate otherreported studies, we
provide a citation.
7.1 Main Findings
7.1.1 Types of Sketches
What types of sketches do designers create? Designers
createlists, tables, GUIs, ER diagrams, class diagrams, code,
draw-ings, and domain specific sketches [8], [55]. Designers
prefergeneral-purpose sketches that support the discussion ofmany
scenarios over sketches specialized for an individualscenario.
While designers sometimes use ER and class dia-grams to design the
system, domain sketches are crucial tosystem design [40].
Do the sketches created vary with respect to the approach
todesign taken? Designers may approach a design problemfrom the
perspective of the user interface, the system, orrequirements,
leading them to work with different types ofsketches. Nevertheless,
all designers made use of a commoncore of sketch types, as their
design process moved beyondtheir initial focus to other aspects of
their design.
How syntactically complex are sketches? Overall, the aver-age
number of distinct visual syntactic elements in sketchesranged from
4.0 for lists to 11.8 for class diagrams. Design-ers very rarely
reordered or rearranged lists, suggestingthat order was either
unimportant or was clear withoutexploration or external
representation.
How do sketches evolve? Sketches increase in syntacticcomplexity
as design conversations progress and designersintroduce new
syntactic elements to record aspects of thedesign [39]. Designers
sometimes borrowed syntactic ele-ments from other notations [14],
[40]. As lists accumulatedadditional syntactic elements, they
sometimes evolved intoclass and ER diagrams [14], [39].
7.1.2 Focus and Transitions
How do designers shift their attention between sketches?
Duringdesign dialogues and periods of review, designers
oftentransfer their focus repeatedly among a small group
ofsketches. These groups of sketches typically provide alter-native
views on some aspect of the design, for examplesketches of
different types or domains, sometimes acrosslevels of abstraction
(cf. [40]). Focus transitions most oftento a sketch with which a
designer has worked recently.They are rarely between sketches
representing distinct alter-natives or points in time.
How long do designers focus on individual sketches? Design-ers
move rapidly between sketches. Designers rarely (12
MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT
THEWHITEBOARD 151
-
5% of total focus periods) focused for more than 30 sec-onds on
a single sketch, and often spent less than 10 sec-onds on a sketch
(66 9%).
How do designers make reference to sketches? We observedthat
designers often referred to adjacent sketches in shortbursts, even
when they do not edit or add to them. Design-ers use momentary
references to gather information, guidethe attention of other
designers by pointing, and combineinformation from sketches by
dividing their focus.
7.1.3 Reasoning Activities
How do designers use sketches to understand and advance thestate
of the design at hand? Sketches play an important role insupporting
reasoning activities: 81 percent of reasoningactivities used
sketches. Sketches most often support rea-soning activities simply
by providing information andsomething to gesture at. Only 22
percent of reasoning activi-ties resulted in an edit to a sketch,
and only 6 percentresulted in the creation of a new sketch.
Which types of reasoning activities do the sketches
support?Sketches support mental simulations, helping designers
topropose new ideas, clarify and refine, and reveal implcia-tions.
Sketches support designers in reviewing theirprogress, helping
designers to check the coverage of require-ments, identify
assumptions, and plan what to address next.Finally, sketches
support the discussion of alternatives, mostoften through the use
of existing sketches rather thanthrough editing (18 percent) or
creating sketches (5 percent).
Are the outcomes of design discussions always recorded in
thesketches supporting these discussions? Sketches did not
repre-sent the entirety of the design or even, for the
partsdescribed, its current state. Designers decided on
solutionswithout writing them down and reused sketches fromrejected
alternatives to support reasoning about differentdesign
alternatives.
7.2 Sketches Support Design Conversations
Designers made constant use of sketches throughout theirdesign
process, using the whiteboard to discuss designalternatives,
tracing paths across the whiteboard to mentallysimulate the
behavior of the systems they designed, andexamining sketches to
review their progress. Yet, whilesketches were central to designers
work, they did not repre-sent the entirety of the design or even,
for the partsdescribed, the current state of the design. Instead,
thedesign existed only in the minds of the designers and in
theconversations in which they engaged. Designers decided
onalternatives without writing them down, referred tosketches as if
they were up to date, and gestured in space atimaginary sketches.
As the design evolved through design-ers conversations, sketches
helped support these conversa-tions, even when the evolution of the
sketches did not keeppace with the evolution of the design.
Designers often worked with sketches that were general-purpose,
enabling them to quickly simulate a wide range ofscenarios by
pointing to and annotating a general-purposesketch, rather than
drawing each scenario in a separatesketch. Much as a schematic of a
car design might support adiscussion of a maintenance issue, or a
timeline might sup-port a discussion of scheduling, general-purpose
sketches at
a higher level of abstraction supported the designers in div-ing
into the discussion of many alternatives and manyissues. In
contrast to other normative and descriptiveaccounts of design at
the whiteboard which emphasize theuse of separate sketches of
alternatives to compare, weightradeoffs, and synthesize [7], [58],
designers almost neverexplored alternatives by creating separate
sketches for eachalternative. Rather, they pointed at existing
sketches, evenpretending that they matched the design that existed
onlyin their minds. One potential explanation comes from thecontext
of our study: designers performed early explorationof a design.
This kind of design task may allow designers tobe more creative and
less constrained.
Designers preference for general-purpose sketches alsoshaped the
types of sketches designers used. Rather thansketching individual
scenarios separately using sequencediagrams, designers used class
and ER diagrams whichafforded the flexibility to point at and
rapidly flesh out mul-tiple scenarios within a single sketch. Even
when simulatingthe behavior of a design over time, designers
evidentlyfound it easier to point at and reason about elements
withina class diagram than to repeatedly draw sequence
diagrams.Similarly, designers working with user interfaces built
gen-eral-purpose mockups, simulating scenarios through point-ing
and annotating rather than sketching each state in astoryboard.
This preference may reflect the demands ofearly, informal design in
which many designs are consid-ered in quick succession; designers
did, for example, reportthat they would typically create sequence
diagrams later inthe design process. And designers did choose to
externalizemore in situations in which the design conversation
becamecomplex. For example, in designing a class as part of alarger
class diagram, one pair (B) provided more detail bymaking a
list.
Implications. These findings have important implicationsfor the
design of both fully interactive whiteboard systemsand capture
technologies such as electronic pens for physi-cal whiteboards.
First and foremost, such tools must dealwith the discrepancy
between design as it exists in design-ers minds and as designers
record it on a whiteboard: infor-mal sketches provide support for
discussing a design, ratherthan a record of the design. This has
fundemental implica-tions for tools intended to capture an accurate
record of adesign. Rather than simply record a static view of a
sketch,finding ways to record the design process itself,
includingthe reasoning activities enabled by sketching, is
cruciallyimportant, as the design does not exist in the sketches
but inthe conversations and interactions they enable. For
example,to capture a sketch used in a mental simulation, it may
bemore helpful to record the strokes designers draw, momentby
moment, and designers corresponding discussion,allowing designers
to later play back a video capturing theentire walkthrough of an
aspect of a system that took placein the mental simulation.
Second, tools automatically capturing sketches for long-term
usagesuch as in design documents, wikis, and issuetrackersmust take
into account that a single sketch may beused to discuss a range of
topics. Given sufficient context, itmay be possible for the
original designer to identify the ratio-nale of a decision from a
sketch. But, as sketches are used formultiple purposes, it may be
insufficient to simply record a
152 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2,
FEBRUARY 2015
-
sketch in one place. Thus, it again may be more effective
forsuch tools to consider recording not simply sketches but
thereasoning activities that occur with sketches, including
thescenarios and alternatives annotated on top of sketches. Andeven
out-of-date sketches may serve an important role incapturing the
discussion of newer scenarios.
7.3 The Approach to Design Influences theSketches Designers
Draw
While all designers were provided with the same materialsand
were given the same prompt, three distinct approachesto design
emerged, which in turn influenced the sketchesdesigners drew. Some
designers chose to focus on thebehavior of the system, leading them
to create systemsketches. Others focused on the user interface,
creatingsketches describing user interactions. Others still
designedfrom requirements, creating list sketches to work
towardtheir designs. Yet all designers made use of a common coreof
sketch types, as their design process moved beyond theirinitial
focus to other aspects of their design.
Previous work has identified the types of sketches pro-duced
during design activities [22], [47], [56]. Our findingsextend these
results, identifying the frequency of individualvisual syntactic
elements within several types of sketches.
Designers only infrequently made use of traditional nota-tions
for software design; of the 155 sketches designers cre-ated, nine
of them used a traditional notation, either a classor ER diagram.
Instead, much of the design process tookplace in lists, used
domain-specific sketch types to modelthe problem domain, or worked
with user interface mock-ups. Even when designing the system,
designers often didso while working with domain sketches, using
them towork through scenarios and generate issues that the
designshould address. Design sketches are simply not limite