PREPRINT VERSION Disinforma*on as Collabora*ve Work: Surfacing the Par*cipatory Nature of Strategic Informa*on Opera*ons KATE STARBIRD, Human Centered Design & Engineering, University of Washington AHMER ARIF, Human Centered Design & Engineering, University of Washington TOM WILSON, Human Centered Design & Engineering, University of Washington In this paper, we argue that strategic information operations (e.g. disinformation, political propaganda, and other forms of online manipulation) are a critical concern for CSCW researchers, and that the CSCW community can provide vital insight into understanding how these operations function—by examining them as collaborative “work” within online crowds. First, we provide needed definitions and a framework for conceptualizing strategic information operations, highlighting related literatures and noting historical context. Next, we examine three case studies of online information operations using a sociotechnical lens that draws on CSCW theories and methods to account for the mutual shaping of technology, social structure, and human action. Through this lens, we contribute a more nuanced understanding of these operations (beyond “bots” and “trolls”) and highlight a persistent challenge for researchers, platform designers, and policy makers—distinguishing between orchestrated, explicitly coordinated, information operations and the emergent, organic behaviors of an online crowd. CCS Concepts: • Human-centered computing → Collaborative and social computing • Social and professional topics → Computing / technology policy KEYWORDS Social media; Information Operations; Disinformation; Media Manipulation Future ACM Reference: Kate Starbird, Ahmer Arif, Tom Wilson. (2019). Disinformation as Collaborative Work: Surfacing the Participatory Nature of Strategic Information Operations. PACMHCI. Vol: CSCW, Article xx. 1 INTRODUCTION In recent years, social media have been strategically leveraged by numerous and diverse actors for political gain—from the ongoing harassment of government-critical media outlets in the Philippines [59], to the manipulation of democratic processes in Britain [14] and the U.S. [58] in 2016, to the “coordinated inauthentic behavior” feeding recent tensions between India and Pakistan [29]. We’ve all been exposed to numerous terms that attempt to describe these growing problems (e.g. fake news, digital pollution [27], information disorder [100]. information war [68]) and give them historical context (active measures [9], disinformation [40], asymmetric warfare [91]). Here, we use the term strategic information operations to encompass efforts by individuals and groups, including state and This work is supported by the National Science Foundation (grants 1715078 and 1749815) and the Office of Naval Research (grants N000141712980 and N000141812012). Authors’ addresses: [email protected], [email protected], [email protected] Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). © 2019 Copyright held by the owner/author(s). 0730-0301...$15.00 https://doi.org/xxxxxxx
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
PREPRINT VERSION
Disinforma*on as Collabora*ve Work: Surfacing the Par*cipatory Nature of Strategic Informa*on Opera*ons
KATE STARBIRD, HumanCenteredDesign&Engineering,UniversityofWashingtonAHMER ARIF, HumanCenteredDesign&Engineering,UniversityofWashington TOM WILSON, HumanCenteredDesign&Engineering,UniversityofWashington
Inthispaper,wearguethatstrategicinformationoperations(e.g.disinformation,politicalpropaganda,andotherformsofonlinemanipulation)areacriticalconcernforCSCWresearchers,andthattheCSCWcommunitycanprovidevitalinsightintounderstandinghowtheseoperationsfunction—byexaminingthem as collaborative “work” within online crowds. First, we provide needed definitions and aframeworkforconceptualizing strategic informationoperations,highlighting related literaturesandnotinghistoricalcontext.Next,weexaminethreecasestudiesofonlineinformationoperationsusingasociotechnical lensthatdrawsonCSCWtheoriesandmethodstoaccountforthemutualshapingoftechnology, social structure, and human action. Through this lens, we contribute a more nuancedunderstandingoftheseoperations(beyond“bots”and“trolls”)andhighlightapersistentchallengeforresearchers,platformdesigners, andpolicymakers—distinguishingbetweenorchestrated, explicitlycoordinated,informationoperationsandtheemergent,organicbehaviorsofanonlinecrowd. CCSConcepts:•Human-centeredcomputing→Collaborativeandsocialcomputing•Socialandprofessionaltopics→Computing/technologypolicy
KEYWORDS Socialmedia;InformationOperations;Disinformation;MediaManipulationFutureACMReference:KateStarbird,Ahmer Arif, Tom Wilson. (2019). DisinformationasCollaborativeWork:SurfacingtheParticipatoryNatureofStrategicInformationOperations. PACMHCI. Vol:CSCW, Article xx.
1 INTRODUCTION In recentyears, socialmediahavebeen strategically leveragedbynumerousanddiverseactors forpoliticalgain—fromtheongoingharassmentofgovernment-criticalmediaoutlets inthePhilippines[59], to themanipulation of democratic processes in Britain [14] and theU.S. [58] in 2016, to the“coordinatedinauthenticbehavior”feedingrecenttensionsbetweenIndiaandPakistan[29].We’veallbeenexposedtonumerousterms thatattempt todescribe thesegrowingproblems (e.g. fakenews,digital pollution [27], information disorder [100]. information war [68]) and give them historicalcontext(activemeasures [9], disinformation [40],asymmetricwarfare [91]).Here,weuse the termstrategic informationoperations toencompasseffortsby individualsandgroups,includingstateand
This work is supported by the National Science Foundation (grants 1715078 and 1749815) and the Office of Naval Research (grantsN000141712980andN000141812012).Authors’ addresses: [email protected], [email protected], [email protected]
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
© 2019 Copyright held by the owner/author(s). 0730-0301...$15.00
https://doi.org/xxxxxxx
2
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
non-stateactors,tomanipulatepublicopinionandchangehowpeopleperceiveeventsintheworldbyintentionallyalteringtheinformationenvironment.Theseoperationsareaglobalphenomenon,withpolitical,social,psychological,educational,andcybersecuritydimensions.Indeed,researchersacrossfieldsareconveningtounderstandtheseoperationsandtocraftsolutionstoaddressthem—throughpolicy,education,andtechnologydesign.
In this paper we argue that strategic information operations are a critical concern for CSCWresearchers and that CSCW researchers canmake important contributions to these conversations.Strategicinformationoperations,andinparticulardisinformation,functiontounderminetheintegrityof the informationspaceandreducehumanagencybyoverwhelmingourcapacitytomakesenseofinformation[66,69].Theythereforestrikeat thecoreofourvalues.Andtheyaffectthingswecareabout—forexample,findinglife-savinginformationduringacrisisevent[85,95],organizingonlineforpoliticalchange[87,96,104],andprotectingonlinespacesfrombullyingandharassment[10,97].
Theonlineaspectsofmoderninformationoperationsrendersthemnewlylegibleforexamination—providingopportunitiestostudytheirmotives,tactics,andeffectsthroughthedigitaltracestheyleavebehind.ThefieldofCSCWoffersmethodsandtheoriesthatcanhelpusanalyzeandmakesenseofthesetraces,particularlybyfocusingonthemutualshapingofthetechnologicalplatforms,socialstructures,andhumanbehavior.Here,wepresent three case studiesofstrategic informationoperations, eachexaminedasaparticularconfigurationofcollaborative“work”withinanonlinecrowd.Thisperspectiveencouragesustofocusontheparticipatorynatureoftheseactivities,movingonfromreductiveandexotified accounts of “bots” and “trolls” to more complex conceptualizations that account for howstrategic information operations integrate into existing online communities and leverage thosecommunitiesindynamicwaystoachievetheirgoals.Throughthislens,wecanattempttoaccountforaspectsoftheseoperationsthatareentirely“orchestrated”byasetofpaidactors,aswellasaspectsthataremore“organic”—i.e.emergingthroughimplicitcoordinationamongdiverseactorswhomaynotsharethesamegoalsastheoperatorsorbeawareoftheirroleintheoperation.
Ourpaper isorganizedas follows.First,wecall outanddefineour terms—strategic informationoperationsanditssubtype,disinformation—providinghistoricalcontextforeach.Next,wepresentourframework for examining these operations as collaborativework, followed bya description of ourmethodologicalapproach.Wethenpresentthreerecent,complementarycasestudiesofinformationoperations—Case1:theparticipationofRussiantrollsinpoliticizedTwitterdiscourseleadinguptothe2016U.S.election;Case2:apersistent,multi-platformcampaigntargetingthe“WhiteHelmets”intheSyrianCivilWar;andCase3:sustainedconspiracytheorizingaboutcrisiseventsthatservesasavectorofpoliticaldisinformation—toillustratehowoursociotechnicalsystemsallowinformationoperationstotarget,integratewith,andleveragetheactivitiesofonlinecrowds.Finally,wediscusstheimplicationsof this intersection (between improvised citizen-based activity and orchestrated efforts) for thoseseekingtoaddressthespreadofproblematicallyinaccurateinformationinonlinesettings.
2 BACKGROUND
2.1 Strategic Information Operations Online Information operations is a term social media companies like Facebook have adopted to describeorganizedcommunicativeactivitiesthatattempttocirculateproblematicallyinaccurateordeceptiveinformation on their platforms [101]. These activities can be called “strategic” in reference toHabermas’s theory of communicative rationality [33],which distinguishes between understanding-orientedcommunicativeactivitiesandstrategiconesthatareorientedtowardsotherpersonsfromapurpose-driven, utilitarian point of view (e.g. the calculative manipulation of others). InformationoperationsasatermoriginateswithintheU.S.intelligencecommunity,whereitreferstoactionsof“softwarfare”thataimtodegradethedecision-makingcapabilitiesofageopoliticaladversarybytargetinghumanperceptionandcognitionratherthanphysicalinfrastructures[44].Certainlythistermremainsunsatisfying, for its abstract unspecificity, and for its connection to militaristic metaphors like“informationwar”thatrisklegitimizingnationalistornativistsentiments.Italsohighlightsparticularactorsandmotivesoverothers,invitingustooverlook—orflattenthroughsimplisticequivalencies—
Disinformation as Collaborative Work 3 thecomplexityandambiguityinherenttomanyothercloselyrelatedactivities,suchasadvertisingandpublic relation campaigns. However, it serves well enough as the uncertain terrain for the task ofbroadeningconsiderationof theworkinvolvedinthesemanipulationefforts,stillmostlythoughtassomethingdonetohumancrowdsratherthansomethinghumancrowdsdo.
The material affordances of social media platforms make them robust infrastructures forinformation operations [11, 12]. The tools and practices that haveevolved to support thework ofadvertisers, suchas effective impressionmanagementor segmentingandmicro-targetingparticularaudiences[67],canalsobe leveragedbygovernmentandpoliticaloperativestaskedwithspreadingcertainnarratives.Socialmedia’scrowd-andalgorithm-driveninformationflows,andtheeasysharingofcontentacrosslargeaudiences,canalsofacilitateinformationoperationsbygeneratingemergentandself-reinforcingeffects.Finally,socialmediafeedscombinedifferenttypesofcontentsuchasnewsandentertainment, collapsing different contexts together [55]. This makes it difficult to discern theintentions behind any given piece ofmedia content, especially given the challenges of establishinginformationprovenanceintheseenvironments.Informationoperationscanexploitthisambiguitybyblendingmisleading rhetoric with accurate and inaccurate content as well as inaccurate sourcinginformation(e.g.byusingfabricatedidentities).Jackhasnotedthatbyclaimingthattheyare“settingtherecordstraight”orconfrontingthestatusquo,informationoperationscandisavowanyintenttomisleadandframetheiragendaaseducationaloremancipatoryinnature[40].Journalistsandsocialscientists can find themselves particularly disadvantaged in these circumstances—unless there isverifiableproofofintenttodeceive,thesegroupsriskreputational,professionalandlegalrepercussionswheninvestigatingormakingclaimsaboutinformationoperations.Meanwhile,politicaloperativescanexploit these professionals’ cautiousness by using plausible deniability as a defense and spreadmisleadingcontentwithoutfacingmajorthreatstotheirowncredibility[40].
2.2 Disinformation Operations Someacademics[21,52,59]andjournalists[69]havetheorizedthatnoteveryinformationoperationnecessarilytriestofostersupportforaparticularmessageoridea.Someoperationsinsteadfocusoncausingadistractionorcreatinguncertaintyinwaysthat“killthepossibilityofdebateandareality-basedpolitics”[69].InherLexiconofLies,Jackcatalogsseveralsuchapproaches[40].Forinstance,thefloodingofconversationalspacesonsocialmediawithpositivemessagesorattemptstochangethesubjecthasemergedtobeamisdirectionaltacticcalledxuanchuanthatisassociatedwiththe“Fifty-CentArmy”inChina[34,47].Withinthispaperwescopeourconsiderationtodisinformation,inpartbecauseourcasestudiesprominentlyfeaturedisinformationasaformofinformationoperations.
Disinformationcanbedefinedasinformationthatisdeliberatelyfalseormisleading[20].However,italsohasamorespecificmeaning,stemmingfromtheetymologicallyrelateddezinformatsiya,atermwith roots in Soviet intelligence operations. Bittman, a former practitioner of disinformation whodefected to theU.S. in 1968 [74] and later became anacademic researcher and teacher, describesdisinformationasoneformofSovietactivemeasures—offensiveinstrumentsofforeignpolicythatseekto extend influence and advance geopolitical goals by distorting the information environment andchanginghowpeopleperceivetheworld.Activemeasuresareakintoantitheticalpublicrelations—theyseektoundermine,delegitimize,anddenigrateapublicimageratherthanworkingtoimproveit[9].
Intheearly20thcenturyLeninanticipatedthatWesternfreedoms(e.g.ofspeechorthepress)couldbeexploitedforthepurposesofsubversionandthespreadofpropaganda[70]andheintegratedthese“informal penetration techniques” into policy [9]. But active measures as we describe them hereemerged in the early 1960s when the role of the Soviet intelligence services shifted from passiveintelligencegatheringtotheactivedisseminationofsubversivemessages[8].Theestablishmentofadedicatedapparatustosupportthisactiverolemarkeda“newerainwhichdistortionconciselyandpurposefullytaintsthenaturalflowofinformationthroughouttheworld”[8].
Sovietactivemeasuresweredesignedtohavedetrimentalconsequencesonspecifictargetsaswellas wider-reaching geopolitical implications such as: undermining political, military or economicstrength;creatingorexacerbatingexistingriftswithinsocieties;discreditingpoliciesorrepresentativesdomestically and/or internationally; and influencing policy decisions. The complexity of theseobjectives dictated that active measures were deployed as part of a long-term strategy: it wasacknowledgedthatasingleactivemeasurewouldhavelittleimpact,butwhendeployedpersistentlyoveraperiodofyearsthecumulativeeffectswouldbesignificantandlong-lasting[9].Itistherefore
4
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
usefultothinkofactivemeasuresgenerally,anddisinformationspecifically,notasaqualityofaspecificpieceofinformation,butasacollectionofinformation-actions—oracampaign.Additionally,evaluatingdisinformationislessaboutthetruthvalueofoneormorepiecesofinformationandmoreabouthowthosepiecesfittogethertoserveaparticularpurpose.Tobeeffective,adisinformationcampaignmustbebasedarounda“rationalcore”ofplausible,verifiableinformationorcommonunderstandingthatcanbereshapedwithdisinformation—forexamplehalf-truths,exaggerations,orlies[9].
Journalists were often targeted as “unwitting agents” [9] in the spread of disinformation—forexample throughanonymous tips thatoffereda “scoop”or alignedwith their existingbeliefs.Theywouldthenunwittinglyintroducethedisinformationintothepresswithoutrealizingthetrueintentionsbehindthetransaction.Itisreportedthatjustasinglesensationalistarticlewithdetailsofaconspiracywasoftenenoughtogenerateinterestfromotheroutlets,facilitatingdisseminationandleadingtoshockamongthepublic—withthehopethatthiswouldpromptactionsuchasaprotest[8].Indeedthereisevidencethatconspiracytheorieshavebeenused,strategically,asavectorforpoliticaldisinformationbyexploitingexistingriftswithinsociety,forexampletheintentionalspreadoftheoriesclaimingthattheCIAorchestratedJFK’sassassination[52,69]andthatU.S.scientistscreatedandspreadAIDSasabiologicalweaponinAfrica[19].Historicalaccountssuggestthatbycraftingamessagethatappealedtoexisting political bias, groups at the extremes of society (e.g. the far left or far right) would beparticularlyamenabletoconspiratorialcontentevenifthesourcewasquestionable[9].
2.3 Contemporary Research of Online (Dis)Information Operations Informationoperations that target socialmediahavebeengrowing in sizeand scopeover thepastdecade.Intheirglobalinventoryoforganizedsocialmediamanipulation,BradshawandHoward[12]report that since 2010, more than half a billion dollars have been spent by political parties andgovernmentstoresearch,develop,andconductoperationsfocusedonmanipulatingpublicopinionoversocialmedia.Theseauthorsalsofoundevidencein2018thattheseoperationsarenowtakingplacein48 countries [12]. Scholars have documented the effects these operations have had on politicalconversationsinarangeofcontexts.InthePhilippines,partiesacrossthepoliticalspectrumhavemadeuse of hierarchized “click armies” that rely on digital workers—and veryminimally on automatedbots—todrownoutdissentingopinions[59].InBrazil,informationoperationsharnessedlargedatasetsofinformationaboutcitizensheldbycorporationsandgovernmentstotargetdifferentaudiencesduringtwoPresidentialcampaigns,onePresidentialimpeachmentcampaign,andtheelectionfortheMayorofRio[4,12].InAmerica,theNationalIntelligenceCouncilhasassertedthatRussiaengagedinsignificanteffortstodisrupt“publicfaithinthedemocraticprocess”[58]duringthe2016presidentialelections.
In these examples, and in existing contemporary research into strategic online informationoperationsmore broadly, the focus is often on explicitly coordinated activity—i.e. those activitiesconductedbyautomated“bots”[e.g.1,22,103]orpaidworkers[e.g.47,59].However,thatperspectiverisks over-simplifying the dynamics of these operations, which often involve actors who are notexplicitlycoordinatedand,insomecases,arenotevenawareof theirrole inthecampaign—e.g.the“unwittingagents”describedbyBittman[9].
2.4 Situating Strategic Information Operations as Collaborative Work Onlineinformationoperationsareparticipatoryinnature.Theirmessagesspreadthrough—andwiththehelpof—online crowdsandother informationproviders.Ourworkviews strategic informationoperationsonlineascollaborativework,aperspectivethatpushesustoexpandourfocusbeyond“bots”and “trolls” to consider of the role of online crowds (unwitting and otherwise) in spreadingdisinformationandpoliticalpropaganda.WeadoptaCSCWperspectivethatprovidesabroadwindowforexaminingtop-down,orchestratedworkaswellasothertypesofcoordination,bothexplicitandimplicit. It allows us to explore and account for both big “W” work that occurs within formalorganizations[45]andlittle“w”workthatemergeswithindistributed,onlinecrowds[e.g.57,64,71,85]—aswellastheintersectionbetweenthetwo.
The fieldofCSCWhasalsoadoptedandevolved sociotechnical theories—e.g. structuration [26],sociomateriality [61], and distributed cognition [39]—to assist in the investigation both of howtechnologyshapesworkwithinformalorganizations[60,61]andofhowonlineenvironmentsfacilitate
Disinformation as Collaborative Work 5 and shape collectivebehavior in the “crowd” [82,86,95].Through this sociotechnical lens,we canattempttoaccountforthemutualshapingoftechnologicalaffordances,socialstructures,andhumanaction. Technological affordances are essentially what the platforms allow us (as users) to do—includingfeaturesoftheinterfacesweuse(e.g.topostor‘like’atweet)aswellasthealgorithmsthatshapewhatweseeofothersandtheircontentandhowothersseeourcontent.Socialstructuresincludeboththenormsorrulesthatguideouractionswithinthesesystems,andtheonlinecommunitiesthattakeshapearoundcertainconversations.Humanagency,i.e.theactionswechoosetotake,areshapedbythesetechnologicalandsocialstructures,butthoseactionsalsofunctiontoshapethesestructuresinturn.Forexample,ournetworksofconnections(our“socialnetworks”)areshapedbythetechnologicalaffordanceswithintheseenvironments(we‘follow’someoneandtheirinformationbecomespartofour‘feed’).Thesenetworksofconnectionssimultaneouslyenableandconstraincertaininteractions(weseethecontentofcertainusersandnotothers)thatfurthershapeouractions.Similarly,thetechnologicalinfrastructurechangestoadapttoourbehaviors,sometimesgradually(throughtheadditionofnewfeatures)andsometimesdynamically(we‘like’anotheruser’spostandthealgorithmadaptstosendusmoreoftheircontent).Furthermore,actionswetakeovertimecontributetothedevelopmentofnorms,guiding the behavior of other users within the environment. This view of the mutual shaping oftechnology,socialstructure,andhumanactionallowsustosurfacesomeofthesecond-andthird-ordereffectsofstrategicinformationoperationsinonlineenvironments.
Through this sociotechnical lens,we can attempt to addressa critical challenge for researchers,platformdesigners,andpolicymakers—distinguishingbetweenorchestrated,explicitlycoordinated,informationoperationsandtheemergent,organicbehaviorsofanonlinecrowd.
3 METHODOLOGICAL APPROACH Perhapsobscuredbythereactionarypresscoverage,onlineplatformslikesocialmediaarelikelyonlyoneelementofmultidimensionalinfluencecampaigns;however,onlineactivitiesleavebehinddigitaltraces thatprovidenewwindowsfor studying theseoperations.Digital tracesare recordsofonlineactivities,leftthroughtheactivitiesthemselves,structuredinawaythatmakesitpossibletostudythem(i.e.tocollect,store,andanalyze),afterthefactandatscale.Thoughthesetracesareoftenincompleteandimperfect,theydomaketheonlinecomponentsofmoderninformationoperationsnewlylegibleforresearchers—allowing us to investigate the motives, tactics, and effects of strategic informationoperationsinmorepublicvenuesandatmorerapidtimescales.Inotherwords,wenolongerhavetorelyupontheaccountsofdefectorsandthegreyliteratureofotherintelligenceoperators;wecanseetheoperationsforourselves.
In investigating these traces, our research expands uponmethodological innovation from crisisinformatics,aninterdisciplinaryfieldwithrootsinCSCW,thatexamineshowpeopleuseinformationand communication technology (ICT) to respond to crisis events such as natural and man-madedisasters[63].Themethodsofcrisisinformaticshaveevolvedtoenabletherapidcollection,storage,andanalysisofdigitaltracedatacreatedthroughthe“massparticipation”withinonlineenvironmentsthatoccursafterdisastersandotherbreakingnewsevents.Asthefieldofcrisisinformaticswastakingshape,Palen andher colleagues argued that ICT-enabled interaction (and the resulting trace data)wouldallowustoexaminehumanreactionstocrisisevents innewways,evenas it facilitatednewresponsesandconfigurationsofresponders[38].ThesesameargumentscanbeappliedtoICT-enabledinformationoperations.
In recent years, we have employed and evolved themethods of crisis informatics [63] first forstudying online rumors and misinformation [2, 53], and then later for studying intentionaldisinformation and information operations [e.g. 3, 84, 102]We employ a grounded, interpretivist,mixed-methodapproachthatdeeplyintegratesqualitativeandquantitativeanalysestoprovideinsightinto human behavior, as mediated by online platforms at scale. This methodological approach isinformedbya sociotechnicalperspective that guidesus toexaminehowmacro-level structuresareshapedbymicro-levelinteractions—andviceversa.
In our work, we repeatedly shift across perspectives, from high-level views (facilitated byvisualizations and descriptive statistics) that help us identify patterns and anomalies, to close-upengagement with the content (qualitatively analyzing tweets, accounts, articles, etc.) to betterunderstandthenatureofthosepatternsandanomalies.Systematiccontentanalysisalsoallowsustogeneratenewhypothesesaboutunderlyingpatterns in the tracedataand conceiveofnewwaysof
6
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
exploring those patterns and testing those hypotheses (e.g. by creating a network graph with aunexplorededgeproperty).Asweiteratebackandforthacrossdifferent“levels”,wegenerate,test,andrefineour interpretiveexplanationsof thenatureandrelationshipsbetween structure, informationdynamics,andhumanaction.
In constructing knowledge from these trace data, our methodology adapts a grounded theoryapproach[15,16]toincludequantitativeandvisualrepresentationsasbothartifactsforinterpretativeanalysis andmethods for stressing and refining our emerging theories. Thesemethods are closelyalignedwithtraceethnography[25]andnetworkethnography[36],whichprovidesimilarlyvaluablelensesforfollowinghumanactivitiesthroughandacrossonlineplatformsandnotinghowtechnologiesandsocialstructures(e.g.norms,socialnetworks)shapeonlinebehavior.
3.1 A Brief Note on Reflexivity Ourapproachacknowledgesthatitisimpossibletoremain“outsideof”ourstudytopicasresearchers.As“humanresearchinstruments”[24]whoarestudyingdisinformation,theanalyseswedevelopinthispaperhavebeenshapedbyourpractical, theoretical,experienced,andinexperiencedlenses.Ratherthantreatingtheseintersubjectiveelementsasareliabilityproblem,wepayanalyticattentiontothemtotryandofferaricherunderstandingofdisinformationasacomplexphenomenon.Todothis,wedrawonqualitativetraditionsforfosteringreflexivityinresearchdesign[7,31,49].Somespecificstepswetaketofosterreflexivityinclude:1)involvingmultipleinvestigatorsfromdifferentbackgroundsand5+countriesinourresearch;2)journalingabout,reflectingon,andholdingdialogsabouthowourworkisshapingandbeingshapedbyourepistemicbeliefs,values,politicalperspectivesandassumptions;and3)brieflyreportingonhowthesethingsmayhavecomeintoplayduringtheresearchprocess.
Crucially, we have observed that our own backgrounds and positions make it difficult toproblematizecertainaspectsofinformationoperations.Forinstance,whentheinformationoperationsin our case studies amplifiedmessages laden with progressive values (shared bymembers of ourresearch team), we found ourselves wrestling with creeping doubt and skepticism about ourinterpretations of these operations as problematic—or as operations at all. We noticed thesedissonancesareintersectional.Forexample,messagesthatcritiqueevidenceusinganti-imperialistoranti-positivistframesgeneratedconsiderabletensionsforthoseofusfromdecolonizedcountriesandwho are invested in the view that knowledge is situated and socially constructed. Surfacing thesetensionshelpedusbegenerativeandintimateinourresearch(becauseitreducesthedistancebetweenusandour“subjects”).Thesetensionssensitizedustohowdisinformationoperationscaneffectivelysowdoubtandcreateconfusion—notjustinimaginaryOthers,butourselves.
4 CASE STUDIES Here,wepresentthreecasestudiesfromourownresearch,eachrelyingonlong-termengagementwiththe specific context—spanninga year ormore of intense study ineach case and incorporating ouroriginalanalyseswithotherresearchers’andjournalists’accountsofthephenomena.Thoughsomeoftheempiricalfindingshavebeencoveredinpriorwork,theaccountspresentedhereprovideadditionalcontextfromcontinuedstudy,newdetailsfromsupplementarydatasources,previouslyunpublishedinsights,andnewanalysesguidedbythesociotechnicalperspectiveoutlinedabove.Thoughsimilarinmanyways,thethreestudiesexaminevastlydifferentkindsofinformationoperations—intermsofhowtheyengagewith,leverage,andshapetheactivitiesoftheonlinecommunitieswithinwhichthey“work”.Thefirstcasedemonstratesahighlyorchestratedcampaign,thesecondshowscultivationofanonlinecommunity,andthethirdrevealshowemergentactivityinanonlinecrowdcanresonatewiththeworkofinformationoperators.Individually,eachcasestudyprovidesadetailedandnuancedlookatthreepersistentoperations,allwithtiestotheRussiangovernment’sinformationapparatus—butalsowithevidenceofothercoordinatedactors.Together,theyprovidecomplementaryperspecticesthatrevealcommonelementsandpatternsof strategic informationoperationsandhighlight the complexityofdisentanglingorganiconlineactivityfromorchestratedcampaigns.
Disinformation as Collaborative Work 7 4.1 Case Study 1: Trolling Operations by the Internet Research Agency Targeting U.S.
Political Discourse (2015-2016) It is now widely recognized that the Internet Research Agency in St. Petersburg (RU-IRA) wasconductingayears-longinformationoperationonsocialmediathatwasleveraged,inpart,toinfluencepoliticalviewsintheUnitedStatesleadinguptothe2016election.Evidencesupportingthisviewhasbeenreportedinacademicresearch[41],investigativejournalism[65],intelligencecommitteereports[58],andbytheplatformsthemselves[92,93,101].
Our lab initially encountered these operations accidentally, while studying other onlinephenomena—in particular, online discourse about the #BlackLivesMattermovement. InNovember2017,wepublishedapaper[87]aboutthathighlypolarizeddiscourseonTwitter,examining“framingcontests”betweenpoliticallyleft-leaning,pro-#BlackLivesMatteraccountsandpoliticallyright-leaning,anti-#BlackLivesMatter accounts. Shortly after publication, the U.S. House of RepresentativesIntelligence Committee released a list of Twitter accounts that had been determined to have beenoperatedbytheRU-IRA[94].Uponfirstseeingthat listofaccounts,werecognizedseveral fromourstudy, including some thatwehad featured inourpaper.After systematically cross-checking thoseaccountsagainstour#BlackLivesMatterdata,wefoundthatRU-IRAaccountswereembeddedin,andinsomecasesquiteinfluentialwithin,both“sides”ofthatpolarizedconversation(seeFigure1).Later,Twitterreleasedafulldatasetofalloftheseaccountsandalloftheirtweets,andwewereabletoseehowtheconversationwehadstudied(#BlackLivesMatter)fitwithinthebroaderoperationsoftheRU-IRA.Asitturnedout,wehadstumbledintoasignificantelementoftheiroperation.
4.1.1 Data and Methods: Thiscasestudyincorporatesfourdifferentstagesofanalysis,eachbasedondifferentdata.Itbeginswithourinitial#BlackLivesMatterstudy[87],relyingondatacollectedfromTwitterrelatedtoshootingeventsin2016.ItthenshiftstofocusspecificallyupontheroleofRU-IRAaccountsinthatsamedataset[3],andthenpullsbacktoexaminetheonlineactivitiesofthoseaccountsmorebroadlythroughqualitativeanalysisbasedonavailabletracedata,accessedthroughtheInternetArchive[90].Finally,itmakesuseofthemorerecentlyreleasedarchiveofRU-IRAoperations[93]toaddimportantcontexttoearlierfindingsbyexaminingtheentirescopeofthoseoperations.Thoughthespecific analysesvariedacross thedifferentpartsof this study, eachwas informedbyourbroadermethodological approach, which deeply integrates qualitative and quantitative (including visual)methodstoprovideagrounded,interpretativeexplanationofthephenomena.
Fig.1.RetweetNetworkGraph:RU-IRAAgentsin#BlackLivesMatterDiscourse.Thegraph(originallypublished[3])showsaccountsactiveinTwitterconversationsabout
#BlackLivesMatterandshootingeventsin2016.Eachnodeisanaccount.Accountsareclosertogetherwhenoneaccountretweetedanotheraccount.Thestructuralgraphshowstwo
distinctcommunities(pro-BlackLivesMatterontheleft;anti-BlackLivesMatterontheright).
8
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
AccountscoloredorangeweredeterminedbyTwittertohavebeenoperatedbyRussia’sInternetResearchAgency.Orangelinesrepresentretweetsofthoseaccount,showinghowtheir
contentechoedacrossthedifferentcommunities.ThegraphshowsIRAagentsactiveinboth“sides”ofthatdiscourse.
4.1.2 Synthesized Findings: RU-IRA Agents Targeted, Infiltrated, and Cultivated Politically Active Communities Online: RU-IRA agents “worked” together through the operation of more than 3000accountsthatpresented themselvesaspeopleandorganizations.Abouthalfwereactive inEnglish-languagediscourse.TheothersprimarilytargetedRussian-speakingaudiences.FocusingontheEnglish-languageaccounts,usingcontentandnetworkanalysis,weidentifiedthreedistincttypesofaccounts:1) “local” news accounts that reposted headlines from actual news in thosemarkets; 2) “hashtagwarrior”accountsthatinitiatedandspreadhumoroushashtagsinagame-likefashiontogainattentionandfollowers;and3)highlypersonalizedaccountsenactingpoliticallyactiveU.S.citizens.Amongthethirdtype,thereweretwodistinctgroups,onewithaccountsthatimpersonatedAfricanAmericansand#BlackLivesMatter activists, and another that impersonated white, politically active conservatives,primarilyfromsouthernU.S.states.
RU-IRA accounts impersonating members of these two politically active groups—the AfricanAmerican“left”andthewhite,conservative“right”—createdaboutone-third(morethan1Mtweets)ofalltheEnglish-languagetweetspostedbyRU-IRAaccounts.Moreimportantly,thisrathersmallsubsetofRU-IRAaccounts(~300)garnered85%ofallEnglish-languageretweets(18.5Mretweets).Inotherwords,accountsimpersonatingactivistswithinthesegroupswereaboutathirdoftheRU-IRA’soverallEnglish-languageoperations,andbyfartheirmostsuccessfulintermsofreceivingtractionintheonlinecrowd.Contentanalysissuggeststhattheseaccountsoperatedtowardsmultipleobjectivesincluding:amplifying political divisions in the U.S.; criticizing and demotivating support for U.S. presidentialcandidateHillaryClintonontheleft;andpromotingcandidateandlaterPresidentTrumpontheright.
Fig.2.Example#BlackLivesMatterrelatedcontentthatwascirculatedbyRU-IRAsocialmediaaccountstodifferentaudiences.
ThereisnowasignificantbodyofevidencedocumentinghowRU-IRAagentstargetedtheAfricanAmericancommunity,inpartbyimpersonating#BlackLivesMatteractivists[3,17,75].Ouranalysesrevealthattheyalsospecificallytargeted,andhadsomesuccessinfiltrating,“grassroots”conservativeactivist communities(seeFigure2 for examples).Wequote “grassroots”here,because someof thecommunitieshavethemselvesbeenshapedbydomesticcampaignstobuildconservativenetworksonTwitter.AnotableexampleisthePatrioticJournalistNetwork(PJNET),agroupofTwitteractiviststhatusecoordinated—andinsomecasesautomated—tweetingpracticestosimulateortriggerviralityforconservativemessages[87].PJNETisanonlineorganizationthathasbeenactivesinceatleast2014.
Disinformation as Collaborative Work 9 The group hasa suite of online tools that encourages on-message tweeting, especially of graphicalmemes. Group members use the #PJNET hashtag in their profiles and tweets, which facilitatesrecruitment into the group and encourages “follow-back” practices among groupmembers. Thesepractices have affected the network structure within the right-leaning community—an effectdocumentedbyStewartetal.[87]in#BlackLivesMatterdiscourseandbySupovitzetal.[89]inastudyononlineactivismaround#CommonCore.TheIRApurposefullytargetedthePJNETcommunity,usingtheir hashtags and cultivating their influencers. The #PJNET hashtag was among the top-30mostretweetedtermsbyRU-IRAaccounts—tweetedmorethan10,000times.Mostofthesewereretweetsof non-IRA accounts who appended #PJNET to their tweets. The RU-IRA repeatedly amplifiedinfluencersinthePJNETcommunity,includingoneaccounttheyretweetedmorethan1140timesandwho,inturn,followedatleastsevendifferentRU-IRAaccounts.
Toinfiltratesuchcommunities,RU-IRAagentscreatedconsistent,persistent,politicalpersonasthatreflected caricatures of U.S. political participants. These personas were formed, in part, throughmimickingtheonlinepresentationsandbehaviorsofrealU.S.citizens.InthefullRU-IRAdata,wecanseeanevolutionof theaccountsastheybecomemoresophisticated—behavingmoreandmore like“real” people online, and specifically converging with the presentation of highly political onlineaccounts.AsociotechnicalperspectivesuggeststhattheactivitiesofRU-IRAaccountswouldalsohaveshapedthesocialstructures—i.e.thesocialnormsandnetworkties—oftheonlinecommunitiestheytargeted,andconsequentlytheactionsofotherusersinthosecommunities.ThishypothesisisreflectedinclaimsthattheRU-IRAcampaignwas,inpart,designedto“sowdiscord”[58,69].Inotherwords,theRU-IRAoperationstargeted,infiltrated,andcultivatedpoliticizedonlinecommunities.
JournalisticaccountsbasedontheaccountofactivistLyudmilaSavchuk,whoworkedundercoverattheInternetResearchAgency’sfacilityinSt.Petersburg[56,78],describethe“big-W”workofRU-IRAagents as taking place within that facility and organized in a top-down fashion, with supervisorsprovidinglistsoftopicsandagentsfocusingtheir“troll”accountsonthosetopicsfortheirshifts.Thoughthisworkwasmanagedbytheformalorganization,individualworkerswereempoweredtocreatetheirownpersonasandimproviseastheycreatedcontentalignedwiththeorganization’sobjectives.Fromthisview,theworkofRU-IRAagentscanbeclassifiedasclearlyorchestrated.
ThoughouranalysesinsomewaysconfirmthisviewoftheRU-IRAoperationsasorchestrated,theyalsorevealaconcertedstrategytointegrateintoorganiconlinecommunities.RU-IRAagentsdidthisina variety ofways—by impersonating activistswithin those online communities, building networkswithin those communities through interactions enabled by the platform (e.g. liking, retweeting,following),andattimesdirectlycontactingother“real”activistsorinfluencerswithinthecommunity.Thereisevidencethat,ina fewrarecases,RU-IRAagentsdirectlycollaboratedwithactivists,e.g. toorganizephysicalprotestsintheU.S.[98].Intheseways,bothindirectlyanddirectly,unwittingactivistsintheU.S.begantocoordinatetheiractionswithRU-IRAagents.
However,itisimportanttostressthattheonlinecommunitiesthemselveswerenotproductsofthoseoperations,butinsteadwerefunctioningasunwittinghosts.Underscoringthatpoint,thefirstRU-IRAtweetsabout#BlackLivesMatterdonotappearuntilabout18monthsafterthemovementbegan.Thissupportsanunderstandingthat,thoughtheyparticipatedintheconversationandmadesomeprogresstowardsintegratingintothatcommunity,theRU-IRAwerenotpartofthefoundationalstagesofthe#BlackLivesMattermovement.Instead,theytargetedanexistingorganicmovementforinfiltrationandcultivation. Though their information sharing and engagement behaviors often aligned with thatmovement’sethos, theirobjectivesweremarkedlydifferent,astheysoughttoguidethecommunitytakingpartinthatmovementtowardspoliticalviewsfavorabletoRussia’slong-termgoals.TheRU-IRAconductedparalleloperationswithintheconservative,pro-Trumponlinecommunity.Onthat“side”oftheconversation,wecanalsoseethemtargetinglessorganicmovements,piggybackingoffofexistingdomesticinfluenceoperations,like#PJNET,whoserootsalsoappeartopredateRU-IRAinfiltration.
4.2 Case Study 2: The Disinformation Campaign Targeting the White Helmets Thissecondcasestudyexaminesstrategicinformationoperationsinthecontextofarmedconflict—theSyrianCivilWar.Inparticular,wefocusonthedisinformationcampaigntargetingtheWhiteHelmets,avolunteerhumanitarian responsegroupthatworks in rebelheldareasofSyria.TheWhiteHelmetsprovide searchand rescueassistanceandmedical aid to peopleaffectedby the conflict—primarilyvictimsofairstrikesperpetratedbytheSyriangovernmentandtheirRussianallies.Thegroupalso
10
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
documentstheirwork,publishingvideosshowingthehumanimpactsof theconflict. In2016, theseeffortshelped togarner internationalattentiontoand sympathy forSyriancitizens sufferingat thehandsoftheirgovernment—attentionthatmighthavemovedtheinternationalcommunitytocallforactionagainstPresidentAssadandhisallies.However,thegroupalsobecamethetargetofamulti-partyonlinecampaign todelegitimize them [50,77].Theywere labelled toolsof foreign influence, calledcriminalsandterrorists,andaccusedofstagingchemicalweaponsattacksandotherevents(seeFigure3foranexample).Toasignificantextent,thiscampaignworked.IfyougoonlineandsearchfortheWhiteHelmets, youwill likelybe overwhelmed by the critical contentmeant to delegitimize them,silencetheirvoices,andjustifytargetingthemforviolence(inviolationofinternationallaw).Thismulti-dimensionaldisinformationcampaignreflectsacomplex,strategicinformationoperationthatincludesbothorchestratedandorganicelements.
4.2.1 Data and Methods: WehavebeenstudyingthisconversationsinceJuneof2017.OurdataincludeanongoingcollectionofTwitterdatarelatedtotheSyrianconflict,scopedtotweetsthatcontainanexplicitreferencetotheWhiteHelmets.Again,ourworkinthiscontextappliesmixedmethods—qualitative,quantitative,andvisual—inagrounded,interpretativeapproach.Weexaminethedataatvarious “levels”, shiftingfrommicro- tomeso- tomacro- scales (andbackagain) tounderstand thecontent (competing narratives), structure (social networks) and dynamics (information flows,interactions)oftheconversation[102].Weconductcontentanalysisoftweetsandaccounts,examinetemporalpatternsofcontentproduction(tolookforthingslikecoordinatedaction),andcreatenetworkgraphs to look at relationships between accounts (to understand the role of social networks andcommunitiesintheproductionofdisinformation).WealsousethelinkswithintweetstolookbeyondTwitteratthesurroundinginformationecosystemsthatcontributetotheWhiteHelmetsdiscourse[84].Most recently,wehavebeenconductinga cross-platformanalysis to lookat the roleofYouTube insupportingthisdiscourse.
Fig.3.Sampletweetamplifyingananti-WhiteHelmetsnarrative.
4.2.2 Synthesized Findings: Cultivating an Online Activist Community; Filling Data Voids with Geopolitical Propaganda: FocusingfirstontheTwitterconversation,onemajorfindingissimplythat
Disinformation as Collaborative Work 11 anti-WhiteHelmetsvoicesoverwhelmanddrownoutpro-WhiteHelmetsvoices.WhenwelookatthesocialstructureoftheWhiteHelmetsdiscourse,wefindtwodistinctcommunitiesofaccounts—onethatproducedcontentinsupportoftheWhiteHelmetsandanotherthatproducedcontentcriticizingtheWhiteHelmets. Interestingly, critical accountsoutnumberedpro-WhiteHelmetsaccounts (byabout40%).Evenmore interestingly, those critical accountsweremore consistently activeandproducedaboutthreetimesasmuchTwittercontentasthepro-WhiteHelmetsaccounts.Thatcriticalcontentreflectedmanyofthenarrativeswedescribedabove—e.g.claimingtheWhiteHelmetsassociatedwithterrorists and accusing them of various crimes. Importantly, the accounts that constituted thecommunityofanti-WhiteHelmetsvoices,especiallythecoreaccountsthatproducedthemajorityofcontentinthatdiscourse,werenotautomated/botaccounts,nordotheyappeartobepredominantlypaid troll accounts. Instead,most seem to be authentic, online “information activists” who devotesignificant personal resources to tweeting in support of this explicitly anti-WhiteHelmets politicalagenda.Manyhavebridgedfromotherrelatedcauses—including“anti-war”activismandsupportofthePalestinian cause. A few prominent accounts in the community appear to be agents of foreigngovernments or other groups (e.g. Hezbollah) who blend into these online activist communities,althoughitisverydifficulttodistinguishbetweenauthenticactivistsandthoseimposters.SomeofthemostinfluentialaccountsturnouttobeWestern“journalists,”whorosetoprominencethroughtheiranti-WhiteHelmets content production (through tweets, blogs, andarticles)—in part due to cross-platform amplification from state-sponsored media outlets (for example, broadcast interviews onRussia’sflagshipmediaoutlet,RT).ThetweetinFigure3wassentfromtheTwitteraccountassociatedwithRT’sUKcurrentaffairsprogram,whichfeaturesaninterviewwithoneoftheprominentjournalistsintheanti-WHconversationanddemonstratesthiscross-platformpromotion.
Within theboundariesofTwitter,wecan seeambient “work”by theactivists, journalists,mediaoutlets,andagentstoamplifyeachother’scontent(throughlikesandretweets),aswellasmoreexplicitcoordinationworkacrossthesediverseactors.Inonecommontactic,anaccountwillcallattentiontoapieceofcontent(e.g.atweetsupportingtheWhiteHelmets)fromothermembersofthegroupbyaddingalistof@mentionstoatweetquotingorlinkingtothatcontent.Here’satemplateofthestructureofoneofthesecall-outtweets:
@activist1: @journalist1 @activist2 @activist3 @journalist2 @activist4 @agent @activist5 Hey, check this out! White Helmets are at it again.
<quoted tweet with a statement supporting the White Helmets or portraying them in a positive light>
Thesemass-mention tweets result in acascadeofreactions, as thementionedaccountsmobilize toamplifytheirgroup’smessage(e.g.retweet@activist1’stweet)andtoexplicitlychallenge(e.g.throughrepliestotheoriginalpost)thepro-WhiteHelmetsclaimand/oritsauthor.Theeffectontheaccountwhosetweetwasquoted/replied-tocanbeexperiencedasaformof“dog-piling”[43].Overtime,theseefforts function both to drown out and drive out (through intimidation and harassment) contentsupportingtheWhiteHelmets.Thoughtheinitiatingaccountneverexplicitlysays,“let’sallgotweetareplychallengingthequotedtweet”,thisactivityfollowsoneofafewsetroutinesthatarerepeatedlyenactedandcanbeeasilydeciphered,copied,andlearnedbynewmembersofthecommunity.Inthisway,the“work”ofthisonlinecommunityislooselycoordinatedthroughasharedsetofpracticesandroutines. Although there are accounts within this community that are affiliated with influenceoperations frommultiple state-actors, there is little evidenceof explicit, top-downorchestrationbythoseactors.However, it is importanttonote that it ispossible andeven likelythatother formsofcoordination are occurringwithin this assemblage of actors that are not visible to researchers, forexamplethroughdirectmessagesonTwitterorothercommunicationplatforms,andactorsthatmayappear tobeauthentic activistsbutwhoare in realitypaidagents.Thishighlights the challengeofunravelingtheorganicfromtheorchestratedthatwewillreturnto.
Looking beyond Twitter to the surroundingmedia ecosystem (using URL links within tweets),revealsasimilarasymmetry inthecontentandstructureof theWhiteHelmets’discourse—i.e.anti-WhiteHelmetsarticlesarecitedfarmoreoftenthanpro-WhiteHelmetsarticles.Throughcloseranalysis(documentedindetailinpreviouswork[84]),wediscoveredatightlyconnectednetworkofwebsitesthat repeatedly share content, via copy-pasted articles (oftenword forword) criticizing theWhiteHelmets (see Figure 4). We termed this content-sharing network a media “echo-system” and its
12
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
structurereflectsseveraldifferentdynamics,includingexplicitcontent-sharingrelationshipsbetweenwebsites,opportunisticappropriationofarticlesoutsidethisnetwork,andtheuseoftoolsthatallowforeasyrepublishingofcontentacrosscertainpublishingplatforms.
Thoughit isdifficult toattributeanyof theseeffectstoexplicitcoordination,patternsofactivitysuggestthatthestructureanddynamicsofthisecho-systemreflectbothorganicpropertiesandtargetedinfluencecampaigns.Amongthemostinfluentialwebsitesinthisecho-systemareasmallnumberofself-described“independent”mediaoutletsandthinktanksthatconsistentlysharemessagesalignedwithRussianand, insomecases, Iraniangovernmentinterests.Other influentialwebsitesaremediaoutletsexplicitlyfundedand/orheavilyinfluencedbytheRussian(RT,Sputnik),Iranian(Farsnews),andSyrian(syrianews.cc)governments.Thesetwotypesofprominentwebsitesproducethemajorityoforiginalanti-WhiteHelmetscontent,whichisthencopied-and-pastedacrossalargernumberofwhatappear to be diverse media outlets with varying ideological orientations—from TruePatriot toActivistPost to VeteransToday to TheFreeThoughtProject. Many of these websites appear to beideologicallyand/orfinanciallymotivated.Itisalsopossiblethatthelistincludesimpostersitesrunbycertain state and even non-state actors, a strategy that was documented in the RU-IRA activitiesdescribedabove.Acloserlookatthisecho-systemrevealsakindofmicro-targetedstrategy,wherethesame anti-White Helmets content is presented within different wrappers (different websites) thatappealtodifferentaudiences.Reflectingsomethingaboutthesharedinfrastructureofdisinformation,thereissignificantoverlapwiththenetworkgraphofdomainssupportingconspiracytheorizingaboutcrisisevents(seeFigures4&6).
Fig.4.DomainNetworkGraph:ContentSharingacrossthe“Alternative”MediaEcho-System.EachnodeisawebsitethathostedanarticlecitedintheWhiteHelmetstweets.Viewisfocusedondomainsthatsharedanti-WhiteHelmetscontent.Nodesareconnectedwhenthesame
articleappearedonbothdomains.Nodesaresizedbythenumberoftweetscitingthatdomain.
Examining these structuresandactivities froma tacticalpointofviewsuggests that informationoperations—inparticulardisinformationcampaigns—usemulti-dimensional,multi-channelmethodstofloodandpolluteinformationspaces.Wecanseethistacticappliedacrosssocialmediaplatformsin
Disinformation as Collaborative Work 13 theanti-WhiteHelmetscampaign.VoicescriticaloftheWhiteHelmetsdominatecontent-productiononseveral socialmediaplatforms, includingTwitter andYouTube,wherea relatively smallnumberofhighlyactiveaccountsconsistentlyproduce,interactwith,andamplifyanti-WhiteHelmetscontent—andmobilizeexplicitlytochallengeanddrownoutpro-WhiteHelmetscontent.Butthetacticsaren’tlimited to socialmedia; they extend to the surroundingmedia ecosystemwhere a relatively smallamountoforiginalcontentisre-sharedandrepurposedacrossalargenumberofwebsites,creatinganecho-effect.Thesetacticscanactto“gamethesystem,”pushingthesetweets,videos,andarticlesupintothetopsearchresultsfor“WhiteHelmets”,essentiallyhijackingtheconversation.Thisactivityfunctionsononeleveltosimplyspreadpreferrednarratives,challenginganddelegitimizingthegroup.Indeed,theanti-WhiteHelmetscampaignseemedtobeeffectiveatoverwhelmingtheWhiteHelmets’effortstodrawgarnersympathyandsolidarityfromWesternaudiencesandatmakingthemtargetsforpoliticalviolenceandretribution.Atanotherlevel,thisactivityworkstopollutethesystemandunderminetrustintheinformationspacemorebroadly.
Ouranalysisrevealstheanti-WhiteHelmetscampaigntobeapersistent,cross-platform,strategicinformationoperationthatwasatleastpartiallyeffectiveintheiraims.Thiscampaignhadsignificantsupport from the Russian disinformation apparatus, including several of their media outlets (RT,Sputnik,IntheNOW).Thereisalsoevidenceofotherstate(Iranian)andnon-state(Hezbollah)actors.However,thiscampaignissignificantlydifferentfromtheRU-IRAoperationsdescribedabove.Thoughtherearesomeorchestratedelements,asignificantportionofthisactivityappearstobeorganic—withconsiderable interplay and resonance between the two. From a sociotechnical perspective, thesegeopoliticalentitiesareintegratedintothesociotechnicalinfrastructureofthismediaecosystem,andtheyshapethesysteminmultipleways,forexamplebyhostingcontentonflagshipmediaoutletsandby pushing content out through ideologically-aligned, “independent” websites. They also activelysupport friendly journalists, publishing their content, amplifying their voices, and enhancing theirprofessionalreputations.Sothiscampaigndoesreflectsomeelementsoforchestration.Buttherearealsomanysincereactorsor“unwittingagents”[9]inthisspace,onlineactivistswhoseviewshavebeenshapedbytheinformationoperation(andwhoseactionsarefurthershapedbythesocialandtechnicalstructureof thesemediaspaces),andwhoactivelysupportthegenerationanddisseminationof theoperation’spreferrednarratives.Inmanycasestheseagentsare“journalists”whosereputationissoclosely tied to their output of disinformation-aligned content that it is difficult to assign a singlemotivationtotheirparticipation.Consideredaspartofaninformationoperation,thisactivityisperhapsbestdescribedascultivatedratherthanorchestrated.
4.3 Case Study 3: The Online Ecosystem Supporting Conspiracy Theorizing about Crisis Events
Thisfinalcasestudyexploreshowconspiracytheoriesofcrisiseventstakeshapeonsocialmediaandwiththesupportofan“alternativemedia”ecosystemthatfeedsandshapesthem.Overandoveragain,andwithincreasingfrequencyinrecentyears,man-madecrisiseventssuchasschoolshootingsandterroristattackshavesparkedonlineconspiracytheorizing(likethetweetinFigure5below)—claimingthattheeventdidnothappeninthewaythatmediaandgovernmentofficialsareportrayingit.
Fig.5.Exampletweetspreadinga“falseflag”narrative.
14
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
Asmassivenumbersofpeopleconvergeonlinetomake senseofa tragic event, a subsetof thatparticipatoryaudiencebeginstobuildandsharetheoriesthattheeventwasa“hoax”stagedby“crisisactors”ora“falseflag”wherethewidelyrecognizedsuspectsarenottherealperpetrators,butarebeingframedbyasecretgroupofpowerfulactors.Onsocialmediaplatforms,wecanseethesetheoriesevolveovertime,astheaudienceassemblesevidencetosupporttheirtheoriesandnegotiatestofindthetheorythat fits the available evidence best. Articles on “alternative news” websites, such as Alex Jones’InfoWars,oftenaddfuelforthetheoriesand“evidence”forthesediscussions.Thoughtheexplanationsaredynamicandadaptedtofiteachparticularevent,thetheoriesareconsistentinthattheybeginwiththeideathatthenarrativebeingpromotedbymainstreammediaandgovernmentofficialsisfalse,andtheyworktofindanother“alternativenarrative”.
Ourresearchteamhasaccidentallybeen studying thisbehavior since2013,whenwe initiatedaresearchprojectfocusedspecificallyonthespreadofonlinerumorsduringcrisisevents.Andinearly2016,webegan to seehow this conspiracy theorizingof crisis eventswas connected topolitically-motivateddisinformation.
4.3.1 Data and Methods: Building from that earlywork, this case study includes insights fromanalyzingnumerous“crisisactors”and“falseflag”rumorsacrossseveraldistincteventsbetween2013and2016,includingtheBostonMarathonBombings,theSanBernardinoshootings,theUmpquaSchoolshootings,andtheParisAttacks.Ourresearchteamidentified,scoped,andinsomecasesdidtweet-by-tweetcodingofthese“rumors”—readingtweetstounderstandhowtherumorstookshape,evolved,andwerechallengedonTwitter.Someoftheseanalysesappearinpublishedwork[e.g.3,53].
Thiscasestudyalsofeaturesamulti-partanalysisofconspiracytheorizingthattookplaceonTwitterintheaftermathofshootingeventsin2016andaninvestigationoftheonlinewebsitesthatsupportedthoseefforts[83].TheseeddataforthisworkwasaTwittercollectionofshooting-relatedterms(e.g.shooting,shooter,gunman,etc.)thatwasactiveforninemonths(JanuarytoOctober)in2016.Usinglinksinthosetweetsandco-sharingpatternscreatedbyusers,wecreateda“domainnetworkgraph”(see Figure 6) that revealed some of the structure of the media ecosystem that surrounded andsupported this conversation.Next,we coded the differentweb domainsaccording towhether theycontainedcontentsupportingtheconspiracytheoriesofthoseshootingevents(Figure6,red),explicitlychallengedthoseconspiracytheories(Figure6,blue),orjustgotpulledintotheconversationforbeinga“straight”newsstoryabouttheevent(Figure6,yellow).Thoughyellowdomainsdidnotexplicitlyshareaconspiracytheoryinourdataset,yellowdomainsthatareintegratedintotheredsectionofthegraphareonesthatconspiracytheoristsoftenturntofortheirnews.Finally,weconductedanin-depth,qualitative content analysis of the different web domains, not limited to content about conspiracytheoriesofcrisisevents,butexaminingtheircontentmorebroadlyaswellastheiraboutpagesandmissionstatements.ThewebdomaincontentanalysiswascompletedinDecember2016.
4.3.2 Synthesized Findings: A View Down the Rabbit Hole at the Disinformation Ecosystem: Analysisof theTwitter data reveals a group ofonline conspiracy theorists,working collectively toproduce,evolve, and amplify their theories with the support of an “alternative” media ecosystem. Theirconspiracy theorizingabout crisis events is not controlledby an outside actor or group. Instead, itemerges“organically”throughtheactivitiesofgroupmemberswhohavecometoshareanepistemologythat views world events as controlled by powerful, sinister actors and “mainstream”media as co-conspiratorsthathelptohidethattruth.Thisgroupdoesnotwaitforanexternaldisinformationagenttogivethemthesignaltostarttheorizing.Instead,eachneweventbecomesinterpretedthroughthesamelens,andbecomesacatalystforanotherroundofconspiracytheorizing.Theparticipantsworktogether to assemble available evidence to support their theories, often adjusting their alternativenarrative to accommodatenewpiecesof information.At times, thegrouparguesandnegotiates todeterminewhichdirectiontheconspiracytheoryshouldtake—e.g.aftertheOrlandoPulsenightclubshootingin2016,severalaccountswentbackandforthdebatingwhethertheeventwasa“hoax”(i.e.itdidn’t really happen) or a “false flag” (i.e. it happened, but not in the way themedia portrayed),eventuallysettlingonthelatternarrativeasabetterfitwithemergingevidence.
Thiscrowdofcrisiseventskepticsappearstobelargelyconstitutedbytruebelievers(thoughthereisevidencethatsomeparticipantshaveothermotivations—bothfinancialandpolitical).Theonlinenatureoftheiractivityallowsforanepistemiccommunitytotakeshapearoundtheseconversations,throughfollowingrelationshipsonTwitter,andinplacesliker/conspiracyonreddit.Italsoallowsfor
Disinformation as Collaborative Work 15 otherpeopleconvergingonlinetomakesenseofacrisiseventtobepassivelyexposedtothisactivity—aswellastheepistemologythatdrivesitandthetechniquesusedtopromoteit.Thus,thisactivitycanacttorecruitothersintothecommunityandteachthemthesocialrulesofparticipatingthere.Studyingthisbehaviorallowsustoseetheintersectionofplatformfeatures,socialandsocio-technicalstructures(networkties,onlinenorms),andhumanbehaviorsthatconstitutethephenomenonofgoing“downtherabbithole”.
Ourstructuralgraphof themediaecosystemsupportingthisconversation(Figure6)revealsthetheorizingtobesupportedbyamixofclickbaitnews,self-described“alternative”and“independent”media,aswellasgovernment-affiliateddomainssuchasRTandSputnikNews(Russia)andPressTV(Iran)andgeopoliticalthinktanksassociatedwithspecificallypro-Russianstances.Thesewebsitesdonotinitiateeachepisodeofconspiracytheorizing,buttheirarticlesdoshape,amplify,andsustainthoseconversations.Webarticlesfunctiontoassemblethedifferentpiecesofevidenceintoasingle,coherentnarrative—an “alternativenarrative” tocompetewiththeone that appearson “mainstream”mediaoutlets.Thesearticlesarethenrepeatedlypostedtosocialmediathroughsharebuttons,andmanygetre-sharedacrossotherwebdomains,expandingtheaudienceandexposure.Thewebsitecontentisalsolessephemeralthanthesocialmediaactivity,helpingtosustaintheconspiracytheoryovertime.
Another significant finding (first reported in [83]), revealed when we lookmore deeply at thebroadercontentonthewebdomains inred(Figure6), is thatmanyof thesewebsiteshostcontentsupportingnotjustoneconspiracytheory,butmany.Mostofthewebsitesinredhostdozensofarticlespromotingdifferentconspiracytheoriesacrossseeminglydisparatetopicsanddomains—e.g.climatechangedenial,anti-vaccinepseudoscience,claimsthatObamaisreallyalizard-likealienandthatHillaryClintonrunsapedophileringunderapizzaplaceinWashingtonDC,theoriesthatashadowycabalofrichJewishmensecretlyruntheworld,9-11trutherism,flatearththeory,etc.Theseareverydifferenttheories,buttheyoften reflecta sharedepistemology—one thatpushespeople toquestion science,journalism,andtheintegrityofdemocraticinstitutions.Relatedly,muchofthecontentonthesedomainswas(asofDecember2016)overtlypolitical,buttheirpoliticswerenottraditionally“right”or“left”.Instead,theypromotednationalistandpopulistmessagesandmanywereconsistentlysupportiveofU.S.PresidentelectTrump,Russia’sPresidentPutin,andthegeopoliticalaimsofRussia.
Fig.6.DomainNetworkGraphofMediaEcosystemthatSurroundingConspiracyTheorizingaboutShootingEventsin2016.Nodesaredomainscitedintweetsreferencing“falseflag”or“crisisactors”conspiracytheoriesofshootingevents.Nodesaresizedbythenumberoftweets
16
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
linkingtothosedomains.Edgesarecreatedwhenthesameusersenttweetslinkingtobothdomains.Edgesareweightedbythenumberofdifferentuserswhotweetedlinkingtobothdomains.Rednodesaredomainsthathostedcontentexplicitlypromotingoneormore
conspiracytheoriesoftheshootingevents.Bluedomainsexplicitlychallengedthoseconspiracytheories.Yellowdomainsneitherexplicitlypromotednorexplicitlychallengedtheconspiracytheoriesofthoseshootingevents,butwerepulledintotheconversationforbeinga“straight
news”articleabouttheshootingevent.
Thisconvergenceofconspiracytheoriesandpopulistpoliticalmessageslikelyhasmultiplecausesanddiverseeffects.Therearetechnicalorstructuralfactorsinthisconvergence,wherebythetoolsofpublicationandcontent-sharingagreementsinthealternativemediaecosystemencouragelow-budgetwebsitestofilltheirpageswiththisfreeorlow-costcontent.Theremayalsobestrategicreasonsforinformationoperatorstopushthiskindofcontentacrossthesemediaecosystems.Ithasbeenarguedthatapersonwhobelievesoneconspiracytheorymaybemoresusceptibletoanother[88],andsothesewebsitescouldbeactingpurposefullyasgatewaysfromoneconspiracytheorytoanother.ThereareechoesofotherSoviet-eraconspiracytheoriesinthisecosystemaswell,demonstratinghowtheeffectsofinformationoperationscanbediffuseandlong-lasting.
Onewaytoviewtheonlineconspiracytheorizingdescribedinthiscasestudyisasalargelyemergentandself-sustainingactivitythatisshapedbothbycurrentsociotechnicalsystemsandsocialstructuresaswellasbylingeringnormsandepistemologiesthathave,inturn,beenshapedinsubtleways,andtosomeextentoverdecades,bystrategicinformationoperations.
5 DISCUSSION Inthispaperwehaveconceptualizedinformationoperationsassociotechnicalphenomenathatrelyonavarietyofactorsandstructurestosuccessfullydisseminateproblematickindsofinformation,andwehavepresentedthreecasestudiesthatdemonstratedifferentkindsofinformationoperations—intermsoftherelationshipsbetweenthoseoperationsandtheonlinecommunitieswithinwhichtheyoperate.Fromahighlevel,wecancategorizethefirstcasestudyashighlyorchestrated,thesecondascultivated,andthethirdaslargelyemergentandself-sustaining.However,wecanalsoseeeachofthesedynamicsatplay,tosomeextent,acrossallthreestudies.Takentogether,thesecasestudiesproviderichinsightinto how information operations manifest within online communities and highlight the diffuseintersectionsbetweentheworkofprofessionalinformationoperatorsandtheactivitiesofadistributed,onlinecrowd.
5.1 Strategic Information Operations as Collaborative Work Weconceptualizetheseintersectionsasaformofcollaborativework,usingasociotechnicalperspective[26,60,61]tohighlightthemechanismsofcoordination.Insomecaseswecanseeevidenceofthosecollaborations in the digital record (e.g. anti-White Helmets activists and agents of informationoperation campaigns mobilizing through @mentions to spread disinformation) or infer them bytriangulatingthepublicallyavailablesocialmediadatawithotheraccountsabouttheinnerworkingsofthe informationoperators (i.e.within theRU-IRA).But inothercases it is less clear thatactorsareexplicitlycoordinatingtheiractions,andinsteadthecollaborationconsistsofconvergentbehaviorsthatreflectakindofintentionalshapingorcultivationofanonlinecommunity.
Thatshapingisnotuni-directional,buthaselementsofimprovisationastheoperatorsworktobothreflectandguidetheactivitiesoftheirhostcommunity(e.g.RU-IRAagentsusing#BlackLivesMatterand#BlueLivesMatterhashtagsintheiraccountsandtweets).Thisdynamicisnotnew[9,13,67,72].Forexample, in Nazi Germany during World War II, orchestrated propaganda campaigns tapped intograssrootsorganizingeffortstocultivateanetworkofneighborhoodvolunteerpropagandistscalled“TheRing”[13].TheRingfilledanimportantroleinNaziGermany’spropagandasystembyadjustingnationalmessagestofitlocalsituationsandbyservingasaninterpersonalchannelforspreadingcertainnarratives.Bytwerkcapturessomeofthevalueofthisapproachbynotinghow“Germansknewthatpartymemberswereobligatedtosaytherightthings. Itwassomethingelsewhenashopkeeperor
Disinformation as Collaborative Work 17 teacherwithnoobviouspartyconnectionmadethepoint”[13].Ultimatelytheseordinarycitizenswereimplicated in the propaganda system, but the system itself initially emerged to help them feelempoweredbyallowingthemto“participateinthegreateventsoftheday”[30].
Moderninformationoperationsrepresentanextensionoftheseexistinglogics,althoughthematerialaffordancesofsocialmediacanbothbroadentheroleofpublicparticipationandmakeitmorevisible.Ourcasestudiesshowhowthisparticipationistakingonnewshapesbydocumenting,forinstance,theactivitiesofthePJNETgroupinStudy1ortheanti-waractivistsandwestern“journalists”inStudy2.Theseexamplesunderscorethefactthatonlineinformationoperationsdonotexcusivelyfollowatop-downmodeloftransmittingdisinformationthroughexplicitlycoordinatedactors,butoftenrelyinsteadonpersuadingaudiencestobecome“unwittingagents”(intheproblematictermsusedbyintelligenceprofessionals)[9]or“citizenmarketers”[67]whotakeituponthemselvestospreadthesemessagesatthe grassroots level. That is to say, information operations in online settings involve work that isinterdependentinnature—i.e.consistingofindependentcontributionsbydifferenttypesofformalandinformal workers that ultimately rely on each other. Some of these workers are unpaid grassrootintermediaries(e.g.forummoderators,activistgrandmothers,andvolunteerorganizers)whooperatelike fans to amplify the operation’s messages and occasionally take those messages forward inunpredictableways,whileothersarepaidprofessionals(e.g.publicrelationsexpertsandkeyopinionleaders)tryingtosupportanddirectthispeer-to-peeractivity.
5.2 Implications for Design of Platform Policy: The Trouble with “Coordinated Inauthentic Behavior”
Thisviewof informationoperationsas collaborativeworkhas implicationsfor thedesignofpolicy.Jacksonetal.[41]havearguedfora“revival”ofCSCWattentiontopolicy,bywhichtheymeanbothatthemacro-leveloflawandgovernmentaswellasthemeso-leveloftheinternalpoliciesthatprivatecompaniesusetoguidetheirownactions.Here,weechothatargumentandhighlightsomeofthewaysthat aCSCWperspectiveon strategic informationoperationscaninformsomeof the currentpolicychallengesthatsocialmediaplatformsface.
Associalmediacompanieshavemovedtoaddressproblemsofmisinformation,disinformation,andothertoxicbehaviorsontheirplatforms,manyhavedevelopedpoliciesfordefiningactorsorbehaviorsthat they findproblematicand for taking specific actions toaddress thoseactorsorbehaviors(e.g.account or content removal, reduction in visibility, demonetization, etc.). These companies areincreasingly communicatingwith thepublic in termsof thesepolicies [e.g.28,29,81,106,107]. Indealingwithdisinformationandinformationoperations specifically, the companieshaverepeatedlyexpressedreticencetofocusoncontent[29,107],oftenstatingthattheydonotwanttobethe“arbitersoftruth”[48,80].Thispositionrevealssomeofthe“knot”thatJacksonetal.[41]conceptualizebetweendesign,practice,andpolicy—astheplatformsattempttonavigateadifficultcompromiseinrelationtoexisting legal statutes and dynamic social norms around “freedom of speech” and in themidst ofincreasingpoliticalpressureandcriticismsof“politicalbias”[32].Insteadofcontent,theplatformshaveelected to focus on authenticity and behavior. For example, Facebook’s policy currently stresses aconceptualizationof“coordinatedinauthenticbehavior”[28].
Butthecaseswehaveexamined—fromaCSCWperspectivethathighlightsthecollaborativeandparticipatory nature of online information operations—signal some inherent difficulties with suchapproaches. Inparticular,ourworkrevealsentanglementsbetweenorchestratedactionandorganicactivity,includingtheproliferationofauthenticaccounts(realpeople,sincerelyparticipating)withinactivitiesthatareguidedbyand/orintegratedintodisinformationcampaigns.Aligningwithresearchon how active participation strengthens attitudes [62], across our studies, we observe ordinarymembersof theonline crowdappearing, over time, to increasinglyview theirparticipationaroundspreading preferred narratives—sharing anti-Hillary content in the “left” cluster in Study 1,disseminatinganti-WhiteHelmetscontent inStudy2,sharingconspiracytheoriesofcrisisevents inStudy3—notonlyintermsofshapingorinfluencingoutcomes,butasastatementoftheirownidentity.Thiswasmadevisibleinexpressivepracticeslikeusingprotestsymbolsandhashtagsintheirsocialmediaprofiles[87],andit isadynamicthatcanbeseeninothercasesofonlineactivism[85]. It isimplicated in some of the convergence between how online volunteers and information operatorspresentthemselvesintheirprofiles.And,itsuggeststhatthoughthepublicisintheoryinformationallydownstreamfromthemessagesbeingpropagatedbyinformationoperations,its“organic”participation
18
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
cangrow in importance and visibility relative to “orchestrated” efforts.The case of the conspiracytheorizersinStudy3,andtosomeextenttheanti-WhiteHelmetsactivistsinStudy2,speaktothisideaof how disinformation can take root in online communities and become, at least partially, self-sustaining.Platformpoliciesdesignedaroundrootingout“coordinated inauthenticbehavior”wouldhavedifficultyaddressingthesecampaignsoncetheyhavereachedthislevelofmaturity.
Furthercomplicatingthispictureisthefactthattheseinformationoperationsarethemselvesshapedbyorganicactivity—asoperatorsroutinely“takeup”andreflectbackmessagesoriginatinginthecrowd.OngandCabaneshintatthisdynamicintheirstudyofnetworkeddisinformationinthePhilippines[59],andweseeitagaininourcasestudyofthedisinformationcampaigntargetingtheWhiteHelmets(Study2).Inbothcases,activistsandoperatorsusedoverlappingtoolsandpracticestoincreaseoutreach—suchastargetingspecificaudiences,harnessingword-of-mouthendorsementsforstrategicpurposes,selectivelyforwardingnewsinwaysthatblurtheboundariesbetweeneducationandpersuasion,andengagingnotonlyin‘rational’dialogbutalsomodelingacontagiousemotionalenthusiasmforotherstofollow [59, 67]. This perspective again demonstrates the difficulty of differentiating between theorchestrated behaviors of strategic information operators and the organic behaviors of the onlinecommunitiesthatthoseoperatorsroutinelytarget.
Theseinsightssuggestthatpoliciesbuiltsolelyoncoordinationandinauthenticitymayfallshortofaddressinginformationoperations,especiallyoncetheirworkhastakenroot.Theyalsoshedlightontosomeofthechallengesthatplatformsfaceindevelopingandimplementingtheirpolicies.Forexample,a more robust approach might consider information operations at the level of a campaign andproblematizecontentbasedonthestrategicintentofthatcampaign.Apolicylikethiscouldempowertheplatformstotakeactionbasedontheprovenanceofinformation—e.g.withinacampaigndesignedtomisleadforpoliticalpurposesortounderminetheintegrityofinformationspaces—ratherthanthetruth value of a piece of content or the authenticity/sincerity of a specific account. However, thisapproachleavestheplatformsinapositionoftakingactiontoremoveorreducevisibilityofcontentthatmaybesharedorevenproducedbyauthenticaccountsofsincereonlineactivists.Andthismightputtheplatform’spolicyatoddswithcommonlyheldvalueslike“freedomofspeech”andplatformgoalssuchasprovidingaplaceforactivists(includingthoseinoppressedgroups)tocongregateandorganize.
Thesearenotsimplechallengesandtheyhavebeenshowntoberesistenttosimplesolutions.Inaddressingthem,wearguethatplatformpoliciesattheintersectionofdigitalexpressionandstrategicinformationoperationscannotbe“neutral”ornarrowlyprocedural;theymustbesubstantive,whichistosaythattheymustdotheirbesttoensurethatthevaluesthatanimateoursocietiesarefaithfullytranslatedtothedigitalenvironment.Craftingsuchpolicieswillinvolvechallengingconversationsthatmustavoidmagicalthinkingofthesortthatfrequentlyentersintotechnologypolicydebates(e.g.thefetishization of new technologies or “disruptive innovation”). Echoing Jackson and colleagues’encouragement [41], we argue here that CSCW researchers should be participating in theseconversations,helpingtoguidethedevelopmentofsocialmediaplatformpoliciesastheyattempttoaddresstheseemerginganddynamicthreatstotheplatformsthemselvesanddemocraticdiscoursemorebroadly.
5.3 Implications for Researchers: Theorizing the Effects of Online Information Operations
Currently, there is still substantial uncertainty and speculation regarding the potential impact ofinformationoperationsinvolvingsocialmediaplatforms.Unlikethedistributionofmaterialartifactssuchaspamphletsandposters,thecirculationofcontentonsocialmediacreatesdigitaltracesthatcanbe systematically analyzedtomap thereachofdisinformation spreadingactivities in superbdetail.However, the intensely participatory nature of strategic information operations that we havehighlightedsuggeststhatitisnotenoughtoseehowmanypeoplewereexposedtoanygivenpieceofmisleadinginformation;wemustunderstandwhattheseaudiencesdowithit.
Here,webelieveitisimportanttomovebeyondquestionsthatfocusonthehighest-levelcorrelationsaffordedbytheanalyticsoflargevolumesofdata.Suchworkiswelcomeandrelevantbutisonlythebeginningofa seriesof inquiries thatneed tobemade intoaudience interactionswith informationoperations. Computational techniques for analyzing large datasets can allow us to do things like
Disinformation as Collaborative Work 19 calculatetheviralpopularityofdeceptivecontentandcorrelateitwithquantifiableshort-termvariablessuchasthenumberofresharesorelectionresults,butbythemselvestheseapproachesarelesswellequippedtomeasurecomplexprocessesofpoliticalidentityformationandlong-termshiftsinculturalandsocialnormsthatarepartofthelogicsoftechniqueslikedezinformatsiya.
Forexample,researchershaverecentlybegunanalyzinglargesocialmediadatasetstotrytomeasuretheeffectsofRussianinterferenceinthe2016U.S.election(whichfactoredintobothourfirstandthirdcasestudies).Someoftheserecentstudies[e.g.79,105]havereportedobservinglimitedeffectsontheshort-termonlinebehaviorsofsocialmediausersthatwereexposedtotheseinformationoperations.However,careshouldbeexercisedwheninterpretingorframingsuchfindingstoavoidsuggestingthattheeffectsofinformationoperationsareminor,orindeedcanevenbeproperlydetermined.Careisparticularly importantwiththepubliccommunicationsofsuchfindingsbecauseofhowtheycanbeappropriatedbynon-experts to informdecisionmaking, aswell aspolitical andethical thinking, inproblematicways[e.g.51].Whileitisindeedapossibilitythattheeffectsofinformationoperationsareminor,itisalsopossiblethattheseeffectsareextremelydifficulttomeasureduetothediffusenatureofstrategicinformationoperations,thecomplexinterplaybetweenorchestratedandorganicaction,andthepotentialsecond-ordereffectsthatoccurthroughchangestothesocialnetworks,socialandpoliticalnorms,andothersociotechnicalstructuresinthebroaderinformationspace.
Thisdifficultyinquantifyingtheeffectsofpropagandaisnotnew.Bittmannotedit,whiledescribingthe impactsofKGBactivemeasures inapre-onlineworld,writing: “Theresultsarenot statisticallymeasurable,ofcourse,buttheKGBevaluatesthesedeceptiongamesinbroaderpolitical terms”[9].Moreover, social scientistshavedisputed the ideaof information campaignsdirectly influencing thepublicsinceatleastthe1950s.Forinstance,KatzandLazarsfeld’sPersonalInfluence[46]arguedagainstunderstandingtheeffectsofsuchcampaignsusingone-stepcause-and-effectmodels,findingthatthepersuasiveeffectofanymediamessagewasstronglymediatedbyopinionleadersandone’scircleofpeers.Morerecently,Marwickhaspersuasivelyarguedforthevalueofemployingsociotechnicalmodelstounderstandtheeffectsofproblematicallydeceptiveinformation,whiledrawinguponactiveaudienceparadigms that reject the idea of treating social media audiences as an undifferentiated mass ofinterchangeable“culturaldupes”[54].
Inprinciple,CSCWandHCIdrivenapproachesshouldfigurestronglyinilluminatingsomeofthesedynamicsconcerningtheeffectsofinformationoperations.CSCWresearchvaluesworkingwith“BigData”tounderstandphenomenathatemergeasaconsequenceoflarge-scalegroupactivities.Atthesametime,thefieldrecognizestheimportanceofunderstanding“smalldata”,thatistherich,qualitativeexperiences of social actors who engage in cultural and political practices. Understanding howindividualsandgroupsmakemeaningfromthesepracticesandinterprettheirpoliticalrelevancewillallowustodevelopmorerobustperspectivesontheeffectsofinformationoperations.
As a closing remark,we offer this reflection, observationand appeal:Notonly can theevolvingdiscourse around information operations benefit from CSCW perspectives, CSCW perspectives canbenefitfromgrapplingwiththesemurkyandmuddyphenomena.Informationoperationsbothcanandhaveused thepro-social cooperativeworktakingplaceoversocialmediaasapointof access.Thisincludes, for instance, grassroots political organizing, digital volunteerism, and the wider work ofsensemakingandsharinginformationduringbreakingnewseventsandcrisissituations[e.g.85,87,98,104].Byconsideringthedynamicsofinformationoperations,wemightourselvesbemoreinnovativeandunderstandingofothersocialphenomenathatwearealreadystudying.Ourresearchcommunityisalsoinvestedinhowwecansupportprocessesofbeingmoreself-criticalandawareaswebuildandusesystems forgroups [e.g.5,73, 76].Thoughinformationoperationsarepolitically chargedandthusdifficulttostudyfroma“neutral”pointofview,theycanhelpusexposeandinterrogateassumptionsandvalue-commitmentshiddentousundernormalconditionsthatneverthelessinformourevaluationand design efforts. Information operations are thus not only important to the study of onlinecollaborations;theyarealsoimportantcontextsfromwhichtostudyonlinecollaborations.
6 CONCLUSION Inthispaper,wehaveframedinformationoperationsasaCSCWconcern—onewithcriticalsocietal
import.Drawingoncasestudiesandhistoricalcontext,weshowthattheworkofonlineinformationoperationsextendsbeyondthenarrowwindowofautomatedorpaidactorssuchasbotsandhiredtrolls—thewindowthattendstogetthemostmediaandresearchattention.Ourworkunderscoresthe
20
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
fact that these operations are participatory, taking shape and persisting as collaborations betweenorchestrated agents and organic crowds.We demonstrate, from a sociotechnical perspective, howinformation operations function at multiple levels, by directly shaping actions perhaps, but moreprofoundlybyshapingthesurroundingsocialstructures—e.g. thenetworksofactiviststhat takeuptheirmessages,alongwiththeirnorms,practices,andideologies.Wenotehowtheseoperationstakeadvantageofandresonatewiththedesignofsocialmediaplatformsthathavebecomecentraltohowinformation is created, shared, and negotiated across the globe. These platforms face mountingchallengestocountertheseemerginganddynamicthreats,andwearguethataCSCWperspectivecanhelpguidethemastheydeveloppoliciesinresponse.Weconcludebyhighlightingtheneedformorerobust,cross-disciplinarytreatmentsofinformationoperationstounderstandtheiremergentfeaturesinatechnologicallymediatedworld.
REFERENCES [1] NorahAbokhodair,DaisyYoo,andDavidW.McDonald.2016.DissectingaSocialBotnet:Growth,Content
andInfluenceinTwitter.InProceedingsofthe2016ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'16).ACM,NewYork,NY,USA,839-851.DOI:https://doi.org/10.1145/2675133.2675208
[2] AhmerArif,JohnJ.Robinson,StephanieA.Stanek,ElodieS.Fichet,PaulTownsend,ZenaWorku,andKateStarbird.2017.Acloserlookattheself-correctingcrowd:Examiningcorrectionsinonlinerumors.InProceedingsofthe2017ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'17).ACM,NewYork,NY,155-168.DOI:https://doi.org/10.1145/2998181.2998294
[3] AhmerArif,LeoG.Stewart,andKateStarbird.(2018).ActingthePart:ExaminingInformationOperationswithin#BlackLivesMatterDiscourse.Proc.ACMHum.-Comput.Interact.2,CSCW,Article20(November2018),27pages.DOI:https://doi.org/10.1145/3274289
[4] DanArnaudo.2017.ComputationalPropagandainBrazil:SocialBotsduringElections.ComputationalPropagandaResearchProject.OxfordUniversity,Oxford,UK.
[5] KarlaBadillo-Urquiola,YaxingYao,OshratAyalon,BartKnijnenurg,XinruPage,EranToch,YangWang,andPamelaJ.Wisniewski.2018.PrivacyinContext:CriticallyEngagingwithTheorytoGuidePrivacyResearchandDesign.InProceedingsofthe2018ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'18).ACM,NewYork,NY,425-431.DOI:https://doi.org/10.1145/3272973.3273012
[6] RonaldM.Baecker,JonathanGrudin,WilliamA.S.Buxton,andSaulGreenberg(Ed.).1995.Groupwareandsocialdynamics:Eightchallengesfordevelopers.ReadingsinHuman–ComputerInteraction.DOI:https://doi.org/10.1016/B978-0-08-051574-8.50079-0.
[7] ChristineA.Barry,NickyBritten,NickBarber,ColinBradley,andFionaStevenson.1999.Usingreflexivitytooptimizeteamworkinqualitativeresearch.Qualitativehealthresearch9,no.1(1999):26-44.
[8] LadislavBittman.1972.TheDeceptionGame:CzechoslovakIntelligenceinSovietPoliticalWarfare.SyracuseUniversityPress,Syracuse.NY.
[9] LadislavBittman.1985.TheKGBandSovietDisinformation:AnInsider'sView.Pergamon-Brassey's,Washington,DC.
[10] LindsayBlackwell,JillDimond,SaritaSchoenebeck,andCliffLampe.2017.ClassificationandItsConsequencesforOnlineHarassment:DesignInsightsfromHeartMob.InProceedingsoftheACMonHumanComputerInteraction1,CSCW,Article24(November2017).DOI:https://dx.doi.org/10.1145/3134659
[11] SamanthaBradshawandPhillipN.Howard.2018.Challengingtruthandtrust:Aglobalinventoryoforganizedsocialmediamanipulation.ComputationalPropagandaResearchProject.OxfordUniversity,Oxford,UK.
[12] SamanthaBradshawandPhillipN.Howard.2018.WhydoesJunkNewsSpreadSoQuicklyAcrossSocialMedia?Algorithms,AdvertisingandExposureinPublicLife.KnightFoundation.Retrievedfromhttps://kf-site-production.s3.amazonaws.com/media_elements/files/000/000/142/original/Topos_KF_White-Paper_Howard_V1_ado.pdf
[13] RandallL.Bytwerk.2010.GrassrootspropagandaintheThirdReich:TheReichringforNationalSocialistpropagandaandpublicenlightenment.GermanStudiesReview33(2010).93-118.DOI:http://dx.doi.org/10.2307/40574929
[14] CaroleCadwalladr.2017.ThegreatBritishBrexitrobbery:howourdemocracywashijacked.(May2017).RetrievedApril3,2019fromhttps://www.theguardian.com/technology/2017/may/07/the-great-british-brexit-robbery-hijacked-democracy
[15] KathyCharmaz.2014.ConstructingGroundedTheory.Sage,ThousandOaks,CA.[16] NormanK.DenzinandYvonnaS.Lincoln(Eds.).1994.GroundedTheoryMethodology:AnOverview.Anselm
StraussandJulietCorbin.HandbookofQualitativeResearch.Sage,ThousandOaks,CA.
Disinformation as Collaborative Work 21 [17] ReneeDiResta,Dr.KrisShaffer,BeckyRuppel,DavidSullivan,RobertMatney,RyanFox,Dr.Jonathan
Albright,andBenJohnson.TheTactics&TropesoftheInternetResearchAgency.NewKnowledge.Retrievedfromhttps://disinformationreport.blob.core.windows.net/disinformation-report/NewKnowledge-Disinformation-Report-Whitepaper.pdf
[18] PaulDourishandVictoriaBellotti.1992.AwarenessandCoordinationinSharedWorkspaces.InProceedingsofthe1992ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'92).ACM,NewYork,NY,USA,107-114.DOI:https://doi.org/10.1145/143457.143468
[19] AdamB.EllickandAdamWestbrook.2018.OperationInfektion.RussianDisinformation:FromColdWartoKanye.RetrievedApril3,2019fromhttps://www.nytimes.com/2018/11/12/opinion/russia-meddling-disinformation-fake-news-elections.html
[20] EnglishOxfordLivingDictionaries.Disinformation.RetrievedApril3,2019fromhttps://en.oxforddictionaries.com/definition/disinformation
[21] RobertM.Faris,HalRoberts,BruceEtling,NikkiBourassa,EthanZuckerman,andYochaiBenkler.2017.Partisanship,Propaganda,andDisinformation:OnlineMediaandthe2016U.S.PresidentialElection.BerkmanKleinCenterforInternet&SocietyResearch.HarvardUniversity,Cambridge,Massachusetts,UnitedStates.
[22] EmilioFerrara,OnurVarol,ClaytonDavis,FilippoMenczer,andAlessandroFlammini.2016.TheRiseofSocialBots.CommunicationsoftheACM59,7(2016).96-104.DOI:https://doi.org/10.1145/2818717
[23] VijayaGaddeandYoelRoth.2018.EnablingfurtherresearchofinformationoperationsonTwitter.(October2018).RetrievedApril3,2019fromhttps://blog.twitter.com/en_us/topics/company/2018/enabling-further-research-of-information-operations-on-twitter.html
[24] CliffordGeertz.1975.Theinterpretationofcultures.Chicago:ChicagoUniversityPress.[25] R.StuartGeigerandDavidRibes.2011.Traceethnography:Followingcoordinationthroughdocumentary
practices.InProceedingsoftheAnnualHawaiiInternationalConferenceonSystemSciences.1-10.DOI:https://doi.org/10.1109/HICSS.2011.455
[26] AnthonyGiddens.1984.TheConstitutionofSociety:OutlineoftheTheoryofStructuration.UniversityofCaliforniaPress,Berkeley,CA.
[27] SamGill.2019.Thepriceofprogress:How"DigitalPollution"ispoisoningdemocracyandwhatwecandoaboutit.(January2019).RetrievedApril3,2019fromhttps://knightfoundation.org/articles/the-price-of-progress-how-digital-pollution-is-poisoning-democracy-and-what-we-can-do-about-it
[28] NathanielGleicher.2019.CoordinatedInauthenticBehaviorExplained.FacebookNewsroom.(December2018).RetrievedJune26,2019fromhttps://newsroom.fb.com/news/2018/12/inside-feed-coordinated-inauthentic-behavior/
[29] NathanielGleicher.2019.RemovingCoordinatedInauthenticBehaviorandSpamFromIndiaandPakistan.(April2019)RetrievedApril3,2019fromhttps://newsroom.fb.com/news/2019/04/cib-and-spam-from-india-pakistan/
[30] JosephGoebbels.1938.DerRundfunkalsachteGroßmacht.SignalederneuenZeit.25ausgewählteRedenvonDr.JosephGoebbels.197-207.
[31] MarilysGuilleminandLynnGillam.2004.Ethics,reflexivity,and“ethicallyimportantmoments”inresearch.Qualitativeinquiry,10.2(2004):261-280.
[32] JessicaGuynn.2019.TedCruzthreatenstoregulateFacebook,GoogleandTwitteroverchargesofanti-conservativebias.USAToday.(April2019).RetrievedJune26,2019fromhttps://www.usatoday.com/story/news/2019/04/10/ted-cruz-threatens-regulate-facebook-twitter-over-alleged-bias/3423095002/
[33] JürgenHabermas.1984.Thetheoryofcommunicativeaction,Vol.2.Beaconpress,Boston,MA.[34] RongbinHan.2015.Manufacturingconsentincyberspace:China’s‘fifty-centarmy’.JournalofCurrent
ChineseAffairs44,2(2015).105-134.[35] DelHarveyandDavidGasca.2018.ServingHealthyConversation.TwitterBlog.(May2018).RetrievedJune
26,2019fromhttps://blog.twitter.com/en_us/topics/product/2018/Serving_Healthy_Conversation.html[36] PhilipN.Howard.2002.Networkethnographyandthehypermediaorganization:Newmedia,new
organizations,newmethods.NewMedia&Society4,4(2002).550-574.DOI:https://doi.org/10.1177/146144402321466813
[37] Y.LinlinHuang,KateStarbird,ManiaOrand,StephanieA.Stanek,andHeatherT.Pedersen.2015.Connectedthroughcrisis:Emotionalproximityandthespreadofmisinformationonline.InProceedingsofthe2015ACMConferenceonComputer-SupportedCooperativeWork(CSCW'15).ACM,NewYork,USA,969-980.DOI:https://doi.org/10.1145/2675133.2675202
[38] AmandaLeeHughesandLeysiaPalen.2009.Twitteradoptionanduseinmassconvergenceandemergencyevents.InternationalJournalofEmergencyManagement6,3-4(2009).248-260.
[39] EdwinHutchins.1995.CognitionintheWild.MITpress,Cambridge,MA.[40] CarolineJack.2017.Lexiconoflies:Termsforproblematicinformation.Data&Society.Retrievedfrom
https://datasociety.net/output/lexicon-of-lies/[41] StevenJ.Jackson,TarletonGillespie,andSandyPayette.2014.Thepolicyknot:Re-integratingpolicy,
practiceanddesigninCSCWstudiesofsocialcomputing.InProceedingsofthe17thACMconferenceon
22
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
Computersupportedcooperativework&socialcomputing(CSCW’14),ACM,NewYork,USA,588-602.DOI:https://doi.org/10.1145/2531602.2531674
[42] KathleenHallJamieson.2018.Cyberwar:HowRussianHackersandTrollsHelpedElectaPresidentWhatWeDon't,Can't,andDoKnow.OxfordUniversityPress,2018.
[43] ShagunJhaver,SuchetaGhoshal,AmyBruckman,andEricGilbert.2018.Onlineharassmentandcontentmoderation:Thecaseofblocklists.ACMTransactionsonComputer-HumanInteraction(TOCHI2018)25,no.2(2018):12.DOI:https://doi.org/10.1145/3185593
[44] JointChiefsofStaff.2014.InformationOperations.JointPublication3-13.DepartmentofDefense,UnitedStates.Retrievedfromhttp://www.jcs.mil/Portals/36/Documents/Doctrine/pubs/jp3_13.pdf
[45] SimonM.Kaplan,WilliamJ.Tolone,DouglasP.Bogia,andCelsinaBignoli.1992.Flexible,ActiveSupportforCollaborativeWorkwithConversationBuilder.InProceedingsofthe1992ACMConferenceonComputer-SupportedCooperativeWork(CSCW'92).ACM,NewYork,USA,378-385.DOI:https://doi.org/10.1145/143457.143552
[46] ElihuKatz,PaulF.Lazarsfeld,andElmoRoper.2017.Personalinfluence:Thepartplayedbypeopleintheflowofmasscommunications.Routledge,London,UK.
[47] GaryKing,JenniferPan,andMargaretE.Roberts.2017.HowtheChinesegovernmentfabricatessocialmediapostsforstrategicdistraction,notengagedargument.AmericanPoliticalScienceReview111,3(2017).484-501.
[48] ArjunKharpal.2017.Facebookdoesn'twanttobethe'arbiterofthetruth',topexecSherylSandbergsays,amidfakenewscriticism.CNBC.com.(April2017).RetrievedJune26,2019fromhttps://www.cnbc.com/2017/04/24/facebook-fake-news-sheryl-sanberg.html
[49] AudreyM.Kleinsasser.2000.Researchers,reflexivity,andgooddata:Writingtounlearn.Theoryintopractice39.3(2000):155-162.
[50] MatthewLevinger.2018.MasterNarrativesofDisinformationCampaigns.JournalofInternationalAffairs71,1.5(2018).125-134.
[51] JohnLeyden.2018.Kremlinsocialmediatrollsaren'tactuallythatinfluential,studyfinds.(Jan2018).RetrievedApril3,2019fromhttps://www.theregister.co.uk/2018/01/30/russian_troll_influence/
[52] HerbertS.LinandJaclynKerr.2017.OnCyber-EnabledInformation/InfluenceWarfareandManipulation.OxfordUniversityPress,UK.
[53] JimMaddock,KateStarbird,HaneenJ.Al-Hassani,DavidE.Sandoval,ManiaOrand,andRobertM.Mason.2015.Characterizingonlinerumoringbehaviorusingmulti-dimensionalsignatures.InProceedingsofthe18thACMConferenceonComputerSupportedCooperativeWork&SocialComputing.ACM,NewYork,NY,228-241.DOI:https://doi.org/10.1145/2675133.2675280
[54] AliceE.Marwick.2018.WhydoPeopleShareFakeNews?ASociotechnicalModelofMediaEffects.GeorgetownLawTechnologyReview2,474(2018).474-512.
[55] AliceE.Marwickanddanahboyd.2011.Itweethonestly,Itweetpassionately:Twitterusers,contextcollapse,andtheimaginedaudience.NewMedia&Society13,1(2011),114-133.DOI:https://doi.org/10.1177/1461444810365313
[56] JolieMyersandMonikaEvstatieva.2018.MeetTheActivistWhoUncoveredTheRussianTrollFactoryNamedInTheMuellerProbe.NPR:Parallels—ManyStories,OneWorld.(March2018).RetrievedJune26,2016fromhttps://www.npr.org/sections/parallels/2018/03/15/594062887/some-russians-see-u-s-investigation-into-russian-election-meddling-as-a-soap-ope
[57] YiftachNagar.2012.Whatdoyouthink?:TheStructuringofanOnlineCommunityasaCollective-SensemakingProcess.InProceedingsofthe2012ACMConferenceonComputer-SupportedCooperativeWork(CSCW'12).ACM,NewYork,USA,393-402.DOI:https://doi.org/10.1145/2145204.2145266
[58] OfficeoftheDirectorofNationalIntelligence.2017.AssessingRussianactivitiesandintentionsinrecentUSelections.NationalIntelligenceCouncil.(January2017).RetrievedApril3,2019fromhttps://www.dni.gov/files/documents/ICA_2017_01.pdf
[59] JonathanCorpusOngandJasonVincentA.Cabanes.2018.ArchitectsofNetworkedDisinformation.TheNewtonTech4DevNetwork.UniversityofLeeds,Leeds,UK.Retrievedfromhttp://newtontechfordev.com/wp-content/uploads/2018/02/ARCHITECTS-OF-NETWORKED-DISINFORMATION-FULL-REPORT.pdf
[60] WandaJ.Orlikowski.1992.Thedualityoftechnology:Rethinkingtheconceptoftechnologyinorganizations.Organizationscience3,3(1992).398-427.
[61] WandaJ.Orlikowski.2007.Sociomaterialpractices:Exploringtechnologyatwork.Organizationstudies28,9(2007).1435-1448.
[62] StuartOskampandP.WesleySchultz.2005.AttitudesandOpinions(3rded.).LawrenceErlbaum,Mahwah,NJ.
[63] LeysiaPalenandKennethM.Anderson.2016.Crisisinformatics-Newdataforextraordinarytimes.Science353,6296(2016).224-225.DOI:https://doi.org/10.1126/science.aag2579
Disinformation as Collaborative Work 23 [64] LeysiaPalenandSophiaB.Liu.2007.CitizenCommunicationsinCrisis:AnticipatingaFutureofICT-
SupportedPublicParticipation.InProceedingsofthe2007CHIConferenceonHumanFactorsinComputingSystems(CHI'07).ACM,NewYork,NY,727-736.
[65] AliciaParlapianoandJasmineC.Lee.2018.ThepropagandatoolsusedbyRussianstoinfluencethe2016election.(February2018).RetrievedApril3,2019fromhttps://www.nytimes.com/interactive/2018/02/16/us/politics/russia-propaganda-election-2016.html
[66] ChristopherPaulandMiriamMatthews.2016.TheRussian"FirehoseofFalsehood"PropagandaModel.RandCorporation,SantaMonica,CA.DOI:https://doi.org/10.7249/PE198
[67] JoelPenney.2017.Thecitizenmarketer:Promotingpoliticalopinioninthesocialmediaage.OxfordUniversityPress,Oxford,UK.
[68] PeterPomerantsev.TheKremlin'sinformationwar.JournalofDemocracy26,4(2015).40-50.[69] PeterPomerantsevandMichaelWeiss.2014.Themenaceofunreality:HowtheKremlinweaponizes
information,cultureandmoney.InstituteofModernRussia,NewYork,NY.[70] DavidRees.1984.SovietActiveMeasure:ThePropagandaWar.InstitutefortheStudyofConflict,London,UK.[71] AleksandraSarcevic,LeysiaPalen,JoanneWhite,KateStarbird,MossaabBagdouri,andKennethAnderson.
2012.“BeaconsofHope”inDecentralizedCoordination:LearningfromOn-the-GroundMedicalTwitterersDuringthe2010HaitiEarthquake.InProceedingsofthe2012ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'12).ACM,NewYork,NY,47-56.DOI:https://doi.org/10.1145/2145204.2145217
[72] JeffreyK.Sawyer.1990.Printedpoison:Pamphletpropaganda,factionpolitics,andthepublicsphereinearlyseventeenth-centuryFrance.UnivofCaliforniaPress,Berkeley,CA.
[73] PhoebeSengers,JohnMcCarthy,andPaulDourish.2006.ReflectiveHCI:articulatinganagendaforcriticalpractice.InProceedingsoftheCHI'06extendedabstractsonHumanFactorsinComputingSystems.ACM,NewYork,NY,1683-1686.DOI:https://doi.org/10.1145/1125451.1125762
[74] DavidK.Shipler.1986.Aftertheydefect…(December1986).RetrievedApril3,2019fromhttps://www.nytimes.com/1986/12/07/magazine/after-they-defect.html
[75] CraigSilverman.2018.RussianTrollsRanWildOnTumblrAndTheCompanyRefusesToSayAnythingAboutIt.(Feb.2018).RetrievedApril17,2018fromhttps://www.buzzfeed.com/craigsilverman/russian-trolls-ran-wild-on-tumblr-and-the-company-refuses?utm_term=.ad65gb5jz#.rdwOw8O6Z
[76] RobertSodenandLeysiaPalen.2018.InformatingCrisis:ExpandingCriticalPerspectivesinCrisisInformatics.InProceedingsoftheACMonHuman-ComputerInteraction2,CSCW,Article162(November2018).DOI:https://doi.org/10.1145/3274431
[77] OliviaSolon.2017.HowSyria'sWhiteHelmetsbecamevictimsofanonlinepropagandamachine.(December2017).RetrievedApril3,2019fromhttps://www.theguardian.com/world/2017/dec/18/syria-white-helmets-conspiracy-theories
[78] AndreiSoshnikov.2015.Thecapitalofpoliticaltrolling.MR7.ru.(March2015).RetrievedJune26,2016fromhttps://mr-7.ru/articles/112478/
[79] AlexanderSpangher,GireejaRanade,BesmiraNushi,AdamFourney,andEricHorvitz.2018.AnalysisofStrategyandSpreadofRussia-sponsoredContentintheUSin2017.arXiv:1810.10033.Retrievedfromhttps://arxiv.org/abs/1810.10033
[80] SuprajaSrinivasan.2018.Wedon'twanttobearbitersoftruth:YouTubeCBORobertKyncl.EconomicTimes/IndiaTimes.(March2018).RetrievedJune26,2019fromhttps://tech.economictimes.indiatimes.com/news/internet/we-dont-want-to-be-arbiters-of-truth-youtube-cbo-robert-kyncl/63438805
[81] AlexStamos.2018.AuthenticityMatters:TheIRAHasNoPlaceonFacebook.(Apr.2018)RetrievedApril17,2018fromhttps://newsroom.fb.com/news/2018/04/authenticity-matters/
[82] KateStarbird.2013.DeliveringPatientstoSacréCoeur:CollectiveIntelligenceinDigitalVolunteerCommunities.InProceedingsofthe2013CHIConferenceonHumanFactorsinComputingSystems(CHI'13).ACM,NewYork,NY,801-810.DOI:https://doi.org/10.1145/2470654.2470769
[83] KateStarbird.2017.ExaminingtheAlternativeMediaEcosystemthroughtheProductionofAlternativeNarrativesofMassShootingEventsonTwitter.In11thInternationalAAAIConferenceonWebandSocialMedia(ICWSM2017),Montreal,Canada,(pp.230-339).
[84] KateStarbird,AhmerArif,TomWilson,KatherineVanKoevering,KatyaYefimova,andDanielScarnecchia.(2018).EcosystemorEcho-System?ExploringContentSharingacrossAlternativeMediaDomains.In12thInternationalAAAIConferenceonWebandSocialMedia(ICWSM2018),Stanford,CA,(pp.365-374).
[85] KateStarbirdandLeysiaPalen.2011.Voluntweeters:Self-OrganizingbyDigitalVolunteersinTimesofCrisis.InProceedingsofthe2011CHIConferenceonHumanFactorsinComputingSystems(CHI'11).ACM,NewYork,NY,1071-1080.DOI:https://doi.org/10.1145/1978942.1979102
[86] KateStarbirdandLeysiaPalen.2013.WorkingandSustainingtheVirtualDisasterDesk.InProceedingsofthe2013ACMConferenceonComputer-SupportedCooperativeWork(CSCW'13).ACM,NewYork,USA,491-502.DOI:https://doi.org/10.1145/2441776.2441832
[87] LeoGraidenStewart,AhmerArif,A.ConradNied,EmmaS.Spiro,andKateStarbird.2017.Drawingthelinesofcontention:Networkedframecontestswithin#BlackLivesMatterdiscourse.InProceedingsofACM
24
PACMHCI, Vol. X, No. CSCW, Article Z. Publication Date: TBD.
Human-ComputerInteraction1,CSCW,Article96(December2017),23pages.DOI:https://doi.org/10.1145/3134920
[88] CassR.Sunstein.2014.ConspiracyTheoriesandOtherDangerousIdeas.SimonandSchuster,NewYork,NY.[89] JonathanSupovitz,AlanJ.Daly,MigueldelFresno,andChristianKolouch.2017.#CommonCoreProject.
RetrievedApril3,2019fromwww.hashtagcommoncore.com[90] TheInternetArchive.AbouttheInternetArchive.RetrievedApril17,2018fromhttps://archive.org/about/[91] RodThornton.2007.Asymmetricwarfare:Threatandresponseinthe21stcentury.Polity,Cambridge,UK.[92] TumblrHelpCenter.2018.PublicRecordofUsernamesLinkedtoState-SponsoredDisinformation
Campaigns.(Mar.2018).RetrievedApril17,2018fromhttps://tumblr.zendesk.com/hc/en-us/articles/360002280214
[93] Twitter.2018.UpdateonTwitter'sReviewofthe2016U.S.Election.(Jan.2018)RetrievedApril17,2018fromhttps://blog.twitter.com/official/en_us/topics/company/2018/2016-election-update.html
[94] UnitedStatesHouseofRepresentativesPermanentSelectCommitteeonIntelligence.2017.ExhibitB(Nov.2017).https://democrats-intelligence.house.gov/uploadedfiles/exhibit_b.pdf
[95] SarahVieweg,AmandaL.Hughes,KateStarbird,andLeysiaPalen.2010.MicrobloggingDuringTwoNaturalHazardsEvents:WhatTwitterMayContributetoSituationalAwareness.InProceedingsofthe2010CHIConferenceonHumanFactorsinComputingSystems(CHI'10).ACM,NewYork,NY,1079-1088.DOI:https://doi.org/10.1145/1753326.1753486
[96] MorganVigil-Hayes,MarisaDuarte,NicholetDeschineParkhurst,andElizabethBelding.2017.#Indigenous:TrackingtheConnectiveActionsofNativeAmericanAdvocatesonTwitter.InProceedingsofthe2017ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'17).ACM,NewYork,NY,USA,1387-1399.DOI:https://doi.org/10.1145/2998181.2998194
[97] JessicaVitak,KalyaniChadha,LindaSteiner,andZahraAshktorab.2017.IdentifyingWomen'sExperiencesWithandStrategiesforMitigatingNegativeEffectsofOnlineHarassment.InProceedingsofthe2017ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'17).ACM,NewYork,USA,1231-1245.DOI:https://doi.org/10.1145/2998181.2998337.
[98] ShaunWalker.2017.RussiantrollfactorypaidUSactiviststohelpfundprotestsduringelection.TheGuardian.Availableat:https://www.theguardian.com/world/2017/oct/17/russian-troll-factory-activists-protests-us-election
[99] YiranWangandGloriaMark.2017.EngagingwithPoliticalandSocialIssuesonFacebookinCollegeLife.InProceedingsofthe2017ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'17).ACM,NewYork,NY,USA,433-445.DOI:https://doi.org/10.1145/2998181.2998295
[100] ClairWardleandHosseinDerakhshan.2017.InformationDisorder:TowardanInterdisciplinaryFrameworkforResearchandPolicyMaking.CouncilofEuropeReport.
[101] JenWeedon,WilliamNulandandAlexStamos.2017.InformationOperationsandFacebook.(Apr.2017)RetrievedMarch17th,2019fromhttps://fbnewsroomus.files.wordpress.com/2017/04/facebook-and-information-operations-v1.pdf
[102] TomWilson,KaitlynZhou,andKateStarbird.2018.AssemblingStrategicNarratives:InformationOperationsasCollaborativeWorkwithinanOnlineCommunity.Proc.ACMHum.-Comput.Interact.2,CSCW,Article183(November2018),26pages.DOI:https://doi.org/10.1145/3274452
[103] SamuelC.WoolleyandPhilipN.Howard.2016.Politicalcommunication,computationalpropaganda,andautonomousagents:Introduction.InternationalJournalofCommunication10(2016).
[104] VolkerWulf,KaoruMisaki,MeryemAtam,andDavidRandall.2013.'Ontheground'inSidiBouzid:InvestigatingSocialMediaUseDuringtheTunisianRevolution.InProceedingsofthe2013ACMConferenceonComputerSupportedCooperativeWorkandSocialComputing(CSCW'13).ACM,NewYork,USA,1409-1418.DOI:https://doi.org/10.1145/2441776.2441935
[105] SavvasZannettou,TristanCaulfield,EmilianoDeCristofaro,MichaelSirivianos,GianlucaStringhini,andJeremyBlackburn.2018.DisinformationWarfare:UnderstandingState-SponsoredTrollsonTwitterandTheirInfluenceontheWeb.arXiv:1801.09288.Retrievedfromhttps://arxiv.org/abs/1801.09288
[106] YouTube.2019.OurOngoingWorktoTackleHate.OfficialBlog.(June2019).RetrievedJune26,2019fromhttps://youtube.googleblog.com/2019/06/our-ongoing-work-to-tackle-hate.html
[107] MarkZuckerberg.2019.MarkZuckerberg:TheInternetneedsnewrules.Let’sstartinthesefourareas.(March2019).RetrievedApril3,2019fromhttps://www.washingtonpost.com/opinions/mark-zuckerberg-the-internet-needs-new-rules-lets-start-in-these-four-areas/2019/03/29/9e6f0504-521a-11e9-a3f7-78b7525a8d5f_story.html
Related Documents
