1
ResearchUsingTwitterSocialMediaDataTwitterisanextensivesourceofsocialmediadata,andunusualinthatithasbothanextensivedatasharinginfrastructureandanexplicitlypermissiveTermsofService.ThistutorialdocumentshowshowtosocialscientistscanuseTwitterasasourceofresearchdata,withoutanytechnicalbackgroundorprogrammingability.Itshowshowtousecommonlyavailabledatacaptureandanalysistools,andthekindsofresearchmethodsandinvestigationsthattheycansupport.NoteonSoftwareRequiredforthisTutorialTofollowthistutorial,youwillneedtousetheChromebrowsertoaccessTwitter’swebsite.Thedatacapturetool(WebDataRA)isaChromebrowserextensionthatinterpretstheTwitterWebpagesandcapturesthemasspreadsheetdatathatyoucanpasteintoMicrosoftExcel(WindowsorMac).YouwillalsouseGephi,anopensourcenetworkanalysisandvisualisationapplicationwhichyoucandownloadandinstallfromgephi.org.ThisexpectsarecentversionoftheJavaRuntimeEnvironment(JRE)tobeinstalledonyourmachine.
InstallingandUsingWebDataRATheWebDataRAwillcaptureTwitter,FacebookandGoogledatafromabrowserandallowyoutopasteatableofinformationdirectlyintoaspreadsheet.ThisdocumentfocusesonitsusewithTwitter.1) InstalltheWebDataRAbrowserextensionintoChromebyvisiting
bit.ly/WebDataRAinChrome,andclickingontheblue“+AddtoChrome”button.Thesmallgreenicon willappearinthetoprightofthebrowserwindow,nexttotheURLbar.
2) GotoTwitter.comandcreateaTwittersearchordisplayatimeline3) ClickontheWebDataRAicon tostartcollectingtweets.Everyfive
secondsthebrowserwillautomaticallyscrolltothebottomofthepagetomakeTwitterloadthenextbatchofresultsandcopythedataautomaticallytotheclipboard.
4) Whenyouhavecollectedenoughresults,pastethedataintoanExcelspreadsheet.5) UseExceltoanalysedata,orexporttootherprogramssuchasGephiorVoyantforother
kindsofanalysis.
2
OverviewWhenyoupastedatafromWebDataRAyouwillcreatefourtablesinyourspreadsheet,appearingbeloweachother.
Thetweetdata,withauthor,mentions,hashtags,textandcountsofretweets,repliesandlikesbrokenoutinseparatecolumns.
Accountoccurrencesummary,acountofthenumberoftimesthateachTwitteraccountappearsinthedatasetasauthororamention(includingthenumberofretweets).
Countsoftheappearancesofeachhashtag.
Atableofedgesoftheconversationalnetwork,i.e.thenumberoftimeseachpairofaccountscommunicatewitheachother.
Youcanusethisdatainvariousways:(a) Thetweetdata(gray)containsthebasicdataabouteachtweet:whatwassaid,
when,bywhoandtowhom.Youcanusethisdatatoformageneraloverviewofthecommunicationovertimeandidentifythemostsignificanttweets.YoucanalsoexaminespecifictweetsandtheircontextbyreferringbacktotheTwittersiteusingeachtweet’sURL.
(b) Theaccounttable(green)showsyouthemostactivetweeters,themostfrequentrepliers,andthemostretweetedusers.Thiswillhelpyouseethekeyactorsinaconversation,andthemainrolesthattheytake.Youcanfollowupbyclickingontheaccountnamestolookattheaccountbiosandtherelevanttimelinesoftheseactorstounderstandwhethertheyarecorporateaccounts,privateindividuals,botsortrolls.
(c) Thehashtagtable(blue)showsyouthemostfrequentlyusedhashtags.Thiscanhelpyouextendyourdatagatheringtolookformoretweetsrelevanttoyourresearchquestion.
(d) Theedgetablewillhelpyoutoseetheinteractionsbetweenactors,andhelpyoutounderstandgroupingsofactors,andthepatternoftheirinteraction.Isakeyaccountdominatingaconversationandtalkingtomanyothers?Aretheyrespondingorjustbeingpassiverecipientsofmarketingmessages?Isthereagroupofequalshavingabalancedconversationwithequalparticipation?
3
UsingExcelforSimpleQuantitativeAnalysesThefollowingsetoftweets(graytable)comesfromaTwittersearchforthephrase“DigitalDetox”.
TheeasiestwaytoseeanoverviewofaTwittertimelineistocreateaPivotTable.Clickonanygraycell,andchoose“PivotTable”fromtheInsertribbon.InthePivotTablebuilder,drag“Author”fromtheFieldNamepanelintothe“Rows”panel,drag“Timestamp”intothe“Columns”panel,anddrag“Author”(again)intothe“Values”panel(itwillautomaticallyturninto“CountofAuthor”).
Thescreenshotaboveshowsthedatesincludedinthetwittersampleasgreencolumnheadings,andtheaccountsthatauthoredtweetsasrowlabelsalongthelefthandside,orderedbymostprolifictweeters.Thevaluesineachcellarethenumberoftweetsauthoredbyaspecificaccountonaspecificday.Youcanadjusttheformattingforconvenience(Inarrowedthecolumnsandslantedthecolumnheadingsandchangedtheangleofthetextto60degreestofit),usethe“Row
4
Labels”controltosortbytheauthorcount(i.e.thenumberoftweetsanauthorcreated)andshowonlytherowswherethetotalauthorcountisgreaterthanachosenthreshold.Youcanalsouseconditionalformattingtocolourthecellstohighlightthemostextremevalues.AllkindsofsummariesandanalysesarepossibleusingExcelonthisdata,including:
• Showingthedistributionofthetweetsamplethroughtime• Identifyingthemostprolificand/orpopularactors,andshowingtheiractivity
throughtime• Showingtheuseofindividualhashtags(thismightbeusefulinabigconversation,or
onethatevolvesoveralongerperiod)• Comparingtherelativeproportionofcontributionsfromdifferentactors/hashtags
AlloftheseanalyseswillleadontootherquestionsthatcanbeaskedbygoingbacktoTwitter.Theaccountnamesintheaccount“authorandmentions”(green)tableareclickable,andopenthepageoftheaccountprofileinyourdefaultwebbrower.Alternatively,tomakeasetofaccountnamesintoclickablehyperlinksgivingbrowseraccesstotheuser’sTwittertimelineandbio,usethefollowingfunction:=HYPERLINK(CONCATENATE("http://twitter.com/",A9),CONCATENATE("@",A9)) whereA9isanexampleofacelladdressthatcontainsaTwitteraccountnamee.g.lescarr.Followingtheaccounthyperlinksforthemostprolificauthorsinthegreentable,weseethattheyareallcommercialactorstooneextentoranother.Account # BioItsTimeToLogOff 30 TimeToLogOffisthehomeofdigitaldetox.We’respearheadingthe
movementtodisconnectregularlyfromdigitaldevicesandreconnectwiththeworldoffline.Wedothisthroughcollectingfactsontheneedfordigitaldetox,runningcampaignstogeteveryoneofftheirscreensandhostingretreats,eventsandworkshops.
DinnerTableMBA 9 Acommercialorganisationworkingtogethertohelpfamiliesbecomemoreconfident,successful,andself-empowered
SpareFoot 8 Astoragecompany.Wemakeiteasytomoveandstoreyourstuff.Reservestorageforfreeandgetyourmindoutoftheclutter.
CultureEffect 5 AuthorofDigitox:HowtoFindaHealthyBalanceforyourFamily’sDigitalDiet
UsingGephiforNetworkAnalysesToseehowthevariousaccountsinteractwitheachotherasanetwork,copyandpastetheyellowtableintoaseparatespreadsheetandsaveitasaCSVfile(callitedgetable.csvor
3.OneclicktoopenTwitteraccountinbrowser
2.Functionturnsaccountnameintolink
1.Functionusesaccountname
5
similar).Loadupthenetworkvisualisationprogram“Gephi”,andstartanewproject.Inthe“DataLaboratory”,choose“ImportSpreadsheet”andloaduptheCSVdataasanedgetable.Youcanthenapplyavarietyofnetworklayoutalgorithmsinthe“Overview”pane.
Thenetworkvisualisationshowsmanyisolatednodes(accounts)inanouterringandacentralcoremadeupofdifferentgroupsofaccounts.Manyofthesearelooselyconnected“chains”of2-6accountswhereoneaccounthasmentionedanother,whichhasmentionedanotherandsoon.Therearemorecomplicatedsubnetworkcomponentsthatdemonstratemoreactivity,asseenbelow.
Thegreencomponentisdominatedbyasinglecorporateaccount(themostprolificaccountinthissample)whoseroleistopromotetheideaofadigitaldetoxandthat“tweetsat”manyotheraccounts,initiatingcommunicationwiththem.Bycontrast,therednetworkconsistsofalargergroupofteachersandeducationprofessionalswhoalreadyparticipateinalargerprofessionalnetworkwithinTwitter,andwhoarediscussingthetopicofdigitaldetoxwithinthatcontext.ManysummariesandanalysesarepossibleusingGephi’snetworkvisualisationtool:
• Showingtheinteractionofthenetworkactors• Identifyingthecommunitiesandactiveparticipantsubgroupswithinthelarger
sample• Identifyingtherolesofdifferentactorsinthecommunicationsnetwork
6
Pleasenote,thatmanynetworkanalysesintheliteratureoftenfocusontheretweetnetwork,butthatisnotpossiblewithWebDataRA,becausetheidentityoftheretweetingaccountsisnotavailabletotheWebbrowser(theWebUserinterfaceorWebapp).Youhavedataonthenumberofretweets(i.e.thepopularityofthetweet),butnottheaccountsthatretweetedtheoriginalmessage.Formoreinformationonasupplementaryservicebeingdevelopedtofillthisgap,seetheendofthisdocument.
UsingVoyantforTextualAnalysesInthegraytable,copythe“SanitisedText”column.Thiscontainsthetextofallthetexts,butwithalltheTwitterfeatures(@names,#hashtags,URLs)removedtoleaveonlytheEnglishtext.GototheVoyant-Tools.orgwebsite,pastethetextintothetextboxandpressthe“Reveal”button.Youwillseeascreenwithseveralpanelsthathelpyouexplorethetextofthetweetsindifferentways.VoyantToolsisatextualcorpusanalyser.Itconsidersthedatathatyouhaveenteredasasingledocumentwheretheindividualtweetsarelikeindividualsentencesorparagraph.TheWordcloud,Reader,TrendandConcordanceallanalysethetextfromthecollectionoftweets.ClickonawordintheWordcloud,andallitsoccurrenceswillbehighlightedintheReaderpanel,it’sfrequencythroughoutthewholedocument(setoftweets)willbedisplayedintheTrendgraph,anditscontextwillbedisplayedintheConcordance.Thishelpsyoutoinvestigatetheuseoflanguageinthecollectionoftweets,andquicklyunderstandwhatisbeingtalkedaboutandhow.Italsohelpsyoutoseehowthelanguagechangesovertime.Thefirstthingthatthisdisplayshowsisthatthemostcommontermsaredigital,detox,anddigitaldetoxbecausetheywerethesearchterms!Toignorethem,addthemtothestopwordslistbyclickingonthe“Options”iconintheWordcloudpanel’sgreyiconbar.
Whenthemouseentersthebar,youwillseethefollowingiconsappear: .The“Options”iconistheslidingbutton,nexttothe“Help”questionmark.
ToedittheStopwordslist,clickonthe“EditList”buttonandaddthethreeextratermstobeignored,oneperline.
7
Youwillseethatthepanelshavere-analysedthetext,ignoringthetermsthatyouhaveaddedtothestopwordlist.
Othermorecomplexvisualisationsandanalysesareavailabletobeused,includingadvancedMachineLearningalgorithmstoclusterkeywordsandsimplegraphvisualisationstoshowkeywordco-occurrence.
StreamGraph DimensionalReductionsofKeywordAppearance
NetworkofKeywordCo-occurrences
8
AnalysisofTwitterLanguageThewordcloudshowsinvisualformthemostcommonlyusedtermsintheTwittersample.Thisisgreatforanimpressionofthetopics,butamoreusefulsummaryisthe“Terms”panel.Using15ofthetop20wordsinthissample,wemighthypothesisethatadigitaldetoxisaboutspendinglesstimeusingaphoneandtheneedtounplugyourselffromsmartphonesocialmediatechnologyapps–it’sahealthchallengeyoutryforaday,aweekorayear.
Aneasywaytoinvestigatetheuseofeachofthesewordsistoselecttheminthetermpanel,andscrollthroughthein-contextuseinthe“Context”panel.Themostcommonlyusedwordis“time”butit’smainuseisnotinthesenseof“spendingtoomuchtime”or“savingtime”butinthesenseofanopportunity(it’stimeto…orit’stimefor…)andfrequentlyasarhetoricalquestion(Isn’tittimeforadigitaldetox?)FindingtweetsthatspeakfrompersonalexperienceManyofthetweetsseemtobepromotionallifestyletweetsinheadlineform(e.g.“WhyYouShouldDoaDigitalDetox”or“DigitalDetoxBenefits:Areyouaddictedtotechnology?”)Toidentifypersonaltweetsfrompeoplewhohavetriedorarethinkingoftryingadigitaldetox,searchfortweetscontainingthepersonalpronoun“I”.Unfortunately,“I”isoneoftheVoyantToolsstoplistwordsandisignored(astoplistconsistsofthecommonbutlow-informationwordsinatextincludingpronouns,prepositionsandconjunctions).AlthoughitispossibletoeditthestoplistinVoyant,IwillsearchforthestringI<space>orI<apostrophe>inthetextofthetweet(tomatchI,Iam,Ihave,Iwill,I’m,I’ve,I’ll).Usethefollowingformula =OR(NOT(ISERROR(FIND("I ",A1))),NOT(ISERROR(FIND("I'",A1)))) todefineanextracolumninthegraytable,andsortthesetbytheresults(TRUEorFALSE).Only17%ofthetweetsinthiscollectionwereidentifiedaspersonalbythissimplerule.(Note,noteverytweetwiththewordIistalkingfromafirstpersonperspective,andmanytweeterswouldmiss“Ihave”or“Iam”fromthebeginningofatweetortext.)Ofthose17%,justoverhalfwereactivelyplanningadigitaldetox,weredoingadigitaldetoxorhaddoneone.(Theironyoftweetingaboutdoingadigitaldetoxisnotlostonsomecommentators!)
9
Itisthenpossibletofollowupwitheachoftheaccountsthathastweetedaboutanactualperiodofdigitaldetoxthattheyhaveundertaken,toexaminetheirTwitteractivitybefore,duringandafterthedetoxandtoidentifyanyquantitativedifferenceintheiruseoftheplatform.Asanexample,Ichoseasingleaccountthathadannouncedthattheywerestartinga“DigitalDetox”todeprivethemselvesofsocialmediaforaday.Iaccessedallthetweetsofthatuserovertheperiod(usingWebDataRA),andcreateda1-authorpivottable(asabove)toaggregatethetweetstoacountperday,andthenplottedthatdataasascattergraphwith7-daymovingaverage.Thegraphshowsthatalthoughtheaccountdidnottweetonthechosendate(markedwitharedcross)thereisnonoticeablechangeintweetingbehaviouraftertheannounceddetoxday.
AdditionalMethods:SentimentAnalysisSentimentanalysiscanhelpyouidentifypositiveornegativecommentsinyoursample.Thisisapopularmethodinindustry,especiallywithbrandmanagementcompanies.Howeveritisacademicallycontested,anddoesnothaveahighdegreeoftransparencyinthelexicalprocessing. Nevertheless,totryitout,pastethe“SantitisedText”columnintoanonlineservicesuchassentigem.com.Considertowhatextenttheresultsseemaccuratetoyou–howwelldoesitidentifypositiveandnegative‘sentiment’inasentence?Whatkindsofinaccuraciescanyousee?And,mostimportantly,despiteanyshortcomings,doesithelpyoutoidentifyanypointsofinterestinyourdataformorethoroughinvestigation?
7% 11%
22%
23%
25%
12%
PersonalTweetsaboutDigitalDetox
WantToDo
PlanToDo
Doing
Did
Suggestions
Opinions
10
AdditionalMethods:RetweetNetworkAsupportingserviceisbeingdevelopedforWebDataRAtoallowyoutorecovertheretweetnetworkaroundthetweetsthatyouhavecollected.Aspreviouslyexplained,theTwitterwebappdoesnotshowtheretweetsofeachtweet,onlythecountofretweets.However,itispossibletorequestthatdatafromtheTwitterAPI,althoughitislimitedto100retweetsofeachtweet.(Ifatweethasbeenretweetedmorethan100times,thoseextratweetswillnotbeaccessible,assuchthisshouldbeconsideredtobeanapproximationoftherealretweetnetwork.)Toobtaintheretweetdataforyourtweets,selectthetwitteridsforwhichyouwishtofindtheretweets,andpastetheseintotheformathttp://pretend.webdataRA/retweets/.PleasenotethattheTwitterAPIlimitsrequestssuchthateachtwitterIDthatyouprovidewilltake12secondstoprocess,andthefinishedresultmaytakeanhourormore.YoucanpastetheresultingdataintoanExcelspreadsheet,saveitasCSVandimportitintoGephiaspreviouslyexplained.IntheGephinetworkvisualisationbelow,nodesarelargerandcolouredmoreintenselyaccordingtohowmuchothernodesretweetthem(theirroleasauthoritiesinthenetwork),butthenodelabelsaresizedaccordingtowhethertheyretweetorareretweeted.Soyoucanseethatsomeaccountsareveryactivebutnotasoriginatorsofinformation,whereasotheraccountsmaybelessactivebutaremoreinfluential.
11
