A New Approach to Bot Detection

NSLab, RIIT, Tsinghua Univ

A New Approach to Bot Detection:Striking the Balance Between Precision and Recall

Fred Morstatter et al.Presented by Jun Yang

2017.3.29

Qing Lyu


What is a bot?

� Social media accounts that are controlled by software.

� Self-declared bots.� Spambots.� Socialbot.

2


What is a bot?

� Innocuous.� Post up-to-date weather, news, historical events, etc.

� Nocuous.� Infiltration.� Influence trending.� Repost or follow specific user.

3


How many?

4

� Over half of the accounts on Twitter are not human.� 5-9% bots produce 24% tweets on Twitter.� 28% of accounts created in 2008 and half of the accounts

created in 2014 have been suspended by Twitter.


Influence

5

� Harvest private users' data.� Sway discussion.� Influence trending hashtags and user statistics.� Lose user experience and trust.� Social media researches.


Bots detection

6

� Classification tasks.


Bots detection

7

� Classification tasks.� Content¾ Different from normal users.¾ URLs.¾ Sentiment.¾ Length.¾ Similarity.¾ Original tweet.


Bots detection

8

� Classification tasks.� User profile¾ Automatically generated accounts with detectable patterns.¾ E-mail addresses.¾ Creation times.¾ Life time.¾ Screen name and verified name.¾ Human typing.


Bots detection

9

� Classification tasks.� Activities¾ Request frequency.¾ IP addresses.¾ Multiple login location.


Bots detection

10

� Classification tasks.� Network structure and connection¾ Mass following and unfollowing behaviors.¾ Statistical and structural features.


Ground Truth Acquirements

11

� Manual annotation� Suspended users list.� Honeypots.


Precision vs Recall

12

� An undetected bot vs an angry user?


Contribution

13

� Two labeled datasets by different techniques.� Textual features by LDA.� Modified approach for higher Recall and F1.


Dataset

14

� Lybya� Querying keywords of Arab Spring.� Collect accounts from 2011.2 to 2013.2� Check whether suspended or removed in 2015.2� 7.5% accounts as bots.


Dataset

15

� Arabic Honeypot� Random tweet or retweet Arabic phrases.� Measures to avoid suspension.� Collect human users that tweet same phrases.� Balanced dataset.


Dataset

16

� Arabic Honeypot� Random tweet or retweet Arabic phrases.� Measures to avoid suspension.� Collect human users that tweet same phrases.� Balanced dataset.


Baselines

17

� Heuristics� Retweet fraction.� Average tweet length.� URLs fraction.� Average time interval.


LDA

18


AdaBoost

19

� Different weak classifiers will focus on different bots.


BoostOR

20

� A modified AdaBoost algorithm to improve Recall.


BoostOR

21

� A modified AdaBoost algorithm to improve Recall.


Discussion of F1-score

22

� Balanced test set.

� Positive:Negative = 3:1

� Positive:Negative = 1:3

Precision Recall F1C1 90.00% 70.00% 78.75%C2 70.00% 90.00% 78.75%




Number of topics

23


Thank you!

Questions?

24

A New Approach to Bot Detection

Documents