GENERATING TESTS FOR MOBILE APPS FROM NATURAL …€¦ · This is fu*king annoying. Fix it already! BEEP BEEP Reviews Classifier IT FU*KING CRASHES!!! The app still crashes every

GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG

REPORTS AND APP REVIEWS

Partially supported by: NSF, IBM, MSR, and Google

Alessandro (Alex) OrsoSchool of Computer Science – College of Computing

Georgia Institute of Technology

Joint work with Marcelo d’Amorim and Paola Spoletini

SNEAK PREVIEW

SNEAK PREVIEW

GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG

REPORTS AND APP REVIEWS

Partially supported by: NSF, IBM, MSR, and Google

Alessandro (Alex) OrsoSchool of Computer Science – College of Computing

Georgia Institute of Technology

Joint work with Marcelo d’Amorim and Paola Spoletini

SNEAK PREVIEW

GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG

REPORTS AND APP REVIEWS

Partially supported by: NSF, IBM, MSR, and Google

Alessandro (Alex) OrsoSchool of Computer Science – College of Computing

Georgia Institute of Technology

Joint work with Marcelo d’Amorim and Paola Spoletini

Field failures are

unavoidable!

SNEAK PREVIEW

GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG

REPORTS AND APP REVIEWS

Partially supported by: NSF, IBM, MSR, and Google

Alessandro (Alex) OrsoSchool of Computer Science – College of Computing

Georgia Institute of Technology

Joint work with Marcelo d’Amorim and Paola Spoletini

Field failures are

unavoidable!

TYPICAL DEBUGGING PROCESS

Very hard to(1) reproduce(2) debug

BugRepository

TYPICAL DEBUGGING PROCESS

Very hard to(1) reproduce(2) debug

BugRepository

OVERARCHING GOAL: help developers (1) investigate field failures,

(2) understand their causes, and(3) eliminate such causes.

Recent survey of Apache, Eclipse, and Mozilla developers:

Information on how to reproduce field failures is the most valuable, and difficult to obtain, piece of information for investigating such failures.[Zimmermann10]

OUR WORK SO FARRecording and replaying executions[icsm 2007, icse 2007]

Input anonymization [icse 2011]

Input minimization [woda 2006, icse 2007]

✘

Mimicking field failures [icse 2012, ase 2013, icst 2014]

Locating and explaining field failures [issta 2012, issta 2013, ASE 2014, TOSEM 2015, TR]

MIMICKING FIELD FAILURESUser run (R) Relevant events

(breadcrumbs)Mimicked run (R’)

MIMICKING FIELD FAILURESUser run (R) Relevant events

(breadcrumbs)Mimicked run (R’)

Twitter crashes when I: 1) Open the app 2) Select a tweet with an image 3) Retweet the selected tweet

MIMICKING FIELD FAILURESUser run (R) Relevant events

(breadcrumbs)Mimicked run (R’)

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

OVERALL VISION

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Reviews Classifier

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

App Test Generator

App Analyzer

Code Analyzer

GUI Analyzer

Bug Reports

Test Steps

App StoreReviews

App Code

App GUI

Domain Knowledge Actions

DictionaryTest Cases

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Reviews Classifier

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

App Test Generator

App Analyzer

Code Analyzer

GUI Analyzer

Bug Reports

Test Steps

App StoreReviews

App Code

App GUI

Domain Knowledge Actions

DictionaryTest Cases

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Reviews Classifier

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Reports

App StoreReviews

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

Bug Report Miner

Bug Reports

Test Steps

IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!

BEEP

BEEP

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

retweet

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

retweet

like

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

notifications

retweet

like

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

notifications

retweet

like

image

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

notifications

retweet

like

image

<action1> <parameter*> <trigger><action2> <parameter*> <trigger>…

App Analyzer

Code Analyzer

GUI AnalyzerTest Steps

App Code

App GUI

Domain Knowledge Actions

Dictionary

reply

notifications

retweet

like

image

<action1> <parameter*> <trigger><action2> <parameter*> <trigger>…

<open twitter> <> <start com.twittwer. app.twitter><retweet> <current tweet> <click >…

App Test Generator

Test Steps

ActionsDictionary

Test Cases

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…

App Test Generator

Test Steps

ActionsDictionary

Test Cases

1) Open twitter2) Select tweet with image3) Retweet➡ Crash

<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…

CURRENT AND FUTURE WORK

CURRENT AND FUTURE WORKCURRENT AND FUTURE WORK

CURRENT AND FUTURE WORKCURRENT AND FUTURE WORK

• Review Classifier : reuse• Bug Report Miner: very preliminary version• App Analyzer : previous work on feature identification and

cross-platform issues• App Test Generator: previous work on Android testing

• Is it feasible? What are the metrics of success?• What technologies are best suited for the tasks involved? Can

we use them in a black-box fashion?• Can we leverage multiple similar reviews/bug reports?• Can we do this interactively (as the users enter their reviews)?

OPEN QUESTIONS/CHALLENGES