Csise15 codehunt

Crowdsourcing Code and Process via Code Hunt

Tao Xie, Judith Bishop, Nigel Horspool, Nikolai Tillmann, Jonathan de Halleux

University of Illinois at Urbana-Champaign, USA; Microsoft Research, USA; University of Victoria, Canada

secret

https://www.codehunt.com

Testing Tool Educational Gaming

Dynamic Symbolic Execution (Pex)

secret

Pex for Fun/Code Hunt: Interactive Gaming for Teaching and Learning

Support

http://research.microsoft.com/pex/ http://pex4fun.comhttps://www.codehunt.com

http://research.microsoft.com/pex/

Pex is Part of Visual Studio 2015!• As new feature of “Smart Unit Tests”

http://www.visualstudio.com/en-us/news/vs2015-preview-vs#Testing

Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In ASE 2014, Experience Papers.http://web.engr.illinois.edu/~taoxie/publications/ase14-pexexperiences.pdf

Coding Duelshttp://pex4fun.com/

Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.http://web.engr.illinois.edu/~taoxie/publications/icse13see-pex4fun.pdf

Coding Duel Competition @ICSE 2011

http://pexforfun.com/icse2011

Code Hunt: Resigned As Gamehttps://www.codehunt.com/

Gameplay

1. User writes code in browser2. Cloud analyzes code – test cases show differences

As long as there are differences: User must adapt code, repeatWhen they are no more differences: User wins level!

secret

code

test cases

Behind the Scene of Code Hunt

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

9

It’s a game!

• iterative gameplay• adaptive• personalized• no cheating• clear winning criterion

secret

code

test cases

Towards a Course Experience

Code Hunt as Crowdsourcing Platform

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

12

Take 1: Code

Code Hunt as Crowdsourcing Platform

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

13

Take 2: Process class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Code Hunt as Crowdsourcing Platform

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

14

Take 3: Code (w/o secret)class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Passing Player Contributed Implementations

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Judge Metrics

Why Code Hunt as Crowdsourcing Platform?

Why Code Hunt as Crowdsourcing Platform?

Crowdsourcing Scale & Open Data

Coding Duels

1,667,521 clicked 'Ask Pex!'

http://pex4fun.com/

2014 Beauty of Programming Contest

Code Hunt can identify top coders

Code Hunt - the APCS (default) Zone

• Opened in March 2014• 129 problems covering the Advanced Placement

Computer Science course• By August 2014, over 45,000 users started• By now, over 725,000 users

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.101.111.121.131.141.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.80

5000100001500020000250003000035000400004500050000

APCS Zone, First three sectors, 45K to 1K

Sector and Level

Play

ers

Code Hunt Usage

Code Hunt has had several hundred thousands of users since launch in March 2014Stats from Visual Studio Analytics over the period 9/19/14 and 10/20/14 indicate 1,150 users a day

Stickiness (loyalty) is very high

APCS 725,199 ICSE 627 BoPQuali 13,778 BopA 1,017 BopB 1,103 BopSemi 1,164 CSTA 61TEALS 14 BootCamp 30 Laser Sept 12 Polska Oct 344 ImCupJan 542 ImCupDec 222ImCupNov 379 ImCupOct 831ImCupSept 257

Total Try Count

Average Try Count

Max Try Count

Total Solved Users

13374 363 1306 1581

Size of Code Hunt Data (so far)

For ImCupSept257 users x 24 puzzles x approx. 10 tries = about 13,000 programs

#users

Code Hunt Data Released to the Public!http://research.microsoft.com/codehuntcommunity/

https://github.com/Microsoft/Code-Hunt

The currently released data set contains the programs written by students (only) worldwide during a contest over 48 hours. There are approximately 250 users, 24 puzzles and about 13,000 programs. 

Judith Bishop, Nigel Horspool, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. Code Hunt: Experience with Coding Contests at Scale. In ICSE 2015 JSEET.http://web.engr.illinois.edu/~taoxie/publications/icse15jseet-codehunt.pdf

http://research.microsoft.com/chese2015/

Summary: Testing Tool Educational Gaming

DSE/Pex

secret

Pex for Fun/Code Hunt:

Interactive Gaming for

Teaching and Learning

Support

http://pex4fun.comhttps://www.codehunt.com

http://research.microsoft.com/codehuntcommunity/

https://github.com/Microsoft/Code-Hunt

Testing Tool Educational Gaming

secretSupport

http://research.microsoft.com/pex/ http://pex4fun.com

Q & AThank you!

http://research.microsoft.com/codehuntcommunity/ https://github.com/Microsoft/Code-Hunt