Constraint Based Synthesis for Beginners

Constraint Based Synthesis for Beginners

PSY 2012

Armando Solar-Lezama

CAP View of Synthesis

Synthesis Methodology Code

A different approach

Synthesis Methodology

CodeDomain Specific Tool

EXHIBIT A: SYNTHESIS OF SQLWith Alvin Cheung and Sam Madden

Motivation•It turns out SQL is challenging to learn

Who would have thought?

•Frameworks simplify program/DB interface

You can access the DB without using SQL Which can lead to some interesting code...

Examples: Explicit Selectpublic Set<Project> getUnfinishedProjects() { Set<Project> unfinishedP = new HashSet<Project>(); List<Project> projects = this.projectDao.getAllProjects();

for (Project project : projects) { if (!(project.getIsFinished())) { unfinishedP.add(project); } } return unfinishedP; }

Get list of projects from the DB

Select unfinished projects

SELECT * FROM Projects WHERE isFinished=FALSE

Examples: Explicit Selectpublic List<WilosUser> getRoleUser(){ List<WilosUser> listUser = new ArrayList<WilosUser>(); List<WilosUser> user= this.userDao.getUsers(); List<Role> role = this.roleDao.getRoles(); for(int i = 0; i < user.size(); i ++){ for(int a = 0; a < role.size(); a++){ if(user.get(i).getRole_id(). equals(role.get(a).getRole_id())){ WilosUser userok = user.get(i); listUser.add(userok); } } return listUser;}

Find users in the Roles list and add them to the output

Start with users and

roles

SELECT u FROM users u, roles r WHERE u.roleId == r.id

Why is this so bad?•These can be performance bottlenecks

Where performance matters, people write SQL

•More controversial arguments can be made

Is this a synthesis problem?•It is

SynthesizerImperative code with loop nests

Equivalent Formula in Relational Algebra

Domain knowledge

about relational algebra

Framing the problemList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}

Precondition: truePostcondition:

Framing the problem•We want to synthesize a post-condition!

•Challenges Defining the language Solving the synthesis problem Generating code

Defining the language•Requirements

Should make reasoning about the program easy

Should have expressiveness comparable to SQL

Language•Just like relational algebra...•...but with lists rather than sets

The program is written in terms of lists, not sets

• •

InvariantsList getUsersWithRoles () { List users = getUsersFromDB(); List roles = getRolesFromDB(); List results = []; int i = j = 0; while (i < users.size()) { while (j < roles.size()) { if (users[i].roleId == roles[j].id) results.add(users[i]); } }return results;}

Verification•Easy if you know the invariants

Z3 can do it in seconds

•All you need are a few axioms:

:...

Synthesis•Synthesis works best in a finite domain

•Invariant generation can be framed as a sketch

1) Model operations in terms of their source code

2) Construct the verification condition leaving invariant and post condition as unknowns

3) Create a sketch for unknowns4) Solve!

Results so far•We have tried this with a dozen loop nests from real open source projects

•All solve in less than 7 minutes Slow, but not very optimized.

EXHIBIT B: AUTOMATED GRADINGWith Rishabh Singh and Sumit Gulwani

The real software problem•The Software Quality problem is a symptom

•The real problem:The demand for software in our society far

exceeds the supply of people skilled enough to produce it

Three pronged attack

Make programmers more productive Make programming more accessible Reduce the cost of training the next generation

Grading Programming Assignments•Test-cases based grading

No precise correctness correlation No student tailored feedback

•Manual grading by TAs Error-prone Time consuming Expensive

•Manual grading will not scale to 100K students

using System;public class Program {public static int[] Puzzle(int[] a) {

int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){

b[count] = a[i];count++;

}return b;

} }

22

Buggy Program for Array Reverse

6:28::50 AM


int[] b = new int[a.Length];int count = 0;for(int i=a.Length-1; i < a.Length-1; i--){


}return b;

} }

23


6:32::01 AM




}return b;

} }

24


6:32::32 AM

No change! Sign of Frustation?


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){


}return b;

} }

25


6:33::19 AM


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);


}return b;

} }

26


6:33::55 AM

Same as initial attempt except Console.Writeline!


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i < a.Length; i--){ Console.Writeline(i);


}return b;

} }

27


6:34::06 AM

No change! Sign of Frustation?


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i <= a.Length; i--){ Console.Writeline(i);


}return b;

} }

28


6:34::56 AM

The student has tried this before!




}return b;

} }

29


6:36::24 AM

Same as initial attempt!




}return b;

} }

30


6:37::39 AM

The student has tried this before!


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i > 0; i--){


}return b;

} }

31


6:38::11 AM

Almost correct! (a[i-1] instead of a[i] in loop body)


int[] b = new int[a.Length];int count = 0;for(int i=a.Length; i >= 0; i--){


}return b;

} }

32


6:38::44 AM

Student going in wrong direction!




}return b;

} }

33


6:39::33 AM

Back to bigger error!




}return b;

} }

34


6:39::45 AM

No change! Frustation!




}return b;

} }

35


6:40::27 AM

No change! More Frustation!!




}return b;

} }

36


6:40::57 AM

No change! Too Frustated now!!! Gives up.

AutoGrader•Automate grading

Find semantic errors Feedback to fix them

Students make similar mistakes

Is this a synthesis problem?•It is

SynthesizerBuggy implementation

Corrections for student program

Error ModelReference Solution

Array Reverse

i = 1

i <= a.Length

Challenge 1: Different Algorithms

Challenge 2: Scalability

1010 different possible candidate corrections

Our Approach•Use data of previous student solutions

•Correction rules based on observed errors

•Create a set of candidate solutions using rules and find closest correct solution.

Aren’t rewrite systems hard?•Can teachers really write rewrite rules?

•Angelic non-determinism helps Ambiguities and redundancies no longer matter

Results: Problems Fixed

F1 F2 F3 F4 F50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Problems fixed

Array ReversePalindromeMaxFactorialisIncreasingSort

Error Models

Frac

tion

of

Prob

lem

s fix

ed

Results: Performance

F1 F2 F3 F4 F50

5

10

15

20

25

30

35

Performance with Error Models

reversepalindromemaxisIncreasingfactorialsort

Error Models

Run

ning

Tim

e (i

n s)

Results: Generalization

palindrome array max isIncreasing sort0

10

20

30

40

50

60

70

80

Generalization of Error models

Specific Error modelArray reverse error model

Num

ber

of fi

xes

Feedback for Tutoring System

OK for Grading,

But not ideal for teaching

Broad research agenda ahead•Transformative for students in under-funded schools

Reduce the resources required to support quality instruction Enable “true” distance education for programming courses

•Same technology could be used for automatic tutoring Identify errors stemming from deep misconceptions (e.g. not understanding difference in values vs. references)

Synthesize small examples that make misconceptions explicit

48

Sketch Tutorial on Saturday

http://bit.ly/sketch2012

Constraint Based Synthesis for Beginners

Documents