Greedy Algorithms and Heuristics
Optimization problems, Greedy Algorithms, Optimal Substructure and Greedy choice
Learning & Development Teamhttp://academy.telerik.com
Telerik Software Academy
Table of Contents
1. Optimization Problems
2. Greedy Algorithms and Failure Cases
3. Optimal Greedy Algorithms
Optimal Substructure and Greedy
Choice
Demo: Proving Optimality of a
Greedy Solution
4. The Set Cover Problem
5. Notable Greedy algorithms
2
Optimization Problems
Not "just" Looking for a Solution
Optimization Problems Finding the best solution of a problem From all solution candidates
i.e. most optimal solution candidate
Not just any solution
Two categories – based on variables Continuous – variable values are
continuous
Discrete – a.k.a. Combinatorial 4
Optimization Problems Importance of optimization problems Optimal solutions reduce cost
Optimal solutions are more likely to be realistically possible
E.g. finding any route between to a city In theory enables getting there
In practice it might be too long to travel
Finding the shortest route is much better
Greedy AlgorithmsPicking Locally Best Solution
Greedy Algorithms Greedy algorithms are a category of algorithms Can solve some optimization
problems
Usually more efficient than all other algorithms For the same problems
Greedy algorithms pick The best solution
From their current position & point of view
i.e. they make local solutions
Greedy Algorithms: Example
Playing against someone, alternating turns
Per turn, you can take up to three coins
Your goal is to have as much coins as possible
Greedy Algorithms: Example
Things to notice in the way you played to winAlways take the max number of coins
i.e. make the current optimal solution
You don't consider what the other player does
You don't consider your actions' consequences
The greedy algorithm works optimally here
Greedy Algorithms A greedy algorithm solves a problem in steps At each step
Algorithm picks the best action available
Best action is determined regardless of future actions or states
i.e. greedy algorithms assume Always choosing a local optimum
Leads to the global optimum10
Greedy Algorithms Main components of a Greedy algorithm A candidate set
A selection function
A feasibility function
An objective function and a solution function
Basically, a greedy algorithms have A set of possible actions
A way to pick the best action at a time
A way to determine the solution is reached
Greedy Algorithms Greedy algorithms exploit problem structure E.g. where the solution is the sum
of the optimal solutions to subproblems
E.g. where greedy choices don't lead to bad overall positions
Many currency systems' denomination units are suited to greedy processing Ironic, isn't it?
Greedy Algorithms: Example
Consider the US currency denominations
Problem: gather a sum of money, using the least possible number of bills/coins Suppose you have infinite supplies
of each denomination
1¢
5¢
10¢
25¢
50¢
1$Sum to make: 1.29$
1$
25¢
1¢
1¢
1¢ 1¢
Greedy Algorithms: Example
Greedy algorithm to make a sum With minimum number of coins
Given the US currency system This will work for most 1-2-5 series
based systemsWe want to achieve the sum SWe start with the sum Q = 0We have a set of coins C = { 1, 5, 10 … }At each step• pick the largest value V from C
such that Q + V less than or equal to S
• Increase Q by V //i.e. add a coin of value VRepeat until Q == S//the number of repetitions is the number of needed coins
Greedy for Sum of CoinsLive Demo
Greedy Failure Cases Greedy algorithms are often not optimal Even can reach the unique worst
possible solutions for some problems
Example: Largest sum path (starting at top)
Example: Coin denominations 4, 10, 25; Greedy will fail to make the sum 41which is 25 + 4 * 4
Optimal Greedy AlgorithmsOptimal Substructure, Greedy Choice
Property, Proving Optimality of a Greedy Approach
Optimal Greedy Algorithms
Suitable problems for greedy algorithms often have these properties: Greedy choice property
Optimal substructure
Any problem having the above properties Guaranteed to have an optimal
greedy solution
Matroids – way to prove greedy optimality If a problem has the properties of a
matroid, it is guaranteed to have an optimal greedy solution
Optimal Greedy Algorithms
Greedy choice property An optimal solution to the problem
begins with a greedy choice
Subproblems that arise can be solved by consequent choices Also enforced by optimal
substructure
Optimal Greedy Algorithms
Optimal substructure After each greedy choice
The problem remains an optimization problem
Of the same form as the original problem
i.e. the optimal solution to the problem contains optimal solutions to the subproblems
Solving a Problem Optimally with
GreedyGreedy for the Activity Selection
Problem and Proving its Optimality
Proving Greedy Optimality
The Activity Selection Problem (a.k.a. Conference Scheduling Problem)
Given a set of activities S = {a1, a2, … an}
Each with a start & finish time: ai = (si, fi)
Activities are "compatible" if they don't overlap i.e. their intervals do not intersect
What is the maximum-size subset of compatible activities? i.e. which is the largest list of
compatible activities that can be scheduled
Proving Greedy Optimality
The Activity Selection Problem Can have several optimal solutions
In the following case {a1, a4, a8, a11} is optimal
Another optimal is {a2, a4, a9, a11}Index 1 2 3 4 5 6 7 8 9 10 11
Start (si) 1 3 0 5 3 5 6 8 8 2 12
Finish (fi) 4 5 6 7 8 9 10 11 12 13 14
Proving Greedy Optimality
A greedy algorithm for the task:
Greedy characteristic of above algorithm Taking the earliest finish activity
gives more time for other activities
i.e. choose the "maximum remaining time"
• Select activity with the earliest finish from S
• Remove activities in S conflicting with selected
• i.e. non-compatible activities are removed
• Repeat the until no activities remain in S
Proving Greedy Optimality
To prove the discussed greedy is optimal Need to prove the problem has a
greedy choice property
Need to prove the problem has optimal substructure
Observations
1. If there exists an optimal solution, it is a subset of activities from S
2. In any solution, the first activity to start is the first to finish
Proving Greedy Optimality
Let A be an optimal solution (subset of S) Sort activities in A by finish time.
Let k be the index of the earliest activity in A
If k = 1 => A begins with a greedy choice
If k != 1 => Let B = (A – {k}) + {1}. Prove B is
optimal:
f1 <= fk (from sorting and k > 1) => f1 doesn't overlap any activity in B, so B is a solution
B has the same size (number of activities) as A
Hence, B is also optimal
Proving Greedy Optimality
So far we proved that: A solution starting with a greedy
choice exists
The greedy choice solution is also optimal
Hence we proved the problem exhibits the greedy choice property There exists an optimal solution,
starting with a greedy choice
Now, we need to prove the problem has optimal substructure
Proving Greedy Optimality
We have selected activity 1 (greedy 1st choice) Thus, we reduced to the same
problem form Without activities in S which
intersect activity 1
If A is optimal to S, then: A' = A – {1} is optimal to S' = all activities of S which start after f1
Can A' be non-optimal to S' ? If exists B' with more activities than A' (from S')
Adding activity 1 to B' gives B with more activities (from S) than A -> contradiction
Proving Greedy Optimality
We just proved the problem has optimal substructure Each greedy choice leads us to a
problem of the same form
The new problem's solution is a subset of the initial problem's solution
i.e. all local solutions joined form the global optimal solution
We have proven both properties, so our greedy algorithm is optimal
Activity Selection ProblemLive Demo
The Set Cover Problem
Using Greedy for Approximation
Set Cover Problem Greedy algorithms can sometimes find optimal solutions This is not their only application
There exist problems, for which An optimal solution is to complex to
find
i.e. calculating an optimal solution is infeasible
Sometimes greedy algorithms provide good approximations of the optimal result
Set Cover Problem The Set Cover Problem (SCP) is such a problem
SCP formulation: Given a set {1,2,…,m} called "the
Universe" (U)
And a set S{{…},…} of n sets whose union = U
Find the smallest subset of S, the union of which = U (if it exists)
So, we have a target set, and a set of sets with repeating elements (i.e. redundant elements) How do we find the smallest number
of sets, which in union make the target set
Set Cover Problem The SCP turns out very complex
The optimal solution is NP-complete
i.e. infeasible for calculations (unless P = NP)
However, relatively good solutions can be achieved through a greedy approach: At each step, pick the set
containing the largest number of uncovered elements
Notable Greedy Algorithms
Several Common Greedy Algorithms
Notable Greedy Algorithms
Dijkstra's algorithm for finding the shortest path between two vertices in a weighted graph (with no
negative cycles)
At each step, of all reached edges, pick: the one that, along with the path to
it, constitutes a minimal sum
The algorithms is proven optimal Immediately after it reaches a
vertex, the path generated to it is guaranteed to be optimal i.e. no need to traverse all vertices
Notable Greedy Algorithms
Prim and Kruskal's algorithms for a minimum spanning tree (MST) are greedy algorithms Prim:
pick the smallest edge, not in the MST so far
Kruskal: pick the smallest edge, connecting
two vertices, not in the same tree
Both algorithms have the same complexity
Notable Greedy Algorithms
The prefix tree generation algorithm in Huffman coding is greedy Greedy: pick the
two smallest-value leaves/nodes and combine them
Left move: 0,Right move: 1
46 15
27
54
100
0
0
0
0
0
1
1
11
1 A: 00B: 100C: 01D: 1010E: 11F: 1011
Frequency % 22 12 24 6 27 9
Symbol A B C D E F
Huffman CodingLive Demo
форум програмиране, форум уеб дизайнкурсове и уроци по програмиране, уеб дизайн – безплатно
програмиране за деца – безплатни курсове и уроцибезплатен SEO курс - оптимизация за търсачки
уроци по уеб дизайн, HTML, CSS, JavaScript, Photoshop
уроци по програмиране и уеб дизайн за ученициASP.NET MVC курс – HTML, SQL, C#, .NET, ASP.NET MVC
безплатен курс "Разработка на софтуер в cloud среда"
BG Coder - онлайн състезателна система - online judge
курсове и уроци по програмиране, книги – безплатно от Наков
безплатен курс "Качествен програмен код"
алго академия – състезателно програмиране, състезания
ASP.NET курс - уеб програмиране, бази данни, C#, .NET, ASP.NETкурсове и уроци по програмиране – Телерик академия
курс мобилни приложения с iPhone, Android, WP7, PhoneGap
free C# book, безплатна книга C#, книга Java, книга C#Дончо Минков - сайт за програмиранеНиколай Костов - блог за програмиранеC# курс, програмиране, безплатно
?
? ? ??
?? ?
?
?
?
??
?
?
? ?
Questions?
?
Greedy Algorithms
http://algoacademy.telerik.com
References & Further Reading
The following materials were very helpful in the making of this lecture and we recommend them for further reading:
Lecture @ Boston University (Shang-Hua Teng) http://www.cs.bu.edu/~
steng/teaching/Fall2003/lectures/lecture7.ppt
Lecture @ University of Pennsylvania http://www.cis.upenn.edu/~
matuszek/cit594-2005/Lectures/36-greedy.ppt
Book on Algorithms @ Berkeley University
http://www.cs.berkeley.edu/~vazirani/algorithms/chap5.pdf