Page 1
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.1
Introduction to Algorithms6.046J/18.401J
LECTURE 16Greedy Algorithms (and
Graphs)• Graph representation• Minimum spanning trees• Optimal substructure• Greedy choice• Prim’s greedy MST
algorithmProf. Charles E. Leiserson
Page 2
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.2
Graphs (review)Definition. A directed graph (digraph)G = (V, E) is an ordered pair consisting of• a set V of vertices (singular: vertex),• a set E ⊆ V × V of edges.In an undirected graph G = (V, E), the edge set E consists of unordered pairs of vertices.In either case, we have |E | = O(V 2). Moreover, if G is connected, then |E | ≥ |V | – 1, which implies that lg |E | = Θ(lgV). (Review CLRS, Appendix B.)
Page 3
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.3
Adjacency-matrix representation
The adjacency matrix of a graph G = (V, E), where V = {1, 2, …, n}, is the matrix A[1 . . n, 1 . . n]given by
A[i, j] = 1 if (i, j) ∈ E,0 if (i, j) ∉ E.
Page 4
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.4
Adjacency-matrix representation
The adjacency matrix of a graph G = (V, E), where V = {1, 2, …, n}, is the matrix A[1 . . n, 1 . . n]given by
A[i, j] = 1 if (i, j) ∈ E,0 if (i, j) ∉ E.
22 11
33 44
A 1 2 3 41234
0 1 1 00 0 1 00 0 0 00 0 1 0
Θ(V 2) storage ⇒ denserepresentation.
Page 5
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.5
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
Adj[1] = {2, 3}Adj[2] = {3}Adj[3] = {}Adj[4] = {3}
22 11
33 44
Page 6
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.6
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
Adj[1] = {2, 3}Adj[2] = {3}Adj[3] = {}Adj[4] = {3}
22 11
33 44
For undirected graphs, |Adj[v] | = degree(v).For digraphs, |Adj[v] | = out-degree(v).
Page 7
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.7
Adjacency-list representationAn adjacency list of a vertex v ∈ V is the list Adj[v]of vertices adjacent to v.
Adj[1] = {2, 3}Adj[2] = {3}Adj[3] = {}Adj[4] = {3}
22 11
33 44
For undirected graphs, |Adj[v] | = degree(v).For digraphs, |Adj[v] | = out-degree(v).Handshaking Lemma: ∑v∈V = 2 |E | for undirected graphs ⇒ adjacency lists use Θ(V + E) storage —a sparse representation (for either type of graph).
Page 8
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.8
Minimum spanning trees
Input: A connected, undirected graph G = (V, E)with weight function w : E → R.• For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
Page 9
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.9
Minimum spanning trees
Input: A connected, undirected graph G = (V, E)with weight function w : E → R.• For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
∑∈
=Tvu
vuwTw),(
),()( .
Output: A spanning tree T — a tree that connects all vertices — of minimum weight:
Page 10
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.10
Example of MST
6 125
14
3
8
10
15
9
7
Page 11
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.11
Example of MST
6 125
14
3
8
10
15
9
7
Page 12
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.12
Optimal substructureMST T:
(Other edges of Gare not shown.)
Page 13
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.13
Optimal substructure
u
vRemove any edge (u, v) ∈ T.
(Other edges of Gare not shown.)
MST T:
Page 14
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.14
Optimal substructure
u
vRemove any edge (u, v) ∈ T.
MST T: (Other edges of Gare not shown.)
Page 15
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.15
u
vRemove any edge (u, v) ∈ T. Remove any edge (u, v) ∈ T. Then, T is partitioned into two subtrees T1 and T2.
T1
T2u
v
Optimal substructureMST T:
(Other edges of Gare not shown.)
Page 16
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.16
u
vRemove any edge (u, v) ∈ T. Remove any edge (u, v) ∈ T. Then, T is partitioned into two subtrees T1 and T2.
T1
T2u
v
Optimal substructureMST T:
(Other edges of Gare not shown.)
Theorem. The subtree T1 is an MST of G1 = (V1, E1), the subgraph of G induced by the vertices of T1:
V1 = vertices of T1,E1 = { (x, y) ∈ E : x, y ∈ V1 }.
Similarly for T2.
Page 17
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.17
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = {(u, v)} ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Page 18
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.18
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = {(u, v)} ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Do we also have overlapping subproblems?•Yes.
Page 19
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.19
Proof of optimal substructure
w(T) = w(u, v) + w(T1) + w(T2).Proof. Cut and paste:
If T1′were a lower-weight spanning tree than T1 for G1, then T ′ = {(u, v)} ∪ T1′ ∪ T2 would be a lower-weight spanning tree than T for G.
Great, then dynamic programming may work!•Yes, but MST exhibits another powerful property which leads to an even more efficient algorithm.
Do we also have overlapping subproblems?•Yes.
Page 20
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.20
Hallmark for “greedy”algorithms
Greedy-choice propertyA locally optimal choice
is globally optimal.
Page 21
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.21
Hallmark for “greedy”algorithms
Greedy-choice propertyA locally optimal choice
is globally optimal.
Theorem. Let T be the MST of G = (V, E), and let A ⊆ V. Suppose that (u, v) ∈ E is the least-weight edge connecting A to V – A. Then, (u, v) ∈ T.
Page 22
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.22
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
u
v
(u, v) = least-weight edge connecting A to V – A
T:
Page 23
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.23
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
u
T:
(u, v) = least-weight edge connecting A to V – A
v
Consider the unique simple path from u to v in T.
Page 24
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.24
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
u(u, v) = least-weight edge connecting A to V – A
vT:
Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A.
Page 25
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.25
Proof of theoremProof. Suppose (u, v) ∉ T. Cut and paste.
∈ A∈ V – A
u(u, v) = least-weight edge connecting A to V – A
vT ′:
Consider the unique simple path from u to v in T. Swap (u, v) with the first edge on this path that connects a vertex in A to a vertex in V – A.A lighter-weight spanning tree than T results.
Page 26
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.26
Prim’s algorithmIDEA: Maintain V – A as a priority queue Q. Key each vertex in Q with the weight of the least-weight edge connecting it to a vertex in A.Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v) ⊳ DECREASE-KEY
π[v] ← u
At the end, {(v, π[v])} forms the MST.
Page 27
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.27
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ ∞∞
∞∞ 00
∞∞
∞∞
∞∞
6 125
14
3
8
10
15
9
7
Page 28
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.28
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ ∞∞
∞∞ 00
∞∞
∞∞
∞∞
6 125
14
3
8
10
15
9
7
Page 29
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.29
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ 77
∞∞ 00
1010
∞∞
1515
6 125
14
3
8
10
15
9
7
Page 30
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.30
Example of Prim’s algorithm
∈ A∈ V – A
∞∞
∞∞ 77
∞∞ 00
1010
∞∞
1515
6 125
14
3
8
10
15
9
7
Page 31
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.31
Example of Prim’s algorithm
∈ A∈ V – A
1212
55 77
∞∞ 00
1010
99
1515
6 125
14
3
8
10
15
9
7
Page 32
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.32
Example of Prim’s algorithm
∈ A∈ V – A
1212
55 77
∞∞ 00
1010
99
1515
6 125
14
3
8
10
15
9
7
Page 33
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.33
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 34
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.34
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 35
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.35
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
1414 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 36
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.36
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 37
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.37
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 38
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.38
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 39
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.39
Example of Prim’s algorithm
∈ A∈ V – A
66
55 77
33 00
88
99
1515
6 125
14
3
8
10
15
9
7
Page 40
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.40
Analysis of PrimQ ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Page 41
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.41
Analysis of PrimQ ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Θ(V)total
Page 42
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.42
Analysis of PrimQ ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
Θ(V)total
|V |times
Page 43
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.43
Analysis of PrimQ ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
degree(u)times
|V |times
Θ(V)total
Page 44
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.44
Analysis of Prim
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
degree(u)times
|V |times
Θ(V)total
Page 45
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.45
Analysis of Prim
Handshaking Lemma ⇒Θ(E) implicit DECREASE-KEY’s.
Q ← Vkey[v] ←∞ for all v ∈ Vkey[s] ← 0 for some arbitrary s ∈ Vwhile Q ≠ ∅
do u ← EXTRACT-MIN(Q)for each v ∈ Adj[u]
do if v ∈ Q and w(u, v) < key[v]then key[v] ← w(u, v)
π[v] ← u
degree(u)times
|V |times
Θ(V)total
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Page 46
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.46
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
Page 47
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.47
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
TotalQ T TDECREASE-KEYEXTRACT-MIN
Page 48
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.48
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
TotalQ T TDECREASE-KEYEXTRACT-MIN
array O(V) O(1) O(V2)
Page 49
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.49
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
TotalQ T TDECREASE-KEYEXTRACT-MIN
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Page 50
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.50
Analysis of Prim (continued)
Time = Θ(V)·TEXTRACT-MIN + Θ(E)·TDECREASE-KEY
TotalQ T TDECREASE-KEYEXTRACT-MIN
array O(V) O(1) O(V2)binary heap O(lg V) O(lg V) O(E lg V)
Fibonacci heap
O(lg V)amortized
O(1)amortized
O(E + V lg V)worst case
Page 51
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.51
MST algorithms
Kruskal’s algorithm (see CLRS):• Uses the disjoint-set data structure (Lecture 10).• Running time = O(E lg V).
Page 52
November 9, 2005 Copyright © 2001-5 by Erik D. Demaine and Charles E. Leiserson L16.52
MST algorithms
Kruskal’s algorithm (see CLRS):• Uses the disjoint-set data structure (Lecture 10).• Running time = O(E lg V).
Best to date:• Karger, Klein, and Tarjan [1993].• Randomized algorithm.• O(V + E) expected time.