Final Notes

MIE 1620: Linear Programming and Network Flows

Final Review Notes

Marlee Vandewouw

November 30, 2014

Contents

1 Introduction to Linear Programming 31.1 General and Standard Form LPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Approximating Convex Optimization Problems using LPs . . . . . . . . . . . . . . . . . . 4

2 Geometry of Linear Programming 42.1 Polyhedra and Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Definitions of a Polyhedrons Corner Point . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.3 Polyhedra and Basic Solutions in Standard Form . . . . . . . . . . . . . . . . . . . . . . . 52.4 Degeneracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Existence and Optimality of Extreme Points . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 The Simplex Method 63.1 Theory Behind the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2 Implementation of the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.3 Finding an Initial BFS for the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Duality 94.1 Fundamental Duality Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94.2 Dual Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3 Duality, Degeneracy and Farkas Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.4 Cones and Extreme Rays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.5 Representation of Polyhedra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Sensitivity Analysis 115.1 Changes in b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.2 Changes in c . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.3 Changes in A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125.4 Add a New Variable/Column . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.5 Add a New Inequality Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135.6 Add a New Equality Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 Network Flow Problems 136.1 Directed and Undirected Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136.2 Graph Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.3 Network Flow Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146.4 Special Cases of the Network Flow Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 156.5 The Network Simplex Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156.6 The Maximum Flow Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.7 Duality in Network Flow Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.8 Network Flow Sensitivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.9 Network Flow Complementary Slackness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.10 The Assignment Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.11 The Shortest Path Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

1

7 Interior Point Methods 207.1 The Primal Path Following Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

8 Large Scale Optimization 218.1 Delayed Column Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218.2 Cutting Plane Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218.3 Dantzig-Wolfe Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

9 Robust Optimization 239.1 General Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239.2 Inverse Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2

Chapter 1

Introduction to Linear Programming

1.1 General and Standard Form LPs

Definition 1.1.1 (LP Problem in General Form). An LP problem in general form is:

minimize cx

subject to aix bi i M1aix bi i M2aix = bi i M3xj 0 j N1xj 0 j N2

Or, more compactly:

minimize cx

subject to Ax b

Definition 1.1.2 (LP Problem in Standard Form). An LP problem in standard form is:

minimize cx

subject to Ax = b

x 0

To transform a general form to a standard form LP, follow the following algorithm.

Algorithm 1 Transform General Form to Standard Form

1: Eliminate the free variables:

xj = x+j x

j

x+j 0xj 0

2: Eliminate inequality constraints:

Ax b Ax s = b, s 0New A matrix: [A I]

New x vector: [x s]

3

1.2 Approximating Convex Optimization Problems using LPs

Definition 1.2.1 (Convex Function). A function f : Rn R is convex if x, y Rn, and [0, 1] wehave:

f (x+ (1 )y) f(x) + (1 )f(y)

Example 1.2.1 (Residual Problem). Since linear functions are convex, the maximum of a set of linearfunctions is also convex. Let cix + di, i I be a set of linear functions. Let f(x) = maxiI{cix + di}.Then:

min f(x) min maxiI{cix+ di}

minx,z

z

s.t. z cix+ di, i I

Chapter 2

Geometry of Linear Programming

2.1 Polyhedra and Convex Sets

Definition 2.1.1 (Hyperplanes and Halfspaces). Let a be a non-zero vector in Rn. Let b be a scalar.Then:

(a) {x Rn|ax = b} is a hyperplane

(b) {x Rn|ax b} is a halfspace (like a constraint)

Definition 2.1.2 (Polyhedron). A polyhedron is a set of the form {x Rn|Ax b} where A is an mnmatrix and b is a vector in Rm. In other words, a polyhedron is the intersection of a finite number ofhalfspaces of the form {x|aix bi, i I}.

Definition 2.1.3 (Convex Set). A set S is convex if x, y S, and [0, 1], we have:

x+ (1 )y S

Theorem 2.1.4 (Polyhedra and Convexity). A polyhedron P = {x Rn|Ax b} is convex.

Definition 2.1.5 (Convex Combination and Hull). Let x1, . . . , xk Rn and 1, . . . , k 0 withki=1 i = 1. Then:

(a) y =k

i=1 ixi is a convex combination of the vectors x1, . . . , xk.

(b) The convex hull of x1, . . . , xk is the set of all possible convex combinations of x1, . . . , xk.

2.2 Definitions of a Polyhedrons Corner Point

Definition 2.2.1 (Extreme Point). x P is an extreme point of a polyhedron P if there do not existy 6= x, z 6= x, y, z P and [0, 1] such that x = y + (1 )z.

4

Definition 2.2.2 (Vertex). x P is a vertex of a polyhedron P if c such that cx < cy, y P , y 6= x.

Definition 2.2.3 (Active Constraint). Any constraint that holds with equality at a particular x is anactive or binding constraint at x.

Definition 2.2.4 (Basic Solution). Consider a polyhedron P Rn defined by linear equality and in-equality constraints. Then:

(a) x is a basic solution if:

(i) all equality constraints are active

(ii) out of all constraints active at x, n of them have linearly independent ai

(b) x is a basic feasible solution (bfs) if x is a basic solution and it is feasible

Theorem 2.2.5 (Equivalence of Corner Point Definitions). Let P be a non-empty polyhedron and letx P . Then, the following are equivalent:

(a) x is a vertex

(b) x is an extreme point

(c) x is a bfs

2.3 Polyhedra and Basic Solutions in Standard Form

Definition 2.3.1 (Polyhedra in Standard Form). In standard form, consider polyhedra of the formP = {x|Ax = b, x 0} where x Rn, b Rm, and A Rmn. Assume the rows of A are linearlyindependent to avoid redundancy, and m n

Definition 2.3.2 (Basic Solutions in Standard Form). In standard form, the constraint Ax = b con-tributes m active constraints, and thus nm of the constraints x 0 must be active. Overall, we needn linearly independent active constraints.

Theorem 2.3.3 (Conditions for Basic Solutions). A vector x Rn is a basic solution (in a standardform polyhedron) if and only if Ax = b and there exist the set of indices B(1), . . . , B(m) such that:

(a) The columns AB(1), . . . , AB(m) are linearly independent

(b) If i / {B(1), . . . , B(m)}, then xi = 0

Definition 2.3.4 (Basic Matrix, Columns, and Variables). Let B be a matrix constructed with thecolumn vectors AB(i), i {B(1), . . . , B(m)}, and xB the column vector consisting of entries xB(i), i {B(1), . . . , B(m)}. Then, B is called a basic matrix, AB(1), . . . , AB(m) are basic columns which form abasis of Rn, and xB is a vector of basic variables (other xj s are nonbasic).

To construct a basic solution, follow the following algorithm.

Algorithm 2 Construct a Basic Solution

1: Pick m indices B(1), . . . , B(m) such that AB(1), . . . , AB(m) are linearly independent.2: Set xj = 0 for j 6= B(1), . . . , B(m).3: Solve Ax = b for unknown xB(1), . . . , xB(m), i.e. BxB = b.

5

2.4 Degeneracy

Definition 2.4.1 (Degenerate Basic Solution). A basic solution x Rn is degenerate if more than nconstraints are active at x.

Definition 2.4.2 (Degenerate Basic Solution in Standard Form Polyhedra). Let P = {x Rn|Ax =b, x 0}, A Rmn. A basic solution x Rn is degenerate if more than nm of the components of xare zero, i.e. if xB(i) = 0 form some i.

2.5 Existence and Optimality of Extreme Points

Definition 2.5.1 (Contains a Line). A polyhedron P Rn contains a line if there exists a vector x Pand a non-zero vector d Rn such that x+ d P for all scalars .

Theorem 2.5.2 (Existence of Extreme Points). Suppose P = {x|Ax b}, where A is an m n matrixwith m n and P is non-empty. Then, the following are equivalent.

(a) P has at least one extreme point

(b) P does not contain a line

(c) There exists n rows of A that are linearly independent

Additionally, the following hold:

(a) Every nonempty bounded polyhedron has at least one bfs

(b) Every nonempty polyhedron in standard form has at least one bfs

Theorem 2.5.3 (Optimality of Extreme Points). Consider:

minxP

cx

If P has at least one extreme point, then either the optimal cost is , or there exists an extreme pointthat is optimal.

Chapter 3

The Simplex MethodFor the following chapter, only standard form problems will be considered where rank(A) = m n.

3.1 Theory Behind the Simplex Method

The objective behind the simplex method is:

(a) If cd 0, d, then terminate.

(b) If d such that cd < 0, move along the direction d.

6

Definition 3.1.1 (Feasible Direction). Let x P . A vector d Rn is a feasible direction at x is thereexists a positive scalar for which x+ d P .

Definition 3.1.2 (Reduced Costs). For each variable xj :

cj = cj cBB1Aj

is the reduced cost. The reduced cost of basic variables is always zero. The reduced cost of nonbasicvariables can be positive, zero, or negative.

Theorem 3.1.3 (Determining Optimality from Reduced Costs). Let x be a bfs with basic matrix B andcorresponding reduced costs c. Then,

(a) If c 0, then x is optimal.

(b) If c < 0, with j a nonbasic index, and x is non-degenerate, then x is not-optimal.

An optimal non-degenerate solution satisfies:

(a) B1b 0 (feasibility - non-negativity x)

(b) c = c cBB1A 0 (non-negativity of reduced cost)

How far along the direction d can we go? Start from the point x and travel to x+ d. If d improves thecost, move as far as possible along the line while remaining feasible. Thus:

= max{ > 0|x+ d P}

We only leave the polyhedron P is some xi becomes negative. There are two cases:

(a) d 0, then x+ d 0, > 0, thus =

(b) di < 0 for some i, then:

xi + di 0 = i xidi

= = min{i|di

3.2 Implementation of the Simplex Method

The following algorithm is revised Simplex Algorithm.

Algorithm 4 Revised Simplex Method

1: Proceed the same as the Simplex Method 1.0, but also keep the matrix B1.2: Maintain the matrix [B1|u], where u is the non-identity row of B1B. Do elementary row operations

so u becomes a matrix with 0s except for the ith element, which becomes 1.

Definition 3.2.1 (Full Tableau Implementation). Maintain and update the following tableau:

cBxB | c1 . . . cj . . . cn xB(1) | | . . . u1 . . . |

... | | . . .... . . . |

xB(l) | B1A1 . . . ul . . . B1An... | | . . .

... . . . |xB(m) | | . . . um . . . |

where cBxB is the negative of the objective function, cj is the entering variable, and the correspondingcolumn is the pivot column, the top row is the zeroth row (the reduced costs), xB(l) is the exiting variable,

and the corresponding row is the pivot row, and ul is the pivot element. ComputexB(i)ui

for all ui > 0.The smallest ratio corresponds to the exiting variable. Perform row operations to make ul = 1, and therest of the uis to equal 0. Perform the same operations to the zeroth row, which will produce the correctresults.

3.3 Finding an Initial BFS for the Simplex Method

Definition 3.3.1 (Auxiliary Problem). Starting with the constraints Ax = b and x 0, introduceartificial variables and form the auxiliary problem:

minimize y1 + . . .+ ym

subject to Axy = b

x, y 0

Use the simplex method to solve the auxiliary problem using the initial bfs x = 0 and y = b. If theoptimal cost is 0, then y = 0, so there exists an x such that Ax = b, x 0. Otherwise, the originalproblem is infeasible. If the auxiliary problem is feasible, if the basic variables are only xs, then dropthe columns corresponding to the ys. Otherwise, drive ys out of the basis until only xs remain.

This leads to the two-phase simplex method:

8

Algorithm 5 Two Phase Simplex Method

1: Phase I:

1: Formulate and solve the auxiliary problem.2: If the optimal cost is greater than zero, the original problem is infeasible. Else, we have a feasible

solution to the original problem.3: Drive out any artificial variables and eliminate redundant constraints (if any), and terminate with

the basic columns of A.

4: Phase II:

1: Start with the bfs x, a basis from the last step of Phase I.2: Compute reduced costs.3: Run the Simplex Method.

Chapter 4

DualityDefinition 4.0.2 (Primal and Dual). For every variable in the primal/dual, there is a constraint in thedual/primal, so the number of constraints in the primal equals the number of variables in the dual, andvice versa. To convert between the two, use the following table:

Primal Dual

min cx max pbs.t. xj ()0 s.t. pAj ()cj

aix = bi pi freexj free p

Aj = cjaix ()bi pi ()0

The dual of the dual is the primal, and equivalent problems have equivalent duals.

4.1 Fundamental Duality Theorems

Theorem 4.1.1 (Weak Duality). If x is the primal feasible solution, and p is a dual feasible solution,then pb cx. Corollary:

(a) If the optimal cost of the primal is , then the dual is infeasible.

(b) If the optimal cost of the dual is +, then the primal is infeasible. item Let x be primal feasible, pbe dual feasible, and cx = pb. Then, x and p are the primal/dual optimal, respectively.

Theorem 4.1.2 (Strong Duality). If a linear program has an optimal solution x, then so does its dual(p) and, moreover, the optimal costs are equal, i.e. cx = (p)b.

The following table can be used to determine the existence of solutions for the primal and dual.PPPPPPPPPDual

PrimalFinite Optimum Infeasible

Finite Optimum 3 7 7

7 7 3Infeasible 7 3 3

9

Theorem 4.1.3 (Complementary Slackness). Let x be primal feasible and p be dual feasible. Then,(x, p) is optimal if and only if:

(a) pi(aix bi) = 0, i

(b) (cj pjAj)xJ = 0, j

4.2 Dual Simplex Method

Definition 4.2.1 (Dual Feasibility). Consider a standard form primal, with a basis B and a primal bfsxB = B

1b 0. B gives rise to a dual basic solution p = cBB1. Dual feasibility is pA c which isequivalent to having non-negative reduced costs, i.e. c 0.

For the dual simplex method, start with a dual feasible solution (and primal infeasible) and iterate untilit is primal feasible while maintaining dual feasibility. It is used when a dual bfs is readily available.

Algorithm 6 Dual Simplex Method

1: Select xB(l) < 0. If non exists, the we have an optimal bfs and the algorithm terminates.

2: Examine the lth row (xB(l), v1, . . . , vn) where vi is the lth component of B1Ai. For each vi < 0,

compute ci|vi| . Let j be such thatcj|vj | = mini:vi 0 for all i then the optimal dual cost

is unbounded, and the algorithm terminates.3: Pivot on vj : Aj enters the basis and AB(l) leaves the basis. Same mechanics (elementary row opera-

tions) as before for the primal simplex method.

4.3 Duality, Degeneracy and Farkas Lemma

Definition 4.3.1 (Dual Degeneracy). Dual degeneracy occurs when more than m reduced costs are equalto 0, which is equivalent to at least one non-basic reduced cost equaling 0.

Theorem 4.3.2 (Corollary of Complementary Slackness). (a) If the primal has a non-degenerate opti-mal solution, the dual has a unique solution.

(b) If the dual has a non-degenerate optimal solution, the primal has a unique solution.

Theorem 4.3.3 (Farkas Lemma). Let A be an m n matrix, and b Rm. Exactly one of the twoalternatives hold:

(a) There exists some x 0 such that Ax = b.

(b) There exists some vector p such that pA 0 and pb < 0.

Theorem 4.3.4 (Farkas Lemma Corollary). Suppose the system Ax b has at least one solutionand let d be some scalar. Then, the following are equivalent:

(a) Every feasible solution to Ax b satisfies cx d.

(b) There exists some vector p > 0 such that pA = c and pb d.

10

4.4 Cones and Extreme Rays

Definition 4.4.1 (Cones). A set C Rn is a cone if 0, and x C, we have that x C.(a) If C is nonempty, then ~0 C always.

(b) P = {x Rn|Ax 0} is a polyhedral cone.

(c) The only possible extreme point of a cone is the zero vector.

(d) A cone with an extreme point is pointed.

Theorem 4.4.2 (Cone Theorem). Let C = {x|Ax 0}. Then, the following are equivalent:(a) The zero vector is an extreme point of C.

(b) C does not contain a line.

(c) n vectors out of a1, . . . , am (rows of A) that are linearly independent.Definition 4.4.3 (Recession Cones). A recession cone at y is the set of all directions d such thaty+d P , P . If P = {x|Ax b}, then the recession cone at y is {d|A(y+d) b, 0}, whichis equivalent to {d|Ad 0}, which is independent of y (similar for equality case).Definition 4.4.4 (Rays). Nonzero elements of the recession cone are called rays.

(a) A non-zero element x of a polyhedral cone is an extreme ray if there are (n 1) linearly independentactive constraints at x.

(b) An extreme ray of a recession cone of P is an extreme ray of P .

Theorem 4.4.5 (Extreme Points and Extreme Rays). Consider minxP cx, where P = {x|Ax b} and

assume P has at least one extreme point. Then, the following are equivalent:

(a) The optimal cost is .

(b) There exists an extreme ray d such that cd < 0

4.5 Representation of Polyhedra

Theorem 4.5.1 (Resolution Theorem). Let P = {x|Ax b} be a nonempty polyhedron with at leastone extreme point. Let x1, . . . , xk be the extreme points and w1, . . . , wr be the complete set of extremerays of P . Let:

Q =ki=1

ixi +

rj=1

jwj | 0,

ki=1

i = 1, j 0

Then, Q = P . The converse holds as well. Also, a non-empty bounded polyhedron is the convex hull ofits extreme points.

Chapter 5

Sensitivity Analysis

5.1 Changes in b

Suppose bi is changed to bi + :

11

(a) Reduced costs are not a function of b, and thus not affected.

(b) Feasibility requires that:

B1(b+ ei) 0 xB + g, where g = B1ei

If satisfies xB(j) + ji 0, j = 1, . . . ,m, the current basis is optimal, where g = [1i, . . . , mi],so:

maxj|ji>0

xB(j)

ji min

j|ji

5.4 Add a New Variable/Column

Suppose a new variable/column is added, making the problem:

minimize cx+ cn+1xn+1

subject to Ax+An+1xn+1 = b

x 0, xn+1 0

(a) Need to check whether the reduced costs satisfy:

cn+1 = cn+1 cBB1An+1 0

If this holds, the current solution is optimal, otherwise, run the primal simplex method.

(b) (x, xn+1) = (x, 0) is a basic feasible solution to the new problem.

5.5 Add a New Inequality Constraint

Suppose a new inequality constraint is added am+1x bm+1. If the old solution x satisfies this constraint,it is still optimal. Otherwise, need to move to a new optimal solution by running the dual simplex method.

5.6 Add a New Equality Constraint

Suppose a new inequality constraint is added am+1x = bm+1. Assume the old x violates this constraint.

Without loss of generality, assume am+1x > bm+1. Construct the auxiliary problem:

minimize cx+Mxn+1

subject to Ax = b

am+1x xn+1 = bm+1x 0, xn+1 0

where M is a big number. Then (x, am+1x b) is a primal bfs, so run the primal simplex number. If

the optimal solution to the auxiliary problem has xn+1 = 0, then we have an optimal solution to the newproblem.

Chapter 6

Network Flow Problems

6.1 Directed and Undirected Graphs

Definition 6.1.1 (Directed Graph). A directed graph G = (N,A) consists of the set of nodes N and setof directed arcs A. An arc is an ordered pair (i, j) where i is the start node and j is the end node.

Definition 6.1.2 (Undirected Graph). An undirected graph G = (N,E) consists of the set of nodes Nand set of edges E joining two nodes.

Definition 6.1.3 (Connected Graph). A graph is connected if for every two distinct nodes i, j N ,there is a path from i to j.

13

6.2 Graph Theorems

Definition 6.2.1 (Tree). An undirected graph G = (N,E) is a tree if it is directed and has no cycles.A node in a tree is a leaf if it is only connected to one node (i.e. has degree 1).

Theorem 6.2.2 (Tree Theorem). (a) Every tree with more than one node has a leaf.

(b) An undirected graph is a tree if and only if it is connected and has |N | 1 edges.

(c) For any two distinct nodes i, j in a tree, there is a unique path between the two.

(d) Adding an edge to a tree will create exactly one cycle.

Definition 6.2.3 (Spanning Tree). Given a connected undirected graph G = (N,E), let E1 E suchthat T = (N,E1) is a tree. Then, T is a spanning tree.

Theorem 6.2.4 (Spanning Tree Theorem). Let G = (N,E) be connected and undirected. Let E0 Ebe a set of edges which do not form a cycle. E0 can be augmented to (E1 E0 E1) so that (N,E1) is aspanning tree.

6.3 Network Flow Formulation

Definition 6.3.1 (Network Flow Problem). A network flow problem (NFP) is:

minimize

(i,j)A

cijfij

subject to bi +

j:(j,i)A

fji =

j:(i,j)A

fij , i N

0 fij uij , (i, j) A

Definition 6.3.2 (Flow Conservation). Corresponding to the first constraint, the total amount of flowinto node i equals the total flow out of i:

iNbi = 0

Definition 6.3.3 (Capacity). Corresponding to the second constraint, the flow through can arc cannotexceed its capacity. If uij =, it is an incapacitated problem.

Definition 6.3.4 (Matrix Notation for the Network Flow Problem). If there are n nodes and m arcs, Ais an n m vector, called the node-arc incidence matrix. The rows equal the number of nodes and thecolumns the number of arcs.

aik =

1 if i is the start node of the kth arc

1 if i is the end node of the kth arc0 otherwise

The matrix notation for flow conservation is Af = b. The sum of all rows of A is the zero vector, and thusthe rows of A are linearly dependent. Form A by throwing away the last row of A, same for b, formingAx = b.

14

Definition 6.3.5 (Circulation). A flow vector that satisfies Af = 0 is a circulation, meaning that thereis no external supply and demand. Consider a cycle C with F being the set of forward arcs in a cycle,and B being the backwards ones. Let the flow vector be:

hcij =

1 if (i, j) F1 if (i, j) B0 otherwise

which satisfies Ahc = 0. The cost of cycle C is:

chC =

(i,j)F

cij

(i,j)B

cij

Given a flow vector f , cycle C and a scalar , the flow vector f + hC is obtained by pushing units offlow around C. The resulting cost change is chC .

6.4 Special Cases of the Network Flow Problem

Definition 6.4.1 (Shortest Path Problem). Find the shortest path between two nodes. This problemhas no capacities, one unit of flow, and distance ij = cij .

Definition 6.4.2 (Maximum Flow Problem). Find the largest amount of flow that can be sent from agiven source to a given sink node. This problem has capacities, and bs = bt, and want to maximize bs.

Definition 6.4.3 (Transportation Problem). Send flow from the sources to the sinks to meet demandat minimal cost. This problem requires

i si =

j dj , and cij are the shipping costs.

Definition 6.4.4 (Assignment Problem). Assigns each source node to a sink node. In this problem, thenumber of sources equals the number of sinks, and it has unit supply and demand.

Definition 6.4.5 (Equivalent Variants of the Network Flow Problem). (a) Every network flow problemcan be reduced to a one source/one sink problem.

(b) Every network flow problem can be reduced to one with no sources or sinks.

6.5 The Network Simplex Algorithm

Definition 6.5.1 (Standard Form Network Flow Problem). Consider the network flow problem:

minimize cf

subject to Af = b

f 0

Then, the standard form network flow problem is:

minimize cf

subject to Af = b

f 0

Definition 6.5.2 (Tree Solution). A flow vector f is a tree solution if it is constructed as follows:

(a) Pick a set T A of n 1 arcs that form a tree (ignore directions)

15

(b) Set fij = 0, (i, j) / T

(c) Determine fij for (i, j) T using Af = b

If a tree solution satisfies f 0 it is a feasible tree solution. To carry out step (c), make n the root, startat the leaves and work towards the root.

Theorem 6.5.3 (Unique Tree Solution). Once a tree T is fixed, a corresponding tree solution is uniquelydetermined. In other words, Af = b, fij = 0, (i, j) / T has a unique solution.

Theorem 6.5.4 (Tree and Basic Solution). A flow vector is a tree solution if and only if it is a basicsolution

Definition 6.5.5 (Change of Basis). Picking a nonbasic variable to enter is equivalent to choosing anarc (i, j) / T , forming a cycle. Push flow around the cycle until a constraint is about to be violated, i.e.choose to push flow equal to:

= min(k,l)B

fkl

The new flow in the cycle is given as:

fkl =

{fkl + if (k, l) Ffkl if (k, l) B

If B is empty, = (extreme ray).

Definition 6.5.6 (Change in Cost). The change in cost is equal to:

(k,l)F

ckl

(k,l)B

ckl

= cijwhere (i, j) is the non-basic entering variable. Defining pn = 0, we get:

cij = cij (pi pj), (i, j) A

Since the reduced costs are zero for basic variables, the pis are easily calculated.

An overview of the network simplex method is provided in the following algorithm.

Algorithm 7 Overview of the Network Simplex Method

1: Start with a tree T and a bfs f . To initialize, assume a network with one source and sink, and addan auxiliary arc from the source to the sink with a high cost M . Let the auxiliary arc be in the treeand all flow goes through it. This is a bfs. Run the NSM using this as the initial bfs.

2: Compute p : pn = 0, pi pj = cij , (i, j T ).3: Compute cij = cij (pi pj), (i, j) / T . If cij , (i, j), then the current bfs is optimal. Else, choose

(i, j) with cij < 0 to enter the basis.4: (i, j) forms a cycle. If all arcs are oriented in the same direction as (i, j), B = , and the optimal

cost is .5: Let = min(k,l)B fkl. Compute the new flow vector, and remove from the basis an arc that now

has zero flow.

Definition 6.5.7 (Network Simplex for Capacitated Problems). Now, consider constraints of the formdij fij uij , (i, j) A. For arcs not in the tree solution, either fij = dij or fij = uij . If cij < 0 andfij = dij , push flow around the cycle in the direction of (i, j). If cij > 0 and fij = uij , push flow aroundthe cycle in the opposite direction of (i, j). The rest of the algorithm is essentially the same.

16

6.6 The Maximum Flow Problem

Definition 6.6.1 (The Maximum Flow Problem). Given capacities uij , no costs, we want to maximizethe flow from the source s to the sink t:

maximize bs

subject to Af = b

bt = bsbi = 0, i 6= s, t0 fij uij , (i, j) A

By setting the costs all to zero, introducing a new infinite capacity arc (t, s) with cost cts = 1, this isequivalent to minimizing

(i,j)A cijfij .

Definition 6.6.2 (Augmented Path). Let f be a feasible flow vector. An augmented path is a path froms to t such that fij < uij for all forward arcs, and fij > 0 for all backward arcs in the path.

Definition 6.6.3 (Labeling). A labeled node i means that there exists an unsaturated path from s ti. A scanned node i means that all neighbours of i have been looked at and have been attempted tobe labeled. If a node i is labeled, and (i, j) is an arc, and fij < uij , then j should be labeled as well.Similarly, if i is labeled and (j, i) is an arc and fji > 0, j should be labeled. This finds an augmentedpath.

Definition 6.6.4 (Maximum Flow that can be Pushed Through a Path). The maximum amount of flowthat can be pushed along a path P is:

(P ) = min{ min(i,j)F

(uij fij), min(i,j)B

fij}

This gives rise to the Ford-Fulkerson algorithm for solving maximum flow problems.

Algorithm 8 Ford-Fulkerson Algorithm

1: Start with a feasible flow f (i.e. f = 0).2: Search for an augmenting path. If non, stop (optimal).3: Else, found P and push (P ) along the path and return to Step 2. If (P ) = , then algorithm

terminated with optimal cost of .

Definition 6.6.5 (Cuts). An s-t cut is a subset S N such that s S and t / S. A capacity of a cutis given by:

C(S) =

{(i,j)A|iS,j /S}

uij

Theorem 6.6.6 (Max-Flow Min-Cut Theorem). (a) If the Ford-Fulkerson algorithm terminates becauseno augmenting path can be found, then the current flow is optimal.

(b) The value of the max-flow equals the value of the minimum capacity cut.

6.7 Duality in Network Flow Problems

Definition 6.7.1 (Dual Network Flow Problem). Consider the standard form network flow problem, andassume the network is connected, uncapacitated and

iN bi = 0. The corresponding dual is:

minimize pb

subject to pA c

p free

17

which is equivalent to:

minimize pb

subject to pA c

p free

Because of the structure of A, dual constraints are of the form:

pi pj cij , ,(i, j) A

6.8 Network Flow Sensitivity Analysis

Suppose bi becomes bi + and bj becomes bj to maintain the flow balance. Assume non-degeneracyand the same basis remains optimal. Then, the change in cost is:

(pi pj)

6.9 Network Flow Complementary Slackness

An optimal solution (f, p) satisfies the complementary slackness conditions:

(a) The first condition is:pi(a

if bi) = 0

If pi 6= 0, then aif = bi for all feasible flows.

(b) The secon condition is:fij(pi pj cij) = 0

If fij > 0, then pi pj = cij . If pi pj < cij , then there exists a way of sending flow from i to j atless cost than using (i, j). Hence, (i, j) should not carry flow, so fij = 0.

6.10 The Assignment Problem

Definition 6.10.1 (The Assignment Problem). Given capacities uij , no costs, we want to maximize theflow from the source s to the sink t:

minimizeni=1

nj=1

cijfij

subject toni=1

fij = 1, j = 1, . . . , n

nj=1

fij = 1, i = 1, . . . , n

fij 0, (i, j) A

There are n people and n projects. The first constraint can be interpreted as for any project j, at mostone person works on it. The second constraint can be interpreted as for any person i, at most one workson one job. If person i cannot do j, then cij =. The flow fij is equal to one if i is assigned to j, andzero otherwise.

18

Definition 6.10.2 (The Bipartite Matching Problem). For this special case of the assignment problem,cij = 0 if and only if person i is compatible with project j. We want to find as many person-projectmatchings as possible. If the optimal cost is zero, there exists a perfect matching.

minimizeni=1

nj=1

cijfij

subject toni=1

fij = 1, j = 1, . . . , n

nj=1

fij = 1, i = 1, . . . , n

which is equivalent to:

maximize

ni=1

ri +

nj=1

pj

subject to ri + pj cij , i, j

Once the pj s are determined, ri can be determined as ri = minj=1,...,m{cij pj} and can be rewrittenaccordingly.

Definition 6.10.3 (Complementary Slackness). The complementary slackness condition can be writtenas:

(a) Flow must be conserved.

(b) If fij > 0, then ri + pj = cij pj cij = maxk{pk cik}

where the second condition can be interpreted as each project k has a reward pk and if peron i is assignedto it, there is a cost cik. The difference between the two is the profit to person i derived from carryingout project k. Thus, it says that each person should be assigned to a most profitable project.

6.11 The Shortest Path Problem

Definition 6.11.1 (The Shortest Path Problem Formulation). Consider a directed graph with cost cij .We want to find the shortest directed path from the origin to the destination. Solve the network flow asa linear program, and if the optimal cost is , there exists a negative cost cycle, otherwise, then anoptimal tree solution exists, which is the tree of shortest paths.

Definition 6.11.2 (Dual of the Shortest Path Problem). A degenerate primal optimal solution impliesthat the dual has a unique optimal solution. Let pi be the cost of sending flow from i to n, which is alsothe length of the shortest path from i to n. If (i, j) T , then pi = pj + cij , otherwise, pi pj + cij .

Definition 6.11.3 (Bellmans Equation). The dual optimal solution p satisfies the following equation:

pi = min{pj + cij}, i = 1, . . . , n 1pn = 0

Definition 6.11.4 (Bellman Ford Algorithm). Let pi(t) be the shortest path from i to n that uses lessthan t arcs. pi(t) = if you cannot do it in less than t arcs. Then, let pi(t + 1) = min{cij + pj(t)}.Keep track of the indices which attain min and then find nodes in the shortest path. The terminationconditions are:

19

(a) If there is no negative cost cycle, then there exists a shortest walk that is also the shortest path withless than n 1 arcs, i.e. pi(n 1) = pi and pi(n) = pi(n 1).

(b) If there exists a negative cost cycle, then pi(t) is for some i.

Definition 6.11.5 (Dijkstras Algorithm). The algorithm can only be used if cij > 0 and if we assumethat cij is defined for all i, j with i 6= n but may be for some parts. The idea is to work backwardsfrom destination nodes.

(a) Find a node l 6= n such that cln cin, i 6= n. Set pl = cln.

(b) For all nodes i 6= l, n, set cin = min{cin, cil + cln}.

(c) Remove node l from the graph and repeat with remaining graph.

Alternatively,

(a) Move back from destination node and label its neighbour.

(b) Pick the smallest one and finalize (move into set of shortest path lengths).

(c) Label neighbours of finalized node and pick the smallest one of the remaining nodes, etc.

Chapter 7

Interior Point Methods

7.1 The Primal Path Following Algorithm

Set up a barrier problem to prevent xj from reaching the boundary xj . Consider the standard formproblem:

minimize cx

subject to Ax = b

x 0

and its corresponding dual:

maximize pb

subject to pA+ s = c

s 0

Define the barrier function as:

B(x) = cx

nj=1

log xj , 0

Then, we want to:

minimize B(x)

subject to Ax = b

20

Solve the barrier problem for successively smaller values of . In particular, as the limit approaches zero,x() x.

Chapter 8

Large Scale Optimization

8.1 Delayed Column Generation

Suppose in the standard form problem, the matrix A has a very large number of columns. All the simplexmethod really needs to do is find a j such that cj < 0 when one exists. Thus, the idea behind delayedcolumn generation is to solve an auxiliary problem: minimize cj . If the optimal value is negative, thenAj enters the basis, otherwise, it is at the optimal solution. The ability to solve this auxiliary problemefficiently depends on the problem structure.The idea is to solve the restricted master problem, which uses the constraints i I {1, . . . , n}. Thisgenerates a solution set (x, p). Then solve the subproblem of minimizing the reduced cost. If the optimalcondition is met, than an optimal solution is found; otherwise, add the appropriate constraint into therestricted master problem.

8.2 Cutting Plane Methods

The cutting plane method recognizes that a primal with many columns is a dual with many rows. It thensolves the relaxed dual problem, for only constraints i I {1, . . . , n}. Suppose p is a solution to therelaxes problem. Then, there are two cases:

(a) p is feasible for the original problem, and thus it is optimal for the original problem.

(b) p is infeasible for the original problem, and thus it must violate one of the original constraints, whichis added to the relaxed problem.

To determine whether an optimal solution to the relaxed problem is feasible for the original problem,minimize ci (p)Ai over all i. If the optimal value is nonnegative, p is feasible and thus optimal to theoriginal problem. Otherwise, the i which causes the value to be negative should be added to the relaxedproblem.

8.3 Dantzig-Wolfe Decomposition

Consider an LP of the form:

minimize c1x1 + c2x2

subject to D1x1 +D2x2 = b0, dimension m0

F1x1 = b1, dimension m1

F2x2 = b2, dimension m2

x1, x2 0, dimension n1, n2, respectively

Let:Pi = {xi 0|Fixi = bi}, i = 1, 2

21

Then, an equivalent problem is:

minimize c1x1 + c2x2

subject to D1x1 +D2x2 = b0

xi Pi, i = 1, 2

Let xji , j Ji be the extreme points of Pi and wki , k Ki be a complete set of extreme rays of Pi. By theresolution theorem, any xi Pi can be represented as the sum:

xi =jJi

jixji +

kKi

ki wki

where ji , ki 0,

jJi

ji = 1, for i = 1, 2. Substituting this in, the constraints can be written as:

jJ1

j1

D1xj110

+ kK1

k1

D1wk100

+ jJ2

j2

D2xj201

+ kK2

k1

D2wk200

=b01

1

Then, give a basis B, p = cBB

1, where p = [q, r1, r2], apply the column generation idea: i.e. examinethe reduced costs for the first subproblem:

For variable j1 : c1xj1 [q, r1, r2][D1x

j1, 1, 0]

= (c1 qD1)xj1 r1

For variable k1 : c1w

k1 [q, r1, r2][D1wk1 , 0, 0] = (c1 qD1)wk1

And solve:

minimize (c1 qD1)x1subject to x1 P1

Do the same for x2. Then, there are three possibilities:

(a) The optimal cost is , i.e. the simplex method finds some extreme ray wk1 such that (c1qD1)wk1