Harald Enzinger, Michael Rath Jan 9, 2012...I Optimal solution lies on the surface of the polyhedron I Optimal solution lies in a corner of the polyhedron I Optimal solution is a global

TU Graz - Signal Processing and Speech Communication Laboratory

Linear Programming and the Simplex method

Harald Enzinger, Michael Rath

Signal Processing and Speech Communication Laboratory

Jan 9, 2012

Harald Enzinger, Michael Rath Jan 9, 2012 page 1/37


Outline

Introduction to Linear Programming

SimplexBasicsComputational DetailsDemonstration

Dual problem of LP



Outline



Dual problem of LP



Formulation of a Linear Program

minimize cTx

subject to Ax ≤ b

Cx = d

I The objective function cTx is a linear function of n decisionvariables x1 to xn

I There can be linear inequality and equality constraints

I The constraints define a feasible set of solutions

I The goal is to find a feasible solution that minimizes theobjective function



Geometric Interpretation of a Linear Equation in 2D

I aTx = b with a =

(a1a2

)x =

(x1x2

)I a1x1 + a2x2 = b → x2 = −a1

a2x1 +

ba2



Proof of Geometric Interpretation

aTx = aT (x⊥+ x‖)

= aTx⊥+ aTx‖

= ‖a‖‖x⊥‖ cos(0◦) + ‖a‖‖x‖‖ cos(90◦)= ‖a‖‖x⊥‖ = b

‖x⊥‖ =b

‖a‖



Generalization to more Dimensions

I aTx = b dimx = nI n = 3: PlaneI n > 3: Hyperplane

I aTx ≤ b or aTx ≥ bI Halfspace that is defined by

(Hyper-) Plane



Set of Equations

a11 · · · a1n...

...am1 · · · amn

x1

...xn

=

b1...bm

I Ax = b

I solution is an affine subspaceI dimension of solution space:

n− rank(A)

I Ax ≤ bI Intersection of m Halfspaces



Visualization of a Linear Program

I Polyhedron represents feasibleregion

I (Hyper-) planes represent constantobjective function value

I Objective function value isproportional to distance from origin

I Optimal solution lies on the surfaceof the polyhedron

I Optimal solution lies in a corner ofthe polyhedron

I Optimal solution is a globaloptimum



Special Cases of feasible region

I No intersection of halfspaces I Optimal solution isunbounded



Canonical Form of a Linear Program

maximize cTx

subject to Ax ≤ b

x ≥ 0

I minimization of cTx is equal to maximization of −cTxI constraint aTx ≥ b is equal to constraint −aTx ≤ −bI constraint aTx = b is equal to constraints

aTx ≤ b and aTx ≥ b

I unbounded variable xi can be split into two bounded variables:xi unbounded → xi = xi1 − xi2 xi1 ≥ 0 xi2 ≥ 0



Transformation to Standard Form

I Transform inequations to equations by introducing slackvariables xn+1 to xn+m

aTi x ≤ bi → aT

i x+ xn+i = bi

Ax ≤ b → (A|I)x = b

a11 · · · a1n | 1 · · · 0...

... |... 1

...am1 · · · amn | 0 · · · 1

x1...xnxn+1

...xn+m

=

b1...bm



Basic Solutions

a11 · · · a1n | 1 · · · 0 | b1...

... |... 1

... |...

am1 · · · amn | 0 · · · 1 | bm

I A basis is a subset of m linearly independent columns

I Basic variables xB are variables that belong to the basis

I Non-basic variables xN are the remaining variables

I A basic solution is found by setting xB = A−1B b and xN = 0

I e.g.xTB = (xn+1 · · ·xn+m) xT

N = (x1 · · ·xn) xB = b xN = 0



Corners of Polyhedron

Relation of Basic Solutions and CornersEvery basic solution corresponds to a corner of the polyhedron.

I xN = 0

I ⇒ Solution lies in intersection of hyperplanes Hj , j ∈ N

I xB = A−1B b is unique

I ⇒ Solution is unique

A unique intersection of n hyperplanes must be a corner.

Additional Properties

I A basic solution / corner is feasible if all xB ≥ 0

I A basic solution / corner is degenerated if there is an xB = 0



Example for Basic Solutions and Edges

I n = 2 variables

I m = 3 constraints

x1 + x2 ≤ 4

2x1 − x2 ≤ 3

x2 ≤ 1

x1 ≥ 0

x2 ≥ 0




1 1 1 0 0 | 42 −1 0 1 0 | 30 1 0 0 1 | 1

I N = {1, 2} B = {3, 4, 5}I Edge is feasible and not degenerated1 0 0

0 1 00 0 1

x3x4x5

=

431

⇒ xT = (0, 0, 4, 3, 1)




1 1 1 0 0 | 42 −1 0 1 0 | 30 1 0 0 1 | 1

I N = {1, 3} B = {2, 4, 5}I Edge is not feasible and not degenerated 1 0 0−1 1 01 0 1

x2x4x5

=

431

⇒ xT = (0, 4, 0, 7,−3)




1 1 1 0 0 | 42 −1 0 1 0 | 30 1 0 0 1 | 1

I N = {4, 5} B = {1, 2, 3}I Edge is feasible and degenerated1 1 1

2 −1 00 1 0

x1x2x3

=

431

⇒ xT = (2, 1, 0, 0, 0)



Outline



Dual problem of LP



The Simplex method

Basic Idea

I Start at corner point → initial basic solution

I Move along edge → increase one variable at a time

I Select variable with largest improvement of z → enteringvariable

I Move to next feasible corner point → select leaving variable

I Repeat until optimal corner point reached → no moreimprovement of z



The Simplex method

Basic Idea



The Simplex method

Requirements

LP in standard form:maximize z = c>x

subject to Ax = bx � 0, b � 0

Convert LP to standard form

I a>i x ≤ bi:

Introduce slack variable → a>i x+ si = bi

Example: 6x1 + 4x2 ≤ 24⇒ 6x1 + 4x2 + s1 = 24

I a>i x ≥ bi:

Introduce surplus variable → a>i x− Si = bi

Example: x1 + x2 ≥ 800⇒ x1 + x2 − S1 = 800



The Simplex method - Computational Details

Initialization

I Build tableau for canonical form

I Use slack variables as starting basic solution

Basic x1 . . . xn s1 . . . sm Solution

z −c1 . . . −cn 0 . . . 0 0 z-row

s1 a11 . . . a1n 1 . . . 0 b1 s1-row...

.... . .

......

. . ....

......

sm am1 . . . amn 0 . . . 1 bm sm-row

→ z-row corresponds to z − c1x1 − c2x2 − . . . cnxn = 0




Optimality condition

I Choose variable to enter the basic solution

I Take the one with the most negative coefficient in objectiveequation (z-row)

I If there is none with negative coefficient, optimality has beenreached

Basic x1 x2 s1 s2 s3 s4 Solution

z −5 −4 0 0 0 0 0 z-row

s1 6 4 1 0 0 0 24 s1-row

s2 1 2 0 1 0 0 6 s2-row

s3 −1 1 0 0 1 0 1 s3-row

s4 0 1 0 0 0 1 2 s4-row




Feasibility condition

I Choose variable to leave the basic solution

I Take the one with the minimum non-negative ratio

I Ratios of {solution/entering variable coefficient} correspond tointercerpts of constraints with entering variable

Basic x1 x2 s1 s2 s3 s4 Solution Ratio

z −5 −4 0 0 0 0 0

s1 6 4 1 0 0 0 24 246 = 4

s2 1 2 0 1 0 0 6 61 = 6

s3 −1 1 0 0 1 0 1 < 0

s4 0 1 0 0 0 1 2 ∞




Swapping Entering and Leaving Variable

I Replace leaving var. in basic solution with entering var.


z −5 −4 0 0 0 0 0

s1 6 4 1 0 0 0 24 s1-row

s2 1 2 0 1 0 0 6 s2-row

s3 −1 1 0 0 1 0 1 s3-row

s4 0 1 0 0 0 1 2 s4-row





I New pivot row = Current pivot row / Pivot element


z −5 −4 0 0 0 0 0

x1 6 4 1 0 0 0 24 x1-row

s2 1 2 0 1 0 0 6 s2-row

s3 −1 1 0 0 1 0 1 s3-row

s4 0 1 0 0 0 1 2 s4-row





I New row = (Current row) - (its pivot col.coeff.) · (New pivot row)


z −5 −4 0 0 0 0 0

x1 1 23

16 0 0 0 4 x1-row

s2 1 2 0 1 0 0 6 s2-row

s3 −1 1 0 0 1 0 1 s3-row

s4 0 1 0 0 0 1 2 s4-row





I Solution of iteration


z 0 − 23

56 0 0 0 20

x1 1 23

16 0 0 0 4 x1-row

s2 0 43 − 1

6 1 0 0 2 s2-row

s3 0 53

16 0 1 0 5 s3-row

s4 0 1 0 0 0 1 2 s4-row




Getting initial basic feasible solution (BFS)

I For canonical form one can take slack variables for initial BFS

I (=) Constraints a>i x = bi:Introduce artificial variable → a>i x+Ri = bi

I (≥) Constraints a>i x ≥ bi:Introduce surplus and artificial variable → a>i x−Si +Ri = bi

Dealing with artificial variables

I Eliminate artificial variables using standard simplex to get BFS

I M-method or Two-phase Method




M-method

I Introduce high penalty into objective function for artificial variablesMaximize z = c>x−MR

I Choose M accordingly to guarantee drop out of artificial variables

I Large M can result in roundoff errors that impair accuracy

Basic x1 x2 x3 R1 R2 Solution

z −4 −1 0 100 100 0

R1 3 1 0 1 0 3

R2 4 3 −1 0 1 6

⇓ Normalization


z −696 −399 100 0 0 −900

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.




Two-phase MethodSolve the LP in two phases:

Phase 1: Introduce new objective function to minimize the sum ofartificial variables{Minimize r =

∑i Ri} ⇒ {Maximize r = −

∑i Ri}

Phase 2: Perform usual simplex with solution obtained from Phase 1


r 0 0 0 1 1 0

R1 3 1 0 1 0 3

R2 4 3 −1 0 1 6

⇓ Normalization


r −7 −4 1 0 0 −9

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.



The Simplex method - Special Cases

Degeneracy

I Happens if tie occurs for minimum ratio in Feasibility condition

I At least one basic variable will be zero in next iteration

I Model has at least one redundant constraint (overdetermined point)

I Cycling if objective value doesnt improve

Alternative Optima

I Objective function parallel to constraint

I All points between corner points optimal solutions



The Simplex method - Demonstration

Solving toy example graphically

minimize −x1 − x2

subject to −x1 − x2 ≤ −2x1 − x2 ≤ 5

3x1 − x2 ≤ 18

3x1 + x2 ≤ 27

−x1 + 5x2 ≤ 25

−x1 + x2 ≤ 3

−4x1 + x2 ≤ 0























Outline



Dual problem of LP



Dual of LP

Definition

I Dual of LP defined from primal (original) LP model

I Optimal solution of one problem also provides solution to the other

Rules to construct dual problemI Define dual variable for each primal constraint

I Define dual constraint for each primal variable

I Primal constraint coefficients define left-hand side coefficients of dual constraint and its objectivecoefficient defines the right-hand side

I Objective coefficients of dual equal right-hand side of primal constraint equations

I Type of optimization switches (max⇔ min)

I Dual constraint type is determined by primal optimization type (min⇒≤, max⇒≥)



Dual of LP

Example

I Use rules on primal in equation form (Standard form)

Primal in equation form Dual variables

Minimize z = 15x1 + 12x2 + 0x3 + 0x4

subject to

x1 + 2x2 − x3 + 0x4 = 3 y12x1 − 4x2 + 0x3 + x4 = 5 y2

x1, x2, x3, x4 ≥ 0

Dual Problem

Maximize w = 3y1 + 5y2subject to

y1 +2y2 ≤ 15

2y1 −4y2 ≤ 12

−y1 ≤ 0

y2 ≤ 0



References

Rainer Burkard,

“Lecture Notes: Mathematische Optimierung”,http://www.opt.math.tu-graz.ac.at/∼hatzl/Vorlesung/MathOptSS11/Opt.pdf

Juncheng Wei,

“Lecture Notes: Linear Programming”,http://www.math.cuhk.edu.hk/∼wei/LP11.html

H.A. Taha,

“Operations Research: An Introduction”,Pearson Prentice Hall, 8th Edition, 2007


Harald Enzinger, Michael Rath Jan 9, 2012...I Optimal solution lies on the surface of the polyhedron I Optimal solution lies in a corner of the polyhedron I Optimal solution is a global

Documents