Introduction to Operations Research - ULisboa · Introduction to Operations Research (Week 6: ... (in the dual) in this case. ... possess optimal solutions x and y, respectively,

Introduction to Operations Research(Week 6: Linear Programming:

Duality and More on Post-Optimality)

Jose Rui Figueira

Instituto Superior TecnicoUniversidade de Lisboa

([email protected])

March 28-29, 2016

This slides are currently available for personal use of IST LEGI FIOstudents in an unpublished draft form only. The slides cannot becopied, reproduced, or distributed in any form.

Part

Duality

Contents

1. Introduction

2. Building the dual from an example

3. Solving the primal solves the dual and vice-versa

4. Transformation rules

5. The Dual (generalization)

6. Main results

7. More relations between primal and dual

8. An exercise

1. Introduction

Brief Introduction

I How to build the Dual of a Primal LP?

I Why is a Dual LP so important?

I It allows to have a different view of the problem.

I It allows to help in the economic interpretation of the problem.

I It allow to develop new mathematical concepts for LP.

I It allows the design of very efficient algorithms.

I References: [Bazaraa et al., 1990, Goldfarb and Todd, 1989,Hillier and Lieberman, 2005, Taha, 2010, Williams, 1993, Williams, 1999]

J.R. Figueira (IST) FIO March 28-29, 2016 5 / 45


The Primal LP Problem

max z(x1, x2) = 3x1 + 4x2 Profit

subject to: 2x1 + x2 6 18 (y1) 1 Silicone

x1 + 2x2 6 15 (y2) 2 High-tech machine

x1 6 8 (y3) 3 Robot 1

x2 6 6 (y4) 4 Robot 2

x1 > 0 Nonnegativity of x1

x2 > 0 Nonnegativity of x2

There are always three components to take into account for building the dual:

I The objective function, which is a minimization (in the dual) in this case.

I Associate with each constraint of the Primal there is a Dual variable (4 dualvariables).

I Associate with each Primal variable there is a Dual constraint (2 dualconstraints).



Building the Dual

min w(y1, y2, y3, y4) = 18y1 + 15y2 + 8y3 + 6y4

subject to: 2y1 + y2 + y3 > 3y1 + 2y2 + y4 > 4y1 > 0

y2 > 0y3 > 0

y4 > 0I Objective function of the Dual: The coefficients of the objective function of the

Dual are the RHS (bi) of the Primal constraints.

I Constraints of the Dual: The RHS of the Dual constraints are the coefficients ofthe decision variables (cj) of the Primal objective function.

I Each Dual constraint is built from each Primal activity: The LHS coefficients of theDual constraint (j) are the technological coefficients (aij) for the Primal activity (j).

I Analyze, for example, the relation between c1 = z1 − c1 and the first constraint ofthe dual LP? 2y1 + y2 + y3 > 3 ⇔ (2y1 + y2 + y3)− 3 > 0.



The Dual LP in an equality form

min w(y) = 18y1 + 15y2 + 8y3 + 6y4 − 0y5 − 0y6

subject to: 2y1 + y2 + y3 − y5 = 3y1 + 2y2 + y4 − y6 = 4

y1, y2, y3, y4, y5, y6 > 0

I Decision variables of the Dual: y1, y2, y3, and y4.

I Surplus variables of the Dual: y5 and y6.

I Since solving the Primal also solves the Dual, where are the values of theses Dualvariables?

I Let us have a look at the optimal primal solution (in the last Tableau).


3. Solving the primal solves the dual and vice-versa

cj 3 4 0 0 0 0Basis xj x1 x2 x3 x4 x5 x6 bi Ratio

0 x6 0 0 1/3 −2/3 0 1 23 x1 1 0 2/3 −1/3 0 0 70 x5 0 0 −2/3 1/3 1 0 14 x2 0 1 −1/3 2/3 0 0 4

t = 3 zj 3 4 2/3 5/3 0 0D zj − cj 0 0 2/3 5/3 0 0 37

I Consider the optimal primal simplex Tableau.

I The Dual solution is in lines zj and cj = zj − cj.

I The Dual decision variables are related with the primal constraints and so with theprimal slack variables. Why?

I Their values are in the line zj under the slack variables of the primal problem. Why?

I Thus, y1 = 2/3, y2 = 5/3, y3 = 0, and y4 = 0. Why?

I What about the values of y5 and y6?


4. Transformation rules

Objective, Constraints, and Variables Types

Primal Dual

Objective Function � Objective Functionmax min

Variables � Constraints> 0 >6 0 6free =

Constraints � Variables6 > 0> 6 0= free



The LP Dual Problem

minm

∑i=1

biyi

subject to:m

∑i=1

aijyi > cj, j = 1, . . . , n,

yi > 0, i = 1, . . . , m.

Or, by adding the surplus (“slack”) variables, ym+1, . . . , ym+j, . . . , ym+n:

minm

∑i=1

biyi

subject to:m

∑i=1

aijyi − ym+j = cj, j = 1, . . . , n,

yi > 0, i = 1, . . . , m + n.



The dual as the primal in the “mirror”

c1...

cj...

cnm

inw

>...

>...

>=

a11...

a1j...

a1nb1

(y1 )

......

......

......

ai1...

aij...

ainbi

(yi )

......

......

......

am1

...am

j...

amn

bm(y

m)

(x1) (xj) (xn)c1 . . . cj . . . cn = max za11 . . . a1j . . . a1n 6 b1

.... . .

.... . .

......

...ai1 . . . aij . . . ain 6 bi...

. . ....

. . ....

......

am1 . . . amj . . . amn 6 bm

DUAL

(yi > 0, i = 1, . . . , m)PRIMAL

(xj > 0, j = 1, . . . , n)


6. Main results

Symmetry Between Primal and Dual.

maxx

{c>x : Ax ≤ b, x ≥ 0

}(P)

miny

{b>y : A>y ≥ c, y ≥ 0

}(D)

and

max(x,s)

{c>x : Ax + s = b, (x, s) ≥ 0

}(P′)

min(y,r)

{b>y : A>y− r = c, (y, r) ≥ 0

}(D′)

where s = (xn+1, . . . , xn+m)> and r = (ym+1, . . . , ym+n)> are the vectors ofslack variables of the primal and the dual problems, respectively.


6. Main results

Two Lemmas on Duality

If (D) is the dual of a problem (P), then the dual of (D) is (P).

Lemma 1 (The Dual of the Dual).

Any feasible solution of (D) yields an upper bound for (P).

Lemma 2 (Upper Bound).


6. Main results

Important Theorems on Duality

Let x ∈ Rn denote a feasible solution for (P) and y ∈ Rm denote a feasiblesolution for (D). Then, the following relation holds: c>x 6 b>y.

Theorem 1 (Weak Duality).

Let x∗ ∈ Rn denote a feasible solution for (P) and y∗ ∈ Rm denote afeasible solution for (D). If c>x∗ = b>y∗, then x∗ and y∗ are the optimalsolutions of (P) and (D), respectively.

Theorem 2 (Strong Duality).


6. Main results

Proving Strong Duality: Farka’s Lemma (A variant)

Let A ∈ Rm×n and let b ∈ Rm. Then exactly one of the two alternativesholds:

1. There exists x ≥ 0, such that Ax = b.

2. There exists y, such that y>A ≥ 0 and y>b < 0.

Lemma 3 (Farka’s Lemma).

This result simply states

1. that a vector is either in a given convex cone, or [it is an exclusive or]

2. that there exists a hyperplane separating the vector from the cone.

(there are no other possibilities.)

Let Ai denote the i− th column of matrix A. Next slide shows a geometric view of this

lemma.J.R. Figueira (IST) FIO March 28-29, 2016 16 / 45

6. Main results

Geometric Illustration of Farka’s Lemma

A1 A2 A3

b

b

y

Hy,0



A Different Optimality Condition

Let (x∗, s∗) ∈ Rn ×Rm denote a feasible solution for (P′) and (y∗, r∗) ∈Rm ×Rn denote a feasible solution for (D′). Then x∗ and y∗ are optimalfor (P) and (D), respectively iff x∗>r∗ = 0 and y∗>s∗ = 0.

Theorem 3 (Complementary Slackness).



Possible results for (P) and (D)

Exactly one of the following statements is true:

1. Both Problem (P) and Problem (D) possess optimal solutions x∗ andy∗, respectively, and c>x∗ = b>y∗.

2. Problem (P) is unbounded and Problem (D) is infeasible.

3. Problem (D) is unbounded and Problem (P) is infeasible.

4. Both problems are infeasible.

Theorem 4 (Fundamental Theorem of Primal-Dual Relationships).


8. An exercise

Linear Programming (LP)

What is the dual of the following primal LP problem?

maxn

∑j=1

cjxj

subject to:n

∑j=1

aijxj 6 bi, i = 1, . . . , m′ ,

n

∑j=1

aijxj > bi, i = m′ + 1, . . . , m′′ ,

n

∑j=1

aijxj = bi, i = m′′ + 1, . . . , m′′′ ,

with xj > 0, for all j = 1, . . . , n.

Exercise 1 (Dual problem).


Part

Economic Interpretation

Contents

1. Introduction

2. Product-mix resource allocation problem

3. The optimal primal simplex Tableau

4. What are the shadow prices?

5. Another view of the problem

6. More on the economic interpretation

7. Lagrangian Duality

1. Introduction

Brief Introduction

I We start again with the resource allocation problem.

I We will see a new view of the problem.

I The relations between dual variables, internal value of resources, shadowprices, and ...

I A different view of duality.


2. Product-mix resource allocation problem

A Classical Maximization Problem

The CorkMaxPro Company is the largest Portuguese producer andexporter of cork made products. Over the last five years, the com-pany has gradually expanded the geography and the volume of itsproduction activities, which consist of producing a product mix (i.e., de-signing and making the product lines or assortments that the companyoffers to its clients), here generically denoted by P1, . . ., Pj, . . ., and Pn.

The company’s plants possess several resources, R1, . . ., Ri, . . ., andRm, to make the different assortments. The quantity available of eachresource is, b1, . . ., bi, . . ., and bm, respectively. The allocation of resourcesto the production is as follows: allocate aij units of the resource Ri to makeone unit of the assortment Pj.

The market absorbs all the production and the revenue per unit ofproduct sold is cj Euros. The Company’s CEO needs to know the next yearproduction plan (activity levels) in such a way that the resource constraintsshould be taken into account and the overall revenue to be earned from theselling of the assortments in the market should be maximized.

Example 1 (A Classical Product Mix Resource Allocation Problem).



Optimal Primal Simplex Tableau

((i) ∈ B) ((i) ∈ B) cj c1 . . . cj . . . cn 0 . . . 0 . . . 0 Sol.

c(i) x(i) xj x1 . . . xj . . . xn xn+1 . . . xn+i . . . xn+m bi

c(1) x(1) a11 . . . a1j . . . a1n a1,n+1 . . . a1,n+i . . . a1,n+m b1...

......

. . ....

. . ....

.... . .

.... . .

......

c(i) x(i) ai1 . . . aij . . . ain ai,n+1 . . . ai,n+i . . . ai,n+m bi...

......

. . ....

. . ....

.... . .

.... . .

......

c(m) x(m) am1 . . . amj . . . amn am,n+1 . . . am,n+i . . . am,n+m bm

zj z1 . . . zj . . . zn zn+1 . . . zn+i . . . zn+m

1 z cj = (zj − cj) (z1 − c1) . . . (zj − cj) . . . (zn − cn) (zn+1 − cn+1) . . . (zn+i − cn+i) . . . (zn+m − cn+m)m

∑i=1

c(i)b(i)

I Primal optimal solution (for the decision and slack variables):x?(i) = bi, i = 1, . . . , m.

z? =m

∑i=1

c(i)bi.

I But, this Tableau provides more information than merely the values of the primal variables.



Some Questions

I For the assortments that are not produced (the ones corresponding to thenon-basic decision variables) how much more expensive should we make theirprices in order to produce them in the future?

I What is the value of an extra unit of a scarce resource?

Simplex provide us the economic information for the answers.

I As for the decision variables the reduced costs give us the necessary priceincreasing.

I As for the value of resources, shadow prices can be seen as the marginaleffect of a decrease or an increase of resources availabilities.


4. What are the shadow prices?

An Important Definition

Associated with an optimal solution are shadow prices (also referred to asdual variables, marginal values, marginal valuations, . . .) for the constraints.The shadow price associated with a particular constraint corresponds to themodification in the optimal value of the objective function per unit increasein the right-hand-side (RHS) value for that constraint.

Shadow prices for abundant resources are equal to zero. The companydoes not give valuation to its abundant resources.

Definition 1 (Shadow Prices on the Constraints).



A Related Minimization Problem

The CorkMaxPro Company’s accountant is confronted with the assess-ment of the value of the company since there is a strong possibility ofrenting it to a Spanish investor for one year. The accountant should reportthe value to the CEO, who keeps in mind that renting the resources mustbe at least as favorable as using the capacity itself (i.e., the rent should beat least equal to the earnings obtained from producing the assortments).

The company’s plants have bi available units of resource Ri, for i =1, . . . , m. What should be the rent per unit of resource? And what shouldbe the minimum value of the overall rent in such a way that the companywill know what minimum offer will be economically acceptable for renting?

Producing one unit of Pj needs a1j units of resource R1, a2j units ofresource R2, . . ., and amj units of resource Rm. The overall rent for suchquantities of resources should be greater than, or equal to, the revenueearned from one unit of the assortment Pj, cj, for j = 1, . . . , n.

Example 2 (Renting the CorkMaxPro Company).



The Linear Programming (LP) Dual Model

minimize w(y) = b1y1 + . . . + biyi + . . . + bmym

subject to: a11y1 + . . . + ai1yi + . . . + am1ym > c1...

......

a1jy1 + . . . + aijyi + . . . + amjym > cj...

......

a1my1 + . . . + aimyi + . . . + amnym > cn

y1, . . . yi, . . . ym > 0



Some Remarks

I The overall valuation provided by the accountant to the CEO is equal to itsoptimal production plan of the original problem. It seems quite natural andthe result comes from the Duality Theorems (this results is always true).

I Abundant resources have a zero valuation. Again, it seems quite naturalbecause we do not use all the availability of resources. It is also a result fromDuality (Slackness Complementarity): If a constraint is non-biding in theoptimal primal Tableau its dual value, in the optimal dual solution, is zero.

I When, for a given product, the imputed costs are higher than its per unitrevenue, the company does not make its production. This is another resultfrom the Duality (Slackness Complementarity). If a constraint for the dualproblem is non-biding, then the associated primal decision variable has a zerovalue. From an economic point of view it means that non-biding constraintshave zero valuations.



Shadow Prices as Opportunity Costs

I An increasing in the availability of a scarce resource results in an opportunityto make more revenue.

I Analogously, a decreasing in the availability of a scarce resource results in aloose opportunity to make more revenue.

I The shadow price represents a cost of the lost opportunity.



Lagrangian Duality (1)

I Define the Lagrangian L that corresponds to the primal problem (P).

L(x, y) = c>x + y>(Ax− b).

I The vector y plays the role of a vector of Lagrange multipliers for theconstraints in the primal problem (P).

I The following definition is a fundamental one in unconstrained optimization.

The point (x∗, y∗) is a saddle point of the Lagrangian L if and only if

1. x∗ ≥ 0, y∗ ≥ 0 and

2. L(x, y∗) 6 L(x∗, y∗) 6 L(x∗, y), for all x ≥ 0, y ≥ 0.

Definition 2 (Saddle Point).



Lagrangian Duality (2)

I The function L(x, y∗) attains a maximum with respect to x at x = x∗ overall x ≥ 0.

I Conversely, L(x∗, y) attains a minimum with respect to y at y = y∗ over ally ≥ 0.

The saddle point conditions of Definition 2 are necessary and sufficient forx∗ to solve (P) and for y∗ to solve (D).

Theorem 5 (Optimality Conditions).



Building the Dual (1)

I Consider our problem (P) as follows, in the equality form,

maxx

{c>x : (Ax− b) = 0, x ≥ 0

}.

I It implies,

g(y) = maxx

{c>x + y>(Ax− b) : x ≥ 0, y ∈ Rm

}.

I For any y ∈ Rm, g(y) provides an upper bound for (P).

I To get the best upper bound, we should

miny

g(y).

I This problem is called the Lagrangian dual and g(y) the dual function.



Building the Dual (2)

I Let us analyze the function g(y).

g(y) = maxx

{c>x + y>(Ax− b)

}= y>b + max

x

{(y>A− c>)x

}.

I It should be noticed that,

g(y) =

+∞ if (y>A− c>)j < 0, for at least one j

y>b if (y>A− c>)j > 0, for all j

I Since +∞ is not an interesting upper bound, we should focus our attentionfor y ∈ Rm such that y>A− c> ≥ 0. The dual (D) can thus be written asfollows.

miny

{y>b : y>A ≥ c, y ∈ Rm

}.

I The only difference is that we started with Ax = b and will have y with freecomponents.


Part

More on Post-Optimality

Contents

Adding a New Constraint

Add the constraint x1 + 2x2 6 8 to:

maximize x1 + 2x2subject to: x1 + x2 6 6

x2 6 3

with x1, x2 > 0.

x1

x2

0 1 2 3 4 5 6 70

1

2

3

4

5

6

7

The previous solution is no more optimal:

cj 1 2 0 0Basis xj x1 x2 x3 x4 bi

1 x1 1 0 1 −1 32 x2 0 1 0 1 3

zj 1 2 1 1cj 0 0 1 1 9

I How to deal with this case?

I Add one line to the tableau.

(related to the constraint)

I This implies to add a column.

(related to the new slack variable, x5)

I Put x5 as a function of x3, x4.

(the two non-basic variables)


Adding a New Constraint

I The new constraint: x1 + 2x2 + x5 = 8.

I Equivalently: x5 = 8− x1 − 2x2.

(where x5 is the slack variable)

I We know that: x1 = 3− x3 + x4.

(from the first row of the tableau)

I We also know that: x2 = 3− x4.

(from the second row of the tableau)

I We have:

x5 = 8− (3− x3 + x4)− 2(3− x4).

x5 = −1 + x3 + x4

−x3 − x4 + x5 = −1I Fill the yellow zones!

I Observe that the solution is optimal but notfeasible!

I It‘s time to use Dual Simplex Algorithm.

I x5 leaves and x4 enters.

The previous solution is no more optimal:

cj 1 2 0 0 0Basis xj x1 x2 x3 x4 x5 bi

1 x1 1 0 1 −1 0 32 x2 0 1 0 1 0 30 x5 0 0 −1

�� −1 1 −1zj 1 2 1 1 0cj 0 0 1 1 0 9

cj 1 2 0 0 0Basis xj x1 x2 x3 x4 x5 bi

1 x1 1 0 2 0 −1 42 x2 0 1 −1 0 1 20 x4 0 0 1 1 −1 1

zj 1 2 0 0 1cj 0 0 0 0 1 8

Multiple Optima!


Initial Dual Simplex Tableau (Iteration 0)

(q ∈ B) (q ∈ B) cj c1 . . . ck . . . cn 0 . . . 0 . . . 0 Solution

cq xq xj x1 . . . xk . . . xn xn+1 . . . xn+` . . . xn+m bi0 xn+1 −a11 . . . −a1k . . . −a1n 1 . . . 0 . . . 0 −b1...

......

. . ....

. . ....

.... . .

.... . .

......

0 xn+` −a`1 . . .�� −a`k . . . −a`n 0 . . . 1 . . . 0 −b`

......

.... . .

.... . .

......

. . ....

. . ....

...0 xn+m −am1 . . . −amk . . . −amn 0 . . . 0 . . . 1 −bm

zj z1 . . . zk . . . zn 0 . . . 0 . . . 01 z zj − cj z1 − c1 . . . zk − ck . . . zn − cn 0 . . . 0 . . . 0 0

minimize 6x1 + 3x2subject to: x1 > 1

x1 + x2 > 2

with x1, x2 > 0.

minimize 6x1 + 3x2subject to: −x1 + x3 = −1

−x1 + −x2 + x4 = −2

with x1, x2, x3, x4 > 0.


Dual Simplex Algorithm: minx{c>x : −Ax ≤ −b, x ≥ 0}

Algorithm 1 Dual Simplex

Input: A ∈ Rm×n, b ∈ Rm, c ∈ Rn.Output: x∗ ∈ Rn

+ such that Ax ≥ b.

1: x∗i ← −bi, i ∈ B = {n + 1, . . . , n + m}, and x∗j ← 0, j ∈ N = {1, . . . , n};2: compute cj, j ∈ N (cj ← −cj);

3: while (∀t ∈ {i : bi < 0, i = 1, . . . , m} : {j : −atj < 0, j ∈ N} 6= {}) do

4: select ` ∈ {i : bi < 0, i = 1, . . . , m} and the corresponding x(`);5: select k ∈ arg min

j∈N

{cj/− aj` : −aj` < 0

}and the corresponding xk;

6: pivoting on −a`k;7: x∗(i) ← bi, i = 1, . . . , m, and x∗j ← 0, j ∈ N ;

8: compute cj, j ∈ N ;9: end while

10: if (∃t ∈ {i : bi < 0, i = 1, . . . , m}) and ({j : −atj < 0, j ∈ N} = {}) then11: x∗ ← −∞ (primal infeasibility);12: end if


A Second Small Example

minimize 6x1 + 3x2subject to: x1 > 1

x1 + x2 > 2

with x1, x2 > 0.

x1

x2

0 1 2 3 40

1

2

3

4

cj 6 3 0 0Basis xj x1 x2 x3 x4 bi

0 x3 −1 0 1 0 −10 x4 −1

�� −1 0 1 −2zj 0 0 0 0cj −6 −3 0 0 0

0 x3

�� −1 0 1 0 −13 x2 1 1 0 −1 2

zj 3 3 0 −3cj −3 0 0 −3 6

6 x1 1 0 −1 0 13 x2 0 1 1 −1 1

zj 6 3 −3 −3cj 0 0 −3 −3 9


Some questions

Questions:

1. Consider the optimal tableau of the previous example.

2. Identify the solution of the dual problem.

3. Build the dual of the primal.

4. Show the optimality of solutions through the complementary slacknessproperty.


Part

Bibliography

References

Bazaraa, M., Jarvis, J., and Sherali, H. (1990).

Linear Programming and Network Flows.John Wiley & Sons, New York, USA, second edition.

Goldfarb, D. and Todd, M. J. (1989).

Linear programming.In Nemhauser, G. L., Rinnoy Kan, A. H. G., and Todd, M. J., editors, Optimization, volume 1 of Handbooks in Operations Research andManagement Science, pages 141–170. North Holland, Amsterdam, The Netherlands.

Hillier, F. and Lieberman, G. (2005).

Introduction to Operations Research.The McGraw-Hill Companies, Inc., New York, USA, eighth edition.

Taha, H. (2010).

Operations Research: An Introduction.Prentice-Hall, Upper Saddle River, New Jersey, USA, 9th edition.

Williams, P. (1993).

Model Solving in Mathematical Programming.John Wiley & Sons, Chichester, UK.

Williams, P. (1999).

Model Building in Mathematical Programming.John Wiley & Sons, Chichester, UK.

Introduction to Operations Research - ULisboa · Introduction to Operations Research (Week 6: ... (in the dual) in this case. ... possess optimal solutions x and y, respectively,

Documents