-
VOL. 11, NO. 6 WATER RESOURCES RESEARCH DECEMBER 1975
Reservoir Management Models MATTHEW J. SOBEL
School of Organization and Management, Yale University, New
Haven, Connecticut 06520
The optimal policies for several discrete time control models of
reservoir storage are characterized. Each characterization is
exploited to simplify the numerical solution of heretofore
formidable problems. Some of the results exploit analogies between
multi-item inventory theory and systems of multiple reser- voirs.
In particular, it is observed that the 'linear decision rule' label
in the water resource literature or- dinarily arises in contexts
where either (1) a myopic policy is optimal, or (2) a
multiple-reservoir optimal release policy, as a function of the
vector of reservoir contents, possesses a Jacobian matrix whose
values are between zero and one. Either (1) or (2) impressively
reduces the computational burden that would otherwise be
carried.
This paper analyzes the structure of optimal policies for
several discrete time control models of reservoir storage. Most of
the models are stochastic and they are prompted by operating
problems of regulating the amounts of water dis- charged from
reservoirs. However, the design problem of selecting the capacity
of a reservoir induces two models in the paper and, of these, one
is deterministic. The form of an op- timal policy is characterized
for each model and then exploited to simplify the numerical
solution of heretofore formidable problems. Several sections of the
paper exploit analogies between multiple-reservoir systems and
multi-item inventory systems. As a consequence, the form of some of
the results below should be familiar to readers acquainted with
inventory theory.
Several purposes are served by proliferating the number of
available reservoir models. Actual reservoir systems display
heterogeneous characteristics, and so a richer set of models
permits adequate representation of more of these systems.
Furthermore, it permits the same reservoir system to be modeled in
several ways. Thus dichotomies in this paper such as
deterministic/stochastic, independent/correlated, and chance
constraints/explicit costs are not presented as a tax- onomy of
actual reservoirs. Instead, they are different ways of idealizing
the same problem of managing a water storage process.
SVMMA}
The following section investigates the reservoir design prob-
lem of computing the smallest reservoir capacity that will ac-
commodate a sequence of predicted inflows. It is shown that a hand
calculation, simpler than previously proposed algorithms, will
solve this problem. Moreover, some stochastic versions of the
reservoir capacity problem can be solved equal- ly easily.
The paper continues by developing an analogy between models of
multiple-reservoir systems and of 'multi-item inven- tory' models.
Cost minimization (or net revenue max- imization) reservoir models
are very closely related to inven- tory models about whose
optimization so much is known. The analogy is exploited in a
stochastic model of a class of multiple-reservoir systems. This
model has a simple myopic solution, and so the sequential problem
degenerates to a se- quence of single-period problems whose
numerical solution is easily obtained. The 'myopic' solution
specifies a point (or vec-
Copyright (D 1975 by the American Geophysical Union.
tor) that receives different labels in different literatures:
'linear decision rule' (water resources), 'base stockage policy'
(opera- tions research), and 'single critical number' (operations
research). In the subsequent section the structure of optimal
policies is explored for multiple-reservoir systems that may lack a
myopic solution. Optimal release policies, as functions of
reservoir contents, are shown to possess a Jacobian matrix whose
values are between zero and one. This property is shown to
accelerate computations impressively.
For expository convenience, the previously mentioned stochastic
models assume that inflows in different periods are ind endent.
This assumption is relaxed in the final section of the paper where
it is shown that the preceding algori.thms and policy structures
extend to general stochastic processes of in- flows and
demands.
NOTATION AND CONVENTIONS
Vector notation is used in the sections of the paper con- cerned
with multiple-reservoir systems. However, inner products are the
only multiplications needed. If a = (a, ..., ai) and = (, ..., i)
are real I vectors, then a. or .a is written to denote the inner
product -'t--[ ottlt. The notation ej is used for the I vector
having all zeros except one in the jth component, j = 1, ...,
I.
All functions that arise are assumed to attain their minima when
they are constrained to compact domains. It is sufficient that the
functions be lower semicontinuous, and this restric- tion poses no
practical difficulties. Whatever functions are differentiated will
possess one-sided derivatives. For defi- niteness, therefore,
derivatives are assumed to be taken from the left.
When I' is a finite set of real numbers or a real-valued func-
tion that attains a constrained minimum, the notation min I'
denotes the smallest member of the set or lowest value of the
function. Similarly, if I' is a finite set of real numbers, then
max I' denotes the largest member of the set. It is convenient to
write (a) + for max {a, 0} when a is a real number. Ifa = (a, ",
at) is a real I vector, then (a) + denotes the I vector whose ith
component is (at) +. The notation P{A} indicates the probability of
an event A.
M any quantities in the following models will vary with time,
and the subscript t denotes the time period to which a variable
pertains. If the variable refers to the ith reservoir in a collec-
tion of I reservoirs, then it carries a superscript of i. For exam-
ple, st denotes the quantity of water stored in reservoir i at the
end of period t. If the superscript i is absent from such a
767
-
768 SOBEL: RESERVOIR MANAGEMENT MODELS
variable, then it denotes the relevant I vector in period t,
e.g., st = ..., sD.
The notation list presents most of the symbols used in the
paper. Generally, capital letters denote the random counter- parts
to lowercase letters, and these capitals are absent from the list.
Exceptions to this rule, such as I, K, and T, are on the list. The
latter part of the paper makes some use of dynamic programing in
general and inventory theory in particular. Some quantities which
are scalars until that point thereafter are functions. For example,
the random drawdown in period t is denoted Xt(u) as a function of
freeboard u at the beginning of the period. Asterisks are sometimes
used to denote an op- timal policy, Xt*( ) being an example. Such
variants of a lowercase symbol, xt in this case, are not shown in
the list.
One last notational peculiarity attends the mixing of water
resource models with inventory theory. The maximal draw- down has
been labeled f in past water models, and the ex- pected cost of an
optimal inventory policy has been denoted ft( ) in inventory
theory. Both these usages occur in the se- quel, but at most one of
them is used in any given section. In fact, the first occurrence
offt( ) is well beyond the last use off to denote maximal
drawdown.
SMALLEST STORAGE CAPACITY FOR PREDICTED NEEDS: A CHEBYSHEV
PROBLEM
This section concerns a deterministic model, namely, a storage
process in which inflows and demand are treated as be- ing known in
advance. The problem, concerning a planning horizon of T periods,
is to determine (1) the smallest storage capacity that accommodates
the inflows and permits the pro- jected demands to be met and (2)
the corresponding schedule of discharges. Such deterministic models
arise in subtasks in simulation programs having random elements.
The optimiza- tion has been posed by ReVelle et al. [1969] as a
linear programing problem. It will be shown here that simple hand
calculations lead to a solution.
Thomas' 'sequent peak' algorithm [Harvard Water Resources Group,
1963] (see Fiering [1967] or Loucks [1970]) solves a problem
related to ours. It is the special case of (1)-(4) below when ft =
co and mt= 0 for all t and it is 'assumed that the inflow and
drafts will be repeated in successive sets of cy- cles of T-years
each' [Harvard Water Resources Group, 1963, pp. 1-6]. This last
assumption is riot made below. In some ap- plications it is an
attractive feature, while in others it is not sought. In some
cases, the assumption forces an increase in the requisite
capacity.
Let st denote the quantity of water in storage at the end of
period t and let rt be the predicted inflow during the period. Then
st_ and st are connected by
St = St- + rt -- Xt (1) where xt denotes the drawdown, the
quantity discharged dur- ing period t.
In reservoirs whose surface area does not vary significantly
with storage volume, leakage and evaporation also are essen- tially
independent of storage olume. Then rt can be regarded as inflow net
of leakage and evaporation, and the algebraic sum is not a function
of storage volume.
Of course (1) is invalid if the right side exceeds the capacity
c of the reservoir, and so an elaboration of (1) is
st = min {c, st_ + rt -- xt} (2) the overflow, if any, being
(st_ + rt -- xt -- c) +. The physical
constraints are
0
-
SOBEL: RESERVOIR MANAGEMENT MOVERS 769
future periods r > t. Let Lt denote the sum of mt and water
required to be held back in t for future use. A policy for which xt
> st-1 + rt -- Lt in a period t leads to infeasibility in some
period r > t. Then Lr = mr Lt = mt + (Lt+l + qt+l - rt+l - mr) +
(8)
t = 1, ..., T- 1
is verified inductively starting with t = T- 1 and proceeding
from t + 1 to t. The minimax capacity problem with (7) in place of
(4) has an optimal solution satisfying
(ft - xt)(st - Lt) = 0 t = 1,..., T (9) The proofs of (9) and
(5) are the same after recognizing that (9) is equivalent to
xt = min {ft, st-1 + rt -- Lt} t = 1," ', T (10) The
computations underlying (8) and (10) (or (6) if rt -- qt
and mt --> mt+l for all t) are trivial by hand. They are
followed by the recursion (1) for sl, "', st. Then the minimally
suffi- cient capacity is c = max {So, '", st}.
Table 1 presents a numerical example of the applications of (1),
(8), and (10). The data, the computations for (8), (10), and (1),
and the end products, the draft and storage decisions, are
given.
STOCHASTIC MINIMAX CAPACITY
An inspection of (5) and (6) shows that the value taken by an
optimal discharge quantity xt does not depend on the data for any
period later than t. In fact, if qt -< xt --< ft and mt
--< st are time-varying restrictions, then rt -- qt for all t
and rnl _> rn2 > " > mr imply that
xt = min Ift, st-1 + rt - mr} (11) is optimal. Here the decision
xt in period t is unaffected by knowledge of events in periods
later than t. Thus the myopic character of (6) is retained by (11)
which implies that a variety of stochastic minimax capacity
problems have an easy solu- tion. Suppose that a stochastic process
(Q1, F1, M1, R0, "', (Qr, Fr, Mr, Rr) determines (qt, ft, mr, rt)
for t = 1, ..., T. The storage levels S1, "', Sr and discharges
,Y1, '", ,Yr are controlled random variables. Let
St = St-1 + Rt-Xt (12)
and suppose that the values st-i, mr, rt, qt, and ft taken by
St-i, Mr, Rt, Qt, and Ft are known when the value of Xt is
selected. A stochastic version of rt -- qt and mt -- mt+l for all t
is
P{Rt- Qt Mt - Mt+l t= 1,'.., T} = 1 (13) There is some decision
rule X1, "', Xr such that
PlMt+l < St Qt < xt < Ft t = 1,..., T} = 1 (14)
Assumptions (13) and (14)guarantee that a feasible rule exists
(with probability one). Let X denote the set of rules (X1, "', Xr)
satisfying (14). The stochastic version of (6) is
't* = min {Ft, St_i* q- Rt - Mt} (15) where So* -- So and
St* : St-i* q- Rt - Xt* t = 1, "', T Second Result
Then the optimality and myopia of (11) imply that (15) is
feasible and induces minimax capacity under assumptions (13) and
(14). This claim is true with probability one. As a consequence,
among all rules satisfying (14) the one given by (15) minimizes the
expectation of the maximum stored quantity.
Let ht be an outcome (or 'realization' or 'point in the sample
space') of the history
Ht -- (So, Q1, F1, M1, X1, R1, "', St-2, Qt-1, Ft-1, Mt_l,
)'t-1, Rt-1, St-1, Qt, Ft, Mt) (16)
It is physically impossible for policy X1, '", Xr to have Xt de-
pend on more information than is contained in Hr. Suppose (14) is
replaced by the chance constraint
and P{mt St "- St _ l q- rt -- mr(mr) (17a) qt --< X't(ht)
ft} - Olt (175)
for all ht and t = 1, ..., T. The values of mr, rt, qt, and st-1
are included in ht and the values of at are assumed to satisfy 0
(at ( 1, t = 1, ...,T.
The restrictiveness of (17) compared to (14) is more illusory
than real. Suppose (13) is valid and that X1, '", Xv is a policy
different from (.15) and let So, "', St be the storage levels
generated by the Xt. It will be shown that maxt {St*t
TABLE 1. A Simple Minimax Capacity Problem
Data
Maximum Minimum Month Draft Draft
t ft qt
Minimum Computations Inflow Storage
rt mt Lt + + qt + - rt + i Lt $t- i + rt - Lt
Decisions
Dratt Xt
Storage St
0 1 10 8 2 10 8 3 10' 8 4 10 8 5 11 9 6 13 11 7 15' 13 8 17' 15
9 14 12
10 12 10 11 11' 8 12 10' 8
0 10 1' -1 1' 9 9 1 10 1' -3 1' 10 10 1 15 1 3 4 12 10 6 21 1 16
17' 10 10 17 17 I 23 24* 10 10 24 13 3 22 25* 12 12 25t
9 2 16 18 16 15 19 6 1 6 7 18 17 8 8 1 1 2* 14 14 2
10 1' -4 1' 11 11 1 12 0 -2 0 13 11 2 11 1 1 12 10 3
*Binding constraint. 'Minimal c = s6 = 25 = max {s0,'", s.}.
-
770 SOBEL: RESERVOIR MANAGEMENT MODELS
_< maxt {St} with probability one. If Xt(ht) < Xt*(ht) for
all ht and t, then St 2 St* for all ht and t; so
P{max {St*} max {St}} = 1 t t
and X* is superior to X. The only other possibility is that
there is the smallest integer t and history ht such that
Xt(ht) > Xt*(ht) = min {ft, st- * + rt - mr} Third Result
Then either (1) Xt(ht) ft, and Xt is infeasible, or (2) Xt(ht)
St-* + rt -- mt = st- + rt -- mr; so mt > st = st- + rt -- X(h
), P{M(h ) _< &(h, X ), and Q(h) _< X(h ) _< F(h)} =
0, which violates (17); so Xt is again infeasible. Therefore (13)
implies that X* in (15) is optimal for the stochastic minimax
problem with chance constraints.
COMMENT ON CHANCE CONSTRAINTS AND POLICY STRUCTURE
'Chance-constrained' programing combines dynamic cost
minimization with probability constraints on some operating
characteristics that would result from a chosen policy. In-
equality (17) is an example of a chance constraint. Several re-
cent reservoir management models (see LeClerc and Marks [1973] and
its references) have included chance constraints, but their authors
have ignored the well-known [Derman and Klein, 1965] possibility
that a randomized policy dominates every unrandomized policy. For a
trivial example, suppose there are two actions labeled 0 and 1.
Exactly one of these two must be taken every period subject to the
constraint that ac- tion I be taken at least 100% of the time.
Action j incurs a cost of $j each time that it is taken, and the
objective is to minimize the average cost per period. Action I
taken every period is the only unrandomized stationary policy that
satisfies the 100% constraint, and the associated average cost is
$1. However, the randomized st. ationary policy that takes action 0
with probability I - and action 1 with probability also satisfies
the constraint, and its average cost is $ < $1.
Water resource models have not yet applied chance con- straints
to joint events. For example, if et is the event being constrained
in (17), then one may seek a solution that satisfies
P{ UI . U/ '" UI r} > a0 in addition to (17). This
observation is pertinent in most water resource contexts; methods
for solving the resulting optimiza- tion problems are discussed by
Miller and Wagner [1965].
FORMULATION OF OPERATING PROBLEMS: APPLICABILITY OF INVENTORY
THEORY
The benefits and costs associated with a particular reservoir
design depend partly on the modes of operation that the design
permits. In other words, design problems and operating problems are
interrelated because the design of a reservoir affects the
composition of the class of feasible operating rules. However, this
interdependence will not be made explicit henceforth, and the
remainder of the paper is devoted to deter- mining the size of
drafts in successive periods. This section of the paper examines
the formulation of operating problems and concludes that their
structure is shared by models found in 'inventory theory.'
Beginning with the seminal work by Massd [1946] (which also
originated dynamic programing), a sizable literature on the
determination of optimal sequential decision rules for
reservoir releases has developed. Little [1955], Gessford and
Karlin [1958], and Amir [1967] significantly advanced Mass6's
results. As a generalization of their models, consider an inter-
connected system of I reservoirs having capacities C , ..., C I and
let C = (0, ..., CI).
Elaborating on the previous notation, let sd denote the amount
of water in the ith reservoir at the end of period t, let xd denote
the amount of water released from it, and let denote the random
amount of water that flows into it. Let st = (St , ''', Stl), Rt =
(Rt 1, ''', Rtl), and xt = (xt', '", xt I) so that (2) is valid
with the present vector notion if rt is replaced by Rt and the min
in (2) is taken separately for each compo- nent i = 1, ..., I.
It is convenient to use st_ - xd, namely, the storage after
discharge before inflow, as the decision variable instead of xt t.
Let
Yt = St-1 i -- Xt Yt = (Yt 1, '' ', Yt I) so that (2) is
equivalent to
st - min {C, Yt q- Rt} (18) If water cannot be pumped from one
reservoir to another, then the constraint on xt is 0 < xt <
st_x or
0 __ Yt -- St- (19) If water can be pumped from any reservoir to
any other during the same period then the constraint is weaker,
namely,
i E Yt i st-t _ 0 _< y t C (20) i i
If water can be pumped only from some reservoirs to others, then
the constraints will be intermediate between (19) and (20). For
specificity in the sequel the 'no pumping' assumption (19) will be
made. However, the essential properties of the follow- ing results
are preserved under (20) or any mixture of (19) and (20). For ease
of exposition the random vectors R, Ro., ... are assumed to be
independent. This assumption will be relaxed at the end of the
paper; if this were not possible, the results would be
valueless.
There is a one-to-one relation between the contents of a
reservoir and its freeboard. Let ud = C - s,_d denote the freeboard
in reservoir i at the beginning of period t and let u, = (ut , "',
ud). Similarly, let v,' = C - yd which can be in- terpreted as the
freeboard after discharging xd before the in- flow Rt occurs:
l)? = C i -- yt i = C i -- St_ + Xt t = Ut + Xt Now (18) and
(19) are equivalent to
ut+ = (vt - R) + u < v < C (21) The purpose of the
transformation (s, y) --, (u, v) is to observe that (21) is
formally the recurrence equation for successive in- ventory levels
in a 'lost sales' inventory system. If vd is the amount of 'goods'
available to satisfy 'demand' Rt , then the
i is -- R? if that difference is subsequent 'inventory level'
ut+ nonnegative. If Rt t tt t, then the excess demand is lost
(lost
= 0 as a result There is a huge literature on op- sales) and ut+
. timal sequential ordering rules (the 'order' is vd - ut ) in
inven- tory systems (as evidenced by the surveys by Veinott [1966]
and Clark [1972]). Therefore the appearance of (21) suggests that
inventory theoretic results and methodology may be ap- plicable to
reservoir management. That conjecture is verified in following
sections.
-
SOBEL: RESERVOIR MANAGEMENT MODELS 771
COSTS AND BENEFITS DEPENDING ONLY ON RESERVOIR LEVELS
Continuing from (21), let Gt(u, v) denote the expected cost in
period t (net of revenues) of a vector xt = v - u of discharges if
storage at the beginning of the period is st_ = C - u. Detailed
models for the construction of Gt( , ) are presented by many
authors such as Massd [1946], Little [ 1955], Gessford and Karlin
[1958], Amir [1967], and Roefs [1968]. Su and Deininger [1972]
present models in which the expected net costs in each period t
depend only on the reservoir level, and so there are functions g (
), '", gr( ) such that
gt(v) Gt(u, tg) t = 1, 2,'' ', T where T is the planning
horizon. The problem in this case is to choose v, ..-, vr
sequentially subject to (21) so as to
T
min E gt(vt) (22) t----1
where E denotes expectation. If the constraints (21) could be
ignored, then the problem
would have the following trivial myopic solution. Let tit be the
biggest (lexicographically) nonnegative vector in [0, C] that
minimizes gt( ); an optimal unconstrained solution is vt = tit (so
xt = tit - C + st-O for all t = l, ..., T. The constraints (21)
jeopardize this solution with the possibility that ut ; tit for
some t. That eventuality is precluded by the assumption
tit-dr _< tit+ t = 1, ..., T- 1,(23) where dt= (dr , "', dt
), dt being the minimum possible in- flow to reservoir i in period
t, and' so dt is the largest number satisfying P{Qt _> dt t} =
1. Of course, u < ti < ti2 < < tit is sufficient for
(23).
To see that (23) implies that vt = tit for all t is feasible,
first observe that u _< ti implies that v = ti satisfies (21)
for t = 1. Suppose that vt = tit is feasible; then
ut+ = (vt - Rt) + = (tit - Rt) + < (tit - dr) + (24) with
probability one by definition of dr. For the ith compo- nent in
(24), either ut - - dt < 0 = (ut - - cltt) + < ut+ -
-t is nonnegative by definition) or 0 < (ut - (because ut + -
_ -t from (23). The inductive conclu- dt) + = ut - -dt < Ut+
sion, with probability one, is that (23) ensures ut < tit,
and so vt = at is feasible for all t = 1, ..., T. Optimality of vt
= tit for all t follows from the fact that the imposition of
constraints (21) cannot reduce the cost of an optimal solution to a
previously unconstrained problem. Fourth Result
In summary, (23) implies that vt = tit for t = 1, ..., T is op-
timal for the problem of (21) and (22). The key step is to change
the decision variable from xt to either Yt or t9t. In either case,
given st_ or ut, there is a one-to-one relation with xt. When (23)
is valid, changing the decision variable replaces a burdensome
T-period dynamic programing recursion in an l- dimensional state
space with T separate /-dimensional minimization problems. If gt( )
is the same for all t, i.e., the cost structure is 'stationary,'
then just one /-dimensional minimization is needed. The whole
problem is said to have a myopic solution because vt = tit does not
stem from a recur- sion. The results above for problems (21) and
(22) with as- sumption (23) make up a special case of properties
that Veinott [1965] derived for multi-item inventory systems. Clark
[1972] surveys recent research on multi-item inventory systems.
Example 1. The problems (21) and (22) encompass costs and
revenues as linear functions of discharge quantities x, ' ', xr.
This fact was first exploited by Veinott and Wagner [1965] (who
mention that Martin Beckmann had recognized it earlier) in an
inventory problem. Suppose At' is the cost (net of benefits) of a
unit quantity released from reservoir i during period t. Let At =
(At , '' ', At ') so that At'xt (inner product) is the net cost of
xt. Let bt(ut) denote the net cost of having st_ = C - ut in
storage at the start of period t. Then the total cost K during T
periods, a random variable, is
T
K = [At'xt -'[- O t(ut+)l t--1
T
= - t--.1
Letting Ar+ -= 0 and using (21) and ut = (vt_ - Rt_O +, T
---- ' )+ t t t=2
+ + T
.... + t--.--1
+ - - Therefore
T
EK = Y'. gt(vt) -- A' u t=l
where A-u is uninfluenced by the decisions v, ..., vr and
gt(v) = At.v - At+.E(v - Rt) + + Ebt[(v - Rt) +] (25) Then the
problem of feasibly minimizing EK is equivalent to (22).
Example 2. As a special case of (25), suppose only a single
reservoir is being regulated so that I = 1 and also At = A, bt(u) =
B(u - 3,) 2, and P{Rt _< r} = r/C (0 _< r < C) for all t =
1, .., T. It is assumed that A > 0, B > 0, 3, > 0, and
2B(C - 3,) < A. For t < T, (25) takes the form
fo fo Cgt(v) = --A (v -- r) dr--[- B (v -- r --)2 dr -'l- B3'2(C
-- v) --I- CAr (25')
for 0 < v < C. Straightforward calculus shows that this
func- tion is concave on [0, C] (because of the inequality above).
Therefore its minimum on [0, C] occurs either at v = 0 or at v = C.
The assumption A > 0, B > 0, and 3, > 0 ensures that gt( )
is minimized at v = 0 if t < T. Because A r+ -= O, sub-
stitution in (25) for t = T yields (25') except that the first term
is absent. It is straightforward to show that gr( ) is convex on
[0, C] and minimized at 0. Then in the notation of (23), tit = 0
for all t; so (23) is satisfied if u = 0, i.e., So = C. Therefore
if the reservoir is full at the outset, then an optimal policy con-
sists of discharging xt = C - st_ - tit = C - C - 0 = 0 at the
beginning of each period t.
Example 3. The problem analyzed by Su and Deininger [1972] may
exemplify (25). They consider a single reservoir (Lake Superior) in
which the decision variable is the quantity discharged and the
expected costs (net of benefits) depend only on the level of the
reservoir at the beginning of a period.
-
772 SOBEL: RESERVOIR MANAGEMENT MODELS
FORMULATION OF OPERATING PROBLEMS: DYNAMIC PROGRAMING
Recall the notation Gt(u, o) for the single-period expected cost
of a vector x = v - u of discharges if initial storages are st_ = C
- u. The preceding section explained that the com- putation of an
optimal policy is often easy if Gt depends essen- tially only on v.
This section concerns more complex models in which that assumption
cannot be made. It is convenient to let p(a, fi) = (a - fi)+ for I
vectors a and fl. Then from (21),
Ut+x = P(t;t, Rt) At the beginning of each period t an outcome
ht of the
history Ht is observed, and then vt. is chosen (xt = vt - ut). A
reservoir release policy for period t is thus an I vector-valued
function Vt( ) ofht such that 0 < ut < Vt(h) < C for all
possi- ble outcomes ht of Ht (ut is specified in ht). Let Vt = (Vt,
'", Vr) denote an arbitrary release policy in periods t, t + 1, ...
T and let ft(Vt[ ht) be the conditional expected net cost in those
periods from following policy Vt given that the history Ht = hr.
That is
T
lb,)= i=t
The criterion of optimality will be ft( [ ). More precisely, a
policy V* = (V*, ..., Vr*) is'defined to be optimal if
ft(Vt*lht) < ft(Vt ht) (26) for all ht and Vt, t = 1, ..., T,
where Vt* = (Vt*, "', Vr*). A straightforward inductive proof
(starting with t: T) shows that if there exists an optimal policy,
then ft(Vt*lht) depends on ht only through ut, t = 1, ..., T, and
ft(Vt*lht) -= ft(ut), the {ft} satisfying
ft(u) = min {G,(u, v) '-k EJ,+,[p(v, R,)]}, (27) u_v_ c
0_u_C,t = 1,"',T(fr+( )=0).Conversely, ifthereisa sequence {ft(
)} satisfying (27), then there exists an optimal policy V* = (V*,.
., Vr*), Vt*(ht) -- Vt*(ut) being any value of v at which the
minimum in (27) is attained.
The recursive equation (27) will be used to characterize an
optimal policy. A virtue of the dynamic programing formula- tion
thus far is the absence of any heuristic devices such as the
'principle of optimality.'
There is a considerable literatur,e on optimal economic growth
under uncertainty and optimal consumption and sav- ings by an
individual during a lifetime. Equation (27) with 0 < v _( u
replacing u _( v _( C is formally equivalent to the recur- sive
equations of 'capital accumulation' which pervade that literature.
However, it is reasonable to assume that p( , r) is a concave
function (for each r) in the capital accumulation con- text whereas
(v - r) + lacks that useful property. References to the capital
accumulation literature and results for (27) when p( , r) is
concave can be found in part 1 of the volume edited by Szegb' and
Shell [1972].
MONOTONE POLICIES
The objective of this section is to demonstrate that there is an
optimal discharge policy Xt*(s), as a function of reservoir levels
s, such that 0 < aXt*'(s)/asj < 1, i,j = 1, ...,I. Then a
higher storage level induces a larger discharge, but the incre-
ment in the discharge does not exceed the increment in storage.
This property will be exploited computationally in the follow- ing
section.
It is useful to introduce the notation
Jt(u, 1)) = Gt(u, l)) q- Eft+xLo(/), Rt) ] so (27) can be
written
it(u) = min Jt(ll, t)) (0 _< U _< C) (29) u_v _ c
It is assumed that Gt( , ) is a weakly subadditive function on
its domain {(u, v): 0 _ u _ r _ C} A, namely, that cross- partial
derivatives of Gt, with respect to uj and ra, are non- positive.
Specifically, henceforth it is assumed for each t that
0 _> Gt(u + bey, v + 'Yen) - Gt(u, l; + 'Yen) -- Gt(u + 6el,
V) + Gt(u, v) (30)
where b, 'Y > 0 such that all four arguments in (30) lie in
A. Example 1 above is a special case of the usual situation
where Gt( , ) contains one term depending on reservoir levels
and another term depending on discharge quantities. Specifically,
if
Gt(u, v) = at(v - u) + bt(u) (31) then nonnegativity of the
cross-partial derivatives of at( ) en- sures (30). The cross
partials are nonnegative, for instance, if at(u- v) -= at*Y][v'- u
] with at* being convex.
The second term in (28), Eft+[o(v, Rt)], is trivially weakly
subadditive in (u, v), and a sum of weakly subadditive func- tions
is itself weakly subadditive. Therefore (30) and (28) imply for
each t that Jt( , ) is weakly subadditive. It follows from (29) and
Topkis [1968] that there is an optimal policy Vt*( ) for (29) such
that
Vt'(u) Vt(u' ) 0 _( u _( u' _( C (32) To interpret (32), let s
and s' be I vectors such that 0 < s' < s O, it is convenient,
following Veinott [1972], to rewrite (29) with the decision
variable as x = v - u instead of v and the state variable as s = C
- u instead of u. Let wt(s) --- ft(C - s) so that wt(s) = min
{Gt(C-- s, C-- s Jr- x)
O_z_x
+ EWt+l(min {C, s- x n t- Rt})} (34) Assumption (30) is too weak
to ensure fhe desired property for Xt*( ). Suppose also that
separation (31) applies to Gt( , ) so that
Gt(C - s, C- s + x) = at(x) + bt(C- s) It is assumed for each t
that at and bt have nonnegative cross- partial derivatives or, more
generally, that
at(x + be:) - at(x) bt(x + be:) - bt(x) (35) are nondecreasing
functions of xn for any k j, j - 1, ., I, b > 0, and 0 < x
< x + bej < C. It follows from (34), (35), and Topkis [1968]
that there is an optimal policy Xt*( ) for (34)
(28)
-
SOBEL: RESERVOIR MANAGEMENT MODELS 773 such that
Xt*(s') < Xt(s) 0 _< s' _< s _< C (36) Fifth
Result
Therefore optimal discharge quantities are increasing func-
tions of the storage quantities. Inequality (36) implies that Xt*(
) is-continuous except at upward jumps and that these are the only
kind ofjumps possible. However, (33) implies that only downward
jumps are possible. Taken together, (33) and (36) imply that Xt*( )
is continuous. Also (36) asserts monotonicity, and so ,It*( ) is
left-differentiable on (0, C], and all the elements of the Jacobian
matrix of Xt*( ) have values between 0 and 1, i.e..
0 < O Xt*(x) _ Os i _< 1 0 < s 0 when I = 1 in
groundwater man- agement models. The mathematical structure
underlying many of Burt's papers (see Burr [1970] for a
bibliography) is closely related to (27) and therefore to capital
accumulation models. A numerical example that illustrates the
computational benefits of monotone optimal policies is presented
after the following section.
ACCELERATED COMPUTATIONS FOR MONOTONE POLICIES
This section explains how (33), (36), and (37) can be ex-
ploited to accelerate the computation of an optimal policy. It is
assumed that variables have been discretized for computa- tions and
that C i is an integral multiple of the common unit storage and
release quantity for reservoir i, i = 1, ..., I. Then (37) is
equivalent to Xt*'(s + ej) {Xt*'(s), Xt*'(s) + 1} (38)
i = 1, ...,I j = 1, ...,I In words, suppose that Xt* is known at
some lattice point s, 0 < s < C. Let s' = s + ej be an
adjacent lattice point such that
S ' s k .i
s ' = s-'[ ' 1 k = j Now ,Yt*(s + e;) is an optimal quantity to
release from reser- voir i if the contents of the reservoirs are
given by s + e;. Thus (38) asserts that either Xt*(s + e;) is the
same as Xt*(s) or else it is Xt*(s) + 1. Therefore the I vector
Xt*(s + e;) must lie on one of the 2 lattice points x adjacent to
Xt*(s) such that x _> Xt*(s). Formally, let M denote the set of
I vectors, each of whose components is zero or one. Thus M contains
2 ele- ments. Now (38) is equivalent to
Xt*(s + e.) {Xt*(s) + m' rn M} (39) By using v - u = x in (29),
the effect of (39) is ft(u- ej) = min l Jr(u- ej, Vt*(u) - m): rn
M} (40)
for 0 < u < C andj = 1, ..., I. For each t, (40) suggests
a recursion in u starting with u = C. Necessarily, Vt*(C) - C
because u < v < C. The lattice points in the/-dimensional
rect-
angle [0, C] can be generated sequentially by starting at u = C
and then repeatedly subtracting/-dimensional unit vectors ej. There
are
II (C q -- 1)-- 1 i--1
lattice points in [0, C] not including the point C. From (40),
at each of these points at most 2 alternatives need to be com-
pared; so for each t, Vt*( ) is determined with fewer than
2r II (C' -3- 1) (41) i--1
comparisons of alternative decisions. A straightforward com-
putation using (29), by comparison, requires searching the lat-
tice points in [u, C] for each u. The rectangle [u, C] contains
I
II (c'- u' + i=l
points, so summing over u [0., C], there are I
Z u [0, tY] i=l
= 2-' II ( C+ 2)( C+ 1) (42) comparisons when the monotone
structure of Vt*( ) is not exploited. Some values of the ratio of
(41) to (42) are presented in Table 2. The computational
improvement of (40) with respect to (29) becomes more pronounced as
I and C increase.
Suppose that (30) is valid but (31) is invalid, or (31) applies
but the cross partials of at and bt are not all nonnegative. Then
in the discrete case, Vt*(u) < Vt*(u + e), but Vt*'(u + e) >
Vt*(u) + 1 is possible for some u, i, and j. The problem is still
simpler than (29) but computationally much more burden- some than
(40).
EXAMPLE
The following example illustrates the use of (40). Suppose I =
1; so a single reservoir is being controlled, its capacity is C =
3, and the planning horizon has T = 3 periods. Benefits and costs
in a period t are assumed to derive from a demand Dt for downstream
use of water and the level of the reservoir at the beginning of
each period. Suppose that the net cost of downstream use is
4(Dr - xt) + + 2(xt - Dr) + and that the net cost associated
with reservoir levels is (st-1 - 2) '. The demands D1, Do., and Ds
are assumed to be independent random variables with the same
distribution:
d P{Dt = d} 0 0.1 1 0.4 2 0.3 3 0.1 4 0.1
TABLE 2. Ratio of Number of Alternative Decisions per Stage in
Accelerated Versus Straightforward Algorithms,
O=C ..... Ct=c
Number of
Reservoirs I
Size of Each Reservoir c
5 10 50
1 0.571 0.333 0.078 2 0.327 0.111 0.006 5 0.187 0.037 2.6 X 10
-6
-
774 SOBEL: RESERVOIR MANAGEMENT MODELS
TABLE 3. Values of G(u, v) in the Example u v=O v=l v=2 v=3
0 7.8 4.4 3.4 4.2 1 6.8 3.4 2.4 2 7.8 4.4 3 10.8
Then Gt(u, v) takes the following form: Gt(u, t;) = 4E(Dt - t; +
u) + + 2E(v - u - Dr) +
+ (3 - u - 2) ' (43) Using the distribution of Dt to evaluate
the expectations results in the values for G(u, v) (see Table 3).
To illustrate the computation of G(u, v), consider G(2, 3):
For example, Xa(2) = 2 + Va(3 - 2) - 3 = -1 + Va(1) = -1 + 2-
1.
It is necessary to evaluate Ef[(v - Rs) +] because Jr(u, v) =
St(u, v) + Eft+[(v - Rt) + ] (45)
It is assumed that R, R., and Rs are independent random
variables with the following common distribution'
r P{Rt = r} 0 0.3 1 o. 1 2 0.1 3 0.3 4 0.2
Use of this distribution and the values off(u) results in the
fol-
G(2, 3) = 4[P{Ot = 0}(0 -- 3 + 2) + + P{Ot = 1}(1 -- 3 '-I- 2) +
+ P{O = 2}(2 -- 3 + 2) + + P{ D = 3}(3 -- 3 + 2) + + P{ D = 4}(4 --
3 + 2)+1 + 2[P{ D, = 0}(3 - 2 - 0) + + P{Dt = 1}(3- 2- 1) + + P{Dt
= 2}(3- 2-- 2) + + P{D = 3}(3- 2- 3) + q- P{Dt = 4}(3 -- 2-- 4)+1
q- (3-- 2-- 2) '
= 410.1(0) q- 0.4(0) q- 0.3(1) -3- 0.1(2) q- 0.1(3)1 q- 210.1(1)
q- 0.4(0) q- 0.3(0) q- 0.1(0)q-0.1(0)1 q- 1' = 3.4 q- 1 = 4.4
The weak subadditivity of G(u, v) can be verified either with
(43) or Table 3. For example,
[G(2, 3) - G(2, 2)] - [G(1, 3) - G(1, 2)] = -3.4- (-1) = -2.4 N
0
Because T = 3 and fr+( ) -- 0, we have J3(u, v) = G(u, v).
Therefore the recursion specified beneath (40) takes the form in
Table 4 for t - 3. To illustrate the computation, consider
/a(1) = min {J a(1, Va(2)), Ja(1, Va(2)- 1)} = min { Ja(1, 3),
Ja(1, 2)} = min {G(1, 3), G(1, 2)} = min {4.2, 3.4} = 3.4
The minimum is attained at J3(1, 2) = G(1, 2), and so Va(1) = 2.
In order to find the corresponding optimal discharge policy,
namely, Xa(s2), we use the identity t)t = C -- st- + Xt
so
Xt(st-O = st_ + Vt(C- st_) - C (44) In the present case, this
yields the following policy:
s,. Xs(s,) 0 0 I 1 2 1 3 2
lowing tabulation of Ef[(v - RO +]:
0 1 2 3
For example, for v = 3,
Era[(3- Rs) +]
- ] 3.4 3.4 3.7 5.7
= fa(3)P{Rs = 0} q- /a(2)P{Rs = 1} '-I- fa(1)P{Rs = 2} +
fa(0)P{Rs > 2}
= 10.8(0.3) + 4.4(0.1) + 3.4(0.1) +'3.4(0.5) = 5.7 Using this
equation and the above tabulation in (45) for
t = 2 results in values for J.(u, v) (see Table 5). For example,
J.(1, 2) = G(1, 2) + Ef[(2 - R0 + ] = 3.4 + 3.7 = 7..1. Table 5 is
used to computef(u) in the same fashion that Table 1 was used to
compute f(u). See Table 6 for the specific recursion. The optimal
discharge policy associated with V.(u) is deter- mined with (44)
and follows:
s, X4s,) 0 0 I 1 2 1 3 2
TABLE 4. Recursive Computation off(u) TABLE 5. Valises of d,.(u,
v) in the Example
u Computation offa(u) f(u) V3(u) v=0 v=l v=2 v=3
3 G(3, 3) = 10.8 10.8 2 min {G(2, 3), G(2, 2)} - min {4.4, 7.8}
4.4 1 min {G(1, 3), G(1, 2)} = min {4.2, 3.4} 3.4 0 min {G(10, 2),
G(0, 1)} = min {3.4, 4.4} 3.4
3 0 11.2 7.8 7.1 9.9 3 1 10.2 7.1 8.1 2 2 11.5 10.1 2 3 16.5
-
SOBEL: RESERVOIR MANAGEMENT MODELS 775
TABLE 6. Recursive Computation off(u) Computation off(u) f(u)
Vs(u)
ds(3, 3) = 16.5 16.5 3 rain {ds(2, 3), d:(2, 2)} = rain {10.1,
11.5} 10.1 3 min {ds(l, 3), ds(l, 2)} = rain {.8.12, 7.1} 7.1 2
rain {ds(O, 2), ds(O, 1)} = rain {7.1, 7.8} 7.1 2
TABLE 8. Recursive Computation of f(u) Computation off(u) f(u)
V(u)
d(3, 3) = 21.0 21.0 3 min {d(2, 3), d(2, 2)} = min {14.6, 15.8}
14.6 3 rain {d;(1, 3), d(1,2)} = rain {12.6, 11.4} 11.4 2 rain
{dffO, 2), d(O, 1)} = rain {11.4, 11.5} 11.4 2
Tables 7-9 present the remaining computations in the same order
in which they are performed.
v A[(v - 0 7.1 1 7.1 2 8.0 3 10.2
CORRELATED INFLOWS
This section is concerned with the impact of conditional
dependence in the inflow sequence of random I vectors Rx, R2, ",
Rr. The preceding stochastic models already encompass
interdependence of the I components of Rt = (Rt x, ''', Rtt). Also,
the stochastic minimax capacity model allowed dependence among Rx,
"', Rr. In general, the impact of dependence is to expand the
dimension of the state space but to leave unchanged the general
form of an optimal policy. This assertion will be illustrated for
the stochastic models in pre- ceding sections.
Let Wt(Rx, '' ', Rt-x) denote a statistic of Rx, "', Rt-x that
is sufficient for Rt, namely,
P{Rt < alWt} = P{Rt < alRx, ..., Rt_x} (46) Such a
statistic always exists because (Rx, "', Rt-O is itself sufficient.
Let ft be the set of all possible values of Wt. Then instead of
(27) or (28) and (29) the generic problem is
Jr(u; co) = min Jr(u, v; co) (47) u _< v _< C
Jr(u, v; w) = Gt(u, v; w) + Eft+x(p[v, Rt], t[w, Rt]) (48)
(O_
-
776 SOBEL: RESERVOIR MANAGEMENT MODELS
NOTATION
at(x) net cost in period t of drawdown vector x. At net unit
cos: of drawdown from reservoir i during
period t. bt(u) net storage cost of having C - u in storage at
start of
period t. c reservoir capacity as endogenous variable.
exogenous capacity of reservoir i. dt minimum possible inflow to
reservoir i in period t. Dt random downstream demand for water
during
period t. E expectation (of a random variable).
ft(u) expected cost of an optimal policy during periods t
through T if C - u is in storage at the start of period t.
ft maximal drawdown during period t (sometimes not indexed by
t).
gt(l)) Gt(u, v) when it does not depend on u. Gt(u, v) expected
net cost in period t of a vector v - u of
drawdowns if storage levels at the beginning of the period are C
- u.
ht history of events up to the beginning of period t. i generic
index number of reservoir superscripted on
other variables. I number of reservoirs in system.
Jr(u, t)) expected cost during periods t, ..., T of drawdown u -
v in period t, when the initial storage was C - u, followed by an
optimal policy.
K total (random) cost during T periods. Lt minimal storage in
period t plus water obliged to be
held back for future use. mt minimal storage at end of period t
(sometimes not
indexed by t). M set of all I vectors each of whose components
is zero
or one.
N expected costs irrelevant to the decision problem. P
probability measure. qt minimal drawdown during period t (sometimes
not
indexed by t). rt inflow during period t. st quantity of water
in storage at end of period t. t generic index number of period. T
length of planning horizon;
ud freeboard in reservoir i a,t beginning of period t. vt
freeboard in reservoir i after discharge but before in-
flow in period t. w(x) f(c- x).
Wt( ) sufficient statistic. xt drawdown during period t. yt t
quantity of water in reservoir i during period t after
drawdown before inflow, equal to st_x - xt . at minimum
probability level of a chance constraint.
p(v, r) storage at end of period if inflow is r and storage was
C - v just before inflow occurred.
3_ domain of the single-period cost function Gt( , ).
Acknowledgments. This manuscript was partially supported by
NSF grant GK-38121 and was presented at the Puerto Rico meeting
of the Institute of Management Sciences/Operations Research Society
of America in October 1974.
REFERENCES
Amir, R., Optimum operation of a multi-reservoir water supply
system, Ph.D. thesis, Stanford Univ., Stanford, Calif., 1967.
Burt, O. R., Optimal resource use over time with an application
to groundwater, Manage. $ci., 11, 80-93, 1964.
Burt, O. R., Groundwater storage control under institutional
restric- tions, Water Resour. Res., 6(6), 1540-1548, 1970.
Clark, A. J., An informal survey of multi-echelon inventory
theory, Nay. Res. Logist. Quart., 19(4), 621-650, 1972.
Derman, C., and M. Klein, Some remarks on finite horizon Marko-
vjan decision models, Oper. Res., 13, 272-278, 1965.
Eisel, L. M., Comments on 'The linear decision rule in reservoir
management and design' by Charles ReYelle, Erhard Joeres, and
William Kirby, Water Resour. Res., 6(4), 1239-1241, 1970.
Fiering, M., Streamflow Synthesis, Harvard University Press,
Cambridge, Mass., 1967.
Gessford, J., and S. Karlin, Optimal policy for hydroelectric
opera- tions, in Studies in the Mathematical Theory of Inventory
and Production, chap. 11, edited by K. J. Arrow, S. Karlin, and H.
Scarf, Stanford University Press, Stanford, Calif., 1958.
Griliches, Z., Distributed lags: A survey, Econometrica, 35(1),
16-49, 1967.
Harvard Water Resources Group, Operations research in water
quality management, Div. of Eng. and Appl. Phys. Harvard Univ.,
Cambridge, Mass., !963.
Iglehart, D. L., Capital accumulation and production for the
firm: Op- timal dynamic policies, Manage. Sci., 12, 193-205,
1965.
LeClerc, G., and D. H. Marks, Determination of the discharge
policy for existing reservoir networks under differing objectives,
Water Resour. Res., 9(5), 1155-1165, 1973.
Little, J. D.C., The use of storage water in a hydroelectric
system, J. Oper. Res. Soc. Amer., 3, 187-197, 1955.
Loucks, D. P., Some comments on linear decision rules and chance
constraints, Water Resour. Res., 6(2), 668-671, 1970.
Mass6, P., Les Rdserves et la Rdgulation de l'Avenir Dans la Vie
Economique, 2 vols., Hermann, Paris, 1946.
Miller, B. L., and H. M. Wagner, Chance constrained programming
with joint constraints, Oper. Res., 13(6), 930-945, 1965.
ReVelle, C., E. Joeres, and W. Kirby, The linear decision rule
in reser- voir management and design, 1, Development of the
stochastic model, Water Resour. Res., 5(4), 767-777, 1969.
Roefs, T. G., Reservoir management: The state of the art, Rep.
320- 3508, IBM Sci. Center, Yorktown, N.Y., 1968.
Russell, C. B., An optimal policy for operating a multipurpose
reser- voir, Oper. Res., 20(6), 1181-1189, 1972.
Sobel, M. J., Chebyshev optimal waste discharges, Oper. Res.,
9(2), 308-322, 1971.
Su, S. Y., and R. A. Deininger, Modelling the regulation of Lake
Superior under uncertainty of future water supplies, in Proceedings
of the International Symposium on Uncertainties in Hydrologic and
Water Resource Systems, vol. 2, pp. 555-575, University of Arizona,
Tucson, 1972.
SzegiS, G. P., and K. Shell, Mathematical Methods in Investment
and Finance, North Holland, Amsterdam, 1972.
Topkis, D. M., Ordered optimal solutions, Ph.D. thesis, Stanford
Univ., Stanford, Calif., 1968.
Veinott, A. F., Jr., Optimal policy for a multi-product,
dynamic, non- stationary inventory problem, Manage. Sci., 12(3),
206-222, 1965.
Veinott, A. F., Jr., The status of mathematical inventory
theory, Manage. Sci., 12(11), 745-777, 1966.
Veinott, A. F., Jr., Sub-additive functions on a lattice in
inventory theory, paper presented at 19th International Meeting,
Inst. of Manage. Sci., Houston, Tex., April 1972.
Veinott, A. F., Jr., and H. M. Wagner, Computing optimal (s, $)
in- ventory policies, Manage. Sci., (5), 525-552, 1965.
(Received February 5, 1974; revised February 13, 1975; accepted
March 20, 1975.)