© 2007 Warren B. Powell Slide 1 The Dynamic Energy Resource Model Lawrence Livermore National Laboratories September 24, 2007 Warren Powell Alan Lamont Jeffrey Stewart Abraham George © 2007 Warren B. Powell, Princeton University
Jan 04, 2016
© 2007 Warren B. Powell Slide 1
The Dynamic Energy Resource Model
Lawrence Livermore National LaboratoriesSeptember 24, 2007
Warren PowellAlan Lamont
Jeffrey StewartAbraham George
© 2007 Warren B. Powell, Princeton University
© 2007 Warren B. Powell Slide 2
The dynamic energy resource model
Questions:» How will the market evolve in terms of the adoption of
competing energy technologies?• How many windmills, and where?• How much ethanol capacity?• How will the capacity of coal, natural gas and oil evolve?
» What government policies should be implemented?• Carbon tax? Cap and trade?• Tax credits for windmills and solar panels?• Tax credits for ethanol?
» Where should we invest R&D dollars?• Ethanol or hydrogen?• Batteries or windmills?• Hydrogen production, storage or conversion?
© 2007 Warren B. Powell Slide 3
The dynamic energy resource model
Features we need:» Multiple time scales
• Model will plan decades into the future, but reflect decisions and processes that occur on hourly, daily, seasonal and yearly levels.
» Multiple forms of uncertainty• We will model dynamic information processes that describe
the evolution of technology, climate, weather, prices and wind.
» Multiple levels of spatial granularity• The model will be able to run at different levels of spatial
aggregation, capturing the geographic substitution of different types of energy.
» Multi-attribute representation of markets• We want to be able to distinguish energy demands to capture
usage and lifestyle patterns.
© 2007 Warren B. Powell Slide 4
Outline
A deterministic modelA stochastic, dynamic energy modelADP for energy capacity management
© 2007 Warren B. Powell Slide 5
Outline
A deterministic modelA stochastic, dynamic energy modelADP for energy capacity management
© 2007 Warren B. Powell Slide 6
Deterministic models
Deterministic, linear programming-based models» Basic model:
» Features:• Can model flows of energy and substitution of energy
resources over time.• Assumes a deterministic view of the world (everything is
known now).
1 1min subject to , 0t t t t t t t tt
c x A x B x R x
© 2007 Warren B. Powell Slide 7
Deterministic models
A single large linear program:
TimeSpace
© 2007 Warren B. Powell Slide 8
Deterministic models
Limitations» Unable to model uncertainty in technology, climate,
prices.
» Unable to model activities at a high level of detail. Large linear program limits the number of rows, which grows rapidly as we use finer representations of resources and markets.
» Traditionally uses a discrete time representation, making it hard to handle fine time scales (e.g. hourly) over long horizons (e.g. 50 years).
© 2007 Warren B. Powell Slide 9
Outline
A deterministic modelA stochastic, dynamic energy modelADP for energy capacity management
© 2007 Warren B. Powell Slide 10
Stochastic, dynamic model
The state of a resource:
Capacity of facilities
Location
Cost
Carbon output
Age
Reserves
ta
© 2007 Warren B. Powell Slide 11
Stochastic, dynamic model
Modeling multiple energy resources:» The attributes of a single resource:
» The resource state vector:
» The information process:
The attributes of a single resource
The attribute space
a
a
A
ˆ The change in the number of resources with
attribute .taR
a
The number of resources with attribute
The resource state vector
ta
t ta a
R a
R R
A
© 2007 Warren B. Powell Slide 12
Stochastic, dynamic model
Modeling market demands:» The attributes of a single type of demand:
» The demand state vector:
» The information process:ˆ The change in the number of demands with
attribute .tbD
b
The attributes of a demand to be served.
The attribute space
b
b
B
The number of demands with attribute
The demand state vector
tb
t tb b
D b
D D
B
© 2007 Warren B. Powell Slide 13
Stochastic, dynamic model
The system state:
, , System state, where:
Resource state (how much capacity, reserves)
Market demands
"system parameters"
State of the technology (costs, pe
t t t t
t
t
t
S R D
R
D
rformance)
Climate, weather (temperature, rainfall, wind)
Government policies (tax rebates on solar panels)
Market prices (oil, coal)
© 2007 Warren B. Powell Slide 14
Stochastic, dynamic model
The three states of our system» The state of a single resource/entity
» The resource state vector
» The system state vector
1
2
3
t
t t
t
a
a a
a
1
2
3
ta
t ta
ta
R
R R
R
, ,t t t tS R D
© 2007 Warren B. Powell Slide 15
Stochastic, dynamic model
The decision variable:
New capacity
Retired capacity
:
Type
Location
Technology
t
for eachx
© 2007 Warren B. Powell Slide 16
Stochastic, dynamic model
Exogenous information:
ˆ ˆ ˆNew information = , ,t t t tW R D
where:
ˆ Exogenous changes in capacity, reserves
ˆ New demands for energy from each source
ˆ Exogenous changes in parameters.
Change in technology
Ch
t
t
t
R
D
ange in climate/weather
Change in prices/market supplies
© 2007 Warren B. Powell Slide 17
Stochastic, dynamic model Hourly
» Daily temperature variation» Wind» Equipment failures
Daily» Fluctuation in spot prices» Short term demand» Major weather events» Transportation delays (movement of coal and oil)
Monthly» Seasonal variation (temperature, water flow for hydro, population shifts)» Medium term weather patterns» Significant supply disruptions (major hurricane, wars)
Yearly» Changes in technology» Demand patterns (SUV’s)» Long term climate cycles (including global warming)» Spatial patterns in population growth» New supply discoveries (major oil fields)» Intervention of foreign governments in markets» Long term supply contracts
tW
© 2007 Warren B. Powell Slide 18
Stochastic, dynamic model
The transition function
1 1( , , )Mt t t t tS S S x W S
t t+1
© 2007 Warren B. Powell Slide 19
Stochastic, dynamic model
Our strategy:» Basic model:
» Features:• Simulation-based – We simulate forward in time using a very
general-purpose transition model.• Handles virtually any form of uncertainty.• Can use a range of policies for different types of decisions,
from simple dispatch rules to more sophisticated policies that look into the future.
1 1
( )
, ,
t t t
Mt t t t
x X S
S S S x W
Make a decision using policy
Update state using system model
© 2007 Warren B. Powell Slide 20
Information and decisionsInformation
T – changes in technologyS – changes in energy suppliesP – changes in energy prices
W – Weather
DecisionsI – Changes in energy capacity infrastructure (new plants, new fields)S – Short term changes in suppliesR – R&D investmentsM – Market response
T T T T
W WW W WW W WW W WW W WWW WWS S S S S S SSP P PP P P P P P P P P P P P P P P P
I I I IR RR R R R R R R RS S S S S S S S S S S S S S S S S SM M M M M M M M M
© 2007 Warren B. Powell Slide 21
Making decisions
Dispatch decisions:» Use the technology with the lowest marginal cost.
» Small linear program to handle substitution of different types of power.
GENERATORS MARKETS
2
1
4
3
A
C
B
I
II
III
ENERGY SOURCES
© 2007 Warren B. Powell Slide 22
Making decisions
Hydro power management» Forecast inflow and outflow to reservoirs to determine
amount available for generating electricity
r0l
1l
L
1 2 3 t
r0l
1l
L
1 2 3 t
r
0l
1l
1l
L
1 2 3 t
2lr
0l
1l
1l
L
1 2 3 t
2l r
1l2l
2l
0l
L
1 2 3 t
r
1l2l
2l
0l
L
1 2 3 t
© 2007 Warren B. Powell Slide 23
Making decisions
Purchasing new capacity:» A decision to add capacity in year t changes the
capacity available in year t+1.
» Resource transition function
Resource state vector at time .t ta a AR R t
1 1ˆ
t t t tR R x R
Resource state vector
Capacity change decisions
Exogenous changes to resources
© 2007 Warren B. Powell Slide 24
Making decisions
Purchasing new capacity:» We want our capacity acquisition decisions to mimic the
intelligence that companies/financial markets make.
» We propose to “simulate Wall St.” by solving the capacity acquisition problem as an optimization problem to find the policy that solves:
» The optimal policy is characterized by Bellman’s equation:
» Problem: Solving this equation is computationally intractable because of the “three curses of dimensionality.”
min ( , ( )) t t t tt
E C S X S
1 1( ) min ( , ) ( )tt t x t t t t tV S C S x EV S
© 2007 Warren B. Powell Slide 25
Approximate dynamic programming
Solving the dynamic program using approximate dynamic programming:» Step 1: Break transition into two steps:
» Step 2: Formulate value function around post-decision state:
» Step 3: Replace value function with approximation
» Step 4: Design strategy for updating the approximation
1 1
Post-decision state variable
ˆ New pre-decision state variable
xt t t
xt t t
R R x
R R R
( ) min ( , ) ( )t
x xt t x t t t t tV S C S x V R
( ) arg min ( , ) ( )t
xt t x t t t t tX S C S x V R
© 2007 Warren B. Powell Slide 26
Part VII - CASTLE Lab NewsCASTLE Lab News
New Modeling Language Captures Complexities of Real-World Operations!
75 cents
Spans the gap betweensimulation and optimization.
CASTLE Lab announced the development of a powerful new simulation environment for modeling complex operations in transportation and logistics. The dissertation of Dr. Joel Shapiro, it offers the flexibility of simulation environments, but the intelligence of optimization. The modeling language will allow managers to quickly test continued on page 3
Thursday, March 2, 1999
© 2007 Warren B. Powell Slide 27
© 2007 Warren B. Powell Slide 28
Outline
A deterministic modelA stochastic, dynamic energy modelADP for energy capacity management
© 2007 Warren B. Powell Slide 29
ADP for energy resource management
oiltx
2008
oiltR ˆ oil
tD ˆ oiltˆ oil
tR
New information 2009
1oiltR 1
oiltx 1
ˆ oiltD 1
ˆ oilt 1
ˆ oiltR
New information
windtxwind
tR ˆ windtD ˆ wind
tˆ windtR 1
windtR 1
windtx 1
ˆ windtD 1
ˆ windt 1
ˆ windtR
coaltxcoal
tR ˆ coaltD ˆ coal
tˆ coaltR 1
coaltR 1
coaltx 1
ˆ coaltD 1
ˆ coalt 1
ˆ coaltR
corntxcorn
tR ˆ corntD ˆ corn
tˆ corntR 1
corntx 1
corntR 1
ˆ corntD 1
ˆ cornt
ˆ corntR
© 2007 Warren B. Powell Slide 30
We have to allocate resources before we know the demands for different types of energy in the future:
ADP for energy resource management
© 2007 Warren B. Powell Slide 31
We use value function approximations of the future to make decisions now:
ADP for energy resource management
© 2007 Warren B. Powell Slide 32
,,1x ntR
,,2x ntR
,,3x ntR
,,4x ntR
,,5x ntR
This determines how much capacity to provide:
ADP for energy resource management
© 2007 Warren B. Powell Slide 33
,1ˆ ( )ntv
,2ˆ ( )ntv
,3ˆ ( )ntv
,4ˆ ( )ntv
,5ˆ ( )ntv
Marginal value:
,,1x ntR
,,2x ntR
,,3x ntR
,,4x ntR
,,5x ntR
ADP for energy resource management
© 2007 Warren B. Powell Slide 34
1, 1,( )xt AB t ABV R
,1,
x nt ABR
Using the marginal values, we iteratively estimate piecewise linear functions.
ADP for energy resource management
© 2007 Warren B. Powell Slide 35
R1t
ktv
ktv
Right derivativeLeft derivative
1, 1,( )xt AB t ABV R
,1,
x nt ABR
Using the marginal values, we iteratively estimate piecewise linear functions.
ADP for energy resource management
© 2007 Warren B. Powell Slide 36
R1t
( 1)ktv ( 1)k
tv
1, 1,( )xt AB t ABV R
,1,
x nt ABR
Using the marginal values, we iteratively estimate piecewise linear functions.
ADP for energy resource management
© 2007 Warren B. Powell Slide 37
Piecewise linear, separable value function approximations:
Piecewise linear, separable:
( ) ( )t t tl tll
V R V R
L
ADP for energy resource management
© 2007 Warren B. Powell Slide 38
Approximate dynamic programming
t
© 2007 Warren B. Powell Slide 39
Approximate dynamic programming
© 2007 Warren B. Powell Slide 40
Approximate dynamic programming
© 2007 Warren B. Powell Slide 41
Approximate dynamic programming
Features» Extremely flexible
• Simulation-based modeling is able to handle high level of detail about energy resources and demands, time scales and uncertainties.
• Can handle mixed policies: – Myopic policies for dispatch problem– Rolling horizon procedures for hydro– Dynamic programming-based policies for capacity
acquisition
» Challenges• Value function approximations have to be designed to handle
the state of technology and climate.• Strategies have to be designed to guide the system to reach
different goals.• Measures for evaluating solution quality (is it realistic? near-
optimal?) need to be designed.
© 2007 Warren B. Powell Slide 42
© 2007 Warren B. Powell Slide 43
The dynamic energy resource model
Uncertainties (exogenous information processes)» Technology:
• Carbon sequestration• The cost of batteries, fuel cells, solar panels• The storage of hydrogen, efficiency of solar panels, …
» Climate: • Global and regional temperatures• Changing patterns of snow storage on mountains• Wind patterns
» Markets: • Global supplies of oil and natural gas• International consumption patterns• Domestic purchasing behaviors (SUV’s?)• Tax policies• The price of oil and natural gas
© 2007 Warren B. Powell Slide 44
The dynamic energy resource model
Alternative ways of solving large stochastic optimization problems:» Simulation using myopic policies – Using rules to determine
decisions based on the current state of the system. Rules are hard to design, and decisions now do not consider the impact on the future.
» Deterministic optimization – Ignores uncertainty (and problems are still very large scale).
» Rolling horizon procedures – Uses point estimates of what might happen in the future. Will not produce robust behaviors.
» Stochastic programming – Cannot handle multiple sources of uncertainty over multiple time periods.
» Markov decision processes – Discrete state, discrete action will not scale (“curse of dimensionality”)
© 2007 Warren B. Powell Slide 45
Dynamic energy resource management Proposed approach: Approximate dynamic
programming» Our research combines mathematical programming, simulation
and statistics in a dynamic programming framework.• Math programming handles high-dimensional decisions.• Simulation handles complex dynamics and high-dimensional
information processes.• Statistical learning is used to improve decisions iteratively.• Solution strategy is highly intuitive – tends to mimic human behavior.
» Features:• Scales to very large scale problems.• Easily handles complex dynamics and information processes.• Rigorous theoretical foundation
» Research challenge:• Calibrating the model.• Designing high quality policies using the tools of approximate
dynamic programming.• Evaluating the quality of these policies.