Adaptive Stochastic Control for the Smart grid

Adaptive Stochastic Control for the Smart grid

Qinghua ShenSmart grid meeting

2

• Introduction to the control of smart grid adaptive stochastic control, smart grid

• Adaptive Stochastic Control basis of stochastic system, policy search and approximation, convergence

• Example: distributed generation despatch with storage ADP for resource allocation, value function approximations

• Challenges

Outline

3

Introduction to the control of smart grid

• Control of the smart grid

goal for control: instantly, corrective and dynamically Self-healing: auto repair or removal of potentially faulty equipment Flexible: rapid and safe interconnection of distributed generation and storage Predictive: statistics, machine learning, predictive models interactive: appropriate information is provided transparently in near real time Optimal : operators and customers efficiently and economically Secure : cyber- and physical-security

4

Introduction to the control of smart grid

• Major Components

5

• Introduction to the control of smart grid adaptive stochastic control, smart grid

• Adaptive Stochastic Control basis of stochastic system, policy search and approximation, convergence

• Example: distributed generation despatch with storage ADP for resource allocation, value function approximations

• Challenges

Outline

6

Adaptive Stochastic Control

• Stochastic system

State variables physical state: energy amount, status of a generator information state: current and historical demand, price and weather belief state: probability distributions

The decisionswhether or charge/discharge, use backup

The exogenous Informationall the dimensions of uncertainty

The Transition Functiongiven the state, decisions and exogenous information, determines next state

The Objective Functionmetrics that governs how we make those decisions and evaluate the performance of policies of the controller designs

7

Adaptive Stochastic Control

• Policies

maps the information in state S to a decision x. , which is the state variables, capturing energy resources Rt, exogenous information pt, and belief state Kt. The problem

is known variously as the value of the a policy or the cost to go function. can be a cost function if we minimize, or contribution function if we maximize.

Cost include generating electricity, purchasing fuel, losses due to energy conversion, cost of repair, and penalties for curtailing loads

policy for what includes whether to charge/discharge, when to run a distributed generator, how much energy draw from grid for every customer in every networks and the utility

8

Adaptive Stochastic Control

• Design a robust policy: four classes

Myopic Policies minimize next-period cost without decisions for future( special structure good) Look-ahead PoliciesOptimize over some time horizon using a forecast of the possible variability of exogenous events such as weather. Forecast can either be deterministic forecasts or stochastic forecasts Policy Function ApproximationsFunctions return an action given a state, without solving any form of optimization, including: rule-based lookup table; Parameterized rules(threshold hold); statistical functions Policy based on Value Function ApproximationsOptimal policy obtained from HJB equation, to avoid curses of dimensionalitya) Approximate to eliminate the expectation; b) replace the value function with a computationally tractable approximation; c) solve the resulting deterministic maximization problem using a commercial solver

9

Adaptive Stochastic Control

• ADP and the Post–Decision State Value function approximation

when structure of a policy is not obvious, estimates the value of being in a state

When x is a vector, solve the maximization problem is problematic(expectation hard to compute exactly)---refer to stochastic search Post-Decision state

Post decision state determined through current state and action

10

Adaptive Stochastic Control

• Design policy

look up tables

Parametric models

With this strategy, we face the challenge of first identifying the basis functions, and then tuning the parameters Nonparametric models

handle high-dimensional, asymptotically unbiased• Kernel regression;• Support Vector regression;• Neural networks;• Dirichlet process mixtures.

11

Adaptive Stochastic Control

• Policy search

Direct policy search Depend on Monte Carlo sampling ----stochastic searchMethods: sequential krigingUsing the knowledge gradientApplied when the policy structure is apparent

Bellman residual minimization for value function approximationsThis is the most widely used strategy for optimizing policies, and encompasses a variety of algorithmic approaches that include approximate value iteration (including temporal difference learning) and approximate policy iteration

12

ASC for distributed generation despatch

• Approximate Dynamic Programming for resource allocation

Resource allocationhow much energy to store in a battery, whether a diesel generator should be turned on, and whether a mobile storage device (and/or generator) should be moved to a congested location.

A general model

Rta is the number of resources with attribute vector a xtad is the number of resrouces we act on with a decision of type d. a decision d can be (-1,0,1) to discharge, hold, or recharge a battergy (0,1) to turn a distributed generator off or on

13

Adaptive Stochastic Control

• Value Function Approximations for resource allocation

Approximate the value function

resource allocation utility function: concavity propertyApproximate value function by the post decision resource vector Separable piece-wise linear function

Estimate piecewise linear concave functions by iteratively stepping forward through time and updating value functions

14

Adaptive Stochastic Control

• Experimental work

evaluate the resultsResource determine the quality of the resulting policy is a major challengefit the value functions for a deterministic problem, and compare the resulting solution to the optimal solution for the deterministic problem, obtained by using a commercial solver limited by the size

15

Challenges

Convergence Only some structure can be proofed to be convergence with approximationConcavity is an important category

For smart gridBeneficial to both utility and end users– enough incentiveThe track of key performance metrics