Approximate Dynamic Programming and Reinforcement Learning for Nonlinear Optimal Control of Power Systems November 4, 2003 Ronald Harley Georgia Institute.

Approximate Dynamic Programming and Reinforcement Learning for Nonlinear

Optimal Control of Power Systems

November 4, 2003

Ronald HarleyGeorgia Institute of Technology

ECS-0231632ECS-0080764

Kumar VenayagamoorthyUniversity of Missouri-Rolla

Adaptive Critic Design: Nonlinear Optimal Control

PlantInformaton

UtilityFunction (U)

Optimal cost-to-gofunction (J)

Critic Networks :To minimize the value (of derivatives)

of J with respect to the states

Derivatives via BP

Model Network(Identifier) : To learnthe dynamics of plant

Model Network

Action Network :To find optimal control

u

Plant

Control

Reinforcement Learning

0

* )()(k

k ktUtJ

STATCOM Control

Simulation Results

100ms SC at PCC,

Line Voltage , Generator Terminal VoltageV

The simplified schematic of the SSSC (160 MVA, 15KV VL-L)

Optimal control for FACTS devicesInternal control for static series synchronous

compensator (SSSC)

Series VSI

SynchronousGenerator

V dc

Inf. bus

v s v rv c

is v x

Turbine-Governor

AVR -Exciter

rexe

SSSC

+

GTO

Control

DHPNCCONVC

idc

Optimal control for FACTS devicesInternal control for SSSC (CONVC)

PI Based internal controller (CONVC) for the SSSC

Synchrnouslyrotating reference

transformations

kk ip

+

+

)ˆ

ˆ(tan

ˆˆ

1

22

cd

cq

dc

cqcd

v

vα

V

vvm

Vdc *

Vdc

+-

id

PI- Vdc

iq

- s

kk ip

iq

PI- iq

cqv̂

+iq*

Vdc

s

kk ip

PI- ip

ip

cdv̂+id*

-

ia ib ic

m

+

GTO gate controlof series VSI

+

P*Q*

Vdc

id

Vectorphase-locked

loop

va

vb

vc

Real and reactivecurrent computation

V r

V'dc

Publication: N.G. Hingorani and L. Gyugyi, “Understanding FACTS-Concepts and Technology of Flexible AC Transmission Systems”, IEEE Press, New York, 2000.

Optimal control for FACTS devicesCase study: 100 ms three phase short circuit test at

receiving-end (infinite-bus)

Rotor angle

0 1 2 3 4 5 620

30

40

50

60

70

80

90

100

110

120

130

Time [s]

d [D

egre

e]

Uncompensated

CONVC

DHPNC

Schematic single-line diagram showing an SCRC with external controller (160 MVA, 15KV VL-L)

Optimal control for FACTS devicesExternal control for series capacitive reactance

compensator (SCRC)

SynchronousGenerator Inf. bus

v s

is

SCRC

v r 0

re2xe2

re1xe1Turbine-

Governor

AVR -Exciter

CsT1

1

Internal Controlof SCRC

VoltageSourceInverter

V dc

v c

+GTO

W

WC

sT1

sTK

X C

Filtering DampingController

CsT1

1

External Control

+

+

Line #1

Line #2

*CX

CX

Optimal control for FACTS devicesDHP based external controller (DHPEC)

Schematic single-line diagram showing the DHP based external controller (DHPEC)

SynchronousGenerator

Inf. bus

vs

is

SCRC

re2 xe2

re1 xe1Turbine-Governor

AVR -Exciter

Internal Controlof SCRC

VoltageSourceInverter

Vdc

vc

+GTO

XC

+

+

Line #1

Line #2

*CX

CX

vr

DHP basedexternal controller

(DHPEC)

Optimal control for FACTS devicesCase study: Step changes X*C [pu]

0 5 10 15 20 25-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

Time [s]

[ra

d/s

]

Fixed XC*

KC=1.5

DHPC

Speed deviation

Application in Multi-Machine power system

FACTS(SCRC)

Gen 31600 MVA

Gen 15000 MVA

Gen 22200 MVA 200 km

AREA 1 AREA 2

T1

T2

T3

T4

T5

T6

500 kV 500 kV 500 kV500 kV 13.2 kV

13.2 kV

13.2 kV

13.8 kV

115 kV

Line 1

Line 2

Line 3

Line 4

Line 5

Z 1

Z 2

1

2

34 5

610

7

DHPNC-G

CONVC-GS1

DHPEC-S

CONVEC-SS2 3

11Industrial load

Residentialload

115 kV 13.8 kV

8 9

Large-scale multi-machine power system

A UPFC in the POWER SYSTEM

InfiniteBusShunt

InverterSeries

InverterVd

c

SeriesInverterControl

ShuntInverterControl

V1

Vdcref

R1, L1

V2V1

V1ref

Z1Synch

Generator

Governor

AVR

Exciter

+

-

UPFC

Z1

V1ref

Vdc

Pref

Pinj

Qinj

Q ref

1 2

Pout,

Qout

Verr

Vdcerr Perr

Qerr

R2, L2

Vr

TurbinePref

Neurocontroller

Neuroidentifier

Q

P

ed

eq

Neurocontroller

Neuroidentifier

Vdc

V

epd

epq

5 6 7 8 9 10 11-50

0

50

100

150

200

250

Time (sec)

Load

Ang

le(°

)

UPFC

PI

5 6 7 8 9 10 11-50

0

50

100

150

200

250

Time (sec)

Load

Ang

le(°

)

UPFC

PI

NC

PI

Responses of the Generator for a 180 ms 3- phase Short Circuit at bus 2 at P=0.8 p.u & Q=0.15 p.u

Load angle

Speed response

4 5 6 7 8 9 10 110.96

0.98

1

1.02

1.04

1.06

1.08

1.1

Time (sec)

Spe

ed (

Pu)

UPFC

PI

4 5 6 7 8 9 10 110.96

0.98

1

1.02

1.04

1.06

1.08

1.1

Time (sec)

Spe

ed (

pu)

UPFC

PI

NC

PI

Micro-Machine Research Lab. at the University of Natal, Durban, South Africa

Gen. #1: Trans. Line Impedance Increase

10 10.5 11 11.5 12 12.5 13 13.5 14 14.5 1520

25

30

35

40

Time in seconds

Load

ang

le in

deg

rees

DHP_CONV

CONV_PSS_CONVCON_CONV

10 10.5 11 11.5 12 12.5 13 13.5 14 14.5

0.97

0.98

0.99

1

1.01

Time in seconds

Ter

min

al v

olta

ge in

pu

DHP_CONV

CONV_PSS_CONV

CONV_CONV