1
Design of Real-time Simulator for Electric Power
Distribution System Based on FPGAs
Zhiying Wang
Xiaopeng Fu
Tianjin University, China
2
Background
Motivation
– Integration of renewables in distribution networks
– Utilization of massive electronics in AC-DC hybrid distribution networks
Application
– Southern Grid of China : flexible DC distribution system pilot prototype verification
– Southern Grid of China : fast RT simulation platform for DC distribution networks
– State Grid of China : RT simulation platform for distribution network with DGs integrated
Components of simulation model
– large-scale distribution networks: high dimension of the model
– power electronics interfaced DG: vast range of time-scales
3
Real-time simulation of distribution network
4
Active Distribution Network FPGA-based RT simulator
2.8 2.9 3 3.1 3.2 3.3 3.4-200
-100
0
100
200
300
400
500
600
t/s
2.993 2.995 2.997 2.999 3.00160
80
100
120
140
PSCAD
FPGA
VP
V1
,c/V
6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:000
100
200
300
400
500
600
700
800
900
t/h
PSCAD
FPGA
S/W
/m2
6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:000
2000
4000
6000
8000
10000
12000
14000
16000
t/h
9:50 10:00 10:10 10:205800
6000
6200
6400
6600
PSCAD
FPGA
PP
B,p
v/W
dynamic behaviors
Overview of Multi-FPGA based Real-time simulator Design
system modelling
numerical algorithm
hardware design
platform design
MG
......
MG
...
MVAC
MVDC
LVDC
5
Simulator Design
6
system decomposition
hardware
design
platform design
coarse-to-fine grained system decomposition
map hardware resource to system modelling
Multi-FPGA system topology, synchronization mechanism, data interaction
7
fillter
transformer
PV
line
+-
+
-
+-
+-+
+PI
PIPI
PI
fillter
transformer
BAT
line
+-
+
-
+-
+-+
+PI
PIPI
PI
inverter control
d-axis electrical system
control system
2
3
4
5
6
11 12
9
10
1
7
8
15 16
13
14
19
20
17
18
22
21
coarsegrained
1
3 4 5
6
9 10 11
2
7 8
17 18 19 20 21
12
13 14 15 16
22
q-axis
d-axis
q-axis
inverter controlfine
grained
Electrical & Control
TLM
fine-grained parallelism
… …
System decomposition at various levels:
Coarse-to-fine grained decomposition of active distribution network
8
FPGA #1 FPGA #2 FPGA #3
Core #1
Core #2
Core #3
Core #5
Core #6
Core #4
Core #7
+-
+
-
+-
+-+
+PI
PIPI
PI
E
C
d
q PI PI
PI PI
PI PIVdc
Is VsIGBT XFM
data transmission
state_number
v(t)
history current source
matrix solving
Electrical system
G-11 G-1
2 G-1N
...
ROM
trigonometric function
exponential function
limiter
adder
compator
transfer function
Control system
supply source
control source
transformer
power electronic device
breaker
meter
electrical elements
RLCG
line
iIF(t-τ)
G-1
v&i
v&i
switch_sig
PWM
iSOURCE
injected node
currents
Bergeron interfacesimpler
FPGA #4 FPGA #5 FPGA #6 FPGA #7 FPGA #8
extendable
module level parallelism
element level parallelism
MANA
ih(t-Δt )
i
system levelparallelism
unit level parallelism
...
Core #...
9
combined mesh and linear array topology for the multi-FPGA system
plug and play
extendable
flexible configuration
maximum utilization of data
transmission channel
Multi-FPGA system topology:
10
· · ·
· · ·
sta_sync_sigFPGA1
(master)
FPGA2
(slave)
· · ·
· · ·
sta_sync_sigFPGA1
(master)
FPGA2
(slave)
· · ·
· · ·
FPGA3
(slave)
· · ·
2t
2t
2t
2t
3t
3t
1t
1t
1t
1t
1t
1t
1t
1t
2t
3t
3t
mint
1 min2t t
2 min3t t
3 min4t t
synchronize the simulation time-step of each FPGA to the same time reference
Single-rate synchronization mechanism Multi-rate Synchronization mechanism
Synchronization mechanism:
11
Multirate Simulation (target power electronics)
12
fast system
physical identification:
Including high-
frequency power
electronic devices
slow system
distribution network
without power
electronic devicesdistribution network
(slow system)
DG
FPGA #1
FPGA #2
FPGA #3
FPGA #...
FPGA #7
FPGA #6FPGA #4
DG(fast system)
v&i
controller
PWM
DG & ESSpower electronic
converters
PMS
M
+-
+
-
+-
+-+
+PI
PIPI
PI
DG
DG
DG
...
DG
...
FPGA #5
DG(fast system)
~~
~
PMSM
~~
......
Controlled signal
Multirate simulation between distributed generation(DG) & distribution network(DN)
13
Multirate simulation between distributed generation(DG) & distribution network(DN)
multirate interfacing method:
slow system Interpolated points
fast system
slow system
fast system
TLM decoupling between DG and DN
interpolation of slow system
averaging of fast system
14
Multirate simulation between electrical system & control system in a DG
multirate interfacing method:
electrical & control system decoupling
directly data interaction
asynchronous multirate simulation
v&i
PWM
DG & ESS(slow system)
PMSM ~
~
~
Controlled signal
power electronic converters(fast system)
controller(slow system)
+-
+
-
+-
+-+
+PI
PIPI
PI
15
Expanding Modeling Capability of Large-scale
Distribution Networks with the Matrix Exponential
Method
Xiaopeng Fu, Lecturer
Tianjin University, China
16
G
k
Ih
k
m
m
Geq
-Geq
-Geq
Geq
-Ih
+Ih
nodal admittance equation
ikm(t)
vk(t) vm(t)
Gu=i
ikm(t)
Ih(t-Δt)
vk(t) vm(t)Geq
RL branch
Electrical System Solution
difference equation using root-matching technique
𝑖km 𝑡 = e−∆𝑡𝑅/𝐿𝑖km(𝑡 − ∆𝑡) +1 − e−∆𝑡𝑅/𝐿
𝑅𝑣km 𝑡
equivalent circuit
etA appears in its Padé-approximant form
17
The Matrix Exponential etA
For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, 𝐞𝑨 is defined as
𝐞𝑨 ∶= 𝑰 + 𝑨 + Τ𝑨2 2! + Τ𝑨3 3! + ⋯
Closely related to state transition of dynamic systems
Solver Integrator Padé Approx. ex ≈ rkm(x)
FE 𝑰 + 𝑡𝑨 𝑟10(𝑥) = 1 + 𝑥
BE 𝑰 − 𝑡𝑨 −1 𝑟01(𝑥) = 1/(1 − 𝑥)
TRAP 2𝑰 − 𝑡𝑨 −𝟏 2𝑰 + 𝑡𝑨 𝑟11(𝑥) = (2 + 𝑥)/(2 − 𝑥)
ARTEMiSArt5
𝐞𝑡𝑨 ≈𝑰 + 2𝑡𝑨/5 + 𝑡𝑨 𝟐/20
𝑰 − 3𝑡𝑨/5 + 3 𝑡𝑨 𝟐/20 − (𝑡𝑨)𝟑/60
LRivk vm
ihist
ivk vm
Geq
Nodal Analysis State Space Analysis
Solver Conductance 𝑮𝒆𝒒 History current source 𝒊𝐡𝐢𝐬𝐭 𝒕
TRAP 𝑅 +2𝐿
Δ𝑡
−1
𝐺eq 𝑣𝑘𝑚 𝑡 − Δ𝑡 − 𝑅 −2𝐿
Δ𝑡𝑖 𝑡 − Δ𝑡
Root Matching*
1 − e−Δ𝑡𝑅/𝐿
𝑅e−Δ𝑡
𝑅𝐿 ∙ 𝑖 𝑡 − Δ𝑡
Linear System Theory
ሶ𝒙 𝑡 = 𝑨𝒙 𝑡 + 𝒃 𝑡 ,has state transition rule:
𝒙 𝑡 = 𝐞𝑡𝑨𝒙 0 + න0
𝑡
𝐞 𝑡−𝜏 𝑨𝒃 𝜏 𝑑𝜏
branch discretizationwith scalar exponential
18
A-stability, Free from numerical oscillation problems, etc.
The matrix exponential formula for nonlinear systems…
Variation-of-constant formula:
𝒙 𝑡 = 𝐞 𝑡−𝑡0 𝑳𝒙0 +න𝑡0
𝑡
𝐞 𝑡−𝜏 𝑳𝑵 𝜏, 𝒙 𝜏 𝑑𝜏
General nonlinear system
ሶ𝒙 𝑡 = 𝑭 𝑡, 𝒙 𝑡 = 𝑳𝒙 𝑡 + 𝑵 𝑡, 𝒙 𝑡linear nonlinear
Matrix Exponential-based Integrators for State-Space Analysis Program
Linear Network RLC branch, PI-section, linear
transformer, etc.
Standard SS equation,
automatically formed
Nonlinear Electric machine,
power converters, dynamic loads,
etc.
Dedicated model library
linear networks
px1 = Ax1+Bu1
y1 = Cx1+Du1
nonlinear injectors
px2 = f (t,x2,u2)
y2 = g(t,x2,u2)
y2
u1
u2
y1
Numerical Discretization at the System Level
Krylov Subspace
For 𝑨 ∈ ℝ𝑛×𝑛 and 𝒗 ∈ ℝ𝑛×𝑛, a q–dimensional Krylov subspace
is defined as
𝑲𝑚 = span 𝒗, 𝑨𝒗,… , 𝑨𝑚−1𝒗 ,
Action of matrix functions: Krylov based approximation
For 𝑲𝑚 𝑨, 𝒗 , Arnoldi iteration gives 𝐕m and 𝐇m = 𝐕m𝐓𝐀𝐕m, which
enables
𝝋𝑝 𝑨 𝒗 ≈ 𝑽𝑚𝝋𝑝 𝑯𝑚 𝑽𝑚T 𝒗
Reduced Order Simulation with Krylov Approximation
19
In the originaldimension 𝑁
In the reduced dimension (𝑚 ≪ 𝑁)
算法 1 Krylov 子空间降阶的指数欧拉仿真算法
𝑡 = 𝑡0;𝒙 = 𝒙0
𝐫𝐞𝐩𝐞𝐚𝐭
compute 𝒇(𝑡,𝒙)
initial guess 𝑚 = 𝑚0
𝐫𝐞𝐩𝐞𝐚𝐭
compute 𝑽𝑚 ,𝑯𝑚 with Arnoldi iteration
construct 𝑯 with 𝑝 = 1
compute 𝐞ℎ𝑯 with scaling & squaring
𝑭 = approximation to ℎ𝝋1(ℎ𝑨)𝒇(𝑡,𝒙)
𝜀 = error estimation 𝑚 = 𝑚new
𝐮𝐧𝐭𝐢𝐥 𝜀 < Tol
𝒙 = 𝒙 + 𝑭 (指数欧拉法)
𝑡 = 𝑡 + ℎ𝐮𝐧𝐭𝐢𝐥 𝑡 = 𝑡end
Algorithm: Exp-Euler with standard Krylov subspace approximation
Matrix Exponential-based Integrators with Krylov Subspace Acceleration
20
Improvement 1: Subspace Reuse
• Continuous linearization => Retaining Jacobian𝒙𝑛+𝑘 = 𝒙𝑛+𝑘−1 + ℎ𝝋1 ℎ𝑱𝑛 𝑭 𝑡𝑛+𝑘−1, 𝒙𝑛+𝑘−1
• Construct formula that exploits Jacobian reuse𝒙𝒏+𝒌 = ෝ𝒙𝒏+𝒌 + 𝒙𝒏+𝒌,ෝ𝒙𝒏+𝒌 = 𝒙𝒏 + 𝑘ℎ𝝋1 𝑘ℎ𝑱𝒏 𝑭 𝒕𝒏, 𝒙𝒏 ,𝒙𝒏+𝒌 = 𝒙𝒏+𝒌−𝟏 + ℎ𝝋1 ℎ𝑱𝒏 𝑱𝒏𝒙𝒏+𝒌−𝟏 + 𝚫𝒈𝒏𝒌
• Adjust Jacobian update rate with error control
Improvement 2: Extended Krylov Subspace
• Standard Krylov subspaces𝐊𝑚 𝑨, 𝒗 = 𝐬𝐩𝐚𝐧 𝒗, 𝑨𝒗,… , 𝑨𝑚−1𝒗
match eigenvalues from largest magnitude
• Extended Krylov subspace𝐊𝛾𝑞,𝑛
𝑨, 𝒗 = 𝐊𝑞 𝑨, 𝒗
⊗ 𝐊𝑛 𝑰 − 𝛾𝑨 −1, 𝒗
More targeted matching of the eigenvalues
polynomial
rational
EI with Kry. Approx. 𝒙n+1 = 𝐞ℎ𝑨𝒙n ≈ 𝑽m𝐞ℎ𝑯m𝑽m
𝐓 𝒙n
ෝ𝒙n = 𝑽m𝐓 𝒙n Order reduction, 𝑽m
𝐓 ∈ ℝ𝑚×𝑁
ෝ𝒙n+1 = 𝐞ℎ𝑯mෝ𝒙n Numerical integration in ℝ𝑚×𝑚
𝒙n+1 = 𝑽mෝ𝒙n+1 Transforming back to ℝ𝑁×𝑁 space
ෝ𝒙n+1 = 𝐞ℎ𝑯mෝ𝒙n
ෝ𝒙n+2 = 𝐞ℎ𝑯mෝ𝒙n+1
⋮
Multiple steps in low dimension
Moment-matching based MOR has 𝐊m 𝑰 − σ𝑨 −𝟏, 𝒃
Different matching criteria
MOR Viewpoint & Further Improvements
21
Efficiency Comparison
The speed up ratios of exponential integrators over other solver are 4.22, 4.55, 9.67, 17.48 for four cases. Computation Time Growth Rate
• SPS, ode23t, ode15s have roughly 𝑁1.7
• Best exponential solver has sublinear growth 𝑁0.66
SolverComputation Time under 10 𝜇s Step Size (s)
Case1: 17 WTGs Case2: 26 WTGs Case3: 66 WTGs Case4: 100 WTGs
SPS 297.31 575.81 3143.61 6036.91
ode23t 269.29 513.80 2663.86 5377.92
ode15s 279.85 537.57 2804.93 6013.67
ExpFix 63.77 171.43 2189.23 5591.23
ExpMRb + RT-Krylov 95.34 112.81 234.16 307.68
Speedup ratio 4.22 ExpFix 4.55 ExpMRb+RT-Krylov 9.67 ExpMRb+RT-Krylov 17.48 ExpMRb+RT-Krylov
Numerical Test of Large-scale Wind Farm
22
Scaling & Squaring Method for the Matrix Exponential
For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, we have 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎
Scaling: choose 𝜎 = 2𝑠 s.t. Τ𝑨 𝜎 is sufficiently small, use 𝐞 Τ𝑨 𝜎 ≈ 𝐫e Τ𝑨 𝜎
Squaring: 𝐞𝑨 recovered by successive squaring 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎≈ 𝐫e
𝑨
𝜎
2𝑠
.
0 87654321
𝐞𝑑𝑡8 𝑨 ≈ 𝐫e
𝑑𝑡
8𝑨𝐞
2𝑑𝑡8 𝑨 = 𝐞
𝑑𝑡8 𝑨 ⋅ 𝐞
𝑑𝑡8 𝑨 𝐞
4𝑑𝑡8 𝑨 = 𝐞
2𝑑𝑡8 𝑨 ⋅ 𝐞
2𝑑𝑡8 𝑨 𝐞𝑑𝑡𝑨 = 𝐞
4𝑑𝑡8𝑨 ⋅ 𝐞
4𝑑𝑡8𝑨
Example: use 𝜎 = 23 = 8 to compute 𝐞𝑑𝑡𝑨
Dense Outputs in Large Step Size Integrations
23
Scaling & Squaring Method for the Matrix Exponential
For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, we have 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎
Scaling: choose 𝜎 = 2𝑠 s.t. Τ𝑨 𝜎 is sufficiently small, use 𝐞 Τ𝑨 𝜎 ≈ 𝐫e Τ𝑨 𝜎
Squaring: 𝐞𝑨 recovered by successive squaring 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎≈ 𝐫e
𝑨
𝜎
2𝑠
.
Large step-size integration with dense outputs makes multirate interfacing easy!
w/o dense output 4 dense outputs per step 16 dense outputs per step
Dense Outputs in Large Step Size Integrations
24
ሶ𝒙 𝑡 = 𝑨𝒙 𝑡 + 𝒈 𝑡, 𝒙 𝑡 +𝑖=1
𝑁
𝑩𝑖𝒚𝑖 𝑡
𝒚𝑖 𝑡 = 𝒉𝑖 𝒙 𝑡 − 𝜏𝑖 + 𝑪𝑖𝒚𝑖 𝑡 − 𝜏𝑖 , 𝑖 = 1,… ,𝑁.
𝒙 𝑡 = 𝑺 𝑡 , 𝑡 ∈ −max 𝜏𝑖 , 0
𝒚𝑖 𝑡 = 𝑯𝑖 𝑡 , 𝑡 ∈ −𝜏𝑖 , 0
Delayed-Differential-Algebraic Equation (D-DAE)
ቐ
State space formulation
with propagative elements
vk vm
ikm imk
...
Lumped-parameter
system m
Lumped-parameter
system n
With Initial Condition
ቐ
Iterative EI Solution for D-DAE (enables 𝛥𝑡 > 𝜏)
𝒙𝑛+1𝑘
= 𝐞𝛥𝑡𝑨𝒙𝑛 + 𝛥𝑡𝝋1 𝛥𝑡𝑨 𝒈 𝑡𝑛, 𝒙𝑛 +
𝑙=1
𝑝+1
𝛥𝑡𝑙𝝋𝑙 𝛥𝑡𝑨 𝑩𝒂𝑙−1𝑘−1
with polynomial fitting
time[1]
tn-τ
[2]
tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ
时步积分
时滞项计算
基于[tn-τ,tn]的时滞项预测
预测步积分
time
tn-τ
[4]
tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ
基于预测步的时滞项校正(1)
校正步积分(1)
time
tn-τ
[2k+2]
tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ
基于校正步(k-1)的时滞项校正(k)
校正步积分(k)
[2k+1]
[3]
预测步
第1个校正步
第k个校正步
已知时滞项
已知时滞项
𝑰h𝑘
𝑡 =
𝑙=0
𝑝𝒂𝑙
𝑘
𝑙!𝑡 − 𝑡𝑛
𝑙
Exploit model discontinuity property
• Initial wave propagation phenomenon => small step
• Wave propagation dying out => large step
Large Step Size Integration with Propagative Elements
Summary• Expandable architecture with multi-FPGA hardware, each
FPGA is functionally complete
• Suitable step size selection fitting subnetwork timescales, enabled by multirate interfacing techniques
• High-fidelity simulation results for studied system
• Full EMT modeling of external system through Krylovsubspace techniques with multirate interface
25