August 14, 2003 Blackout Summary Based on Interim Report of the United States – Canada Power Outage Task Force November 19, 2003
Mar 26, 2015
August 14, 2003 Blackout
Summary Based on Interim Report of theUnited States – Canada Power Outage Task Force
November 19, 2003
2
U.S.-Canada Interim Report
● Released November 19, 2003● Result of an exhaustive bi-national
investigation Working groups on electric system, nuclear
plant performance and security Hundreds of professionals on investigation
teams performed extensive analysis
● Interim report produced by the teams and accepted by the bi-national Task Force
3
Overview
● Overview of power system and reliability
● Pre-outage conditions on August 14● Trigger events and start of cascade● Wide area cascade● Root causes● Next steps
4
Power System Overview
5
Reliability Overview
● Balance generation and demand● Balance reactive power supply and
demand● Monitor flows and observe thermal limits● Observe power and voltage stability limits● Operate for unplanned contingencies● Plan, design and maintain a reliable
system● Prepare for emergencies
Reliably operate the system you have!
6
3 Interconnections / 10 NERC Regions
7
NERC Control Areas
8
NERC Reliability Coordinators
9
Footprints of Reliability Coordinators in Midwest
10
NERC Immediate Response to Blackout
● First hours Worked closely with NERC Reliability
Coordinators Identified what had tripped and extent
of outage Assessed restoration efforts Maintained open line with DOE/FERC Communicated with DHS, White
House, and NRC
● First days Assigned project manager Established Steering Group with
industry executive experts Began organizing investigation teams 90+ volunteers + entire NERC staff
11
Investigation Organization Overview
Steering Group
MAAC/ECAR/NPCC Coordinating Group
MAAC
ECAR
NPCC
MEN Study Group
Project Planning and Support
Sequence of Events
Data Requests and Management
Investigation Team Lead
System Modeling and Simulation Analysis
NERC & Regional Standards/Procedures
& Compliance
Transmission System Performance,
Protection, ControlMaintenance & Damage
Operations - Tools, SCADA/EMS
Communications Op Planning
System Planning, Design, & Studies
Root Cause AnalysisCooper Systems
Generator Performance, Protection, Controls
Maintenance & Damage
U.S – CanadaTask Force
Vegetation/ROW Management
Frequency/ACE
Restoration
Investigation Process Review
12
Data Gathering and Analysis
● Three fact-finding meetings August 22 September 8-9 October 1-3
● Onsite interviews and inspections● Secure database of outage information● Extensive corroboration of data to
determine facts● Analysis by teams of technical experts
13
Root Cause Analysis
● Logical structure for investigating complex problems
● Identifies changes, conditions, actions, or inactions at each causal step
● Starts with final event and drills back through each branch of causal tree.
● Asks “why?” at each step.● Accurate, reliable, defensible understanding
of the root causes.
Successfully used to investigate root causes of PJM voltage stability condition in July 1999 and established history in nuclear and defense industries.
14
Root Cause Analysis Phases
Pre-Existing ConditionsE.g. voltages, wide- area transfers,
line and generator outages, etc.
Sammis – StarStar – South Canton
Hanna – JuniperChamberlin - Harding
BLACKOUT16:15
16:06
15:05
Initial Focus
15
August 14 Conditions Prior to Blackout● Planned outages
Cook 2, Davis Besse nuclear plants East Lake 4, and Monroe 1
● Transfers high to northeast U.S. + Ontario Not unusually so and not above transfer limits
● Critical voltage day Voltages within limits Operators taking action to boost voltages
● Frequency Typical for a summer day
● System was within limits prior to 15:05, on both actual and contingency basis
16
Warm But Not Unusual for August
17
August 14 Imports to Northeast-Central Compared to 6/1 to 8/13/2003
18
Voltages Prior to 15:05 EDT August 14
19
Frequency Typical for Summer Day
20
Blackout was NOT Caused by
● Heavy wide-area transfers● Low voltages, voltage collapse● Lack of IPP voltage/reactive support● Frequency anomalies● Cinergy outages starting at 12:08● East Lake 5 trip at 13:31
Contributing factor to later events, but not by itself causal to the blackout
● DPL Stuart-Atlanta trip at 14:02 Contributing factor to loss of MISO real-time
monitoring, but not electrically significant
21
Outage Sequence of EventsTransmission Map Key
O N T A R I O
Transmission Lines
765 kV
500 kV
345 kV
230 kV
Transmission Lines
765 kV
500 kV
345 kV
230 kV
22
East Lake 5 Trip: 1:31:34 PM
ONTARIO
2
1
ONTARIO
23
East Lake 5 Exciter Failure Causes Trip
24
Stuart Atlanta Trip: 2:02 PM
ONTARIO
25
MISO State Estimator and Reliability Analysis
● MISO state estimator and contingency analysis ineffective from 12:37 to 16:04 State estimator not solving due to missing
information on lines out in Cinergy then DPL
Human error in not resetting SE automatic trigger
● Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates
26
FirstEnergy Computer Failures
● 14:14 Alarm logger fails and operators are not aware No further alarms to FE operators
● 14:20 Several remote consoles fail● 14:41 EMS server hosting alarm processor and other
functions fails to backup● 14:54 Backup server fails
EMS continues to function but with very degraded performance (59 second refresh)
FE system data passed normally to others: MISO and AEP AGC function degraded and strip charts flat-lined
● 15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition
● No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips
27
Phone Calls to FirstEnergy
● FE received calls from MISO, AEP, and PJM indicating problems on the FE system but did not recognize evolving emergency 14:32 AEP calls regarding trip and reclose of Star-
S. Canton 15:19 AEP calls again confirming Star-S. Canton
trip and reclose 15:35 Calls received about “spikes” seen on
system 15:36 MISO calls FE regarding contingency
overload on Star-Juniper for loss of Hanna-Juniper 15:45 FE tree trimming crew calls in regarding
Hanna-Juniper flashover to a tree PJM called MISO at 15:48 and FE at 15:56
regarding overloads on FE system
28
Chamberlin-Harding (3:05:41)
29
Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper
30
(3:05:41) Hanna-Juniper(3:32:03)
31
Hanna Juniper Confirmed as Tree Contact atLess than Emergency Ratings of Line
32
Effects of Ambient Conditions on Ratings
33
(3:05:41)(3:32:03)
Star- S. Canton (3:41:35)
34
Situation after Initial Trips 3:05:41 – 3:41:35
ONTARIO
35
Canton Central – Tidd(3:45:41)
36
138 kV Lines Overload and Cascade Near Akron
Simulated 138 kV Line Loadings
0
20
40
60
80
100
120
140
160
180
200
Outages
% o
f N
orm
al R
atin
gs
(Am
ps)
Dale-W.Can138 kV
W.Ak-PVQ22 138 kV
Cham-W.Ak138 kV
E.Lima-N.Fin 138 kV
CantC Xfmr
W.Ak-PVQ21 138 kV
Babb-W.Ak138 kV
E.Lima-N.Lib 138 kV
Clov-Torrey138 kV
Da
le-W
.Ca
n1
38
kV
W.A
k 1
38
kV
Bk
r Fa
ilure
E.L
ima
-N.F
in1
38
kV
Ca
ntC
Xfm
r
W.A
k-P
V Q
21
13
8 k
V
Ba
bb
-W.A
k1
38
kV
E.L
ima
-N.L
ib1
38
kV
Clo
v-T
orre
y1
38
kV
Sta
r-S.C
an
t3
45
kV
Ha
nn
a-J
un
34
5 k
V
Ha
rd-C
ha
mb
34
5 k
V
Ch
am
-W.A
k1
38
kV
37
15
:05
:41
ED
T
15
:32
:03
ED
T
15
:41
:35
ED
T
15
:51
:41
ED
T
16
:05
:55
ED
T
0
20
40
60
80
100
120
140
% o
f N
orm
al R
atin
gs
Can
ton
Cen
tral Tran
sform
er
Bab
b-W
.Akro
n 138 kV
Hard
ing
-C
ham
berlin
Han
na-
Jun
iper
Star-S
.Can
ton
Clo
verdale-T
orrey 138 kV
E.L
ima-N
ew L
iberty 138 kV
W.A
kron
-Pleasan
t Valley 138 kV
E.L
ima-N
.Fin
lay 138 kV
Ch
amb
erlin-W
.Akro
n 138 kV
W.A
kron
138 kV B
reaker
Dale-W
.Can
ton
138 kV
Sammis-Star
138 kV Cascade Contributes Furtherto Overload of Sammis-Star
38
Sammis-Star(4:05:57.5)
39
Sammis-Star Zone 3 Relay Operateson Steady State Overload
40
Actual Loading on Critical Lines
0
400
800
1200
1600
12:00 13:00 14:00 15:00 16:00
Time - EDT
Flo
ws
(M
W)
Harding - Chamberlin
Hanna - JuniperStar - South Canton
Sammis - Star
East Lake 5 Trip
Harding - Chamberlin Line Trip
Hanna - Juniper Line Trip
Sammis - Star Line Trip
Star - South Canton Line Trip
41
Actual Voltages Leading to Sammis-Star
250
270
290
310
330
350
370
15:00 16:00Time - EDT
Vo
ltag
e (k
V)
Star
Hanna
Beaver
Perry
100% Voltage
95% Voltage
90% Voltage
Sammis - Star
345 kV Line Trip
Star - South Canton345 kV Line Trip
Hanna - Juniper345 kV Line Trip
Harding - Chamberlin345 kV Line Trip
Gaps in Data Records
42
Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM
RemainingPaths
43
345 kV Lines Trip Across Ohio to West
ONTARIO
44
Generation Trips 4:09:08 – 4:10:27 PM
ONTARIO
45
345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM
46
Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM
47
Power Transfers Shift at 4:10:38.6 PM
48
Eastern Eastern Michigan (Detroit) UnstableVoltage and Frequency Collapse and Pole Slipping
Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38
49
Generator Trips to 16:10:38
50
Generator Trips – Next 7 Seconds
51
Overloads on PJM – NY Ties 4:10:39 PM
52
PJM – NY Separating 4:10:44 PM
53
Cleveland – Toledo Island 4:10:39 - 4:10:46 PMCleveland Blacks Out
54
Northeast Completes Separation from Eastern Interconnection 4:10:43 – 4:10:45 PM
North of LakeSuperior
55
Conditions at Niagara Indicate Progressively Worsening Stability Conditions with Prior Events
56
Island Breaks Up: 4:10:46 – 4:13 PM
57
Frequency in Ontario and New York during BreakupNiagara Generation Stays with Western NY
58
Generator Trips – After 16:10:44
59
Areas Affected by the BlackoutService maintained
in some area
Some Local Load Interrupted
End of the Cascade
60
Blackout Root Cause Group 1FE Situational Awareness
● FE did not ensure a reliable system after contingencies occurred because it did not have an effective contingency analysis capability
● FE did not have effective procedures to ensure operators were aware of the status of critical monitoring tools
● FE did not have effective procedures to test monitoring tools after repairs
● FE did not have additional high level monitoring tools after alarm system failed
61
Blackout Out Root Cause Group 2Vegetation Management
● FE did not adequately manage tree growth in its transmission rights of way
62
Blackout Cause Group 3Reliability Coordinator Diagnostics
● MISO’s state estimator failed due to a data error.
● MISO’s flowgate monitoring tool didn’t have real-time line information to detect growing overloads
● MISO operators couldn’t easily link breaker status to line status to understand changing conditions.
● PJM and MISO ineffective procedures and wide grid visibility to coordinate problems affecting their common boundaries
63
Voltage support/reactive supply
Reliability communications
Computer failure response & notifications
Emergency action plans and capabilities
Operator training for emergencies
Vegetation management
Near-Term Industry ActionsResponses from Control Areas and
Reliability Coordinators Due December 15
64
Next Steps
● U.S./Canada Power Outage TF hearings Public hearings to allow comment on report and
input on recommendations December 4 December 5 December 8 – Toronto
Industry technical conference December 10 – Philadelphia
● NERC next steps NERC executive committees December 11 NERC committees meet January 13-14 Continue investigation
Near term analysis and recommendations in support of U.S. Canada Task Force
Long term analysis and recommendations for NERC