PSERC Blackout of 2003: Description and Responses · 11/7/2003 · PSERC Introduction • The blackout investigation is on-going with no conclusions about root causes announced as

Blackout of 2003: Description and Responses

Dennis RayExecutive Director

Power Systems Engineering Research Center

Updated November 7, 2003

PSERCPSERC’s Mission

Universities partnering with industry in a collaboratory to:• Engage in forward-thinking about

industry challenges • Conduct collaborative research for

innovative solutions to those challenges

• Facilitate interchange of ideas • Educate the next generation of power

industry engineers

PSERCCollaborating Universities

• Cornell University - Robert J. Thomas, Center Director

• Arizona State University - Gerald Heydt

• University of California at Berkeley - Shmuel Oren

• Carnegie Mellon University - Sarosh Talukdar

• Colorado School of Mines - P.K. Sen

• Georgia Institute of Technology - Sakis Meliopoulos

• Howard University - James Momoh

• University of Illinois at Urbana - Peter Sauer

• Iowa State University - Vijay Vittal

• Texas A&M University - Mladen Kezunovic

• Washington State University - Anjan Bose

• University of Wisconsin-Madison - Robert H. Lasseter

• Wichita State University - Ward Jewell

Industry MembersMidAmerican Energy

National Grid USANat’l Renewable Energy Lab

New York ISONxtPhase

OncorPacific Gas and Electric

PJM InterconnectionPowertech

PowerWorld Corp.RTE – French TSOSalt River Project

Siemens, EMISSouthern CompanySteel Tube Institute

TVATri-State G&T

U.S. DOEWestern Area Power Admin.

ABBAlliant EnergyALSTOM EAI

American Electric PowerAmerican Transmission Co.

Arizona Public ServiceCalifornia ISO

CenterPoint EnergyCooperative Research Network

Duke EnergyEntergy

EPRIERCOTExelon

GE Power SystemsInstitut de recherche d’Hydro-Québec

(IREQ)ISO New England

Korea Elec. Power Research Institute

PSERCPSERC Resources

• Web site: http://pserc.org• Blackout of 2003 web page

• Blackout description (with event timelines)• Data measurements• Investigations• PSERC background information• Grid reliability studies• Links to related sites

PSERCIntroduction

• The blackout investigation is on-going with no conclusions about root causes announced as of the date of this presentation. This presentation illustrates the complexity of the issues facing investigators and does not attempt to comprehensively list those issues. Many of the details related to the blackout have not yet been released.

• This presentation does not necessarily reflect the views of PSERC’s member universities or industrial members.

• An accompanying simulation using 1998 FERC 715 data illustrates the consequences of selected events along the blackout timeline.

PSERCSources• The presentation uses publicly available materials

from the U.S./Canadian Joint Outage Task Force, International Transmission Co., Electrotek, EnerNex, Midwest Independent Transmission System Operator (MISO), National Electric Reliability Council, New York Times (including transcripts from the MISO control center), PSERC researchers, SoftSwitching Technologies, TVA, testimony before the House Energy and Commerce Committee and the Michigan PSC Report on August 14th Blackout.

• All of the source materials are linked through the Blackout of 2003 web page on the PSERC web site. Also, there are links embedded in text, graphics and slides in this presentation. With an internet connection, simply clicking on the linked objects will take you to the source documents.

PSERCBlackout Statistics• Area of 50 million people in eight states and

two provinces • Approximately 61,800 Megawatts (MW) of load

• PJM Interconnection – 4,000 MW• Midwest ISO – 18,500 MW• Hydro Quebec – 100 MW• Ontario IMO – 21,000 MW• ISO New England – 2,500 MW• New York ISO – 24,400 MW

• 34,000 miles of transmission (out of 150,000 miles in U.S.)

• More than 290 generating units• Thousands of substations, switching

facilities, circuit-protection devices, etc.

PSERCNorth American Interconnections

PSERCDisturbance Characteristics• Frequency excursions throughout the

Eastern Interconnection• Generation > Load means frequency rises• Generation < Load means frequency falls

• Voltage excursions principally in the blackout region• Voltage declined, leading to voltage collapse in

some cases.• Overvoltage also occurred in some instances

after load “tripped.”• Insufficient local supply of reactive power (VAR’s)

leads to declining voltage and potentially to voltage collapse (see PSERC background papers for more info).

PSERCFrequency in Chattanooga, TN

System frequency normally close to 60 Hz (or 60 cycles per second)

Sudden loss of load results in excess generation, thus

frequency jumps

Automatic and operator responses lower frequency

Source: TVA

PSERCFrequency Excursions Seen Throughout

the Eastern Interconnection

Source: EnerNex

PSERCFrequency Measured in Ann Arbor, MI (includes on-site backup generation)

Approximate time when significant MI customer outages began

Michigan Electric Coordinated System (MECS)

PSERCData from SoftSwitching Technologies’ I-Grid Monitors.

Red dots indicate measured low voltage events. No monitors shown in Canada.

PSERCWithin 42 seconds voltage problems

appear in New York.

PSERCWithin the next 19 seconds voltage

problems are wide-spread.

Voltage Collapse at a Site on Staten Island

Source: Electrotek

PSERCSome Major Parties• ISOs/RTOs/Market Operators/Reliability

Coordinators• Independent Market Operator in Ontario• Midwest Independent Transmission System

Operator• ISO New England• New York ISO• PJM Interconnection

• Retail Service Providers: AEP, Cinergy, DTE Energy, FirstEnergy, ConEd, among many others

• Transmission Companies: International Transmission Co. (ITC), Michigan Electric Transmission Co. (METC), among others

PSERCRTO/ISO Map

Regional Wholesale Power Transaction Pattern on 8/14

• 2,500 MW to the Independent Market Operator of Ontario (IMO) (from Mid-America Interconnected Agreement (MAIN) and Hydro-Quebec (HQ)

• 1,300 MW to the New York Independent System Operator (NYISO) (from MAIN, HQ, and Mid-Atlantic Area Council (MAAC)

• 2,500 MW to MAAC (from ECAR and MAIN). The general transaction pattern was west to east (3,500 MW) and south to north (2,500 MW).

• Afternoon schedule included: 1,500 MW from American Electric Power (AEP) to MISO, about 1,500 MW into MECS, about 1,000 MW into IMO, and about 1,800 MW (from AEP) and 700 MW (from DPL) into FE.

• Of the net schedule into FE, approximately 1,650 MW was for retail suppliers.

Source: House Testimony by Peter Burg

PSERCEvents Leading to the Blackout:

12:05:44 – 1:31:34 PM Generator trips

1. 12:05:44Conesville Unit 5 (rated 375 MW). Returns to service at 3:17. Producing 150 MW at 4:00.

2. 12:45 until 1:14:04Greenwood Unit 1 (rated 600 MW). Boiler fuel trip. Returns to service at 1:57. Producing 400 MW at 4:00.

3. 1:31:34Eastlake Unit 5 (rated 597 MW). Tripped while restoring the voltage regulator from manual to automatic control.

Transmission Lines

765 kV500 kV345 kV230 kV

Transmission Lines

765 kV500 kV345 kV230 kV 1

EventsLine openingPath openingGenerator trip

Event number1

EventsLine openingPath openingGenerator trip

Event number

Source: Joint Task Force and House testimony

PSERC2:02 PM Transmission line disconnects

in southwestern Ohio

4. 2:02Stuart – Atlanta 345 kV (DPL).Line locked out. Cause not available.

PSERC2:36 Phone call from Cinergy

to MISO

“I hate to worry you, but I think we’re a trip away from …. setting a little history.”

Note: Cinergy was not in the final blackout area.

Source: “Overseers Missed Big Picture as Failures Led to Blackout.”The New York Times, September 13, 2003.

PSERC2:36 At MISO

Coordinating the response to Cinergy'stroubles involved at least six officials at Midwest ISO.

Actions hindered by failure of an ISO program, called a state estimator, that helps to monitor grid conditions.

"The state estimator has been down for an hour and a half," said MISO at 2:36.

Source: NYT, 9/13

PSERCITC: Flows are normal.

PSERCFE: Voltages below normal

“Hours before the Aug. 14 blackout, operators at FirstEnergy, the utility that serves much of northeastern Ohio, noticed a peculiar thing: voltage across their system was below normal, a sign of insufficient reactive power. Even though FirstEnergywas importing thousands of megawatts from southern Ohio, "we were exporting VAR's outside the FirstEnergy system the whole day," said Charles E. Jones, a senior vice president. ‘That's not normal.’ ”

Source: “Elusive Force May Lie at Root of Blackout” New York Times, 9/23

5. 3:05:41 Harding-Chamberlain 345 kV (FE)Phase-to-ground fault. Cause unknown; flow 40% of rating at trip.

6. 3:32:03Hanna-Juniper 345 kV (FE)Tree contact; flow 85% of rating at trip.

7. 3:41:33Star-South Canton 345 kV (FE/AEP)Phase-to-ground fault.Cause unknown; flow 85% of rating at trip.

3:05:41 – 3:41:33 Transmission lines disconnect between eastern Ohio and

northern Ohio

PSERCITC: Flows remain steady.

PSERC3:36 MISO Calls FirstEnergy

MISO asks “…what is going on over there?” FirstEnergy says they are unsure."I wonder what is going on here," MISO says at 3:36 p.m. "Something strange is happening."

Source: NYT, 9/13

PSERCITC: Flows remain steady

PSERC3:48 PJM call to MISO

At this point, PJM and MISO think only two lines have failed. Actually, eight have failed. In response to a question about the first line failure, a MISO controller says, “I have not had a chance to investigate it. There is too much going on right now.”

Source: NYT, 9/13

PSERC3:57 FirstEnergy Call to MISO

FirstEnergy told the MISO that the voltage on a major line was dangerously low. "Do you have any idea on what is going on?“

MISO replied that the Hanna-Juniper line was out, adding, "I am wondering if it is still out.“

"We have no clue," FirstEnergy replied. "Our computer is giving us fits too. We don't even know the status of some of the stuff around us.“

MISO comments: “I can’t get the big picture of what’s going on. Strange things are all happening at the same time.”

Source: NYT, 9/13

PSERC3:45:33 – 4:08:58 Remaining transmission lines

disconnect from eastern into northern Ohio

8. 3:45:33Canton Central-Tidd 345 kV (FE).Line reclosed in one minute, with 345/138 kV transformers isolated.

9. 4:06:03Sammis-Star 345 kV Tripped and locked out due to line overload.(22% overload at trip)

Note: Numerous 138 kV line trips are also occurring.

Flows between ITC and FE reverse. Voltages on Mich. grid decline.

4:08:58 – 4:10:27 Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan

10. 4:08:58 Galion-Ohio Central-Muskingum 345 kV(AEP/FE). 3% overload at trip.

11. 4:09:06East Lima-Fostoria Central 345 kV (AEP)45% overload at trip.Reclosed within 2 mins.

12. 4:09:23-4:10:27Kinder Morgan (rated 500 MW; loaded to 200 MW at trip.)

PSERCITC-FE flows jump by 2000 MW

ITC-IMO flows reverse.

PSERCITC: Voltages begin collapsing

in mid-Michigan

PSERC4:10:00 – 4:10:38 Transmission lines disconnect

across Michigan and northern Ohio, generation trips off-line in northern Michigan and northern Ohio, and

northern Ohio separates from Pennsylvania13. 4:10 – Harding-Fox (FE) 345 kV

14. 4:10:04 – 4:10:45 – Twenty generators along Lake Erie in northern Ohio (loaded to 2174 MW total)

15. 4:10:37 – West-East Michigan 345 kV

16. 4:10:38 – Midland Cogeneration Venture (loaded to 1265 MW)

17. 4:10:38 – Transmission system separates northwest of Detroit

18. 4:10:38 – Perry-Ashtabula-Erie West 345 kV

Source: Joint Task Force

PSERCSummary of the Situation at 4:10:38

PSERCComment on Protective Relaying

in Michigan“The earlier NY blackouts of the mid-sixties resulted in the installation of technical safety devices ("underfrequency relaying") throughout the transmission grid which undoubtedly…protected the security of the grid in many cases in the past. Unfortunately, in this instance, such equipment was designed to address an imbalance in load and generation (a frequency event), not overuse of the system resulting in voltage collapse as we saw within Michigan, and had little value in mitigating the August 14 event.”Joseph Welsh, CEO, International Transmission Co. in testimony before the House Energy and Commerce Committee, September 4, 2003. The Michigan PSC August 14 Blackout Report says that the protective equipment in Michigan worked as designed.

PSERCITC in fast voltage collapse.

ITC-IMO flow rises to nearly 2800 MW

4:10:40 – 4:10:44 Four transmission lines disconnect between Pennsylvania

and New York

19. 4:10:40Homer City-Watercure Road 345 kV

20. 4:10:40Homer City-StolleRoad 345 kV

21. 4:10:41South Ripley-Dunkirk 230 kV

22. 4:10:44East Towanda-Hillside 230 kV Source: Joint Task Force

PSERC4:10:41 Transmission line disconnects and

generation trips in northern Ohio

23. Fostoria Central-Galion (AEP/FE) 345 kV

24. Perry 1 nuclear unit (rated 1252 MW)

25. Avon Lake 9 unit (rated 616 MW)

26. Beaver-Davis Besse 345 kV

PSERC4:10:42 – 4:10:45 Transmission paths disconnect in

northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection

27. 4:10:42Campbell unit 3

(rated 820 MW)

28. 4:10:43Keith-Waterman 230 kV

29. 4:10:45 Wawa-Marathon 230 kV

30. 4:10:45 Branchburg-Ramapo 500 kV

PSERCBlackout in ITC System

4:10:46 – 4:10:55 New York splits east-to-west. New England (except southwestern Conn.).

The Maritimes separate from New York and remain intact.

31. 4:10:46– 4:10:55 New York-New England trans-mission lines disconnect

32. 4:10:48 New York transmission splits east-west

4:10:50 – 4:11:57 Ontario separates from New York west of Niagara Falls and west of St.

Lawrence. Southwestern Connecticut separates from New York and blacks out.

33. 4:10:50– The Ontario system just west of Niagara Falls and west of St. Lawrence separates from New York.

34. 4:11:22 – Long Mountain – Plum Tree 345 kV

35. 4:11:57 – Remaining transmission lines between Ontario and eastern Michigan separate

PSERCFrequency Varies By Area

as Grid Separates

PSERCAffected Area (4:13 p.m.)

ONTARIO

Area affected by blackout(Service maintained in isolated “islands”)

PSERCThe Day the Earth Stood Still(1950 Movie)

Professor Barnhard was talking to his secretary Hilda about the worldwide electric blackout that had been caused by alien visitors to Earth. He asked her, “Does this make you feel insecure?” and she said “Yes.”

His response was “I am glad,” because he hoped that the blackout would bring about change.

Source: Michigan PSC Report on the August 14th Blackout. Nov. 5, 2003

PSERCInvestigations

• Canadian/U.S. Joint Outage Task Force• Working Group: Electric System, Nuclear

Power, Security• Two phases: (1) find blackout’s cause and

(2) formulate recommendations• Technical support by Consortium for

Electric Reliability Technology Solutions (with PSERC and national lab researchers)

• National Electric Reliability Council (NERC)

PSERCInvestigations

• Congress• U.S. House Committee on Energy and Commerce.

W.J. "Billy" Tauzin, Louisiana, Chairman• U.S. Senate Committee on Governmental Affairs,

Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia. George V. Voinovich, Ohio, Chairman

• Federal Energy Regulatory Commission Midwest ISO-PJM RTO Issues Inquiry

• State Commissions and Legislatures

PSERCExcerpts from Testimony Before the House Energy and Commerce Committee (Sep. 4)

• “…events on our system, in and of themselves, could not account for the widespread nature of the outage.” “We strongly believe that such a widespread loss of power could only result from a combination ofevents, not from a few isolated events.”H. Peter Burg, Chairman and CEO, FirstEnergy Corp.

• “The opening of the lines isolated our system.…. avoided cascading outages across the AEP System and probably far beyond… I don't know why all systems didn't perform in a similar manner.”Linn Draper, Chairman, President and CEO, American Electric Power

• “Just before 4:11 p.m. EDT, voltage on our system began fluctuating and declining and frequency began to drop. Low system frequency triggered sensors that actuated an automatic, four-step under-frequency load shedding program disconnecting approximately 50% of our load. The voltage continued to fluctuate and did not recover. There was a loss of generation and transmission and the system shut down very quickly.”Eugene R. McGrath, Chairman, President and CEO, Consolidated Edison Company of New York

PSERCComments in Testimony

• “On August 14, 2003, with absolutely no warning, the ITC transmission grid experienced severe electric flows (which were a result of energy demands of electric customers other than those residing in Michigan) which collapsed our grid and the grids of our interconnected neighbors…. This event is akin to a "tsunami" hitting an unsuspecting coastal community.”Joseph Welch, CEO, International Transmission Company

• “….it appears that a significant mismatch between load and generation developed in the Midwest (though we received no notice of the emerging conditions at the time). This caused large and abrupt swings in power flows and frequency, and protective systems tripped several transmission lines in New York State at approximately 4:10 p.m. This was the first indication we received of a disturbance.”Nick Winser, Group Director Transmission, National Grid Transco

PSERCComments in Testimony

• “During the course of the hour preceding the cascading event, after the loss of a large generating unit in northern Ohio had alreadyoccurred, several transmission line outages also occurred in theOhio area. During this period the Midwest ISO operator was in contact with the neighboring Reliability Coordinator at PJM as well as control operators within our territory. At this point in time, the issues did not seem to implicate a regional problem.”James P. Torgerson, President and CEO, Midwest ISO

• “Up until the event, our system was operating normally, well within applicable criteria and with adequate reserves. There had been aroutine scheduling call with adjoining control areas that occurred only 11 minutes before the event (at 4:00:00 p.m.) without indication of a problem.”William J. Museler, President and CEO, New York ISO

PSERCComments in Testimony• “PJM became aware of significant impacts on its system from an

external disturbance at approximately 4:10 pm.” “…the system worked as it was designed---through the automatic operation of relays PJM was able to isolate problems which effectively separated it from the outage and “kept the lights on” for the overwhelming majority of its customers.”Phil Harris, President, PJM Interconnection

• “…we were able to isolate most of the New England grid from the rest of the power system. The New England system was designed and maintained properly and worked as expected. Automatic protective relay devices on the transmission lines opened as intended, interrupted the transmission lines and opened the tiesinterconnecting our system with New York. This mechanical actionseparated most of New England from the disturbance to the west.”Gordon van Welie, CEO, ISO New England

PSERCResponses to the Blackout

• Grid technology• Grid management and planning• Policy issues• R&D and education

Note: The responses listed have already been suggested in various forums. Certainly more detailed responses will be discussed in the weeks and months to come as investigation results become available.

PSERCResponses: Grid Technology

• Improve the fundamental infrastructure (invest in hardware)

• Increase information for monitoring and control

• Improve models and tools for analyzing “security” of operations and finding solutions

• Enhance the grid’s intelligence (i.e., more automation)

• Rethink the grid structure and how the grid is used

PSERCResponses: Grid Management

and Planning• Improve operational controls (such as

mandatory operating standards, more centralized decision-making, reactive power support, etc.)

• Improve coordination and communications for reliable operation

• Reassess maintenance practices• Find regional solutions • Eliminate fragmentation in ISO/RTO areas• Expand market-based solutions

PSERCResponses: Policy Issues

• Grid investment incentives• State construction approval and siting

(processes, incentives, etc.)• Federal backstop transmission siting

authority• Regional Transmission Organizations

(RTO’s) with mandatory participation and less fragmentation

• Transmission pricing

PSERCResponses: Policy Issues

• Mandatory and enforceable operating standards

• FERC’s Standard Market Design Platform

• Back to the basics: reliable supply at a reasonable cost

• Comprehensive energy bill vs. a focused reliability bill

PSERCObservations• Grid is being used in ways not intended when

designed, is more heavily loaded, and its use is more unpredictable than in the past.

• Large-scale outages happen more often than you might expect.

• Fixing the specific cause of this blackout won’t fix all potential causes of future blackouts (too many unlikely failure combinations).

• It’s a virtually impossible objective to prevent all blackouts. And it would be very expensive to try to do so.

• A reasonable question is “Blackouts – how often do we want them?”

PSERC“Grid 2030” – A National Vision for

Electricity’s Second 100 Years

• Elements to be designed• A national electricity “backbone” • Regional interconnections which include Canada and Mexico • Local distribution, mini- and micro-grids providing services to

customers from generation resources anywhere on the continent • National Electric Delivery Technologies Roadmap

(being finalized)• Design the architecture, develop the critical technologies,

accelerate technology acceptance, strengthen market operations, build partnerships

• DOE Office of Electric Transmission and Distribution to lead effort to modernize the grid

Source: Testimony of Jimmy Glotfelty, Director of OETD, before the House Subcommittee on Energy, Sept. 25, 2003

PSERCEPRI’s Electricity Sector Framework for the Future

The vision is one of a highly reliable, affordable, environmentally friendly power system that provides essential public services and supports the economic aspirations of all classes of customers.• Digital control of the power delivery network• Integrated electricity and communications for the user• Transformation of the meter into a two-way energy/information

portal• Integration of distributed energy resources into the network• Robust advanced power generation portfolio, including coal

refining

PSERCR&D and Education Needs

• R&D• Transmission and Distribution Technology• Systems• Markets

• Education• Next generation of power engineers need a

quality education.• Providing that quality education requires a

strong university R&D program.

PSERCNew T&D Research Areas

in PSERC

• Data integration and enhanced functions• Managing an aging infrastructure• Distribution and transmission automation• New devices and related control concepts• New control and protection paradigms

PSERCNew Systems Research Areas

in PSERC

• Improved computational methods• On-line security assessment • Improving transmission capacity• System issues related to distributed

energy resources• New technologies for improving

system performance

PSERCNew Markets Research Areas

in PSERC

• Transmission investment• Seams problems between ISO’s/RTO’s• Market monitoring and mitigation• Market goals and verification• Reliability assessment large

interconnected grids

PSERCR&D Expenditures as a Percent of Revenue

Drugs &

sOffic

h.Elec

tronic

Electric

al Eqt

Durable

Aircraf

Motor v

esReta

itectu

ralCon

r prod

Fabric

ated m

Furnitu

d fixt

Commun

icatio

Printin

itsFerr

lear P

Mining

Food S

Restau

Bldg M

ateria

Electric UtilitiesNuclear Power

Source: This graph was prepared by EPRI using NSF data.

U.S. Power Engineering Undergraduate Enrollments

1960 1980 20000

Source: G.T. Heydt and V. Vittal, “Feeding Our Profession,” IEEE Power & Energy Magazine, Vol.1, Issue 1, Jan/Feb 2003, pp 38-45

U.S. Power Engineering Graduate Enrollments

1970 1975 1980 1985 1990 1995 20000

200M.S.E.E.

PSERCDeclining Number of Power

Engineering Faculty (and Programs)

number ofrecent retirements

facultysize

U.S. Power Faculty: 2003 Survey Results

number ofrecent hires

Source: 40 responses of the survey reported by P. W. Sauer, G. T. Heydt and V. Vittal, “The State of Electric Power Engineering Education,” IEEE Transactions on Power Systems, Special Section on Education, to appear in 2004

PSERCRemaining Years of Service by

Power Engineering Faculty

Research and Education are important elements of our critical

infrastructure, too.

PSERC Blackout of 2003: Description and Responses · 11/7/2003 · PSERC Introduction • The blackout investigation is on-going with no conclusions about root causes announced as

Documents

The Mathematics of the Great U.S. Blackout August 14, 2003.....

August 14, 2003 Blackout

Impacts and Actions Resulting from the August 14, 2003...

PSERC Overview Presentation - Power Systems Engineering

Blackout in Sweden and Eastern Part of Denmark 2003-09-23

Blackout 2003

Analysing the Causes of the Italian and Swiss Blackout,...

Blackout USA- Canada 2003

Impacts and Actions Resulting from the August 14, 2003 ........

Blackout models

PSerc Seminar PSERC · PDF fileSWITCHYARD INTERFACE DATA...

Technical Analysis of the August 14, 2003, Blackout 14...

Final Report on the August 14, 2003 Blackout in the...

Gorur t26g Pserc Final Report (1)

August 14, 2003 Blackout Final Report U.S.-Canada Power...

Rackliffe Pserc Smart Grid Forum Mar09