Experience with Hydro Generator Expert Systems

GE Energy

Experience withHydro GeneratorExpert Systems

As presented at the Iris Rotating Machine Conference

June 2008, Long Beach, CA

Peter Lewis, IrisJohn Grant, GE EnergyJ. Evens, NYPA

GE Energy | GER-4488 (07/08)

Current technological advances in condition monitoring are

employing an increasing number of complex sensors and

advanced monitors to diagnose the operating status and condition

of hydro generators and turbines. Advanced systems routinely

employed may include bearing vibration, air gap, partial discharge,

and flux monitoring. Proper interpretation of this often complex

information can lower operating and maintenance expenses, in

addition to reducing unscheduled outages and catastrophic

failures. However, the volume of available data from these

monitors, and the extensive interpretation necessary to evaluate

the complex waveforms and spectrums, can overwhelm plant

personnel and resources. Sophisticated software and algorithms

are often necessary to correlate and interpret this data to establish

the overall generator and drive train condition.

HydroX™ (for Hydro Expert) is a knowledge-based expert system

program for on-line monitoring of hydro-generators. Working with

the New York Power Authority, the system was developed over five

years by Iris Power and GE's Bently Nevada* team. After a further

two years of prototype evaluation at NYPA’s St. Lawrence Power

Project on two 60 MVA generators, the validated system is now

commercially available.

The successful development of HydroX was predicated by several key

factors, including:

1. Available and cost effective on-line monitors for critical

components of the turbine and generator.

2. Expertise in the form of hydro-generator design, operation,

and maintenance knowledge that could be codified into expert

system rules.

3. A suitable commercial software platform or expert system shell.

4. A site where the system could be deployed and evaluated.

Each of these factors is discussed below in greater detail.

On-line MonitorsAs part of an upgrade and life extension project of their hydraulic

fleet which began in the late 1990s, NYPA identified several key

technologies necessary to more completely monitor a large hydraulic

turbine and generator. In some cases, although on-line monitors were

available, their cost or complexity made them prohibitive for inclusion

into an expert-based monitoring system like HydroX. As well as the

normal process data, specialized monitors that were considered

critical to the expert system diagnostics include on-line air-gap,

bearing vibration, stator partial discharge and core temperature and

vibration. Over time, competition in the market place led to several

Experience with Hydro Generator Expert Systems

monitors suitable for this hydro-generator monitoring [1]. In the case

of partial discharge monitoring, where no solution existed, a

cooperative R&D effort between NYPA and Iris Power led to the

development of a cost effective on-line PD monitor called

HydroTrac™ [2].

Knowledge BaseOne of the first diagnostic expert systems for on-line turbo-generator

monitoring was developed in the 1980s by EPRI and was called

GEMS[3]. Although a later attempt to create a commercial system

based on this research prototype failed, many of the machine

behavior models developed for GEMS were later very relevant to

HydroX. In addition, this system clearly demonstrated the need for

some form of probabilistic reasoning as complex machine monitoring

is never fully deterministic. The technical success of GEMS spawned

follow on work by EPRI and others in the area of hydro generator

monitoring[4].

Recognizing the loss of machines expertise in the hydro industry,

in the late 1990s NYPA initiated a research project to interview

generator design, operations, and maintenance personnel to

document a diagnostic rule set for an expert system like HydroX.

Although at the time, suitable monitors and sensors were still under

development, and no suitable software platform existed, it was felt

that documenting the rules was a critical first step. This was a multi-

year effort using experts from OEMs, industry, academia, and utility

engineering and operations staff. The result of this project was the

knowledge base that was used later to create HydroX.

Expert System ToolsSince the 1980s, expert systems have been a topic of research

aimed at automating monitoring and diagnostics for complex

industrial equipment. Early attempts involved the use of specialized

computer hardware and software which were not robust or ready

for industrial applications. With the growth in popularity and

capabilities of desktop PCs, it became possible to develop

distributed client-server applications. During the 1990s a prototype

system called ACMS (Advance Condition Monitoring System) was

fielded on such a platform but proved too unreliable, slow, and

difficult to configure to be commercially viable. Other vendors

developed expert system shell programs[5] however, these systems

suffered from a lack of standard interfaces to sensing and

monitoring systems. During this period vendors tended to create

islands of technology which were incapable of communicating

with each other.

GE Energy | GER-4488 (07/08) 1

Only in the past few years have practical PC-based tools been

available for development and commercial deployment of expert

system based plant monitoring systems. System 1* is such a tool,

and contains standard interfaces such as OPC clients/servers

which allow it to communicate with external third party monitors

and sensors. In addition, it contains a rule based inference engine

and provides tools for users to develop decision based logic.

System 1 also provides a number of analysis and visualization tools

that enhance the rule engine by allowing end-users to view data

(historical and current) and rule results in a variety of ways.

Evaluation SiteAn ideal time to install the sensors/monitors necessary to support

a system like HydroX is during a plant refurbishment/upgrade. At

the St. Lawrence Power Project, NYPA was undertaking a plant life

extension project sequentially on 16 units and this project provided

the perfect platform for evaluating the HydroX rule-set. During

each unit’s upgrade, additional sensors were installed to support

the expert system and interfaces to the plant control and

monitoring systems were created. Using the acquisition portion of

HydroX, data was collected from these systems over time on

several units, making it possible to identify machine specific

behavior and characteristics. The generalized rule-set created

during the knowledge base development was then customized

through a “tuning” algorithm. These tuning rules were created to

account for specific generator behaviors due to subtle differences

in manufacture or external factors such as seasonal changes in

ambient conditions.

HydroX FeaturesHydroX is a condition-baseddiagnostic system for hydro-turbine/

generators. The system is basedonacommercial PC-basedasset

management tool called System1. System1 is adistributed software

product basedonaSQLServer databaseandcontains components for

data collection from remote systemsviaOPC, aproduction rule engine for

processinguser defined rules, andadesign tool for developingand testing

rules anddeveloping customuser interfaces. The rules are thebasis for the

HydroXSystemand represent the knowledgebaseof the expert system.

Individual ruleswere created toprocess input data intomoreuseful relevant

data. Processeddata is than fed throughvariousanalysis algorithms

embedded in rules again, or to providedecision support. Oftenmultiple rules

are createdandgrouped into “Rulepaks” that aremeant toprovide specific

analysis functionality. In thismanner, anexpert systemcanbecreated that

canencodeexpert knowledge intoanautomatedanalysis system.

Utilizing the knowledge base developed earlier with NYPA, a

modular set of HydroX Rulepaks were created in System 1.

Encoding each major sensor group in its own Rulepak facilitated

the customization of HydroX for the available machine sensor data

at different sites. If a particular monitor such as PD is not available,

then the rules dealing with those inputs can be easily removed,

leaving the rest of the system functional. Some Rulepaks

incorporate corroboration algorithms that can communicate with

other Rulepaks in order to raise confidence in a diagnosis. In this

manner HydroX offers a comprehensive system that can draw

upon multiple data paths to reinforce its diagnostic accuracy. The

addition of more monitoring systems often will lead to a better

diagnosis.

One particular challenge in any expert system is dealing with

uncertainty in the data analysis. System 1 has built-in mechanisms

for indicating the severity of a problem. In HydroX this was

extended to utilize a Mycin like uncertainty scheme[6] to combine

facts from various sensor inputs into a diagnosis with a certainty

factor. As sensor readings vary further from expected values, or

multiple indications of a problem become apparent, the certainty

in the diagnosis of a fault condition increases.

Where possible, the prediction of “expected” value for sensors is

made based on mathematical models of machine parameters that

are then tuned for the specific unit. These predicted values are

then compared to the actual measured values and deviations are

analyzed by the rules to compute a diagnosis. For example, the

2

Figure 1. System 1 components

Software Components

GCS Computer SCADA Computer HydroTrac Controller

Bently 3500

System 1 Platform and Database

System 1 Data Acquisition

HydroX DisplayHydroX RulePak System 1 Config

GE Energy | GER-4488 (07/08)

predictions of thrust bearing pad temperatures are made based on

the thrust bearing oil temperature and the MW load of the

machine. This basic equation is then customized to account for

heating/cooling time constants of the machine with load, and to

the actual readings obtained at full load for each sensor which

vary due to sensor location and other physical properties.

For many sensors, the alarm thresholds may be significantly

different depending on the mode of the machine. HydroX has rules

to determine the machine mode and where necessary, different

thresholds and even rules are executed dependent on this mode.

The specific modes HydroX recognizes are: standstill, mechanical

runup/rundown, rated-speed de-energized field, field energized but

unsynchronized, synchronized unloaded, load transient and loaded

thermally stable. An example of this behavior would be air gap

measurements, where significantly different nominal air gaps can

be expected depending on the machine state. HydroX uses this

information to set mode-specific thresholds for alarms making the

system very sensitive to small variations in readings.

The machine mode is also used in several instances to calculate

and alarm on the trend of sensor values. The trend of nominal air

gap, during field flashing for example, can indicate a specific type

of problem that trending at nominal machine load would not

detect.

Current industry trends are to move to more automated plants,

with less on-site expertise and operations staff. As described

above, HydroX can calculate and trend key features and synthesize

summary indications from complex data sets from monitors such

as vibration, air-gap, PD, etc. Using these intermediate indicators,

along with diagnostic rules, an expert system like HydroX can filter

and focus attention to abnormal values, and provide diagnosis of

specific faults as well as possible remedies. In addition, trending of

such parameters over years can indicate long-term degradation

that may otherwise go undetected until damage limits are

breached.

NYPA InstallationAs part of a plant modernization project, Unit 18 at the St.

Lawrence Power Project was removed from service to be

refurbished/up-rated. During this work, additional sensors and

monitors were installed to instrument the unit for HydroX. In

addition to the conventional unit monitoring connected to the

GE Energy | GER-4488 (07/08) 3

Figure 2. Graph showing comparison of actual and predicted bearing vibrationbased on unit load and bearing oil temperature

Figure 3. Depiction of expected air gap trend for different machine states

Figure 4. Trend plot of measured air-gap changes during a startup at NYPA

Figure 6. HydroX data interfaces

is that since the units are coming off a major overhaul, the number

of faults has been minimal. In addition, many of the long-term

trending rules for conditions such as partial discharge can take

years to calculate and are just now providing useful values.

plant control system, additional sensors and monitors were added

for partial discharge, bearing vibration, core vibration, back of core

temperatures, and air-gap.

As each of the 16 units in the plant are refurbished (a 10-year

program), the identical sensor set is installed and connected to

HydroX. Once completed, all 16 units will be monitored.

A group of two dedicated PC computers run the HydroX

components; the data acquisition system, the SQL Server

Database, the Diagnostic Rule Engine and the User Interface.

These computers were installed on a separate LAN, and interfaced

to the other necessary plant systems (Generator Control System to

obtain conventional unit sensor data, HydroTrac for PD data, and a

Bently Nevada 3500 rack for air gap and vibration data). The

interfaces to external systems were accomplished using an OPC

Data Interface[7].

Experience to date:Over the past several years, the prototype HydroX has been moni-

toring Unit 18 (and now several other units as they are refurbished

and instrumented). One difficulty with this approach to deployment

4 GE Energy | GER-4488 (07/08)

Figure 5. HydroX sensor set

5

One significant problem that only became apparent as additional

units were connected to HydroX related to the tuning of the rules.

The models and algorithms used to provide predicted sensor val-

ues require substantial tuning for various constants, which can

only be done once the unit is in service. For the deployment of a

successful commercial system, it is not practical for a Field Service

Engineer to be on-site waiting on a unit start-up, and for possibly

weeks after that to collect data for the various machine states

needed to tune the rules. For this reason, a set of “auto-tuning”

rules were written. These rules track data during initial unit opera-

tion, and automatically calculate and enter the specific constants

needed for the various predicted sensor values. The rules use linear

regression to determine the dependency of two independent vari-

ables on a given sensor input. This dependency is usually calculat-

ed during startup as the machine will see the greatest span of

measurements for a given input.

The creation and testing of these rules was a significant and unan-

ticipated effort, but was clearly necessary if HydroX was to be a

commercial success.

A similar problem was found with the setting of alarm limits for

measured values. There are a multitude of custom values that

must be set for HydroX to calculate malfunction certainties proper-

ly. These values are usually known by plant personnel and used for

basic alarming of critical parameters. There are still many values

that may not be known by plant personnel and also, the sheer

multitude of values would make the collection of these values and

customization of the system extremely time consuming. In many

cases these values can be based on given machine standards.

HydroX was built to address this issue by incorporating an auto-

matic tuning system for alarm limits. For example, stator winding

temperature limits are set according to winding insulation classes

(i.e., NEMA), such standards are used in HydroX to automatically

Figure 7. Partial sample logic of an auto-tuning rule

GE Energy | GER-4488 (07/08)

choose the proper limits based on machine construction parame-

ters. HydroX also allows the end-user to set these values manually

and override the automatic values if required.

A final lesson that can be taken from this experience concerns the

reliability of the system. In general, a hydro turbine and generator

is a true model of reliability with some units in continuing service

after 50 years. Unfortunately the same cannot necessarily be said

for the components used to monitor them. It is far more likely that

a sensor, data acquisition system, computer or network will experi-

ence a problem than a hydro generator will. Problems with some

sensors failing and computer components have occurred since the

original installation of the system in 2005. Software and operating

system problems can also occur in any system relying heavily on

computer systems and network interfaces. In particular, plant net-

work security has been a source of problems, as network security

becomes ever more stringent forcing frequent upgrades of soft-

ware, hardware and protocols—all of which may require reconfigu-

ration of the various components in HydroX.

Future PlansBased on the successful deployment on two units at St. Lawrence,

a commercial System 1 Rulepak for HydroX has been created. Over

time this system will be installed on all 16 units at St. Lawrence. It

is expected, that during future deployments at other sites, new

interfaces will be developed to sensors and monitors from other

vendors. Standardized protocols like OPC make this a relatively

simple effort. Obvious future extensions to the system would be to

include support for pump storage units which are often critical and

highly stressed assets.

ConclusionsThe HydroX system is an advanced expert system that will help

utilities protect hydro turbine-generators while reducing the cost of

operation by transitioning from preventive to condition-based

maintenance. The system combines advanced fault detection

knowledge from multiple industry experts with modern data

acquisition systems in order to empower maintenance technicians

and experts alike by providing them with real-time, easy to

understand information. By providing automated data collection

and analysis, the system minimizes the vast volume of data that

would otherwise have to be collected and analyzed manually. This

also leads to a greater wealth of data but without jeopardizing the

speed and accuracy of analysis as can be the case when too much

data is present. HydroX also reduces the number of annoying

“nuisance alarms” by providing a corresponding certainty with

each diagnosis. It is expected that an expert system like HydroX

can extend machine life, reduce forced outages, and reduce

operation and maintenance expenses.

REFERENCES1. J.F. Lyles et al, “Using Diagnostic Technology for Identifying

Generator Maintenance Needs”, Hydro Review, June 1993, p. 58.

2. B.A. Lloyd, S.R. Campbell, G.C. Stone, “Continuous On-line PD

Monitoring of Generator Stator Windings”, IEEE Trans EC, Dec.

1999, p. 1131.

3. G.S. Klempner, A. Kornfeld, and B. Lloyd, “The generator expert

monitoring system (GEMS) experience with the GEMS prototype,”

EPRI Utility Motor and Generator Predictive Maintenance

Workshop, December 1991.

4. A. Roehl and B. Lloyd, “A developing standard for integrating

hydroelectric monitoring systems” EPRI Motor and Generator

Conference, Orlando, Nov. 1995.

5. Nilsen, S., OECD Halden Reactor Project, Inst. for Energiteknikk;

“Experiences made using the expert system shell G2, Tools for

Artificial Intelligence”, 1990, Proceedings of the 2nd International

IEEE Conference, 6-9 Nov 1990, page(s): 520-529

6. Rule Based Expert Systems: The MYCIN Experiments of the

Stanford Heuristic Programming Project, BG Buchanan and EH

Shortliffe, eds. Reading, MA: Addison-Wesley, 1984

7. OPC Foundation – www.opcfoundation.org

HydroX is a trademark of the New York Power Authority.HydroTrac is a trademark of Iris Power Engineering, Inc.* Bently Nevada and System 1 are trademarks of General Electric Company.

6 GE Energy | GER-4488 (07/08)

GE Energy | GER-4488 (07/08)

Notes

GE Energy | GER-4488 (07/08)

GE Energy | GER-4488 (07/08)

©2008, General Electric Company. All rights reserved.

GER4488 (07/08)