PI Helps Solve NT Server Performance Issues and Network Incidents

PI Helps Solve NT Server Performance Issues and

Network Incidents

Or

The PI’s Have It.

April 3, 2000 Dow Corning Corp.

Project Started October 1999 - Formal Project Started Goals

Capture NT Server Performance Data In PI. Capture Network Performance Data In PI.

Tools A development PI Server (from Dow Corning) A development PI License (from OSI Software) Multiple copies of PI Perfmon Interface (from OSI

Software) A development NetScout License (from NetScout

Systems Inc.)

April 3, 2000 Dow Corning Corp.

What NT Performance Data Is Being Captured in PI?

NT Performance Counters through the PI Perfmon interface.

We have had it running since May 17, 1999 in an un-official use as an aid to Jon Peterson

April 3, 2000 Dow Corning Corp.

Early FindingsMemory on our PI Servers was eroding.

PI Trend Available. PI Batch was the main problem.

April 3, 2000 Dow Corning Corp.

PI-Batch ProcessMemory Leak

April 3, 2000 Dow Corning Corp.

Early FindingsMemory on our PI Servers was eroding.

PI Trend Available. PI Batch was the main problem.

PI 3.2.357.8 came out with a fix for PI Batch. (Good Job Jon.)

PINetManager had a leak.

April 3, 2000 Dow Corning Corp.

PINetMgr had a leak.

April 3, 2000 Dow Corning Corp.

Early FindingsMemory on our PI Servers was eroding.

PI Trend Available. PI Batch was the main problem.

PI 3.2.357.8 came out with a fix for PI Batch.

PINetManager had a leak.PI 3.2.357.17 fixed the leak.

April 3, 2000 Dow Corning Corp.

Later FindingsPI 3.2.357.17 when run on NT 4.0 SP5

also fixed a leak in PIArchss.

April 3, 2000 Dow Corning Corp.

PIArchss Process Before SP5

April 3, 2000 Dow Corning Corp.

Later FindingsPI 3.2.357.17 when run on NT 4.0 SP5

also fixed a leak in PIArchss.Ensign Agents (another tool used in

Dow Corning) can go nuts and lock up one of our NT MSSQL Servers

April 3, 2000 Dow Corning Corp.

Ensign Agents

April 3, 2000 Dow Corning Corp.

Later FindingsPI 3.2.357.17 when run on NT 4.0 SP5

also fixed a leak in PIArchss.Ensign Agents (another tool used in

Dow Corning) can go nuts and lock up one of our NT MSSQL Servers

Exchange Server Lost it’s STORE process.

April 3, 2000 Dow Corning Corp.

Exchange Server

Notice the LOSS then change of STORE process from Exchange

April 3, 2000 Dow Corning Corp.

NT SystemsCurrently Being Watched! PI Server MSSQL Servers Exele's – EdictVB Server FOXBoro's NT version (Our Control System

Support Group is thrilled.) Intellution SCADA servers (Our Control

System Support Group is thrilled.) Automatic Data Collection ADC Server Exchange Servers

April 3, 2000 Dow Corning Corp.

Locations Being MonitoredMidland – Michigan USABarry Wales - UKChiba - Japan

April 3, 2000 Dow Corning Corp.

What Network Data Is Being Captured In PI? Started in October 1999. NetScout can be configured to use MSSQL. We added two tables to the MSSQL to create

an association between NetScout counters and PI Tags.

Basic Network data is captured from our Global WAN.

We are using 1 minute summarized data from NetScout.

April 3, 2000 Dow Corning Corp.

Impact of Network on Exchange Server

Exchange Server ->

April 3, 2000 Dow Corning Corp.

Network Benefits of PI DataSpotted several instances of Over

Utilization (Can almost predict instances of Network outages for users.)

Helped to determine a Baseline to determine the need for purchased band width and line speed. ($ can be found here.)

April 3, 2000 Dow Corning Corp.

In Summary We helped OSI touch up some of it's code. We spotted some troublesome software from

other vendors. We help to trouble shoot NT Server issues

and outages. We help to baseline our Global Network. We saved some $.

April 3, 2000 Dow Corning Corp.

How Did We Do It?PI Tag Configurations for PI PerfmonPI Perfmon InterfaceSQL Server Use for NetScoutPI Tag Configurations for NetScoutLessons Learned

April 3, 2000 Dow Corning Corp.

Lessons Learned 1 Run PI Perfmon on the server it is monitoring.

This uses the PIAPI and Buffering to push the data to a PI Server.

Trying to PULL data uses to much network bandwidth.

Trying to PULL data leaves you vulnerable to Network Outages.

Remember we are doing this across a wide area network.

April 3, 2000 Dow Corning Corp.

Lessons Learned 2Really think through the compression

and exception deviation percentages. Memory usage varies a LOT. CPU usage varies Even More. Regression Analysis ProcessBook Addin

can save you a lot of stats time.

April 3, 2000 Dow Corning Corp.

Lessons Learned 3 You CAN use SPC tools on an NT Server

Process. NT Server Process should run acceptablely under

the rules of SPC. You can find out what impacts you and sends you

out of control. Alarming based on Out Of Control can be done. Alarming based on Process Conditions is also

possible.

April 3, 2000 Dow Corning Corp.

Lessons Learned 4OSI Believes in Continuous

Improvement. PI Batch Improvement PI NetMgr Process PI Archss Process

April 3, 2000 Dow Corning Corp.

Lessons Learned 5When It comes to Network Stats, It is

hard to tell when you have enough. NetScout Probes Offers a lot of data. At

this point Dow Corning is only capturing the tip of the iceberg.

OSI is working with NetSout to get at much more of the data.

April 3, 2000 Dow Corning Corp.

Lessons Learned 6Watch Data in Long and Short Views.

Some items have data that takes weeks to develope.

Some items are best seen in the second they occur.

Only a long term historian can give you this long and short view of events.

Get the Data Have Some Fun