Getting of your data into PI Getting all of your data into PI or: Teach yourself to be an interface expert in 7 days…

Getting Getting all of your of your data into PIdata into PI

or: Teach yourself to be an interface expert in 7 days…or: Teach yourself to be an interface expert in 7 days…

Everyone here has an EMS/DCSEveryone here has an EMS/DCS

• You have at least one PI interface collecting data (hopefully) from this system.

• The control system was the impetus for getting PI, but there are many other data sources out there.

• Collect as much data as your system can reasonably handle – someone will want it eventually.

But what about the rest of it?But what about the rest of it?

EMS related data• UDS• SE• SA• Day ahead forecast• Dispatcher actions

Market and other data• LMP• Weather• Computer system and

network performance• other?

Remember to send a Christmas card toRemember to send a Christmas card to the developer of Batch File Interfacethe developer of Batch File Interface

• EMS = Elderly Mainframe System• UNIX systems love ASCII files

(Windows too to a lesser extent)• Files can hold a surprising amount

of data in a reasonable amount of space (if you don’t keep them around for too long)

• Files can be easily manipulated before being passed to BFI – which is good because BFI can be a bit particular…

Batch File Interface MagicBatch File Interface Magic

1. Batch Files are not antiquated technology – they are proven technology.

2. Format, Format, Format.

3. Run multiple instances of BFI on one machine. Increases throughput and allows the use of multiple directories.

4. BFI clusters well – although it may not be supported.

BFI in use at PJMBFI in use at PJM

• UDS– Data is generated every 5 minutes– Custom parsing routine runs prior to BFI

(.NET service)• State Estimator

– ~200,000 tags– Every 30 seconds– Pre-processing done in VB.NET

• Backload of data from previous historians

Relational DatabasesRelational DatabasesThe other “flat file”The other “flat file”

• EMS data tends to be available via proprietary interfaces (ICCP) or flat files, business data tends to live in databases (Oracle, SQL Server, etc.)

• RDBMS interface works well – is more complicated to configure than BFI

• Cooperation from database administrators is essential for success

RDBMS at PJMRDBMS at PJM

• LMP data– Most sought after piece of data– “Lives” in an Oracle database– RDBMS interface connects to a dedicated view in the

database and uses a dedicated database user ID

Now it gets *really* interestingNow it gets *really* interesting……

• The Day Ahead load forecast– PI doesn’t want it, well

the archive anyway.– Dispatchers need it– Data is in an Oracle

database

Enter the Enter the COM connector!COM connector!

• Allows data that isn’t actually in the PI archive to look like it is.

• Points are configured similarly to “real” PI points

PI NetMgr

Snapshot

Request

Response RPC Call

RPC Response

COMConnector

COM call

Return Data ForeignDataSystem

Redirector

OLEDB COM ConnectorOLEDB COM Connector

• Used to access data not stored in PI, Used to access data not stored in PI, but accessible via OLEDB.but accessible via OLEDB.

• Since the PI Archive subsystem is Since the PI Archive subsystem is not involved, future timestamps are not involved, future timestamps are not a problemnot a problem

• Can be though of as similar to a Can be though of as similar to a DataSet in Process Book, but with DataSet in Process Book, but with the administration centralized and the administration centralized and handled as other PI tags (not stored handled as other PI tags (not stored in Process Book files all over the in Process Book files all over the network)network)

Pros and Cons of the COM connectorPros and Cons of the COM connector

• PROs– It works– Dirt simple for the

users– Once you master this,

it’s a practical way to interface to some strange stuff

• CONs– Complicated

(relatively)– Future data not well

represented in ProcessBook (markers only)

– People will ask you 3,276,352 times if this could be done in ACE

Getting even farther outside the boxGetting even farther outside the box

• The PI SDK has brought the ability to read/write data to PI from a script.

• If you can write a script to read the data, you can write it to PI.

• “Web scraping” is a fancy name for parsing a text file (HTML formatted, but text none the less)

• Internet Explorer is easily scriptable• Snooze alert – here comes the code:

Set oIE = CreateObject("InternetExplorer.Application")oIE.Navigate2("http://wwwa.accuweather.com/adcbin/public/curcon_local.asp?

partner=accuweather&zipcode=94203&metric=0")While oIE.Busy

WScript.Sleep(100)WendstrHTML = oIE.Document.body.innerText

• Parse strHTML like any other text. Then use the SDK to write it to PI

Why do I think that I just opened Why do I think that I just opened Pandora's box?Pandora's box?

• There is no practical limit to the type of data that can be collected this way

• Think twice before revealing this ability to your users (and please don’t tell mine!)

• Every line of useful script requires 30 lines of error control.

• High overhead – you’ll not be collecting 1000’s of tags like this.

• Consider building a text tag to write script errors to. Monitor this text tag with Processbook (or another script – egad!)

The one who dies with the most The one who dies with the most interfaces wins, right?interfaces wins, right?

• What about all those computers?• How do you know

– They’re alive– They’re keeping up– You’re not about to be paged

• The answer is Perfmon, Ping, SNMP, and TCP Response – a.k.a. ITMonitor

IT monitoring is important!IT monitoring is important!

• If your systems are overload, hung or otherwise dead, you don’t collect any data!

• Perfmon – performance counters from your Windows systems

• SNMP – performance counters from your UNIX systems and network devices

• PING – is there anybody out there?

Caveats…Caveats…

• Your IT people may or may not want to give you access to performance data.

• Check with your OSISoft account rep about purchasing these interfaces stand alone.

Whew! Where are we now…Whew! Where are we now…

• Siemens Spectrum• ICCP• Batch File• RDBMS• SDK Scripting• COM Connector• Perfmon• SNMP• Ping

But there’s always room for 1 more:But there’s always room for 1 more:

PItoPIPItoPI

•Used to replicate data Used to replicate data between PI systemsbetween PI systems

•Geographic dispersionGeographic dispersion

•OLAP processingOLAP processing

•Fault ToleranceFault Tolerance

•Additional compressionAdditional compression

ConclusionsConclusions

Life Today• PJM is using 10

different interface technologies to collect data.

• This number won’t be going down anytime soon (#11 has been

selected)

Life Tomorrow• More market data

– More use of RDBMS– More use of COM

connectors

• Attempts to leverage the SDK to reduce the number of file based interfaces

Question

Question

ss??