Ecotools Case Study : Database Monitoring at BNP Paribas Dennis Adams BNP Paribas.

Ecotools Case Study : Database Monitoring at

BNP ParibasDennis Adams

BNP Paribas

What’s in a name : Paribas Capital Markets

Now part of BNP Paribas Investment Banking € Euro-Denominated Bonds.Equities, Bonds and Derivatives Trading

Systems in most major financial centres Infrastructure in territories and central

London Data Centre.Finance and Reporting in Paris.

Databases and OS Types in Production

HP/UX DecUNIX Solaris NT 4 TOTALSybase 8 16 8 5 37Ingres 2 39 0 0 41TOTALS 10 55 8 5 78

Tokyo, Singapore, Hong Kong, Paris, London, New York

Our Responsibility: Data Management Group

Managing DBMS ServersBased in London, remote Support to

territories (out-of-hours callouts !)Liaise with London Operations 24h/dayOther Teams

Systems Management - hardware & OSNetworks - LAN & WANApplication Teams - Application Support

Objectives when purchasing Ecotools

Monitoring of Sybase DBMS detail down to SQL statementIngres as an additional requirement

ReliableWAN based Event AlertingLinks to TNG for Central AlertingExtract trend data for capacity planning

How we use Ecotools

Command Line Control Files for Scenarios

GUI

Monitored Agent Machines

Ecotools Repository

Alerts

How we use Ecotools

Single Central Solaris Console (V6.2.1)Continuous running 24 X 7Minimal use of the Ecotools GUIScenarios controlled from ECOCLI

start/stop UNIX shell scripts.

Alerts picked up by “tail” of ecotools logsummary on daily in-house web page

Configuring Domains

Domain = Logical grouping of ServersUnit of “discovery” within EcotoolsMonitor “All Servers” together

Configuring Domains

Group by Business Unit / DBMS Type / Territory ?

Lots of small Domains ?speed up “discovery”

Few no of large Domains ?easier to get to individual machines

COMPROMISE: DOMAINS OF MAX 25ORGANISED BY APPLICATION TYPE

(ALMOST)

Creating Scenarios

Scenario = Basic Collection Task One or more agents (cache hit, log size)One or more machines (“All Servers”)Time Interval (“10 Minutes”)

Either using the GUI or from .CTL fileCREATE USING GUI, SAVE AS A CTL

TEMPLATE... SED/AWK START FROM UNIX SHELL

Managing Scenario Control Files

Lots of Small Control files ?One UNIX impf process per scenarioa single machine can have 15 “request”s80 *15 impf processes on console (!!!)ps -ef | more

Few no of Large Control files ?Unix VI ctl file when new machine added.Lots of machines per ctl file makes restart a

problem

Managing Control Files

EVERY MACHINE HAS IT’S OWN CONTROL FILES (MAXIMUM OF 3)Use SED string substitution to created from

standard TemplatesEcotools Startup logic...

For file in *.ctldo

ecoclieco: run $file

done

GUI Stability Issues

GUI “hangs” when Alerts arrive.Determined by ECOCLI_BLINKALARMECOCLI_ALARMINTERVAL =1200 default =

2*CliLogInterval

The GUI Crashes !Corrupts master imdb - lose all dataBetter in version 6.2.1 (latest patches)

SWITCH OFF ECOCLI_BLINKALARM, USE UNIX SCRIPT TO TAIL LOG FILE

Managing Ecotools “Views”

View = Repository for performance dataconsists of indexed/ flat file directory

Need X months data - capacity planningCan keep views going for several weeks

but “imdb” = 12MB, .datalog = 1.8GB.

CREATE NEW VIEW EVERY MONTHSET “NO OF DATA POINTS=1500” IN

CONTROL FILES (OR LOSE DATA)

Trend Data for Capacity Planning

Several Month’s data requiredStart a new GUI for every single view ??SOLUTION : extract data to CSV files for

reading into databaseThere is no supported utility to do this.SHELL SCRIPTS - GREP / SED / AWK

UNSUPPORTED HACK !

Unicenter/TNG Integration

In-house implementation calling KSH script from Ecotools Scenario Language

Can be implemented as “MgrAction” or “AgtAction”

MgrAction = run on Solaris ConsoleGMRPY_CDBSTOP errors - not scalable

AgtAction = run on Agent MachineNOT YET PROVEN

Summary : Ecotools Experience so far

Evaluation of version 4.n in 1997Monitoring of Euro changeover in 1998Y2K tracking and Reporting 1999Version 6.1 chosen as Y2K versionSoon replaced by 6.2.1 with patches.Currently Monitoring nearly 80

Production Systems from London.

Looking at Ecotools V.7

Looks impressive on paperSQL server storing collected dataGenuine 3-layer architectureBetter User interface

Concerns:Support for Legacy Operating SystemsDec UNIX versionsAbility to create own “shell script” agents.Earlier Versions of Sybase

Ecotools Future at BNP Paribas

Evaluate V7 for monitoring NT/Sybase Production Systems.

Need UNIX shell agents for NT console Objective: move to centralised NT

console - eventuallyKeyword: STABILITY

Success Stories

Sybase Settlement System - TokyoReserved log space low

caused by backup server failing and transaction logs not being properly dumped. Restarted backup server

Potential system hang avertedDelayed Settlement = We get Fined

Success Stories

Ingres Equity Derivative Trading - New York

Large Table Approaching Ingres Architectural Limit of 2GBPlanned outage to remodify table into

multiple locations to allow it to grow

Equity Trading Outage avertedKeep the Traders Trading

Success Stories

Ingres Forex Trading - ParisIngres Error Log reported potential

UNIX Filesystem Error one eveningAlerted Paris Unix Team, who decided

to ignore the warningSystem Crashed overnightBut at least we could say “I told you so”

Final Proof

Accepted within DMG GroupIntegrated with our way of workingWe are now a more Pro-Active TeamLooks good on the CV, provided you

call it... E-COTOOLSLunch Time ?