CERN - IT Department CH-1211 Genève 23 Switzerland t Oracle and Streams Diagnostics and Monitoring Eva Dafonte Pérez Florbela Tique Aires.

CERN - IT DepartmentCH-1211 Genève 23

Switzerlandwww.cern.ch/it

Oracle and Streams Diagnostics and

Monitoring Eva Dafonte Pérez

Florbela Tique Aires Viegas



Agenda

• Oracle Enterprise Manager• Streams monitoring tool• Local monitoring tools• Network diagnostic tools• Open questions



Oracle Enterprise Manager

• Set of centralized management tools – administration – configuration management– end-to-end monitoring – security capabilities

• Proactive monitoring and alerting• Monitoring service performance and usage• Automation, schedule jobs, patch

management

Overview













• Thresholds configuration• Metrics for the servers’ load• Run some advisors to try and pinpoint

performance or configuration issues• Can Tier1 use CERN OEM to monitor their

databases?

Open questions



Streams monitoring tool

• Objectives:– Replication topology– Status of streams connections– Error notifications– Monitor streams performance (latency, throughput, …)

– Monitor resources related to the streams performance (Streams Pool memory, Redo generation)

• Architecture:– ‘Strmmon’ daemon written in Phython

• collects streams and instances info + repository• errors and warnings

– End-user web application

http://oms3d.cern.ch:4889/streams/main.phpOverview




Monitor view

Connection view




Database list




Connection dashboard view

Detailed Streams view




Graph generator




• New features– Error tab (web application)

• list of errors that have been reported by streams processes

– Availability tab (web application)• Percentage availability of each instance provided with

availability plots.

– New metrics (monitor)• CPU consumption• Physical bytes

– Read

– Written




Errors List

CPU consumption




Availability




• Proposition of future features– Weekly reports(number of transactions applied, number of

LCRs streamed etc)?– More notifications via mail(high latency,high CPU

utilization etc.)?– Some automatization in streams administration?

• Detecting common failures (e.g. propagation hangs)

• Proceed procedure to solve the failures

• Streams errors report:– Any action necessary at Tier1?– Who is testing what?

• Email alerts– RAL still does not receive notifications

Open questions



Local and Network monitoring

• Is OEM sufficient?• Which other tools?• To which metrics we should pay attention?• “Homemade” tools for backup monitoring:

– RAL, …

• Local monitoring with Nagios– Is this reasonable?– Any experience?

Open questions

• Triumf (slides)• BNL (slides)



Overall

• What, specifically, Tier-1s should monitor on their own databases?

• What CERN want to know about the sites?• What Tier-1 sites need to know about the

CERN databases?• What Tier-1 sites need to know about other

Tier-1 sites?

CERN - IT Department CH-1211 Genève 23 Switzerland t Oracle and Streams Diagnostics and Monitoring Eva Dafonte Pérez Florbela Tique Aires.

Documents

t streams

department ch

tool availability slide

detailed streams view

streams diagnostics

written slide

streams performance

tool database list slide