Configuration, service instrumentation & service management

1

Enabling Grids for E-sciencE

Joachim Flammer Service Management

Configuration,Configuration,service instrumentation service instrumentation & service management& service managementJoachim Flammer

on behalf of the JRA1 Integration team

2



Overview

• Comments• Requirements• Service interactions• Common issues• Implementation scenarios• DM JMX example• Discussion points

3



A few comments to start …

• Timeline– Discussion in the area of configuration an service management is going

on for several months already

– If we want to have something in gLite 1.x we don’t have much time left

• Framework– So far discussion concentrated much on configuration, dynamic

configuration, …

– … however the scope is much wider and is about proper service management and instrumentation

• Priorities– In our opinion a proper working service management is a key corner

stone of gLite

– It will up to the project to decide for the priorities

– Discussion started in PTF …

– … but we need to have a proposal ready

4



Requirements

• Do we need service instrumentation ?• There has been a lot of discussion …

… clear project made statement about the requirement

– JRA1 / SA1 presentation in Athens– Discussions with SA1– Discussions with user community– PTF Requirements Database

5



PTF Requirements Database

• Many entries related to configuration / management / logging & error handling

• Entries are coming from different sides (SA1, applications, …)• Some examples:

– 100724 Grid services should have an administrative interface for:

• monitoring • creating alarms • taking out sites/services by grid operation center etc...

They should have common, extensible, set of APIs – 100727

All grid services must be able to use standardized interface for error and log messages and files. Log level has to be adaptable for e.g. different level of debugging. Minimal level has to guarantee audit trails.

– 100726 Grid services must enable the measurement of quantities relevant to

the verification of service level agreements (SLA)

6



Service interactions types

Two types of interactions:• Service interacts with system

– Configuration

– Logging

• Manager/Information system interacts with – service

getting service information (version number, uptime, …) operations on service (start, stop, clear, …) event notification (errors, full disks, ….)

– container– system

7



Service interaction with System

• Configuration– reading/writing of configuration

• Logging– common logging format– logging to different destinations

• Saving of system state– recovery after shutdown

8



Service Information

• State– deployed, started, stopped – Status: running/not running

• Relationships– Dependencies (e.g. needs

service A, B, C)– Replacement services (is

duplicate of service D,E)

• Generic information– Name– Version– Package signature– Interface version

• Configuration– Values– Last updates

• Timing– uptime– present time

• Service performance metrics– Last processing time– Longest processing time

• Logging– Log information– Number of errors

• Service specific internal information– Number of used DB

connections– Number of requests

• Available services in container / node

9



Service operation

• Status operation– start, stop, pause

• Internal operations– pinging– Different kind of service specific operations

cleanup connections reset redirect log information add user ….

10



Service events

• Service wants to inform about special events– Service started, stopped, ….– Fatal error occurred– Internal problems

disk running full performance low need more coffee ….

11



The challenge …

• Heterogeneous system– Web services

C++ Java

– Daemons

• Common service management interfaces– basic common set for all services

– can be extended by an individual service

• Reuse existing capabilities – minimize impact on services– e.g. start/stop

• Several information sources– service– container– node

• Standard outside communication– standard outside SOAP interface:

CIM, WBEM, WSDM, ….– self descriptive

• Separate management from business logic– service users are interested in

service API– management application are

interested in management API– separate WSDL interfaces

12



Implementation scenario

• We foresee a two-fold structure– one global management service per node

gateway to the outside management collects information from different sources

• Service(s)

• Container(s)

• System directs commands to different destinations

• Service(s)

• Container(s)

• System

– service specific parts in each service Provides common functionality

• Loading of configuration

• Interface to management service

13



Node system logic

Tomcat

gLiteService

Manager


servicelogic

servicelogic

gLite management application

container logic

log files

gLiteService

gLiteService

Service A

SO

AP

? SOAP ? Service B

SO

AP

JMX

configuration

WBEM/CIM/WSDM via SOAP

Information system

14




• Implementation scenarios– Central gLite Service Manager

Java web service Implemented in Tomcat JMX for internal management Combining different sources/destinations Outside connectors: WBEM / CIM / WSDM via SOAP …

– gLite service Java web service

• inside tomcat• Use JMX for communication central gLite Service Manager

C++ web service• (extended) ServiceConfigurator for communication with central

gLite Service Manager, e.g. via SOAP Daemons

• Remote/Local calls for communication with central gLite Service Manager

15



gLiteService - configuration

• Example for gLiteService– Managing configuration

Load from central place Handling of complex data type Enable querying of configuration from outside

– Example service: Datamanagement Fireman MySQL catalog(org.glite.data.catalog-service-fr-mysql) Java webservice inside tomcat Uses JDBC pool connection to MySQL Sets up the JDBC connection via JNDI Uses two classes

• Service classorg.glite.data.catalog.service.composite.FiremanCatalogSoapBindingImpl

• DB connection class

org.glite.data.common.helpers.DBManager

16



Tomcat

gLite Manager


servicelogic

Web browser

log files

gLiteService

Fireman Service

SO

AP

JMX

configuration

HTTP

Configuration MBean

HTTPadapter

JMXserver

17



Example configuration file<config> <parameters>

<test.value description=“This is a test value.” value=“HelloWorld” type=“string”/>…

</parameters>

<container name=“egee_db”description=“EGEE Metadata Database JNDI DataSource”type=“org.apache.commons.dbcp.BasicDataSource”jndi.reference.name=“jdbc”>

<parameters> <username

description=“The connection username.”jmx.readable=“true”jmx.writeable=“false”type=“string”value=“egee”/>

<driverClassNamedescription=“The fully qualified Java class name of the JDBC driver.”jmx.readable=“true”jmx.writeable=“false”type=“string”value=“org.gjt.mm.mysql.Driver”/>…

</parameters> </container></config>

18



Turning on the central configuration

Step 1: Turn on the central configuration

Implement the GliteService class specifying your service name to– loads configuration from central location

here: /opt/glite/etc/config/glite-SERVICENAME.cfg.xml

– makes configuration information available to central JMX server Data Management service

19




...public class FiremanCatalogSoapBindingImpl

extends ServiceImpl implements FiremanCatalog {

private DBManager firemanDBmanager = null;

...

public FiremanCatalogSoapBindingImpl(){super();

try{ firemanDBmanager = new DBManager(“jdbc/egee_db”);

} } catch (DBException e){ ... } } ...}

20




import org.glite.management.service.GliteService;...public class FiremanCatalogSoapBindingImpl

extends ServiceImpl implements FiremanCatalog {

private DBManager firemanDBmanager = null; private GliteService gLiteService; private String serviceName = “catalog-service-fr-mysql”; ...

public FiremanCatalogSoapBindingImpl(){super();gLiteService = GliteService.instance(serviceName);try{ firemanDBmanager = new DBManager(“jdbc/egee_db”); // firemanDBmanager = new DBManager(“egee_db”, serviceName); // firemanDBmanager = new DBManager(“egee_db”, gLiteService);}

} catch (DBException e){ ... } } ...}

21



Getting the configuration

Step 2: Getting the configuration information

There are three possibilities

1. via JNDI

2. via JMX

3. via GliteService

22



Getting the configuration: via JNDI

public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException {

try { m_db_pool_name = pool; Context initCtx = new InitialContext();

Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name);

} catch (NamingException e) { ...

} }

23





try { m_db_pool_name = pool; Context initCtx = new InitialContext();

// Context envCtx = (Context) initCtx.lookup("java:comp/env"); Context envCtx = (Context) initCtx.lookup("glite"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name);

} catch (NamingException e) { ...

} }

24




• Necessary adoptions:– Change of JNDI path

• Advantages– Only need to change JNDI path – Easy to change in between gLite & non-gLite system– Can use JNDI features: notification, …– Dynamic configuration possible if supported by application– Independent of central JMX server

• Disadvantages– Duplication of efforts

JMX for central & remote management JNDI for application

25



Getting the configuration: via GliteService


try { m_db_pool_name = pool;

Context initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name);

} catch (Exception e) { ...

} }

26




import org.glite.management.service.GliteService;

public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; //public DBManager(String pool) throws DBException { public DBManager(String pool, GliteService gliteService) throws DBException {

try { m_db_pool_name = pool;

// Context initCtx = new InitialContext(); // Context envCtx = (Context) initCtx.lookup("java:comp/env"); // m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name);

m_dataSource = (DataSource) gliteService.getConfigurationParameter(m_db_pool_name);

} catch (Exception e) { ...

} }

27




• Necessary adoptions:– Change JNDI lookup by GliteService function call– Pass GliteService to classes that use configuration

• Advantages– Easy retrieval of values– Dynamic configuration possible if supported by application– Independent of central JMX server

• Disadvantages– Extra features (e.g. notification) need to be implemented by

hand– Duplication of efforts (JMX for central, JNDI for application)

28



Getting the configuration: via JMX

import javax.management.*;

public class DBManager { private MBeanServer mbs;

public DBManager(String pool, String serviceName) throws DBException {

m_db_pool_name = pool;try{ List srvList = MBeanServerFactory.findMBeanServer(null);

ListIterator srvListItr = srvList.listIterator(); while(srvListItr.hasNext()) {

MBeanServer localMbs = (MBeanServer) srvListItr.next();if (localMbs.getDefaultDomain().compareTo("glite") == 0)

mbs = localMbs; }

ObjectName objectName = new ObjectName(serviceName + configurationSubMBeanPath);

m_dataSource = (DataSource) mbs.getAttribute(objectName, m_db_pool_name); } catch (Exception e){

29



Getting the configuration: via JMX

• Changes:– Lookup of central MBean server– Lookup of MBean instead of JNDI Object

• Advantages– Fully integrated solution

central management and local configuration coupled

– Standard management solution– Dynamic configuration possible– Can use all JMX features (notification, old/new value ..)

• Disadvantages– Depends on central JMX server

30



Discussions points

• Feedback– Comments– Ideas

• Requirements from your side– What do you want to see in the management interface?– What requirements do you have?– Priorities? What do you want to see first?

• Discussion on implementation possibilities– Architecture– Implement GliteService as

Parent class Add-on class

– JNDI / GliteService / JMX• C++ configurator

– News– Feedback (Eli, …)

• JMX studies– News– Feedback (Paolo, …)

Configuration, service instrumentation & service management

Documents