Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162

ibm.com/redbooks Redpaper

Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Budi DarmawanAleem Subhedar

Celena TanHoward Anglin

Huang ChuanRohit Dhall

Planning for performance of management infrastructure

Implementing with multiple servers

Performing mass update of agents

Front cover

http://www.redbooks.ibm.com/



December 2007

International Technical Support Organization

© Copyright International Business Machines Corporation 2006, 2007. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.

Second Edition (December 2007)

This edition applies to Version V6.0 of ITCAM for Response Time Tracking (product number 5698-A75) and Version 6.0 of ITCAM for WebSphere (product number 5698-A71).

Note: Before using this information and the product it supports, read the information in “Notices” on page vii.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixThe team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixBecome a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. Overview of IBM Tivoli Composite Application Manager implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Application management with IBM Tivoli. . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.1 IBM Tivoli systems management portfolio . . . . . . . . . . . . . . . . . . . . . 31.1.2 IBM Tivoli Composite Application Manager solution . . . . . . . . . . . . . . 5

1.2 Scope of and concerns relating to large-scale implementation. . . . . . . . . . 61.2.1 Defining large-scale implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.2 Concerns and considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3 Overview of IBM Tivoli Composite Application Manager. . . . . . . . . . . . . . . 81.3.1 Understanding ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 81.3.2 Understanding IBM Tivoli Composite Application Manager

for Response Time Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.4 Document organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 2. Planning for ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . 172.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.2 Product architecture of ITCAM for WebSphere. . . . . . . . . . . . . . . . . . . . . 182.3 Deciding on the size of the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.2 Sizing estimation for ITCAM for WebSphere managing server. . . . . 222.3.3 Data collector overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4 Implementation options for ITCAM for WebSphere. . . . . . . . . . . . . . . . . . 252.4.1 Designing the managing server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.4.2 Deploying a large number of data collectors. . . . . . . . . . . . . . . . . . . 28

2.5 Communication and security considerations. . . . . . . . . . . . . . . . . . . . . . . 292.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.5.2 Firewall and port consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.6.1 Failover and fault tolerance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

© Copyright IBM Corp. 2006, 2007. All rights reserved. iii

Chapter 3. Installing ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 333.1 Installing ITCAM for WebSphere managing server . . . . . . . . . . . . . . . . . . 34

3.1.1 Installation configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.1.2 Database installation and configuration . . . . . . . . . . . . . . . . . . . . . . 353.1.3 WebSphere Application Server considerations. . . . . . . . . . . . . . . . . 413.1.4 Configuring the split server of the managing server . . . . . . . . . . . . . 423.1.5 Installation and setup of split server . . . . . . . . . . . . . . . . . . . . . . . . . 433.1.6 Verifying the installed components in a split environment . . . . . . . . 473.1.7 Adding additional publish servers and archive agents . . . . . . . . . . . 48

3.2 Deploying data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.2.1 Setting up the silent installation process . . . . . . . . . . . . . . . . . . . . . . 503.2.2 Installing the data collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.2.3 Installing and configuring the data collector together . . . . . . . . . . . . 543.2.4 Configuring data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.2.5 Automatically discovering the installation parameters . . . . . . . . . . . 61

3.3 Configuring and setting up SSL communication . . . . . . . . . . . . . . . . . . . . 643.3.1 Managing server Secure Socket Layer setup . . . . . . . . . . . . . . . . . . 643.3.2 Data collector Secure Socket Layer setup . . . . . . . . . . . . . . . . . . . . 653.3.3 Working with custom certificates. . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 4. Maintenance of ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . 754.1 Operating ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764.2 Performance and availability of the managing server . . . . . . . . . . . . . . . . 76

4.2.1 Performance of the WebSphere Application Server . . . . . . . . . . . . . 764.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.2.3 Data trimming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.3 Backup and recovery configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.1 ITCAM for WebSphere backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.2 WebSphere configuration backup . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.3 Database backup and restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

4.4 Log files and configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.4.1 Managing log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.4.2 Managing the configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.5 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.5.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.5.2 Updating ITCAM for WebSphere managing server. . . . . . . . . . . . . . 874.5.3 Updating ITCAM for WebSphere data collectors . . . . . . . . . . . . . . . 90

Chapter 5. Planning for ITCAM for Response Time Tracking . . . . . . . . . . 935.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.2 Product architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.3 Sizing the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

iv Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

5.3.2 Sizing estimation for ITCAM for Response Time Tracking . . . . . . . . 975.4 Deployment of ITCAM for Response Time Tracking. . . . . . . . . . . . . . . . . 98

5.4.1 Designing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 985.4.2 Deploying the management agent . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.5 Communication and security considerations. . . . . . . . . . . . . . . . . . . . . . 1005.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.5.2 Firewall and port considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.6.1 Failover and fault tolerance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Chapter 6. Installing ITCAM for Response Time Tracking. . . . . . . . . . . . 1056.1 Clustering the management server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

6.1.1 Preparing the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.1.2 Installing the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1076.1.3 Installing WebSphere Application Server . . . . . . . . . . . . . . . . . . . . 1096.1.4 Installing WebSphere Load Balancer . . . . . . . . . . . . . . . . . . . . . . . 1186.1.5 Installing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 1246.1.6 Checking the configuration of the RTT cluster application . . . . . . . 128

6.2 Deploying the management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 1296.2.1 Silent installation of the management agent . . . . . . . . . . . . . . . . . . 1296.2.2 Command-line interface for management components . . . . . . . . . 1306.2.3 Defining management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 131

6.3 Setting up Secure Sockets Layer certificates . . . . . . . . . . . . . . . . . . . . . 1336.3.1 Secure Sockets Layer for ITCAM for Response Time Tracking . . . 1336.3.2 Working with custom certificates. . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Chapter 7. Maintenance of ITCAM for Response Time Tracking . . . . . . 1377.1 Operational issues pertaining to a large environment . . . . . . . . . . . . . . . 1387.2 Performance and availability of management server . . . . . . . . . . . . . . . 138

7.2.1 Performance of WebSphere Application Server . . . . . . . . . . . . . . . 1387.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

7.3 ITCAM for Response Time Tracking files . . . . . . . . . . . . . . . . . . . . . . . . 1407.3.1 Backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1407.3.2 Managing the log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

7.4 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.4.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.4.2 Updating ITCAM for Response Time Tracking management server1427.4.3 Updating ITCAM for Response Time Tracking management agents143

Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Contents v

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

vi Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM application programming interfaces.

© Copyright IBM Corp. 2006, 2007. All rights reserved. vii

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Redbooks (logo) ®pSeries®z/OS®AIX®CICS®Database 2™DB2 Universal Database™DB2®ETE™

ETEWatch®IBM®IMS™Lotus Notes®Lotus®Monitoring On Demand®MVS™Notes®Operating System/400®

OMEGAMON®OS/400®Rational®Redbooks®Tivoli®WebSphere®Workplace™

The following terms are trademarks of other companies:

Oracle, JD Edwards, PeopleSoft, Siebel, and TopLink are registered trademarks of Oracle Corporation and/or its affiliates.

Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance, Inc. in the U.S. and other countries.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.

Enterprise JavaBeans, EJB, Java, JavaBeans, JDBC, JMX, JNI, JRE, JVM, J2EE, Solaris, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Outlook, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

viii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Preface

This IBM® Redpaper discusses large-scale implementation of IBM Tivoli® Composite Application Manager for WebSphere® and IBM Tivoli Composite Application Manager for Response Time Tracking. Large-scale implementation is typically characterized by the number of monitoring agents deployed and the number of transactions load-managed. A typical large-scale implementation of a monitoring product contains the following challenges:

� Keeping up the performance of the monitoring tools to accommodate the processing load from the agents.

� Automation of installation, update, and maintenance of monitoring agents based on silent installation and automated update.

� Specific day-to-day maintenance actions to ensure performance and availability of the monitoring solution.

This IBM Redpaper addresses these issues with regard to the implementation of ITCAM for WebSphere and ITCAM for Response Time Tracking on distributed platforms. The discussion is divided into planning issues, implementation guides, and maintenance considerations.

The team that wrote this RedpaperThis Redpaper was produced by a team of specialists from around the world working at the IBM International Technical Support Organization (ITSO), Austin Center.

Budi Darmawan is a Consulting IT Specialist at the IBM ITSO, Austin Center. He writes extensively and teaches IBM classes worldwide on all areas of Tivoli systems management products. Before joining the ITSO in 1999, Budi worked as Solution Architect and Implementer in Integrated Technical Services, IBM Indonesia. His current interests include availability management, z/OS® systems management, and Java™ programming.

Aleem Subhedar is a Staff Software Engineer with India Software Labs in Pune, India. He has seven years of experience in AIX® and Middleware System Administration. He holds a degree in Chemistry from Pune University. His areas of expertise include AIX, pSeries®, and related system technologies. He is an IBM Certified System Expert. His areas of interest include pSeries virtualization and high availability.

© Copyright IBM Corp. 2006, 2007. All rights reserved. ix

Celena Tan is a Managing Consultant with IBM Software Group Services in Australia. She has 14 years of experience in the IT field. She holds a Masters of Technology from National University of Singapore and a Bachelor of Electrical Engineering (Hons) from the University of Tasmania. Her areas of expertise include ITCAM family products and rational testing, and change and configuration management products.

Howard Anglin is a Deployment Expert for ITCAM for WebSphere, Response Time Tracking, IBM Tivoli Monitoring in the United States. He has worked with various large customers, and in his role as an IT Specialist he has resolved deployment, integration, and performance issues. He has nine years of experience in the software test and development field with emphasis on the WebSphere Application Server. He holds a Bachelor of Science in Electrical Engineering from Manhattan College, Riverdale, New York. Howard began his career at IBM in the pSeries Hardware Group as a Test Engineer developing automation solutions for the production line. He then transferred to the software group.

Huang Chuan is a Senior Test Lead of IBM China CSDL lab. He has five years of experience in software developing and over six years of experience in software product testing. He has led the ITCAM for Response Time Tracking test project for several releases. He holds a degree in Computer Science from the University of Electronic Science and Technology of China.

Rohit Dhall is an IT Architect with GBS, IBM India. He has 10 years of IT experience in technologies like client-server computing, Web-based transactional systems, data warehousing, and data mining. His major expertise is in designing, implementing, and tuning large-scale Internet banking, eMortgage, and anti-money laundering solutions for the banking and financial sector. He is EXIN ITIL® certified and also holds certification in Java and EJB™ from Brainbench. His current interests include SOA and IBM Virtualization offerings.

Thanks to the following people for their contributions to this project:

Donna Martin, Noel Lewis, Tony Williams, Marco De Gregorio, Sushanto PanditIBM Software Group, Tivoli Software

John HortonAuthor of the first edition of Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time TrackingLarge-Scale Implementation of IBM Tivoli Composite Application Manager, REDP-4162

Julie CzubikInternational Technical Support Organization, Poughkeepsie Center

x Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Become a published authorJoin us for a two-week to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will team with IBM technical professionals, Business Partners, or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcomeYour comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this Redpaper or other Redbooks® in one of the following ways:

� Use the online Contact us review book form found at:

ibm.com/redbooks

� Send your comments in an e-mail to:

[email protected]

� Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. HYTD Mail Station P0992455 South RoadPoughkeepsie, NY 12601-5400

Preface xi

http://www.redbooks.ibm.com/residencies.html

http://www.redbooks.ibm.com/residencies.html



http://www.redbooks.ibm.com/contacts.html

xii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Chapter 1. Overview of IBM Tivoli Composite Application Manager implementation

This chapter provides an overview of the large-scale implementation issues for IBM Tivoli Composite Application Manager. This chapter covers the following topics:

� 1.1, “Application management with IBM Tivoli” on page 2

� 1.2, “Scope of and concerns relating to large-scale implementation” on page 6

� 1.3, “Overview of IBM Tivoli Composite Application Manager” on page 8

� 1.4, “Document organization” on page 15

1

© Copyright IBM Corp. 2006, 2007. All rights reserved. 1

1.1 Application management with IBM Tivoli

Computer-based applications are the lifeblood of modern enterprises. Most business processes are driven by the so-called computer application that promotes productivity, automates processing, and minimizes human errors. These applications enable business persons to focus on what must be done, instead of how to do it. However, as business processes rely more on these applications, the applications become critical to the business. The applications must be available for the execution of the business processes.

Most applications evolved from centralized applications typically managed by the information technology (IT) department or mainframe-based applications, where all the application layers are maintained from the central mainframe. Today, applications tend to have multiple layers, often distributed across different servers, different platforms, and different components. These applications are called composite applications. This complicates the management of applications on matters such as operational settings, problem determination, and performance management.

Applications as a business-critical entity must be available with adequate response time for users to perform their tasks. With application components spread throughout the enterprise, problem determination and performance management are typically complicated. There is no clear path for finding which component faces the problem. Is it the database? A network problem? The application server experiencing a bottleneck? A user machine stall? Sometimes, these components even belong to different organizations.

Figure 1-1 shows a typical composite application. This is used by multiple users through the Internet and intranet. It consists of multiple application layers, each with its own abstraction level. Some of the applications have the original back end in the mainframe transactions.

Figure 1-1 Composite application

2 Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Composite applications are regarded as the ultimate application management challenge, as they span different application servers that communicate with each other. This architecture enables modular application development, where changes in a layer may not affect other layers, but introduces the complexity of multiple components.

This paper demonstrates how to implement the IBM Tivoli Composite Application Manager family of products in a large-scale environment. This chapter introduces IBM Tivoli product portfolio and how IBM Tivoli Composite Application Manager product fits.

1.1.1 IBM Tivoli systems management portfolio

IBM Tivoli product solutions are aligned towards an overall IBM IT Service Management approach. Figure 1-2 shows the IBM IT Service Management portfolio structure.

Figure 1-2 IBM IT Service Management

This approach provides Information Technology Infrastructure Library-aligned automation work flows. Future offerings will provide an open standard-based and configuration management database-based solution, as well as a workflow engine.

IT Operational Management Products

IT Service Management Platform

IT Process Management Products

Best Practices

Change and ConfigurationManagement Database

Server, Network & Device

ManagementStorage

ManagementSecurity

ManagementBusiness

ApplicationManagement

Service Delivery

& SupportService

DeploymentInformation

ManagementBusinessResilience

IT CRM & Business

Management







Best Practices




ManagementStorage

ManagementSecurity

ManagementBusiness



ManagementStorage

ManagementSecurity

ManagementBusiness


Service Delivery

& SupportService



IT CRM & Business

Management

Service Delivery

& SupportService



IT CRM & Business

Management

Chapter 1. Overview of IBM Tivoli Composite Application Manager implementation 3

The operational management pillar shown in Figure 1-2 on page 3 is divided into software families. The availability solution addressed in business application management and server, network, and device management can be viewed as an integrated offering, as shown in Figure 1-3.

Figure 1-3 IBM Tivoli software portfolio

As shown in Figure 1-3, the Tivoli software portfolio is divided into the following components:

� Resource monitoring

Measures and manages IT resource performance, including servers, databases, and middleware.

� Composite application management

Monitors and manages an application and its components, and understands applications from the availability standpoint.

� Event correlation and automation

Correlates and automates events or faults that are generated by resource monitoring, application monitoring, or both to provide a concise root-cause analysis of failure in the environment.

� Orchestration and provisioning

Provides the ability to deploy or redeploy servers or components as requested on demand to fulfill processing requirements, if the necessity arises as indicated by the correlation engine.

Business Service Management

Event Correlation and Automation

Resource Monitoring

Orchestration and Provisioning

Composite Application Management

Security Storage


� Business service management

Provides a high-level view of business status as reflected by its underlying monitoring components. The view is either in real time or based on a service-level agreement.

1.1.2 IBM Tivoli Composite Application Manager solution

The IBM Tivoli Composite Application Manager family resides in the application management pillar of the Tivoli software portfolio. The current application management portfolio consists of the following products:

� ITCAM for Response Time Tracking V6.1

� ITCAM for Response Time V6.2

� IBM Tivoli Composite Application Manager for service-oriented architecture (SOA) V6.1

� ITCAM for WebSphere V6.1

� ITCAM for J2EE™ V6.1

� ITCAM for Web Resources V6.2

� ITCAM for CICS Transactions V6.1

� ITCAM for IMS Transactions V6.1

� IBM Tivoli OMEGAMON® XE for Messaging V6.0

Figure 1-4 shows the scope of composite application management.

Figure 1-4 Composite application management

Response Time Tracking

WBI messagingWeb Services calls

WebSphere performance

CICS/IMS transaction


Manage the overall composite application from the following sides:

� Get the user side of response time and availability with ITCAM for Response Time Tracking.

� Get IBM WebSphere middleware performance and analyze in-depth resource usage through ITCAM for WebSphere.

� Manage messaging from IBM WebSphere Business Integration MQ Series using IBM Tivoli OMEGAMON XE for IBM WebSphere Business Integration. For more details, refer to Implementing IBM Tivoli OMEGAMON XE for WebSphere Business Integration V1.1, SG24-6768.

� Manage message flow in an SOA environment and collect metrics for Web service calls using IBM Tivoli Composite Application Manager for service-oriented architecture (SOA).

� Provide the integration view with a mainframe-based, back-end application such as Information Management System (IMS™) or Customer Information Control System (CICS®) using ITCAM for IMS Transactions or ITCAM for CICS Transactions.

1.2 Scope of and concerns relating to large-scale implementation

This paper discusses large-scale implementation of IBM Tivoli Composite Application Manager. It specifically provides information about the implementation of ITCAM for WebSphere and ITCAM for Response Time Tracking in large-scale environments. The discussion is about large-scale implementation in distributed and mainframe environments, and includes the following topics:

� 1.2.1, “Defining large-scale implementation” on page 6� 1.2.2, “Concerns and considerations” on page 7

1.2.1 Defining large-scale implementation

There are several indications relating to large-scale implementation. These indications are based on the following factors:

� The number of application servers to be monitored

Each application server must have an agent installed to be monitored and managed. With the number of application servers ranging from hundreds to thousands, additional care must be taken to manage the deployment, maintenance, and processing of the managing server.


� The transaction rates on application servers

The transaction rates contribute to the overhead of the monitoring system. A balance of data collection and system health must be achieved. A large number of transactions potentially require larger management server processing.

� The number of network sites

The number of network sites typically corresponds to the potential bottlenecks between the sites. The bottlenecks may be from production data, monitoring data, or a security requirement such as a firewall.

� The requirement for high availability or fail over

This additional requirement, although not directly related to the scale, is typically a must for a large-scale implementation.

� The existence of multiple managed spaces that a site must handle

Managed space is defined as a group of environments with a single management database and a set of management server processes. Different managed spaces are usually used to separate the production and development environments. They are also used to prepare and test the changes to the management environment.

1.2.2 Concerns and considerations

Following is a list of concerns and considerations that are specific to a large-scale environment:

� Server size

As this is a large-scale implementation, sizing the servers to manage the environments is critical. The placement, configuration, and specification of a single server or multiple servers must be predetermined in order to avoid bottlenecks in processing. This sizing must also take into consideration special processing requirements such as debugging and troubleshooting and data collection and recovery.

� Deploying agents

The number of agents that must be deployed are enormous and prohibitive to being performed manually. Automated efforts must be included in the ability to deploy and implement the agents automatically with minimal manual intervention. This must cover initial deployment, fix pack implementation, and maintenance action.


� Security

This includes confidentiality support and firewall support.

– Confidentiality support secures information transfer between the agents and the servers.

– Firewall support allows the sites to be secured, with management action still flowing through in order to effectively manage the environment.

� Reliability

Fail over and fault tolerance are critical to maintain while monitoring business-critical applications. The reliability factor must be promptly addressed and ensured.

� Maintenance

Changes do happen, as with deployment. These changes must be applied to both the servers and the agents. Special consideration must be provided for a large-scale implementation with changes on both the servers and the agents. While server consideration applies to preserving, monitoring, and data collection with minimal downtime, agent consideration relates to automating the deployment process with minimal manual intervention and outage.

This paper deals with and addresses these concerns for ITCAM for Response Time Tracking and ITCAM for WebSphere implementations.

1.3 Overview of IBM Tivoli Composite Application Manager

This section explains the following topics:

� 1.3.1, “Understanding ITCAM for WebSphere” on page 8

� 1.3.2, “Understanding IBM Tivoli Composite Application Manager for Response Time Tracking” on page 11

1.3.1 Understanding ITCAM for WebSphere

This section provides an overview of ITCAM for WebSphere. The discussion includes the following topics:

� “Features and functions” on page 9� “Components” on page 9� “Platforms supported” on page 10


For more information about ITCAM for WebSphere, visit the following Web site:

http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/

Features and functionsITCAM for WebSphere helps increase the performance and availability of business-critical applications by providing facilities for real-time problem detection, analysis, and repair. Correlation spanning Java 2 Platform, Enterprise Edition (J2EE), Customer Information Control System, and Information Management System, and diagnostics at the method level pinpoint code problems to help resolve problems quickly and reduce support and operations costs.

Today’s business processes often depend on a number of complex applications. Although most businesses have traditional monitoring tools to manage individual resources at a high level, many lack an integrated solution to automatically monitor, analyze, and resolve problems at the service, transaction, application, and resource levels. As a result, operations and development may take a long time to identify, isolate, and fix composite application problems.

ITCAM for WebSphere is an application management tool that helps maintain the availability and performance of on demand applications. It helps you to quickly pinpoint, in real time, the source of bottlenecks in application code, server resources, and external system dependencies. This product also provides detailed reports that you can use to enhance the performance of your applications. ITCAM for WebSphere provides in-depth, WebSphere-based application performance analysis and a tracing facility.

ITCAM for WebSphere enables multiple levels of analysis to get a complete view of the application, depending on the requirement. From production-level monitoring to detailed heap and method debugging, it digs into Structured Query Language (SQL) performance analysis without the need for database monitors. It provides SQL information and information about calls that were made through Java Database Connectivity (JDBC™). ITCAM for WebSphere provides a composite status correlation for transactions that use Customer Information Control System and Information Management System as the back end.

ComponentsITCAM for WebSphere contains the following components:

� Managing server

This acts as the central component that manages and administers the data collectors. It stores that data in a relational database repository. A Web-based


http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/

application is provided to show monitoring results. This interface is also called the visualization engine.

� Data collector

This runs on the application server and collects performance information for the managing server.

� Tivoli Enterprise Monitoring Agent

This collects information that shows the status of the WebSphere Application Server and sends it to the Tivoli Enterprise Monitoring Server for display on the Tivoli Enterprise Portal. The Tivoli Enterprise Monitoring Agent is installed on individual machines where data collectors reside. This component is moved to IBM Tivoli Composite Application Manager for Web Resources in Version 6.2.

Platforms supportedFor a complete platform coverage list, refer to the following Web site:

http://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itcam6.html

Table 1-1 provides an overview of the platforms supported for ITCAM for WebSphere V6.

Table 1-1 Platforms supported for ITCAM for WebSphere

Component Software

Managing server operating system

� IBM AIX V5.2 and V5.3� Solaris™ 8 and Solaris 9 (SPARC)� Hewlett-Packard UNIX® (HP-UX) 11i 1� Windows® 200 Server or Advanced Server with

Service Pack 4 (SP4)� Windows 2003 Server Standard Edition/Enterprise Edition

(SE/EE)� Red Hat Enterprise Linux® (RHEL) 3.0 and 4.0� SUSE Linux Enterprise Server (SLES) 8 and 9

Managing server database � IBM DB2® V8.1 Fix Pack 6 (FP6) or IBM DB2 V8.2� Oracle® 8i SE R3 8.1.7, Oracle 9i SE R2 9.2, Oracle 10g

Managing server WebSphere WebSphere Application Server V5.1.x or WebSphere Application Server V6.x



1.3.2 Understanding IBM Tivoli Composite Application Manager for Response Time Tracking

This section provides an overview of ITCAM for Response Time Tracking. It discusses the following topics:

� “Features and functions” on page 9� “Components” on page 12� “Platforms supported” on page 14

For more information about ITCAM for Response Time Tracking, visit the following Web site:

http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

Features and functionsITCAM for Response Time Tracking proactively recognizes, isolates, and resolves transaction performance problems by using robotic and real-time techniques. It is an end-to-end transaction management solution that monitors user response time and helps you to visualize the transaction’s path through your application systems, including the response time contributions of each step. ITCAM for Response Time Tracking uses Application Response Measurement (ARM) technology to track the response time of a distributed application.

Data collector platform � AIX V5.2 and V5.3� Solaris 8 and 9 SPARC� HP-UX 11i 1� Windows 200 Server or Advanced Server with SP4� Windows 2003 Server SE/EE� RHEL 3.0 and 4.0� SLES 8 and 9� Red Flag Advanced Server (RFAS) 4.0 and 4.1(xLinux)� IBM Operating System/400® (OS/400®) V5.2 and V5.3� IBM z/OS V1.4, V1.5, or V1.6

Customer Information Control System

V2.2, V2.3, and V3.1

Information Management System

V7.1, V8.1, and V9.1

Component Software


http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

Today’s business processes often depend on composite applications that span Web servers, J2EE application servers, integration middleware, and mainframe systems. Although most businesses have traditional monitoring tools to manage individual resources, many lack an integrated solution to automatically monitor, analyze, and resolve user response time problems. As a result, it may take a long time to identify, isolate, and fix distributed transaction performance problems.

ITCAM for Response Time Tracking enables you to follow the path of a user transaction end-to-end across your business infrastructure. You can drill down to each step the transaction takes as it travels across multiple systems, and measure how each component of a transaction contributes to the overall response time. The entire transaction analysis process is transparent to customers and application developers. It collects transaction performance through robot and browser simulation, in-depth J2EE server instrumentation, and feedback from Customer Information Control System and Information Management System.

ITCAM for Response Time Tracking feeds the Tivoli Enterprise Monitoring Server to provide a comprehensive performance management solution on Tivoli Enterprise Portal. This enables the development of custom monitoring workspaces for managing enterprise applications.

ComponentsITCAM for Response Time Tracking consists of the following components:

� Management server

This acts as the central point of contact for ITCAM for Response Time Tracking. It consists of a WebSphere-based J2EE application that performs the management and administrative functions. The management server stores data in a central database repository.

� Store and Forward Agent

This relays traffic to and from the management agents. Typically, the Store and Forward agent is used in a firewall environment. It consolidates the port requirements for the connectivity.

� Management agent

This performs the monitoring function. Typically, it investigates the performance of the distributed application, depending on the management components deployed on it. The components that you can deploy are:

– Generic Windows workstation

This allows deployment of IBM Rational® Robot to measure transaction performance.


– Client Application Tracker

This uses IBM ETEWatch® scripts to collect performance information. Default monitoring is available for measuring IBM Lotus® Notes® and Microsoft® Outlook® performance.

– Synthetic Transaction Investigator (STI)

This performs Web-based transactions and measures the resulting response time.

– Quality of Service monitoring agent

This collects information about user performance by acting as reverse proxy between the user and the Web server.

– JavaTM 2 Platform, Enterprise Edition (J2EE) monitoring agent

This instruments and collects performance information about J2EE-based application servers such as WebSphere or WebLogic.

– Web Response Monitor component

– Rational Performance Tester

– Tomcat and JBoss monitoring component

– Generic Application Response Measurement (ARM) agent

This collects ARM events from a custom-instrumented application.

� Tivoli Enterprise Monitoring Agent for Tivoli Enterprise Monitoring Server

This feeds data from the ITCAM for Response Time Tracking server to display on the Tivoli Enterprise Portal.


Platforms supportedFor a complete platform coverage list, visit the following Web site:

http://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/Version60.html

Table 1-2 provides an overview of the platforms supported for ITCAM for Response Time Tracking V6.0.

Table 1-2 Platforms supported for ITCAM for Response Time Tracking

Component Software level

Management server operating system

� Microsoft Windows 2000 Server with SP4� Windows 2000 Advanced with SP4� Windows 2003 Server SE or EE� IBM AIX V5.2 or V5.3� Solaris 9 or 10� HP-UX 11i 1� RHEL 3.0 or 4.0� SLES 8 or 9

Management server database � Oracle 9i SE 9.2� IBM DB2 V8.1 ESE with FP3+ (required for WebSphere

Application Server V5.1.x)� IBM DB2 V8.1 ESE with FP6a+ (required for WebSphere

Application Server V6.x)� IBM DB2 V8.2

Management server WebSphere � WebSphere Application Server V5.1.x or later versions� WebSphere Application Server V6.0.1.x or later versions

Management agent platform � Windows 2000 Professional, Server or Advanced Server with SP4

� Windows 2003 Server SE or EE� Windows XP Professional with SP1� IBM AIX V5.2 or V5.3� Solaris 9 or 10� HP-UX 11i� RHEL 3.0 or 4.0� SLES 8 or 9� RFAS 4.0 or 4.1 (xLinux)� z/OS V1.4, V1.5, or V1.6� OS/400 V5.2 or V5.3



1.4 Document organization

This paper discusses the following topics:

� Before the implementation

Chapter 2, “Planning for ITCAM for WebSphere” on page 17, and Chapter 5, “Planning for ITCAM for Response Time Tracking” on page 93, discuss the planning and sizing considerations.

� The implementation

Chapter 3, “Installing ITCAM for WebSphere” on page 33, and Chapter 6, “Installing ITCAM for Response Time Tracking” on page 105, discuss additional steps that are required, such as reliability and automation considerations.

� After the implementation

Chapter 4, “Maintenance of ITCAM for WebSphere” on page 75, and Chapter 7, “Maintenance of ITCAM for Response Time Tracking” on page 137, discuss maintenance considerations and operational concerns relating to a large-scale implementation.



Chapter 2. Planning for ITCAM for WebSphere

This chapter provides information about areas that must be considered during the planning phase of IBM Tivoli Composite Application Manager implementation in a large environment. This chapter discusses the following topics:

� 2.1, “Planning considerations” on page 18� 2.2, “Product architecture of ITCAM for WebSphere” on page 18� 2.3, “Deciding on the size of the servers” on page 21� 2.4, “Implementation options for ITCAM for WebSphere” on page 25� 2.5, “Communication and security considerations” on page 29� 2.6, “Reliability and high availability” on page 32

2


2.1 Planning considerations

This section discusses the following aspects pertaining to large-scale implementations (see also 1.2.2, “Concerns and considerations” on page 7):

� Understanding the product architecture

This allows you to make the correct decisions. Section 2.2, “Product architecture of ITCAM for WebSphere” on page 18, describes the architecture for ITCAM for WebSphere.

� Sizing the servers

This is important to correctly acquire adequate servers and choose a sound software configuration option. Section 2.3, “Deciding on the size of the servers” on page 21, describes one approach.

� Understanding the servers’ configuration options and agent deployment

This is discussed for ITCAM for WebSphere in 2.4, “Implementation options for ITCAM for WebSphere” on page 25.

� Planning for communication security

This is a mandatory step for an enterprise with business-critical and sensitive information in a transaction environment. Section 2.5, “Communication and security considerations” on page 29, discusses confidentiality and firewall requirements.

� Discussing reliability, failover, and disaster recovery issues

These are the other mandatory aspects pertaining to a critical business process on a large enterprise. Section 2.6, “Reliability and high availability” on page 32 discusses this.

2.2 Product architecture of ITCAM for WebSphere

This section discusses the product architecture of ITCAM for WebSphere. This understanding is critical to plan and decide about the server configuration and other implementation issues. See also IBM Tivoli Composite Application Manager V6.1 Family Installation, Configuration, and Basic Usage, SG24-7151.

ITCAM for WebSphere V6.0 evolved from WebSphere Studio Application Monitor (WSAM) and IBM Tivoli OMEGAMON XE for WebSphere. ITCAM for WebSphere observes and reports on the health of Java 2 Platform, Enterprise Edition-based applications. It tracks the progress of applications as they traverse through Java 2 Platform, Enterprise Edition (J2EE) application servers,


middleware adapters and transports, and database calls, to back-end systems such as Customer Information Control System (CICS) or Information Management System (IMS) to extract business data or to invoke mainframe business processes.

The tracking of applications produces request traces, where the events in a request’s life are recorded and stored in a monitoring repository database. ITCAM for WebSphere captures the CPU and the elapsed internal times when events are called and exited, measuring as far down as the CPU consumed and the elapsed internal times charged to individual methods in J2EE classes. The methods or events taking the most time are marked as an application’s parts that deserve attention for runtime improvement studies and code optimizations.

ITCAM for WebSphere does not require modification of any J2EE or mainframe application code. Java Virtual Machine Tool Interface (JVMTI) interfaces and primitives, along with WebSphere Performance Management Interface (PMI) and z/OS System Measurement Facility (SMF) 120 records, are ITCAM for WebSphere’s principal data sources. The monitoring data is collected and analyzed to offer a wealth of information about the health of J2EE applications and their servers.

Many system-level performance metrics are collected and reported about J2EE application servers. The status of the servers and their resources, particularly at vital checkpoints such as CPU utilization, memory usage, and the status of internal components such as database connection pools, Java Virtual Machine (JVM™) thread pools, Enterprise JavaBeans™ (EJB) usage, and request processing statistics, are very important in locating real-time problems with J2EE applications. ITCAM for WebSphere brings attention to these critical indicators with real-time, graphical displays of their values and their trends over a span of time.

ITCAM for WebSphere is a distributed performance monitoring application for application servers. Its components are connected through IP network communication. The central component of ITCAM for WebSphere, the managing server, is its heart and brain. It collects and displays various performance information from application servers.

The application servers run a component of ITCAM for WebSphere called data collector, which is a collecting agent that runs in the application server and sends monitoring information to the management server. These data collectors operate independently of each other.

Chapter 2. Planning for ITCAM for WebSphere 19

Figure 2-1 shows the overall architecture of ITCAM for WebSphere.

Figure 2-1 ITCAM for WebSphere architecture

The application monitor comprises the following main parts:

� Managing server

A managing server comprises several Java-based components that provide the environment to collect and present management data.

� Data collector agent

A data collector agent runs on each monitored application server, whether J2EE, Customer Information Control System (CICS), or Information Management System (IMS), and communicates essential operational data to the managing server. Unique sampling algorithms maintain low CPU and network overhead, while providing application-specific performance information.

Web Server

Application servers withITCAM for WebSphere

Data collectors

Browser interfaceITCAM

for WebSphereManaging Server

Tivoli EnterpriseMonitoring Server

andTivoli Enterprise

Portal Server

I


2.3 Deciding on the size of the servers

The scale of the implementation must decide the size of the servers to be used. Sizing determines the hardware configuration and implementation consideration of the servers. This section discusses the following topics:

� 2.3.1, “Sizing parameters” on page 21

� 2.3.2, “Sizing estimation for ITCAM for WebSphere managing server” on page 22

2.3.1 Sizing parameters

The following parameters must be considered before deciding on the size of the servers:

� The number of data collectors for ITCAM for WebSphere

This value assumes that the application servers run a similar load profile. If the application servers have several load profiles, consider them in different groups.

� The transaction rate for application servers

The number of transactions executed for each minute, when multiplied with the number of data collectors or monitoring agents, gives the total amount of transaction information captured for a given period.

� The complexity of a transaction

It is not easy to understand the complexity of a transaction. This requires a more subjective approach than transaction rate counting, which can be retrieved from the transaction data or the application log. The relative complexity of transactions is determined by the number of method calls per transaction. Typically, the number of methods a complex transaction invokes is around four to six times that of a simple transaction.

� There are some product-specific parameters that affect sizing considerations. These parameters are built to filter out unimportant or insignificant information from the data that is collected. These parameters are:

– Data collection filter– Sampling rate– Monitoring level – Listening policy mask– Instrumentation level


2.3.2 Sizing estimation for ITCAM for WebSphere managing server

Specific to ITCAM for WebSphere, consider the following parameters for sizing:

� Communication bandwidth� Memory size� Processing requirement� Database size

Communication bandwidthSeveral communication traffic flows exist between the managing server and the data collector. The communication traffic flows are:

� Initial communication with the kernel to collect configuration information

This only happens in the initial connection when the data collector is started. This configuration information consists of sending the configuration and managing server Java archives.

� Management information to modify data collection level, sampling interval, or logging level from the kernel

This happens by request or when scheduled by Monitoring On Demand®. The size of this communication is small and negligible.

� Visualization engine requests for current active transactions

The impact of these requests depends on the following factors:

– The transaction rate and the average transaction response time that make up the average number of in-flight transactions

– The number of concurrent Web console users who may request the in-flight transaction information

� Transaction information is streamed to the publish server as it happens

This is the largest contributor to network load. It uses up the largest amount of network bandwidth. The formula is as follows:

– Monitoring in level 1: transaction rate x 353

– Monitoring in level 3: (transaction rate x 353) + (transaction rate x method call x 172)

Important: Sizing estimation for ITCAM for WebSphere managing server must be estimated for a worst-case scenario, that is, in the state that level 3 monitoring is run for the highest number of data collectors concurrently.


As an illustration, we use a sample environment with a transaction rate of 3,000 requests per minute on level 1 and 300 requests per minute on level 3 monitoring. The average method calls is 500 methods per requests. The transaction bandwidth required is:

– Level 1 transaction load: (3000 transaction / 60 seconds) x 353 = 17,650 bytes/sec

– Level 3 transaction load: (30 transaction / 60 seconds) x 353 + (30 / 60) x 5,000 x 172 = 430,176 bytes/sec

As shown in this example, the majority of network usage is spent on level 3 analysis. In a real production environment, for the majority of time, ITCAM for WebSphere runs on level 1. Therefore, the communication requirement is low. However, prepare an installation to occasionally increase monitoring in level 3 for problem determination purposes.

Memory sizeMemory requirement is typically important for the following components:

� Kernel

The memory size of the kernel is directly related to the number of data collectors. The typical size of 64 MB in the setenv.sh may have to be increased for more than 50 data collectors.

� Publish server

The memory size is related to the number of transactions the publish server has to process, with some consideration to the transaction complexity factor, that is, the number of methods invoked. The publish server’s memory must be adequate to handle the data size between garbage collector intervals. For garbage collection per minute, you must accommodate a minute’s worth of data. In the example provided in “Communication bandwidth” on page 22, the total size of publish server memory for processing the load must be around 4.3 x 60 x (1.5) = 387 MB. Note that the base publish server was already using around 100 MB of storage.

� Archive agent

This requires memory as a subset to the publish server and is masked by the sampling percentage from the publish server. The archive agent uses more memory than the sampling rate percentage, as it performs Java Database Connectivity (JDBC) database calls.


� Visualization engine memory size

This depends on the number of users who are connected and the activities that they perform. Users are categorized into the following groups:

– Users monitoring the availability screens– Users collecting performance reports– Users monitoring in-flight threads

Modify the visualization engine’s memory size by using the WebSphere Application Server administration console.

Memory sizes for ITCAM for WebSphere components are defined in the setenv.sh file that is sourced by all overseer components.

Processing requirementThe processor requirement for ITCAM for WebSphere is directly related to the transaction rate. The largest processor usage is for the following components:

� Publish server: to process transaction data � Database engine: for interface to the database� Archive agent: to perform SQL calls� WebSphere Application Server: to process user requests

Database sizeThe typical database size requirement depends on:

� The number of application server statistics� The transaction volume to be stored � The complexity of transaction� The duration to keep the data

Database table information that increases in size during ITCAM for WebSphere execution is:

� requests: number of requests x 353 bytes

� methods: number of methods x # requests in L3 x 172 bytes

� pmidata: number of data collectors x (3600/polling interval) x 73 bytes

� serverstats: number of data collectors x (3600/polling interval) x 107 bytes

� volumestats: number of data collectors x (3600/polling interval) x 74 bytes

� memorydata: number of data collectors x (3600/polling interval) x 115 bytes

� gcdata: number of data collectors x (3600/garbage collection interval) x104 bytes


2.3.3 Data collector overhead

Monitoring with ITCAM for WebSphere has overhead related to data collectors running on a production WebSphere Application Server. The overhead is minimal for data collectors running on level 1 monitoring. This is typically around a 2–3% increase of CPU time with no notable memory or disk input/output (I/O) requirement.

When the monitoring level is increased, the processing overhead of ITCAM for WebSphere data collectors also increases. This increase is due to the fact that ITCAM for WebSphere collects more data from more sources. A typical level 2 monitoring generates around a 10% increase in processing usage, while a level 3 monitoring generates around 25–30% overhead.

This means that level 2 or level 3 monitoring must be used sparingly in your production environment. To change the monitoring level for purposes of problem determination, schedule it to start and then step back to level 1 automatically in order to reduce the impact on users.

2.4 Implementation options for ITCAM for WebSphere

Depending on the size of your implementation, there are some considerations for implementing ITCAM for WebSphere. This section discusses the following topics:

� 2.4.1, “Designing the managing server” on page 25� 2.4.2, “Deploying a large number of data collectors” on page 28

2.4.1 Designing the managing server

The ITCAM for WebSphere managing server consists of the following products:

� IBM DB2 Universal Database™ Enterprise Server or Oracle database server� WebSphere Application Server� ITCAM for WebSphere managing server


Figure 2-2 shows the conceptual relationship between the components.

Figure 2-2 ITCAM for WebSphere components

The following ITCAM for WebSphere components are displayed in Figure 2-2:

� Kernels

These control the managing server. There are always two copies of kernels running on an ITCAM for WebSphere managing server for redundancy and failover. The kernels register components as they join the managing server, periodically renew connections and registrations with components and data collectors, and collect server and component availability information.

� Publish servers

These receive application and system event data from the data collectors, gather and compute request-level information about performance metrics such as response times, and implement the trap monitoring and alerts features.

� Archive agents

These receive monitoring data from the publish servers and store the monitoring data in ITCAM for WebSphere’s repository.

� Global publishing server

This collects information from the publish servers and correlates all parts and pieces of multi-server requests, such as requests from J2EE servers to execute Customer Information Control System (CICS) or Information Management System (IMS) programs.

Kernel (KL)Provide services on:

- Lookup- Registration- Recovery- Configuration

Publish Server (PS)

Archive Agent (AA)

Global Publish Server (SAM)

Polling Agent (PA)

Message Dispatcher (MD)

Visualization EngineProvide services on:-Administration-Availability-Problem Determination-Performance Management

OCTIGATEdatabase

Publ

ish

traffi

c

Sna

psho

t tra

ffic


� Message dispatcher

This is a conduit for messages from ITCAM for WebSphere using e-mail and Simple Network Management Protocol (SNMP) facilities.

� Polling agent

This collects data from Web servers for Apache 2.0 and later versions.

� Visualization engine

This is a Web-based graphical user interface (GUI) with access to graphics, ITCAM for WebSphere performance reports, real-time views of different slices of monitoring data, ITCAM for WebSphere internal commands, and event-driven functions. The visualization engine runs on a J2EE server such as WebSphere Application Server.

Although ITCAM for WebSphere provides the facility to install all the components in a single wizard, which is called embedded installation, individually installing each component allows more flexibility in terms of verifying each component and configuring them to suit your requirements. The considerations that you must keep in mind when installing the components are:

� Database

You can install the database locally on the managing server or on a separate database server. ITCAM for WebSphere provides database configuration scripts to assist with the configuration of a remote database.

Utilizing a remote database, regardless of whether it is a DB2 Universal Database or an Oracle database, relieves the processing load on the managing server. An environment with hundreds of data collectors generates a large amount of data flowing into the database. This amount increases considerably if the data collectors are set to run monitoring at level 2 or level 3, even for a short period of time.

A remote database allows database query processing and recording to be processed using dedicated hardware, instead of sharing with the main managing server that is already busy with processing the transaction information.

� WebSphere Application Server

The visualization engine of the managing server acts as the administration console for ITCAM for WebSphere. The visualization engine is deployed on a WebSphere Application Server JVM that resides in a standalone application server or an application server that is a part of a network deployment environment.

We recommend that you install the visualization engine on a separate application server JVM that is not monitored by ITCAM for WebSphere data collectors, especially in a network deployment environment. This reduces any


possible conflicts that may arise with respect to ITCAM for WebSphere. In addition, if an issue does arise, problem determination will be somewhat easier due to the separation.

� ITCAM for WebSphere components

Configure the managing server to handle large amounts of data by adding additional components, such as the publish servers and the archive agents. When adding the publish servers and the archive agents, the distribution of data is handled by the managing server. The amount of data being written to the database is handled more efficiently as well.

Another major consideration for the managing server is the split server installation. This option provides the managing server with the overseer processes that exist on separate machines, including the kernel, which provides load balancing and failover capabilities.

There are benefits to this type of configuration when there are hundreds of data collectors providing data to the managing server. This type of setup not only allows the managing server to handle more memory and disk space usage, but also provides a failover capability. For more information about split server installation, refer to 3.1, “Installing ITCAM for WebSphere managing server” on page 34.

2.4.2 Deploying a large number of data collectors

Installation of a small amount of ITCAM for WebSphere data collectors is performed by using the graphical-based installation and configuration wizard provided by the product. When presented with the task of deploying hundreds of data collectors into an environment, the graphical interface is no longer a good option. This non-interactive automated installation method is commonly known as silent installation.

The use of silent installation provides a means to deploy a larger number of data collectors in a more efficient manner. When performing the silent installation, information about the WebSphere environment must be known ahead of time. A response file will be used during the installation, and if incorrect information is used, may result in a failed install.

When performing the silent installation of data collectors, the WebSphere Application Server version must be taken into account, as V6 introduced the usage of profiles. The response files for silent installation are different for various versions of the WebSphere Application Server. In some cases, when two versions of WebSphere Application Server are present, it is better to have two separate master response files.


Installing the Tivoli Enterprise Monitoring Agent (TEMA) is also an option of the silent installation. Although Tivoli Enterprise Monitoring Agent can be installed using silent installation, more configuration must be performed to connect to IBM Tivoli Monitoring V6.1 Tivoli Enterprise Monitoring Server.

Mass automated installation is also possible by using a software distribution or provisioning solution such as the IBM Tivoli Configuration Manager.

There is an additional consideration for deploying data collectors on a machine that has multiple application servers installed. Consider installing a separate data collector directory set for each application server, because applying a fix pack for data collectors requires you to stop the application server. You have a more flexible scheduling option with separate data collector installation for each application server.

2.5 Communication and security considerations

Communication and security issues are vital to the inter-networked world that we live in. Applications and their management infrastructure must be secured in order to protect resources from unauthorized sources. This section discusses the following planning considerations:

� 2.5.1, “Communication security” on page 29� 2.5.2, “Firewall and port consideration” on page 30

2.5.1 Communication security

Communication security relates to the confidentiality of the information transmitted over a network. Management information that is used by IBM Tivoli Composite Application Manager products may contain details about application processing internals. This requires the content of the management information to be secured from being accessed by unauthorized sources.

WebSphere securityWebSphere security plays a significant role in a large-scale implementation. In some cases, WebSphere security is not enabled during the test phase of an implementation, but in a production environment. This requires certain additional considerations. The WebSphere user must have the appropriate permissions to, for instance, issue a wsadmin command.

The configuration of data collectors involves the use of Java Command Language (JACL) scripts, and can fail when there is a permission problem.


If any of the application servers on which the data collector is installed has WebSphere security enabled on it, the entire ITCAM for WebSphere environment must have it enabled as well. This includes WebSphere security being enabled on the ITCAM for WebSphere managing server.

Secure Sockets Layer communicationSecure communication between the managing server and the data collector is a viable option if there is a requirement for data to be encrypted during transmission. Using Secure Sockets Layer (SSL) provides secure data transmission from the data collector to the managing server and must appease corporate security requirements, if necessary. Additional configuration must take place on the managing server and the data collector when enabling SSL. A certificate key generator is included with the product. This key generator provides the facility to use custom-generated keys.

A best practice is to complete the default installation of the managing server and the data collector and then enable SSL for both. This isolates problems (that is, whether the problem is caused by the basic installation or the SSL configuration).

2.5.2 Firewall and port consideration

Firewall and port issues arise when the data collectors are on a different site, location, or subnet from the managing server. Problems such as name resolution occur if the Domain Name System (DNS) is not set up correctly on either the managing server or the data collectors. Routing problems occur if the Internet Protocol (IP) addresses used belong to different subnets. The entire network environment must be looked into in order to determine where a firewall, router, or bridge may exist.


Figure 2-3 shows the communication port requirement for ITCAM for WebSphere.

Figure 2-3 Communication port requirements

The managing server requires open ports for each kernel and publish server. The data collector requires open ports for the command agent and the event agent. The port consolidator requires a port to communicate to the managing server. Use a single port consolidator to consolidate communication from multiple data collectors.

A port consolidator is useful to limit the number of ports required for communication between the data collector and the managing server. Port consolidation is a viable option if there is a limit to the number of ports that can be opened on the firewall. Additional configuration must be carried out on the data collector, including the configuration of the data collector to go through the port consolidator, and starting the port consolidator process.

PortConsolidator

DCCommand Agent

DCEvent Agent

KL1

KL2

PS1

PS2

DCCommand Agent

DCEvent Agent

DCCommand Agent

DCEvent Agent


2.6 Reliability and high availability

This section discusses reliability issues that relate to failover and disaster recovery.

2.6.1 Failover and fault tolerance

Split server configuration for ITCAM for WebSphere or the clustering server for ITCAM for Response Time Tracking consists of having two or more management servers running on separate physical machines. Hardware or software errors do occur on a machine and cause the server to cease functioning. Using the separate server configuration, the secondary server can handle the entire load until the failing machine is recovered.

The switchover to the secondary managing server is not automatic. Manual intervention must take place for the failover to be successful. There are specific ITCAM for WebSphere components that can only be run on one managing server. They must therefore be started on a secondary server, such as the global publish server or the message dispatcher, if the primary server goes down.

2.6.2 Disaster recovery

There are three areas where a backup is necessary for disaster recovery with respect to the IBM Tivoli Composite Application Manager server:

� Database

Perform database backup regularly to collect the most up-to-date information. Use the database utility function to perform the backup function.

� WebSphere Application Server configuration for the server and agent or data collectors

Perform a backup for these for them to be restored in a disaster recovery scenario.

� IBM Tivoli Composite Application Manager servers

These servers must be physically considered for recovery in the disaster recovery site.


Chapter 3. Installing ITCAM for WebSphere

This chapter discusses the installation and deployment procedure for ITCAM for WebSphere. This chapter provides information about the following topics:

� 3.1, “Installing ITCAM for WebSphere managing server” on page 34� 3.2, “Deploying data collectors” on page 50� 3.3, “Configuring and setting up SSL communication” on page 64

3


3.1 Installing ITCAM for WebSphere managing server

This section discusses some of the considerations for installing the ITCAM for WebSphere managing server and provides information pertaining to the following topics:

� 3.1.1, “Installation configuration” on page 34� 3.1.2, “Database installation and configuration” on page 35� 3.1.3, “WebSphere Application Server considerations” on page 41� 3.1.4, “Configuring the split server of the managing server” on page 42� 3.1.5, “Installation and setup of split server” on page 43� 3.1.7, “Adding additional publish servers and archive agents” on page 48

3.1.1 Installation configuration

The installation configuration for IBM Tivoli Composite Application Manager for WebSphere managing server in a large-scale environment is shown in Figure 3-1.

Figure 3-1 Installation configuration for ITCAM for WebSphere managing server

Depending on the scale of your implementation, choose to implement one or more of the following structures:

� Remote database, so that the database processing overhead does not hinder the processing of the managing server, although the database server must be connected through a local network to the managing server to minimize the networking overhead for storing the data over the network.

Managing Server machines Database Server

OCTIGATE


� The managing server split-server configuration facilitates the load to be spread into several machines, besides allowing fault tolerance of the managing server components.

The configuration that is discussed here covers the database, application server, and the managing server overseer components.

3.1.2 Database installation and configuration

As mentioned in 2.4, “Implementation options for ITCAM for WebSphere” on page 25, utilizing a remote database for the managing server helps reduce memory consumption and processing load in a large-scale implementation. This section discusses how to deploy the database on a remote server using IBM Database 2™ (DB2) Universal Database (UDB), Enterprise Server Edition.

Instructions about how to set up and configure a remote database are available in Chapter 13 of IBM Tivoli Composite Application Manager for WebSphere Installation and Customization Guide, GC32-9506.

Implementing ITCAM for WebSphere with a remote database involves the following steps:

1. Installing the DB2 UDB, Enterprise Server Edition on the database server machine.

2. Installing DB2 Client Application Enabler on the managing server machines and visualization engine machines. These clients are necessary for communicating with the remote database server.

3. Cataloging database information about the DB2 clients.

4. Defining the OCTIGATE database structure on the remote database server machine.

The software installation is not discussed here, as this must be performed in accordance with the database documentation.

DB2 Universal Database server considerationsThe DB2 UDB installation includes defining the distribution files and file systems for the database itself. In a large environment where performance is critical, follow these recommendations:

� Store the database files in a separate physical disk rather than the operating system paging and DB2 UDB system files. This allows you to minimize disk contention for accessing the database files.

Chapter 3. Installing ITCAM for WebSphere 35

� Allocate adequate memory for DB2 UDB application pools to ensure performance. The most important pool is the DB2 bufferpool, which is used to buffer the database data for quick and efficient access.

� Use a faster disk for database storage, possibly separating the file systems for recovery log and data.

Database installation scriptsITCAM for WebSphere provides remote database creation scripts in the tar file supplied in the managing installation image:

<ms-installation-image-directory>/base/scripts/db2-remote-scripts.tar

The scripts in the tar file are aimed at a UNIX-based environment installation or a Windows-based environment with Microsoft Windows Services for UNIX (SFU) installed.

On untarring the db2-remote-scripts.tar file, you will be presented with various installation scripts in the bin directory that is created. The primary executable is db2install. Depending on the platform, different syntax exist for executing the db2install command. The following example uses amuser as the owner ID and admin as the default user from the visualization engine.

� For Windows, the syntax is:

db2install.bat jdbc_user jdbc_password adminVEuser instance_user instance_password

For example:

db2install.bat amuser ampasswd admin db2inst1 dbpasswd

� For UNIX, the syntax is:

db2install.sh db2username db_user absolute_path_of_install_directory adminVEUser

For example:

./db2install.sh db2inst1 amuser /tmp/db2/ admin

The script uses the su command several times to switch the user as the DB2 instance owner. We recommend that you log in as the root user, so that you are not prompted for the DB2 user password multiple times. If you log in as the root user, add the root user to the DB2 administrator group, typically called db2grp1.

You must also define the installation owner user in the operating system of the database machine. This user is typically called amuser and requires administrative access.


Cataloging the database on the DB2 Universal Database clientThe DB2 client must be able to connect to the actual DB2 OCTIGATE database. This connection is defined using the following definitions:

� Node definition

This indicates where the DB2 UDB server resides.

� Database definition

This indicates the database selection in the node.

Start the DB2 UDB command-line interface using the db2cmd command in a Windows-based platform or by sourcing the db2profile on a UNIX-based platform.

To catalog the node, issue the following command:

db2 catalog tcpip node <ipnode> remote <ipaddr> server <ipport>

Following is a description of the parameters in this command:

� ipnode: the database node name.

� ipaddr: the TCP/IP host name or address the client addresses the server by.

� ipport: the port number on which the DB2 server listens. The default port number is 50000.

To catalog the database, issue the following command:

db2 catalog database OCTIGATE at node <ipnode>

In this command, ipnode is the database node name as defined by the catalog tcpip node command.


The actual OCTIGATE database must exist to connect to the database. The connection is performed using the db2 connect command to OCTIGATE. Figure 3-2 shows a successful connection.

Figure 3-2 Successful connection to OCTIGATE

We recommend that you create the database first and then catalog it. This is to help you to test the connection to the database from the DB2 client immediately. Although you can catalog the database without testing the connection, testing it ensures connectivity.

Changing database name from OCTIGATE

To change the database name OCTIGATE to another name, there are some modifications required on:

� Installation and configuration scripts� Runtime database connection

Example 3-1 illustrates the changes required in db2configuration.sh to modify the database name from OCTIGATE to CAMFWAS.

Example 3-1 Modification to db2configuration.sh

# Drop the octigate database firstecho "$DBUSER> db2 drop database camfwas"db2 drop database camfwas#Fix SQL1005N problem.#One of the causes is that octigate database may be created by other DB2 instance#so it still exists in system catalog.echo "$DBUSER> db2 catalog database camfwas"

# . /home/db2inst1/db2profile# db2 connect to OCTIGATE user db2inst1 using db2passw

Database Connection Information

Database server = DB2/AIX 8.2.0 SQL authorization ID = DB2INST1 Local database alias = OCTIGATE

Note: Although we do not recommend changing the database name of the ITCAM for WebSphere managing server from OCTIGATE, some installations may have strict naming conventions for the database name.


db2 catalog database camfwasecho "$DBUSER> db2 drop database camfwas"db2 drop database camfwasecho "$DBUSER> db2 uncatalog database camfwas"db2 uncatalog database camfwas# Create the octigate databaseecho "$DBUSER> db2 create database camfwas using codeset UTF-8 territory US"db2 create database camfwas using codeset UTF-8 territory USstatus="$?"echo "$DBUSER> db2empfa camfwas"db2empfa camfwas# Configure DB2 and the Octigate Databaseecho "$DBUSER> db2 -tf $2"db2 -tf $2# echo "Change IBMDEFAULTBP size to -1 so BUFFPAGE setting will be used"db2 connect to camfwas...

Example 3-2 illustrates the changes required in db2createschema.sh to modify the database name to CAMFWAS.

Example 3-2 Modification to db2createschema.sh

# Connect to DB2echo "$DBUSER> db2 connect to camfwas"echo "$DBUSER> db2 -tf $DBSCRIPT"su - $DBUSER -c "db2 connect to camfwas; db2 -tf $DBSCRIPT; db2 terminate"#Change the database table to update the user "admin"echo "$DBUSER> db2 connect to camfwas"status="$?"echo "$DBUSER> db2 update users set username = '$ADMINUSER', ext_user = '$ADMINUSER' where username = 'admin' and ext_user = 'admin'"su - $DBUSER -c "db2 connect to camfwas; db2 \"update users set username = '$ADMINUSER', ext_user = '$ADMINUSER' where username = 'admin' and ext_user = 'admin'\" ; db2 terminate"

Example 3-3 illustrates a successful database connection and creation. Start the DB2 UDB command-line interface using the db2cmd command. Then connect to the database as the schema user (for example, amuser) and list the table.

Example 3-3 Checking database

db2 => connect to camfwas user amuser using ampasswdDatabase Connection InformationDatabase server = DB2/LINUX 8.2.4SQL authorization ID = AMUSERLocal database alias = CAMFWASdb2 => list tables


Table/View Schema Type Creation time------------------------------- --------------- ----- --------------CONFIG AMUSER T 2007-06-07-11.25.24.284983CTG_SR_GATEWAY AMUSER T 2007-06-07-11.25.25.150959...2007-06-07-11.25.22.927237VOLUMESTAT AMUSER T 2007-06-07-11.25.23.838550WBI_BO_VIEW AMUSER V 2007-06-07-11.25.25.555333WBI_REQUEST_OVERVIEW AMUSER T 2007-06-07-11.25.25.435186WEBSERVERCHARTDATA AMUSER T 2007-06-07-11.25.24.772717

65 record(s) selected.

Two further modifications are required in the managing server to reference the database:

� $ITCAM_HOME/bin/setenv.sh

To change the database name from OCTIGATE to CAMFWAS, change the definition of JDBC_DRIVER_URL to point to CAMFWAS. For example:

JDBC_DRIVER_URL=jdbc:db2://lima.itsc.austin.ibm.com:50000/camfwas

� Set the JDBC properties in the WebSphere Application Server where visualization engine is running:

a. Log on to the WebSphere Application Server administrative console.

b. Go to Resources → JDBC → Data Sources.

c. Select ITCAMDataSource.


d. Modify the database name in the DB2 Universal Database Property to reflect the name of the ITCAM Database. For example, Figure 3-3 illustrates the change to the database CAMFWAS.

Figure 3-3 Configuration for ITCAMDataSource

3.1.3 WebSphere Application Server considerations

ITCAM for WebSphere requires the WebSphere Application Server V5 or V6 to run the visualization engine. The WebSphere Application Server runs the visualization engine Web application that acts as the ITCAM for WebSphere Web console. The WebSphere Application Server can be a standalone server or a part of a network deployment environment.

Considerations for operating the Web consoleIn case of a visualization engine error, minimize the load of the non-managing server application and ensure access to the WebSphere administration console. This is not a problem for a network deployment environment because the administration console runs independently of the application server. For a


standalone server with WebSphere Application Server V5, do not use the default server1, as it runs the administration console application. The reasons why you must not do this are:

� There will be some overhead from the administration console.

� If ITCAM for WebSphere stops functioning, you are left without an administration console and have to update the eXtensible Markup Language (XML) files manually.

3.1.4 Configuring the split server of the managing server

The split server configuration of the managing server consists of running two kernels on separate physical machines. Implementing the ITCAM for WebSphere managing server in this manner provides failover capabilities for reporting data collectors.

Figure 3-4 shows the split-server configuration in a two-system environment. Note that this figure shows only the managing server components.

Figure 3-4 Split server configuration

Note: Currently, the alerts and traps setting requires the publish server to contact a local message dispatcher process. We recommend that you have a message dispatcher on server1.

Managing server - 1 Managing server - 2

Kernel 1

Publish Server 1Publish Server 2

Archive Agent 1Archive Agent 2

Visualization EngineMessage Dispatcher

Global Publish Server

Kernel 2

Publish Server 3Publish Server 4

Archive Agent 3Archive Agent 4


As Figure 3-4 on page 42 shows, there are two managing servers:

� Managing server 1

This acts as the main server. In a split server configuration for ITCAM for WebSphere, the following components can reside only on one of the managing servers, particular the primary server:

– Visualization engine– Message dispatcher– Global publish server

� Managing server 2

This acts as a backup server and offloads the processing of the primary server.

3.1.5 Installation and setup of split server

Both of the managing servers have the following requirements:

� For a Windows-based server, install Microsoft Services for UNIX.

� Access to the OCTIGATE database, either locally or using the database client software. Our example sets the DB2 client similarly for both of the managing server machines.

� Java environment, which is typically acquired from the WebSphere Application Server installation on the machine. Even though you do not install WebSphere Application Server, you still have to adjust JAVA_HOME to a suitable Java Runtime Environment (JRE™). The WebSphere’s j2ee.jar must also be present. Copy $WAS_HOME/java and $WAS_HOME/lib/j2ee.jar from the primary managing server to the secondary server.

For ease of maintenance, we recommend that both servers run the same level of software. One approach is to tar the primary managing server directory and restore it on the secondary managing server. Alternatively, you can also install the managing server software on both the machines.


Perform the following changes to the configuration files on one or both the servers:

� $ITCAM_HOME/bin/setenv.sh

This file determines the configuration information for the managing server.

a. To allow a split server, change the definition for KERNEL_HOST01 and KERNEL_HOST02 to point to each managing server, assuming that you use the same port definitions for both the servers. Each kernel requires three ports for operation. Perform this activity on both the servers, as shown in Example 3-4.

Example 3-4 Modification of setenv.sh

KERNEL_HOST01=srv152.itsc.austin.ibm.comPORT_KERNEL_CODEBASE01=9122PORT_KERNEL_RFS01=9120PORT_KERNEL_RMI01=9178KERNEL_HOST02=srv153.itsc.austin.ibm.comPORT_KERNEL_CODEBASE02=9123PORT_KERNEL_RFS02=9121PORT_KERNEL_RMI02=9119

b. Edit the $ITCAM/etc/ms.properties file and set the kernel.hosts property, as shown in Example 3-5.

Example 3-5 Example of ms.properties

kernel.hosts=srv152.itsc.austin.ibm.com:9122:9120:9178:9178,srv153.itsc.austin.ibm.com:9123:9121:9119

c. Adjust the setting for JAVA_HOME accordingly. This must refer to the same Java Virtual Machine (JVM) environment that is similar to the WebSphere Application Server. To check the Java version, use the java -version command to make sure that you are using the same JVM as your WebSphere Application Server.


� $ITCAM_HOME/etc/ve.properties

This is the configuration for the visualization engine. Reflect the correct kernel codebase and rfs addresses that are stored in the value of kernel.codebase and kernel.rfs.address, as shown in Example 3-6. Perform this activity for both the managing servers.

Example 3-6 Modification of ve.properties

kernel.codebase=http://srv152.itsc.austin.ibm.com:9122/kernel.core.jar

http://srv153.itsc.austin.ibm.com:9123/kernel.core.jarkernel.rfs.address=srv152.itsc.austin.ibm.com:9120

srv153.itsc.austin.ibm.com:9121

� For all the components that have been started in Managing server 2, edit the corresponding control files for these components. Make sure that the entries pointing to kernel codebase and ports like RMI and RFS are pointing to the correct location. We edited the aactl.sh and psctl.sh files. These were the components running on managing server 2, as shown in Figure 3-4 on page 42.

Example 3-7 Modification of aactl.sh

CODEBASE_ADDRESS02=$KERNEL_HOST02:$PORT_KERNEL_CODEBASE02KERNEL_CODEBASE=”http://$CODEBASE_ADDRESS02/kernel.core.jar”RFS_ADDRESS=”KERNEL_HOST02:$PORT_KERNEL_RFS02”RMI_CODEBASE=”http://$CODEBASE_ADDRESS02/archiveagent-intf.jar http://$CODEBASE_ADDRESS02/archiveagent.jar”


� $ITCAM_HOME/bin/am-start.sh

This is the startup script for ITCAM for WebSphere. Modify this script to indicate which components must be started on which machine. Refer to Figure 3-4 on page 42 to comment out the appropriate component. The secondary managing server must suppress the kernel 1 and kernel 1 watch dog. The primary managing server must suppress the kernel 2 and kernel 2 watch dog. Similarly, you have to add appropriate commands to start the individual components on each server. Our sample am-start.sh for managing server 1 is shown in Example 3-8. You must also shorten the wait time for the watch dog, as there is only one kernel to start in each machine.

Example 3-8 Sample content of am-start.sh for managing server 1

${CYANEA_HOME}/bin/amctl.sh wd1 start${CYANEA_HOME}/bin/amctl.sh aa1 start${CYANEA_HOME}/bin/amctl.sh aa2 start${CYANEA_HOME}/bin/amctl.sh ps1 start${CYANEA_HOME}/bin/amctl.sh ps2 start${CYANEA_HOME}/bin/amctl.sh md start${CYANEA_HOME}/bin/amctl.sh sam1 start

� $ITCAM_HOME/bin/am-stop.sh

Perform the similar modification in the am-stop.sh script as well, in order to eliminate errors when stopping the components.

Note: For am-start.sh and am-stop.sh, start the primary managing server first. Use an automation mechanism to coordinate this process. One such automation mechanism is the use of a remote execution shell.


3.1.6 Verifying the installed components in a split environment

Start ITCAM for WebSphere on both the managing servers using the am-start.sh command. Once all the components are started on both the machines, verify the split server configuration using the Web console. Select Administration → Managing server → Self Diagnosis. In the component window, the running components and their Internet Protocol (IP) addresses are displayed, as shown in Figure 3-5.

Figure 3-5 Self-diagnostic window that shows the managing server components

Note: The polling agent has been removed from this version of ITCAM for WebSphere. You cannot add or configure the polling agent, even though an empty node for the polling agent is displayed in the application monitor component tree in the Web console.


3.1.7 Adding additional publish servers and archive agents

The basic configuration has only two publish servers and two archive agents. To further distribute the load, add publish servers or archive agents. Typically, in a split-server environment (shown in Figure 3-4 on page 42), two publish servers and two archive agents per machine are defined.

The instructions for adding additional publish servers and archive agents are derived from IBM Tivoli Composite Application Manager for WebSphere Installation and Customization Guide, GC32-9506. Defining new components includes the following modifications:

1. Run the following command from the $ITCAM_HOME/bin/ folder:

./add-ps.sh publish_server_port_number

This adds a new publish server. The output of the command is shown in Example 3-9

Example 3-9 Output of executing add-ps.sh

[root@srv153 bin]# ./add-ps.sh 9105Current number of ps = 2New number of ps = 3Creating a new /opt/IBM/itcam/WebSphere/MS/etc/ps3.properties, using ps1.properties as a templatetesting UUID=f0039d70-0f1b-dc01-c4dc-000c293bdac6Use UUID=f0039d70-0f1b-dc01-c4dc-000c293bdac6Creating a new /opt/IBM/itcam/WebSphere/MS/etc/log-ps3.properties by copying log-ps1.propertiesDone modifying setenv.shDone modifying am-start.shDone modifying am-stop.shDone modifying am-forcestop.sh

2. Run the following command from the $ITCAM_HOME/bin/ folder:

./add-aa.sh archive_agent_port_number

Restriction: You can add only publish servers and archive agents. There are only two instances of kernels and a single instance of global publish server, message dispatcher.


This adds a new archive agent. The output of the command is shown in Example 3-10.

Example 3-10 Output of executing add-aa.sh

[root@srv153 bin]# ./add-as.sh 9031Current number of AA = 2New number of AA = 3Creating a new /opt/IBM/itcam/WebSphere/MS/etc/aa3.properties, using aa1.properties as a templatetesting UUID=d078edc1-0e1b-dc01-2da7-000c293bdac6Use UUID=d078edc1-0e1b-dc01-2da7-000c293bdac6Creating a new /opt/IBM/itcam/WebSphere/MS/etc/log-aa3.properties by copying log-aa1.propertiesDone modifying setenv.shDone modifying am-start.shDone modifying am-stop.shDone modifying am-forcestop.sh

3. Modify the am-start.sh and am-stop.sh scripts to add the new components for both servers:

a. For am-start.sh, if you have added two more archive agent aa3 and aa4, respectively, and published server ps3 and ps4, the following entries must be made (Example 3-11).

Example 3-11 Addition to am-start.sh

MS_home/bin/amctl.sh ps3 start MS_home/bin/amctl.sh ps4 start MS_home/bin/amctl.sh aa3 start MS_home/bin/amctl.sh aa4 start

b. For am-stop.sh, use the commands shown in Example 3-12.

Example 3-12 Addition to am-stop.sh

MS_home/bin/amctl.sh ps3 stop MS_home/bin/amctl.sh ps4 stop MS_home/bin/amctl.sh aa3 stop MS_home/bin/amctl.sh aa4 stop


3.2 Deploying data collectors

This section lists the configuration options for installing the data collector. You modify the configuration options either in the response file or at the command line to run the silent installation.

You have the option to only install the data collector, both install and configure the data collector, or only configure the data collector using this procedure. If you only want to configure the data collector, perform this the configuration after the data collector has been installed.

Data collector deployment for a large installation requires a custom-automated silent installation and configuration of ITCAM for WebSphere data collectors on the target application servers.

This section discusses the following topics:

� 3.2.1, “Setting up the silent installation process” on page 50� 3.2.2, “Installing the data collector” on page 53� 3.2.3, “Installing and configuring the data collector together” on page 54� 3.2.4, “Configuring data collectors” on page 56� 3.2.5, “Automatically discovering the installation parameters” on page 61

3.2.1 Setting up the silent installation process

The silent installation process is supported by the install-DC command, with the following syntax:

install-DC -silent -options inputfile

The input file can either be modified from a supplied sample or generated using the installation wizard. In Version 6.1, the installation wizard and the configuration tool wizard both have the ability to generate the silent installation option file. Table 3-1 lists the available sample option files for the wizard. These options files are supplied in the silent sub-directory of the installation image.

Table 3-1 Options files list

Application server type Sample option file name

WebSphere Application Server V6 DC6.opt

WebSphere Application Server V5 DC.opt

WebSphere Application Server DC_was.opt

WebSphere Portal Server DC_portal.opt


The install settings in the option files are commented using leading number characters (###). If you enable an option, you must supply a value. A blank or a null value is not acceptable. The silent installation option requires advance knowledge of the target machine, including the target WebSphere environment. A prefilled response file may not be efficient in a large environment. An external mechanism is required to collect the information and populate the silent installation control files beforehand. Some of this information requires the use of naming conventions. See 3.2.5, “Automatically discovering the installation parameters” on page 61.

Table 3-2 lists the parameters for installation.

Table 3-2 Installation parameters

WebSphere Process Server DC_ps.opt

WebSphere Enterprise Service Bus DC_esb.opt DC_esb.opt

Workplace™ Collaboration Service Mail Server DC_wcsmail.opt

Option for upgrading WebSphere V5 or V6 data collector V6.0 into V6.1

migrate_DC.opt

Application server type Sample option file name

Option string Description

-V disableOSPrereqChecking=true | false Whether to check the operating system level before the installation.

-V LICENSE_REJECT_BUTTON=true | false Must be false.

-P installLocation=value Specifies the path of the installation directory for the data collector.

-V LOG_DIR=value This specifies the writable directory to which the installation and configuration programs will write log files.

-W LogSetting.consoleOut=true | false Specifies whether to display messages issued by the installation and configuration program on the console.

-V DC_WD_JAVAHOME=value Path of the Java home directory.

-V DC_CC_TEMA=true | false Whether to install Tivoli Enterprise Monitoring Agent collector interface.

-V LAUNCH_CONFIG=true | false Specifies whether to launch or defer the data collector Configuration Tool after installation.

-V DEFER_CONFIG=true | false


Table 3-3 lists the parameters for configuration.

Table 3-3 Configuration parameters

-V GC_LOG_PATH=value If Tivoli Enterprise Monitoring Agent is true, specifies the path for the garbage collector log.

-W LogSetting.logLevel=level Level: FATAL, ERROR, WARNING, INFO, DEBUG_MIN, DEBUG_MID, DEBUG_MAX, ALL.

Option String Description

-V LOG_DIR=value Writable directory to which the installation and configuration programs write log files.

-V LAUNCH_CONFIG=true | false Specifies whether to launch or defer the data collector configuration tool after installation.

-V DEFER_CONFIG=true | false

-V DC_CCUC_CONFIG=true | false Specifies whether to configure or unconfigure a data collector.

-V DC_CCUC_UNCONFIG=true | false

-V DC_RECONFIG_ALLOW=true | false Indicates whether reconfiguring the data collector is allowed.

-V DC_MSKS_SERVERNAME=value Fully qualified host name of the managing server.

-V DC_MSKS_CODEBASEPORT=value Remote file system port for the managing server.

-V DC_MS_AMHOME=value Installation directory of the managing server.

-V DC_CC_ITCAMFWAS=true | false Whether the application monitor interface support will be installed.

-V DC_CAS_WAS=true | false Specifies which type of data collector to configure, WebSphere Application Server, Enterprise Service Bus, Process Server, or WebSphere Portal Server.-V DC_CAS_ESB=true | false

-V DC_CAS_PS=true | false

-V DC_CAS_WPS=true |false

-V APP_SERVER_NAMES=value Path to the application server.

-V WS_NODE_NAME=value Application server node name.

-V DC_WD_JAVAHOME=value Location of the Java home directory.

-V DC_WD_PROFILENAME=value Applies only to Version 6 application servers.

-V DC_WD_WASBASEDIR=value Location of the application server base directory.

-V DC_WD_WASVER=value Version of the application server.


3.2.2 Installing the data collector

A sample option file for our WebSphere Application Server V6.1 is shown in Example 3-13. This installation is on the Linux-based server.

Example 3-13 Sample option file for WebSphere Application Server V6.1

# Log Parameters-V LOG_DIR="/var/ibm/tivoli/common"-V disableOSPrereqChecking="true"-W LogSetting.logLevel="ALL"-W LogSetting.consoleOut="false"-P installLocation="/opt/IBM/itcam/WebSphere/DC"-V DC_WD_JAVAHOME=/opt/IBM/WebSphere/AppServer/java-V LICENSE_ACCEPT_BUTTON="true"-V LICENSE_REJECT_BUTTON="false"-V LAUNCH_CONFIG="false"-V DEFER_CONFIG="true"-V DC_CC_TEMA="true"-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/gc.log"

-V DC_ASL_HOSTNAME=value Administrative application host name where the data collector will be installed.

-V DC_ASL_SOAPPORT=value SOAP interface port for the administrative application.

-V WAS_BASEDIR=value Only for ESB, portal server, and process server, indicates path and version of the base WebSphere Application Server.-V WAS_BASE_VERSION=value

-V DC_CC_TEMA=true | false Whether to run secondary collector for Tivoli Enterprise Monitoring Agent.

-V GC_LOG_PATH=value If Tivoli Enterprise Monitoring Agent is true, specifies path for the Garbage Collector log.

-V TEMA_OFFLINE_ALLOW=true | false Allows Tivoli Enterprise Monitoring Agent to be off line.

-V FIREWALL_ENABLED=true | false Whether the data collector machine is behind a firewall.

-V PROBE_RMI_PORT=value If the firewall is enabled, define RMI port for probe agent.

-V PROBE_CONTROLLER_RMI_PORT=value If the firewall is enabled, define RMI port for controller.

-W LogSetting.logLevel=level Level: FATAL, ERROR, WARNING, INFO, DEBUG_MIN, DEBUG_MID, DEBUG_MAX, ALL.


Run the following silent installation command to silently install the data collector:

./setup_DC_<platform>.bin -silent -options <PATH>/DC6.opt

The option file in Example 3-13 on page 53 does not launch the configuration tool. You can check the installation result in the /var/ibm/tivoli/common/CYN/logs/trace-install.log file. When the installation is successful, the end of trace-install.log contains the message in Example 3-14.

Example 3-14 Sample trace-install.log for a successful data collector installation

<LogText><![CDATA[Exit, return value = The InstallShield Wizard has successfully installed ITCAM for WebSphere Data Collector 6.1.Choose Finish to exit the wizard.]]></LogText>

3.2.3 Installing and configuring the data collector together

A sample option file for our WebSphere Application Server V6.1 is shown in Example 3-15. This installation is on the Linux-based network deployment server called srv107 with the managing server on srv152. Notice that the option -V LAUNCH_CONFIG="true" since we are both installing and configuring the data collector.

Example 3-15 Sample option file for WebSphere Application Server V6.1

# Install Parameters-V LOG_DIR="/var/ibm/tivoli/common"-V disableOSPrereqChecking="true"-W LogSetting.logLevel="ALL"-P installLocation="/opt/IBM/itcam/WebSphere/DC"-V DC_WD_JAVAHOME=/opt/IBM/WebSphere/AppServer/java-V LICENSE_ACCEPT_BUTTON="true"-V LICENSE_REJECT_BUTTON="false"-V LAUNCH_CONFIG="true"-V DEFER_CONFIG="true"-V DC_CC_TEMA="true"-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/gc.log"

# Configuration Parameters

-V DC_CCUC_CONFIG="true"-V DC_CCUC_UNCONFIG="false"-V DC_RECONFIG_ALLOW="true"-V DC_OFFLINE_ALLOW="false"-V DC_MSKS_SERVERNAME="srv152.itsc.austin.ibm.com"-V DC_MSKS_CODEBASEPORT="9122"


-V DC_MS_AMHOME="/opt/IBM/itcam/WebSphere/MS"-V DC_CC_ITCAMFWAS="true"-V DC_CC_TEMA="true"-V DC_CAS_WAS="true"-V DC_CAS_ESB="false"-V DC_CAS_PS="false"-V DC_CAS_WPS="false"-V DC_CAS_WCSMAIL="false"-V APP_SERVER_NAMES="cells/srv107Node01Cell/nodes/srv107Node02/servers/server1"-V WS_NODE_NAME="cells/srv107Node01Cell/nodes/srv107Node02"-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/ProcSrv01"-V DC_WD_PROFILENAME="ProcSrv01"-V DC_WD_WASBASEDIR="/opt/IBM/WebSphere/AppServer"-V DC_WD_WASVER="60"-V DC_ASL_HOSTNAME="srv107.itsc.austin.ibm.com"-V DC_ASL_SOAPPORT="8880"-V DC_ASL_USERNAME="NULL"-V DC_ASL_PASSWD="NULL"-V WAS_BASEDIR="/opt/IBM/WebSphere/AppServer"### TEMA Parameters##-V DC_CC_TEMA="true"-V GC_LOG_PATH="/var/ibm/tivoli/common/CYN/logs/GC"-V TEMA_OFFLINE_ALLOW="false"### DC Host Parameters##-V AM_SOCKET_BINDIP="srv107.itsc.austin.ibm.com"-V FIREWALL_ENABLED="false"

For additional flexibility and the reuse of response files, you can manually type some configuration options on the command line, instead of including them in the response file. This is useful for providing:

� Unique options: to reuse the response file in multiple installations and configurations, but some of the options are unique to each.

� Password protection: to safeguard the password by manually entering it during each installation and configuration. If you record the password in the option (.opt) file, the password is unencrypted and visible to anyone who opens the file.


The following command sets administrative server passwords on the command line and relies on a response file to provide all other configuration details:

./setup_DC_lin.bin -silent -V DC_ASL_PASSWD="tivoli" -options /<PATH>/DC6.opt

3.2.4 Configuring data collectors

As a machine may have more than one application server, we can now configure additional application servers on that machine. This is typically performed using the config_DC utility that is installed in the ITCAM_HOME/config_DC directory. You may also need to run the configuration if you defer launching the configuration tool, as in 3.2.2, “Installing the data collector” on page 53.

There are two ways of configuring the data collectors without interaction:

� Using the config_dc -silent utility and modifying the dcInpouts.txt

The drawback of this is that all the error messages require a graphical user interface (Windows or X11) to be available and responded upon.

� Using the Java Command Language (JACL) script configDataCollector.jacl

Running config_dcExample 3-16 shows the options file that allows multiple data collector configurations to be performed in a single run.

Example 3-16 Sample properties for configuring two servers in the same node

# Configuration Parameters

-V DC_CCUC_CONFIG="true"-V DC_CCUC_UNCONFIG="false"-V DC_RECONFIG_ALLOW="true"-V DC_OFFLINE_ALLOW="false"-V DC_MSKS_SERVERNAME="srv152.itsc.austin.ibm.com"-V DC_MSKS_CODEBASEPORT="9122"-V DC_MS_AMHOME="/opt/IBM/itcam/WebSphere/MS"-V DC_CC_ITCAMFWAS="true"-V DC_CC_TEMA="false"-V DC_CAS_WAS="true"-V DC_CAS_ESB="false"-V DC_CAS_PS="false"-V DC_CAS_WPS="false"

Note: Configuration options specified in the response file take precedence over those entered in the command line.


-V DC_CAS_WCSMAIL="false"-V APP_SERVER_NAMES="cells/khartoumCell01/nodes/khartoumNode01/servers/ClientSvc"-V WS_NODE_NAME="cells/khartoumCell01/nodes/khartoumNode01"-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/AppSrv01"-V DC_WD_PROFILENAME="AppSrv01"-V DC_WD_WASBASEDIR="/opt/IBM/WebSphere/AppServer"-V DC_WD_WASVER="61"-V DC_ASL_HOSTNAME="khartoum.itsc.austin.ibm.com"-V DC_ASL_SOAPPORT="8879"-V DC_ASL_USERNAME="NULL"-V DC_ASL_PASSWD="NULL"-V WAS_BASEDIR="/opt/IBM/WebSphere/AppServer"### TEMA Parameters##-V DC_CC_TEMA="false"-V GC_LOG_PATH="NULL"-V TEMA_OFFLINE_ALLOW="false"### DC Host Parameters##-V AM_SOCKET_BINDIP="khartoum.itsc.austin.ibm.com"-V FIREWALL_ENABLED="false"### Configuration Parameters##-V DC_CCUC_CONFIG="true"-V DC_CCUC_UNCONFIG="false"-V DC_RECONFIG_ALLOW="true"-V DC_OFFLINE_ALLOW="false"-V DC_MSKS_SERVERNAME="srv152.itsc.austin.ibm.com"-V DC_MSKS_CODEBASEPORT="9122"-V DC_MS_AMHOME="/opt/IBM/itcam/WebSphere/MS"-V DC_CC_ITCAMFWAS="true"-V DC_CC_TEMA="false"-V DC_CAS_WAS="true"-V DC_CAS_ESB="false"-V DC_CAS_PS="false"-V DC_CAS_WPS="false"-V DC_CAS_WCSMAIL="false"-V APP_SERVER_NAMES="cells/khartoumCell01/nodes/khartoumNode01/servers/ServerSvc"


-V WS_NODE_NAME="cells/khartoumCell01/nodes/khartoumNode01"-V DC_WD_PROFILEHOME="/opt/IBM/WebSphere/AppServer/profiles/AppSrv01"-V DC_WD_PROFILENAME="AppSrv01"-V DC_WD_WASBASEDIR="/opt/IBM/WebSphere/AppServer"-V DC_WD_WASVER="NULL"-V DC_ASL_HOSTNAME="khartoum.itsc.austin.ibm.com"-V DC_ASL_SOAPPORT="8879"-V DC_ASL_USERNAME="NULL"-V DC_ASL_PASSWD="NULL"-V WAS_BASEDIR="/opt/IBM/WebSphere/AppServer"### TEMA Parameters##-V DC_CC_TEMA="false"-V GC_LOG_PATH="NULL"-V TEMA_OFFLINE_ALLOW="false"### DC Host Parameters##-V AM_SOCKET_BINDIP="khartoum.itsc.austin.ibm.com"-V FIREWALL_ENABLED="false"

Run the following silent configuration command:

/opt/IBM/itcam/WebSphere/DC/config_dc/config_dc.sh -silent -options itcam_dc_config.opt

The file dcInputs.txt gets created during the install process of the data collector and is later used by the data collecter configuration wizard, along with an options file (.opt).

During the configuration process the dcInputs.txt file (see Example 3-17) gets further updated and is used by the remainder of the configuration process.

Example 3-17 Sample dcInputs.txt file

TEMP_DIR=/tmp/ibm_am_installer_dc/TOOLKITHOME=/opt/IBM/itcam/WebSphere/DC/toolkitJAVA_HOME=/opt/IBM/itcam/WebSphere/DC/_jvm/jreAM_HOME_CONFIG_DC=/opt/IBM/itcam/WebSphere/DC/config_dcOS=linuxFILESEP=/DCHOME=/opt/IBM/itcam/WebSphere/DC/itcamdcITCAM61HOME=/opt/IBM/itcam/WebSphere/DCUNCONFIGURE_SUCCESSFUL=true


DMGR_RMI=TIVOLI_COMMON_DIR=/var/ibm/tivoli/commonOS_VERSION=2.6.9-42.ELsmpCONNECTION=SOAPJDK_VERSION=WS_RMI_PORT=CONFIGURE=trueJDK_TYPE=COBRAND=ITCAMREPLACE_JAVA_SECURITY_POLICY=trueLOG_COMMON_DIR=/var/ibm/tivoli/commonOS_NAME=RedHatEnterpriseLinuxAM_HOME=/opt/IBM/itcam/WebSphere/DC/itcamdcLICENSE=true

Example 3-18 shows a successful silent configuration entry in trace-install.log.

Example 3-18 Sample trace-install.log for a successful data collector configuration

<LogText><![CDATA[DATA: Configure Finish: Successfully configured the application servercells/khartoumCell01/nodes/khartoumNode01/servers/ClientSvccells/khartoumCell01/nodes/khartoumNode01/servers/ServerSvc.]]></LogText>

Running configDataCollector.jaclUse a JACL script called configDataCollector.jacl that exists in the config_DC directory. The input file for this script is found in the temporary directory that is created from the installation.

To install the data collector into a secondary application server in the node laredoNode02 with profile vbdpro01, we use the parameter file shown in Example 3-19.

Example 3-19 Sample properties for configuring data collector

server=server1(cells/laredoNode02Cell/nodes/laredoNode02/servers/server1|server.xml#Server_1144783086875)server.genericJvmArguments=-Xbootclasspath/p:${AM_HOME}/lib/bcm-bootstrap.jar;${AM_HOME}/lib/ppe.probe-bootstrap.jar -Xrunam

Note: The <PATH>/DC/dc_config/dcInputs.txt file must exist in order for the data collector configuration to succeed.


-Djlog.propertyFileDir.CYN=${AM_HOME}/etc -Djlog.qualDir=laredoNode02.server1server.systemproperty0=com.ibm.websphere.classloader.plugin=com.cyanea.bcm.websphere.BcmPluginserver.systemproperty1=java.rmi.server.codebase=file://${MS_AM_HOME}/lib6.0/ppe.probe.jar file://${MS_AM_HOME}/lib/ppe.probe-intf.jar file://${MS_AM_HOME}/lib6.0/ppe.was_6.jar file://${MS_AM_HOME}/lib6.0/ppe.probe-bootstrap.jar server.systemproperty2=appserver.platform=was6010server.systemproperty3=am.appserver=server1server.systemproperty4=java.security.policy=C:\\ITCAM\\DC/etc/datacollector.policyserver.systemproperty5=deploymentmgr.rmi.connection=server.systemproperty6=am.profilename=vbdpro01server.systemproperty7=am.nodename=laredoNode02server.systemproperty8=am.home=C:\\ITCAM\\DCserver.environment0=PATH=C:\\ITCAM\\DC/lib;server.environment1=QUALDIR=laredoNode02.server1 server.environment2=_JVM_THREAD_DUMP_BUFFER_SIZE=25000000server.customservice0.displayName=amserver.customservice0.classpath=server.customservice0.enable=trueserver.customservice0.description=Custom Service for ITCAMfWAS DCserver.customservice0.classname=com.cyanea.ws6.ProbeServiceserver.pmiservice.initialSpecLevel=connectionPoolModule=H:wsgwModule=N:orbPerfModule=H:cacheModule=N:webAppModule=H:threadPoolModule=H:wlmModule=M:webServicesModule=N:beanModule=X:jvmRuntimeModule=H:systemModule=M:servletSessionsModule=H:transactionModule=M:j2cModule=Hserver.am.install.root=C:\\ITCAM\\DCserver.am.ms.home=/C:/IBM/itcam/WebSphere/MS

Apart from the apparent configuration values, the first line indicates the server ID. Retrieve this from the appropriate server.xml file by finding the xmi:id=”Server_ string. The command we use to activate this property is:

. $WAS_HOME/bin/setupCmdLine.bat$WAS_HOME/bin/wsadmin.bat -conntype SOAP -hostname laredo -port 8881 -f $ITCAM_HOME/config_DC/configDataCollector.jacl attr.properties


3.2.5 Automatically discovering the installation parameters

To analyze the installed WebSphere software in a machine, use the vpd.properties file. Depending on your platform, the vpd.properties file resides on the following directories:

� On AIX: /usr/lib/objrepos/vpd.properties� On Windows: %WINDIR%\vpd.properties or %HOMEPATH%\WINDOWS� On Linux, Solaris, HP-UX: /vpd.properties or /root/vpd.properties

The entries in the vpd.properties file are delimited by a long | character field similar to that shown in Example 3-20.

Example 3-20 vpd.properties file entries

WSBAA60|6|0|0|0|6.0.0.0|1=WebSphere Application Server|IBM WebSphere Application Server| |IBM| |6.0.0.0|C:\IBM\WebSphere\AppServer|0|1|WSBAA60CoreRuntime|6|0|0|0|6.0.0.0|1|0|2|ref_33390|1|WSBAA60|6|0|0|0|6.0.0.0|1|0|false|"_uninst" "uninstall.jar" "uninstall.dat" ""|true|3|WSBAA60|6|0|0|0|6.0.0.0|1

The following list is aimed at helping you identify the information:

� The first identifier is the product ID.

� WSB* identifies the base WebSphere Application Server.

� The second field provides the version number.

� The twelfth field helps retrieve the WebSphere Application Server installation directory.


Retrieve the WebSphere Application Server instances by analyzing the directory structure of the installation. The profiles for WebSphere Application Server V6 are in the profiles directory. From the WebSphere home path, get the application server from the config subdirectory. Figure 3-6 shows a sample directory structure.

Figure 3-6 Configuration directory structure

In the cell level, the application server type is seen. In Figure 3-6, the file is profiles/default/config/cells/laredoNode01Cell/cell.xml. Example 3-21 shows our sample cell.xml.

Example 3-21 Sample cell.xml

<?xml version="1.0" encoding="UTF-8"?><xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:topology.cell="http://www.ibm.com/websphere/appserver/schemas/5.0/topology.cell.xmi" xmlns:ipc="http://www.ibm.com/websphere/appserver/schemas/5.0/ipc.xmi"> <xmi:Documentation> <contact>WebSphere Application Server v5.0 Default Configuration Files v1.7 7/31/02</contact> </xmi:Documentation> <topology.cell:Cell xmi:id="Cell_1" name="laredoNode02Cell" cellDiscoveryProtocol="TCP" cellType="STANDALONE"> </topology.cell:Cell></xmi:XMI>


In Example 3-21 on page 62, the bold type indicates the cell type — a STANDALONE cell or a DISTRIBUTED network deployment-based cell.

� For a standalone cell, retrieve the port information from serverindex.xml under the node name folder, for example, config/cells/laredoNode02Cell/nodes/laredoNode02/serverindex.xml. Example 3-22 shows parts of the server index.

Example 3-22 Sample serverindex.xml

<?xml version="1.0" encoding="UTF-8"?><serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="laredo.itsc.austin.ibm.com">

. . .

<specialEndpoints xmi:id="NamedEndPoint_1144783085594" endPointName="SOAP_CONNECTOR_ADDRESS"> <endPoint xmi:id="EndPoint_1144783085594" host="laredo.itsc.austin.ibm.com" port="8881"/> </specialEndpoints>

. . .

</serverindex:ServerIndex>

� For a distributed cell, first select the core group information to retrieve the node name of the deployment manager. The core group information is stored in a sub-directory under the cell called coregroups\DefaultCoreGroup\coregroup.xml. Get the association between the node and the dmgr server, as shown in Example 3-23.

Example 3-23 Excerpt from coregroup.xml

. . .<coreGroupServers xmi:id="CoreGroupServer_1130943100094" nodeName="laredoCellManager01" serverName="dmgr"/>

. . .

The serverindex.xml is found under the appropriate node, for example, cells/laredoCell01/nodes/laredoCellManager01. The SOAP_CONNECTOR_ADDRESS is similar to that shown in Example 3-22.


Based on the above consideration, a shell script can be written to retrieve all the available application servers on a machine and build the response files individually to perform the installation.

Another option is to use an automated software distribution solution such as IBM Tivoli Configuration Manager or IBM Tivoli Provisioning Manager.

3.3 Configuring and setting up SSL communication

ITCAM for WebSphere provides the facility to communicate with data collectors securely over TCP/IP networks using Secure Socket Layer (SSL). Default certificates and keystores are set up during initial installation. SSL is not enabled by default. Therefore, manual configuration is necessary in order to have the managing server and data collectors communicate over a secure connection. This section discusses the following topics:

� 3.3.1, “Managing server Secure Socket Layer setup” on page 64� 3.3.2, “Data collector Secure Socket Layer setup” on page 65� 3.3.3, “Working with custom certificates” on page 67

3.3.1 Managing server Secure Socket Layer setup

Enabling SSL and using the default certificate information to verify SSL communication works must be your first step. This is to rule out any possible errors while attempting to generate and use your own certificates and keystores.

1. Go to the ITCAM _HOME/etc directory. The kl1.properties and kl2.properties kernel configuration files contain the following parameters that must be activated for Secure Sockets Layer:

security.enabled=truecodebase.security.enabled=true

2. Set the system properties on the WebSphere Application Server where the visualization engine is running. Add the system properties using the WebSphere administration console. Where you set the properties depends on the version of the WebSphere Application Server.

– certificate.path: MS_home/etc/mgmttomgmt.cer – keystore.location: MS_home/etc/CyaneaMgmtStore – keystore.storepass: cyanea94612 – keystore.keypass: cyanea94612 – nodeauth.userid: cyaneamgmt


Figure 3-7 shows these settings in the administration console.

Figure 3-7 New SSL-related settings

3. Restart the kernel components and the visualization engine application server to activate the new settings.

3.3.2 Data collector Secure Socket Layer setup

The SSL setup is stored in the datacollector.properties file. This file is located in the etc subdirectory of the data collector installation directory. Activate the definition shown in Example 3-24.

Example 3-24 SSL settings for datacollector.properties file

security.enabled=truecertificate.path=/opt/IBM/itcam/WebSphere/DC/etc/dctomgmt.cer;/opt/IBM/itcam/WebSphere/DC/etc/dctoproxy.cerkeystore.location=/opt/IBM/itcam/WebSphere/DC/etc/CyaneaDCStorekeystore.storepass=oakland94612keystore.keypass=oakland94612nodeauth.userid=cyaneadccomm.use.ssl.dc=true


Remove the system-generated datacollector.properties file and restart the data collector. The system-generated property file name is similar to <nodename>.<servername>.datacollector.properties.

On restarting the data collector, the message shown in Example 3-25 is generated in msg-dc.log, which is located in the Tivoli common log directory. Once you receive the message, communication between the managing server and the data collector is encrypted.

Example 3-25 SSL active information

<Message Id="CYND5237I" Severity="INFO"> <Time Millis="1143583253402"> 2006-03-28 17:00:53.402-05:00</Time> <Server Format="IP">wsamaix1.tivlab.raleigh.ibm.com</Server> <ProductId>CYN</ProductId> <Component>CYN.msg.datacollector</Component> <ProductInstance>6</ProductInstance> <LogText><![CDATA[CYND5237I Using SSL server socket for component: CommandAgent. All clients connecting to this component MUST connect via SSL, else the client socket may hang till the timeout expires. No user action is required.]]></LogText> <Source FileName="com.cyanea.command.CommandAgent" Method="start"/> <TranslationInfo Type="JAVA" Catalog="com.ibm.tivoli.probe.resource.LogMessages" MsgKey="CYND5237I"></TranslationInfo> <Principal>wsamaix1.tivlab.raleigh.ibm.com/9.42.153.102</Principal></Message>

Note: Only the CommandAgent port uses SSL. Other communications opened by the data collector, such as the ProbeController port and the data collector - Publish Server port, do not use SSL. Therefore, when SSL is enabled, only the data in the channels connected to the CommandAgent port is encrypted.


All the data processed in the CommandAgent channel is encrypted when SSL is enabled. Table 3-4 shows the data classification.

Table 3-4 Data content classification

3.3.3 Working with custom certificates

This section discusses one approach to using custom certificates. There are different ways of configuring the certificates. The approach that we use categorizes the machines into three types: the managing server, the port consolidator, and the data collector. Each type can communicate with the other type and understand one another.

Classification Data

Command and control data Configuring and unconfiguring data collector

User actions related to threads � Starting and stopping JVM threads� Changing thread priorities� Getting thread priorities and thread status� Requesting drill information to see cookies, and so on� Generating thread dumps� Getting thread stack traces

System information � Application server information� Operating system platform information� JVM information

Application information � All the applications installed on the monitored application server

� Application binaries and location information� Thread pool information related to Java Message Service

(JMS), Java 2 Enterprise Edition (J2EE) Connector Architecture (JCA), Java Transaction API (JTA), Servlet, Enterprise JavaBeans (EJB), and so on

� Data source information

Performance data All performance monitor interface (PMI) data

Transport data � Object Request Broker (ORB) data� SOAP ports

Memory information � Obtaining JVM Heap Snapshot™ data� Performing memory leak analysis� Performing heap dump


Three keystores must be generated for custom certificates. The three keystores for a distributed environment implementation are:

� MSStore

This contains the following managing server certificates:

– mgmttomgmt.cer (cn=cyaneamgmt) – dctomgmt.cer (cn=cyaneadc) – proxytomgmt.cer (cn=cyaneaproxy)

� DCStore

This contains the following data collector certificates:

– proxytodc.cer (cn=cyaneaproxy) – mgmttodc.cer (cyaneamgmt)

� ProxyStore

This contains the following port consolidator certificates:

– mgmttoproxy.cer (cn=cyaneamgmt) – dctoproxy.cer (cn=cyaneadc)

The keystore files are generated using the keytool command, which comes with Java Runtime Environment (JRE). In order to run the keytool command, you must be in the JAVA_HOME/bin directory or have it specified in the PATH. Perform a quick check by entering the keytool command. A help list showing the syntax must be visible on the console.

In order to better manage the keystores, we create a directory called keyfiles in ITCAM_HOME/etc and run the keytool command from this directory.


Creating MSStoreThe first keystore that must be created is for the managing server. The name of the keystore can be customized to fit your environment. In the examples provided here, NEWMSStore is used as the keystore name. Because three aliases must be added to this keystore, the command must be run three times, once for each alias.

� For communication between the managing servers, create the alias mgmttomgmt for the keystore NEWMSStore with the command shown in Example 3-26.

Example 3-26 Command to create the alias mgmttomgmt

keytool -genkey -alias mgmttomgmt -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 -dname "cn=cyaneamgmt, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

This command creates the NEWMSStore file and the mgmttomgmt key in it. The password, keypass, and storepass can be anything you want them to be.

� For communication from the managing server to the data collector, create the alias dctomgmt using the command shown in Example 3-27.

Example 3-27 Command to create the alias dctomgmt

keytool -genkey -alias dctomgmt -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 -dname "cn=cyaneamgmt, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

This command adds the dctomgmt alias into the NEWMSStore file.

� For communication from the managing server to the port consolidator, create the alias proxytomgmt to the keystore using the command shown in Example 3-28.

Example 3-28 Command to create the alias proxytomgmt

keytool -genkey -alias proxytomgmt -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 -dname "cn=cyaneamgmt, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"


Creating DCStoreThe second set of keystores that must be created is for the data collector. As with the managing server keystore, customize the data collector keystore name to your environment. In this example, we use NEWDCStore.

� Create the alias proxytodc for keystore NEWDCStore using the command shown in Example 3-29.

Example 3-29 Command to create the alias proxytodc

keytool -genkey -alias proxytodc -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

� Create the alias mgmttodc for the NEWDCStore file using the command shown in Example 3-30.

Example 3-30 Command to create the alias mgmttodc

keytool -genkey -alias mgmttodc -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

Creating ProxyStoreProxyStore is the last keystore that must be created before extracting the certificates. This is used by the port consolidator in a distributed environment. We use the file NEWProxyStore for this purpose.

� Create the alias mgmttoproxy for the keystore NEWProxyStore using the command shown in Example 3-31.

Example 3-31 Command to create the alias mgmttoproxy

keytool -genkey -alias mgmttoproxy -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"


� Create the alias dctoproxy using the command shown in Example 3-32.

Example 3-32 Command to create the alias dctoproxy

keytool -genkey -alias dctoproxy -keyalg RSA -keysize 1024 -sigalg MD5withRSA -validity 2000 -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2 -dname "cn=cyaneaproxy, OU=Tivoli, O=IBM, L=Raleigh, ST=NC, C=US"

Extracting certificates from the keystoresOnce all the three keystores are created, extract the certificates from each keystore and give them to their communication partners. These certificates are used to validate the SSL connection between the components. Figure 3-8 shows the keystores and certificate exchanges.

Figure 3-8 Keystores and certificate exchanges

Managing Server

Port Consolidator

Data Collector

MSStore DCStoreProxyStore

mgmttomgmt

dctomgmt

proxytomgmt dctoproxy

mgmttoproxy

proxytodc

mgmttodc

mgmttomgmt

dctomgmtproxytomgmt

dctoproxymgmttoproxy proxytodc

mgmttodc

key

certificates


Do the following:

� Extract all the certificates from the managing server keystore NEWMSStore. The export commands are shown in Example 3-33.

Example 3-33 Commands to extract certificates from the managing server keystore

keytool -export -alias mgmttomgmt -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 -file mgmttomgmt.cerkeytool -export -alias dctomgmt -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh 2 -file dctomgmt.cerkeytool -export -alias proxytomgmt -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 -file proxytomgmt.cer

� Extract all the certificates from NEWDCStore by using the commands shown in Example 3-34.

Example 3-34 Commands to extract all the certificates from NEWDCStore

keytool -export -alias proxytodc -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2 -file proxytodc.cerkeytool -export -alias mgmttodc -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2 -file mgmttodc.cer

� Extract all the certificates from NEWProxyStore by using the commands shown in Example 3-35.

Example 3-35 Commands to extract all the certificates from NEWProxyStore

keytool -export -alias mgmttoproxy -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2 -file mgmttoproxy.cerkeytool -export -alias dctoproxy -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2 -file dctoproxy.cer

After completing the extraction of all the certificates, the directory must have seven .cer files. Import these certificates to their appropriate partners using these commands:

� For NEWMSStore, use the commands shown in Example 3-36.

Example 3-36 Commands to import certificates: NEWMSStore

keytool -import -alias mgmttomgmt -file mgmttomgmt.cer -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 keytool -import -alias mgmttoproxy -file mgmttoproxy.cer -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2 keytool -import -alias mgmttodc -file mgmttodc.cer -keypass raleigh1 -keystore ./NEWMSStore -storepass raleigh2


� For NEWDCStore, use the commands shown in Example 3-37.

Example 3-37 Commands to import certificates: NEWDCStore

keytool -import -alias dctomgmt -file dctomgmt.cer -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2 keytool -import -alias dctoproxy -file dctoproxy.cer -keypass raleigh1 -keystore ./NEWDCStore -storepass raleigh2

� For NEWProxyStore, use the commands shown in Example 3-38.

Example 3-38 Commands to import certificates: NEWProxyStore

keytool -import -alias proxytomgmt -file proxytomgmt.cer -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2 keytool -import -alias proxytodc -file proxytodc.cer -keypass raleigh1 -keystore ./NEWProxyStore -storepass raleigh2

Deploy these newly generated keystores to the appropriate machines. Store NEWMSStore on the managing servers, NEWProxyStore on the port consolidator machines, and NEWDCStore on the data collector machines. Copy the certificate files to the respective machines, too.

Update the appropriate configuration entries specified in 3.3.1, “Managing server Secure Socket Layer setup” on page 64, and 3.3.2, “Data collector Secure Socket Layer setup” on page 65. Restart all the components to activate the SSL with the custom certificates.



Chapter 4. Maintenance of ITCAM for WebSphere

This chapter discusses the issues relating to maintenance of ITCAM for WebSphere in a large-scale environment. This chapter discusses the following topics:

� 4.2, “Performance and availability of the managing server” on page 76� 4.2, “Performance and availability of the managing server” on page 76� 4.3, “Backup and recovery configuration” on page 83� 4.4, “Log files and configuration files” on page 85� 4.5, “Performing product maintenance” on page 87

4


4.1 Operating ITCAM for WebSphere

This chapter discusses some operational issues regarding a large ITCAM for WebSphere environment. The focus is on the issues that arise largely because of environment size. Following is a list of these issues along with details about the sections that describe them:

� Section 4.2, “Performance and availability of the managing server” on page 76, shows that as the size of the environment grows, even a slight inefficiency can cause a major problem. A large-scale environment requires the performance and availability of the managing server to be soundly maintained.

� Section 4.3, “Backup and recovery configuration” on page 83, shows that backup and recovery of configuration is a necessity. The ability to quickly identify and recover system configuration is critical. The large size of the environment prohibits ad hoc backup and recovery.

� Section 4.4, “Log files and configuration files” on page 85, shows that log and configuration files must be maintained. There are two aspects of file maintenance: keeping the size of the log files under control and ensuring that the correct property files are used.

� Section 4.5, “Performing product maintenance” on page 87, discusses implementation of WebSphere, Database 2 (DB2), and ITCAM for WebSphere patches. Typically, with the number of data collectors involved, this can be a huge task.

4.2 Performance and availability of the managing server

Performance and availability of the ITCAM for WebSphere managing server are mainly related to the IBM WebSphere Application Server that hosts the visualization engine and the OCTIGATE database. This section discusses the following topics:

� 4.2.1, “Performance of the WebSphere Application Server” on page 76� 4.2.2, “Database maintenance” on page 77� 4.2.3, “Data trimming” on page 78

4.2.1 Performance of the WebSphere Application Server

There are several critical tuning factors for the visualization engine. The sole use of the visualization engine is to provide users with information regarding the enterprise activity, not collecting and analyzing the activity. This eases the role of the visualization engine to not being completely critical.


Tuning of the visualization engine is mainly to ensure that the WebSphere Application Server has adequate memory. This also relates to the fact that the visualization engine has to run on one of the managing servers that runs other Java-based kernel components.

Each logged-in user and the activity of each user requires additional storage to be acquired by the visualization engine. The Java garbage collection process also plays an integral part in providing adequate memory resource.

4.2.2 Database maintenance

Database performance is largely related to how the data is stored physically and how well the database manager understands the nature of the data. This section primarily discusses the use of DB2 as the database manager.

The primary tools to ensure this are the REORG and RUNSTAT tools. You can run both of these utilities from the DB2 utilities from the DB2 command line.

� The REORGCHK utility reads the table information and analyzes the necessity of running reorg. REORGCHK uses the current statistics or generates its own statistics.

The REORG utility reorganizes the database structure to ensure that data is allocated coherently and in a typical access sequence requested by most queries. The syntax for running REORG is:

DB2 REORG TABLE creator.table_name INDEX primary_index_name

� The RUNSTATS utility collects the statistics of the tables and indexes to ensure that the database manager chooses the most optimized method to get the data requested by an Structured Query Language (SQL) request. The syntax for running RUNSTATS is:

DB2 RUNSTATS ON TABLE creator.table_name WITH <runstat_options>

Typically, you must run REORGCHK for frequently modified tables and then determine whether you want to run REORG on those tables. After all the modifications have been performed, run RUNSTATS to ensure that the statistics are current and that DB2 obtains the most optimal path to the data an Structured Query Language (SQL) requested.

Chapter 4. Maintenance of ITCAM for WebSphere 77

ITCAM for WebSphere supplies a script to run the RUNSTATS utility. This script is called run-stat-cmds and resides in the bin directory. This command selects specific tables that are necessary to perform RUNSTATS on. The syntax of this command depends on the platform you execute. It can be either one of the following:

� run-stat-cmds.bat OCTIGATE amuser

� export JDBC_USER=amuserexport JDBC_PASSWORD=ampasswdrun-stat-cmds.sh OCTIGATE

The table that ITCAM for WebSphere constantly modifies is the REQUEST table. This table must be checked frequently. The run-stat-cmds automatically checks this table and stores its output in the logs/reorg.log file. If the cluster ratio of the REQUEST table is lower than 90, run REORG on this table and rerun RUNSTATS on the REQUEST table. This REORG command is as follows:

db2 "REORG TABLE AMUSER.REQUEST INDEX R_TIME "

We recommend a regular, maybe monthly, maintenance window to REORG all the tables for ITCAM for WebSphere and perform a complete RUNSTATS.

4.2.3 Data trimming

Another aspect of database performance is processing the data size. Because monitoring information is accumulated, access to current information may be degraded, as the database has to process more information than it must. Old data must be trimmed to allow more efficient processing. This process is application-specific, as the database manager does not know the relationship between the data. This also minimizes the disk space usage for ITCAM for WebSphere.

There are two methods of pruning the tables. Run both of them, as they address different sets of tables. The pruning methods are:

� Automatic pruning by the archive agent

This prunes data every two hours and keeps data for 48 hours. See “Archive agent trimming” on page 79.

� Manual pruning using the datatrim.sh command

We recommend that you run this regularly. A daily run to minimize the amount of data to be pruned may be appropriate. See “Running datatrim.sh” on page 80.


Archive agent trimmingOnce enabled, the managing server archive agent automatically prunes the following tables:

� MEMORY_DATA � GC_DATA � MQI_QUEUEMGR_SR_OVERVIEW� MQI_QUEUE_SR_OVERVIEW� CTG_SR_OVERVIEW� WEBSERVERCHARTDATA� IMSTHREADS� VOLUMESTAT� PORTALSTATS

When data trimming is enabled, pruning occurs every two hours, keeping 48 hours worth of data in each table. These tables, and the criteria for trimming the tables, are hard-coded in ITCAM for WebSphere, and cannot be changed.

Automatic data trimming using the archive agent is enabled by setting the ENABLE_DATATRIMMER property to true in the etc/aa1.properties file or the etc/aa2.properties file, or both. If both the properties files have data trimming enabled, there is a built-in delay for the second archive agent in order to prevent the occurrence of database deadlocks.

Stop and start the archive agents to pick up the changes made to the ENABLE_DATATRIMMER property using the following commands:

<MS_HOME>/bin/aactl.sh stop aa1<MS_HOME>/bin/aactl.sh start aa2

Note: You may see a datatrimmer.sh script in the <MS_HOME>/bin directory. This script is not recommended and is not discussed in this paper.

Note: If you are using more than two archive agents, do not enable the data trimmer for the other archive agents. Only aa1 and aa2 must have the ENABLE_DATATRIMMER=true property.


Running datatrim.shThe datatrim.sh script is a command-line utility that can be scheduled or run manually to trim data that is no longer required from the database. As it is shipped with ITCAM for WebSphere, this script prunes data from the following tables:

� REQUEST� METHOD� SERVERSTATS� PMISTATS� VOLUMESTAT

APAR PK17797 allows additional tables to be trimmed, including the following:

� MEMORY_DATA� IMSEVENTS

The datatrim.sh script carries out two major processes, marking and trimming:

� Mark deleting process.

This process updates the rows for the REQUEST and METHOD tables that will be deleted in the data trim process. This method is controlled by the etc/deleterelatedtables.xml control file shown in Example 4-1.

Example 4-1 Deleterelatedtables.xml control file

<?xml version="1.0" encoding="UTF-8"?><tableinfo>  <commitcount>2000</commitcount> <useoracle>false</useoracle> <table> <name>request</name> <daystokeep>7</daystokeep>  <columnname>end_time</columnname>  <startdate></startdate> <enddate></enddate>  <adminserver></adminserver> <appserver></appserver> <idcolumn>request_id</idcolumn><delayinterval>15</delayinterval> <relatedtables> <name>method</name>


<columnname>start_time</columnname> </relatedtables> </table></tableinfo>

� Data trim process

This process deletes the data marked in the REQUEST and METHOD tables as a result of the marking process. This process is controlled by the etc/deletesingletable.xml control file shown in Example 4-2.

Example 4-2 Deletesingletable.xml control file

<?xml version="1.0" encoding="UTF-8"?><tableinfo> <commitcount>2000</commitcount> <useoracle>true</useoracle> <table> <name>serverstats</name> <daystokeep>7</daystokeep>  <columnname>current_time</columnname>  <startdate></startdate> <enddate></enddate>  <adminserver></adminserver> <appserver></appserver> <serverid>probe_id</serverid><delayinterval>5</delayinterval> </table> <table> <name>pmistats</name> <daystokeep>7</daystokeep>  <columnname>current_time</columnname>  <startdate></startdate> <enddate></enddate> 


<adminserver></adminserver> <appserver></appserver> <serverid>probe_id</serverid><delayinterval>5</delayinterval> </table></tableinfo>

For both the eXtensible Markup Language (XML) files in Example 4-1 on page 80 and Example 4-2 on page 81, the following apply:

� The parameter commitcount controls the number of records to be committed to a database in a transaction. Change the commitcount setting from a default value of 2000 to something higher if you have large numbers of records to be deleted. This speeds up the trimming process. The commitcount number applies to all the tables specified in the XML files.

� The useoracle setting is no longer used. Instead, the datatrim.sh script uses the Java Database Connectivity (JDBC) driver to determine the type of database (DB2 or Oracle) being used.

� For each individual table, set either daystokeep or the startdate and enddate pair settings. If daystokeep is set, it overrides the startdate and enddate values.

To invoke datatrim.sh, log in using the ITCAM for WebSphere owner, typically called amuser. Run the utility from the managing server. It will write its output in the log/datatrim.log file. The syntax of the command is:

$MS_HOME/bin/datatrim.sh octigate amuser ampasswd

The result and status of the datatrim.sh command can be analyzed in the logs/datatrim.log file.

The datatrim.sh command prunes the tables, but does not optimize or reclaim free space. Perform these actions by using the REORG utility in DB2 or by exporting and reimporting the tables in Oracle.

We recommend that you change the daystokeep value to the number of days of data that you want to keep, and then run the datatrim.sh script every night. Running the script every night facilitates the removal of fewer records from the database more often. Running the command less frequently can result in the script running for extended periods of time.


While it is not necessary to stop the archive agents (<MS_HOME>/bin/aactl.sh stop) prior to running the datatrim.sh script, stop the archive agents to avoid database deadlocks.

4.3 Backup and recovery configuration

The backup and recovery procedure covers both WebSphere backup and database backup. The aim of a backup policy is to quickly restore the operation. Another objective is to minimize the backup’s impact on the operation. This section discusses the following topics:

� 4.3.1, “ITCAM for WebSphere backup” on page 83� 4.3.2, “WebSphere configuration backup” on page 83� 4.3.3, “Database backup and restore” on page 84

4.3.1 ITCAM for WebSphere backup

Back up the installation directory structure of ITCAM for WebSphere using any backup utility. The restore of the file system of ITCAM for WebSphere must be owned by the ITCAM for WebSphere user ID, which is typically amuser.

4.3.2 WebSphere configuration backup

WebSphere configuration backup must be considered for both the managing server and the data collector. Depending on whether you want to reinstall or restore the backup, initiate a backup. As WebSphere configuration is not a dynamic entity, taking a backup of the initial implementation and a backup procedure to be implemented before and after a configuration change must be sufficient.

To dump the WebSphere Application Server configuration, use the backupConfig command. The syntax for the backupConfig command is:

backupConfig <file> -nostop -logfile <filename> -user <name> -password <pswd>

To restore the WebSphere configuration, use the restoreConfig command. The syntax for restoreConfig command is:

restoreConfig <file> -nostop -logfile <filename> -user <name> -password <pswd>

Note: Stopping the archive agents leads to loss of performance data.


4.3.3 Database backup and restore

For backing up the DB2 UDB, use the DB2 UDB backup utility that is launched by using the DB2 command-line processor. The syntax for the DB2 backup utility is:

db2 backup database <dbname> user <username> using <password> to <file>

A successful backup returns a message that is similar to:

Backup successful. The timestamp for this backup image is : 20041116104855

You must either keep the timestamp or rename the file according to the timestamp. The timestamp is required to restore the database.

To restore the database, execute the following command:

db2 restore database <dbname> user <user> using <password> from <file> taken at <timestamp> <option>

An example of running the restore with an option to prohibit rolling forward is shown in Example 4-3. Rolling forward is a feature that allows the post-backup changes to the database to be applied from the transaction redo log. A typical distributed DB2 installation does not activate this feature.

Example 4-3 Restoring DB2 database

> db2 restore database octigate user itcamusr using passwd from /tmp/octigate taken at 20041116104855 without rolling forwardSQL2539W Warning! Restoring to an existing database that is the same as the backup image database. The database files will be deleted. Do you want to continue ? (y/n) yDB20000I The RESTORE DATABASE command completed successfully.

Note: A directory path for the target file must exist. Manually create the path before executing the backup command. If not, you will get an exception similar to:

SQL2036N. The path for the file or device "/tmp/octigate" is not valid.


4.4 Log files and configuration files

Perform the file maintenance process regularly to reduce the amount of disk space used and to manage monitoring configuration. Log files maintenance applies to both managing servers and data collectors, with configuration file maintenance applying more to data collectors. This section discusses the following topics:

� 4.4.1, “Managing log files” on page 85� 4.4.2, “Managing the configuration files” on page 86

4.4.1 Managing log files

Log files in ITCAM for WebSphere are used for problem determination of the product’s working mechanism. There are several locations for log files, depending on the components that generate them. These locations are:

� Tivoli common directory

This is where most of the log files are stored. Some typical directory names are:

– %ProgramFiles%\ibm\tivoli\common (Windows)– /var/ibm/tivoli/common (UNIX)

The directory under the common directory is typically identified by a three-character product code such as CYN for ITCAM for WebSphere. In the product directory, there are sub-directories:

– First-failure data capture (FFDC) for first-failure data capture application

The FFDC directory contains diagnostic information pertaining to the program exceptions that are captured.

– Logs subdirectory of Tivoli common directory

The logs subdirectory is typically filled with log files and must be maintained.

� The logs subdirectory of the installation path

These are the logs that are typically initiated either before the product is active, such as standard output and standard error redirection, or from a leftover log file from a previous release.

Some utility commands such as datatrim.sh also use this directory to store their output.

Because the log files in the common directory have a maximum size and predefined number of generations, they cannot exceed a certain size. This path must be maintained and checked for ad hoc logging that may use up the space.


We recommend that you use a separate file system to contain the common directory.

The contents of the logs subdirectory of the installation path must be regularly cleaned up manually to prune the content.

4.4.2 Managing the configuration files

Figure 4-1 shows the configuration files for data collectors inter-relation. In Version 6.1, there is a significant change in the structure of the configuration files. This is due to the changes of the byte code instrumentation mechanism.

Figure 4-1 Configuration file structure

From time to time, you may have to modify some XML configuration files to enable certain product features. There is currently no mechanism in the product to make this switch. Employ an external automation mechanism to switch these configurations back and forth.

Configuration change is typically performed by changing the toolkit_custom.properties and several related xml files under the itcamdc/etc directory. This is to add or remove the classes to be used for testing with memory allocation probe or locking probe.

$DC_home

itcamdc/etc

runtime

<wasver>.<wasnode>,<server>

custom_requests.xmllock_analysis.xml

memory_leak_diagnosis.xmlmethod_entry_exit.xmltoolkit_config_ctg.xmltoolkit_config_ims.xmltoolkit_config_jdo.xmltoolkit_config_kwj.xmltoolkit_config_mqi.xmltoolkit_config_ra.xml

custom/toolkit_custom.propertiescyn-cclog.propertiescynlogging.properties

jiti.properties*.datacollector.properties

*.datacollector.policy*.kwjdc.properties*.toolkit.properties

*.toolkit.xml


4.5 Performing product maintenance

Various considerations must be taken into account when dealing with the application of product maintenance, fixes, and fix packs. In a large environment this must be performed remotely and automated with minimal impact to the production system. This section discusses the following topics:

� 4.5.1, “Getting software updates” on page 87� 4.5.2, “Updating ITCAM for WebSphere managing server” on page 87� 4.5.3, “Updating ITCAM for WebSphere data collectors” on page 90

4.5.1 Getting software updates

Software updates are available at the following IBM Web sites:

� For ITCAM for WebSphere:

http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliCompositeApplicationManagerforWebSphere.html

� For DB2 Universal Database:

http://www-1.ibm.com/support/docview.wss?rs=71&uid=swg27007053

� For WebSphere Application Server:


Typically, you do not have to get updates for DB2 or WebSphere on the managing server. ITCAM for WebSphere only supports a certain level of these products. However, you may sometimes have to fix an impending problem with a certain software level.

4.5.2 Updating ITCAM for WebSphere managing server

The ITCAM for WebSphere managing server updates must be performed as a scheduled maintenance. This means that:

� You must have a backup of the management server system.� You must perform an orderly shutdown of the management server.� Users must be notified of the system’s unavailability.

The managing server updates for ITCAM for WebSphere software are typically performed using a silent installer. The ITCAM for WebSphere fix packs consist of two sets of files that are distributed as tar files:

� 6.1.0-TIV-ITCAMfWAS_SVR-FP000n.tar� ITCAMfWAS_V6_UpdateInstaller.tar


http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliCompositeApplicationManagerforWebSphere.html



Fix packs and interim fixes are distributed as tar files, which are installed using the silent command-line update installer. Multiple fixes may be applied by combing all *.update files found in the <fix-dir>/update directory to a common update directory.

The procedure to apply the updates is derived from the readme for the update installer V6.1.0.pdf.

1. Modify the silentUpdate.properties file to reflect your environment. Example 4-4 illustrates the changes required for our managing server, srv152.

Example 4-4 The silentUpdate.properties

# product locationproduct.location=/opt/IBM/itcam/WebSphere/MS# updates locationupdates.location=./updates######################################################################## Managing Server properties# These properties apply only for the MS update installation,######################################################################## change to false if database should not be updatedupdateDb=true# change to false if VisualEngine should not be updatedupdateVe=true# TODO: uncomment this property and specify correct WAS_HOMEupdateVe.wasHome=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01updateVe.was.soap.host=srv152.itsc.austin.ibm.comupdateVe.was.soap.port=8880updateVe.was.node=srv152Node01updateVe.was.server=server1updateVe.was.user=rootupdateVe.was.password=rpasswd#ms.oracle.driver.jar=${ORACLE_JDBC_DRIVER_PATH}/ojdbc14.jar#ms.oracle.driver.path=# Set this property to reconfigure MS processes to bind to the specified address.#ms.bind.host.name=172.17.11.82######################################################################## Uninstall properties######################################################################## the updates to be uninstalleduninstall.updates=last


2. Ensure that the following environment is set prior to running the silent updater:

export JAVA_HOME=/opt/IBM/WebSphere/AppServer/javaexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=JAVA_HOME:$CLASSPATH

3. Prepare the installation by executing:

./silentUpdate.sh -prepareInstall

4. Do not proceed with the installation. Clean the results of the previous command by executing:

./silentUpdate.sh -cleanPrepared

5. Install by executing:

./silentUpdate.sh -install

6. If the install failed, rollback is achieved by:

./silentUpdate.sh -rollback

7. Display updates that have been installed by executing:

./silentUpdate.sh -displayInstalledUpdates

Example 4-5 shows an example of the output after running the command.

Example 4-5 Output from silentUpdate.sh -displayInstalledUpdates

# ./silentUpdate.sh -displayInstalledUpdatesUpdate installer version 6.1.0, build 20070513145421Logging details into /opt/IBM/itcam/WebSphere/MS/logs/update/update_20070616101145.logAction: display installed updatesUpdates installed 5/31/07 2:51 PM 6.1.0.1.1 Interim Fix 1 after Fix Pack 1Updates installed 5/31/07 2:09 PM 6.1.0.1.5 Interim Fix 5 after Fix Pack 1Updates installed 5/31/07 2:08 PM 6.1.0.1.4 Interim Fix 4 after Fix Pack 1Updates installed 5/31/07 12:20 PM 6.1.0.1.3 Interim Fix 3Updates installed 5/31/07 11:56 AM 6.1.0.1 Fix Pack 1Finished successfully


4.5.3 Updating ITCAM for WebSphere data collectors

Data collectors are typically run in a production environment. Updating these machines requires strict change control. Because data collectors run as an integral part of the WebSphere Application Server, updating the data collectors typically involves restarting the application server. When dealing with a clustered environment, alternately restarting the members of a cluster does not affect application availability, while a standalone application server means that the application is unavailable.

The primary prerequisite of this fix pack is generally to have the managing server updated before installing the fix pack to the data collectors. Installation of data collectors can be performed in stages. All the data collectors do not have to be updated together.

As with the managing server updates, the data collector updates, too, come as two tar files:

� 6.1.0-TIV-ITCAMfWAS_MP-FP00nn.tar.� ITCAMfWAS_V6_UpdateInstaller.tar

The following discussion is derived from our experience in installing fix pack 1 of ITCAM for WebSphere V6.1. This fix pack is distributed as 6.1.0-TIV-ITCAMfWAS_MP-FP0001.

1. Download and untar the update installer tar file ITCAM_V61_UpdateInstaller.tar.

2. Download and untar the fixes tar files. You may untar all available updates to the same updates directory. Both fix packs and interem fixes are may be applied together. Example 4-6 lists the fixes that we apply.

Example 4-6 A list of update files to be applied

6.1.0-TIV-ITCAMfWAS_MP-FP0001.update6.1.0.0-TIV-ITCAMfWAS_MP-IF0003.update6.1.0.0-TIV-ITCAMfWAS_MP-IF0004.update6.1.0.0-TIV-ITCAMfWAS_MP-IF0005.update6.1.0.0-TIV-ITCAMfWAS_MP-IF0006.update6.1.0.1-TIV-ITCAMfWAS_MP-IF0001.update

Note: Use the following steps following the DataCollector Fixpack1 update or after multiple fix updates that includes Fixpack 1 - 6.1.0.1-TIV-ITCAMfWAS_MP-IF0001.


3. The update installer requires the Java Runtime Environment Version 1.3.1 or later. Either include the java command into PATH or define the JAVA_HOME environment variable to point to the Java Runtime Environment home directory. In UNIX or Linux, run:

export JAVA_HOME=/opt/WebSphere/AppServer/java

4. The update installer is a non-interactive silent installer that works based on the response file. The default name of the response file is silentUpdate.properties, and it is supplied with the update installer. Update the response file, as in Example 4-7.

Example 4-7 The silentUpdate.properties for applying fixes to the data collector

product.location=/opt/IBM/itcam/WebSphere/DCupdates.location=./updatesupdates.gclog=/tmp/gclog.loguninstall.updates=last

5. Stop the WebSphere Application Servers that have the data collector installed.

6. In order to prepare for installation, change to the directory where the update installer scripts are stored. The preparation is recommended for checking the environment. Run the command ./silentUpdate.sh -prepareInstall. A typical output of the prepareInstall is shown in Example 4-8.

Example 4-8 The prepare for installation

# ./silentUpdate.sh -prepareInstallUpdate installer version 6.1.0, build 20070513145421Logging details into /opt/IBM/itcam/WebSphere/DC/logs/update/update_20070608170706.logAction: prepare installFinished successfully

7. To update the data collector with all the update files in the updates sub-directory, run the command ./silentUpdate.sh -install. A typical output of the install command is shown in Example 4-9.

Example 4-9 Running the patch installation

# ./silentUpdate.sh -installUpdate installer version 6.1.0, build 20070513145421Logging details into /opt/IBM/itcam/WebSphere/DC/logs/update/update_20070608171205.log

. . .


8. When the installation has completed, you can verify the data collector version using the command. ./silentUpdate.sh -displayInstalledUpdates. The status in our environment is shown in Example 4-10.

Example 4-10 Patch installation result

# ./silentUpdate.sh -displayInstalledUpdatesUpdate installer version 6.1.0, build 20070513145421Logging details into /opt/IBM/itcam/WebSphere/DC/logs/update/update_20070608172553.logAction: display installed updatesUpdates installed 6/8/07 5:13 PM 6.1.0.1.1 Interim Fix 1 6.1.0.1 Fix Pack 1 6.1.0.0.6 Interim Fix 6 6.1.0.0.5 Interim Fix 5 6.1.0.0.4 High CPU usage for TEMA PMI collection 6.1.0.0.3 JITI supportFinished successfully

9. In a WebSphere Application Server V6.1 environment, activate the OSGI configuration initialization. From the WebSphere /profiles/<profile-name>/bin directory, run the osgiCfgInit.sh script. You should expect no messages from this script.

10.Restart the application server to activate the data collectors.

Some patches may have to be updated manually through the WebSphere administration console. This is typically to add certain options on WebSphere. To perform the updates silently, build a Java Command Language (JACL) script.

Note: At least one server must first be configured before updating the version level of the data collecter.


Chapter 5. Planning for ITCAM for Response Time Tracking

This chapter discusses areas that must be considered during the planning phase of IBM Tivoli Composite Application Manager implementation in a large environment. The topics discussed in this chapter are:

� 5.1, “Planning considerations” on page 94� 5.2, “Product architecture” on page 94� 5.3, “Sizing the servers” on page 96� 5.4, “Deployment of ITCAM for Response Time Tracking” on page 98� 5.5, “Communication and security considerations” on page 100� 5.6, “Reliability and high availability” on page 102

5


5.1 Planning considerations

As discussed in 1.2.2, “Concerns and considerations” on page 7, the following aspects of a large-scale implementation are discussed in this section, too:

� Understanding the product architecture

This allows you to make the correct choices. Section 5.2, “Product architecture” on page 94, describes the architecture for ITCAM for Response Time Tracking.

� Sizing the servers

This is important to correctly acquire adequate servers and choose a sound software configuration option. Section 5.3, “Sizing the servers” on page 96, describes an approach that can be used.

� Understanding the servers’ configuration options and agent deployment

This is discussed for ITCAM for Response Time Tracking in 5.4, “Deployment of ITCAM for Response Time Tracking” on page 98.

� Planning for communication security

This is a mandatory step for an enterprise with business-critical and sensitive information in a transaction environment. Section 5.5, “Communication and security considerations” on page 100, discusses confidentiality and firewall requirements.

� Reliability discussion, failover, and disaster recovery

These, too, are mandatory aspects of a business-critical process in a large enterprise. Section 5.6, “Reliability and high availability” on page 102, discusses this issue.

5.2 Product architecture

This section discusses the product architecture of ITCAM for Response Time Tracking. This understanding is critical for planning and deciding about server configuration and other implementation issues. See also IBM Tivoli Composite Application Manager V6.1 Family Installation, Configuration, and Basic Usage, SG24-7151.

ITCAM for Response Time Tracking V6.0 has evolved from IBM Tivoli Monitoring for Transaction Performance V5.3 and inherits the major components and functions of IBM Tivoli Monitoring for Transaction Performance V5.3.


Figure 5-1 shows the components of ITCAM for Response Time Tracking.

Figure 5-1 Components of ITCAM for Response Time Tracking

Basically, ITCAM for Response Time Tracking is controlled from the management server, which provides a centralized repository of policy, configuration, and data for the ITCAM for Response Time Tracking environment.

The rest of ITCAM for Response Time Tracking is the management agent. The management agent performs response time data collection and performance functions on behalf of the management server. The management agent is a single agent that can have different components deployed on it to perform different functions.

The management server and management agent typically operate in an unrestricted port environment. When there is a firewall between them, they restrict the port usage to communicate. The firewall typically requests that they use a single communication port to talk back and forth. This is where the Store and Forward agent comes in. It bundles the communication between the management server and the management agent to use a single port to pass through the firewall. Because the Store and Forward agent can be cascaded, in this sense, there can be a chain of Store and Forward agents passing through multiple layers of firewalls.

RTTManagement

server

RTTStore and

forward agent

RTT management

agent

RTT management

agent

RTT management

agent

RTT management

agent

FIR

EWA

LL

FIR

EWA

LL

RTTStore and

forward agent

RTT management

agent

Tivoli EnterpriseMonitoring

Agent

Chapter 5. Planning for ITCAM for Response Time Tracking 95

A special management agent resides on z/OS machines. The management agent on z/OS machines has the transaction server component activated to receive performance information from the Customer Information Control System (CICS) and Information Management System (IMS) data collector.

In ITCAM for Response Time Tracking V6.0, information from the management server can be forwarded to the Tivoli Enterprise Monitoring Server for display in the Tivoli Enterprise Portal. This is achieved using the Tivoli Enterprise Monitoring Agent for ITCAM for Response Time Tracking.

5.3 Sizing the servers

Due to the large scale of the implementation, the size of the servers assumes importance. Sizing determines the hardware configuration and implementation consideration of the servers. This section discusses the following topics:

� 5.3.1, “Sizing parameters” on page 96� 5.3.2, “Sizing estimation for ITCAM for Response Time Tracking” on page 97

5.3.1 Sizing parameters

To evaluate the size of the management server, you must understand the data collection mechanism for ITCAM for Response Time Tracking. Overall, the load for the management server in a steady state environment mainly relates to the collection of response time information and serving the Web console for operators. Additional work may be required to perform the following functions:

� Running a discovery policy

� Performing administrative tasks from the Web console or the command-line interface (CLI)

� Filing the uploading activities from the Synthetic Transaction Investigator or the Rational Robot

� Fulfilling an ad hoc request to reload data from the Web console

The other additional tasks are not addressed in this Redpaper as these are not significant in a steady-state production environment.

Reporting load parametersReporting load is mainly local access to Database 2 (DB2) Universal Database. This does not have a significant network overhead. The processing overhead is the processor requirement to serve the users and DB2 buffering concerns.


This load is typically proportional to the number of operators with direct access to ITCAM for Response Time Tracking. However, in ITCAM for Response Time Tracking V6.0, the majority of view-only access for response time information can be consolidated into the Tivoli Enterprise Portal, which allows a single Tivoli Enterprise Monitoring Agent to collect information from the ITCAM for Response Time Tracking management server.

With the Tivoli Enterprise Portal solution in place, typical Web console users must be limited to administrative staff, because they can be easily contained.

Response time measurement The response time information is recorded by the Application Response Measurement (ARM) agent process from various components that perform monitoring policies, such as listening and playback. This information is uploaded to the management server using the eXtensible Markup Language (XML) data structure in a predefined and scheduled interval. The size of the data transferred is dependent on the number of response time data collected:

� For playback policies, you have a fixed amount of response time information based on the playback occurences on the time interval.

� For listening policies, the response time depends on the number of transactions actually invoked, the sampling rate, and the number of monitored components invoked by the transactions.

5.3.2 Sizing estimation for ITCAM for Response Time Tracking

Specific to ITCAM for Response Time Tracking, consider the following parameters for sizing:

� “Communication bandwidth” on page 97� “Management server processing” on page 98� “Database size” on page 98

Communication bandwidthCommunication between the management server and the management agent primarily consists of:

� Initial loading of policies and schedules

This happens when the management agent is started. The size depends on how many policies are deployed to the management agent.

� Data retrieval for response time data

Data retrieval is typically performed every hour. You can also request an ad hoc retrieval from the Web console when data is retrieved.


� Additional actions such as deploying a monitoring component or assigning a new monitoring policy to an agent do not contribute to a large workload in a production environment.

Management server processingThe management server is constructed in a single, large enterprise application. The processing of the management server application is affected by the following parameters:

� The number of connected management agents� The size of data retrieved for each interval� The number of connected Web console users

Database sizeThe size of the database depends on the amount of response time information loaded from the management agents. The response time information is collected from policies such as:

� Discovery policies� Listening policies� Playback policies

5.4 Deployment of ITCAM for Response Time Tracking

The considerations for installing and configuring ITCAM for Response Time Tracking are covered in this section. The topics discussed here are:

� 5.4.1, “Designing the management server” on page 98� 5.4.2, “Deploying the management agent” on page 99

5.4.1 Designing the management server

The ITCAM for Response Time Tracking management server consists of the following products:

� DB2 Universal Database Enterprise Server or Oracle database server� WebSphere Application Server� ITCAM for Response Time Tracking management server application

The ITCAM for Response Time Tracking management server is a single contact point for all the management agents and the user Web console. This in turn leads to the ITCAM for Response Time Tracking management server being a single point of failure for response time monitoring.


In a large environment, we recommend that you separate the management server into a multiple-server implementation in a clustered environment. Figure 5-2 shows a clustered environment.

Figure 5-2 Clustered environment

The environment shown in Figure 5-2 introduces the following:

� Load balancing using the WebSphere Edge Server for access to the management servers

� Separation of load for database access, as the database server is now a separate, shared machine

� Better reliability because if one of the management servers fails, the other must still be able to process management request

5.4.2 Deploying the management agent

Management agent deployment for a large environment requires the following planning:

� Naming convention

Each agent must be easily identifiable by using a program to understand its role.

� Silent installation

Automated, non-interactive installation is a better option to deploy the agent to hundreds or thousands of agents.

WebSphere Node Deployment Cluster

WebSphere Application Server

DB2

WebSphere Application Server

WebSphere Edge Server


� Communication requirement

Each agent must communicate to the management server. If there is a firewall, create a Store and Forward agent to minimize the holes in the firewall rule.

� Mass automated installation

Perform this by using a software distribution or provisioning solution such as IBM Tivoli Configuration Manager.

Perform management agent configuration and component deployment by using a command-line interface. Scripting the deployment facilitates efficient, automated, consistent, and a less error-prone deployment.

5.5 Communication and security considerations

Communication and security issues are vital in the inter-networked world we live in. Applications and their management infrastructure must be secured to protect resources from unauthorized sources. This section discusses some of the planning considerations pertinent to this area:

� 5.5.1, “Communication security” on page 100� 5.5.2, “Firewall and port considerations” on page 101

5.5.1 Communication security

Communication security relates to the confidentiality of the information transmitted over the network. Management information that is used by IBM Tivoli Composite Application Manager products may contain details about the application processing internals. This requires that the content of the management information be secured from unauthorized access.

WebSphere securityWebSphere security plays a significant role in a large-scale implementation. In some cases, WebSphere security is not enabled during the test phase of an implementation, but during the production environment. This requires additional considerations such as the WebSphere user having the appropriate permissions to, for example, issue wsadmin commands.

The configuration of data collectors involves the use of Java Command Language (JACL) scripts, and can fail when there is a permission or rights problem.


If any of the application servers on which the data collectors are installed has WebSphere security enabled, the entire ITCAM for WebSphere environment must have it enabled as well. This includes WebSphere security being enabled on the ITCAM for WebSphere managing server.

Secure Sockets Layer communicationSecure communication between the managing server and the data collector is a viable option if there is a requirement for data to be encrypted during transmission. Using Secure Sockets Layer (SSL) facilitates secure data transmission from the data collector to the managing server, and must appease corporate security requirements, if necessary. Additional configuration must be carried out on the managing server and the data collector when enabling the SSL setup. A certificate key generator is included with the product. This key generator provides the facility to use custom-generated keys.

A best practice is to complete the default installation of the managing server and the data collector and then enable SSL for both. This isolates problems (that is, whether the problem is caused by the basic installation or the SSL configuration).

5.5.2 Firewall and port considerations

Firewall and port issues can arise when the data collectors are on a different site, location, or subnet from the managing server. Problems such as name resolution can occur if the Domain Name System (DNS) is not set up correctly on either the managing server or the data collector. Routing problems may occur if the IP addresses used belong to different subnets. The entire network environment must be looked at to determine where a firewall, router, or bridge can exist.

Figure 5-3 shows the communication port requirements for ITCAM for Response Time Tracking.

Figure 5-3 Communication port requirements

The management server running WebSphere Application Server listens at the listening port for a Hypertext Transfer Protocol (HTTP) request and for a Simple Object Access Protocol (SOAP) request. The management agent talks either directly to these ports or in a consolidated manner through the Store and Forward agent. The Store and Forward agent acts as the proxy agent to

Management server

Store and ForwardAgent

Management agent

Store and ForwardAgent

HTTP port SnF port SnF port


consolidate communication from the management agent to the management server.

The Store and Forward agent can be cascaded to facilitate communication through multiple firewalls. This facility is provided, as some agents may be located across the Internet to monitor the response time from a remote agent.

5.6 Reliability and high availability

This section discusses reliability issues such as those relating to failover and disaster recovery.

5.6.1 Failover and fault tolerance

Clustering implementation for the ITCAM for Response Time Tracking management server facilitates a more reliable management of the management servers to allow one of the servers to take care of the entire management load when the other server fails. The clustering implementation uses WebSphere Edge Server load balancing to ensure that both the management servers are used.

The WebSphere Edge Server component automatically redirects traffic to the available machine. In the ITCAM for Response Time Tracking management server, there is no indication of the primary server and the secondary server. They all perform an equal role in a WebSphere cluster.

5.6.2 Disaster recovery

There are three areas where backup is necessary for disaster recovery, with regard to the IBM Tivoli Composite Application Manager server:

� Database

Database backup must be performed regularly in order to collect the most up-to-date information. Use the database utility function to perform the backup function.

� WebSphere Application Server configuration for the server and the agent or the data collector

This must be backed up so that it can be restored in a disaster recovery scenario.


� IBM Tivoli Composite Application Manager servers

These must also be physically considered to facilitate recovery at the disaster recovery site.



Chapter 6. Installing ITCAM for Response Time Tracking

This chapter provides information about the installation and deployment procedure for ITCAM for Response Time Tracking. This chapter discusses the following topics:

� 6.1, “Clustering the management server” on page 106� 6.2, “Deploying the management resources” on page 129� 6.3, “Setting up Secure Sockets Layer certificates” on page 133

6


6.1 Clustering the management server

The clustering action of installing the ITCAM for Response Time Tracking management server is performed using a WebSphere Network Deployment cluster. The database must be located in a remote machine to overload the processing and eliminate dependency on either of the machines. This database may also run on a database 2 (DB2) cluster solution. The overall installation requirement is described in Figure 6-1.

Figure 6-1 Installation requirement

Communication from the management agents and the Web console must take place through the WebSphere Edge Server, which distributes the load for all the management servers. This section discusses the following topics:

� 6.1.1, “Preparing the operating system” on page 106� 6.1.2, “Installing the database” on page 107� 6.1.3, “Installing WebSphere Application Server” on page 109� 6.1.4, “Installing WebSphere Load Balancer” on page 118� 6.1.5, “Installing the management server” on page 124

6.1.1 Preparing the operating system

Operating system preparation includes the following tasks:

� Creating a user ID to own the ITCAM for Response Time Tracking server. We create a user called itcamusr.

� Modifying the following operating system (OS) parameters:

– Kernel parameters for Hewlett-Packard UNIX (HP-UX) or Solaris

See IBM Tivoli Composite Application Manager for Response Time Tracking Installation and Configuration Guide, GC32-9482.

– User right assignment for Windows

The ITCAM for Response Time Tracking user requires Act as part of the operating system and Log on as service rights.

Database server

DB2 Universal Database server

Management server

WebSphere Application Server Node DeploymentITCAM for RTT clustering

server

Edge server

WebSphere Edge Component

Load balancer


� Allocating the file systems that are required for:

– DB2 product

/usr/lpp/db2 or C:\Program Files\IBM\SQLLIB

– DB2 instance

/home/db2inst1 or C:\DB2

– WebSphere Application Server:

/opt/IBM/WebSphere or C:\Program Files\IBM\WebSphere

– Tivoli shared path

/var/IBM/tivoli/common or C:\Program Files\IBM\tivoli\common

– Product path

/opt/IBM/itcam or C:\Program Files\IBM\itcam

6.1.2 Installing the database

This section discusses the preparations for using a DB2 database. We use IBM DB2 Universal Database Enterprise Server Edition V8.2 with Fix Pack Level 5.

The DB2 database takes up space on the current installed drive. Make that sure you have enough space to accommodate the ITCAM for Response Time Tracking database.

Chapter 6. Installing ITCAM for Response Time Tracking 107

The databse installation steps are:

1. Launch the DB2 installation launch pad and select Install Product. In the DB2 Setup Wizard window that appears (Figure 6-2), we install the base DB2 server without data warehousing by selecting Typical.

Figure 6-2 DB2 options

2. We install with mostly default options. The instance user ID that we create is db2admin.

3. ITCAM for Response Time Tracking comes with a script to create the database schema. Use this script to preinstall the database. Optionally, you can also have the wizard create the database. This script is in the installation compact-disc read-only memory (CD-ROM) in the disk1/utilities/<dbversion> directory. The DB2 creation script is called crtdb_db2. The format for running the script is as follows:

crtdb_db2 version dbname dbuser dbpasswd country_code create_db was_user

In this syntax:

– version specifies the DB2 version, such as 8.1 or 8.2.

– dbname is the name of the database being created, such as ITCAMRTT.


– dbuser specifies the user name for the DB2 instance owner.

– dbpasswd specifies the password for dbuser.

– country_code is the locale name as understood by DB2.

– create_db indicates whether to create a new database, Y or N.

– was_user is the user that the WebSphere Application Server is running under.

In our environment, we issue the following command:

crtdb_db2 8.2 ITCAMRTT db2admin xxxxxxxx US Y itcamusr

4. ITCAM for Response Time Tracking management server connects directly to the database using Java Database Connectivity (JDBC) type 4. Cataloging the database locally on the ITCAM for Response Time Tracking management server is not required if the database is installed remotely.

6.1.3 Installing WebSphere Application Server

We use WebSphere Application Server V6.0.2 and install the network deployment configuration. The installation first creates the network deployment manager, dmgr instance. Individual application servers are then installed using the application server profile. We use the default AppSrv01 profile to connect to the deployment manager. Any of the machines can be the deployment manager, which is used mainly to administer the cluster and not an operational runtime tool.

Note: DB2 supports only eight characters for the database name.

Important: You must have the proper DB2 environment to issue this command. On Windows, run this under the DB2 command window. On UNIX platforms, source the db2profile under the DB2 instance owner home directory.


The WebSphere Application Server installation process is:

1. Launch the WebSphere Application Server Version 6.0 Network Deployment edition installation wizard, which installs the base WebSphere code. We install WebSphere Application Server in C:\IBM\WebSphere\AppServer (Figure 6-3).

Figure 6-3 Installation path for WebSphere Application Server


2. As can be seen in Figure 6-4, we do not install the sample applications. The Javadocs are optional because they do not create additional processing requirements. Click Next.

Figure 6-4 WebSphere Application Server components


3. The WebSphere Application Server installation wizard shown in Figure 6-4 on page 111 facilitates the launch of the profile creation wizard. Select Create an Application Server profile, as shown in Figure 6-5, and click Next.

Figure 6-5 Profile creation

4. After the installation concludes, launch the profile creation wizard again. On Windows, start this by using the following command:

$WAS_HOME\bin\ProfileCreator\pctWindows

Alternately, use the wasprofile command to create a deployment manager profile to host the deployment manager.

Note: We use default values for ports and other options.

Note: The deployment manager profile must only be created in the deployment manager node.


5. We create a standard deployment manager profile Dmgr01 from the Profile creation wizard (Figure 6-6).

Figure 6-6 Application server profile

6. Install any necessary WebSphere Application Server fix packs. In our environment, we install Refresh Pack 2 for WebSphere Application Server V6.0.2. Copy the updateinstaller directory into the WebSphere directory. The refresh pack must be uncompressed into the WebSphere home directory and launched from there.

When applying WebSphere fix packs on Windows servers, Java may not be found even after sourcing the WebSphere Application Server environment with setupCmdLine.bat. In this case, Java Virtual Machine (JVM) must be indicated manually using the following command:

update -is:javahome %JAVA_HOME%

Note: Download the WebSphere fix packs or refresh packs from the following URL:

ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpacks/<was_version>/<refreshpack or fixpack>/<Platform>

Our code is downloaded from the following URL:

ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpacks/was60/refreshpack2/Windows



You may have to enclose JAVA_HOME within quotes if you have a space in the installation path, such as program files (Figure 6-7).

Figure 6-7 Updating WebSphere Application Server

7. Start federating the application servers to the deployment manager cluster. Perform this from the deployment manager administrative console:

http://dmgr_host:9061/ibm/console

8. In the administrative console, select System Administration → Nodes. Click Add Node, as shown in Figure 6-8.

Figure 6-8 Federating application server

C:\Program Files\ibm\WebSphere\AppServer\updateinstaller>echo %JAVA_HOME%C:\Program Files\ibm\WebSphere\AppServer\java

C:\Program Files\ibm\WebSphere\AppServer\updateinstaller>update -is:javahome "%JAVA_HOME%"


9. The add node wizard guides you to connect to the appropriate node, as shown in Figure 6-9. The node wizard connects to the administration server to collect information and create the node agent instance. The node agent talks with the deployment manager to perform administrative tasks for the deployment manager. It also includes the server1 instance into the deployment manager cluster.

Figure 6-9 Add node wizard


10.Federate all the application server instances that you want to include in the ITCAM for Response Time Tracking clustering server. After the nodes have been federated, create a cluster. Outage from a single application server results in the load being forwarded to other members of the cluster. A cluster is created from the Administrative Console → System Administration → Servers → Cluster menu. Click New, as shown in Figure 6-10.

Figure 6-10 Cluster list


11.Create the cluster, as shown in Figure 6-11, for defining the name and its members. At the completion of each step, click Next. Click Finish when done. Save the WebSphere configuration.

Figure 6-11 Cluster creation

12.The WebSphere cluster must be authenticated uniformly. To perform this action, activate global security and use an external Lightweight Directory Access Protocol (LDAP) server for authentication. This paper does not discuss this, as implementation of different LDAP servers differs considerably.

Note: LDAP is required, as local OS authentication cannot span a server for a UNIX or a Linux machine. If you are using a Windows domain user authentication, you can use the Windows domain users for the cluster security.


13.Install the IBM Hypertext Transfer Protocol (HTTP) server. Install this server from the installation media of the WebSphere Application Server in the IHS subdirectory. Install the IBM HTTP Server using the default options in the C:\IHS directory. You must also install the IBM HTTP Server plug-in to the IBM HTTP Server system.

14.Generate the IBM HTTP Server plug-in. The plug-in generates the configuration file from the connected WebSphere server. To recreate the plug-in configuration file, run the configurewebserver1 command from the bin subdirectory of the plug-in installation directory.

15.Modify the IBM HTTP Server configuration file (httpd.conf) for additional ports to receive communication from the management agent, typically 9082. Add the following directive to the httpd.conf file:

Listen 9082

The WebSphere Application Server is ready.

6.1.4 Installing WebSphere Load Balancer

The WebSphere Load Balancer is a component of the WebSphere Edge Component. The WebSphere Load Balancer is installed using the WebSphere Edge Component installation media. This is the same media that you used for installing the WebSphere Caching Proxy for the Store and Forward agent. We use the dispatcher component of the Load Balancer. Conceptually, the dispatcher component works as shown in Figure 6-12.

Figure 6-12 Load Balancer diagram

The Load Balancer requires two Internet Protocol (IP) addresses. The physical address, called the non-forwarding address, is used to access the box directly.

srv150

srv151

srv105 (non forwarding)rtt

request


However, the Load Balancer uses a second address that is an alias to the physical IP address. Any request to this address is forwarded to either of the backend servers.

Perform the following tasks to install the WebSphere Load Balancer:

1. The WebSphere Load Balancer requires a loopback address to be defined as a forwarding address.

– In AIX, issue the following command:

ifconfig lo0 alias address netmask mask

– In the Windows environment, add the Microsoft loopback adapter device and assign the cluster IP address.

2. On the welcome panel (Figure 6-13), click Next.

Figure 6-13 Welcome dialog


3. In the license agreement dialog (Figure 6-14), accept the license agreement by clicking Yes.

Figure 6-14 License agreement

4. The installer then checks the prerequisites. If the check is successful, it then displays the language selection panel (Figure 6-15).

Figure 6-15 Language selection


5. Figure 6-16 shows type of installation. We use a custom installation and click Next. Note that we use content-based routing.

Figure 6-16 Select the installation components


6. The next panel (Figure 6-17) shows the summary of selections made and starts the installation upon clicking Next.

Figure 6-17 Installation summary dialog

7. Finally, the installation result is shown Figure 6-18. Click Finish to complete the installation and exit the installation wizard.

Figure 6-18 Installation summary and result


8. Start our content-based routing server. We are running a Linux server. Run the cbrserver command to start the server.

9. The interaction for defining the load balancer is using a command line interface. This is performed using the cbrcontrol command.

a. Start the executor:

cbrcontrol executor start

b. Define the cluster:

cbrcontrol cluster add rtt.itsc.austin.ibm.com

c. Define the ports that the cluster listens to:

cbrcontrol port add rtt.itsc.austin.ibm.com:9081

d. Select the backend servers that the Load Balancer will try to balance. Add all the WebSphere cluster members into this definition:

cbrcontrol server add rtt.itsc.austin.ibm.com:9081:srv150.itsc.austin.ibm.comcbrcontrol server add rtt.itsc.austin.ibm.com:9081:srv150.itsc.austin.ibm.com

e. Add the rule based on which content base routing would route/load balance the requests. Right-click the port name and select Add Rule Content:

cbrcontrol rule add rtt.itsc.austin.ibm.com:9081:defRule type content pattern uri=*/tmtpUI/*

f. Associate the servers in the cluster to the rule defined in previous step e:

cbrcontrol rule useserver rtt.itsc.austin.ibm.com:9081:defRule srv150.itsc.austin.ibm.comcbrcontrol rule useserver rtt.itsc.austin.ibm.com:9081:defRule srv151.itsc.austin.ibm.com

g. Start the manager:

cbrcontrol manager start

Note: WebSphere Load Balancer assumes that you have the Java Runtime Environment (JRE) javaw executable in the system path. If you do not have the javaw in the system path, add this before proceeding.

Note: You can choose the type of rule depending on your need. For this illustration we use content type as the type of rule.


h. Start the advisor on the port configured in step c on page 123:

cbrcontrol advisor start http 9081

10.The result of the configuration is illustrated in Example 6-1.

Example 6-1 Checking result of configuration

[root@srv105 logs]# cbrcontrol server report rtt.itsc.austin.ibm.com:9081Cluster: rtt.itsc.austin.ibm.com Port: 9081-------------------------------------------------------------| Server | Map port | Total | CPS |-------------------------------------------------------------| srv151.itsc.austin.ibm.com || | 9081 | 0 | 0 || srv150.itsc.austin.ibm.com || | 9081 | 0 | 0 |-------------------------------------------------------------

Your WebSphere Load Balancer is now ready.

6.1.5 Installing the management server

In a clustering environment, the management server is installed using a command-line script. In WebSphere, the installation process requires access to the DB2 Java Database Connectivity (JDBC) driver. Copy the Java subdirectory of the DB2 installation or install the DB2 client locally for this.

Before starting the installation process, get all the following installation parameters:

� WebSphere Application Server information:

– Installation path

– Administrator user and password

– Profile name, cell name, node name, and server name

– Admin console port

– Simple Object Access Protocol (SOAP) connector port

Note: The above-mentioned steps can also be performed by GUI. Run lbadmin to start the content-based routing GUI. For more information about Load Balancer and various configuration commands refer to Load Balancer Administration Guide, GC31-6858.


– (Optional) Lightweight Directory Access Protocol (LDAP) user name and password

– (Optional) Secure Sockets Layer (SSL) information

• Client Key file name and password• Client Trust file name and password

� Database information:

– Database server host name– Database server port number– Database name for DB2 or system ID for Oracle– Database user ID and password– (DB2 only) JDBC driver path

The installation of the ITCAM for Response Time Tracking clustering management server uses a special CD-ROM image. It does not include the installation wizard. Installation is performed interactively using an installation script. The installation must have both the cluster CD-ROM and the common CD-ROM images copied to the same installation directory. Example 6-2 shows a sample copy command for UNIX platforms.

Example 6-2 Sample copy command for UNIX platforms

cp -r RTTMSclustr/shareddir/* <shared dir> cp -r RTTMScommon/disk2/shareddir/* <shared dir>

Example 6-3 shows a sample copy command for Windows.

Example 6-3 Sample copy command for Windows

xcopy /S/I RTTMSclustr\shareddir <shared dir>xcopy /S/I RTTMScommon\disk2\shareddir <shared dir>

The installation script is a Java Command Language (JACL) script that configures and installs tmtpCluster.ear into your cluster. Invoke the JACL script using the wsadmin command from the network deployment server. The installation command is:

wsadmin –username <userid> -password <password> –f itcamfrtt.jacl all

The existing options for itcamfrtt.jacl are:

� all: This performs all the installation procedures such as install and configure.� configure: This configures the ear application only.� install: This installs the ear application only.� unconfigure: This unconfigures the ear application.� uninstall: This uninstalls the ear application.


� removeall: This removes ear application and configuration.

Detailed instructions for running the JACL script are provided in ITCAM for RTT: Installing a Management Server in a WebSphere Cluster Environment, SC32-1804. Example 6-4 shows our installation result.

Example 6-4 Running itcamfrtt.jacl

------------------------------------------------ ITCAMfRTT Install/Configuration Script

Operation: all Silent: false------------------------------------------------

Have the JVM heap settings been verified? (yes|no) [yes]:Is global security enabled? (yes|no) [yes]:Have all nodes been federated and network connectivity verified? (yes|no) [yes]Please enter the cluster name [ITCAMfRTTCluster]:Available Nodes: srv150Node01 srv151Node01Select the desired node [srv150Node01]:Please enter the cluster member name [ITCAMfRTTServer1]:Please enter the path to the shared directory [/mnt/clustershare]:/rttshareIs this a Windows node (yes|no) [no]:Current Cluster Nodes and Members: srv150Node01 - ITCAMfRTTServer1

Add more cluster members (yes|no) [yes]:Available Nodes: srv150Node01 srv151Node01Select the desired node [srv150Node01]: srv151Node01Please enter the cluster member name [ITCAMfRTTServer2]:Please enter the path to the shared directory [/rttshare]:Is this a Windows node (yes|no) [no]:

Current Cluster Nodes and Members: srv150Node01 - ITCAMfRTTServer1 srv151Node01 - ITCAMfRTTServer2

Add more cluster members (yes|no) [yes]: no

--------------------------------------------------Cluster information obtained...--------------------------------------------------

Please enter the WebServer/LoadBalancer port [81]:


Please enter the HTTP transport port for ITCAMfRTTServer1 [9081]:Please enter the HTTP transport port for ITCAMfRTTServer2 [9081]:

Collecting Database/Datasource Information------------------------------------------------

Select the backend database type (db2|oracle|oracle10G) [db2]:Please enter the database driver path for ITCAMfRTTServer1 [c:/sqllib/java]:/home/db2inst1/sqllib/javaPlease enter the database driver path for ITCAMfRTTServer2 [/home/db2inst1/sqllib/java]:Please enter the database name (location) [ITCAMRTT]: tmtpdbPlease enter the DB2 database hostname [localhost]: lima.itsc.austin.ibm.comPlease enter the DB2 database port number [50000]: Please enter the database username [db2admin]: db2inst1Please enter the database password [password]: XXXXXXXX------------------------------------------------- Collecting Security Information for JMS-------------------------------------------------

Please enter the system username [LocalOSUserID]: wasadminPlease enter the system password [password]: XXXXXXXX------------------------------------------------- Collecting Information for Install-------------------------------------------------

Please enter the full path name of the EAR file [tmtpCluster.ear]:/rttclust/RTTMSclustr/ear/DB2UDB_V81/tmtpCluster.ear

------------------------------------------------ Configuring Cluster and Cluster Members Scope: srv150Cell01(cells/srv150Cell01|cell.xml#Cell_1)------------------------------------------------ Creating Cluster ITCAMfRTTCluster... ITCAMfRTTCluster created successfully!Creating Cluster Member ITCAMfRTTServer1... ITCAMfRTTServer1 created successfully!Enabling SIB Service on ITCAMfRTTServer1... SIB Service enabled successfully!Creating Cluster Member ITCAMfRTTServer2... ITCAMfRTTServer2 created successfully!Enabling SIB Service on ITCAMfRTTServer2... SIB Service enabled successfully!

------------------------------------------------ Cluster Configuration Completed!!!------------------------------------------------

------------------------------------------------ Installing ITMTP-Cluster-------------------------------------------------


Installing application {ITMTP-Cluster}... Application Name: ITMTP-Cluster Ear file: /rttclust/RTTMSclustr/ear/DB2UDB_V81/tmtpCluster.ear Target Cluster: ITCAMfRTTCluster Authentication Alias: TMTPDataSourceAuthDataStarting application install...

------------------------------------------------ ITMTP-Cluster Installation Completed!!!------------------------------------------------

Do you wish to restart the cluster (yes|no) [yes]:

Starting ITCAMfRTTCluster...Ripple start of ITCAMfRTTCluster initiated.

6.1.6 Checking the configuration of the RTT cluster application

After the installation completes, log on to the WebSphere server administration console and check the Map virtual hosts for Web modules setting of RTT cluster application. See whether they have been mapped to the itcamfrtt_host correctly. Change the virtual host to itcamfrtt_host if they are not currently mapped to it. Select OK and restart the application. See Figure 6-19.

Figure 6-19 Virtual hosts definition of RTT cluster application


6.2 Deploying the management resources

The management agent in ITCAM for Response Time Tracking is a neutral component. Management components are deployed after the installation is complete, using either the Web interface or the command-line interface (CLI).


� 6.2.1, “Silent installation of the management agent” on page 129� 6.2.2, “Command-line interface for management components” on page 130

6.2.1 Silent installation of the management agent

Installing the management agent by using the silent installation method allows unattended installation of a large number of agents to be performed. The silent installation of the ITCAM for Response Time Tracking management agent is performed using the following command:

setup_MA -silent -options <responsefile>

The response file sample is provided in the responsefiles subdirectory of the installation media. The options in the response file are divided into:

� License agreement setting for management agents

-G licenseAccepted=true

� Installation directory for management agents

-P ma.installLocation="C:\IBM\itcam\RTT\MA”

� Installation settings

-W msConnection.offline="true"-W msConnection.hostName="fully_qualified_host_name" -W msConnection.userName="tmtpuser" -W msConnection.password="tmtppassword" -W msConnection.sslValue="[true | false]" -W msConnection.portNumber="9446" -W msConnection.maPort="1976" (This is the default value.) -W msConnection.epKeyStore="c:/tmtp/agent.jks" -W msConnection.epKeyPass="changeit" -W serviceUser.user="TMTPAgent" (Windows only) -W serviceUser.password="tivoli" (Windows only)

� Global response settings

-G replaceExistingResponse="yesToAll | yes | noToAll | no" -G replaceNewerResponse="yesToAll | yes | noToAll | no" -G removeExistingResponse="yesToAll | yes | noToAll | no"


-G removeModifiedResponse="yesToAll | yes | noToAll | no"

� Log settings

[-W logSettings.level="trace_level"] [-W logSettings.consoleOut="true | false"]

A sample response file MA.opt is under the management agent installation file directory. You can use it to proceed the silent installation by modifying the default setting in the sample response file.

6.2.2 Command-line interface for management components

To deploy the management agent, use the CLI. The CLI for ITCAM for Response Time Tracking is provided in the MS_Home/downloads/cli directory. This directory, which can be copied to any machine, provides the functionality from a command line. The CLI is useful for providing a scripted deployment of ITCAM for Response Time Tracking.

Copy the entire directory from MS_Home/downloads/cli. If you are running a Windows system, copy w32util.dll to %WINSYSDIR%\w32util.dll. Modify tmtpcli.bat or tmtpcli.sh to define CLI_HOME to the path where you put the directory. In our environment, we run the CLI from the management server, which is installed in C:\IBM\itcam\RTT\MS. Therefore, CLI_HOME is set to:

CLI_HOME=C:\IBM\itcam\RTT\MS\downloads\cli

You must have JAVA_HOME defined to the JRE 1.4 environment. With the WebSphere Application Server, you can run setupCmdLine.bat or setupCmdLine.sh to establish your environment.

Initially, generate the tmtpcli.properties file using the CreateCliProperties command. Depending on whether you activate global security, you may have to use Hypertext Transfer Protocol Secure (HTTPS) protocol and specify the KeyStore file and its password. The Universal Resource Locator (URL) for the management server is the primary URL for the application server, and not the Web console URL. In our environment we issue the following command:

tmtpcli -CreateCliProperties –MsUrl http://khartoum:9081/ –UserName itcamrtt –Password xxxxxxx

See IBM Tivoli Composite Application Manager for Transaction Tracking Reference Guide, SC32-9485.


Deploy the monitoring components using the CLI. Each management component type requires different invocation parameters.

� Deploying quality of service component:

tmtpcli -DeployQoS –AgentName laredo –ProxyHostname laredo.itsc.austin.ibm.com –ProxyPort 7081 –OriginHostname laredo.itsc.austin.ibm.com –OriginPort 9081 -Wait

� Deploying synthetic transaction investigator component:

tmtpcli -DeploySti –AgentName geneva –Wait

� Deploying generic Windows component:

tmtpcli –DeployGenWin –AgentName ser1 –ProjectName trader001 –ProjectPassword pwd –ProjectUserId Administrator –Wait

� Deploying Application Resource Measurement (ARM) custom application component:

tmtpcli –DeployCustomApp –AgentName laredo –ArmAppName Trader

� Deploying Java 2 Platform, Enterprise Edition (J2EE) component:

tmtpcli –DeployJ2ee –AgentName laredo –ServerType WebSphere –Version 6.0 –ServerHome C:\IBM\WebSphere\AppServer –ServerName laredo1 –ProfileName AppSrv01 –AutoRestart –NetworkDeploy –DeployManagerName laredo.itsc.austin.ibm.com –AdminPort 9060 –Wait

6.2.3 Defining management resources

Management resource definition for a large-scale implementation is cumbersome if performed using the Web console. Define these management resources, such as agent groups, policy groups, operators, and schedules, by using the command-line, too:

� Defining policy groups:

tmtpcli -CreatePolicyGroup <policy group name>

� Defining agent groups:

tmtpcli -CreateAgentGroup <agent group name> -Agents <comma separated agent list>

� Defining users:

tmtpcli -CreateUser <user>

� Defining schedules:

tmtpcli -CreateSchedule <xml filename>


The schedule itself must be defined in an eXtensible Markup Language (XML) file. Example 6-5 shows the schedule XML file definition.

Example 6-5 Schedule XML file

<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <ns1:ScheduleData xmlns:ns1="urn:TMTP"> <arg0 href="#every5min"/>

</ns1:ScheduleData> <multiRef id="every5min" soapenc:root="0" soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xsi:type="ns2:com.ibm.tivoli.transperf.core.ejb.common.ScheduleData" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:ns2="typeNS"> <created xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</created> <creator xsi:type="xsd:string"></creator> <deleted xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</deleted> <deletor xsi:type="xsd:string"></deletor> <description xsi:type="xsd:string">Changed temporally by Danilo</description> <end xsi:type="xsd:int">0</end> <endDateTime xsi:type="xsd:dateTime">1970-01-01T00:00:00.000Z</endDateTime> <endTime xsi:type="xsd:string">00:00</endTime> <isDeleted xsi:type="xsd:boolean">false</isDeleted> <iteration xsi:type="xsd:int">5</iteration> <iterationType xsi:type="xsd:int">0</iterationType> <iterationUnit xsi:type="xsd:string">MINUTES</iterationUnit> <lastUpdated xsi:type="xsd:dateTime">2005-10-14T15:37:07.672Z</lastUpdated> <lastUpdator xsi:type="xsd:string"></lastUpdator> <name xsi:type="xsd:string">Every_5_5min</name> <objectVersion xsi:type="xsd:int">1</objectVersion> <run xsi:type="xsd:int">1</run> <runFri xsi:type="xsd:boolean">false</runFri> <runMon xsi:type="xsd:boolean">false</runMon> <runSat xsi:type="xsd:boolean">false</runSat> <runSun xsi:type="xsd:boolean">false</runSun> <runThu xsi:type="xsd:boolean">false</runThu> <runTues xsi:type="xsd:boolean">false</runTues>


<runWed xsi:type="xsd:boolean">false</runWed> <start xsi:type="xsd:int">0</start> <startDateTime xsi:type="xsd:dateTime">2005-10-14T21:36:00.656Z</startDateTime> <startTime xsi:type="xsd:string">00:00</startTime> <timeZone xsi:type="xsd:string">US/Central</timeZone> <type xsi:type="xsd:string">PLAYBACK_SCHEDULE</type> <useDST xsi:type="xsd:boolean">true</useDST> <uuid xsi:type="xsd:int">221</uuid> </multiRef>

6.3 Setting up Secure Sockets Layer certificates

To secure communication between various components of ITCAM for Response Time Tracking, activate SSL. ITCAM for Response Time Tracking provides a default set of certificates to let you implement SSL encryption using the default setting.

The default setting has the communication encrypted, and provides the quickest way to implement SSL. It also lets you test the function with the SSL enabled, but without the risk of potential authentication problems with custom certificates. Anybody with the installation image can extract the certificate and unscramble the information.


� 6.3.1, “Secure Sockets Layer for ITCAM for Response Time Tracking” on page 133

� 6.3.2, “Working with custom certificates” on page 134

6.3.1 Secure Sockets Layer for ITCAM for Response Time Tracking

Secure Sockets Layer in ITCAM for Response Time Tracking is handled using the Java Secure Socket Extension (JSSE). In this configuration, SSL certificates and keys are stored in a jks (Java key store) file. Management of these jks files is performed using the IBM GSKIT.

Note: The IBM GSKIT is different from the keytool command that is supplied in the Java environment.


Perform Secure Sockets Layer activation directly with the installation wizard or the silent installation options when you install ITCAM for Response Time Tracking. The default port assignments for SSL communication are:

� Management server port: 9446

� Store and Forward Agent port: 443 for the management agent or downstream Store and Forward Agent, and 1976 for the management server or upstream Store and Forward Agent

To activate SSL communication after you have ITCAM for Response Time Tracking running without SSL, perform the following tasks:

1. Activate SSL on the management server.

The SSL setting on the management server is stored in the WebSphere Application Server settings. Changing to SSL requires modification in the following paths:

– Select Security → SSL and modify host/TMTPSettings_Without_ClientAuth and host/TMTPSettings_With_ClientAuth to specify the jks file and its password.

– Select Environment → Manage WebSphere Variables and modify the TMTP_KEYSTORE and TMTP_KEYPASS variables. The TMTP_KEYPASS password contains an encrypted password that can be created by using pass_util.jar.

2. Activate SSL on the Store and Forward agent.

The Store and Forward agent uses WebSphere Caching Proxy. SSL modification requires changes in ITCAM for Response Time Tracking and WebSphere Caching Proxy.

– Modify the SnF/config/endpoint.properties file and the change endpoint.keystore and endpoint.keypass parameters.

– Modify etc/cp/en_US/ibmProxy.conf and the KeyRing and KeyRingStash variables.

3. Activate SSL on the management agent.

Modify the config/endpoint.properties file and change the endpoint.keystore and endpoint.keypass parameters.

6.3.2 Working with custom certificates

You may want to implement your own set of custom certificates. The basic operation of certificates lets you define your identity certificate to encrypt the


information that you send and lets everyone you trust have a public key to decrypt your information.

Although it is possible to define a separate certificate identity for every machine that communicates, the sheer number of machines and operation changes involved in managing them are prohibitive, especially in a large environment. We implemented the following set of certificates:

� Management server certificate

� Store and Forward agent certificate

� Management agent certificate that communicates directly to the management server

� Management agent certificate that communicates through the Store and Forward agent

Configuration involves the following steps:

1. The jks files are built and managed using the iKeyMan command. The command resides in the WebSphere Application Server’s bin directory. Create the following jks files:

– For the management server: jks file for the management server. We call this file prodms.jks.

– For the Store and Forward agent: jks file for the Store and Forward agent. We call this file prodsnf.jks.

– For the management agent: jks file for the management agent. We call this file prodma.jks.

Besides these files, the iKeyMan tool also generates a password key file with the same name and extension of arm.

2. Run the gsk7ikm command from the WebSphere Application Server’s gskit7install directory. This is used to create a key database (kdb file) for the Store and Forward agent, and possibly the quality of service proxy. The quality of service SSL is not discussed in this paper as it relates to the application requirement.

This task produces a kdb file, a stash file (sth) (where the kdb password is stored), and an arm file.

3. The jks and kdb file sets must be installed and activated on the ITCAM for Response Time Tracking, as discussed in 6.3.1, “Secure Sockets Layer for ITCAM for Response Time Tracking” on page 133.



Chapter 7. Maintenance of ITCAM for Response Time Tracking

This chapter discusses the issues relating to the maintenance of ITCAM for Response Time Tracking in a large-scale environment. This chapter discusses the following topics:

� 7.1, “Operational issues pertaining to a large environment” on page 138� 7.2, “Performance and availability of management server” on page 138� 7.3, “ITCAM for Response Time Tracking files” on page 140� 7.4, “Performing product maintenance” on page 142

7


7.1 Operational issues pertaining to a large environment

This chapter discusses operational issues regarding a large-scale ITCAM for Response Time Tracking environment, and focuses on the issues that arise largely because of the environment size. The topics discussed in this chapter are:

� 7.2, “Performance and availability of management server” on page 138

As the size of the environment grows, even a slight inefficiency can cause a major problem. A large-scale environment requires the performance and availability of the managing server to be soundly maintained.

� 7.3, “ITCAM for Response Time Tracking files” on page 140

This is a necessity. This section discusses specific backup and recovery considerations and maintenance of files.

� 7.4, “Performing product maintenance” on page 142

This discusses the implementation of WebSphere, Database 2 (DB2), and ITCAM for Response Time Tracking patches. Typically, with the number of data collectors involved, this can be a huge task.

7.2 Performance and availability of management server

Performance and availability of the ITCAM for Response Time Tracking management server is related mainly to the IBM WebSphere Application Server that hosts it and the availability of its database. This section discusses the following topics:

� 7.2.1, “Performance of WebSphere Application Server” on page 138� 7.2.2, “Database maintenance” on page 139

7.2.1 Performance of WebSphere Application Server

In ITCAM for Response Time Tracking, the management server runs entirely on the WebSphere Application Server process. The performance of the management server and the Web interface are greatly dependent on the performance of the WebSphere Application Server. This makes the performance tuning of the WebSphere Application Server critical.


The WebSphere Application Server basically runs on a Java Virtual Machine (JVM). The typical performance indicators for a JVM are:

� Adequate processing power from the original machine

The clustering environment ensures that you can distribute the processing power across enough server machines. These machines must be dedicated and must not be running any unnecessary tasks.

� Class loading disk performance

WebSphere boot classes and other jar files from ITCAM for Response Time Tracking must be stored in a contiguous and fast disk. If possible, separate the system paging area with the jar files.

� Adequate memory size for processing

As JVMs use a lot of memory, provide adequate memory for the server. You may want to specify the minimum JVM size from the WebSphere Application Server configuration. You must be able to find the size after running the server for a while. Allocating memory on the fly is expensive.

Another aspect of JVM memory tuning is the garbage collection. The frequency, as well as the garbage collection options, must be tuned to not interfere with the processing and waste critical memory space.

� Network connection to the database engine and from the WebSphere Edge Server

Primary network connection considerations are request and uploading data to the management server, and data to be put into the database. If possible, the cluster and the WebSphere Edge Server must be running, using a separate local network to minimize packet collision with other applications on the Ethernet bus.

7.2.2 Database maintenance

The performance of the management server is also related to database performance, which is in turn related to how the data is stored physically and how well the database manager understands the nature of data. This section is primarily aimed at discussing the use of DB2 as the database manager.

The primary tools to ensure this are the REORG and RUNSTAT tools. Run both of these utilities from the DB2 utilities from the DB2 command line. Following is a description of these tools:

� The REORGCHK utility reads the table information and analyzes the necessity of running reorg. REORGCHK used current statistics or generates its own statistics.

Chapter 7. Maintenance of ITCAM for Response Time Tracking 139

The REORG utility reorganizes the database structure to ensure that data is allocated coherently and in a typical access sequence requested by most queries. The syntax for running REORG is:

DB2 REORG TABLE creator.table_name INDEX primary_index_name

� The RUNSTATS utility collects the statistics of the tables and indexes to ensure that the database manager chooses the most optimized method to get the data requested by a Structured Query Language (SQL) request. The syntax for running RUNSTATS is:

DB2 RUNSTATS ON TABLE creator.table_name WITH <runstat_options>

Typically, you should run REORGCHK for frequently modified tables, and then determine whether you want to run REORG on those tables. After all the modifications are performed, run RUNSTATS to ensure that the statistics are current and the DB2 obtains the most optimal path to the data that an SQL requested.

We also recommend a regular, maybe monthly, maintenance window to REORG all the tables for ITCAM for Response Time Tracking and perform a complete RUNSTATS.

Database performance also relates to the size of the data itself. Regular pruning of the collected data must be performed to ensure database performance. This pruning action must be carried out before any REORG or RUNSTATS utilities are performed.

7.3 ITCAM for Response Time Tracking files

This section discusses the file considerations for ITCAM for Response Time Tracking and provides information about the following topics:

� 7.3.1, “Backup and recovery” on page 140� 7.3.2, “Managing the log files” on page 141

7.3.1 Backup and recovery

The backup and recovery procedure for ITCAM for Response Time Tracking is similar to the backup and recovery procedure for ITCAM for WebSphere discussed in 4.3, “Backup and recovery configuration” on page 83. It consists of server backup, WebSphere Application Server configuration, and DB2 database backup.


An additional requirement for backup may arise from the WebSphere Edge Server. Typical backup of the product installation directories must be adequate.

7.3.2 Managing the log files

Log files in ITCAM for Response Time Tracking are used for problem determination of the product’s working mechanism. Several locations exist for the log files, depending on the components that generate them. The locations are:

� Tivoli common directory

This is where most of the log files are stored. The typical directory names are:

– %ProgramFiles%\ibm\tivoli\common (Windows)– /var/ibm/tivoli/common (UNIX)

The directory under the common directory is typically identified by a three-character product code, such as BWM for ITCAM for Response Time Tracking. In the product directory, the following sub-directories exist:

– First-failure data capture (FFDC) for first failure data capture application

The FFDC directory contains diagnostic information pertaining to the program exceptions that are captured.

– The logs subdirectory of Tivoli common directory

This is typically filled with log files and must be maintained.

� WebSphere log files

WebSphere log files are located on each WebSphere Application Server instance under the path $WAS_HOME/profiles/<profile>/logs/<server>.

As these log files have a maximum size and a predefined number of generations, they cannot exceed a certain size. This path must be maintained and checked for ad hoc logging that may use up the space. We recommend that you use a separate file system to contain the common directory.

Note: WebSphere Application Server backup must be performed on the network deployment engine instead of on individual application servers.


7.4 Performing product maintenance

Various considerations must be taken into account when dealing with the application of product maintenance, fixes, and fix packs in a large environment, as this must be performed remotely, and automated with minimal impact to the production system. This section discusses the following topics:

� 7.4.1, “Getting software updates” on page 142

� 7.4.2, “Updating ITCAM for Response Time Tracking management server” on page 142

� 7.4.3, “Updating ITCAM for Response Time Tracking management agents” on page 143

7.4.1 Getting software updates

Obtain the software updates from the following IBM Web sites:

� For ITCAM for Response Time Tracking:

http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliCompositeApplicationManagerforResponseTimeTracking.html

� For DB2 Universal Database:


� For WebSphere Application Server:


Typically, you do not have to obtain updates for DB2 or WebSphere on the managing server. ITCAM for Response Time Tracking only supports a certain level of these products. However, you may sometimes require a certain software level to fix a problem.

7.4.2 Updating ITCAM for Response Time Tracking management server

The ITCAM for Response Time Tracking management server updates must be performed as a scheduled maintenance. This means that:

� You must have a backup of the management server system.� You must perform an orderly shutdown of the management server.� Users must be notified of the system’s unavailability.

The patch for ITCAM for Response Time Tracking is distributed as a self-extracting executable called 6.0.0.0-TIV-RTT-FP000n_<platform> with the


http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliCompositeApplicationManagerforResponseTimeTracking.html



bin or exe extensions for each supported management server platform. It also provides zip files for cluster-based installation.

Non-cluster installation is performed by launching the executable and following the prompt, while cluster-based implementation is performed by extracting the zip file and running the supplied Java Command Language (JACL) scripts with the wsadmin command. You may have to update the ITCAM for Response Time Tracking database using the supplied script programs. See the corresponding readme files for more information.

Restart the WebSphere Application Server after applying the update.

7.4.3 Updating ITCAM for Response Time Tracking management agents

Perform management agent updates in ITCAM for Response Time Tracking directly from the management server process. Use the automatic agent through the Web console. Select System Administration → Agent Updates. Install the agents manually using the following command:

<MA_BASEDIR>\jre142\jre\bin\java -Dtmtp.user.dir=<MA_BASEDIR> -jar ma_fixpack.jar -silent

Reinstall the Java 2, Enterprise Edition (J2EE) component after updating the agent. Use the Web interface or the command line for this. For a large installation, perform both the steps using the command-line interface, as shown in Example 7-1.

Example 7-1 Updating J2EE components using the command line

tmtpcli -RemoveComponent -AgentName <agent> -ComponentType J2EE_WAS tmtpcli -DeployJ2ee –AgentName <agent> –ServerType WebSphere –Version 6.0 –ServerHome serverHome –ServerName serverName –ProfileName profileName [–AutoRestart] [–IsSecure –AdminUserName <adminUserName> –AdminPassword <adminPassword>] [–NetworkDeploy –DeployManagerName <deployManagerName> –AdminPort <adminPort>] [–Wait]

Restriction: Manually install the management agent for z/OS and OS/400 using a platform interface such as System Modification Program/Extended (SMP/E).



acronyms

AIX Advanced Interactive eXecutive

API Application programming interface

ARM Application Response Measurement

BCM Byte Code Modification

CD-ROM Compact-disc read-only memory

CLI Command-line interface

CPU Central processing unit

DB2 Database 2

EJB Enterprise JavaBeans

ETE™ End-to-end

FFDC First-failure data capture

FMID Function modification identifier

GUI Graphical user interface

HTML Hypertext Markup Language

HTTP Hypertext Transfer Protocol

HTTPS HTTP Secure

IBM International Business Machine Corp.

ITIL Information Technology Infrastructure Library

ITM IBM Tivoli Monitoring

ITSO International Technical Support Organization

J2C Java2 Connectivity

J2EE Java 2 Platform, Enterprise Edition

JAR Java archive

JDBC Java Database Connectivity

Abbreviations and

© Copyright IBM Corp. 2006, 2007. All rights reserved

JMX™ Java Management Extensions

JNI™ Java Native Interface

JRE Java Runtime Environment

JVM Java virtual machine

JVMPI Java Virtual Machine Profiler Interface

JVMTI Java Virtual Machine Tool Interface

LDAP Lightweight Directory Access Protocol

MSC Multiple Systems Coupling

MVS™ Multiple Virtual Storage

PCB Program control block

PDF Portable Document Format

PMI Performance monitor interface

RMI Remote Method Invocation

RTE runtime environment

SNMP Simple Network Management Protocol

SOAP Simple Object Access Protocol

SQL Structured Query Language

SSH Secure SHell

SSL Secure Sockets Layer

STI Synthetic Transaction Investigator

TCP/IP Transmission Control Protocol/Internet Protocol

TEMA Tivoli Enterprise Monitoring Agent

TEMS Tivoli Enterprise Monitoring Server

TEP Tivoli Enterprise Portal

. 145

TEPS Tivoli Enterprise Portal Server

UDB Universal Database

UML Universal Markup Language

URI Universal Resource Identifier

URL Universal Resource Locator

WSDL Web Services Definition Language

XML eXtensible Markup Language


Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper.

IBM RedbooksFor information about ordering these publications, see “How to get IBM Redbooks” on page 149. Note that some of the documents referenced here may be available in softcopy only.

� Installing WebSphere Studio Application Monitor V3.1, SG24-6491

� Unveil Your e-business Transaction Performance with IBM TMTP 5.1, SG24-6912

� WebSphere Studio Application Monitor V3.2 Advanced Usage Guide, SG24-6764

Other publicationsThese publications are also relevant as further information sources:

� ITCAM for Response Time Tracking publications

– IBM Tivoli Composite Application Manager for Response Time Tracking Administrator’s Guide, SC32-9483

– IBM Tivoli Composite Application Manager for Response Time Tracking Checking Performance and Availability, SC32-9484

– IBM Tivoli Composite Application Manager for Response Time Tracking Installation and Configuration Guide, GC32-9482

– IBM Tivoli Composite Application Manager for Response Time Tracking: Installing a Management Server in a WebSphere Cluster Environment, SC32-1804

– IBM Tivoli Composite Application Manager for Response Time Tracking Prerequisites, SC32-9486

– IBM Tivoli Composite Application Manager for Response Time Tracking Problem Determination Guide, SC32-9513


– IBM Tivoli Composite Application Manager for Response Time Tracking Reference Guide, SC32-9485

– IBM Tivoli Composite Application Manager for Response Time Tracking V6.0 Program Directory, GI11-4099

� ITCAM for WebSphere publications

– IBM Tivoli Composite Application Manager for WebSphere Installation and Customization Guide, GC32-9506

– IBM Tivoli Composite Application Manager for WebSphere: Installing and Configuring the Tivoli Enterprise Monitoring Agent, SC32-1801

– IBM Tivoli Composite Application Manager for WebSphere Operator’s Guide, SC32-9508

– IBM Tivoli Composite Application Manager for WebSphere Problem Determination Guide, SC32-9509

– IBM Tivoli Composite Application Manager for WebSphere: Tivoli Enterprise Monitoring Agent Problem Determination Guide, SC32-1800

– IBM Tivoli Composite Application Manager for WebSphere User’s Guide, SC32-9507

Online resourcesThese Web sites and URLs are also relevant as further information sources:

� Microsoft link for InstallShield error

http://support.microsoft.com/default.aspx?scid=kb;en-us;295278

� Microsoft Windows Services for UNIX

http://www.microsoft.com/windows/sfu/default.asp

� Open group Web site for Application Response Time Management

http://www.opengroup.org/arm

� Product prerequisites information







http://support.microsoft.com/default.aspx?scid=kb;en-us;295278

http://www.microsoft.com/windows/sfu/default.asp

http://www.opengroup.org/arm

� Product Web pages

http://www-306.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

http://www-306.ibm.com/software/tivoli/products/composite-application-mgr-websphere/

� WebSphere fix pack links


ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpacks/was60/refreshpack1/Linux

How to get IBM RedbooksYou can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site:

ibm.com/redbooks

Help from IBMIBM Support and downloads

ibm.com/support

IBM Global Services

ibm.com/services

Related publications 149



http://www.ibm.com/support/

http://www.ibm.com/support/

http://www.ibm.com/services/

http://www.ibm.com/services/

http://www-306.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

http://www-306.ibm.com/software/tivoli/products/composite-application-mgr-websphere/



ftp://ftp.software.ibm.com/software/websphere/appserv/support/fixpacks/was60/refreshpack1/Linux




Index

Aapplication management 2

pillar 5application monitor 20

data collector agent 20managing server 20

Application Response MeasurementARM 97

Application Response Measurement, See ARMapplication server 19archive agent 23, 26, 48

trimming 79ARM 11

Application Response Measurement 97

Bbackup

database 84ITCAM for WebSphere 83

backupConfig command 83

Ccertificate key generator 30CICS 6

Customer Information Control System 96CLI

command-line interface 96Client Application Tracker 13cluster 106

environment 99CMDB 3command-line interface 130

CLI 96commands

backupConfig 83config_DC 56configurewebserver1 118CreateCliProperties 130crtdb_db2 108datatrim.sh 80db2cmd 37, 39db2install 36

© Copyright IBM Corp. 2006, 2007. All rights reserved

gsk7ikm 135iKeyMan 135install-DC 50java 44keytool 68restoreConfig 83run-stat-cmds 78SetupCmdLine.bat 113setupCmdLine.bat 130su 36tmtpcli 130update 113wasprofile 112wsadmin 29

communicationport requirement 31security 100

communication security 29composite application 2confidentiality support 8config_DC command 56configuration files 44, 86configuration management database, See CMDBconfigurewebserver1 command 118CreateCliProperties command 130crtdb_db2 command 108custom certificate 67custom certificates 134Customer Information Control System

CICS 96Customer Information Control System, See CICS

Ddata collection mechanism 96data collector 19, 96

additional 56deploying 28deployment 50

data retrieval 97data trimming 78database

cataloging 37connection pools 19

. 151

engine 139installation 107installation scripts 36maintenance 139table information 24

gcdata 24memorydata 24methods 24pmidata 24requests 24volumestats 24

database maintenance 77datatrim.sh command 80DB2

Client Application Enabler 35database 107Universal Database

Enterprise Server 25, 98installation 35

db2cmd command 37, 39db2install command 36designing managing server 25disaster recovery 32, 102

EEJB 19

usage 19embedded installation 27Enterprise JavaBeans, See EJBETEWatch 13

Ffailover 102fault tolerance 102file system 107firewall 101

considerations 30support 8

Gglobal publish server 43gsk7ikm command 135

IIBM GSKIT 133IBM IT Service Management approach 3IBM Tivoli Composite Application Manager for Re-

sponse Time Trackingbackup, recovery 140components 12

management agent 12management server 12Store and Forward Agent 12

deployment 98features 11file systems 107functions 11installation, deployment 105maintenance 137management agents, update 143management server, processing 98management server, update 142overview 11platforms supported 14server sizing

communication bandwidth 97IBM Tivoli Composite Application Manager for Web-Sphere 19

architecture 19–20availability 6backup, recovery configuration 83component

Tivoli Enterprise Monitoring Agent 10components 9

archive agent 26, 28data collector 10global publishing server 26kernels 26managing server 9message dispatcher 27polling agent 27publish server 26, 28visualization engine 27

data sources 19features 9functions 9implementation 25managing server 20, 25

communication bandwidth 22database size 24sizing 22

overview 8platforms supported 10product architecture 18

IBM Tivoli Composite Application Manager for Web-Sphere managing server


processing requirement 24IBM Tivoli Composite Application Manager portfolio 5

for IBM CICS Transactions 5for IBM IMS Transactions 5for Response Time Tracking 5for SOA 5for WebSphere 5IBM Tivoli OMEGAMON® XE for IBM Web-Sphere Business Integration 5

IBM Tivoli Composite Application Manager Re-sponse Time Tracking

designing management server 98IBM Tivoli software portfolio 4

business service management 5composite application management 4event correlation and automation 4orchestration and provisioning 4resource monitoring 4

iKeyMan command 135IMS 6, 19

Information Management System 96in-depth resource usage 6Information Management System

IMS 96Information Management System, See IMSinstallation parameters 61install-DC command 50IT Infrastructure Library, See ITILITCAM for Response Time Tracking

planning considerations 94product architecture 94sizing parameters 96sizing servers 96

ITCAM for WebSpherecomponents 26database installation, configuration 35features 9installation of managing server 34

configuration 34maintenance 75operational issues 76

ITCAM for WebSphere managing serversizing

memory size 23ITIL 3

JJ2EE

application server 19monitoring agent 13

JACL scripts 29java command 44Java Database Connectivity, See JDBCJava Runtime Environment, See JREJava Virtual Machine Tool Interface, See JVMTIJDBC 9JRE 43JVM thread pools 19JVMTI 19

Kkernel 23, 26

parameters 106keystore 68

extraction of certificates 71keytool command 68

Llarge-scale environment 3

definition 6large-scale implementation

concerns, considerations 7deploying agents 7maintenance 8reliability 8security 8sizing servers 7

listening policies 97load balancing 99log files 85, 141

Mmanagement agent 95, 135

deployment 99management resources

definition 131deployment 129

management server 95installation 124

parameters 124performance, availability of 138

managing server 19global publishing server 26

Index 153

managing server split server 35mass automated installation 29, 100message dispatcher 27, 43Microsoft Windows Services for UNIX, See SFUmodular application development 3monitoring components 131Monitoring On Demand 22monitoring policies 97

Nname resolution 30naming convention 99

OOCTIGATE database 35OMEGAMON XE 18operating system preparation 106operational management pillar 4Oracle database server 25, 98overhead

data collector 25overseer components 24

PPerformance Management Interface, See PMIplanning considerations 18playback policies 97PMI 19policy

discovery 98listening 98playback 98

polling agent 27, 43port 101

consolidation 31issues 30

product maintenance 87, 142pruning tables 78publish server 23, 26, 48

QQuality of Service monitoring agent 13

RRational Robot 12, 96Redbooks Web site 149

Contact us xi

reliability 102remote database 34reporting load 96request table 78response time 6

data 97response time measurement 97restoreConfig command 83run-stat-cmds command 78

SSecure Sockets Layer communication 30Secure Sockets Layer, See SSLsecurity

communication, firewall, port usage 29service level agreement 5setenv.sh 23SetupCmdLine.bat 113setupCmdLine.bat 130SFU 36silent installation 28, 50, 99, 129Simple Network Management Protocol, See SNMPsizing servers 21

parameters 21complexity of transaction 21data collection filter 21instrumentation level 21listening policy mask 21monitoring level 21number of data collectors 21sampling rate 21transaction rate 21

SNMP 27software updates 87, 142split server

configuration 42installation, setup 43

SSL 30, 64certificates 133communication 101data collector setup 65managing server setup 64

SSL communication 30STI 13Store and Forward Agent 95, 135su command 36Synthetic Transaction Investigator 96Synthetic Transaction Investigator, See STI


TTivoli common directory 141Tivoli Enterprise Monitoring Agent 13Tivoli Enterprise Monitoring Server 96Tivoli Enterprise Portal 10, 96tmtpcli command 130tmtpcli.properties 130

Uupdate

managing server 87update command 113user right assignment 106utilities

REORGCHK 139RUNSTATS 140

utilityREORGCHK 77RUNSTATS 77

Vvisualization engine 24, 27, 41, 43, 76

Wwasprofile command 112WebSphere

log files 141security 100

WebSphere Application Server 25, 98, 109installation 109performance 76, 138

WebSphere Edge Componentinstallation 118

WebSphere Edge Server 106WebSphere security 29WebSphere Studio Application Monitor 18WebSphere Studio Application Monitor, See WSAMWindows workstation 12wsadmin command 29WSAM 18

Index 155


®

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICALINFORMATION BASED ONPRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

Redpaper


Planning for performance of management infrastructure

Implementing with multiple servers

Performing mass update of agents

This IBM® Redpaper discusses large-scale implementation of IBM Tivoli® Composite Application Manager for WebSphere® and IBM Tivoli Composite Application Manager for Response Time Tracking. Large-scale implementation is typically characterized by the number of monitoring agents deployed and the number of transactions load-managed. A typical large-scale implementation of a monitoring product contains the following challenges:� Keeping up the performance of the monitoring tools

to accommodate the processing load from the agents.

� Automation of installation, update, and maintenance of monitoring agents based on silent installation and automated update.

� Specific day-to-day maintenance actions to ensure performance and availability of the monitoring solution.

This IBM Redpaper addresses these issues with regard to the implementation of ITCAM for WebSphere and ITCAM for Response Time Tracking on distributed platforms. The discussion is divided into planning issues, implementation guides, and maintenance considerations.

Back cover

Large scale implementation of ibm tivoli composite application manager for web sphere and response time tracking redp4162

Technology

program filesibmwebsphereappserverupdateinstallergt

programfilesibmtivolicommon

structured

extensible

response time

response time

response time

response time