NZS-4532 - Bringing Historical Data to Life with IBMs SMF Data Engine

Bringing Historical Data to Life with IBM’s SMF Data Engine Alan PlaceSenior Development Manager

Barry Klutz

zManage Offering Manager

Please Note:

1

• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

• Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

• The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.

• The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

SMF Introduction

2

• SMF data has been around for years - the bread and butter of the traditional Systems

Programmer

IBM System Management Facility

The standard way of writing z/OS records of activity to a file

Allows reporting or analysis of I/O, CPU utilisation, network activity, software usage, etc

• In today’s world there are pressures that are forcing us to rethink how we deal with this data

– Loss of skills

– Exponential increases in data volumes

– New technology bringing ever-more complicated metrics into the picture

– Economic pressures

This is the story of SMF at IBM and how we are using this

valuable data source to help customers run a more

efficient, cost effective business

SMF - Traditional Data Usage

3

Tivoli Decision Support for z/OS (TDSz)

• TDSz collects, organizes, and converts raw standard systems management data

into business relevant information that can help improve operational planning,

cost management, responsiveness, and decision-making process in organization

to effectively manage economic performance of the IT investment.

• Addresses performance reporting, capacity management, resource availability

management and accounting needs in an enterprise. Generates customized

reports for communicating and exchanging valuable information between

different departments in an enterprise.

• Removes customer pain points of high cost competitive tools and competitors’

proprietary database.

Accounting & Chargeback

Performance

Service Level Reporting

Extract, Categorize, Store

• Measure SLA compliance

• Quantify increased IT resource consumption or

abnormal spikes

• Compare trends to pinpoint where consumption

increased

• Converts raw data into business-relevant information

• Basis for mainframe accounting

Capacity Management Analytics (CMA)Released in 2012 for a complete Capacity Management Solution

• Collection Engine (TDSz)

• Predictive Engine (SPSS)

• Reporting Engine (Cognos)

SMF – Beyond Traditional Reporting

4

Reporting

• Granularity / Statistical

• Forecasting / Prediction

• Application performance model

• Correlation of data / relationships

• Use beyond Capacity Management

Correlate & Forecast

• Granularity / Statistical

• Forecasting / Prediction

• Application performance model

• Correlation of data / relationships

• Use beyond Capacity Management

SMF – Using the Past to look into the Future

CMA uses predictive analytics to help

organizations use their data to make better

decisions by drawing reliable, data-driven

conclusions based on past and current events.

Future capacity requirements can be

forecasted to help ensure that sufficient

capacity is available when the business

needs it.

Capacity

Planning and

Forecasting

Question: Would I have enough capacity to handle my business growth in

the next three months?

5

SMF – Data as a Analytics tool

CMA Near real-time Anomaly Detection

Provides anomaly detection analysis on CICS transaction data. Helps customer find out which CICS

transaction is anomaly. And customer can use our result to tuning or fix problem of their production

environment.

6

Based on transaction CPU utilization and elapsed time

6

SMF – The TDSz Processing Engine

7

Input

Data

Update

Definition

TDSz Core ProcessingCollects, manipulates, aggregates and stores log data

(SUM, AVG, MIN, MAX, etc)

Multiple options:

• Log archive

• Logstream

• SMF Buffer

• IMS Log

• Any fixed record

Record

Definition

Lookup

Tables

DB2

Log

Procedure

Record

Procedure

Combination allows logic to

be added (eg threshold

processing)

Assigns alternative values

to data or ranges (eg

peak/offpeak, MIPS rating,

banking applications)

Additional processing logic to

handle composite records (eg

IMS), split records, etc.

Special handling for complex or

different SMF formats

Defines what data is written

and the format

Bespoke language for defining

data types and SMF record

formatting

SMF – Using the data to create Insight

8

Tivoli Decision Support for z/OS 1.8.2

• Traditionally TDSz relied on customer experience and knowledge

• Using SMEs from z/OS, DB2, CICS and IMS the Key Performance metrics were created

• Cut down metrics to quickly check the state of the system

• Performance improved dramatically

• Simple reports to drill down into the problem areas

A Big Data view of SMF

9

IBM® DB2® Analytics Accelerator for z/OS (IDAA)

A high performance appliance

• Integrates IBM zSystems® infrastructure and IBM PureData™

• Powered by IBM Netezza® technology

Now we can have SMF Data in a repository for lightning fast queries

Storage is cheaper and allows data analysts to store months of data

Allows a deep dive of an event that may have happened weeks ago

SMF data as a SaaS Offering

10

IBM z Operational InsightsA New Way to Optimize your z Systems

Cut costs. Save time. No installation necessary.

Rapid, cloud-based analytics enable you to:

see quantified savings upfront

identify what actions to take next

compare your performance to others

Try the open beta today, for free! ibm.biz/try-zoi

Actionable insights for CICS, MQ , IMS & WAS in minutes, not hours - just add operational data.

SMF data as a SaaS Offering Benefits

11

Near Real Time SMF data

12

• SMF data can be processed more efficiently from a pure operational view

– A large North American bank cuts SMF logs every half an hour and collects/loads data using TDSz. Data is

available in DB2 within 50 minutes

Becomes an early warning system

No need to wait for the following day

• The logstream is another way of bringing data in sooner…………..

The SMF Data Engine

13

• Uses the standard SMF processing embedded in TDSz

• No DB2 pre-req

• Reads from the logstream

• Easily installed (hours instead of days)

• Highly targeted data customised for the problem being investigated

Available to any product with a need to process SMF data

TDSz reinvented as the standard way to read and process SMF data in

near real-time

Bringing SMF Data closer to Real Time

14

IBM Operations Analytics for z/OS Log Analyzer (IOAz)

Centralized,

Distributed, Cloud,

Resilient Architectures

Increase Data Volume

Logs,

Traces,..Events

Metrics

Transactions

ConfigCore files

Contains the first production version of the SDE to provide insight into why messages are

happening and potential solutions to reduce MTR

The Challenge - Find the right needle in

one of many haystacks – QUICKLY

IOAz was created as a debug aid as part of the Predict, Search and Optimize strategy that

underpins the whole of the zManage portfolio

SMF Data - Near Real Time in Action

15

IBM Operations Analytics for z/OS Log Analyzer

The Power of Targeted SMF

16

Use Case: System Programmer wants detailed performance data

for a specific period of time to diagnose a performance problem.

Alert Request

Specific, tailored

definitions for type of

problem type

SDE

Batch

Job

Subm

it

DB2

Load

CSV

Output

destination and

format built into

requestTDSzReports

Locate

Start/end time

Record types and

fields

SMF

Data

Engine

The Future of SMF data

17

SDE Common Approach

Logstream

Hot

DataTier 1

Long term

Trending

Data

Short term

Analytical

Data

Near Real Time Tier 2

Real Time

BatchLog

Archive

Tier 3

SMF

Direct

• Single mechanism for handling all SMF data

• Read once – write many

• Designed to function in both online and batch mode.

Has the flexibility to target data and populate all three data tiers while keeping CPU consumption under control.

Multiple insights

from multiple

products

Other InterConnect Sessions

18

Tuesday 23-02-16 8:30am - 9:30am

4555 IT Analytics Keynote: IT Analytics for the Enterprise Barry Klutz

Tuesday 23-02-16 4:00pm - 5:00pm

2051 A New Era of Mainframe Optimization: IBM Operational Insights for z Systems

Steven Horsman

Wednesday 24-02-16 8:30am - 9:30am

4561IBM Operations Analytics for z Systems Client Experience Barry Klutz

Thursday 25-02-16 10:30am - 11:15am

5063 IT Operations Analytics = Bridging Business and IT Ann Dowling

Notices and Disclaimers

19

Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission

from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of

initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS

DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE

USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.

IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers

have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in

which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials

and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or

their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and

interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such

laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.

20

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not

tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.

Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the

ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT

NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual

property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,

FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,

Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,

PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,

StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business

Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM

trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee

Portal to complete your session surveys from your

smartphone,

laptop or conference kiosk.