Advanced Technical Skills (ATS) North America © 2010 IBM Corporation 1 z10 CPU MF Overview and WSC Experiences SHARE Session 2113 March 16, 2010 John Burg IBM Washington Systems Center
Advanced Technical Skills (ATS) North America
© 2010 IBM Corporation1
z10 CPU MF Overview and WSC Experiences
SHARE Session 2113
March 16, 2010
John Burg
IBM Washington Systems Center
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation2
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.
The following are trademarks or registered trademarks of other companies.* Registered trademarks of IBM Corporation
* All other products may be trademarks or registered trademarks of their respective companies.
Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
AlphaBlox*APPN*CICS*CICS/VSE*Cool BlueDB2*DFSMSDFSMShsmDFSMSrmmDirMaintDRDA*DS6000DS8000ECKDESCON*FICON*FlashCopy*
GDPS*HiperSocketsHyperSwapIBM*IBM eServerIBM logo*IMSLanguage Environment*Lotus*Large System Performance Reference™ (LSPR™)Multiprise*MVSOMEGAMON*Parallel Sysplex*Performance Toolkit for VMPowerPC*PR/SMProcessor Resource/Systems Manager
RACF*Redbooks*Resource LinkRETAIN*REXXRMFS/390*Scalable Architecture for Financial ReportingSysplex Timer*Systems Director Active Energy ManagerSystem/370System p*System StorageSystem x*System zSystem z9*System z10
Tivoli*Tivoli Storage ManagerTotalStorage*VSE/ESAVTAM*WebSphere*xSeries*z9*z10z10 BCz10 ECz/Architecture*z/OS*z/VM*z/VSEzSeries*
Trademarks
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation3
TopicsCPU MF Introduction– What it is and how to enable– Lessons Learned
Key Performance Metrics– CPI, Problem State, z10 Cache / Memory Hierarchy– Formulas
WSC Experiences with SMF 113s– HiperDispatch Theory and Measurement– Aligning SMF 113s with CPU Activity by Logical CPU – z/OS 1.11 Dynamic Channel Path Management (DCM) Measurement
Future Direction– Workload Characterization
Summary
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation4
z10 CPU Measurement Facility Introduction
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation5
What is the z10 CPU Measurement Facility New hardware instrumentation facility “CPU Measurement Facility” (CPU MF)
– Available on System z10 EC GA2 and z10 BC– Supported by a new z/OS component (Instrumentation), Hardware Instrumentation
Services (HIS)
Potential Uses – for this new “cool” virtualization technology– COUNTERS
• Supplement Current Performance Metrics• Workload characterization
– SAMPLING• ISV product improvement • Application Tuning
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation6
High Level DescriptionData collection done by System z10 hardware– Low overhead– Little/No skew in sampling– Access to information which is not available from software
• Information about how software and hardware interact
2 Basic Modes– COUNTERS – SAMPLING
New IBM Research article – “IBM System z10 performance improvements with software & hardware synergy”– http://www.research.ibm.com/journal/rd/531/jackson.pdf
– Contact IBM team for copy of the article
This presentation will focus on COUNTERS
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation7
Requirements and Steps to utilize z10 CPU MFRequirements for CPU MF
• System z10 machine must be at GA2 Driver 76D – Bundle #20 or higher• z10 BC sub-capacity models must be at Bundle #20 or higher
• z/OS LPAR being measured must be at z/OS 1.8 or higher with APARs: • OA25755, OA25750, and OA25773• OA27623 also recommended to add “CPU Speed” to SMF 113s and HIS COUNTERS output• Not supported for z/OS running as a z/VM guest
• Steps to utilize CPU MF1. Configure the z10 Server to collect CPU MF Data
• Update LPAR Security Tabs (See appendix)
2. Configure HIS on z/OS to collect CPU MF Data *• Set up HIS Proc• Set up OMVS Directory• Collect SMF 113s via SMFPRMxx
3. Collect CPU MF Data • Start HIS – Modify with Begin/End – for COUNTERS or SAMPLING• “F HIS,B,TT=‘Text',PATH='/his/',CTRONLY,CTR=ALL”
4. Analyze the CPU MF Data • SMF 113s
//HIS PROC
//HIS EXEC PGM=HISINIT,REGION=0K,TIME=NOLIMIT
//SYSPRINT DD SYSOUT=*
Remember CTR=ALL to get Extended Counters!
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation8
CPU MF (COUNTERS) for Performance and Capacity SizingWhat is CPU MF (COUNTERS)?
– A new z10 capability to measure z10 cache / memory hierarchy characteristics
How can it be used today?– To supplement current performance metrics (e.g. from SMF, RMF, DB2, CICS)– As a secondary data source to understand why performance may have changed
What it may be used for in the Future? – To help characterize workload for Capacity Sizing with zPCR
What CPU MF is not – It is Not a substitute for traditional performance nor capacity metrics – It does Not indicate the capacity being achieved by the LPAR or processor
You can calculate “MIPS”, but it has no relationship to commonly used capacity ratings
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation9
CPU MF – Lessons Learned since August 2009CPU MF Performance Metrics can be used to help understand why performance changed
Customers are successfully running CPU MF COUNTERS collecting SMF 113s – Over days and months without any reported performance impact– Feedback from Volunteers is this is very easy to enable, with a minimal time investment
SMF 113 Logical CPU IDs are equal to the SMF 70 Logical CPU IDs– Can match up SMF 113s & SMF 70s to identify GCPs, zIIPs or zAAPs– Can see the unique Vertical Polarity Logical CPs cache/memory characteristics
• E.G. Vertical Mediums may have higher L2 Remote activity
In multi-book z10 ECs there can be L2 Remote Activity even if <=12 GCPs– Because of I/O activity from SAPs as the data is initially stored in the Remote L2
Utilize the Counter Version Number fields to map to technology – Number is increased for a change in meaning or number of counters
• SMF113_2_CTRVN1 – Basic or Problem-State counter sets• SMF113_2_CTRVN2 – Crypto or Extended counter sets
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation10
Key Performance Metrics
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation11
CPU MF and HIS provide a z/OS logical view of z10 Resource Usage and Cache Hierarchy Sourcing
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
PR/SM
z/OSLP0 LP1 LP4
Book
CPU MF
HIS
LPAR / Logical CP view:Memory Accesses
Cache
•L 2 Accesses (local and remote)
•L1.5 Accesses
•L1 Sourced from Hierarchy
•Instructions and Cycles
•Crypto function
LPAR
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation12
Introducing CPU MF Key Performance Metrics:
Potential Workload Characterization z10 L1 sourcing from cache/memory hierarcy
These numbers come from a synthetic Benchmark and do not represent a production workload
CPI – Cycles per Instruction
PRBSTATE - % Problem State
L1MP – Level 1 Miss %
L15P – % sourced from L1.5 cache
L2LP – % sourced from Level 2 Local cache (on same book)
L2RP – % sourced from Level 2 Remote cache (on different book)
MEMP - % sourced from Memory
LPARCPU - APPL% (GCPs, zAAPs, zIIPs) captured and uncaptured
SYSID HOUR CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPUSYSD 9 AM Hour 9.1 2.6 4.0 78.2 9.8 0.1 11.9 2.8SYSD 8 AM to 5 PM 6.5 2.6 4.0 85.3 11.4 0.1 3.3 2.0
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation13
CPU MF can help provide z10 cache/memory resource change insights
CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPU
L1 Hit %
CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPU
L1 Hit %
Decreased Contention
Increased Contention
Increased Residency Time
Decreased Residency Time
=
=
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation14
Formulas Potential Workload Characterization z10 L1 sourcing from cache/memory hierarcy
Metric Calculation – note all fields are deltas between intervals
CPI B0 / B1 PRBSTATE (P33 / B1) * 100L1MP ((B2+B4) / B1) * 100L15P ((E128+E129) / (B2+B4)) * 100L2LP ((E130+E131) / (B2+B4)) * 100L2RP ((E132+E133) / (B2+B4)) * 100MEMP (((E134+E135) + (B2+B4-E128-E129-E130-E131-E132-
E133-E134-E135)) / (B2+B4)) * 100LPARCPU ( ((1/CPSP/1,000,000) * B0) / Interval in Seconds) * 100
B* - Basic Counter Set - Counter Number
P* - Problem-State Counter Set - Counter Number
See “The Set-Program-Parameter and CPU-Measurement Facilities” SA23-2260-0 for full description
E* - Extended Counters - Counter Number
See “IBM The CPU-Measurement Facility Extended Counters Definition for z10” SA23-2261-0 for full description
CPSP is SMF113_2_CPSP “CPU Speed”
CPI – Cycles per Instruction
PRBSTATE - % Problem State
L1MP – Level 1 Miss %
L15P – % sourced from L1.5 cache
L2LP – % sourced from Level 2 Local cache (on same book)
L2RP – % sourced from Level 2 Remote cache (on different book)
MEMP - % sourced from Memory
LPARCPU - APPL% (GCPs, zAAPs, zIIPs) captured and uncaptured
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation15
WSC Experiences
HiperDispatch Theory and Measurement
Aligning SMF 113s with CPU Activity
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation16
HiperDispatch attempts to align Logical CPs with PUs in the same Book
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
PR/SM
z/OSLP0 LP1 LP4
Book
CPU MF
HIS LPAR / Logical CP view:Memory Accesses
Cache
•L 2 Accesses (local and remote)
•L1.5 Accesses
•L1 Sourced from Hierarchy
•Instructions and Cycles
•Crypto functionLPAR
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
PR/SM
z/OSLP0 LP1 LP4
CPU MF
HIS
Book
LPAR
HD=NO HD=YES
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation17
From CPU MF, HiperDispatch=YES May Decrease the L2 Remote %
Potential Workload Characterization z10 L1 sourcing from cache/memory hierarcy
CPI – Cycles per Instruction
PRBSTATE - % Problem State
L1MP – Level 1 Miss %
L15P – % sourced from L1.5 cache
L2LP – % sourced from Level 2 Local cache (on same book)
L2RP – % sourced from Level 2 Remote cache (on different book)
MEMP - % sourced from Memory
LPARCPU - APPL% (GCPs, zAAPs, zIIPs) captured and uncaptured
CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPU
L1 Hit %
=
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation18
HiperDispatch – Measurement from an Internal Benchmark
These numbers come from a synthetic Benchmark and do not represent a production workload
CPI – Cycles per Instruction
PRBSTATE - % Problem State
L1MP – Level 1 Miss %
L15P – % sourced from L1.5 cache
L2LP – % sourced from Level 2 Local cache (on same book)
L2RP – % sourced from Level 2 Remote cache (on different book)
MEMP - % sourced from Memory
LPARCPU - APPL% (GCPs, zAAPs, zIIPs) captured and uncaptured
L1 Hit %
HiperDispatch=YES resulted in a 12% improvement as measured
by the traditional CPU/Transaction metric
HD ? 2097 E40 CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPU
NO 1x32w 9.6 52.0 5.1 56.6 26.3 11.3 5.8 2880
YES 1x32w 8.5 52.0 5.0 62.0 25.0 7.4 5.6 2571
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation19
Traditional CPU Activity - SMF 70s
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation20
SMF 113s Can Be Aligned with SMF 70s - CPU Activity
Since SMF 113 CPU IDs = SMF 70 Processor IDs, this enables
•Identifying SMF 113 CPU ID types: GCPs, zAAPs, and zIIPs
•Identifying CPU MF characteristics by Logical CPU
•e.g. Vertical Medium Polarity Remote L2 Activity
zIIPs Vertical Medium and Remote L2
MONTDAY SYSID HOUR POOL PRID POLAR WEIGHT LOGPSHR VPOL CPI PRB L1MP L15P L2LP L2RP MEMP LPARCPU STATEOCT 13 ATS1 14 1 0 69.91 100.0 HIGH 6.5 55.8 5.8 80.0 17.1 0.2 2.6 46.4OCT 13 ATS1 14 1 1 69.91 100.0 HIGH 6.1 55.0 4.5 77.6 18.6 0.2 3.7 48.6OCT 13 ATS1 14 1 2 69.91 100.0 HIGH 5.7 58.6 3.8 77.7 18.0 0.3 4.0 32.9OCT 13 ATS1 14 1 3 69.91 100.0 HIGH 5.6 60.1 3.7 78.7 17.1 0.2 4.0 35.7OCT 13 ATS1 14 1 4 69.91 100.0 HIGH 6.7 52.8 6.0 79.0 18.2 0.2 2.6 45.4OCT 13 ATS1 14 1 5 69.91 100.0 HIGH 6.4 53.8 4.5 75.4 20.6 0.2 3.8 44.7OCT 13 ATS1 14 1 6 69.91 100.0 HIGH 5.8 59.7 3.7 75.6 20.0 0.2 4.3 30.7OCT 13 ATS1 14 1 7 69.91 100.0 HIGH 5.8 60.6 3.8 76.5 19.1 0.2 4.1 37.1OCT 13 ATS1 14 1 8 69.91 100.0 HIGH 6.8 49.7 5.9 80.9 15.1 1.1 3.0 33.2OCT 13 ATS1 14 1 9 69.91 100.0 HIGH 6.6 48.5 4.6 77.3 17.2 1.4 4.1 30.8OCT 13 ATS1 14 1 10 69.91 100.0 HIGH 5.9 57.9 3.7 78.5 14.9 2.2 4.5 19.8OCT 13 ATS1 14 1 11 61.03 87.3 MED 6.6 61.3 3.5 78.5 4.5 11.0 6.0 26.6OCT 13 ATS1 14 1 14 0 0.0 LOW ***** 0.0 36.5 13.9 8.2 1.5 76.4 0.0OCT 13 ATS1 14 1 15 0 0.0 LOW ***** 0.0 36.5 14.5 8.3 1.0 76.2 0.0 OCT 13 ATS1 14 6 12 355 99.8 MED 4.5 0.0 1.9 79.1 12.1 0.4 8.4 1.0OCT 13 ATS1 14 6 13 230 64.6 MED 8.0 0.0 3.1 59.3 10.6 19.2 10.9 0.0
From RMF 70 Records - CPU Activity From CPU MF SMF 113 Records
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation21
CPUMF HiperDispatch Summary
CPU MF can be used– To explain why we saw the improvement from
HiperDispatch=YES from the traditional metrics
– To identify the Logical CPU cache / memory characteristics by Engine Type and/or Vertical Polarity
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation22
WSC Experiences z/OS 1.11 FICON Dynamic Channel Path Management (DCM)
Advanced Technical Skills (ATS) North America
© 2010 IBM Corporation23
New with z/OS 1.11 Dynamic Channel Path Management (DCM)
WLM can automatically manage ESCON, FICON Bridge (FCV), and FICON I/O paths connected to DASD subsystems in response to changing workload demands.
The objectives for FICON DCM are: – 1) Simplify I/O configuration planning and definition
– 2) Reduce the customer skills required to manage z/OS
– 3) Dynamically balance I/O channel resources
– 4) More efficient use of channel resources
– 5) Enhance availability
Advanced Technical Skills (ATS) North America
© 2010 IBM Corporation24
ATS ConfigurationIBM Systems & Technology Group
FICON DCM Evaluation Topology
LPAR Cluster A LPAR Cluster B
LCSS0
LPA1
SYSC
LPA2
SYSDLPA3 LPB1 LPB2
Switch C4 Switch C6 Switch C7
LCU3-6Axx LCU4-6BxxLCU2-69xxLCU1-68xx
S1B8
M3BB
M2BA
S1BC
M1B9
LCSS2LCSS1
Switch C5
Our Test utilized CHPIDS B8 (Static), B9 (Managed), and BA (Managed) to
LCU1 68xx
Advanced Technical Skills (ATS) North America
© 2010 IBM Corporation25
What we didDefined 2 CU Groups– CU Group 1 – 6800 and 6900 addresses - Static Channel B8, DCM Managed Channels B9, BA, BB – CU Group 2 – 6A00 and 6B00 addresses - Static Channel BC, DCM Managed Channels B9, BA, BB
DCM will always look to see that a CU Group has 2 CHPIDs for Availability, including Static CHPIDS– If less than 2 then it will “permanently” assign Managed channels to ensure there are 2– In out case since we only have 1 static CHPID assigned to each CU Group, it gave 1 Managed CHIPD
Drove Channel load to 8 Devices in CU Group 1 over 2 CHPIDS (B8 and B9) ~ 61% Busy each– We utilized CHDRVR internal tool to drive I/O to the devices via job JPBURGC8 with DCM disabled
Once DCM enabled, the load caused the “Free” Managed Channel (BA) to be moved to CU Group 1 – Evidenced by messages IEE303I and IEE302I messages
• We saw the individual Vary Paths (OFF) to each address first (from the Source CU Group)• Followed by the Individual Vary Paths (ON) to each address next (to the Target CU Group)
With the 3rd Channel coming over, the Channels Utilization dropped to ~56% Busy each – but now 3 of them– Overall Channel Load increased ~37%
This led to – A decrease in I/O response time and an increase in I/Os per Second ~36%– For JPBURGC8 an increase in throughput and a decrease in CPU/EXCP
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation26
z/OS 1.11 DCM – Channel Busy
SMF 73 Channel Busy
DCM ON
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation27
z/OS 1.11 DCM – I/O Response and Rate – Job Improvements
JPBURGC8DCM ? CPSEC EXCPS CPU / EXCPs DASD I/Os per secNo 3.78 516,021 0.0000073 1,720Yes 5.01 701,618 0.0000071 2,339DCM Improvement ==> 1.36 0.97 1.36
SMF 74 DASD Subsytem I/O Response SMF 74 DASD Subsytem I/O Rate
JPBURGC8 SMF 30 Characteristics
DCM ON
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation28
z/OS 1.11 DCM - CPU MF Characteristics
SYSID MON DAY SH Hour CPI PRBSTATE L1MP L15P L2LP L2RP MEMP LPARCPU DCMSYSC FEB 17 P 10.75 6.1 4.3 5.1 88.4 10.5 0.2 0.9 11.2 OFFSYSC FEB 17 P 11.00 6.1 4.3 5.1 88.4 10.8 0.2 0.7 11.2 OFFSYSC FEB 17 P 11.25 6.1 4.3 5.1 88.3 10.8 0.2 0.8 11.2 OFFSYSC FEB 17 P 11.50 6.1 4.2 5.1 88.4 10.5 0.2 1.0 11.7 OFF
DCM OFF 6.1 4.3 5.1 88.4 10.6 0.2 0.8 11.3 OFF
SYSC FEB 17 P 11.75 5.8 4.5 5.2 89.2 10.1 0.2 0.6 12.8 ONSYSC FEB 17 P 12.00 5.9 4.5 5.3 89.2 10.0 0.2 0.7 12.8 ONSYSC FEB 17 P 12.25 5.7 4.5 5.3 89.1 10.2 0.2 0.6 12.4 ONSYSC FEB 17 P 12.50 5.8 4.5 5.3 89.0 10.1 0.2 0.7 12.7 ONSYSC FEB 17 P 12.75 5.8 4.5 5.3 89.3 10.0 0.2 0.5 12.5 ONSYSC FEB 17 P 13.00 5.8 4.5 5.3 89.3 10.0 0.2 0.5 12.6 ONSYSC FEB 17 P 13.25 6.0 4.5 5.3 89.0 9.7 0.2 1.2 13.1 ON
DCM ON 5.8 4.5 5.3 89.2 10.0 0.2 0.7 12.7 ON
With DCM ON
•CPI is down and LPARCPU is up (due to load), indicative of improvement
•Problem State is up, indicative of LPAR Busy increase
Conclusion: The best performance view is at Job Level from SMF 30 statistics
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation29
Summary
With DCM=ON– The I/O driver job saw a 36% throughput increase from traditional SMF 30 statistics
• Along with a reduced CPU/EXCP (-3%) and improved DASD I/O response times – CPU MF provided a secondary view of the job improvement environment
DCM can help: – Simplify I/O configuration planning and definition
– Enhance availability
See Techdoc WP101544 for more information about DCM• http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101544
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation30
Future Direction
Workload Characterization
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation31
Workload Characterization Future Vision
Future vision is to potentially combine SMF 23s and SMF113s – To help identify workload characteristics and to provide better input for capacity
planning and performance – 80 Customer/Partitions thru 3/12
SMF 23s SMF 113s
Workload Characterization
Looking for “Volunteers” – (2 days, 24 hours/day, SMF 23s, 70s, 71s, 72s, 113s per LPAR)
If interested send note to [email protected]
No deliverable will be returned
Benefit: Opportunity to ensure your data is used to influence clustering analysis
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation32
z10 CPU Measurement Facility COUNTERS Summary
Traditional metrics continue to provide the best view of Performance – CPU MF can help explain why a change occurred
In the future CPU MF may help characterize workloads for Capacity Sizing
CPU MF has a very low overhead to run and is easy to implement– Less than 1/100 of a second for HIS address space in 15 minute interval– Customers are successfully running CPU MF in Production Today
Volunteers are still needed for our Workload Characterization study – Feedback from Volunteers is this is very easy to enable, with a minimal time investment
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation33
Acknowledgements Many people contributed to this presentation including:
Riaz Ahmad
Greg Boyd
Harv Emery
Gary King
Frank Kyne
Dennis Ng
Steve Olenik
Dale Riedy
Bill Rooney
Brian Smith
Bob St John
Kathy Walsh
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation34
Thank You
for attending!
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation35
Appendix
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation36
Documentation• MVS Commands SA22-7627-19
• Setting up hardware event data collection 1-39
• The Set-Program-Parameter and CPU-Measurement Facilities SA23-2260-0 • Full description of Basic, Problem-State and Crypto Counter Sets
• IBM The CPU-Measurement Facility Extended Counters Definition for z10 SA23-2261-0 • Full description of Extended Counter Set
• WSC Short Stories and Tall Tales• SHARE Summer 2009 Denver - Session 2136 – John Burg
• Techdoc in late 1QT 2010 • This presentation and a detailed write up for enabling CPU MF – John Burg
• ITSO Red Book reference Planned for 4QT 2010– Exploiting System z LPAR Capacity Controls - SG24-7846. 2 Part Book:
• Part 1 - CPU MF• Part 2 - HiperDispatch, Group Capacity Controls, hard/soft capping• Draft available ~April 1
http://www.redbooks.ibm.com/redbooks.nsf/home?ReadForm&page=drafts
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation37
CPU MF Update – Lessons Learned since March 2009L1 Miss % can be determined from CPU MF COUNTERS
z10 EC must be at bundle #20 or higher for CPU MF COUNTERS
IRD considerations– If CPU goes offline, only activity within internal is recorded in an Intermediate record, then
• If no activity in follow on 15 minute interval(s), Intermediate record is not cut for the CPUID– No Final record when HIS is ended
• When activity resumes, Intermediate record is written for CPUID
New APAR OA27623 to add “CPU Speed” to SMF 113 and to HIS COUNTERS output– Processor speed for which the hardware event counters are recorded. Speed is in cycles /
microsecond - “4404” for z10 EC – SMF 113 new field: SMF113_2_CPSP - 4 byte binary– Simplifies conversion of Cycles into “Time”
Customers are successfully running CPU MF COUNTERS (and collecting SMF 113s) over 24 hours
Analyze the “major” LPARs on a z10 at the same time
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation38
APAR OA27623 “CPU Speed” – HIS COUNTERS Output
These numbers come from a synthetic Benchmark and do not represent a production workload
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation39
How it worksHardware Instrumentation Counters
CPU nCPU 1CPU 0…
….CNT SMF 113s
Modify HIS,Begin
<15 min intervals>
Modify HIS,End
Raw counters to SMF
One record per CPU
Delta counters, (End- Begin), to .CNT file
One file for all CPUs
HW
OS
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation40
What data is in the CPU MF – per Logical CPBasic Counters (and Problem) per CPU – (1)
– Cycles
– Instructions
– L1 Cache Sourcing basic information
Crypto Counters per CPU – (1)
– Counts and Cycles by Crypto function
Extended Counters – per CPU (Model Dependent) – (2)
– Cache Hierarchy Information and more• z10 L1 Sourcing detailed information
z10 L1 Cache Hierarchy Sourcing
1 - See “The Set-Program-Parameter and CPU-Measurement Facilities” SA23-2260-0 for full description2 - See “IBM The CPU-Measurement Facility Extended Counters Definition for z10” SA23-2261-0 for full description
See Appendix for Basic, Problem and Crypto Counters
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation41
What we didSet up CPU MF on WSC z10 and z/OS 1.10
Started/Modified HIS and collected SMF 113s and *.CNT Data
– Ran “COUNTERS” mode, COUNTERS=ALL (Basic, Problem, Crypto, Extended) via:
– “F HIS,B,TT='EncrypCounters2',PATH='/his/',CTRONLY,CTR=ALL”
Ran DASD dumps• DASD dumps sequentially over 20 minute duration • With option: ENCRYPT(CLRTDES) -
Built sample reports with a REXX exec– Used *.CNT output to as input– Validated with SMF 113s– Reports
• Basic Counters• Basic / Extended Counters - z10 L1 Cache Hierarchy Sourcing Report• Crypto Counters
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation42
SYSHISyyyymmdd,hhmmss.CNT Output unformatted– HIS019I EVENT COUNTERS INFORMATION– FILE NAME: SYSHIS20090207.161102.CNT
– COMMAND: MODIFY HIS,B,TT='EncrypCounters2',PATH='/his/',CTRONLY,CTR=ALL– COUNTER VERSION NUMBER 1: 1 COUNTER VERSION NUMBER 2: 1
– COUNTER SET= BASIC– COUNTER IDENTIFIERS:– 0: CYCLE COUNT– 1: INSTRUCTION COUNT– 2: L1 I-CACHE DIRECTORY-WRITE COUNT– 3: L1 I-CACHE PENALTY CYCLE COUNT– 4: L1 D-CACHE DIRECTORY-WRITE COUNT– 5: L1 D-CACHE PENALTY CYCLE COUNT
– START TIME: 2009/02/07 16:11:02 START TOD: C3B6ADBE7AD83D26– END TIME: 2009/02/07 16:31:19 END TOD: C3B6B24700FC45A5– COUNTER VALUES (HEXADECIMAL) FOR CPU 00:– 0- 3 0000004689BEBF20 0000000433831366 0000000014CF0790 000000021B57E0D8– 4- 7 000000002A620C97 0000000B25C43DBC ---------------- ----------------
– START TIME: 2009/02/07 16:11:02 START TOD: C3B6ADBE7AD95826– END TIME: 2009/02/07 16:31:19 END TOD: C3B6B24700FD3625– COUNTER VALUES (HEXADECIMAL) FOR CPU 01:– 0- 3 00000048CFB22F1D 000000048D23D49A 00000000154D89E5 0000000229B662EA– 4- 7 000000002C1F067B 0000000B8087F6A7 ---------------- ----------------
– START TIME: 2009/02/07 16:11:02 START TOD: C3B6ADBE7ADABCA6– END TIME: 2009/02/07 16:31:19 END TOD: C3B6B24700FE1525– COUNTER VALUES (HEXADECIMAL) FOR CPU 04:– 0- 3 00000021DE76A328 0000000A8F16E5E9 0000000000022392 00000000008AC8F2– 4- 7 000000001B92F07B 000000035E926CFD ---------------- ----------------
– COUNTER SET= PROBLEM-STATE– COUNTER IDENTIFIERS:– 32: PROBLEM-STATE CYCLE COUNT– 33: PROBLEM-STATE INSTRUCTION COUNT– 34: PROBLEM-STATE L1 I-CACHE DIRECTORY-WRITE COUNT– 35: PROBLEM-STATE L1 I-CACHE PENALTY CYCLE COUNT– 36: PROBLEM-STATE L1 D-CACHE DIRECTORY-WRITE COUNT– 37: PROBLEM-STATE L1 D-CACHE PENALTY CYCLE COUNT
Description
Start / End timeCounters per CPU - 00
Counters per CPU - 01
Counters per CPU - 04
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation43
Sample Report – Basic Counters
These numbers come from a synthetic Benchmark and do not represent a production workload
Normalized Basic Counters to per Second
L1 Index and Directory Write Counts used In Cache Hierarchy Sourcing
L1 Miss % can be derived from CPU MF information
•Instruction Count is the base. If instructions are not in z10 L1 Cache, then they must be “Sourced” from the z10 hierarchy. The Total “Sourced” is the Total Write Count, the “Misses”
•L1 Miss % = Directory Write Counts (I+D) / Instruction Counts
•3.2% = (580,653.65 + 1,572,649.35) / 68,15,013.72
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation44
Sample Report – Basic / Extended Counters z10 L1 Cache Hierarchy Sourcing
Various Sources from Extended Counters
Total L1 Sourcing from Basic Counters
These numbers come from a synthetic Benchmark and do not represent a production workload
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation45
CPU MF and HIS provide a z/OS logical view of z10 Resource Usage and Cache Hierarchy Sourcing
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
Memory
L2 Cache
L 1.5
CPU
L 1
L 1.5
L 1
CPU
L 1.5
L 1
CPU
PR/SM
z/OSLP0 LP1 LP4
Book
CPU MF
HIS
LPAR / Logical CP view:Memory .27%
Cache
L 2 (local and remote) 20.92%
L1.5 78.8%
L1 Sourced from above Hierarchy
2.15 Million / Sec
3.2% L1 Miss %
96.8% L1 Hit %
These numbers come from a synthetic Benchmark and do not represent a production workloadLPAR
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation46
Sample Report – Crypto Counters
This information may be useful in determining:
•When and What encryption function is occurring (Count)?
•How many cycles are being used?
The encryption facility executed both SHA functions and TDES functions for this specific test.
Since CPU MF is new, this information is not available from RMF today
Need to analyze more Customer data
These numbers come from a synthetic Benchmark and do not represent a production workload
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation47
Image Profile Security Customization for HIS
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation48
Counter DataBasic Counter Set
– Cycle count– Instruction count– Level-1 I-cache directory write count– Level-1 I-cache penalty cycle count – Level-1 D-cache directory write count– Level-1 D-cache penalty cycle count
Problem State Counter Set– Problem state cycle count– Problem state instruction count– Problem state level-1 I-cache directory write count– Problem state level-1 I-cache penalty cycle count – Problem state level-1 D-cache directory write count– Problem state level-1 D-cache penalty cycle count
Extended Counter Set– Number and meaning of counters are model dependant
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation49
Counter DataCrypto Activity Counter Set (CPACF activity)– PRNG function count– PRNG cycle count– PRNG blocked function count– PRNG blocked cycle count– SHA function count– SHA cycle count– SHA blocked function count– SHA blocked cycle count– DES function count– DES cycle count– DES blocked function count– DES blocked cycle count– AES function count– AES cycle count– AES blocked function count– AES blocked cycle count
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation50
Layout: (SMF manual, HISYSMFR macro)Standard SMF record header (‘1C’x bytes)SMF record control information
TOD when SMF record is written, etc.Offset, length, and number of data sections
Data sectionTOD when counter data was capturedCPU number
Offset, length, and number of Counter Set SectionsOffset, length, and number of Counter SectionsCounter Set Sections
Counter Set type (1=BASIC, 2=PROB, 3=CRYPTO, 4=EXT)Bit mask identifying the counters being recorded in array
e.g. ‘FC00000000000000’x => counters 0-5 are validCounter Sections – 8-byte counter values (contiguous)
SMF Record type 113, subtype 2
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation51
z9 ECCPU
1.7 Ghzsuperscalar
CachesL1 private 256k i, 256k dL2 shared 40 mbs / bookbook interconnect: ring
z10 ECCPU
4.4 Ghzredesigned pipelinesuperscalar
CachesL1 private 64k i, 128k dL1.5 private 3 mbsL2 shared 48 mbs / bookbook interconnect: star
z10 versus z9 hardware comparison
...
Memory
L2 Cache
L1
CPU
L1
CPU
L1
CPU
...
Memory
L2 Cache
L1.5
CPUL1
L1.5
CPUL1
L1.5
CPUL1
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation52
DCM Requirements
Software Dependencies - OA28321
– All systems in LPAR cluster must be z/OS 1.11
Hardware Dependencies– Processor – All currently supported processors (z900 and up)
– Channels – All currently supported FICON channels
– Coupling Facility required if running multi-system Switches
– Must have control unit port (CUP) function
– Must be defined in the IODF
– Control Unit – No special microcode needed
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation53
DCM Management Strategy
Channel Utilization*
– Three ranges• Good (0 – 25)• Average (26 – 50)• Poor (51 and above)
Port I/O Intensity†
– Three ranges• Good (0 – 1000)• Average (1001 – 3000)• Poor (3001 and above)
•* From RMF Channel Path Activity Report•† From RMF ESS Link Statistics Report
Advanced Technical Support – Washington Systems Center
© 2010 IBM Corporation54
Link Port StatisticsControl Unit Information
FICON DCM Process
Data CollectionCollect performance data
WLMCollect data from all systems in LPAR ClusterReturn data averaged over period of time
DCM AlgorithmsCheck for bottlenecks, performance issuesDetermine what changes can be madeSelect the best changeChange the I/O configuration – Dynamic I/ODynamic I/O
Add, replace or delete CHPID from control unit
Switch Information
Channel Statistics
DCM Change
Possible changes1. Add A0 to 50002. Add A1 to 50003. Add A0 to 40004. Add A1 to 4000
Possible changes1. Add A0 to 50002. Add A1 to 50003. Add A0 to 40004. Add A1 to 4000
Control Unit
5000
Control Unit
5000
Control Unit
4000
Control Unit
4000Static
CHPID Static
CHPID
Managed CHPID (A0)
Managed CHPID (A0)
Managed CHPID (A1)
Managed CHPID (A1)
Static CHPID
Static CHPID
80%
80%53%
53%
53%
New Old