1 October, 2007 PICMG Hardware Platform Management
2
Introduction to Pigeon Point Systems
• Delivering world-class management components for open modular platforms– Focused on delivering outstanding PICMG management
controllers– Dependable for specification compliance, interoperability
and responsive support– Proven in dozens of companies and with years of
interoperability experience• Founded in 1997, headquartered in Scotts Valley, California• Please visit www.pigeonpoint.com for more details
• See www.pigeonpoint.com/library.html#presentations for the most recent update of this presentation
3
Broad Range of Tier 1 Customers for Pigeon Point™ Platform Management
• Notable examples in key market segments:– Telecom Equipment Manufacturers: Alcatel-Lucent, Nokia,
Siemens– Integrated platform providers: Motorola ECC, Sun
Microsystems– ATCA/AMC board developers: above plus Adlink Technology,
Advantech, Bittware, CES, Diversified Technology, Emerson NP Embedded Computing, Interphase, McAfee, NMS Communications, Qualcomm, Portwell, Znyx Networks
– Electronic packaging companies (shelf vendors): Pentair/Schroff/Electronic Solutions, Rittal/Kaparel
• See www.pigeonpoint.com/customers.html for public list
Pigeon Point, µShelf Manager™ and µCarrier Manager™ are trademarks of Pigeon Point Systems
4
Pigeon Point Leadership Cited by Venture Development Corporation (VDC)
• May, 2007 VDC (www.vdc-corp.com) study of ATCA/µTCA market assessed “build vs. buy” choices for management subsystems
• Key VDC conclusions include:– Most Shelf Managers shipped with ATCA platforms are
based on designs from Pigeon Point– Among IPM Controllers not designed in-house, nearly all
are based on Pigeon Point’s off-the-shelf designs– Most µTCA platform vendors use or plan to use shelf
management technology from their primary supplier for ATCA shelf management technology, Pigeon Point Systems
5
The PICMG “TCA” Landscape
• Six years of development for 4 major spec families; goal to maximize sharing of platform management facilities
• Specification development complemented and strengthened by regular interoperability testing
• Result: sophisticated, but complex, platform management architecture based on IPMI, tuned to PICMG needs
6
Overall Purpose of PICMG Hardware Platform Management
• Monitor & control low-level aspects of boards, modules and other Field Replaceable Units within a shelf
• Watch over basic health of the shelf and its constituent FRUs, report anomalies, take corrective action when needed
• Provide accessibility for inventory information & sensor readings
• Archive (and forward, as appropriate) event reports and failure notifications from boards, modules and other intelligent FRUs
• Manage power, cooling & interconnect resources in the shelf• Enable visibility into a shelf for a logical System Manager—
some mix of software + “swivel chair folk”
7
Overall Approach to PICMG Hardware Platform Management
• Focus on low level h/w management– Mandatory facilities on all boards, modules and shelves
• Adopt Intelligent Platform Management Interface (IPMI) 1.5 Revision 1.1 as foundation– IPMI widely used in PC and Server industry
• Emphasize interoperability among independently implemented components
• Encourage and leverage ecosystem of “off-the-shelf”management components– Re-using such components can be a win for product
developers • Allowing them to focus on their value adds• Ensuring excellent interoperability when mature,
compliant components are chosen– Pigeon Point components used as examples here
10
AdvancedTCA Shelf Manager: Key Services
• Provide access to inventory information for all FRUs• Manage power consumption and backplane interconnects
– Using self-describing requirements in FRU info• Manage cooling resources for the shelf, responding to FRU-
configured temperature threshold events• Manage distributed collection of sensors
– Self-describing in Sensor Data Records (SDRs)• Collect events in a persistent store and optionally perform
configurable actions in response; IPMI facilities include:– Non-volatile System Event Log that stores N event
records intended for interpretation via SDRs– Platform Event Filtering (PEF) that provides mechanism
for configurable actions on events (such as pages or SNMP traps)
• Provide visibility to System Manager on all the above as desired
11
AdvancedTCA Shelf Manager: Key Services (Cont.)
• System Manager Interface: logical connection to shelf-external management– Specification-required for interoperability:
• IPMI LAN Interface, including Remote Management Control Protocol (RMCP)
– Likely Shelf Manager-specific additions: command line & SNMP interfaces
• Optionally dual redundant Shelf Manager– Assumed from same vendor; coordination protocols not
covered in specs• Specification allows broad implementation freedom for
logical Shelf Manager function
12
One Way to Build Dedicated ShMCs: Use Off-the-shelf Mezzanine Modules
• SO-DIMM-sized Pigeon Point ShMM-500R + shelf-specific carrier yields dedicated ShMC– Avoids massive “from-scratch” development– Small size of mezzanine makes for great physical
flexibility for dedicated ShMC placement in the shelf• See www.pigeonpoint.com/library.html#userdocs for Pigeon
Point Shelf Manager user documentation
13
ShMC Redundancy and Full Connectivity to System Manager
• Spec provides for dual ShMC redundancy w/ auto failover of ShMC IP addresses– Details of coordination are
left to implementation– ShMM-500R details
shown• R2.0 ECN-001 (ShMC cross-
connects) allows both ShMCs to be connected with both hubs for better robustness– Mandated by CP-TA ICD– ECN development led by
Pigeon Point
14
System Manager Interface
• Multiple Sys Man interfaces on most ATCA Shelf Managers– IPMI LAN Interface (RMCP) the
only ATCA-required interface– Typical ShMCs have at least
SNMP & command line, also• Problem: only RMCP has spec-
defined framework and semantics– RMCP is quite low level– SNMP is widely interesting, but:
• Management Info Base (MIB) is necessarily Shelf-Manager-specific
15
SA Forum’s Hardware Platform Interface (HPI) Provides a Solution
• HPI, layered above RMCP, can provide specification-defined:– Platform management API– SNMP MIB implementation
• Portable System Manager uses RMCP or either HPI interface– Preserves System Manager
investments• SAF’s HPI-to-ATCA Mapping
specification provides crucial compatibility assurance
16
OpenHPI: an Effective HPI Enabler
• Provides highly adaptable foundation with plug-ins for particular platform types (see www.openhpi.org)
• Already used for wide range of platforms, including ATCA & IBM BladeCenter
• Remote Procedure Call (RPC) enables HPI clients that are remote from OpenHPI daemon
• Pigeon Point OpenHPI has plug-in that delivers SAF mapping spec compliance w/ Pigeon Point Shelf Manager– Communicates via RMCP– On-/off-ShMM HPI service– Pigeon Point User Guide posted:
www.pigeonpoint.com/library.html#userdocs
17
AdvancedTCA IPM Controller (IPMC): Key Facilities
• IPMI-1.5-specified commands and FRU Information• Key AdvancedTCA extensions:
– Dual redundant IPMB-0 connection to Shelf Manager– Hot swap state management for FRUs (including
represented FRUs)– Electronic keying (commands + FRU Info) for point-to-
point & bused backplane interconnects– LED management, including color, lamp test– Fan control for interoperable fan trays– Payload power control & negotiation w/ shelf
19
Overall Approach to Layering AdvancedMC Facilities Above ATCA
• Fit smoothly into established PICMG 3.0 conventions• Avoid impacting PICMG 3.0 R1.0 Shelf Managers w/ AMC.0• Reduce requirements on Module Management Controller
(MMC) to limit its cost and footprint• Require Carrier IPM Controller (IPMC) to represent MMC as
a full-fledged ATCA FRU to ShMC; Carrier IPMC:– Does power negotiation on AMC’s behalf– Does AMC E-Keying similarly to ATCA, but without
involvement of Shelf Manager– Makes Module SDRs visible to Shelf Manager
• Preserve IPMI foundation in MMC– Full IPMI sensor infrastructure optionally available for
MMC sensors
20
IPMC Build Option: Acquire Mature Solution(Pigeon Point Example Here)
• Integration-ready schematics
• Bench top HW for ramp up• Renesas H8S/216x and
Atmel AVR variants• Minimal board footprint
• Firmware source code• Common firmware base
for H8S and AVR• GNU tool chain supplied• Designed for easy
configuration
21
Renesas H8S-based ATCA IPMC and AMC Carrier IPMC; Atmel AVR-based MMC
• Single H8S for IPMC or Carrier IPMC– H8S/216x family used pervasively
in IPMI management applications– Excellent headroom for future
growth:• 33 Mhz 16-bit CPU• Up to 512 KB Flash• 40 KB SRAM• 6 I2C ports
– Allows advanced features:• LPC as payload interface• Serial over LAN with Intel
82571 NICs• Atmel AVR excellent for MMC role• Additional logic block per AMC site
22
Recent/Upcoming Developments with ATCA/AMC
• AMC.0 R2.0 adopted in November, 2006– Many clarifications and other improvements for hardware
platform management– New clock E-Keying facility
• PICMG HPM.1, the IPM Controller Firmware Upgrade spec, adopted in May, 2007– Spec development led by Pigeon Point; see Pigeon
Point article below for introduction and background– “Interoperable Firmware Upgrades for PICMG
Management Controllers”, RTC Magazine, August 2007• PICMG 3.0 R3.0 now under development in PICMG
23
HPM.1 Based Firmware Upgrades Address Key Need in PICMG Platform Management
HPM.1:• Defines interfaces for Upgrade Agents & IPM Controllers• Defines format for upgrade images w/ opaque upgrade data• Allows agent to upgrade firmware in arbitrary controllers
24
PICMG 3.0 R3.0 Progress Report
• Key focus: substantial requirements engineering edit– All requirements numbered for precise references:
• Within PICMG 3.0 and from other PICMG specs• Within profiles defined by related organizations such
as SCOPE or CP-TA• Within ATCA customer/vendor specifications
– Requirements restructured, if necessary, for that goal• Additional substantial effort on clarifications, accounting for
many of 1,700+ CRs (Change Requests) beyond enumeration-related changes– ~500 of those CRs in hardware platform management
• Important functional enhancements/extensions, also• Earliest adoption date: January, 2008
25
Other Functional Enhancements in Preliminary PICMG 3.0 R3.0
• Defined IPMI interface for telco alarm monitoring/control– Can be represented either by Shelf Manager or in-shelf
IPM Controller– Benefits generic System Manager applications
• Improved LED services, including identification of textual and graphic legends associated with each LED– Enables more complete/explicit operator interfaces in
generic System Manager applications• New “Set FRU Extracted” command provides standardized
mechanism for System Manager to communicate operator knowledge of physical FRU removal
• Additional option for IPMB-0 buffers on intelligent FRUs– Simplified buffer can allow improved noise margins
28
Overall Approach to Hardware Platform Management in µTCA
• Support AMC.0 AMC modules without change– Provide “carrier” environment that is equivalent to an
ATCA carrier in terms of interfaces with the AMCs• Reuse as much as possible of existing PICMG management
infrastructure– Including concepts, terminology, possibly
implementations– Minimize barriers to reuse of ATCA System Manager
applications with µTCA• Choose architectural approaches that facilitate the cost and
modularity goals of the µTCA initiative
29
µTCA Shelf Manager (ShM): Local or Remote to Carrier Manager
• ShM co-located w/ Carrier Manager– Especially relevant for a low-
end, one-carrier shelf– Interface between Shelf &
Carrier Managers is private• ShM located remotely from
Carrier Manager– Shelf-Carrier Manager
Interface mandated as IP– Certainly the choice for a
multi-carrier shelf– Many options for “off-MCH”
Shelf Managers
30
µTCA Shelf Manager Responsibilities
• Represent the entire shelf (including up to 16 MicroTCA carriers, each represented by a Carrier Manager)– Preserve representation offered by an ATCA Shelf
Manager as much as possible• Each carrier appears as an IPMC-managed “board”
with managed FRUs (all the modules in that carrier)– Maintain System Event Log and Sensor Data Record
Repository• Requires tracking shelf-wide FRU population
– Remote Management Control Protocol interface to System Manager mandated, as with ATCA
• Manage cooling for the shelf– Cooling Units in one carrier may cool other carrier(s)– Shelf Manager has data structure that establishes
relationships between “cooler(s)” and “coolee(s)”
31
Non-responsibilities for µTCA Shelf Manager, Compared to ATCA
• No power management; done by Carrier Manager, instead– Tight coordination with in-carrier Power Modules
necessary• No E-Keying; done by Carrier Manager, instead
– Specification addresses only intra-carrier E-Keying– Inter-carrier fabric connections are out of scope
• No use of IPMB-0 to communicate with “boards” (actually, carriers) in the shelf– Shelf-Carrier Manager Interface replaces IPMB-0 as
shelf-wide management link with carriers, via their Carrier Managers
32
Carrier Manager and MCMC(s) Provide Intra-Carrier Management
• Logical Carrier Manager represents entire carrier to Shelf Manager– Uses bused IPMB-0 to
connect with EMMCs– Uses radial IPMB-Ls to
connect with MMCs– Uses I2C to access Carrier
FRU Info Device– Can be implemented on dual
redundant basis• MicroTCA Carrier Management
Controller (MCMC) provides local management for MCH– Sensors for MCH and E-
Keying for MCH fabrics
33
Carrier Manager “Owns” All Resources of its MicroTCA Carrier
• FRUs (including AMCs, Power Modules, Cooling Units, possibly OEM Modules) presented in fixed ranges of FRU IDs for each type– Example: Shelf Manager needs to change fan speeds
on a Cooling Unit• Issues fan speed command to Carrier Manager w/
appropriate FRU ID• Carrier Manager forwards to appropriate CU
• Carrier Manager handles power budgeting, allocation, monitoring for all Power Modules in the carrier– Similar to AMC.0 Carrier IPMC responsibilities, but using
independently implemented modular Power Modules• Carrier-wide Device Sensor Data Repository merges sensor
data from all modules in the carrier– Sensor data for a given module is added/subtracted from
carrier-wide repository when module inserted/extracted
34
MCMC Handles Local Management of its MCH Module
• Implementations may combine the MCMC function with the Carrier Manager function on one MCH-based microcontroller
• Nevertheless, these are distinct logical functions, which is especially important in dual MCH architectures:– One logical Carrier Manager for entire carrier executes
on one of the two MCHs, can switch over if necessary– Two MCMCs are always active, since they represent the
local resources of each MCH, including sensors & fabrics
• Carrier FRU Info stored in I2C SEEPROM allows MCH (inc. Carrier Manager, MCMC) to automatically adapt to carrier specifics
35
µTCA Power Modules (PMs)
• Need to both:– Preserve power-related
interfaces to AMCs– Support modular
implementations with optional redundancy
• PMs provide:– Radial power channel to
each module site– Presence-detect & enabling
for module (E)MMCs– Autonomous power on for
selected FRUs at startup– Optional redundant PM with
hardware takeover for one failed peer
• PMs rely on Carrier Manager for policy decisions
36
µTCA Cooling Units
• Support fan management commands originally designed for ATCA
• Scope of cooling effects may cross MicroTCA Carrier boundaries
• Fan geography data structures in Shelf FRU Info define fan <-> FRU relationships
• Like PMs, managed by Enhanced MMCs (EMMCs)– Equivalent to MMCs,
but with dual IPMB-0
37
AMC.0-compliant AMCs Operate in Either µTCA or ATCA Carriers
• Interfaces presented by the carrier are functionally identical– Redundancy of fabric
and IPMB-L connections may need special care
• Principles that motivated AMC.0 architecture benefit MicroTCA.0, as well
38
Key µTCA Choices for Hardware Platform Management with AMC Modules
• Fully preserve ATCA carrier interface for hardware platform management
• Require carrier controller to represent MMC as a full-fledged FRU to µTCA Shelf Manager; Carrier Manager:– Does power budgeting/allocation on AMC’s behalf,
directly and without involvement of Shelf Manager– Does AMC E-Keying similarly to ATCA, but without
involvement of Shelf Manager– Makes SDRs for all modules in the µTCA carrier visible
to Shelf Manager as if owned by the Carrier Manager
39
Recent/Upcoming Developments with µTCA/AMC
• AMC.0 R2.0 adopted in November, 2006– Many clarifications/improvements for hardware platform
management– New clock E-Keying facility
• PICMG HPM.1, the IPM Controller Firmware Upgrade spec, adopted in May, 2007– Applies to MTCA.0, AMC.0 and PICMG 3.0 controllers– Spec development led by Pigeon Point
• Some revision areas for MTCA.0 already queued, but no active revision effort for base spec under way– Main queued revision area: synching w/ AMC.0 R2.0– Also under way: Rugged µTCA, with a first “dot spec”
(MTCA.1) focusing on air-cooled contexts
40
Rugged µTCA: Hardware Platform Management Needs
• Few areas of necessary extension identified thus far– One key need: a way for system components to self-
identify temperature ranges and ruggedness levels
41
µTCA Management Controller Build Option: Acquire Existing Solution (e.g. Pigeon Point)
• Integration-ready schematics• Bench top HW for ramp up• MMC w/ Atmel AVR• MCMC, EMMC w/ Renesas H8S
• Firmware source code• GNU tool chain supplied• Simple and flexible
configuration
42
Pigeon Point MCH ManagementSolution Widely Used
• Covers MCMC, µCarrier Manager, µShelf Manager• Excellent interoperability with other compliant µTCA
components demonstrated in PICMG (µ)TCA-IWs• Used in MCHs from Emerson, Motorola and Advantech (see
above) with others in development
43
• 15 TCA interop workshops, 14 w/ management testing– ~15-25 participating companies in each event
• ~40 test plans for different functional areas– Developed by participants; PPS provided major subset
• Workshops add emphasis on new PICMG architectures as testable products become available– Primary focus of main workshops has been ATCA/AMC– µTCA emphasis now ramping up
• In addition, µTCA-IWs have provided additional focus on µTCA– 9 µTCA-IWs thus far, including #9 in Germany, 9/07
• Communications Platform Trade Association (CP-TA) aims to augment PICMG Interops with formal testing– Initial focus is ATCA components– Expected to add formal testing for AMC & µTCA, also
PICMG TCA-IWs Validate Specs and Products
44
Implementing PICMG Management Controllers Takes Serious Engineering
• Strongly consider using existing components
• Pick validated, interop-tested components
• Participate in TCA-IWs with your own products
• Six years of PICMG h/wplatform management spec work has yielded:– PICMG 3.0: 187 pages– AMC.0: 98 pages– MTCA.0: 110 pages– HPM.1: 66 pages
45
Summary: ATCA/µTCA/AMC Hardware Platform Management…
• Represents a significant PICMG effort to: – Define interoperable extensions to IPMI– Maintain and enhance these extensions through
interoperability testing and further specification work– Maintain maximum consistency among different platform
architectures such as ATCA, AMC and µTCA• Constitutes a serious engineering project at any of the shelf,
board or module levels– Even if development starts from existing PICMG 2.9 or
conventional IPMI components• Can significantly benefit from the use of off-the-shelf
components that are validated and interoperability-tested
47
Speaker Background
Mark Overgaard founded Pigeon Point Systems in 1997 to focus on products and services supporting the adoption of open modular platforms to replace proprietary architectures in the telecommunications market. He is a leader in the technical subcommittees of PICMG, including the management aspects of AdvancedTCA, AdvancedMC and MicroTCA. The Pigeon Point platform management components are widely used for shelf, board and module level hardware platform management for all three of the major PICMG platform architectures. Previously Mark was VP, Engineering at Lynx Real-Time Systems (a Unix-compatible RTOS supplier) and TeleSoft (a major supplier of embedded development solutions for Ada).