Network Equipment Compliance and Installation Challenges

© 2014 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

Network EquipmentCompliance and Installation Challenges

Robert E. FullerLead Member of Technical StaffAT&T Labs

October 2014


2

• Lead Free Solder/RoHS 6/6 Compliance

• Thermal Management, Airflow Requirements, Impacts, and Solutions

• Cable Management

• Power Distribution

• Equipment Size and Weight

Topics


3

Lead Free Solder/RoHS 6/6 Compliance Challenges• Industry concerns about solder joint reliability in lead

free products remain

• EU Exemption expires July 2016

• Vendors need to provide clear understanding of • what products are currently compliant, what will be

discontinued, and what will be re-designed to be compliant

• The impacts on sparing and maintenance procedures


4






Topics


5

• Heat and Power Estimation - Manufacturer’s Specified Heat Values vs. Actual Measured Values.

• Cooling the Chassis – Difficult to air cool equipment beyond 20 – 25 KW heat loads within acoustic noise limits. Trade off: air flow vs. electronics.

• Removing the Heat – How to cool and re-circulate the hot exhaust air from air cooled equipment. Older building are not designed for high heat loads.

• Improving PUE (Power Usage Effectiveness)

• AT&T Standards

Thermal Management, Air Flow Requirements, Impact, and Solutions


6

Challenges of Heat and Power estimation

• Manufacturer’s Specified Heat Values vs. Actual Measured Values

Results reporting Requirements

• AT&T ESP (Engineering and Space Planning) Form – Requires separate and distinct values for Nominal and Maximum

• Part specific values (e.g., chassis, cards, etc.)

• ATIS (Alliance for Telecommunications Industry Solutions)– Charts heat and power at 0, 25%, 50%, 75%, 100% (max) equipment utilization

• System total values (measured, calculated or estimated)

Heat and Power Estimation


7

• Air Cooling: Difficult to air cool equipment beyond 20 – 25 KW heat loads within acoustic noise limits, power, air impedance (rear door, grill, etc. issues), high speed exhaust. Chassis design trade off: air flow vs. electronics.

• Liquid cooling (submersive, direct contact)

• Hybrid: Air cooling with liquid cooling to hottest components

Cooling The Chassis


8

Increased hot/cold aisle temperature gradient leads to greater cooling efficiency. This can be achieved by:

• Consistent front to rear airflow

• In cabinet hot/cold isolation

• Hot or cold aisle containment

Removing the Heat:Efficiency Improvements


9

Consistent Airflow: Typical Solution for Side Airflow Chassis

Airflow Redirector - Plenum


10

Consistent Airflow: Typical Solution for Fuse Panels

Rear ViewFront View

Heat Ramp


11

Cabinet Modifications:Hot/Cold Aisle Isolation

Filler panels, top air blocking panel,

silicon pass-through inserts, foam inserts


12

Cabinet Modifications: Top Panel Isolation

Top panel brush and cover plate kit


13

Hot/Cold Aisle Containment

• “Bathtub” Solutions

• Can be hot or cold aisle isolation depending upon the particular cooling infrastructure

• 3 sided containment

• 5 sided containment


14


All distributed cooling systems require containment to effectively separate the supply and return air. Containment increase the temperature gradient (delta T) between these air masses by reducing mixing of supply and return air.


15


Three sided containment will provide 70% effectiveness in separating supply and return air.


16


Five sided containment will provide 96% effectiveness in separating supply and return air.

It is not necessary to provide difficult top containment to get significant results from adding containment to equipment areas.


17

Aisle Containment Installation ChallengesAisle Containment in Partial Cabinet Line-Ups


18

Aisle Containment Installation Challenges

Containment Doors and Panels


19

Distributed Refrigerant Cooling (DRC)

DRC Solutions

• In row cooling - requires hot/cold aisle separation

• Back of cabinet cooling

• DRC direct to equipment


20

Distributed Cooling Installation Challenges

Equipment may not have the necessary airflow to support passive rear door coolers nor air flow re-directors to push air down and to the sides to in-row cooler intakes. Small, high speed exhaust outlets compound the problems.

Possible solutions include active doors (not currently commercially available), placing in-row coolers in adjacent aisles directly behind the equipment, or making aisle space into a giant in-row cooler plenum.


21

Economization

• Economization is using the natural exterior temperature as a cooling source and is becoming a requirement for all cooling systems.

• The key to effective “Economization” is to increase supply air temperatures modestly and drive return air temperatures as high as possible to allow for the greatest window of “economization hours” in any climate.

• Economization can be done with heat exchangers to address air quality concerns or may be done with direct air exchange.

• The use of “Economization” and “Containment” will generate an average 40% improvement of PUE in a mild climate. Cool climates will see a greater savings while warm climate will see less but still significant savings.


22

AT&T Thermal Standard

• AT&T’s Thermal Standard is ATT-812-000-705. The standard details traditional cooling systems, raised floor systems, DRC systems, multiple containment strategies, economization and thermal management space solutions.

• A support document numbered ATT-812-000-705 ANNEX has been created to discuss and illustrate manufacture specific solutions and products in support of ATT-812-000-705.

• AT&T’s Thermal Standards are proprietary documents and require approval for access.


23






Topics


24

Cable Management ChallengesOffice cable racking and chassis cable management are size constrained• Typical cable types are cat 5/6 cable and 1.6/2.0 mm breakouts/jumpers

• Fiber cable management may not accommodate fiber drip loops

• Chassis may support up to 600 cat 5/6 or 1000 individual fiber ports (2000 strands)

• High density MPO connectors used to increase port density require the use of breakout cables to the optical distribution frame

• Rack mount equipment may not have any vertical cable management Large quantities of cat 5/6 require custom cable management Large quantities of fiber are difficult to house in vertical FPS

• Distribution cable with 900 micron fiber used on front is upjacketed to 2.0 mm

• Breakout cable uses 1.6mm cordage end-to-end • Smaller within bay, but more costly and takes more rack space


25

Cat 5/6 UTP

“Thin” versions of Cat 6 cable are approximately 20% smaller, 10% lighter, and

more flexible than standard Cat 6 cable

Less congestion on card faces, in equipment and rack cable managers, and in overhead ladder racking leading to lower installation costs and improved ease of maintenance


26

Fiber Initiatives – 1.2 mm Jumpers (Simplex and Duplex)

Fiber Cordage Used to Manufacture Jumpers

3.0mm, 2.0mm, 1.6mm, 1.2mm

simplex BIF cordage

3.0mm, 2.0mm, 1.2mm

duplex BIF cordage


27

Fiber Initiatives – 1.2 mm Jumpers (Simplex and Duplex)

1.2 mm jumpers are 44% smaller than 1.6mm jumpers = 1.5 times more fibers in the same space

1.2 mm jumpers are 64% smaller than 2.0 mm jumpers = 2.8

times more fibers in the same space

Both 1.2mm simplex (shown) and duplex jumpers successfully trialed in AT&T network and now in production use

1.6mm

72 fibers 72 fibers

1.2 mm


28

Fiber Initiatives – 1.2 mm Breakout CableCurrently 1.6mm breakout cable is used when 900 micron distribution cable up-jacketed to 2.0 mm is too large

1.2 mm is 42% smaller, 30% lighter than 1.6mm and allows 2.8X more 1.2 mm fibers in the same space than 2.0 mm up-jacketed fiber

1.2mm 12 fiber breakout cable Penalty: 40% larger than 900 micron for 72 count cable

1.6mm 12 fiber breakout cable Penalty: 80% larger than 900 micron for 72 count cable


29

Fiber Initiatives – 96 Strand Micro-Cables

Eight subgroups each with twelve 250 micron fibers


30

Fiber Initiatives – Backbone Micro Cables

72f Distribution Cable – 900 micron fiber18.5mm dia.: 6.6x cross-sectional area of microcable

72f Breakout Cable – 1.2mm fiber22.0mm dia.: 9x cross-sectional area of microcable72f Breakout Cable – 1.6mm fiber24.9mm dia.: 12x cross-sectional area of microcable

Prototype 72f Micro Cable – 250 micron fiber: 7.2 mm Diameter

12f 1.2mm Breakout Cable: 6.4 mm Diameter

12f 1.6mm Breakout Cable: 7.6 mm Diameter


31

Fiber Initiatives – MPO Connectors

• Used to increase density and versatility of physical ports compared to SC and LC connectors

• High density low speed (1 GE) ports where card faceplate does not have sufficient room for LC connectors

• 10 x 10 GE CFP in 100 GE “electrical” port adds versatility and reduces costs for composite 100 GE signal compared to full 100GE

• 2 x 4 x 10GE in 100 GE “electrical” port adds versatility and reduces costs for composite 40 GE signal compared to full 40GE


32

Fiber Challenges – MPO Pin Outs


33



34


10 x 10GE

1

1

10

10

Receive

Transmit

1

10

10

1

Receive

Transmit

OR

Back to back roll over cable

Back to back cross over

cable

Back to back hybrid cable


35


10 x 10GE

1 10

Transmit

1 10

OR

1 10

Receive

Receive

10 1

Transmit


36


1 14 4

4 x 10GE

1 14 4

4 x 10GE

2 x 4 x 10GE

1

1 4

1Receive

Transmit

4 4

1 4

Receive

Transmit

Legs are a straight and roll over cable variant


37


8 x 12 x 1GE (e.g. 96 port 1 GE card)

1

1

12

12

Receive

Transmit

1

12

12

1

Receive

Transmit

OR


38


1

1

12

12

Receive

Transmit

OR ???

OR

2

4

5

3

43

2 5


39

MPO Cable and Connector Challenges

MP01MPO2MPO3MPO4MPO5MPO6MPO7MPO8MPO9MPO10MPO11MPO12

MPO13

MPO15MPO14

MPO16MPO17MPO18MPO19MPO20MPO21MPO22MPO23MPO24

KEY

UPPER ROWLOWER ROW

Cable and breakout must be compatible with chassis cable management, FPS, and rack installations

Optics “Pin outs” vary by manufacturer

Connector numbering varies by manufacturer

A5

A4

B1B2B3B4

B5B6B7B8B9

B11


40

Fiber Challenges – MPO Breakout Cables to LGX• Pin outs can vary for the same size and speed

connector and will vary for same size connector for different speeds.

• MPO – MPO cable pin outs are specific to the optics at each end and can be plugged in with the ends backwards.

• Need different breakout cable pin outs (MPO – SC/LC) based on connector pin out and size.

• Fiber strand colors must correspond to the T/R pairs at the breakout end. This means different cable builds, not just different labeling. (slide?)


41

Fiber Initiatives – MPO Breakout Cables

900 micron breakout to

LGX

1.6 mm breakout to equipment front

panel


42

Fiber Initiatives – MPO to MPO Cables

24 fiber MPO-MPO

Pin out varies by both application and optics

• Straight

• Cross-over

• Rolled

• Hybrid

• And more

And how do you know?


43






Topics


44

Power Cable Management Challenges

• Chassis may require up to 36 DC power cable pairs or 24 single phase 220 VAC cords

• Typically no power cable management is provided

• AC cords typically do not “lock” into the chassis receptacles

• N + 1 power supply redundancy schemes are not practical to implement in the field (would require N + 1 BDFB loads)

• A + B power feed redundancy to each power module results in double the number of feeds and little reliability improvement


45

• A DA Plant is located near the equipment it serves, saving copper

• Scalable from 850A – 3800A depending upon required battery reserve

• In some cases, requires a new battery technology Sodium Metal Halide, aka “Sodium Salt”

• At room temperature, battery is table salt, nickel, iron, and alumina

Solution: Distributed Architecture (DA) Power Plant


46

DC Power Plant Architecture Features

• Single large DC power plant for office

• Large DC batteries in the power room (basement)

• Long DC primary power circuits

• Requires BDFBs

• Requires short secondary distribution

• Requires bay FAPs for most NEs

• Voltage drop from BDFB to NE is typically limited to 0.5V loop

• Short AC feeds to rectifiers

• Heavy lead acid battery technology

• Multiple small DC power plants for areas

• Small DC batteries in the equipment area

• “Primary” distribution in the traditional sense is eliminated

• BDFBs are eliminated

• “Secondary” distribution technically becomes primary

• Bay FAPs can be eliminated for most NEs

• Voltage drop from PBD to NE = 1.75V loop, effectively eliminating all transition devices; i.e., cable size is #2 AWG for most 125A circuits

• Longer AC feeds to rectifiers – more PDSCs

• Lighter alternate battery technology

Centralized Architecture Distributed Architecture (DA)

DA Plants = less copper


47

Solution: Mini “BDFB”

Six 600A loads, 10 fuse positions/load

Equipment Cost Savings of ~50%


48

Solution: Bridging Clips

Bridging Clips are used on Demarcation Fuse Panels to provide 2 or more outputs for a single input.

Bridging clips are used with high current (125A) BDFBs where

individual power inputs are fused at >50A and the total load of multiple fuse outputs will not exceed 125A.

Special cases include planning for growth and redundant chassis power

supply inputs.

This saves secondary power cable installation costs, BDFB fuse

positions, cable rack and cable hole congestion = $$$.


2:1 Bridging Clips

49


2:1 Bridging Clip Example

50


4:1 Bridging Clip Example

51

16 60A redundant feeds 4 125A feeds


52






Topics


Equipment Size and Weight Challenges

• Rack mount units may exceed the 44 ru standard 7’ rack/cabinet VPO and/or may be very deep

• Both standalone and rack mount chassis may exceed 1500# when fully populated.

• “Green” shipping means everything in one >1500# crate – how do you move it from loading dock to installation site especially if FRUs must be removed first

• Challenge to deliver to and transport equipment within the office (too tall, too heavy, cannot be tilted)

• Challenge to install in cabinet due to size and weight

• Cabinet/rack must support 1500# or more

53


54

Cabinet and Rack Design Challenges

• Must incorporate hot/cold aisle separation while allowing front to rear cable access and provide vertical cable management

• Mounting rail design must support plenums needed to correct side air flow.

• Heat ramps to correct top to bottom airflow (e.g. fuse panels) must allow for various equipment offsets, depths, etc.

• Cabinet must support over 1500# while maintaining zone 4 seismic compliance

• Cabinets, if required, must accommodate equipment over 44 ru high

• Cabinet design must allow for in-cabinet cable management


55

Cabinet ModificationsReduced Top and Bottom Cowlings


56

Cabinet ModificationsPocket Rails, Base Chassis Support


57

Questions?

AT&T & Our Vendors

Partnering for Success

Network Equipment Compliance and Installation Challenges

Documents

att marks

att logo

respective owners

heat values

challenges of heat

lead free solderrohs

air flow requirements

high heat loads