"Internet Influence on Server Evolution,"

Internet Influence on

Server Evolution

CMG 2000 ConferenceOrlando, FloridaDec 11, 2000

John BaudrexlIntel Corporation

Technology & Research LabsLabsLabsIntelIntel

© 2000 Intel Corporation

2

LabsLabsIntelIntel*All trademarks and brands are the property of their respective owners

Agenda Net Impacts The I/O Squeeze Solution: InfiniBand* Technology A Clustering Vision Some Issues Summary

Note: The following material represents the views of the author only, and is not intended to contract, promise, or represent any current or future Intel products.


3


net/www are growing rapidly 1.5B webpages, 1M added daily 400M users, 95M host machines 13M domain names

china: 534K domains in 1 week bandwidth demand driven by:

new user access technologies new applications

larger, latency-sensitive data immense rise in IP traffic

IP BW doubles every 100 days mobile access on the rise

burgeoning e-biz adds fuel forcing buildout of high- bandwidth optical networks

Net Impacts

‘99

# URLs on the WWW

Data Size (KB)

0

100

10000

E-mail Image Hi ResImage

Movie

1000

10

File

100000


4


internet growth giving rise to specialized service providers (SPs) NAPs, ISPs, ASPs, search engines, cacheing, etc colocation services

web/E-biz hosting, storage outsourcing, electronic brokers gears of internet economy now rely on SPs to

be agile, scalable, highly available SPs deploying powerful, rack-mounted gear

mix of networking, storage devices many single function servers

3 traditional tiers: front-end, mid-tier, back-end SPs beginning to influence server design

data centers are becoming a crossroads... requirements: security, all the ‘bilities’

Net Impacts


5


Net Impacts many SPs are locating near NAPs for better performance (response)

17 router hops on ave often tens of ms apart caching technologies are proliferating

more data center floor space needed difficult to build big ones expensive: $500-$700/sf not uncommon retrofits not much cheaper managed services hold key to improved ROI global situation

in the background, Moore’s Law keeps time... so what does all this mean?


6


incentive/capability to make servers result: new “higher density” servers

powerful, memory-rich more units in less volume, floor space easier to deploy and manage rapidly/remotely configurable reasonable cost more modular, available, reliable the server is the FRU - service it offline

prediction: high density storage also coming 10,000 petabytes of storage anticipated online

worldwide by 2005 much of it stored in data centers

smaller!

Net Impacts

“1U is too big.”


7




8


The I/O Squeeze shrinking form factor colliding with:

increasing Hz, shorter parallel busses CPU and memory power dissipation (heat) size of power supply/heat removal components size of on-board I/O infrastructure

very careful signal/layer layout required thermal challenges abound

in opposition with reliability not much room for fans, PS, VRMs, etc. liquid-based cooling solutions being pondered

can’t pack enough I/O close enough anyway without violating PCI standards specs...


9


PCI bus technology: 1994 - industry readily embraced spec

32-bit/33MHz propagated rapidly first generation chipsets had limited scalability

1996 – first 64/33 bus available 1998 – first 64/66 bus available 2000 – first PCI-X bus (100MHz) available trends -

hot plug busses tuning/awareness has led to better performance

getting good feeds and speeds nowadays seeing 80-90% efficiency on modern busses BW exceeds most I/O interface technology req’s multi-channel cards emerging

The I/O Squeeze


10


PCI bus technology issues -

memory mapped “load-store” ops stall faster CPUs no standard driver card behavior scalability - out of slots, out of I/O reliability - bus parity err causes system reset stretching a parallel PCI bus is impractical

cost (more pins, cables, connectors, etc.) skew/signal integrity issues increased power needed

The I/O Squeeze


11


The I/O Squeeze

lower speeds = more slots, longer busses multiple cards/chls can lead to chaotic bus behavior

higher speeds = fewer slots, shorter bus lengths 133MHz pushing limit of modern parallel busses one slot looks like internal bus extension

||||||

||||||

|||||

||||||

|||||

64/6664/66||||||

||||||

|||||

||||||

|||||

||||||

||||||

|||||

||||||

|||||

64/1064/1000

||||||

||||||

||

64/1364/1333

MemCtrl

I/OHub

Memory

I/OBridgeI/O

BridgeI/OBridgeCPU

™

CPU™

high density system boundary

PCI Slot

s

traditional system boundary


12




13


InfiniBand* Technology...a new interconnect

Rev 1.0 spec at: www.infinibandta.org


14


InfiniBand* Technology new industry specification endorsed by majority server OEM, I/O people many new concepts - products avail end ’01

allows separation of CPU/memory from I/O reliable, scalable, switched, packet-based, hardware-routed, serial message-passing fabric architecture designed to endure at least 10 years

defines protocol, pgm model, wire, interop, err handling expected to replace PCI in servers

new type of I/O bridge no PCI slots anymore, plug in hoses instead

based on VI Architecture: QP/CQ concept, rDMAs high speed serial pt-to-pt links (2.5Gb/sec)

widths: 1x = 500MB/sec, 4x = 2GB/sec, 12x = 6GB/sec bi-dir range of service classes, flow control, QoS “verbs” software interface for IPC, I/O

many other features – see website


15


InfiniBand* Technology

RouterRouter

CACA

NetworkNetwork

Link

Link

SysSysMemMem

CPUCPU

CPUCPU

MemMemCntlrCntlr HCAHCA LinkLink

SwitchSwitch LinkLink

Link

Link

TCATCA TargetTarget

TCATCA

TargetTargetH

ost I

nter

conn

ect

Hos

t Int

erco

nnec

t

RouterRouterLinkLinkSource: InfiniBand* Trade Association

one interconnect for storage, net, IPC


16




17


A Clustering Vision skinny powerful servers

1-2 IA processors, maybe even Itanium™ gobs of memory (>4GB if you want) all-InfiniBand* I/O – no PCI slots completely remote management external 48V power supply

legacy free no floppy, no keyboard, mouse, video, CD-rom,

serial, parallel - all gone! no hard drive – boot from anywhere reduces volume, minimizes human interaction

only 2 kinds of hoses attached power cord InfiniBand* cables


18


A Clustering Vision cluster together as needed

deploy lightweight IPC protocol such as VIA out-of-band management and configuration

attach external I/O devices (gateways) storage (IBA-FC/SCSI), network (IBA-LAN/WAN) etc.

add intelligent, scalable network services distributed, smart packet processing at line rates

build in large distributed memory use lightweight IPC transport over IBA remote memory access << local/ext disk access fast response time – replaces some cache tech meets e-commerce scalability requirement

new price/perf potential for e-Biz IA platforms usually best price/performance IBA* chipsets low cost/port over time


19


A Clustering Vision ten nodes

20 Itanium™ (“McKinley”) processors gobs of MIPS, flops

80GB distributed RAM >1TB internal HDD 40GB/s bi-directional redundant I/O BW

two 4x IBA* links per node

6kW peak power 3kW average

total 10U - 18” high!


20




21


thermal challenges some compelling solutions emerging rack may also now require thermal engineering

products implementing InfiniBand* links still a year away

partners will gain early access true legacy free systems still a pipe dream

product quality BIOS, backward compatibility work underway in this direction

clusters are hard to implement and manage reliable failover hard enough performance clustering harder still linux helps some lots of momentum in the industry to change this

Some Issues


22


increased density impacts the data center power delivery

W/sf2 req’s increasing faster than industry supply cooling

new construction: raised floor retrofit: ceiling supply only

cost effectiveness site selection demands proximity to power/net grid reality is multi-dimensioned, complex

provision for greater backup capability more batteries, generators, conversion/delivery

environmental zoning prevents storage of sufficient diesel fuel! heat plume an EPA concern

despite apparent economies of scale,“super” data centers will be rare

Some Issues


23


Product Heat-Density Trends

Some Issues


24


electrical power consumption in U.S. today: 8% internet, 5% computing projection: by 2010, 30% of U.S. consumption will go to IT

data center power density trend mainframe days: 20-40 W/ft2

“standard” today: ~60-100 W/ft2

(way) high end today: ~200 W/ft2

some folks looking for 300-500 W/ft2

cooling physics 1 ton of AC, 500cfm airflow at 20oF ΔT dissipates ~4kW generating 1 ton of AC takes ~1kW of input power harder to remove heat than to supply power

equipment heat generation (commonly operate at ~50% of nameplate listed below) SMP servers today: 7U = 700W; 4U = 400W; 2U = 200W; 1U = 150W 3U Clariion 5000-series 10 drive FC JBOD disk array = 350W 11U Cisco Catalyst 6000-series 32Gb LAN/WAN switch = 1.3kW

a 42U rack: occupies 15-20 ft2, accounting for walkways and access 100 W/U * 42U * 50% nameplate ÷ 20 ft2/rack

power density requirement ~105 W/ft2

Issues: Density FactoidsSource: Intel ASL research


25




26


Summary internet growth creating crossroads at

service providers, data centers Moore’s Law, SPs are driving server

evolution to higher densities pushing I/O out of the box data center infrastructure not ready for higher

density equipment impacts need to develop mitigation strategy

IBA* Architecture fits data center req’s convergence creates compelling sol’n space

high density large memory InfiniBand* Architecture


27


Backup Material


28


0

50

100

150

200

250

300

350

400

450

1997 1998 1999 2000 2001

Year

Max

Ban

dwid

th (

MBp

s)

ATMEthernetFC1394SCSI

Source: In-Stat, Electronic Trend

Chart based on bi-directional bandwidths, and onestimated availability of new standards/technologies

I/O Interface Bandwidth Trend: Increasing!


29


HOW LONG TO REACH 30% PENETRATION? (U.S. Internet Council, Apr. 1999).

Internet -- 7 years Television – 17 years Telephone – 38 years Electricity – 46 years.

# ONLINE WORLDWIDE 1998 – 95.43 million people. (eMarketer eStats 1999). 1998, Dec. – 144 million (IDC, Dec. 1999). 1999, Dec. – 240 million (IDC, Dec. 1999). 2000, Sept. – 377 million (Nua Internet Surveys (www.nua.ie)2002 – over 490 million (Computer Industry Almanac, Nov. 1999). 2005 – over 765 million (Computer Industry Almanac, Nov. 1999).

Net Explosion

Source : An IDC Telebriefing, March 4, 1999

0100200300400500600700800

1997 1998 1999 2000 2001 2002 2003

DevicesUsers

http://www.usic.org/usic99/usic_state_of_net99.htm


30


HOST COMPUTERS: In July 1999 there were 56.2 million "host" computers supporting web pages. In July 1997 there were 19.5 million host computers, with 3.2 million hosts in July 1994, and a mere 80,000 in July 1989. (Internet Software Consortium – Internet Domain Survey).

Monthly Average Hosts in Millions (Nov. 2000, Telecordia Technologies www.netsizer.com)Month 1998 1999 2000 Jan30.3529 44.2292 70.2238Feb31.3906 48.0616 72.9501Mar32.4267 50.4808 75.1010Apr33.5716 53.4689 77.1504May34.5158 55.8092 80.1905Jun35.4504 57.3499 82.8749Jul36.4862 59.2922 85.8418Aug37.2169 61.3617 88.6271Sep38.5637 63.3344 91.5614Oct39.9339 65.6407 94.2659Nov41.5190 67.3174 -Dec43.0331 68.8020 -

Net Explosion


31


BACKBONE CAPACITY: The capacity of the Internet backbone to carry information is doubling every 100 days. (U.S. Internet Council, Apr. 1999).

DATA TRAFFIC SURPASSING VOICE: Voice traffic is growing at 10% per year or less, while data traffic is conservatively estimated to be growing at 125% per year, meaning voice will be less than 1% of the total traffic by 2007. (Technology Futures, Inc March 2000).

EMAIL VOLUME: Average U.S. consumer will receive 1,600 commercial email messages in 2005, up from 40 in 1999, while non-marketing and personal correspondence will more than double from approximately 1,750 emails per year in 1999 to almost 4,000 in 2005 (Jupiter Communications, May 2000).

DOMAIN NAMES: There are 12,844,877 unique domain names (e.g. Cisco.com) registered worldwide, with 428,023 new domain names registered each week. (NetNames Statistics 12/28/1999).

Net Explosion


32


TOTAL AMOUNT OF DATA: 1,570,000,000 pages, 29,400,000,000,000 bytes of text, 353,000,000 images, and 5,880,000,000,000 bytes of image data. (The Censorware Project, Jan. 26, 1999).

NEW DATA EACH DAY: In just the last 24 hours, the web has added 3,180,000 new pages, 59,700,000,000 new bytes of text, 716,000 new images, and 11,900,000,000 new bytes of image data. (The Censorware Project, Jan. 26, 1999).

WEB PAGE LIFE SPAN: The average life span of a web page is about 44 days. (The Censorware Project, Jan. 26, 1999).

TRACKING WEB CONTENT CHANGES: To keep up with the changes to web content, you'd need to download about 873,000,000,000 bytes of information per day, which would mean you'd need a connection capable of downloading 10,100,000 bytes per second. (The Censorware Project, Jan. 26, 1999).

Net Explosion


33


DOCUMENTS ON WEB: The web now contains over one billion unique, indexable documents. (Emarketer/ Inktomi Corp., Jan. 2000).

CORPORATE EMAILS: In 2004, companies will send more than 200 billion e-mails. (Forrester Research, February 2000).

DOWNLOAD SPEEDS IMPROVING: Average homepage took 4.73 seconds to download by the end of 1999, 27% faster than at the beginning of the year (6.49 seconds). (Keynote Systems, April 2000).

NEW WEB PAGES: On average, more than 3 million Web pages were created every day in 1999. (IDC, Apr. 2000).

WEB HITS/DAY 1999: U.S. web pages averaged one billion hits per day (aggregate) in October 1999. (eMarketer/Media Metrix, Nov. 1999).

Net Explosion