DETER Testbed Status Kevin Lahey (ISI) Anthony D. Joseph (UCB) January 31, 2006.

DETER Testbed Status

Kevin Lahey (ISI)

Anthony D. Joseph (UCB)

January 31, 2006

Current PC Hardware

ISI

● 64 pc3000 (Dell 1850)● 11 pc2800 (Sun V65x)● 64 pc733 (IBM Netfinity

4500R)

UCB

● 32: bpc3000 (Dell 1850)● 32: bpc3060 (Dell 1850)● 32 bpc2800 (Sun V60x)

Approx 1/3 of nodes are currently down for repair or reserved for testing

Special Hardware

ISI• 4 Juniper M7i routers• 2 Juniper IDP-200 IDS• 1 Cloud Shield 2200• 2 McAfee Intrushield 2600

UCB Minibed• 8-32 HP DL360G2

Dual 1.4GHz/512KB PIII

Current Switches

ISI• 1 Cisco 6509

(336 GE ports)• 7 Nortel 5510-48T

(48 GE ports each)• Gigabit Switch

Interconnects

UCB• 1 Foundry FastIron 1500

(224 GE ports)• 10 Nortel 5510-48T

(48 GE ports each)• Gigabit Switch

Interconnects

UCB Minibed• 6 Nortel 5510-48T

(48 GE ports each)

Current Configuration

... ......

...

pc733s

pc2800s

pc3000s

Cisco 6509

Nortel 5510

1Gb

...Junipers

... ......

...

bpc2800s

bpc3000s

Foundry

FastIron

1500

Nortel5510s

1Gb (expandable)

...

bpc3060s

ISI UCB1Gb VPN

New Hardware for 2006ISI● 64 Dell 1850, identical to previous pc3000s

– Dual 3GHz Xeons with 2GB RAM, but with 2MB cache instead of 1MB,

and 6 interfaces instead of 5

● 32 IBM x330 (dual 1GHz Pentium IIIs with 1GB RAM)

UCB● 96+ TBD nodes, depending on overhead recovery● Full Boss and Users nodes:

– 2 HP DL360 Dual 3.4GHz/2MB Cache Xeon, 800MHz FSB, 2GB RAM– HP Modular Smart Array 20s: 12 x 500GB SATA drives (6TB)

Combined● Nortel 5510-48T and 10Gb-capable Nortel 5530-24T switches

New ISI Configuration

... ...

......

pc733s

pc2800s... ... ... ...

... ...

2 x 10Gb

... ... ... ... ... ... ... ...

pc3000s

... ... ... ...... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...

pc1000s

pc3000s

1Gb (10Gb later) 1Gb

...Junipers

Cisco 6509

Nortel 5510

Nortel 5510

Nortel 5510

DETER Clusters

ISI

UCB

Progress (1)

● People:– New ops guy (Kevin Lahey @ isi) getting up to speed

● Reliability:– Daily backups for users and boss, 1-time tarballs for all other

nodes– More robust Nortel switch configuration– ISI or UCB users/boss machines can run either or both

clusters

● Security: “Panic Switch” to disconnect from Internet

Progress (2)

● Emulab Software:– Unified boot image for –com1 and –com2 machines– DNS servers, IP addresses in Database– Click image with polling

● Incorporated state of Emulab as of about 9/30/05– Debugged at UCB, then installed at ISI– Firewall and experimental nodes must be resident on the

same switch– Release/update procedure is still problematic – for

discussion in testbed breakout

In-Progress (1)

● Reliability:– Automating fail-over between clusters (DB mirroring /

reconciliation scripts)

● Security:– Automatic disk wiping on a project/experiment basis– Automating leak testing for control/experiment networks

● Performance:– Redoing the way emulab-in-emulab handles the control net

(saves 1 experimental node interface)– Improving the performance of the VPN/IPsec links– Supporting a local tftp/frisbee server at UCB

In-Progress (2)

● Federation:– Supporting running federated experiments between separately

administered Emulabs using emulab-in-emulab● Netgraph module to rewrite 802.1q tags as they pass through a VPN

tunnel (similar to Berkeley and ISI link)

● Configuration:– Incorporating EMIST setup/visualization tools into Dashboard

● New Emulab Hardware Types:– Supporting IBM BladeCenters (currently testing with 12x2 BC)– Routers as first-class objects

Network Topology

● Open hypothesis: Inter-switch links may be a

bottleneck– Foundry/Cisco-Nortel and Nortel-Nortel– Adding multiple 10GE interconnects

● Exploring alternate node interconnection topologies– Example: connecting each node to multiple switches

● Potential issue: Assign is a very complex program– There may be all sorts of gotchas lurking out there

Other New Nodes on the Horizon

● Secure64● NetFPGA2● pc3000s with 10 interfaces● Research Accelerator for MultiProcessing (RAMP)

– 1,000 200-300 Mhz FPGA-based CPUs – Some number of elements devoted to FSM traffic

generators– Many 10GE I/O ports– ~$100K for 8U box at 1.5KW