InfiniBand Presentation

INFINIBAND“Bandwidth Out Of The Box”

Presented By-

Shekhar Kumar

074, CS-’B’

Guided By- Latha S.

Nair

Contents

Introduction

I/O Architectures- Fabric vs Bus

Technical Overview

InfiniBand Layers

InfiniBand Architecture

Advantages & Drawbacks

Markets

Conclusion

References

Introduction

A switch-based serial I/O interconnect architecture

Operates at a base speed of 2.5 Gb/s or 10 Gb/s in each direction (per port)

Provides both QoS (Quality of Service) and RAS (Reliability, Availability and Serviceability)

Superset of the Virtual Interface Architecture

Used in high performance computing and data centres

I/O Architectures- Fabric vs Bus

Feature Fabric Bus

Topology Switched Shared Bus

Pin Count Low High

Number Of End Points Many Few

Max Signal Length KMs Inches

Reliability Yes No

Scalable Yes No

Fault Tolerant Yes No

I/O Architectures Contd…

Shared Bus ArchitectureCPU

System

Controller

I/O Controller

I/O Controller

I/O Controller

I/O Controller

I/O Controller

PCI to PCI

Bridge

PCI to PCI

Bridge

System

Memory

I/O Architectures Contd…

Switched Fabric Architecture

Technical Overview

Switch-based point-to-point interconnect architecture

Each individual link is based on a four-wire 2.5Gb/s bidirectional connection.

Architecture defines a layered hardware protocol (Physical, Link, Network, Transport Layers) as well as a software layer

Each link can support multiple transport services for reliability

InfiniBand Feature Set

Layered Protocol- Physical, Link, Network, Transport, Upper Layers

Packet based communication Quality Of Service Three link speeds1. 1X- 2.5 Gb/s, 4 Wire2. 4X- 10 Gb/s, 16 Wire3. 12X- 30 Gb/s, 48 Wire PCB, Copper and Fiber cable

interconnect Remote DMA support Multicast and Unicast support

InfiniBand Layers

InfiniBand Layers Contd…

1) Physical Layer

Defines both electrical and mechanical

characteristics for the system

Includes cables and receptables for

fibre and copper media, backplane

connectors

Defines three link speeds, 1X,4X,12X

Each individual link is a 4-wire

differential connection that provides a

full duplex connection at 2.5 Gb/s


2) Link Layer

Encompasses packet layout, point-to-point operations, and switching within a local subnet

Packets- Two types: Management and Data packets

1. Management Packets- Used for link configuration and maintenance

2. Data Packets- Carry up to 4k bytes of a transaction payload

Switching- Devices within a subnet have a 16-bit Local ID assigned by the subnet manager. This LID is used for addressing.


Link Layer

QoS- supported through virtual lanes(VL)

VLs are separate logical communication links which share a single physical link

Data Integrity

Two CRCs per packet, Variant CRC and Invariant CRC

16-bit VCRC includes all fields in the packet and is recalculated at each hop

32-bit ICRC covers only the fields that do not change from hop to hop


Network Layer Handles routing of packets from one

subnet to another(within a subnet a network layer is not required)

Packets contain a global route header(GRH)

GRH contains the 128-bit IPv6 addressTransport Layer Responsible for in-order packet

delivery, channel multiplexing and transport services

Also handles transaction data segmentation when sending, and reassembly when receiving



Contd… Defines multiple devices for system

communication: channel adapter, switch, router, and a subnet manager

At least one channel adapter is required for each end node

Subnet Manager sets up and maintains link

Channel adapters and switches contains Subnet Management Agent(SMA) for handling communication with the subnet manager

Channel Adapters

Connects InfiniBand to other devices

Two types: Host Channel adapter(HCA) and Target Channel Adapter(TCA)

HCA- Provides interface to a host device and supports all software verbs defined by InfiniBand

TCA- Provides connection to an I/O device from InfiniBand with a subset of HCA features necessary for specific operations of each device

Switches and Routers

Switches and Routers

Contd… Switch

Contains more than one InfiniBand port and forwards packets from one of its port to another based on the LID

Can be configured to forward either unicast packets or multicast packets

Can consume or generate management packets

Router

Forwards packets from one subnet to another without consuming or generating packets

Reads the Global Route Header to forward the packet based on IPv6 address

Advantages

Superior performance

Low-latency

High-efficiency

Fabric consolidation and low energy

usage

Reliable, stable connections

Data Integrity

Highly interoperable environment

Drawbacks

Complex in design

Few platforms supports it as yet

Bleeding edge, for now, so users will

need to perform extensive testing

Markets

Application Clustering

Inter-Processor Communication(IPC)

Storage Area Networks

CONCLUSION

Greater processor performance

Supports RAS

Works both in-the-box and enables

Bandwidth Out of the Box

Supports scalability

REFERENCES

1. InfiniBand Trade Association www.infinibandta.org

2. InfiniBand Network Architecture, MindShare, Inc.

By Tom Shanley

3. InfiniBand Architecture, Development and Deployment: A strategic guide to server I/O solutions

By William T. Futral

4. Qos in InfiniBand Subnetworks:

Alfaro,F.J.; Sanchez, J.L.; Duato, J.; Department de Informatica, Univ. de Castilla-La Mancha, Albacete, Spain

InfiniBand Presentation

Technology