PCI Express over IP - Accelerated - Missing Link Electronics · PCI Local Bus from software view (addressing, driver, configuration, …) PCIe devices implement a set of registers

(c) 2016 MLEcorp.com EW2016-Session18-MLE-FPGAAccellWithHLS 1

PCI Express over IP - Accelerated

We are a Silicon Valley based technology company with Offices in Germany. We are Partner of leading electronicdevice and solution providers and have been enabling key innovators in the automotive, industrial, test & measurement markets to build better Embedded Systems, faster.

Our Mission isTo develop and market technology solutions for Embedded Systems Realization via pre-validated IP and expert application support, and to combine off-the-shelf FPGA devices with Open-Source Software for dependable, configurable Embedded System platforms

Our Expertise isI/O connectivity and Acceleration of data communication protocols, additionally opening up FPGA technology for analog applications, and the integration and optimization of Open Source Linux and Android software stacks on modern extensible processing architectures.

Dr. Endric Schubert, Univ. Ulm & Missing Link ElectronicsAndreas Braun, Missing Link ElectronicsUlrich Langenbach, Fraunhofer HHI

(c) 2016 MLEcorp.com EW2016-Session18-MLE-PCIeOverIP 2

Introduction and Motivation

PCIe to Ethernet Throughput Matching


PCI Express

Transaction

Data Link

Physical

Application

PCIe Layer

HeaderStart LCRCSeq. Payload ECRC

Transaction Layer Packet (TLP)

4 B 2 B 12/16 B 128/256 B 4 B 4 B

● PCIe replaces the PCI Local Bus (backwards compatible)● Full-duplex serial transmission ● At 8GT/s line rate (Gen3) on up to 32 lanes● Packet-based protocol with four layers

● Data Link layer, physical layer: Reliable transport on the link● Transaction layer:

● Transport of application data, device configuration, interrupts, Quality of service

● TLP categories: Memory, I/O, configuration, message


PCI Express

● PCIe is point-to-point. Hierarchical system topologies via switches

CPUPCIe

Endpoint

Endpoint

Endpoint

RootComplex(RC)

DP

PCIe

PCIe

PCIeUP

DP

Switch

● ID based routing (bus/device/function number) and address based routing● Transactions require completion (split-transaction) or posted transactions

possible


PCI Express

● PCI Local Bus from software view (addressing, driver, configuration, …)● PCIe devices implement a set of registers (configuration space)● PCIe topology needs to be explored at the beginning of system start-up● Enumeration of devices by completing Configuration-TLPs

BRBR

BR

Switch

Bus1 Bus2 Bus4

EP

EP

RC

Bus3

Range problem: Physical line length of PCIe on PCBs is limited to the centimeter range


State-of-the-art

InfiniBand:

ExpEther:

Source: Nec Corporation

Source: Mellanox

● Standard HPC interconnect

● PCIe Gen2 over 10GbE networks by NEC.

PCIe external cabling:● Standard for copper cables


TCP / IP


Network Processing


Network Protocol Acceleration Platform Architecture


PCIe over IP

Transaction

Data Link

Physical

Application

PCIe Layer

HeaderStart LCRCSeq. Payload ECRC

Transaction Layer Packet (TLP)

4 B 2 B 12/16 B 128/256 B 4 B 4 B

IP

MAC

Ethernet

TCP

Network Layer

Application

TCP

IPMAC

TLP

Ethernet

Src.MACPreamble FCSSFD Dst.MAC Type/Len

8 B 1 B 6 B 6 B 2 B 46 B – 1500 B 4 B

TLP

TCP/IP Packet

TLP TLP TLP TLP TLP


Concept of PCIe-over-IP

FPGA 1

TLP2TCP

TCP2TLP

FPGA 2

TCP/IP

Ethernet

PCIe PCIe End-point

Host PC Device

TCP/IP 1 TCP/IP 2

Root Complex

UPSWport

DNSWport

Networkstack

Networkstack

TLP2TCP

TCP2TLP

TLP 1PCIe 1 PCIe 2 TLP 2

...... ...PCIe 1 PCIe 2TLP 2

...TLP 1

TCP/IP TLP 1 TLP ...

Distributed PCIe Switch based on “XPressRICH3” PCIe IP Core from PLDA


Implementation

Interfaceadapter

+TLP-aggr.

UPPCIePHY

NPAP

Sidebandcon.

TLPcon.

Server config

DNportedit

10GbEPHY

Interfaceadapter

+TLP-aggr.

DPPCIePHY

NPAP

Sidebandcon.

TLPcon.

Config+

Completer+

Device edit

10GbEPHY

LANMain-board

Device LAN

FPGA1 – Upstream Side

FPGA2 – Downstream SideClient config

PERST#

PERST#


Implementation

RC EPPCIe PCIe

10GbE

UP Port DN Port

Local Remote

ZC706ZC706

FMC-to-PCIeNVMe SSD

CPU

Setup


Implementation

TLP aggregation

TCPIP

MAC

TLP


ImplementationLocal switch

UP Port DN Port

Distr. switch with TLP aggregation

RC EP

PCIe

UP Port

Local

ZC706

FMC-to-PCIeNVMe SSD

CPU

DN PortPCIe

RC EPPCIe

PCIe

10GbE

UP Port DN Port

Local Remote

ZC706ZC706

FMC-to-PCIeNVMe SSD

CPU

(c) 2016 MLEcorp.com EW2016-Session18-MLE-FPGAAccellWithHLS 16

Conclusion

● Reliable “tunneling” of PCI Express via TCP/IP● Fully transparent to PCIe Root Complex and Operating System ● FPGA processing enables bandwidth at 10 GigE line rates

● Based on “XPressRICH3” PCIe IP Core from PLDA

● Please visit us at the show:

HHI – Hall 4 Booth

MLE – Hall 2 Booth 2-421

Xilinx – Hall 1 Booth 1-205

PCI Express over IP - Accelerated - Missing Link Electronics · PCI Local Bus from software view (addressing, driver, configuration, …) PCIe devices implement a set of registers

Documents