Spring Camp 2013 1 Getting ready for day 2 Yesterdays tree was moved to NetFPGA- 10G-live-BACKUP-Day1/ IF you edited code cp NetFPGA-10G-live-BACKUP- Day1/projects/crypto_nic/hw

Spring Camp 2013 1

Getting ready for day 2

• Yesterday’s tree was moved to NetFPGA-10G-live-BACKUP-Day1/

• IF you edited codecp

NetFPGA-10G-live-BACKUP-Day1/projects/crypto_nic/hw/ pcores/crypto/ hdl/verilog/crypto.v \

NetFPGA-10G-live/ projects/crypto_nic/hw/pcores/crypto/hdl/verilog/ nf10_crypto_v1_00_a.v

• Edit your nf10_crypto_v1_00_a.v– Rename the module to nf10_crypto

Spring Camp 2013 2

NetFPGA Spring CampDay 2

Presented by: Andrew W. Moore

withMarek Michalski, Neelakandan Manihatty-Bojan, Gianni Antichi

Georgina Kalogeridou, Jong Hun Han, Noa Zilbermanaided by

Yury Audzevich, Dimosthenis Pediaditakis

Poznan University of TechnologyMay 20 – 24, 2013

http://NetFPGA.org

Spring Camp 2013 3

Previously Covered

•Infrastructure– Tree– Build System– EDK environment

•The Life of a Packet Through the NetFPGA– Hardware Datapath – Interface to software: Exceptions and Host I/O

•Implementation– Module Template– Write Crypto NIC using a static key

•Simulation and Debug– Write and Run Simulations for Crypto NIC

Spring Camp 2013 4

Tutorial Outline

•Registers

– Explain Register System– Use AXI Lite registers modules to implement register– Add register access stimulus to define Crypto NIC encryption key– Update Simulations

•Build and Test Hardware– Build– Explanation of Hardware Tests– Write and run Hardware Tests

• Verify value: 0xFFFFFFFF• Verify value: 0xFF00FF00• Verify value: 0x55555555

•Group Discussion– Project Ideas– Scope of work that can be accomplished in 2-3 days

•Team up for Projects– Team leaders will describe projects

Spring Camp 2013 5

Crypto Module State Diagram

DetectPacket’sHeader

Payload

Header+Payload

Spring Camp 2013 6

Section I: Registers

Spring Camp 2013 7

Specifying the Key via a Register

• Can set the key via a register instead

• Need to understand the register system

• Register system:– Specify registers stimulus in the reg_stim.axi

file– Implement registers in module

• Use instances of ipif_regs and axi_lite_ipif_1bar• Other instances available as well (2bar, 3bar, table)

Spring Camp 2013 8

Register bus

ModuleS_AXI_AWADDR

S_AXI_WDATA

S_AXI_CLK

S_AXI_ARESETN

S_AXI_AWVALID

S_AXI_WVALID

S_AXI_BREADY

S_AXI_ARADDR

S_AXI_ARVALID

S_AXI_RREADY

S_AXI_WSTRB

S_AXI_RDATA

S_AXI_RRESP

S_AXI_RVALID

S_AXI_WREADY

S_AXI_BRESP

S_AXI_ARREADY

S_AXI_BVALID

S_AXI_AWREADY

WR

ITE

RE

AD

WR

ITE

RE

AD

Spring Camp 2013 9

ipif_regs Module// -- IPIF REGS ipif_regs #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), .NUM_WO_REGS (NUM_WO_REGS), .NUM_RW_REGS (NUM_RW_REGS), .NUM_RO_REGS (NUM_RO_REGS) ) ipif_regs_inst ( .Bus2IP_Clk ( Bus2IP_Clk ), .Bus2IP_Resetn ( Bus2IP_Resetn ), . . . .wo_regs ( wo_regs ), .rw_regs ( rw_regs ), .ro_regs ( ro_regs ) );

Spring Camp 2013 10

axi_lite_ipif_1bar (in every core)// -- AXILITE IPIF axi_lite_ipif_1bar #( .C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH), .C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH), . . . .C_BAR0_HIGHADDR (C_HIGHADDR) ) axi_lite_ipif_inst ( .S_AXI_ACLK ( S_AXI_ACLK ), .S_AXI_ARESETN ( S_AXI_ARESETN ), . . . .S_AXI_AWREADY ( S_AXI_AWREADY ),

// Controls to the IP/IPIF modules .Bus2IP_Clk ( Bus2IP_Clk ), . . . .IP2Bus_Error ( IP2Bus_Error ) );

Spring Camp 2013 11

Additional Related Modules

• axi_lite_ipif_2bar– Same as axi_lite_ipif_1bar but with support

for 2 address ranges• axi_lite_ipif_3bar

– Same as axi_lite_ipif_1bar but with support for 3 address ranges

• ipif_table_regs– Read and write to a table

Spring Camp 2013 12

Adding Registers Logic (1)

• Registers are arranged in memory in the following order:– Write only registers– Read/Write registers– Read only registers

• Define the number of registers used of each type, e.g.:– localparam NUM_RW_REGS = 2; localparam NUM_RO_REGS = 1;

• Use these parameters for the ipif_regs module– No need to define a parameter that is not used(e.g. there are no WO registers)

Spring Camp 2013 13


• Ipif_regs exposes 3 registers busses:– wo_regs, rw_regs, ro_regs

• Each bus is (NUM_REGS*C_S_AXI_DATA_WIDTH) wide– Each register is assigned C_S_AXI_DATA_WIDTH bits

• Usage examples:assign ro_regs = {version_reg,counter_reg1,counter_reg2};

always @(posedge Bus2IP_Clk) if (~Bus2IP_Resetn) begin dummy_reg <= 'h0; end else begin dummy_reg <= rw_regs [C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) + 31 :

C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) ]; end

Spring Camp 2013 14


Registers usage:•RO, WO, RW refers to software access•Write only registers can be written only by the software• Read only registers are set by hardware and

read by software or hardware • Read/Write registers can set by software and

read by software or hardware

Spring Camp 2013 15

Testing Registers with Simulation

Spring Camp 2013 16

Simulating Register Access

DUT

== !=

PASS

reg_stim.axi reg_stim.log

compare

FAIL

system_axisim_tb

Legend:- DUT: Design Under Test- stim: stimulus- tb: testbench- sim: simulation

1. Define register stimulus

2. The testbench executes the stimulus

3. Simulation accesses are written to a log file

4. A script can compare expected and actual valuesAnd declare success or failure

Spring Camp 2013 17

Registers Stimulus (1)

cd ~/NetFPGA-10G-live/projects/crypto_nic/hwvim reg_stim.axi

An example of write format :

Address

Data

Byte Enable strobe

Spring Camp 2013 18

Registers Stimulus (2)

cd ~/NetFPGA-10G-live/projects/crypto_nic/hwvim reg_stim.axi

An example read format :

Address

Spring Camp 2013 19

Registers Access Log

READ

WRITE

Time

cd ~/NetFPGA-10G-live/projects/crypto_nic/hwvim reg_stim.log

Spring Camp 2013 20

Replacing Static Key

• In the crypto project, replace the static key with the key from the registers– Provide an Enable registers

• Update your simulations to set the key

Spring Camp 2013 21

To execute a register simulation

cd ~/NetFPGA-10G-live/projects/crypto_nic/hwmake simreg

Spring Camp 2013 22

Section II: Build and Test Hardware

Spring Camp 2013 23

Synthesis

• To synthesize your projectCHECK your pao file:

cd ~/NetFPGA-10G-live/projects/crypto_nic/hw/pcores/nf10_crypto_v1_00_a/data/edit nf10_crypto_v2_1_0.pao

For synthesisuncomment lines 39,40,43 comment out lines 41 and 42

cd ~/NetFPGA-10G-live/projects/crypto_nic/hwmake

Spring Camp 2013 24

Hardware Tests

• Test compiled hardware

• Test infrastructure provided to – Read/Write registers – Read/Write tables – Send Packets – Check Counters

Spring Camp 2013 26

Python Libraries

• Start packet capture on interfaces

• Clear all tables in hardware

• Create packets– MAC header– IP header– PDU

• Read/Write registers

• Read/Write reference router tables– Longest Prefix Match– ARP– Destination IP Filter

Spring Camp 2013 27

Hardware Test Examples

from ~/NetFPGA-10G-live/<project>•reference_nic

– Simple crossover test• test/hw_external_crossover

•older 1Gbps Reference Router examples– Packet Forwarding

• test/both_packet_forwarding

– Longest Prefix Match• test/both_lpm_generic

– Send and Receive • test/hw_send_rec

Spring Camp 2013 29

Creating a Hardware Test (2)

• Your task:edit ~/.bashrc and change NF_DESIGN_DIR to be crypto_nicreopen your Terminal (to force this change)cd ~/NetFPGA-10G-live/projects/cp --archive reference_nic/test crypto_nic/.

This creates a directory of hardware tests just like the reference_niccd ~/NetFPGA-10G-live/projects/projects/crypto_nic/testcp hw_external_loopback hw_crypto_encrypt

• Now edithw_crypto_encrypt/run.py

to create your tests.

Spring Camp 2013 30

Running Hardware Tests

• Use command nf_test.py– Required Parameter

• sim hw or both (right now only use hw)– Optional parameters

• --major <major_name>• --minor <minor_name>

both_crypto_encrypt

• Run the commandnf_test.py hw --major crypto --minor encrypt

majormajor minorminor

Spring Camp 2013 31

Section III: Interface with Software

Spring Camp 2013 32

NetFPGA-Host Interaction (recap)

–Register reads/writes via ioctl system call with wrapper functions:

• rdaxi(int address, unsigned *rd_data);• wraxi(int address, unsigned *wr_data);

eg:rdaxi(0x7d4000000, &val);

–Useful command line utilitiescd ~/NetFPGA-10-live/projects/crypto_nic/sw/host/apps./rdaxi 0x7d4000000./wraxi 0x7d4000000 0x1

Spring Camp 2013 33

Recap

Build a complete NetFPGA design

Learn:• Module creation (Verilog)• Reference pipeline integration• Verification via simulation• Verification via hardware tests• Interaction with software

Spring Camp 2013 34

Step 1. Program NetFPGA-10G Guidelines

1) Prepare a bit file, nf10.ko driver for a NetFPGA cardcd ~/NetFPGA-10G-live/projects/crypto_nic/sw/host/driver; makeOR, do your make from the DESIGN directorycd ~/NetFPGA-10G-live/projects/crypto_nic; make

Reference_nic driver is used for most projects.

2) cd ~/NetFPGA-10G-live/projects/crypto_nic/bitfiles3) Load a bit file for programming FPGA ~/NetFPGA-10G-live/tools/scripts/impact_run.sh <bit_file_name.bit>

eg ~/NetFPGA-10G-live/tools/scripts/impact_run.sh crypto_nic.bit

4) Reboot machine (only required once each machine power-up)5) repeat step 4)

Spring Camp 2013 35


If you are developing new DMA systems steps 6 onward apply6)To find out loaded bitfile image, run

$ lspci –d *:4244 –vxx

Spring Camp 2013 36


7) ~/NetFPGA-10G-live/tools/scripts/pci_save_restore.sh save dma

Now, when the FPGA needs to be programmed, run only step 4) that programs FPGA, restores PCIE configuration, and loads nf10.ko driver.

Spring Camp 2013 37


8) Go to ./NetFPGA-10G-live/projects/reference_nic/sw/host/driver9) Compile nf10 driver

$ make10) Load nf10 driver by run insmod

$ insmod nf10.ko

Spring Camp 2013 38


11) Run dmesg to find out the kernel driver$ dmesg

Spring Camp 2013 39

Impact (the FPGA loader)

• Xilinx impact has a peculiar behaviour sometimes (this is a known Xilinx impact bug)

If the batch mode of impact does not work, a GUI interface appears….

These next slides will guide you through using the GUI.

Appendix

Spring Camp 2013 40

Step 1. Use of Impact GUI for Program FPGA

Impact GUI can be used for programming FPGA. This steps shows how programs FPGA using Impact GUI.

1)Source Xilinx tools $ source /opt/Xilinx/13.4/ISE_DS/settings64.sh

2)Go to NetFPGA-10G directory and run impact $ impact

Appendix

Spring Camp 2013 41


3) Tick ‘create a new project’ and click OK.

Appendix

Spring Camp 2013 42


4) Tick ‘Configure devices using Boundary-Scan(JTAG) and click OK

Appendix

Spring Camp 2013 43


5) Click Yes

Appendix

Spring Camp 2013 44


6) Select a bit file and click OPEN.

Appendix

Spring Camp 2013 45


7) Click No.

Appendix

Spring Camp 2013 46


1) Click Cancel.

Appendix

Spring Camp 2013 47


8) Click Cancel.

Appendix

Spring Camp 2013 48


9) Double click ‘Program’10) This process ONLY loads the FPGA.

Appendix

Spring Camp 2013 49

Section IV: Wrap-up

Spring Camp 2013 50

Thoughts for Developers

• Build Modular components– Describe shared registers – Consider how modules would be used in larger systems

• Define functionality clearly – Through regression tests– With repeatable results

• Disseminate projects– Post open-source code– Document projects on Web, Wiki

• Expand the community of developers– Answer questions on the Email list – Collaborate with your peers to build new applications

Spring Camp 2013 51

NetFPGA.org

Spring Camp 2013 52

• Build an accurate, fast, line-rate NetDummy/nistnet element

• A flexible home-grown monitoring card

• Evaluate new packet classifiers – (and application classifiers, and other neat network apps….)

• Prototype a full line-rate next-generation Ethernet-type

• Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)

• Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware)

• Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w)

• Hardware supporting Virtual Routers

• Check that some brave new idea actually works e.g. Rate Control Protocol (RCP), Multipath TCP,

• toolkit for hardware hashing

• MOOSE implementation

• IP address anonymization

• SSL decoding “bump in the wire”

• Xen specialist nic

• computational co-processor

• Distributed computational co-processor

• IPv6 anything

• IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….)

• Netflow v9 reference

• PSAMP reference

• IPFIX reference

• Different driver/buffer interfaces (e.g. PFRING)

• or “escalators” (from gridprobe) for faster network monitors

• Firewall reference

• GPS packet-timestamp things

• High-Speed Host Bus Adapter reference implementations– Infiniband– iSCSI– Myranet– Fiber Channel

• Smart Disk adapter (presuming a direct-disk interface)

• Software Defined Radio (SDR) directly on the FPGA (probably UWB only)

• Routing accelerator– Hardware route-reflector– Internet exchange route accelerator

• Hardware channel bonding reference implementation

• TCP sanitizer

• Other protocol sanitizer (applications… UDP DCCP, etc.)

• Full and complete Crypto NIC

• IPSec endpoint/ VPN appliance

• VLAN reference implementation

• metarouting implementation

• virtual <pick-something>

• intelligent proxy

• application embargo-er

• Layer-4 gateway

• h/w gateway for VoIP/SIP/skype

• h/w gateway for video conference spaces

• security pattern/rules matching

• Anti-spoof traceback implementations (e.g. BBN stuff)

• IPtv multicast controller

• Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters)

• DES breaker

• platform for flexible NIC API evaluations

• snmp statistics reference implementation

• sflow (hp) reference implementation

• trajectory sampling (reference implementation)

• implementation of zeroconf/netconf configuration language for routers

• h/w openflow and (simple) NOX controller in one…

• Network RAID (multicast TCP with redundancy)

• inline compression

• hardware accelorator for TOR

• load-balancer

• openflow with (netflow, ACL, ….)

• reference NAT device

• active measurement kit

• network discovery tool

• passive performance measurement

• active sender control (e.g. performance feedback fed to endpoints for control)

• Prototype platform for NON-Ethernet or near-Ethernet MACs– Optical LAN (no buffers)

Stuck for a NetFPGA project?Well I’m not sure about you but here is a list I created:• Build an accurate, fast, line-rate NetDummy/nistnet element

• A flexible home-grown monitoring card• Evaluate new packet classifiers

– (and application classifiers, and other neat network apps….)• Prototype a full line-rate next-generation Ethernet-type• Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)• Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware)• Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w)• Hardware supporting Virtual Routers• Check that some brave new idea actually works

e.g. Rate Control Protocol (RCP), Multipath TCP, • toolkit for hardware hashing• MOOSE implementation• IP address anonymization • SSL decoding “bump in the wire”• Xen specialist nic• computational co-processor• Distributed computational co-processor• IPv6 anything• IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….)• Netflow v9 reference• PSAMP reference• IPFIX reference• Different driver/buffer interfaces (e.g. PFRING)• or “escalators” (from gridprobe) for faster network monitors• Firewall reference• GPS packet-timestamp things• High-Speed Host Bus Adapter reference implementations

– Infiniband– iSCSI– Myranet– Fiber Channel

• Smart Disk adapter (presuming a direct-disk interface)• Software Defined Radio (SDR) directly on the FPGA (probably UWB only)• Routing accelerator

– Hardware route-reflector– Internet exchange route accelerator

• Hardware channel bonding reference implementation• TCP sanitizer• Other protocol sanitizer (applications… UDP DCCP, etc.)• Full and complete Crypto NIC• IPSec endpoint/ VPN appliance• VLAN reference implementation• metarouting implementation• virtual <pick-something>• intelligent proxy• application embargo-er• Layer-4 gateway• h/w gateway for VoIP/SIP/skype• h/w gateway for video conference spaces• security pattern/rules matching• Anti-spoof traceback implementations (e.g. BBN stuff)• IPtv multicast controller• Intelligent IP-enabled device controller (e.g. IP cameras or IP powermeters)• DES breaker• platform for flexible NIC API evaluations• snmp statistics reference implementation• sflow (hp) reference implementation• trajectory sampling (reference implementation)• implementation of zeroconf/netconf configuration language for routers• h/w openflow and (simple) NOX controller in one…• Network RAID (multicast TCP with redundancy)• inline compression• hardware accelorator for TOR• load-balancer• openflow with (netflow, ACL, ….)• reference NAT device• active measurement kit• network discovery tool• passive performance measurement• active sender control (e.g. performance feedback fed to endpoints for control)• Prototype platform for NON-Ethernet or near-Ethernet MACs

– Optical LAN (no buffers)

Spring Camp 2013 53

Project Ideas for the NetFPGA• NetFPGA-10G Test Harness• Drop 1-in-N packet module• Rate-limited module• Event capture module• Statistics and Counters• Measurement sketch• Advanced OPL • Input / Output scheduler• 40G Port• VLAN Tagging• ….

• Ideas from NetFPGA-1G

Spring Camp 2013 54

Visit http://NetFPGA.org

Spring Camp 2013 55

• Wednesday restart from 9am• This evening is time to try hw testing &

synthesis crypto_nic design if you didn’t manage that

• Make your group, leader&project

• Specific, Realistic• Time-bounded

• Modest is good…..

Spring Camp 2013 56

Section IX: Conclusion

Spring Camp 2013 57

Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller,

Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo

AcknowledgmentsNetFPGA Team at Stanford University (Past and Present):

NetFPGA Team at University of Cambridge (Past and Present):

Andrew Moore, David Miller, Muhammad Shahbaz, Martin ZadnikMatthew Grosvenor, Gianni Antichi, Neelakandan Manihatty-Bojan,

Georgina Kalogeridou, Jong Hun Han, Noa Zilberman

All Community members (including but not limited to):

Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer,

Eric Keller , Tatsuya Yabe, Bilal Anwer,Yashar Ganjali, Martin Labrecque

Kees Vissers, Michaela Blott, Shep Siegel

Spring Camp 2013 58

Thanks to our Sponsors:

• Support for the NetFPGA project has been provided by the following companies and institutions

Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.This effort is also sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA8750-11-C-0249. This material is approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.

Spring Camp 2013 1 Getting ready for day 2 Yesterdays tree was moved to NetFPGA- 10G-live-BACKUP-Day1/ IF you edited code cp NetFPGA-10G-live-BACKUP- Day1/projects/crypto_nic/hw

Documents

highaddr axi

axilite ipif axi

ipif regs ipif

regs slide

aresetn s

awready s

aclk s

awaddr s