Troubleshooting Techniques for - NXP Semiconductorscache.freescale.com/files/training/doc/ftf/2014/FTF-NET-F0246.pdf · External Use TM Troubleshooting Techniques for QorIQ eTSEC

External Use

TM

Troubleshooting Techniques for QorIQ eTSEC and DPAA PlatformsFTF-NET-F0246

A P R . 2 0 1 4

Rotariu Marian

TM

External Use 1

Session Introduction

• Presents sensitive areas and debugging tips for FSL DPAA & FSL eTSEC architectures

• This information is based on true stories during development and debug activities inside FSL.

• Step-by-step HOW-TO for functional & perfomance issues• The troubleshooting techniques are presented from a software

point of view

TM

External Use 2

Session Objectives

• After completing this session you will be able to: − Debug and solve the most common types of FSL DPAA and eTSEC

issues − Estimate better bring-up activities list for your product− Configure and fine-tune the DPAA and eTSEC for different scenarios

TM

External Use 3

Agenda• Architecture Overview• Configuration Workflow• Tools and Debugging Mechanisms• Issue Examples• Session review

TM

External Use 4

Architecture Overview

eTSEC Architecture Overview

DPAA Architecture Overview

Product Use Cases

TM

External Use 5

eTSEC Architecture

DTS

irq

BD Ring

eTSEC Driver

Linux Network

Stack

Rx Routine

Device Tree Parsing Module

CPUeTSEC

DMA

Scheduler

Parser

Memory

BD Ring

Tx Routine

RCW

TM

External Use 6

eTSEC Family

• P1 Family− P1010/P1014− P1020/P1011− P1021/P1012− P1022/P1013− P1025/P1015

• P2 Family− P2020/10

TM

External Use 7

DPAA Architecture

DTS

Tx Port

Rx Port

FMan

MAC

QMan

FMan Driver

Device Tree

Parsing Module

QMan Driver

BMan Driver

Linux Network

Stack

BMan

SERDESCPU

DPAA Driver

Rx routine

DMA

RAM

bufferpool

Tx routine

RCW

PCD

TM

External Use 8

DPAA Family

• P1 Family− P1023/P1017

• P2 Family− P2040/P2041

• P3 Family− P3041

• P4 Family− P4080/P4040

• P5 Family− P5020/P5010− P5040/P5021

• T4 Family− T4240/T4160

• B4 Family− B4860/B4420/B4460

TM

External Use 9

Product Use Cases

SW cusomization

HW cusomization

Freescale reference HWFreescale SDK

- Reference Manuals- Infocenter- Support

Custom HWFreescale SDK

- RCW, U-boot, DTS fix-up- MDIO MUX-ing

Custom HWCustom SW

- different number of ports- different phy model- different drivers

Freescale Reference HWCustom SW

- check compatibility- same HW init as standard SDK

TM

External Use 10

Configuration Workflow

Software components inside DPAA and eTSEC

Boot-up sequence

Linux Messages

TM

External Use 11

Software components in eTSEC and DPAA architectures

RCW U-Boot uCode Device tree Kernel Rootfs

• RCW (Reset Configuration Word)• U-Boot• uCode− used for programming some FMan hardware blocks (only in DPAA)− version should be compatible with FMan Driver version

see $(SDK_parent_dir)/fm-ucode/readme

• Device tree− most device tree errors are displayed at kernel bootup by the corresponding kernel driver− most often, issues are generated by a dts configuration error.

(only in DPAA)

PPCBoot imagemicrocode binary device tree blob PPCBoot imagePPCBoot imageRCW binary

TM

External Use 12

Software components in eTSEC and DPAA architectures

RCW U-Boot uCode Device tree Kernel Rootfs

• Kernel− Bootargs

standard bootargs (root, console, maxcpus)custom bootargs• usdpaa_mem, bportals, qportals• fsl_fm_max_frm• fsl_fm_rx_extra_headroom

− Kconfigmemory configuration (number of buffers for Eth drivers, thresholds)FMan dynamic resource allocation algorithm

• Rootfs− depends on the SDK version− iperf, netperf should have the same version on interconnected boards

(only in DPAA)

PPCBoot imagemicrocode binary device tree blob PPCBoot imagePPCBoot imageRCW binary

TM

External Use 13

Reset Configuration Word

RCW U-Boot

• PBL RCW as seen from U-Boot00000000: AA55 AA55 010E 0100 0E5A 0000 0000 0000 

00000010: 1E1E 0000 0044 0000 A07F 6AF0 D340 2000 

00000020: FE80 0000 0500 0000 0000 0000 0000 0001 

00000030: 0000 0000 C0DA 80F6 A000 0000 0000 0000 

00000040: 0000 0000 0000 0000 0813 8040 7003 80FC

• Frequencies for CPU, DDR, Platform, FManClock Configuration:

CPU0:1499.985 MHz, CPU1:1499.985 MHz, CPU2:1499.985 MHz, CPU3:1499.985 MHz,

CPU4:1499.985 MHz, CPU5:1499.985 MHz, CPU6:1499.985 MHz, CPU7:1499.985 MHz,

CCB:799.992 MHz,

DDR:649.994 MHz (1299.987 MT/s data rate) (Asynchronous), LBC:99.999 MHz

FMAN1: 599.994 MHz

FMAN2: 599.994 MHz

QMAN: 399.996 MHz

PME: 599.994 MHz

• SerDes Protocol− Number of ports, slot position

• PBL verification guide

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-time

• Naming conventiona_bcdef_g:

a = 'R' if RGMII is supported / 'N' if not available/not used

b = What is available in Slot 1

c = What is available in Slot 2

d = What is available in Slot 3

e = What is available in Slot 4

f = What is available in Slot 5

For the Slots (b..f):

'N' if not available/not used

'P' if PCIe

'X' if XAUI

'R' if SRIO

'S' if SGMII

'A' is AURORA

g = 'hex value of serdes protocol value'

TM

External Use 14

U-Boot

• versionU-Boot 2013.10QorIQ-SDK-V1.6 (Jan 30 2014 - 13:33:12)

• uCode versionFman1: Uploading microcode version 106.2.14

Fman2: Uploading microcode version 106.2.14

• detected portsFM1@DTSEC2, FM2@DTSEC1, FM2@DTSEC2, FM2@TGEC1, e1000#0 [PRIME]

• verify ping=> ping $serveripUsing e1000#0 devicehost 192.168.1.1 is alive

• write software components− DTS, kernel, sysroofs, rcw, ucode, u-boot

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

• Environment=> env pribaudrate=115200bdev=sda3eth1addr=00:04:9F:02:87:0Deth2addr=00:04:9F:02:87:0Eethact=e1000#0ethaddr=00:04:9F:02:87:0Cethprime=e1000#0fdtaddr=c00000fdtfile=p4080ds/p4080ds.dtbfman_ucode=eff40000hwconfig=fsl_ddr:ctlr_intlv=cacheline,bank_intlv=cs0_cs1;usb1:dr_mode=host,phy_type=ulpi;usb2:dr_mode=peripheral,phy_type=ulpi

TM

External Use 15

U-Boot

• rewrite RCW on a B4860QDS using tftpping $serverip;

setenv filesize;

tftp 100000 /($tftp_dir)/rcw_4sgmii_4pcie_2xfi_1600mhz_1866ddr.bin;

erase 0xee000000 +$filesize;

cp.b 100000 0xee000000 $filesize

• rewrite U-Boot on a B4860QDS using tftpping $serverip;

setenv filesize;

tftp 100000 /($tftp_dir)/u-boot-B4860QDS-git-r34.bin;

erase 0xedf80000 +$filesize;

cp.b 100000 0xedf80000 $filesize

• set Linux bootargssetenv bootargs isolcpus=1-4 usdpaa_mem=256M bportals=s0 qportals=s0;

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 16

U-Boot

• write DTS, uImage and rootfs tftp 1000000 $tftp_dir/uImage;

tftp 2000000 $tftp_dir/sysrootfs;

tftp c00000 $tftp_dir/p4080ds.dtb;

• bootup Linux kernelbootm 1000000 2000000 c00000

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 17

Linux Init

• BMan initializationBman portals initialised

Bman: BPID allocator includes range 32:32

• QMan initializationQman portals initialised

Qman: FQID allocator includes range 256:512

Qman: FQID allocator includes range 32768:32768

Qman: CGRID allocator includes range 0:256

Qman: pool channel allocator includes range 1025:15

• FMan driver initalizationFreescale FM module (Dec  3 2013:15:50:20), FMD API version 21.1.0

Freescale FM Ports module (Dec  3 2013:15:50:24)

fsl_mac ffe4e8000.ethernet: FMan MEMAC

fsl_mac ffe4e8000.ethernet: FMan MAC address: 00:04:9f:02:60:78

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 18

Linux Init

• Ethernet driver initializationfsl_dpa: FSL DPAA Ethernet driver ()

fsl_dpa: Probed interface eth0

• PHY and MDIO initializationlibphy: Fixed MDIO Bus: probed

libphy: Freescale XGMAC MDIO Bus: probed

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 19

Linux InitPre-Boot Loader Boot Loader

Linux Init Linux Run-timeRCW U-Boot

• BMan / QMan self-tests (Kconfig)BMAN: --- starting high-level test ---BMAN: --- finished high-level test ---bman_test_thresh: startbman_test_thresh: buffers are inthread 0: startingthread 0: exiting...(cropped output)...bman_test_thresh: doneqman_test_hotpotato startingCreating 2 handlers per cpu......(cropped output)...Received final (8th) frameqman_test_hotpotato finishedqman_test_high startingVDQCR (till-empty);...(cropped output)...scheduled dequeue (till-empty)Retirement message receivedqman_test_high finished

TM

External Use 20

Linux Init

• BMan initialization failureBman err interrupt handler present

Can't get /buffer-pool@9 property 'fsl,bpid'

• QMan driver initalization failureQman err interrupt handler present

• Ethernet driver initialization failurefsl_dpa ethernet.16: dev_get_drvdata(ffe4f0000.ethernet) failed

fsl_dpa: probe of ethernet.16 failed with error -22

fsl_dpa_proxy-proxy ethernet.16: dev_get_drvdata(ffe4f0000.ethernet) failed

fsl_dpa_proxy-proxy: probe of ethernet.16 failed with error -22

• MAC driver initalization errorfsl_mac ffe4f0000.ethernet: of_get_mac_address(/soc@ffe000000/fman@400000/ethernet@f0000) failed

fsl_mac: probe of ffe4f0000.ethernet failed with error -22

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 21

Linux Run-time

• Check that interface was initializedifconfig -a

ip addr

• Add ip address and raise the interfaceifconfig fm1-mac5 192.168.10.1/24 up

• Communication tests− pingping 192.168.10.1

− netperf/iperfnetperf -H 192.168.10.1

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 22

Linux Run-time

• Receiving erroneous framesfsl_dpa: dpaa_eth.c:1326:_dpa_rx_error() fsl-dpa: non-zero error counters in fman statistics (sysfs)

fsl_dpa ethernet.18 fm1-10g: FD status = 0x00040020

• Erroneous frame flags#define FM_FD_ERR_CLS_DISCARD 0x00020000 /**< classification discard */

#define FM_FD_ERR_EXTRACTION 0x00008000 /**< Extract Out of Frame */

#define FM_FD_ERR_NO_SCHEME 0x00004000 /**< No Scheme Selected */

#define FM_FD_ERR_KEYSIZE_OVERFLOW 0x00002000 /**< Keysize Overflow */

#define FM_FD_ERR_COLOR_RED 0x00000800 /**< Frame color is red */

#define FM_FD_ERR_COLOR_YELLOW 0x00000400 /**< Frame color is yellow */

#define FM_FD_ERR_ILL_PLCR 0x00000200 /**< Illegal Policer Profile selected */

#define FM_FD_ERR_PLCR_FRAME_LEN 0x00000100 /**< Policer frame length error */

#define FM_FD_ERR_PRS_TIMEOUT 0x00000080 /**< Parser Time out Exceed */

#define FM_FD_ERR_PRS_ILL_INSTRUCT 0x00000040 /**< Invalid Soft Parser instruction */

...(all the flags are defined in fm_ext.h from FMD)...

Pre-Boot Loader Boot LoaderLinux Init Linux

Run-timeRCW U-Boot

TM

External Use 23

Tools and Debugging Mechanisms

DPAA Debugging Mechanisms

Advanced Multicore Debug Architecture

Freescale Packet Analysis Tool

Linux Tools

sysfs, debugfs, procfs for DPAA

ethtool for eTSEC

Pattern for networking debug in DPAA and eTSEC

TM

External Use 24

DPAA Hardware Debug Mechanism

• SoC paradigm+ great integration that simplifies board-level layout+ signal integrity− lowered debug visibility

• multicore SoC− concurrency − increased potential for unintended interactions

TM

External Use 25

Advanced Multicore Debug Architecture

Datapath

DDR

Debug Module

CPU

Agent

CCSR

DCSR

SerDes

QManFMan

Nexus

JTAG

Eth

to remote host

TM

External Use 26

Packet Analysis Tool for DPAA Debug

• Packet-Oriented System Level Performance Analysis

• SoC Data Plane Configuration Debug

• Packet Processing Latency Analysis

• Packet Processing Critical Resource Monitoring

TM

External Use 27

sysfs for DPAA Ethernet Driver

• General sysfs entries for Ethernet drivers/sys/class/net/

• Frames queues used by the Ethernet driverroot@b4860qds:/sys/class/net/fm1‐mac5# cat fqidsRx error: 259Rx default: 260Rx PCD: 14848 ‐ 14975Tx confirmation (mq): 261 ‐ 284Tx(recycling): 285 ‐ 308Tx error: 309Tx default confirmation: 310Tx: 311 ‐ 334

• Buffer pools managed by the Ethernet Driver

root@b4860qds:/sys/class/net/fm1‐mac5# cat bpids32

Linux Kernel

DPAA EthernetDriver

RX Port TX Port

refillsBP

MAC

TM

External Use 28

debugfs for Ethernet Driver• Counters in DPAA Ethernet Driverroot@p4080ds:~# cat /sys/kernel/debug/powerpc/fsl_dpa/eth0

DPA counters for fm1-mac5:

CPU irqs rx tx recycle confirm tx sg tx err rx err bp count

0 63058 27024 27027 0 36034 0 0 0 96

1 9008 9006 9009 0 2 0 0 0 82

Total 72066 36030 36036 0 36036 0 0 0 178

• Congestion status (same debugfs location as the counters)Device congestion stats:

Device has been congested for 0 ms.

CGR id 0 avg count: 0

Device entered congestion 0 times. Current congestion state is: not congested.

DPAAEthernetDriver

NAPI

QMan

notificationssent to FMan

DCPSP datapath to FMan ports

interruptsCPU

TM

External Use 29

debugfs for Ethernet Driver• Rx Error typesDPA RX Errors:

CPU        dma err  phys err  size err   hdr err  csum err

0         0         0         0         0         0

1         0         0         0         0         0

Total          0         0         0         0         0

• Special types of errorsDPA ERN counters:

CPU cg_td wred err_cond early_w late_w fq_td fq_ret orp_z

0 0 0 0 0 0 0 0 0

1 0 0 0 0 0 0 0 0

Total 0 0 0 0 0 0 0 0

TM

External Use 30

Sysfs for Frame Manager• Sysfs general location/sys/devices/ffe000000.soc/ffe400000.fman

• BMI Port counters/sys/devices/ffe000000.soc/ffe500000.fman/ffe589000.port/statistics/*

• Memory mapped common registers for DMA, BMI, QMI

/sys/devices/ffe000000.soc/ffe400000.fman/fm_regs

• Memory mapped registers for Parser/sys/devices/ffe000000.soc/ffe400000.fman/fm_prs_regs

• Memory mapped registers for Key Generator

/sys/devices/ffe000000.soc/ffe400000.fman/fm_kg_regs/sys/devices/ffe000000.soc/ffe400000.fman/fm_kg_pe_regs

• Memory mapped registers for Policer/sys/devices/ffe000000.soc/ffe400000.fman/fm_plcr_regs

• Memory mapped registers for MAC device

/sys/class/net/fm1‐mac5/mac_regs

FManMAC

BMI Rx PortTx Port

DMA

QMI

Internal Memory

Tx Port Rx Port

Parser

Key Generator

Policer

to Main Memory

to QMan

to physical interfaces

to BMan

TM

External Use 31

Debugfs and Sysfs for Buffer Manager

• Buffer pool state in debugfsroot@b4860qds:/sys# cat ./kernel/debug/bman/query_bp_state

bp_id free_buffers_avail bp_depleted

0            no              no

...

32           yes             no

33           no              no

• Buffer pool countroot@b4860qds:/sys# cat /sys/devices/ffe000000.soc/ffe31a000.bman/pool_count/32

1024

• Error interruptsroot@b4860qds:/sys/devices/ffe000000.soc/ffe31a000.bman# cat err_isr

0x00000000

root@b4860qds:/sys/devices/ffe000000.soc/ffe31a000.bman/error_capture# cat sbec_0

0

root@b4860qds:/sys/devices/ffe000000.soc/ffe31a000.bman/error_capture# cat sbec_1

0

BMan

Memory

aquire/release buffers; to FMan BMI

depletion and error irqs; to CPU

TM

External Use 32

sysfs and debugfs for Queue Manager

• sysfs location for QMan/sys/devices/ffe000000.soc/ffe318000.qman

• debugfs location for QMan/sys/kernel/debug/qman

• Information regarding one particular queueroot@b4860qds:/sys/kernel/debug/qman# echo 160 > query_fq_np_fields

root@b4860qds:/sys/kernel/debug/qman# cat query_fq_np_fields

Query FQ Non Programmable Fields Result fqid 0xa0

force eligible pending: no

retirement pending: no

state: Out of Service

fq_link: 0x0

odp_seq: 0

orp_nesn: 0

...(cropped output)...

is: ics_surp contains a surplus

ics_surp: 0

byte_cnt: 0

frm_cnt: 0

...(cropped output)...

QMan

Software portal

Channel

Communication with FMan

WQ

Direct connected portal

Communication with CPU

TM

External Use 33

ethtool for eTSEC

• display all the countersroot@p2020rdb:~# ethtool -S eth2 | grep err

• software countersrx‐large‐frame‐errors: 0

rx‐short‐frame‐errors: 0

rx‐non‐octet‐errors: 0

rx‐crc‐errors: 0

rx‐overrun‐errors: 0

rx‐busy‐errors: 0

rx‐babbling‐errors: 0

ethernet‐bus‐error: 0

...(cropped output)...

• hardware countersrx-fragmented-frames: 0

rx-jabber-frames: 0

rx-dropped-frames: 0

tx-byte-counter: 261232

tx-packets: 136

MAC Layer

eTSECRx Int Tx Int

Linux Kernel

DM

AeTSEC Ethernet Driver

Linux Network StackMemory

TCP/IP CSUM

Tx Queue Scheduler

Tx FIFO FIFO Ctrl

Rx FIFO FIFO Ctrl

MAC Address

Filter

L2/L3/L4 Parser

Rx Filing Engine

TM

External Use 34

Pattern for networking debug in DPAA and eTSEC

1. Test Device counters*2. Linux stack counters*3. Ethernet driver counters*4. MAC registers*5. BMI and QMI registers6. BMan buffer pool

counter7. QMan counters

Linux Kernel

QMan

FMan

RX F

Qs

TX FQ

s

Network Stack

DPAA Ethernet Driver

MAC

Ports

QMI

BM

IRX

TX

BMan

RAM

bufferpool

Test Device1

2

3

4

5

6

7

3

(*) steps used for eTSEC

TM

External Use 35

Issue Examples

Configuration Issues

MDIO Bus Registration

Functionality Issues

VLAN Compatibility

Performance Issues

Out of Memory

Interrupt Coalescing

TM

External Use 36

MDIO Bus Registration

• Linux output:mdio_bus ffe4fc000: MDIO read errorfsl_dpa: dpaa_eth.c:156:_dpa_rx_error() fsl‐dpa: non‐zero error counters in fman statistics (sysfs)mdio_bus ffe4fc000: MDIO read errorfsl_dpa ethernet.18 fm1‐mac3: Err FD status = 0x00040000

• MDIO Bus was not getting registered− mismatch between phyhandle names in uboot and linux.

fm1mac1: ethernet@e0000 { /* DTSEC1/10GEC3 */

phy-handle = <&sgmii_phy_s3_1e>;phy-connection-type = "xgmii";

};

TM

External Use 37

VLAN Discovery

• After configuring a VLAN interface, traffic not reaching from eTSECto a non-VLAN device

• Frames from eTSEC get dropped on the nonVLAN dev• use tcpdump to display the entire frame• disable Tx VLAN insertion H/W acceleration feature for eTSEC

ethtool –k eth0 txvlan off

eTSEC device non-eTSEC deviceVLAN 0

eth0eth0eth0.1

TM

External Use 38

Out of Memory Issue

• DPAA cloud computing• Harddisk bottleneck• Linux Page Cache

mechanismecho 10000 > /proc/sys/vm/min_free_kbytes

echo 200 > /proc/sys/vm/vfs_cache_pressure

DPAA Device

Storage Devices

Users

tcp_client

tcp_write

Samba share

/smb/a//smb/b/

TM

External Use 39

Interrupt Coalescing Issue

• CPU utilization seems too high − sar 1

• Tx overruns reported by ifconfig for the transmitting deviceeth1      Link encap:Ethernet HWaddr 00:04:9f:00:13:01  

inet addr:172.16.1.1  Bcast:172.16.255.255  Mask:255.255.0.0UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1TX packets:41362129 errors:28 dropped:0 overruns:87034 carrier:0collisions:0 txqueuelen:1000 RX bytes:309431480 (295.0 MiB)  TX bytes:2372850676 (2.2 GiB)Base address:0xa000 

• Reduce the Rx/Tx interrupt rate the device, by rasing the “interrupt coalescing” values− ethtool –c eth1

• Verify interrupt rate− cat /proc/interrupts | grep eth

`

TM

External Use 40

For Further Information

• SME Contact information− Marian Rotariu

[email protected]

• Supporting Docs, References− SDK Documentation

B4860 QorIQ Qonverge Multicore Baseband Processor Reference ManualP4080 QorIQ Integrated Host Processor Reference ManualLinux Ethernet DriverLinux DPAA Ethernet Primer

− http://www.freescale.com/

TM

External Use 41

Session SummaryBy now, you should be able to:

Debug and solve the most common types of FSLDPAA and eTSEC issues

Configure and fine-tune the DPAA and eTSEC fordifferent scenarios

TM

© 2014 Freescale Semiconductor, Inc. | External Use

www.Freescale.com