Audio Video Bridging on i.MX Applications Processorsread.pudn.com/downloads699/doc/comm/2814014/FTF-SHB-F1211.… · 1722a – AVTP (Draft) ... − Represents 125 µs worth of audio

External Use

TM

Audio Video Bridging on i.MX Applications ProcessorsFTF-SHB-F1211

J U N E . 2 0 1 5

Rui Sousa | Software EngineerDavid Babin | Hardware Engineer

TM

External Use 1 #FTF2015

Agenda

• AVB Overview

• Hardware Requirements

• AVB Implementation Challenges

• AVB Software Stack

• AVB Hardware Platforms

• AVB Hardware Media Clock Capture/Recovery

• Demo Platform

• Q&A

TM


AVB Overview

TM


Audio Video Bridging (AVB)

• Definition: Set of standards, protocols and technologies for the transport of time sensitive streams (audio, video and control) in bridged LANs

• Purpose: Replace existing analog and multi-standard digitized stream transport with a commonly deployed technology, offering:− Bounded and low transport latency

− Reliable delivery (with guaranteed bandwidth and network resources)

− Time synchronization

• Advantages− Can replace complex and costly wiring with a single and existing technology:

Ethernet

− Based on open Standards

− Coexists with legacy IP data traffic

− Self-managed

TM


Audio Video Bridging Standards

IEEE Standards (https://www.ieee.org)

• 802.1BA-2011 AVB (Audio Video Bridging)

• 802.1AS-2011 gPTP (Generalized Time Precision Protocol)

• 802.1Qat-2010 SRP (Stream Reservation Protocol)

• 802.1Qav-2009 FQTSS (Forwarding and Queuing of Time Sensitive Streams)

• 1722-2011 AVTP + MAAP (Audio Video Transport Protocol, MAC Address

Acquisition Protocol)

• 1722.1-2013 AVDECC (Audio Video Discovery, Enumeration, Connection

Management and Control)

• 1722a D13 AVTP

TM


Audio Video Bridging Standards

AVnu Alliance (http://www.avnu.org)

• Community promoting AVB/TSN technologies in an interoperableecosystem

• Certification program

• Market specific workgroups− Automotive

− Professional Audio/Video

− Consumer Electronics

− Industrial

• Automotive profile

TM


802.1AS gPTP

• Provides time synchronization across bridged LAN networks− Within 1us peak to peak for 7 hops

• Based on IEEE 1588-2008 PTPv2

• gPTP domain with several interconnected time aware systems (end stations and bridges) synchronized to a grandmaster clock

• Best Master Clock Algorithm (BMCA)− Network is able to select best source of timing (based on ANNOUNCE

messages)

• Propagation delay measurement between peers of a point-to-point link− Media dependent (802.3 -> Pdelay Requests/Responses)

• Propagation of Synchronization messages− Bridges update SYNC messages based on peer delay measurements and

own delay (residence time)

TM


802.1Qat - SRP

• Provides reservation of bandwidth and resources along the network path between Talker and Listener(s)− Bounded maximum latency− Reliable delivery (if reservation is successful)

• Two traffic classes− Identified by VLAN priority and ID− Class A (2ms transit time, 125us observation interval)

− Class B (50ms transit time, 250us observation interval)

• Talkers declare available streams and their properties• Listeners declare their intention to receive a stream• Bridges adjust/merge/propagate declarations

− Possibly changing Talker/Listener declarations if reservation fails

TM


1722 - AVTP

• Defines transport protocol for Audio & Video− Layer 2 protocol over 802.3, 802.11, …− Data, control and timing

• Several media formats supported (based on IEC 61883)− Raw audio (IEC-61883-6)− MPEG2-TS (IEC-61883-4)

− …

• AVTP presentation time− Based on gPTP

− Used to synchronize presentation time (across several Listeners)

− Used to recover media clock

TM


1722a – AVTP (Draft)

• New backwards compatible revision• Several new/native media formats introduced

− AVTP audio format (AAF, Raw audio, Int, Float, 16/24/32bit, 8-192 KHz, …)

− Compressed Video format (H264, MJPEG, JPEG2000)− SDI Video format− Raw Video format

• Improved distribution of clock timing (not limited to media clock)− Dedicated clock reference format (CRF) stream− Several timestamps per packet

• AVTP control format• Encrypted data transport encapsulation• Encrypted/Authenticated control transport encapsulation

TM


1722.1 - AVDECC

• Provides the ability to discover devices on the LAN, enumerate their properties, manage their connections and control them

• Talker/Listener/Controller Entities• Formal/Full description of Entity capabilities (Entity model):

− Descriptor based

− Stream inputs/outputs

− Analog/Digital Audio/Video inputs/outputs

− Media formats (sample rates, channels, aspect ratio, frame sizes, codecs, …)

− Clock domains

− Signal routing

− …

• Talker/Listeners Advertise their presence and answer to commands• Controllers Discover/enumerate Talker/Listeners and issue commands

TM


AVnu Automotive Profile

• Objectives− Reduced start up time

− Reduced transport overhead

− Reduced end station load

• gPTP− No BMCA, fixed port role, fixed grandmaster, stored delay measurements

• SRP− Fixed configuration

− New traffic classes: C (15ms transit time, 1.333us observation interval), D (15ms transit time, 1.451us)

• AVTP− Reduced number of supported formats (AAF, Compressed Video, 61883-6)

• AVDECC− Not used

TM


Audio Video Bridging Network

• Talker end stations

• Listener end stations• Talker & Listener end stations

• AV Bridges

• gPTP domain− gPTP Master

− gPTP Slave(s)

• SRP domain(s)• AVB domain(s) (gPTP ∩ SRP)

• Media clock domain(s)− Media clock master

− Media clock slave(s)

TM


Audio Video Bridging Network Streams

Talker end station AV BridgeListener end

station

AV Bridge AV Bridge Talker end stationTalker end station

Listener end station

Listener end station

Talker & Listener end station

TM


Audio Video Bridging Network gPTP

gPTP Master AV Bridge gPTP Slave

AV Bridge AV Bridge gPTP SlavegPTP Slave

gPTP Slave gPTP SlavegPTP Slave

TM


Audio Video Bridging Network Media clock

Media clock slave

AV BridgeMedia clock

slave

AV Bridge AV BridgeMedia clock

slaveMedia clock

Master

Media clock slave

Media clock slave

Media clock slave

TM


Audio Video Bridging Network Automotive

Talker Left Mirror Camera AV Bridge

Talker Right Mirror Camera

Listener RSEListener RSE

Talker Rear View Camera

Talker & Listener Head Unit

TM


Audio Video Bridging Network Pro Audio

AV BridgeListener

LoudspeakerListener

Loudspeaker

Talker & Listener Mixing table

Talker & ListenerMic & Monitor

AV Bridge

AV Bridge

AV Bridge

Listener Loudspeaker

Listener Loudspeaker



Listener Recording

TM


Hardware Requirements

TM



• End stations− 802.3 Ethernet� Full duplex 100Mbps

� 1588 (PTPv2) MAC support

− 802.11 Wireless� 802.11n + 802.11v

• Local clock source (+-100ppm, 40ns granularity)• Media clock recovery (highly recommended)• Ethernet Rx classification/multiqueue (optional, Listener)• Tx credit based shaper (per stream)/multiqueue (optional, Talker)

TM



• AV Bridges− gPTP support− FQTSS support

− SRP support

• Automotive profile− Static SRP configuration

TM


AVB Implementation Challenges

TM


AVB Timing Challenges

• AVB defines low latency Class A traffic with a maximum latency of 2 msover 7 hops

• AVB defines higher latency Class B traffic with a maximum latency of 50 ms over 7 hops

• AVB 61883-6 Audio Format suggests “A Class A isochronous packet is generated every 8 kHz”− Represents 125 µs worth of audio− For an audio CD type (16 bit sample, stereo) stream @ 48 kHz, this means 6

samples � 24 bytes of payload

• The talker must send media data at 125 or 250 µs interval (extra slide)• The listener must provide 2 or 50 ms buffer capacity (worst cases of “must

wait” before presenting samples), and deliver on time at the Class interval rate

• Consider video playback scenario (extra slide)

TM


Shaping for Multiple Streams (IEEE 802.1Qav)

Class A Credit-based

Shaper

Class B Credit-based

Shaper

Static P

riority

Ethernet M

AC

Best Effort Traffic

Software Scheduler

Streaming Source #A

Streaming Source #B

Streaming Source #N

…

Typical AVB Ethernet IPwith Class-based Shaper

• Class A/B traffic needs to be shaped/scheduled with a 125/250 µs interval rate

• Class queuing requires that only MaxIntervalFrames of each stream may be scheduled within one scheduling cycle (125 µs for class A, 250 µs for class B)

⇒ The software scheduler must run at the class observation interval of 125 or 250 µs

“stream-based” Ethernet AVB hardware IP, e.g. on i.MX 6SLX or MPC5748G, reduces the shaping processing load (support hw queues per traffic class with „Launch Time“ capability)

TM


Video Streaming Scenario Challenge

• Scenario: Head unit streams movie to the Rear Seat Entertainment unit:

− Video data is received by the RSE, audio data by the Ethernet amplifier

− Consumer video (e.g. H.264 main profile) decompression has some very significant latency (typically 50-100 ms)

− Audio and video need to be “lip synced” with ~ 10 msaccuracy

⇒Some significant amount of audio data needs to be stored temporarily

� Option A: In the Ethernet amp � larger amount of RAM required

� Option B: Streaming source (head unit) needs to buffer audio sufficiently and send audio delayed compared to video

Amplifier

HeadUnit

Rear Seat Entertainment

Ethernet Switch

Video D

ata

Audio D

ata

TM


Media Clock Challenges

• Cross Time stamping at the Talker side (outgoing stream)

Source: Dave Olsen

Timestamp generation must be made without latency (matching the

exact sampling operation)

TM


Media Clock Challenges

• Reconstruction at the Listener side (incoming stream)

Source: Dave Olsen

Clock regeneration must be made without latency (at the perfect

matching time between timestamps

and gPTP time)

TM


AVB Software Stack

TM


AVB Software Stack

• Multi-platform− Runs on several Freescale platforms: i.MX, VFxxx− OS/Hardware independent core code, greatly reducing re-development

cost− Platform abstraction layer

• Scalable− Allows usage of subsets of the AVB stack to meet specific market needs− Offers variety of performance/cost levels according to the platform

• AVB Hardware Support− Able to run both on platforms with standard Ethernet IP or with Ethernet

AVB IP

TM


AVB Software Stack

• All AVB components are provided (as defined by IEEE)− GPTP− MSRP, MVRP

− FQTSS

− AVTP− MAAP

− AVDECC

• Media Application Support− AVB library provides API for Media Applications

− Demo integration with ALSA & GStreamer, implementing Media Server, Audio Amplifier, Video Player…

TM


AVB Stack Diagram

Control application

IEEE 1722

AVTPMedia

IEEE802.1QatSRP

Bandwidth Reservation

Ethernet IEEE 802.3 MAC

IEEE802.1Qav FQTSSQueuing & Shaping

IEEE 1722 MAAP

IEEE 1722.1AVDECC

AVB Management & Control

IEEE 802.1ASgPTP

Timing & SynchronizationTCP/IP

stack

Alsa/Gstreamer/…

DMA

I/O

Audio/Video Driver

Control API

Media API

Operating System

Media Clock

Recovery

Productized component

IEEE 1722 AVTP ControlOS

Abs

trac

t

HW

Abs

trac

t

TM


AVB Stack Diagram

• Automotive Profile

Control application

IEEE 1722

AVTPMedia

IEEE802.1QatSRP

Bandwidth Reservation

Ethernet IEEE 802.3 MAC

IEEE802.1Qav FQTSSQueuing & Shaping

IEEE 1722 MAAP

IEEE 1722.1AVDECC

AVB Management & Control

IEEE 802.1ASgPTP

Timing & SynchronizationTCP/IP

stack

Alsa/Gstreamer/…

DMA

I/O

Audio/Video Driver

Control API

Media API

Operating System

Media Clock

Recovery

Productized component

IEEE 1722 AVTP ControlOS

Abs

trac

t

HW

Abs

trac

t

TM


Linux AVB Stack (i.MX 6)

• Linux 3.10.17 with real-time extensions• Kernel patch for deep integration with ENET driver• AVB kernel module for system services

− Shared memory− Network tx/rx− Rx classification− Tx QoS scheduler− Media clock capture/recovery

• AVB userspace task/thread for each stack component− Linux specific event loop− Platform independent stack implementation

• Media interface driver + user library/API• Example media applications

− Audio (Alsa), Video & Audio (GStreamer)

TM


Linux AVB Stack (i.MX 6)

gPTP

Linux PHC subsystem AVB Tx Shaping

ENET 1588 timer

ENET Low-level driver

Regular network stack

MAAP

PKT RX/TX

CLK control

Timer

HAL SRP AVTP

ENET PTP driver

ENET MAC

Media stack /

applicationOther

processes

Kernel

User-space

Media clock driver

AVB Rx bypassqueues

AVDECC

OS abstraction layer

Audio interfaces

SDMAMedia clock

Audio/VPU/GPU drivers

KernelMedia interface driver

AVB library

Videointerfaces

TM


Platforms

• Primary target is i.MX 6Quad/6Dual, running Linux OS

• For secondary targets (SoCs and OSs)− i.MX 6SoloX, Linux OS Cortex-A5

i.MX 6 Series i.MX 6SoloX VFxxx Controller

ARM

Cortex-A9

ARM

Cortex-A9

ARM

Cortex-M4

ARM

Cortex-A5

ARM

Cortex-M4

Linux

OSX X X

Priorities• i.MX 6Quad/6Dual (ARM Cortex-A9) on Linux

OS for Automotive• i.MX 6SoloX (ARM Cortex-A9) on Linux OS for

Automotive/Prosumer Audio• VF6xx (ARM Cortex-A5)

TM


AVB Stack Roadmap

Color Legend

Released

Committed

Roadmap Date

Original Release

Major Release

Version/Beta

Service Pack

2015 2Q 3Q 4Q1Q

Release Date: Apr. 2015

• Device/OS support

− i.MX 6, VF6xx

− Linux RT

• Features

− AVTP A/V packetizer

− FQTSS Class A, B, BE

− Media Clock Recovery

& Capture

− gPTP, SRP, AVDECC

− AVB API

• Evaluation Package

− Audio Sampler, Server, Amplifier with GUI control

− Video Server, multi-Players

− Audio/Video Sync

− ALSA & GStreamer

AVB Beta3

• New Features

− Multi-stream synchronization

− MJPEG video transport

− AVB API extended

• New Evaluation Package

− A/V Server, multi-Players with GUI control

− Camera source

• Evaluation Package

− Extended Media Applications, integrated withGStreamer and ALSA.

AVB Beta (Final)

• New Device/OS− i.MX 6 SoloX

• New Features− FSL gPTP− Completion &

Stabilization� AVTP, FQTSS, Media

Clock

� SRP, AVDECC

� AVB API

• New Evaluation Package− Extended A/V

application examples

• Development Package

− Yocto based

• New Device/OS

− Kinetis: K6x, K7x

(2016)

− MQX RTOS (2016)

• New Features

- Additional AVTP

formats for A/V, Control

streams

- MAAP

- Preparation for Avnu

Certifications

(Automotive, ProAudio)

• New Demo Application

− Surround Cameras

− Audio amplifier on Kinetis(Cortex M4/ MQX)

AVB

AVB 1.x

Target Date: June. 2015 Target Date: Sep. 2015 Target Date: Q4 2015, Q1 2016

AVB 1.0

TM


AVB Hardware Platforms

TM


AVB i.MX 6Quad/6Dual/6Solo

• MAC support for IEEE-1588• EPIT hardware timer for Tx scheduling• External media clock recovery PLL (Cirrus CS2000)

TM


AVB i.MX 6SoloX

• MAC support for IEEE-1588• Tx credit based scheduler

− Three transmit queues (typically SR Class A, B and best effort)

− Timestamp launch time allows per stream scheduling

• Rx classification− Three receive queues (typically SR Class A, B and best effort)

− VLAN priority based

• EPIT hardware timer for Tx scheduling (lower rate)• Internal adjustable media clock PLL• GPT for media clock cross time stamping

TM


i.MX 6 AVB Development Hardware

• SABRE Automotive Infotainment Rev2 CPU Card with i.MX 6Quad or i.MX 6DualLite

• New “Flexible Ethernet Expansion” Card with options for:

− Atheros Gb PHY (same PHY as on current SABRE hardware for software compatibility)

− 10/100 Broadcom BroadR-Reach ®

PHY

− 10/100 Broadcom BroadR-Reach ® 4-port switch

• First hardware available – 1st Half 2014

• Final boards available – January 2015

TM


AVB Hardware Media Clock Capture/Recovery

TM


i.MX 6Q/6DL Talker or Head Unit, Audio Master Clock = Local

L.O.24.576 MHz

CS2000Clk Sync

i.MX 6Q/6DL

Audio ClkInput

ESAI CODEC on Base Board

Audio ClkInput

Selector

Head UnitMaster Mode

ListenerSlave ModeRef Clk

I2C

Timing Ref(Clk In)

i.MX 6Q/6DL

1588_ Event_0_In

1588_ Event_0_Out

SDMA_Event_1 (input)

1588_Event2_Out

MCLK

MLE145170Programmable

Divider

SPI

750 Hz – 8 kHz

Test point scope

connections for

Events & MCLK

Selector

ListenerHead

Pulse Stretc

h

TM


i.MX 6Q/6DL Listener, Audio Master Clock = Synthesi zed

L.O.24.576 MHz

CS2000Clk Sync

i.MX 6Q/6DL

Audio ClkInput


Audio ClkInput

Selector



I2C

Timing Ref

(Clk In)

i.MX 6Q/6DL

1588_ Event_0_In

1588_ Event_0_Out


1588_Event2_Out

MCLK


Divider

SPI

750 Hz – 8 kHz

Selector

ListenerHead

Pulse Stretc

h

Test point scope

connections for

Events & MCLK

TM


• AVB Signal Assignment

i.MX 6Quad/6Dual

Ref AVB Associated Name MX6Q Ball & ALT Mode Comment

TP43 ENET_1588_EVENT0_IN ENET_TXD1 (ALT4) Legacy system function = MLB CLK

TP40 ENET_1588_EVENT0_OUT GPIO_19 (ALT1) Legacy system function = ENET INT_B

TP41 ENET_1588_EVENT2_OUT ENET_RX_ER (ALT4) Legacy system function = USB_OTG_ID

TP40 SDMA Event 1 DISP0_DAT17(ALT4) Legacy system function = RGB Disp (cluster)

TP42 MCLK, 24.576 MHz CLK2_P/N config’d as input Audio master clock

TM


i.MX 6SX, External Chips Eliminated Due to On-chip AVB MCR

L.O.24.576 MHz

CS2000Clk Sync

i.MX 6SX

Audio ClkInput


Audio ClkInput

Selector



I2C

Timing Ref(Clk In)

i.MX 6SX

1588_ Event_0_In

1588_ Event_0_Out


1588_Event2_Out

MCLK


Divider

SPI

750 Hz – 8 kHz

Selector

ListenerHead

Pulse Stretch

Test point scope

connections for

Events & MCLK

TM


Demo Platform

TM


AVB Demo Platform

SABRE AI CPU board

SABRE AI CPU2 + Base

board

SABRE AI CPU2 + Base

board

SABRE AI CPU2 board

AVB Switch

0

1

2

3

Media Server

A/V Unit

Video Display

Audio Amplifier

Ethernet AVB

SD Card

LVDS

LVDS

Display

Display

Display

LVDS

TM


Q&A

TM

© 2015 Freescale Semiconductor, Inc. | External Use

www.Freescale.com

Audio Video Bridging on i.MX Applications Processorsread.pudn.com/downloads699/doc/comm/2814014/FTF-SHB-F1211.… · 1722a – AVTP (Draft) ... − Represents 125 µs worth of audio

Documents