This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Remind: Solution Provider• Both of SGW-C/U will be provided by NetCracker• Who will emulate HSS? Emulated by NEC/NetCracker
Action & Discussion Items• Logistics - Rack - one at ON.Lab, one at Cavium• Topology representation at ONOS• Packet tracing (Signaling, Data)• Integration• Clarify H/W requirements• Network configuration/map (range of IPs, Ports for each Nodes) should be given to partners (by
Cavium)• Interface between VNF manager and XOS• Method of traffic classification (for Local vs Non-Local)- Also need to decide whether we need ‘Central EPC’ or not.
spine
Leaf
spine
Internet PGW-D
ONOS
OF
vBBUeNB (1)
vBBUeNB (2)
X2, S1-U, S1-MME, S5(GTP-U)
Edge Services(Cache)
UE
GUI
SON
agent
Statistics
Leaf Leaf
RRUSector RRU
SectorUE
MME
Topology & Flow Path (Modified)
3
PGW-C
SGW
For this POC, Only PGW-C will talk to ONOS North-bound
Traffic Classification
Traffic Classification for directing to Local(Edge) Service or to Central Service
Possible options APN (Access Point Name)
• MME assigns PGW based on the APN requested by UE (application)• Easies way from the point of network• However, it needs some setting or application configuration on UE
So, this method may be used for predefined users or applications Not flexible in practice but simple way to showcase
Interception• Tap the traffic behind the BBU and direct it local VNF if it is intended to be handled at the Edge
or send it to the central core otherwise.• If we use destination IP for decision rule, it is inside the GTP header Needs some function or Node which can inspect the packet (GTP)
DPI (Deep Packet Inspection)
LIPA (Local IP Access) & SIPTO (Selected IP Traffic Offload)
Local EPC vs Central EPCScenario options By using only Video Cache at Mobile Edge (without using EPC)
• Once the traffic is classified for ‘local’, it is directed to local cache• No need of local EPC• Doesn’t support Handover. (some solution exist that support handover). But, ‘Okay’ with local usage.
By using both local EPC and Cache• Need local EPC• Support handover• Good to showcase distributed core strategy
Conclusion (Suggestion)• Use ‘APN’ method for traffic classification• Use Emulation (by TeraVM) for central EPC pair
Discussion▪Agent on ONOS to interact with PGW-C▪ How to handle GTP Tunneling▪ Integration efforts
- Integration w/o ONOS first. - Then, go with XOS/ONOS
▪ Rack details▪ Renew APPs APIs, Parameter list▪ XOS : static vs dynamic VM spin-off
- Need to show on-demand deployment at least for the initial setup.
▪ Project Management - SPRINT, JIRA, Trello
Use Cases (Demo Scenarios to show)#1 Local Enterprise Intranet Services Good way to show the benefit of utilizing localized mobile network infrastructure
• Use APN assigned to Enterprise employees for internal usage• Traffics for designated to the Enterprise will be directed to the ‘Local Enterprise Slice’, while employees can still use
normal mobile services through central mobile network• Can show local cache, Intranet, local DNS, Analytics etc.
Can show handover scenario for both internal and inter-cell (micro~macro)
#2 Cell on Demand (ANR) Scenario
• 2 Cells show on SON GUI• Turn lots of U.E traffic form TeraVM• Show congestion on Cells from GUI• Spin-off new Cell• Automatic neighbor & relationship (by SON)• Possible MLB(Mobile Load Balancing) execution
VNF-M interworking (NetCracker) NetCracker’s VNF-Manager interfacing with XOS
will abstract the complexities of VNFs below it Can Cavium’s vBBU and Radisys’ PGW also be
handled in the same way?• It will be good If possible• Need to find out what should be done from Cavium’s
vBBU and Radisys’ PGW
eSON by ACORD framework eSON register on XOS as a service using ACORD vBBUs talk with XOS (connected by using API) eSON will be noticed for the event from BBUs by XOS eSON can access DB formed by XOS/ACORD to get information from vBBUs Same logic can be applied to SGWs, PGWs and MMEs
Works to do
Work to do (summary)
1. EPC integration effort
2. TOSCA model for XOS
3. eSON plug-in for using XOS/ACORD frame (notification)
4. Discovery of Mobile Topology to show GUI
5. VNF-Manager interworking (NetCracker MME & SGW. Also for vBBU & PGW)
ONOS (Virtualization, Slicing) + OpenStack (Multi Domain) + XOS
5GMobility Stack
over Multiple RATs
4GMobility Software Stack : vBBU, VOD,
vCDN, vDNS
Residential Residential
Software Stack vOLT, vSG,
vRouter, vCDN
EnterpriseEnterprise
Software Stack: VPN, VOD, vCDN, ...
Front- Haul To RRUs
Back Haul
RRU OperatorMobile Core
Scope of project for ONS 2016 March 15th Deliverables:
M-CORD ( Mobile Edge) Concept Evaluate delay performance & flow control options for disaggregation of EPC Realize the benefit of mobile edge & service chaining
CORDFabric
VirtualizedBBUs
Edge ServiceFunctions
DistributedEPC
CORDPlatform
EPC ControlFunctions
CentralizedEPC
BackboneSwitch
Video Caching vSGW
vPGW
XOS MME
OpenStack
ONOSPCRF
vSGW
vPGW
vBBU
vBBU
vBBU
Option: UP/CP separation
CORDFabric
ONOS (Virtualization, Slicing) + OpenStack (Multi Domain) + XOS
5GMobility Stack over
Multiple RATs 4G
LTE: vBBU, vCDN, vDNS
Road-map
What should we demo? • Mobile Services at the edge (Video Caching)• SON Configuration for optimization of eNBs • M-CORD Platform for abstraction of Mobile components • Disaggregated EPC model – demonstrate Handover through OF Flow-Table modification
Mobile Edge March Demo Test Scenario:
The Mobility Topology (JSON/Python Script) will be pushed to ONOS if not discoverable The topology will include what has been configured in the Mobile Edge Rack
ONOS GUI will be used to validate topology as known to ONOS and configured . This includes the actual hardware ( Aero flex as UE Hosts, Spine & leaf as connectivity of mobile RAN and control elements
(RRUs, vBBUs, links among vBBUs as representation of RANs RAN connectivity to mobile EPC controls ( MME, SGWs, PGWs, PDN, Service Element )
Initialization Scenarios1-Use Aroflex to validate no UE attachments exists. 2-Use Aeroflex Traffic generator to initiate x number of UE attachments 3-Use ONOS GUI to show connectivity's are established between UEs and Service Termination points
Mobile Edge Cache utilization Scenarios: TBD
Handoff Scenario: TBD
Additional Features and functionality needed for each building blocks Identify development gaps
• ONOS (ON.Lab)
• XOS (ON.Lab)
• RRU ad Edge BBUs ( Cavium)
• Edge P/SGW data plane ( Radisys)
• MME, PGW Control ( NetCracker)
• Orchestration control / VNF management ( XOS/ON.Lab, Radisys)
• eSON application (Airhop)
• TeraVM test VM (Cobham / Aeroflex)
• Smartphones / UEs (Cavium)
• Caching SW (leverage from CORD project)
Cavium
BBU-C
BBU
NetCracker
vMME
vSGW
S1-MME S11S1-U
Radisys
PGW-C
PGW-US5
Mobile CORD ONOS Stack
?
ServiceSGi
XOS
OpenStack
Ope
nFlo
w
PoC: Who does what?
CORDFabric
Update PoC Ownership DiagramAdd SGW/RRU/eSON server etc.
Project Definition • Mobile Edge HW Configuration & Components• Demo POD Details
Radisys Disaggregated Packet Gateway Components
Radisys packet gateway solution consist of two components
Packet gateway Data Plane Component (PGW-D)• Function: Performs gateway functions like GTP tunneling/de-tunneling, packet statistics, charging, lawful
intercept, NAT etc.• Form Factor: x86 based server, CENTOS 7.0, KVM. Based on OVS architecture.• Interfaces
• 3GPP S5 interface with Serving Gateway.• SGi interface with external packet data network.• Open Flow 1.4 interface with ONOS controller.
Packet gateway Control Plane Component (PGW-C)• Function: Implements control aspect of gateway functions like GTPc signaling and application logic.• Form Factor: KVM based Virtual machine, CENTOS 7.0.• Interfaces
• Integration with ONOS north bound interface to program PGW-D.
4 x UE
1x Front Haul Switch
3x vBBU2x RRU
Mobile ONOS
CORD Fabric
TeraVM
Mobile XOS
EPC (MME)
EPC (PGW-D)
PGW-C Add physical server location
for each of the entity
22
POC HW Configuration (need Update) XOS/OpenStack
ONOS
CORD Fabric
Service VNF
vPGW Data, MME, & PGW Control
vBBU (2-3 )
Fronthual Switch ( Ethernet)
vPGW Data, MME, & PGW Control
ON.Lab / x86 Service Decomposition and Orchestration
Provider / HW Role & Characteristics
ON.Lab / x86 Topology Abstraction, Event Processing & Forwarding Control
ON.Lab / x86 Acceleration for Mobility
Airhop (TBD) eSON Application
RadisysNetCracker (x86) Disaggregation of SGW and PGW data plane from control plane
Cavium / ThunderX OF OV Enabled
Optical Cross Connect or Ethernet Switch
Centralized MME, SGW and PGW data plane ( Home Operators EPC) TBD, Simulated (x86)
Interconnect vBBUs to Remote Radio Units
Cavium / OCTEON Fusion Remote Radio Unit providing LTE Uu interfaceRRUs
• (Excel spread-sheet with list of parameters used for provisioning, optimization)
Signaling Diagram – System Initialization
Possible signal flow diagram(s) forsystem initialization
fabric
RRHSector
vBBUeNB (1)
vBBUeNB (2)
vBBUeNB(5)
vBBUeNB(3)
vBBUeNB (4)RRH
RRHSector
RRHSector
RRHSector
RRHSector
RRHSector
X2 X2
X2X2
X2
vMME P-GW
PDN
S-GW
S11 S5
S1MMES1U
Mobile CORD ONOS Data Model Graph: Topology & Sector level Abstraction
Flow table Control Handoff
New Attachments
Link
Port
Link
Link
Link
Link
Port
RRHSector
RRHSector
RRHSector
RRHSector
RRHSector
UE
vPGW-C
Applications eSON ONOS-GUI Applications
Performance & Delay Budgets ( Update )
• Handoff• Mobile CORD
• eSON Real-time update Loops ?? • TBD
• Service Chaining ,• Signaling • Data Path
Backups & Notes
28
BBU SGW-D PGW-D
M-CORD Switch Fabric ( Spin & Leaf)
BBU-C MME SGW-C PGW-C
UE
RRH
Forwarding Controlled by ONOSChaining managed by XOS
3GPPSignaling Traffic
ONOS
ONOS NBI/SBI3GPPData Traffic
PCRF HSS
SDN(ONOS) Control for Mobile Network
InternetEdge Cloud
Architecture Goal
for future
spine
Leaf
spine
Internet
RadisysSGW-D/PGW-D
ONOS
OF
vBBUeNB (1)
vBBUeNB (2)
X2, S1-U, S1-MME, S5(GTP-U)
RRUSector
RRUSector
Edge Services(CDN, DNS..)UE
GUI
PGW-C
Event that make ONOS toChange the Flow-Table
Static Addresses
eSON Airhop-VM Receives
State-updatesFrom eNBs, etc
agent
SGW-C
StatisticsInitial Network configuration
through JSONUE Attachment Control Flow :1- UE RRU MME(3GPP Authentication / Location Update Process. Emulation)2 - MMERRU(Default Bearer Establishment Procedure Below:)3 - MMEONOS SGW/PGW (to create Flow Entry)4 - UE gets IP through ‘Attach Accept’ (MME eNB UE) (Assigned from PGW-C)
UE During Handover:1 – UE starts handover request2 - Standard Handover procedure over X2 static interfaces 3 - MMEONOS SGW (to ONOS updates Flows for UE)
UE Data Flow(Service Request) :1- UE RRU ENB UE gets Data Channel assignment2- MME eNB, S1 Bearer set-up3. Signals ONOS to create Flow Entry(OF will not be used talk to eNB for establishing UE mobility related Flow Entry?OF is just utilized for making Flow Entry from eNB to SGW, which is static)
Leaf Leaf
RRUSector
RRUSector
UE
Websocketswith JSON
Control FunctionsGroup
MME
Topology & Call Flow
For POC#1, OF will be mainly used In managing “Data traffic” relatedFlow Table entries
Architecture Goal
for future
spine
Leaf
spine
Internet
RadisysSGW-D/PGW-D
ONOS
OF
vBBUeNB (1)
vBBUeNB (2)
X2, S1-U, S1-MME, S5(GTP-U)
RRUSector
RRUSector
Edge Services(CDN, DNS..)UE
GUI
PGW-C
Event that make ONOS toChange the Flow-Table
Static Addresses
eSON Airhop-VM Receives
State-updatesFrom eNBs, etc
agent
SGW-C
StatisticsInitial Network configuration
through JSON
Leaf Leaf
RRUSector
RRUSector
UE
Websocketswith JSON
Control FunctionsGroup
MME
Topology & Call Flow (in detail-③)
UE Data Flow (Service Request):1- UE RRU ENB UE gets Data Channel assignment2- MME eNB, S1 Bearer set-up(OF will not be used talk to eNB for establishing UE mobility related Flow Entry?OF is just utilized for making Flow Entry from eNB to SGW, which is static)??
Sequential arrow diagram will be
added
Architecture Goal
for future
CORD PoC_1High Level Overview: Segmented Gateway: Control Path Component
Suggestion for PoC Use Case
Goal: Flow based selection of centralized or local user plane mobility anchor • Simpler: We can keep same LTE interfaces between eNB, MME and SPGW-C.• Part of ETSI NFV demo “SDN Enabled Virtual EPC Gateway”.
Goal: Signaling optimization for IoT, persisting core network bearer• Will need work to optimize S1-MME and S11 signaling, breaks standard interfaces.• Connection less communication for IoT device falls in this category (http://web2-clone.research.att.com/export/sites/att_labs/techdocs/TD_101553.pdf)
Goal: Effectiveness of Open Flow for SGPW-D user plane• Extension for GTP tunnel match and action.• Extension for charging capability (raise asynchronous alarm when data used is within certain margin).• Extension for hierarchical metering needed in PGW (bearer and APN level metering).• Extension for downlink packet buffering needed in SGW-D during paging.
Beyond March Demo Wish List• Connectionless Services to minimize the signaling overhead• DNS @ the edge • Mhealth Applications @ the Edge
Need to add scenario to depictEdge Core usage
Mobile Edge Use case Examples:
UE/Flow performance QOE visualizationGeo-Analysis & behaviorsCell Planning ad Analysis Coverage Analysis Device Impact Radio Block Usage Analysis
Network Performance MonitoringCapacity and monitoring & Analysis Correlation & root cause analysisAd-hoc on demand analysis
Self Organizing real time feedback loop Self healing load balancingEnergy Saving optimizationAuto- neighbor relationship
CORD PoC_1High Level Overview: Segmented Gateway
Main Components of Segmented Gateway PoC
34
CaviumvBBU-D SGW-D PGW-D
M-CORD Switch Fabric ( Spin & Leaf)
BBU-C MME SGW-C PGW-C
UE
RRH
Forwarding Controlled by ONOSChaining managed by XOS
3GPPSignaling Traffic
ONOS
ONOS NBI/SBI
3GPPData Traffic
PCRF HSS
Vm (OF)
Ethernet/CPRI
Sector/Cell
Data Path Component
• Serving Packet gateway data path function.
• Packet gateway data path function.
Control Path Component
• MME functions• Signaling and control plane of serving
and packet gateway.
Radisys FlowEngineData/Forwarding Plane Elements
Architecture Goal
for future
CORD PoC_1High Level Overview: Segmented Gateway: Data Path Component
OFOF
Main Components of Segmented Gateway PoC: Data Path Component
Component: Radisys FlowEngine providing data/forwarding components for PGW and SGWLocation: Distributed and Centralized EPC CORD RacksForm Factor: x86 COTS server (*and EZchip NPU system for high scale/performance) • *NOTE: Radisys FlowEngine also supports deployment as leaf switch.
High Level Block Diagram: Software switch, OVS based architecture
NIC
ovs-vswitchd (control)ovsdb-server(m
anagement)
Open Flow
1.4
Kernel, KVM
User Space
Hardware
DPDK
OV
SD
B
Gateway Specific Open Flow Tables
Table 0
Ingress
Table 1
Access Control
Table 2
Bearer Mapping
Table 3
Value Added Service
Table 4
Forwarding
CORD PoC_1High Level Overview: Segmented Gateway: Data Path Component
Main Components of Segmented Gateway PoC: Data Path Component
Data Path Function Highlights
• Support PGW and SGW data path function, scalable for millions of UEs/devices.
• Flexible definition of gateway function in terms of OF table sequence.
• Ingress table is responsible for sending signaling plane messages to Signaling application via OF PACKET_IN option. It also identifies whether packet is upstream or downstream and saves direction in metadata.
• Bearer mapping table performs gateway core function • de-tunneling (upstream), tunneling (downstream) packets. Rely on OF extension to support GTP tunnels.• For downstream packets, map it to bearer based on Service Data Flow (SDF) and traffic flow template (TFT).• Metering (not in PoC) and Charging.
• Value added service is a placeholder for specific function that can be part of data plane like lawful intercept, NAT, flow redirection for DPI.
Control Interface: Compliant to Open Flow 1.4 and extensions to support PoC use case.Management Interface: OVSDB based interface.
CORD PoC_1High Level Overview: Segmented Gateway: Control Path Component
OFOF
Main Components of Segmented Gateway PoC: Control Path ComponentLocation : Act as one of CORD application.
Control Path Component
Use Case: Optimize LTE signaling overhead
Why: LTE core signaling (eNB -> MME, MME -> SGW) during idle to active (and vie versa) transition is not scalable for IoT deployments.
How: Make eNB to SGW bearer connection permanent, not to be teared down during UE idle state transition.
Additional use case: Connection less communication for IoT device, in case UE and eNB support it. http://web2-clone.research.att.com/export/sites/att_labs/techdocs/TD_101553.pdf
Function: Optimized, Colocated MME, SGW and PGW control plane function.
• Optimize standard LTE interfaces S11, S1-C, S5-C and S8-C.
• During idle state transition SGW-C (and BBU-c) will not be asked to delete its S1-U bearer, SGW-D keeps the bearer information. OF flow timeout mechanism can be used to tune when bearer gets deleted in SGW-D.
• (TBD) Interface with vBBU to keep GTP bearer intact during idle transition.
• When UE goes to active state and established RRC connection, no core signaling is needed to modify bearer in SGW-D.
• Form Factor: All of MME, SGW-C and PGW-C can be part of a VM with S11, S5-C/S8-C replaced by internal interface.
MME
SGW-C PGW-C
S11
S5
S1-MME Decomposed Gateway
Application
ONOS
SGW-D/PGW-D
BBU-C
Architecture Goal
for future
Action Items:• JSON config file to provide the topology to ONOS - future work: Ali implementing link provider (discussion to support mobility)• Segment routing ???• Number of flow rules
CORD PoC_1Decomposed LTE core gateways: Open Items
• Potential partner for gateway control application• Need to work closely to define OF extensions and features need for data plane.
• Overall use cases of PoC_1• How does packet gateway function fit in overall use case targeted for PoC_1?• Is the use case end-to-end voice/video call over CORD network?
• Does the infrastructure exist to simulate end-to-ends call flows (eNB’s, mobiles, OOT service etc.)?• Is deployment options of gateway (access or centralized based on traffic type) part of PoC?• Is the goal to show throughput/latency aspect as well?• Will data monitoring and charging (via PGW data plane) be part of PoC.
• Platform• What type of x86 server is available to run gateway data plane?
Resources & Logistics• Resources (and their location) from Radisys perspective.• On.Lab resources to coordinate gateways based PoC. Does On.Lab get involved in development effort for Poc?
Expectation on what becomes open source
SDN controlled aggregation and backhaulAs you mentioned, if all switches/routers within aggregation and backhaul are SDN controlled, we can define proprietary forwarding mechanism (route based on 10.10.0.3). I see two main challenges with this approach Large Flow Tables: The flow we will create in all switches will be per UE/IoT. This assumes capabilities in all intermediate OF switches to be able to
handle large flows (per UE/IoT). With most switches based on COTS switching silicon (like Broadcom), there will be limits on maximum number of flows that can be supported (probably thousands or hundreds of thousands).
OF extension: In case we need certain specific OF extensions, all the intermediate switches need to support that. Do we still need anchor gateway (PGW)Within LTE, the GTP tunneling plays two roles. First one is to enable mobility of devices by preserving its IP address. This one is not needed for stationary IoT devices. Second role of PGW is to act as anchor for UE traffic so that operators services/policies can be applied to UE traffic. This includes services like DPI, NAT, firewall, lawful intercept, charging, parental control etc. provided on SGi interface (implemented via service chaining). I would assume that some of these services are important for IoT devices as well. When tunneling is applied, it’s easy to route all UE/IoT traffic to PGW anchor where all these services can be applied. In a flat network (without any tunneling), is there a way define an single anchor point for specific device’s ingress/egress traffic? This is probably related to first point, if all intermediate switches can be OF controlled at UE/IoT device level, we can define a switch as anchor for device. One option: static, single tunnel from eNB to SGW and SGW to PGWOne way to address above concerns is to keep tunnel from eNB to SGW-D (and SGW—D to PGW-D), however, no specific to IoT device. There will be one tunnel from eNB to potential SGW-D’s (and from SGW-D’s to PGW-D’s); this tunnel is created statically. We keep most of procedure same (IP address assignment by PGW, UE using this address for its session). With this approach Only eNB and SGW-D/PGW-D solution need to maintain millions of UE/IoT specific flows. All intermediate switches can be plain white box switch. Controller need to program UE/IoT specific flows in eNB/SGW-D/PGW-D nodes and not it intermediate switches. This reduces complexity in
intermediate switches, they continue to route/switch packet based on other header fields. PGW still acts as anchor to apply operators policy and services for IoT device. Still achieves initial goal to remove signaling overhead in individual tunnel creation/deletion for IoT devices. No change to UE software. Not having tunnel per IoT device, certainly limits QoS provisioning on per IoT device basis.
Packet Flow through the MCORD Platform(update)
Preliminary : Operation Steps for Initial Attach Backups
Connections and State Transition : Before / During / After X2 Handover Backups
Connections and State Transition : Before / During / After X2 Handover (Con’t) Backups
Handover Preparation Backups
Handover Execution Backups
Handover Completion Backups
Connections and States Before / After Service Request Backups
Procedure for UE Triggered Service Request Backups
Procedure for UE Triggered Service Request (Con’t) Backups
Procedure for Network Triggered Service Request Backups
Procedure for Network Triggered Service Request (Con’t) Backups
Information in Evolved Packet System Entity Before Service Request Backups
Information in Evolved Packet System Entity After Service Request Backups
Questions / Assumptions:
• Radisys contribution to PoC is around segregated Packet gateway in LTE core network. Idea is to split the traditional gateway in data plane (GTPu, S5/S8) and control plane (GTPc S5/S8, Diameter). The control plane will sit on top of ONOS to program data plane via OF interface. Radisys has data plane solution based on OVS architecture and provides OF/OVSDB interface. We are in discussion on how to handle the control plane aspect.
• We have the eSON parameter list from Airhop that provide the requirements for ONOS NB APIS eNB to eSON APIS:configure_ue_periodic_measurement_report configure_ue_a4_measurement_reportset_pciupdate_ocnupdate_q_offset