Tutorial at the AICCSA ’2007) May 13-16, 2007, Amman, Jordan
Control Problems in Multi-service, Multi-platform Telecommunication Networks
Instructor: Prof. Franco Davoli Department of Communications, Computer and Systems Science (DIST), and Italian National Consortium for Telecommunications (CNIT) – University of Genoa Research Unit University of Genoa Via Opera Pia 13 I-16145 Genova Italy Tel.: +390103532732 Fax: +390103532154 E-mail: [email protected] Summary: Telecommunication Networks are complex, distributed, large-scale systems. Networking is basically a resource allocation activity, in a broad sense, where resources are typically represented by bandwidth, storage and processing capacity. The optimization of these resources, whose main final goals are to provide Quality of Service (QoS) to the users at reasonable and competitive prices and to maximize the revenue of the network operator, can be viewed under two different, related perspectives: for network planning purposes and for real time control of network operations. The importance of these aspects is enhanced by the ever-increasing presence of transfer modes based on statistical multiplexing paradigms (e.g., the Internet) and of multiple services within all types of networks. Moreover, complexity is added to the problem by the heterogeneity of networking platforms: though a sort of common paradigm at the network layer and above is that of the Internet Protocol Suite (and related QoS mechanisms), there are a number of different physical transport environments, which have widely different characteristics in terms of transmission capacity, error resilience, operational complexity, and scalability. The aim of the tutorial is to explore different areas in networking (QoS-Internet, cellular networks and wireless LANs, satellite networks, optical networks) from the point of view of resource allocation and QoS control, to outline the main problem areas and to point to some common control techniques arising in the different environments. Outline of Topics: Control Problems in Telecommunication Networks
• General aspects and common ground • Problem areas in networking control • Heterogeneous networking environments • Quality of Service • Timescales - control, management, planning • Networking technologies
A-PDF MERGER DEMO
Optimal Control of Dynamic Systems
• Representations of dynamic systems • Controlled Markov chains • Markov Decision Processes • Functional and parametric optimization • Dynamic Programming • Optimization Techniques
Call Admission Control, Bandwidth Allocation, Congestion Control
• Admission policies • Service separation - Decoupling low- and high-level constraints • Bandwidth allocation • Dynamic routing of flows • Pricing • Applications in wired and wireless networking platforms
Instructor’s short bio: Franco Davoli received the ‘laurea’ degree in Electronic Engineering in 1975 from the University of Genoa, Italy. Since 1990 he has been Full Professor of Telecommunication Networks at the University of Genoa, where he is with the Department of Communications, Computer and Systems Science (DIST). From 1989 to 1991 and from 1993 to 1996 he was also teaching classes in Telecommunication Networks at the University of Parma, Italy. His past research activities have included adaptive and decentralized control, large scale systems, routing and multiple access in packet-switched communication networks, packet radio networks. His current research interests are in bandwidth allocation, admission control and routing in multiservice networks, wireless mobile and satellite networks and multimedia communications and services. He has co-authored over 250 scientific publications in international journals, book chapters and conferente proceedings. In 2004, he has been the recipient of an Erskine Fellowship from the University of Canterbury, Christchurch, New Zealand, as Visiting Professor. He has been Principal Investigator in a large number of research projects, and has served in several positions in the Italian National Consortium for Telecommunications (CNIT), including the direction of the National Laboratory for Multimedia Communications in Naples in the period 2002-2004; he is currently Vice-President of the CNIT Management Committee. He is a Senior Member of the IEEE. Target Audience: students, researchers, instructors and professionals interested in the field of telecommunication networks. A minimum knowledge of basic networking principles is required.
1
Control Problems in Multi-service,Multi-platform Telecommunication Networks
SAMPLE SLIDES
Italian National Consortium for TelecommunicationsItalian National Consortium for Telecommunicationscnitcnit
Franco DavoliFranco DavoliDepartment of Communications, Computer and Systems Department of Communications, Computer and Systems
Science (DIST), University of Genoa, ItalyScience (DIST), University of Genoa, Italy
CNITCNIT--University of Genoa Research UnitUniversity of Genoa Research [email protected]@dist.unige.it
cnitcnit
2
OutlineOutline
Control problems in Telecommunication Control problems in Telecommunication NetworksNetworks
General aspects and common groundGeneral aspects and common groundProblem areas in networking controlProblem areas in networking controlHeterogeneous networking environmentsHeterogeneous networking environmentsTimescales Timescales -- control, management, planningcontrol, management, planning
cnitcnit
3
OutlineOutline (cont’d)(cont’d)
Call Admission Control, Bandwidth Allocation Call Admission Control, Bandwidth Allocation and Routingand Routing, , Congestion ControlCongestion Control
Admission policiesAdmission policiesService separation Service separation -- Decoupling lowDecoupling low-- and highand high--level level
constraintsconstraintsBandwidth allocation in ATM and IP terrestrial, mobile Bandwidth allocation in ATM and IP terrestrial, mobile wireless and satellite networks wireless and satellite networks -- Dynamic routing of flowsDynamic routing of flowsSchedulingSchedulingPricingPricing
cnitcnit
4
OutlineOutline (cont’d)(cont’d)
Instances of control techniques in specific Instances of control techniques in specific environmentsenvironments
Access networksAccess networks
Mobile wireless networks and satellite networksMobile wireless networks and satellite networks
CrossCross--layer approacheslayer approaches
Approximation techniques and parametric Approximation techniques and parametric optimizationoptimization
cnitcnit
5
IntroductionIntroduction
A large number of dynamic control and resource A large number of dynamic control and resource allocation problems arise in almost all types of allocation problems arise in almost all types of communication networks.communication networks.Among others, some examples of the most Among others, some examples of the most commonly found ones are:commonly found ones are:
Connection Admission Control (CAC)Connection Admission Control (CAC)Bandwidth AllocationBandwidth AllocationCongestion ControlCongestion ControlRoutingRoutingSchedulingSchedulingPower control in wireless networksPower control in wireless networks
cnitcnit
6
IntroductionIntroduction (cont’d)(cont’d)
Such problems are encountered in different Such problems are encountered in different networking environments, cabled and wireless; networking environments, cabled and wireless; basicallybasically
TDMTDM--based structures (circuitbased structures (circuit--switched telephone switched telephone networks; mobile radio networks, even in conjunction networks; mobile radio networks, even in conjunction with CDMA; satellite networks)with CDMA; satellite networks)ATM networksATM networksIP networks with IP networks with DiffServ/IntServ DiffServ/IntServ paradigms, MPLSparadigms, MPLSOptical networks (with Optical networks (with MPMPλλSS, GMPLS), GMPLS)Wireless networksWireless networks
The various structures may appear together (in The various structures may appear together (in particular, IPparticular, IP--overover--X)X)
cnitcnit
7
IntroductionIntroduction (cont’d)(cont’d)
Control problems appear at various architectural layersControl problems appear at various architectural layersPhysical and Data Link: dynamic power control and fade Physical and Data Link: dynamic power control and fade countermeasures, bandwidth allocation among users and services, countermeasures, bandwidth allocation among users and services, multiple access, …multiple access, …Network: dynamic bandwidth allocation, routing, Call Admission Network: dynamic bandwidth allocation, routing, Call Admission Control, packet scheduling, …Control, packet scheduling, …Transport: elastic bandwidth allocation, congestion control, …Transport: elastic bandwidth allocation, congestion control, …Application: congestion control (e.g., TCPApplication: congestion control (e.g., TCP--friendly applications), friendly applications), rate adaptation, pricing, …rate adaptation, pricing, …
CrossCross--layer approacheslayer approaches (i.e., exploiting information from (i.e., exploiting information from other layers for control purposes) are often advisable, other layers for control purposes) are often advisable, especially at lower layers in noisy environments.especially at lower layers in noisy environments.
cnitcnit
8
Purpose of the TutorialPurpose of the Tutorial
To give an overview of some control To give an overview of some control issues and techniques commonly used issues and techniques commonly used in telecommunication networksin telecommunication networksTo show instances of their application in To show instances of their application in different networking environmentsdifferent networking environmentsTo point out possible open problems To point out possible open problems and research areasand research areas
cnitcnit
9
Goals of controlGoals of control
The ultimate goal is to provide some level of The ultimate goal is to provide some level of Quality of ServiceQuality of Service ((QoSQoS) to the entities (data ) to the entities (data units, connections, applications, end users) units, connections, applications, end users) that are being considered, depending on the that are being considered, depending on the specific layer or the “granularity” (or on the specific layer or the “granularity” (or on the scope or “width”) they are looked upon.scope or “width”) they are looked upon.According to ITUAccording to ITU--T E800, T E800, QoS QoS is interpreted is interpreted asas
The overall effect of performanceThe overall effect of performance--enabling enabling services that determine the degree of satisfaction services that determine the degree of satisfaction of a service user.of a service user.From the viewpoint of the telecommunication From the viewpoint of the telecommunication network, network, QoS QoS translates into the capability of the translates into the capability of the network to guarantee a specific service level.network to guarantee a specific service level.
cnitcnit
10
Quality of ServiceQuality of Service
Indeed, the term Indeed, the term QoS QoS has a number of has a number of interpretations, which range from the quality interpretations, which range from the quality perceived by the service user to a set of performance perceived by the service user to a set of performance (in general, layer(in general, layer--specific) parameters that is specific) parameters that is necessary to specify to obtain the desired level of necessary to specify to obtain the desired level of service.service.E.g., one may distinguishE.g., one may distinguish–– Intrinsic Intrinsic QoS QoS : directly provided by the network and described in : directly provided by the network and described in
terms of objective indicators, like loss (of data units or conneterms of objective indicators, like loss (of data units or connections) ctions) and transfer delay.and transfer delay.
–– Perceived Perceived QoS QoS -- PP--QoSQoS: as subjectively measured by the : as subjectively measured by the Mean Mean Opinion Score (MOS).Opinion Score (MOS).
–– Assessed Assessed QoS QoS : as referred to the user’s willingness to continue : as referred to the user’s willingness to continue using a service. Related to using a service. Related to PP--QoSQoS, but also dependent from the , but also dependent from the pricing mechanism, the support guaranteed by the provider and pricing mechanism, the support guaranteed by the provider and other commercial and market aspects.other commercial and market aspects.
cnitcnit
11
Service Level SpecificationService Level Specification
QoS QoS provisioning is often offered in terms of provisioning is often offered in terms of objective indicators, by using aobjective indicators, by using aService Level Specification Service Level Specification -- SLSSLS..The SLS is a set of performance indexes and The SLS is a set of performance indexes and of their required values that together define of their required values that together define the service offered to a given traffic.the service offered to a given traffic.The SLS is the technical part of an The SLS is the technical part of an agreement, negotiated between service user agreement, negotiated between service user and provider, relatively to the characteristics and provider, relatively to the characteristics of the service itself and to the associated set of the service itself and to the associated set of metrics (of metrics (Service Level Agreement Service Level Agreement -- SLASLA).).
cnitcnit
12
ApplicationsApplications
Which applications need some form of Which applications need some form of QoSQoS? ? All applications requiring a specified level of All applications requiring a specified level of “guarantee” from the network “guarantee” from the network –– Services for Services for the the transport transport of aggregate of aggregate
information information ((bandwidth from providersbandwidth from providers, VPN), VPN)In the access networkIn the access networkIn the In the backbone backbone networknetwork
–– VideoconferencingVideoconferencing, , videotelephonyvideotelephony–– VoIPVoIP, Internet , Internet TelephonyTelephony–– TeleTele--medicinemedicine–– TeleTele--educationeducation–– Remote ControlRemote Control–– Emergency Emergency ((Disaster RecoveryDisaster Recovery) ) applicationsapplications–– ……
cnitcnit
13
Requests for Requests for QoSQoS
Market Market requestsrequestsWidespread diffusion Widespread diffusion of the of the Internet Internet Protocol Protocol SuiteSuite as as a “a “universaluniversal” ” platformplatformNeed Need of of mechanisms to provide quality mechanisms to provide quality endend--toto--endend QoS QoS on IP on IP networks networks and and across across multiple multiple domainsdomains..
cnitcnit
14
ITUITU--T T QoS Classes QoS Classes ((for for IP)IP)
Traditional applications of best-effort IP networks5
Low loss (short transactions, streaming data flow)4Data transactions, interactive3
Data transactions, highly interactive2Real-time, delay jitter sensitive, interactive1
Real-time, delay jitter sensitive, highly interactive0
CharacteristicsQoSClass
ITU-T Y-1541
cnitcnit
15
QoS QoS metrics (IP)metrics (IP)
IPLR IPLR -- IPIP Packet Loss Packet Loss RateRateIPTD IPTD -- IPIP Packet Packet TransferTransfer DelayDelayIPDV IPDV -- IPIP Packet Delay VariationPacket Delay VariationIPER IPER –– IPIP Packet Error Packet Error RateRateSkewSkew ((average value average value of the of the delay delay difference among packets belonging to difference among packets belonging to differentdifferent, , mutually synchronizedmutually synchronized, , media)media)……
cnitcnit
16
QoS QoS metrics metrics -- RequirementsRequirements
1 x 10-4Upper limitIPER
Un-specified
1 x 10-31 x 10-31 x 10-31 x 10-31 x 10-3Upper limit on packet loss rate
IPLR
Un-specified
Un-specified
Un-specified
Un-specified
50 ms50 msUpper limit on1-10-3 quantile of
IPTD less min IPTD
IPDV
Un-specified
1 s400 ms100 ms400 ms100 msUpper limit on average IPTD
IPTD
Class 5 Un-
specified
43210Performance Parameter
QoS Classes
cnitcnit
17
QoS QoS metrics metrics -- An exampleAn example
cnitcnit
18
QoS QoS controlcontrol
Functionalities and ToolsFunctionalities and Tools
Identification Identification of of traffic flowstraffic flowsCall Admission Call Admission ControlControlTraffic EngineeringTraffic EngineeringScheduling (Scheduling (Service Service discipline)discipline)Flow Flow and and congestion congestion controlcontrolQoS QoS RoutingRoutingResource AllocationResource Allocation
cnitcnit
19
QoS QoS control control -- Time scalesTime scales
packet time
round-triptime
connection time
long term
packet marking CAC
traffic shaping
scheduling
resource reservation
flow control
QoS routing
A wide range of time scales, with orders of magnitude from A wide range of time scales, with orders of magnitude from few few µµs to minutes, hours and days. Accordingly, a set of s to minutes, hours and days. Accordingly, a set of (related) resource allocation and control problems, spanning(related) resource allocation and control problems, spanning•• Network ControlNetwork Control•• Network ManagementNetwork Management•• Network PlanningNetwork Planning
cnitcnit
20
QoS QoS mappingmapping
Services offered by lower layers should Services offered by lower layers should provide provide QoS QoS mappingmapping functions for the functions for the benefit of higher layers benefit of higher layers Implementing endImplementing end--toto--end guarantees (if at all end guarantees (if at all possible!) would imply cooperation among possible!) would imply cooperation among layerslayersHoweverHowever, care , care should be taken should be taken in crossin cross--layer layer approachesapproaches, in , in order not to disrupt order not to disrupt architectural principles that ensure architectural principles that ensure interoperabiltyinteroperabilty
cnitcnit
21
Transport technologiesTransport technologies
Higher layers - IP
SONET/SDH(ATM)
TDM
Ethernet Other…
WDM
cnitcnit
22
QoS QoS control control -- TechnologiesTechnologies
““Throwing bandwidth Throwing bandwidth at theat the problemproblem””ATMATMIP (IPv4 and IPv6)IP (IPv4 and IPv6)
DiffServ DiffServ // IntServIntServ
MPLSMPLS
cnitcnit
23
Technologies Technologies -- MPLSMPLS
Label Switching Label Switching -- Traffic engineering capabilitiesTraffic engineering capabilities
IPIP ATMATMMPLSMPLS QoS QoS handlinghandling
Control Control functionsfunctionsSimpleSimpleUniversally widespreadUniversally widespread
C1
C3
C2IP
IP 5IP 10
IP
LER
LSR
LERLSRLSR
cnitcnit
24
Technologies Technologies -- MPLSMPLS
32-bits
Label MPLS
Label (20 bits) TTLSEXP
IP
–– LabelLabel–– ExperimentalExperimental–– Stacking bitStacking bit (indicates presence of more (indicates presence of more labelslabels))–– Time to liveTime to live
IP on guaranteed performance networkIP on guaranteed performance network
cnitcnit
25
Technologies Technologies -- MPLSMPLS
C1
C3
C2IP
IP 5IP 10
IP
LER
LSR
LERLSRLSR
–– LER (LER (Label Edge RouterLabel Edge Router) at ) at ingress applies ingress applies LabelLabel to packet to packet and and sends sends over over correct correct LSP (LSP (Label Switched PathLabel Switched Path).).
–– LSRs LSRs ((Label Switched RouterLabel Switched Router) ) switch packetswitch packet, , swapping labelsswapping labels–– Egress Egress LER LER eliminates label eliminates label and and forwards packet with forwards packet with IP IP forwarding forwarding
procedureprocedure
cnitcnit
26
Technologies Technologies -- Flow IDFlow ID
ATMATM
IPv4IPv4
IPv6IPv6
8 7 6 5 4 3 2 1Bit
1 2 3 4 5
Ottetti
GFC VPI
VPI
VCI
VCI
VCI PT RES CLP
HEC
Version Type of Service Total Length
Identification Flags Fragment Offset
Source Address
Destination Address
IHL
Time To Live Protocol Header checksum
Options Padding
TCP/UDP Port - Sourcee TCP/UDP Port - Destination
Version Class Flow LabelPayload Length Next Header Hop Limit
Source Address
Destination Address
cnitcnit
27
QoS QoS control functionscontrol functions
Traffic flow identificationTraffic flow identificationCACCACRate control, Rate control, traffic shaping traffic shaping and and filteringfilteringBandwidth allocationBandwidth allocation
Scheduling Scheduling ((service service discipline)discipline)Flow Flow and and congestion congestion controlcontrolQoS RoutingQoS Routing
cnitcnit
28
Markov Chains, Markov Chains, MDPsMDPs, and Optimization, and Optimization
Discrete Time Markov ChainsDiscrete Time Markov ChainsContinuous Time Markov ChainsContinuous Time Markov ChainsMarkov Decision ProcessesMarkov Decision ProcessesDynamic ProgrammingDynamic ProgrammingInfiniteInfinite--horizon optimizationhorizon optimizationNumerical techniquesNumerical techniquesControl Issues in TDM, ATM and IP networksControl Issues in TDM, ATM and IP networksExamplesExamples
cnitcnit
29
Representation of networks asRepresentation of networks asdynamic systemsdynamic systems
Telecommunication networks are most often modeled Telecommunication networks are most often modeled as multias multi--dimensional dimensional complexcomplex dynamicdynamic stochasticstochasticsystems. This means they are interconnected systems. This means they are interconnected subsystems, whose state depends on time, and subsystems, whose state depends on time, and whose behavior may be driven by external random whose behavior may be driven by external random variables. In particular, they may be most often variables. In particular, they may be most often represented as represented as queueing queueing systemssystems..There are many, often equivalent, representations of There are many, often equivalent, representations of complex dynamical systems, e.g., in terms of inputcomplex dynamical systems, e.g., in terms of input--output differential equations, transfer function output differential equations, transfer function matrices, state equations. One of the most commonly matrices, state equations. One of the most commonly used for used for queueing queueing systems and networks is in terms systems and networks is in terms of of Markov chainsMarkov chains..
cnitcnit
30
Markov ChainsMarkov Chains
Markov chains are Markov processes whose timeMarkov chains are Markov processes whose time--dependent random variables (the dependent random variables (the statestate of the Markov of the Markov chain) can assume values in a discrete set (the chain) can assume values in a discrete set (the statestatespacespace), either finite or ), either finite or countably countably infinite.infinite.The Markov property is essentially a conditional The Markov property is essentially a conditional independence of the future evolution on the past (the independence of the future evolution on the past (the whole history of the process being summarized in the whole history of the process being summarized in the current state).current state).Basically, the chain can be seen as modeling the Basically, the chain can be seen as modeling the position of an object in a discrete set of possible position of an object in a discrete set of possible locations over time, the next location being chosen at locations over time, the next location being chosen at random from a distribution that depends only on the random from a distribution that depends only on the current one.current one.
cnitcnit
31
Discrete Time Markov ChainsDiscrete Time Markov Chains
Definition 1.1.Definition 1.1. A stochastic processA stochastic processatat consecutive points of observation 0,1,…,n,… is aconsecutive points of observation 0,1,…,n,… is a
DTMC if, for allDTMC if, for all
X0, X1,...,Xn ,...{ }
n ∈N o , xn ∈S
Pr Xn +1 = xn +1 Xn = xn, Xn −1 = xn −1,..., Xo = xo{ }=
= Pr Xn +1 = xn+1 Xn = xn{ }Let . The quantitiesLet . The quantitiesS = 0,1, 2,...{ }
pij = Pr Xn +1 = jXn = i{ }= Pr X1 = jXo = i{ }are the oneare the one--step transition probabilities of a step transition probabilities of a homogeneoushomogeneouschain, i.e., whose conditional chain, i.e., whose conditional pmf pmf is independent of time.is independent of time.
∇∇
cnitcnit
32
DTMCDTMC (cont’d)(cont’d)
IrreducibilityIrreducibility
Definition 1.2.Definition 1.2. A transition matrix P on the state space A transition matrix P on the state space SS isissaid to be said to be irreducibleirreducible if it is possible for a Markov chain withif it is possible for a Markov chain withTPM P to move from any state TPM P to move from any state ii to any other state to any other state jj in finitein finitetime, i.e., if there is a path between any two states in thetime, i.e., if there is a path between any two states in thecorresponding transition diagram. A DTMC is corresponding transition diagram. A DTMC is irreducibleirreducible if itsif itsTPM P is irreducible.TPM P is irreducible.Theorem 1.1.Theorem 1.1. An irreducible DTMC has An irreducible DTMC has at most one invariantat most one invariantdistributiondistribution (it certainly has one if it is finite). A DTMC with(it certainly has one if it is finite). A DTMC withone invariant distribution is said to be one invariant distribution is said to be positive recurrentpositive recurrent..
The invariant distribution measures the fraction of time thatThe invariant distribution measures the fraction of time thatthe DTMC spends in the various statesthe DTMC spends in the various states..
∇∇
∇∇
cnitcnit
33
DTMCDTMC (cont’d)(cont’d)
Periodicity and Periodicity and ErgodicityErgodicity
LetLet
be the greatest common divisor of the number of steps be the greatest common divisor of the number of steps nn such thatsuch thatthe DTMC can go from state the DTMC can go from state ii back to itself in n steps (for anback to itself in n steps (for anirreducible DTMC, d is the same for all states).irreducible DTMC, d is the same for all states).Definition 1.3.Definition 1.3. Let P be an irreducible TPM on Let P be an irreducible TPM on SS. If . If d > 1d > 1, then P, then Pis said to be is said to be periodic with period dperiodic with period d. If . If d = 1d = 1, then P is said to be, then P is said to beaperiodicaperiodic..Theorem 1.2.Theorem 1.2. For an For an irreducibleirreducible and and aperiodicaperiodic DTMC with invariantDTMC with invariantdistribution , the limit exists, is independent of the idistribution , the limit exists, is independent of the initial statenitial stateand coincides with the unique steadyand coincides with the unique steady--state probability vector.state probability vector.
An An irreducibleirreducible, , aperiodicaperiodic DTMC with all states being DTMC with all states being positivepositiverecurrentrecurrent is said to be is said to be ergodicergodic..
d = GCD n ≥ 1(Pn )i ,i > 0{ }
π ˜ π
∇∇
∇∇
cnitcnit
34
Continuous Time Markov ChainsContinuous Time Markov Chains
CTMC’s CTMC’s can be viewed as can be viewed as DTMC’s DTMC’s with an with an
infinitesimally small time unit.infinitesimally small time unit. However, a more However, a more direct definition can be used.direct definition can be used.To this aim, we recall the properties of an To this aim, we recall the properties of an exponentially distributed r.v. exponentially distributed r.v. τ:τ:
The r.v. The r.v. ττ is exponentially distributed with rate is exponentially distributed with rate λλ > 0 if> 0 if;;
If If ττ is exponentially distributed with rate is exponentially distributed with rate λλ, then, then---- ττ is is memorylessmemoryless, i.e.,, i.e.,
Pr τ > t{ }= e−λt , t ≥ 0
E τ{ }= 1 λ
Pr τ > s + t τ > s{ }= Pr τ > t{ }, ∀s, t ≥ 0
cnitcnit
35
DTMCDTMC (cont’d)(cont’d)
ExamplesExamples
PStationaryprobabilityvector(s)
˜ P ˜ π Uniquesteady-stateprobabilityvector
1 1 1 0
0 1
Infinitelymany
˜ P = Pn = P,
∀n
˜ π = π(0) ˜ P == π(0) None
1
1 0 1
1 0π = .5 .5[ ] P n does not
converge˜ π does not
existNone
. 5. 5 . 5
. 5
.5 .5
.5 .5π = .5 .5[ ]
˜ P = Pn = P,
∀n
˜ π = π(0) ˜ P =
= .5 .5[ ]
π = ˜ π =
= .5 .5[ ]
1
1
0 1
0 1π = 0 1[ ]
˜ P = Pn = P,
∀n
˜ π = π(0) ˜ P =
= 0 1[ ]= πNone
( ˜ π 0 = 0 )
10
0 1
0 1
0 1None that covers
the whole state space
cnitcnit
36
CTMCCTMC (cont’d)(cont’d)
∇∇
Definition 1.4.Definition 1.4. Let Let SS be a countable set. A rate matrix be a countable set. A rate matrix QQ on on SSis a collection of real numbers s.t.is a collection of real numbers s.t.
Definition 1.5.Definition 1.5. Given a countable set Given a countable set SS, a rate matrix Q on , a rate matrix Q on SS,,and an initial distribution and an initial distribution ππ, the CTMC is defined, the CTMC is definedas followsas follows
Choose with distribution Choose with distribution ππ in in SS;;If , select a random time If , select a random time ττ that is exponentially distributed with that is exponentially distributed with rate ; define X s.t. ;rate ; define X s.t. ;At time At time t=t=ττ, the process jumps from the initial value i to a new value , the process jumps from the initial value i to a new value j, selected independently of j, selected independently of ττ s.t.s.t.
The construction resumes from there, independently of the procesThe construction resumes from there, independently of the process before s before τ.τ.
Q = qij, i, j ∈S{ }0 ≤ qij < ∞ , ∀i ≠ j∈S , and
−qii = qi ≡ qijj≠i∑ < ∞ , ∀i ∈S
X = Xt, t ≥ 0{ }
x0x0 = i
q i Xt = i for 0 ≤ t < τ
Pr Xτ = jX0 = i, τ{ }= Γij ≡ qij qi , j ≠ i
∇∇
cnitcnit
37
CTMC CTMC (cont’d)(cont’d)
Again, the only information about the trajectory of X Again, the only information about the trajectory of X up to time t that is useful for predicting the trajectory up to time t that is useful for predicting the trajectory after time t is the current value . after time t is the current value . Note, in passing, that in processes having nonNote, in passing, that in processes having non--exponential holding times, but conditionally exponential holding times, but conditionally independent successive jumps (e.g., M/G/1 independent successive jumps (e.g., M/G/1 queueing queueing systems, with Poisson arrivals and general service systems, with Poisson arrivals and general service time distribution), it can be useful to study the time distribution), it can be useful to study the (discrete time) Markov chain arising from the process (discrete time) Markov chain arising from the process considered at certain jump times (considered at certain jump times (embedded Markov embedded Markov chainchain). Such processes are called ). Such processes are called semisemi--MarkovMarkov..(It is remarkable that the steady(It is remarkable that the steady--state distribution of the state distribution of the embedded DTMC in the M/G/1 case is the same as the one of embedded DTMC in the M/G/1 case is the same as the one of the original the original nonnon--Markovian Markovian process. This is due to the soprocess. This is due to the so--called called PASTAPASTA property (property (Poisson Arrivals See Time AveragesPoisson Arrivals See Time Averages)).)).
Xt
cnitcnit
38
CTMC CTMC (cont’d)(cont’d)
ξ0 τ 0
ξ1
τ1
ξ2 τ 2
ξn τ n
t
Xt
Pr ξ0 = i 0 ,ξ1 = i1, ...,ξn = i n{ }= πi0Γi0 i1
Γi1i2⋅⋅ ⋅Γin −1in
cnitcnit
39
Controlled Markov ChainsControlled Markov Chains
In both DiscreteIn both Discrete--Time and ContinuousTime and Continuous--Time Markov Chains, Time Markov Chains, the elements of the transition or rate matrices may be the elements of the transition or rate matrices may be dependent on a variable dependent on a variable uu, whose values in a set , whose values in a set U(i)U(i)determine the transition probability or rate, given state i. determine the transition probability or rate, given state i. We can write We can write or to evidence the functional or to evidence the functional dependence, or think of the elements of the matrices as dependence, or think of the elements of the matrices as being being parametrized parametrized by u, which represents a control action. by u, which represents a control action. We talk in this case of a We talk in this case of a controlled Markov chaincontrolled Markov chain..In many cases of interest, the sets In many cases of interest, the sets U(i)U(i) may be finite. In may be finite. In general, the action u stems from a general, the action u stems from a control lawcontrol law (or (or strategystrategyor or policypolicy), which determines the action as a function of ), which determines the action as a function of available information on the process state, either available information on the process state, either deterministically (deterministically (pure policypure policy) or on the basis of a ) or on the basis of a probability distribution over the action space (probability distribution over the action space (randomized randomized policypolicy).).
pij(u) q ij(u)
cnitcnit
40
Markov Decision ProcessesMarkov Decision Processes
In general, the information available on the process at time t In general, the information available on the process at time t (either discrete or continuous) may be denoted by I(t), and may (either discrete or continuous) may be denoted by I(t), and may represent a whole collection of past observations on the represent a whole collection of past observations on the system’s state (either perfect or partial); the goal of the contsystem’s state (either perfect or partial); the goal of the control rol law may be the (functional) minimization of some average cost law may be the (functional) minimization of some average cost (or maximization of average revenue), over a time horizon that (or maximization of average revenue), over a time horizon that may be finite or infinite. This is the general setting (in an may be finite or infinite. This is the general setting (in an extended sense) of extended sense) of Markov Decision Processes (MDP)Markov Decision Processes (MDP)..A good deal of control problems arising in telecommunication A good deal of control problems arising in telecommunication networks (e.g., multiple access, CAC, flow control, dynamic networks (e.g., multiple access, CAC, flow control, dynamic bandwidth allocation among traffic classes) admit a general bandwidth allocation among traffic classes) admit a general formulation in terms of formulation in terms of MDPsMDPs. In some instances, there may . In some instances, there may even be more than one decisional agent, and such agents may even be more than one decisional agent, and such agents may possess different information on the system’s state, leading to possess different information on the system’s state, leading to formulations in terms of game or team theory.formulations in terms of game or team theory.
cnitcnit
41
Markov Decision Processes Markov Decision Processes (cont’d)(cont’d)
As far as finite control horizons are concerned, even problems As far as finite control horizons are concerned, even problems with partial or imperfect (e.g., noisy) observations of the with partial or imperfect (e.g., noisy) observations of the system’s state may be treated efficiently, whenever it is system’s state may be treated efficiently, whenever it is possible to extract from the whole set of past observations possible to extract from the whole set of past observations (which is growing with time!) a finite(which is growing with time!) a finite--dimensional set of dimensional set of quantities that, loosely speaking, contain all the information iquantities that, loosely speaking, contain all the information in n I(t) that is necessary for control purposes. Such a set is calleI(t) that is necessary for control purposes. Such a set is called d a a sufficient statisticsufficient statistic (or (or information stateinformation state).).Over infinite control horizons (that are of interest because Over infinite control horizons (that are of interest because they may be characterized by stationary (timethey may be characterized by stationary (time--invariant) invariant) optimal control laws), little exists regarding the case of optimal control laws), little exists regarding the case of imperfect information. imperfect information. RecedingReceding--horizonhorizon approximations approximations ((repetitive controlrepetitive control), either ), either closedclosed--looploop or or openopen--loop feedbackloop feedbackmay sometimes be viable solutions.may sometimes be viable solutions.In order to expose the basic principles, in the following we In order to expose the basic principles, in the following we limit our treatment to the case of perfect state information.limit our treatment to the case of perfect state information.
cnitcnit
42
MDP MDP (cont’d)(cont’d)
A note on representations of system’s dynamicsA note on representations of system’s dynamics
It may happen that a networking problem can be It may happen that a networking problem can be formulated more directly in terms of (stochastic, i.e., formulated more directly in terms of (stochastic, i.e., driven by some driven by some noisenoise variables) state equations, of variables) state equations, of the typethe type
rather than of Markov chains. As far as discrete time rather than of Markov chains. As far as discrete time and discrete (finite or infinite countable) state spaces and discrete (finite or infinite countable) state spaces are concerned, it is straightforward to reformulate are concerned, it is straightforward to reformulate the dynamics in term of a Transition Probability the dynamics in term of a Transition Probability Matrix.Matrix.
xk+1 = fk (xk , uk , wk)
statestate controlcontrolexogenous stochasticexogenous stochastic
variablevariable
cnitcnit
43
MDP MDP (cont’d)(cont’d)
Control lawsControl laws
In the In the nonnon--randomizedrandomized ((purepure) case, consider control laws ) case, consider control laws of the formof the form
(where t represents a discrete time instant (decision(where t represents a discrete time instant (decisionepoch) where the process changes state).epoch) where the process changes state).
As regards the As regards the randomizedrandomized case, the control law takes on case, the control law takes on the formthe form
s.t. when the process enters state i at time t, action u iss.t. when the process enters state i at time t, action u ischosen with probability . Obviously,chosen with probability . Obviously,
ut = γ t (i) , i ∈S, u t ∈Ut(i), t = t 0 ,t1 ,...
νt = ˜ γ t (i, u) , i ∈S, u ∈Ut(i)
˜ γ t (i, u)
0 ≤ ˜ γ t(i,u) ≤ 1, ∀ i ∈S, u ∈Ut(i) and ˜ γ t(i, u) = 1, ∀i ∈Su∈U t (i)
∑
cnitcnit
44
MDP MDP (cont’d)(cont’d)
Cost (or revenue) Cost (or revenue) functionalsfunctionals
If a cost is associated to the state and control values, of the If a cost is associated to the state and control values, of the type type
then the whole cost of the process can be written as a sumthen the whole cost of the process can be written as a sumor an integral over time. We distinguish the cases of discreteor an integral over time. We distinguish the cases of discrete--and continuousand continuous--time processes, over finite and infinite timetime processes, over finite and infinite timehorizons, respectively. Let represent a polihorizons, respectively. Let represent a policy (acy (awhole set of strategies).whole set of strategies).
DTMC, finite horizonDTMC, finite horizon: :
DTMC, infinite horizonDTMC, infinite horizon::
gt(x t, u t) = gt xt , γ t (xt )[ ]
J ˜ γ (x0) = E gt Xt , γ t(Xt)[ ]x0t=0
N −1
∑
J ˜ γ , av(x0 ) = limN→∞
1
NE gt Xt , γ t (Xt)[ ]
t =0
N −1
∑ x0
J ˜ γ , disc (x0) = limN→ ∞
E α tg t Xt, γ t(Xt )[ ]t=0
N −1
∑ x0
DiscountedDiscountedcostcost
Average expectedAverage expectedcostcost
˜ γ = γ0 , γ1, ...{ }
cnitcnit
45
MDP MDP (cont’d)(cont’d)
Cost (or revenue) Cost (or revenue) functionalsfunctionals
CTMC, finite horizonCTMC, finite horizon::
where is the instant of the where is the instant of the NN--th th jump of the process. This jump of the process. This continuous time cost can be easily continuous time cost can be easily discretized discretized over events over events (by using (by using uniformizationuniformization), to yield a sum over n as in the ), to yield a sum over n as in the discrete case (actually, the same is possible, with some discrete case (actually, the same is possible, with some more complication, if the final time is a fixed instant T, more complication, if the final time is a fixed instant T, rather than random).rather than random).CTMC, infinite horizonCTMC, infinite horizon::
again, there is an equivalent discrete problem that can beagain, there is an equivalent discrete problem that can be obtained by obtained by uniformizationuniformization. The discounted versions are also. The discounted versions are also possible.possible.
J ˜ γ c (x0) = E gt Xt , γ t (Xt)[ ]
0
τN
∫ dt x0
τN
J ˜ γ , avc (x0 ) = lim
T→∞
1
TE gt Xt, γ t(Xt )[ ]
0
T
∫ dt x0
cnitcnit
46
MDP MDP (cont’d)(cont’d)
Cost (or revenue) Cost (or revenue) functionalsfunctionals
In networking problems, the cost (or revenue) is In networking problems, the cost (or revenue) is most often associated with:most often associated with:
LossLoss of data units (segments, packets, cells, …) in finite of data units (segments, packets, cells, …) in finite buffersbuffersBlockingBlocking of connection or flow requests at the network edge of connection or flow requests at the network edge or in the transition across network boundariesor in the transition across network boundariesDelayDelay of data units (in individual buffers or endof data units (in individual buffers or end--toto--end)end)ThroughputThroughput (or (or goodputgoodput) at various architectural layers ) at various architectural layers (data link, network, transport, application)(data link, network, transport, application)Net user gainNet user gain or satisfaction (benefit less price paid for or satisfaction (benefit less price paid for resource utilization)resource utilization)Network or service provider’s Network or service provider’s revenuerevenue