The use of agents in the control of fixed and radio networks John Bigham Department of Electronic Engineering Queen Mary, University of London email: [email protected]
The use of agents in the control of fixed and radio networks
John Bigham
Department of Electronic Engineering
Queen Mary, University of London
email: [email protected]
Structure of talk Look at examples of the application of
agent technology to resource management in telecoms networks ATM networks AMPS networks 3G networks
Show some common threads architecture relation between autonomy &
communication suggest where we need to go further
What are agents ? - Definition used here:
Distributed software that exhibits features of: Autonomy: the ability to have control over its own
actions and states. An agent is able to make decisions and complete actions based on its internal representation of the world without direct intervention.
Social Ability: the ability to interact with other agents via some kind of communication language in a co-ordinated manner. Agents may co-operate or compete.
Reactivity: the ability to perceive changes in the environment and react timely and appropriately.
All of this depends on the code!
Types of agents
Intelligent Agents “Intelligent” pieces of software that communicate
through a standardised message-passing language (Agent Communication Language)
Like traditional signalling Mobile Agents
Agents that move – i.e. code that moves More reluctance to use mobile agents because of
the risk of viruses and rogue agents This talk will restrict itself to intelligent agents
controlling networks
For the real time systems described
All have to make some kind of decision in real time.
All need to be ready to make this decision All agents we have built can be viewed as
consisting of Reactive and Planning layers This applies to
user agents, resource management agents, negotiation agents….
5
Reactive and planning layers Reactive layer copes with immediate problems:
Equivalent to control plane actions Needs to be fast acting Scope for decision making limited View of environment limited
Otherwise large communications overhead Planning layers
Equivalent to management plane actions Aims to balance resources over a longer timescale
Works in the background Successful planning minimises need for reactive action and
reduces communication overheads But cannot be 100% correct
Can have multiple reactive layers Can have multiple planning layers
The boundary is grey
6
Kinds of Agent The different kinds of agent are determined by
their roles establishing an SLA, managing the radio
resource their ownership
e.g. SP or NP their location
e.g. at a RNC, at a point of presence Some roles are closely related to business structure
So agent logic depends on the business model used Some roles are related to the details of the resource
management So agent logic depends on the physical model
Different if ATM, IP, radio
Example: Controlling a fixed network
Intelligent resource management in ATM networks using a society of interacting agents – Impact project.
real time control of a real physical network Implemented on test bed at TeleDanmark in Århus Concentrated on setting up and charging for
connections Management of ATM was an issue at the time
much applicable to IP About using agents in a telecommunications
environment, not about research into agents per se.
8
Context : Value chain
customer user
Service provider
Network provider
Transport service
Higher-layer services$
Content provider
$
$
$ Value Value chainchain
Content can be Content can be paid for directly paid for directly
or via SPor via SP
Business model
Customer
NP 2
SP1
$
SP2 SP3 SPM
$$$
•Customer can access any Service Provider (SP)Customer can access any Service Provider (SP)
$
Modelling the business layers allows…
Service providers: to relate offer price to spot costs of bandwidth to build into the offer, information related to the
customer: tariff profile & volume discount Customers
to consider future use when accepting offer: e.g. if one SP offers a better volume discount, it
might be better to use that even if current price is higher.
11
Transport network of a service provider
Consist of bandwidth reservations through a core network
Consist of bandwidth reservations through a core network
RA
RA communicates with SPA to change resources for its source-
destination pair
Service provider agent communicates with NPA to change resources available
CA
PUA
Connection Agent gets bids from RAs for source-destination pair
Proxy User Agent requests connection
SPA
Control of ATM network
Architecture
Connection set-up PUA requests connection with parameters CA asks RAs for bid on that request RAs respond:
OK Wait while capacity is transferred Wait while connections are transferred NO
CA chooses (on some criterion) and connection notifies winning RA
Connection is set up. This is all at the reactive layer - FAST
These options do not exist with conventional
signalling
PC2
PC5
PC1PC4
PC3
42000
25000
44000
24000
Physical link
S-D pair 4-2 (3 VPs)
S-D pair 5-2
Example of the relaxation in the reactive layers
Free
Used
Biggestcapacity PC2
PC5
PC1PC4
PC3
25000
44000
24000
24000
20000
42000
First connection PC4-PC2 goes to VPwith the biggest capacity
Free capacity reduced
Allocation of connection to VPsmaximise minimum residualNow the biggest capacity
PC2
PC5
PC1PC4
PC3
22000
20000
22000
20000
24000
4000
20000
Total spare capacity 26000
250005000
20000
This is the load scenario that is used for the demonstrations
Adding more gives:
Now we wish to add another connection from PC5 to PC2 of 20,000 units
There is not enough capacity With no capacity transfer we get
rejection Capacity transfer between the two VPs
on the route between PC5 and PC2 allows the connection to be made.
PC2
PC5
PC1PC4
PC3
22000
20000
22000
20000
24000
4000
20000
Slow connection - allows capacity transfer
Total spare capacity 26000
5000
20000
Connection request occurs
Not enough capacity - connection blocked
But there is sufficient overall capacity
PC2
PC5
PC1PC4
PC3
Re-routing allows another connection
Not enough capacity for another connection PC5-PC2Re-route “blue” connectionNow capacity for the other connection
6000
20000
5000
20000
20000
24000
0
40000
4000
40000
But …. planning carried on the background might have been re-adjusting the capacities to avoid this reactive layer action Indeed it does Could demonstrate this……….
Architecture
Connection Allocation
SP Capacity Transfer
SP Connection Transfer
SP Tactical Planning
SP Strategic Planning
NP Strategic Planning
purely reactive
purely planning
capacity cannot be readily transferred
capacity is not immediately available
call allocation policies for RA
e.g. Selfish or Cooperative
Queen Mary – EPSRC support
Done within IMPACT
Can enrich the reactive layer by allowing time for optimisation
planning
Managing mobile networks
Initially on AMPS networks – more work in the literature there.
fixed set of frequencies, fixed shapes Extended to 3G Problem with mobile is that unlike fixed networks
the resources at access point (the air interface) are severely limited
More difficult problem Objectives
Optimise network capacity Maintain network coverage Guarantee Quality of Service
Conventional channel assignment : spacing out the frequencies
Channel Allocation Schemes Fixed Channel Assignment -
FCA Modified FCA schemes
channel borrowing with/without locking
load sharing Dynamic Channel Allocation
Centralised versions provide better results as they have a wider view, but:
Scalability problems Lack of robustness Heavy signalling load Not flexible
A
A
A
A
A
A
A
Current schemes are reactive
Reactive algorithms reach a point of saturation Signalling load becomes enormous Resources are there but cannot be
reached No real flexibility Investigated negotiation and co-operation
with other BSs to improve the efficiency of channel allocation.
Agent manages a cluster: Co-operation Strategy
B’
B’
B’
B’
A
A’
A’
A’
A’
A’
B’
B’
A’
Hot spot in one cluster For capacity, agent
managing cluster wants to negotiate moving frequency from neighbour
Into cells which will interfere with neighbouring cluster
Removing frequency from neighbour reduces its capacity
Negotiate with other clusters to see which can be borrowed with least effect Clusters can refuse Charge a price
Internal Architecture
Reactive Layer Responsible for fast response to incoming
traffic Implements a channel allocation algorithm
Local Planning Layer Responsible for channel re-assignment Takes into account efficiency of channel
usage Monitors efficiency of reactive layer
Co-operative Planning Layer Load balancing implemented by agent co-
operation Utility function to choose best region Market-based-control for management
handoffs
Co-operative planninglayer
Local planninglayer
Reactivelayer
Results
LoadLoad
Blocking probability)Blocking probability)
Stylised formStylised form
FCA
reactive layer
cooperative layer invoked
In this region, planning layer copes and co-operative layer is not invoked.
Challenges in 3G
New types of service Increasing numbers of
Service Providers Any-to-any marketplace Radio access technology
--- WCDMA all cells (of same type)
use same frequency interference is a
limiting factor needs accurate power
control cell breathing
Resource management is more complicated than in FDMA/TDMA systems
Work described has grown out of the SHUFFLE IST Project applying agent
technology to controlling 3G
this approach is sketched
Agents in SHUFFLE architecture lie in two kinds of plane
Negotiation plane holds agents concerned with making agreements between SPs & NPs on SLAs Primarily related to the business model
Each resource plane holds agents concerned with resource network management to meet the SLAs for each NP
UA
SPNA
NPNA1
NPRA1 NPRA2Handover & cell selection
QoS ManagementRadio Resource Management
NPNA2
Negotiation planeNPNA1 NPNA2
NPA1
NPA2
Negotiation plane
Resource planes
SPA
SPNA
Resource Agents
plane=agents with related roles
Three main groups of actors, each represented by agents:
Users
• Subscribe to service providers
• Make requests for connections
Service Providers• Brokers
• Offer services such as traffic information, videoconferencing
Network Operators• Own and control the physical networks
• Carry the connections requested by users
Connection Requests in a Free Market
UA
SPRA1
SPRA2
NPRAa
NPRAb
NPRAc
1. User makes a connection request
2. User’s UA contacts a SPRA to make the request
3. SPRA chooses a NPRA and relays the request
4. NPRA agrees to admit connection and informs SPRA
5. SPRA informs UA to which NPRA to connect
6. UA contacts NPRA - connection is set up
Use reputation
reinforcement learning
Layered control structure of resource agent
Co-operative planning layer Change antenna
radiation pattern Local planning layer
Prioritise Forcibly change QoS
Reactive layer Assignment to a Node
B Connection Admission
Control Possible change in
QoS
Local Planning Layer – acts on changing QOS (policy) within cell
Reactive Layer FAST
po
licy
po
licy
po
licy
assignment
OK?
set up connection
Exceptionhandler
reject
Sta
tus
rep
ort
ing
Yes
No
Modified request
No
YesCACOK?
Co-operative Planning Layer – changes radiation patterns between cells to meet wider goals
Request
Sta
tus r
ep
ort
ing
Sta
tus r
ep
ort
ing
Assignedrequest
Assignment: A hot spot occurs
Assignment Cost =f(distance from base station, power in direction of mobile)+ g(interference in base station if connected).
+ h(impact on non-assigned base stations if connected). ’
Assignment
lightly loaded
moderately loadedhighly loaded
Bid by each base station
Both normal and CBB graphs did not utilise social blocking. Theta is made to vary between 0.2 and above with no upper limit for CBB. Kuri’s prediction is calculated as 0.7918.
Simple Planner
Measure SIR statistics in a cell If minimum QoS threshold is not reached,
increase interference penalty for cell weight in g(interference).
requests neighbour cells to decrease their interference penalties (their ) if they want to
this tends to make new connections go to neighbours rather than overloaded cell
If maximum QoS threshold is exceeded, decrease interference penalty request neighbour cells to increase their interference
penalties MODIFYING POLICY AT THE LOWER LAYER
Other actions of the planner:if radio resources become scarce what are sensible resource plane actions ?
To meet SLA we can Prioritise calls ……. Reduce QoS ……..
All local planning actions Change radio configuration
Requires more cooperation
Behaviour of a conventional 3G network
Conventional 3G network
Cell structure blind to Call Traffic Changing
Low Flexibility
Dynamic reconfiguration
Dynamic reconfiguration
Dynamic reconfiguration
Reactive to Call Traffic Changing
Higher Flexibility Higher Capacity Low additional
cost Can use
Distributed Control
configuration achieved using
smart antenna technology + coordination
The basic idea of a smart antenna is to use antenna patterns that are not fixed but adapt to current radio conditions.
Smart antennas are usually categorised as either fully adaptive or switched-beam
Fully adaptive smart antenna
i n te r f e r e r
d e s i r e d u s e rVisualised as pointing a narrow beam in the direction of the wanted mobile and nulls in the directions of interferers
The complexity and cost of the adaptive beam former is still seen as a major disadvantage.
simple beam formers are used to make sector patterns
Do not require a fully adaptive antenna. Instead employ a fairly simple beam-former
The cooperative ‘intelligence’ is provided by the the resource agents.
Multiple beams from a sector combined to provide uniform and shaped coverage
A coverage snapshot achieved through cooperation between cells
reduced coverage
increased coverage
Scenario 1
Simulation ResultsScenario 2
Relative merits of AgentTel and fully adaptive smart antennas
Higher potential gain (X2 instead of X1.3)
Less complex and
cheaperSimpler – can be
mounted on the top of the
tower or uses fewer
connectors main points of failure
Beam forming easier
patented approach
Fully adaptiveAgentTel
How can it be achieved physically?
Tapered amplitude illumination provides
vertical pattern
Individual element excitation is fixed by a network within the antenna so only one RF input signal is required.
RF in
The simple linear array used in conventional base stations
Basic smart antenna
Amp&
Phasecontrol
Amp&
Phasecontrol
Amp&
Phasecontrol
Amp&
Phasecontrol
RF drive signal
A1 P1
A2 P2
A3 P3
A4 P4
Array of conventional base-station antennas gives the basis of a smart antenna
RESULTS:Scenario Descriptions: Scenario 2 (for example)
3000 traffic units in the whole area. 100 base stations, each has the capacity
to serve 36 traffic units 20% of traffic is uniformly distributed in
the whole area. Other 80% of traffic is distributed in 40
hot spots with normal distribution
Simulation Results
Generation
Sys
tem
Uplin
kC
apaci
ty
0 250 500 750 10001700
1800
1900
2000
2100
2200
2300
2400
2500
2600
2700
2800
Fixed (circular) pattern
First method (2.8)
Second method (2.9)
Scenario 1Scenario 2
shows benefit from shaping can be > 20%
faster convergence
slow
convergence
Allocation by service type possible
Gold
Silver
Bronze
Users Served:12% Gold17% Silver35% Bronze
Users Served:23% Gold29% Silver6% Bronze
Time
Real time change: base stations agree patterns by negotiation
Message type• Request for help (RFH)• Request for pattern (RFP)• Return from antenna agent (RFA)– Return from base station agent (RFB) • Request for commitment (RFC)• Acknowledgment of commitment (AOC)• Scheme expires (SE)• Cancel Transaction (CT)• Change Your Coverage (CYC)• Change Your Pattern (CYP)
Ask
Respond
Ask for commitment
Respond with
commitment
Perform change in pattern
Negotiation Process34 35 36
44 45 46 47
65 66
54 55 56
BS45 BS55BS46 BS66Ant45RFP(1,2,3,4,5)
RFA(3,4)
RFH(4)
RFH(3)
RFA(322)
Ant46 Ant55 Ant66
RFP(41,42,43)
RFA(41)
RFP(31,32,33)
RFA(32)
RFH(32)
RFP(321,322,323)
RFA(323)
RFB(323, price323)
RFA(33)
RFA(31)RFB(41, price41)
RFA(43)
RFA(42)
RFA(5)
RFA(321)RFB(32, price32+price323)
RFC(41)
AOC(41)
CYC(41)
CYP(4)
RFA(5)
CYP(41)
Ask
Respond
Ask for commitment
Respond with
commitment
Perform change in pattern
A RFH issued in two cases
When U>T, i.e. the observed utilisation U at a base station exceeds a chosen threshold T, or
during hypothetical reasoning in response to another RFH when helping would cause the utilisation threshold at the helping base station also to be exceeded so further requests have to be made. There is a HTK
Triggering on U>T could be formulated in terms of an interference constraint But not too bad a model if assume perfect power
control
If U>T then a reduction in coverage area is required
New hypothesised coverage shapes are generated from the current shape by removing locations.
Candidate locations for removal are the locations on the boundary of the
existing shape, and the next levels in These locations, are given a value
with respect to a base station that is a weighted sum of two selfish criteria and one social criterion
location
The value of a location to a BS is……
proportional to the profit at the location inversely proportional to a power of the distance. The capability of neighbouring base stations to help.
here the utilisation of the nearest immediately neighbouring base station.
Depends on the type of user…….. Also used to find the price for helping
' ( ( ))( )
( ) ( )z
kk U NN i
d iV i P i
Generating hypotheses
Locations are repeatedly removed, in a “randomised” way, until the threshold T is no longer exceeded.
This is repeated 5-10 times in the current implementation
to generate hypothesised patterns LOTS of detail missing!
Key aspects are the hypothesis generation Value functions Need to reason with patterns the physical system
can generate
Rotating antenna test tower
Not just simulation
Fixed transmitantenna
Antennapattern
measurementreceiver
stop simulator and connect a selected base station to real antenna is pattern generate what we expected!
Alan Dick & Co outdoor range
Summary Tried to show that distributed resource
management is feasible and effective mechanism for resource management in fixed and radio networks.
Shown three examples where Performance is improved Increased flexibility is created
Design is not a monolithic hierarchy, rather coordinated hierarchical pillars of control
Coordination between pillar layers increases a planning becomes more dominant
Each layer helps to find more resource
LoadLoad
Blocking probabilityBlocking probability
Localised reactive algorithms
Local planning layer
cooperative layer
In this region, planning layer copes and co-operative layer is not invoked.
global optimisation
reactive layer
Each layer creates components of policy applied in lower layers
Cooperative
Local planning
Reactive
time for response
policy prescribed & uses little knowledge of neighbours
creates policy of lower layer + asks neighbours to modify their policy (voluntary)
e.g. maxmin/fragmentation
e.g. CBB + reservation
try to reach mutually advantageous agreements on policy of lower layers (policy +meta policy)
SLAs & other business drivers
create aspect of policy
autonomy
amount of inter component communication
Is this very different from distributed control?
No much of it is similar to hierarchical and distributed
control Of course there are a variety of approaches for this
also, e.g. subsumption architecture Even ideas like reputation, tit for tat, tit-for-tat with
forgiveness,…… have close analogues in reinforcement learning ewma
So a large part of what is often called agent technology is not really different But still extremely useful!
Higher layers
However agent work has extended modelling to higher layers to include social interactions In competitive as well as cooperative environments
In competitive models some people argue against optimisation of an agent’s resource, but for finding e.g. Nash Equilibria A game theory approach Some negotiation mechanisms try to find equilibria
Others argue that optimisation is still relevant, but we need to use recursive modelling E.g. a NP should model a SP’s behaviour in order
for the NP to make optimal resource allocationsAnd vice versa
Example:Negotiating SLAs (including micro SLAs)
SPRA1
SPRA2
NPRAa
NPRAb
NPRAc
SPRA chooses a NPRA
SPNA1
SPNA2
NPNAa
NPNAb
NPNAc
Model each other and use these models in the individual optimisations laurissa
Chose not to model the NPRAs, just to select “optimal” – but reward and punish over time
n.b.
Does the agent metaphor give advantages?
Yes It works down through all the levels of the
business Including a competitive environment Supports a distributed mix of reactive and
planning elements Allows the modeller to place their components
into context An increasing number of tools available
But it is only a very general idea
other technologies beginning to offering similar infrastructure capabilities
and these are becoming ubiquitous Some based on client server and peer to
peer ideas developed for the internet XML, SOAP messages….ACL JavaSpaces ….. Distributed BB systems Servers, EJB … “business” agents Jini……
In fact agent systems may well be built on this technology, rather than on “agent platforms”
But people working with these technologies will face the same problems that researchers in agent technologies have been looking at
So concepts supporting communication when there is not a global
unifying ontology adjustable autonomy negotiation strategies property/concept/code re-use when
functionality has to change …….
Are key in making the system work
Can be implemented/installed bottom up
Start of with the reactive layer
Add local planning functionality
Add cooperation and integration with the business layer
Plan to build a real system at Athen’s new airportSLAs for high QoS customers (i.e. high profit) customers
Tracking these customers around the airportEnsuring emergency and priority users are served
Without starvation to all others– which can itself cause panic
It is becoming real!
But just as we feel we are understanding one area…..
new pressing requirements include coping with: Flexible business models and rapid provision of
new services Operator mistakes Fraud a daily reality
Malicious attacks from insiders and outsiders Increased vulnerability as moves to IP
Voice->IP , high bw unsecure connections-> DOS Need to manage interactions between
“safeguard” agents managing the management layers and resource agents managing the resource layers a very difficult problem…… help appreciated!