Evolutionary Game Dynamics. Classic Game Theory: Selfish players are involved in a game and each of them has to decide between different strategies in.

Evolutionary Game Dynamics

• Classic Game Theory:

Selfish players are involved in a game and each of them has to decide between different strategies in order to maximize a payoff, which depends on the strategies of all players

• Evolutionary game theory:

Entire groups of players are involved in a game. Each group is programmed to use some strategy. Strategies with high payoff, fitness, spread through the overall population. The payoff of a player depends on the strategies of the rest of the players and hence on the frequency of each strategy within the overall population. This yields a feedback loop whose dynamics is the object of evolutionary game theory

Game Theory BackgroundConsider a game with:• Two players I, J• Strategy spaces • Payoff matrices A, B nxm. (resp. ) denotes the payoff of player I

(resp. J) when I uses Strat(I) and J uses Strat(J)

• Mixed strategy: Player I Player J the probability of player I using the probability of player J using

)e,(eStrat(I) n1 )f,(fStrat(J) n1

ija ijbie jf

nT

n Sxx ),,( 1 xm

Tn Syy ),,( 1 y

ixjfjy

ie

}1,0|),,{(1

1

n

iiinn xxxxS }1,0|),,{(

11

m

jjjmm yyyyS

• The expected payoff for player I when he uses x and J uses y is

• (x,y) is a Nash Equilibrium, (NE) if:

• A game is symmetric if:

Strat(I)=Strat(J) and

• is a symmetric NE if

AyTx

n

n

Szzxyx

Szyxyz

BB

and AATT

TT

TBA nSx

nSzxxyz AA TT

Replicator Dynamics

• Population of n types• denotes the frequency of type i• State of the population• Assume that are differentiable functions of time t• Individuals encounter randomly and engage in a symmetric

game with payoff matrix A• Expected payoff for an individual of type i:

• Average payoff in the population state x:

ix

nT

n Sxx ),,( 1 x

ix

ninii xaxa 11)A( x

xx AT

Replicator Equation

iii xxx

log

nii

iii

xx

nixx

)A()A(for and 0 :0 i.e. pointsRest

),,1( )A)A((

1 xx

xxx T

Per capita growth:

Folk theorem of evolutionary game theory

• If z is a symmetric NE then it is a rest point

• If z is a strict symmetric NE then it is asymptotically stable

• If rest point z is the limit of an interior orbit then it is a symmetric NE

• If rest point z is stable then it is a symmetric NE

stability asymptotic stability

Existence of NE

• Remarks:

1. Every interior rest point is a NE

2. There exists at most one interior rest point, and one rest point for each pure strategy

• Theorem:

Every game has at least one NE nSx

Evolutionary Stable Strategy (ESS)

• Types of population correspond to mixed strategies

i=(1,…,N)

• Average payoff for an individual type p(i) against an individual of type p(j):

• describes the type frequency in the population

• Average strategy within the population:

• Replicator equation:

nS)(p i

Ap(j)p(i)T

nSx)p()p( ixix

))Ap())p()(p(( xx Tii ixx

• ESS is a strategy which if adopted by a population of players, no alternative strategy that is initially rare can invade

• Mutant Strategy: Incumbent Strategy:

• For a symmetric 2-player game: where x is the frequency of the invader • Definition: An ESS in a symmetric 2-player game is a mixed strategy if:

1. It is a NE, i.e. and

2. If then

nSp nS^

p

)]pAppAp)(-(1-Ap)p-App()[1(^^^^

TTTT xxxxx

)p,p(^^

^^^

pAppAp TT ^^^

pAppAp TT AppApp^TT

Dynamics of two strategy games• Players I,J

• Payoff matrix

• State of population• Replicator equation:

• This equation admits only three outcomes

dc

baA

)))((1(

))())((1(

1111

21111

dbxbdcaxxx

AxAxxxx

)e,(eStrat 21

2121 1 ),( xxxx x

Hawk-Dove (HD) Game

• Large population of animals

• Occasionally two animals engage in a competition over a pray

• Two strategies:

1. Hawk: Always fight for pray. Either gets the whole pray or gets injured.

2. Dove: Never fights for pray. Withdraws from Hawk, splits pray with Dove.

• Payoff matrix:

V/20

VC/2-V/2 AH

D

H D

• V/2-C/2 0

u(H,H) u(D,H) (H,H) is the unique symmetric NE and since u(D,D)<u(H,D) (H,H) is the ESS

• V/2-C/2<0

We use the replicator equation which here is:

Rest points x=1, x=0, x=V/C. Stable interior rest point, the only symmetric NE. Stable coexistence of both types with x=(V/C,1-V/C) is the ESS

)c-V)(1(21

1111 xxxx

Prisoners dilemma game

• Same assumptions

• Payoff matrix c>a>d>b

• Replicator equation:

• No interior equilibrium, one frequency converges to 0, the other to 1.In this example Fink is the ESS

dc

baA

0)( with

)))((1(

1

1111

dbxbdca

dbxbdcaxxx

Coordination Game• Same assumptions

• Payoff matrix a>c , d>b

• Like BoS but in this case both players prefer one outcome over the other

• Interior rest point exists and is unstable. The two pure strategies are also NE and both are attracting, bistability Two ESS.

dc

baA

)()( bdcabdx

Evolutionary dynamics in competitive routing

Non-cooperative game with finite number of players and infinite space strategy

• Model

Network {V,L,f} Users i=1,…,N

Links l=1,…,M user I sends fixed amount Λi from s(i) to t(i)

User i rate on link l:

Cost payed from user i for link l:

Cost payed from user i for all links used by it

• Each user has to determine the way his traffic is split in order to minimize his cost

L),(ff ll lllll ba )(f

il

)()( lllill

il baC

)()(1

lll

M

l

il

i baC

• Nash Equilibrium (NE)

λ is a Nash equilibrium if:

i=1,…,N and strategy of user i

• Potential game

The game admits a potential function P where:

• Consequences:

1. NE exists

2. NE is unique

• Asynchronous best response updates converge to the unique NE

),()( iiii CC i

l

L

ll

N

i

N

i

il

il

L

l

l ba

11 1

22

1

)()(2

)P(

Replicator Dynamics

• User i rate Population i global mass of infinitesimal users

• Proportion of population i that uses strategy m:

• Cost of users in i that use link l:

• The marginal cost of users in i that use link l:

• The mean marginal population cost

iii

im

imx

N

il

iill

iill

il bxaxxC

1

))(()(

lii

lli

N

i

illi

l

i

li

l bxaxaC

xf

1

)(

M

l

il

il

i xfxxF1

)()(

• Replicator Equation:

• If the NE is:

and it is an interior rest point the replicator dynamics

• The NE is asymptotically stable and any solution trajectory starting from an internal initial condition converges to the NE

• So in ESS exists a stable coexistence of the equilibrium flow for all links and all populations

)]()([ li

lii

lil xfxFxx

m

l l

lil

aaxli

1

*

11)( ,

lbbl

Possible Application Load Balancing

• Routing protocol with load dependent metric

• Able to perform load balancing across unequal cost links, sending more packets to lower metric routes and vice versa

• Replicator dynamic analysis with marginal costs can be implemented in order to find an algorithm that performs optimal load balancing

• The algorithm can be executed in routers firmware to ensure optimal routing with load balancing in real time

An application of evolutionary game theory in TCP congestion avoidance

Congestion Control Protocols

• Various TCP versions with different congestion avoidance algorithms

• Aggressiveness varies from version to version

• Examples:

TCP New-Reno: approximately AIMD

HSTCP: approximately AIMD

Scalable TCP: MIMD• AIMD (Additive Increase Multiplicative

decrease) algorithm adjusts the flow to the available capacity by increasing the window size by a packets every RTT and when it detects congestion it decreases the window size to b times it’s value.

For TCP New-Reno a=1, b=1/2

The Model

• Two populations (i=1,2) of AIMD TCP connections with αi, βi αi linear increase rate βi multiplicative decrease rate xi transmission rate at time tAssumptions:i) RTT same for all connections ii) Light traffic (in terms of connections, not workload) iii) Losses occur whenever sum of rates reaches capacity C: x1(t)

+x2(t)=Civ) Losses are synchronized• TCP connection i is more aggressive than a connection j if αi ≥ αj

and βi ≥ βj

• Population dynamics:

• Let xn and yn be the transmission rates of connection i and j, respectively, just before a loss occurs.

• We have xn +yn = C.• Just after the loss, the rates are β1xn and β2yn. The time, measured in RTTs, it takes

to reach again C is:

which yields the difference equation:

where

the solution is

Where

1 2

1 2

n nn

C x yT

a a

1 21 1 1

1 2n n n n

a Cx x a T qx

a a

1 2 2 1

1 2

a aq

a a

1 20

1 2

1( )

1

nn

n

a C qx q x

a a q

: 1 ii

• Since |q|<1 For

• Average throughput of connection 1

• Average loss rate (same for connections 1 and 2)

• In wireless networks energy is a costly resource, so loss rate causes energy inefficiency

Fitness for player i: where λ is a tradeoff parameter

n

1 2 1 2 1 2

1 2 1 2 2 1 1 2 2 1

1. 1

a C a C Cx T

a a q a a a a

1

1 1 2

1 2 2 1

1

2hp

aT C

a a

1 2

1 2

1 1( )a a

RT C

ii hpJ T R

Evolutionary Stable Strategy• Aggressive (Hawk) (αH,βH), Not aggressive (Dove) (αD,βD)• For all λ: J(H,H)<J(D,D), J(D,H)<J(D,D), J(H,D) larger than all• For small λ: J(H,H)>=J(D,H) so (H,H) is the ESS• For large λ: J(H,H)<J(D,H) Mixed HD structure• Two versions of TCP will coexist, x is the frequency of the aggressive

version

• η1,η2 positive so equilibrium point x* decrease linearly on λ

• Conclusion : Applications that are more sensitive to losses would be

less aggressive at ESS

* 1 2

3

( )x

1 2 1 2 1 2 2 11 2 3

1 2 2 1 1 2

1 1 ( )1 1( ) , ( ), ( ) ,

2 4 2 2 ( ) ( )

aC C

C

• Classic replicator dynamics:

The fitness acquired at time t has an impact on the rate of growth of thepopulation size that uses it instantly.

• Replicator dynamics with delay (More realistic model)

The fitness acquired at time t will impact the rate of growth τ time later

• K positive constant, accelerates the speed of convergence to the ESS• The delay τ is related to the time scale for example of switching from the use of one protocol to another or upgrading a

protocol

( ) ( ) ( , ( )) ( ) ( , ( )) ( ) ( ) ( , ) ( ) ( , ) ( ).

( ) ( )i i j i j j kj j j k

x t x t K J i x t x t J j x t x t K x t J i j x t J i k x t

( ) ( ) ( , ( )) ( ) ( , ( ) ( ) ( ) ( , ) ( ) ( , ) ( ).

( ) ( )i i j i j j kj j jk

x t x t K J i x t x t J j x t x t K x t J i j x t J i k x t

• Replicator dynamics with delay yield to the mixed ESS if:

τK<θ

where

• For the evolutionary protocol to converge to an ESS the product τK must have a constant value and smaller than θ

• For large delay τ, the fitness of a strategy acquired at t-τ should have a smaller impact on the rate of growth and hence slower convergence to the mixed ESS

1 2

1 21 2

( , ) ( , ), ( , ) ( , )

δ=δ δ , θ=2δ δ

J B A J A A J A B J B B

• Mixed ESS (H,D)=(3/4,1/4)

• Top figure: Impact of gain parameter K (τ=1 ).While τK<θ x(t) converges to the ESS. For higher values of K the convergence is faster. For K=15, τK<θ oscillates around the ESS

• Down Figure: Impact of delay τ

K=1. While τK<θ x(t) converges to the ESS.For τ=15 the system is unstable. It oscillates around the ESS

Evolutionary Games and Population Dynamics Maintenance of cooperation on public goods games

Evolutionary Games and Population Dynamics: Maintenance of cooperation on public goods games

The problem of cooperation investigated by means of the game theoretical models of:1)Prisoners Dilemma for pair wise interactions2)Public Goods Games for groups or interacting individuals

Public Goods Games:Subjects(players) secretly choose how many of their private tokens to put into a public pot. Each subject keeps the tokens they do not contribute +even split of the tokens in the pot-Contributors:”cooperators “

-Those who do not contribute:”defectors” or “free riders”

Payoffs: PD(k)=rkc/N PC(k)= PD(k)-c

N-size of the group, k-number of cooperators, c-cost,

r-multiplication factor

for N=2 it can be mapped onto Prisoners Dilemma

Public Good Games: Basic Idea

•When population density varies=>Effective group size S of the public goods varies

• Small population densities => Small S (and vice versa)

•Large population densities favors defection Defection leads to a decrease in population density and hence a decrease in S which in turn, favors cooperation

GAME DYNAMICS AND POPULATION DYNAMICS

Fraction of cooperators : x

Fraction of defectors :y

Fraction of empty space :z=1-x-y

Normalized population density x+y

(0=extinction,1=maximal population density)fc ,fD average payoffs of cooperators and defectors respectively at any point of time

d- constant death rate and birth rate proportional to average payoffs and zPopulation Dynamic Model:

(Natural extension of

replicator dynamics)

The average fitness of defectors and cooperators can be given:

where

( ),

( ),

( ) ( )

.

.

. . .

C

D

C D

x x zf d

y y zf d

z x y x y d z xf yf

1(1 )

1 (1 )

( ),

N

D

C D

x zf r

z N z

f f F z

1 1( ) 1 ( 1)

1

NN r z

F z r zN z

Game Dynamics and population Dynamics Homogenous Populations

Homogenous Populations

Population Dynamics of cooperators in absence of defectors for different death rates d

Cooperators unable to survive for

and the only stable equilibrium is x=0(exstinction)

for d<dmax two interior equilibria appear:One stable branch at higher x (solid line)One unstable branch at lower x(dashed line)

Cooperators thrive at sufficiently high densities but go extinct otherwise

1max ( 1)( 1)

N

Nd d r N N

Game Dynamics and population Dynamics Heterogenous Populations

Cooperators survive even when facing exploitation by defectors. The relative proportion of cooperators is f=x/(x+y) and we have:

There are four different dynamic systems depending on the existence of interior equilibrium points. Equilibrium points are of the form where solves and

(a) Q exists and is a stable focus. Cooperators and defectors

can coexist . Trajectories spiral towards Q , except for

low population densities or abundant defection, in which

case population goes extinct.

2

1

(1 ) ( ),(1 )

(1 )( ( 1)(1 ) )

. ..

.N

x y y xf zf f F z

z

z z fz r z d

( , )Q z f

z

( ) 0F z

1( ( 1)(1 ))

Nf d z r z

Game Dynamics and population Dynamics Heterogenous Populations

(b) Increasing death rates push the unstable Q upwards until it disappears . This leads to persistence of cooperators and elimination of defectors. The population vanishes for low initial population densities or abundant defection

(c)For smaller multiplication factors r , Q becomes unstable and the population always goes extinct. Trajectories originating in the vicinity of Q approach z=1 in an oscillatory manner with increasing amplitude

Game Dynamics and population Dynamics Heterogenous Populations

(d) When eliminating the unstable focus Q by increasing d, the population continues to go extinct but the oscillatory dynamics has disappeared

Possible application in P2P networks

• P2P network of N nodes

• Nodes controlled by selfish users

• Cooperators: Download and caching files

• Defectors: Only download, not caching files

• Large population densities favors defection leading to a decrease in population density and hence a decrease in effective interaction group which in turn favors cooperation

• For intermediate frequencies of populations we can have a stable coexistence of both types or dominance of cooperators

Possible application in mobile ad-hoc networks

• Mobile ad-hoc network with N nodes• Each node supports a multihop routing algorithm where nodes

function as relays• Nodes are equipped with non-replenishable energy sources• Selfish behavior: conserve power, don't act as relays• Cooperative behavior: act as relays• Large population densities favors defection leading to a decrease in

population density and hence a decrease in effective interaction group which in turn favors cooperation

• For intermediate frequencies of populations we can have a stable coexistence of both types or dominance of cooperators

Congestion control for applications sensitive to delay

• Applications sensitive to delay prefer UDP over TCP• The congestion control is implemented by the applications• Dynamic rate adjustment by the application during the session by

degrading for example the amount of redundancy• When a congestion event is detected the rate is reduced• Aggressive sessions may not reduce the rate at the first congestion

event waiting to see if the congestion has been eliminated through the other sessions reducing the rate

• Or aggressive sessions may reduce the rate at the first congestion event but the reduction is smaller than the one that takes place in the non-aggressive sessions

• Evolutionary game theory can predict the dominance or coexistence of the different protocols

References

• J. Hofbauer, K. Sigmund. Evolutionary Game Dynamics. Bulletin (New Series) of the American Mathematical Society. Volume 40, Number 4, Pages 479-519

• R. Cressman. Evolutionary Dynamics and extensive form games. The MIT Press

• E. Altman, Y. Hayel, H. Kameda. Evolutionary dynamics and potential games in non-cooperative routing. INRIA Sophia Antipolis.

• E. Altman, R. ElAzouzi, Y. Hayel, H. Tembine. An Evolutionary Game approach for the design of congestion control protocols in wireless networks. INRIA Sophia Antipolis.

• C. Hauert, M. Holmes, M. Doebeli. Evolutionary games and population dynamics: maintenance of cooperation in public good games. Proc. R. Soc. B 2006 273, 2565-2571