An Introduction to Modeling Techniques and Levels, and ... · Modeling the Immune System – W8 An Introduction to Modeling Techniques and Levels, and Ordinary Differential Equations.

Modeling the Immune System – W8

An Introduction to Modeling Techniques and Levels, and

Ordinary Differential Equations

Modeling: why and what ?• The grand challenge:

Constitute a computer model that represents all we know about the immune system, which can be – stored– searched– compared – and analyzed

• We are not yet there, but you may help one step

http://research.microsoft.com/towards2020science/

Modeling

Individual behaviorsand local interactions

Global structuresand collective

decisions

• Modeling to understand microscopic to macroscopic transformation

• Modeling as interface to artificial systems

Ideas forartificialsystems

The Crucial Role of Modeling

Goal of the Modeling Module• Understand what a model • Master the concepts underlying the most

frequently used models• Know how to simulate them• Learn by example how they were used• Get a feeling of the limitations of the

current modeling techniques and their experimental validation

Contents of the Modeling Module• Introduction

– Swarm-intelligent systems– Modeling levels and techniques– Examples from animal and robotic populations

• ODE models (Ordinary Differential Equations)– Deterministic– Based on Rob de Boer’s lecture notes– Apply it to Annotated Paper DePillis et al

Contents of the Modeling Module• Stochastic macroscopic models

– Markov chain models – random– Based on Jean-Yves Le Boudec’s lecture notes– Apply it to Annotate Paper Fraser et al

• Stochastic microscopic models – Cellular Automata as example– Based on research papers

Outline of This Lecture

• Swarm-intelligent systems– Mechanisms– Comparison between natural societies and IS– Modeling levels and techniques– Comparison between robotic societies and IS

• An introduction to ODE models– Logistic equation– Steady state analysis– Phase plots

Is the Immune System a Swarm-Intelligent System?

(Examples from Social Insects)

A Typical SI System: Ant Colonies• Natural system

– social insect societies• Unit coordination

– distributed control + environmental template– individual autonomy– self-organization

• Communication– direct local communication (peer-to-peer)– indirect communication through signs in the

environment (stigmergy) • Scalability

– A few to milions of units• Robustness

– redundancy– balance exploitation/exploration– individual simplicity

Immune System: a SI System?• Natural system

– immune system• Unit coordination

– distributed control + environmental template (e.g. chemical gradients, different organs)

– cell autonomy (e.g. decisional, perception-to action, energetic)

– self-organization• Communication

– direct local communication: cell-to-cell contact– indirect communication: e.g., cytokines

• Scalability– Up to hundreds of billions of units. Lower

bound?• Robustness

– redundancy– balance exploitation/exploration (e.g., antibody

production)– individual simplicity (in comparison to the

whole system level)

Neutrophils

Mast cellsMacrophages

Cells:

B-Cells

T-CellsDC-Cells

Molecules:- Acute phase proteins- Cytokines- Chemokines- Antibodies- …

B

T

…

Two Key Ideas in Swarm-Intelligent Systems

1. Self-Organization

2. Stigmergy

Self-Organization

• Set of dynamical mechanisms whereby structure appears at the global level as the result of interactions among lower-level components

• The rules specifying the interactions among the system's constituent units are executed on the basis of purely local information, without reference to the global pattern, which is an emergent property of the system rather than a property imposed upon the system by an external ordering influence

Basic Ingredients of Self-Organization

• Multiple interactions• Randomness• Positive feedback

– E.g., recruitment of other cells• Negative feedback

– E.g., limited number of available cells

Characteristics of Natural Self-Organized Systems

• Creation of spatio-temporal structures– E.g., AC: foraging trails, nest architectures, clusters of objects, ...;

IS?

• Multistability(i.e., possible co-existence of several stable states)– E.g., ants exploit only one of two identical food sources, build a

cluster in one of the many possible locations, ...; IS?

• Existence of bifurcations when some parameters change– E.g., termites move from a non-coordinated to a coordinated phase

only if their density is higher than a threshold value; IS?

StigmergyGrassé P. P., 1959

• “La coordination des taches, la regulation des constructions ne dependent pas directement des oeuvriers, mais des constructions elles-memes. L’ouvrier ne dirige pas son travail, il est guidé par lui. C’est à cette stimulation d’un type particulier que nous donnonsle nom du STIGMERGIE (stigma, piqure; ergon, travail, oeuvre = oeuvre stimulante).”

• [“The coordination of tasks and the regulation of constructions does not depend directly on the workers, but on the constructions themselves. The worker does not direct his work, but is guided by it. It is to this special form of stimulation that we give the name STIGMERGY (stigma, sting; ergon, work, product of labor = stimulating product of labor).”]

It defines a class of mechanisms exploited by social insects to coordinate and control their activity via indirect interactions.

Stigmergic mechanisms can be classified in two different categories: • quantitative (or continuous) stigmergy• qualitative (or discrete) stigmergy

Stimulus

Answer

S1

R1

S2

R2

S3

R3

time

S 4

R4

S 5

R5

Stop

Definition

Stigmergy

→ probably the most widely used in the IS

An Example from Ant Colonies

Nest

Food Source

Symmetric Bridge Experiment(Deneubourg, 1989)

• Stigmergy?

• Self-organization?– Spatio-temporal structures?– Multi-stability?– System bifurcation?

• Ingredients of SO:• Multiple interactions?• Randomness?• Positive feedback?• Negative feedback?

Bridge with two Branchesof the Same LengthExperimental Results

= ( k + A i ) n

PA( k + A i ) n + ( k + B i ) n

= 1 - PB

A i : number of ants having chosen branch A

B i : number of ants having chosen branch B

Microscopic Model(Deneubourg 1990)

Probabilistic choice of an agent at the bridge’s bifurcation points; Montecarlo simulation

PA and PB : probability for the ant i+1 to pick up the branch A or B respectively

n (model parameter): degree of nonlinearityk (model parameter): degree of attraction of a unmarked branch

Ai

Aii PifB

PifBB

≤>+

=+ δδ1[1

Ai

Aii PifA

PifAA

>≤+

=+ δδ1[1

iBA ii =+ δ = uniform random variable on [0,1]

0

0,25

0,5

0,75

1

1 10 100

k=1 n=5

k=1 n=2

k=1 n=1

P APPAA

Parameters of the Choice Function

• The higher is n and the faster is the selection of one of the branches (sharper curve); n high corresponds to high exploitation

• The greater k, the higher the attractivity of a unmarked branch and therefore the higher is the probability of agents of making random choices (i.e. not based on pheromones concentration deposited by other ants); k high corresponds to high exploration

0

0,25

0,5

0,75

1

1 10 100

k=10 n=5

k=10 n=2

k=10 n=1

P APPAA

Number of ants havingchosen branch A

Number of ants havingchosen branch A

k highk low

Bridge with two Branchesof the Same Length

Model vs. Experiments

40

50

60

70

80

90

100

0 500 1000 1500 200040

50

60

70

80

90

100

0 500 1000 1500 2000

experiment

model

Total number of ants having traversed the bridge

% o

f ant

pas

sage

s on

the

dom

inan

t bra

nch

Parameters that fit experimental data:n = 2k = 20

An Example of Varying the Environmental Conditions

• Two branches (A and B) differing in their length (length ratio r) connect nest and food source

• Test for the optimization capabilities of ants

© J.-L. Deneubourg

Asymmetric Bridge Experiment(Deneubourg, 1989)

Food Source

Nest

• The previous simple microscopic model based on Montecarlo techniques should be modified to take into account distance/traveling time corresponding to the geometry of the asymmetric bridge

• This is an alternative option at the microscopic level: multi-agent system, more computationally intensive (agent trajectory simulated but no embodiment).

Asymmetric Bridge –Microscopic Modeling

© Marco Dorigo, ULB, 1999

An Example from the IS?

• Idea of the bridge experiment: – Selected lab experiment (i.e. in vitro) vs. real

environment (i.e. in vivo)• What’s the “ bridge experiment” for the IS?

– Has been done?– What are the difference in experimental

conditions? – What are the technical challenges in gathering

data?

Similarities between an Ant Colony and the Immune System

• Swarm-intelligent system (self-organization, stigmergy, distributed control, individual autonomy etc.)

• Multi-caste, inter-individual diversity comparable within the same caste

• Individuals endowed with sensors, actuators, information processing capabilities

• Mobility

– IS: more “hierarchical” structure (e.g., 3 barrier lines), more morphologically different cells (hardware specialization)

– AC: mostly passive environment – open loop or relatively slow closed loop between environment and collective system (e.g. evolution over generations); IS: active environment (body), relatively fast closed-loop

– IS: individual intelligence level low (no learning? no internal representations?) AC: navigation, learning, etc.

– IS: very high numbers; AC: high numbers– IS: population reproduction (birth/death) crucial; AC: most

of studies (particularly the modeling one) during life span– Experimental techniques different; data for modeling also

differentConsequences at the modeling level (tools, resulting models)?

Differences between an Ant Colony and the Immune System

Modeling Levels and Techniques

(Examples from Swarm Robotics)

• Even if individual node control deterministic, the interaction with the environment/teammates in the real-world is noisy and barely predictable → probabilistic models

• Swarm-intelligent systems exploit self-organization as main coordination mechanism: among key ingredients of self-organization (see week 1 and 2) there is randomness, an ingredient at the core of the exploration-exploitation balance of these systems. Coordinated collective behavior base on self-organization is statistically predictable using appropriate probabilisticmodels!

Rationale for Probabilistic Modelling –Swarm-Intelligent Real-Time Systems

• Different levels, different system parameterscertain low-level parameters (e.g., body morphology, sensor characteristics, trajectories ) can be captured in an explicit and more accurate way with microscopic models; others (e.g., birth numbers, death numbers, number of cells of a given species) can be captured also at higher level

• Different levels, different generalization powerthe higher the abstraction, the better the generalizing power (e.g., other experimental constraints, other class of experiments, outline common fundamental blocks)

• Different levels, different computational costquantitatively accurate models have often to be solved numerically: the higher the abstraction, the lower the computational cost

Benefits of Multi-Level Modelling

Modeling Levels

Target swarm system: multiple units

ModelsMacroscopic models: single representation for the swarm

Microscopic models: unit represented individually • S&A-based simulator• Multi-agent system

• Point simulator• …

• Rate equations• Master equations

• Stochastic reaction networks• …

Abs

trac

tion

Expe

rim

enta

l tim

e

Multi-Level Modeling Methodology

Ss SaSs SaSs Sa

∑ ∑′ ′

′−′′=n n

nnn tNtnnWtNtnnW

dttdN )(),|()(),|()(

Ss Sa

Target system (physical reality): info on controller, S&A, communication, morphology and environmental features

Microscopic – Module-based: intra-robot (e.g., S&A, transceiver) and environment (e.g., physics) details reproduced faithfully

Microscopic – Agent-based: multi-agent models, only relevant robot features captured, 1 agent = 1 robot

Macroscopic: rate equations, mean field approach, whole swarm

Com

mon

met

rics

Microscopic Level– Individual-centered (e.g., cell-centered in IS); accurate at the

individual level; multi-agent model– One can ask questions about diversity, including each individual

different from each other– States are representing for instance: current behavior, position,

orientation, etc.– Might still be represented as Markov process in continuous time

but state space might not be countable (e.g., continuous spatialspace); of course, in simulation spatiotemporal discretizationshappen according to machine resolution/simulation settings which results in anyway extremely large number of states

– Usually computational rather than mathematical models– Computational time scales at least linearly with the number of

units– Multiple runs needed for statistical significance on the

population average behavior

Macroscopic Level– Population-centered (e.g., IS-centered, for a specific aspect);

accurate at the population level; single abstraction model– One can ask questions about diversity on pre-establishing castes

at the price of major extension of models (e.g., new set of ODE for each caste)

– States are representing for instance: number of current individuals of a given type, in a given behavior, etc.

– Markov process in continuous time, countable space – Usually mathematical, numerically solvable models (simplified

models can be analytically tractable)– Computational time is independent of the number of units in the

system– Depending on the implementation, population average behavior

can be predicted (rate equations, ODE) or absolute numbers (stochastic network models)

Modeling Assumptions

Modeling Assumptions 1. Probabilistic FSM description for environment and

multi-agent system; arbitrary state granularity2. Semi-Markovian properties: the system future state

is a function of the current state (and possibly of the amount of time spent in it)

3. Nonspatial metrics for swarm performance4. Mean field approach (well-mixed system): mean

spatial distribution of agents over multiple runs assumed to be homogeneous, as they were randomly hopping on the arena

5. Linear superposition of object/robot detection areas(sparse object/robot distribution; no overcrowding, no detection areas overlapped)

Assumptions 1 and 2• We work with states

pin poutTx

Sx

Sx: state xTx: duration of state xpin, pout: probabilities to entry and leave state x

• States can characterize both robots and the environment

Assumptions 3,4,5• Trajectories and object location do not count -> 1D

Montecarlo simulation• N objects of type i -> N x (prob. to encounter i)

O1O1

O2

O2O2

2D physical space 1D probability space

O2

O1

Free space

From Microscopic to Macroscopic Models:

Theoretical Background

Microscopic Level

∑ ∑′ ′

∆+′−′∆+=

−∆+=∆

n ntnptnttnptnptnttnp

tnpttnptnp),(),|,(),'(),|,(

),(),(),(

p(n,t) = probability of an agent to be in the state n at time tIf Markov properties fulfilled (neglect distributions):

inflow outflow

Probability the agent was in a given state n’

Transition probability

ttnttnptnnW

t ∆′∆+

=′→∆

),|,(lim);|(

0Transition rate

Sum over all possible states n’ the agent can be in

Macroscopic Level

∑ ∑′ ′

′ ′−′=n n

nnn tNtnnWtNtnnW

dttdN )(),|()(),|()( Rate Equation

(time-continuous)

inflow outflown, n’ = states of the agents (all possible states at each instant)Nn = average fraction (or absolute number) of agents in state n at time tW = transition rates (linear, nonlinear);

∑ ∑′ ′

′ ′−′+=+n n

nnnn kTNkTnnTWkTNkTnnTWkTNTkN )(),|()(),|()())1((

Time-discrete version. k = iteration index, T time step (often left out)

Left and right side of the equation: averaging over the total number of agents, dividing by ∆t, limit ∆t → 0; neglect distributions of the stochastic variables (mean field approach):

Time Discretization: The Engineering Receipt

1. Assess what’s the time resolution needed for your swarm performance metrics

2. Consider unit-to-unit interaction essence: probabilistic/deterministic, asynchronicity role, …

3. Choose whenever possible the most computationally efficient model: time-discrete less computationally expensive than emulation of continuity (e.g. Runge-Kutta, etc.); in our systems/metrics there is no evidence of decreased prediction accuracy

4. Advantage of time-discrete models: a single common sampling rate can be defined among different abstraction levelsOften not even a tradeoff: just use the appropriate

instrument for the appropriate problem!

Time-discrete vs. time-continuous models:

Model Parameters• Gray-box approach: a priori information about the system

is exploited – multi-agent system– # of agents– technological and environmental constraints

• Models should not only explain but have also predictive power: the mapping between model parameters and design choices should be as explicit as possible (the higher the abstraction level the more difficult it is)

• Two types of parameters for micro-AB and macro: – State durations (e.g., interaction times with objects)– State-to-state transition probabilities (e.g., encountering

probabilities)

Linear Example:Wandering and Obstacle

Avoidance

A Simple Linear Model

© Nikolaus Correll 2006

Example: search (moving forwards) and obstacle avoidance

A simple Example

Nonspatiality& microscopiccharacterizationDeterministic

robot’s flowchart

Search Avoidance

Start

Obstacle?YN

Search Avoid., τa

Start

Obstacle?pa

ps

1-pa

Probabilistic agent’s flowchart

Ss Sa

pa

τa

ps

PFSM (Markov Chain)

Linear Model – Probabilistic Delay

Search Avoidance, Ta

Ta = mean obstacle avoidance durationpa = probability of moving to obstacle av.ps = probability of resuming searchNs = average # robots in searchNa= average # robots in obstacle avoidanceN0 = # robots used in the experimentk = 0,1, … (iteration index)

Ns(k+1) =

Na(k+1) =

Ns(k)

N0 – Ns(k+1)

ps=1/Ta

+ psNa(k)- paNs(k)

pa

Ns(0) = N0 ; Na(0) = 0

Linear Model – Deterministic Delay

Search Avoidance, Ta

Ta = mean obstacle avoidance durationpa = probability moving to obstacle avoidanceNs = average # robots in searchNa= average # robots in obstacle avoidanceN0 = # robots used in the experimentk = 0,1, … (iteration index)

Ns(k+1) =

Na(k+1) =

Ns(k)

N0 – Ns(k+1)

1

+ paNs(k-Ta)- paNs(k)

pa

! Ns(k) = Na(k) = 0 for all k<0 !Ns(0) = N0 ; Na(0) = 0

Linear Model – Sample Results

Micro-AB to macro comparison(same robot density but wall surfacebecome smaller with bigger arenas)

Micro-AB to micro-MB comparison(different controllers, static scenarios, allocentric measures)

Na*/N0

Steady State Analysis• Nn(k+1) = Nn(k) for all states

n of the system → Nn*

• Note 1: equivalent to differential equation of dNn/dt = 0

• Note 2: for time-delayed equations easier to perform the steady-state analysis in the Z-space but in t-space also ok (see IJRR-04)

• For our linear example (time-delay option):

aas Tp

NN+

=1

0*

Group size

Ex.: normalized mean number of robots in search mode at steady state as a function of time for obstacle avoidance

aa

aaa Tp

TpNN+

=1

0*

aas Tp

NN+

=1

0*

Nonlinear Example –Stick-Pulling

A Case Study: Stick-Pulling

Proximity sensors

Arm elevationsensor

Physical Set-Up Collaboration via indirect communication

• 2-6 robots• 4 sticks• 40 cm radius arena

IR reflectiveband

Systematic Experiments

Real robots Module-based model

•[Martinoli and Mondada, ISER, 1995]•[Ijspeert et al., AR, 2001]

Experimental and Realistic Simulation Results

• Real robots (3 runs) and realistic simulations (10 runs)• System bifurcation as a function of #robots/#sticks

Nrobots > Nsticks

Nrobots ≤ Nsticks

Geometric Probabilities

sgg

sg

aww

rR

arr

ass

pRp

ppAAp

NppAApAAp

=

==

−===

2

1

0

/)1(

//

Aa = surface of the whole arena

From Reality to Abstraction

Deterministic robot’s flowchart

Probabilistic agent’sflowchart

Markov Chain (PFSM)Nonspatiality& microscopiccharacterization

Full Macroscopic Model

• 6 states: 5 DE + 1 cons. EQ• Ti,Ta,Td,Tc ≠ 0; Τxyz = Τx + Τy + Τz• TSL= Shift Left duration• [Martinoli et al., IJRR, 2004]

)()()()()()()();()()(])()([)()1(

22

121

iasRaswcdascdagcascag

cgasacgagsRwggss

TkNpTkNpTkNTkTkNTkTkNTkTkkNppkkkNkN

−+−+−−∆+−−∆+−Γ−∆+++∆+∆−=+

For instance, for the average number of robots in searching mode:

∏−

−−=

−=Γ

=∆

−−=∆

SL

SLg

Tk

TTkjsgSL

ggg

dggg

jNpTk

kNpk

kNkNMpk

)](1[);(

)()(

)]()([)(

2

22

011

with time-varying coefficients (nonlinear coupling):

Swarm Performance Metric

C(k) = pg2Ns(k-Tca)Ng(k-Tca)

e

T

k

T

kCe

∑== 0

t

)( (k)C

: mean # of collaborations at iteration k

: mean collaboration rate over Te

Collaboration rate: # of sticks per time unit

Results (Standard Arena)

Micro-MB (10 runs)Micro-AB (100 runs)Macro (1 run)

Discrepancies due to continuous vs. discrete quantities with small numbers; rate equation simplification on stat distributions

Results: 4 x #Sticks, #Robots and Arena Area

Micro-MB (10 runs)Micro-AB (100 runs)Macro (1 run)

Reducing the Macroscopic Model

Τi,Τa,Τd,Τc << Τg →Τi=Τa=Τd=Τc=0

Goal: reach mathematical tractability

Nonlinear coupling!

Reduced Macroscopic Model

Search Grip

Ns = average # robots in searching modeNg= average # robots in gripping modeN0 = # robots used in the experimentM0 = # sticks used in the experimentΓ = fraction of robots that abandon pullingTe = maximal number of iterationsk = 0,1, …Te (iteration index)

Ns(k+1) =

Ng(k+1) =

Ns(k) – pg1[M0 – Ng(k)]Ns(k)

successful

+ pg2Ng(k)Ns(k)

unsuccessful

+ pg1[M0 – Ng(k-Τg)]Γ(k;0)Ns(k-Tg)

N0 – Ns(k+1)

∏−=

−=Γk

Tkjsg

g

jNpk )](1[)0;( 2

Ns(0) = N0, Ng(0) = 0Ns(k) = Ng(k) = 0 for all k<0

Initial conditions and causality

Results Reduced Micro-AB Model

• 4 robots, 4 sticks, Ra = 40 cm • 16 robots, 16 sticks, Ra = 80 cm

• Micro-AB (100 runs) and macro models overlapped• Only qualitatively agreement with micro-MB/real robots results

Steady State Analysis (Reduced Macro Model)

• Steady-state analysis [Nn(k+1) = Nn(k)] → It can be demonstrated that :

g

optg RM

NforT+

≤∃1

2

0

0

with N0 = number of robots and M0= number of sticks, Rg approaching angle for collaboration

• Counterintuitive conclusion: an optimal Tg can exist also inscenarios with more robots than sticks if the collaboration is very difficult (i.e. Rg very small)!

∝

approaching angle for collaboration

Analysis Verification (Micro-AB and Macro Full Model)

gg RR101~ =

20 robots and 16 sticks (optimal Tg)

Example: (collaboration very difficult)

• can be computed numerically by integrating the full model ODEs or solving the full model steady-state equations

Optimal Gripping Time• Steady-state analysis → can be computed analytically in

the simplified model (numerically approximated value):

gc

g

gg

optg R

forR

NRpT

+=≤

−

+−

−=

12

21

)1(2

1ln

)2

1ln(

10

1

βββ

β

optgT

with β = N0/M0 = ratio robots-to-sticks

[Lerman et al, Alife Journal, 2001], [Martinoli et al, IJRR, 2004]

optgT

Journal PublicationsStick Pulling

• Li, Martinoli, Abu-Mostafa, Adaptive Behavior, 2004-> learning + micro-AB

• Martinoli, Easton, Agassounon, Int. J. of Robotics Res., 2004-> real + micro-MB + micro-AB + macro

• Lerman, Galstyan, Martinoli, Ijspeert, Artificial Life, 2001-> micro-MB + macro

• Ijspeert, Martinoli, Billard, and Gambardella, Auton. Robots, 2001-> real + micro-MB + micro-AB

Object Aggregation

• Agassounon, Martinoli, Easton, Autonomous Robots, 2004-> micro-MB + macro + activity regulation

• Martinoli, Ijspeert, Mondada, Robotics and Autonomous Systems-> real + micro-MB + micro-AB

Applicability of Methods Developed for Swarm

Robotics to the IS

Can we apply these Methods to the IS? • Yes:

– Assumptions are ok– Embodied units endowed with sensors, actuators, computation, several

communications forms – Diversity among units and components (manufacturing and

assembling), noise, nonlinearities, physical world laws are all there

• No:– Microscopic information to be imported is not always available, not

accurate, or extremely costly to be acquired in IS (e.g., individual cell control, physical substrate, environmental information, etc.)

– Computational efficiency for micro optimized for hundreds of units and not billions

– Calibration methods based on different experimental conditions– Role of reproduction completely different: crucial in the IS, essentially

not considered in swarm robotics (conservation laws exploited)

Can we Modify this Method for the IS? Perhaps: open question …. – Certain limitations are easy to relax (e.g., taking into account birth

and death; optimizing computational time), other are more difficult (e.g., what microscopic information from the literature can we import, what experimental data can be acquired)

– Such multi-level methods could in principle guide experimental research in the IS, if some seed results is achieved …

– Current methods are non-spatial but they can easily be expanded to spatial methods; more difficult to maintain compactness (danger: state explosion) but further methods have been developed (e.g., graph-based); no default time/space discretization such as in Cellular Automata