CSE 222A CSE 222A Graduate Networking Graduate Networking Winter 2003 Winter 2003 Lecture 10: High Lecture 10: High - - Performance Router Design Performance Router Design Stefan Savage Stefan Savage
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CS
E 2
22A
CS
E 2
22A
Gra
duat
e N
etw
orki
ngG
radu
ate
Net
wor
king
Win
ter
2003
Win
ter
2003
Lect
ure
10: H
igh
Lect
ure
10: H
igh --
Perf
orm
ance
Rou
ter D
esig
nPe
rfor
man
ce R
oute
r Des
ign
Stef
an S
avag
eSt
efan
Sav
age
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Ove
rvie
wO
verv
iew
Rou
ter b
asic
sIn
terc
onne
ctio
n ar
chite
ctur
e◆
Inpu
t Que
uing
◆O
utpu
t Que
uing
◆V
irtua
l out
put Q
ueui
ng◆
Sch
edul
ing
Futu
re b
ottle
neck
sC
ase
Stud
ies
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Wha
t’s
in a
rou
ter?
Wha
t’s
in a
rou
ter?
Phys
ical
com
pone
nts
◆O
ne o
r mor
e in
put i
nter
face
sth
at re
ceiv
e pa
cket
s◆
One
or m
ore
outp
ut in
terf
aces
that
tran
smit
pack
ets
◆A
cha
ssis
(box
+ p
ower
) to
hold
it a
ll
Func
tions
◆Fo
rwar
dpa
cket
s◆
Dro
ppa
cket
s (c
onge
stio
n, s
ecur
ity, Q
oS)
◆D
elay
pack
ets
(QoS
)◆
Tran
sfor
mpa
cket
s? (E
ncap
sula
tion,
Tun
nelin
g)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Wha
t a
rout
er d
oes:
Wha
t a
rout
er d
oes:
the
norm
al c
ase
the
norm
al c
ase
Rec
eive
inco
min
g pa
cket
from
link
inpu
t int
erfa
ceLo
okup
pac
ket d
estin
atio
n in
forw
ardi
ng ta
ble
◆(d
estin
atio
n, o
utpu
t por
t(s))
Valid
ate
chec
ksum
, dec
rem
ent t
tl, u
pdat
e ch
ecks
um
Buf
fer p
acke
t in
inpu
t que
ueSe
nd p
acke
t to
outp
ut in
terf
ace
(inte
rfac
es?
Mca
st)
Buf
fer p
acke
t in
outp
ut q
ueue
Send
pac
ket t
o ou
tput
inte
rfac
e lin
k
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Wha
t a
rout
er lo
oks
like?
Wha
t a
rout
er lo
oks
like?
Cis
co 2
500
Link
sys
DE
FSR
81
19”
8”
Cap
acity
: <10
Mbp
sC
apac
ity: <
10M
bps
1.75
”
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Wha
t a
rout
er lo
oks
like
(2)
Wha
t a
rout
er lo
oks
like
(2)
Cis
co G
SR
124
16Ju
nipe
rM16
0
6ft
6ft
1919””
2ft
2ft
Cap
acit
y:
Cap
acit
y:
160G
b/s
160G
b/s
Pow
er: 4
.2kW
Pow
er: 4
.2kW
3ft
3ft
2.5f
t2.
5ft
1919””
Cap
acit
y:
Cap
acit
y:
80G
b/s
80G
b/s
Pow
er: 2
.6kW
Pow
er: 2
.6kW
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Wha
t a
rout
er lo
oks
like
(3)
Wha
t a
rout
er lo
oks
like
(3)
Plu
risTe
rapl
ex20
w/7
Rac
ks
Cap
acit
y:
Cap
acit
y:
>1T
b/s
>1T
b/s
Pow
er: 4
5kW
P
ower
: 45k
W
(~25
0 ho
mes
?)(~
250
hom
es?)
1 ro
om
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Hig
hH
igh
-- per
form
ance
rou
ters
perf
orm
ance
rou
ters
Gea
red
to c
ore
and
dist
ribut
ion
serv
ice
need
s◆
Req
uire
men
ts: h
igh
spee
d &
hig
h de
nsity
Why
do
we
care
?◆
Moo
re’s
Law
slo
wer
than
link
spe
ed g
row
th (a
nd B
W d
eman
d)»
OC
48c
(2.5
Gbp
s), n
ow, 1
28ns
/pac
ket
»O
C19
2c (1
0Gbp
s), i
n de
ploy
men
t, 33
ns/p
acke
t»
OC
768c
(40G
bps)
, 200
2-3,
8ns
/pac
ket
◆N
eed
high
den
sity
/low
pow
er to
man
age
PO
P c
ompl
exity
»$2
0-10
0k &
2-4
00W
per
por
t, 50
% p
orts
freq
uent
ly fo
r in
tern
al c
onne
ctiv
ity»
DW
DM
can
hel
p w
ith th
e fo
rmer
, but
requ
ires
mor
e in
terfa
ces
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Func
tion
al a
rchi
tect
ure
Func
tion
al a
rchi
tect
ure
Res
erva
tion/
Adm
issi
on
Con
trol
Rou
ting
Prot
ocol
s
Rou
ting
Tabl
eC
lass
ifica
tion
Rul
es
Fire
wal
l
Pack
etC
lass
ifica
tion
Switc
hing
Forw
ardi
ngTa
ble
Out
put
Sche
dulin
g
Con
trol
Pla
ne•C
ompl
ex•P
er-c
ontro
l act
ion
•May
be
slow
Dat
a pl
ane
•Sim
ple
•Per
-pac
ket
•Mus
t be
fast
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Pac
ket
clas
sifi
cati
onP
acke
t cl
assi
fica
tion
Forw
ardi
ng◆
Long
est p
refix
mat
ch o
f des
tinat
ion
agai
nst f
orw
ardi
ng ta
ble
◆R
etur
ns (o
utpu
t por
t, N
ext-h
op M
AC
hea
der)
tupl
e◆
Key
issu
e: fo
rwar
ding
tabl
e gr
owth
◆G
eorg
e w
ill ta
lk a
bout
this
nex
t tim
e
QoS
tagg
ing
◆C
erta
in tr
affic
tagg
ed w
ith h
ighe
r prio
rity
◆P
er fl
ow (s
rcip
, src
port,
dst
ip, d
stpo
rt), p
re s
ourc
e or
des
tpr
efix
, per
pro
toco
l (N
apst
er, e
tc…
)
Fire
wal
l rul
es◆
Blo
ck a
cces
s to
TC
P p
acke
ts w
ith d
stpo
rt !=
80
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Inte
rcon
nect
arc
hite
ctur
eIn
terc
onne
ct a
rchi
tect
ure
Inpu
t & o
utpu
t co
nnec
ted
via
switc
h fa
bric
Kin
ds o
f sw
itch
fabr
ic◆
Bus
◆C
ross
bar
◆S
hare
d M
emor
y
How
to d
eal w
ith
tran
sien
t con
tent
ion?
◆In
put q
ueui
ng◆
Out
put q
ueui
ng◆
Com
bina
tion
Inpu
tO
utpu
tSw
itch
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Firs
t G
ener
atio
n R
oute
rsFi
rst
Gen
erat
ion
Rou
ters
CPU
Rou
teTa
ble
Buf
fer
Mem
ory
MA
C
Line
Car
d
Shar
ed B
us(s
)
MA
C
Line
Car
d
MA
C
Line
Car
d
•Sin
gle
CPU
and
shar
ed m
emor
y;•A
ll cl
assi
ficat
ion
by
mai
n C
PU
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Sec
ond
Gen
erat
ion
Rou
ters
Sec
ond
Gen
erat
ion
Rou
ters
CPU
Rou
teTa
ble
MA
C
Line
Car
d
Shar
ed B
us(s
)
Forw
ardi
ngC
ache
Buf
fers
MA
C
Line
Car
d
Forw
ardi
ngC
ache
Buf
fers
MA
C
Line
Car
d
Forw
ardi
ngC
ache
Buf
fers
Dire
ct D
MA
on
cach
e hi
t
Cac
he o
fre
cent
rout
es
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Thi
rd G
ener
atio
n R
oute
rsT
hird
Gen
erat
ion
Rou
ters
MA
C
Line
Car
d
Forw
ardi
ngTa
ble
Buf
fers
CPU
Car
d
Rou
ting
Tabl
e
CPU
MA
C
Line
Car
d
Forw
ardi
ngTa
ble
Buf
fers
Switc
h Fa
bric
•Sha
red
inte
rcon
nect
(usu
ally
cro
ssba
r)•C
entr
aliz
ed s
ched
uler
•Ful
l for
war
ding
tabl
ein
line
car
d•F
ixed
cel
ls (w
hy?)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Out
put
queu
ing
Out
put
queu
ing
Out
put i
nter
face
s bu
ffer
pack
ets
Pro
◆S
impl
e al
gorit
hms
◆S
ingl
e co
nges
tion
poin
t
Con
◆N
inpu
ts m
ay s
end
to th
e sa
me
outp
ut◆
Req
uire
s sp
eedu
p of
N
Inpu
tO
utpu
tSw
itch
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Inpu
t qu
euin
gIn
put
queu
ing
Inpu
t int
erfa
ces
buffe
r pa
cket
s
Pro
◆S
ingl
e co
nges
tion
poin
t◆
Sim
ple
to d
esig
n al
gorit
hms
Con
◆M
ust i
mpl
emen
t flo
w c
ontro
l◆
Low
util
izat
ion
due
to H
ead-
of-L
ine
(HoL
) Blo
ckin
g »
Util
ilim
ited
to 2
-2^
.5=5
8%
Inpu
tO
utpu
tSw
itch
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Hea
dH
ead
-- of
of-- L
ine
Blo
ckin
gLi
ne B
lock
ing
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Vir
tual
Out
put
Que
ues
Vir
tual
Out
put
Que
ues
(cou
rtes
y N
ick
McK
eow
n)(c
ourt
esy
Nic
k M
cKeo
wn)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
IQ +
Vir
tual
Out
put
Que
uing
IQ +
Vir
tual
Out
put
Que
uing
Inpu
t int
erfa
ces
buffe
r pa
cket
s in
per
-out
put
virt
ual q
ueue
s
Pro
◆S
olve
s bl
ocki
ng p
robl
emC
on◆
Mor
e re
sour
ces
per p
ort
◆C
ompl
ex a
rbite
r◆
Stil
l lim
ited
by in
put/o
utpu
t co
nten
tion
(sch
edul
er).
◆R
R: 1
=1/e
= 6
3%
Inpu
tO
utpu
tSw
itch
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Sw
itch
sch
edul
ing
Sw
itch
sch
edul
ing
Prob
lem
◆M
atch
inpu
ts a
nd o
utpu
ts◆
Res
olve
con
tent
ions
◆N
o ou
tput
pac
ket d
rops
◆M
axim
ize
thro
ughp
ut◆
Do
it in
con
stan
t tim
e…M
any
algo
rithm
s fo
r uni
form
traf
fic a
ssum
ptio
n◆
E.g
. TD
M, M
axim
um s
ize
bipa
rtite
mat
ch◆
App
roxi
mat
e an
swer
s (e
.g. i
SLI
P, s
ubm
axim
alm
atch
)R
ecen
t res
ult (
Dai
et a
l, 20
00)
◆M
axim
al s
ize
mat
chin
g +
spee
dup
of tw
o gu
aran
tees
10
0% u
tiliz
atio
n fo
r mos
t tra
ffic
assu
mpt
ions
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Mod
ern
rout
erM
oder
n ro
uter
IQ +
VoQ
+ O
Q◆
Spe
edup
of 2
◆C
entra
l sch
edul
er◆
Fixe
d-si
zed
inte
rnal
cel
ls
Pro
◆C
an a
chie
ve u
tiliz
atio
n of
1◆
Can
sca
le to
mul
tiple
Tb/
s
Con
◆M
ultip
le c
onge
stio
n po
ints
◆C
ompl
exity
Inpu
tO
utpu
tSw
itch
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Typ
ical
fun
ctio
n br
eakd
own
Typ
ical
fun
ctio
n br
eakd
own
Inpu
t int
erfa
ce◆
Forw
ardi
ng◆
Virt
ual o
utpu
t que
uing
Switc
h◆
Sch
edul
ing
inpu
t int
erfa
ce re
ques
ts to
out
put i
nter
face
s◆
Mul
ticas
t sch
edul
ing
Out
put i
nter
face
◆Q
ueue
pac
kets
for t
rans
mis
sion
◆C
lass
ifica
tion
◆B
uffe
r man
agem
ent (
whi
ch p
ktto
dro
p)◆
Sch
edul
ing
(whi
ch p
ktto
sen
d fro
m b
uffe
r)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Nex
t bo
ttle
neck
sN
ext
bott
lene
cks
Buf
ferin
g at
hig
h sp
eed
◆S
RA
M d
ensi
ty to
o lo
w fo
r BW
*D o
f 40G
bps
link
◆D
RA
M to
o sl
ow◆
SR
AM
mem
ory
man
agem
ent a
s ca
che
for D
RA
M
Sche
dule
r and
arb
iter o
verh
ead
◆Li
mits
siz
e of
sw
itch
and
link
BW
◆Tw
o-st
ate
switc
h (C
hang
et a
l, 20
00);
no s
ched
uler
Hig
h de
nsity
(100
’s-1
000’
s of
line
car
ds)
◆P
hysi
cal d
ista
nce
to s
uppo
rt de
nsity
; ele
ctric
al li
nks
degr
ade
◆O
ptic
al li
nks;
opt
ical
cro
ss c
onne
ct (M
EM
s,tu
nabl
ela
sers
)
Tim
e to
mar
ket,
Pow
er/H
eat
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Cas
e st
udy
Par
trid
ge e
t al
98
Cas
e st
udy
Par
trid
ge e
t al
98
IQ +
VO
Q +
OQ
15 li
ne-c
ards
eac
h w
ith u
p to
16
inte
rfac
es◆
Forw
ardi
ng e
ngin
es s
epar
ate
from
lc(w
hy?)
Poin
t-to-
Poin
t sw
itch
◆50
Gbp
s ca
paci
ty, 2
5% lo
st to
ove
rhea
d◆
32M
pps
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Rou
ter
Arc
hite
ctur
eR
oute
r A
rchi
tect
ure
pack
ethe
ader
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Impl
emen
tati
onIm
plem
enta
tion
Line
car
ds◆
53M
hz F
PG
A, I
/O p
roce
ssin
g, q
ueue
man
agem
ent
Forw
ardi
ng e
ngin
e◆
430M
hz 2
1164
Alp
ha u
sing
L2
as ro
ute
cach
e, L
3 ho
lds
all
»12
,000
rout
es..
clai
m 9
5% h
it ra
te…
reas
onab
le?
◆Lo
ts o
f pai
n w
ith u
PI/O
bei
ng n
ot q
uite
righ
t◆
Doe
sn’t
chec
k IP
che
cksu
m.
Is th
is o
k?Sw
itch
◆S
ched
uler
FP
GA
Con
trol
pla
ne◆
Net
BS
Dru
nnin
g G
ateD
on 2
33-M
hz 2
1064
◆P
ushe
s ro
utin
g ta
bles
to F
Es
and
hand
les
exce
ptio
nal
pack
ets
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Sum
mar
y: I
nnov
atio
nsS
umm
ary:
Inn
ovat
ions
1.Ea
ch F
E ha
s a
com
plet
e se
t of t
he ro
utin
g ta
bles
2.A
sw
itche
d fa
bric
is u
sed
inst
ead
of th
e tr
aditi
onal
sha
red
bus
3.FE
s ar
e on
boa
rds
dist
inct
from
the
line
card
s4.
Use
of a
n ab
stra
ct li
nk la
yer h
eade
r5.
Incl
ude
QoS
pro
cess
ing
in th
e ro
uter
Que
stio
ns◆
Wou
ld y
ou u
se a
mic
ropr
oces
sor f
or fo
rwar
ding
toda
y?◆
How
impo
rtant
is p
rogr
amm
abili
ty?
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Cas
e st
udy:
McK
eow
n et
al9
8C
ase
stud
y: M
cKeo
wn
et a
l98
Tiny
Ter
ra: a
n in
spira
tion
for C
isco
GSR
(nee
BFR
)IQ
+ V
OQ
+ O
Q
32 in
terf
aces
at 1
0Gbp
s ea
ch32
x32
cros
sbar
(320
Gbp
s ag
greg
ate)
64bi
t cel
lsLi
ne c
ards
ASI
C-b
ased
iSLI
Psc
hedu
ling
algo
rithm
◆G
reed
y, s
ubm
axim
albi
parti
te m
atch
ing
◆E
asy
to im
plem
ent
(2N
arb
iters
)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Con
clus
ion
Con
clus
ion
It is
feas
ible
to b
uild
hig
h sp
eed
rout
ers
◆40
Gbp
s lin
k sp
eeds
◆10
-20
Tbps
aggr
egat
e ca
paci
ty
But
…◆
Lim
ited
prog
ram
mab
ility
◆H
igh
com
plex
ity, s
low
tim
e to
mar
ket
»Ju
nipe
r I2
AS
IC 2
.5M
gat
es»
Typi
cal O
C19
2 LC
~30
M g
ates
! »
Sta
rting
to re
quire
sig
nific
ant o
n-ch
ip S
RA
M◆
Nex
t gen
(OC
3072
160
Gbp
s LC
) may
be
clos
e to
cro
ss-o
ver
poin
t for
CM
OS
(luc
kily
, not
cle
ar th
ere
is d
eman
d an
yway
)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Ope
n is
sues
Ope
n is
sues
Net
wor
k pr
oces
sors
◆
Qui
ck T
TM +
hig
h pe
rform
ance
◆A
pplic
atio
n-sp
ecifi
c pr
ogra
mm
abili
ty (f
or h
igh-
touc
h fe
atur
es)
Softw
are
desi
gn◆
Neg
lect
ed, b
ut c
ritic
al
Opt
ical
sw
itchi
ng◆
Look
s lik
e ci
rcui
t sw
itchi
ng, v
ery
low
pow
er◆
Ok
to lo
se p
acke
t sw
itch
effic
ienc
y if
BW
is c
heap
and
relia
ble
◆C
urre
ntly
ver
y ex
pens
ive
to li
ght a
lam
bda
(am
ps, l
aser
s)
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
For
next
tim
e…Fo
r ne
xt t
ime…
Geo
rge
Varg
hese
will
be
pres
entin
g on
IP L
ooku
ps
Ther
e is
a p
acke
t loo
kup
surv
ey p
aper
to re
ad th
at
will
be
post
ed to
nigh
t.
Febr
uary
18,
200
3C
SE 2
22A
--Le
ctur
e 10
–R
oute
r Des
ign
Som
e sl
ides
cou
rtes
y of
Nic
k M
cKeo
wn
Poi
nts
of P
rese
nce
(P
oint
s of
Pre
senc
e ( P
OP
sP
OP
s ))
A B C
POP1
POP3
POP2
POP4
D E F
POP5
POP6
POP7
POP8
Related Documents
