Indirect (dynamic) networks

Indirect (dynamic) Networks

1

Indirect (dynamic) Networks

• Communication between any two nodes has to be carried through some switches.• Classified into:• Crossbar network•Multistage interconnection networks (MIN)• Nonblocking networks• Rearrangeable nonblocking networks• Blocking networks

• Single-stage interconnection networks

2

Crossbar (nonblocking, 1-hop)

3

An N x M crossbarThe cost increases as N x M.

Switch point(crosspoint)

M outputs

N inputs

N:1 multiplexer

A large crossbar

4

0

1

2

3

MINs

5

Three-stage MIN

first stage

second stage

third stage

N inputs

M outputs

Nonblocking MIN• Any input port can be connected to any free output port without

affecting the existing connections.• Same functionality as a crossbar.• The Clos network is proposed to reduce the cost of crossbar by

decreasing the number of switches (1953).

6

History (telephone net.)• Early telephone networks were built from elector-mechanical

crossbars.• Key developments in telephone switching include the Clos network

in 1953 and the Benes network in 1962.• Many large telephone switches today are still built from Clos or

Clos-like networks.

7

3-stage Clos network (m,n,r)

8

Nonblocking: m >= 2n-1

Rearrangeable: 2n-1 > m >= n

N inputs

N outputs

1

2

r

1

n

1

n

1

n

1

2

m

1

2

r

1

n

1

n

1

n

n×m r×r m×n

Rearrangeable nonblocking MIN

• Any input port can be connected to any free output port with rearrangement of paths for the existing connections.• Less stages or smaller switches than a nonblocking networks. • The best known example is the Benes network.

9

Benes network

10

An 8x8 Benes network (Eσ ３－１ E σ ２

－ 1 Eσ ２ Eσ ３ E)

N(2log2N - 1) / 2 switches, 2log2N - 1 hops, E: stage

0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

σ ３－１ σ ２

－１ σ ２ σ ３

010

001

(2,2,2) clos

(2,2,2) clos

(2,4,2) Clos

2x2 switch (radix: 2)

11

straight exchange lowerbroadcast

upperbroadcast

Permutation 　

•Permutations are traffic patterns in which all traffic from each source is directed to one destination.• It can be compactly represented by a permutation

function that maps source to destination.•Bit (digit) permutation are those in which each bit

(digit) of the destination address is a function of one bit (digit) of the source address.• di = sj or dx = f(sx)

12

Permutation functions (1/2)

Communication from node address x(an,an-1,an-2,…, a1) to• Perfect shuffle: σ(x)=(an-1,an-2,…, a1,an)• k sub-shuffule: σk(x)=(an,an-1,…, ak+1,ak-1, … , a1,ak)• k supershuffule: σk(x)=(an-1,an-2,…, an-k+1, an, an-k, … , a1)• Inverse perfect shuffle: σ-1(x)=(a1 , an , an-1,an-2,…, a2)• k sub-inverse shuffule: σk

-1 (x)=(an , …, ak+1 , a1,ak,…, a2)

• Butterfly: β(x)=(a1 , an-1,an-2,…, a2 , an)• k sub-butterfly: βk (x)=(an,…, ak+1 , a1,ak-1, … , a2,ak)• k superbutterfly: βk (x)=(an-k+1, an-1,…, an-k+2 , an,an-k, … , a1)

13

Permutation functions (2/2)

Communication from node address x(an,an-1,an-2,…, a1) to• Bit reversal: ρ(x)=(a1,a2,…, an-1, an)• k sub-bit reversal : ρk(x)=(an,an-1,…, ak+1,a1, a2,… , ak-1,ak)• k super-bit reversal: ρk(x)=(an-k+1,an-k+2,…, an-1, an, an-k, … , a1)• shift: α(x)= | x+1| (2n), where modulo by 2n

• k sub-shift: αk(x)= | x+1| (2n) +floor(x/2k)2k

• k super-shift: αk (x)= | x+ 2n-k | (2n) • Exchange: Ei(x)=(an,an-1, …, ~ai,…, a1)

14

Perfect shuffle (1/2)

15

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

Perfect shuffle

σ(x)=(an-1,an-2,…, a1,an)Inverse perfect shuffle

σ-1(x)=(a1,an an-1,an-2,…, a2)

Perfect shuffle (2/2)

16

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

3 sub-inverse shuffule

σ3-1

(x)=(a1 , a3 , a2)2 sub-inverse shuffle

σ2-1

(x)=(a3, a1, a2)

Butterfly

17

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

butterfly (bit reversal)

β3(x)=(a1 , a2 , a3)

　　＝ ρ ３ (x)

2 sub-butterfly

β2(x)=(a3, a1, a2)

shift

18

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

shiftα(x)= | x+1 | (23)

2 sub-shift

α 2(x)=| x+1 | (22)+floor(x/ 22) 22

exchange

19

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

E3(x) = (~a3, a2, a1) E2(x) = (a3, ~a2, a1)

Blocking MIN

• A free input/output pair is not always possible because of conflicts with the existing connections.• Minimizing the number of switches and stages.• Unidirectional MIN (Cenju-3, IBM PR3)• Bidirectional MIN (IBM SP, TMC CM5)

20

Unidirectional MINs•Channels and switches are unidirectional.•At least ceiling(logkN) stages are required for a MIN

of N input and output ports with k×k switches.•Every path through the MIN crosses all the stages

(same length).•Examples: baseline, butterfly, indirect cube

logN stages for N inputs/outputs with 2x2 switches( binary logN-fly in the form of k-ary n-fly)the number of communication patterns are (√N)N

21

Baseline networks (1/2)(B = Eσn

－１ Eσn-1－１ E ・・・ Eσ ２

－ 1 E)

22

000

001

010

011

100

101

110

111

000

001 (UUL)

010 (ULU)

011

100

101

110

111

abc cab cbaσ ２

－ 1σ ３－ 1

001

Baseline networks (2/2)

• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),select upper link when bi=0, otherwise select lower link on (n-i+1) stage (MSB

to LSB).(path is determined only by the destination address)

23

switch

upper linklower link

inputs

Omega networks (1/2) Ω= (σE)n

24

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

σσσabc bca abccab

Omega networks (2/2)

• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),set the switch as straight when ai^bi=0, otherwise set it as exchange on (n-i+1)

stage (MSB to LSB).

25

straight exchange

Indirect binary n-cube (1/2)(C = Eβ ２ Eβ ３ ・・・ E β ｎ Eσ － 1)

26

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

σ-1β ３β ２

000001

100101

010

110

011

111

abc acb abcbca

Indirect binary n-cube (2/2)

• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),set the switch as straight when ai^bi=0, otherwise set it as exchange on i-th

stage (LSB to MSB).

• A k-ary n-fly network consists of kn source/destination terminals, n stages of kn-1 k×k crossbar switches.• Degree of the switches: 2k• Diameter: n+1 or logkN +1 where N = kn

27

Relation among blocking networks • For omega, indirect n-cube and Banyan(Y):

Ω-1=C=Yσ 　So they are almost functionally equal (isomorphic).• The communication patterns are different, but same number of

combinations ((√N)N).• Smaller number of communication patterns than nonblocking

networks (N!).

28

Bitonic sort 　

29

8 inputs sorter4 inputs sorter2 inputs sorter

A LB H

Batcher-banyan network

30

BitonicSorter

(Batcher stage)

Ω

(banyan

stage)

Non-blocking

5∞31∞･･∞

135∞∞･･∞

1

3

5

receiver Ordered addresses

Tree saturation (e.g. on Omega)

31

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

Combining: combine two messages

hotspot

Fault-tolerant MINs

• Blocking MINs provide a single path between any input/output pair.• If a link becomes congested or fails, the unique path property can

easily disrupt the communication (no path diversity).• A solution is to provide multiple routing paths by adding extra

stages or channels.

32

Fault-tolerant MINs (Omega’s case)

33

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

σσσAdditionalstage

Fault-tolerant MINs (Baseline’s case)

34

000

001

010

011

100

101

110

111

000

001

010

011

100

101

110

111

σ ２－ 1σ ３

－ 1

Bidirectional MINs (BMINs)

•Channels and switches are bidirectional.• Information can be transmitted simultaneously in

opposite directions between neighboring switches.•Paths are established in BMINs by crossing stages in

forward, then turnaround, and finally backward.•Several paths are possible in forward direction, but

a single path is available in backward direction.

35

Bidirectional Banyan networks

36

000

001

010

011

100

101

110

111

β ３β ２

000001

100101

010

110

011

111

000001

100101

010

110

011

111

forward

backward

turnaround

A fat tree BMIN

37

forward backward

Hybrid networks

• Combine mechanisms from shared-medium networks and direct or indirect networks.• Increase bandwidth and reduce the distance between nodes.• Examples:

• Multiple backplane buses• Hierarchical buses• Cluster-based networks

38

Multiple backplane buses

39

device device device

bus

bus

bus

Hierarchical buses

40

Global bus

Cluster busbridge

Cluster-based networks

41

Cluster bus

ClusterCluster Cluster

ClusterCluster Cluster

router

History (Clos networks, etc.)•Clos introduced non-blocking networks in1953.•Benes discovered rearrangeable non-blocking

networks in 1965.•Bachter introduced the bitonic sort and the

Batcher sorting network in 1968.•Knuth’s book in 1998 treats sorting networks in

detail.•Multicast in Clos networks was first considered by

Masson and Jordan(1972), and has since been studied by many researchers.

42

History (indirect networks)• Processor-memory networks emerged in the late 1960s

in the form of indirect netwoks.• The smallest machine employed crossbar, whereas

larger sytems used butterfly, etc..• Since the early 1990s, a variant of the Clos and Benes

networks of telephony has emerged in SMP (MPP) networks in the form of the fat tree.• Today, the high pin bandwidth of router chips relative

to message length motivates the use of networks of higher node degree, such as Clos and fat free.

43