Page 1
Scheduling Techniques for Hybrid Circuit/Packet
NetworksHe Liu, Matthew K. Mukerjee, Conglong Li, Nicolas Feltman,
George Papen, Stefan Savage, Srinivasan Seshan, Geoffrey M. Voelker, David G. Andersen, Michael Kaminsky, George Porter, and Alex C. Snoeren
Page 2
Hybrid circuit/packet networks: Can we get best of both?
Circuit Switching Packet Switching
High link rate
Low link rate
Limits on num flows+ Explicit flow setup
Any-to-any communication
This talk: commun. restricted to pairs
+ non-trivial circuit reconfig time
Page 3
Hybrid networks in datacenters*
*Helios [SIGCOMM ’10], c-Through [SIGCOMM ’10], Flyways [SIGCOMM ’11], …
RACK RACK RACK RACK RACK RACK RACK RACK
Circuit Packet
Page 4
Hybrid networks in datacenters*
*Helios [SIGCOMM ’10], c-Through [SIGCOMM ’10], Flyways [SIGCOMM ’11], …
A B CABC
Datacenter workloadsare skewed
Senders
Receivers
Page 5
Senders
ReceiversHybrid networks in datacenters*
*Helios [SIGCOMM ’10], c-Through [SIGCOMM ’10], Flyways [SIGCOMM ’11], …
A B CABC
Prior work showsmajor benefits
(“Keep circuits busy!”)
Page 6
Hybrid networks in datacenters*
*Helios [SIGCOMM ’10], c-Through [SIGCOMM ’10], Flyways [SIGCOMM ’11], …
Packet switch as “escape hatch”
for difficult demands
Need an explicit sequence of circuit
configurations
Page 7
Hybrid networks in datacenters*
How do proposed hybrid networks schedule demand?They don’t! They use an oracle
*Helios [SIGCOMM ’10], c-Through [SIGCOMM ’10], Flyways [SIGCOMM ’11], …
Page 8
Does prior scheduling work help?
iSLIP BvN
Works per-packet (can’t identify “difficult” to schedule demands)
Works on all demand simultaneously
(can identify “difficult” demands)
Both assume circuit reconfig is free (use many configurations)
(Fast heuristic per-packet) Optimal
Page 9
Does prior scheduling work help?BvN
Both assume circuit reconfig is free (use many configurations)
Time
iSLIP
Page 10
Contribution: Solstice
• Characterizing
• Optimal formulation and bounds
• A fast, heuristic-based hybrid scheduler “ ”
• of Solstice: performs 2-3x better than previous solutions and within 14% of optimal up to a 128-port switch
the hybrid scheduling problem
Solstice
Evaluation
Page 11
Outline
• Model
•
•
•
the hybrid scheduling problem
Solstice
Evaluation
Page 12
Goal of hybrid scheduling
?
A B CABC
Schedule all demand in minimum duration
Page 13
Goal of hybrid scheduling
Schedule all demand in minimum duration
duration = total circuit transmission time+
number of configurations * circuit reconfig cost
Configuration selection and duration
What goes topacket switch
Page 14
The hybrid scheduling problem
?
A B CABC
Page 15
Understanding circuitsKey properties
of circuit switching
Configuration
Duration
A B CABC
Page 16
Understanding circuitsKey properties
of circuit switching
Configuration
Duration
Src Dest
A
B
C
A
B
C
A B CABC
A
B
C
A
B
C
Src Dest
Configuration
Duration
Page 17
The hybrid scheduling problem
?Circuit Switch Packet Switch
A
B
C
A
B
C
A B CABC
Page 18
The hybrid scheduling problem
?Circuit Switch Packet Switch
A
B
C
A
B
C
A B CABC
A B CABC
How to compute the output?
Page 19
Outline
• Model
• The hybrid scheduling problem
• Solstice
• Evaluation
Page 20
Solstice: intuition
• What can we take advantage of?
• Skew in workload (i.e., large + small demands)
• Circuit switch is efficient for large demands
• Packet switch is efficient for small demands
Page 21
Solstice: key ideaSchedule largedemands first
on circuit
A B CABC
D
D
Page 22
Solstice: key ideaSchedule largedemands first
on circuit
Circuit SwitchLoop
A B CABC
D
D
A
B
C
A
B
C
D D
Page 23
Solstice: key idea
Circuit Switch
A B CABC
D
D
Schedule largedemands first
on circuitLoop
A
B
C
A
B
C
D D
A
B
C
A
B
C
D D
Page 24
Solstice: key ideaTerminate when
remainder can be served by
packet switch
Circuit Switch
A B CABC
D
D
A
B
C
A
B
C
D D
A
B
C
A
B
C
D D
Packet SwitchA B C
ABC
D
D
Page 25
Solstice: key idea
Large demand alwayson circuit switch
(efficient)
Small demand alwayson packet switch
(efficient)Circuit SwitchA
B
C
A
B
C
D D
A
B
C
A
B
C
D D
Packet SwitchA B C
ABC
D
D
Page 26
Solstice: example
A B CABC
D E
DE
Page 27
Solstice: exampleTotal
Duration
0Reconfig time = 2.5
. 5 40 . 5 5 . . 40 5 . . . 10 4045 5 . . . . 40 10 . .
Page 28
Solstice: exampleServableon Packet
0Total
Duration
0Reconfig time = 2.5 Packet BW = 1/10
. 5 40 . 5 5 . . 40 5 . . . 10 4045 5 . . . . 40 10 . .
Page 29
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
0Reconfig time = 2.5 Packet BW = 1/10
. 5 40 . 5 5 . . 40 5 . . . 10 4045 5 . . . . 40 10 . .
Page 30
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
0Reconfig time = 2.5 Packet BW = 1/10
. 5 40 . 5 5 . . 40 5 . . . 10 4045 5 . . . . 40 10 . .
Page 31
. 5 40 . 5 5 . . 40 5 . . . 10 4045 5 . . . . 40 10 . .
A
B
C
A
B
C
D
E
D
E
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
0Reconfig time = 2.5 Packet BW = 1/10
40
Page 32
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
0Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 33
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
40Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 34
Solstice: exampleServableon Packet
0Threshold
32Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 35
Solstice: exampleServableon Packet
4.25Threshold
32Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 36
Solstice: exampleServableon Packet
4.25Threshold
16Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Threshold too large
Page 37
Solstice: exampleServableon Packet
4.25Threshold
8Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 38
Solstice: example
A
B
C
A
B
C
D
E
D
E
Servableon Packet
4.25Threshold
8Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Incompleteconfiguration
Page 39
Solstice: exampleServableon Packet
4.25Threshold
4Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E . 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
Page 40
Solstice: example
A
B
C
A
B
C
D
E
D
E
Servableon Packet
4.25Threshold
4Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
A
B
C
A
B
C
D
E
D
E
. 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
MultipleConfigs
Page 41
. 5 . . 5 5 . . . 5 . . . 10 . 5 5 . . . . . 10 . .
A
B
C
A
B
C
D
E
D
E
Solstice: exampleServableon Packet
4.25Threshold
4Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
Page 42
Solstice: exampleServableon Packet
4.25Threshold
4Total
Duration
42.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E . 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Page 43
Solstice: exampleServableon Packet
4.25Threshold
4Total
Duration
47.5Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E . 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Page 44
Solstice: exampleServableon Packet
4.25Threshold
4Total
Duration
50Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E . 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Page 45
Solstice: exampleServableon Packet
5Threshold
4Total
Duration
50Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E
. 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
. 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Page 46
Servableon Packet
5
Solstice: exampleThreshold
4
. . . . . . . . . . . . . . . . . . . . . . . . .
. 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Total Duration
50Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E
Page 47
Servableon Packet
5
Solstice: exampleThreshold
4
. . . . . . . . . . . . . . . . . . . . . . . . .
. 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .
Total Duration
50Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E
ScheduledAll Demand
Page 48
Servableon Packet
5
Solstice: exampleThreshold
4
. . . . . . . . . . . . . . . . . . . . . . . . .
Total Duration
50Reconfig time = 2.5 Packet BW = 1/10
40
A
B
C
A
B
C
D
E
D
E
5
A
B
C
A
B
C
D
E
D
E
ScheduledAll Demand
2 Configs
. 5 . . . . . . . 5 . . . 5 . 5 . . . . . . 5 . .Remainderon Packet
Page 49
Outline
• Model
• The hybrid scheduling problem
• Solstice
• Evaluation
Page 50
Inefficiencies
• Two possible ways to be inefficient
• Circuit reconfig “downtime”
• Link is active but sender has no data to send
• Focus on first one in this talk; second in paper
Page 51
Metrics
• Total schedule duration
• total circuit transmission time + number of configurations * reconfig time
• Number of circuit configurations used in schedule
Page 52
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)Schedule duration
Better
Worse
Page 53
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)Schedule duration
Better
Worse
Lower Bound
Page 54
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)Schedule duration
Upper Bound
Better
Worse
Lower Bound
Page 55
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)
Solstice
Schedule duration
Solstice at most 14% from optimal for these counts
Better
Worse
Page 56
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)
Solstice BvN
Schedule duration
Better
Worse
Page 57
8 12 16 32 64 128Number of ports
0
2
4
6
8
10
12
14
16
Tim
e(m
s)
Solstice BvN iSLIP
Schedule durationSolstice
2-3x better
iSLIP poor overall. Why?
BvN gets worse here. Why?
Better
Worse
Page 58
8 12 16 32 64 128Number of ports
0
50
100
150
200
Num
confi
gs
Solstice BvN iSLIP
Number of configurations
iSLIP uses many configurations
throughout
BvN almost 3x number of
configurations
Better
Worse
Page 59
Other results in paper
• Effects of skew (up to 10% worse)
• Effects of sparsity (up to 26% worse)
• Runtime complexity analysis
• Augmenting Solstice with offline computation
Page 60
Conclusion
• Characterizing the hybrid scheduling problem
• Optimal formulation and bounds
• A fast, heuristic-based hybrid scheduler “Solstice”
• Evaluation of Solstice: performs 2-3x better than previous solutions and within 14% of optimal up to a 128-port switch
Page 61
Scheduling Techniques for Hybrid Circuit/Packet
NetworksHe Liu, Matthew K. Mukerjee, Conglong Li, Nicolas Feltman,
George Papen, Stefan Savage, Srinivasan Seshan, Geoffrey M. Voelker, David G. Andersen, Michael Kaminsky, George Porter, and Alex C. Snoeren
Page 63
Solving Optimally• Use an ILP to solve the problem
• Provides intuition and a lower bound (i.e., this is the best you can do)
• has to work through n! possible configurations
• Slow (e.g., 12-port takes minutes, 16-port takes hours)
Page 64
Stuffing . 13 10 70 4 . . 14 12 6965 . . 12 1415 33 2 . 1112 7 3 1 .
. 14 10 70 4 . . 17 12 6971 . . 13 1415 70 2 . 1112 14 69 3 .
= Variable row/col sum (but max is 98)
Fixed row/col sum (98) =
Pick large entries and increase until maxed
Page 65
Runtime complexity• Dominated by circuit configuration selection
• (equivalent to finding a perfect matching in a bipartite graph)
• O(n * non-zero entries in rows) + O(n * log2 n)
• If we assume number of non-zero entries is constant (in line with prior work on dc workloads), total is O(n2 * log2 n)
Page 66
Simulation Setup• 20 microsecond reconfiguration time
• accumulation period of 3 ms, schedule all traffic
• 100Gbps circuit links, 10Gbps packet links
• sparse matrices (4 large and 12 small demands per port on 64 port switch)
• skewed matrices (small demands total 30% of demand)
• 100 random matrices for testing
Page 67
8 12 16 32 64 128Number of ports
0
20
40
60
80
100
Util
izat
ion
(%)
iSLIP BvN Solstice
Circuit utilizationNumber of configshas great impact
on circuit utilization
Worse
Better
Page 68
8 16 24 32 40 48 56 64Number of demands per port
2.02.53.03.54.04.5
Tim
e(m
s)
2.8 3.
1 3.3 3.
5 3.6
3.7
3.7 3.
9Solstice
Sparsity
Default
Worse
Better
only 26% worse
. 3 . . . . . 4 . . . . . 4 . . . . . 142 . . . .
32 13 10 70 424 29 14 12 6965 1 10 12 1415 33 2 42 1112 7 3 1 3
Page 69
Skew
0 5 10 15 20 25 30 35 40 45 50Bandwidth requested by small flows (%)
2.02.53.03.54.04.5
Tim
e(m
s)
2.7 2.
9 3.0
3.0
3.0 3.1
3.1
3.1 3.2 3.2
3.3
Solstice
Worse
Better
only 6% worseDefault
. 3 . 40 2 2 . 4 . 40 1 45 . 4 . . 2 48 . 142 . 3 4 .
. 43 . 40 4242 . 41 . 4041 45 . 44 . . 42 48 . 4142 . 43 44 .