Dynamic Load Balancing Tree and Structured Computations CS433 Laxmikant Kale Spring 2001
Dec 28, 2015
Dynamic Load Balancing Tree and
Structured ComputationsCS433
Laxmikant Kale
Spring 2001
When to send work away:
• Consider a processor with k units of work, with P other processors, – assume that a message takes 100 microsecs to reach:
• 20 microseconds send-processor overhead,
• 60: network latency
• 20 receive processor overhead
– If each task takes t units of time to complete, under what conditions should send them out to others (vs. doing it itself)?
– E.g. if t=100 microseconds? 50? 1000?
Key observation: the “master” spends 40 microseconds on coordination for a task, although the latency is 200 microsecs
Tree structured computations
• Examples: – Divide-and-conquer
– State-space search:
– Game-tree search
– Bidirectional search
– Branch-and-bound
• Issues:– Grainsize control
– Dynamic Load Balancing
– Prioritization
Divide and Conquer
• Simplest situation among the above– Given a problem, a recursive algorithm divides it into 1 (2) or
more subproblems, and solutions to the subproblems are composed to create a solution
– Example: adaptive quadrature
– Consider a simpler setting:
• Fib(n) = fib(n-1) + fib(n-2)
• Note: the fibonacci algorithm is not important here
– Issues:
• subtrees are unequal size, so can’t assign work a priori
• Fire tasks in parallel: but too fine-grained
Dynamic load balancing formulation:
• Each PE is creating work randomly• How to redistribute work?
• Initial allocation• Rebalancing
• Centralized vs distributed
Reading Assignment
• Adaptive grainsize control:– http://charm.cs.uiuc.edu go to publications, 95-05
• Prioritization and first-solution search:– http://charm.cs.uiuc.edu go to publications, 93-06
• Dynamic Load Balancing for tree structured computations:– Vipin Kumar’s papers (link to be added shotrly)
– http://charm.cs.uiuc.edu go to publications, 93-13
• A few more papers will be posted soon..
Adaptive grainsize control:
• Strategy 1: cut-off depth – (but must have an estimate of the size of the subtree)
• Strategy 2: stack splitting– Each PE maintains a stack of nodes of the tree
– If my stack is empty, “steal” half the stack of someone else
– Which part of the stack? Top? Bottom?
Adaptive grainsize control:
• Strategy 3:– Objects (tree nodes) decide whether to make children
available for other processors by calling a function in the runtime
– runtime monitors the size of its Queue (stack), and possibly size of other processor’s queues
Adaptive grainsize control:
• Strategy 3: Objects decide how big they want to grow– Monitor execution time (number of tree nodes evaluated)
– If the number is above a threshold:
• Fire some of my nodes as independent objects to be mapped somewhere else
– Problem: you sometimes get a “Mother” object that just keeps firing lots of smaller objects
• Solution: above the threshold, split the rest of the work into two objects and fire them off.
Dynamic load balancing
• Centralized:– maintain top levels of tree on one processor
– serve requests for work on demand
• Variation:– hierarchical:
Fully Distributed strategies
• Keep track of neighbors• Diffusion/Gradient model• Neighborhood averaging• What topology to use:
– Machine’s
– Hypercube
– Denser?
Gradient model
• Misnomer:too broad a name• Actual strategy:
– Processor arranged in a topology
• (may be virtual, but the original purpose was to use real)
– Each processor (tries to) maintain an estimate of how far it is from an idle processor
– Idle processors have a distance of 0
– Other processors: periodically send their numbers to nbrs
• My distance = 1 + min(neighbor’s distance)
– If my distance is more than a neighbor’s, send some work to it
• Work will “flow” towards idle processor
Neighborhood averaging
• Assume virtual topology• Periodically send my own load (queuesize) to neighbors• Each processor:
– Calculate avergae load of its neighborhood
– If I am above average, send pieces of work to underloaded neighbors so as to equalize them
• Estimate of work:– Assume the same for each unit
– Use better estimate if known
Randomized strategies
• Random initial assignment:– As work is created, assign it to a PE
– Problems: no way to correct errors
– Each message goes across processor: communication overhead
• Random demand:– If I am idle, ask a randomly selected processor for work
– If I get a demand, send half of my nodes to the requestor
– Good theoretical properties
– In practice: somewhat high overhead
Using Global Average
• Carry out a periodic global averaging to decide the average load on all processors
• If I am above average:– send work “away”
– Alternatively, get a vector of overload via global averaging, and figure out whom to send what work
Using Global Loads
• Idea:– For even a moderately large number of processors, collecting
a vector of load on each PE is not much more expensive than the collecting the total (per message cost dominates)
– How can we use this vector without creating serial bottleneck?
– Each processor know if it is overloaded compared with avg.
• Also knows which Pes are underloaded
• But need an algorithm that allows each processor to decide whom to send work to without global coordination, beyond getting the vector
– Insight: everyone has the same vector
– Also, assumption: there are sufficient fine-grained work pieces
Global vector scheme: contd
• Global algorithm: if we were able to make the decision centrally:
Receiver = nextUnderLoaded(0);
For (I=0, I<P; I++) {
if (load[I] > average) {
assign excess work to receiver, advancing receiver to the next as needed;
}
To make a distributed algorithm run the same algorithm on each processor! Except ignore any reassignment that doesn’t involve me.
Tree structured computations
• Examples: – Divide-and-conquer
– State-space search:
– Game-tree search
– Bidirectional search
– Branch-and-bound
• Issues:– Grainsize control
– Dynamic Load Balancing
– Prioritization
State Space Search
• Definition:– start state, operators, goal-state (implicit/explicit)
– Either search for goal state or for a path leading to one
• If we are looking for all solutions:– same as divide and conquer, except no backward
communication
• Search for any solution: – Use the same algorithm as above?
– Problems: inconsistent and not monotonically increasing speedups,
State Space Search
• Using priorities:– bitvector priorities
– Let root have 0 prio
– Prio of child:
– parent + my rank
p01 p02p03
p
Effect of Prioritization
• Let us consider shared memory machines for simplicity:– Search directed to left part of the tree
– Memory usage: let B be branching factor of tree, D its depth:
• O(D*B + P) nodes in the queue at a time
• With stack: O(D*P*B)
– Consistent and monotonic speedups
done
unexplored
active
Ideal Stack-stealing Prioritized
Need prioritized load balancing
• On non shared memory machines?• Centralized solution:
– Memory bottleneck too!
• Fully distributed solutions:• Hierarchical solution:
– Token idea
Bidirectional Search
• Goal state is explicitly known and operators can be inverted– Sequential:
– Parallel?
Game tree search
• Tricky problem:• alpha beta, negamax