ch11n

8/7/2019 ch11n

1/18

Chapter 11Chapter 11

Limitations ofLimitations of

Algorithm PowerAlgorithm Power

Copyright 2007 Pearson Addison-Wesley. All rights reserved.

8/7/2019 ch11n

2/18

11-2Copyright 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin Introduction to the Design & Analysis of Algorithms, 2nded., Ch. 11

Lower BoundsLower Bounds

Lower boundLower bound: an estimate on a minimum amount of work: an estimate on a minimum amount of work

needed to solve a given problemneeded to solve a given problem

Examples:Examples:

number of comparisons needed to find the largest elementnumber of comparisons needed to find the largest element

in a set ofin a set ofnn numbersnumbers

number of comparisons needed to sort an array of sizenumber of comparisons needed to sort an array of size nn

number of comparisons necessary for searching in a sortednumber of comparisons necessary for searching in a sorted

arrayarray

number of multiplications needed to multiply twonumber of multiplications needed to multiply two nn--byby--nn

matricesmatrices

8/7/2019 ch11n

3/18


Lower Bounds (cont.)Lower Bounds (cont.)

Lower bound can beLower bound can be

an exact countan exact count

an efficiency class (an efficiency class (;;))

TightTightlower bound: there exists an algorithm with the samelower bound: there exists an algorithm with the sameefficiency as the lower boundefficiency as the lower bound

ProblemProblem Lower boundLower bound TightnessTightness

sorting (sorting (comparisoncomparison--basedbased)) ;;((nnloglog nn) yes) yes

searching in a sorted arraysearching in a sorted array ;;(log(log nn) yes) yeselement uniquenesselement uniqueness ;;((nnloglog nn) yes) yes

nn--digit integer multiplicationdigit integer multiplication ;;((nn) unknown) unknown

multiplication ofmultiplication ofnn--byby--nn matricesmatrices ;;((nn22) unknown) unknown

8/7/2019 ch11n

4/18

8/7/2019 ch11n

5/18


Trivial Lower BoundsTrivial Lower Bounds

Triviallower boundsTriviallower bounds: based on counting the number of items: based on counting the number of itemsthat must be processed in input and generated as outputthat must be processed in input and generated as output

ExamplesExamples finding max elementfinding max element ---- n steps or n/2 comparisonsn steps or n/2 comparisons

polynomial evaluationpolynomial evaluation

sortingsorting

element uniquenesselement uniqueness

Hamiltonian circuit existenceHamiltonian circuit existence

ConclusionsConclusions may and may not be usefulmay and may not be useful

be careful in deciding how many elementsbe careful in deciding how many elements mustmust be processedbe processed

8/7/2019 ch11n

6/18


Decision TreesDecision Trees

Decision treeDecision tree a convenient model of algorithms involvinga convenient model of algorithms involvingcomparisons in which:comparisons in which:

internal nodes represent comparisonsinternal nodes represent comparisons

leaves represent outcomes (or input cases)leaves represent outcomes (or input cases)

Decision tree for 3Decision tree for 3--element insertion sortelement insertion sort

a < b

b < c a < c

s

s

s

a < c b < c

a < b < c

c < a < b

b < a < c

b < c < a

no yes

abc

abc bac

bcaacb

yes

a < c < b c < b < a

no

8/7/2019 ch11n

7/18


Decision Trees and Sorting AlgorithmsDecision Trees and Sorting Algorithms

Any comparisonAny comparison--based sorting algorithm can be representedbased sorting algorithm can be representedby a decision tree (by a decision tree (for each fixedfor each fixed nn))

Number of leaves (outcomes)Number of leaves (outcomes) uu nn!!

Height of binary tree withHeight of binary tree with nn! leaves! leaves uu loglog22nn!! Minimum number of comparisons in the worst caseMinimum number of comparisons in the worst case uu loglog

22nn!!

for any comparisonfor any comparison--based sorting algorithm,based sorting algorithm, since the longestsince the longestpath represents the worst case and its length is the heightpath represents the worst case and its length is the height

loglog22nn!! }} nn loglog22n (n (by Sterling approximationby Sterling approximation))

This lower bound is tight (mergesort or heapsort)This lower bound is tight (mergesort or heapsort)

Ex. Prove that 5 (or 7) comparisons are necessary and sufficient for

sorting 4 keys (or 5 keys, respectively).

8/7/2019 ch11n

8/18


Adversary ArgumentsAdversary Arguments

Adversary argumentAdversary argument: Its a game between the adversary and the (unknown) algorithm.: Its a game between the adversary and the (unknown) algorithm.The adversary has the input and the algorithm asks questions to the adversaryThe adversary has the input and the algorithm asks questions to the adversaryabout the input. The adversary tries to make the algorithm work the hardest byabout the input. The adversary tries to make the algorithm work the hardest byadjusting the inputadjusting the input (consistently).(consistently). It wins the game after the lower bound timeIt wins the game after the lower bound time(lower bound proven) if it is able to come up with two different inputs.(lower bound proven) if it is able to come up with two different inputs.

Example 1: Guessing a number between 1 andExample 1: Guessing a number between 1 and nn using yes/no questionsusing yes/no questions

(Is it larger than(Is it larger than xx?)?)

Adversary: Puts the number in a larger of the two subsets generated by last questionAdversary: Puts the number in a larger of the two subsets generated by last question

Example 2: Merging two sorted lists of sizeExample 2: Merging two sorted lists of size nn

aa11

8/7/2019 ch11n

9/18


Lower Bounds by Problem ReductionLower Bounds by Problem Reduction

Fact: If problemFact: If problem QQ can be reduced to problemcan be reduced to problem PP, then, then QQ is at leastis at least

as easy asas easy as PP, or equivalent,, or equivalent, PPisis at least as hard asat least as hard as QQ..

Reduction fromReduction from QQ toto PP: Design an algorithm for: Design an algorithm for QQ using an algorithmusing an algorithm

forfor PPas a subroutine.as a subroutine.

Idea: If problemIdea: If problem PPis at least as hard asis at least as hard as problemproblem QQ, then a lower, then a lower

bound forbound for QQis also a lower bound foris also a lower bound for P.P.

Hence, find problemHence, find problem QQwith a known lower boundwith a known lower bound that canthat can

bebe reducedtoreducedto problemproblem PPin question.in question.

Example:Example: PPis finding MST foris finding MST for nn points in Cartesian plane, andpoints in Cartesian plane, andQQis element uniqueness problem (known to be inis element uniqueness problem (known to be in ;;((nnloglognn))))

Reduction from Q to P: Given a set X= {x1, , xn} of numbers (i.e. an

instance of the uniqueness problem), we form an instance ofMST in the

Cartesian plane: Y= {(0,x1), , (0,xn)}. Then, from an MST for Ywe

can easily (i.e. in linear time) determine if the elements in Xare unique.

8/7/2019 ch11n

10/18


Classifying Problem ComplexityClassifying Problem Complexity

Is the problemIs the problem tractabletractable,, i.e., is there a polynomiali.e., is there a polynomial--time (O(time (O(pp((nn))))algorithm that solves it?algorithm that solves it?

Possible answers:Possible answers:

yes (give example polynomial time algorithms)yes (give example polynomial time algorithms)

nono

because its been proved that no algorithm exists at allbecause its been proved that no algorithm exists at all

(e.g., Turings(e.g., Turings haltingproblemhaltingproblem))

because its been be proved that any algorithm for it wouldbecause its been be proved that any algorithm for it would

require exponential timerequire exponential time

unknown.unknown. How to classify their (relative) complexity using reduction?How to classify their (relative) complexity using reduction?

8/7/2019 ch11n

11/18


Problem Types: Optimization and DecisionProblem Types: Optimization and Decision

Optimization problemOptimization problem: find a solution that maximizes or: find a solution that maximizes orminimizes some objective functionminimizes some objective function

Decision problemDecision problem:: answer yes/no to a questionanswer yes/no to a question

Many problems have decision and optimization versions.Many problems have decision and optimization versions.

E.g.: traveling salesman problemE.g.: traveling salesman problem

optimizationoptimization: find Hamiltonian cycle of minimum length: find Hamiltonian cycle of minimum length

decisiondecision: find Hamiltonian cycle of length: find Hamiltonian cycle of length ee LL

Decision problems are more convenient for formal investigation ofDecision problems are more convenient for formal investigation oftheir complexity.their complexity.

8/7/2019 ch11n

12/18


ClassClass PP

PP: the class of decision problems that are solvable in O(: the class of decision problems that are solvable in O(pp((nn)) time,)) time,wherewhere pp((nn) is a polynomial of problems input size) is a polynomial of problems input size nn

Examples:Examples:

searchingsearching

element uniquenesselement uniqueness

graph connectivitygraph connectivity

graph acyclicitygraph acyclicity

primality testing (finally proved in 2002)primality testing (finally proved in 2002)

8/7/2019 ch11n

13/18


ClassClass NPNP

NPNP((nondeterministic polynomialnondeterministic polynomial): class of decision problems whose): class of decision problems whoseproposed solutions can be verified in polynomial time = solvableproposed solutions can be verified in polynomial time = solvableby aby a nondeterministicnondeterministic polynomialalgorithmpolynomialalgorithm

AA nondeterministic polynomialalgorithmnondeterministic polynomialalgorithm is an abstract twois an abstract two--stagestage

procedure that:procedure that:

generates agenerates a solutionsolution of the problem (on some input) byof the problem (on some input) by guessingguessing

checks whether this solution is correct in polynomial timechecks whether this solution is correct in polynomial time

By definition, it solves the problem if its capable of generating andBy definition, it solves the problem if its capable of generating and

verifying a solution on one of its triesverifying a solution on one of its tries

Why this definition?Why this definition?

led to development of the rich theory called computationalled to development of the rich theory called computationalcomplexitycomplexity

8/7/2019 ch11n

14/18


Example: CNF satisfiabilityExample: CNF satisfiability

Problem: Is a boolean expression in its conjunctive normalProblem: Is a boolean expression in its conjunctive normalform (CNF) satisfiable, i.e., are there values of itsform (CNF) satisfiable, i.e., are there values of itsvariables that make it true?variables that make it true?

This problem is inThis problem is in NPNP. Nondeterministic algorithm:. Nondeterministic algorithm:

Guess a truth assignmentGuess a truth assignment Substitute the values into the CNF formula to see if itSubstitute the values into the CNF formula to see if it

evaluates to trueevaluates to true

Example:Example: ((A |A | BB || CC)) && ((AA || BB)) && ((B |B | D |D | EE)) && ((D | ED | E))

Truth assignments:Truth assignments:

A B C D EA B C D E

0 0 0 0 00 0 0 0 0

. . .. . .

1 1 1 1 11 1 1 1 1

Checking phase:Checking phase: OO((nn))

8/7/2019 ch11n

15/18


What other problems are inWhat other problems are in NPNP??

Hamiltonian circuit existenceHamiltonian circuit existence Partition problem: Is it possible to partition a set ofPartition problem: Is it possible to partition a set ofnn

integers into two disjoint subsets with the same sum?integers into two disjoint subsets with the same sum?

Decision versions of TSP, knapsack problem, graphDecision versions of TSP, knapsack problem, graphcoloring, and many other combinatorial optimizationcoloring, and many other combinatorial optimizationproblems.problems.

All the problems inAll the problems in PPcan also be solved in this manner (butcan also be solved in this manner (butno guessing is necessary), so we have:no guessing is necessary), so we have:

PP NPNP

Big (million dollar) question:Big (million dollar) question: PP== NPNP??

8/7/2019 ch11n

16/18


NPNP--Complete ProblemsComplete Problems

A decision problemA decision problem DD isis NPNP--completecomplete if it is as hard as anyif it is as hard as any

problem inproblem in NPNP, i.e.,, i.e.,

DDis inis in NPNP

every problem inevery problem in NPNPisis polynomialpolynomial--time reducibletime reducible toto DD

Cooks theorem (1971): CNFCooks theorem (1971): CNF--sat issat is NPNP--completecomplete

NP-complete

problem

NPproblems

8/7/2019 ch11n

17/18


NPNP--Complete Problems (cont.)Complete Problems (cont.)

OtherOther NPNP--complete problems obtained through polynomialcomplete problems obtained through polynomial--

time reductions from a knowntime reductions from a known NPNP--complete problemcomplete problem

Examples: TSP, knapsack, partition, graphExamples: TSP, knapsack, partition, graph--coloring andcoloring and

hundreds of other problems of combinatorial naturehundreds of other problems of combinatorial nature

known

NP-complete

pro

lem

NPpro lems

candidate

for NP -

completeness

8/7/2019 ch11n

18/18


PP == NPNP?? Dilemma RevisitedDilemma Revisited

PP == NPNPwould imply that every problem inwould imply that every problem in NP,NP, including allincluding allNPNP--complete problems,complete problems, could be solved in polynomial timecould be solved in polynomial time

If a polynomialIf a polynomial--time algorithm for just onetime algorithm for just one NPNP--completecompleteproblem is discovered, then every problem inproblem is discovered, then every problem in NPNPcan becan besolved in polynomial time,solved in polynomial time, i.ei.e.. PP == NPNP

Most but not all researchers believe thatMost but not all researchers believe that PP{{ NPNP,, i.e.i.e. PPis ais aproper subset ofproper subset ofNP.NP.IfIfPP{{ NP,NP, then the NPthen the NP--completecompleteproblems are not inproblems are not in P,P, although many of them are veryalthough many of them are very

useful in practice.useful in practice.

NP-complete

pro

lem

NP pro lems

ch11n

Documents