Algorithms and Algorithmics Part I - IDA > HomeTDDD63/info/slides/TDDD63_Fo10_4sl.pdf · Algorithms and Algorithmics Part I Algorithms ... An expression is deﬁned in terms of ...

Algorithms and Algorithmics

Part IAlgorithms

Computability TheoryComputational Complexity Theory

Cryptography

What is an Algorithm?• Let’s focus on gastronomy!

• Given a kitchen containing

• a baker, supply of ingredients, baking utensils, an oven, etc.

• Baking a cake is

• a process that is carried out from the ingredients, by the baker, with the aid of the oven, and most significantly, according to the recipe!

• Ingredients: input to the process

• Cake: output of the process

• Recipe: the algorithm (an abstract entity)

• the formal written version: program (Software)

• Utensils, oven, baker: (Hardware)

Baking a CakeIngredients

Cake

RecipeOven

UtensilsBaker

EffectiveProcess

HardwareSoftware

Input

OutputComputational

Thinking!

A real recipe: Mousseline au chocolat

Melt chocolate and 2 tablespoons water in double boiler. When meltedstir in powdered sugar; add butter bit by bit. Set aside. Beat egg yolks until thick and lemon-colored, about 5 minutes. Gently fold in chocolate. Reheat

slightly to melt chocolate, if necessary. Stir in Rum and vanilla. Beat egg whitesuntil foamy. Beat in 2 tablespoons sugar; beat until stiff peaks form.Gently

fold whites into chocolate-yolk mixture. Pour into individual serving dishes. Chill at least 4 hours. Serve with whipped cream, if desired.

8 ounces of semisweet chocolate pieces, 2 tablespoons of water, 1/4 cup of powdered sugar, 6 separated eggs, etc.

6-8 servings of Mousseline au chocolat

Level of Detail / Basic Actions

“Stir in powdered sugar” Basic action because it is assumed the hardware knows how to do it.

Level of detail is very important when it comes to an algorithm’s elementary instructions.

• Should be tailored to fit the hardware’s capabilities• Should also be appropriate for the comprehension level of the potential reader or user of the algorithm.

Ambiguities or Fuzzy Phrases not allowedin specifying instructions to a computer!

“about 5 minutes”“served with whipped cream if desired”Is it the serving or addition of WC

that is dependent on person’s desires?

Short Algorithms for Long Processes

Given a list of personnel records, one for each employee in a certain company, each containing the employee’s name, personal details, and salary, find the total

sum of all salaries of all employees

1. Make a note of the number 0;2. proceed through the list, adding each employee’s salary to the noted number;3. having reached the end of the list, produce the noted number as output.

Problem:

Algorithm:

The text of the algorithm is short and fixed, yet the process it invokes varies with the length of the input list and can be very, very long!

A fixed algorithm prescribing many processes of varying lengths, the precise duration and nature of the process being dependenton the inputs to the algorithm. In fact, the potential choice of inputs may be infinite!

We have:

The Algorithmic Problem and its Solution

Characterization of all legal inputs

Characterization of desired outputs as a function of inputs

Algorithmic Problem

and

Algorithmic Solution

Any legal input

Desired Output

Algorithm A

The problem and its solution• An algorithmic problem consists of:

• a characterization of a legal, possibly infinite, collection of potential input sets, and

• a specification of the desired outputs as a function of the inputs

• Assumption

• a description of allowed basic actions or a hardware configuration, together with its built-in basic actions is provided in advance.

• Each of the actions must be carried out in a finite amount of time.

• An algorithmic solution consists of:

• an algorithm, composed of elementary instructions prescribing actions from the agreed upon set

• the algorithm, when executed for any legal input set, solves the problem, producing the output as required.

Solving Algorithmic Problems

• Algorithmic problems can be incredibly complex and take years of work to solve satisfactorily

• Many problems do not admit satisfactory solutions

• Many problems do not admit solutions at all!

• For many problems, the status as far as good algorithmic solutions is still unknown.

Course Book Definition

An algorithm is an ordered set of unambiguous finitelyexecutable steps that defines a terminating process

An algorithm must contain control instructions that determine the sequence or order in which the executable

steps or instructions are carried out

Control Flow or Ordering in Program Execution

• Sequencing: Statements are to be executed (or expressions evaluated) in a certain specified order.

• Selection: Depending on some runtime condition, a choice is to be made among two or more statements or expressions

• The most common selection constructs are if and case (switch)

• Iteration: A given fragment of code is executed repeatedly, either a certain number of times, or until a certain run-time condition is true.

• Iteration constructs include for/do, while, and repeat loops

• Procedural Abstraction: A potentially complex collection of control constructs ( a subroutine) is encapsulated in a way that allows it to be treated as a single unit, usually subject to parameterization

• Recursion: An expression is defined in terms of (simpler versions of ) itself, either directly or indirectly

• the computational model requires a stack on which to save information about partially evaluated instances of the expression.


• Concurrency: Two or more program fragments are to be executed/evaluated “at the same time”

• either in parallel on separate processors,

• or interleaved on a single processor in a way that achieves the same effect.

• Exception handling and speculation: A program fragment is executed optimistically, on the assumption that the expected condition will be true. If the condition turns out to be false

• execution branches to a handler that executes in place of the remainder of the protected fragment ( in the case of exception handling), or

• in place of the entire protected fragment (in the case of speculation)

• In this case, the language implementation must be able to rollback or undo any visible effects of the protected code.

• Nondeterminacy: The ordering or choice among expressions or statements is deliberately left unspecified, implying that any alternative will lead to correct results.


• These 8 principle categories cover all the control-flow constructs and mechanisms found in most programming languages

• Though the syntax and semantics details vary from language to language, thinking in these terms will make it easier to

• learn new languages

• evaluate the tradeoffs among languages

• and design and reason about algorithms in a language-independent way!

Computational Problems

• An “algorithm” is an informal intuitive concept, but associated with it is the concept of a computational process.

• Many attempts have been made to provide formal definitions of what the most general mathematical notion of a computational process is.

• In other words, a formal equivalent of the informal notion of an algorithm.

• For example:

• A Turing machine that halts on all inputs is the precise formal notion corresponding to the intuitive notion of an algorithm.

• The area that studies such issues is called Computability Theory

• It asks the fundamental question:

• What can be computed by a computational device?


Computational Problems that CANbe solved by an algorithm

Computational Problems thatCANNOT be solved by any algorithm

Computability Theory

CANNOT be solved in any practicalsense due to excessive time/space

requirements


CAN be solved in a practicalsense with reasonable time/space

requirements

Computational Complexity Theory

Tractable

Intractable Class NP

Class P


Some History and Context

Leibniz: Step Reckoner

•A universal artificial mathematical language•All human knowledge could be represented in this language•Calculational rules would reveal all logical relationships among these propositions•Machines would be capable of carrying out such calculations

Calculus Ratiocinator

Let usCalculate!

Leibniz (1646 - 1716)

Early use of binary system(not in step reckoner)

In 1673, Leibniz built the first true four-function calculator. His unique, drum-shaped gears formed the basis of many successful calculator designs for the next 275 years, an unbroken record for a single underlying calculator mechanism.

Leibniz Step Reckoner

AdditionSubtractionMultiplicationSquare root extraction

Hilbert (1862-1943)

23rd Problem: Does there exist an algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem)

2nd Problem: Establish the consistency of the axioms for the arithmetic of real numbers

1st Problem: Decide the truth of Cantor’s Continum Hypothesis

24 problemsfor the

20th century

Boole (1815-1864)

Turned “Logic” into Algebra

Classes and terms (thoughts) could be manipulated using algebraic rules resulting in valid inferences

Logical deduction couldbe developed as a branch

of mathematics

Subsumed Aristotle’s syllogismsIn essence Leibniz’

calculus rationator (lite)

Boolean Logic

Frege (1848-1925)

The 1st fully developed system of logic encompassing all of the deductive

reasoning in ordinary mathematics.

•1st example of formal artificial language with formal syntax•logical inference as purely mechanical operations (rules of inference)

Begriffsschrift “Concept Script”

Intention was to show that all of mathematics could be based on logic! (Logicism)

Gödel (1906-1978)

Showed the completenessof 1st-order logic in his PhD Thesis

The logic of PM (and consequently PA)

is incomplete

There are true sentences not provable

within the logical system

As a consequence, the consistency of the

mathematics of the real numbers can not be proven within any

system as strong as PA

Develop metamathematics inside a formal logical system by encoding propositions as numbers

Hilbert’s 2nd Problem

Turing (1912-1954)

23rd Problem: Does there exist an algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem)

Turing wanted to disprove the 23rd problem

To do this, he had to come up with a formal characterization of the generic process underlying

the computation of an algorithm

He then showed that there were functions that were not effectively computable including the

Entscheidungsproblem!

As a byproduct he found a mathematical model of an all-purpose computing machine!

“Effective” ComputationWhat is it, that humans do, when humans “compute”?

What actions are undertaken by a human agent when manipulating a finite set of symbols according to fixed rules?

Strip this undertaking of any particular notion of mental activity, thought, imagination, creativity....

What are the essentials of human computation itself?

Observe!: This is 1930. There are no computers in existence!

Turing Machine• Blueprint for a Turing Machine (1st of 2 parts):

• Architectural Part:

• An infinitely long tape divided into squares stretching in two directions

• there will never be a shortage of space

• A finite set of symbols

• 0 and 1 (can use any finite alphabet)

• executes absolutely simple, primitive operations on these symbols.

• A reading head

• Can scan squares one at a time

• Can move to the left or right one square at a time, or not at all

• Can inscribe, or erase symbols on squares

• A finite set of states

• The states correspond to various finite configurations of a Turing machine’s reading head.

Turing Machine• Blueprint for a Turing Machine (2nd of 2 parts):

• Procedural Part:

• The behavior of a Turing machine is controlled by a finite series of instructions

• Each instruction is governed by

• the state of the reading head and

• the current symbol being scanned

• The Turing machine understands only two commands:

• The first instructs the machine what to write or erase

• The second, whether it is to move to the left or the right by one square

• Each command given to a Turing machine has 4 parts:

• If the machine is in this state, and it is scanning that symbol, then it must write this symbol or that symbol, and move one square to the left or one square to the right, or not at all.

Effective Computability: Turing Machine

• finite alphabet of symbols• finite set of states• infinite tape marked off with squareseach of which is capable of carrying a single symbol• mobile sensing-and-writing head that can travel along the tape one square at a time• state-transition diagram containing the instructions that cause changes to take place at each step

A Turing machine is computing a function: a mapping from input to output!

In the Course Book

A Turing Machine for Incrementing a Value

Uses a state transition table instead of finite state diagram

An Example: Palindromes

move-a

move-b

test-a

test-b

mark YES NO return

a/a, R b/b, R

a/#, R

b/#, R a/a, R b/b, R

#/#, L

#/#, L

#/#, L

#/#, L a/#, L

b/#, L

a/a, L b/b, L

#/#, L

#/#, R

b/b, L

a/a, L

a b b a # a b a a # Examples:

Can generalize to any finite alphabetCan encode decision problems (recall hilbert’s 23rd problem!)

# #

Functions

• Function: A correspondence between a collection of possible input values and a collection of possible output values so that each possible input is assigned a single output

• Computing a Function: Determining the output value associated with a given set of input values

• Effectively Computable Function: Output values can be determined algorithmically from input values.

• Turing Computable Function: Any function that can be computed from a Turing machine in the following manner:

• Place input value in binary form on a tape

• Run the Turing machine associated with the function until it halts

• Read the output value from the tape

On Algorithms and Computable Functions

• The notion of an algorithm is a human artifact

• It stems from the human concept of effectively getting something done (mentally).

• In essence, it may be synonymous with anything a human mind can effectively do.

• Controversial!

• The notion of computability is a formally defined mathematical concept.

• Turing Computability is one such formal definition.

• For any computable function, there is a Turing machine which effectively computes that function.

Church-Turing ThesisTuring machines are capable of solving any effectively computable algorithmic problem! Put differently, any algorithmic problem for which we can find an algorithm that can be programmed in some programming language, any language, running on some computer, any computer, even one that has not yet been built, and even one requiring unbounded amounts of time and memory space for ever larger inputs, is also solvable by a Turing machine!

Turing Machine

(Partial) Recursive Functions: Gödel,KleeneLambda Calculus: ChurchPost Production Systems: PostTuring Machines: TuringUnlimited Register Machines: Cutland

A Truly Astonishing and Remarkable (formally well-grounded )Claim!

Scheme, C++, C, Ada, Phython, Java, Ruby, C#,

LISP ,.......

Pentium, Multicore, Big Blue, super computer, ...

Uncomputable Functions?• Are there well-defined functions that are not effectively computable?

• If so, how would we prove such a thing?

• Turing did just that!

• First he required a formal mathematical notion of effective computability.

• Turing Machine

• His well-defined problem was that asked by Hilbert:

• Does there exist an algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers? (Entscheidungsproblem)

• He formally proved that the function (decision problem) associated with this question is not effectively computable. There is no Turing machine that can compute this function!

• The fact that there are functions which are not computable, places limitations on what a computer can do.

• The result may in fact place limitations on what the human mind can do also....but this is a controversial (but important) topic.

An Unsolvable Problem: The Halting Problem

R X

Does R halt on X?

Yes No

If R(X) terminates

If R(X) diverges

A Program Potential Input

Is there a Turing machine which computes this function?

Before answering, let’s define a bare bones language for writing programs!

Universal Programming Language

• It would be tedious to represent algorithms at such a low level of abstraction using Turing Machines.

• A Universal Programming Language is one in which a solution to any computable function can be expressed

• Most popular programming languages are universal in this sense.

• Turing Completeness

• A system of data-manipulation rules (such as a computer’s instruction set, a programming language, etc.) is said to be

• Turing complete or computationally universal if it can be used to simulate any single-taped Turing machine.

Bare Bones Language

• Variables:

• refer to bit patterns interpreted as non-negative integers

• Three Assignment Statements:

• clear name

• incr name

• decr name

• One control structure:

• while name not 0 do: <statement sequence> end;

Examples

Multiply X x Y

Swap variable values

Universality of Bare Bones

• Has been shown formally to be Turing Complete

• Relation to a Turing machine:

• Input: Initial values of all variables

• Output: Final values of all variables

• The Bare Bones program itself directs the computation of a function

Back to the Halting Problem

R

Encode the program R in bare bones as a bit sequence

We define a program as self-terminating if executingthe program with all its variables initialized to the

program’s encoded representation leads to a terminating process

Testing a Program for Self-Termination

Observe! Any program is either self-terminating or not self-terminating

Examples: Testing a Program for Self-Termination

While X not 0 do;

incr X; end:

Clear X While X not 0

do; incr X; end:

Binding X to the program’s encoded representation results

in the program not halting

By definition it isnot self-terminating

Binding X to the program’s encoded representation results

in the program halting

By definition it isself-terminating

Self-referential Programs

Proving the Unsolvability of the Halting Problem

First: Propose the existence of a program that,

given any encodedversion of a program

will halt with variableX equal to 1 if theinput represents aself-terminating

program, or with X equal to 0 otherwise.

ProposedProgram

Assume:

Then: If such a program exists, we could modify it by

adding awhile-endstructure

ProposedProgram

to producea new

programwhile Xnot 0 do;end;

Now: If this new program wereself-terminating and

execution would reach this point

with X equal to 1 ProposedProgram

we started it withits own encoding

as its input

while Xnot 0 do;end;

so execution would become trapped in this loop forever

i.e., if the new program is self-terminating,then it is not self-terminating!!!

Case 1

However: If this new program werenot self-terminating and

execution would reach this point

with X equal to 0 ProposedProgram

we started it withits own encoding

as its input

while Xnot 0 do;end;

so execution of this loop would be

skipped

i.e., if the new program is not self-terminating,

then it is self-terminating!!!

and executionwould halt;

Case 2

Consequently:

the existence ofa new program

ProposedProgram

The existence ofthe proposed

program

while Xnot 0 do;

that is neitherself-terminatingnor non self- terminating

So the existence of the proposedprogram is impossible!

wouldlead to

ProposedProgram

Contradiction

Negate the assumption

Another Proof

• Using Diagonalization

Universal Turing Machine• One consequence of the Church-Turing thesis is the existence of universal

algorithms.

• A universal algorithm A has the ability to act like any algorithm whatsoever

• Input: the description of any algorithm P

• Input: any legal input value to the algorithm, X

• It runs, or simulates P(X), halting if P(X) halts with the outputs that P(X) would normally give had P been running on X

• A computer or interpreter is very much like a universal algorithm

• It is presented with a program P in a language with legal input X, to P; It executes P on X and outputs the result.

• Stored Program Concept (program as data)

• Universal though means that it is insensitive to the choice of language or machine, which a computer or interpreter is not.

Universal Turing Machine: A blueprint for a general purpose computer

algorithm A

program P input X

Run P on X

P implements A; is written in language L2

Universal program U, written in language L1; simulates the effect of a program in L2 on an input

LISP: Eval Programs as data

Formal mathematical

abstraction of a general computing

device

Ace Computer

Observe: U can simulate any computer in existenceObserve: L1 can simulate algorithms in its own language L1

Let both L1 and L2 be the language of Turing Machines!

Conclusions• Gödel’s result:

• Any formal system expressive enough to axiomatize arithmetic is and always will be incomplete!

• There are mathematical formulas which are true but can not be shown within the formal system to be true!

• One such formula is the consistency of the axioms of arithmetic themselves.

• Turing’s result:

• There is no algorithm that can determine the truth or falsity of any logical proposition in a system of logic that is powerful enough to represent the natural numbers (Entscheidungsproblem)

• As a side effect:

• Turing proposed a formal model of effective computability

• Turing defined a mathematical abstraction of a computing device

• Both Gödel and Turing formally showed the limits to formal systems and the degree of mechanization possible.

• If mind is mechanism, what does this say about human intelligence?

• What does this say about the goals of artificial intelligence?



Computational Problems thatCANNOT be solved by any algorithm


CANNOT be solved in any practicalsense due to excessive time/space

requirements


CAN be solved in a practicalsense with reasonable time/space

requirements

Computational Complexity Theory

Tractable

Intractable Class NP

Class P

Algorithms and Algorithmics Part I - IDA > HomeTDDD63/info/slides/TDDD63_Fo10_4sl.pdf · Algorithms and Algorithmics Part I Algorithms ... An expression is deﬁned in terms of ...

Documents