“Why Design Information is Required to Find Improbable Complex Targets”

“Why Design Information is Required to Find Improbable

Complex Targets” Robert J. Marks IIDistinguished Professor Of Electrical and Computer [email protected]

Abstract

Engineers use models of science to improve quality of life. Computational intelligence is such a useful engineering tool. It can create unexpected, insightful and clever results. Consequently, an image is often painted of computational intelligence as a free source of information. Although fast computers performing search do add information to a design, the needed information to solve even moderately sized problems is beyond the computational ability of the closed universe. Assumptions concerning the solution must be included. For targeted search, the requirement for added information is well known. The need has been popularized in the last decade by the No Free Lunch theorems. Using classic information theory, we show the added information for searches can, indeed, be measured. The total information available prior to search is determined by application of Bernoulli's principle of insufficient reason. The added information measures the information provided by the evolutionary program towards achieving the available information. Some recently proposed evolutionary models are shown, surprisingly, to offer negative added information to the design process and therefore perform worse than random sampling.

Menu

The Fossil Record Computer Search Evolutionary Search Bernoulli’s Principle of Insufficient Reason Blind Search No Free Lunch Theorems Active Information Examples of Active Information Final Thoughts

The Fossil Record “In general, it is usually impossible or impracticable to test hypotheses about

evolution in a particular species by the deliberate setting up of controlled experiments with living organisms of that species. We can attempt to partially to get around this difficulty by constructing [computer] models representing the evolutionary system we wish to study, and use these to test at least the theoretical validity of our ideas.” J. L. Crosby (~1965)

“The only way to see evolution in action is to make computer models” because “in real time these changes take aeons, and experiment is impossible.” Heinz Pagels (~1985)

David Fogel, The Fossil Record, IEEE Press

“The Darwinian idea that evolution takes place by random hereditary changes and selection has from the beginning been handicapped by the fact that no proper test has been found to decide whether such evolution was possible and how it would develop under controlled conditions.” Nils Barricelli 1962

Menu


What is Computer Search?Searching for a Good Pancake Recipe

How long do we cook on side one?

How long do we cook on side two?

Ten possibilities on each side: 100 combinations.

Searching for a Good Pancake Recipe

Determining Fitness: Tim the Taster

FITNESS: RANKS EACH PANCAKE ON A SCALE OF ONE TO TEN

DESIGN CRITERION: 9 or above is an acceptable pancake.

FITNESS LANDSCAPE: The plot of all 100 fitness values


TARGET: 9 combinations meet the design criterionPROBABILITY OF SELECTION: p = 9/100 = 0.09

LANDSCAPE, TARGET and p ARE INITIALLY UNKNOWN


ADD ONE MORE UNKNOWN: HEAT SETTING On THE STOVE

Now there are 1000 possible combinations.

THE CURSE OF DIMENSIONALITY!


Cooking the whole pancake

1. Pancake Mix: 1 to 10 cups

2. Eggs: 0 to 9

3. Milk: 1 to 10 cups

4. Water: 0 to 9 cupsTHE CURSE OF DIMENSIONALITY!

5. Salt: 1 to 10 pinches

6. Butter for Skillet: 0 to 9 pads

7. First Side Timing: 10 times

8. Second Side: 10 possibilities

9: Stove Setting: 10 Possibilities

Now there are ONE BILLION possible

combinations.

Implicit Targets (Convergence)


Now there are ONE BILLION possible

combinations.

CRAFTED FITNESS LANDSCAPE:e.g. Revisit two parameter cooking example. Stove is electric and blows a fuse if either side of the pancake is cooked for more than a minute and half.


Let a computer do it...


Computer Search

Using software simulation, find a recipe that meets design criteria.

There are MANY search procedures

100 Pancake Recipes Per Second

Defining “Impossible”

Menu


What is Evolutionary Search?

Simulation of Darwinian Evolution on a Computer

A set of possible solutions

Computer Model

How good is each solution?

Keep a set of the best

solutions

Duplicate, Mutate &

Crossover

Survival of the fittest

Mutation

Next generation

Search in Engineering Design

Yagi-Uda antenna (1954) Can we do better? Engineers…

1. Create a parameterized model

2. Establish a measure design’s fitness

3. Search the N-D parameter space

Parameters that give results better than Yagi-Uda

T

Space of all parameters.

Designed by Evolutionary Search at NASA

http://ic.arc.nasa.gov/projects/esg/research/antenna.htm

Random Search: You are told “Yes & No” (Success and no success)

TargetTarget

Blind Search

UHF

http://www.youtube.com/watch?v=50uW0b7tWiM

Directed Search: Information is given to you...

TargetTarget e.g.

•Warmer!

•Steepest Descent

•Conjugate Gradient Descent

•Interval Halving

Menu


Bernoulli’s Principle of Insufficient Reason

“...in the absence of any prior knowledge, we must

assume that the events have equal probability”

Jakob Bernoulli, ``Ars Conjectandi'' (``The Art of Conjecturing), (1713).

e.g. A Lottery. If you have one of a million tickets, your chance of winning is

the same as everyone else:

000,000,11

p


Laplace agreed:“[When] we have no reason to

believe any particular case should happen in preference to any other”

Arne Fisher, Charlotte Dickson, and William Bonynge, Mathematical Theory Of Probabilities & Its Applications To Frequency Curves & Statistical Methods, Macmillan (1922)

Menu


Blind Searches... Monkeys at a typewriter…

27 keys

Apply Bernoulli's principle of insufficient reason

“in the absence of any prior knowledge, we

must assume that the events have equal

probability

Jakob Bernoulli, ``Ars Conjectandi'' (``The Art of Conjecturing), (1713).

Information Theoretic Equivalent: Maximum Entropy

(A Good Optimization Assumption)

How Does Moore’s Law Help in Blind Search?

Computer today searches for a target of B =10000 bits in a year.

Double the speed.

Faster Computer searches for a target of B + 1 =10001 bits in a year.

Converting Mass to Computing Power

Minimum energy for an irreversible bit Von Neumann-Landaurer limit =

ln(2) k T = 1.15 x 10 -23 joulesMass of Universe ~ 1053 kg. Convert

all the mass in the universe to energy (E=mc2) , we could generate 7.83 x 1092 Bits

1. Assuming background radiation of 2.76 degrees Kelvin

Define Impossible:

Anything requiring 1093 bits or more.

Seth Lloyd, “Computational Capacity of the Universe”, Physical Review Letters 88(23) (2002): 7901-7904.

Seth Lloyd in Physical Review, says 10120

Expected number

= NL

How Long a Phrase? Target

L

IN THE BEGINNING ... EARTH

JFD SDKA ASS SA ... KSLLS KASFSDA SASSF A ... JDASF J ASDFASD ASDFD ... ASFDG JASKLF SADFAS D ... ASSDF

.

.

.

IN THE BEGINNING ... EARTH

4.7549 bits per letter for 26 letters and a

space

How Long a Phrase from the Universe?

p=N-L

7.83 x 1092 bits = NL log2 NL

For N = 27,p=N-L

L = 63 characters

ppB log

Number of bits expected for a random search

How Long a Phrase from the Multiverse?

Using Lloyd’s 10120 bits for 101000 universes

gives 836 letter search.

Does Quantum Computing Help?

Quantum computing reduces search time by a square root.

L. K. Grover, “A fast quantum mechanical algorithm for data search”,

Proc. ACM Symp. Theory Computing, 1996, pp. 212--219.

Targeted Search

Probability Search Space

Pr()=1

TargetT

t

Pr[tT ] =

||||

T

Fitness

Each point in the parameter space has a fitness. The problem of the search is finding a good enough fitness.

Acceptable solutions

T

Search Algorithms

Steepest Ascent

Exhaustive

Newton-Rapheson

Levenberg-Marquardt

Tabu Search

Simulated Annealing

Particle Swarm Search

Evolutionary Approaches

Problem: In order to work better than average, each algorithm implicitly

assumes something about the search space and/or location of the

target.

Menu


No Free Lunch Theorem

With no knowledge of With no knowledge of where the target is at and where the target is at and no knowledge about the no knowledge about the

fitness surface, one fitness surface, one search performs, on search performs, on

average, as good as any average, as good as any another.another.

No Free Lunch Theorem Made EZ

x

yFind the value of x that maximimizes the fitness, y.

Nothing is known about the fitness, y.

Another illustration...52 Cards.Find the Ace of Spades

Cullen Schaffer (1994)A Conservation Law for Generalization Performance

“About half of the people in the audience to which my work was directed told me that my result was completely obvious and common knowledge–which is perfectly fair. Of course, the other half argued just as strongly that the result wasn’t true.”

(Private Correspondence)

Quotes on the need for added information for targeted search …

1. “…unless you can make prior assumptions about the ... [problems] you are working on, then no search strategy, no matter how sophisticated, can be expected to perform better than any other” Yu-Chi Ho and D.L. Pepyne, (2001).

2. No free lunch theorems “indicate the importance of incorporating problem-specific knowledge into the behavior of the [optimization or search] algorithm.” David Wolpert & William G. Macready (1997).

1. ``Simple explanantion of the No Free Lunch Theorem", Proceedings of the 40th IEEE Conference on Decision and Control, Orlando, Florida, 2. "No free lunch theorems for optimization", IEEE Trans. Evolutionary Computation 1(1): 67-82 (1997).

Therefore...

Nothing works better, on the average, than random search.

For a search algorithm like evolutionary search to work, we require active information.

Evolutionary Search...

Evolutionary search is “able to adapt solutions to new problems and do not rely on explicit human knowledge.” David Fogel*

* (emphasis added D. Fogel, Review of “Computational Intelligence: Imitating Life,” IEEE Trans. on Neural Networks, vol. 6, pp.1562-1565, 1995.

BUT, the dominoes of an evolutionary program must be set up before the are knocked down. Recent results (NFL) dictate there

must be implicitly added information in the crafting of an

evolution program.

Evolutionary Computing

e.g. setting up a search requires formulation of a “fitness function” or a “penalty function.”

Michael Healy, an early pioneer in applied search algorithms, called himself a “penalty function artist.”

Can a computer program generate more information than it is given

If a search algorithm does not obey the NFL

theorem, it “is like a perpetual motion machine

- conservation of generalization

performance precludes it.” Cullen Schaffer (1994)

3. Cullen Schaffer, 1994. “A conservation law for generalization performance,”in Proc. Eleventh International Conference on Machine Learning, H. Willian and W. Cohen, San Francisco: Morgan Kaufmann, pp.295-265.

9

6

?

Conservation of Information

A computer can create information no more than an iPod can create

music

Menu The Fossil Record Computer Search Evolutionary Search Bernoulli’s Principle of Insufficient Reason Blind Search No Free Lunch TheoremsActive Information Examples of Active Information Final Thoughts

Conservation of Information

“The [computing] machine does not create any new information, but it performs a very valuable transformation of known information.”

Leon Brillouin, Science and Information Theory (Academic Press, New York, 1956).

Shannon Information Axioms Small probability events should have more

information than large probabilities.– “the nice person” (common words lower info)– “philanthropist” (less used more information)

Information from two disjoint events should add– “engineer” Information I1

– “stuttering” Information I2

– “stuttering engineer” Information I1 + I2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

1

2

3

4

5

6

7

Shannon Information

)(log2 pI

p

I

Measuring probability in

coin flipsBITS !!!!

Targeted Search

||||]Pr[

TTtp

Probability Search Space

TargetT

Bernoulli's Principle of Insufficient Reason = Maximum

Entropy Assumption

ninformatio available

log2

pI

Endogenous Information

TargetT

||||log

log

2

2

T

pI

This is all of the information we can get from the search. We can get no more.

Endogenous Information: Interval Halving

bits 42log161log 4

22

I

Tno = 0

no = 0

yes = 1

yes = 1

4 bits of information = 0011

This is a “perfect search”

.pq

,pq

Search Probability of Success.Choose a search algorithm...

.ppS

qLet

be the probability of success of an evolutionary search.

From NFL, on the average, if there is no active information:

If information has been added

.1q

Active Information Definition

q

pI 2log

IpI

1log2

= all of the available information

1. For a “perfect search”,

reference

Checks:

Active Information

q

pI 2logChecks:

0log2

p

pI

= no active information

2. For a “blind query”, .pq

Active Information

TargetT

No active information: UNIFORM

Active Information: MORE PROBABLE

AREAS

Active Information can be NEGATIVE

Menu


EXAMPLES of ACTIVE INFORMATION

What is the source of active information?

There must be active information in some way guide this process.

Where does it come from? Or is there something we are not considering? ...

The universe is not old enough nor big enough to allow the evolution of

complex life.

Active Information in Random Searches...

QI 2log

1. Active information is not a function of the size of the space or the probability of success – but only the number of queries.

2. There is a diminishing return. Two queries gives one bit of added information. Four queries gives two bits. Sixteen queries gives four bits, 256 gives 8 bits, etc.

Active Information in Partitioned Search...

METHINKS*IT*IS*LIKE*A*WEASEL

XEHDASDSDDTTWSW*QITE*RIPOCFL XERXPLEE*ETSXSR*IZAW**LPAEWL MEQWASKL*RTPLSWKIRDOU*VPASRL


yada yada yada

Dawkins required

43 iterations

2. Active Information in Partitioned Search...


QI Q 2log

For random search

QLI Q 2log

Hints amplify the added information by a factor of L.

For Partitioned Search

N= 27 characters, 26 in alphabet

For perfect search using partitioning information, set

QLII Q 2log

bits 133 II Q

Since L=28, if we set

iterations 27Qit follows that...

ACTIVE INFORMATION in Partitioned Search...

Iterations From

Information

Comparison


L= 28 characters, 27 in alphabet

iterations 27Q

Reality: For Partitioned Search

iterations 101.1973 40Q

For Random Search

There is a lot of active information!

Active Information in Partitioned Search...

Domain knowledge can be applied differently resulting in varying degrees of active information

The knowledge used in The knowledge used in partitioned search can partitioned search can be used to find all the be used to find all the

letters and spaces in an letters and spaces in an arbitrarily large library arbitrarily large library using only 26 queries.using only 26 queries.

Weasel Wear on EvoInfo.orgGeorge Montañez

1.1. Specify a target of bits of length Specify a target of bits of length LL2.2. Initiate a string of random bits.Initiate a string of random bits.3.3. Form two children with mutation (bit flip) probability of Form two children with mutation (bit flip) probability of

. . 4.4. Find the best fit of the two children. Kill the parent and Find the best fit of the two children. Kill the parent and

weak child. If there is a tie between the kids, flip a coin.weak child. If there is a tie between the kids, flip a coin.5.5. Go to Step 3 and repeat.Go to Step 3 and repeat.

(WLOG, assume target is all ones)(WLOG, assume target is all ones)

2. Single Agent Mutation (MacKay)Single Agent Mutation (MacKay)

2. Single Agent Mutation (MacKay)

2. Active FOO InformationFOO = frequency of occurrence E 11.1607% M 3.0129%

A 8.4966% H 3.0034%

R 7.5809% G 2.4705%

I 7.5448% B 2.0720%

O 7.1635% F 1.8121%

T 6.9509% Y 1.7779%

N 6.6544% W 1.2899%

S 5.7351% K 1.1016%

L 5.4893% V 1.0074%

C 4.5388% X 0.2902%

U 3.6308% Z 0.2722%

D 3.3844% J 0.1965%

P 3.1671% Q 0.1962%

Information of nth Letter

)log( nn pI Average information=Entropy

)log( nn

n ppH

Concise Oxford Dictionary (9th edition, 1995)

English Alphabet Entropy

English– Uniform

– FOO

– Added information

characterper bits 76.4)27log(

H

characterper bits 36.3

)log(

nn

n ppH

characterper bits40.1I

Kullback-Leibler Distance between FOO and Maximum Entropy

Asymptotic Equapartition Theorem

Target

A FOO structuring of a long message restricts search to a subspace uniform in .

T

FOO Subspace

For a message with L characters with alphabet of N letters...

elements LNelements 2 LLI N

Asymptotic Equapartition Theorem

For King James Bible using FOO, the active information is

I+ = 6.169 MB.Available Information

I = 16.717 MB Can we add MORE information?

digraphstrigraphs

ev

The NFL theorem has been useful to address the "sometimes outrageous claims that had been made of specific optimization algorithms“

S. Christensen and F. Oppacher, "What can we learn from No Free Lunch? A First Attempt to Characterize the Concept of a Searchable,“ Proceedings of the Genetic and Evolutionary Computation (2001).


Search for the Search ? The endogenous information for

the search increases exponentially with respect to the active information

nats IeI

First Possibility: The Wrong Problem:Nonteleological Evolution

Pure Darwinism claims evolution does not have a target and is therefore “nonteleological”.

One Problem: All evolution emulations are teleological.

Second Problem: Simon Conway Morris’s “Convergence” (Implicit teleology.)

Morris’s Convergence & Basener’s Ceiling

Simon Conway Morris explores the evidence demonstrating life’s almost eerie ability to navigate to a single solution, repeatedly.

William Basener shows dynamic processes,

including evolution, hit a ceiling.

CONVERGENCE

Second Possibility:

Parallel Universes: More Matter

Parallel Universes– (Quantum) The big bang was a wave function one of

whose states was our universe. – (String Theory) The big bang was the product of the

chance touching of multidimensional branes. Problem: The multiverses are still not big enough

to explain life with no external information. Problem: These are pure conjecture and require a

faith in scientific materialism.

Third Possibility:

Panspermia: More Time

Panspermia is a theory supported by Sir Francis Crick, Nobel Laureate for discovering DNA.

•Panspermia claims life was planted on earth by aliens. Directed panspermia, supported by Crick, says this was done on purpose.

•Problem: This is pure conjecture and requires a faith in science fiction. It only displaces the problem.

Fourth Possibility:

The Universe had a Creator

This is taught in the Bible of the Jew, Christian and Muslim. From Genesis:

1. “In the beginning God created the heavens and the earth.”

2. “Now the earth was formless and empty, darkness was over the surface of the deep, and the Spirit of God was hovering over the waters.” No Information

Fourth Possibility:

The Universe was Created by God

And God said, "Let there be light," ... "Let the water under the sky be gathered to one place, and let dry ground appear." "Let the land produce vegetation." "Let the water teem with living creatures, and let birds fly..." "Let the land produce living creatures according to their kinds..." "Let us make man in our image, in our likeness, and let them rule over the fish of the sea and the birds of the air, over the livestock, over all the earth, and over all the creatures that move along the ground."

Information & Structure

"Torture numbers, and they'll confess to anything." Gregg EasterbrookGregg Easterbrook

EVOLUTION SIMULATIONS

Schneider’s EV

Equivalent to inverting a perceptron:

24 weights on

[-511, 512]

Bias

[-511, 512]

Fixed binding site locations.

error

String of 131 nucleotides

The Function...

The Results...

The Illusion...The Illusion...

Finding a specific stream of 131 bits has a Finding a specific stream of 131 bits has a probability of...probability of...

Endogenous InformationEndogenous Information

II = 131 bits= 131 bits

p p = 2= 2-131-131= 3.67 = 3.67 xx 10 10-38-38

N N = 704 generations x 64 genomes = 704 generations x 64 genomes per generation = 45,056 queriesper generation = 45,056 queries

for a perfect searchfor a perfect search

Active information rate = 3 Active information rate = 3 millibits per querymillibits per query

Source of active information: perceptron Source of active information: perceptron structure and error measure.structure and error measure.

HAMMING ORACLE:

Lots of Active Information

ev

Active Information

Solution in 131 Queries Guaranteed

EV Ware on EvoInfo.orgGeorge Montañez

AVIDA - DOVER Trialhttp://ncse.com/files/pub/legal/kitzmiller/trial_transcripts/2005_0928_day3_am.pdf

http://ncse.com/files/pub/legal/kitzmiller/trial_transcripts/2005_0928_day3_am.pdf

AVIDA

NAND logic

AVIDA: Active InformationStair Step Active Information

AVIDA: Active InformationStair Step Active Information

THE TARGET

(EQU)

This logic is minimal, but not

unique

AVIDA: Active InformationStair Step Active InformationOne Population:

3600 Organisms Evolved Using At Most About 10.8 Billion Instructions

Same Run, Using No Stepping Stones...

AVIDA:

Active Information

Endogenous Information of AVIDA 85 instructions per digital organism, 1420 runs of 10.8 billion instructions per run using importance sampling

Winston Ewert, William A. Dembski and R.J. Marks II, “Evolutionary Synthesis of Nand Logic: Dissecting a Digital Organism,'' Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics, October 11-

14, 2009, San Antonio, Texas, USA.

bits40I(20 nucleotides)

AVIDA:

Active Information



Active information introduced by AVIDA, including stair steps. 353 runs each. Normalized active information per

instruction is in nanobits.

AVIDA uses Stair Step Information. Remove a stair, the active info

AVIDA:

Ratchet Active Information



Using AVIDA Resources Better.

RATCHET: One organism. Perform mutation. If lower fitness, repeat. If the same or better, keep the mutation.

AVIDA:

Ratchet Active Information



Using AVIDA Resources Better.

Removing Obvious Deleterious Instructions

Confession...

Minivida on EvoInfo.org

Menu


AVIDA:

Final Thoughts



Summary of Points...

Targeted Evolutionary Computing Schaffer’s Perpetual Motion Machine for Information

Active information can be measured analytically or through simulation.

Some models of simulated evolution have negative active information. Random chance works better.

Active information should be reported in all published models of simulated targeted evolution.


Summary of Points...

As seen in ev, AVIDA and the WEASEL,

EVOLUTIONARY ALGORITHMS CREATE

NO INFORMATION The info is in the algorithm and can be

mined more efficiently using other algorithms

Next Steps…

Coevolution

Information Measures

EvoInfo.org


John Maynard Keynes disagreed...– Bertrand’s Paradox– Social Science Applications– Distribution of Reciprocals

)(frequency 1 h)(wavelengt

pdf pdf


Bertrand’s Paradox– Due to Definition of “random”, not Bernoulli R.J. Marks II, Handbook of Fourier Analysis and Its Applications, Oxford University Press, (2009).

Social Science Applications– Incomplete or Vague Definition of Search Space

Distribution of Reciprocals– Not true for discrete domains...


Distribution of Reciprocals– Not true for discrete domains and some-to-many mappings

If the probability of success in is p, and in is p, then the probability that p > p is one half.

William A. Dembski and R.J. Marks II, “Bernoulli's Principle of Insufficient Reason and Conservation of Information in Computer Search,'' Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics, October 11-14, 2009, San Antonio, Texas, USA.

Finis

Finis

Finis

“Why Design Information is Required to Find Improbable Complex Targets”

Documents

design information

available information

needed information

free source of information

negative added information

total information available

classic information

computer models