Class 28: Entropy

Class 28: Entropy

cs1120 Fall 2011David Evans28 October 2011

A giant python swallows an alligator in Everglades National Park, Fla..Paul Zenk/PBS Nature “Invasion of the Giant Pythons” (yesterday’s New York Times)

2

Plan

Story so Far: Liberal Arts RecapReversing in PythonShannon’s Information Theory: EntropyReturning Quiz 3, PS5, etc.

Course RoadmapSynthesis

Analysis

Ch 2

: Lan

guag

e

Ch 3

: Pro

gram

min

g

Ch 4

: Pro

cedu

res

Ch 5

: Dat

aCh

6: M

achi

nes

Ch 7

: Cos

t

Ch 8

: Sor

ting

and

Sear

chin

g

PS5,

Ch

9: S

tate

PS6,

Ch

10: O

bjec

ts

Intr

acta

bilit

y

PS7,

Ch

11: I

nter

pret

ers

Ch 1

2: C

ompu

tabi

lity PS

8, 9

You

are

here

Ch 1

: Com

putin

g

cs1120 Main Topics

Language: How to describe information processes by defining procedures (Chapters 3, 4, 5, 9, 10)Programming with procedures, lists, recursion (PS1-4)Programming with state (PS5), objects (PS6), languages (PS7)

Logic: How to predict properties about information processes (Chapter 6, 7, 8)Predicting how running time grows with input sizeAre there problems which can’t be solved by algorithms? (Ch 12)

Machines: How to efficiently implement information processes (Chapter 6) (not much on this)Machines, Digital LogicHow to implement a Scheme interpreter (Ch 11, PS7)

From Lecture 1:

• Grammar: study of meaning in written expression

• Rhetoric: comprehension of verbal and written discourse

• Logic: argumentative discourse for discovering truth

• Arithmetic: understanding numbers

• Geometry: quantification of space

• Music: number in time

• Astronomy

BNF, RTN, rules of evaluation for meaning

Interfaces between components (PS6),

evaluation rules

Rules of evaluation, if, recursive definitions

adding with logic,representing numbers

Curves as procedures, fractals (PS3)

Ada, GEB Chapter

Nothing!

Triv

ium

Qua

driv

ium

But…read the Neil DeGrasse Tyson essay!

7

Lists (Tuples) in Python

Scheme (list)> (define p (list 1 2 3))> (car p)1> (cdr p)(2 3)> (cdr (cdr (cdr p)))null

Python (tuple)>>> p = (1, 2, 3)>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()

Python Shortcuts

>>> p = (1, 2, 3)>>> p[2]3>>> p[3:]()>>> p[:2](1, 2)

list[i] Evaluates to ith (counting from 0) element of tuple/list

Runs in (approximately) constant time!

9

Mutable Lists (Lists) in Python

Scheme (mutable list)> (define p (mlist 1 2 3))> (mcar p)1> (mcdr p)(2 3)> (mcdr (mcdr (mcdr p)))null

Python [List]>>> p = [1, 2, 3]>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()

Using Mutable Lists

>>> p = [1, 2, 3]>>> p[0]1>>> p[0] = 2>>> p[2, 2, 3]>>> p.append(4)>>> p[2, 2, 3, 4]

Note: this is different from mappend!It take a single element, not a list.

Scheme Application:(<verb> <operands>)Python procedure call:<verb>(<operand>)Method call:<object>.<verb>(<parameters>)

Trick Question

>>> p = (1, 2, 3)>>> p[0] = 2Traceback (most recent call last): File "<pyshell#23>", line 1, in <module> p[0] = 2TypeError: 'tuple' object does not support item assignment

Reversing a Mutable List

1

p:

2 3

Scheme:

1 2 3

p:

Python:

mlist-reverse!

(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))

Literal Translation(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))

Literal Translation

>>> p=[1,2,3]>>> sreverse(p)>>> p[3, 2, 1]

def sreverse(p): if len(p) == 1: return else: rest = p[1:] first = p[0] sreverse(rest) p[0] = rest[0] p[1:] = rest[1:] p.append(first)

(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))

This is correct, but no sane Python programmer would do it this way!

Python for

Statement ::= for Varible in Expression: Block def gaussSum (n): sum = 0 for i in range(1, n+1): sum = sum + i return sum

(define (loop index result test update proc) (if (test index) (loop (update index) (proc index result) test update proc) result)) (define (gauss-sum n) (loop 1 0 (lambda (i) (<= i n)) (lambda (i) (+ i 1)) (lambda (i sum) (+ i sum))))

Reverse in Python

1 2 3

p:

Note: there is a built-in list method, p.reverse().

def reverse(p): for i in range(0, len(p) >> 1): p[i], p[len(p) – 1 - i] = p[len(p) – 1 - i], p[i]

int(len(p) / 2) len(p) >> 1

19

Information Theory

20

Entropy

H = -pi log2 pi

21

(Ideal) Coin Toss

H = -pi log2 pi

flickr cc:Daniel Dale

22

Nucleotide

N {A, C, G, T}Intuition: need two bits to distinguish four possible values

00 = A, 01 = C, 10 = G, 11 = T

H = -pi log2 pi

23

24

Claude Shannon’s Juggling W. C. Fields

Real Coin Toss?

25

Real Coin TossAccording to Susan Holmes, 51% of the time, a coin will land same way it started. [link]

H = -pi log2 pi

26

27

Are Randall Munroe’s entropy estimates correct?Left for course blog discussion.

28

Class Sorting using “Odette”

Assuming alphabet randomly distributed: O is 15th out of 26 letters in alphabet

0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...

= 0.988…

29

Class Sorting using “Odette”

The real class: 41 out of 49 students have names before “Odette”!

0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...

-((41/49) * log_2 (41/49) + (7/49) * log_2 (7/49)) = 0.6162235892609252

30

Better Class SortingFront of Class

Hannah – Li-Chang

Anthony - Casey

crm Caroline - Gregory

mdc Michael - Yuan

Charge

Office hours:Peter – right nowSunday, 1-6pmEveryone should be making progress on PS6

Monday: Trick-or-Treat Protocols!Inheritance