NI Introduction

8/17/2019 NI Introduction

http://slidepdf.com/reader/full/ni-introduction 1/76

BC-0504Natureza da Informação

David Correa Martins [email protected]

Introduction and some bits about theHistory of Information Theory



• Prof. David Correa Martins Junior (CMCC)• Email: [email protected]• Email subject: [N! "ubject

• Delta #uildin$ ("%o #ernardo)& room '

• *esearc+ interests• Pattern reco$nition a,,lied to #ioinformatics and "-stems

#iolo$-.



Course Goas

/oal: 0o ,resent t+e foundations of t+e nformation Nature

o

.e.& t+e main conce,ts about information re!resentation and"uantification.



#rere"uisites

No formal ,rere1uisites 2i$+ sc+ool ,rere1uisites "et t+eor- Combinatorial anal-sis Probabilit- and statistics 3o$arit+m4s ,ro,erties 0e5t inter,retation



$ssessments % Grade

$ssessments

Midterm E5am: 50& & November rd

6inal E5am: 50&& December 7t+

*eta8e E5am: December 9t+ 'e!ace the smaest of the t(o e)am *rades +!en to anyone (ho (ants to try to im!ro,e the fina*rade

ub.ect/ $ content



$ssessment % 1ina *rade

$ssessment 7& avera$e ; < =&> avera$e < 7& ; # ?&> avera$e < =&> ; C &> avera$e < ?&> ; D <vera$e &> ; 6



Tidia

+tt,:AAtidiaBae.ufabc.edu.br

3o$ in usin$ -our institutionalcredentials. 0+e ,rofessor illre$ister t+e students



Course materia•Course materia/ 2i be made a,aiabe at

Tidia 3htt!/tidia-aeufabcedubr6 on thesection 7'e!osit8rio 3'esources69

• Cass sides• :)ercise ists and ans(ers

•I;#+'T$NT/ study the materia <B:1+':< the cass

• Brin* your o(n "uestions to so,e them andfaciitate earnin*

• =+> are the main res!onsibe for your earnin*• The content is e)tensi,e/ study throu*hout thecourse 3not .ust before the e)ams6

http://tidia-ae.ufabc.edu.br/

http://tidia-ae.ufabc.edu.br/



:stimated chrono*ram

Content

22, 28/09 Course introduction, Data, Information, Knowledge,History of Information Theory, Semiótica

29/09, 06/10 Enumeration systems, base conversion, it, ooleanalgebra

13, 20/10 Codes, source coding, digital!analogic conversion

26/10, 27/10 Information Theory03/11 Midterm exam



Crono*rama estimado

Conteúdo09 e 10/11 Efficient coding and data com"ression

17 e 23/11 Error detection and correction

24/11 e01/12 #eural coding and genetic code

07/12 E$ercises and review class08/12 Final exam

15/12 Retake exam



*ecommended readin$:Decodin$ t+e niverse. C+arles "eife ('>>?) Pen$uin #oo8s.(Este livro, embora de caráter divulgativo, é o que captura melhor o foco multidisciplinar dadisciplina. Apresenta um roteiro que pode ser preenchido abordando cada tópico com uma

profundidade maior . Eiste somente um eemplar na biblioteca. Eiste vers!o em pdf"

"istemas Di$itais: fundamentos e a,licaFes. 0+omas 3.6lo-d. (Essencial para eplicar os sistemas de numera#!o, código de $amming, a

convers!o A%& e &%A e o teorema de amostragem de 'hannon)*quist. A álgebra booleanatambém está bem eplicada. Eistem vários eemplares na biblioteca"

"istemas de Comunica%o <nalG$icos e Di$itais. "imon2a-8in ('>>H). (A +eoria da nforma#!o, compress!o de dados e detec#!o%corre#!o

de erros"

<n ntroduction to nformation t+eor-. "-mbols& si$nals and

Noise. Jo+n *. Pierce. Dover. (Embora um pouco ultrapassado, apresentaos desafios encontrados pelos pioneiros da teoria de informa#!o. -ostra as solu#esencontradas para estes desafios"



Informationa $)is

0+e advances of science and tec+nolo$- aremulti,l-in$ our ca,acit- to collect& ,rocess& ,roduceand use information leadin$ it to ne levels neverreac+ed before. t brin$s:

ne o,,ortunities ne social 1uestions more science and tec+nolo$- advances& fosterin$ a

virtuous c-cle



Informationa $)is

1undamentas and !rocesses/ nformation Nature: +at is nformation and +o cane re,resent or measure itI

nformation Processin$: mani,ulation and informationtreatment& under bot+ +uman and com,uter as,ects(,rocessin$)

nformation communication: transmission anddistribution of t+e information and its im,act



Naturea danforma%o

0ransforma% oda

nforma%o

Comunica%oda

nforma%o

(conceitual)0eGrica

(su,orte)0ecnolo$ia

(utilia%o)2umana

<bstrata Concreta "ocial

➢K #it➢Entro,ia➢<nalo$. L Di$ital➢Ca,ac. "+annon

➢0. nforma%o

➢0. Com,uta%o➢Kr$. Com,utadores➢Com,ress%o Dados➢Cri,to$rafia➢Com,le5idade

➢0. ComunicaFes➢Ca,acidade canal➢Canal $aussiano➢nforma%o$entica➢Codifica%o

➢"mbolos e"inais➢*udo➢Proc. EstocOsticos

➢Krdem e Desordem➢Caos

➢"entidosA,erce,%o➢Co$ni%o e <%o➢nteli$ncia➢Conscincia

➢MemGria

➢<,rendiado➢Crebro➢Con+ecimento➢*a%oAEmo%o➢*edes "ociais

➢3in$ua$em2umana➢nternet➢"oc. nforma%o➢Econ. nforma%o➢*e$ula%oAQtica

➢Proc. "inais➢0ransformadas➢Pro$rama%o➢Minera%o Dados➢0radu%o

➢"ist.ComunicaFes➢*edes e 0rOfe$o➢EletrRnica➢6otRnica➢Novas 0ecnolo$ias



$mostras





S+at t+is course is reall- aboutI

% 0+e course Naturea da nforma%os+os +o t+e nformation is ,resent inour lives.

% Not just at t+e tec+nolo$ic level(telecommunications& com,uters&internet)

% #ut also at t+e biolo$ic (DN<& brain) and+uman (lan$ua$e& semiotics) levels



&ata, nformation and /no0ledge



Data

% Data: basic element that quantifies orqualifies something

& Do not carry any intrinsic meaning byitself & Initial perception about the subject

& Identified by symbolic characteristics

& Ex.: screw 6 weights 6! grams –



Data

% " set of facts about the world#% $hey are usually quantifiable#

% %an be easily captured and stored incomputational de&ices#% Do not carry meaning and neither

can be used in judgements#% 'o actions can(t be made based on just data.



Data

% $ypes & "lphanumeric & )roduct code* price* amount* etc.

& Images & )hotographs

& "udio

& +ideo & etc...



Data

asic facts

E'g'( sho""ing at the su"ermar)et

*+ -,./eer 012.3

-+ .,//Cheese.404-

2+ 5,//6uice.2504

-+ 3,//utter 2.*4.

0+ -,./7il) 0*2.*

3+ 5,./Chocolate-35-2

Qantit!"ndi#idal $ri%e&e'%ri(tionCode



Information% Information: interpreted and

contextuali,ed data & -equires data interpretation & Is the result of processed data* it(s useful for

decisionma/ing.% It answers questions li/e 0who1* 0what1* 0where1 and

0when1.

& Ex.: screw 23 is the hea&iest of the group

&ata 1 processing 2 information



Information

% " conjugated data set that hasrele&ance and purpose#

% %an be transformed by humans and

be submitted to judgment & 'emiotics 4quality of information5 & Information analysis to produce

/nowledge% %onstitute base for action 4decision

ma/ing5



Information

Transformed data with aditional information

+ 24,//33Total+ 1,//*+ -,./eer 012.3

+ .,//-+ .,//Cheese.404-

+ -3,//2+ 5,//6uice.2504

+ 3,//-+ 3,//utter 2.*4.

+ -3,//0+ -,./7il) 0*2.*

+ 4,//3+ 5,./Chocolate-35-2

)otal $ri%e Qantit!"ndi#idal $ri%e&e'%ri(tionCode

&air* products sold



Information

Can hel" to increase the "rofit

+ 24,//33Total+ 1,//*+ -,./Cerve8a012.3

+ .,//-+ .,//9uei8o.404-

+ -3,//2+ 5,//Suco.2504

+ 3,//-+ 3,//7anteiga2.*4.

+ -3,//0+ -,./:eite0*2.*

+ 4,//3+ 5,./Chocolate-35-2

$re*o )otalQantidade$re*o "ndi#idal&e'%ri*+oCdi-o

3everage products sold



nowledge% nowledge: ability to create a

model and suggest actions ordecisions to ta/e & %omprehension* analysis and synthesis

start at the /nowledge le&el & -equired le&el to ta/e smart decisions

% -esponde quest7es do tipo 0como1

& Ex.: 8hea&ier( definition* comparison rules*procedures 4models5

nformation 1 processing 2 4no0ledge(processing 2 eperience, training, etc."



nowledge

% " structured and organi,ed set ofinformation#

% -equires intelligent human judgement

& 'emiotics: assignment of meaning toinformation#

% 9ffers rules* comparisons*deductions and implications



nowledge

% Ex. supermar/et: & $he items and ; are frequently bought

together & <il/ and chocolate & 'o put them near each other



=ierarchy

+his hierarch* represents the sequence bet0een the elements, but

also the volume of each one



Data x Information x

nowledge





Data x Information x

nowledge

•&ata are ra0 numbers or labels.• A grid segmentation could produce information (e.g. more or less

populous segments" about these ra0 data.•5ules that describe 0hen a segment is red or green, for eample, are

part of 4no0ledge



6hat is information7

• Latin etimology

• Different perspecti&es

8 >iology 8 ?inguistics 8 )hysics 8 %omputer 'cience

What is the relationship between information andcommunication? 8 Communication: information exchange between the

actors

from in "into", 1 formare 9to form, shape:

(into form, delineate"



• Information is inversely proportional to

the probability of occurrence of a fact 8 =igher probability facts are less informati&e 8 Examples: 8 $he first time we listen a disc* it brings new musical

/nowledge. >ut after listen to it many times* we can

predict the next accords* so this disc brings no moreinformation to us.

8 @e all can predict the missing 091 at the word 0?A+E1*because it(s a common word. 'o it is unnecessary towrite that character at that position.

8 Boal: to quantify the information. $he meaningor 0quality1 is irrele&ant a priori.

8 $he information quality or meaning is ofinterest to semiotics

Information $heory



$istor* of nformation +heor*



;laude E. 'hannon (=>?@@" is the 9father: of nformation +heor*. $is

boo4 9+he -athematical +heor* of ;ommunication: 0as published in ==

e seu livro publicado em ==

http://images.google.com.br/imgres?imgurl=http://www.maths.sussex.ac.uk/Staff/JWPH/PROF/Shannon.jpg&imgrefurl=http://www.maths.sussex.ac.uk/Staff/JWPH/PROF/index.html&usg=__BAshUFz98ZsXqbA-KYJRtEWyZVk=&h=326&w=268&sz=7&hl=pt-BR&start=1&tbnid=eXf-JVI3YMG0pM:&tbnh=118&tbnw=97&prev=/images%3Fq%3DClaude%2BE.%2BShannon%26gbv%3D2%26hl%3Dpt-BR%26sa%3DG




• Em =B, 'hannon published the paper 9 A Mathematical

Theory of Communication:, republished as boo4 net

*ear

• 3efore him, isolated 0or4s 0ent step b* step to0ards ageneral theor* of communication

• )o0ada*s the +heor* of ;ommunication (or nformation

+heor*" is a huge research area 0ith man* boo4s ands*mposiums about the subCect

Shannon





• nformation +heor* is a broad theor* that involves a lot of

math.

• Bit is the fundamental measure to quantif* information

• At DA3; there is a specific course about nformation

+heor*

Information Theory





• nformation theor* allo0s us toF

• 'a* ho0 man* informative bits could be sent per second through

a certain communication channel

• -easure the rate in 0hich a source can produce information

• 'a* ho0 to represent efficiently, or encode, messages to be

transmitted through some channel .

• 'a* ho0 0e can avoid transmission errors

Information Theory




$he pilars of Information $heory

C $hermodynamic studies

C

%ryptography and computers built atthe @orld @ar II

C $ransmission technology* starting with<orse code and telephony



Entropy and thermodynamics



>olt,mann* the father ofthermodynamics and information theory

• Entropy equation inscribed at>olt,mann(s tomb

• $he bases of Information $heory are pro&ided by

thermodynamic studies >ut >olt,mann didn(t /now

that his studies would be thebases of Information $heory



Industrial -e&olution

• F6G. @attin&ented the

steammachine• -esearch to

impro&e the

machineefficiency



%arnot(s =eat engine



%arnot(s =eat engine

%oldsource

wor/=eat

Engine

=ot source

G

G?

=ot source

%oldsource

=eat'ource

G

G?

wor

Refrigeration



-udolf %laussius

• It is impossible tostop the tendencyto thermodynamicequilibrium of theuni&erse 4H6!5

• In other words* 0the

entropy alwaystends to increase1

mposs e to stop t e



mposs e to stop t etendency to thermodynamic

equilibrium %old source

=ot source

@or/=eat

Engine

G

G?

=eatsource

G

G?

@or/=eat

Engine

G

G?



II $hermodynamics law

• Entropy measures thethermodynamics equilibrium

• <ore entropy means more

equilibrium• $he second thermodynamics law

says that 0the entropy amount of any

system tends to increase with timeuntil its maximum &alue1



'/ log4@5

• Entropy equationinscribed at >olt,mannJstomb

• "toms of a gas tend to

disperse itself uniformly.• $he uni&erse entropy

always increases

• ?ater in the course wewill see the 0bridge1between the entropies of>olt,mann and 'hannon



%ryptography and

%omputers at @orld @ar II



0"K is short of water1

• $hese wordschanged thecourse of the

war at )acific.• $he

cryptographic

japanese code4LM235 wasdecoded by

americans



0?ets attac/ "KN1

%ommander -ochefort"dmiral ;amamoto

Hets attac4 AI A777 6hat is A777

6ould A be -id0a* sland7



0<idway Island is out of waterN1

%ommander -ochefort "dmiral ;amamoto

-id0a* sland is out of 0aterI A is out of 0aterI

A$AIII got *ouIII



End of @ar

• "mericans waitedfor the japanesearri&al

• Kour Lapaneseaircraft carrierswere destroyed.

• $he end of @orld@ar II began



European 0Oboats1



Enigma• $he german "rthur

'cherbius in&ented0Enigma1* a machine

to encrypt messages• x!P differentstates possible

• >rute force would

require: each atomon uni&ersecomputing a trillionof /eys per second*since the beginningof uni&erse

$uring and colleagues bro/e



$uring and colleagues bro/ethe Enigma code

• "ll characters wereswapped at each time

• $he corresponding

symbol to character0K1 was ne&er 0K1 itself

• Osual sentences 4e.g.

the weather is goodtoday5 allowed thedestruction of Oboatsand the end of war.

$h h i l $ i



$he theoretical $uring<achine

• $he machine reads*writes and deletesbits in an endlessstring

• $he $uring <achinehas 0uni&ersal

computability1• I.e. any current

computer is a $uring

<achine



$he allied cryptography helpedto end the @orld @ar II andstarted the Information Era

$uring

contributed tothe end of @orld@ar II



%ryptography

• " simple cryptographic method is toreplace each letter 4character5 by theletter placed M positions forward in

the alphabet• %aesar cipher

• $ry to decode the following message:

Q xgp&q jqlg gu&cxc ow/&q hqtg9 &ento hoje esta&a muito forte

n 2



-edundancy

• Bs TvBnBs dB cBBncBB e dB tBcnBlB$BB BstTBmBltB,lBcBndo Bs nBssBs cB,BcBdBdBs dB

cBlBtBr& trBtBr& $BrBr B BtBlBBrBnfBrmBBes.

9 b i / d f d d



9ur brain ta/es ad&antage of redundancy todecode texts as the following:

• De aorcdo com uma peqsiusa de umauinr&esriddae ignlsea* nRo ipomtra emqaul odrem as ?teras de uma plra&aa

etRso* a Sncia csioa iprotmatne T que apiremria e Stmlia ?teras etejasm no lgaurcrteo. 9 rseto pdoe ser uma bUguana ttaol*que &coV anida pdoe ler sem pobrlmea.

Itso T poqrue nWs nRo lmeos cdaa ?teraisladoa* mas a plra&aa cmoo um tdoo.'ohw de bloa.



9r li/e this one...

• 3$ )QOM! $$9 3-+ P)MP3 )P-P<!3$-P- %9<9 M933P %P>XP%9M'BO KPY- %93P3 <)-339MP"M$3N-)P- M339N M9 %9<X9 3$P+P

<! %9<)?%PD9* <P3 M'$P ?M=P 'OP<M$ +P D%K-PMD9 9 %ZDB9QOP3 PO$9<P$%P<M$* '< )-%'P-)M3P- <O$9* %-$9[ )9D K%P-

>< 9-BO?=939 D339N 'OP %P)P%DPD<-%N )P-P>\M3N



$echnologies forInformation $ransmission



!rigins of the modern

Information Theory+here are mathematical analogies bet0een the

nformation entrop* and the entropies of

+hermod*namic and 'tatistical -echanic, but

the modern nformation +heor* are rooted at the

origins of electronic communication



Telegraph origins

• BJBF 'amuel 3. -orse 0or4ed 0ith Alfred Kail on acode 4no0n toda* as the -orse codeF• Alphabetic characters are represented b* spaces, dots anddashes

•Electronic transmission 0as achieved representing spaces b*current absence, dots b* short currents, and dashes b* longcurrents

• ;ombinations of these s*mbols 0ere associated 0ith characters

• E (the most frequent character in English" 0as associated to a

single dot.

<orse %ode



<orse %ode

The "orse code and



The "orse code andInformation Theory

• mportant questionF• ;ould another mapping (dots, dashes, spaces" impl* to faster

telegraphic transmissions of English tets7

• Ans0erF• Dsing the modern nformation +heor* 0e found that the

transmission rate gain 0ould be about LM at most.

• +his suggests that -orse intuitivel* attac4ed on of the mainproblems addressed b* nformation +heor*.



• Himitations summar*• Himits related to the signal transmission speed

• nterferences (noises"

• $ard to distinguish among man* possible current values

• Himited current intensit* to avoid destruction of the cableinsulation

• A more precise mathematical anal*sis 0as required

Telegraph limitations

ib i h



Contributions to theInformation Theory

• -an* people contributed mathematicall* to the informationtheor* at NN centur*F• Horde /elvin (6illiam +homson"

• Aleander Oraham 3ell (telephone inventor in BPL"

• $enri Qoincaré• Rliver $eaviside

• -ichael Qupin

• O. A. ;ampbell (A++;"

• 3ut the biggest contribution 0as obtained b* Sosephourier



#ourier$s contribution

• ourier based his 0or4s on the sine function

•'ho0ed that an* function (including electric signals"

can be decomposed in a sum of sines 0ith differentamplitudes, phases and frequencies.

% l ff$ d Wi $



%olmogoroff$s and Wiener$scontributions

n the =@s 5ussians and Americans solved

independentl* the problem of estimating the

correct signal from an un4no0n nois* signal.




#inally Shannon

• 'o 0hen 'hannon published his 0or4s in =P,

much 0as alread* done

• n a certain 0a*, he summariTed and producedne0 4no0ledge about all these problems

previousl* studied b* other researchers





Shannon$s contribution

• 3ut 0e could sa* that his great contribution 0as to ans0erthe follo0ing questionsF• $o0 could 0e encode (0ith electric signals" a message from a

source to transmit it as fast as possible through a channel that

introduces noises 0ith a certain pattern7• How fast could 0e transmit certain message through specific

channels without errors7

• All 0e have seen until no0 is part of 0hat 0e call

nformation +heor*. n this course 0e 0ill see introductor*topics about this theor*.




To do before ne&t class

• 'tud* the slides and ans0er the eercise listcorresponding to the first 0ee4 ('emiotics"

•'ection U5epositórioV (5esources" at +idia

• )o need to deliver the list for grading, but it is

fundamental to do the eercises and stud*

before classes

● and consequentl* for the eams

NI Introduction

Documents