Ee315 probability muqaibel ch1

Chapter 1. Probability

1. Set definitions• Element, Subset, Proper subset, Set, Class, Universal set=S, empty=null set=𝜙.• Countable, uncountable,, finite, infinite.• Disjoint, mutually exclusive.

2. Set operations• Venn diagram, equality, difference, union=sum, intersection=product,

complement.• Algebra of sets (commutative, distributive, associative)• De Morgan’s law• Duality Principle

3. Probability introduced through sets and relative frequency4. Joint and conditional probability5. Independent events6. Combined experiments7. Bernoulli trials8. Summary

1Dr. Ali Muqaibel

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

• •

-

"Ii

- • _ E

•

-

- 7

•

• • • • •

.. - . •

-. ------- ----_ .. -

\0 - .

- _. ---------------------------- -- -

e.",,-

---- - - -

-- - e ...... f \-~) - ILV·U ' -f ~A __ Ss-el . 7 •

I J:.~'j fvI.- .. . " - -- -

. IM.c~-ktk \ ) 1«)10. 'v.le 1"'- ,. w - • - • •

IV\. • -~

, t~vvq' .11401<'6

, "tA. V '-- ) , ,

•

&,~ , . 11..~16 - I '" \ -. • -

-- - -~ _. -----,---------

- =-

CIS ,

- --•

-----"-> , ----

"

I

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

-

.

- -

--- -fj .;... .. ::. ---. ---=~ 3> ,-- - _ c ~_ D, - <,c ,s.._K.5 __ _ •

-

_ _ (Cr, _ D L F _" _. _ - --- ,/ ----------------------, ,

-- -- '"-' _._-

____ r.x-(

____ \u=~~. J6c:us\j<M . ___ _ _ _____ _

sPJ w/J.~ N e1e~do -ifo3-e..r~

~---- --N -------.-------------

- - -- -

- - , - - - • • - -t~

N . ~J c:2 .-. ~Li ,v .::-~

7 '.

~----- - ------------------

--------------------- - --

'----------------~----------- - ... --- - ._- -•

. _---_. _. -- ------------_._.----------------_ .. ----------•

~. . • • •

--------------------- -------- -----,

----_. ---- -------------,-------,--

•

•

•

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

I ,

--~-~'-~-'~. ~. ~~' ~~~~~~' ----------------------------

,

-------- ------------------

:...-------- ~----, -, , ---~--;-----------

_____ a

G .. ---- ,.-.s -01\k :Mk E.,,* b

-- __ U ) ___ _

- ---- -- ------ _._-----

'-r-----

,

--------, ---------- ._----

,

? ,

f\ - Q .,6' <q <....\.c .... £L lOO t;' • . - - •• • ,

~ J b ~ ~.s 0.(' I

< 13 - __ a 'I I ' M - ••

~, S'

B - < I 0 , _._-

--, - ~.' - t.' < d ::- _ "'. , ~ 8 IS :::::- .,(,...,..- ,- , - --, - -- - .. -,

--------- , , - '--- - ~~ " . -~-- - -'~- - - -- . - --,

--'------ - ---------------

- w~ _____ .. _ ...

~ ( . .,..-b ..... 6Y\O-j''-'-_ , . - -- ------ --t...QV _, __ _ - ----

A ,LLA ,=,",""LL_-'. :.. .:

• •

n

'" ---- -

1'I :: I

1'--= ,

_ ..

---- --

•

3 - ---------_.

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

I 2 : ! £ i • 2 2 zi 2

- ' . -'

, ,

- s~_" 'or) -...!.SwL...,;-.:....:. '---;. ,;'_ , ---,-A -

•

,

u u --- -[)g~-.-(3..fJ 8---'------ -----, ---

lJ f) - tLl) ( ~, u.c -

u _ n , U A f} g) (..f , .1\ n c

a !,> t ) C I ~a.I( (I," t~tH __ _ f1 /) --efl UJs) u (

o (1 (\ () =. _ .-8. LA. (I5.LLL..) - A \1 is ~U G_

------------------------, -------------------

----

-- --- ----

----_ . • -

•

\ f\ \) I~ ( ? 0..' ---- - - - - ---------- .. .I-,J

-, .. • • ,

•

,

"

---

---- --•

-- '----- ~.---,-.' .-

/ -_-,---,A,-,,_U~ ___ __ __ ====

,- ,--, -

•

rr--.J.."'"l ~l ? 3 -

-::-" _________________________________________________________ ~ ___ -=·===:'v~~:' _

__ (1 <: • _ --=-~--,,-< S

U A . - n 6r) .,. '- --

_~ _ = ~==== :---=_-IfLO . _= __ &J'Lk

, -

•

------------ -

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

,

• -

t -

•

-

\

I <:;:

7 7 ==

•

I .'

'1 , •

,

,

\

•

•

-------_. ------------------------------------.- ----------------_._,- ----------

-

,

I e ,--, ._--- - ~----- - -

,-------- '--------, " --

-=9 -C~b'v:\.\A.;).V'd .-----------,---

,

•

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

We

•

•

•

p,.~b 1------

• .,

./

i

•

•

, ------------

-- 2 .. , ) ----------------------

"

, •

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

_ akL c:.LC"_ . '-= L-_

_______ --'s ':: 0 < ;y <.. \ Co •

• • , .

- --

~~--------f~-~- -'~ .. ---------------I .N

•

--"-. - ;;..,.......

-

(3 -•

•

- r B)

f

• ..A

(>,,!e .. "' . 1 _____ _

•

" ,. ~-~J __ _

• ..... . 7 •

/

•

~ I

" (

" '. , • --< it 1 ________ --' (

/ -,

,

1-1-/--, ( " t) , 1 ' . ' . - - _ ' _ -~ I '

~~~3~6---------------------------------------~=-~~~~,

. - ,

" U A !!J

~ -, 1,'''',- '[ , •

1.:- , '1 ' ::: I

• -- -- - - ,

I ~

• , I •

Zf f I -

, ,

A 1:)

-

-

•

, ,

-(

• • " , ,

I I • :?-( - . - • -,

= ,a. ;; .. ... _ -- - --/6

~ >:.-' --- -

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

, VI o by;,..,

1 / l ,.

•

I I 1

1 I

- •

. __ =-<~ ~ _____ '-,_. __ :L1- 51

--- '-' _1 -1.) - l2 L?J ..... <2 _ ______ ~ /'Z / Xi) ____ .

- :$ 3 B,u-') _ --UI 4-1- Q -= .. ::.... 11: L2..;:,..<..v _

•

-

5 •

r)

•

-

Dr. Ali Muqaibel EE315

EE315 Dr. Ali Muqaibel

•

,

-

- A \ -\ B - - f fLOJ~~) ___ _

•

-= - -

_____ 1 III jS -

- --

•

--

, ,

\:. ' ~ 'r-I{~ -

-.S..\B_ - y> (. s () IS) .. - 1 PC P,

f ( F\ n )5) \"' ( c.. (\ i3) - _. _. - - .- - -:---------•

- n:(3)

-

fee,) P(~) • ,

-----

, ,

~- - - _______ ~. _____________ . ___ !2 ~

'J

"

r. , I •

'>

'--_ c..::=--__ .>::..,; '-__ :""'=:"';

- 32.

- 2'2> lao

.-

•

Total Probability

18

1 1

( ) ( )N N

n

n

n

n

A AB BSA A

11

( ) [ ( )] ( )N N

n n

nn

P A P A B P A B

Dr. Ali Muqaibel

Given 𝑁 mutually

exclusive events 𝐵𝑛, 𝑛 =1,2, … . 𝑁, that divides the

universe

𝐵𝑚 ∩ 𝐵𝑛 = 𝜙for all 𝑚 ≠ 𝑛

𝑛=1

𝑁

𝐵𝑛 = 𝑆

Total Probability (proof)

mutually exclusive

𝑃 𝐴 =

𝑛=1

𝑁

𝑃 𝐴 𝐵𝑛 𝑃 𝐵𝑛

Bayes’ Theorem

19

1 1

( ) ( )( )

( ) ( ) ( ) ( )

n n

n

N N

P A B P BP B A

P A B P B P A B P B

Dr. Ali Muqaibel

1.4 Joint and Conditional Probability

𝑃 𝐵𝑛 𝐴 =𝑃 𝐵𝑛 ∩ 𝐴

𝑃 𝐴

𝑃 𝐴 𝐵𝑛 =𝑃 𝐴 ∩ 𝐵𝑛𝑃 𝐵𝑛

𝑃 𝐵𝑛 𝐴 =𝑃 𝐴 𝐵𝑛 𝑃 𝐵𝑛𝑃 𝐴

• Given one conditional probability,𝑃 𝐴 𝐵 , we can find the other, 𝑃(𝐵|𝐴)

• Another form using the total probability to represent 𝑃(𝐴)

Example: Total Prob. & Bayes’ Theorem

20

Ex 1.4-2:

Dr. Ali Muqaibel

1.4 Joint and Conditional Probability

Binary Symmetric Channel (BSC)

𝐵1: 1 before the channel𝐵2: 0 before the channel𝐴1: 1 After the channel𝐴2: 0 After the channel

A priori probabilities

𝑃 𝐵1 = 0.6, 𝑃 𝐵2 = 0.4Conditional Probabilities or transition prob.

𝑃 𝐴1 𝐵1 , 𝑃 𝐴1 𝐵2 , 𝑃 𝐴2 𝐵1 , 𝑃 𝐴2 𝐵2

Find 𝑃 𝐵1 𝐴1 𝑎 𝒑𝒐𝒔𝒕𝑟𝑜𝑖𝑜𝑟𝑖 𝑝𝑟𝑜𝑏𝑃 𝐵2 𝐴1 , 𝑃 𝐵1 𝐴2 , 𝑃 𝐵2 𝐴2

Example: Solution

21

1 1( ) ?P B A

1 1 1 1 1 2 2( ) ( ) ( ) ( ) ( ) 0.9 0.6 0.1 0.4 0.58P A P A B P B P A B P B

1 1 11 11 1

1 1

( ) ( )( ) 0.9 0.6 54( ) 0.931

( ) ( ) 0.58 58

P A B P BP B AP B A

P A P A

2 1 1 1( ) ? 1 ( )P B A P B A

1 2( ) ?P B A 2 2( ) ?P B A

Dr. Ali Muqaibel

𝑃 𝐵1 𝐴1 =𝑃 𝐴1 𝐵1 𝑃 𝐵1𝑃 𝐴1

Find 𝑃 𝐴2 =?

Look at examples in the text book 1.4.3 & 1.4.4

1.5 Independent Events

22

( ) ( ) ( ) & independent ( ) ( )

( ) ( )

P B A P B P AA B P B A P B

P A P A

Def: Two events & are (statistically) independent ifA B

( ) ( ) ( ), ( ) 0, ( ) 0P A B P A P B P A P B

( ) ( ) ( )S A AB B B B A B A

(( ) ( ))P B P B A P B A

& independent A B ( ) ( ) ( ) ( ) ( )

( )[1 ( )] ( ) ( )

( ) P B P B A P B P B P A

P B P A

P B

P P

A

B A

& independentA B

Dr. Ali Muqaibel

Statistically independent : the probability of one event is not affected by the occurrence of the other event, 𝑃 𝐴 𝐵 = 𝑃 𝐴 , and 𝑃(𝐵|𝐴) = 𝑃(𝐵)

What about the complement?

Comments about independence• Sufficient Condition for independence (not just

necessary) 𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃 𝐵

• Independence simplify many things “Usually assumed based on physics”.

• If 𝑃 𝐴 > 0 & 𝑃 𝐵 > 0, 𝑡ℎ𝑒𝑛𝑃 𝐴 ∩ 𝐵 = 0 ⇒ 𝑀𝑢𝑡𝑢𝑎𝑙𝑙𝑦 𝑒𝑥𝑐𝑙𝑢𝑠𝑖𝑣𝑒

𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃(𝐵) independent

• For two events to be independent 𝐴 ∩ 𝐵 ≠ 𝜙, because if they are disjoint occurrence of one exclude the other.

• Example:• 𝑨 “select a king”

• 𝑩 “select a jack or queen”

• 𝑪 “select a heart”

• Find 𝑷 𝑨 ,𝑷 𝑩 , 𝑷 𝑪 , 𝑷 𝑨 ∩ 𝑩 ,𝑷 𝑨 ∩ 𝑪 , 𝑷 𝑩 ∩ 𝑪 and test dependence.

Dr. Ali Muqaibel 23

𝑃 𝐴 =4

52

𝑃(𝐵) =8

52

𝑃(𝐶) =13

52

𝑃 𝐴 ∩ 𝐵 = 0 ≠ 𝑃 𝐴 𝑃 𝐵 =32

522

𝑃 𝐴 ∩ 𝐶 =1

52= 𝑃 𝐴 𝑃 𝐶

𝑃 𝐵 ∩ 𝐶 =2

52= 𝑃 𝐵 𝑃 𝐶

𝐴 is independent of 𝐶 and 𝐵 is independent of 𝐶 , but 𝐴 & 𝐵 are dependent.

Multiple Events

24

1 2 3Def: 3 events , , & independent A A A

1 2 1 2( ) ( ) ( )P A A P A P A ( ) 0iP A 2 3 2 3( ) ( ) ( )P A A P A P A

1 3 1 3( ) ( ) ( )P A A P A P A 1 2 3 1 2 3( ) ( ) ( ) ( )P A A A P A P A P A

Ex:21 3{1,2}, {2,3}, {1,3}AAA {1,2,3,4}S

( ) ( ) ( ),i j i jP A A P A P A i j

32 31 21( ) ( ) ( ) ( )A AA AP P P PA A 2 31, , & NOT independentAAA

21 3, , & pairwise independentAAA

1 3 312 2Fact: , , & independent & ( ) independentA AA AA A

1 3 312 2Fact: , , & independent & ( ) independentA AA AA A

Dr. Ali Muqaibel

1.5 Independent Events

Independence by pairs is not sufficient all combinations must be satisfied

Comments about independence of multiple events • If 𝑁 events are independent, then any one of them is independent of

any event formed by unions, intersection, and complements of the others.

𝐴1 is independent 𝐴2 => 𝐴1 is independent 𝐴2

• If independent intersection is changed to product𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 ∩ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 𝑃 𝐴3

𝑃 𝐴1 ∩ 𝐴2 ∪ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 ∪ 𝐴3• This is assuming full independence. Pairwise is not sufficient.

Example: Drawing four ace with replacement and without repleacement

With replacement 𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 ∩ 𝐴4 = 𝑃(𝐴1)𝑃(𝐴2) 𝑃(𝐴3) 𝑃(𝐴4) =4

52

4≈ 3.5 10−5

Without replacement 𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 ∩ 𝐴4 = 𝑃(𝐴1)𝑃(𝐴2|𝐴1)

𝑃(𝐴3|𝐴1 ∩ 𝐴2) 𝑃(𝐴4|𝐴1 ∩ 𝐴2 ∩ 𝐴3) =4

52×3

51×2

50×1

49≈ 3.69(10−6)

Dr. Ali Muqaibel 25

Combined Experiments

• Combined experiments are made of sub-experiments.

• Examples (Wind speed, pressure)

• Repeating an exp. 𝑁 times: Flipping a coin 𝑁 times.

• For two independent (𝑆1, Ω1, 𝑃1) with 𝑁 elements and (𝑆2, Ω2, 𝑃2) with 𝑀 elements, we can form a combined experiment (𝑆, Ω, 𝑃) with 𝑁𝑀elements where 𝑆 = 𝑆1 × 𝑆2 = 𝑠1, 𝑠2 , 𝑠1 ∈ 𝑆1 , 𝑠2 ∈ 𝑆2

• Example 1: 𝑆1 = 𝐻, 𝑇 , 𝑆2 = {1,2,3,4,5,6}

• 𝑆 = { 𝑇, 1 , 𝑇, 2 , 𝑇, 3 , 𝑇, 4 , 𝑇, 5 , 𝑇, 6 , 𝐻, 1 , 𝐻, 2 , 𝐻, 3 ,𝐻, 4 , 𝐻, 5 , 𝐻, 6 }

• Example 2: 𝑆1 = 𝐻, 𝑇 , 𝑆2 = 𝐻, 𝑇

• 𝑆 = 𝐻,𝐻 , 𝐻, 𝑇 , 𝑇, 𝐻 , 𝑇, 𝑇

• Events on the combined Space, 𝑠1 ∈ 𝐴, 𝑠2 ∈ 𝐵, 𝐶 = 𝐴 × 𝐵

• 𝐴 × 𝐵 = 𝐴 × 𝑆2 ∩ (𝑆1 × 𝐵)

Dr. Ali Muqaibel 26

Probability of Combined Experiments

27

3 independent experiments ( , , ), 1,2,3i i iS P i

( , , )S PCan define a combined probability space

1 2 3S S S S

1 2 3 1 1 2 2 3 3( ) ( ) ( ) ( ), i iP A A A P A P A P A A

1 2 3

Permutation

!( 1) ( 1)

!

n

r

nP n n n r

n r

Combination

!

! !

n n

r r n r

0

( )n

n r n r

r

nx y x y

r

binomial expansion

Dr. Ali Muqaibel

Permutation: # of possible sequences (order important) (not replaced)Combination: # of possible sequences (order not important)(not replaced)# decreases by 𝑃𝑟

𝑟 =𝑟!

0!= 𝑟!

Examples: Permutation and Combination

• Example: drawing 4 cards from 52 card deck. How many possibilities are there.

•𝑃452 =

52!

52−4 != 52(51)(50 49 = 6,497,400

• Example: A coach has five athletes and he wants to make a team made of 3.How many teams can he make?

•𝐶35 =

5!

2!3!= 10 . Same as choosing 2 for the spare team.

𝐶𝑟𝑛 = 𝐶𝑛−𝑟

𝑛 .

Other notations are also possible.

Dr. Ali Muqaibel 28

1.7 Bernoulli Trials

29

Basic experiment - 2 possible outcomes ( or )A A

Bernoulli Trials - repeat the basic experiment timesN

( ) ( ) 1P A p P A p

(Assume that elementary events are independent for every trial.)

({ occurs exactly times}) (1 )k N kkk

P AN

p p

Ex 1.7-1: ( ) 0.4P A 3N

2 13

(2 hits) 0.4 (1 0.4) 0.2882

P

Dr. Ali Muqaibel

Hit or miss , win or lose, 0 or 1

We are firing a carrier with torpedoes.𝑃 ℎ𝑖𝑡 = 0.4. It will sunk if two or more hits. We are firing three torpedoes.

Continue Example :Bernoulli Trials

30

Ex 1.7-3: ( ) 0.4P A 120N

3 03

(3 hits) 0.4 (1 0.4) 0.0643

P

({carrier sunk}) (2 hits) (3 hits) 0.352P P P

50 70120

(50 hits) 0.4 (1 0.4) ?50

P

large N De Moivre-Laplace approximation

Poisson approximation

120! ?

Dr. Ali Muqaibel

0 33

(0 hits) 0.4 (1 0.4) 0.2160

P

1 23

(1 hits) 0.4 (1 0.4) 0.4321

P

Given we are firing for 3 seconds. Firing rate

2400 per minutes. Find 𝑃{𝑒𝑥𝑎𝑐𝑡𝑙𝑦 50 ℎ𝑖𝑡𝑠}

De Moivre-Laplace & Poisson Approximations

• Stirling’s Formula: 𝑚! ≈ 2𝜋𝑚1

2𝑚𝑚𝑒−𝑚, for large 𝑚.

• Error less than 1% even for 𝑚 = 10.

• Using Stirling’s formula, De Moivre-Laplace Approximation

𝑃 =𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈

1

2𝜋𝑁𝑝 1 − 𝑝𝑒𝑥𝑝 −

𝑘 − 𝑁𝑝 2

2𝑁𝑝 1 − 𝑝

• 𝑁, 𝑘, 𝑎𝑛𝑑 𝑁 − 𝑘,𝑚𝑢𝑠𝑡 𝑏𝑒 𝑙𝑎𝑟𝑔𝑒 , 𝑘 𝑚𝑢𝑠𝑡 𝑏𝑒 𝑛𝑒𝑎𝑟 𝑁𝑝 to assure small numerator.

• If 𝑁 is very large and p is very small De Moivre-Laplace approximation fails, we can use Poisson approximation:

𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈

𝑁𝑝 𝑘𝑒−𝑁𝑝

𝑘!• 𝑁 large and 𝑝 is small.

Dr. Ali Muqaibel 31

Example: De Moivre-Laplace Approximation

• Back to the torpedoes example. Given we are firing for 3 seconds. Firing rate 2400 per minutes. Find 𝑃{𝑒𝑥𝑎𝑐𝑡𝑙𝑦 50 ℎ𝑖𝑡𝑠}

• 𝑁 = 3𝑠𝑒𝑐 ×2400𝑏𝑢𝑙𝑙𝑒𝑡𝑠/𝑚𝑖𝑛

60 𝑠𝑒𝑐/𝑚𝑖𝑛= 120 𝑏𝑢𝑙𝑙𝑒𝑡𝑠.

• 𝑘 = 50, 𝑁𝑝 = 120 0.4 = 48, 𝑁 1 − 𝑝 = 120 0.6 = 72

• 𝑁, 𝑘 & 𝑁 − 𝑘 = 70 are all large. 𝑘 = 50 is near 𝑁𝑝 = 48

• 𝑃{50 ℎ𝑖𝑡𝑠}=𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈

1

2𝜋𝑁𝑝 1−𝑝𝑒𝑥𝑝 −

𝑘−𝑁𝑝 2

2𝑁𝑝 1−𝑝

=1

2𝜋(48) 0.6𝑒𝑥𝑝 −

50 − 48 2

2(48) 0.6= 0.0693

Dr. Ali Muqaibel 32