Chapter 1. Probability
1. Set definitions• Element, Subset, Proper subset, Set, Class, Universal set=S, empty=null set=𝜙.• Countable, uncountable,, finite, infinite.• Disjoint, mutually exclusive.
2. Set operations• Venn diagram, equality, difference, union=sum, intersection=product,
complement.• Algebra of sets (commutative, distributive, associative)• De Morgan’s law• Duality Principle
3. Probability introduced through sets and relative frequency4. Joint and conditional probability5. Independent events6. Combined experiments7. Bernoulli trials8. Summary
1Dr. Ali Muqaibel
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
• •
-
"Ii
- • _ E
•
-
- 7
•
• • • • •
.. - . •
-. ------- ----_ .. -
\0 - .
- _. ---------------------------- -- -
e.",,-
---- - - -
-- - e ...... f \-~) - ILV·U ' -f ~A __ Ss-el . 7 •
I J:.~'j fvI.- .. . " - -- -
. IM.c~-ktk \ ) 1«)10. 'v.le 1"'- ,. w - • - • •
IV\. • -~
, t~vvq' .11401<'6
, "tA. V '-- ) , ,
•
&,~ , . 11..~16 - I '" \ -. • -
-- - -~ _. -----,---------
- =-
CIS ,
- --•
-----"-> , ----
"
I
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
-
.
- -
--- -fj .;... .. ::. ---. ---=~ 3> ,-- - _ c ~_ D, - <,c ,s.._K.5 __ _ •
-
_ _ (Cr, _ D L F _" _. _ - --- ,/ ----------------------, ,
-- -- '"-' _._-
____ r.x-(
____ \u=~~. J6c:us\j<M . ___ _ _ _____ _
sPJ w/J.~ N e1e~do -ifo3-e..r~
~---- --N -------.-------------
- - -- -
- - , - - - • • - -t~
N . ~J c:2 .-. ~Li ,v .::-~
7 '.
~----- - ------------------
--------------------- - --
'----------------~----------- - ... --- - ._- -•
. _---_. _. -- ------------_._.----------------_ .. ----------•
~. . • • •
--------------------- -------- -----,
----_. ---- -------------,-------,--
•
•
•
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
I ,
--~-~'-~-'~. ~. ~~' ~~~~~~' ----------------------------
,
-------- ------------------
:...-------- ~----, -, , ---~--;-----------
_____ a
G .. ---- ,.-.s -01\k :Mk E.,,* b
-- __ U ) ___ _
- ---- -- ------ _._-----
'-r-----
,
--------, ---------- ._----
,
? ,
f\ - Q .,6' <q <....\.c .... £L lOO t;' • . - - •• • ,
~ J b ~ ~.s 0.(' I
< 13 - __ a 'I I ' M - ••
~, S'
B - < I 0 , _._-
--, - ~.' - t.' < d ::- _ "'. , ~ 8 IS :::::- .,(,...,..- ,- , - --, - -- - .. -,
--------- , , - '--- - ~~ " . -~-- - -'~- - - -- . - --,
--'------ - ---------------
- w~ _____ .. _ ...
~ ( . .,..-b ..... 6Y\O-j''-'-_ , . - -- ------ --t...QV _, __ _ - ----
A ,LLA ,=,",""LL_-'. :.. .:
• •
n
'" ---- -
1'I :: I
1'--= ,
_ ..
---- --
•
3 - ---------_.
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
I 2 : ! £ i • 2 2 zi 2
- ' . -'
, ,
- s~_" 'or) -...!.SwL...,;-.:....:. '---;. ,;'_ , ---,-A -
•
,
u u --- -[)g~-.-(3..fJ 8---'------ -----, ---
lJ f) - tLl) ( ~, u.c -
u _ n , U A f} g) (..f , .1\ n c
a !,> t ) C I ~a.I( (I," t~tH __ _ f1 /) --efl UJs) u (
o (1 (\ () =. _ .-8. LA. (I5.LLL..) - A \1 is ~U G_
------------------------, -------------------
----
-- --- ----
----_ . • -
•
\ f\ \) I~ ( ? 0..' ---- - - - - ---------- .. .I-,J
-, .. • • ,
•
,
"
---
---- --•
-- '----- ~.---,-.' .-
/ -_-,---,A,-,,_U~ ___ __ __ ====
,- ,--, -
•
rr--.J.."'"l ~l ? 3 -
-::-" _________________________________________________________ ~ ___ -=·===:'v~~:' _
__ (1 <: • _ --=-~--,,-< S
U A . - n 6r) .,. '- --
_~ _ = ~==== :---=_-IfLO . _= __ &J'Lk
, -
•
------------ -
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
,
• -
t -
•
-
\
I <:;:
7 7 ==
•
I .'
'1 , •
,
,
\
•
•
-------_. ------------------------------------.- ----------------_._,- ----------
-
,
I e ,--, ._--- - ~----- - -
,-------- '--------, " --
-=9 -C~b'v:\.\A.;).V'd .-----------,---
,
•
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
We
•
•
•
p,.~b 1------
• .,
./
i
•
•
, ------------
-- 2 .. , ) ----------------------
"
, •
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
_ akL c:.LC"_ . '-= L-_
_______ --'s ':: 0 < ;y <.. \ Co •
• • , .
- --
~~--------f~-~- -'~ .. ---------------I .N
•
--"-. - ;;..,.......
-
(3 -•
•
- r B)
f
• ..A
(>,,!e .. "' . 1 _____ _
•
" ,. ~-~J __ _
• ..... . 7 •
/
•
~ I
" (
" '. , • --< it 1 ________ --' (
/ -,
,
1-1-/--, ( " t) , 1 ' . ' . - - _ ' _ -~ I '
~~~3~6---------------------------------------~=-~~~~,
. - ,
" U A !!J
~ -, 1,'''',- '[ , •
1.:- , '1 ' ::: I
• -- -- - - ,
I ~
• , I •
Zf f I -
, ,
A 1:)
-
-
•
, ,
-(
• • " , ,
I I • :?-( - . - • -,
= ,a. ;; .. ... _ -- - --/6
~ >:.-' --- -
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
, VI o by;,..,
1 / l ,.
•
I I 1
1 I
- •
. __ =-<~ ~ _____ '-,_. __ :L1- 51
--- '-' _1 -1.) - l2 L?J ..... <2 _ ______ ~ /'Z / Xi) ____ .
- :$ 3 B,u-') _ --UI 4-1- Q -= .. ::.... 11: L2..;:,..<..v _
•
-
5 •
r)
•
-
Dr. Ali Muqaibel EE315
EE315 Dr. Ali Muqaibel
•
,
-
- A \ -\ B - - f fLOJ~~) ___ _
•
-= - -
_____ 1 III jS -
- --
•
--
, ,
\:. ' ~ 'r-I{~ -
-.S..\B_ - y> (. s () IS) .. - 1 PC P,
f ( F\ n )5) \"' ( c.. (\ i3) - _. _. - - .- - -:---------•
- n:(3)
-
fee,) P(~) • ,
-----
, ,
~- - - _______ ~. _____________ . ___ !2 ~
'J
"
r. , I •
'>
'--_ c..::=--__ .>::..,; '-__ :""'=:"';
- 32.
- 2'2> lao
.-
•
Total Probability
18
1 1
( ) ( )N N
n
n
n
n
A AB BSA A
11
( ) [ ( )] ( )N N
n n
nn
P A P A B P A B
Dr. Ali Muqaibel
Given 𝑁 mutually
exclusive events 𝐵𝑛, 𝑛 =1,2, … . 𝑁, that divides the
universe
𝐵𝑚 ∩ 𝐵𝑛 = 𝜙for all 𝑚 ≠ 𝑛
𝑛=1
𝑁
𝐵𝑛 = 𝑆
Total Probability (proof)
mutually exclusive
𝑃 𝐴 =
𝑛=1
𝑁
𝑃 𝐴 𝐵𝑛 𝑃 𝐵𝑛
Bayes’ Theorem
19
1 1
( ) ( )( )
( ) ( ) ( ) ( )
n n
n
N N
P A B P BP B A
P A B P B P A B P B
Dr. Ali Muqaibel
1.4 Joint and Conditional Probability
𝑃 𝐵𝑛 𝐴 =𝑃 𝐵𝑛 ∩ 𝐴
𝑃 𝐴
𝑃 𝐴 𝐵𝑛 =𝑃 𝐴 ∩ 𝐵𝑛𝑃 𝐵𝑛
𝑃 𝐵𝑛 𝐴 =𝑃 𝐴 𝐵𝑛 𝑃 𝐵𝑛𝑃 𝐴
• Given one conditional probability,𝑃 𝐴 𝐵 , we can find the other, 𝑃(𝐵|𝐴)
• Another form using the total probability to represent 𝑃(𝐴)
Example: Total Prob. & Bayes’ Theorem
20
Ex 1.4-2:
Dr. Ali Muqaibel
1.4 Joint and Conditional Probability
Binary Symmetric Channel (BSC)
𝐵1: 1 before the channel𝐵2: 0 before the channel𝐴1: 1 After the channel𝐴2: 0 After the channel
A priori probabilities
𝑃 𝐵1 = 0.6, 𝑃 𝐵2 = 0.4Conditional Probabilities or transition prob.
𝑃 𝐴1 𝐵1 , 𝑃 𝐴1 𝐵2 , 𝑃 𝐴2 𝐵1 , 𝑃 𝐴2 𝐵2
Find 𝑃 𝐵1 𝐴1 𝑎 𝒑𝒐𝒔𝒕𝑟𝑜𝑖𝑜𝑟𝑖 𝑝𝑟𝑜𝑏𝑃 𝐵2 𝐴1 , 𝑃 𝐵1 𝐴2 , 𝑃 𝐵2 𝐴2
Example: Solution
21
1 1( ) ?P B A
1 1 1 1 1 2 2( ) ( ) ( ) ( ) ( ) 0.9 0.6 0.1 0.4 0.58P A P A B P B P A B P B
1 1 11 11 1
1 1
( ) ( )( ) 0.9 0.6 54( ) 0.931
( ) ( ) 0.58 58
P A B P BP B AP B A
P A P A
2 1 1 1( ) ? 1 ( )P B A P B A
1 2( ) ?P B A 2 2( ) ?P B A
Dr. Ali Muqaibel
𝑃 𝐵1 𝐴1 =𝑃 𝐴1 𝐵1 𝑃 𝐵1𝑃 𝐴1
Find 𝑃 𝐴2 =?
Look at examples in the text book 1.4.3 & 1.4.4
1.5 Independent Events
22
( ) ( ) ( ) & independent ( ) ( )
( ) ( )
P B A P B P AA B P B A P B
P A P A
Def: Two events & are (statistically) independent ifA B
( ) ( ) ( ), ( ) 0, ( ) 0P A B P A P B P A P B
( ) ( ) ( )S A AB B B B A B A
(( ) ( ))P B P B A P B A
& independent A B ( ) ( ) ( ) ( ) ( )
( )[1 ( )] ( ) ( )
( ) P B P B A P B P B P A
P B P A
P B
P P
A
B A
& independentA B
Dr. Ali Muqaibel
Statistically independent : the probability of one event is not affected by the occurrence of the other event, 𝑃 𝐴 𝐵 = 𝑃 𝐴 , and 𝑃(𝐵|𝐴) = 𝑃(𝐵)
What about the complement?
Comments about independence• Sufficient Condition for independence (not just
necessary) 𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃 𝐵
• Independence simplify many things “Usually assumed based on physics”.
• If 𝑃 𝐴 > 0 & 𝑃 𝐵 > 0, 𝑡ℎ𝑒𝑛𝑃 𝐴 ∩ 𝐵 = 0 ⇒ 𝑀𝑢𝑡𝑢𝑎𝑙𝑙𝑦 𝑒𝑥𝑐𝑙𝑢𝑠𝑖𝑣𝑒
𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝑃(𝐵) independent
• For two events to be independent 𝐴 ∩ 𝐵 ≠ 𝜙, because if they are disjoint occurrence of one exclude the other.
• Example:• 𝑨 “select a king”
• 𝑩 “select a jack or queen”
• 𝑪 “select a heart”
• Find 𝑷 𝑨 ,𝑷 𝑩 , 𝑷 𝑪 , 𝑷 𝑨 ∩ 𝑩 ,𝑷 𝑨 ∩ 𝑪 , 𝑷 𝑩 ∩ 𝑪 and test dependence.
Dr. Ali Muqaibel 23
𝑃 𝐴 =4
52
𝑃(𝐵) =8
52
𝑃(𝐶) =13
52
𝑃 𝐴 ∩ 𝐵 = 0 ≠ 𝑃 𝐴 𝑃 𝐵 =32
522
𝑃 𝐴 ∩ 𝐶 =1
52= 𝑃 𝐴 𝑃 𝐶
𝑃 𝐵 ∩ 𝐶 =2
52= 𝑃 𝐵 𝑃 𝐶
𝐴 is independent of 𝐶 and 𝐵 is independent of 𝐶 , but 𝐴 & 𝐵 are dependent.
Multiple Events
24
1 2 3Def: 3 events , , & independent A A A
1 2 1 2( ) ( ) ( )P A A P A P A ( ) 0iP A 2 3 2 3( ) ( ) ( )P A A P A P A
1 3 1 3( ) ( ) ( )P A A P A P A 1 2 3 1 2 3( ) ( ) ( ) ( )P A A A P A P A P A
Ex:21 3{1,2}, {2,3}, {1,3}AAA {1,2,3,4}S
( ) ( ) ( ),i j i jP A A P A P A i j
32 31 21( ) ( ) ( ) ( )A AA AP P P PA A 2 31, , & NOT independentAAA
21 3, , & pairwise independentAAA
1 3 312 2Fact: , , & independent & ( ) independentA AA AA A
1 3 312 2Fact: , , & independent & ( ) independentA AA AA A
Dr. Ali Muqaibel
1.5 Independent Events
Independence by pairs is not sufficient all combinations must be satisfied
Comments about independence of multiple events • If 𝑁 events are independent, then any one of them is independent of
any event formed by unions, intersection, and complements of the others.
𝐴1 is independent 𝐴2 => 𝐴1 is independent 𝐴2
• If independent intersection is changed to product𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 ∩ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 𝑃 𝐴3
𝑃 𝐴1 ∩ 𝐴2 ∪ 𝐴3 = 𝑃 𝐴1 𝑃 𝐴2 ∪ 𝐴3• This is assuming full independence. Pairwise is not sufficient.
Example: Drawing four ace with replacement and without repleacement
With replacement 𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 ∩ 𝐴4 = 𝑃(𝐴1)𝑃(𝐴2) 𝑃(𝐴3) 𝑃(𝐴4) =4
52
4≈ 3.5 10−5
Without replacement 𝑃 𝐴1 ∩ 𝐴2 ∩ 𝐴3 ∩ 𝐴4 = 𝑃(𝐴1)𝑃(𝐴2|𝐴1)
𝑃(𝐴3|𝐴1 ∩ 𝐴2) 𝑃(𝐴4|𝐴1 ∩ 𝐴2 ∩ 𝐴3) =4
52×3
51×2
50×1
49≈ 3.69(10−6)
Dr. Ali Muqaibel 25
Combined Experiments
• Combined experiments are made of sub-experiments.
• Examples (Wind speed, pressure)
• Repeating an exp. 𝑁 times: Flipping a coin 𝑁 times.
• For two independent (𝑆1, Ω1, 𝑃1) with 𝑁 elements and (𝑆2, Ω2, 𝑃2) with 𝑀 elements, we can form a combined experiment (𝑆, Ω, 𝑃) with 𝑁𝑀elements where 𝑆 = 𝑆1 × 𝑆2 = 𝑠1, 𝑠2 , 𝑠1 ∈ 𝑆1 , 𝑠2 ∈ 𝑆2
• Example 1: 𝑆1 = 𝐻, 𝑇 , 𝑆2 = {1,2,3,4,5,6}
• 𝑆 = { 𝑇, 1 , 𝑇, 2 , 𝑇, 3 , 𝑇, 4 , 𝑇, 5 , 𝑇, 6 , 𝐻, 1 , 𝐻, 2 , 𝐻, 3 ,𝐻, 4 , 𝐻, 5 , 𝐻, 6 }
• Example 2: 𝑆1 = 𝐻, 𝑇 , 𝑆2 = 𝐻, 𝑇
• 𝑆 = 𝐻,𝐻 , 𝐻, 𝑇 , 𝑇, 𝐻 , 𝑇, 𝑇
• Events on the combined Space, 𝑠1 ∈ 𝐴, 𝑠2 ∈ 𝐵, 𝐶 = 𝐴 × 𝐵
• 𝐴 × 𝐵 = 𝐴 × 𝑆2 ∩ (𝑆1 × 𝐵)
Dr. Ali Muqaibel 26
Probability of Combined Experiments
27
3 independent experiments ( , , ), 1,2,3i i iS P i
( , , )S PCan define a combined probability space
1 2 3S S S S
1 2 3 1 1 2 2 3 3( ) ( ) ( ) ( ), i iP A A A P A P A P A A
1 2 3
Permutation
!( 1) ( 1)
!
n
r
nP n n n r
n r
Combination
!
! !
n n
r r n r
0
( )n
n r n r
r
nx y x y
r
binomial expansion
Dr. Ali Muqaibel
Permutation: # of possible sequences (order important) (not replaced)Combination: # of possible sequences (order not important)(not replaced)# decreases by 𝑃𝑟
𝑟 =𝑟!
0!= 𝑟!
Examples: Permutation and Combination
• Example: drawing 4 cards from 52 card deck. How many possibilities are there.
•𝑃452 =
52!
52−4 != 52(51)(50 49 = 6,497,400
• Example: A coach has five athletes and he wants to make a team made of 3.How many teams can he make?
•𝐶35 =
5!
2!3!= 10 . Same as choosing 2 for the spare team.
𝐶𝑟𝑛 = 𝐶𝑛−𝑟
𝑛 .
Other notations are also possible.
Dr. Ali Muqaibel 28
1.7 Bernoulli Trials
29
Basic experiment - 2 possible outcomes ( or )A A
Bernoulli Trials - repeat the basic experiment timesN
( ) ( ) 1P A p P A p
(Assume that elementary events are independent for every trial.)
({ occurs exactly times}) (1 )k N kkk
P AN
p p
Ex 1.7-1: ( ) 0.4P A 3N
2 13
(2 hits) 0.4 (1 0.4) 0.2882
P
Dr. Ali Muqaibel
Hit or miss , win or lose, 0 or 1
We are firing a carrier with torpedoes.𝑃 ℎ𝑖𝑡 = 0.4. It will sunk if two or more hits. We are firing three torpedoes.
Continue Example :Bernoulli Trials
30
Ex 1.7-3: ( ) 0.4P A 120N
3 03
(3 hits) 0.4 (1 0.4) 0.0643
P
({carrier sunk}) (2 hits) (3 hits) 0.352P P P
50 70120
(50 hits) 0.4 (1 0.4) ?50
P
large N De Moivre-Laplace approximation
Poisson approximation
120! ?
Dr. Ali Muqaibel
0 33
(0 hits) 0.4 (1 0.4) 0.2160
P
1 23
(1 hits) 0.4 (1 0.4) 0.4321
P
Given we are firing for 3 seconds. Firing rate
2400 per minutes. Find 𝑃{𝑒𝑥𝑎𝑐𝑡𝑙𝑦 50 ℎ𝑖𝑡𝑠}
De Moivre-Laplace & Poisson Approximations
• Stirling’s Formula: 𝑚! ≈ 2𝜋𝑚1
2𝑚𝑚𝑒−𝑚, for large 𝑚.
• Error less than 1% even for 𝑚 = 10.
• Using Stirling’s formula, De Moivre-Laplace Approximation
𝑃 =𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈
1
2𝜋𝑁𝑝 1 − 𝑝𝑒𝑥𝑝 −
𝑘 − 𝑁𝑝 2
2𝑁𝑝 1 − 𝑝
• 𝑁, 𝑘, 𝑎𝑛𝑑 𝑁 − 𝑘,𝑚𝑢𝑠𝑡 𝑏𝑒 𝑙𝑎𝑟𝑔𝑒 , 𝑘 𝑚𝑢𝑠𝑡 𝑏𝑒 𝑛𝑒𝑎𝑟 𝑁𝑝 to assure small numerator.
• If 𝑁 is very large and p is very small De Moivre-Laplace approximation fails, we can use Poisson approximation:
𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈
𝑁𝑝 𝑘𝑒−𝑁𝑝
𝑘!• 𝑁 large and 𝑝 is small.
Dr. Ali Muqaibel 31
Example: De Moivre-Laplace Approximation
• Back to the torpedoes example. Given we are firing for 3 seconds. Firing rate 2400 per minutes. Find 𝑃{𝑒𝑥𝑎𝑐𝑡𝑙𝑦 50 ℎ𝑖𝑡𝑠}
• 𝑁 = 3𝑠𝑒𝑐 ×2400𝑏𝑢𝑙𝑙𝑒𝑡𝑠/𝑚𝑖𝑛
60 𝑠𝑒𝑐/𝑚𝑖𝑛= 120 𝑏𝑢𝑙𝑙𝑒𝑡𝑠.
• 𝑘 = 50, 𝑁𝑝 = 120 0.4 = 48, 𝑁 1 − 𝑝 = 120 0.6 = 72
• 𝑁, 𝑘 & 𝑁 − 𝑘 = 70 are all large. 𝑘 = 50 is near 𝑁𝑝 = 48
• 𝑃{50 ℎ𝑖𝑡𝑠}=𝑁𝑘𝑝𝑘 1 − 𝑝 𝑁−𝑘 ≈
1
2𝜋𝑁𝑝 1−𝑝𝑒𝑥𝑝 −
𝑘−𝑁𝑝 2
2𝑁𝑝 1−𝑝
=1
2𝜋(48) 0.6𝑒𝑥𝑝 −
50 − 48 2
2(48) 0.6= 0.0693
Dr. Ali Muqaibel 32