Introduction to stochastic processIntroduction to stochastic process
Dr. Nur Aini Masruroh
Outlines Outlines
Concept of probability
Random variables
Expected value
Conditional probability
Limit theorem
Stochastic processes
Probability in Industrial EngineeringProbability in Industrial Engineering
Random arrivals of jobsRandom service timeRandom requests for resourcesProbability of queue overflowService delaysScheduling problemsFlow control and routingRevenue managementRisk assessment for decision analysisEtc
Basic probabilityBasic probability
Random experiment: an experiment whose outcome cannot be determined in advanced
Sample space (S): set of all possible outcomes Event (E): a subset of sample space, it occurs if the
outcome of the experiment is an element of that subset A number P(E) is defined and satisfies:
0 ≤ P(E) ≤ 1 P(S) =1 For any sequence of events E1, E2, … that are mutually
exclusive
11
)(i
ii
i EPEp
Random variablesRandom variables
Consider a random experiment having sample space S. A random variable X is a function that assigns a real value to each
outcome in S The distribution function F of the random variable X is defined for
any real number x by
A random variable X is said to be discrete if its set of possible values is countable. For discrete random variables,
A random variable is called continuous if there exists a function f(x), called the probability density function, such that
xXPxF ,)(
xy
yXPxF )(
B
dxxfBXP )(inis For every set B
Random variablesRandom variables
Since it follows that
The join distribution function F of two random variables X and Y is defined by
F(x, y) = P{X≤ x, Y ≤ y} The distribution functions of X and Y,
Fx(x) = P{X≤ x} and FY(y) = P{Y≤y}
It can be obtained from F(x, y) by making use of the continuity property of the probability operator.
Continuity property:
Similarly,
F(x, y) = Fx(x)FY(y) if X and Y are independent
x
dxxfxF )()( )()( xFdx
dxf
xXPyYxXP nn
,lim
),(lim)(or),(lim)( yxFyFyxFxFx
Yy
X
Cumulative Distribution Function (CDF)Cumulative Distribution Function (CDF)
CDF, F, of the random variable X is defined by all real numbers b, -∞ < b < ∞, by F(b) = P(X ≤ b)
F(b) denotes the cumulative probability that the random variable X takes on a value that is less or equal to b (the total probability mass)
CDF is a non-decreasing function: P(X≤ a) ≤ P(X ≤ b) if a < b and thus F(a) ≤ F(b) for a < b
F(∞) = 1 and F(-∞)=0: bounded CDF is a right continuous function: for any b, F(b+), value
of F(b) just to the right of b, equals to F(b) as b+ get closer to b. The CDF is right continuous at all values, but may be left discontinuous at some values
Probabilities from CDFProbabilities from CDF
P(X ≤ b) = F(b) P(X<b) = F(b – ε) = F(b-)
ε is a small number
P(X=b) = P(X ≤ b) – P(X < b) = F(b) – F(b-) P(a < X ≤ b) = F(b) – F(a) P(a ≤ X ≤ b) = F(b) – F(a-) P(a ≤ X < b) = F(b-) – F(a-) P(a < X < b) = F(b-) – F(a)
Probability Density Function (pdf)Probability Density Function (pdf)
The pdf of a continuous random variable X is defined by the derivative of the continuous CDF at differentiable intervals : dF(b)/db = f(b)
The pdf of a continuous random variable tells us the density of the mass at each point on the sample space
The value of pdf is not probability, for example, if f(x) = 2x, 0 ≤ x ≤ 1, and 0 otherwise, f(1) = 2 is obviously not a probability value
Graphically, pdf is the area under f(x) from ain interval a,b.
Note: if pmf has meaning itself, the value itself for pdf has no meaning
b
a
dxxfbXaP )()(
Discrete random variable: summaryDiscrete random variable: summary
A random variable X associates a number with each outcome of an experiment
A discrete random variable takes on a finite number The probabilistic behavior of a discrete random variable
X is described by its probability mass function (pmf), p(u) P(uj) = p(X=uj)
All the probabilistic information about the discrete random variable X is summarized in its pmf
A discrete random variable X has a CDF F(b) which is: Right-continuous Staircase CDF
Continuous random variable: summaryContinuous random variable: summary
No mass to define pmf: every event has zero probability Example: p(height of people= 173.7897654) 0 However P(147< height of people < 187) 1
The set of possible values are uncountable while the set of possible values were finite or countably infinite for a discrete random variable
Sample space is not a discrete set, but a continuous space (or interval)
A continuous random variable X has a CDF F(b) which is: Continuous at all b, -∞ < b < ∞ Differentiable at all b (except possibly at finite set of points)
Expected valueExpected value
Expectation of the random variable X,
The variance of the random variable X is defined by
Var X = E[(X – E[X])2]
= E[X2] – E2[X]
Two jointly distributed random variables X and Y are said to be uncorrelated if their covariance defined by
Cov (X, Y) = E[(X – EX)(Y – EY)]
= E[XY] – E[X]E[Y]
x
XxXxP
Xdxxxf
xxdFXE
discreteisif}{
continueisif)(
)(][
Expected valueExpected value
Expectation of a sum of random variables is equal to the sum of the expectations
Variances:
n
ii
n
ii XEXE
11
[
),(2)(11
ji
n
i jii
n
ii XXCovXVarXVar
Conditional probabilitiesConditional probabilities
In general, given information about the outcome of some events, we may revise our probabilities of other events
We do this through the use of conditional probabilities The probability of an event X given specific outcomes of another
event Y is called the conditional probability X given Y The conditional probability of event X given event Y and other
background information ξ, is denoted by p(X|Y, ξ) and is given by
0)|(for)|(
)|(),|(
Yp
Yp
YXpYXp
Bayes’ TheoremBayes’ Theorem
Given two uncertain events X and Y. Suppose the probabilities p(X|ξ) and p(Y|X, ξ) are known, then
X
XYpXpYp
where
Yp
XYpXpYXp
)||()|()|(
)|(
),|()|(),|(
Factorization rule for joint probabilityFactorization rule for joint probability
Limit theorem Limit theorem Strong Law of Large Numbers
If X1, X2, … are independent and identically distributed with mean μ, then
Central Limit Theorem If X1, X2, … are independent and identically distributed with
mean μ and variance σ2, then
Thus if we let where X1, X2, … are independent and identically distributed, then the Strong of LLN states that, with probability1, Sn/n will converge to E[Xi]; whereas the CLT states that Sn will have an asymptotic normal distribution as n → ∞
1...
lim 1
n
XXP n
n
dxean
nXXP x
an
n
2/12
2
1...lim
n
i in XS1
Stochastic processStochastic process
X(t) is the state of the process (measurable characteristic of interest) at time t
• the state space of the a stochastic process is defined as the set of all possible values that the random variables X(t) can assume
• when the set T is countable, the stochastic process is a discrete time process; denote by {Xn, n=0, 1, 2, …}
• when T is an interval of the real line, the stochastic process is a continuous time process; denote by {X(t), t≥0}
Stochastic processStochastic process
Hence,• a stochastic process is a family of random
variables that describes the evolution through time of some (physical) process.
• usually, the random variables X(t) are dependent and hence the analysis of stochastic processes is very difficult.
• Discrete Time Markov Chains (DTMC) is a special type
of stochastic process that has a very simple dependence among X(t) and renders nice results in the analysis of {X(t), t∈T} under very mild assumptions.
Example of stochastic processesExample of stochastic processes
Refer to X(t) as the state of the process at time t A stochastic process {X(t), t T} is a time indexed ∈
collection of random variables
X(t) might equal the total number of customers that have entered a supermarket by time t
X(t) might equal the number of customers in the supermarket at time t
X(t) might equal the stock price of a company at time t
Counting processCounting process
Definition: A stochastic process {N(t), t≥0} is a counting process if
N(t) represents the total number of “events” that have occurred up to time t
Counting process Counting process
Examples: If N(t) equal the number of persons who have entered a
particular store at or prior to time t, then {N(t), t≥0} is a counting process in which an event corresponds to a person entering the store
• If N(t) equal the number of persons in the store at time t, then {N(t), t≥0} would not be a counting process. Why?
If N(t) equals the total number of people born by time t, then {N(t), t≥0} is a counting process in which an event corresponds to a child is born
If N(t) equals the number of goals that Ronaldo has scored by time t, then {N(t), t≥0} is a counting process in which an event occurs whenever he scores a goal
Counting processCounting process
A counting process N(t) must satisfy N(t)≥0 N(t) is integer valued If s ≤t, then N(s) ≤ N(t) For s<t, N(t)-N(s) equals the number of events that have
occurred in the interval (s,t), or the increments of the counting process in (s,t)
A counting process has Independent increments if the number of events which occur in
disjoint time intervals are independent Stationary increments if the distribution of the number of
events which occur in any interval of time depends only on the length of the time interval
Independent incrementIndependent increment
This property says that numbers of events in disjoint intervals are independent random variables.
Suppose that t1< t2≤ t3< t4. Then N(t2)-N(t1), the number of events occurring in (t1,t2], is independent of N(t4)-N(t3), the number of events occurring in (t3, t4].