Characterization Nor Me Der God i City

Characterization and sufficient conditions fornormed ergodicity of Markov Chains∗

A.A. Borovkov and A. HordijkInstitute of Mathematics, Novosibirsk, Russia

and Mathematical Institute, Leiden University, Netherlands

June 30, 2003

Abstract

Normed ergodicity is a type of strong ergodicity for which it holdsthat the convergence of the n-th step transition operator to the sta-tionary one is in operator norm. We derive a new characterization of itand we clarify its relation with exponential ergodicity. The existenceof a Lyapunov function together with two conditions on the uniformintegrability of the increments of the Markov Chain is shown to bea sufficient condition for normed ergodicity. Conversily, the sufficientconditions are also almost necessary.

1 Introduction

In this paper we study a special form of normed ergodicity for discrete-timeMarkov Chains(MC). Normed ergodicity is a generalization of the so-calledstrong ergodicity (see [18]). In strong ergodicity the n-th step transition ker-nels converge to the stationary one exponentially fast in the operator norminduced by the supremum norm. Strong ergodicity is implied by one of theoldest conditions for ergodicity, the so-called Doeblin condition (see [18]).

∗The research of this paper has been supported by the INTAS-projects: ”AsymptoticAnalysis of Stochastic Networks” and ”The Mathematics of Stochastic Networks”.

1

Unfortunately, the Doeblin condition is not satisfied in most multidimen-sional MC, since it requires that the recurrence time to a compact set is uni-formly (in the initial state) bounded. So strong ergodicity does not hold formost stochastic networks. Therefore, Hordijk introduced in the early eight-ies a new concept of normed ergodicity. In this generalization the operatornorm induced by a weighted supremum norm was used. It turned out thatinteresting applied probability models, especially queueing networks underrather broad conditions, do satisfy this ergodicity. Since in [13] for the firsttime applicable sufficient conditions were derived and subsequently Meyn andTweedie devoted a complete chapter of their book (see chapter 16 of [17]) tothis type of ergodicity, the notion of normed ergodicity has been applied innumerous papers. In this paper we study a special type of normed ergodicityand we focus on the relation with the ergodicity results in the recent book [1].We derive a new characterization, which clarifies this relation. In additionto the exponential ergodicity (see [1]) normed ergodicity requires one morelarge deviation relation (see theorem 2).

Let us denote by X(x, n) the state of the MC at time t = n, given the ini-tial state x at time t = 0. The second main theorem of this paper shows thatfor a Harris chain the existence of a Lyapunov function say g, together withthe uniform integrability in x of ξ(x) , g(X(x, 1)) − g(x) and of ξ(x)eµξ(x)

for some µ > 0, is sufficient for normed ergodicity. This provides new veri-fiable conditions for normed ergodicity. They are also nearly necessary andtherefore, as it seems to us, most general. The theorem generalizes a result of[17] (see Theorem 16.3.1 of [17]). It implies normed ergodicity with weightedsupremum norm

‖f‖l = supx∈X

|f(x)||l(x)| ,

where l = eµg. We call this (µ, ρ, g)− ergodicity, where ρ is the convergencerate. We also show that these conditions are also necessary for (µ, ρ, g)-ergodicity, with the exception of the uniform integrability of ξ(x) which isclose to be necessary.

Normed ergodicity was introduced in the early eighties by Hordijk andindependently by Kartoshov ( see [12] where it was applied for Blackwelloptimality (its revision is published as [4]) and [16] where a characterizationof it was given). In the paper [13] it was shown for a countable Markov chainthat normed ergodicity is equivalent to geometrical recurrence. This wasthe first applicable characterization of normed ergodicity. Inspired by this

2

theorem a similar result was proved in [17] for a Harris chain. Let us reviewthe history of normed ergodicity a bit more. Hordijk used it in his analysisof Blackwell optimality for a Markov decision chain (MDC). With respectto the probabilistic structure a MDC is a collection of MC. The ergodicityof a MDC is equivalent to the uniform ergodicity of a collection of MC.The Doeblin condition has been extensively used in MDC. In [11] variousequivalent conditions were derived, e.g. it is equivalent to the condition thatthe expected recurrence times to a finite set are bounded uniformly in allinitial states and all MC in the collection. In [7] (see also [8]) it was provedthat the Doeblin condition implied strong ergodicity uniformly in all MC. Asmentioned above the drawback of strong ergodicity is that it is too strong formost applied models and also unsuitable in case of unbounded cost structurein MDC. Therefore, as it seems, normed ergodicity is of more interest. In [13]it was shown for a countable MC which may have one or several classes ofessential states (a so-called multichained MC), that the existence of a strongLyapunov function implies normed ergodicity (see also the introduction ofthat paper for a more extended history of normed ergodicity, which wascalled there µ−geometric ergodicity where µ is eµg in this paper). In [5] thegeneralization to countable MDC with multiple classes of essential states wasshown, the extension to MDC with a general state space is done in [14] undera Harris-type condition.

The paper is organized as follows. In section 2 the definition of (µ, ρ, g)−ergodicity is given and a theorem is stated with sufficient conditions forit. In section 3 the new characterization consisting of four conditions isderived for (µ, ρ, g) − ergodicity. Three of these conditions are used in theanalysis of exponential ergodicity of [1], and one is new. In addition to theexponential ergodicity normed ergodicity requires one more large deviationrelation, which is explicitly stated. The new characterization is not easilyapplicable, but it gives valuable new insight in the necessary conditions fornormed ergodicity.

Section 4 gives a rather complete analysis of the relations between Lya-punov and strong Lyapunov functions. In our opinion, it gives the definitiveanswer to the question when does recurrence imply µ-geometric recurrenceof [13], or in the language of [17], the relation between drift and geometricdrift condition. Indeed, as it seems to us, the uniform integrability in x ofξ(x) , g(X(x, 1)) − g(x) and ξ(x)eµξ(x) for some µ > 0 is the most generaland most natural condition for converting a Lyapunov to a strong Lyapunovfunction. The section generalizes results of [6], [19] and [17].

3

In section 5 the new characterization is used in order to show that theexistence of a Lyapunov function say g, together with the above mentioneduniform integrability conditions, is sufficient for (µ, ρ, g) − ergodicity. Twotheorems are given, in the proof of the first one theorem 16.0.1 of [17] isused. The second theorem of this section is a direct application of the newcharacterization as derived in section 3. The theorem is more general thenthe results in [17], by the more general uniform integrability conditions andalso by the fact that the Lyapunov function, or drift function, is not requiredfor the one-step transition kernel, it is sufficient to have it for any powerof the transition kernel. In the recent publications [10] and [9] this moregeneral theorem is used. The section also provides a theorem on the necessitythat g is a Lyapunov function and the uniform integrability in x of ξ(x) ,g(X(x, 1)) − g(x) and ξ(x)eµξ(x) for some µ > 0 for (µ, ρ, g) − ergodicity.The uniform integrability in x of ξ(x) , g(X(x, 1)) − g(x) is however not anecessary condition, a counterexample is given.

This paper is a revised version of the sections 1-5 of [2]. As it seems, inmost applications of stochastic networks it is straightforward to check thatthe uniform integrability conditions hold in case of service times of Cramer-type. However, the construction of a Lyapunov function may be complicated.In the companion paper, which is a revised version of the sections 6-7 of [2],we construct for a class of stochastic networks a linear Lyapunov functionvia the fluid limit. Moreover, the results of this paper are applied to a d-node Jackson network with batch arrivals. In [13] it was shown as a keyapplication, that normed ergodicity holds for the 2-node Jackson networkwith an exponential bounding function l = eµg, where g is linear in thequeue sizes. It has been an open problem since then whether a similar resultis true for d-node networks with d ≥ 3. This result readily follows from theresults of this paper combined with those of [3].

2 Definition of normed ergodicity

Let X(n) = X(x, n), n ≥ 0 be a Markov Chain (MC) with initial state x, ona measurable state space (X ,B), and transition probability

P (x, n,B) = P (X(x, n) ∈ B).

We assume that the MC is ergodic with stationary probability measure π(B)for B ∈ B. For µ > 0, ρ a function from (0, µ] to [0, 1), and g a function

4

from X to R+ we introduce (µ, ρ, g)− ergodicity.

Definition 1 The MC X(n) is called (µ, ρ, g)− ergodic if there exists someconstant c such that for all n ≥ 0, and all λ ∈ (0, µ] the following relationholds

supx

∫|P (x, n, dy)− π(dy)| eλ(g(y)−g(x)) < c(ρ(λ))n (1)

This definition of ergodicity for a fixed λ is in fact equivalent to ergodicityin the bounded supremum norm defined with the bounding function l , eλg.Let Vl denote the normed space of all real-valued functions f on X with thefinite l-norm

‖f‖l = supx∈X

|f(x)||l(x)|

and the associated operator norm for a linear operator, say T : Vl −→ Vl isdefined by

‖T‖l = sup‖f‖l≤1

‖Tf‖l .

Let the linear operator T be defined by Tf(x) ,∫

(P (x, 1, dy)−π(dy))f(y))for f ∈ Vl. Then the relation (1) can be expressed as

‖P n − π‖l < cβn, (2)

where l = eλg, β = ρ(λ). This relation has played a fundamental role in thetheory on Markov Decision Chains (see the references given in the introduc-tion and [15] for recent results) and in the stochastic stability of MC (see[17] where it is called V -uniform stability). Note that (µ, ρ, g)− ergodicity,which we also call normed ergodicity for short, is a more specific condition, itrequires geometric convergence for all λ in a interval and specifies the bound-ing function l as lλ , eλg. In this paper we shall use the following notationfrequently,

V(N) , {x : g(x) ≤ N}. (3)

Note that the set V(N) depends on the parameter N . We suppress this inour notation where it is clear which N is used, or where we just assume thatV = V(N) for some N . We need the following Harris-type condition for theset V .

5

Condition 1 (H) There exist n0 ≥ 0, φ(.) a probability measure on (X ,B),and p ∈ (0, 1) such that

infx∈V

P (X(x, n0) ∈ B) ≥ pφ(B) (4)

for all B ∈ B .

We shall call a MC which satisfies the condition (H) a Harris chain. Lya-punov type functions are well-known but have different versions in the liter-ature, we will use the following definitions.

Definition 2 The function g is a Lyapunov function (L.f.) for MC X(n) if

Eg(X(x,m))− g(x) ≤ −ε + cIV (x), (5)

for some m ≥ 1, ε > 0 and c < ∞.

Definition 3 The function v is a strong Lyapunov function (s.L.f.) for MCX(n) if for some N ,

Ev(X(x,m))− v(x) ≤ −εv(x) + cI{x:v(x)≤N}(x), (6)

for some m ≥ 1, ε > 0 and c < ∞.

The relation between L.f. and s.L.f. will be analyzed in section 4. It willbe shown there that if g(x) is a L.f. and Eeµ(g(X(x,m)−g(x)) < ∞ for someµ > 0 then under uniform integrability conditions the function v(x) = eµg(x)

is a s.L.f. It is easily seen that (6) with g(x) ≥ 0, implies that

‖Pm‖v < ∞.

Also it is straightforward to show (see remark 1) that (6) implies for some v′

with v′(x) = v(x), v /∈ V that

‖V Pm‖v′ ≤ (1− ε′), (7)

for some ε′ > 0, where the taboo transition operator V P is defined by

V Pf(x) =

∫

X\VP (x, 1, dy)f(y), x ∈ X , (8)

and V P n is the n-th power of V P . In [13] it is shown for X a countable statespace and X(n) an aperiodic and multichained MC that ‖V P n‖v ≤ (1 − ε),for some ε > 0 and some n > 0 and V a finite set, is equivalent to relation(2). The following theorem for fixed λ and X a general state space followsfrom theorem 16.0.1 in [17].

6

Theorem 1 If MC X(n) satisfies condition (H) and it has a strong Lya-punov function v(x) = eµg(x) for some µ > 0, then it is (µ, ρ, v)− ergodic forsome ρ > 0.

In this paper we derive a characterization of (µ, ρ, g)− ergodicity and itwill give us a generalization of theorem 1, which is more easily applicable.

3 Characterization of normed ergodicity

In this section we develop an approach to (µ, ρ, g) − ergodicity based onthe rate of convergence in the ergodic theorem. This approach provides theconnection with estimates in the ergodic theory developed in [1], and givesnecessary and sufficient conditions for (µ, ρ, g) − ergodicity. We use thefollowing notations and condition,

∆(x, n) , supB|P (x, n, B)− π(B)|

= 12

∫|P (x, n, dy)− π(dy)| dy

andVn , {x : g(x) < hn} for a fixed h > 0.

Let τ(x) denote the entrance time of set V, i.e.

τ(x) , min{k ≥ 1 : X(x, k) ∈ V }.

The following condition is used in [1]

supx∈V

P (τ(x) > k) < ce−βk, for some c < ∞ and β > 0. (9)

The relations (4) and (9) imply the following exponential ergodicity relation(see [1] theorems 2.2 and 3.1),

supx∈Vn

∆(x, n) < ce−γn, for some c < ∞ and γ > 0 (10)

and the following large deviations estimates:

π(Bv) < ce−γv for Bv = {x : g(x) > v}, (11)

7

supn

supx∈Vn

P (g(X(x, n)) > v) < ce−γv, γ > 0. (12)

We introduce one more large deviations inequality for some β > 0, γ > 0and x such that g(x) > hn, h > 0,

P (x, n, Ax,n,j) < ce−γj, ∀j ≥ 0 (13)

whereAx,n,j , {y : g(y) > g(x)− βn + j}.

Note that if g is a L.f. with ε = 2β then with high probability X(x, n) ∈{y : g(y) < g(x)−2βn}, so Ax,n,j is a domain of large deviations for X(x, n),which explains the relation (13).

We can now formulate our first main result, which gives a characterizationof normed ergodicity. In this paper the letter c with or without index, willdenote a constant, which are not always the same, especially if they are usedin different places.

Theorem 2 The relations (10), (11), (12) and (13) are necessary and suf-ficient for (µ, ρ, g)− ergodicity.

Proof. We first prove that the conditions are sufficient, and we use thenotations

∆k = {y : g(y) ∈ [k, k + 1)}∆+

k = {y ∈ ∆k : P (x, n, dy) ≥ π(dy)}∆−

k = ∆k\∆+k .

Then the integral in (1) with λ = µ is bounded by

c∑

k

(P (x, n, ∆+k )− π(∆+

k ))eµ(k−g(x)) − c∑

k

(P (x, n, ∆−k )− π(∆−

k ))eµ(k−g(x)),

(14)for c > eµ. We consider now several cases.

Case 1: g(x) ≤ hn, where the h is the same as that in the definition ofVn. For any β > 0 the first sum

∑k in (14) is bounded by

≤∑

k≤βn

+∑

k>βn

8

using (10) we find

∑

k≤βn

≤ c∑

k≤βn

e−γn+µ(k−g(x))

≤ c1e−γn+µβn

≤ c1e− γn

2 ,

for µ ≤ γ2β

. From (11) and(12) we have

∑

k>βn

≤ 2c∑

k>βn

e−γk+µk

≤ c1e− γβn

2 ,

for 2µ ≤ γ. The second sum∑

k corresponding to ∆−k in (14) goes similarly.

Hence

supx:g(x)≤hn

∫|P (x, n, dy)− π(dy)| eµ(g(y)−g(x)) < cρn,

forµ ≤ ( γ

2β∧ γ

2) and ρ ≥ (e−

γ2 ∨ e−

γβ2 ).

Case 2: g(x) > hn.In this case we do not need ergodicity (relation (10)) and we only use

estimates for

I1 =

∫π(dy)eµ(g(y)−g(x))

and

I2 =

∫P (x, n, dy)eµ(g(y)−g(x)).

From (11) for µ ≤ γ2

I1 ≤ c∑

k

π(∆k)eµ(k−g(x))

≤ c1e−µg(x) ≤ c1e

−µhn. (15)

For I2 we have a more complicated estimation, denote

I3 = c∑

k≤g(x)−βn

P (x, n, ∆k)eµ(k−g(x))

9

andI4 = c

∑

k>g(x)−βn

P (x, n, ∆k)eµ(k−g(x)),

where β will be chosen later on. Then

I2 ≤ I3 + I4,

and we continue with bounds for I3 and I4.

I3 ≤ c1e−µβn, (16)

since∑

k P (x, n, ∆k) ≤ 1.

I4 ≤ c∑j≥0

P (x, n, ∆g(x)−βn+j)eµ(−βn+j)

≤ ce−µβn∑j≥0

P (x, n, Ax,n,j)eµj,

and for β > 0 satisfying (13) we find the following inequality

≤ c1e−µβn

∑j≥0

e−(γ+µ)j ≤ c2e−µβn

where we take µ < γ2. Together with (16) this gives

I2 < ce−µβn.

Hence,

supx:g(x)>hn

∫|P (x, n, dy)− π(dy)| eµ(g(y)−g(x)) < cρn

1 ,

withρ1 ≥ (e−µh ∨ e−µβ).

Consequently, as follows from the bounds in both cases the relation (1) holdsfor

µ ≤ ( γ2β∧ γ

2) and ρ ≥ (e−

γ2 ∨ e−

γβ2 ∨ e−µh ∨ e−µβ). (17)

We next show that the conditions (10) to (13) are also necessary.The relation (1) can be rewritten as

supx

∫|P (x, n, dy)− π(dy)| eµg(y) ≤ cρneµg(x).

10

Because, eµg(y) ≥ 1 and g(x) ≤ hn for x ∈ Vn, it follows from this relationthat

supx∈Vn

∆(x, n) ≤ cρneµhn ≤ ce−γn,

for γ = − ln ρ2

and h ≤ γµ. Hence (10) holds for these values of γ and h. We

prove next that (11) follows from (1) for n = 0. Indeed, if we take any fixedstate x0 then it follows from (1) that

∫π(dy)eµg(y) ≤ c1 < ∞. (18)

Recall the notation Bv of relation (11), since eµg(x) > eµv is satisfied forx ∈ Bv we find from a Chebyshev type inequality that

π(Bv) ≤ c1e−µv,

which implies relation (11) with γ = µ. The proof of relation (12) goes asfollows.

0 ≤ supn

supx∈Vn

P (g(X(x, n)) > v)

≤ π(Bv) + supn

supx∈Vn

∫

Bv

|P (x, n, dy)− π(dy)|

≤ ce−γv + e−µv supn

supx∈Vn

∫

Bv

|P (x, n, dy)− π(dy)| eµg(y)

≤ ce−γv + ce−µv supn

supx∈Vn

ρneµg(x)

≤ ce−γv + ce−µv supn

ρneµhn ≤ c1e−γv,

for h = − ln ρµ

and γ < µ. For the proof of relation (13) we write

P (x, n, Ax,n,j)

≤∫

Ax,n,j

P (x, n, dy)eµ(g(y)−g(x)+βn−j)

≤ e−µj+βµn[

∫

Ax,n,j

(P (x, n, dy)− π(dy))eµ(g(y)−g(x)) +

∫

Ax,n,j

π(dy)eµ(g(y)−g(x))]

≤ e−µj+βµn[cρn + e−µg(x)

∫

Ax,n,j

π(dy)eµg(y)].

11

Since∫

Ax,n,jπ(dy)eµg(y) < ∞, as follows from (18), we have for x such that

g(x) > hn,

P (x, n, Ax,n,j) ≤ e−µj+βµn[c(ρn + e−µhn)]

≤ ce−µj,

for β ≤ (− ln ρµ∧ µh). This completes the proof.

In section 5 we come back to this characterization and we will use it in theproof that (µ, ρ, g)-ergodicity holds under an uniform integrability conditionif g is a Lyapunov function.

4 On Lyapunov functions

In this section we analyze the relation between Lyapunov and strong Lya-punov functions. Using Chebyshev’s inequality related results have beenobtained in [6], [19] and [17]. However, it seems that theorem 3 gives morenatural and applicable conditions. We shall use uniform integrability condi-tions in various forms. Let ξ(x), x ∈ X be a family of random variables. Weneed the following conditions:

Condition (UIξ): The r.v. ξ(x) are uniformly integrable (u.i.).Condition (UIξ)λ: There exist λ > 0 such that ξ(x)eλξ(x) are uniformly

integrable (in x). We will especially use these conditions for

ξ(x) , g(X(x, 1))− g(x),

and we denote the conditions for this choice of ξ as (UI) and (UI)λ.The conditions (UIξ) and (UIξ)λ imply that

a(x) , Eξ(x)

are uniformly bounded in x and for ξ0(x) , ξ(x)− a(x)

supx

−M∫

−∞

tP (ξ0(x) ∈ dt) → 0, (19)

supx

∞∫

M

teλtP (ξ0(x) ∈ dt) → 0, (20)

12

as M →∞.It is well-known that if ξ(x) are u.i. then

supx

E |ξ(x)| < ∞. (21)

In (UIξ)λ we assume that eλξ(x) are u.i. for some λ > 0. This is implied (seelemma 1 below) by

supx

Eeλ′ξ(x) < ∞, (22)

for some λ′ > λ. Conversely, the u.i. of eλξ(x) implies (22) for all λ′ ≤ λ.Similarly, the u.i. of ξ(x)eλξ(x) follows from

supx

E∣∣∣ξ(x)eλ′ξ(x)

∣∣∣ < ∞, (23)

for some λ′ > λ, and it implies (23) for all λ′ ≤ λ. These relations for

ξ(x) , g(X(x, 1))− g(x) (24)

imply that,sup

xEeλ′ξ(x) = ‖P‖v , with v = eλ′g(x),

hence relation (22) for this choice of the ξ(x) is equivalent to ‖P‖v < ∞.The relation (6) with v(x) = eµg(x) and m = 1 can be rewritten as

Eeµξ(x) ≤ (1− ε) + cIV (x), x ∈ X , (25)

from which relation (22) follows. It becomes clear that relation (22) is anatural condition in the context of (µ, ρ, g)-ergodicity. For the use lateron we summarize in the following lemma the relationships between theseconditions, its proof is straightforward and will be omitted.

Lemma 1 The following implications hold:

1. (22) for λ > 0 implies (23) for λ′ : 0 < λ′ < λ,

2. (23) for λ > 0 implies (22) for λ′ : 0 < λ′ ≤ λ,

3. if ξ(x) ≥ c > −∞ then (22) for λ > 0 implies (21) and the condition(UIξ).

13

4. (22) for λ > 0 implies the u.i. of eλ′ξ(x) and the condition (UIξ)λ′ for0 < λ′ < λ.

We need another technical lemma in which we use the following notations,

a(x) , Eξ(x) and ψ(x, λ) , Eeλξ(x).

Lemma 2 The conditions (UIξ) and (UIξ)λ imply that for any γ > 0, thereexists λ′ > 0 such that ∀λ : 0 < λ < λ′

supx

(ln ψ(x, λ)− λa(x)) < γλ. (26)

Proof. Denote ξ0(x) , ξ(x)−a(x) and φx(λ) , Eeλξ0(x) then Eξ0(x) = 0and

φx(λ) = 1 + λφ′x(λ), λ ∈ (0, λ),

φ′x(λ0) =

∫teλ0tP (ξ0(x) ∈ dt) =

∫t(eλ0t − 1)P (ξ0(x) ∈ dt). (27)

It follows from (19) that we can choose M such that for all x ∈ X

−∫

t<−M

tP (ξ0(x) ∈ dt) <γ

4

and from (20) that M also satisfies

∫

t>M

teλ0tP (ξ0(x) ∈ dt) <γ

4.

Choose λ′ ≤ λ such that∣∣t(eλ′t − 1)

∣∣ < γ2

for |t| ≤ M , then for λ ≤ λ′

φ′x(λ) ≤ φ′x(λ′) ≤ γ.

Hence by (27)

ln ψx(λ)− a(x) = ln Eeλξ0(x) = ln φx(λ) ≤ ln(1 + λγ) ≤ λγ,

which completes the proof.Note that the lemma says that the r.h.s. of (26) is o(1) uniformly in x as

λ → 0. We can now state the main result of this section.

14

Theorem 3 The following implications hold:

1. If v(x) ≥ 1 is s.L.f. then g(x) , ln v(x) is L.f.,

2. If conditions (UI) and (UI)λ, λ > 0, hold and g is L.f. then for some

µ0 > 0 the function v(x) , eµg(x) satisfies (6) for µ : 0 < µ < µ0 andhence v is s.L.f.

Proof. We first prove the second implication. The condition (UI)λ im-plies relation (23). By Lemma 1 we have that the relation (23) impliesrelation (22) for µ ≤ λ. Hence the relation (6) is satisfied for these values ofµ if x ∈ V . It remains to show that

Eeµξ(x) ≤ 1− ε, x /∈ V,

or sufficiently small µ. This follows from Lemma 2 since according to (5)

supx/∈V

a(x) = supx/∈V

Eξ(x) ≤ −ε,

and relation (26) implies that for x /∈ V ,

ln Eeλξ(x) ≤ −λε

and soEeλξ(x) ≤ e−λε,

from which (20) follows.To finish the proof we show the first implication. Since the function ln is

a convex function, we have for x /∈ V

Eg(X(x,m))− g(x) = E lnv(X(x,m))

v(x)

≤ ln Ev(X(x,m))

v(x)

≤ ln(1− ε) < ε′ < 0,

for some ε′. This completes the proof.

Note that due to Lemma 1 condition (22) implies that for an increasingLyapunov function g with bounded derivative, it holds that v = eλg is astrong Lyapunov function in case the jumps are bounded from below. Thisis often the case in random walks, e.g. the left skip free random walks (see[19]).

15

5 Applicable conditions for

normed ergodicity

In this section we give two proofs for the assertion that the existence of L.f.g and the conditions (UI) and (UI)λ imply (µ, ρ, g)-ergodicity.

The first proof is based on theorem 16.0.1 of [17]. Moreover, it is related totheorem 16.3.1 of [17]. However, the theorem below seems to be stronger sincethe relations (UI) and (UI)λ are more general applicable than the conditionsof theorem 16.3.1 of [17].

Theorem 4 Assume that the MC is a Harris chain and suppose that theconditions (UI) and (UI)λ hold. Then the MC is (µ, ρ, g)− ergodic if g is aLyapunov function with m = 1 in condition (L).

Proof. The proof follows immediately from theorem 3 together withtheorem 16.0.1 in [17].

Using the characterization of normed ergodicity as it is derived in theorem2 we give a direct proof of this result. Actually, we will prove a slightly moregeneral theorem. Before we prove the theorem more notation and a technicallemma is needed.

DenoteAn , {g(X(x, 1)) > N, . . . , g(X(x, n)) > N}

Lemma 3 Assume that g is a Lyapunov function with m = 1, ε > 0 andV = {x : g(x) ≤ N}, and suppose that the conditions (UI) and (UI)λ. Thenthere exists λ′ > 0 such that for all λ : 0 < λ < λ′

P (g(X(x, n)) > B; An) ≤ e−λ(B−g(x))− ελn2

.

Proof. Denote

ξi , g(X(x, i))− g(X(x, i− 1)), Zn ,n∑

i=1

ξi,Fi , σ(X(0), . . . , X(i))

Pn , P (g(X(x, n)) > B; An) = P (Zn > B − g(x); An),

sinceg(X(x, n)) = g(x) + Zn.

16

The function g is Lyapunov for m = 1, hence

E(ξi p Fi) ≤ −ε on the set {ω : g(X(x, i− 1)) > N}.

The Chebyshev inequality gives

Pn ≤ e−λ(B−g(x))En, (28)

where

En , E(eλZn ; An−1) = E(eλξn+λZn−1 ; An−1)

= E(E(eλξn p Fn−1)eλZn−1I(An−1)).

From the Lyapunov condition it follows that

an , an(ω) , E(ξn p Fn−1) ≤ −ε for ω ∈ An−1.

Using lemma 1 we find that

En ≤ e−λεE(E(eλ(ξn−an) p Fn−1)eλZn−1I(An−1)) ≤ e−

λε2 En−1,

for sufficiently small λ. Hence

En ≤ e−ελn2 .

This inequality together with relation (28) completes the proof.

We are now ready to give a direct proof that the existence of a Lyapunovfunction together with the uniform integrability conditions is sufficient forthe conditions of theorem 2 and therefore the (µ, ρ, g)-ergodicity holds.

Theorem 5 Assume that the MC is a Harris chain and suppose that theconditions (UI) and (UI)λ are satisfied. Then the MC is (µ, ρ, g) − ergodicif g is a Lyapunov function for any m ≥ 1 in condition (5).

Proof. We first suppose that the relation (5) holds for m = 1, and weprove that this together with conditions (H) and (UI)λ imply the relations(10), (11) and (12). Therefore we show that MC X(n) satisfies the conditions(Ig,MC)1 and (Ig,MC)2 of [1] (see [1] p.16 and lemma 2.1). For conveniencewe write them here as follows. For some µ > 0

17

(I)1 : P (τ(x)− αg(x) > k) < g(x)e−µk for x /∈ V.

(I)2 : supx∈V

P (g(X(x, 1) > k) < e−µk.

The latter condition follows from (UI)λ. For the first condition we have

P (τ(x) > n) = P (An)

whereAn = {g(X(x, 1)) > N, . . . , g(X(x, n)) > N}.

Substitute N for B in lemma 3. Then for n = αg(x) + k

P (An) ≤ e−λ(N−g(x))− ελn2 = e−λN− ελk

2+g(x)(1−αε

2 )λ.

For α = 2ε

we have

P (An) = P (τ(x) > αg(x) + k) ≤ e−λN− ελk2 .

This proves (I)1. Together with (I)2 it proves (I, MC) (see lemma 2.1 in [1]).Applying now the theorems 2.2 and 3.1 of [1] gives that the conditions (10),(11) and (12) are satisfied. We next prove that also condition (13) holds.Therefore we bound

P (x, n, Ax,n,j) = P (g(X(x, n)) > s),

for s = g(x)− βn + j. Let θ be the last hitting time of set V before n, i.e.

θ , max{k ≤ n : g(X(k)) ≤ N, g(X(k + 1)) > N, . . . , g(X(n)) > N}

andk = n if g(X(k)) > N, . . . , g(X(n)) > N, k = 1, . . . , n.

Recall that in relation (13) we have that g(X) > hn. Hence

P (g(X(x, n)) > s) ≤n−1∑i=1

P (g(X(x, n)) > s) + P (g(X(x, n)) > s; An)

18

and by lemma 3

≤n−1∑i=1

supz∈V

P (g(X(z, n− i)) > s) + P (g(X(x, n)) > s; An)

≤ cne−γs + e−λ(s−g(x)−λn2 ≤ cne−γ(hn−βn+j) + eλ(βn−j)− ελn

2

≤ c1(e−γj + e−λj), for β ≤ (h

2∧ ε

2),

which completes the proof of (13).If g is a Lyapunov function for m > 1 then we consider the MC X(n) ,

X(nm) with transition operator Pm. By lemma 1 the condition (UI)λ implies(22) and hence we have ‖P‖h < ∞ for h , eλg(x), from which

∥∥P k∥∥

h< ∞ for

k ≥ 0. For MC X(n) the normed ergodicity follows similarly as for the proofof m = 1. Note that the relations (21) and (23) for ξ(x) = g(X(x, 1))− g(x),imply that they also hold for ξ(x) = g(X(x,m))− g(x). Hence,

‖P nm − π‖h < cρn,

from well-known norm inequalities we find for n = km+r with 0 ≤ r ≤ m−1that

‖P n − π‖h ≤∥∥P km − π

∥∥h.(‖P − π‖h)

r ≤ c1ρn1 ,

where ρ1 = ρ1m . Hence also the normed ergodicity for the MC X(n) follows.

This completes the proof of the theorem.

Remark 1 . Let us point out that the conditions (I, MC) and (I)1 followreadily if we have that v with v(x) ≥ 1 for all x is a s.L.f.

Suppose v is s.L.f. then it directly follows from relation (6) that

‖Pm‖v < ∞

For simplicity in notation let us continue with m = 1. Relation (6) gives,

Pv ≤ (1− ε)v + cIV ,

where c can be taken as c , supx∈V Pv(x). Define

v′(x) ,{

v(x), x /∈ V,c1v(x), x ∈ V,

19

and recall the notation defined in relation (8). Then

V Pv′(x) ≤{

(1− ε)v′, x /∈ V(1− ε)v(x) + c, x ∈ V.

If we choose c1 such that

(1− ε)v(x) + c ≤ (1− ε)c1v(x),

we have that

V Pv′ ≤ (1− ε)v′. (29)

The relation (29) with v′ ≥ e, where e is the function e(x) , 1 for all x,implies the conditions (I)1 and (I, MC). Indeed, it follows that

P (τ(x) > n) = (V P ne(x)) ≤ (V P nv′(x)) ≤ (1− ε)nv′(x).

Hence,supx∈V

P (τ(x) > n) ≤ supx∈V

v′(x)(1− ε)n

and so the condition (I, MC) is satisfied. Similarly,

P (τ(x) > n + αv′(x)) ≤ (1− ε)n+αv′(x)v′(x), x /∈ V.

Hence also the condition (I)1 holds.

Except the uniform integrability of ξ(x), x ∈ X , all conditions in theorem5 are also necessary. Before we prove this we give a counterexample whichshows that (UI) may not hold in case of normed ergodicity. Denote ζ(x) ,(X(x, 1)−x) and define the MC X(n) by ζ(x) = −x, X = R+. Then clearlythe family ξ(x) with g(x) = x, x ∈ X , is not uniformly integrable and so (UI)does not hold. But, X(x, 2) = 0, x ∈ X , and hence the MC is (µ, ρ, g)-ergodicwith ρ ≡ 0.

Theorem 6 Assume that the MC X(n) which satisfies (UIζ) with ζ(x) ,(X(x, 1) − x), is (µ, ρ, g)-ergodic. Then X(n) satisfies the condition (UI)λ

and it holds that g is a Lyapunov function.

Proof. By definition the relation (1) holds for n = 0. Hence ‖π‖h < ∞,with h = eµg. Relation (1) for n = 1 gives that ‖P − π‖h < ∞ hence also

20

‖P‖h < ∞, which is equivalent to condition (22) with ξ(x) as in (24). Lemma1 then implies that (UI)λ holds for λ : 0 < λ < µ. For proving that g is aLyapunov function we choose n such that the r.h.s. of relation (1) for n issmaller than ε < 1

3. Then with the notation of (3) for V(N) we have for N

sufficiently large that∫

V c(N)

P (x, n, dy)eµg(y) ≤∫

V c(N)

|P (x, n, dy)− π(dy)| eµg(y) +

∫

V c(N)

π(dy)eµg(y)

< 2εeµg(x). (30)

The condition (UIζ) implies that for any δ > 0, there is an M ≥ 1 such thatfor ζn(x) , (X(x, n)− x)

−∫

t<−M

tP (ζn(x) ∈ dt) < δ,

for all x. Hence for m > N + M and x ∈ V cm

∫

V(N)

P (x, n, dy)eµg(y) ≤ δeµN ≤ ε, (31)

for δ < εe−µN . Combining (30) and (31), we find for x ∈ V cm that

∫P (x, n, dy)eµg(y) ≤ 3εeµg(x).

This shows that relation (6) is satisfied for V = Vm and x ∈ V cm . In order to

complete the proof of relation (6) for all x we only need that

supx∈Vm

∫P (x, n, dy)eµg(y) < ∞,

which follows from the fact that ‖P n‖h < ∞. Hence, eµg(x) is a strongLyapunov function and theorem 3 then gives that g is a L.f. This completesthe proof.

6 Conclusion

Normed ergodicity has shown to be a valuable and useful type of ergodicity,it has been used in numerous papers. In this paper we have given:

21

• a new characterization of normed ergodicity

• we analyzed the relation with Borovkov’s ergodicity analysis in [1]

• we derived as it seems, the most general and natural relation betweenLyapunov and strongly Lyapunov function

• we derived new and more general sufficient conditions for normed er-godicity, which have been used in [10] and [9]

• we showed that these conditions are with the exception of one alsonecessary.

References

[1] Borovkov, A. A. (1998) Ergodicity and Stability of Stochastic Processes.Wiley, Chicester.

[2] Borovkov, A. A. and Hordijk, A. (2000) On normed ergodicity of Markovchains. Technical Report MI 2000-40 Leiden University.

[3] Borovkov, A. A. and Hordijk, A. (2003) On linear Lyapunov functionsand normed ergodicity for stochastic networks. In preparation.

[4] Dekker, R. and Hordijk, A. (1988) Average, sensitive and Blackwelloptimal policies in denumerable Markov decision chains with unboundedrewards. Math. Oper. Res. 13, 395-421.

[5] Dekker, R., Hordijk A. and Spieksma, F. M. (1994) On the relationbetween recurrence and ergodicity properties in denumerable Markovdecision chains. Math. Oper. Res. 19, 539-559.

[6] Fayolle, G., Malyshev, V. A. and Menshikov, M. V. (1995) Topics in theConstructive Theory of Countable Markov Chains. Cambrigde Univer-sity Press, Cambridge.

[7] Federgruen, A. , Hordijk, A. and Tijms, H. C. (1978) Recurrence con-ditions in denumerable state Markov decision processes. In DynamicProgramming and its Applications (M. L. Puterman, ED.). AcademicPress, New York, pp.3-22.

22

[8] Federgruen, A. , Hordijk, A. and Tijms, H. C. (1978) Simultaneousrecurrence condition on a set of denumerable stochastic matrices. J.Appl. Prob. 15, 842-847.

[9] Heidergott, B and Hordijk, A (2003) Tayler series expansions for sta-tionary Markov chains. Technical Report MI 2003-02, Leiden University.To appear in: Adv. Appl. Prob. 35.

[10] Heidergott, B, Hordijk, A and Weisshaupt, H.(2002) Measure–ValuedDifferentiation for Stationary Markov Chains. EURANDOM report2002-027.

[11] Hordijk, A. (1974) Dynamic Programming and Markov Potential The-ory. Mathematical Centre Tract 51, C. W. I. , Amsterdam.

[12] Hordijk, A. and Dekker, R. (1983) Average, sensitive and Blackwell op-timal policies in denumerable Markov decision chains with unboundedrewards. Report No.˜83–36, Institute of Applied Mathematics and Com-puting Science, Leiden University.

[13] Hordijk, A. and Spieksma, F. M. (1992) On ergodicity and recurrenceproperties of a Markov chain with an application to an open Jacksonnetwork. Adv. Appl. Prob. 24, 343-376.

[14] Hordijk, A. and Spieksma, F. M. and Tweedie, R. L. (1995) Uniformstability for general state space Markov decision processes. Technicalreport.

[15] Hordijk, A. and Yushkevich, A. A. (1999) Blackwell optimality in theclass of all policies in Markov decision chains with a Borel state spaceand unbounded rewards. Math. Meth. Oper. Res. 50, 421-448.

[16] Kartoschov, N. (1985) Inequalities in theorems of ergodicity and stabilityfor Markov chains with common phase space.Theory Prob.˜Appl. 30,247–259.

[17] Meyn, S. P. and Tweedie, R. L. (1993) Markov Chains and StochasticStability. Springer, London.

[18] Neveu, J. (1965) Mathematical Foundations of the Calculus of Proba-bility. Holden Day, San Francisco.

23

[19] Spieksma, F. M. and Tweedie, R. L. (1994) Strenghtening ergodicity togeometric ergodicity for Markov chains. Stoch. Models. 10, 45-75.

24