The Right Way to do Reinforcement Learning
with Function Approximation
Rich SuttonAT&T Labswith thanks to
Satinder Singh, David McAllester, Mike Kearns
The Prize• To find the “Right Way” to do RL with FA
– sound (stable, non-divergent)– ends up with a good policy– gets there quickly, efficiently– applicable to any (discrete-time, finite-state) MDP– compatible with (at least) linear FA– online and incremental
• To prove that it is so
• Tensions:– Proof and practice often pull in different directions– Lack of knowledge negative knowledge– We have not handled this well as a field
critical to viability of RL !
Outline• Questions• History: from policy to value back to policy• Problem Definition
– Why function approximation changes everything
• REINFORCE• Policy Gradient Theory• Do we need values? Do we need TD?
– Return baselines - using values without bias– TD/boostrapping/truncation
• may not be possible without bias• but seems essential for reducing variance
QuestionsIs RL theory fundamentally different/harder with FA? yes
Are value methods unsound with FA? absolutely not
Should we prefer policy methods for other reasons? probably
Is it sufficient to learn just a policy, not value? apparently not
Didn’t we already do all this policy stuff in the 1980s? only some of it
Can values be used without introducing bias? yes
Can TD (bootstrapping) be done without bias? I wish
Is TD much more efficient than Monte Carlo? apparently
Is it TD that makes FA hard? yes and no, but mostly no
So are we stuck with dual, “actor-critic” methods? maybe so
Are we talking about genetic algorithms? No!
What about learning “heuristic” or “relative” values. Are these policy methods or value methods? policy
The Swing towards Value Functions
• Early RL methods all used parameterized policies• But adding value functions seemed key to
efficiency• Why not just learn action value functions and compute policies from them!
• A prediction problem - almost supervised• Fewer parameters• Enabled first proofs of convergence to optimal
policy• Impressive applications using FA• So successful that early policy work was bypassed
Q*(s,a) =E r1 +γr2 +γ2r3 +L s0 =s,a0 =a,π *
π * (s) =argmaxa
Q* (s,a)CleanerSimplerEasier to use
Q-learningWatkins, 1989
• Theory hit a brick wall for RL with FA• Q-learning shown to diverge with linear FA• Many counterexamples to convergence• Widespread scepticism about any argmax VF
solution– that is, about any way to get conventional convergence
• But is this really a problem? • In practice, on-policy methods perform well • Is this only a problem for our theory?• With Gordon’s latest result these concerns seem
to have been hasty, now invalid
The Swing away from Value Functions
Why?
Diagram
Why FA makes RL hard
• All the states interact and must be balanced, traded off• Which states are visited is affected by the policy
A small change (or error) in the VF estimate can cause a large, discontinuous change in the
policy can cause a large change in the VF estimate
Diagram of What Happens in— Value Function Space —
inadmiss
able value fu
nctions
value fu
nctions c
onsiste
nt with
True V*Region of π*
best admissable
policy
Original naïve hope
guaranteed convergenc
eto good
policy
Res gradient et al.
guaranteed convergenc
eto less
desirable policy
Sarsa, TD() & other on-policy methods
chatteringwithout
divergence or guaranteed
convergence
Q-learning, DP & otheroff-policy methods
divergence possible
…and towards Policy Parameterization
• A parameterized policy (PP) can be changed continuously
• A PP can find stochastic policies– the optimal policy will often be stochastic with FA
• A PP can omit the argmax (the action-space search)– necessary for large/continuous action spaces
• A PP can be more direct, simpler• Prior knowledge is often better expressed as a PP
• A PP method can be proven convergent to a local optimum for general differentiable FA
REINFORCE Williams, 1988
Defining the Problem (RL with FA)
Part I: Parameterized PoliciesFinite state and action sets
Discrete time
Transition probabilities
Expected rewards
Stochastic policy
possibly parameterized
S A N
t 0,1,2,3, K
ps s a Pr st 1 s st s,at a
sa E rt 1 st s,at a
π (s,a) Pr at a st s
π w.l.o.g. n
n N
s ′ s a rπ , p
L L e.g.,
Examples of Policy Parameterizations
Gibbs
or =1Normalization
featuresof
= weightsRanking #s,one per action
ActionProbabilities
π (st ,a)
Gibbsor
=1
featuresof and
Ranking # forrepeat
ActionProbabilities
π (st ,a)
st
st a
a Aa
Ranking #s are mechanistically like action valuesBut do not have value semanticsMany “heuristic” or “relative” values are better viewed as ranking #s
More Policy ParameterizationsContinuous actions work too, e.g.:
GaussianSampler
featuresof
= weightsimplicitly determine the
continuous distribution
st
mean of
std. dev. of a t
a t
a t
π (st ,a)Much stranger parameterizations are possible
e.g., cascades of interacting stochastic processes
such as in a communications network or factory
We require only that our policy process produces
at according to some distribution π (st,a)∇π (st ,a)
perhaps π (st ,a)
Choose π to maximize a measure of total future reward,called the return
Values are expected returns
Defining the Problem (RL with FA) II
Rt
Vπ(s) E Rt st s,π
Qπ(s,a) E Rt st s,at a,π
Optimal policies
π * arg maxπ
Vπ(s) s S π *
(s,a) arg maxa
Qπ(s,a)
Value methods maintain a parameterized approximation to a value function,
And then compute their policy, e.g.,
Vπ, Q
π, V
π *
, or Qπ *
π (s) arg maxa
ˆ Q (s,a)
FA Breaks the Standard Problem Def’n!
Discounted case
One infinite, ergodic episode: s0, a0, r1, s1, a1, r2 , s2 , K
Let be the space of all policiesLet be all policies consistent with the parameterizationProblem: depends on s!
no one policy—in —is best for all statesstates compete for control of
Need an overall (not per state) measure of policy quality, e.g.,
argmaxπ∈Π
Vπ (s)
J(π) = Vπ (s)dπ (s)s∑
dπ
(s)asymptotic fraction of time spent in s under πBut! Thm: J(π) is independent of J(π) = ave. reward/step
Rt =1
1−γ rt+1 +γrt+2 +γ2rt+3 +L( )Return:
RL Cases Consistent with FA
Average-reward case
J(π) = average reward per time step underπ
Rt = rt+k −J (π )[ ]k=1
∞
∑
Episodic case
J(π) =Vπ (s0 )
Rt =rt+1 +γrt+2 +γ2rt+3 +L +γT−k−1rT
Many epsiodes, all starting from
One infinite, ergodic episode: s0, a0, r1, s1, a1, r2 , s2 , K
s0
Outline• Questions• History: from policy to value back to policy• Problem Definition
– Why function approximation changes everything
• REINFORCE• Policy Gradient Theory• Do we need values? Do we need TD?
– Return baselines - using values without bias– TD/boostrapping/truncation
• may not be possible without bias• but seems essential for reducing variance
Do we need Values at all?
Extended REINFORCE (Williams, 1988)
Δ t = α Rt
∇θπ (st ,at )
π (st ,at )Δ = Δt
t∑ offline
updating
episodiccase
Thm: Eπ Δ{ } =α∇ J (π )There is also an online, incremental implementation using
eligibility traces
Converges to a local optimum of J for general diff. FA!Simple, clean, a single parameter...Why didn’t we love this algorithm in 1988??
No TD/bootstrapping (a Monte Carlo method) thought to be inefficient Extended to average-case
(Baxter and Bartlett, 1999)
Policy Gradient Theorem
Thm: ∇ J(π ) = dπ (s) Qπ (s, a)∇θπ (s,a)a
∑s
∑Marbach & Tsitsiklis ‘98Jaakkola Singh Jordan ‘95Cao & Chen ‘97Sutt McAl Sing Mans ‘99Konda & Tsitsiklis ‘99Williams ‘88
how often soccurs under π
does not involve
∇d
π (s) !
Policy Gradient Theory
Thm: ∇ J(π ) = dπ (s) Qπ (s, a)∇θπ (s,a)a
∑s
∑
how often soccurs under π
= dπ (s) Qπ (s,a) − b(s)[ ]∇θπ (s,a)a
∑s
∑ for any b : S → ℜ
∇π(s,a)
a∑ = 0 ∀s
= dπ (s) π (s,a) Qπ (s,a) − b(s)[ ]∇θπ (s,a)π (s,a)a
∑s
∑
how often s,a occurs under π
=Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
t
∇ J(π ) = Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st , at )
π (st , at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
=Eπ Rt
∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪REINFORCE
t
∇ J(π ) = Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st , at )
π (st , at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
=Eπ rt +1 + γV π (st +1) − V π (st )[ ]
∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
OR
≈Eπ rt+1 + γ ˆ V π (st +1) − ˆ V π (st )[ ]
∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪actor-critic
≈Eπ Rtλ − ˆ V π (st )[ ]
∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
OR
general form,includes all above
possibleTD/bootstrapping
idealbaseline?
=Eπ Rt
∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪REINFORCE
t
∇ J(π ) = Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st , at )
π (st , at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Conjecture: The ideal baseline is
b(s) =Vπ (s)
In which case our error term is an advantage
Qπ (st,at)−Vπ (st) =Aπ (st,at)Baird ‘93
No bias is introduced by an approximation here:
b(s) = ˆ Vπ (s)
How important is a baseline to the efficiency of REINFORCE?
Apparently very important, but previous tests flawed
Random MDP Testbed
• 50 Randomly constructed episodic MDPs– 50 states, uniform starting distribution– 2 actions/state– 2 possible next states per action– expected rewards (1,1); actual rewards +(0,0.1)– 0.1 prob of termination on each step
• State aggregation FA - 5 groups of 10 states each• Gibbs action selection• Baseline learned by gradient descent• Parameters initially• Step-size parameters
= ˆ V π (s)
=0 w = 0
Effect of Learned Baseline
.01 .1 1 10.0
10.8
No Baseline
α
01
02
01
2
J(π)after 50
episodes
Much better to learn a baseline approximating Vπ
REINFORCE withper-episode updating
∇ J(π ) = Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st , at )
π (st , at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Can We TD without Introducing Bias?
Thm: An approximation Q can replace Q without bias if it is of the form
ˆ Q (s,a) =wT ∇π (s,a)π (s,a)
and has converged to a local optimum.
Sutton et al. ‘99Konda & Tsitsiklis ‘99
However!Thm: Under batch updating, such a Q results in exactly the same updates as REINFORCE. There is no useful bootstrapping.
Empirically, there is also no win with per-episode updating
Singh McAllester Suttonunpublished
Effect of Unbiased Linear Q^
.01 .1 1 0.1
1.7
J(π)after 50
episodes
α
REINFORCE Unbiasedat best
ˆ Q
per-episode updating
TD Creates Bias; Must We TD?
Is TD really more efficient than Monte Carlo?
Apparently “Yes”,but this question deserves a better answer
accumulatingtraces
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
RANDOM WALK
50
100
150
200
250
300
Failures per100,000 steps
0 0.2 0. 0.6 0.8 1
CART AND POLE
00
50
500
550
600
650
700
Steps perepisode
0 0.2 0. 0.6 0.8 1
MOUNTAIN CAR
replacingtraces
150
160
170
180
190
200
210
220
230
20
Cost perepisode
0 0.2 0. 0.6 0.8 1
PUDDLE WORLD
replacingtraces
accumulatingtraces
replacingtraces
accumulatingtraces
RMS error
Is it TD that makes FA hard?
• Yes, TD prediction with FA is trickier than Monte Carlo– even the linear case converges only near an optimum– nonlinear cases can even diverge
• No, TD is not the reason the control case is hard
• This problem is intrinsic to control + FA• It happens even with Monte Carlo methods
small change in value
discontinuouschange in policylarge change in
state distribution
large
Small Sample Importance Sampling -
A Superior Eligibility Term?Thm: ∇ J(π ) = dπ (s) Qπ (s, a)∇θπ (s,a)
a∑
s∑
how often soccurs under π
= dπ (s) Qπ (s,a) − b(s)[ ]∇θπ (s,a)a
∑s
∑ for any b : S → ℜ
∇π(s,a)
a∑ = 0 ∀s
= dπ (s) π (s,a) Qπ (s,a) − b(s)[ ]∇θπ (s,a)π (s,a)a
∑s
∑
how often s,a occurs under π
=Eπ Qπ (st ,at ) − b(st )[ ]∇θπ (st ,at )
π (st ,at )
⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
t
n?
n?
Questions
Is RL theory fundamentally different/harder with FA? yes
Are value methods unsound with FA? absolutely not
Should we prefer policy methods for other reasons? probably
Is it sufficient to learn just a policy, not value? apparently not
Can values be used without introducing bias? yes
Can TD (bootstrapping) be done without bias? I wish
Is TD much more efficient than Monte Carlo? apparently
Is it TD that makes FA hard? yes and no, but mostly
no
So are we stuck with dual, “actor-critic” methods? maybe so