Incorporating Advice into Agents that Learn from Reinforcement Presented by Alp Sardağ
Jan 14, 2016
Incorporating Advice into Agents that Learn from
Reinforcement
Presented by
Alp Sardağ
Problem of RL• Reinforcement Learning usually requires a
large number of trainning episodes.
HOW OVERCOME?• Two approachs:
• Implicit representation of the utility function• Allowing Q-learner to accept given advice at
any time and, in a natural manner.
Input Generalization• To learn how to play game (chess 10120
states), impossible to visit all these states• Implicit representation of the function: A form
that allows to calculate the output for any input, much more compact than tabular form.
Example:
U(i) = w1f1(i)+w2f2(i)+...+wnfn(i)
Connectionist Q-learning• As the function to be learned is characterized by a
vector of weights w, neural networks are obvious candidates for learning weights.
The new update rule:
w w + (r+Uw(j)-Uw(i))wUw(i)
Note: TD-gammon learned better than Neurogammon
Example of Advice Taking
Advice: Don’t go into box canyons when opponents are in sight.
General Structure of RL learner
A connectionist Q-learning augmented with advice-taking.
Connectionist Q-learning• Q(a,i) : utility function maps state and actions to
numeric values.
• Given a perfect version of this function the optimal plan is to simply choose, in each state that is reached, the action with the maximum utility.
• The utility function is implemented as a neural network, whose inputs describe the current state and whose outputs are the utility of each action.
Step 1 in Taking Advice• Request the advice: Instead of having the
learner request the advice, the external observer provides advice whenever the observer feels it is appropriate. There are two reasons for this:• It places less burden on the observer• It is an open question how to create the best
mechanism for having a RL agent recognize its needs for advice.
Step 2 in Taking Advice• Convert the advice to an internal representation: Due to the
complexities of natural language processing, the external observer express its advice using a simple programming language and a list of task-specific terms.
• Example:
Step 3 in Advice Taking• Convert the advice into usable form: Using
techniques from knowledge compilation, a learner can convert high level advice into a collection of directly interpretable statements.
Step 4 in Advice Taking• Use ideas from knowledge-based neural networks:
Install the operationalized advice into the connectionist representation of the utility function.• Converts a ruleset into a network by mapping the
“target concepts” of the ruleset to output units and creating hidden units that represent the intermediate conclusion.
• Rules are intalled incrementally installed into networks.
Example
Example Cont.
Advice added, note that the inputs and outputs to the network remain unchanged; the advice only changes how the function from states to the utility of actions is calculated.
Example Cont.A multistep plan:
Example Cont.A multistep plan embedded in a REPEAT:
Example Cont
A dvice that involves previously defined terms:
Judge the Value of Advice• Once the advice is inserted, the RL agent returns to
exploring its environment, thereby integrating and refining the advice.
• In some circumstances, such as game learner that can play against itself, it would be straightforward to empirically evaluate the advice.
• It would also be possible to allow the observer to retract or counteract bad advice.
Test Bed
Test environment: (a) sample configuration (b) sample division Of the environment into sectors (c) distance measured by the agent sensors(d) A neural network that computes utility of actions.
Methodology• The agents are trained for a fixed number of episodes for
each experiment. • An episode consists of placing the agent into a randomly
generated, initial environment, and then allowing it to explore until it is captured or a treshold of 500 step is reached.
• Environment contains 7x7 grid 15 obstacles, 3 enemy agents, and 10 rewards.• 3 random generated environment.• 10 randomly initialized network.
• Average total reinforcement is measured by freezing the network and measuring the average reinforcement on a testset.
Advices
Testset Results
Testset Result
Above table shows how well each piece of advicemeets its intent.
Related Work Gordon and Subramanian (1994)
developed a system similiar to that one. The agent accept high-level advice of the form IF condition THEN ACHIEVE goal. It operationalizes these rules using its background knowledge about goal achievement. The resulting rules are then refined using genetic algorithms.