The learned dog Class 9: Discrimination, generalization, concept formation and interactions between PC and OC.

The learned dog

Class 9: Discrimination, generalization, concept formation and interactions between PC and OC.

Agenda

• The quizzes

• Quick continuation on Lindsay & reinforcement

• Generalization and discrimination

• Concept formation

• Interactions between PC and OC

Quizzes

• Graded quizzes will be back this week.

• I have posted the answers on the website under handouts.

• Quick review: BB & CB

Quiz review - 8For each of the following situations, tell us whether it represents an example of Pavlovian conditioning? If you don’t think it is an example of Pavlovian Conditioning, explain why not. If you think it is, identify the conditioned stimulus, the unconditioned stimulus, and the conditioned/unconditioned response. (45%)

a. Your cat appears at your feet whenever you use the can opener.

PC CS: can opener US: food/odor of food C/UR: appearing

Connection made when being fed canned food or smelling food when any can of food is opened. Appearing is an orienting response. It’s not an operant response because the presence of the food smell is not contingent on the behavior.

Quiz review - 8

b. The sound of crinkling cellophane will bring most pet dogs running to the sound.

PC CS: crinkling cellophane US: food/odor of foodC/UR: running

Connection made with treats in noisy cellophane human food such as chips in crinkly bags. Once again running to the sound is an orienting response and not an operant response because the resulting smell of food is not contingent on the animal’s action.

c. Your dog lies down when you put your hand in your pocket.

OC Dog is doing a specific probably trained behavior; hand reaching to pocket for treat. Treat is contingent upon behavior of down, and the dog is using the motion of your hand toward the pocket as a cue to lie down. Hmmm, there is some PC here as well as usual as you will see when looking at how cues are learned

Quiz review - 8

f. A dog gets sick in the car on its first ride and does not want to get in the next time

PC CS: Car US: Feeling Sick UR: Throwing up The CR is avoiding the car

d. When I pick up my coat or put on my shoes my dogs all run to the door.

PC CS: Coat US: Going out C/UR: run to the doorCoat on = go out. Running to the door is an orienting response to the CS.

e. My dog sits as I reach for the doorknob.

OC Dog has to do a behavior to get to go out, the reach for door knob is the cue, yes this part is PC

Quiz review - 8 g. Your thunder phobic dog begins to react when the sky gets dark with

clouds and the wind picks up.

PC CS: Dark & wind US: Thunder

C/UR: Fearful response

h. Your puppy was frightened by a sound and saw a person with a hat

right after, this puppy is now afraid of people wearing hats.

PC CS: Hats US: Sound C/UR: Fear

This is an example of Weird Fear. The sound both sensitizes the puppy puts her in a fearful state. In this sensitized, fearful state, the hat, which in general is a neutral stimulus becomes a salient feature of a context that the puppy subsequently associates with being scared. This is also an example of backward conditioning (the US appears before the CS.)

Quiz review - 8

i. Your dog knows the word “sit” means to sit; you want to teach the dog to sit on the word “pickles”. You begin saying the word ‘pickles” right before you say “sit”, soon the dog is sitting on the word pickles

PC / Second Order Conditioning CS1: sit CS2: picklesUS: R+ food C/UR: sit.

This is a nice example of PC within the context of OC. The association of “pickles” with “sit’ is second order conditioning, and that is PC. But it happens because “sit” predicts a context in which if the dog sits they will get a treat, and that is OC.

j. To get your dog to step over a jump you lure with a piece of food.

OC Luring is one method of using operant conditioning. Stepping over the jump = get reward. Getting the food is contingent on performing the behavior.

Quiz review 8 & 9k. A fearful dog gets over his fear by pairing the scary object with a

yummy treat.PC CS: Scary object US: Yummy treat C/UR: eating yummy treatHere we are using PC to desensitize or counter condition to an already bad association to turn it around and attempt to make a good association. The scary object becomes a cue that something good is going to appear. As long as the food appears regardless of what the dog does then it is PC.

9. Who made popular the “pack theory” way of training dogs in the 70’s?Bob Bailey Cesar Milan The Monks of New Skete

Karen Pryor

BB says: Hmmm… , I thought Karen Pryor got the idea of using a clicker from the sound that the alpha wolf makes when it is about to regurgitate food.

Lindsay on Reinforcement...

Control and prediction

• ‘Successful control depends on adequate prediction, and adequate prediction depends on successful control.’

• In other words it’s all about control & prediction...

• Control: if I want to attain a given outcome, do I have one or more reliable strategies for achieving that outcome. That is, if I perform the strategy, my expectation is that I will achieve the desired outcome.

• Prediction: can I predict the imminent/future occurrence of biologically significant events so as to be in a good position, now, to take advantage of them, or to avoid their occurrence.

• Note, prediction is only useful if there are reliable strategies to control outcomes based on those predictions.

Learning as the process of forming and refining expectations

• Learning is the process of forming and refining expectations in light of what actually happens...

• Was the actual outcome, better than expected, worse than expected, or exactly what was expected?

• Learning occurs most rapidly when the mismatch between expectation and reality is greatest, all things being equal.

• Most of the same rules, e.g. incremental predictive value, apply whether the prediction is based on sensory input, self-action, or a combination.

Punishment & reinforcement occurs when outcomes don’t match expectations

• Attractive Outcome

• Better than expected: surprise (R)

• Worse than expected: disappointment (P)

• Aversive Outcome

• Better than expected: relief (R)

• Worse than expected: startle (P)

An example...

• What is the dog controlling, what are the expectations and when are they confirmed or violated? When is the most learning occurring?

• Dog has learned that jumping up on the door seems to make it open when they want to go outside to chase that nasty squirrel

• Your partner decides enough is enough, so you wait until the dog has 4 on the floor before you open the door

• You then go on to only open the door when the dog has been sitting quietly and stays seated while the door is open.

Lindsay’s idea of diverters and disrupters...

• A ‘diverter’ is a positive outcome that is different in kind than what the dog expects...

• Dog has an expectation if they sit in response to ‘sit’ they will get a cookie, but you present them with a ball. It causes surprise, but Lindsay doesn’t consider it a reinforcer at that moment because the dog wasn’t working to control the ball’s appearance. But over time it could become one.

• Throwing food on the ground as a diverter?

• A ‘disrupter’ is something that momentarily disrupts behavior.

• Once again, the first appearance is not considered punishment because dog wasn’t acting so as to avoid or escape its appearance.

Lindsay’s idea of diverters and disrupters...

• ‘Diverters and disrupters are means for initiating new behavior without first punishing or extinguishing already established behavior. Both diverters and disrupters are marking events that set the stage for establishing a new set of control-prediction expectancies with which to organize new behavior’

Why have I subjected you to this rant?

• It provides a clean conceptual framework in which the words don’t get in the way.

• Prediction and control as the goal of learning, and the process of learning as that of forming and updating expectations about...

• the world (PC)

• the effect of one’s actions on the world so as to get the good and avoid the bad. (OC)

• By focusing on the difference between expectations and outcomes as the driver for learning, it provides a useful framework for thinking about training

• The role of surprise to push behavior in one direction or another

• But also, the importance of consistency and predictability

Interactions between Pavlovian and Operant Conditioning...

Interactions between PC & OC

• Conceptually it is easy to distinguish between PC and OC

• PC: relation between 2 stimuli, whether CR occurs or not does not affect occurrence of US

• OC: relation between a response and an outcome. No intended relationship between a stimulus and the outcome.

• In practice, and especially in animal training, they tend to get very blurred indeed.

• So the question, as SWR put it, isn’t whether they interact but how?

PC and OC as different aspects of one underlying process focused on learning causality...

• 3 classes of causality...

• Stimulus predicts a future event that does not depend on action on the part of the animal: S->0, or P(O|S)

• Performance of an action predicts a future event that is independent of stimuli in the world: R->O or P(O|R)

• A stimulus is predictive of a future event, if the animal subsequently performs a particular action. S->(R->O) or P(P(O|R)|S)

blocking and overshadowing can occur between OC &

PC...

Light more reliable predictor of food than lever press

Schwartz, B., E. A. Wasserman, et al. (2002). Psychology of Learning and Behavior. New York, NY, W. W. Norton & Company, Inc.

blocking and overshadowing can occur between OC & PC...

Here running is a better predictor of food


Evidence of an hierarchical association

Lever press means one thing in one context and another in another context

These interactions are present in everyday training...

• If you reward a behavior whether it is done in the context of a cue or not, it will take the dog a lot longer to learn the cue...

• There is no incremental predictive value to be had by attending to the cue.

• The incremental predictive value occurs when you stop rewarding spontaneous presentations of the behavior

• If you reward the dog regardless of whether it performs a desired behavior within a given context, it will take the dog a lot longer to learn the behavior

• There is no incremental predictive value associated with performing the behavior.

Stimulus control of behavior

Stimulus control...

• Not only does an animal need to know what to do in order to achieve some outcome, but it needs to know when to do it.

• Learn that certain features in the environment signal a context in which if they perform a given action, a desired outcome will follow, at least more reliably than in the absence of those features...

• Example:

• Hear ‘sit’ -> Perform ‘sit’ -> Receive a cookie

• When the behavior is more likely in the presence of a given stimulus than in its absence, it is said to be under ‘stimulus control’

ABCs

Antecedent Behavior Consequence

hear ‘sit’ perform ‘sit’ get cookie

Stimulus Response Outcome

Discrimination & Generalization

How does an animal learn what features of the environment to attend to?

• First and foremost, may be biased to attend to certain kinds of features: auditory, visual, movement, olfactory...

• Innate bias (modality & temporal)

• Learned bias

• Discriminating between features based on differential outcomes:

• Sitting in response to ‘sit’ reliably produces a cookie. Lying down in response to ‘sit’ reliably doesn’t produce a cookie... Hmmm...

Example of discrimination

• Tends to go from general to specific...

• Form expectation that ‘light on’ signals reliable context to perform behavior.

• Observe expectation true in some cases not in others

• Form & test new expectation: ‘red-light-on’ is a more reliable predictor than ‘light on’


Discrimination is revealed through

choice

Since it is inferred need to be very careful to make sure animal is responding to the cue that you think it is, or should be, attending to!!!!


generalization flip-side of discrimination

• Features are often continuous

• frequency, loudness, brightness, speed, tone, strength...

• Animal needs to learn what differences matter and what differences don’t

• Generalization is the process of identifying what features don’t matter, and/or what differences don’t matter


Generalization & discrimination...

• Flat response curve suggests generalization, peaked curve suggests discrimination

• Why might you see a flat response curve?

• Animal not attending to feature

• Animal has learned it doesn’t matter


Discrimination/generalization & incremental predictive value...

• Does the feature, or variation in the feature have incremental predictive value with respect to other features and/or the reliability of the behavior in producing a given response?

• Among equally predictive features which gets chosen (over-shadowing)

• Inherent salience

• Learned salience

• Verbal cues are a perfect example of this, each cue should get progressively easier to train

Discrimination training

• Animals can learn to discriminate based on training

• Differential outcomes in the presence or absence of particular features.

• Here, color initially overshadows vertical line


Discrimination training cont...

• There is an important idea here: animals act as if they only discriminate if there is evidence that it is worth their while to do so (slackers at heart...)

• Top: continuous tone

• Middle: peck rewarded in presence of tone, not in its absence

• Bottom: peck rewarded in presence of one tone, & explicitly not in presence of another tone.


Transfer of discrimination training

• Degree to which training transfers depends in part on the nature of the variation

• Extradimensional: different class of features are important

• Intradimensional: different values within existing class are important

• The latter is easier as one might imagine.


Color Importan

t

Shape Importan

t

Color Importan

t

Peak shift: peak is shifted away from counter-

example

Spence’s theory explains peak shift as function of interaction between + curve and - curve

Curve is sum of top 2 curves

Relative or absolute...

• Initially trained with S+ as 40W and S- as 10W (i.e., one key brighter than the other)

• Tested with choice of 2 buttons, one 50W, one 70W. Which do they choose...

• Spence predicts 50: closest to original S+

• Transposition predicts 70: brighter one of the pair


Another example of transposition

In this choice experiment animal seemed to use relational rule

Relational rule: Light over dark, turn right

Absolute rule: 5 &7 turn right


Animals may in fact use both rules...

• Note the protocols are different...

• In transposition tests, animal is given a choice

• In generalization/shift tests, strength of response to a single, but varying stimulus is measured.

• When animals have a choice may use relative rule, when given a single but varying example, may use generalization rule.

Concept formation (very cool chapter by the way, well worth reading)

What is a concept, and how do you know if an animal has a concept?

• Keller & Schoenfeld turned the question around so they could speak of behavior that was conceptual. That is, ‘when a group of objects gets the same response, when they form a class the members of which are reacted to similarly, we speak of a concept’.

• In other words, the existence of a concept is observed via a common response by the animal to a set of entities all of which must share, in the animal’s mind, some group of common properties (features).

• dogs, cats, squirrels, prey, toys, food...

• Generalization occurs within a class, discrimination occurs between classes.


Artificial vs. natural concepts...

• While we often talk about identifiable features, it is a lot harder to precisely enumerate the features associated with natural objects. For example, what is the necessary and sufficient set of ‘features’ needed in order to identify a dog, or a tree?

• Even if we come up with a set, not clear that it will be the same set as a given animal may use.

• None the less, there is lots of evidence that animals can learn natural concepts. In fact, it is really pretty cool...

Presence or absence

• Herrnstein’s experiment :

• Pigeon’s trained to peck key when they see an example of a category (tree, water, fish, particular person) and not to peck key when it is not an example of the category

• Trained and tested on 1 category at a time. Testing included novel examples

• Turned out to be ‘fledgling’s play for pigeons”: quick and accurate Schwartz, B., E. A. Wasserman, et al. (2002). Psychology

of Learning and Behavior. New York, NY, W. W. Norton & Company, Inc.

Multiple natural concepts...

• Training set up

• Pigeon shown image of an object belonging to one of 4 categories

• They have a choice of pressing 1 of 4 buttons, each of which is assigned to a given category

• Press right button & they get a treat...

• Button is like saying the name of the category


Multiple natural concepts, cont...

• Training set up

• 40 daily trials in which they saw 10 different exemplars of 4 categories (cats, flowers, cars & chairs)

• Content of images varied a great deal, incl. cases in which objects were partially obscured


Multiple natural concepts, results...

• 80 percent correct after 30 days of training (chance would be 25%)

• When tested with novel examples of categories they were still 67% accurate!

• Interestingly, when researchers varied the number of different examples vs. repeating exposure to the same examples...

• Took the pigeons longer to learn the category, but...

• were more accurate in classifying novel instances of the categories (better able to generalize)

• Also did better with natural categories than made up/pseudo categories

Conceptualization: primary vs. secondary generalization

• Examples presented so far assume that the category is based on perceptual similarity...

• the actual features that are being used to discriminate among categories may be unknown, but the assumption is that it is being done on the basis of perceptual features.

• known as primary generalization

• Secondary generalization occurs when the discrimination is based on logical features, or something other than perceptual features...

Discrimination reversal paradigm

• Set of images are arbitrarily broken into 2 categories, A and B.

• On alternate days, images in either A or B are rewarded, i.e. if on the last session images in category A were rewarded, then in this session images in category B are rewarded.

• The only common link between the images in a category is whether they are rewarded on a given day...

• Pigeons quickly learn to figure out on the basis of the outcome of a few examples whether it is a category A day or a category B day, and respond to the remaining examples accordingly.

Joint/super-ordinate category learning

87% accuracy to reassigned stimuli and 72% to non-reassigned stimuli (C2 and C4)

Form 2 new aggregate

categories from previously trained

categoriesTrain new

response to previously

trained category

Test response to other

members of aggregate category


Same & Different

• Matching to sample and oddity testing

• Animal is trained to observe a ‘sample’, e.g. a colored key that the pigeon pecks

• Animal is then presented with 2 additional choices and must choose the one that is the same (match to sample) or different (oddity)


Same vs. Different

• Pigeons did much better when the level of effort directed toward the sample was higher than lower...

• When 1 peck was required: 76% accurate on trained stimuli, and only 55% on novel stimuli.

• When 20 pecks were required accuracy for both trained and novel stimuli went up to 80%.

• Maybe increased pecks resulted in increased attention, or maybe pigeons use a different paradigm to solve the problem under the two scenarios.

• A couple of important lessons here!!!

Oddity testing: base response on whether all

images the same

Pigeons could reliably identify same or different with both trained and novel images


Accuracy of same/different goes up with number of

icons

83% accuracy on trained displays. 71% on displays made up of 16 novel icons


Qualitative assessment of same vs. different or quantitative measure of variability in image

• Wasserman argues that pigeons are making an assessment of variability in the image...

• In fact, the pigeons are accurate on ‘same’ images whether there are 2 icons or 16. There is 0 variability in either case.

• However, the variability goes up when the icons are different (16 different images vs. 2 different images.) In fact, pigeons have a very difficult time with the same/different test when only 2 icons are involved.

The big conclusion...

• There is clear evidence that animals behave conceptually, that is, they produce a common response to entities that seem to share, in their mind, some set of common features.

• They are capable of generalizing within a class, and discriminating across classes.

• In general, the more different training examples they are given, the better able they are to identify novel instances as being part of the same category.

• Note: a given animal’s concepts, and the features they use to form those concepts may be, and undoubtedly are, very different than ones we may form and use.

The learned dog Class 9: Discrimination, generalization, concept formation and interactions between PC and OC.

Documents