Sniffy, The Virtual Rat

Project 2: Chapter 10: Exercises 22, 23, 24, 25, 27

Basic Operant Phenomena: Magazine Training, Shaping, Extinction, and Spontaneous Recovery

This exercise is based on the materials in the Sniffy Pro manual. To get a feel for the software download the Sniffy Demo software available on this site, and print off this document to follow step by step. Some users may find it helpful to run the Tutorial, and see Project 5 before they try it by themselves.

Using the Sniffy Software to Understand Operant Conditioning

When the program starts up you can expect to see Sniffy rearing up, grooming himself, and exploring the chamber. You can observe and manually record any of the behaviors that Sniffy performs. However, the response that psychologists generally study in an operant chamber is bar pressing. In research laboratories, psychologists use computers to control the presentation of food and other stimuli and to record bar-presses, and the Sniffy Pro program simulates these functions.

The Operant Chamber window is the window where you see Sniffy moving about

On the back wall you see a lever the so-called bar, that you will train Sniffy to press, and food hopper, a water spout, a bell and a speaker. The Lab Assistant window provides you with useful suggestions about what to do next or about the status of your current Sniffy experiment. You will need to open two additional windows for this exercise: Display the Operant Associations mind window by selecting it from the Mind Windows section of the Windows menu. If the Cumulative Record window is not visible, make it visible by selecting it from Cumulative Records section of the Windows menu.

The Operant Associations mind window displays the strength of three associations that Sniffy will learn when you train him to press the bar. This window is a called a "mind window" because it shows you the important psychological processes going on inside the Sniffy Pro program. Like all the Sniffy Pro program's mind windows, it has a blue background. The soundÐfood association is the association between the sound of the food pelletÐdispensing mechanism and the fact that a pellet of food is available in the hopper. You will teach Sniffy this association during the next exercise on magazine training. The barÐsound association is the association between the bar and the sound produced by the food-dispensing mechanism. When Sniffy is trained to press the bar, Sniffy learns that the bar is the device whose manipulation causes the sound that signals the presence of a food pellet in the hopper. Action strength is Sniffyâs association between a particular behavior pattern and obtaining food. This is particularly relevant for behaviors other than bar-pressing (see Project 5).

The Cumulative Record window will enable you observe changes in Sniffy's bar-pressing rate. Like all the windows in the Sniffy Pro program that provide measures of Sniffy's behavior, the Cumulative Record window has a white background.

Operant Conditioning: Technique

Like a real rat, Sniffy will press the bar 2 or 3 times every 10 minutes even before you train him to do so. To train Sniffy to bar-press more frequently, you will administer a food pellet (a positive reinforcement) for each bar press immediately after the start of the response. Thus, you will be using positive reinforcement to increase the frequency with which bar pressing occurs. You can accelerate this learning process by using a technique called shaping. Shaping is the technical name for a procedure used to train an animal to do something by reinforcing successive approximations of the desired target behavior. In this procedure, the trainer (teacher) leads the subject (learner) to progress by small steps.

To do this you must first magazine train Sniffy to associate a click sound with a reward.

The magazine in an operant chamber produces a distinctive mechanical sound when it drops a food pellet into the hopper; and it is easy to transform this sound, which has no intrinsic power to act as a reinforcer, into a secondary reinforcer by pairing the sound with food delivery in a way that causes Sniffy to associate the sound with food presentation. The procedure that turns the magazine sound into a secondary reinforcer is called magazine training. It is this sound we use as a reward during shaping.

Exercise 22: Magazine Training

Magazine training is a technique that involves using what amounts to a classical conditioning procedure to turn an originally neutral stimulus into a secondary reinforcer. The idea is to operate the magazine to present pellets of food in such a way that Sniffy learns to associate the sound of the magazine with the availability of a food pellet in the hopper. One of the associations that the Operant Associations mind window displays is the soundÐfood association. By keeping an eye on the Operant Associations mind window, you can watch Sniffy develop an association between the magazine sound and food.

Here are the steps that you need to follow to magazine-train Sniffy.

Start with a new Sniffy file.
Use the Save As command in the File menu to save the file under an appropriate name (e.g., Ex22-MagTrain) in your Sniffy Files or Sniffy Pro for Windows folder.
Display the Operant Associations mind window by selecting it from the Mind Windows section of the Windows menu.
The message in the Lab Assistant window reads, ãTo set up an experiment, choose Design Classical Conditioning Experiment or Design Operant Conditioning Experiment from the Experiments menu. Sniffy is not being reinforced automatically. If you want to issue a reinforcement, press the space bar or click on the bar.ä The last sentence of the Lab Assistant message is the relevant instruction for this experiment.
Wait until Sniffy closely approaches the food hopper. Then deliver a food pellet, either by pressing the space bar on your computer keyboard or by pointing the cursor at the bar and clicking your (left) mouse button.
To save time at the start, you may want to give Sniffy several pellets in rapid succession before he wanders away from the hopper.
After Sniffy has received several pellets, you can let him wander away a short distance before giving him the next pellet.
Keep an eye on the Operant Associations mind window. When the height of the SoundÐFood bar reaches about one-quarter of the way up the scale, the message in the Lab Assistant window will change to read, ãSniffy is developing an association between the sound of the hopper and the presentation of food. However, this association is not strong enough to properly train Sniffy. Continue to present food when Sniffy is near the hopper.ä Follow the Lab Assistantâs advice. Continue to reinforce Sniffy when he is near the hopper.
Because you will want to be able to use this file as the basis for training Sniffy to perform several different behaviors, during magazine training do not consistently give Sniffy a food pellet after he performs any particular behavior or category of behavior.
When the height of the SoundÐFood bar in the Operant Associations mind window reaches about three-quarters of the way up the scale, the Lab Assistant will display the message, ãSniffy appears to have developed an association between the sound of the hopper and the food. You can now use the sound as a reinforcer to shape Sniffyâs behavior.ä
Sniffyâs magazine training is now complete.
Save the file. (Please note that he Demo version does not allow saving)

Exercise 23: Shaping Sniffy to Press the Bar

Shaping is the technical name for a procedure used to train an animal to do something by reinforcing successive approximations of the desired target behavior. The idea is to pick a behavior that the animal performs fairly often and that is similar in some way to the target behavior you want the animal to perform eventually. Reinforcing this first approximation of the behavior will cause Sniffy to perform that behavior more frequently. Eventually, Sniffy will perform a variant of the behavior that more closely resembles the target behavior. That variant then becomes your second approximation, and you require him to repeat that variant to obtain reinforcement. As the second approximation is performed more frequently, Sniffy will eventually emit another variant that resembles the target even more closely, and so on.

Shaping an animal takes patience, careful observation, and good timing. It is a skill that you learn with practice. Sniffy is easier to shape than a real rat, partly because he never becomes satiated for food and partly because his behavioral repertoire is smaller than a real ratâs. Nevertheless, shaping Sniffy is challenging enough for you to get some idea of both the frustration and the eventual feeling of triumph that shaping an animal engenders.

Here is a detailed description of the steps that you should follow to shape Sniffy to press the bar.

Continue with the animal you have magazine trainied or Open the file from Exercise 22 that we suggested you call Ex22-MagTrain.
Select the Save As command from the File menu to give the file an appropriate new name (e.g., Ex23-ShapeBP) and save it in the Sniffy Files folder on your computerâs hard drive. Saving the file with a new name before you start shaping Sniffy preserves your original magazine-training file for future use. You will need it for a number of subsequent exercises.
If the Operant Associations mind window is not already visible, display it by selecting it from the Mind Windows section of the Windows menu.
If Cumulative Record 1 is not visible, make it visible by selecting it from the Cumulative Record section of the Windows menu.
Select the Design Operant Conditioning Experiment command from the Experiment menu.
In the dialogue box that appears, select Bar Press from the Target Behavior section of the drop-down menu under Reinforcement Action. After making the selection, click on the OK button to close the dialogue box.
Note that the terms CRF and Bar Pressing appear in the cumulative record. CRF stands for Continuous Reinforcement, which means that the program will automatically reinforce every bar press. The Bar Press notation indicates that the cumulative record is recording bar presses.
As your first approximation to bar pressing, give Sniffy a pellet of food when he rears up facing the back wall anywhere in the operant chamber.
Gradually require Sniffy to rear up closer and closer to the wall.
Whenever Sniffy rears up directly in front of the bar, there is a chance that he may press it. If he does press the bar, he will hear the magazine sound, receive a food pellet, and the barÐsound association will start to develop. After several reinforced bar presses, a red column will appear above the words ãBarÐSoundä and ãAction Strengthä in the Operant Associations mind window.
When the column above BarÐSound in the Operant Associations mind window reaches about one-quarter of the maximum height, the message in the Lab Assistant window will change to read, ãSniffy is developing an association between the bar and the sound. Continue and Sniffy should be trained soon.ä
Each time Sniffy presses the bar, watch closely what he does after eating the food pellet. He may press the bar again a second time either immediately or after rearing up near the bar a time or two. If he does press the bar again, you know youâre making progress. Allow him to continue pressing the bar as long as he will do so. However, if he rears up more than twice without pressing the bar again, continue to reinforce rearing up.
If you are patient, the time will come when Sniffy will press the bar eight to ten times in rapid succession. At that point, you can stop shaping, sit back, and watch the progressive effect of reinforcement as Sniffy continues to press the bar more and more frequently.
Watch the rising levels of the BarÐSound and Action Strength columns in the Operant Associations mind window. Sniffyâs training is complete when the message displayed by the Lab Assistant changes to read,ãSniffy appears to be trained properly. You may experiment with different schedule effects.ä
When Sniffy is fully trained, select the Save command from the File menu to preserve your trained Sniffy for future use.

After 30 to 45 minutes of attempting to shape Sniffy according to the instructions given above, Sniffy should be pressing the bar at least 20 times during each 5-minute interval delineated by the alternating solid and dotted vertical lines in the cumulative record. If your attempt at shaping fails to obtain that minimum result, something is wrong.

One possibility is that you may not be reinforcing instances of rearing up toward the back wall as outlined above or not reinforcing these behaviors quickly enough. For users who encounter difficulty with shaping, we have provided a section of the Reinforcement Action menu in the Design Operant Conditioning Experiment dialogue box called the Shaping Tutor. If you are having trouble reinforcing Sniffyâs rearing behaviors:

Choose the Design Operant Conditioning Experiment command from the Experiment menu.
Under the Shaping Tutor subsection of the Reinforcement Action menu, choose Rear Up Facing Back.
The Sniffy Pro program will now automatically reinforce all instances of rearing up facing the back wall, including all bar presses. Watch which actions the program automatically reinforces.
When you believe that you are capable of effectively reinforcing these behaviors manually, reopen the Design Operant Conditioning Experiment dialogue box and choose Bar Press from the Target Behavior section of the Reinforcement Action menu.
If you are unable to reinforce Sniffy effectively by hand, watch what is going on as the program automatically reinforces rearing up facing the back wall. Once Sniffy presses the bar 8 to 10 times in a 1- or 2-minute period, reopen the Design Operant Conditioning Experiment dialogue box and choose Bar Press from the Target Behavior section of the Reinforcement Action menu. The program will complete Sniffyâs training automatically.
When Sniffyâs training is complete, the Lab Assistant will display the statement ãSniffy appears to be trained properly. You may experiment with different schedule effects.ä Save the file.

Exercise 24: Cumulative Records: Visualizing Sniffyâs Responding

The cumulative record in normally shown under the Operant Chamber window. If it has been closed or is not visible, display it by selecting it from Cumulative Responses submenu of the Windows menu. Note that as time passes, the visible part of the cumulative record automatically scrolls to the right to follow Sniffyâs current behavior. If you want to look at something that happened earlier, you can use the scroll bar at the bottom of the window to scroll back to the left.

Changes in the bar-pressing rate are measured in the Cumulative Record. The horizontal dimension represents time, and the vertical dimension represents bar presses.

As time elapses a line is drawn horizontally across the page. Each time Sniffy presses the bar the lines moves up a notch and continues along horizontally. The slope of the rising lines on the graph represents the speed with which Sniffy is responding. The steeper the slope, the more rapidly Sniffy was pressing. Reinforced responses are marked by the short, oblique lines drawn through the record.

If you let it run long enough, the Sniffy Pro program will produce a series of 10 Cumulative Record windows which are accessible under the Cumulative Records section of the Windows menu. Each Cumulative Record window depicts Sniffyâs bar-pressing performance during two hours of Sniffy Pro program time. Program time and clock time are not the same thing. The relationship between program time and clock time depends on both the speed of your computer and the animation speed setting in the Preferences menu. The fact that there is a maximum of 10 Cumulative Record windows means that no Sniffy experiment can last more than about 20 hours in program time. After that program time limit has been reached, you can examine and save your results, but you cannot add any additional stages to that particular Sniffy Pro file.

The cumulative record depicted here shows the acquisition of bar pressing as a consequence of shaping in a ãtypicalä experiment. No two cumulative records are ever exactly alike, because Sniffy never behaves in exactly the same way in any two experiments. However, if you were a successful shaper, the part of your cumulative record from Exercise 23 that recorded Sniffyâs acquisition of the bar-pressing response should resemble the one shown.

Here are the important features and some characteristic things about the record shown above that you can expect to see in your own cumulative record:

The flat, horizontal portions of the line show periods when Sniffy was not pressing the bar.
Sniffy begins to press the bar slowly and intermittently at first, then more and more frequently and steadily
The ãheightä of Sniffyâs cumulative record is always 75 responses. There are always 75 responses between any two consecutive pen resets. When the pen resets the first time, Sniffy has made 75 responses. When it resets the second time, he has made 150 responses, and so on. Knowing this will come in handy later on when you need to figure out how many responses Sniffy has made.
In addition to the dark vertical lines that the cumulative record produces when the pen resets from the top of the record to the bottom, there are thinner, alternating solid and dotted vertical lines spaced at five-minute intervals. You will need to know this when we ask you to measure the time it takes to complete each stage of training.

Exercise 25: Extinction

After training Sniffy to press the bar, you might wonder what would happen if you stopped reinforcing bar presses. Extinction is the technical name for the behavior changes that occur when a previously reinforced behavior no longer produces reinforcement.

Here is what you need to do to set up and run an extinction experiment.

Continue with the file containing your trained Sniffy from Exercise 23.

Use the Save As command to save the file under an appropriate new name (e.g., Ex25-Ext, for ãextinctionä) in the Sniffy Files folder on your computerâs hard drive. This step is important because it preserves your original trained Sniffy file for future use. You will need your trained Sniffy file for the next and other exercises.
Choose the Design Operant Conditioning Experiment command from the Experiment menu.
When the dialogue box opens, the button labeled Continuous and the reinforcement action Bar Press are selected because you have been reinforcing all of Sniffyâs bar-press responses. Reinforcing every response is a procedure called continuous reinforcement.
To select extinction, point the cursor at the button labeled Extinction and click your (left) mouse button.
Be sure there is a check mark in the box next to Mute Pellet Dispenser. (If the check mark isnât there, place the cursor over the box and click your (left) mouse button to put a check mark in the box.) Setting up extinction with Mute Pellet Dispenser selected means that Sniffyâs bar presses will no longer produce food pellets and that he will no longer hear the magazine sound as a consequence of bar pressing. In other words, both the primary reinforcer (food) and the secondary reinforcer (the magazine sound) are turned off. This is the standard extinction procedure. After checking to be sure that you have made the correct settings, click the OK command button.
After you click OK, the dialogue box will disappear, and the Sniffy Pro program will begin running again, but Sniffyâs bar presses will no longer be reinforced.
If you want to speed up the experiment, select the Isolate Sniffy (Accelerate Time) command from the Experiment menu.

Immediately after you click OK, your cumulative record will look something like the one shown next. Note that the Sniffy Pro program marks the cumulative record to show the point at which extinction starts and informs you that the magazine sound is muted.

As a consequence of stopping reinforcement, Sniffyâs rate of bar pressing will eventually decline until he presses the bar no more often than he did before he was trained. However, the first effect of extinction is to increase Sniffyâs bar-pressing rate. This increase in response rate is called an extinction burst, and it commonly occurs when an animal is switched from continuous reinforcement to extinction.

A second thing to note about the standard extinction procedure is that the Operant Associations mind window shows that extinction results in the elimination of the barÐsound association and action strength. However, the soundÐfood association remains intact. The barÐsound association and action strength dissipate because bar presses no longer produce the sound. The soundÐfood association remains intact because Sniffy never hears the sound without receiving a food pellet.

The extinction criterion, on an eye-ball estimate basis, is no more than 2 presses within a 5-minute period. The lab Assistant window reports the rate of bar-presses per minute, calculated over the last 10 presses. When the rate drops below .5 presses per minute (i.e., 2.5 presses in 5 minutes) we shall consider bar pressing extinguished. Sniffyâs rate of pressing has returned to within the normal pre-training level (.25 to .50 per minute).

When that point is reached:

Mark the cumulative record by executing the Mark Record command in Experiment menu.
Save your extinction file by selecting the Save command from the File menu.

When the extinction criterion is reached, you should estimate the number of responses that Sniffy made between the onset of extinction and the time when the criterion was reached. You should also estimate the time required to reach the extinction criterion.

When estimating the number of responses and the time elapsed during extinction, remember that:

The thin alternating dotted and solid vertical lines on the cumulative record mark off 5-min periods in program time.
Sniffy always makes 75 responses between two successive pen resets.

Determining how many responses Sniffy has made during extinction and the time that extinction required will typically require you to estimate fractional parts of 75-response vertical pen excursions and fractional parts of 5-min time intervals. You can elect either to make precise measurements or to do ãeyeballä estimates.

If you want to make precise measurements, you should print your cumulative record and make the appropriate measurements with a ruler. To print your cumulative record:

Select the Cumulative Record window by pointing the cursor at it and clicking your (left) mouse button once.
Select the Print Window command from the File menu.

To do an ãeyeballä estimate, look at the cumulative record and estimate the appropriate horizontal (time) and vertical (response) movements as a proportion of 5-min intervals and 75-response vertical pen excursions. For example, have another look at the cumulative record shown above that displays the point at which the change from continuous reinforcement to extinction occurred. The change occurred partway through a 5-min interval between two vertical lines and partway up from the bottom of the cumulative record.

With regard to time, the change occurred not very long before a vertical line marking the end of a 5-min interval. Letâs say that it occurred about 4/5 or 80% of the way through the 5-min interval. Four-fifths of 5 is 4, so our estimate is that the change occurred about 4 min into the 5-min interval, so that about 1 min remained before the cumulative record reached the time marker.
With regard to the number of responses that Sniffy made after the previous time marker and before the switch to extinction, it looks as though the cumulative record had moved no more than 2 or 3 responses up from the bottom. We know that there are 75 responses between pen resets, which means that Sniffy had about 72 or 73 responses to go before the pen reset again.

Exercise 27: Spontaneous Recovery

A single extinction session is not enough to permanently reduce the frequency of an operant response to its pre-training frequency. If an animal that has apparently been fully extinguished is removed from the operant chamber, allowed to rest in its home cage for 24 hours, and then returned to the operant chamber for a second extinction session, its response rate at the start of the second session will be greater than it was at the end of the first extinction session. This rest-produced reappearance of an extinguished operant response is called spontaneous recovery.

To simulate the phenomenon with Sniffy:

Continue with a file in which Sniffy has been trained to press the bar and then extinguished. Or open the file you saved earlier (e.g., Ex25-Ext)
Use the Save As command to save the file under a new appropriate name (e.g., Ex27-SponRec) in the Sniffy Files folder on your computerâs hard drive.
Choose Remove Sniffy for Time-Out from the Experiment menu. To simulate taking a rest, Sniffy will disappear momentarily and then reappear.
If you want to speed up the experiment, select the Isolate Sniffy (Accelerate Time) command from the Experiment menu.

When Sniffy reappears, his bar-pressing rate will be higher than it was at the end of extinction, but lower than it was before extinction. Immediately after the timeout, your cumulative record should resemble that shown below.

Check the Operant Associations mind window shortly after starting your spontaneous recovery experiment. At the beginning of the spontaneous recovery experiment, the extinguished items are partly restored. This partial reappearance of the extinguished items is the ãpsychologicalä reason why Sniffy presses the bar more often at the beginning of the second extinction session than he did at the end of the first extinction session.

Let the Sniffy Pro program run until Sniffy meets the extinction criterion again. Compare the number of responses made and the time required to reach the extinction criterion during this second extinction session with the number of responses and time required during the first extinction session. This comparison will reveal that the second time around Sniffy makes fewer responses and takes less time to reach the criterion.

Thank you for exploring Project 2 with us! See Projects 3, 4 and 5 for more advanced exercises with Sniffy, the Virtual Rat.