Experimental research

• 1. Experimental Research ByShafqat Rasool

2. Experimental research answers the question What if?The researchermanipulates independent variables (e.g., type of treatment, teaching method,communication strategy) andmeasures dependent variables (anxiety level, English comprehension, s atisfaction) in order to establish cause-and-effect relationships between them.Theindependent variable is controlled or set by the researcher. Thedependent variable is measured by the researcher.An experiment is a prescribedset of conditions which permit measurement of the effects of a particular Treatment. 3. Threats to experimental researchInternal validityInternal invalidity asks the question, Are the measurements I make on my dependent (i.e., the variable I measure) variable influenced only by the treatment, or are there other influences which change it?External validity To how much the results are generalized to target population. 4. Internal validity History,Maturation,Testing, Instrumentation, Statistical regression, Differential selection, Experimental mortality, And selection-maturation interaction.The John Henry effect and experimental treatment diffusion. 5. History History refers to events other than the treatment that occur during the course of an experiment which may influence the post-treatment measure of treatment effect. If the explosion of the nuclear reactor in Chernobyl, Ukraine had occurred in the middle of a six-month treatment to help people reduce their anxiety of nuclear power,Must occur during the experiment. Controlled be control group 6. Maturation Subjects change over the course of an experiment. These changes can be physical, mental, emotional, or spiritual. Perspective can change. The natural process of human growth can result in changes in post-test scores quite apart from the treatment.Question: How would a control group control this source of internal invalidity 7. Testing A common research design is to give a group a pre-test, a treatment, and then a post-test . If the same testis used both times, the group may show an improvement simply because of their experience with the test. This is especially true when the treatment period is short and the tests are given within a short time. 8. Instrumentation If you use different tests for pre- and post-measurements, then the change in pre- and post-scores may be due to differences between the tests rather than the treatment.The best remedy is to use randomization and a post-test only design. 9. Statistical regression Statistical regression refers to the tendency of extreme scores, whether low or high, to move toward the average on a second testing. Subjects who score very high or very low on one test will probably score less high or low when they take the test again. That is, they regress toward the mean. Do not study groups formed from extreme scores. Study the full range of scores. 10. Differential selection If we select groups for treatment and control differently, then the results may be due to the differences between groups before treatment.Randomization solves this problem by statistically equating groups. 11. E xperimental mortality Experimental mortality, also called attrition, refers to the loss of subjects from the experiment. If there is a systematic bias in the subjects who drop out, then posttest scores will be are biased. For example, if subjects drop out because they are aware that theyre not improving as they should, then the post-test scores of all those who complete the treatment will be positively biased. Your results will appear more favorable than they really are. How does use of a control group solve the problem of attrition? 12. Selection-Maturation Interaction of Subjects Interaction means the mixing or combining of separate elements. If you draw a group of subjects from one school to serve as the treatment group, and a second group from a different schools to serve as a control, you could well find -- beyond the simple problem of selection differences (Are the two groups equivalent?) --a mixing of selection and maturation factors to compound the extraneous influence on your measurements.For example, if the two schools differ in the average age of their members, they may well respond to the treatment differently due to inherent maturational factors.Randomly selecting all subjects from a defined population solves this problem. 13. The John Henry Effect John Henry, the legendary steel drivin man, set himself to prove he could drive railroad spikes faster and better than the newly invented steam-powered machine driver. He exerted himself so much in trying to outdo the "experimental" condition that he died of a ruptured heart.If subjects in a control group find out they are in competition with those in an experimental treatment, they tend to work harder. When this occurs, differences between control and treatment groups is decreased, minimizing the perceived treatment effect. 14. Treatment diffusion Similar to the John Henry effect is treatment diffusion. If subjects in the control group perceive the treatment as very desirable, they may try to find out whats being done. Both the John Henry Effect and Treatment Diffusion can be controlled if experimental and control groups are isolated. 15.

• Threats to external validity

• Reactive effects of testing

• Treatment and Subject Interaction

• Testing and Subject Interaction

• Multiple Treatment Effect

16. Reactive effects of testing Subjects in your samples may respond differently to experimental treatments merely because they are being tested. Since the population at large is not tested, experimental effects may be due to the testing procedures rather than the treatment itself. This reduces generalizability. One type of reactive effect is pretest sensitization.Another type of reactive effect is post-test sensitization. The posttest can be, in itself, a learning experience that helps subjects to put all the pieces together. Different results would be obtained if the treatment were given without a posttest. While researchers must make measurements, care must be taken to measure treatment effect, not add to it, with a post-test. 17. Treatment and Subject Interaction Subjects in a sample may react to the experimental treatment in ways that are hard to predict. This limits the ability of the researcher to generalize findings outside the experiment itself. If there is a systematic bias in a sample, then treatment effects may be different when applied to a different sample. 18. Testing and Subject Interaction Subjects in a sample may react to the process of testing in ways that are hard to predict. This limits the ability of the researcher to generalize findings outside the experiment itself. If there is a systematic bias of test anxiety or test-wiseness in a sample, then treatment effects will be different when applied to a different sample 19. Multiple Treatment Effect Normally we find a single treatment in an experiment. If, however, an experiment exposes subjects to, say, three treatments (A, B, and C) and test scores show that treatment C produced the best results, one cannot declare treatment C the best. It may have been the combination of the treatments that led to the results.Treatment C, given alone, may produce different results. 20.

• Different types of experimental research can be conducted depending on the nature of subjects and the instruments, and the way data are collected and analyzed.

• • - Will there be a control group?

• • - How many subjects will there be?

• • - Will the subjects be randomly selected?

• • - Will each group be pretested?

• • - How will the obtained data be analyzed?

• • What factors may affect the internal validity?

• • What factors may affect the external validity

21. True-Experimental Design Quasi-Experimental design Pre-Experimental Design

• • • Pretest/posttest control group design

• • • Posttest only control group design

• • • Solomon four group design

• • • Non-equivalent control group

• • • Time series

• • • Multiple time series

• • • One-short case study

• • • One-group, pretest/posttest

• • • Static group comparison

Single variable designs Factorialdesigns Types of Experimental Research 22. True Experimental group Design Advantages of the true-experimental design include: Greater internal validity Causal claims can be investigated Disadvantages: Less external validity (not like real world conditions) Not very practical True Experimental Designs Experimental designs are consideredtrue experiments when they employ randomizationin the selection of their samples and control for extraneous influences of variation on the dependent variable. The three designs we will consider in this section are thebest choices for an experimental dissertation. These are the pretest-posttest control group design, the Posttest Only Control Group design, and the Solomon Four Group design. 23. Quasi-experimental Design Without proper randomization Lack of rigorous statistical scrutiny Some advantages of the quasi-experimental design include: Greater external validity (more like real world conditions) Much more feasible given time and logistical constraints Disadvantage: Not as many variables controlled (less causal claims) 24. Pre-experimental Design Lacking in several areas of the true-experimental criteria. No random selection in most of the cases. Employment of just single group that receives treatment, no control group. The advantages are: Very practical Set the stage for further research Disadvantages: Lower validity 25. True Experimental Design The posttest only control group design. The pretest posttest control group design. The Solomon four group control design. 26. Quasi-experimental DesignNonequivalent Control Group DesignTime Series Multiple Time Series 27. Pre-experimental Design The one-shot case studyOne group Pretest Posttest studyThe static group comparison study 28. If there is only one independent variable that can be manipulated, then asingle-variable design is used. If there are two or more independent variables, and at least one can be manipulated, then a factorial design should be chosen. 29. Single-variable designs.These studies are classified under three main headings depending on the degree of control maintained on other variables:1. Pre-experimental designs (low degree of control)2. True experimental designs (high degree of control)3. Quasi-experimental designs (medium degree of control) 30.

• Pre-experimental designs

• Classified depending on whether there is an involvement of one or two groups, and whether the groups are posttested only, or both are pretested and posttested:

• One-shot case studies

• One-group pretest-posttest design:

• Static-group comparison design:

31. One-shot case studies:One group is exposed to the treatment, and only a posttest is given to observe or measure the effect of the treatment on the dependent variable within the experimental group. Since it is applied on a single group, there is no control group involved in this design.First of all, the chosen group is exposed to the treatment ,and then it is tested only once for the purpose of measuring the degree of change on the dependent variable after the treatment. 32. One-group pretest-posttest design:One group is pretested and exposed to the treatment, and then posttested. This is called a one-group pretest-posttest design because the two tests are administered to the same group.The first one is administered at the beginning of the treatment and the second one at the end. 33. Static-group comparison design At least two groups are involved. After one group receives the treatment, all groups are posttested.This design has better control over most of the variables 34.

• Advantages of pretest design

• • Equivalency of groups

• • Can measure extent of change

• • Determine inclusion

• • Assess reasons for and effects of mortality

• Disadvantages of pretest design

• • Time-consuming

• • Sensitization to pre-test

35. True experimental designsHave the highest level of control among the three single-variable experimental designs because the subjects within the groups arerandomly assigned for each group . When subjects are randomly assigned, there is higher control of the internal validity as well as the external validity. Moreover, there is always acontrol groupto compare the results of the subjects in the experiment with other subjects of similar status that have not been exposed to the treatment. 36. True experimental research may be designed with or without a pretest on at least two groups of randomly assigned subjects. The classification of true experimental designs is made accordingly :1. The posttest-only control group design2 .The pretest-posttest control group design3. Solomon four-group design 37. Posttest Only Control Group Subjects are randomly selected and assigned to two groups. Due to randomization, the two groups are statistically equal. No pretest is given. One group receives the Treatment RXO 1 RO2 Example. Third graders are randomly assigned to two groups. Then one group receives a special study on the life ofIqball. Both are tested on their knowledge of Iqball at the conclusion of the study. Analysis. The difference between group means (O1 and O2) can be computed byan independent groups t-test. 38. 39. Pretest-Posttest Control Group Two randomly selected groups are measured before (O1 and O3) and after (O2 and O4) one of the groups receives a treatment (X). RO 1XO 2RO3O4 Example. Third graders are randomly assigned to two groups and tested for knowledge of Arithmetic. Then one group gets a specialstudy on Arithmetic. Both are then tested again. 40. 41. Analysis. The t-test for independent samples (Chapter 20) can be used to determineif there is a significant difference between the average scores of the groups (O2 and O4). You can also compute gain scores (O2 - O1 and O4 - O3) and test the significance of the average gain scores with the matched samples t-test. This designs only weakness is pre-test sensitization and the possibleinteraction between pretest and treatment. 42. Solomon four-group designTakes the effect of pretest and posttest sensitivization into consideration.It is the combination of pretest-posttest control group (G1 and G2) and posttest only control group (G3 and G4) designs. In this case,subjects are randomly selected and placed into four groups; 43. 44. Example. Third graders are randomly assigned to 1 of 4 groups. The knowledgeoflanguage is measured in groups 1 and 2. Groups 1 and 3 are given a special study on the language learning. When the special study is over, all four groups are tested. Analysis. One-way ANOVA can be used to test the differences in the four posttestmean scores (O2, O4, O5, O6). The effects of the pretest can be analyzed by applying a t-test to the means of O4 (pretest but no treatment) and O6 (neither pretest or treatment). The effects of the treatment can be analyzed by applying a t-test to the means of O5 (treatment but no pretest) and O6 (neither pretest or treatment). Subject maturation can be analyzed by comparing the combined means of O1 and O3 against O6. 45. - the first and the second groups are retested;- the first and the third groups are exposed to the treatment, and the second and the fourth groups are taken as control groups;- all four groups are posttested.This design provides the best result but it requires a large sample so that enough subjects could be assigned to four groups.When the sample is large, administering the tests becomes difficult, time and energy consuming. 46. Quasi-experimental Designs The termquasi- (pronounced kwahz-eye) meansalmost, near, partial, pseudo, orsomewhat . Quasi-experimental designs are used when true experiments cannot be done. A common problem in educational research is the unwillingness of educational administrators to allow the random selection of students out of classes for experimental samples. Without randomization, there are no true experiments. So, several designs have been developed for these situations that are almost true experiments, or quasi-experimental designs.Well look at three:the time series,the nonequivalent control group design,and the counterbalanced design. 47. Time Series Establish a baseline measure of subjects by administering a series of tests over time (O1 through O4 in this case). Expose the group to the treatment and then measure the subjects with another series of tests (e.g., O5 through O8). O1O2O3O4XO5O6O7O8 Comments. Since there is no control group, one cannot determine the effects of history on the test scores.Instrumentation may also be a problem (Are the tests equivalent?)the reactive effects of repeated testing of subjects is a source of external invalidity. 48. Nonequivalent Control Group Design Subjects are tested in existing or intact groups rather than being randomly selected. The dotted line in the diagram represents non-equivalent groups. Both groups are measured before and after treatment. Only one group receives the treatment. O 1XO 2--------------------- O3O4 Comments. This design should be used only when random assignment is impossible. It does not control forselection-maturation interactionstatistical regression.pretest sensitization. 49. Counterbalanced Design Subjects are not randomly selected, but are used in intact groups. Group 1 receives treatment 1 and test 1. Then at a later time, they receive treatment 2 and test 2. Group 2 receives treatment 2 first and then treatment one. Time 1 2 Group1X1OX2O Group2X2OX1O Example. Two third grade classes receive two special studies on language: one inclassroom and the other on a computer. Class 1 does the classroom work first, followed by the computer; class 2 does the computer work first. Both groups are tested after both treatments. 50. Analysis. Use the Latin Squares analysis (beyond the scope of this text). Comments. Since randomization is not used in this design, selection-maturation interaction may be a problem. Multiple treatment effect is a possible source of external invalidity. 51. 52. Thank You