Experimental design

EXPERIMENTAL DESIGN

Group 5

EXPERIMENTAL DESIGNPresenter: Ngoc Cam

Content

1. What is experimental design?2. Types of experimental design:– Pre- Experimental design– True experimental design– Quasi-experimental design– Ex post facto design– Factorial design

What is Experimental Design?

• Experimental design refers to a plan for assigning experimental units to treatment conditions.

• Experimental research answers the question “What if?” The researcher manipulates independent variables (e.g., type of treatment, teaching method, communication strategy) and measures dependent variables (anxiety level, reading comprehension) in order to establish cause-and-effect relationships between them.

• The independent variable is controlled or set by the researcher. The dependent variable is measured by the researcher

A good experimental design serves three purposes• Causation. It allows the experimenter to make

causal inferences about the relationship between independent variables and a dependent variable.

• Control. It allows the experimenter to rule out alternative explanations due to the confounding effects of extraneous variables (i.e., variables other than the independent variables).

• Variability. It reduces variability within treatment conditions, which makes it easier to detect differences in treatment outcomes.

PRE-EXPERIMENTAL DESIGNPresenter: Nguyen Ngoc Cam

Definition

• Pre-experiments are the simplest form of research design. In a pre-experiment either a single group or multiple groups are observed subsequent to some agent or treatment presumed to cause change.

• Data collected with these designs is highly suspect.

Types

1. One-shot case study design 2. One-group pretest-posttest design3. Intact group design

1. One-shot case study design

• Characteristic:– No control or comparison group– Experimental instruction or treatment are

given in a period of time– At the end, have a test

• The result: neither valid nor generalizable

2. One-group pretest-posttest design

• Characteristic:– No control or comparison group– Experimental instruction or treatment are

given in a period of time– Have two tests, at the beginning and at the

end

• Changes in the outcome of interest are presumed to be the result of the intervention or treatment

• Improvement over the one shot case study because you have measured the gains that the subjects have made rather than just looking at how well everyone did at the end

Validity of Results

• An important drawback of pre-experimental designs is that they are subject to numerous threats to their validity

• it is often difficult or impossible to dismiss rival hypotheses or explanations.

• researchers must exercise extreme caution in interpreting and generalizing the results from pre-experimental studies.

• One reason that it is often difficult to assess the validity of studies that employ a pre-experimental design is that they often do not include any control or comparison group.

• Without something to compare it to, it is difficult to assess the significance of an observed change in the case.

• The change could be the result of historical changes unrelated to the treatment, the maturation of the subject, or an artifact of the testing.

• Even when pre-experimental designs identify a comparison group, it is still difficult to dismiss rival hypotheses for the observed change.

• This is because there is no formal way to determine whether the two groups would have been the same if it had not been for the treatment.

• If the treatment group and the comparison group differ after the treatment, this might be a reflection of differences in the initial recruitment to the groups or differential mortality in the experiment.

Advantages and Disadvantages

• Advantages:• Pre-experiments can be a cost-effective way

to discern whether a potential explanation is worthy of further investigation.

• Disadvantages:• Pre-experiments offer few advantages since it

is often difficult or impossible to rule out alternative explanations. The nearly insurmountable threats to their validity are clearly the most important disadvantage of pre-experimental research designs.

Intact Group Design and True Experimental design

Presenter : Minh Sang

Intact Group Design

This is the design that most classroom researchers use.

Step 1 : Select 2 classes to make 2 groups. One is experimental group and the other is control group ( You can decide it by the flip of a coin )

Step 2 : You give the treatment ( experimental instruction ) to the experimental group, not the control group

Intact Group Design

Step 3 : Give the 2 groups a posttest. After the posttest, we can have the result for the research.

In short, an intact group design is : G1 x T1G2 T1

G1 : Experimental GroupG2 : Control GroupT1 : Posttest

Intact Group Design

Example : You want to investigate the effect of

grammar correction on the writing skills of ESL students

Step 1 : Select two similar groups of ESL ss. Flip the coin to decide which group is the experimental and control group

Step 2 : Give the Exp Group the treatment ( grammar correction ) and do nothing with the Control group.

Intact Group Design

Step 3 : Give a pottest to 2 groups. After the test, you will have a conclusion that grammar correction is effective or not ( If the Exp Group has the higher scores, it means that your treatment – grammar correction is effective )

True Experimental Design

This design is used for situations in real life, when we don’t have any particular groups or classes, teams for our research.

It is similar to the intact group design. and you may have a pretest for the Pretest pottest control group design

True Experimental Design

Pottest only control group : this is nearly the same as the the intact group design. The difference is that we choose the members for the group randomly :

So we have : G1 ( random ) X T1 G2 ( random ) T1

True Experimental Design

Pretest pottest control group design : We may have a pretest for this design :

G1 ( random ) T1 X T2G2 ( random ) T1 T2

So why do we have the T2 ( prettest ) ?

True Experimental Design

The T2 ( prettest ) is given when the time you have between the prettest and pottest is not considerable ( not sufficient ) and it may affect the conclusion of your research.

It is when you give a prettest to test the knowledge, ability…of the 2 groups that you have. After all, your conclusion should be much more defensible.

QUASI-EXPERIMENTAL DESIGN

Presenter: Minh Dang

• Quasi-experimental design is practical

compromises between true

experimentation and which we wish to

investigate.

• Quasi-experimental design is susceptible

(easily effected) to some of the questions

of internal and external validity

• By using Quasi-experimental design, we

control as many variables as we can and

also limit the kinds of interpretations we

make about cause-effect relationships and

hedge the power of our generalization

statements

Time-series design

• Because of limitations sometimes it is impossible to have a control group

• use time-series design to deal with the lack of control group

• Time-series design use several pretest and several postest

• No treatment during the pretests know the changes when there are no treatments

• After some pretests treatment some posttests changes from the treatment

• more accurate comparison, conclusion

• Line 1: no effect

• Line 2: negative effect

• Line 3: positive effect treatment is

effective

Equivalent time sample design

• The treatment is introduced and reintroduced between every other pretests and posttests

• Test 1 treatment test 2 treatment test 3 treatment …

In short

• Quasi-experimental design: control many variables and reduce limitations

• Time-series design: pretests treatment posttests

• Equivalent time-sample design: test treatment test treatment …

EX POST FACTO DESIGNSPresenter: Huu Loc

EX POST FACTO DESIGNS

When researchers control the threats to internal and external validity, they are trying to find a direct relationship between the independent and dependent variables.

In other words, they select the population, sample, treatments, and variables in order to find a cause-and-effect relationship between the variables.

example

Not randomly select your Ss, organize your control and treatment groups, and control for factors aside from the media lessons which might influence the results

Can not draw causal relationships between your media materials and Ss' improvement in ability to turn down requests gracefully in English.

You may have created a series of media lessons on how to say no to requests in English.

When considering all the factors that you would need to control, you might think that designing a true experimental research project is almost impossible.

But , it should not mean that we have to give up approximating the ideal as much as possible. X causes Y is an extremely difficult thing to do unless the research is carefully designed and as many extraneous factors are controlled as possible.

When there is no possibility of random selection of Ss, instead of abandoning the research, we simply have to limit the domain of our claims.

oWe have to avoid making cause and effect statements.

EX POST FACTO designs are often used when the researcher does not have control over the selection and manipulation or the independent variable.

Researchers look at the type and/or degree of relationship between the two variables rather than at a cause-and-effect relationship.

example We can study the relationship between scores on

a school- leaving exam in ESL and teachers' ratings for the Ss using an ex post facto design.

We can see if there is a certain amount of agreement between the two sets of scores. Any relationship between the scores of the groups would not be related to any instructional program we had given them before the test.

The designs are called ex post facto. The researcher has no control over what has already happened to the Ss.

The treatment has been given prior to the research project.

There’re 2 EX POST FACTO designs

o Correlational designs

o Criterion group design

Correlational designs are the most commonly used subset, in which a group of Ss may give us data on two different variables. o For example, students planing to study in the US

take the TOEFL. Many universities also have entrance exam to administer to students. We can then look at the relationship of Ss’ scores on one test to their scores on the other.

o Or, foreign students may be asked to take both the Graduate Record Exam (GRE) and an English placement exam prior to admission to a university.

o The score for each S on one test can be compared with the score on the other, allowing us to see whether whose students who score high on one lest also score high on the other.

The schematic representation of this design would be

T1 T2

• It’s no causal relationship between the two variables --> the distinction between independent and dependent variables is not well defined.

• It is arbitrary to call one or the other the independent variable.

• But, it is usually the case that the investigator may be more concerned with one than the other and may therefore label the first the independent variable and the second the dependent variable and show this by the labels X and Y.

In a CRITERION GROUP DESIGN, two groups of Ss are compared on one measure.

In this design, two groups of Ss are compared on one measure. With this design, you might, for example, measure the reading peed of Iranian and French students, assuming you want to see how related or different they might be.

The design would look like this: G1 T1 G2 T1

You can change the design into a two-criterion design by considering level of language proficiency as well as their native language.In this case the criterion group design forms a factorial design.

FACTORIAL DESIGNSPresenter: Ngan Giang

DEFINITION

• Is simply the addition of more variables to the other designs

• There will be more than one independent variable considered

• The variables may have one or many levels

2 x 2 example

Room Temperature

Test Difficulty (Level) 50 degrees (Level) 90 degrees

(Level) Hard Test Hard Test in 50 degrees Hard Test in 90 degrees

(Level) Easy Test Easy Test in 50 degrees Easy Test in 90 degrees

We are interested in studying the effect of room temperature on test taking. To do this, we compare test scores of students who take a test in a 90 degree room vs. those who take a test in a 50 degree room.

• Factor 1: Treatment– psychothera

py– behavior

modification

• Factor 2: Setting– inpatient– day

treatment– outpatient

• higher scores mean the patient is doing worse. 

• day treatment is never the best condition.

• psychotherapy works best with inpatient care and behavior modification works best with outpatient care.

THE PROS AND CONS

• Factorial designs are extremely useful to psychologists and field scientists as a preliminary study, allowing them to judge whether there is a link between variables, whilst reducing the possibility of experimental error and confounding variables .

• The factorial design, as well as simplifying the process and making research cheaper, allows many levels of analysis. As well as highlighting the relationships between variables, it also allows the effects of manipulating a single variable to be isolated and analyzed singly.

• The main disadvantage is the difficulty of experimenting with more than two factors, or many levels. A factorial design has to be planned meticulously, as an error in one of the levels, will jeopardize a great amount of work.

Summary

1. What is experimental design?2. Types of experimental design:– Pre-experimental design– True experimental design– Quasi-experimental design– Ex post facto design– Factorial design

Thanks for your attention!

• Group 5:1. Dinh Quoc Minh Dang2. Vo Huu Loc3. Nguyen Dinh Minh Sang4. Nguyen Ngoc Cam5. Tran Thi Ngan Giang