Postgraduate Course 7. Evidence-based management: Research designs.

Postgraduate Course

7. Evidence-based management:

Research designs

Postgraduate Course

Empirical circle

Postgraduate Course

Research designs: 4 elements

Randomization

Comparison

Before vs after

Prospective vs retrospective

Postgraduate Course

Randomized controlled study

Controlled before-after study

Cohort / panel study

Before-after study

Post-test only study

Survey

Case study

Experiment

Quasi-experiment

Observational study

Longitudinal study

Cross-sectional study

Qualitative study

Research designs: terms

Postgraduate Course

CBA: controlled before-after study

In a controlled before-after study two or more

groups are compared with each other, usually

comprising one group in which an intervention is

carried out (experimental group) and one group

where no or an alternative intervention is

conducted (control group).

Postgraduate Course

=

CBA: controlled before-after study

Postgraduate Course

In the case of randomization, the groups compared with each

other are selected entirely randomly, for example by drawing lots.

This means that each participant (or other unit such as a team,

department or company) has an equal chance of being in the

intervention or control group. In this way, the influence of any

distorting factors is spread over both groups so that these groups

are as comparable as possible with each other with the exception

of the intervention.

Better (but not always feasible): randomization

Postgraduate Course

=

even

oneven

Randomized controlled study (RCT)

Postgraduate Course

Cohort / panel study

Starting point: intervention/exposure (independent variable)

Longitudinal study where large groups of people or companies (cohort / panel) are followed over a long period to see (prospective) whether differences occur among the groups.

=

Postgraduate Course

Case-control study

Starting point: outcome (dependent variable)

Longitudinal study in which one group of people or companies with a particular outcome is compared retrospectively with a group that does not have this outcome.

=

✔

✗

VS

VS

Postgraduate Course

Examples of (flawed) case-control studies

Postgraduate Course

versus

Study in which data are obtained or particular

characteristics are measured of a population before

versus after an intervention / exposure / event, to

measure the effect or correlation.

Before-after study

Postgraduate Course

=

Posttest-only

Postgraduate Course

Cross-sectional study

Study in which data of a statistically significant sample of

a population (managers, CEO’s, employees) is gathered

at one point in time. It provides a snapshot of the current

condition but does not explain cause and effect.

Cross-sectional studies

include surveys

Postgraduate Course

Cross-sectional study

?

?

?

Postgraduate Course

Study in which a large number (qualitative or quantitative)

of aspects of a single case (organization or team) was

investigated in depth over a long period within the case’s

own context.

Case study

Postgraduate Course

Experimental research

‘True’ experiment: RCT

‘Quasi’ experiment: CBA

Experimental research refers to studies where the

researcher manipulates one (or more) variable and

controls the other variable(s) to determine whether there

is a causal relation between the manipulated variable and

the outcome.

Postgraduate Course

Observational research

Natural experiment

Cohort / panel study

Case-control study

Observational research refers to studies where the

researcher merely observes but does not intervene,

with the intention of finding correlations among the

observed data

Postgraduate Course

Qualitative research explores and tries to understand

people's beliefs, experiences, attitudes, behaviour and

interactions. It generates non-numerical data. The best-

known qualitative research-methods include in-depth

interviews, focus groups, documentary analysis and

participant observation.

Qualitative research includes

Case studies

Ethnographic studies

Field research

Grounded theory approach

Qualitative research

Postgraduate Course

Longitudinal research

Cohort / panel study

Repeated measures

Times series

A longitudinal study is an observational study that

involves repeated observations (measurements)

of the same variable(s) over long periods of time

(sometimes years or even decades).

Postgraduate Course

Mixed methods

Postgraduate Course

Better than a single study: a replication study

Better than a replication study:a systematic review / meta analysis

If there were 100 studies, 99 of which gave a ‘negative’ result (where, say, the new intervention appeared to be not effective), while one had a ‘positive’ result (were the intervention appeared effective), it would obviously be a mistake to consider only the single positive study.

But ….

Postgraduate Course

Systematic review / meta-analysis

The intention behind a systematic review is to identify as fully

as possible all the scientific studies of relevance to a particular

subject and to assess the validity and authority of the evidence

of each study separately. As the name indicates, a systematic

review takes a systematic approach to identifying studies and

has the methodological quality critically appraised by multiple

researchers independently of each other, as a consequence of

which the review is transparent and reproducible and can be

monitored. The use of statistical analysis techniques in a

systematic review to pool the results of the individual studies

numerically in order to achieve a more accurate estimate of

the effect is termed a “meta-analysis”.

Postgraduate Course

?

Systematic review / meta-analysis

Postgraduate Course

1. Randomized controlled study

2. Controlled before-after study

3. Cohort / panel study

4. Before-after study

5. Post-test only study

6. Survey

7. Case study

A. Experiment

B. Quasi-experiment

C. Observational study

D. Longitudinal study

E. Cross-sectional study

F. Qualitative study

Which terms belong together?