Part 2 of 3 By: Danielle Davidov, PhD & Steve Davis, MSW, MPA INTRODUCTION TO RESEARCH: MEASUREMENT.

Part 2 of 3 By: Danielle

Davidov, PhD &Steve Davis,

MSW, MPA

INTRODUCTION TO RESEARCH:

MEASUREMENT

1) Threats to research studies

2) Steps in the research design process

3) Identifying and defining variables

4) Validity and reliability of measurement

OUTLINE

Starts After the research question has been developed and refined

The who, what, when, where, and how of research

It comprises the Materials and Methods and Limitations sections of publications

RESEARCH DESIGN

The goal of research design is to provide the most valid and correct answer to the question i.e., we want to make sure we are “doing it right”

This is done by minimizing the threats to the soundness of your study’s conclusion(s): CHANCE

BIAS

CONFOUNDING

WHY IS DESIGN IMPORTANT?

The threat that the study’s findings are merely the result of random processes (chance) i.e., the findings are a “fluke”We can’t do much to control random error

Also referred to as: Type 1 Error Random Error Unsystematic Error

STUDY THREATS: CHANCE

The threat that the study’s results are due to an unfair preference given to one group or a set of outcomes in a studyWe can try to control bias in our study design and subject recruitment

Also referred to as: Systematic Error

STUDY THREATS: BIAS

The threat that the association or relationship observed in the study is influenced by or related to another variableWe can control for this in our study design, subject recruitment, and data analysis techniques

STUDY THREATS: CONFOUNDING

We try to minimize the three main threats during all stages of design process, which are:

1) Identifying and Defining Variables*2) Selecting Measurement Methods*3) Selecting (Sample) Subjects4) Selecting a Research Design5) Establishing an Analysis Plan

*We will be talking about steps 1 and 2 in this presentation

STEPS IN RESEARCH DESIGN

What do you want to measure? (Identify) Ex) Patient satisfaction levels with ultrasound vs. history and physical

exam only

How do you want to operationalize “patient satisfaction?” (Define) Ex) Answers of “Good”, “Very Good”, or “Excellent” on a survey

given to patients about the care they received in the emergency department

IDENTIFY & DEFINE VARIABLES

Classifying Variables:

Independent Variable The variable that has an effect on or influences the

dependent variable. This is the FACTOR/INTERVENTION

i.e.) History and Physical Exam or Ultrasound + H & P

Dependent Variable The variable that is affected by, or dependent upon,

the independent variable. This is the OUTCOME i.e.) Patient Satisfaction

IDENTIFICATION & DEFINITION OF VARIABLES

Classifying Variables (continued)

Confounding Variables –a variable that is related to both the independent and the dependent variable CONFOUNDER or CONTROL variable Common confounders/controls in medical research:

Age Gender Race Severity of Illness

IDENTIFICATION & DEFINITION OF VARIABLES

Controlling for Confounding Variables Not adequately controlling for confounding variables can

have disastrous consequences on your research Identify and define as many as possible

From previous literature From clinical observations From theory

IDENTIFICATION & DEFINITION OF VARIABLES

What if we didn’t consider these important variables when examining the relationship between the Independent and Dependent variables???

Operationalizing variables The process of defining variables in a measurable way.

IDENTIFICATION & DEFINITION OF VARIABLES

Levels of Measurement(NOIR)

Nominal

Ordinal

Interval

Ratio

IDENTIFICATION & DEFINITION OF VARIABLES

Lower

Higher

Characteristic data that cannot be rank ordered

This data is “categorical” – made up of “categories”, not “levels” or “increments” Ex) Ice cream flavors—vanilla is not “better” or “more”

than chocolate Examples: Gender, Race, Student, Marital Status, State or Country of Residence, Insurance Status, Discharge Status, etc. Yes/No Responses are Nominal This type of data is usually “descriptive”

Used to describe a population or sample

NOMINAL LEVEL DATA

Data that can be rank ordered but that do not have measurable distances between each level of rank

Likert Scales - Strongly Disagree to Strongly Agree Class rank - Freshman, Sophomore, Junior, Senior; PGY-I, PGY-II,

PGY-III Degree of illness: None, Mild, Moderate, Severe

Senior is a higher rank than Freshman, but there is no way to quantify how much higher Senior is vs. “Freshman” or how much “more” illness those with a severe illness have compared to those with a mild illness

ORDINAL LEVEL DATA

Data that can be ordered and that have a measurable distance between each level The Interval Scale - Distances between positions are

equal, but "0" is an arbitrary assignment. For example, with temperature, each degree is equally distant from another, but "0" does not mean that there is no temperature. It is simply a reference point on the scale.

The Ratio Scale - All positions are equally distant and "0" means that the value is truly "0". If you have "0" money, you have none. But if you have $200, you have twice as much as a friend who has $100.

Examples of Interval/Ratio Data: Age Height Weight Many Clinical Serum Levels Blood Pressure

INTERVAL/RATIO DATA

Defining variables at higher levels of measurement allows the use of statistical tests that have more Power Power = the probability of finding a true relationship of

difference if it genuinely exists

It is usually better to collect data at higher levels of measurement and then collapse into categories later Ex) Age

What is your age? ____ (best) vs.

What is your age? vs. 18 – 25 26 – 35 36 – 45 etc.

vs.

Under 40 & over 40

LEVELS OF MEASUREMENT AND POWER

Once you have defined and operationalized your research question’s variables, you must decide how to measure them and/or what measurement tool you will use.

There are two forms of error that we must minimize when selecting measurement methods and/or tools: Random error (CHANCE) Nonrandom error (BIAS AND CONFOUNDING)

SELECTING A MEASUREMENT METHOD

To minimize random error we choose a tool or method that is RELIABLE Reliability – The extent to which a measurement method or

tool produces the same results over several measurements AKA precision

Threats to Reliability Observer error: different measurements from the same or

different observers (i.e., blood pressure readings) Instrument error: different measurements from the

instrument itself due to extraneous environmental factors Subject error: different measurements from the natural

biological variability among humans

RELIABILITY

How to assess Reliability: Repeat measurements on the same subject.

Give a survey at two different time points Take blood pressure at two different time points

Use more than one observer. Assess inter-rater agreement

Have two different people take blood pressure

How to maximize Reliability: Standardize the measurement methods

Choose surveys and instruments that have been proven to be reliable

Train observers

Refine & update instruments

Repetition Averaging the measures can cancel out error.

ASSESSING & MAXIMIZING RELIABILITY

To minimize nonrandom error we choose a measurement method and/or tool that is VALID Validity – The extent to which a measurement method and/or

tool measures what is sets out to measure AKA Accuracy

Threats to Validity Observer bias: conscious or unconscious distortion in the

perception and/or reporting of the measurement Subject bias: bias-distortion of self-reported measurements due to

subjects beliefs and biases Hawthorne Eff ects and Social Desirability

Instrument bias: consistently biased or inaccurate measurements due to such things as worn parts or mechanical malfunction

Lack of a clear gold standard: No “best” instrument out there Abstract/behavioral variables: These things are diffi cult to

measure Patient satisfaction, pain, quality of life, intelligence

VALIDITY

Strategies for maximizing Validity Blinding

Ex) Do not allow physician who is taking blood pressure readings to know which subjects are receiving blood pressure medication

Deception Ex) Do not allow subjects to know which “group” they are in Give placebos

Instrument Calibration Make sure instruments are working properly

Use standardized/validated surveys and assessment tools Find these from literature searches Usually better to use “pre-made” surveys or instruments than

creating one from scratch

MAXIMIZING VALIDITY

Identify and define your variables at the VERY beginning of your study Don’t forget your control or confounding variables!

Using higher levels of measurement is better!

Choose instruments and data collection tools that are: RELIABLE – produce the same results over time (precise) VALID – produce results that represent “the truth”

(accurate)

IN SUMMARY

Go through “Introduction to Research Part 3: Sampling and Design”

NEXT STEPS

Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing Clinical Research. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2001:37-49

Spector PE. Research Designs. Newbury Park, CA: SAGE Publications, Inc.; 1981. ISBN: 0-8039-1709-0

http://www.research-assessment-adviser.com/levels-of-measurement.html

REFERENCES