Top Banner

Click here to load reader

Rebecca A. Grier, Ph.D. Institute for Defense Analyses · PDF file Rebecca A. Grier, Ph.D. Institute for Defense Analyses . 7/28/2014-2 DOT&E Guidance on Surveys

Oct 14, 2020

ReportDownload

Documents

others

  • Surveys in Test & Evaluation: The Good, The Bad, & The Ugly

    Rebecca A. Grier, Ph.D. Institute for Defense Analyses

  • 7/28/2014-2

    DOT&E Guidance on Surveys

    • Surveys are an important aspect of DOT&E evaluation of effectiveness and suitability

    • Surveys are appropriate for quantitatively measuring operator and maintainer thoughts and opinions

    • Use surveys only when appropriate

    • Employ best practices for writing and administering surveys

    – Memo provides a best practices guide attachment

  • An Overview

    What Will Be Covered  What can’t be assessed

    by survey  Some best practices for

    writing questions

    What Won’t Be Covered  Selecting academic

    surveys  Formatting surveys  Selecting response option

    type  DOE with surveys  Analysis of survey data  Interviewing techniques

    28 July 2014 3

  • Wants

    What is a Survey?

    28 July 2014 4

    A systematic collection & analysis of data relating to

    the thoughts of a population.

    Physical System

    Requirements

    What do you see?

    A systematic collection & analysis of data relating to the

    thoughts of a population.

    A systematic collection & analysis of data relating to the

    thoughts of a population.

    Thoughts are relative; Requirements are not!

    Thoughts

    Beliefs

  • Examples Where Measurement by Survey Is Not Appropriate

    • “The ship has protective clothing for every crew member.”

    – Count the protective clothing & compare to number of crew.

    • “Engine exhaust levels in the mission bay do not exceed safety limits ...”

    – Measure exhaust levels with Portable Emissions Measurement System (PEMS) & compare to safety limits

    • “Temperatures in primary work spaces were adequate.” – Measure by a thermometer & compare to requirements in

    MIL-STD-1472G – e.g., 5.5.2.1.4 part g: Limited thermal tolerance zones. Where hard physical work is to be required for

    more than two hours, an environment not exceeding… 25 °C (77 °F) shall be provided. … shall be decreased 5.0 °C (9.0 °F) for complete chemical protective uniforms, 4.0 °C (7.0 °F) for intermediate clothing systems, and 3.0 °C (5.0 °F) for body armor.

    28 July 2014 5

  • Survey Purpose & Goal

    28 July 2014

    A systematic collection & analysis of data relating to the thoughts of a population.

    Who Role Sources of Error

    Commissioner Defines Survey’s Purpose & Uses Information from Survey • Not Enough Information • Wrong Information

    Participant/ Respondent Gives Data

    • Answers Different Question • Thinks Too Much • Doesn’t Think Enough

    Analyst Translates Data Into Information • Unable to analyze data • Data Aggregation

    1. Collect specific data for a pre-defined purpose according to rules (i.e., not random thoughts).

    2. The data collection rules determine the validity of the data & the statistical analyses possible.

  • Question Writing Best Practices

    28 July 2014 7

    Knowledge Liability: Respondents Have Enough Information to Answer the Question

    Singularity: Only 1 Idea Per Item

    User Friendly: Items Do Not Require a Lot of Thought or Interpretation (e.g., short, clear, specific)

    Neutrality: Items Do Not Imply Value Judgments Items Are Not Emotionally Charged

    Independence: Responses Will Not Affect Responses to Other Questions (Branching, Redundancy, etc…)

    - Especially important when aggregating responses - Item Response Theory & Internal Reliability

    Available response options match items All possible response options

    Identifier Item Response 1. How many parts of a question are there?

    1 2 3 4 5

  • 1. The system is efficiently reliable.

    User Friendly

    Unclear what the goal of the question is. Therefore no recommended rewording.

    28 July 2014 8

  • 2. Rate the overall ability of the system to provide situation awareness.

    Not Adequate Adequate DK/NA

    2b. If not adequate, rate the degree to which this deficiency degrades effectiveness.

    Very Low Low Moderate High Very High

    28 July 2014 9

  • 2. Rate the overall ability of the system to provide situation awareness.

    Not Adequate Adequate DK/NA

    2b. If not adequate, rate the degree to which this deficiency degrades effectiveness.

    Very Low Low Moderate High Very High

    Knowledge Liability

    28 July 2014 10

  • 2. Rate the overall ability of the system to provide situation awareness.

    Not Adequate Adequate DK/NA

    2b. If not adequate, rate the degree to which this deficiency degrades effectiveness.

    Very Low Low Moderate High Very High

    Knowledge Liability Independence

    28 July 2014 11

  • 2. Rate the overall ability of the system to provide situation awareness.

    Not Adequate Adequate DK/NA

    2b. If not adequate, rate the degree to which this deficiency degrades effectiveness.

    Very Low Low Moderate High Very High

    Knowledge Liability Independence

    All Possible Responses

    6. I trust the information provided by the system.

    28 July 2014 12

    Strongly Agree

    Somewhat Agree

    Slightly Agree

    Slightly Agree

    Somewhat Disagree

    Strongly Disagree

  • 3. Is the training materials complete?

    28 July 2014 13

  • 3. Is the training materials complete?

    Knowledge Liability User Friendly – grammar

    5. I felt as if I needed more training.

    28 July 2014 14

  • 4. Rate the acceptability of system’s Launch Acceptability Region (LAR) displays

    provided to support accurate and timely system employment.

    28 July 2014 15

  • 4. Rate the acceptability of system’s Launch Acceptability Region (LAR) displays

    provided to support accurate and timely system employment.

    Knowledge Liability

    28 July 2014 16

  • 4. Rate the acceptability of system’s Launch Acceptability Region (LAR) displays

    provided to support accurate and timely system employment.

    Knowledge Liability Singularity

    28 July 2014 17

  • 4. Rate the acceptability of system’s Launch Acceptability Region (LAR) displays

    provided to support accurate and timely system employment.

    Knowledge Liability Singularity

    User Friendly (repeated and unnecessary words)

    3. The Launch Acceptability Region (LAR) displays are helpful. 28 July 2014

    18

  • 5. The SSO functionality increases my productive time within the clinic by reducing the amount of time I spend logging into different applications

    when documenting the healthcare provided.

    28 July 2014 19

    Strongly Agree

    Somewhat Agree

    Slightly Agree

    Slightly Agree

    Somewhat Disagree

    Strongly Disagree

  • 5. The SSO functionality increases my productive time within the clinic by reducing the amount of time I spend logging into different applications

    when documenting the healthcare provided.

    Neutrality User Friendly – 27 words!

    4. The SSO function is useful.

    28 July 2014 20

    Strongly Agree

    Somewhat Agree

    Slightly Agree

    Slightly Agree

    Somewhat Disagree

    Strongly Disagree

  • 6. Rate the overall usefulness of the report: Accuracy

    28 July 2014 21

    Completely unacceptable

    Largely unacceptable

    Somewhat unacceptable

    Somewhat acceptable

    Largely acceptable

    Completely acceptable

  • 6. Rate the overall usefulness of the report: Accuracy

    Knowledge Liability Response Options Don’t Match Item

    User Friendly (What is being rated: report acceptability, usefulness, or accuracy?)

    5. The content of the report is useful.

    28 July 2014 22

    Completely unacceptable

    Largely unacceptable

    Somewhat unacceptable

    Somewhat acceptable

    Largely acceptable

    Completely acceptable

    Strongly Agree

    Somewhat Agree

    Slightly Agree

    Slightly Agree

    Somewhat Disagree

    Strongly Disagree

  • 7. Based on your responses above, rate the acceptability of the system.

    28 July 2014 23

  • 7. Based on your responses above, rate the acceptability of the system.

    Independence User Friendly

    (It is the analyst’s job to rate the acceptability of the system)

    8. I would like to use this system to accomplish the mission.

    28 July 2014 24

  • Some Words to Avoid In Writing Surveys

     Accu

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.