5/6/2011 1 Usability, Reliability, and Validity of Remote Autism Diagnostic Observation Schedule (ADOS) Module 4 Administration Jamie L. Schutte, MS, CRC, Michael McCue, PhD, Bambang Parmanto, PhD, & Wayan I. Pulantara, MS University of Pittsburgh Outline I. Background – Adults with High- Functioning Autism – Diagnosing ASD in Adults II. Remote Assessment III. Methods I. ADOS II. VISyTER IV. Clinical Usability IV. Procedural Usability Concerns V. Maintaining Standard Administration Procedure V. Reliability VI. Validity
11
Embed
Usability, Reliability, and Validity of Remote Autism ... · PDF file5/6/2011 1 Usability, Reliability, and Validity of Remote Autism Diagnostic Observation Schedule (ADOS) Module
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
5/6/2011
1
Usability, Reliability, and Validity of Remote Autism Diagnostic Observation
Schedule (ADOS) Module 4 Administration
Jamie L. Schutte, MS, CRC, Michael McCue, PhD,
Bambang Parmanto, PhD, & Wayan I. Pulantara, MS
University of Pittsburgh
Outline
I. Background
– Adults with High-Functioning Autism
– Diagnosing ASD in Adults
II. Remote Assessment
III. Methods
I. ADOS
II. VISyTER
IV. Clinical Usability
IV. Procedural Usability Concerns
V. Maintaining Standard Administration Procedure
V. Reliability
VI. Validity
5/6/2011
2
Autism Spectrum Disorders
• ASDs are developmental disorders
• ASDs are usually diagnosed in early childhood
– Behavioral observation
– Developmental history
Statement of the Problem
• From 2002 to 2006, ASD prevalence increased an average of 57%
• Though ASD onset is prior to age 3 years, it is a lifelong disability
• Adults with ASD may be currently undiagnosed or misdiagnosed
3,326
8,450
15,670
1,407
3,825
10,140
19,587
-
5,000
10,000
15,000
20,000
25,000
2005 2010 2015 2020
Number of individuals diagnosed with autism in Pennsylvania ≥ 21 years of age
no new diagnosis assuming new diagnoses proportional to current rate
5/6/2011
3
Problem (cont.)
• The Autism Diagnostic Observation Schedule (ADOS) has recently become part of the gold standard in ASD diagnosis
• ADOS training opportunities are relatively limited and potentially difficult to access
• Proposed Solution: Remote ADOS module 4 administration system
Telehealth
Telemedicine
Telerehabilitaiton Telepsychiatry
Teleassessment
Telehealthcaree-Health/ Education
The remote administration of systematic procedures for observing and describing
behaviors with the aid of numerical scales or fixed categories, through use of interactive
videoconferencing between a client (and usually a technician) at a local site and a remotely located
assessment expert.
5/6/2011
4
Remote Assessment
Strengths
• Improved access
• Improved quality
• Reduced health-care costs/cost containment
Potential Limitations
• Policy
• Reliability and Validity
ADOS Module 4
• Adolescents/adults
• Verbally fluent
• 10-15 activities with focus on behaviors important in the diagnosis of ASD
• Cutoff scores for both a narrow diagnosis of Autism and a broader diagnosis of PDD
5/6/2011
5
Versatile and Integrated System for Telerehabilitation
(VISYTER)
Clinician-side
5/6/2011
6
Figure 5• Screenshot of the Autism Diagnostic Observation
– Participants performed a remote ADOS module 4 assessment on a volunteer (mock client)
• Psychometric standardization:– “Were you forced to break standard face-to-face
administration procedure? (Yes or No.)
– If yes, how? (Narrative response)
– If yes, to what degree? (Likert scale, 1 to 7)
5/6/2011
9
Results:Procedural Usability Concerns
• Feedback:
– Typos on scoring form
– A glitch in the system that wipes out notes if the administrator forgets to his “save” regularly
– The problem of visual stimuli not staying on the screen simultaneously with the administrator taking notes
• PSSUQ:
Results:Maintaining Standard
Administration Procedure
• Of the 14 ADOS activities administered, only 6 were noted by at least one subject to force breaking standard, face-to-face administration
• Mean ranking was 2.4, indicating that generally breaks in standardization were minimal
5/6/2011
10
Conclusions
• The usability study identified major system deficits and elicited suggestions for making the system more usable
• A variable affecting the usability of the system was familiarity, proficiency, and comfort with computer technology
Conclusions, cont.
• When examining an alternative administration method for a previously standardized assessment, it is important to evaluate whether or not the new administration has equal reliability and validity to the way the evaluation was originally standardized
• In this study, the method of evaluating potential drift was a semi-structured interview with open- and close-ended question administered after each item
• This method could be considered a starting point for other researchers interested in evaluating whether or not remote administrations of standardized procedures are valid
Study 1: Clinical Usability Study 2: Reliability Study 3: Validity
Given the number ofadults with autism in need
and the lack of trained clinicians, assessment services are
not available at thelevel required.
Adults with Autism Spectrum Disorders
(ASDs)
• Diagnostic criteria:
•Impairment in social interaction
•Restricted and repetitive interests and
behavior
•Impairment in communication
•“Urgent public health concern”
• 57% prevalence increase from 2002-2006
•Life-long disability, despite clinical and
research focus on children
Diagnosing ASD… in Adults• No genetic or medical test
• Adults may be undiagnosed or misdiagnosed
• Behavioral presentation – different in adults vs.
children
• Developmental history
• Onset prior to 3 years
• Difficult to access for adults
• Rule out – MR, psychosis, cognitive or
psychiatric disability
• Standardized assessments – adults receive little
attention
Autism Diagnostic Observation Schedule
(ADOS)
• Important component of “gold standard” ASD
diagnosis
• Semi-structured, standardized assessment of:
• Communication
• Social interaction
• Play or imaginative use of materials
• Must be administered by an experienced and
highly trained clinician
• # and location of trained administrators does
not meet need
Study sample
N=14
Face-to-face ADOS
(n=14)
Remote
ADOS
Face-to-face
ADOS
Was
hout
-3 m
onth
s
Remote ADOS
(n=14)
(Random assignment)
Teleassessment
Remote ADOS Module 4 Administration System
Within-subject crossover design:
Aim 1: Clinical usability and participatory development
•Phase 1 – an experienced ADOS administrator (Schutte) will participate, from the beginning, in the design and development of the ADOS teleassessment system
•Phase 2
•Methods
•Participatory study design
•Experienced clinicians who are not part of he development team will conduct a usability evaluation
•Participants – clinically or research reliable ADOS administrators with at least 1 year experience
Aim 2: Develop a remote administration procedural manual
Hypotheses
• ADOS scores will not differ significantly between remote and face-to-face assessment conditions
• Participants will indicate equivalent satisfaction between remote and face-to-face assessments
Participants
• Inclusion: Adults with PDD diagnosis
• Exclusion
• ADOS module 4 evaluation within 90 days
• Significant hearing or visual impairment
Power
• Recruiting 28 participants, with an assumed dropout rate of 20%, will yield a final sample size of n=22
• Yields 80% power to detect a significant result for ICC
Aim
• To estimate diagnostic accuracy of a remotely administered ADOS module 4
Test methods
• Reference standard – Diagnosis by 2 psychiatrists
• Readers of the remote ADOS administration and reference standard will be independent and blind to the results of other test
Participants
• Inclusion: Dx of Autism, ASD, NS, or NT; adolescent/adult, native English speaker
• Exclusion: ADOS module 4 evaluation with 90 days; significant hearing or visual impairment
Power
• For overall classification accuracy, a sample size of 50 yields a precision level of 0.1 using a 95% confidence interval