Every thing you want to know about surveys… And aren't afraid to ask! Carl Berger January 30, 2001 All sources are available, look on the last slide.

Every thing you want to know about surveys…

And aren't afraid to ask! Carl Berger

January 30, 2001

All sources are available, look on the last slide.

But it’s really not about surveys…

It’s about helping Presidents, Provosts, and CIO's to make decisions!

But… you've got to know the territory!

(Gratuitous comments and/or insights…on the bottom of some slides)

Why?

Institutional ReadinessBasis of strategic plan for transformationFaculty interviews and MERLOTNot e-mail survey

Paul Hagner, Interesting Practices and Best Systems in Faculty Engagement and Support

NLII White Paper January 25, 2001

Paul's paper is must reading!

NLII Readiness

Readiness and Bridges Task Forces Entrepreneurial Faculty and “Second-Wave” Faculty Differences in

EngagementThese two groups of faculty, while united in their commitment to quality learning

environments, are very different in both their technical capabilities and their attitudinal readiness to embrace these new technologies. It would be a serious mistake for administrators to make allocation decisions based solely on the characteristics of the “entrepreneurs,” since their needs and their motivations can differ greatly from the “second-wave” faculty.

… an “enabling environment” is a precondition to institutional change.

universal student access,reliable networks,multiple opportunities for training and consulting, and“a faculty ethos which values experimentation and toleration of falters.”

But how do you know?

Guess Traditional Sources Conventional Wisdom Anecdotes Or…

You could ask them!

Even more important…

Helping decision makers Those who have little time to

develop deep knowledge Present in ways that decisions seem

to pop up Or don’t!

Lots of time its realizing that this data snipit can't help a decision!

Topics Asking the right question Asking the question right! Getting a great response (What is a

great response?) Decision Graphics Hidden meanings An example and some surprising?

results

Advantages to asking (survey)

Real data, almost always dumps myths

Understand all groups, early adopters, early majority, late majority, luddites

Faculty, Students, Administrators…

The real pay-off is that they understand that they (faculty, students, administrators) are part of the process!

Disadvantages of asking

Expensive Time consuming Will they understand it when were

done? And worse yet… you may find out that which you

don’t want to know!

About two years ago we invested heavily in NetG. (an on-line training program) Good idea?… stay tuned…

Take the plunge, needs, wants and reality

Enough to give credence Representative Need to get luddites Keep cost down Understandable And… Getting the right questions but even more… Getting the right analysis and presentation!

We chose paper surveys and non-respondent follow-up to make sure weweren't favoring techie types!

On the shoulders of …

Flashlighthttp://www.tltgroup.org/programs/flashlight.html

UCLAhttp://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm

Michiganhttp://sitemaker.med.umich.edu/cberger/reports

Berkeley, Cal State System and Others on the way, Stanford, Penn State, Minnesota but best…

YOU!Beg, borrow or steal good items and question styles. But give credit!

The 12 Step Program to Success

1. Select audience 2. Categories 3. Initial development with faculty,

students 4. Test with small group 5. Contact 6. Follow-up 7. Data entry

8. Analysis9. Presentation

10. Distribution11. Feedback

12. They want next versionNumber 12 is the measure of success. (Along with using the results for decisions)

Asking the right questionDon't jump too quickly to a survey:The Ehrmann technique (Open focus groups and listen,

listen, listen)

A majority of three (If you hear the same from three folks, unsolicited then it's a survey item)

Now build a surveySmall sample trial (Try out some survey types)

Single page (A few 1-pagers with several groups)

Check, check and check again for interpretation and errors

Try for 1 big paper survey every 2 years to prevent "surveyed to death"

Steve Ehrmann of Flashlight has the best question development technique!Gary Gatien of UM develops excellent questions

Asking the question right

Don’t ask “Do you use…?” Ask “How often do you use…” Don’t ask “Do you use either a or b?” Ask “Check all that apply” Don’t ask “How often did you use it last

week…” Ask “When you used it the most how

often did you use…”

Make questions do multiple duty. "How often" also tells yes, no, + lots more

Underlying constructs

Think of an underlying scale Try to avoid yes or no Use common or natural intervals Not…

Often…Frequently…sometimes…once…never

But…1/day…1/week…1/month…1/term…1/yr

With a little creativity you can create a log scale from the last one!

Non respondents1. Make a non respondent list.2. Compare against a representative list. We used our LDAP.3. Look for over or under represented groups.4. Over sample them in relation to the population. (make

comparable)5. Then follow up as above. Send e-mail, campus mail, and finally

call.6. Don't use phone or personal calls to get promises of response.7. Either carry out the survey over the phone or go to their office.8. Stop when you have a 'good feeling.’9. Try long enough to get a significant group or collapse trying.10. Finally, include, successful non respondents to check out

some unusual claims.Well... not quite a 12 step program but that's about two weeks of work

in my evaluation course (without the interactive lab to give a feeling for the frustration of tracking down and analyzing results from nonrespondents).

Data entry Enter for analysis and presentation Use different codes for missing, not

applicable, or filled out incorrectly Look out for multiple missing data

codes. Try out a sample set analysis before

it is too late. Do a small amount (25%)of double

entry to get reliability measures.

Make all missing data codes 9999. Data entry persons like 9, 99, 999 But you then can't do a simple search and replace for missing data.

Data analysis

Look out if you average. Medians may be better.

Report stats sparingly. But look for variation. Look for out of range. (stats can do) Simple stats, and complex ones

(factor analysis, MDS, etc)

Outlying data can really move averages. Also don't be afraid of advanced stats. New computer programs can help you visualize complex data. Try StatView for some real eye openers.

Presentation

Minimize tables for comparison Use graphs.

http://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm

You could spend hours looking for conclusions…

Same data as a graph…(unmodified Microsoft graph…bleh…We'll fix this later.)

Just converting it to a chart isn't more helpful…

0

5

10

15

20

25

30

UNC

Men

UNC

Total

4yr

Women

daily

2-3/week

1/week

1-3/month

never

Chart junk…

False 3DTerrible background

<-- Lousy ColorsPoor Layout

Worse…No way to see real differences

These errors are caused by Accepting Excel defaults.

With a little work the results-->

are cleanand clear

Tufte would love this one!

Student Computer Ownership

26

3

36

6

27

27

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Recommended (89) Required (36)

Undergraduates (206) MBA (202) EMBA (121)

Decision graphics

Making data clear with graphics and..

Using graphics to help decision makers

Combines complex chart data Uses visual design theory Uses perception theory

Decision Graphics started in the 80's to help parents understandIndividualize Learning Programs for special education students.

From tables to a decision graphic (7 steps) Step one:… Start with a table of data Question 26 Use Tech

for…

What a table to try to figure out! But at least it is sorted by the total ofAlready use, would like very much, would like somewhat.

118323961768284757412961not at all

3720223230383745752796don't knowabout it

275270576845464540251717would likesomewhat

999064492932232327151310would likevery much

513362437123143already use

e-maillist

Coursewebpage

E-discuss

group

AudioVisual

MMPresentation

CompSimulation

Writingtutorials

TutorialsSelfpaced

Textanalysis

DistTeach

StreamingVideo

Question 26 Use Tech for…

Step 2… Convert table to an Excel Graph

I'll never know why MS builds starts with such terrible charts!

0

20

40

60

80

100

120

140

already use would like very much would like somewhat

don't know about it not at all

Step 3: Convert to a bar chart (set legend below the chart) Question 26 Use Tech

for…

0 50 100 150

Streaming Video

Dist Teach

Text analysis

Self paced

Tutorials

Writing tutorials

Comp Simulation

MM Presentation

AV

E-discuss

Course Web Page

E-mail list

already use would like very much would like somewhat don't know about it not at all

Step 4: Change to a stacked bar chart… Question 26 Use Tech

for…

Aha…now that sorting makes sense as you look along the dark blue bars.

0 50 100 150 200 250

Streaming Video

Dist Teach

Text analysis

Self paced

Tutorials

Writing tutorials

Comp Simulation

MM Presentation

AV

E-discuss

Course Web Page

E-mail list

already use would like very much would like somewhat don't know about it not at all

Step 5: Expand bars (100% wide and less space between)…

Question 26 Use Tech for…

0% 20% 40% 60% 80% 100%

Streaming Video

Dist Teach

Text analysis

Self paced

Tutorials

Writing tutorials

Comp Simulation

MM Presentation

AV

E-discuss

Course Web Page

E-mail list

already use would like very much would like somewhat don't know about it not at all

Step 6:Change colors to flow from cool to warm Question 26 Use Tech

for…

0% 20% 40% 60% 80% 100%

Streaming Video

Dist Teach

Text analysis

Self paced

Tutorials

Writing tutorials

Comp Simulation

MM Presentation

AV

E-discuss

Course Web Page

E-mail list

already use would like very much would like somewhat don't know about it not at all

Step 7: What the heck.. Shade those colors… Question 26 Use Tech

for…

Bottom Line… Our faculty want course web pages but no distance teaching!

0% 20% 40% 60% 80% 100%

Streaming Video

Dist Teach

Text analysis

Self paced

Tutorials

Writing tutorials

Comp Simulation

MM Presentation

AV

E-discuss

Course Web Page

E-mail list

already use would like very much would like somewhat don't know about it not at all

Remember the UNC data?(unmodified Microsoft graph)

0

5

10

15

20

25

30

UNC

Men

UNC

Total

4yr

Women

daily

2-3/week

1/week

1-3/month

never

With color metrics…

Conclusion… Not much difference!

0% 20% 40% 60% 80% 100%

UNC Men

UNC Women

4yr Men

4yr Women

daily 2-3/week 1/week 1-3/month never

Ranking versus top three…

Ranking works with few choices.Selecting top three will take care of

ranking 4 choicesSelecting top three is easier to takeResults easier to analyze and display

Easier to take

But the results are revealing…

What method do you like to use to learn technology?

And guess which University just spent big bucks for on-line computer classes?

0 100 200 300 400 500 600

On-line computer classes

Telephone consultants

Computer seminars

Univ computer classes

On the job training

Self taught from Books etc.

Self taught from exploring, experimenting

Help from friends, families, colleagues

Most preferred way of learning 2nd preferred way of learning 3rd preferred way of learning

A little factor analysis

Yep, we've got leading and second wave faculty plus the good old AV types

Oblique Solution Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7 Number

Web editor 0.70 0.01 0.05 0.02 0.00 0.05 0.07 234

Web text/graphics 0.67 0.04 0.02 0.06 0.12 0.10 0.01 296

Software tools for Web 0.56 0.06 0.06 0.08 0.01 0.00 0.13 180

Publishing 0.47 0.22 0.19 0.11 0.05 0.15 0.11 329

Web audio/video 0.42 0.02 0.03 0.06 0.00 0.30 0.10 135

Multimedia 0.31 0.16 0.11 0.12 0.10 0.03 0.21 190

Presentation 0.03 0.68 0.00 0.06 0.06 0.07 0.07 494

Statistics 0.17 0.66 0.05 0.12 0.04 0.09 0.06 306

Spreadsheets 0.04 0.64 0.05 0.10 0.10 0.06 0.07 558

Databases 0.19 0.40 0.10 0.12 0.03 0.06 0.19 327

Comp Proj 0.14 0.36 0.06 0.04 0.02 0.22 0.06 354

3-D modeling 0.03 0.01 0.75 0.01 0.00 0.03 0.02 91

Modeling 0.04 0.11 0.70 0.01 0.02 0.09 0.13 167

High performance computing 0.05 0.17 0.60 0.04 0.12 0.15 0.02 54

Computer anim/graphics 0.15 0.03 0.58 0.06 0.04 0.07 0.08 175

Graphics 0.22 0.37 0.40 0.10 0.03 0.10 0.10 363

Movies/filmstrips 0.04 0.00 0.05 0.78 0.00 0.06 0.00 176

Video tape/Audio tape 0.03 0.10 0.06 0.70 0.09 0.23 0.05 424

Overheads/transparencies 0.00 0.39 0.11 0.54 0.03 0.05 0.19 600

E-mail 0.02 0.14 0.04 0.10 0.82 0.02 0.03 728

Browser 0.12 0.09 0.03 0.17 0.69 0.03 0.00 708

Word processing 0.05 0.19 0.02 0.18 0.58 0.04 0.05 723

Video Conferencing 0.09 0.10 0.03 0.02 0.04 0.71 0.00 94

Web Conferencing 0.23 0.03 0.00 0.03 0.05 0.59 0.11 78

Digital Video 0.04 0.04 0.09 0.05 0.06 0.52 0.15 94

Project Management 0.00 0.05 0.03 0.02 0.03 0.09 0.64 119

Calendar 0.03 0.15 0.09 0.21 0.12 0.10 0.63 321

Courseware 0.23 0.14 0.19 0.30 0.03 0.26 0.44 78

The U of Michigan case study

The 1999 Faculty Survey Distributed in February 1999 1500 faculty, stratified random sample 19 Schools and Colleges 743 responses Results to CIO in August 1999 Released in to the public in March

2000

Survey Categories

Use Resources Support

Q29 Base Academic Unit

Had to use a log scale. Med School rules! (We had one school with more responses than faculty, what a story!)

1 10 100 1000

Medical School

Literature, Science & Arts

College of Engineering

none

Dental School

Library

School of Nursing

School of Business

School of Music

School of Art and Design

School of Natural Resources

School of Social Work

College of Pharmacy

Law School

School of Education

School of Information

School of Public Health

Architect & Urban Planning

Kinesiology

ISR

MERIT

School of Public Policy

Questions 29-35Demographics…

Q30Appointment

Q33Job classification

Q34Tenure

Regular600

Full professor176

Tenure289

Adjunct70

Associate professor127

Non tenure434

Temp50

Assistant professor161

Q31Full or Part

Instructor24

Q 35Gender

Full time640

Lecturer96

Male467

Part time86

Other140

Female280

Question 27Would you use the web for…

Very little use now but… pent up demand for next wave use.

Q10 How often do you use…

Surveyed just as our CourseTools was coming on line. Next survey???

0% 20% 40% 60% 80% 100%

E-mail

Word processing

Browser

Spreadsheet

Presentation

Graphics

Publishing

Web editor

Database

Stats

Calendar

Web software

Multimedia

Modeling

Project Management

Courseware

every day 1-2 week several/month 1/month several/year 1/year never

Question 15 Concerns…

Bottom Line… It's time, reliability and support!

Adding students

<-Faculty1999 UM Faculty Survey

Students ->2000 UMStudent Survey

0 100 200 300 400 500 600

On-line computer classes

Telephone consultants

Computer seminars

Univ computer classes

On the job training

Self taught from Books etc.

Self taught from exploring, experimenting

Help from friends, families, colleagues

Most preferred way of learning 2nd preferred way of learning 3rd preferred way of learning

Not too different, second wave for both?

0 50 100 150 200 250

Telephone consultants

Online computer classes

Computer seminars

On the job training

UM computer classes

Self taught from books

Self taught from exploring

Help from friends

First choice Second choice Third choice

Credits

CIO and SACUACIO StaffITD StaffOIT StaffISR743 faculty members!

Special thanks to:Kati BauerSteven BurdickJose Marie GriffithsGary GatienKaren KostNicole KirgisKathleen McClatcheyEric Rabkin

Flashlight, part of the TLT group

Online database In depth rather than broad Excellent source Part of a broad program

http://www.tltgroup.org/programs/flashlight.html

The Full MontyThe Michigan 1999 Faculty Survey and the 2000 Student Survey Form

Original blank survey (.doc) [Faculty and Student]

and…Initial results (.pdf) [Faculty]

but wait…there’s more…This presentation (.ppt)

are available at www.carat.umich.eduFollow links to projects and scroll to faculty

survey…

Or…if you want a special question answered [email protected]

Thanks for coming. We can help decisions and improve learning and teaching!