Senior Scientist, SINTEF IKT Adjunct Associate Professor ... · •Chapter 11 – Analysing Qualitative data •Book chapter + some examples –Stages of analysis –Grounded theory

Amela Karahasanović

Senior Scientist, SINTEF IKT

Adjunct Associate Professor, DESIGN

amela@sintef.no

www.sintef.no

Sintef IKT, Forskningsveien 1

Plan for today

• Chapter 11 – Analysing Qualitative data

• Book chapter + some examples – Stages of analysis

– Grounded theory

– Content analysis

– Quiz before the break

– Text analysis

– NVivo example

– Multimedia analysis

2

• Do you like my app?

3

• What they are doing when they are using your app?

• What kind of problems they might have?

• What they do in their spare time?

4

• Quantitative data: time, error rate, rankings

5

0

5

10

15

20

25

30

35

40

45

1 3 5 6 7 10 11 12 13 17 18 20 21 23 24 25 26

Unified Library Application - Quit

Unified Library Application

Title Information

ReturnItem

Reserve Title

Lend Item

Insert Title Window

Find Title

Compilation

Subject_id 15 Task 2

Sum of Seconds

Minutes

Visited

• Qualitative data: interview notes, survey responses, video and audio material

• Quantitative versus qualitative data analysis

– Involves human coding (biased, subjectivity)

– Learning by doing and reading the literature/theories

6

14:04 What prohibited you in making the optimal decisions? Definitely, both radar screen was very important, by that I could decide if an airplane was ready for push back. The human/machine interface is important. This update took too long time, maybe 2-3 sec. It should be 1 sec maximum. The interfaces are important, overlapping aircraft symbols, is confusing, the clarification is not clear. My mental work flow will slow down. Another point is that the preview of what is coming next has to be well defined.

• Four controllers – 30 minutes each; question about

decisions, the process in control tower, the tools that were used, the experiment

7

Stages of qualitative analysis

• (1) Information about substance (group of users, their behaviour); identify its major components – Substance: decision making processes of ATM

controllers; components: communication, interface, procedures, airport characteristics

• (2) Study properties of each component and their relations – Characteristics of communication; how it is related

with other components

• (3) Using knowledge on each component to understand the substance

8

Online behaviour of Internet users

• (1) Behaviour influenced by personality education and ICT experience

• (2) Study these factors and relations; theories behind

• (3) Examine how each component influence user behaviour

• Experience and knowledge of the researcher critical for the interpretation

9

Grounded theory

• Not a theory

• Qualitative research method

• Goal – develop theory grounded in a systematically collected and analysed data

• Theory -> Hypothesis -> Study -> Data-> Y/N

• Study ->Data -> Theory

– Several rounds; reverse engineering

10

Remember!

• No pre-formed hypothesis!

• No favorite solutions!

• Creativity and open mind!

• Let the data to lead you!

11

Procedures for grounded theory

• Open coding – Identify phenomena

• Development of the concepts – Group phenomena into concept

• Grouping concepts into categories – Grouping and interpretation

• Formation of a theory – Crate inferential and predictive statements on

phenomena

12

Advantages

• Systematic approach to analysis of qualitative data

• Allows generating theory grounded in data and coding

• One can study data early on

13

Disadvantages

• One can be overwhelmed by details

• Theories might be difficult to evaluate

– Textual data, less strict measures, coding

• Might be biased

• Keep in mind

– Be open-minded and creative

– Listen to data

14

Content analysis

• Systematic and replicable technique for compressing text into categories by coding

• Any technique for making inferences by objectively and systematically identifying specified characteristics of messages

• Semi-quantitative method

15

Content

• Media content

– Books, journals, TV programs, websites, blogs, film, music

• Audience content

– Surveys, questionnaires, interviews, focus groups, diaries, observations

– Text, video, audio

16

Why do we need content?

• Interviews, open-ended questions…

• Evaluating a new version of Word

– Number of errors

– Solution time

– Satisfaction

– > Explanations

– > Suggestions for improvement

– > Group interaction (video)

17

Quiz time

• What are the disadvantages of qualitative analysis?

• What is grounded theory?

• What are the advantages of grounded theory?

• What is content analysis?

• What types of content have you collected in your study?

• How are you going to analyse your content?

18

Before you start content analysis

• Clear definition of the data set – Interviews of 7 Telenor costumers

• Online community – Public, private messages

– Period of time

– Population (how often they visit, do they post any messages)

• Context – Security (government versus game industry)

19

Analysis - coding

• More than word counting

• A priory coding

– Codes from the literature

– Analysis

– Several coders

– Reliability check if coding is consistent

– Work fine for known domains

20

Emergent coding

• Appropriate for new topics

• Several researchers examine the data and develop key coding categories

• Comparison, discussion, common list

• Multiple coders do the coding

• Reliability measures calculated; if ok proceed with the coding; if not go back

21

Identifying coding categories

• Very important as they lead the analysis

• Demanding

• Codes are coming from

– Theoretical framework

– Researchers interpretation (research denoted concepts)

– Participants (in-vivo codes)

22

Theoretical framework

• We start research by literature review and identifying theoretical framework related to our research topic

• Difficulties experienced by senior citizens when using computers – Human capabilities : cognitive, physical,

perceptual

• Taxonomies – Categories of users, tasks, errors

23

Researcher denoted concepts

• Identify patterns, opinions, behaviour in your data -> codes

• Open coding

• "I was looking for 'find' …and it was not there…It was so irritating" -> find, frustration

24

• In vivo-codes – Participants have a good descriptions

– Use it in your coding

– "Curriculum integration" from the one parent's response, name of a TV show in the analysis of QoE

• Building a code structure – Participants express same ideas id different ways

– Code list – nomenclature

– Several levels (different levels of details)

25

Coding the text

• Read the text (watch the video) before start the coding

• Difficult to find anything interesting – too many interesting things

• Procedure:

– Look for specific items

– Ask questions constantly about the data

– Making comparisons constantly at various levels

26

Look for key items

• Some statements have more valuable information – Objectives: computers for education

– Actions: click on

– Outcomes: error message appeared

– Consequences: I stopped using it

– Causes: my old laptop

– Context: I was in bus

– Strategies: I first browse

27

Ask questions about data

• The art of asking questions in a larger context

– Sensitizing questions

• What is happening here? What did the user click? How did she reach www.ifi.uio.no?

– Theoretical questions

• What is relationship between two factors? How does interaction change over time?

28

Making comparisons of data

• Compare instances under different categories – Frequency for different capabilities (physical,

cognitive, perceptual) for elderly

• Compare the results between different groups – Age, background, family support

• Compare to the previous published results – Same/contradictory, related studies

• Computer software – NVivo, Concordance, SPSS TextSmart

29

Ensuring high-quality analysis

• Subjective analysis – Which category? Are they in the same group? Is 'good'

and 'ok' the same?

• Validity – Use of well-established and well-documented

procedures to increase the accuracy of findings; Did we get it right?

• Reliability – Consistency of results; Would other researchers

make the same conclusions based on the same data set?

30

Validity

• Construction of a database with the collected data material: raw data (notes, documents, photos…) results f the analysis

• Also increase reliability

• Data source triangulation

– Interviews, observations, diaries

• Avoid having pet theories

• Consider alternative theories

31

Reliability

• Same word might have different meanings

• Body language, face expressions, drawings might have different meanings

• Large studies -> different coders analyse different data subsets

• Different people should code in the same way

32

• Intra-coder reliability (stability)

– Whether the same coder do the same throughout the whole process; Would he do the same next time? (50% A, 30% B, 20% C)

• Inter-coder reliability (reproducibility; investigator triangulation)

– Whether different coders would do the same? Multiple coders with different backgrounds

33

• To achieve good reliability

– Good coding instructions

– Training

– Test on the limited amount of data

• Reliability measure

% agreement = the number of cases coded the same way by multiple coders/ the total number of cases

34

• Coders can do the same by chance

• Cohen's Kappa (0-1; 0 – coded the same by chance; 1 – perfect reliability

• K=(Pa – Pc)/(1-Pc)

• Pa – percentage of cases on which the coders agree

• Pc – percentage of agreed cases by chance

• More that 60% is satisfactory

35

36

Coded by both coders

Coded by

chance

Expected agreement when the data is coded by chance 0.37*0.39=0.14 K=(Pa – Pc)/(1-Pc) Pa = 0.26 + 0.12 + 0.35 = 0.73 Pc = 0.14 + 0.04 + 0.18 = 0.36 K= 0.58

Subjective versus objective coders

• Subjective/inside coders

– Designed the study, developed the coding scheme, collected the data

– (+) know the literature; know the topic; easier to interpret the data; minimal training

– (-)might be biased and unable to see new patterns, new behaviour

• Objective/outside coders

37

Analysing multimedia content

• Low cost, reach data source

• Not easy to analyse

• Coding is needed – time-consuming, tedious, impractical

• Same principles as for text

• How to make it easier – Select a subset of a data set

– Partially automated approach • Human coders annotate the data; the detector used to

automatically annotate the rest

38