Assessing adult literacy The aim, use and benefits of standardized screening tools

Assessing adult literacy

The aim, use and benefits of standardized screening tools

"You teach a child to read, and he or her will be able to pass a literacy test."

1. Research questions

2. Tools and methods

3. Why screen? Why not?

4. What to expect from screening?

Presentation

Screening tools

What is the aim, use and benefits of screening tools in the process of identifying low literate adults?

Standardized screening tools

Practical usability

Broad and standardized screening instrument desirable?

Research questions

Policy on literacy

Flemish government

Increase of literacy in the Flemish population

Policy documents• Strategic plan on “Increasing literacy”• Operational plan on “Increasing literacy”

Objective: phased and systematic screening of literacy among adults

Tools and methodsThree qualitative research methods

A literature survey of literacy skills and the screening of those skills

A document analysis (qualitative content analysis) of 31 existing screening devices

Semi-structured in-depth interviews with 33 key figures from Flanders and The Netherlands

• Low literate individuals and representatives• Screening professionals• Policymakers• Academic experts• Professionals working in secondary

education

What is screening?

Literacy screening implies …

• … based on behaviour or performance which may or may not be induced

• … the literacy skills of an individual or group

• … are assessed and evaluated • … using a benchmark or norm• (in a short protocol)

Screening: how?

Five types of screening instruments

Test – measuring an induced literacy performance using a device developed beforehand (which may or may not be standardized)

Proxy measurement - mapping out factors that show high correlation with low literacy

Self-assessment – making an estimate of one’s own literacy performance level on the basis of structured questions

Interview / discussion – oral questioning of the extent of literacy on the basis of a questionnaire

Observation – consciously observing behaviour with a view to describing and estimating literacy skills

Screening: why (not)?

To strengthen adult literacy, the first step is the indentification of low-literacy (on an individual level).

Still… not al social domains (civil society organisations, health care, work place organisations, …) seem equally open to assessment by means of a standardized screening tool.

Reasons The interpretation and operationalisation of literacy itself

The adequate functionality norm for literacy (and its effects)

The very limited diagnostic information (and what to do with it?)

Multi-literacies

Multiple viewpoints

Level of literacy performance (“the degree of literacy” of an individual)

• = vertical dimension

The range of contexts and situations in which an individual can function using written language

• = … horizontal dimension

● Different user perspectives (micro, meso, macro)

● Different sorts of literacy, different sorts of text (prose literacy, document literacy, numeracy, digital literacy, …)

Multi-literacies

Multi-literacies

“Literacy as a cover term is so broad it must almost be defined for each occasion on which it is used.”

(Kintgen et al, 1988)

“(…) there are no clean cut logical or empirical criteria that can help settle disputes about what functional

literacy is or is not. These facts should not be taken as a sign of inadequacy of the definition. The

definiendum itself is a fuzzy reality and it should not be presented as if it were not.”

(De Glopper, 1992)

Multi-literacies

Consequences for screening

It seems impossible to screen all aspects of literacy…

• together• to the same extent• in the timeframe of a quick screening

protocol

Content analysis of existing screening devices: no single instrument focuses on literacy as a whole… they test separate skills (writing, listening, …) or one sort of literacy (often prose and document literacy)

The cut-off point (1)The cut-off / norm?

When is a person or group functionally literate? Where is the boundary between having and not having

adequate literacy skills (the ‘at risk’ line)

Should we use one?

From a theoretical point of view: Yes – how can we evaluate literacy skills when there

is no norm? It is necessary to identify the ‘at risk’ group opposed to a ‘not at risk’ group

But no – it can never do justice to the complexity of literacy

One or more? One - for the totality of the population More than one – for each subgroup or population

category

The cut-off point (2)The cut-off / norm?

Who should define or specify the cut-off criteria?

Intense societal or public debate International comparison … or the screened individual?

Practical consequences

Too high – a large population “at risk”… targeted policy is impossible… over-problematisation / people talked into believing they have a problem

Too low… a wrongful acceptance of a lack of minimum skills

! Screened individual has to “accept” the cut-off point…

Screening: informationWhat information to expect from screening?

Realism has to rule Only a cursory and generalised portrait of performance…

(illusory effect) Offers no evidence or knowledge of the interplay between

literacy skills and experiences, no diagnostic information

But… … this makes explicit what otherwise would stay unnoticed

or intuitive

Important For most interviewees, screening is just a first step that

should not take place without a possible… Diagnosis Follow-up Training

Screening: expectations

Because research data warn for unrealistic expectations…

Any (new) screening instrument should

Critical success factors … exhibit several essential features

= demonstrate adequate levels of validity, reliability= it should be neutral, fair, efficient

… should be guided by the particularities of the target group A well-defined target group is necessary A group with a uniform needs profile A group that can be reached for screening (for

instance: schools, providers of vocational training courses, …)

The use of a screening tool

Perhaps the most important element: not the screening tool itself but the way it is used…

screening should be

Be part of a formative process

Screening results should motivate the screened individual to embark on an educational process

Start from the needs of the person

Aim: strengthen adult literacy

… Not simply counting heads or labelling people ‘at risk’

Assessing adult literacy

The aim, use and benefits of standardized screening tools

Queestions, comments, feedbag, …feedback

Download the research report (in Dutch)

www.hiva.be

www.cteno.be

Document analysis

Quickscan Intaketoets alfabetisering NT2

NT2 toets online

Speuren naar cijfers en letters ItemDito Nivor-toetsen lezen en schrijven

Domino 2 Diss NT2 profieltoets

Domino 3 Tool Staatsexamen NT2

Domino 4 NT2 Profieltoets alfabetisering

Centraal examen

Tibo - toets instroom beroepsopleiding

Naturalisatietoets Spel-direct

Voorbeeldtoetstaken NT2 – 1.2 Taal- en rekentoets voor reïntegratie

Staal. Schriftelijke taalvaardigheid…

Voorbeeldtoetstaken

NT 2 – 1.1

Taal en rekenen Instaptoets 1.2-2.1

Digibo 3.0 Voortgangstoetsen Havo en vwo

Intaketoets NT2 Digitale schrijftoetsen VMBO