GLAD NSS RESEARCH PROJECT ‘I can’t believe it’s not better’: The Paradox of NSS scores for Art & Design David Vaughan and Mantz Yorke This research was.

GLAD NSS RESEARCH PROJECT

‘I can’t believe it’s not better’: The Paradox of NSS scores for Art & Design

David Vaughan and Mantz Yorke

This research was sponsored by the ADM Subject Centre of the Higher Education Academy and by the HEAD Trust

Aim to help colleagues to:

• Better understand why the subjects receive the ratings on the NSS that they do; and

• Identify how best they can respond to expectations and the enhancement of the student experience

Detailed data from the 2007 NSS:- broadest JACS grouping of 19 subject areas; and - the finest grouping of 107 subjects(+ further analysis of 2008 and 2009 data and student pre-entry qualifications/experience)

Interviews with the seventeen key informants + a focus group to:

- gain an insight into how, and to what effect, institutions have responded to NSS outcomes

- perceptions of the factors affecting student responses to the NSS

The paradox:

- Creative Arts & Design make a substantial contribution to the national economy

- It is a national success story

- Yet when it is viewed through the lens of the NSS, it does not come out well in comparison with other subject areas

Our ‘take’ on the NSS:

It has been validated statistically

However, it is flawed- conceptually (including not suiting all subjects/HEIs equivalently)- technically- it mixes up inputs and outcomes

Crudely:Inputs = T & L; A&F; Academic Support; O & M; Learning ResourcesOutcomes = Personal Development; Overall Satisfaction

Our ‘take’ on the NSS.....

It seems to suffer from a ‘halo effect’- most scales correlate highly with Overall Satisfaction

Is it a valid representation of the students’ overall experience?- the late Malcolm Eley’s research on the Australian CEQ suggests not

The NSS is here to stay (for the time being, at least – but watch out for instruments focusing on student engagement, as in the US and Australia)

So we have to live with it, even if we don’t like it

And we have to admit that even if it’s a gross ‘measure’, we can still get messages from it

A (semi-)permanent fixture

Messages:

Creative Arts & Design perform relatively weakly in the NSS, save in Assessment & Feedback

40%

50%

60%

70%

80%

90%

100%

T&L A&F AcSup O&M LRes PersDev OvSatis

Med/DentAllied MedBio SciVet Sci etcPhys SciMath&Comp SciEngineeringTechnologiesArchit etcSocial studiesLawBus&Admin StMassComm & DLinguistics etcEuro Langs etcEast Langs etcHist&Phil StudCreative A&DEducation

Creative Arts & Design do relatively poorly (NSS 2008)

Percentage agreement

2009 data not available in this form

Surridge (2009, p.18)

Reference subject area: Law

JACS subject area T&L A&F AcSup O&M LRes PDev OvSatis

Medicine and Dentistry +/ns/ns -/-/- -/-/- -/-/- +/+/+ +/+/+ ns/-/-

Subjects Allied to Medicine ns/ns/+ ns/ns/ns +/+/+ -/-/- +/ns/+ +/+/+ ns/ns/ns

Biological Science ns/-/ns +/ns/ns +/+/+ +/+/+ ns/ns+ -/-/- ns/ns/ns

Veterinary Science ns/ns/ns -/-/- ns/ns/ns -/-/- ns/ns/ns ns/ns/ns -/-/-

Agriculture and Related -/-/- ns/ns/ns +/ns/+ -/-/- -/ns- -/-/- -/-/-

Physical Sciences +/ns/ns +/+/+ +/+/+ ns/ns/ns +/+/+ +/ns/ns +/+/ns

Mathematical Science -/-/- +/+/+ +/+/+ +/ns/+ +/+/+ -/-/- ns/ns/ns

Computer Science -/-/- -/-/- +/ns/ns -/-/- +/+/+ -/-/- -/-/-

Engineering and Technology -/-/- ns/ns/- +/+ns -/-/- +/+/+ -/-/- -/-/-

Architecture, Building and Planning -/-/- ns/-/- ns/ns/ns -/-/- ns/ns/- ns/ns/- -/-/-

Social Studies -/-/- +/+/ns ns/ns/ns -/-/- -/-/- -/-/- -/-/-

Business and Administration -/-/- ns/+/ns ns/ns/ns -/-/- ns/+/+ ns/ns/ns -/-/-

Mass Communication -/-/- ns/ns/- ns/ns/ns -/-/- ns/ns/ns -/-/- -/-/-

Languages +/+/+ +/+/+ +/+/+ ns/ns/ns -/-/- ns/ns/- +/ns/ns

Historical and Philosophical Studies +/+/+ +/+/+ +/+/+ +/+/+ -/-/- ns/ns/ns +/+/+

Creative Arts and Design -/-/- +/ns/ns ns/-/- -/-/- -/-/- -/-/- -/-/-

Education -/-/- +/+/+ ns/+/ns -/-/- -/-/- +/+/+ -/-/-

Combined ns/ns/ns +/ns/ns +/ns/ns ns/ns/ns ns/ns/ns -/-/- ns/ns/ns

Two subjects -/-/- +/+/+ +/ns/ns -/-/- -/-/ns -/-/- -/-/-

Three or more subjects -/-/- +/+ns +/ns/ns -/-/- ns/-/ns ns/-/- -/-/-

Three strikes…

3 yrs above av 3 yrs below av

Messages.......

Creative Arts & Design perform relatively weakly in the NSS, save in Assessment & Feedback

Institutions offering major Creative Arts & Design programmes vary widely in their NSS scores (as will be seen later)

There may be internal explanations for why the data are as they are, but these won’t be appreciated by the general public

Disaggregating to JACS 107 subjects:

Percentage agreement T&L 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement A&F 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement AcSup 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement O&M 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement LRes 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement PerDev 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Percentage agreement OvSatis 2007, selected subjects (JACS 107)

0

20

40

60

80

100

Disaggregating to institutions with studio-based provision:

Overall Satisfaction, Creative Arts & Design 2007

T&L Asst & Fdbk

Org & ManAcad Sup

Learn Res Pers Dev

Regarding Learning Resources, Creative Arts & Design seem to do a little less well than laboratory-based subjects

The vertical bars here represent the range of % agreement in the various HEIs

Interview survey:

- NSS clearly here to stay - results taken seriously

- Results a “wake up call”

- Some staff rested on laurels - “we are better than others so we don’t have a problem”

- None complacent or defensive

- More a sense of frustration

- Majority of results at best weakly positive • esp. Organisation & Management and Assessment &

Feedback

- Despite actions such results continue

Consideration of NSS:

- Now part of annual monitoring - alongside other evidence

- Detailed action planning

- Internal surveys in advance or alongside - same or similar questions

• often similar messages but NSS scoring/comments generally harsher

• stronger pressure for action from external NSS survey?

Actions taken by institutions:

- Emphasis on actions to enhance student experience

- Communicate actions better through dialogue

- Regular feedback through websites - focus groups - meetings

e.g. student reps taking “tea with the Dean”

- Poster campaigns - “You said it – We did it”

- Realistic marketing/prospectus information:

• what is promised is delivered

• student expectations realistically informed

• student support services reviewed - increased or more focused provision

Underlying NSS concerns:

- How students in Creative Arts interpret questions • view that the pedagogy of Creative Arts & Design does

not sit well with questions • more for highly timetabled lecture-based subjects

Underlying NSS concerns.....

- Questions not designed to draw out • how personal development supported• whether a well-rounded educational experience received 

- Important to reflect on the student experience based Creative Arts pedagogy:

• to improve our understanding of it • our ability to explain it

Organisation & Management

- Data since 2005 - consistently lower NSS ratings than other subject areas:

• apparent that the drivers behind the data are complex

• range of institutional factors have a bearing - institutional structures - major or minor changes in the physical environment

- how Creative Arts are located or perceived within the institution

Interview survey comments include:

Often a direct correlation between good course leadership and good NSS scores and vice versa

Importance of better communication with students: – “it’s not rocket science to do things like alerting

students to the cancellation of a lecture or laying down expectations about the availability of part time teaching staff”

– recognised students often confused about who to go to for information

– increased emphasis on information/course administrative services

Interview survey comments include.....

Greater emphasis on management of student (and staff) expectations– “what’s the deal?”– what students can expect of institution– what institution should expect of student

– some revisiting or resurrecting Student Charters

Interview survey concerns included.....

- creative arts much more complex - self-determined and negotiated - than highly timetabled lecture-based courses

- much more forward planning and negotiation required to procure specialist resources to complete a project

Therefore NSS question:

“I have been able to access specialist equipment when I needed to”

positive reaction less likely - equipment had to be booked into/queued for - (in contrast to having it available on demand)

Interview survey concerns included.....

Pedagogy of Creative Arts could mean that:

“the more a student has creative freedom the more they are likely to experience what they believe to be a failure of Organisation & Management”. 

NSS question:

“the timetable works effectively as far as my activities are concerned”

could lead to negative comment that:

“the timetable was not effective as it got in the way of their self-determined creative project work”

Many other factors affecting Organisation & Management

but often returned to was the pedagogy of Creative Arts– our understanding of it – our ability to explain it to our students (and to others)

Assessment & Feedback

Assessment & Feedback has consistently attracted low ratings across the whole of HE 

Survey comments suggest feedback more of an issue than summative assessment in Creative Arts.

Actions taken include:

Much work undertaken to raise awareness of students and staff:  – production of assessment handbooks highlighting

Assessment & Feedback – discussions to explain formative and summative

feedback and encourage engagement 

Despite initiatives very little impact on the NSS results:  – do staff - as well as students - fully understand

assessment and feedback models in use? – can we assume accepted models work adequately?

Other questions raised:

– do students really understand learning outcomes/assessment criteria - how they are used?

– are statements in handbooks sufficient without further exemplification?  

– in our efforts to make things more explicit “have we made everything over-complex and difficult to understand?”

Other questions raised.....

Close working relationship between staff and students can also lead to misunderstandings:

– “I thought you said I was doing OK (feedback) but now I’ve not got the marks I expected (summative assessment)”;

– formative feedback given in passing often challenging/intended to stimulate thinking but can even be seen as an instruction

Other questions raised.....

Feedback often not recognised or understood: – “the only form of feedback students in Art & Design

recognise is that tailored to them as individuals - what do I have to do to get a better mark (i.e. 1st or 2.1)?”

– “Our students wouldn’t recognise they were receiving feedback even if we all went round wearing T-shirts that said You are now getting feedback”

Some reflections...NSS results being viewed and considered very seriously

There appear no general problems with student recruitment or commitment to courses in Creative Arts & Design

Memorable experiences – good or bad – often recent - are likely to have a significant effect on NSS scores and comments

Issues that seem to exercise students the most are:– the stimulation of the teaching– engagement of, and contact with, academic staff– feedback (possibly a need to manage perceptions)– Organisation & Management of programmes– access to specialist resources/equipment

So... Why the paradox? Who’s at fault? Fact or anecdote?

Special pleading will not be taken seriously

Why are graduate surveys often much more positive?

Pedagogy of HE Creative Arts & Design:- Do we take it for granted?

- Do we explain it?

- Do we understand it?

- Do we do enough to prepare our students for HE study?

- Do we know what our students expect/expected? - We probably need to ask them but not through the NSS

The full report is available on the HEA-ADM website:

http://www.adm.heacademy.ac.uk/projects/adm-hea-projects/national-student-survey-nss-project