Hosted by: Funded by: Tensions of Measurement: community investment activities in housing associations David Mullins and Vanessa Wilkes, TSRC Housing Studies Association Conference 15 th April 2011
Feb 22, 2016
Hos
ted
by:
Fund
ed b
y:
Tensions of Measurement: community investment activities
in housing associationsDavid Mullins and Vanessa Wilkes, TSRC Housing Studies Association Conference
15th April 2011
Summary• Sector Context• Evidence Base • Levels of Measurement– 5 levels of measurement– Questions and Issues
• Tensions– External Drivers– Internal Drivers
• Conclusions• Further Research
Sector Context • Well established industry, large enterprises
(50% market share in social housing)• Entrenched hybridity (social & commercial) • Core business highly regulated & measured • Non-core ‘community investment’ (CI)
activities symbol of independence • Legitimacy & control issues brought to fore –
2008 Housing & Regeneration Bill & NHF Audit• New tools emerging to measure social impact
of CI activities (in-house & off the shelf)
Evidence Base• Scoping study – emerging approaches to
measurement of social performance (2010)– literature review & checklist (17 tools to set priorities, measure
outcomes or link these to corporate goals)– 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced) – 2 workshops (now 3, plus follow up tracking e.g. of first sector specific
tool)
• NHF Neighbourhood Audits 2008, 2011– Sector overview of CI (64% coverage by stock) – Inputs/Outputs (£435mil CI in 6,800 projects 2008; leverage 1.6:2.7)
• Linked PhD – Impacts & outcomes, depth case studies – More on this later!
The levels of impact measurementOECD definition of impact measurement
“the impact of all changes, positive or negative, attributable to the intervention”
• Also – intended and unintended effects and long-term as well as short-term
Original argument (Zappala and Lyons 2009):• 3 levels of impact measurement
– Whole sector– Individual organisations– Individual projects
• Tools, frameworks and methodologies not easily transferred between levels• Framework developed in relation to housing study (Mullins 2010) to 5 levels
5 levels of impact measurement
1. Sector level
2. Organisational level
3. Team level
4. Project level
5. Individual
Transferability between levels is dependent on:
• How the organisation adopts and uses the tool
• Having the necessary skills and expertise
• Relationship between internal and external data needs
• A corporate awareness of what is required from impact measurement
• The understanding and support by staff
• Maximising the usage of the data at more than one level
Questions and IssuesWhole sector– Is it possible or feasible to measure impact?
• Dependent on collective approach and understanding• Expensive • Housing sector – disparate projects and circumstances
– Draw on information from level 2Organisation / Team / Project– Reason / motivation
• External requirement?• Internal legitimisation / accountability?• A learning tool?
– Availability of skills and expertise• Not necessarily new methodologies but alternative application• Interpretative and subjective decisions
– Draw on information from levels below
External Tensions
External Drivers
• Funding • Sector Promotion• Dances around regulation• Common outcomes /
community budgeting • Wider third sector policy
influences (e.g. promotion of SROI)
Emerging Tensions• Data for Funders
– Conflicts with internal needs– Multiple monitoring– Timescales – Mission Drift
• Sector promotion– ‘more than a good story’– Limited no of PIs obscure
complexities of context & impact
Internal Tensions
Internal Drivers • Accountability & Reporting
(variety of audiences)– Boards & Trustees– Management – Making sure people enjoy the
projects • Benchmarking• Organisational Power Play
– Promoting visibility of CI within corporate scorecards
– Aligning CI activities to corporate goals
Emerging Tensions Commercial & Social Goals• Social Dividend from
commercial success– Tendency toward harder
measures – leverage• Contribute to core business
– Attribution problems – leads to focus of CI on tenants
Corporate v Project Levels– Who sets the goals? – which CI outcomes get
reported?
Conclusions
• Early days but likely to grow• High Expectations in advance of evidence!• Tensions around types of measures and uses
made of the data (both external & internal)• Is it worth it? (practical issues, costs
motivation/training)• Underlying tensions reflect hybrid and large
scale of these organisations & competing interests (e.g. Core business v CI)
Further Research 1. The role of community investment measurement tools
within the organisation– How is community investment integrated and understood?– What measurement tools are used, what is their strategic fit
within the organisation? 2. Data considerations (internal and external)– How is data reported and used?– What level of understanding does it provide?
3. The practical implications of measurement– Does impact measurement cause organisations to rethink
their mission, goals or values? – What are the operational issues and practical considerations
which affect measurement, such as length of contracts