An Evaluation of the Vermont Agency of Natural Resource’s Environmental Indicator Report Series

An Evaluation of theVermont Agency of Natural Resource’sEnvironmental Indicator Report Series

May 26, 2004

Developed at the request of ANR Deputy Secretary, Canute Dalmasse

In partial fulfillment of the requirements of the Vermont Department of Personnel’s

Public Manager Program

Sheila DuranleauScott ShaferKen Valentine

“. . . We will use this year to evaluate the effectiveness of the publication, how it is valued by its audience, and what we can do to improve the report in future years . . .”

• Phase I: Survey report recipients to determine the recognition level and perceived value of the report.

• Phase II: Document the ANR report production process, research

similar efforts of other State Agencies, and compare the outcomes, processes, and associated investments to identify opportunities for improvement.

Do you remember seeing any of these reports? Yes No Have you read one or more? Yes No Was it (or were they) useful? Yes No

If so, for what? (check all that apply) To learn more about the ANR and its mission As an educational resource for teachers or students To learn more about key environmental issues To identify and track environmental indicators and trends For personal interest Other_______________________________________________ If it was not useful, why not?______________________________

Is the content easy to read and understand? Yes No

If the ANR were to discontinue the report, would you miss it? Yes No

What’s the best medium or format to convey this kind of information? Printed Report

(as is) TV

Radio

Newsletter

Web Site

Interactive CD

Newspaper

Other

____________

Other Comments?_________________________________________________________

Can we contact you for further information? Yes No (Name and Phone or e-mail)______________________________________ Thank You!!!

Phase I:• 1086 surveys mailed to “external” report recipients

• 436 surveys mailed to “internal” recipients --- ANR staff

Do you remember seeing any of these reports? 75% Yes 25% No Have you read one or more? 93% Yes 7% No Was it (or were they) useful? 79% Yes 9% No

If so, for what? (check all that apply) 45% To learn more about the ANR and its mission 28% As an educational resource for teachers or students 60% To learn more about key environmental issues 64% To identify and track environmental indicators and trends 46% For personal interest 8% Other_______________________________________________ If it was not useful, why not?______________________________

Is the content easy to read and understand? 91% Yes 0% No

If the ANR were to discontinue the report, would you miss it? 37% Yes 33% No

What’s the best medium or format to convey this kind of information? Printed Report

(as is) 46% TV

10% Radio

7% Newsletter

9% Web Site

36% Interactive CD

4% Newspaper

18% Other

_____3%_____

External Results – 11% return rate

Some Comments from Outside . . .

It's great! Don't discontinue

It's beautiful but, I'm sure, costly

I think it must be cost prohibitive

Save $, put it on the web

Not the best use of limited state dollars

Could be less fancy

I have been in office since 2001 and have not seen one

Do you remember seeing any of these reports? 90% Yes 10% No Have you read one or more? 84% Yes 16% No Was it (or were they) useful? 81% Yes 19% No

If so, for what? (check all that apply) 43% To learn more about the ANR and its mission 20% As an educational resource for teachers or students 43% To learn more about key environmental issues 40% To identify and track environmental indicators and trends 60% For personal interest 13% Other_______________________________________________ If it was not useful, why not?______________________________

Is the content easy to read and understand? 86% Yes 6% No

If the ANR were to discontinue the report, would you miss it? 29% Yes 59% No

What’s the best medium or format to convey this kind of information? Printed Report

(as is) 29% TV 6%

Radio 6%

Newsletter 4%

Web Site 49%

Interactive CD 4%

Newspaper 22%

Other _____4%_____

Internal Results (ANR staff) – 11% return rate

• The printed report is beautifully done. Intent of report needs to be defined

• I've never understood why we do this . . .

• Not enough substance; too much fluff and bragging

• Needs to be succinct . . .

• Waste of paper for those not interested. More people would see it on a website

• Too much paper for an environmental report - walk the walk

Some Comments from ANR staff . . .

Time

Num

ber

of P

ages

Num

ber

Dis

trib

uted

Tot

al H

ours

Est

imat

ed L

abor

*

Pro

duct

ion

& P

rint

ing

Tot

al

Tot

al p

er c

opy

ANR Environmental Indicators Report 20 (color) 10,440 650 11,908$ 17,400$ 29,308$ 2.81$ AOT Performance Report 20 (color) 2,500 380 6,962$ 3,400$ 10,362$ 4.14$ AHS 54 (B&W) 300 500 9,160$ 1,200$ 10,360$ 34.53$ Corrections 170 (B&W) 600 720 13,190$ 10,000$ 23,190$ 38.65$

Costs

Phase II: Comparison with other state agency reports

•Your ANR report is much more polished (and has a more external focus)

•The labor involved is similar

•Total costs are relatively high, but unit costs are low

•Much of the cost is in production, rather than printing (reducing numbers won’t lower costs very much)

•Time and costs to produce this report are reasonable

Strengths

• The report is beautiful and well written

• It goes to the right people (mostly)

• It is read and used

• It’s production is fairly efficient

Gaps

• Changing message conveys a lack of focus

• Many perceive it as an extravagance

• Distribution, Distribution, Distribution

Recommendations

• Focus and standardize the message

• Ensure that the product supports the message

--- Efficiency !!!

• Reinvent the distribution– Biennial production

– Fewer hard copies with increased on-line availability

– Develop and maintain a recipient database

Questions, Comments, Thoughts . . . ?