Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of 112 Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of.

Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of 112Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of 110

PLANNING FUNDABLE GRANT PROJECTS

Danielle Cunniff Plumer, Ph.D., M.S.I.S.

dcplumer associates

[email protected]

Ann Hodges, M.L.I.S., C.A.

University of Texas at Arlington

[email protected]

Planning Fundable Grant Projects

mailto:[email protected]



Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 2 of 112

Course History

• Based in part on “Digital Project Planning and Management Basics,” part of the Library of Congress’ Catalogers Learning Workshop Series • Original design, April 2008

• Mary S. Woodley, CSU Northridge

• Revised June 2009, April 2011• Texas State Library and Archives Commission &

Amigos Library Services• To the extent possible under law, the Texas State Library and Archives

Commission and Amigos Library Services have waived all copyright and related or neighboring rights to this work. Copyright in quoted material is not affected by this waiver.



Grant Components• Needs Statement

• What is the need?

• What data demonstrates that need?

• Project Plan• What is the project idea?

• How does it address the need?

• What are the project goals and objectives?

• Project Timeline and Budget• What resources (time, money, staff) do you need?

• What are the project activities and when do they happen?

• Project Evaluation• How will you know your project was successful?



Workshop Outline

• Statement of Need

• Project Planning

• Budgets and Funding

• Project Management

• Evaluation and Assessment



Workshop Objectives

• Understand the steps used in planning projects

• Learn the skills needed to develop a project budget and funding strategy

• Identify the elements of project management and how they can be used in grant-funded projects

• Define criteria for assessment and selection of tools to measure project success



Class Introductions

• Identify• Yourself

• Institution

• Types of projects you have worked on or are interested in

• Your role in these projects




Section 1:

Statement of Need



Laws of Grantwriting

1. Know who you are and what you want to accomplish

2. Know what needs to be done and how you want to achieve it

3. Know your funding source

You + Funder



Developing Fundable Grant Projects

• Start with a good idea

• Poor ideas will not be successful regardless of how well you write the grant

• Sell your idea• Good ideas are often not funded because you didn’t

explain it well or you didn’t follow the instruction

• Repeat as needed• Many good projects are not funded on their first

application



Grant Development Process• Develop an idea, then match the idea to a funder

• Study the funder’s instructions

• Develop a timeline for proposal preparation

• Understand evaluation criteria and scoring

• Understand review process and who the reviewers are likely to be (individual scoring leads to group scoring)

• Get input on proposal before submission – from colleagues (an expert in the subject, a non-expert, and a good writer) and from program officer

• Allow time for your organization’s bureaucracy

• Meet the submission deadline



Needs Statement

• Interesting to read and accurate

• Related to purposes and goals of your organization

• Supported by evidence (data)

• Realistic – issue/situation you can really do something about over the course of the grant

• Stated in terms of the target audience (clients or beneficiaries)



Targeting the Audience• Characteristics of the target audience:

• Population size

• Language skills

• Disabilities

• Education level

• Geographic area

• Summarize the similarities and differences between the target audience and the broader population your organization serves



Wants and Needs

• Wants• Wishes or desires of the target audience(s) that may be

taken into consideration in determining the most appropriate solution or solutions to meet an identified need

• Needs• Gap between the desired results, on the one hand, and

the current condition of the target audience, product, or service, on the other



Identifying Needs

• Target audience says:• We want you to use social media to promote services

• We want to see more of your collections online

• We want your collections to be around forever



Actual Need

• Target audience says:• We want you to use social media to promote services

• NEED: Portions of the target audience are not aware of services and collections

• We want to see more of your collections online

• NEED: Portions of the target audience are not able to use collections in person

• We want you to preserve collections into the future

• NEED: Collections are at risk due to environmental, institutional, or format-related factors



Scope and Significance

• Many grants include instructions that the grant have state or national significance• Significance can include:

• Methods of the grant (replicability)

• Target audience: peer institutions

• Content of the grant (materials to be digitized, processed, etc.)

• Target audience:

• K-12 (look at Common Core & state standards)

• Special populations (genealogists, higher education students, faculty)




Section 2:

Project Planning



Project Management

• Project management is the discipline of planning, organizing and managing resources to bring about the successful completion of specific project goals and objectives.

• "Project Management." Wikipedia. Available from http://en.wikipedia.org/wiki/Project_management. Internet; accessed 7 July 2009.


http://en.wikipedia.org/wiki/Project_management


What is a project?

• A finite endeavor• having specific start and completion dates

• undertaken to meet particular goals and objectives

• usually to bring about beneficial change or added value

• "Project Management." Wikipedia. Available from http://en.wikipedia.org/wiki/Project_management. Internet; accessed 7 July 2009.


http://en.wikipedia.org/wiki/Project_management


People and Projects

• Project managers may or may not be people managers

• Key project management skills:• Negotiation

• Listening and facilitation

• Time management

• Leadership



Types of Projects

• Collection Arrangement and Description

• Finding Aid Conversion

• Digitization

• System Implementation or Migration• Institutional Repository

• Discovery Layer

• Archival Management System

• Library Catalog



Defining Projects

• Why you are undertaking the project

• What you want the project to achieve

• For whom you are undertaking the project

• When you will achieve it

• How you will achieve it

• Institute of Museum and Library Services. "NLG Project Planning: A Tutorial." Available from http://www.imls.gov/project_planning/index1.asp. Internet; accessed 7 July 2009.


http://www.imls.gov/project_planning/index1.asp


Defining Projects: Why?

• Why do we want to do this?• What are our institution’s needs?

• What are our institutional priorities?

• Who decides the priorities?

• What do we want to achieve?• How will we know when we’ve succeeded?



Defining Projects: What?

• What are we trying to accomplish?• How does this relate to our organizational mission?

• What strategic goals will this project support?

• What have we already accomplished?• What activities of the organization can be redirected

towards the digital project?



Defining Projects: For whom?

• Who is our target audience?• Primary audience

• Secondary audience

• How are their needs being met?

• Where are the gaps in service?

• What skill, knowledge, or behavior can be improved?



Defining Projects: Constraints• Scope

• What is the total amount of work required to complete the project?

• Time• What is the timeline?

• When will the project start? When will it end?

• Cost• What resources are available?

• To develop the project

• To sustain the products



Exercise: Defining Projects

• Define your project• See case studies in student manual for examples

(or develop your own)

You will be working with this project for the remainder of this workshop, so try to identify a project that will be meaningful to you.



Steps in the Planning Process

• Identify the key players

• Conduct an “environmental scan”

• Know your users• Needs and Gap Analysis

• Create a planning team who will be responsible for researching, planning, and managing the project



Stakeholders

• Any individual, group, or organization that influences your project or is affected by it• External: Partners, boards, funders, consultants, target

audiences or users

• Internal: Management, team members, other staff



Stakeholder Roles

• Project Director

• Task Performers

• Project Sponsor

• Subject Matter Experts

• User Representatives



Steering Committee

• Group composed of key decision makers

• In a partnership project, often involves one representative from each of the partners• Collaborative decision-making

• Agreements define responsibilities



Advisory Board

• Group composed of influential outsiders• Provides key skills or knowledge that the project team

may not have

• Lend reputation to project

• Increases trust in project outcomes



Communication Strategy

• Stakeholders want to know what is happening in your project

• Communication tools:• Meetings

• Presentations

• Email lists

• Blogs

• Social Media



The Environmental Scan• Gather information

• Look both inside and outside the organization

• Scan for both content and context

• Use both formal and informal information

• Analyze and interpret this information

• Apply this analyzed information in decision making



Formal Information Sources

• US Census statistics

• Institutional statistics • Size and scope

• Visitation statistics

• Customer service statistics

• Website analytics and usage logs

• Published studies

• Surveys and focus groups



Formal Information Sources• National Center for Educational Statistics

• Academic Library Peer Comparison Toolhttps://nces.ed.gov/surveys/libraries/compare/Default.aspx

• Integrated Postsecondary Education Data System (IPEDS) Data Centerhttp://nces.ed.gov/ipeds/datacenter/

• Public Libraries• IMLS Public Libraries Survey (PLS)

http://www.imls.gov/research/public_libraries_in_the_united_states_survey.aspx

• Non-Profit Organizations• Charity Navigator

http://www.charitynavigator.org/

• GuideStarhttp://www.guidestar.org/


https://nces.ed.gov/surveys/libraries/compare/Default.aspx

http://nces.ed.gov/ipeds/datacenter/



http://www.charitynavigator.org/

http://www.guidestar.org/


Formal Information Sources

• Research and Innovation • Center for the Future of Museums

• http://www.aam-us.org/resources/center-for-the-future-of-museums

• Center for the Future of Libraries

• Ithaka S+R

• http://www.sr.ithaka.org/research-publications

• New Media Consortium

• Horizon Report: http://www.nmc.org/horizon-project

• Pew Research Center

• Internet Project: http://www.pewinternet.org/


http://www.aam-us.org/resources/center-for-the-future-of-museums



http://www.sr.ithaka.org/research-publications

http://www.sr.ithaka.org/research-publications

http://www.nmc.org/horizon-project

http://www.pewinternet.org/

http://www.pewinternet.org/


Informal Information Sources

• “Usual, Reasonable, and Customary” Standard• Anecdotes and observations from peers

• Email lists and discussions in field

• Popular culture• Blogs & Internet sources

• Television

• Film



Organizational SWOT

Strengthsof the organization

helpful to achieving the goal

Weaknessesof the organization

harmful to achieving the goal

Threatsin the environment

harmful to achieving the goal

Opportunitiesin the environment

helpful to achieving the goal



Sample SWOT

Strengths Qualified professional staff Existing set of print finding aids

Weaknesses IT department controls website No budget for software to

create XML

Opportunities ArchivesSpace hosting

available Funding available from

collection donor

Threats New version of standard due to

be released

Objective: Implement EAD at our institution



Outputs and Outcomes• Outcome:

• Changes or gains in knowledge, skills, attitudes, behaviors, status, or life condition that you would like the target audience to achieve

• Output:• A measure of the amount, quality, or volume of use of

the service or product that you want the project to achieve



Identifying OutcomesType of Outcome Definition Example

Knowledge What someone knows

Students will increase their knowledge and understanding of history

Skill What someone can do

Students and teachers will be able to search a database of digitized objects

Attitude What someone feels or thinks about something

Students will demonstrate increased interest in history

Behavior How someone acts High school social studies teachers will find, analyze, and use digitized primary source materials in their lesson plans

Status Someone's social or professional condition

The number of high school graduates in the community will increase

Diffi

cult

y o

f m

easu

rem

ent

incr

ease

s



Output or Outcome?

•Students will have an increased knowledge of and appreciation of history.

Output OutcomeOutcome



Output or Outcome?

•The project will digitize approximately 11,500 assets related to exhibits, educational activities, and scientific phenomena.

Output Outcome



Output or Outcome?

•High school social studies teachers will be able to incorporate digital primary source materials into a lesson plan.

Output Outcome



Output or Outcome?

•We will develop an online forum where teachers can discuss primary sources and lesson plans.

Output Outcome



Exercise: Outputs & Outcomes

• Identify 2-4 output measures for the services or products developed by your project

• Identify 2-4 outcomes for your target audience



Checking the Plan• Will the project achieve the desired outcome(s) for your target audience(s)?

• Are the project goals achievable, given the opportunities and constraints your organization may face in the near future?

• Do the key stakeholders support the decisions that have been made?

• Remember: • “No plan survives first contact with the enemy!”

--Helmuth von Moltke the Elder, Militarische Werke, 1897




Section 3:

Budgets and Funding



Budgets and Funding

•Objective• Learn the skills needed to develop a project budget and funding strategy



Project-Based Budgeting• Assumes that most costs in an organization can be assigned to a given project• Indirect costs make up a given % of budget

• Sum of the budgets for all projects in the organization total the organization budget



Factors to Consider• Every project is unique; costs will vary depending on:• Intended results of the project

• Staff and equipment costs

• Application development

• Data preservation and migration is an ongoing, rather than a once-in-a-lifetime activity



Categories of Cost

• Staffing• Salary plus fringe benefits

• Services• Outsourcing

• Consultants

• Operational expenses• Equipment and Supplies

• Indirect (Facilities and Maintenance)

• Each cost category includes both grant & cost share, whether cash or in-kind



Budgets - Definitions

What is cost sharing (matching funds), you ask?

Newsflash

Definitions on various federal websites often are not very helpful if you don’t already understand the concept being “defined”!

Thus, the following definitions are adapted from several sources.



Cost Share

• “that portion of the project costs that is not charged to IMLS funds. These costs may be supported by your cash outlays; contribution of property and services; and in-kind contributions…”

• Cost share is often required by a funder to help support the project and to ensure that the organization requesting funding has a financial stake in the project.

• Cost share may be referred to as matching, but matching funds can also come from a third party.



In-Kind• Goods, commodities, or services (rather than money)

that help support a project. • Examples are the value of staff time, donated volunteer time,

and donated consultant time that support project activities, and equipment donated to the project.

• In-kind contributions must be necessary to accomplish project objectives, allowable according to funder guidelines, auditable, and incurred during the grant period.

• Cost-share by the applicant in a request for funding usually takes the form of in-kind contributions. (adapted from IMLS)



Direct Costs• Include:

• Faculty, technical and student salary

• Tuition

• Travel

• Supplies & equipment

• Services

• Consultants

• Other costs

• Human subject incentives

• Animal costs

• Direct costs can be assigned to a specific project with a high degree of accuracy. (Source = ASU)



Indirect Costs• Costs that cannot readily be isolated or identified with just one

project or activity. Indirect costs are often referred to as "overhead" or "administrative costs." • Examples are charges for utilities, general insurance, use of office space

and equipment that you own, telephone service, postage, and the salaries of the management and administrative personnel of the organization. (Source = IMLS)

• Watch for wording by funders that they disallow IDC. If instructions are unclear or do allow IDC, ask the granting agency for a statement that it is not allowed.

• F&A: Facilities & Administrative costs, formerly known as indirect costs or overhead.



Note on Indirect (IDC)



Federally Negotiated Rate

• Rate determined by negotiated agreements between federal agencies and non-profit organizations establishing the percentage that can be recovered by the organization receiving an award to reimburse indirect costs.



Staffing• Labor costs (generally) represent the largest percentage of costs in a digital project

• Salary• Full or part-time staff

• Temporary staff hired for project

• Benefits• Range from 20% to 30% of salary



Staffing Costs

• Salaries

• Benefits • Health

• Sick Leave

• Vacation

• Holidays

• Training

• Attendance at conferences and meetings



Total Compensation Package



Staff Model – Complex

Assistant Dean

Digital ProjectsUnit Head

ProjectDevelopment Coordinator

DepartmentalAdministrator

Digitization Lab Manager

ApplicationDevelopers (3)

Project Coordinators (2)

MetadataSpecialists (2)

Imaging Technicians (2)

StudentAssistants



Staff Model – Simple

Manager

Curator ofCollections

EducationCurator

AdministrativeAssistant

ImagingIntern

TranscriptionVolunteers

EducationAssistant



Project Based Budget – StaffPosition Annual

SalaryProject Responsibility % of

Time

Director $50,000 Administration and reporting

5%

Curator of Collections

$37,500 Oversees project intern and volunteers; performs quality control

30%

Imaging Intern $12,000 Grant-funded position. Does day-to-day scanning tasks

100%



Equipment: Hardware• Desktop computers

• Scanners

• Digital cameras

• Audio/video conversion equipment

• Server for storage/delivery

• Server for streaming audio/video

• Long-term maintenance/replacement



Sample Equipment Costs• Scanner

• $600 (Epson V700 flatbed)

• $3,000 (Epson 10000XL large-format flatbed)

• $25,000 (BookEye 3 planetary scanner)

• Digital cameras

• $1,000 (Canon XSi, plus upgraded lens)

• $5,500 (Canon Eos 5D Mark II, plus upgraded lens)

• $2,000 (semi-pro copy stand system with lights)



Equipment: Software• In-house application development:

• Requires skilled programmers

• How and by whom will the system be updated, enhanced, and maintained?

• Purchase of an off-the-shelf product:• Procurement processes must be followed

• Are ongoing maintenance costs and upgrades included?

• Software-as-a-Service model



Outsourcing Services• Software development

• Database• Websites

• Software-as-a-Service (SaaS)• Digitization

• Large format• Audio, video, microfilm

• Transcription • Specialty markup



Outsourcing Decisions

Can work be performed in-house with existing staff and equipment?

Is the cost per item with existing staff and equipment

less than the cost per item for outsourcing, quality being equal?

Can the equipment purchase be justified over a 3-5 year

period based on the number of items?

(Cost-benefit analysis)

Will doing the work in-house result in improved quality or reduced costs for future projects?

In-house Outsource

yes

yesno

no

yes no

yes no



Proposed Project Budget

• Salaries & Benefits

• Travel

• Capital Equipment

• Operational Expenses• Supplies & materials

• Outsourcing

• Consultants

• Educational (Student) Expenses

• Indirect Costs



Sample Budget


Project Archivist

Project Manager Project Director Total NHPRC UTA

Salary FY14 $ 44,000 $ xx,xxx $ xx,xxx

COLA 3% $ 1,320 $ x,xxx $ x,xxx

Salary FY15 $ 45,320 $ xx,xxx $ xx,xxx

Effort % on project 100% 50% 5%

Salary on project $ 45,320 $ xx,xxx $ x,xxx

% NHPRC 80.025% 0% 0%

% UTA 19.975% 100% 100%

Project salary NHPRC $ 36,267 0 0 $ 36,267

Project salary UTA $ 9,053 $ xx,xxx $ x,xxx $ xx,xxx

Fringe FY15 $ 16,768 $ x,xxx $ x,xxx

Fringe on project $ 16,768 $ x,xxx $ x,xxx

Project fringe NHPRC $ 13,419 - - $ 13,419

Project fringe UTA $ 3,349 $ x,xxx $ x,xxx $ xx,xxx

bluesheet sal/fringe subtotals per person $ 12,402 $ xx,xxx $ x,xxx

PC workstation $ 1,200 $ 1,200

XML software $ 300 $ 300

Total Direct Costs $ 99,376

IDC $ 994

Total Project Cost $ 100,369

Cost Share Required (50%) $ 50,185

Direct Cost per organization $ 49,686 $ xx,xxx


Grant Budget Justification

• State the source of the budget numbers provided

• Include any details that will help the grant reviewers understand the budget

• Specific supporting documents may be required• 501(c)(3) exemption letter

• Indirect rate negotiation letter



Tips for Success• Follow the directions in the grant guidelines

• Look at examples of similar projects funded by agency• Take advantage of staff review, if available

• Ensure that proposed budget and timeline are realistic• Check with other institutions in your area to compare

costs• Ask vendors for non-binding quotes• Double-check the math!




Section 4:

Project Management



Project Management

•Objective• Identify the elements of a project management plan and how they can be used in digital projects



Project Lifecycle

Plan Develop

Perform

Sustain?

Evaluate



The Role of Pilot Projects• Scale and outputs of pilot project need to be smaller than in proposed project• Project type and resources should be similar• Can be a subset of the proposed project

• Pilot project activities• Keep careful records of cost and time• Document procedures and processes used in the

project



Project Resources• Assigned to a project by project director

• With agreement of resource managers

• Personnel (e.g., permanent staff, temporary staff and consultants) • Often have a fixed percentage of time assigned to a

particular project

• Materials, equipment, supplies, and special services (e.g., telecommunications)



Project Phases• A phase in a project is a set of logically connected activities that share one or more common milestones

• Often used in software development to mark major or minor releases of the product



Project Activities• An activity is a self-contained element of work performed during the course of a project

• Activity components:• Expected duration

• Expected cost

• Expected resource requirements



Project Tasks• A task is a component of a project activity

• Includes work that could realistically be assigned to or completed by a single project resource

• Can be subdivided into sub-tasks

• Task components:• Expected duration

• Expected cost

• Expected resource requirements



Project Scheduling• Include start and finish dates of each of the activities • Duration: how long, measured in hours or days, it will

take to accomplish the activity

• Relationships among the activities • Dependencies: One activity cannot be started or

completed until another activity is completed

• Resource allocation constraints: Same resource is assigned to two or more activities at once



Project Milestones

•Show accomplishment• Logically related tasks and/or activities• Interim targets• Targets

•Resources are never assigned to milestones



Work Breakdown Structure

• Tree structure• End objective

• Work components

• Terminal elements

• Activities and deliverables

• Must include all elements necessary to achieve the end objective (100% Rule)



Work Breakdown Structure

Goal

Objective #1

Activity A Activity B

Objective #2

Activity C



Grant Design ChartPhase

Activities & Milestones

ActivitiesStart /End

Resources Outcomes Evaluation Budget

Phase I: Goal

Objective 1:

Objective 2:

Objective 3:

Objective 4:



Goal: Increase the amount of primary source information available to meet researcher needs by enhancing access to finding aids for archives and manuscripts collections in the University of Texas at Arlington Libraries Special Collections.

Objectives Tasks/Activities Begin/EndDates Personnel Outcomes (short/mid/long)

Evaluation Budget

Increase online access to 173 finding aids that did not qualify for initial TARO encoding project

A-1: review finding aids and supply missing elements of information

Increase by 173 the number of finding aids available online. Reduce by 173 the backlog of unpublished finding aids at UT Arlington.

A-2: encode finding aids in XML according to DACS and EAD

A-3: publish encoded finding aids on TARO web site

Increase online access to 125 finding aids that failed quality control process of initial TARO encoding project

B-1: review finding aids and correct existing errors

Increase by 125 the number of finding aids available online. Reduce by 125 the backlog of unpublished finding aids at UT Arlington.

B-2: encode finding aids in XML according to DACS and EAD

B-3: publish encoded finding aids on TARO web site


Goal: Increase the amount of primary source information available to meet researcher needs by enhancing access to finding aids for archives and manuscripts collections in the University of Texas at Arlington Libraries Special Collections.

Objectives Tasks/Activities Begin/EndDates Personnel Outcomes (short/mid/long)

Create online access to 173 archival finding aids that did not qualify for initial TARO encoding project

A-1: recruit project archivist (obtain Provost’s approval for position, advertise position, and accept applications)

July 1, 2014-July 25, 2014

McClurkin, Wagner

Pool of candidates is identified from which project archivist will be hired.

A-2: hire project archivist (review applications, interview, make hiring decision, appoint successful candidate)

July 28, 2014-August 15, 2014

McClurkin, Wagner

Essential project member is selected and reports for work.

A-3: train project archivist August 18, 2014-August 22, 2014

Wagner Project Archivist acquires skills and information needed to effectively perform the work required.

A-4: review finding aids and supply missing elements of information

August 25, 2014 - December 31, 2014

project archivist

Completed finding aids are ready for encoding.

A-5: encode finding aids in XML according to DACS and EAD


project archivist

Increase by 173 the number of finding aids encoded.

A-6: review work of project archivist


Wagner Accurate and complete finding aids are ready for publication. Quality of work of project archivist affirmed/any deficiencies in the work of the project archivist addressed.

A-7: transmit encoded finding aids to TARO for publication


Wagner Increase by 173 the number of finding aids available online.


Estimation Challenges• Lack of experience in estimating cost and duration of particular activity• Draw on experience with similar tasks • Draw on expertise of others; get estimates from multiple

sources, including similar projects at other institutions • Develop explicit methods and procedures for cost and

schedule estimates

• Technical glitches• Employ thorough planning and good quality

management practices • Assume the worst in making estimates



More Estimation Challenges• Changes to the project plan

• Conduct a risk assessment that identifies the overall stability of the project environment

• Put change management procedures in place

• Psychological factors • Place skeptics on the cost estimating team; no

individual estimate should be permitted to go unchallenged

• Political pressures • Establish project selection procedures at the

organization level



Exercise: Allocating Resources

• Divide your project into phases (remember that phases do not need to be chronologically sequential).

• Identify 1-4 activities and any needed tasks in each phase, with estimated duration.

• Indicate resources needed (in general terms). You do not need to estimate costs.

• Identify which output or outcome applies to each phase and activity.



Progress Reports• Narrative summary of work accomplished during a given time period• Project activities

• Interim targets and milestones achieved

• Project challenges• Changes to project schedule, deliverables, etc.

• May or may not include a financial report

• Should include any deviations from original grant design




Section 5:

Evaluation and Assessment



Evaluation and Assessment

•Objective• Define criteria for assessment and selection of tools to measure success of a digital project



Outcome-Based Evaluation• The measurement of benefits to people achieved as the result of a project

• Identifies observations that can credibly demonstrate change or desirable conditions

• Systematically collects information about these indicators and uses that information to show the extent to which a program achieved its goals



Types of Assessment

Assessment Strategies

How Much We Do Inputs and outputs: statistics, gate counts, Web use logs, and other measures of quantity and productivity

How Well We Do It Customer satisfaction, quality benchmarks, rankings

How Much We Cost / What We’re Worth

Return on investment and cost:benefit calculations

What Good We Do / Why We Matter

Outcomes measurement, impact assessment

IMLS, Outcome Based Evaluation, “Purposes.” Available at http://www.imls.gov/applicants/overview.shtm



Grant Partners

• Institutional Review Board (IRB)• Anytime anything is to do with human subjects

• Yes, surveys!

• assess improvement in skills

• evaluate claims of improved ease of use



Tasks Inputs Outputs OutcomesImmediate Intermediate Long-Term

Tasks and sub-tasks

Staff timeEquipmentMoney

Items produced by project

Changes produced during project

Changes produced within 1 year of achieving project goal

Changes produced within 5 years of achieving project goal

Digitizing Images

1.5 FTE Technical staff.5 FTE Metadata entry

1 digital library system (including at least 5000 digital objects with metadata)

Students have improved access to primary source materials

Students are able to search for specific primary source materials

Students have increased appreciation of history

Logic Model



Output Statements• Use measurable units

• Examples:• Number and/or percent of items digitized• Number and/or percent of teachers who use materials from the

Web site

• Include desired characteristic refinement• Examples:

• Number and/or percent of items digitized that meet technical standards

• Number and/or percent of teachers from underperforming schools who use materials from the Web site



Sources and Intervals• Specify sources of output data

• Examples• …as measured by the number of images

available in the production Web site• …as determined by a survey of teachers

• Specify frequency or interval of measure• Examples

• Tracked monthly• Performed quarterly



Output Targets

• Target number that you want to achieve at specified points during the project• Example

• By the end of the second quarter of the project, 50% of items from target collection will be digitized and made available on production server, as determined by monthly item inventory



Targets should be SMART

•Specific•Measurable•Achievable•Relevant•Time-based



Outcome Based Approach

• Outcomes are (generally) not directly measurable

• Look for indicators of change• Applied to specific audience

• Measured against a benchmark through data collection

• Quantitative

• Qualitative



Indicators

• What observable evidence of accomplishments, changes or gains can I use to show that the project has achieved the desired outcome? • Who: “The number and/or percent of target audience”

• What: “demonstrate” “produce” “report”

• How much: target amount greater than baseline



Sample Indicators• 80% of teachers from underperforming schools report that they used a minimum of five primary source documents from the project Web site in lesson plans during the last quarter

• 50% of researchers reported that they were able to find a specific document within 5 minutes or less using the search function on the project Web site during the last month.



Benchmarks

• Represent the starting point of a project• Determine what you plan to measure at the onset of the project

• Examples:• How many students and faculty use the archives and special collections for research?

• How many assignments on local history are answered by library resources, and which resources are used?



Quantitative Measures

• Collection statistics• Number of inquiries • Transaction logs

• Number of visits to the sites• Referring urls• IP address of user• Date and time of searches• Number of searches• Types of searches



Qualitative Outcomes

• Qualitative in terms of accessibility, usability, functionality, user satisfaction and expectations• Focus groups• Surveys• Interviews

• Note: check with institution concerning guidelines for using human subjects



Exercise: Evaluation

• Consider what tools you can use to measure success in your project.

• Identify baselines you need to take at the beginning of your project and specific outcome indicators you will track.



Contact Us

• Danielle Cunniff Plumerdcplumer [email protected]

• Ann Hodges, Director of Grants and ResearchUniversity of Texas at Arlington [email protected]


Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of 112 Society of Southwest Archivists, New Orleans, LA, May 28, 2014 Slide 1 of.

Documents

planning projects

new orleans

grantfunded projects

digital project planning

project evaluation

project budget

project idea

grant components