Welcome! If you joined the conference via telephone, please enter your audio pin if you havent already. (This is an example, your audio pin will be different)

Welcome! If you joined the conference via

telephone, please enter your audio pin if you haven’t already. (This is an example, your audio pin will be different)

If you joined with a microphone and headset or speakers (VoIP), please be sure your device volume settings are properly adjusted.

If your mic & headset is not working, please try unplugging and re-plugging in your device.

If you have any questions, please enter them in the questions module.

We will begin promptly at the top of the hour.

DOC110: Evaluating Document Assembly Projects

September 26, 2008

GoToWebinar InterfaceMaximize/minimize your screen

with the chevron symbol Telephone participants need to

enter their audio pin Please share your questions

and knowledge!◦ Ask a question in the

questions log◦ Raise your hand

PanelistsKate Bladow, Pro Bono NetSteve Rapp, Idaho Legal Aid

ServicesBristow Hardin, Legal Services

Corp.

Today’s TrainingWhat is evaluation?Why evaluate?How do you evaluate a program?

◦ Planning for an evaluation◦ The process◦ Case study: Idaho Legal Aid Services

What is LSC looking for in evaluation plans and reports?

Evaluation DefinedSystematically collecting data to

answer questions of interest.

Reasons to Evaluate a Program

To find out if it was successfulTo build buy-inTo document lessons learned and

potential best practicesTo identify areas for improvementTo justify the cost and effort

Evaluation Process

1.Draft an evaluation plan.2.Design and test data

collection instruments.3.Collect baseline data.4.Collect additional data.5.Analyze data and report

findings.6. Integrate necessary changes.7.Repeat steps.

Draft an Evaluation Plan

Example Evaluation PlanProject Goal: Use A2J author and HotDocs to create guided interviews and court form templates that improve the ability of low income people to easily,

effectively and efficiently produce and file court forms without the assistance of an attorney.

Project Objectives Strategies/Activities Evaluation Questions Evaluation Data

Objective 1Use A2J Author and HotDocs to create guided interviews and court form templates that enable pro se litigants to easily produce accurate pro se pleadings.

Produce [insert number] guided interviews and [insert number] interactive court forms.

Identify forms/interviews to automate Hire forms developer to help develop

guided interviews and automated forms

Work with courts to ensure that interviews/forms are accurate, consistent with court requirements and will be acceptable to the courts.

Conduct usability testing of templates to ensure users (clients and/or advocates) can easily understand and use the interviews/forms. (Note: forms should be at the eight-grade reading level of lower.)

Modify templates to address any problems identified in usability testing

Publish forms/interviews on statewide website

What forms/interviews were selected for automation?

Was developer hired to develop guided interviews and automated forms?

What interviews were produced? What forms were automated? Have courts confirmed that the forms are

accurate, consistent with court requirements and acceptable for use?

What usability testing was conducted? What were the results of the usability testing? What changes were made to the automated

forms based on the initial testing? What automated forms and guided interviews

been published on the statewide website? To what extent have the automated forms and

guided interviews made it easier for self-represented litigants to prepare accurate court pleadings?

Confirmation that developer hired. List of interviews and automated

court forms produced. Court confirmation that forms are

forms are accurate, consistent with court requirements and acceptable for use

Description of usability testing Results of usability testing Description of notable changes

made to forms and interviews following initial testing

List of forms/interviews published on statewide website

User interviews/surveys with:• Users• Court staff

User feedback from the website Field observation data

Objective 2 Increase the availability and use by low-income persons of automated forms

Partner with courts and other stakeholders to develop and implement outreach initiatives to:

• foster courts’ acceptance of the forms

• maximize clients’ use of forms Partner with public libraries,

courthouses and other groups to increase access points where clients can:

• access and complete forms • receive assistance in completing

forms

What automated forms and guided interviews have been posted the statewide website?

Was an outreach plan developed? What outreach activities were conducted? To what extent are the different forms being

used by clients? How many judicial districts/courts have

accepted the automated forms? How many new access points have been

established where clients can access the forms?

At how many access points can clients receive assistance in completing the forms?

Copy of outreach plan Description of outreach activities Statistics from national server on

number of times different forms have been accessed and/or downloaded

List of judicial districts that have accepted the automated forms

List of new access points (with description of services available to clients)

Design and test instruments

Time TrialsFocus GroupsUsage StatisticsUsability TestsDocument

ComparisonsInterviewsSurveys

Collect baseline data

Allows you to compare before and after.

Can be collected while the document assembly content is being developed.

Keep this data collection consistent with your later data collection.

Collect additional data

Happens after implementation.

Stay consistent with your early data collection.

Analyze data and report findings Project Goals and Objectives Evaluation Data and Methodologies Summary of Major Accomplishments Recommendations and Future Steps In-Depth Analysis of Accomplishments Factors affecting project

accomplishments Strategies to address major challenges Major lessons and recommendations

Integrate necessary changesUse the feedback that you

gathered.

Tweak the program and then collect more evaluation data to find out if it really was a positive change.

Repeat

Plan

Design

CollectAnalyze

Report

Integrate

Case StudyIdaho Legal Aid Services

Idaho Project HighlightsPartnership between Idaho Legal Aid Services and the Idaho Supreme Court

Funding provided by the Legal Services Corporation and the State Justice Institute

Successful – good feedback and good filings

©2008 National Center for State Courts

A Conversation with LSC

Questions?Kate Bladow –

kbladow@probono.netSteve Rapp -

steverapp@idaholegalaid.orgBristow Hardin – hardinb@lsc.gov

Resources

DA and Legal Aid Specific

Document Assembly Supporthttp://www.probono.net/dasupport

LSC Evaluation Plan and Report Formshttp://tig.lsc.gov/eval_nonwebsite_grants.php

SelfHelpSupport.orghttp://bit.ly/y4c4s

Overview and TheoryW.K.K. Foundation Evaluation

Handbookhttp://bit.ly/1fg3n4

Outcome Measurement Resource Networkhttp://bit.ly/4rD2eY

Basic Guide to Program Evaluationhttp://bit.ly/eHHjW

Intelligent Measurement Bloghttp://intelligentmeasurement.wordpress.com/

Point Khttp://bit.ly/HQL2k

Surveys

LSNTAP Survey Toolhttp://bit.ly/2ARmCF

Creating Effective Online Surveyshttp://bit.ly/1bLpwg

Likert Scaleshttp://bit.ly/4nlZlh

Usability Testing

Usability Methodshttp://usability.gov/methods/

Assess Your Site’s Navigation & Structurehttp://bit.ly/1Bc1Pv

Useit.comhttp://www.useit.com/

Focus Groups

How to Use Focus Groupshttp://bit.ly/NHLBA

Data AnalysisExcel Skills for Advocates

http://www.lsntap.org/OFF105W.K.K Foundation Data Analysis

http://bit.ly/44nU2hHow to Analyze Data

http://bit.ly/3sRnvPOverview Introduction to Statistics

http://bit.ly/3TpaMdThe Complete Idiot's Guide to

Statisticshttp://bit.ly/2t0Hen or http://bit.ly/4hRQok

Attributions Image – Slide 6 – SMercury98

http://www.flickr.com/photos/smercury98/ Image – Slide 8 – markymarko

http://www.flickr.com/photos/cybercafe/ Image – Slide 10 – roboppy

http://www.flickr.com/photos/roboppy/ Image – Slide 10 – gergenzola

http://www.flickr.com/photos/gergenzola/ Image – Slide 10 – Eric Spiegelman

http://www.flickr.com/photos/ericspiegelman/ Image – Slide 10 – jimmedia

http://www.flickr.com/photos/jimmediaart/ Image – Slide 10 – cambodia4kids

http://www.flickr.com/photos/cambodia4kidsorg/ Image – Slide 10 – ifindkarma

http://www.flickr.com/photos/ifindkarma/ Images – Slide 13 – johnbullas

http://www.flickr.com/photos/johnbullas/ Image – Slide 27 – Faeryan

http://www.flickr.com/photos/faeryan/ Content – Slides 7, 8, 10-13, 16-18 – Adapted from Court Leadership and Self-

Represented Litigation Package: Module 7http://bit.ly/3qjLJ4