Plenary - Eye on the Prize: Sharing and Using Program Review Results Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education Leadership and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego State University 3590 Camino Del Rio North San Diego, California, U.S.A. 619-594-8318 [email protected]
32
Embed
Plenary - Eye on the Prize : Sharing and Using Program Review Results
Plenary - Eye on the Prize : Sharing and Using Program Review Results. Marilee J. Bresciani, Ph.D. Professor, Postsecondary Education Leadership and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego State University 3590 Camino Del Rio North - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Plenary - Eye on the Prize:Sharing and Using Program Review Results
Marilee J. Bresciani, Ph.D.Professor, Postsecondary Education Leadership and
Co-Director of the Center for Educational Leadership, Innovation, and Policy
San Diego State University3590 Camino Del Rio North
Developing specific plans for modifications and improvements
Informing decision making, planning and budgeting, including resource re/allocation
Linking and, as appropriate, aggregating program review results to the institution’s broader quality assurance/improvement efforts
Bresciani, M.J.
How do you envision using your results?
In order for these Uses to Occur, An Institution Needs…(Bresciani, 2006)
Set priorities around institutional values
Communicate a shared conceptual framework and common language
Systematically gather data that actually evaluates these priorities
Bresciani, M.J.
Sharing & Using Program Review Results
Challenges occur because: Results not linked to outcomes,
college/institutional goals, or strategic prioritiesTemplate fix is the solution here (p.144)
Those who receive results are unsure what to do with them Presentation of data Interpretation of the data Professional development
Sharing & Using Program Review Results, Cont. Challenges occur because:
The routing for discussion of results and decisions is not clearClarified by roles and responsibilitiesCommunication routing
Expectations of use of results is not clearClarifying how results will be used by
whom , when
What are your Challenges and how could they be readily addressed?
Bresciani, M.J.
Real Life Considerations for the Write-Up
The Audience For whom is the data? Change language for different audiences
if necessary The Story
What point are you trying to make and for whom?
What decision needs to be made and who needs to make it?
The Format Depends on the audience
Bresciani, M.J.
Reporting Strategiesfrom Gary Hanson, Ph.D.
Know your data Know your audience Tell the story
Identify meaningful indicators to shape the story
Examine indicators for patterns Begin with the end in mind
Tie the data to the outcomes Involve the end users in the process
Bresciani, M.J.
Reporting Strategies, Cont.from Bresciani, Zelna, and Anderson, 2004
Identify the values of your constituents and find out how your constituents prefer to see data and reports. Especially important for IR people who are the
“keepers of the data” Continual process of refinement
Students (or those whom you evaluated) can be extremely helpful in your writing and dissemination of results and decisions made.
Be sure to link the data and decisions made to the outcome and the program being assessed (Maki, 2001).
Bresciani, M.J.
Reporting Strategies, Cont. from Bresciani, Zelna, and Anderson, 2004
Timing is everything when delivering results and decisions.
Prepare to defend your outcome, your evaluation method, your results, and the decisions made based on those results.
If you need help interpreting the data, get it.
Consider multiple layers of reporting (broad and general to detailed and specific)
Bresciani, M.J.
Share Examples - Reporting What “need” generated the data
requested? What is the purpose of the data request?
What story needs to be told? Who will be using the data?
What are their values? …preferences for receipt of data?
Who needs to be involved in preparing the data? …presenting the data?
What story are you trying to tell? What key points are you trying to make?
Bresciani, M.J.
Sharing Examples - Providing Data
What data do your constituents want available to inform their decisions? How often do they want it? In what format?
How transparent do they want the data to be? What is the communication flow?
Do they need comparisons? What type of comparisons will be most
meaningful to them?
Some Examples
1. Various Ways to Tell the Story2. Using Dashboard Indicators to Inform OBPR Focus
Example Results SLO 12) Apply research to practice, particularly in their area of specialization and focus Student feedback and the evaluation of learning artifacts from 610, 795 A&B as well as the
learning portfolio reflect a misalignment in the curriculum. We have made slight changes to the ED 690 course to align it to better prepare students for ED 795A & B by offering a stronger transition to preparing students to write a literature review and to reflect on a problem statement and purpose statement. Faculty will continue to visit about further opportunities to improve this alignment over the coming year.
While the faculty explored the possibility of offering ED 690 in spring only and designing a section just for SA students and a section just for rehab students; this solution was not possible due to the in-affordability of offering this solution.
Students reported wanting more control over the selection of their research topic. Student groups will be given more autonomy in selecting their research project topic.
Students working in groups will continue to evaluate their peers. However, due to student feedback, the peer evaluation rubric will be revised to include aspects that students wanted to evaluate their peers on and students will be educated more frequently on how the rubric scores will influence their peer grades.
Results indicate that many students felt that they learned about each different aspect of a research project better in a group environment rather than individually. Students commented that they would prefer to be in a smaller group in the future, with a recommended group size of 4-5 students per project. The group size will be decreased from 8-9 students per group to a group size ranging from 3-5.
We will re-evaluate this outcome in 2008 to see if we made any improvement. Student Exit Survey –We will continue the exit survey each year but explicitly ask students to
respond to the extent that they learned each program SLO, rather than ask them for satisfaction of each course.
How do you perceive your {Primary Academic Advisor}?
AG AR BA ED EN GE GS LA SC VM totalefficient 180 116 153 139 192 92 99 171 163 141 1446
Percent of students indicating the type of advisor they go to most often, by college.
Bresciani, M.J.
Same Information
0%
20%
40%
60%
80%
100%
AG AR BA ED EN GE GS LA SC VM
None
Faculty
Departmental
College
Bresciani, M.J.
More focused InformationMedian
College
Departmental
Faculty
None
Bresciani, M.J.
Real–Life Reporting Reminders
Keep your audience in mind If you have to draft varying
reports/summaries of results for your varying audiences… do it
Report data in context of issues or outcomes
Provide a detailed version and an “executive summary”
Use graphs wisely
Bresciani, M.J.
Real–Life Reporting Reminders, Cont.
Timing IS really EverythingDon’t under-estimate the power of “trying out” drafts on key decision-makers
Interpret your data so that it informs program improvement, budgeting, planning, decision-making, or policies.
Report limitations honestly
Sharing & Using Program Review Results Questions In an ideal world, who would you want to review the results of program review? Does your answer vary by type of
program (prof accreditation or not) or level of program (UG, GR) ?
What reflection questions would you provide those reviewing the results to guide their interpretation of and therefore use of the results?
Sharing & Using Program Review Results Questions, Cont. What are the articulated expectations
for use of the results? How would results be disseminated? Who would be involved in interpreting
the results and what is their role? Are their clear paths for
communication flow of results, interpretation of results, decisions, and recommendations?
Sharing & Using Program Review Results Questions, Cont. Is one type of data more influential
over another type in the interpretation of the results?
Is it clear who is involved in formulating decisions that are based on interpretation of results?
Is it clear on which level the decision resides?
Mentor Group Reflection Questions Is there anything I need to change on
my OBPR template so that it is clear how results align with outcomes at each level?
Am I clear about who needs to see the results from program review and how they prefer to see it in order to inform the necessary decisions?
Do I need to provide any professional development for anyone so they know how to use the results to inform decisions?