Closing the Gap: From Data to Decision Making

Closing the Gap:From Data to Decision Making

Stefinee Pinnegar, Ph.D.Pat Esplin, Ph.D.

Brigham Young University

academy.byu.edu

Closing the gapfrom data to decision making

Intractable Problems •Our world is confronted with problems that occur over and over and are difficult to address•Providing solutions at a particular place and a particular time can inform others

Overarching Questions •Organizational goals can be catalysts for identifying and responding to the intractable problems within your setting•There is power in translating your goals into questions that guide assessment and action.

Assessment Strategies •Identifying available data embedded within normal activities of the organization provides a rich source of information•Using an assessment cycle transforms data to actionable information

Decision Making •Organizations become more strategic when they base actions on meaningful data•Learning organizations seek improvement by gathering information about the effectiveness of their actions in an assessment cycle.

1. _____________________________________2. _____________________________________3. _____________________________________4. _____________________________________5. _____________________________________6. _____________________________________

Intractable Problems

•What are some intractable human issues that always underlie problem solving in education?

1. Literacy2. Economic health/poverty3. Diversity4. Health5. Family6. Power

Intractable Problems

•Our (student & society) intractable problems

Overarching Questions

•How can we effectively support student success here?

• Underlying questions–Who?–What?–When?–Where?–Why?—Mission statement– How?

The existing knowledge, understanding, abilities, motives

and conceptions/ styles of learning students already have

Quality of learning achieved

How studentsapproach learning

and studying

How course material is selected, organized, presented, assessed

University teachers’ ways of thinking about teaching

How students perceive the teaching-learning

environment

Conceptual framework indicating influences on student learning

How a teaching-learning environment is designed

and implemented

Noel Entwistle, ESRC/TLRP ETL report 3, March 2003

What students are expected to learn and understand

Constructive alignment

1. _____________________________________2. _____________________________________3. _____________________________________4. _____________________________________5. _____________________________________6. _____________________________________

Overarching Questions

•What are your institution’s/program’s basic or overarching questions?

Assessment Strategies

•Levels of on-going assessment

Classroom

Program/ Department

Institution

Assessment Cycle

Your institutionMission/Values

Assessment criteria (How will we know if we

have done this well?)

Learning – So what?(What did we learn? What does it mean?)

Outcomes(What do we want

to achieve?)

Improvement(Now what? Policy, resource changes?)

Assessment results (What happened?)

Assessment activities (Design, carry out)

InternalBYUDeveloped

Employers of BYU Graduates

Survey

Graduate Student

Exit Survey

External

BYU Freshman Surveys

• Pre-arrival• End-of-year

Suite of BYU Instruments

Timeline

NSSE

Non-Returning Student Survey

Freshman Sophomore Junior Senior Grad Std Post-college

NSSEBCSSE

AQ2 (10 year)

Alumni Questionnaire

Senior Survey

Internal /In use

Internal / In DevelopmentKey External

Aims of a BYU EducationFoundation

Internal / Planned

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

Assessment Strategies

•What kinds of data do you already have?•What kinds of data do you need?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

Data you have Data you need

Overarching Questions• How can we effectively support

student success here?

Assessment Strategies• What data do we already have?• What data do we need?

• Note: each word in the question directs our efforts:

• How• Can• We• Effectively• Support• Student success• Here

Assessment Strategies

•Who matters at my institution?

Freshman Academy

1. _____________________________________2. _____________________________________3. _____________________________________4. _____________________________________5. _____________________________________6. _____________________________________

Assessment Strategies

•Who matters at my institution?

Assessment Strategies

•What matters to us?

Freshman Academy (FA) is a one-semester program that helps first-year students make important connections within the university during their first semester at BYU. By creating the best teaching-learning environment possible, FA strives to provide an academic environment that facilitates the development of well-rounded students and bridges the gap between high school and college. FA does this by helping students connect with each other, faculty members, academic majors, the University Core, The Aims of a BYU Education, and other university resources.

Freshman Academy Mission Statement

1. _____________________________________2. _____________________________________3. _____________________________________4. _____________________________________5. _____________________________________6. _____________________________________

Assessment Strategies

•What matters at my institution?

StudentPeer

mentor

Other studentsFaculty

Academic Experience

Development of the total Person

Aims of BYU Education• Spiritually strengthening• Intellectually enlarging• Character building• Life-long learning & service

Academic majors

University resources

Bridging the GapFreshman Academy Mission

What matters to stakeholders?

Cost Effectiveness

High impact Low impact

High cost

Low cost

Student retention

Student performance (grades)

Student learning

Time spent at the university

Expe

ctati

ons

1 yr 4 yr3 yr2 yr

When do things matter?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

Assessment Strategies

•When do you do things?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

What things do you do when? What do you know about them?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

Assessment Strategies

•What do you know about student success at your institution?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

What things do you know? How do you know them?

Analysis of Existing Data

Points in time

Criteria Demographics

BackgroundExpectations Engagement

ConnectionsLearning Habits Performance

Pre-semester •BCSSE•BCSSE•Learning Goals Inventory•Boyer Model

•High school engagement•FA interactions•Housing placement

•Approaches to study•Time spent in high school

•ACT•GPA/EGPA analysis•Summer reading reflection

Week 3 •Time logs•PM “How well do you know”•Meals with faculty•Time logs

•Weekly surveys•Classroom observations•Time logs

Mid-term •Socio-grams•Meals with faculty

•Weekly surveys•Classroom observations

End of Semester

•End of semester survey•Student reflections•Peer mentor reflections summaries

•Peer mentor reflection summaries

•Final learning papers•Student reflections•Peer mentor reflection summaries

•Student reflections•Peer mentor reflection summaries

Follow-up 2nd Semester

•Modified Temple survey•Use of campus resources•NSSE•Peer mentor story cycles

•Use of time•(ETL)•Peer mentor story cycles

Assessment Strategies• What do you know about: How can

we effectively support student success here?

Decision Making • What do we do now?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

What do you know? What can you do?

High ACT

Low ACT

High HS GPA

Low HS GPA

Significant time preparing for class

Low level Engagement

Limited time preparing for class

High level Engagement

Assessment Strategies

•Students who matter

Assessment Strategies• What do you know about: How

can we effectively support student success here?

Decision Making • What do we do now?

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

1. ________________2. ________________3. ________________4. ________________5. ________________6. ________________

What do you know? What can you do?

Before semester• Application• Registration• Autobiogra

phies• Surveys• Contact

First 3 weeks• Interviews• How well

do you know?

Mid-term• Faculty

meetings• Student

engagement

End of Semester• Interviews• Transition

ratings• Student

engagement

Using typical activities and timelines

Closing the gapfrom data to decision making

Intractable Problems •Our world is confronted with problems that occur over and over and are difficult to address•Providing solutions at a particular place and a particular time can inform others

Overarching Questions •Organizational goals can be catalysts for identifying and responding to the intractable problems within your setting•There is power in translating your goals into questions that guide assessment and action.

Assessment Strategies •Identifying available data embedded within normal activities of the organization provides a rich source of information•Using an assessment cycle transforms data to actionable information

Decision Making •Organizations become more strategic when they base actions on meaningful data•Learning organizations seek improvement by gathering information about the effectiveness of their actions in an assessment cycle.