Leading to Conclusions Analyzing and Interpreting Data Ben Silliman, Youth Development Specialist North Carolina 4-H.
Post on 21-Dec-2015
215 Views
Preview:
Transcript
Leading to ConclusionsAnalyzing and Interpreting Data
Ben Silliman, Youth Development SpecialistNorth Carolina 4-H
Qualitative Analysis
Leaders also need a way to process those “open ended” observations & comments for decision-making
• To understand the experiences, perceptions, motivations, intentions, and behaviors that – may not yet be coded in an accurate and efficient survey– may offer more in-depth, in-breadth data than a survey
• To engage an audience in dialogue and partnership– valuing them as people and partners– better understanding their needs for education, empowerment
Leading Figures
Tania Leon
Classical music performer,
composer, and conductor
From the organization of her own creative genius to the instantaneous processing of sounds and movements in an orchestra, this multi-talented artist goes beyond the quantitative data (tempo, pitch, volume) to identify and interpret patterns and themes in music and dance. Effective work with a UNC students or other collaborators often depends more on sifting insights than reviewing numbers-thus the value of qualitative analysis.
Purpose of Qualitative Analysis
• Analyze and Interpret Narrative Data from
– Interviews: focus groups, individual interviews– Open-ended questions: self-generated descriptions
or graphics– Observations: on-site observations of groups or
individuals, content analysis of products
Note: qualitative data most insightful when triangulated with other observations and quantitative responses
Purpose of Qualitative Analysis
• Gain insights on a specific theme or audience(vs. generalizing to broader groups)
– Leadership needs of this Council– Strengths of these regional Councils relative to stages
of leadership maturity– Views of youth leaders attending this year’s national
conference about an organizational conflict (or opportunity)
Organizing Qualitative Data
• Pre-Organization (based on theory, experience)
– Established categories used to group comments• Fit ideas to a framework• Focus on predictable categories
• Emergent Organization (grounded theory)
– Frequently mentioned themes (content, process, relationship) named, comments clustered
• Allow critical points to arise from respondents
Organizing Qualitative Data Examples
• Pre-Organization (based on theory, experience)– Leadership Challenge roles: vision, modeling, creative
processing, engaging & empowering, caring– Project Management strengths: recruiting, planning, mobilizing,
implementing, evaluating
• Emergent Organization (grounded theory)
– County Council Leadership Questionnaire:• “Nobody has experience with this”…Content expertise• “People won’t talk with each other”…Relationship issues• “Events are chaotic, thrown together”…Process, organization
Emergent consensus: More training & guidance needed
Qualitative Analysis Steps*
Handling Narrative Data
• Absorbing: Read and re-read to thoroughly understand input before categorizing
• Focusing: Clarifying the purpose for evaluation
• Categorizing: Coding and organizing comments
• Identifying Patterns & Connections: Grouping and inter-relating comments
*adapted from Taylor-Powell, E., & Renner, M. (2003). Analyzing qualitative data. Madison, WI: UW Extension Service bulletin 3658-12.
Qualitative Analysis Step 1: Absorbing What was Said
Practical Step #1: Retreat and Reflect
• Debriefing: Be sure to transcribe or translate comments “as is” in order to process accurately
• Valuing: Examine the quality (focus, breadth, depth) of data to determine worth for examining
• Text-Check: Consider diverse interpretations, limitations, context in any set of comments
• Self-Check: Consider personal biases that affect how you see (and what you don’t see) in the text
Qualitative Analysis Step 1: Absorbing What was Said
Practical Step #2: Don’t lose it
• Marking: Be sure the source and context are clearly indicated and transferred with each item
• Sorting: Physically or electronically group the –marked- comments together
• Duplicating: Copy originals and keep them in a separate location
• Storing securely: Maintain confidentiality by holding material under lock and key
Qualitative Analysis Step 2: Focusing What You Notice
• Remember the Purpose of the Evaluation > What was the research or evaluation question, anyway?
• Identify key questions or emphases– By topic: vision, management skills, caring for others– By time period: public-private settings, early-mid-late
project, routine and crisis events– By case, individual, or group: event, personal,
interpersonal dynamics and systemic issues in or across events
Qualitative Analysis Categorize What You Find
Sample: Open-ended Questions on how Community Service projects teach leadership
As a youth professional, you believe that “hands-on” experiences in community service fosters a variety of leadership skills. As a follow-up with youth leaders who planned and implemented a community service project, you ask an open-ended question:
What did your experience in this community service project teach you about leadership? (Use ”I” statements to describe your learning)
Qualitative Analysis Step 3: Categorize What You Find
• Label and Group individual items > Mark text using pre-set or emergent categories
Feedback from youth after a Community Service Project:What did your experience in this community service project teach
you about leadership? (note broad categories, need to interpret)– Values: responsibility/flexibility, integrity/compromise, acceptance/
accountability, diversity/unity– Concepts: vision/action, task-oriented/people-oriented,
compassion/accountability, product/process– Behaviors: talking/listening, working/playing, thinking/doing,
working alone/working together– Relationships: many/few, close/distant, homogeneous/diverse,
Leading Figures
Jane McKimmon
North Carolina
Extension Educator & Administrator
As an innovative leader, Dr. McKimmon often asked “open-ended” questions, then reflected and prepared a synthesis to create a program. Later, this experience enabled her to present structured evaluations, more efficiently processing client feedback. Although these interactions were often intuitive, data was gathered and interpreted systematically. These are the key elements of qualitative evaluation.
Qualitative Analysis Step 3: Categorize What You Find
• Label and Group individual items > Use existing question or program structure to group comments
Feedback from youth after a Community Service Project:What did your experience in this community service project teach
you about leadership? (note details used to focus descriptions)– Planning the project– Recruitment of helpers– Gaining support– Implementation (doing the project)– Problem solving during the project– Evaluating results
Qualitative Analysis Step 3: Categorize What You Find
• Re-read and distinguish sub-categories > Refine analysis based on research and richness of the data
Feedback from youth after a Community Service Project:(sample comments on communicating as a leader)
– One-way: Recognize the importance of step-by-step instructions
I tried to explain, but forgot and left things out…– Talk/Feedback: Understand the importance of feedback
I learned to check with each person to be sure he understood his job…and was excited about it…
– Two-way: Regular exchange of ideas
Even though I knew the plan, I found that other people gave me good ideas as the project went along …
Qualitative Analysis Step 3: Categorize What You Find
• Relate data to appropriate (multiple) categories> Apply comments to all relevant categories (without over-using)
Feedback from youth after a Community Service Project:(sample comments reflecting innovation)
I tried to explain, but forgot and left things out…
– Communication: ability to give clear directions– Planning: implies development of a project plan
Qualitative Analysis Step 4: Identify Patterns & Connections
• Group within category> Distinguish by similarities and differences
Feedback from youth after a Community Service Project:(sample categories for communication)
Level X: We just argued all the time and the project did not get done…
Level 0: Nothing…I wasn’t the leader…
Level 1: I tried to explain, but forgot and left things out…
Level 2: I learned to check with each person to be sure
he understood his job…and was excited about it…
Level 3: Even though I knew the plan, I found that other people gave me good ideas as the project went along …
Qualitative Analysis Step 4: Identify Patterns & Connections
• Group across categories> Distinguish patterns for key variables
Feedback from youth after a Community Service Project:> Cross-compare for individual experience or observer rating
Level X: Not unusual for new leaders (20% of comments)
We just argued all the time and the project did not get done...
Level 0: Pretty typical for first-time leaders (33%)
Nothing…or…I wasn’t the leader…
Level 1: Typical for new leaders; not for experienced leaders (38%)
I tried to explain, but forgot and left things out…
Qualitative Analysis Step 4: Identify Patterns & Connections
• Group across categories (continued)> Distinguish patterns and frequency for key variables
Feedback from youth after a Community Service Project:> Cross-compare for individual experience or observer rating
Level 2: Rare for new leaders; frequent for experienced leaders (15%)
I learned to check with each person to be sure he understood
his job…and was excited about it…
Level 3: Exceptional, even for experienced leaders (4%)
Even though I knew the plan, I found that other people
gave me good ideas as the project went along …
Qualitative Analysis Step 4: Identify Patterns & Connections
• Try to explain patterns in the data> Explain correlations or cause-effect relationships
Feedback from youth after a Community Service Project:> Growth in leadership (communication) through community service
-Roles in leading community service experiences foster interpersonal perspective-taking and skills of leaders (cause-effect)
-As youth mature, their experience in community service as well as their communication skills will probably both increase (correlation)
-Those who seek leadership skills via service experiences are likely empowered to be more effective leaders later (reciprocal relationship)
Leading Figures
Billy Graham
Preacher, Organization Leader
Raised in Charlotte, NC, Dr. Graham became a practical leader to co-workers, spiritual leader to millions, but also served as leader to an organization holding large meetings through volunteer labor. Careful listening to comments and observation of events—qualitative evaluation--enabled staff to sensitively and creatively respond to opportunities and needs of participants in diverse locations and venues.
Qualitative Analysis Step 4: Identify Patterns & Connections
• Generate & answer questions from patterns> What context or process factors were typical of leaders who gained most (communication or other skills) from community service experience? (or was it just a matter of experience/maturity?)
Feedback from youth after a Community Service Project:> Context/Process factors that foster leadership communication
-Re-read Top 19% (Level 2 & 3) descriptions/quotes of youth
-Review registration/application materials of Top 19% youth
-Research or review descriptions of community service projects, including mentoring style of older youth or adults working with youth leaders
Qualitative Analysis Step 4: Identify Patterns & Connections
• Explore ‘no gains’ and unanswered questions> Why did most youth experience no gains in communication skills?
> How much ‘learning’ involved direct experience vs. observation of others? (and how accurate are reports of their own experiences)
Reflecting on feedback from leaders:> Accounting for ‘no gains’
-’Hands-on’ experience is not enough (more mentoring needed)
-The ‘open-ended’ question was too open (more focus needed)
> Unanswered questions
-May be implied in some responses
-Would have required
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> Main threads and lessons (related to communication)-About 1 in 5 youth show evidence of growth in communication skills
-Prior experience and adult mentoring are more typical for youth who demonstrated higher levels of communication skills
-Quotation that captures how project leadership improves listening:
Even though I knew the plan, I found that other people
gave me good ideas as the project went along …
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> New learning-For this group (at least) there is a greater-than-expected need for coaching (experiential training and individual mentoring) on communicating effectively as a leader (perspectives and skills)
-Picture Graph: Percent of leadership participants who gained each level of communication skills
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> Application to other settings and audiences-Coaching on how to communicate as a leader should begin early and be repeated often, esp. in non-formal (vs. classroom) situations
-Role modeling is critical for adults and older youth as youth leaders as both the attitude and skills of good communicators
-Picture: Youth leader directing end-of-project debriefing
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> Interpretation to diverse stakeholders-Parents/Volunteer Leaders:
Project leadership requires a major time commitment and willingness to work outside one’s comfort zone…As “hands-on” project leaders, many are learning to be clear and caring communicators. Using evaluation feedback from our most effective youth leaders, all youth leaders will be trained and mentored in key skills such as giving directions, active listening, and involving all team members in planning and problem solving during projects and other club activities.
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> Interpretation to diverse stakeholders-Grant Funders/Commissioners:
Community service teaches important values such as compassion for others and giving back to one’s community. Projects also develop leadership and teamwork, critical skills in today’s workplace and civic arenas. Our ongoing program evaluation indicates that many youth leaders are gaining skills to communicate effectively but most need more training and mentoring…Thus we are requesting funding for a program assistant to train volunteer leaders as leadership coaches. This training will support a five-fold increase in community service among youth, providing in an estimated $50,000 benefit to the county.
Qualitative Analysis Step 5:Reaching from Insight to Action
• Summarize major themes & connections
Feedback from youth after a Community Service Project:
> Interpretation to diverse stakeholders-Business Partners:
Today’s community service project leaders are tomorrow’s business and civic leaders…Many youth are learning practical skills such as giving directions, listening effectively, and engaging team members in problem solving. Please consider volunteering as a Mentoring New Generation Leaders volunteer and help us train 100 more youth leaders in the county to become effective project planners, organizers, and communicators while engaging in needed community service.
Qualitative Analysis Step 5:Reaching from Insight to Action
Capture ideas & action in a published report• Format
– Program Fact Sheet, Brief– Brochure– DVD/Video
• Purpose—connect with stakeholders to– Inform: describe benefits of the program and efforts to
improve it– Invite: seek investments of time, money, and other
resources to support and expand the program
top related