The Framework of Qualitative Evaluation in Online HRD of In-service Teachers Yoshida, Masami, Ph.D. Professor Faculty of Education, Graduate School of.

The Framework of Qualitative Evaluation in Online HRD of

In-service Teachers

Yoshida, Masami, Ph.D.Professor

Faculty of Education, Graduate School of Humanities and ScienceChiba University, Japan

2

Study Outline

• In-service teachers’ training– Computer room– E-Learning (TCU-LMS)– Online assignment

• Qualitative evaluation– Forth Generation– Criteria for Validity: Guba’s criteria– QSR NVivo – Interpretive Approach– Scoring Policy: TRIZ

• Action Chart

3

Teachers’ License

• Personalization of education– Small sized school– Small sized classroom– Team teaching– Increasing special education

• Need expand adaptabilities of each teacher in terms of expertise– From pre-service (number) to in-service training of

enhancement• Continuing education and HRD for teachers

– Summer vacation– Credit– License

4

Incoherence

1. Entire distance method was not permitted by stipulations2. Objective-oriented evaluation was not fairly suitable to

evaluate teacher-students learning under constructivist paradigm of education

This study argues effects of new way of absolute scale evaluation

5

The Course Details

• “Educational Methodology and Skills” : 1 credit• Use two days (9:20-16:40) • E-learning was used as prime method of learning and all le

sson contents were accessed through online (LMS: TCU-LMS, the Japanese-language version)

• The 38 teacher-students who had at least 12 years teaching experience

• Content: 15 sections, include 182 Web links (knowledge resource: 22, others are vital communication pages), 94 pages in printed volume

• Motivate teacher-students to activate potential resources & expansive learning activity through net surfing

Historical Changes of Evaluation Target

7

First Generation: Period of Educational Measurement

• 1900-• To determine potential ski

lls of people• Indicators of social skills• Used to enforce conscripti

on, arrangement of study groups in a school

• Intelligence quotient, aptitude tests, personality tests were developed

• Influence of Thorndike’s theory of learning: law of readiness (Hergenhahn & Olson, 2000)

learner

10 Widely recognized view pointsWide adaptability of results

Universal property

・Intelligence Quotient・Aptitude Test

・Personality Test

Indicators of Social Skills

First Generation

8

Second Generation: Period of Educational Evaluation

• 1930-• Expanded secondary and

tertiary education• Educational Indicators• To know changes of

students’• Needs to value

educational effectiveness• Continuous monitoring:

time series based study• Statistical methods to

judge significance• Achievement test,

standardized test• Influence of theory of

John Dewey

learner

10Time Series

Qualitative & QuantitativeOverall view

Improvement of Education

・Achievement Test・National Exam.

・Standardized Test・Inferential Statisticslearner

Time Elapsed

Educational Indicators

Second Generation

9

Third Generation: Objective Based Evaluation

• 1960-• Accountability of

educational institutes • Indicators from

curriculum • Objectives based

evaluation• Used various educational

materials and media (Stake, 1967)

• Formative evaluation and masterly learning

• Standard score (T-score) for education

Learner

10 Objectives of InstructionCurriculum Improvement

・Objective Oriented Evaluation・Formative Evaluation・Masterly Learning

・Standard Score for Education

Indicators from Curriculam

Third Generation

10

Forth Generation: Responsive Constructivist Evaluation

• 1990-• Scientific evaluation frequently le

d managerialism (Guba & Lincoln, 1989)

• Indicators from specific classroom

• Constructivist paradigm• Qualitative evaluation; Naturalisti

c inquiries• Evaluation criteria were anti-preo

rdinate, determined through observation and discussion of stakeholders

• Pay more attention on validity than reliability

Leaner

10 Qualitative AnalysisThick Description

Inductive Data AnalysisOutcomes

・Action Research・Naturalistic Inquiries

・Validity

Indicators in Specific Classroom

Forth Generation

11

Inductive Procedure

Positivism Approach (Generation 1st-3rd):Positivism Approach (Generation 1st-3rd): Evaluation = Rule (situation, action)This aims to explain action and predict another situation to define a rule that is available in other cases (Belk, Sherry Jr., & Wallendorf, 1988)Use deductive data analysis.

Interpretive Approach (Generation 4th):Interpretive Approach (Generation 4th):Evaluation = Interpretation (situation, action)This aims to interpret action and recognize specific situation.Use inductive data analysis

Methods

13

Purposive Sampling

• Final assignment: designed each student to post proposals in a Web board as homework

“Describe your proposal that improves learning of your students to be more self-directed. Especially, explain the way how you intend to apply educational methodology and knowledge of technologies.”

• Purposive Sampling– All descriptions of proposals were introduced into QSR Nvivo7, a

nd processed qualitative data analysis– Valued by stakeholders: an instructor and a coordinator– Use naturalistic inquiries, that is similar to moderation (Gipps, 199

4)– 7 x 4 treatment by seven action criteria (action chart) and four ade

quacy criteria (validity).

14

Action Criteria

All proposals were arranged into the table framework of the action chart that was postulated by Mills (2000), where the following elements are positioned.

1. Research question (findings)2. Recommended action that targeted a given finding.3. Who responsible for specific actions. (responsibility)4. Who needs to be consulted or informed about the findings

of the lesson and the associated actions?5. Who will monitor or collect the effects of actions.6. Dates when the actions and monitoring will occur.7. Any resources and media that will be needed to carry out

the action.

15

Adequacy Criteria

Each element of an action chart was reviewed based on Guba’s four criteria of the qualitative paradigm, and valued trustworthiness.

1. Credibility (CR): roughly parallel to internal validity. It poses the questions of whether the explanation fits the description and whether the description is credible. It refers teacher’s ability to take into account all of the complexities.

2. Transferability (T): roughly parallel to external validity, generalizability of explanation. This concern only case-to-case domain specific data transfer in a naturalistic qualitative study view. It is teachers’ beliefs that everything they propose is context bound.

3. Dependability (D): roughly parallel to reliability. Proposal is responsible for ensuring that the process of lesson is logical, traceable and clearly documented.

4. Confirmability (CO): roughly parallel to objectivity. Data and interpretations of the proposals are not figments of the teacher-student’s imagination, but are clearly derived from the learned contents.

16

Scoring Policy• Developed through the discussion of stakeholders• TRIZ and invention cards were used to determine the warrantable scoring policy

(Altshuller & Altov, 1994).– No.2: taking out– No.5: merging– No.40 composite materials

• Decided a way to count positive points in valid measures that were applied from lesson contents as long as negative points did not exceed positive points

• If a negative point exceeds over a positive point, we valued the action chart one level below.

Evaluation Positive Point

A 5 more

B 4

C 3

D 2

F 1 or less

Findings

Possible adequacy criteria were marked on elements.

Then, positive and negative adequacy criteria were assigned.

18

Action Chart

19

Results

• For reference, scores of achievement test were compared (Table above).– The mean of achievement test

on the evaluation levels were not significantly different at the alpha =.05 level with one-way ANOVA procedures.

• The calculation results of mean positive score by criterion (Table below).– Credibility and Confirmability

had higher score than Transferability and Dependability.

Evaluation NumberMean Achievement

Test Score

A 14 81.2

B 17 81.6

C 5 78.5

D 2 80.0

F 0 -

Evaluation-Criterion

CR T D CO

A 1.9 1.2 1.2 1.6

B 1.6 1.1 1.0 1.4

C 1.2 1.0 1.0 1.0

D 1.0 0 0 1.0

Mean 1.6 0.71 0.66 1.4

Discussion

21

Appeared Expansive Learning

• Vigorous learned things were appeared as their applied methods

• An expansive learning process in which lesson contents and information of other communities are used together for generating new idea.

• It was prominent that using e-learning and Web surfing enabled them to invoke potential information that could not be possible to overview in the traditional F2F lecture with limited knotworking.

• The author sometimes sensed insufficient descriptions – Insufficient context– Lack: details explanation, situational suitability, cares about

limitation of methodology, and involvement of other staff in the process

22

Appeared Changes

• Qualitative evaluation method was vigor for interpretation and easy to lead diagnostic opinions.

• The proposals were made in the situation of their peculiar classroom, but adequacy criteria have enabled to value their varied proposals

• Figure shows summed number of how many plural criteria were marked together in one element of an action chart– This data showed that their plans

were valid to show combined marking

Credibility55.0%

Transferability85.2%

Confirmability75.5%

Dependability72.0%

9319

11

3

12

YOSHIDA Masami

Professor of Chiba University

[email protected]