© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 05/09/2015 - 1 Evaluation, test and development of soft skills Evaluation of the test: results from students´projects.

© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 04/19/23 - 1

Evaluation, test and development ofsoft skills

Evaluation of the test: results from students´projectsWhat do test measure?Testing soft skills

Methods of measuring and improving soft skillsespecially planning games

Prof. Dr. Ulrich Holzbaur AAUAS - Aalen University of Applied Science

faculty of economics, dept. for engineering management STZ Applied Management, Aalen

Steinbeis Tranfer Centre for Applied Management, Aalen


Own experiences/ backgroundProf. Dr. U. D. Holzbaur

Involved in education, teaching, training , leadership since 1974 Actively involved in training/teaching with Army, University Studies for secondary education Head, airborne radar systems group, EADS

Professional experience in Flight Trainer Design with EADS Prof. at AAUAS since 1990, students´projects in:

Computer Science and Software Development Environmental Management and Project Management First grade students´research projects 100 master thesis at Ulm University and AAUAS

LARS project „Planning Games“ Grant from the Studienkommission Hochschuldidaktik Baden-Württemberg

(university education commission) 1997 to 2002 workshops on planning games 1997 to 2002 Development of planning games for sustainability, management, skills


Evaluation of the tests by students´ teams

Situation: within their 1st and 7th semester, students work on a project in teams of 4 to 5 1st term: methods of project management and research 7th term: quality- and environmental management Two joint projects:

Public transportation system for students´ requirements, students´ticket Evaluation of the test

Participants: Pascal Cromm, Antje Discher: tests 4 + 5 students´ teams as object (participants) 1 + 1 students´ teams doing the evaluation

Students´ project run from march 2003 to July 2003 (final report)


Example of the results from tests

-30

-20

-10

0

10

20

30

No

rma

lbe

reic

h li

eg

t b

ei -

48

bis

+ 4

8

Gruppen

Interkulturelle Kommunikation

Soft Skills

ÖPNV

Event

Grüner Gockel

"W"


Evaluation or measurement of skills

With respect to any property PX of a person X we have several values which we may try to measure:

PX "how am I" is the "real" value of X however that may be defined. I(PX) "index" is an indicator for X, e.g. the classical IQ defined by an IQ test. S(PX) "how do I see me" is the self evaluation of the individual. F(PX,Y) "how do others see me?" is the external evaluation by another person or

group of persons Y, this is the image or aura of a person. S(F(PX,Y)) "how do I assume to be seen by others?" "do I know my own image?" F(S(PX),Y) "how do others assess my self-estimation?" F(F(PX,Y),Z) "how do others assess my image?"

In addition, the property PX leads to some performance P in a real world situation R or to some result P in a test situation T.

With regard to testing, we have I(PX) = P(PX,T). The 180°-feedback measures P(F(PX,Y=peer-group), T=180°-feedback)


What do tests measure?

The tests measure P(F(PX,Y),T) in comparison to P(S(PX),T) by using the same test.

The real blind spot ist the difference between F(PX,Y) and S(F(PX,Y))which could be assessed by testingP(F(PX,Y),T) in comparison to P(S(F(PX,Y)),T).


Test principle and structure

skills and behaviour

Self-assessment

Perception by others

180°- feedback

test


What can tests measure? e.g. intercultural competence

Ethical Competence: dealing with different sets of values To decide in an ethical conflict

Personal Competence and awareness: perception of different values, behavior and scripts in different cultures

To see when and whether a intercultural difference occurs Social Competence/ Management competence

To be able to integrate different cultures, to be able to lead a team with members from different cultures

Problem solving competence To solve problems embedded in or arising from

an intercultural difference Knowledge

To know the parameters that determine a cultural behavior Facts

To know the dos and donts of specific cultures

tests


Results from the students´questionnairs

Project: questionnaire for self-assessment questionnaire to assess the team members

questionnaire developed by students´s team to assess the test

self

180°-feedback

evaluation of the test


Evaluation Typical distribution of students´ answers: fine/ good 20%, poor: bad: 20% too complex (> 50 %) took too much time ( > 50 %)

typical responses (free test) tests are too long and too complex many questions that are similar problems to see the meaning students get tired of filling the questionnaires not adapted to / adequate for students´situation

suggestions (free text) test should be anonymous role play would make more sense same method within all test

Evaluation and typical response


Overall evaluation by the students:do the test make sense?

Fandest Du diese Tests sinnvoll?

0

2

4

6

8

10

12

14

16

10=s

ehr g

ut 9 8 7 6 5 4 3 2 1

0=se

hr sc

hlech

t

Bewertung

Hä

ufi

gk

eit

Innovation/Kreativität & Risikobereitschaft

Team und Projektmanagement

Interkulturelle Kompetenz


Problems with translation and meaning

e.g.: „Ich verhalte mich unvermutet“ -> unexpected -> spontaneous behaviour positive ? creativity ? within a management/ leadership context, this is a rather negative property:

unberechenbar = unpredictable, unfair.


Overall evaluation of the quality of the tests:points (max = 10)

innovation/ risk taking skills: 5.5 team & project management skills: 5.5 intercultural competences: 4.2

Suggestions: Test should be shorter, more precise and systematic Try to compare

self assessment to „real“ skills perception by others to self assessment of perception

There should be some method / tool to measure the skills Test should be in a setting, preceded or based on some common experience

e.g. an extensive project experience.

Summary


Planning Game:teaching and experiment

planning game:management game,simulation gamewargame

Model of the real world Simulation Controlled experiment Method of teaching Method of learning

Next, let us lower the sales figures and measure his stress symptoms


Planning Games

Learning by playing Didactics Situations

Planning Game: Initial Conditions = Starting Position Decisions to be made Dynamics = Rule Criteria

Conceptual Basis: Dynamics Decision Goal Didactics

Planning game is more than hard + soft - ware


Planning games

Role-Based Computer-Based

Facts

SelfProblem solving

Your role is ... You are ....You are responsible for ...You will be evaluated ...You may ...

Your role is ... You are ....You are responsible for ...You will be evaluated ...You may ...


Simulation games

Wargames and strategic games The Settlers of Catan, …..

Economic Simulation Games with graphic visualization Beer, Zoo Tycoon, …

Number-Based Economic Simulation Games TOPSIM, FondManager, ….

Management Simulations Project management,

Internet Based Games MARGA, TOPSIM, StockMarketTraining

Paper and Board Simulation Factory,

Startup Computer Simulation (typically based on MS-Excel) UGS-SIM, UGS-Game


Computer based planning gamese.g.: TOPSIM Startup

strategic start-up simulation Phases

Idea and business plan Foundation - constitutive decisions Market entry and competition Evaluation

business areas Production (high-tech sports equipment) Commerce (eCommerce, books/CDs) Services High-Tech (bio-technology)

Problem areas/ challenges Business / finance Technical Social


Other examples of planning games for management skills (paper based)

Number factory Players “produce” numbers Each player has different

Information Role and responsibility Behaviour

Planning takes place on different levels: Production (to calculate the mumbers) Organization (mass production, push-pull, R&D, ) Communication (rules, culture)

Albuchmühle Players plan the future for a mill Core aspects:

Marketing Environmental and risk management

Players have hidden agenas

Your role is ...

You are ....

You are responsible for ...

You will be evaluated ...

You may ...

You want to ...

You are planning to ...

Your role is ...

You are ....

You are responsible for ...

You will be evaluated ...

You may ...

You want to ...

You are planning to ...


Öko² - oeko2 Planning Game for Sustainable Development

Evaluation of the player according to the relevant variables


How to use planning games for skills development

Education (university, school): Special course within the curriculum Embedded method within courses on special subjects

management, quality/ project/ risk/ environmental management accounting, marketing, controlling, enterpreneuring any other subject

Startup for newcomers: Facts Skills Motivation

Lifelong learning: Test & Evaluate … Gain & Improve & Enlarge … Evaluation of skills


Planning games match entrepreneurship skills

Planning games provide an optimal match for the competences needed by entrepreneurs:

Give the potential entrepreneur the Facts and Knowledge Problem solving competence Social Competence and Intercultural Competence Personal Competence and self-esteem Ethical Competence

which are needed to Plan: start a company and make the right decisions Communicate: get the money, personnell and customers Manage: run the company Control the operations and establish the management systems Decide optimally in routine and critical situations

Business Simulations allow to identify misplanning and risks


How to use planning games for skills assessment

Start with planning games Dedicated games for

Skills Aspects Properties


Summary

Test have been evaluated by students´ teams too complex too much questions with little differentiation inhomogeneous answering and evaluation system what do the tests measure?

Planning Games are a valuable tool to teach, experiment, learn evaluate soft and hard skills Create a basis for skill evaluation

Planning games can support soft skills Facts, knowledge and understanding: I KNOW Motivation and self-esteem: I CAN Action: I WILL

© Prof. Dr. Ulrich D. Holzbaur, AAUAS STZAM 05/09/2015 - 1 Evaluation, test and development of soft skills Evaluation of the test: results from students´projects.

Documents

aauas stzam

applied management

methods of project management

test situation

studentsticket evaluation

external evaluation

self evaluation

test participants