QUALITY INDICATOR 13 APRIL 2013. Learning outcomes: Explain the meaning of quality indicators Describe the characteristics of quality indicators Synthesis.

QUALITY INDICATOR 13 APRIL 2013

Learning outcomes: Explain the meaning of quality indicators Describe the characteristics of quality indicators Synthesis the importance of quality indicators in HEI Explain studies related to the application of quality indicators in HEI Quality Indicators Indikator Kualiti dalam Pendidikan Tinggi (Quality Indicators in Higher Education)

INDICATORS Customer satisfaction Performance

Teras PSPTN Meluaskan akses dan meningkatkan ekuiti; Menambah baik kualiti pengajaran dan pembelajaran; Memperteguh penyelidikan dan inovasi; Memperkasakan institusi pengajian tinggi; Mempergiatkan pengantarabangsaan; Membudayakan pembelajaran sepanjang hayat; dan Mengukuhkan sistem penyampaian KPT.

Three aspects: Client quality what students and employers want from the service. Professional quality whether the service meets needs as defined by professional providers and whether it carries out techniques and procedures which are believed to be necessary to meet clients needs. Management quality the most efficient and productive use of resources within limits and directives set by higher authorities. Ovretveit (1992) Quality Indicators Aspects of quality in higher education

Three categories: Academic standards of courses what the students demands of the students, the extent it meets staff needs. Teaching quality staff training, appraisal systems, teaching evaluation, reports from external examiners, patterns of employment of graduates, students feedback. Student achievement completion rates, class of degrees obtained. ONeil (1994) Quality Indicators Judging quality in higher education

The need for competent staff Management responsibility Quality policy Quality manuals Quality planning Quality Indicators Issues to be addressed

HEIs worldwide have undergone reform to improve quality HEIs have implemented systematic and formalized quality assurance processes to achieve greater efficiency and accountability Establishment of quality models and organizations to audit and review university performance Institutional and national quality models and performance indicators are vital components to raise the standard of HEIs Quantitative performance indicators are used to provide international comparisons Quality Indicators Performance cultures in higher education

To ensure education provided by HEIs equips students for employment and provide the country with a highly skilled workforce that support economic growth. To contribute to educational, social, and political values. Quality Indicators Rationale for performance indicators

To monitor own performance for comparative purposes. To facilitate the assessment and evaluation of institutional operations. To provide information for external quality assurance audits. To provide information to the government for accountability and reporting purposes (Rowe, 2004). Quality Indicators Purposes of performance indicators in HEIs

Ensure accountability for public funds Improve the quality of higher education provision Stimulate competition within and between institutions Verify the quality of new institutions Assign institutional status Underwrite transfer of authority between the state and institutions Facilitate international comparisons Ref: Chalmers (2008) Quality Indicators The use of performance indicators in HEIs

Simple indicators expressed in absolute figures and are intended to provide an unbiased description of a situation or process. Performance indicators imply a point of reference; for example, a standard, objective, assessment, or comparator, are relative rather than absolute in character. Involve value judgements. General indicators externally driven and are not indicators in the strict sense; they are frequently opinions, survey findings or general statistics. Ref: Chalmers (2008, quoted from Hanney, Henkel & Kogan, 1997) Quality Indicators Defining performance indicators

Currently there is no common definition of performance indicators. PI cannot be considered as facts, but are goal, value and context laden, and utilized in different ways depending on the performance model being used. PI are defined as measures which give information and statistics context; permitting comparisons between fields, over time and with commonly accepted standards. They provide information about the degree to which teaching and learning quality objectives are being met within higher education sector and institutions. Ref: Chalmers (2008, p.3) Quality Indicators Defining performance indicators

Input Process Output Outcome The types can be categorized as quantitative indicators and qualitative indicators. Ref: Chalmers (2008, p.3) Quality Indicators Types of performance indicators

define as those associated with the measurement of quantity or amount, and are expressed as numerical values. Input indicators Human, financial and physical resources in supporting institutional programmes, activities and services. Output indicators Output reflects the quantity of outcomes produced, including immediate measurable results, and direct consequences of activities implemented to produce results. Do not demonstrate quality of education, but quantities of outcomes. Ref: Chalmers (2008) Quality Indicators Quantitative Indicators

associated with observation based descriptions, rather than an exact numerical measurement or value. Relate to or involve comparisons based on qualities of non-numerical data such as policies and processes for assessing students learning, the experience, the content of a mission statement. Outcome Indicators Focus on the quality of educational program, activity and service benefits for all stakeholders. Insightful, meaningful and accurate since they are related to the objectives of higher education. Students are treated as customers. Ref: Chalmers (2008, p.5) Quality Indicators Qualitative Indicators

Process Indicators include the means used to deliver educational programmes, activities and services within the institutional environment. qualitative information on teaching and learning such as policies and practices. Considered as most practical, useful and appropriate measures of quality teaching and learning. Ref: Chalmers (2008) Quality Indicators Qualitative Indicators

Look at 13 process indicators Mission, Vision and Objectives Teaching and Learning Plans and Policies Teaching and Learning Indicators Internal and External Performance Funds for Teaching and Learning Organizational Unit Review Curriculum Review Assessment and Feedback Policies Graduate Attribute Statement Student experience Ref: Chalmers (2008) Quality Indicators Research conducted in Australia

Look at 13 process indicators Professional Development Appointment and Promotion Criteria Review of Academic Staff performance Recognition of Excellence in Teaching and Enhancing Student Learning Experience Ref: Chalmers (2008) Quality Indicators Research conducted in Australia

Look at four dimensions of teaching practice Institutional climate and systems commitment to the enhancement, transformation and innovation of learning. Measure student experience and level of satisfaction. Diversity relates to ethnic, cultural and socioeconomic diversity as well as students abilities, talents and learning approaches. Ref: Chalmers (2008) Quality Indicators Research in Australia Quality Teaching

Look at four dimensions of teaching practice Assessment the assessment tasks of student enrolled in the program of study. Look at the design, delivery and administration, provision of feedback, moderation, and review of assessment. Engagement and learning community student engagement, i.e. the student commitment and engagement with their own education. Also includes staff engagement. Ref: Chalmers (2008) Quality Indicators Research in Australia Quality Teaching

Study in Hong Kong and China Seven factors contributing to the quality of education.They are: 1 course content; 2 concern for students; 3 facilities; 4 assessment; 5 medium of instruction; 6 social activities; and 7 people.

Course content items The chance that your study fulfils your personal needs The appropriateness of requirements for your course The chance to develop your abilities and prepare for your career The quality of material emphasized in course The usefulness of the module components offered in your career development The usefulness of the module components in fulfilling your personal needs

Concern for student The availability of advisers from whom students can seek help The interest that student advisers take in the progress of their students The ability of advisers to help students develop their course plan The willingness of the university management to take the opinions of students The channels for reflecting students ideas to the management

Facilities The availability of quiet places to study in the university The cleanliness of most facilities used by students The amount and availability of library facilities The places provided for students to relax and lounge during the day The amount and availability of computing facilities The amount and availability of sports and recreational facilities

Assessment The chance that you do well if you work hard The appropriateness of the standard of modules offered The appropriateness of the assessment system Detailed lecture notes are distributed The amount of work required in most modules The amount of time you must spend studying to get a passing grade The likelihood of getting distinction if you work hard

Medium of instruction Lectures be conducted in Language understood by students Tutorials be conducted in language understood by students

Social activities The activities and clubs you can join in the university The social events that are provided for students in the university

People The friendliness of students and the opportunity to make close friends The chances to meet people with the same interests as you have

Conclusion of the study by Kwan & Ng, 1999 Hong Kong and Chinese students are very practical and only focus on study-related matters rather than social life in campus. Students in the States are more interested in campus life but the importance of facilities has not been mentioned. It seems that Hong Kong and Chinese students regard university education as an investment and thus stress course content and facilities.

DISCUSSION Jelaskan elemen yang dipilih oleh anda sebagai pelajar untuk dinilai dalam konteks kepuasan pelanggan. Apa kriteria yang diguna untuk penilaian indeks kepuasan anda sebagai pelanggan?

Peranan universiti dan ahli akademik (Universitys and academics roles) Perkembangan kurikulum mengikut keperluan pasaran (Curruculum development according to market needs) Penyelidikan, pembangunan dan pengkormesialan dalam sistem inovasi kebangsaan (Research, development and commercialization in the national innovation system) Kaedah pengajaran dan pembelajaran (Teaching and Learning Methods) Perluasan akses dan mobiliti pengetahuan (Accessibility and knowledge mobility) Quality Indicators Higher Education in Malaysia Challenges

Pekeliling Kemajuan Pentadbiran Awam Bilangan 2 Tahun 2005 Garis Panduan Bagi Mewujudkan Petunjuk-petunjuk Prestasi Utama Atau Key Performance Indicators (Kpi) Dan Melaksanakan Pengukuran Prestasi Di Agensi Kerajaan

Terminologi Petunjuk Prestasi Utama (KPI) ialah salah satu kaedah bagi mengukur prestasi perkhidmatan agensi-agensi Kerajaan Perkhidmatan Teras Bidang tanggungjawab agensi sejajar dengan visi (core business): dan misi agensi; Proses Utama Fungsi-fungsi di bawah perkhidmatan teras yang (core process): perlu dilaksanakan bagi menghasilkan perkhidmatan untuk pelanggan; Key Performance Petunjuk-petunjuk prestasi utama yang ditentukan Indicators (KPI): sebagai asas mengukur prestasi;

NKRA Education Crime (Public Safety) Corruption Low Income Households Rural Basic Infrastructure Urban Public Transportation

Globalisasi dan piawaian melalui pemeringkatan dan penarafan (Globalization and standardization through development and accreditation) Peluang guna tenaga (Resource utilization opportunities) Quality Indicators Higher Education in Malaysia Challenges

20 public universities 21 polytechnics 37 community colleges >400 registered private colleges 21 private universities and university colleges 11 local university branch campuses + 5 foreign university branch campus Areas for indicators: Academic staff Educational programs Student selectivity Educational resources Governance The method used will be peer review. Quality Indicators Malaysia - Current scenario

Different ranking approaches: League table each university is assigned a specific rank. Higher ranks indicate higher quality, lower ranks indicate lower quality. Quality criteria and indicators are used in this ranking methodology to assess universities. Each indicator such as research impact as the number of citations per faculty in the Thompson Scientific Database or teaching quality as in THES are given weight. This approach are applied to all universities assessed. Quality Indicators How are universities ranked?

Different ranking approaches: A ranking of individual disciplines or departments instead of whole institutions. A multidimensional concept of university quality instead of a one-size-fits-all approach, taking into account the diversity of academic institutions, missions and goals as well as language and cultural specifics. A separate measurement and presentation of single indicators. A presentation of ranking results in rank groups (top, middle, bottom groups) instead of league tables. Quality Indicators How are universities ranked?

Ranking serves several purposes: Responds to demands from consumers for easily interpretable information on the standing of HEIs. Stimulates competition among universities. Provides some rationale for allocation of funds. Helps to differentiate among different types of institutions and different programs and disciplines. Contributes to the definition of quality of HEIs within a particular country. (source: Berlin Principles of Ranking of Higher Education Institutions, 2006 http://www.che.de/downloads/Berlin_Principles_IREG_534.pdfhttp://www.che.de/downloads/Berlin_Principles_IREG_534.pdf) Quality Indicators Purposes of ranking of HEI

Purposes and goals of rankings: Be one of a number of diverse approaches to the assessment of higher education inputs, processes, and outputs. Be clear about their purpose and their target groups. Recognize the diversity of institutions and take the different missions and goals of institutions into account. Provide clarity about the range of information sources for ranking and the messages each source generates. Specify the linguistic, cultural, economic, and historical contexts of the educational systems being ranked should be aware of possible biases. (source: Berlin Principles of Ranking of Higher Education Institutions, 2006) Quality Indicators Berlin Principles of Ranking of Higher Education Institutions

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Design and weighting indicators: Be transparent regarding the methodology used for creating the rankings. Choose indicators according to their relevance and validity. Measure outcomes in preference to inputs whenever possible. Make the weights assigned to different indicators (if used) prominent and limit changes to them.

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Collection and processing of data: Pay due attention to ethical standards and the good practice recommendations articulated in these Principles. Use audited and verifiable data whenever possible. Include data that are collected with proper procedures for scientific data collection. Apply measures of quality assurance to ranking processes themselves. Apply organizational measures that enhance the credibility of rankings.

Quality Indicators Berlin Principles of Ranking of Higher Education Institutions Presentation of ranking results: Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed. Be compiled in a way that eliminates or reduces errors in original data, and be organized and published in a way that errors and faults can be corrected.

Quality Indicators Purposes of Quality Indicator System Colorado State, USA: Encouraging continuous improvement by institutions in achieving high levels of performance. Measuring institutional performance and accountability. Determining funding recommendations and the funding distribution for the higher education system. Build public support for increased funding for higher education.

Quality Indicators Some Quality Indicators Baccalaureate graduation rates Achievement scores of graduating students on various comprehensive examinations, tests, and /or professional specific licensure or certification examinations Graduates employed or continuing their education Institutional support expenditures administration expenditure, expenditures per student Undergraduate class size Faculty teaching workload

Categories of institutions APEX university Research intensive General The structure of Malaysian Qualifications Agency (MQA) One-stop center for institutions for registration and accreditation of courses Quality Indicators Current scenario

Accountability and Quality The concept of accountability and quality assessment in higher education is an international phenomenon In America, many regions are moving toward performance incentive funding. In Europe and Australia, the central government is directly involved in establishing quality mechanisms through: Quality control, Quality audit Quality assessment The agencies involved are like the Higher Education Quality Council and the Higher Education Statistics Agency (HESA) Quality Indicators

The focus of attention in performance indicators in the U.S. has been cost efficiency, scientific and technical education, management of intellectual property produced at universities, and undergraduate education. Less attention is paid to graduate education and research. Categories of costs for higher education: Education and general: faculty and staff salaries, equipment, libraries, administrative and basic academic computing, and certain capital or such as rent. Cost for sponsored research Costs of student living: room, board, clothing, laundry, entertainment, and etc. Cost of foregone earnings: While disengaged from the productive work force. Quality Indicators Accountability and Quality

The Movement in Setting Performance Indicators in Higher Education (U.S.A.) 1980s Era concerned with growth in enrollments and access was over Emerging issues include: Public accountability Quality Productivity Undergraduate education In 1986, all 50 states and the District of Columbia had developed initiatives to improve the undergraduate education

Shift from growth funding (formula funding) toward funding outcomes, results, and performance These efforts paralleled developments in Europe and Australia 1990s The development of performance indicators differs from that in 1980s From voluntary institutional improvement to a system of mandated public accountability By 1994, 18 states had developed indicator systems Greater centralization of authority Issue raised: Will the federal government assume greater centralized control of higher education through areas such as accreditation and financial aids by using a set of national goals and performance standards?

The Future of Higher Education The White Paper 2003 Higher education must expand to meet rising skill needs The social gap among those entering university remains too wide Many of our economic competitors invest more in higher education Universities are struggling to employ the best academics Funding per student fell 36% between 1987 and 1997 Universities need stronger links with business and industry

Reports of Institutional Effectiveness (EOIE) Virginias Public Institutions of Higher Learning Annual report to provide meaningful information on the academic quality and operational efficiency of Virginias public institutions To provide evidence of institutional effectiveness the extent to which institutions accomplish their missions and students achieve their educational goals.

Structure of the Reports (Five Points) Institutions mission The mission statement sets a vision for the institution and defines how it will get there. Institutional profile In-depth views of enrollment and projections of future enrollment. System-wide measures Include 14 performance measures focused on operational efficiency and factors associated with academic quality: Example: Classroom and laboratory space utilization, percentage of professionally accredited programs and etc.

Institution-specific measures Represent unique aspects of the mission that the college or university chose to highlight Core competency reports Explore student general education assessments in written communication and technology/information literacy.

Performance Indicators of California Higher Education, 2001 Describes the scope of the current set of indicators reported by the Commission, and highlights recent trends based on current information related to these indicators. This report are divided into five categories: Population Context, Fiscal Context, Student Preparation, Student Access, and Student Outcomes

Key Performance Indicators (KPIs) Key Performance Indicators are quantifiable measurements, agreed to beforehand, that reflect the Critical Success Factors of an organization. They defer depending on organization KPIs must: Reflect the organizational goals Be key to its success quantifiable

Curtins efficiency and effectiveness Key Performance Indicators (KPIs) Teaching and learning Effectiveness indicators Quality of graduates Quality of teaching Student progress and achievement Input Efficiency indicators Teaching and learning expenditure Student progress and achievement

EFFICIENCY & EFFECTIVENESS EFFICIENCY means: saving TIME, MONEY or EFFORT Efficiency measures the resources used to attain a certain level of output EFFECTIVENESS means how well the the job gets done. i.e. the quality of the output. Effectiveness measures the extent to which outcomes have been achieved

The End of Quality 6 th Quality in Higher Education International Seminar (Birmingham, UK) May. 2001 Three Major themes: 1.Has external quality review has its day? 2.Has control of quality been usurped by the market and by information technology? 3.Does the development of mass education necessarily mean the end of quality?

Transforming Quality 7 th Quality in Higher Education International Seminar (Melbourne) Oct. 2002 Three main themes: 1.To reconceptualise how higher education engages with access, employability and funding issues 2.What constitutes a high quality learning process and outcomes 3.How might quality evaluation be transformed to help improve the quality of the experience and of the learning?

8 th Quality in Higher Education International Seminar (Sheffield) May. 2003 Two major themes: 1.How does student feedback inform quality?To what extent do institutions need to adopt new procedures to make student feedback effective? 2.What does the White Paper encourage a closer link between quality and learning?

The First Session of the Regional Follow-up Committee for the World Conference on Higher Education (WCHE), 2- 3 November 2000, Kuala Lumpur Malaysia Recommendations to Member Countries: 1.Need for ongoing efforts to broaden access taking into account the disadvantaged groups (women and ethnic minorities) 2.Provide increased support for staff development and research 3.More participation of women in higher education particularly in decision making level

Indicators of Research Quality in Higher Education The vast majority of discoveries have been made in higher education environment (Dill, 1986). Review of literature on research productivity highlighted several indicators which include: Productivity dollars Productivity publications Peer evaluation

Productivity dollars The number of dollars generated by research was the most often cited measurement of success Those universities that are ranked higher, their faculty have are adept at obtaining research grants Productivity publications The number of publications is frequently used as an indicator of quality in research The research that is published is taken as an indication of its quality The types of publication which determine its quality: Journal articles, monographs, chapters, books Quality: reputation of publication in discipline, distribution of publication, refereed vs. non-refereed journals, invited chapters/papers

Output versus Impact factor of publications Output refers to how prolific the the research is producing acceptable articles/books Impact was determined by checking citations of the articles over a period of years OUTPUT & IMPACT FACTOR

Peer Evaluation Assemble a group of peers to review the research efforts and make a determination of the quality of those efforts The concerns of such approach include: The visiting group doesnt fully understand the work of the unit/individual being reviewed (especially when it is multidisciplinary) PEER EVALUATION

Curtins Efficiency and Effectiveness Key Performance Indicators (KPIs) Research and Development Effectiveness indicators Research Performance Index Research Quantum Comparison between Curtin and all Australian Universities Research Funding Research Publications

Efficiency Indicators: Research Performance Index Research Funding Research Publications Effectiveness measures the extent to which outcomes have been achieved Efficiency measures the resources used to attain a certain level of output Efficiency and Effectiveness Key Performance Indicators (KPIs)

Publications and Number of Ph.D. Graduates 200176 200289 2003112 Publications 20001,864 (319 international journals, 212 in local journals) 20011,815 (303 international journals, 204 in local journals) 20022,507 (496 international journals, 328 local journals) Malaysia was ranked #63 by MASTIC (Pusat Informasi Sains dan Teknologi Malaysia) in terms of production referred journals

Critical Success factors (Research) Wide academic base and facilities Graduate students Screening of IRPA application and monitoring Evaluation of research and innovation Research culture and administrative support Strong research networking Research cluster development Incentives & in-house competition as incubator

Steps to be Taken by Institutions to Promote Excellence in Teaching Define what they mean by excellent in teaching Having well-defined criteria about excellent teaching and standards for weighting and rating of teaching/research/service Weigh teaching more heavily Increase sophistication of teachers Promote excellent teaching, not just excellent teachers Not treat promotion as a separate issue (Gibbs, 1995)

Theoretical Definition for Excellent Teachers The character of the professor Values, personality, and social intelligence The knowledge of the professor Disciplinary and pedagogical understanding The actions of the professor Problem-solving behaviors The responses of the students Learning outcomes

Quality Teaching in Higher Education Flexibility in approaches to teaching and learning (including assessment) Good organization of subject matter and course, including relevance and coherence of content and planned teaching/learning activities Effective communication Knowledge and enthusiasm for subject matter and teaching Facilitation of learning through student interaction and active experience Respect for and positive attitude toward students Critically reflective orientation to teaching including effective use of feedback to guide learning and improve teaching Appropriateness and fairness in assessment and grading Reeders, E, & Marshall, H. 1996

GOOD TEACHING: THE TOP TEN REQUIREMENTS By Richard Leblanc, York University, Ontario, 1998. 1.Good teaching is as much about passion as it is about reason. 2.Good teaching is about substance and treating students as consumers of knowledge. 3.Good teaching is about listening, questioning, being responsive, and remembering that each student and class is different. 4.Good teaching is about not always having a fixed agenda and being rigid, but being flexible, fluid, experimenting, and having the confidence to react and adjust to changing circumstances

6.Good teaching is also about style This is very important -- good teaching is about humor. 7.Good teaching is about caring, nurturing, and developing minds and talents. 8.Good teaching is supported by strong and visionary leadership, and very tangible institutional support -- resources, personnel, and funds. 9.Good teaching is about mentoring between senior and junior faculty, teamwork, and being recognized and promoted by one's peers. 10.At the end of the day, good teaching is about having fun.

Quality in College Teaching: A Research Approach Flexibility in approaches to teaching and learning (including assessment) Good organization of subject matter and course, including relevance and coherence of content and planned teaching/learning activities Effective communication Knowledge and enthusiasm for subject matter and teaching

Facilitation of learning through student interaction and active experience Respect for and positive attitude toward students Critically reflective orientation to teaching including effective use of feedback to guide learning and improve teaching Appropriateness and fairness in assessment and grading

QUALITY INDICATOR 13 APRIL 2013. Learning outcomes: Explain the meaning of quality indicators Describe the characteristics of quality indicators Synthesis.

Documents

quality heis

meaning of quality indicators

quality indicators rationale

management quality

professional quality

client quality

national quality models

quality of new institutions