The Hong Kong University of Science and Technology Teaching …qa.ust.hk/tlq_report/chi_version/annex_c/index.pdf · The Hong Kong University of Science and Technology Teaching and

Annex C (a)

37

The Hong Kong University of Science and Technology

Teaching and Learning Quality Process Reviews

Self-Evaluation Document

July 2002

38

Undergraduate and Taught Postgraduate Education

Introduction

In March 2000, the University Council reaffirmed that the Mission of HKUST was identical with the “Objects” incorporated in The Hong Kong University of Science and Technology Ordinance (Chapter 1141, Laws of Hong Kong); that is,

(a) to advance learning and knowledge through teaching and research, particularly (i) in science, technology, engineering, management and business studies; and (ii) at the postgraduate level; and (b) to assist in the economic and social development of Hong Kong.

While there has never been any question that HKUST was intended to be a research-

intensive university, it is no accident that, in this statement, “learning” and “teaching” appear before “research.”

This statement of the mission does not provide very specific guidance as to the University’s objectives in the areas of learning and teaching. The Council, however, supplemented its year 2000 adoption of a Mission and Vision statement with five specific Objectives, of which two are particularly relevant to the educational mission.

The first of the five Objectives relates to students:

To give all students, undergraduate and postgraduate alike, a broadly based university experience that includes: superior training in their chosen fields of study; a well-rounded education that enhances the development of their creativity, critical thinking, global outlook, and cultural awareness; a campus life that prepares them to be community leaders and lifelong learners.

An objective relating to campus culture describes the type of environment in which

learning and teaching are to take place:

To provide an open environment and atmosphere conducive to the exchange of knowledge, views, and innovative ideas among students, faculty, staff, and visiting scholars.

The management philosophy and framework of HKUST are encapsulated in the

preamble to the terms of reference of the University Administrative Committee (UAC), as follows:

The Council of the Hong Kong University of Science and Technology is the "supreme governing body" of the University. Academic policies are determined by the Senate. The detailing and implementation of academic and administrative policies are the responsibility of the University's Administration. Within the Administration proper management entails the assignment of clear responsibility for specific results to particular individuals within the organization and the delegation to them of the necessary authority including resources to carry out their responsibilities. In the context of the

39

administration of a university where the knowledge and expertise required for proper decision-making are widely disseminated throughout the organization a high degree of delegation and decentralization is not only appropriate but necessary.

Institutional Level

The challenge in deriving a framework for education quality work (EQW) at HKUST was to clearly articulate expectations that could be used as a basis for accountability within the line management structure. The President is both the chief executive and chief academic officer of HKUST, but the officer delegated primary authority for administration of educational programs and assurance of their quality and improvement is the Vice-President for Academic Affairs (VP-AA). The office of the VP-AA supports teaching and learning in three ways:

• It coordinates, develops, and monitors the implementation of academic programs and policies.

• It serves as the Secretariat for three Senate committees that have a major role in education quality work: the Committee on Teaching and Learning Quality (CTLQ), the Committee on Undergraduate Studies (CUS), and the Committee on Postgraduate Studies (CPS).

• It supervises a number of academic support units that assist in the delivery of teaching and learning, including the Library, the Language Center, and the Center for Enhanced Learning and Teaching (CELT).

In addition, the VP-AA is the final reviewer of applications for substantiation and promotion of academic staff, and so plays a major role in quality assurance of teaching staff. At the institutional level, quality assurance and improvement are supported in several ways.

Policy Guidance The office of the VP-AA has developed and maintains extensive documentation on policy relating to academic programs and their implementation. The latest versions of all policies and procedures are available to staff and students in an on-line Academic Programs Manual at

http://www.ust.hk/~webaa/APM/home.htm.

Included in the APM are detailed instructions for development, review, and implementation of academic programs and courses. For example, an undergraduate program template defines certain required elements of all degree programs that relate to the objective of providing a well-rounded education. It also specifies an approval process that requires a review of an initial “early warning” proposal by the relevant Senate committee, which can (and does) raise questions that must be addressed in the final proposal that will be sent via the same committee to the Senate for approval. It provides guidelines for preparation of the final proposal that require information on program aims and objectives, need and demand for the program, and the nature of consultations and surveys done prior to proposing the program. Each individual course in a program is also subject to review and approval by the relevant Senate committee, and must be supported by information on learning objectives, course content and teaching strategies, and assessment of student learning.

EQW Resources Institution-wide resources to support teaching and learning quality assurance have been consolidated on-line at our Teaching and Learning Quality portal.

40

http://publish.ust.hk/vpaao/tlq/index.html

This includes access to such information as the outcome of first round of TLQPR, the framework for the second round, and suggestions for good practice. Policy elements related to teaching and learning quality assurance will soon be consolidated in an on-line QA Manual to be published on this portal.

Resources for teaching improvement are provided by CELT, an institution-wide support unit reporting to the VP-AA. A description of its services, and a wealth of information for staff and students, is available on the CELT web site at http://celt.ust.hk/.

Quality Assurance Evidence Apart from a policy and coordinating role, there are important institutional contributions to the information needed in teaching and learning quality assurance activities. Under direction from the office of the VP-AA, CELT administers the comprehensive end-of-semester course evaluations, and the office makes the summary results available to the University community on the web. The details of the evaluations are also provided to Departments and passed on to academic review committees considering matters of contract renewal, substantiation, and promotion. The Academic Affairs office also undertakes ad hoc studies of matters related to EQW. For example, as secretariat for CTLQ, a survey of staff with experience as academic reviewers was conducted to clarify perceptions of the nature and significance of various kinds of evidence of teaching quality. A survey of all teaching staff on their perceptions of the teaching and learning environment was commissioned. Several studies have been carried out to benchmark student assessment at HKUST against other local and overseas universities.

Monitoring EQW The Senate assigned to CTLQ the role of monitoring teaching and learning quality assurance activities through the annual collection of reports from the Schools. Based on an analysis of these reports, CTLQ identifies good practices that it reports to the Senate and disseminates throughout the University for consideration and emulation by other units. Where problems are identified, the CTLQ and its secretariat follow up with the School to address them. Following release of the framework for the second round of the TLQPR, the format for the annual reports was modified to require an EQW Profile for each Department and School, as well as for the Language Center. This gives us a comprehensive view of the quality assurance framework and methods for addressing the five domains of EQW for each unit.

School Level

The Deans are accountable for teaching and learning quality assurance within the four Schools. Each School has a School Board that plays a role similar to the Senate with respect to academic matters pertaining to the School. The School Boards for Science, Engineering, and Business and Management have committees similar to the Senate’s CUS and CPS. In the case of the School of Business and Management, there are separate committees for taught and research postgraduate matters. The School of Humanities and Social Science, which has no undergraduate major programs, has only a postgraduate studies committee at the School level; however, both of the two Divisions of the School have an undergraduate studies committee to handle the quality assurance and other aspects of their important contributions to the undergraduate education of the students from other Schools.

Each School has one or more committees of external advisors. The School of Science Advisory Committee consists of 12 external reviewers (two members each for the five

41

Departments and the AMCE program) who are world-renowned academics/experts in the specific areas of studies. The School of Humanities and Social Science also has an Advisory Committee of distinguished scholars, mostly from overseas. The School of Engineering has both a School and Departmental Advisory Committees that draw on local and overseas advisors from both academic and industry backgrounds. The School of Business and Management has an Advisory Committee of overseas academics and a Corporate Advisory Board of business leaders from Hong Kong and overseas.

Departmental Level

By mandate of the Senate, each Department has a Student-Staff Liaison Committee (SSLC) that provides a channel for students to report on their experiences and concerns regarding their teaching and learning programs and an opportunity for the Department's staff to brief students on its actions and plans for improving the curriculum and delivery of the program. Minutes of SSLC meetings are reported to the CTLQ via the Secretariat in the office of the VP-AA. A few Departments choose to have separate SSLCs for specific student target groups.

Other mechanisms for carrying out education quality work in the Departments vary according to the history and culture of the Department. Most have Undergraduate and Postgraduate Committees paralleling the CUS and CPS of the Senate. In addition, a number have a Curriculum Committee, and in recent years about one-third has introduced committees that focus specifically on teaching and learning quality issues. A few small Departments make use of a Department Committee or Meeting, with all teaching staff as members, to address the same set of issues and concerns.

Coherence

The philosophy of delegated authority and accountability leads, as noted in the brief discussion of Departmental QA mechanisms above, to a variety of approaches. This is the result of a deliberate choice not to impose a uniform structure that might encourage a compliance culture. Coherence of the overall system is achieved, in part, by the holding the line management, from the VP-AA through the Dean to the Department Head, accountable for results. One practical means through which coherence of the system is maintained is the interlocking of the various levels through membership of the responsible bodies. For example, at the institutional level, each of the Committees (CUS, CPS, and CTLQ) has two representatives from each School, one appointed by the Dean (and invariably an Associate Dean), and one elected by the School Board. The Secretary of each key committee is one of the Associate Vice-Presidents, but the committee is chaired by a senior faculty member appointed by the President. Student representatives on each committee are nominated by the Students’ Union.

Administrative follow-up is facilitated by the fact that the VP-AA holds a monthly Deans Meeting at which issues related to teaching and learning can be raised and discussed. The close working relationship between the office of the VP-AA and staff in the Deans’ offices ensures that problems are resolved and decisions implemented.

New policies and initiatives related to teaching and learning ate developed after consultation with faculty and staff. Depending on the nature of the issue involved, this may be through the organizational channels or directly with the staff. In the former case, Departments are asked to hold their own discussions and forward comments and suggestions

42

to the Dean, who adds the perspective of the School before forwarding the input to the VP-AA. In other cases, faculty and staff have been asked to comment directly on a consultation paper, which is usually made available to them on the web and often discussed at an open forum.

External Review and Accreditation

A major difference among the Schools in teaching and learning quality assurance is the role played by external accreditation. The School of Engineering, whose programs enroll 40% of all HKUST students, and the School of Business and Management, which enrolls a further 35%, both rely on external accrediting bodies as an integral part of their quality assurance framework. Thus, about three-quarters of all students at HKUST benefit from the additional scrutiny of their programs by independent external experts in light of internationally validated standards.

The School of Engineering, like other engineering programs in Hong Kong, has its programs accredited by the Hong Kong Institution of Engineers (HKIE). As a member of the Washington Accord, the HKIE is able to ensure that our programs meet internationally recognized standards for the engineering professions and are on a par with those of other world-class engineering schools. During the accreditation visit, the accreditation team assesses qualitative factors such as facilities, intellectual environment, morale, professional attitudes and the quality of staff, students and graduates. The visit also includes reviews of examination papers, laboratory instructions and reports, project reports and other materials demonstrating student performance; and meetings with the senior management of the University, the Dean, program leaders, academic and support staff, students, alumni, advisory committee members and employers of graduates.

In April 1999, the School of Business and Management became the first in Asia to be granted accreditation by AACSB. This was also the first time that full accreditation had been given to a school less than ten years old. The School’s Accounting program, in addition to its separate AACSB recognition (the first ever granted to an accounting program outside North America), is also accredited locally by the Hong Kong Society of Accountants. Under the AACSB criteria, institutions are expected to

• Guide educational delivery by a carefully constructed mission • Select and support students to produce outstanding graduates • Deliver degree programs with qualified faculty • Structure learning through relevant curricula • Contribute to knowledge through research and scholarship

In July 2000, the School also became the first in Asia to receive EQUIS accreditation

from the European Foundation for Management Development. The EQUIS quality label is the result of an international system of strategic audit and accreditation designed by Europeans for the assessment of institutions in widely different national contexts. The EQUIS system is deliberately designed to promote continuous improvement at all levels.

Issues Arising from the First Round of TLQPR

The issues covered below are those that were originally addressed in the TLQPR Progress Report submitted on 2 March 1998, and in view of the space limitations of the Self-

43

Evaluation Document, only an update will be provided here. A common theme for all of these areas is continuous improvement and development as we work towards establishing a distinctive teaching and learning quality culture at HKUST.

Accountability

The issue of accountability of teaching and learning quality within the line management framework of HKUST was a recurrent theme in the first TLQPR Report. Means to address the issue of accountability in assessing teaching performance, curriculum design, and curriculum management were described in the Progress Report. Our basic strategy in strengthening accountability has been incremental improvement of policies and processes in consultation with all levels of the organization, in order to promote development of a home-grown teaching and learning quality culture. The establishment and evolution of the Senate Committee on Teaching and Learning Quality (CTLQ) illustrates this approach. The membership of CTLQ was initially drawn from the members of the CUS and CPS elected by the school Boards, in order to being to CTLQ the experience of these members in dealing with the operational matters handled by those committees. It also served to ensure that the quality assurance perspective was enhanced in the operations of the CUS and CPS.

Unlike the CUS and CPS, which have regular business on curriculum and program matters brought before them for decision, CTLQ had to define its own role. Its initial tasks were to monitor compliance with new policies adopted by the Senate following the first round of TLQPR, and it was also assigned the task of receiving reports from the Schools on their teaching and learning quality assurance activities—what we would now call their education quality work. When CTLQ was first established, the academic units in the University were already in the process of producing reports for an Academic Review, aimed at comparing the status of programs with their original plans. These were comprehensive reports, but as they covered teaching programs, the initial annual report on teaching and learning was extracted by the Secretariat from the documents prepared as part of this planning exercise. This gave CTLQ a broad picture of the status of teaching and learning, and a clearer idea of what needed to be included in future annual reports.

In the following year, CTLQ issued guidelines to the Departments and Schools for preparation of their annual reports, based on the framework for TLQPR as established for the first round. Departmental reports were sent to the Deans’ offices and incorporated into School reports to CTLQ. This exercise provided a much more detailed picture, but also highlighted a weakness in the accountability system. For the most part, the School reports were just summaries of information from the Departments (with the full Department reports attached), and gave little insight into the role of the School administration in assuring quality.

The next round of reports used similar guidelines as to content, but required that each School prepare a report to the CTLQ, without forwarding copies of the Departmental submissions. Schools were asked to provide more analysis of Departmental activities, and to provide more information on activities at the School level. The result was a set of much more useful and thoughtful submissions.

In the most recent year, 2001-02, using the new TLQPR framework as a basis, CTLQ decided to ask for submissions from both Departments and Schools in order to have a comprehensive EQW Profile for the institution. However, to reinforce the importance for the Schools to demonstrate responsibility for their own role in assuring teaching and learning quality, the Departments were asked to submit their reports directly to the CTLQ Secretariat.

44

They copied their submissions to the Schools, but only shortly before the deadline for the submission of the School Profiles. This ensured that the School submissions would focus on the School’s role rather than summarizing the work of Departmental units.

During the period since its establishment, CTLQ has developed its own identity and agenda, has conducted a number of important studies of teaching and learning issues, and proposed new policies to strengthen the quality assurance framework. In recognition of its new maturity, the Senate has agreed that in future its members will be elected specifically to this committee by the respective School Boards.

Priorities for Teaching and Research

We have continued and expanded the efforts noted in the Progress Report. As one example, the Center for Enhanced Learning and Teaching (CELT), newly established at the time of that Report, has developed a high profile within the institution. Based on a survey of all academic staff, roughly half have attended a workshop or other event organized by CELT. Its promotion of web-based tools for teaching and learning has encouraged the engagement of many faculty members in activities related to their teaching. The 26 workshop sessions for instructors, demonstrators and TA’s on how to use WebCT and LearningSpace, and about online learning in general, attracted total enrolment of 344 staff. More than 200 on-line courses of various levels of sophistication have been developed.

Note that the focus on technology-enabled teaching was not primarily motivated by pedagogical issues, but by the alignment of these approaches to the natural interests of a large segment of the faculty. This strategy of engagement is intended to encourage a healthy balance between teaching and research in the day-to-day concerns of the academic staff.

In the process, we have discovered that the issue of time on task, as raised in the TLQPR report, may not be the key issue. One of our teaching staff, in describing his development from an inexperienced teacher to a winner of the Michael Gale Award for Distinguished Teaching, noted that the additional time he spends in doing a better job as a teacher has eliminated the sense of frustration and inadequacy he used to experience in the classroom, leaving him mentally fresher and more productive as a researcher. This message, that professional satisfaction as a teacher can reinforce and support other areas of professional performance, is one that we will be trying to get across to our entire faculty.

Research will continue to be a high priority at HKUST, given the mission and vision articulated by the University Council. A survey of staff with experience as members of academic review committees clearly indicates that most give greater emphasis to research than teaching in arriving a their recommendations. Nevertheless, the survey of the teaching staff indicates a clear understanding on their part that the standard for what is regarded by reviewers as effective teaching is not low, and that achieving it is important to their careers.

Sharing Best Practice

As noted in the Progress Report, one of the functions of the Committee on Teaching and Learning Quality, as given in its terms of reference, is to “disseminate examples of good practice and make recommendations to the Schools and Departments based on its review of teaching and learning quality.” CTLQ has identified a number of such practices, and reported them to the Senate. A Letter of Recognition was sent from the Committee to the academic units concerned in appreciation of their work, and copies were disseminated to all units. A

45

collection of these recommended practices has been posted on the Teaching and Learning Quality portal referenced above.

Another major effort aimed at the dissemination of best practice was the Teaching and Learning Symposium organized on campus by CELT in December 2001. Titled “Teaching Innovations: Fostering a Creative and Collaborative Learning Environment,” the symposium highlighted innovative teaching approaches (not all technology-based), and presented awards for individual and group efforts to promote teaching and learning quality. Papers and presentations, in both printable form and streaming video, have been made available through the CELT website.

Organizational Culture

Changes in organizational culture are less easy to document than the specific organizational and policy changes that are intended to induce them. It is particularly hard for those immersed in the culture to judge the extent and nature of these changes, and we look forward to the TLQPR Panel’s report for an external perspective.

We do believe there has been continued development of an understanding of the importance of teaching in the HKUST culture. Not all aspects of this are entirely positive: a survey of all teaching staff on the environment for teaching at HKUST reveals a great deal of apprehension on the part of staff about the pressure they feel from this emphasis, which has not been accompanied by a reduction in the pressure to be productive scholars. There remains more to do to achieve a “creative and collaborative learning environment” or the kind of campus culture envisioned in the Council’s objective, but we believe that teaching and learning is now firmly embedded in the mix of concerns that will shape this development.

Reducing Variance

Relying on the first TLQPR Panel’s observation that “diversity can represent a source of strength,” we have consciously decided to permit individual units a fair degree of independence in how they structure their internal quality assurance mechanisms. The approach has been to build on the existing and familiar structures rather than to impose a uniform structure from above. However, we have promulgated policies that set expectations for the results produced by these mechanisms, and have required the adoption of some new ones where they have been lacking.

Monitoring of the operation of these processes has been strengthened by the creation of CTLQ and the role of the VP-AA’s office as its Secretariat. In particular, more attention has been paid to internal and external benchmarking.

Curriculum Design

As this is one of the domains discussed below, an update on current practices need not be included here. It may be appropriate to note at this point that there have been many curricular changes introduced since the first TLQPR that respond to societal needs. One particularly successful example was the introduction of a new teacher-training program, with a curriculum developed and delivered jointly by HKUST and the Hong Kong Institute of Education. Examples of other curriculum developments aimed at expanding options for students include the introduction of additional minor programs and policies on the award of dual degree titles.

46

Teaching Assistants

The training program that was introduced to improve the performance of postgraduate teaching assistants (TAs) has continued to develop, and has been supplemented by systematic performance evaluation and a scheme for recognition of good performance. This integrated program will be described in more detail in the supplement on accomplishments.

Student Input and External Feedback

As noted in the Progress Report, the Senate mandated all Departments to set up Student-Staff Liaison Committees. Reports of the meetings of these committees have been sent to CTLQ via the office of the VP-AA and indicate that the discussions deal with significant concerns. In the most recent round of reports, there is good information on plans to address the concerns identified. For future reports, Departments will be asked to include follow-up information on the effectiveness of their efforts to deal with these concerns.

Adding the accreditation of programs in the School of Business and Management to program accreditation in Engineering has made HKUST unique in Hong Kong as far as the percentage of its programs subject to periodic external scrutiny is concerned. The accreditation processes for business, accounting, and engineering programs provide valuable input to the objectives, design, and standard of assessment of all these programs. Advisory committees, and less formal advice from distinguished visiting faculty, provide important input on program and curriculum matters. Direct feedback from employers has been difficult to obtain, but we do systematically gather information on the impressions of our students gained by employer recruiters who visit the campus, and feed back the results to the Departments concerned. Many of the questions in our graduate employment survey, which is shared with all units, also provide insights into program outcomes.

Recognition and Incentives

The University-wide Michael Gale Medal for Distinguished Teaching has been supplemented by awards at the School level in all four Schools. These are described on the Teaching and Learning Quality Portal. The office of the VP-AA has also worked closely with the House II Student Association to raise the visibility and enhance the prestige of the annual “Best 10 Lecturers Election” that they first introduced in 1996. With the help of an on-line web-based polling system, the student participation rate in the election has significantly increased. The award ceremony in the Atrium, with the VP-AA personally making the awards to the students’ choices of their best teachers, celebrates the importance we attach to the student-teacher relationship.

The Domains of Education Quality Work—An EQW Profile

As reported above, in 2001-02 each Department and School was required to provide an EQW Profile. This document includes a section on the unit’s quality assurance framework, five sections addressing the domains of education quality work, and a section on support for research postgraduate education. The units were provided with guidelines that included information on the TLQPR Framework, and a series of questions relating to the issues identified in the Framework.

47

While we regard the submissions as satisfactory and quite useful, they are uneven in coverage and depth. This was not unexpected, since the approach and terminology differed from previous years, but there are several follow-up activities under way to improve them. First, the submissions for each domain have been sent to designated members of the Senate to provide an additional independent source of advice on what is being done well and what needs improvement. Second, we will choose an example for each section of the Profile from a unit that is judged to be doing well in that area, to form a “Model EQW Profile” as a means of sharing best practice. Academic units will be asked to make reference to this, and invited to update their own Profiles where possible and appropriate. We will provide access to the full collection of Profiles on the network, and are considering whether it is desirable to allow units to make updates on-line. The Model EQW Profile will itself be updated as better practices evolve and emerge.

Design of curricula

The assurance and improvement of quality in curriculum design is guided by policies and guidelines from all levels of the organization, and informed by inputs from a broad array of internal and external sources. At the institutional level, a framework for curriculum design provides coherence to curricula, and assures that overarching educational goals are incorporated in each degree program. In particular, the commitment to providing a well-rounded education for all undergraduates is embodied in the requirement that about 24% of the course work in every program be devoted to general education, which must include exposure to the disciplines of the humanities and social science and additional course work outside the School in which the program resides.

The delivery of superior training in a particular field of study is facilitated by designation of courses as core or required courses, designated electives, and free electives. All students in a program must successfully complete all required courses. Core courses are required courses that must be taken in a designated semester of study. This structure ensures a common base of knowledge and skills for all graduates of the program. Designated electives provide a degree of flexibility for students to pursue particular interests within the program; this may involve choosing courses from a specified list or simply taking courses of an appropriate level in a specified discipline. Free electives are included in most programs to provide an additional degree of flexibility for students to broaden the base of their education. Based on this framework, each program has a template and students are provided with a recommended course of study that fits the template. To assist students to meet the program requirements, students in a particular semester of a program are pre-registered in the appropriate core and required courses.

New programs and revisions to existing programs must meet the requirements of this framework, and this is reviewed at the School and University levels before the proposals are sent to the Senate for discussion and approval. Substantive program changes are not uncommon, as Departments adapt their curricula to changing needs and advances in knowledge. Most such program decisions are initiated at Department level, although in the School of Business and Management, where the BBA degree for all majors has broad-based business education at its core, program initiatives may also originate at the School level. At the taught postgraduate level, where most new programs and many existing programs are interdisciplinary, the Schools may also play a more active role.

In addition to providing a level of review of Department proposals, the Schools also coordinate and encourage curriculum review and development in the Departments. The

48

School of Science, for example, has coordinated a review of all existing course syllabuses. The School of Business and Management, as already noted, is closely involved in the curriculum design process to ensure that its own objectives for the BBA are met: to develop successful business professionals with a broad-based business and management education; global perspective; strong language and communication skills and the ability to integrate theory and practice.

Input to curriculum design comes at both the School and Department level, with the emphasis dependent upon the particular style and nature of the disciplines within the School. The School of Science relies more on Departmental judgments, but Departments seek advice from students, alumni, academic advisors (both through members of the Advisory Committee and on an ad hoc basis), and where appropriate, industry. A major source of student input is through the Departmental Student-Staff Liaison Committees, but the School also has such a committee at the School level. An illustration of the process is provided by the Department of Biochemistry, whose Teaching Quality Assurance Committee holds an annual meeting of all teaching staff to identify areas considered of major historical importance, current “hot” research areas, and subfields likely to become important in the future; and to consider how to incorporate these into the curriculum. It also makes use of input from employers in Hong Kong’s fledgling biotechnology industry, which has, for example, resulted in more emphasis on more practical applications of laboratory science in the final year curriculum.

The curriculum framework for each discipline in the School of Engineering is informed by the requirements of professional accreditation. In addition, the Departments actively seek input from academic and industry advisors, both local and overseas, international professional bodies, employers, alumni and students. The Department of Mechanical Engineering provides one example. It consults on curriculum matters both an Academic Advisory Board, consisting of three Mechanical Engineering professors from world-class universities, and an Industrial Advisory Board, composed of eight members who are senior industrialists in Hong Kong.

The senior administration of the School of Business and Management maintains regular contact with its Corporate Advisory Board, a group of senior executives from both local and overseas firms. The School also has an Academic Advisory Committee comprised of Deans and former Deans of major business schools in the U.S., Canada, and Europe. The two groups meet together during the biennial meetings of the Academic Board in Hong Kong. Departments within the School have their own similar mechanisms for keeping the curriculum up-to-date and relevant. For example, the Department of Accounting gets feedback from employers through a variety of mechanisms, including the Department Advisory Board, the department’s Internship Program, the Accounting Students’ Society Advisory Board, and annual receptions with Big 5 accounting firms.

At the undergraduate level, the School of Humanities and Social Science does not have majors of its own, but offers a range of offerings to support the educational objective of providing a broad-based education. To give more structure and expand opportunities for students to pursue their interests in these fields, both Divisions have introduced minor programs; in fact, the proposal to do so led to adoption of new policies and procedures governing minor programs that have resulted in development of minors in other Schools. The School has been particularly sensitive to input from students; for example, suggestions that the Division of Social Science offer more courses about Hong Kong and the environment have resulted in additional course offerings on Hong Kong politics, Hong Kong culture, Hong Kong society, and environmental issues.

49

The objective of enhancing students’ creativity and critical thinking has not been neglected, and has been a subject of discussion at many levels of the organization. A number of Departments have introduced courses with this as one of the course objectives. For example, in the School of Science, the Physics 180-280-380 sequence of one-year seminar and tutorial courses brings students and faculty together in small group settings to discuss a variety of concerns. Most Engineering Departments have a three-year sequence of courses on Academic and Professional Development, and the School has developed a new course ENGG 311, Engineers in Society. Courses of this type have a focus on career-related issues, but provide an opportunity to stimulate students’ independent thinking, improve their communications skills, and engage them in “outside-the-box” discussions.

Final year projects, a feature of the curricula in the School of Engineering, help students acquire team skills, require them to think independently and creatively, and sharpen their communication and presentation skills. An annual display of final year project results in the Academic Concourse reinforces students’ pride in their accomplishments and serves as an example to other students. The School of Science is currently working on an adaptation of the final year project model suitable to its needs.

In an institutional effort to address these learning issues head on, CTLQ gave a mandate to the Center for Enhanced Learning and Teaching (CELT) to coordinate and initiate multidisciplinary, non-traditional courses to help students broaden their horizons, develop their integrated and critical faculties, and be creative. Courses developed and offered to date include Effective Learning & Thinking Skills, The Creative Process, Arts & Technology, and Multimedia Communication Skills for Internet Applications. The last of these, despite its technical emphasis, aims to develop students’ mastery of presentation and communication skills, and focuses on creative applications of the technology. The new offerings have attracted great interest from students, and current resources are far short of allowing us to meet the demand.

Design of teaching and learning processes

Department and School curriculum, program, and quality assurance committees have become increasingly involved in discussions of pedagogical issues and sharing of good practice. In addition, many Department and School-wide meetings and forums have been held to discuss various aspects of teaching and learning in greater depth. The Department of Computer Science is a typical case: it has conducted a series of teaching forums to share teaching tips—the Department Head conducted the first such forum, and other faculty led later ones. Teaching tips have also been the topic of several departmental retreat discussions.

The Center for Enhanced Learning and Teaching (CELT) has a major institutional responsibility for supporting and encourage the efforts of teaching staff to innovate and improve. Its workshops and seminars have been well attended, with roughly half of the teaching staff having indicated in a survey that they had attended at least one. The schedule for the Spring semester 2002 is typical: there were about 20 workshops, ranging from very practical topics such as use of Front Page and WebCT to policy discussions, such as “Teaching and Learning after the Education Reform in Secondary Schools - Challenges to Universities,” to pedagogical sessions such as the workshop on “Problem-based and Case-based Cooperative Learning.”

CELT also provides teaching development services, ranging from ad hoc consultation to co-development of instructional projects for on-line and classroom applications. It has

50

played a significant role in stimulating the development of on-line courses on campus, and adoption of the Personal Response System (PRS) as a tool for increasing interaction in lecture situations. (PRS is a hardware/software system developed at HKUST that is currently being used in both higher and secondary education institutions in Hong Kong and overseas.) The Center’s IDEAS portal is a rich source of resources on these and other promising pedagogical innovations.

Faculty in many Departments have carried out teaching development projects, supported by internal funds, external Teaching Development Grants, and a combination of the two. A Symposium on Teaching and Learning, organized by CELT in December 2001, featured more than 30 presentations, highlighting the results of such projects and assessing the effectiveness of new approaches. A full record of the Symposium, including both hard copy of the papers and streaming video of the sessions, is available on the CELT web site.

A focus on measurement of learning outcomes is a recent development. To stimulate Departmental discussions and attention, CELT held two seminars on this topic and related assessment issues on 7 and 8 May 2002. Each Department nominated a representative to attend one of the two sessions, which were conducted by Dr Mavis Kelly. CELT has recently added resources on “Assessment for Learning” to its IDEAS portal.

Design of student assessment and use of assessment results

Since classes began at HKUST in October 1991, the use of continuous assessment has been an almost universal feature of its educational programs. Under this system, a student’s course grade (which is primarily a summative assessment) is based on performance on a variety of tasks, which typically include a final examination, one or more mid-term tests, quizzes, essays, reports, assignments, presentations and other forms of classroom participation, and laboratory exercises, although not all these are likely to be present in each course. The assessment of these tasks can have formative value, as well as contributing to the summative evaluation of the course grade. Experience has shown that a summative component, no matter how small, is necessary to ensure student completion of the tasks, even if (such as in the case of homework assignments) the main purpose is to provide the student with feedback on his or her progress.

Various aspects of the assessment system also play a formative role with respect to the delivery of the course. Instructors can, and do, make adjustments to their presentation of course material in response to assessments showing that students are having difficulty with particular concepts. Even performance on final examinations can provide significant feedback to the program. For example, in the Department of Biochemistry, the TQAC discusses assessment results with instructors, with a view to identifying strengths and weaknesses in knowledge acquisition. If, say, the Year 2 examination results indicate that students have residual problems with some particular concept in molecular biology, a Year 3 instructor in an allied field will be asked to revisit the problem area with the class.

From an institutional perspective, a number of policies and procedures are in place to govern student assessment and ensure that educational objectives and assessment requirements reinforce one another. For example, since the ultimate certification function of assessment for undergraduates is the award of a specific class of honors, a number of policies have been adopted to guide how the result is obtained. The focus of these policies is something called the Graduation Grade Average or GGA, which is a weighted average of course grades subject to certain exclusions. The boundaries between the classes of

51

graduation honors are defined in terms of the GGA. In computing the GGA, courses taken by a student in the first year of study are given half the weight—this policy was introduced to reduce the pressure on new students, who find the multitude of assessment tasks they are required to perform, not to mention the need to sit for roughly 12 final examinations in their first two semesters, a difficult adjustment. The policy has evolved from a proposal made to the Senate by the Students’ Union early in the history of HKUST. In addition, to avoid discouraging students from exploring beyond the bounds of their curriculum, courses that represent “excess” credits beyond the minimum required for graduation may, at the option of the student, be excluded from the calculation of the GGA. This not only removes the disincentive to attempt such courses, but also rewards those who do well. To ensure that the results of this system are reasonable, there is a periodic benchmarking exercise reviewed by the CTLQ (and prior to its establishment, by the CUS) of the breakdown of graduates by honors classifications. To date, this has shown that the figures for HKUST graduates fall between those for HKU and CUHK and hence can be taken as representing a standard consistent with local practice.

The standard of grading, because of its importance in the certification function, is a matter of constant discussion and concern. The Senate has adopted guidelines that are superficially similar to normative assessment, with a specified percentage at each grade level. However, the policy explicitly states that grading each course “on a curve” with a preset proportion of the grade levels to be assigned is generally not acceptable. Rather, the guidelines attempt to provide a common understanding of the normal proportions of grade levels over time and from course to course, department to department, and school to school. Departments have a number of mechanisms to monitor course grades and to address regular and significant departures from the guidelines. Course grades must be approved by Department Heads before they are official, and Department Heads are expected to discuss unusual grading practices with the faculty concerned. A number of Departments have formalized this expectation. For example, in the Humanities Division of the School of Humanities and Social Science, the Division Head regularly reviews the grading results of all courses and discusses specific results with individual instructors whose grading patterns differ, by a large margin, from the percentages in the Senate guidelines. If necessary, changes are to be made before submission of grade sheets. Suggestions are raised regarding how to improve grading practices and how to ensure fair assessment of student performance.

Most Departments offer multi-section courses, and all of them, in response to Senate policy adopted following the first round of TLQPR, have designed mechanisms to ensure consistency of assessment across the sections. A common method is appointment of one of the instructors as course coordinator, who oversees the setting of examination questions and grading of examination papers by the group of instructors teaching the course.

At the institutional level, a compilation of grade distributions for all courses, Departments, and Schools has been a regular topic of discussion at the CUS. The Committee has noted that there has been a consistent deviation from the Senate guidelines, but that this has remained fairly stable over time. Since the outcome of the grading system, the award of honors has so far been consistent with local standards, no changes in the way the guidelines are implemented have been proposed, but the Committee remains alert to the possibility of grade inflation over time. As a further check on the grading standard, the office of the VP-AA has completed a study benchmarking the standard against that of quality universities overseas. This was done by comparing the performance of 65 HKUST students while on exchange for a semester at an institution in the U.S. with the grades they had obtained at

52

HKUST prior to the exchange. A similar comparison of was made for students on exchange at HKUST from abroad.

Another important aspect of the assessment system is in monitoring student progress and identifying students in academic difficulty. Generally, a student who fails to achieve an average of C- in a semester is put on academic probation in the following semester. Students on academic probation must take a reduced study load and seek academic counseling from their academic advisor/counselor in the department. If necessary, they may also be referred to a Counselor of the Student Affairs Office. A student may have the probation lifted by achieving a semester grade average of C- or above; otherwise, probation continues for another semester. A student who fails to achieve C- or better during the second probationary semester is subject to dismissal.

The emphasis on assessment for grades, despite the formative aspects of a number of the grading practices, is natural in the Hong Kong context, where matriculating students have been “educated” in an environment where preparation for public examinations have dominated the landscape. HKUST has not been immune to this examination-and-grade orientation, but is beginning to explore some of the more modern approaches to assessment integrated into the learning experience. The course Effective Learning & Thinking Skills was developed with these approaches as a key underpinning. The course is designed to integrate effective learning and thinking skills into personal development and academic activity. It aims at fostering a proactive lifelong learning spirit and enabling students to understand, monitor, manage, evaluate and reflect on their own learning and thinking strategies. An important part of the course is a web-based personal development profiling system called PEAKS (for Portfolio of Essential Attributes, Knowledge and Skills). This tool allows the students to conduct self-assessment and peer assessment of progress through the experiential leaning activities incorporated in the course. Students who have learned how to use the tool can continue to accumulate information about their knowledge and skills as they progress through their programs, and use it to generate portfolios that can be assessed by instructors, prospective employers etc.

Initial experience with this course indicates that students find it serves a significant need. Based on this response, we have introduced two pilot programs for Fall 2002 that will integrate the course into two degree programs, one from the School of Science and one from the School of Business and Management. Faculty in the programs will work with the course instructor to tailor the learning activities to reinforce learning objectives of the respective programs. Two new courses are being developed, one designed to develop leadership and team skills and the other, career skills. One of the pilot programs, the BSc Mathematics (Statistics Option) will use this entire three-course sequence so that students will take one such course each year. The other participating program (BBA Global Business) already has a career development component, but will use the leadership and teams skills course and incorporate elements related to entrepreneurship to provide a course on leadership and entrepreneurship planned as part of the original program curriculum

Implementation quality

The ability or potential to be a good teacher is an important consideration in faculty recruitment. Departments require candidates for appointment to present a seminar, and their presentation and communication skills are closely examined at that time. Some, such as the Department of Civil Engineering, also require a candidate to give a lecture to the

53

Department’s students, and for those candidates with prior teaching experience, ask for evidence of teaching effectiveness.

The development of new staff as teachers is recognized as an important responsibility by Departments. Some formally assign faculty mentors to new staff, others give new faculty assignments to multiple section courses where more experienced teachers can provide regular reinforcement and guidance. Recognition of good teaching is also widespread. There are School awards as well as the University-wide annual selection that leads to recognition at the Congregation for distinguished teaching. These formal awards are supplemented by other signs of recognition; for example, the Dean of Business and Management sends a Dean’s Appreciation Letter to faculty members who have achieved a score of 80 or above on the standard end-of-semester student course evaluations.

The University-wide system of student evaluation of courses and instructors, described in more detail in the section on accomplishments, provides a reliable and comprehensive source of data on student views of course elements, such as content and workload, and of instructor performance. This system has been collecting data on virtually every course, every semester for a number of years. Summary information on each course is available to the entire University community on-line. This provides feedback to instructors on how they stand relative to their peers. Department and school averages are also shown. In recent years, the course data has been supplemented by a statistical breakdown by class size and program level, to provide more meaningful comparisons.

Department, School and University committees that review candidates for substantiation and promotion consider teaching performance very carefully. This is well known to the candidates; so much so that, according to a survey of teaching staff, the methods for evaluating teaching are a major concern for those being evaluated. This has led to a campus-wide consultation on the use of peer review, and a policy proposal incorporating the suggestions of this process will be tabled at CTLQ in Fall 2002. A number of existing mechanisms incorporating peer review are already in place. The School of Science, for example, has a School-wide standing Peer Review Committee to assess teaching performance. Its reviews are conducted, on a voluntary basis, upon the written request of the faculty member. They are comprehensive reviews based on a statement of the faculty member’s approach to teaching; course outlines and relevant teaching materials, and unannounced visits to classes. Another approach is taken by the Division of Social Science, where every member of the Division Review Committee has to attend and evaluate a lecture by an applicant for substantiation or promotion.

To further refine our perceptions of what constitutes effective teaching, and to formalize our commitment to its importance, HKUST is in the process of developing an official University Statement on Quality Teaching. As a first step, a selected group of faculty members who have themselves been identified as quality teachers and/or who have served as a member of academic review committees at the School or University levels were asked to participate in a survey on the contents of the statement. The survey consisted of a number of proposed statements about the attributes of teachers in performing their teaching role. These have been adapted from a number of reputable sources, including the Seven Principles for Good Practice in Undergraduate Education promulgated by the American Association for Higher Education (AAHE) and the Guidelines for Effective University Teaching issued by the Australian Vice-Chancellors’ Committee (AVCC). A draft Statement on Quality Teaching incorporating the feedback from the first stage of consultation was disseminated to the academic community for consideration and input. Responses were obtained through

54

Department and School channels, by direct submission from individual faculty members, and through open discussion at a Faculty Forum on the issue in March 2002. A revised draft will be tabled at CTLQ in Fall 2002 with an aim of adoption by the Senate before the end of the year.

Mechanisms for evaluation of individual performance are thus well developed, although constantly under review and being improved. Measuring the quality and coherence of programs and curricula as a whole is more difficult and less well developed, but a number of indicators are regularly examined. An annual Graduate Employment Survey, conducted for both undergraduate and postgraduate program completers, not only directly measures employer acceptance of graduates, but also gets information on recent graduates’ perceptions of how well their education has prepared them for the world of work. The items bearing on this question are job satisfaction, relevance of the course of studies to the job, and achievement of major objectives. These surveys are conducted and compiled by the Office of Student Affairs and distributed to staff in all Schools and Departments.

Many Departments receive feedback from employers about the performance of graduates through their formal and informal advisory mechanisms, but we have not attempted to carry out a comprehensive survey of all employers of our graduates. As attractive as this seems in principle, indications are that all but the largest employers are unwilling to devote the resources necessary to making returns of detailed surveys on a regular basis—to what would inevitably be at least all of the public universities in Hong Kong. However, we do regularly collect useful feedback from major companies conducting job interviews in our Career Center. For example, in the three academic years 1997-2000, the Career Center collected feedback from an average of 61 interviewers from 13 major companies. Ratings on seven aspects of interview performance were obtained; namely, manner, enthusiasm, appearance, communication skills, general knowledge, and English proficiency. Both qualitative and quantitative feedback from the major employers obtained in this way is forwarded to the career coordinators of the respective academic units, and is discussed at career coordinator meetings. Most of these characteristics of graduates are indicators of holistic educational outcomes not measured by performance in courses, and include the benefits of activities in and outside the classroom that build personal confidence and develop leadership skills.

Commitment of resources to education quality work

Resources for the quality assurance functions of education quality work are largely represented by the staff time and effort devoted to these tasks. There is a considerable commitment of manpower at all levels of the institution. The Office of the Vice-President for Academic Affairs, with seven senior staff and seven clerical staff, devotes a major fraction of its staff time in support of quality assurance activities. The commitment of other staff (mostly academic staff representatives) to the work of the Senate committees supported by the office is estimated to be about 300 staff-hours per year, for attendance at meetings alone. At the School level, the work of the School Boards and faculty committees is estimated to involve 1,500 staff-hours annually. There was quite a bit of variation in the estimates provided by those Departments that felt comfortable with making them, but a range of about 500-1,000 staff-hours per year per Department is probably conservative.

With respect to resources devoted to quality improvement and teaching development, substantial amounts of both earmarked grants from UGC and internal funds have been devoted to a wide range of projects. As these have been regularly reported in detail to the

55

UGC in a series of progress and final reports, the details will not be included with this document. More difficult to identify with any precision is School and Departmental expenditure on their ongoing efforts to improve curricula and the delivered quality of education. One example of this type of work is the sponsorship of faculty members in the School of Business and Management in the International Teachers Program operated by the School of Business at NYU: five professors were sent to the program at a cost of about $500,000. The use of an external program in this case made it easy to derive the financial commitment, but this is not the case with the smaller in-house activities. Individual units have a one-line budget for most departmental non-staff cost items. The expenditure is captured in budget lines for analysis purposes, but these represent a fairly high level of aggregation. For example, the expenses for workshops and seminars will not distinguish between those related with teaching or research. In order to provide better information on how Schools and Departments utilize their funds to support education quality work, a new line for this purpose has been added to the budget structure from fiscal year 2001-02.

A major institutional resource commitment to the quality improvement aspects of education quality work has been made during recent years through the creation and development of the Center for Enhanced Learning and Teaching (CELT). At the time of the first round of TLQPR, centralized support for this function was provided by the Instructional Development Unit (IDU) within the Educational Technology Center. This small unit (of two senior staff and one clerical staff) offered highly professional assistance to teaching staff, organized events related to pedagogical issues, and participated with teaching staff in projects to improve teaching and learning. However, its small size and positioning in a much larger unit responsible for a wide range of support services limited its impact, visibility and effectiveness. A reorganization of academic support units combined the staff of the IDU with a number of staff that had been supervised by the School of Engineering, working on projects related to educational innovation. A former member of the academic staff with expertise in applications of multimedia was recruited as the Director of the new unit, CELT, which was placed directly under the Vice-President for Academic Affairs. Through a combination of reallocated internal resources (adding approximately $5 million to the annual recurrent funding of IDU) and external funding, CELT has grown to an operation of about 25 staff, with sections for Programs, Teaching Technologies, and Instructional Development.

Conclusions

At the conclusion of the first round of TLQPR in 1995-96, the Panel stated

We came away impressed with the quality of staff with whom we visited, and with the University's potential as a provider of high-quality teaching and learning. However, our review left us with a number of concerns.

This last sentence was one of the few examples of understatement in the TLQPR Report. A rough count identified 57 negative comments about teaching and learning quality processes at HKUST among the Report’s recommendations and observations of unit visits. The challenge this posed to the institution was immediate and obvious. During the intervening years, faculty and staff at all levels have responded to this challenge and worked to ensure that the “University's potential as a provider of high-quality teaching and learning” is being realized.

56

Education quality work and institutional mission

The commitment to education is explicit in the University’s mission, and the objectives adopted by the Council specify the goals we wish to achieve. We have set ourselves the task of building a system for quality assurance and continuous improvement worthy of our mission and objectives. We have also made it a priority to do this within an organizational framework of delegated responsibility, where diversity is encouraged. This was, we believed, necessary for the system to take root in the existing academic culture, to enrich it, and become embedded in it as central part.

In pursuing these goals, there has been a heavy reliance on dialogue between students and staff, particularly through the Student-Staff Liaison Committees that were mandated by the Senate following the first round of TLQPR. Such mechanisms were among the best practices from other institutions revealed in that exercise and adopted at HKUST. It has also been necessary to build up the system through discussions between the academic administration and the academic staff, both in formal committee settings and through forums and other channels to engage the rank and file. These mechanisms have helped to develop a sense of collective responsibility for teaching and learning quality.

This process has resulted in program and pedagogical changes focused on student learning. These have included ways to stimulate peer learning and make use of peer assessment, to develop critical thinking skills, and creativity. In some cases, the discussion has illuminated how an existing component, such as the final year project, contributes to these aims. This sharpens the focus of design of the project course and gives faculty a better understanding of how to assess project outcomes. The quality assurance framework has also made us take a look at traditional assessment practices and to benchmark them in order to validate their standard and verify that they do measure significant learning outcomes.

In making incremental changes and improvements, in both the educational product and the quality assurance processes, the institutional structure has been strengthened to bring greater coherence to the system. While diversity is still an obvious feature, we monitor the operation of the unit systems to ensure they address the relevant concerns. There has also been gradual convergence through the sharing of good practices.

Today, it is true that research remains a high priority and high profile activity at HKUST. But at this period in the University’s development, it is also fair to say that teaching does not stands in the shadow of research, but rather is very much on the front burner as a concern of the institution and of individual faculty.

Current initiatives and work in progress

The obvious work in progress is the continued development and maturation of a coherent set of quality assurance process and practices. As noted elsewhere, in the next year the Senate will be asked to adopt a Statement on Quality Teaching that will affirm our commitment to teaching and learning, and clarify the expectations we have of academic staff. We will propose improvements to the ways we measure the performance of individual teaching staff. We will start several pilot programs aimed at integrating new ways to stimulate and assess learning outcomes not measured by traditional content-based assessment tool into regular programs. With the help of funding from the Teaching Development Grant, we will initiate a multi-year series of projects aimed at improving teaching and learning, in which outcomes from one cycle of projects are used to inform the next.

57

While we are already making use of some basic measures of learning outcomes, we are still in the early stages of understanding how best to measure student progress in cognitive development, and how best to make assessment part of a larger set of conditions that promote change. All Departments sent representatives to an initial workshop on these issues in May 2002, and a set of resources on “Assessment for Learning” have been collected on the IDEAS portal maintained by CELT. This will be a key area for further development in the next few years.

Finally, improving the overall student experience, which includes informal as well as formal learning opportunities, and various aspects of campus life, will be a priority. A task force has been appointed by the President to review the current situation and make recommendations for change.

Annex C (b)

58

The Hong Kong University of Science and Technology

Teaching and Learning Quality Process Reviews

Self-evaluation Supplement on Research Postgraduate Education

July 2002

59

I. Self-evaluation Supplement A. Overview 1. Organization of research postgraduate education programs

The mission statement of the University distinctly stipulates that HKUST as an educational institution is ‘to advance learning and knowledge through teaching and research, particularly ….. at the postgraduate level.’ From the outset, the University aimed at becoming a leading world-class research university in this region. Consistent with this goal, all nineteen departments and divisions of the University offer research degree programs up to the PhD level. Governance of research degree programs is organized through four levels of committee structure - including postgraduate studies committees at the departmental, school and University levels, and lastly the University Senate.

2. Distribution of research postgraduates (RPgs) among schools Among the four schools, Engineering has the largest percentage share of RPg enrolment (54.7%). This is followed by Science (29.4%), Business and Management (8.1%), and Humanities and Social Science (7.8%). The following table provides a breakdown of the distribution of faculty and RPg student numbers among the schools. Although the Business School employs about a quarter of the faculty of the University, its percentage share of RPg enrolment is relatively small. This pattern is consistent with that in the 1998 PESG review visit. Faculty members of the school have focused their teaching efforts on the development and delivery of high-quality coursework programs as well as executive education programs. This seems to be a universal pattern among business schools.

School Total RPg % Faculty Strength % Faculty vs RPg Ratio Science 268.5 29.4% 111 26.0% 1:2.4 Engineering 500.0 54.7% 148 34.7% 1:3.4 Business & Management 74.0 8.1% 117 27.4% 1:0.6 Humanities & Social Science 71.5 7.8% 51 11.9% 1:1.4 Total 914.0 100.0% 427 100.0% 1:2.1

3. Improvements made to research postgraduate programs

3.1 Issues raised in the Postgraduate Education Sub-Group (PESG) visitation in 1998 – In the

last review conducted by PESG on research postgraduate education of the University in the summer of 1998, the University, in its self-critique, identified six areas for improvements to be made. These include : (i) Supervision and monitoring of study progress, (ii) Programs with high attrition rates; (iii) Supply of quality local RPgs, (iv) the English Language proficiency of RPgs, (v) Course availability, and (vi) Communication and environment. Part of the observations of the PESG Panel coincided with areas (i), (v) and (vi) above. In addition, PESG suggested that IT training for Humanities and Social Science (HSS) students be strengthened. It also commented on the adequacy of library resources allocated to HSS and some RPgs were too engrossed in their study that they did not have time to benefit from other aspects of postgraduate education.

3.2 Courses of action taken – The issues raised in the 1998 PESG visitation have been duly considered by

the appropriate committees, administrative offices, schools and departments concerned. Actions have been taken to address the issues raised, many of which will be covered in latter parts of this report.

B. Assurance of the quality of research education

1. Admission of students – Selectivity

Although the authority to admit students to research degree programs rests with individual academic departments, all departments adhere to the minimum admission requirements applicable to all postgraduate programs of the University as stipulated by the University Senate. Applicants with non-standard qualifications whom departments wish to admit are individually considered by the Senate Committee on Postgraduate Studies. In addition to the University-wide admissions requirements, departments may impose other admission requirements as appropriate. At the departmental level, matters concerning the admission of students are either considered by a postgraduate admissions committee or, in the absence of such, the departmental postgraduate studies committee. In an effort to enhance the quality of its RPg intake, beginning 1998, the University, through arrangements of the Ministry of Education of China, recruited finishing undergraduates in the top 2% of their own cohort from major Mainland universities to attend RPg programs at the University. They have helped to enhance the quality of RPg

60

students. For the recruitment of local students, departments have put efforts into attracting the University’s own finishing undergraduates to pursue research studies. The quality of RPg students has been improving. In terms of selectivity for the two types of RPg programs, the average number of MPhil applications received in the 2001-02 year was 2.9 for one local student place, whereas that for a non-local place was 12.5. For PhD programs, the number of local applications has been 2.3 and that of non-locals 4.7 for each student place. Comparing the number of local and non-local applications received, the latter has always far exceeded the former. On statistical information on PG students, the University publishes annually a statistical booklet on PG students, which is disseminated to all departments and units concerned for their reference.

2. Monitoring of student progress At the time an RPg student commences work on thesis research, a thesis supervision committee comprising the supervisor and two other faculty members is appointed to oversee the student’s work. The supervisor meets with the student fairly regularly, whereas the thesis supervision committee may meet at least once a semester to consider the progress made by the student. In addition, an RPg student is required to write an annual progress report, which is reviewed by the thesis supervision committee and discussed with the student. The committee is required to consider the report and rank the student’s performance as well as progress. If a student’s progress is found to be unsatisfactory, a warning will be issued to ensure that the student will take appropriate action to improve. If no improvement is made in the next semester, the student may be required to either take Leave from Study or transfer to an MPhil or taught masters program.

3. Program completion rate

The completion rates of MPhil and PhD programs vary between the two types of programs and also from year to year. In general, the completion rate of MPhil programs has stood consistently high at above 80% for cohorts between 1995/96 and 1997/98. The completion rate for PhD programs is comparatively low, at 53.3%, 46.8% and 33.6% respectively for cohorts of the same period at this point of time. The completion rate for PhDs will improve for all the three cohorts as the percentages of those continuing are 9.7%, 11.7% and 34.4% for the same cohorts. Many are finishing this summer. For both MPhil and PhD programs, these are improved percentages compared with those of cohorts of earlier years reported in the 1998 PESG visitation exercise. There are reasons for the longer time needed to complete PhD studies at the University. In general, RPg programs of the University require longer time to complete than similar programs at sister institutions because of the heavier coursework element required. Normally, the longer time it takes for students to complete a program, the greater is the attrition rate. For the PhD cohort admitted in 1999/2000, a total of 24 (16.6%) out of 145 withdrew in their first two years of study. The figure is reduced to 15 (7.6%) out of 198 for the 2000/01 cohort. For 2001/02, the number of withdrawal after one year has been 10 (6.0%) out of 166 admitted. There are indications that these figures are improving. For PhD programs that take an average of 4 to 5 years, a graduation rate of 50% to 60% is considered reasonable. This is an improvement from 30% plus reported in the last PESG review exercise, when the University was in its infancy stage.

4. Qualifying examination for PhD students

All PhD students, on completion of their coursework requirement, are required to pass a qualifying examination. Failing the qualifying examination twice means that the student will not be allowed to continue his study. This process helps to screen out those not suitable for pursuing PhD thesis study. In 1999/2000, a total of 6 (4.9%) out of 123 PhD students failed their Qualifying Examination. In 2000/01, 11 students (8%) out of 137 sat and failed their qualifying examination and in 2001/02, there have been 7 (5.2%) out of 134 such students thus far.

C. Support provided for research students 1. Academic support for student learning - Thesis supervision

When a student commences his thesis research, a thesis supervisor is assigned. A thesis supervision committee of three faculty members, of which the supervisor is a member, is formed to supervise and monitor the thesis work. In the event a supervisor leaves the University, a new supervisor will normally be appointed from among the remaining members of the committee in order to maintain some continuity. Normally, in appointing a PhD thesis supervision committee, at least one member must have had experience in the supervision of PhD students. For young faculty members with little or no experience in supervision work, they are normally assigned to supervise MPhil students initially to gain experience before they participate in the supervision of PhD students. In terms of the quality of thesis supervisors, at the time of joining the University, all faculty members must have a doctoral

61

degree from a major university. Prior to joining the University, the majority of faculty members have had teaching and research experiences in research universities of international reputation, or experience working in leading research institutes.

2. General provision for student learning - Enhancement of English Language skills and

availability of courses

2.1 Enhancement of English Language skills - As classes and theses are in English, proficiency in English is important to all students. Although many students, at the time of admission, have shown proof of their English proficiency through their test scores in international English Language tests, such test scores are not always reliable. To address the issue, for example, the School of Engineering now requires its RPg students to complete a purpose-designed English Language course. Other RPgs may attend English Language courses designed specifically for postgraduates. The English Writing Center of the Language Center also provides assistance to postgraduates in the writing of thesis and research papers. Some departments also employ communication tutors to help students improve their English writing skills.

2.2 Availability of courses – In a small number of departments/divisions, because of limited faculty

resources, the range of courses available in any semester may not be large. To a student, this means few courses that can match the research interest of the student may be available in any semester. To remedy this situation in the short term, students may choose to enroll for independent study courses which are customized courses that can match the academic interests of individual students.

3. Other general provisions for student learning - Support services and resources for research students

3.1 Apart from providing an excellent environment for conducting research, the University also provides

distinctive support services to its RPg students. In terms of computer support, the University now adopts a decentralized approach. Intensive scientific computations are mainly performed in computer clusters within departments. Orientation sessions on the use of computing and audio-visual facilities are conducted for RPg students who have to perform TA work. The Information Technology Services Center also provides direct technical support for RPg students in the setting up of Web-severs and addressing IT security issues.

3.2 The University library has been very proactive in promoting its services to all students. It offers

orientation classes and tours for all new students. A self-guided tour and a self-paced online information literacy tutorial are available to familiarize students with library services and facilities. Workshops on database guides, searching the Web, and specific research topics are organized at intervals for all students. Of the 551 study carrels,15 have been designated for PG students from the School of Humanities and Social Science. The latter arrangement is in part one of Library’s responses to an observation of the 1998 PESG Panel about an imbalance in resource allocation for those from Humanities and Social Science.

3.3 To prepare RPg students to perform teaching assistant (TA) duties, a wide range of workshops covering

topics from ‘Handling Difficult Situations’ and ‘Helping Students to Learn’, to ‘Leading Discussion Groups’ and ‘Using Technology in Teaching’ are organized by the Center for the Enhancement of Learning and Teaching (CELT). CELT provides training, resources and technical support to TAs to enable them to be more effective and efficient in performing their TA work. In addition, CELT oversees the work of a team of TA co-coordinators who serve as liaison persons with TAs in their own department.

3.4 New RPg students who have to work in laboratories are given training in laboratory safety so that they

are fully aware of safety precautions and issues in laboratory work. 3.5 The University provides research travel grants to its RPg students to conduct field work outside Hong

Kong and to attend international conferences if they have papers accepted for presentation or posting. 3.6 The Student Counseling Service and Careers Center of the Student Affairs Office provides very useful

support to prepare PG students for their careers. Apart from a regular electronic newsletter ‘PG Career Link’ (9 issues a year), it also organizes workshops and career talks exclusively for PGs. Attendance at these events has been very encouraging.

4. Communication with research students

The establishment of communication channels among PG students, faculty and University administration was an area identified in the self-evaluation in the 1998 PESG review to be in need of improvement. Much has been done to address this. Measures taken include the establishment of departmental Staff-Student Liaison Committees, the

62

creation of a PG Website containing information on PG studies, as well as an electronic newsletter on PG matters to keep all informed of PG matters, the establishment of a TA Co-coordinators’ Committee to co-ordinate TA matters in departments, the biennial announcement in the PG newsletter on the grievance system for PG students, and lastly the active recruitment of PG students representatives as members of Senate and the Senate Committee on Postgraduate Studies.

D. Areas for continuous improvement 1. Monitoring of student progress and early warning for weak students As the number of RPg students is becoming sizeable, it is important that an effective and efficient system is in

place to ensure that students who have not been performing satisfactorily in their studies are identified early so that appropriate action is taken to warn and guide the student. The bulk of the work of monitoring student progress falls on individual thesis supervision committees. As a whole, more effort can be put into strengthening the monitoring role of thesis supervision committees. Guidelines can be issued on how the committees can help monitor the work of RPg students. While the annual progress reporting exercise is a useful mechanism that serves this purpose, there is also room for improvement particularly in terms of how the report is channeled and actioned on. Methods of accounting for the course of action taken in cases where unsatisfactory performance has been reported will be reviewed. Such measures will help to strengthen the monitoring of student progress.

2. Student participation in university life

University education is not just about study and research work. An observation of the PESG Panel in the 1998 review was that some students were too engaged in their work such that they had no spare time to participate in professional and social interaction. In addressing this issue, extracurricular activities for PG students have been organized for PG students. More can be done to encourage students to participate in this type of event. In addition, more needs to be done to encourage PG students to participate in committee work that deals with PG issues.

3. Enhancement of general skills of RPg students

To prepare students for their future careers, the rigorous training in research skills in their areas of specialization is not adequate. They need to be skilled in communication, professional and human interaction, and other lifelong skills to face the growing challenges of modern knowledge-based society. As suggested in the PESG Panel in their 1998 visit, such skills would be very useful to RPg students when they engage themselves in work in academe, business, industry or government. Some work has been done to help students plan their future career path but more work, possibly in the form of workshops, talks, discussion groups and training courses can be organized to train RPg students to be better prepared for the challenges that lie ahead of them.

4. Publicizing good practices

The sharing of good practices, which is very useful in stimulating the culture of self-improvement, does not only provide recognition to those involved, it also provides a role model for other departments. Efforts to publicize good practices have been sporadic. As the channels of disseminating information to PG students and faculty are readily available, what is needed is a concerted effort of RPg students, departments and administrative offices dealing with PG students to be made aware of good practices adopted by individual departments or supervisors and report them to all stakeholders on campus.

E. Conclusion

Compared with many established universities, this University, with a history of ten years, is relatively young. Today, although its student enrolment and faculty size have both stabilized, the academic culture and traditions of the University are still evolving and maturing. To echo its mission statement, one of the University’s priorities has been to provide its researchers, both faculty and RPg students, with the best environment for conducting research. In striving for this goal, the University acknowledges the need for continuous improvements to be made to the quality processes and practices of research postgraduate education. Efforts in the next few years will focus on the areas identified above. These efforts will undoubtedly help to enhance the learning experience of RPg students as well as the quality of the research degree programs of this University.

Prepared by PG Secretariat Office of the Vice-President for Academic Affairs Hong Kong University of Science and Technology Attachment (Not enclosed) - List of Websites and Sample Documents Illustrating Quality Assurance Processes in Action

Annex C (c)

63

The Hong Kong University of Science and Technology

Teaching and Learning Quality Process Reviews

Self-evaluation Supplement on Continuing Education Units

July 2002

64

Preamble

In the first few years after the University began its first class in October, 1991, its main efforts had focused on matters such as curriculum development, student and staff recruitment and the preparation of laboratory facilities for both teaching and research. It was not until 1994/95 that the first batch of continuing education courses was offered by the Executive Education Team of the School of Business and Management. The School of Engineering soon began offering professional education programs to the public to meet the needs of engineering professionals. In mid-1999, HKUST College of Lifelong Learning, a wholly-owned subsidiary of the University, was established and within a year, it started offering its first in-house produced online language course to the public. Today the size and operation of the units are still relatively small, compared with those of the CEUs of the other UGC-funded institutions.

1. Governance and organization of Continuing Education at HKUST

Within the University, there are currently a total of three units which are providers of continuing education programs. They are:

(i) Executive Education Team, School of Business and Management, (ii) The Continuing and Professional Education Programs Team, School of

Engineering, and (iii) HKUST College of Lifelong Learning

The School of Business and Management, the first unit that ventured into the

offering of continuing education programs to the public back in 1994/95, focuses their work on executive education. In more recent years, as a result of market demand, a number of self-financed coursework postgraduate programs have been introduced. The School of Engineering started by offering short professional courses and training workshops through its continuing and professional education team. It introduced a small number of self-financed postgraduate programs beginning 1997/98. The newest CEU is the HKUST College of Lifelong Learning (CL3), a wholly-owned subsidiary of the University. When it was first established in 1999, it identified a niche market and focused its efforts on the offering of online English Language courses through both the online and face-to-face modes. To date, CL3 has not yet engaged itself in the offering of credit-bearing programs.

These three, until today, are the key providers of continuing education programs at

the University. They operate independently of each other, both in terms of management, financials and operations. Of the three, CL3 has been playing a central liaison role in the handling of general enquiries concerning continuing education programs of the University. In addition, it has also been the University’s link with external bodies and Government agencies dealing with continuing education matters. The continuing education units in the two schools report directly to the dean of the School concerned, whereas CL3 reports to the Vice-President for Academic Affairs.

65

2. Types of continuing education courses offered

The programs and courses offered by the three continuing education units of the University may be categorized into four main types – (i) self-financed credit-bearing postgraduate programs of the University; (ii) non-credit courses; (iii) credit-bearing award programs offered by overseas institutions; and (iv) customized corporate training programs.

As the scope of CEU programs to be included in this TLQPR review covers only

programs which are credit-bearing and offered by the University, very few programs offered by CEUs of the University fall into this category. In all, only programs in Category (i) above would be included in this TLQPR review. 2.1 Self-financed credit-bearing postgraduate programs – To date, credit-bearing programs have been offered by the two Schools only. These include programs such as the Executive Master of Business Administration (EMBA), the Master of Technology Management (MTM) in Information Technology, the Graduate Diploma in Computer Forensics (GradDip), just to name a few. The full list of programs is provided in Appendix 1. 2.2 Non-credit courses – All the three CEUs offer both short and longer courses. The latter may lead to the award of non-credit bearing certificates or diplomas. These courses would be outside the scope of this TLQPR review.

2.3 Credit-bearing programs offered by overseas institutions – CL3 recently introduced two online credit-bearing programs offered by institutions in the US. As CL3 merely acts as an administrative representative and has no part to play in the content or the award of the programs, these programs also fall outside the scope of this review.

2.4 Customized corporate training programs – All three CEUs offer customized training for the corporate sector. As these are in-house training courses, they are also not the concern of this review.

A full list of award-bearing programs, both credit-bearing and non-credit bearing,

is provided in Appendix 1. 3. Quality assurance mechanisms for programs offered by CEUs

3.1 Quality assurance for credit-bearing programs

The credit-bearing programs mentioned in 2.1 above are programs at the postgraduate level. As these are programs offered by the individual Schools, the approval process for new programs and the processes concerning curricular changes are identical to those of the regular UGC-funded programs. As these are all self-financed programs, the administrative as well as financial aspects of each program are considered by a new committee, the UAC Sub-Committee on Continuing and Professional Education (SCCPE), which is established by the University Administrative Committee (UAC). Program proposals, having endorsed by the School authority, would need to be

66

considered by this committee for their financial viability before they are forwarded to the Senate Committee on Postgraduate Studies for considering the academic aspects of the proposed programs.

Each of the credit-bearing programs, from its conception stage in the department, has to be reviewed by five layers of committees, including the departmental postgraduate studies committee, the School Postgraduate Studies Committee, the UAC Sub-Committee on Continuing and Professional Education, the Senate Committee on Postgraduate Studies and finally the University Senate. The review process is no different from other degree programs of the University except there is an extra committee that oversees the financial aspects of the programs.

Administrative matters and support services for self-financed credit-bearing

programs are co-coordinated by an independent team in each of the School offices. Students on these programs are treated no different from other students of the University. They have access to the University Library, the computer facilities and other student amenities of the University.

3.2 Quality assurance for non-credit programs

A new board of study entitled the Continuing Education Review Board (CERB)

has been established by Senate to review programs and courses that lead to the award of diplomas, which are programs that are of a minimum of 100 hours of instruction. It is intended that CERB will be the board of study, equivalent to a school-level body, that oversees and considers credit-bearing programs that are initiated by CL3. CERB comprises faculty representatives from all the four Schools of the University.

4. Looking ahead

4.1 In recent years, in view of the changing economic situation, there has been a shift for a certain sector of the workforce to transform itself into a knowledge-based labour force. To partially address this change in the labour market in Hong Kong, the Government has been encouraging the populace to engage in lifelong learning pursuits to enhance their professional knowledge and general skills. Recently, the Government has launched the Continuing Education Fund (CEF) Scheme to subsidize people who pursue studies that are relevant to the economic development of Hong Kong. The three CEUs have already contributed towards the pool of CEF approved courses. There are plans for more courses covering a wider range of topics that fall within the focused areas identified by Government to be offered in the foreseeable future. This will provide interested students with a wider choice of topics and would meet the needs of a wider range of students. 4.2 All three of the CEUs will soon be offering some of their programs and courses on the Mainland. In some instances, this may involve the joint offering of programs with Mainland universities. As the human resources market on the Mainland is huge, it is quite conceivable that more programs will be offered to students on the Mainland. Rules and regulations on the offering of educational programs by outside educational bodies operating on the Mainland are still unavailable. It is anticipated that the monitoring of educational programs operating on the Mainland, if introduced by the

67

education authorities, may impact on the plans of CEUs of the University which are beginning to offer programs on the Mainland. 4.3 When there are three CEUs operating continuing education programs in the University, there may be some small overlap in their course offerings. This situation, in general, does not affect credit-bearing programs as there are University committees that serve to co-ordinate program proposal matters that would ensure that overlap of program offerings does not occur. For non-credit bearing programs, the overlap issue is more likely to occur. This may not be an issue if market demand for the type of courses is huge. If demand is small, it may lead to keen competition within the same institution. Whether this is healthy competition can be a point of contention. There may be a need for a co-coordinating body to oversee, at the University level, the offering of continuing education programs. This body may help to ensure that program offerings from all the internal CEUs are well co-coordinated. 4.4 There is a lack of guidance on a qualifications framework on programs which do not carry credits. This has led to a proliferation of award titles such as diplomas, certificates, professional certificates, advanced diplomas, etc. There may be a need for a common award framework to be established so that within the same institution, there is consistency in the award structure being adopted by the CEUs.

5. Conclusion

The CEUs of the University are still in their infancy stage of development. Given the general pressure exerted by society for people to continuously improve themselves intellectually, professionally and also at the personal quality levels, there are plenty of opportunities for the CEUs to expand the range of their course offerings to meet societal needs. For a University built and supported by public funding, it is only natural that it should play an active part in providing quality continuing education programs to the community that has been supporting its operation. This complements the part of the mission statement of the University which stipulates that the University should ‘… assist in the economic and social development of Hong Kong’.

The evolution of the CEUs of the University is a process that will likely take a few

more years to stabilize. The next few years will be important times of development for the CEUs. As mentioned in 4.2 above, apart from the Hong Kong market, opportunities for CEUs to expand their offering of programs to the Mainland are abound. It would be desirable for the CEUs to review their mission, as well as manpower strength and resources, so that they can focus their efforts on the areas of strength they have identified for themselves and offer quality continuing education programs that meet the needs of society.

Prepared by the Office of Vice-President for Academic Affairs Hong Kong University of Science and Technology

Annex C (d)

68

The Hong Kong University of Science and Technology

Teaching and Learning Quality Process Reviews

Self-evaluation Supplement on Exemplary Accomplishments and Areas for Improvement

July 2002

69

Exemplary Accomplishments

Given the developments over the six years since the first round of TLQPR, it seems most appropriate to discuss two concrete and practical areas that the institution has developed to improve the quality of teaching and learning and the assurance of that quality. Both have potential impact across a number of domains of the framework, but their most significant characteristic is that they illustrate a commitment to continuous improvement. We regard these as exemplary because we believe they are further developed and more comprehensive than any similar programs in Hong Kong institutions. Their development was particularly underpinned by the principles of continuous improvement ad collaborative responsibility. Initiatives for further improvement are included in the discussions of these areas below.

Training and Evaluation of Teaching Assistants

The first TLQPR Report recommended that

Variations in performance among postgraduate students acting as teaching assistants (TAs) should be reduced…. While the quality provided by some TAs is exemplary, others appear to be inadequately trained-some even appear to lack adequate language skills. We heard that TAs are "encouraged" to make use of the Educational Technology Centre. One school has made such training mandatory and its extension to other schools is being pursued, but we believe the degree of impetus should be strengthened and diligence in follow-up improved.

This was among the first recommendations addressed, since the training made available by the ETC was already in place and the TLQPR Report provided the impetus for Schools and Departments to buy into its importance. In 1996-97 the training of TAs was made mandatory, but relevance of the program to Departmental needs was an issue, and compliance was uneven. By the following year, a group of “TA Coordinators” had been appointed. These were mostly experienced TAs receiving studentships, who were relieved of teaching responsibilities in order to work with ETC and the Departments to ensure that new TAs attend the most appropriate workshops, help identify specific discipline or Department related training needs, and serve as a communication channel for TA concerns.

The program was taken over by CELT upon its formation. The TA Coordinators, by this time formed into a TA Coordinator Committee, provided feedback from the TAs on the program and helped the staff of CELT to modify the program to meet their needs. As of the current year, TA Coordinators (TACs) work at two levels:

At the department level, they assist the PG Coordinators or their designees to coordinate the work of the TAs and provide the needed peer support in their daily operations. Their duties vary from department to department. Some common duties include organizing a departmental Briefing or Orientation for new TAs, implementing and analyzing student evaluation of TA performance, and providing one-to-one assistance to TAs

At the institutional level, the TA Coordinators, as members of the TA Coordinators' Committee meet regularly throughout the academic year to plan and organize support services for all TAs at HKUST. Working Groups have been formed for several major areas:

• TA Training and Development, e.g. organizing orientation, workshops, informal sharing sessions • TA Evaluation (feedback from students), both paper and online modes • Information and resource services to TAs (such as the creation of the TA Homepage and online

discussion group) • Recognition to good TAs and dissemination of effective practices

As indicated by the formation of this last Working Group, attention is paid to providing recognition for

participation in the training program and for good performance by TAs. There is an annual Certificate Presentation Ceremony at which TAs who have completed training modules are recognized for this, and certificates of appreciation are awarded to TA Coordinators. There is a selection of a Best TA based on student input, and the work of the TAs is highlighted in publications and on the web.

The elements currently incorporated in the program have developed over several years through discussions between and among the CELT staff, the TA Coordinator group, TAs themselves, and Departmental academic staff. The program will continue to improve through constant review. One of the outcomes of this

70

process is recognition that training of TAs is a first step in preparing the next generation of academics to be quality teachers. Under consideration are proposals to extend the training to a more formal introduction to education-related topics such as pedagogy and curriculum design. This may take the form of a voluntary certificate program for postgraduate students interested in pursuing academic careers.

In the meantime, we are building on the experience with the TA training program by introducing a new policy that provides advanced PhD students an opportunity to gain first-hand teaching experience. This recognizes that the increasing focus on teaching and learning in higher education makes such experience a valuable asset for a new graduate competing for an academic post. The program will be limited to students with good communication skills and a demonstrated interest in teaching, who have completed the TA training program, and have satisfactorily performed as a TA, as supported by student course evaluation. Students selected for the program will be allowed to teach a section of a regular course under the supervision and guidance of an experienced member of the faculty. The faculty supervisor will provide feedback on course materials and on assessment, observe the student’s teaching in the classroom, offer suggestions that help to improve the teaching style and methods, and review assignments, tests and examination scripts graded by the student.

Teaching Evaluation

The first TLQPR Report listed “mandatory student course evaluation questionnaires” as an initiative “enhancing teaching and learning quality” and urged that it “be continued and strengthened as part of the broader quality programme.” The mandatory use of end-of-semesters has, indeed, been continued and to a significant degree strengthened by better use of the results and understanding of what they measure. However, selecting this as an area of exemplary achievement is likely to be controversial. There are widespread reservations about the use or misuse of student evaluations among the academic staff at HKUST—although these probably differ in no great degree from the reservations expressed by teaching staff throughout the academic world about such practices. These arguments have not been ignored, and are freely aired in the materials available to faculty on University web sites. For example, there is a strong critique of the end-of-semester questionnaire by Professor Eric Mazur, a highly respected proponent of teaching quality improvement, in a streaming video of an interview at HKUST linked to the CELT home page. His conclusion, that the true measure of effective teaching can only be seen in the performance of students some years after graduation, may be valid, but the need for timely feedback on staff performance simply cannot wait for such ideal information to be obtained.

Despite the reservations noted above, this topic will be presented as an exemplary achievement in terms of improvements in the systematic collection and dissemination of student evaluation data, identification of staff attitudes and concerns through a Survey on Undergraduate Teaching at HKUST, studies of the data in light of those concerns, and broadening the base of teaching evaluation as a way to address the concerns.

The course evaluation questionnaires are completed by students near the end of each semester, and cover virtually all courses. The current version, which has remained stable for a number of years in order to provide continuity and consistency in the interpretation of the results, has 23 multiple-choice type questions and six open-ended ones. Since the Fall of 1996, the average scores on overall impression of course (Question 3) and instructor (Question 14) have been published to the University community on the web. The full results of the evaluation are made available to the faculty member and the Department.

In Spring 1999, the paper-based survey, administered during class time, was replaced by a Web-based system called COSSET (Centralized On-line System for Summative Evaluation of Teaching). COSSET was the end product of a project funded by the Teaching Development Grant and was thoroughly pilot-tested before being adopted for general use. Students now complete the course evaluation questionnaire on the Web. Faculty perceptions of the change to an on-line system were, on average, somewhat negative. More than one-quarter felt that the results provided a less accurate assessment of their teaching; only a very small number felt it was more accurate. Analysis of questionnaire responses, however, provides no evidence that suggests any significant change in the pattern of results.

One persistent impression has been that COSSET, which relies on voluntary student participation outside the supervision of a classroom setting, produces far lower response rates. This is simply not the case; there has been no significant change. As a response to the concern about response rate, however, from Spring semester 2000 classes with enrolment or response rate judged to be too low for reliable comparison have not been included in the on-line compilation. Another issue revealed by the Survey is that nearly half of all faculty surveyed did not feel the student ratings were an accurate reflection of their teaching. Despite this, a substantial

71

majority found the evaluations at least “somewhat useful” for teaching improvement. For those who felt that the ratings did provide an accurate reflection of their teaching, more than half felt the evaluations “useful” or “very useful.”

Of greater concern is the fact that more than one-quarter of teaching staff report that they often feel pressure to “inflate grades” to please students (and hence get better evaluation scores). There is no evidence of any significant general inflation of grades, so presumably most faculty who feel this pressure do not yield to it. However, it reflects a general view that evaluation scores are higher when faculty are more generous in grading. This may or may not be the case: there is a significant (negative) correlation between class size and evaluation scores, and grade averages are also higher for small classes, so the cause-effect mechanism has not been established. To help reviewers make better judgments about teaching, we have provided a breakdown of average evaluations cores by class size grouping and program level since Fall 2000. In an attempt to understand the relationship between evaluation scores and grades, if any, a recent small-scale study was made of a group of courses of roughly the same size. It found essentially no correlation between evaluation scores and grades, with the important exception of failing grades. This preliminary finding will be further explored, and its implications for the reliability of the evaluations will be addressed.

Faculty concerns about the student evaluations reflect their fear that poor evaluations will have adverse consequences when they are being reviewed for substantiation or promotion. This is indirect evidence of the fact that teaching performance is taken seriously in such reviews, and that this has been communicated to the staff. While there is no reason to believe that student opinions of a teacher are irrelevant, there are valid arguments against over-reliance on them. The solution is not to discontinue the use of student evaluations, but to supplement them with additional sources of evidence. It is to this end that internal consultation on the issue of peer review has been carried out in the 2001-02 academic year. Peer input can provide a complementary perspective on teaching performance. The proposed Statement on Quality Teaching that is in the process of development will also guide faculty in preparing their applications so that they provide evidence addressing the various expectations we have of effective teachers.

Areas for Improvement

In the spirit of continuous improvement, there are ongoing efforts to make improvement in many areas, including those mentioned above as exemplary. Where appropriate, reference to these efforts has been incorporated in the main body of the Self-Evaluation Document, which is the primary reason we did not include them here. For the purpose of this Supplement, therefore, we will first focus on improvement of the coherence of the teaching and learning quality assurance system as a whole. A brief discussion of assessment for learning follows.

Coherence of quality assurance processes

The management philosophy and organizational structure make management of the resultant diversity a significant challenge to the coherence of quality assurance processes. While we have documented substantial progress in meeting this challenge in the Self-Evaluation Document, consolidating the gains and building on them to improve the system clearly must be a major priority for all levels of the University. The approach to this task will embody the principles of collaborative responsibility and sharing of good practice, and is aimed at addressing the degree of coherence.

The starting point for further progress will be the unit level EQW Profiles prepared in 2001-02 as the latest version of the annual reports on teaching and learning to the Senate Committee on Teaching and Learning Quality (CTLQ). We will propose changing the focus from an annual report to continuous updating of the Profiles as improvements are made, although a summary of the updates will be compiled for CTLQ to review, and it will report its findings annually to the full Senate.

Two steps will be taken to help units in this task. First, a “Model EQW Profile” will be assembled by combining sections from different units that represent the best of what has been submitted in the initial exercise. Second, for each unit the section that most needs improvement will be identified and the unit will be asked to consider improving either its processes or its description of its processes, making reference to the model profile. This is intended to hasten the dissemination of good practice. Identification of the sections needing improvement will be initially proposed by the CTLQ Secretariat in the office of Vice-President for Academic Affairs, with the Committee making the final decision.

72

This cycle can continue as long as seems appropriate, with a specific area targeted for improvement being identified for each unit at least annually, and units encouraged to update their profiles whenever they make improvements to any area. The model profile will also undergo regular improvement, either when it is judged that a different unit’s updated submission provides a better model in a particular area or when the original unit improves its original submission.

Assessment for Learning

A second area identified for improvement has been selected as a priority because we are only in the initial stages of addressing its implications. This area, assessment for learning, involves the interaction of the quality principles of measuring learning outcomes, and a focus on the processes of student learning.

A number of learning opportunities have been developed in recent years that contribute to this focus in an ad hoc fashion. A recent example is the High Tech Entrepreneur Program, an experimental course in the School of Engineering in which enterprising students, in a realistic business theme project, learn and practice entrepreneurship under the guidance of industry consultants and university professors. The program requires creative and independent work by the participants, emphasizes working collaboratively, and hones their communication and presentations skills. This year, selected teams presented their projects at a public event organized in conjunction with a session of the HKUST Forum on the Future Development of Hong Kong. Student assessment for grades in this course is thus based on student characteristics not normally measured in usual content-based course assessments. Programs like this go a long way towards bringing learning outcomes into focus, but a systematic understanding of how assessment is linked to these outcomes is lacking.

A new initiative, described in the Self-Evaluation Document, will begin in Fall 2002 with the introduction of courses designed to develop critical thinking, creativity, and leadership skills into two specific degree programs on a pilot basis. In these courses, the link between assessment and learning outcomes is explicit, and students will make use of an on-line portfolio assessment tool specifically developed for this purpose. If this approach proves effective, from the point of view of both students and the faculty in the participating programs, a version of it may be adapted for most or all degree programs. If so, it will be necessary to train program faculty to teach the course under the general supervision of a coordinator. This, in turn, will provide an opportunity to disseminate the underlying concepts and principles to a wider group of faculty, with potential positive spin-off for better assessment for learning in other courses.

In addition to these program initiatives, the benchmarking exercises on student assessment standards conducted by the CTLQ Secretariat will be continued and expanded to explore the development of more relevant outcome measures.