Copyright Elaine David & Pam Heath-Johnston University of Connecticut, 2005 This work is the intellectual property of the authors. Permission is granted.

Copyright Elaine David & Pam Heath-JohnstonUniversity of Connecticut, 2005

This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

Developing Effective Production Turnover Procedures

University ITS

EDUCAUSEOctober 18, 2005

Elaine David, Assistant Vice PresidentPam Heath-Johnston, Organizational Consultant

| University ITS

Problem

• Consultant Report

• Department Experiences

• Request by Acting CIO, (Fall, 2003)

| University ITS

Consultant Report

• Our Department hired an outside consultant to look at three areas, one being production turnover procedures• It was found that procedures varied by area; an old system was being used (AutoDoc) that needed to be eliminated; and that it is a very paper-driven process resulting in redundancy, duplication of effort and cost

| University ITS

Department Experiences

• Due to various areas doing “their own thing,” it was hard to have a consistent and standard process• Therefore, many areas did not do what we would consider standard operating procedures with regard to testing, notification and communication, documentation and follow-on support. Some areas didn’t bother to do anything

| University ITS

From our CIO

“To coordinate a policy of standards for turnover of production changes within our department that address testing (minimum expectations), communication (proactive notification of project lead to list of users), documentation (information about changes), and follow-on support (continued support until stability of product has been ascertained)”

| University ITS

Investigative Research

• Multiple ongoing meetings/discussions• Review of current practices/procedures• What is turnover to production?

– Notification/communication– Documentation– Testing– Support

Multiple Ongoing Meetings/Discussions

• Topics addressed with managers included:– Types of changes to be considered– Current practices– Minimal testing expectations– Information expected of the primary

individual associated with the change– Vehicles available for communicating

change– Follow-up support expectations

| University ITS

Review of Current Practices & Procedures

• Current procedures were documented by domain area

• Information gathered included:– List of the various types of production

changes that take place within the area– For each type of change, current practice

with respect to testing, informational notification, installation documentation, and follow-on immediate support

| University ITS

What is Turnover to Production?

• Notification/Communication– No consistent guidelines of why, when, how and to

whom to communicate

• Documentation– No consistency as to the type of documentation

required or where to store the documentation

• Testing– Testing varied from no test platform (network) to

using a test platform and doing unit, system, integration and acceptance testing

• Support– Inconsistent levels of support provided

| University ITS

| University ITS

Overall Implementation

• Clarification of Definitions• Confirming Roles and Responsibilities• Reconfiguring Daily Status group to PEOG• Procedural Changes-approvals, reviews, use of USD• Testing• Documentation• Communication to Community

| University ITS

Clarification of Definitions• What is meant by ‘Change?’

– Any implementation of new functionality– Any interruption of Service– Any repair/modification of existing functionality– Any removal of existing functionality

• Types of Change– Emergency: requires immediate response to correct a production problem or prevent a problem from occurring– Planned: includes major change requiring a project plan; normal maintenance; non-emergency fixes; enhancements; modifications resulting from other changes; vendor upgrades/fixes; replacement of one piece of equipment with another – Administrative: changes to user data rather than changes to software/hardware

| University ITS

Confirming Roles and Responsibilities

Defined roles and responsibilities for:

• Managers• Production Environment Oversight Group (PEOG)• Requester/Implementer of Changes• Viewer of Change

| University ITS

Reconfiguring Daily Status Group to PEOG

• Daily Status Group: Representatives from most domain areas met daily to react to change logs that they had no knowledge of or authority to represent

• PEOG: The group above still meets daily but was rechartered with members from each domain area who are fully informed and empowered to represent their area. This group is responsible for review of preventive and planned changes

Procedural Changes• All non-emergency changes must be ‘approved’• A change request must be created (in the USD

Problem Management System) 3- 7 days prior to implementation for (non-emergency) changes– What? When? Why? – Predicted effects on service during and after

change• All change requests will be distributed to PEOG

for review and discussion prior to implementation date

• Changes will be included on the appropriate UITS web page

• Changes will be reviewed (post-mortem) by PEOG

| University ITS

Testing

• Whenever feasible, a test environment will be created to allow for testing of changes prior to implementation in production

• Testing is required for all non-emergency changes (e.g. unit, system, integration, parallel, acceptance, post-implementation verification)

• Each area will be responsible for developing testing standards appropriate to their types of changes

• Test plans are to be documented and made available to the organization through the USD system

| University ITS

Documentation

• The following minimal documentation must be maintained for changes to production:– History of change with date of

change, who made change, disposition of change

– Test plan and Back-out plan

| University ITS

Communication to Community

• Two websites:– Network and Systems status page (

http://itstatus.uconn.edu/)

– Upcoming Changes page (http://itstatus.uconn.edu/changes.html)

• Change log indicates what is to be included on the web page and when it should be announced

| University ITS

Metrics

• Initially: Are the new procedures being followed.– Are changes being entered into USD system with

sufficient lead time and appropriate classification (emergency vs. non-emergency vs. unknown?

– See Metrics 1 and Metrics 2– Are changes being communicated to PEOG group? Yes– Are we notifying our customers? Not conclusive, needs

further investigation– Are we documenting our risk analysis and test/back-out

plans? Not conclusive, needs further investigation• Long term: Is our production environment more

stable as a result of these new procedures?

| University ITS

Metrics 1 – Lead TimeThis chart reflects the % of change logs in the production environment. Anything

less than 3 days is not acceptable or should only be an emergency.

0%

10%

20%

30%

40%

50%

60%

% of change logs

7/ 1 -9/ 30

10/ 1 -1/ 31

1/ 1 -5/ 30

6/ 1 -8/ 30

Dates

<3 days >3days

Metrics 2- Classification All change logs are classified as planned or emergency. Unknown refers to change

logs never classified by the technician.

0%

10%

20%

30%

40%

50%

60%

70%

% of change

logs

7/ 1 -9/ 30

10/ 1 -1/ 31

1/ 1 -5/ 30

6/ 1 -8/ 30

Dates

Emerg Non emer Unknown

| University ITS

Future Focus

• In utilizing the Problem Management System (USD), we need to ensure staff are attaching test and back-out plans for all changes. It is being done for only a few changes

• We are measuring a few items such as how many changes entered, in how many days ahead of time and types of changes. We need to measure other areas

• We need to enhance the Problem Management System to more easily provide the necessary reports to staff for better planning

Lessons Learned

• Developing effective production turnover procedures is an ongoing process. Once all recommendations have been implemented, we will need to continue to review and refine them.

• We cannot expect to implement all of the recommendations at one time. We need to prioritize the recommendations, phase them in, and measure the results over time.

• There are still some issues to resolve so we need to continue to focus and reinforce changes made.

• When turnover to production was begun, we needed to view it as a project. We should have identified and involved the process owner from the beginning of the project.

• These changes need constant reinforcement to support the new expectations and standards.

| University ITS

Current Contact Information

• Kathie Sorrentino, Assist Director, [email protected]

• Clyde Talley, Manager, [email protected]

• UConn Help Desk: 860.486-4357• Website:

http://uits.uconn.edu/home.html• UITS, University of Connecticut U-3138,

196 Auditorium Rd, Storrs CT 06269-3138

EDUCAUSE Poster BoardContact Information

• Elaine David, Assistant Vice President, [email protected]

• Pam Heath-Johnston, Organizational Consultant, [email protected]

| University ITS