Module 8: I2R Change Control 1.1 Gatekeeper Audit

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

Module 8:I2R Change Control 1.1

Gatekeeper Audit

April 2007


987654321

Change Control Training AgendaThis training will cover nine short modules on the New Change Control Process:

1 Training Introduction and Change Request Types

2 Risk Assessment

3 Quality Assurance

4 SOX Assessment

5 Architecture Review Board

6 Change Control Board

7 Release Management Office

8 Gatekeeper Audit

9 Merging with the New Process and Support Info


987654321

Program Change Control ObjectivesThe Program Change Control (PCC) audit process helps mitigate the following risks that can affect the quality, performance, or integrity of Cisco’s systems:

Unauthorized changes are promoted to production

Untested software is promoted to production

By the end of this module, you will learn about:

Program Change Control (PCC)

SOX Testing Requirements

Gatekeeper Role and Responsibility

Gatekeeper Cheat Sheet: What a Gatekeeper looks for

Gatekeeper Engagement Process


987654321

Program Change Control Risk Management

The following controls were implemented to mitigate risks:

Control Descriptions(Prior to Production Deployments)

Requests Documented

Obtain documented requests for program changes

Change Approval Obtain documented business/change control approval

Test/Prod Environments

Make (develop and test) changes in development/test/staging environments

Testing Test and validate changes

Test and Migration Approval

Obtain business/change control approval after testing completion for production migration


987654321

Classifications of Change TypesAll change requests for information systems must be classified into change types. In addition, the association of change requests with change types is mandatory for all I2R system components to:

Enable consistent and objective identification of appropriate tests and test documentation for each change request

Enable consistent testing and test documentation for information system changes based on change type

REQUESTS

Change Types


987654321

Change TypesEach change request must be classified into one of the following change types:

Aesthetic Application User Interface GUI change

Application Code Changes

System Configurations and Setups

Business Configurations and Setups

Other Technical System Level and Performance Change

Data Changes

Application Changes That Impact Interface


987654321

Classification of Test LevelAll change requests and their associated testing must be categorized into defined test level.

In addition, association of change requests with test level is mandated for all I2R system components to:

Provide external auditors with consistent test documentation

Ensure that each information system is testing similar changes in a similar fashion throughout Cisco


987654321

Classifications for Test LevelEach change request requires a minimum test level:

Validation Testing

IT Acceptance Testing

Business Acceptance Testing ValidationTesting

ITAcceptance

BusinessAcceptance


987654321

Minimum Level of Testing MatrixThe below matrix shows the test level that your change type requires:

Type of Change Validation Testing

IT Acceptance

Testing

Business Acceptance

Testing Aesthetic Application User interface GUI changes

X

Application Code Changes X

System Configurations and Setups X

Business Configurations and Setups X

Other Technical System Level and Performance Changes

X

Data Changes X

Application Changes That Impact Interface X

Lowest Level Highest Level

For example… if your change request is classified as an “Application Code Change”, you are required to perform “Business Acceptance Testing”. Your test documentation and tester must adhere to guidelines defined for “Business Acceptance Testing”.


987654321

Gatekeeper Role and ResponsibilitiesThe Gatekeeper serves as an internal auditor to ensure that I2R IT complies to and passes external audits. Gatekeeper’s responsibilities include:

Accountability for ensuring changes migrated to production environment adhere to PCC compliance requirements

Enforcing controls to ensure that unauthorized and untested changes do not get into production

Providing Kintana approval to authorize push of code to production


987654321

Not Role and Responsibilities of the Gatekeeper

Below are not roles and responsibilities of a gatekeeper:

Recommending scenarios and test cases for your change request

Preventing defects for your change request

Authorizing production implementation and deployments unless CCB has provided production approval

Exception: Active emergency P1/P2 alliance cases

Chasing after you to provide Kintana approval


987654321

Gatekeeper Cheat SheetBefore approving a change request for production, the Gatekeeper must verify that all necessary compliance items are checked off the list:

One PCC ITG Request is appropriately created with all relevant details complete

Change description in One PCC ITG request is understandable to third parties

Quality Center Requirement is associated with One PCC ITG request and completed test documentation is available

Test documentation adheres to level of test requirements based on the change request’s change type

Test executioner in Quality Center is appropriate according to the level of test requirement

One PCC ITG Request number and associated Kintana package are found in scope document or build list

The individual that implemented the change (creator/developer of Kintana package) is not the same as the individual that tested the change (tester in Quality Center)

Valid and approved One PCC ITG request number is referenced in the Kintana package

Evidence of business delegation and sign-off for IT to perform testing on behalf of the business is attached to One PCC ITG request or updated in the request notes

Justification and business sign-off as to why code with failed test steps or failed test cases should still be deployed to production is attached to One PCC ITG request or updated in the request notes


987654321

One PCC and Application ConfigurationsThere must be evidence of approval for application configuration work to proceed. The change requester must ensure that Change Control Board (CCB) production approval is obtained for

application configuration deployment requests.

If the application configuration work is part of a project request or tactical request, make sure that application configuration QC Defect Issue number references a parent One PCC ITG request number

The parent change request must have created a One PCC ITG request and pass PCC audits. Else, the dependent application configuration request cannot be deployed to production

Approved parent One PCC ITG request will be used for deployment of the dependent application configuration setup


987654321

One PCC and Cross Functional Patches

The CA-IT patch lead is only required to create One PCC ITG Request for all patches going into a planned release. The Functional Track is responsible for:

Ensuring that the Gatekeeper has been given the QA sign-off evidence that the patching has resolved issues, enhanced functionality, or did not break existing functionality

Attaching evidence in One PCC ITG request or updating the request notes

Associating Quality Center Requirement with One PCC ITG request and providing completed test documentation

QA


987654321

How to Request a Gatekeeper Audit

To request a gatekeeper audit, the change requester must follow these steps:

Upon completion of the last test cycle, send an email to [email protected]. For details to include in the email, see detailed gatekeeper audit email info

See standard release exception production approval cut-off time or non-standard release exception production approval cut-off time for Gatekeeper review and approval turnaround expectations

During Release planning and scope identification, the Release Manager schedules a meeting with the Gatekeepers


987654321

Planned Release Production Approval Flow


987654321

Release Exception Production Approval Flow


987654321

Emergency Bug Fix Production Approval Flow


987654321

Quarterly vs. Monthly Release Test Cycles

Quarterly and Monthly releases have different test cycles. To ensure that a change request is included in a release, the following must be confirmed:

Quarterly Releases – Release Manager must end the last test cycle 2 - 3 weeks before the Readiness Review

Monthly Releases – Release Manager must end the last test cycle 1 week before the go-live date

The change requester is responsible for ensuring that all cut-off times are met

The change requester must adhere to Gatekeeper engagement procedures

If the change request misses the intended deployment window, the change requestor must re-seek CCB approval for deployment in another deployment window

Notes If you do not adhere to the Gatekeeper engagement procedure, your request will not be deployed to production


987654321

Emergency Bug Fix DeploymentSince Emergency Bug Fix (EBF) often requires immediate deployment without adequate time to go through the regular approval cycle, it has slightly different requirements outlined below:

EBF deployment is only available for active P1/P2 alliance case

Requires Gatekeeper approval based on verbal certification

Requires post deployment Change Control Board (CCB) review of the change request

Requires post deployment CCB approval documented in change request notes

Requires change requester to provide Gatekeeper with completed PCC template and relevant test case documentation (in Quality Center) no later than 24 hours after the change was deployed into production


987654321

PCC Download and Upload Directories for EBF

For an EBF, download PCC template:

https://ework.cisco.com/Livelink/livelink.exe?func=ll&objId=10871606&objAction=Open

Remember to follow instructions in PCC template to…

Adhere to file naming conventions

Link completed PCC template to the One PCC ITG request number

Completed PCC documents can be uploaded to:


987654321

Module 8: Quiz

1. A Gatekeeper does not do what?

A. Serves as an internal auditor

B. Prevents defects

C. Recommends test cases and test case scenarios

D. Verifies that implementer of the change is not the same as the tester for the change

2. What is the total number of Gatekeeper engagement processes?

A. 1

B. 2

C. 3

D. 4


987654321

Module 8: Quiz Cont.3. Evidence of testing should not look like what?

A. Only state an approval of testing completion

B. Provide who, what, when, how in the test case

C. Show test case with results recorded

D. Show failed test steps/cases without justification for code push

4. What is the minimum level of testing required by SOX for System Configuration and Setup change type?

A. Validation Testing

B. IT Acceptance Testing

C. Business Acceptance Testing

D. Validating Testing and Business Acceptance Testing


987654321

Gatekeeper Audit SummaryAs we conclude this module, we hope that you have a thorough understanding of the following three key points:

The purpose of the Program Change Control (PCC) process is to audit and ensure that unauthorized and untested software are not put into production

The Gatekeeper does not recommend testing scenarios for your change request or prevent defects. However, the Gatekeeper might indirectly reduce the chances of defects because of Project Change Control (PCC) audits

If you do not adhere to the Gatekeeper engagement procedure, your request will not be deployed to production


Backup Slides

Gatekeeper Audit


987654321

Creation of a Request

3. Fill in all required information denoted by a red asterisk

3. Fill in all required information denoted by a red asterisk

1. Select the Request option under the create menu

1. Select the Request option under the create menu

2. Select the request type from the drop down menu

2. Select the request type from the drop down menu


987654321

Associating a Test Case to a Requirement

Test cases need to be associated to their corresponding requirement so that testing information will flow to ITG and be visible for approvals

The test cases must be associated to the systematically generated requirement in order to be associated to the correct request in ITG.

Testing Information ITG Request


987654321


On the Reqs Coverage tab

Once you select the Reqs Coverage button the Requirements tree will appear on the right side of the screen

Click on the Select Reqs buttonClick on the Select Reqs button


987654321


1. Highlight the systematically generated requirement that the test case needs to be associated to

1. Highlight the systematically generated requirement that the test case needs to be associated to

2. Click the Green Arrow to associate the Requirement

2. Click the Green Arrow to associate the Requirement


987654321

Associating a Kintana Package to an ITG Request

To associate a Kintana Package to the corresponding One PCC ITG request, the One PCC Request ID field located on the User Data tab of the Kintana package must be populated with the One PCC ITG request number

Prior to the package’s deployment into a test environment, the Kintana workflow will automatically perform the validation to ensure that the package’s associated One PCC ITG request has been initially approved by CA CCB

Prior to the package’s deployment into production, the Kintana workflow will automatically perform the validation to ensure the One PCC ITG request has been finally approved by CA CCB


987654321

Associating a Kintana Package to an ITG Request

In the One PCC ITG Request ID field on the User Data tab of the Kintana Package click the Table Icon

to display all the One PCC ITG requests that the Kintana Package can be associated to. The pick list shows all ITG requests that have been approved and are not in a closed status

In the One PCC ITG Request ID field on the User Data tab of the Kintana Package click the Table Icon

to display all the One PCC ITG requests that the Kintana Package can be associated to. The pick list shows all ITG requests that have been approved and are not in a closed status


987654321

Control Description / Requirement Example Evidence

UPDATE/PATCH POLICY:

Regular and critical patch/code updates from outside vendors are reviewed per the update/patch policy and implemented per the program change control process

Documented vendor patch/software update process

All vendor supplied patch/software patches adhere to program change control deployment processes

PCC Control # 1


987654321


APPROVAL OF THE CHANGE:

A change request must be reviewed and approved by the appropriate owner or managerial representative before the request can be worked on.

Appropriate segregation of duties exists. The individual who submits/owns that change request cannot approve the commencement of work on that change request.

Change request approval must be documented and retained in a centralized repository. Auditors must be able to easily trace change requests back to approvals.

A list of approved approvers for change requests must be maintained and archived.

Change approval must also state the minimal level of testing required based on the nature of the change to implement. The change request submitter/owner must ensure that testing for the request is compliant to the stated minimal level of testing requirements. Please see minimal level of testing section of details.

An approved approver or representative of a change control board updates the change request’s case notes with the following:

“Approved”

Approver user ID or name

Approval date

PCC Control # 2

Back to slide #4: PCC Risk Management


987654321


USER APPROVAL OF TESTING AND MIGRATION OF CODE:

If business validation or acceptance testing is required for the change request, then the appropriate business sign-off is

required to certify the test results.

Executed test cases with clearly documented test results

If business validation or acceptance testing is required, test cases show that a business user performed successful testing

If IT documented or performed testing on behalf of the business, appropriate business sign-off for testing has been updated in the One PCC ITG request notes by an appropriate business user

If IT documented or performed testing on behalf of the business, appropriate business sign-off for testing is attached to the associated One PCC ITG Request (i.e. pdf or html copy of email evidence)

PCC Control # 3



987654321


FINAL APPROVAL FOR PRODUCTION DEPLOYMENT:

Final change request approval for code push to production is given by an approved approver

A list of approved approvers for change requests must be maintained and archived

Final approver in Kintana code migration workflow must be an approved Gatekeeper (known as IT Manager/ QA Reviewer in the tool)

The following scenarios are NOT allowed under SOX segregation of duties rules:

A = Person A, B = Person B

Developer Tester Final Approver

A A B

A B A

B A A

An approved approver or representative of a change control board updates the change request’s case notes with the following:

“Approved”

Approver user ID or name

Approval date

Final Push to Production approval step in Kintana workflow was performed by an approved Gatekeeper (known as IT Manager/ QA Reviewer in Kintana)

PCC Control # 4



987654321


SEPARATE TEST AND PRODUCTION ENVIRONMENTS:

Testing and production occurs on separate databases / servers / technical environments

For an audit, use EMAN tools and/or DBA Oracle 11i website to show proof that test and

production environments are separate

PCC Control # 5



987654321

Description Examples

Changes to the appearance of the application. This does not include changes to business logic to enable appearance changes to the application.

Changes to ensure proper fields and buttons available

Changes to ensure user interface pick lists are correct

Changes to ensure default values are correctly populated

Changes to text content on a website

Change Type - Aesthetic Application User Interface GUI Change

Back to slide #6: Change Types


987654321


Changes to application code that can impact business logic/rules/functionality, values, calculations, decision paths, or transactions.

Change to interactive and batch processes

Changes to the database

Changes to database triggers and stored procedures

Alterations/additions to schemas, tables, views, privileges

Includes changes to reports

Includes major database upgrades or patches that can impact business logic or data

Change Type - Application Code Changes



987654321


Configuration changes and/or setup changes that do not impact business logic and/or functionality

Changes to Oracle job scheduling

Change Type - System Configurations and Setups



987654321


Changes that only require configurations and/or setup changes. These changes can have potential impacts to business logic and/or functionality. However, these requests require no code changes.

Changes to an approval step in a workflow

Change Type - Business Configurations and Setups



987654321


Technical changes to an application that do not impact business logic or functionality

Changes to database table space

Changes to application, operating system, or database memory allocation

Changes to database indexes

Change Type - Other Technical System Level and Performance Changes



987654321


Data fix that adds/updates/deletes information from database tables that cannot be accommodated by the SCM-PDF workflow due to magnitude of the change. This data change can be associated with a release data migration or it can be as part of a production data fix.

Updates to a descriptive flex field associated with a large number of customer records

Insert large number of records into a database table

Purge large number of records from an interface

staging table

Change Type - Data Changes



987654321


Changes to application code that impacts data exchange between the application and other applications or systems

Code change to interface that affects the logic to bring MFG Standard Cost from ODS to CTSPRD

Code change to interface to filter out ODS bad data from prices that need to be pulled into CTSPRD

Change Type - Application Changes That Impact Interface



987654321

DescriptionExample of Change That Could be Tested This Way

Test Documentation Required Data Elements

Testing performed by the business or IT where the change can be verified by visual inspection.

This type of testing occurs when the change does not require validation of business logic or transactions.

Aesthetic user interface change Date of validation

Name of person who performed validation

Description of performed validation steps

Test Level – Validation Testing

Back to slide #8: Classifications for Test Level


987654321



Testing performed and documented by technical resources.

This type of testing checks the quality of system-related changes to the application with no risk to business logic or

application controls.

Performance test due to index changes

Date of test

Name of tester

Test description

Test steps

Expected results for each test step

Actual results for each test step

Pass/Fail for each test step

Test Level – IT Acceptance Testing



987654321



Testing performed and documented by application users to validate business transactions are appropriately processed.

This type of testing checks the quality of changes to business logic or functionality by using business test conditions or scenarios.

Changes to reports, calculations, approval routing, logic within application code

Date of test

Name of tester

Test description

Test steps

Expected results for each test step

Actual results for each test step

Pass/Fail for each test step

Test Level – Business Acceptance Testing



987654321

Detailed Gatekeeper Audit Email Info

To request a gatekeeper audit, the change requestor can send an email to [email protected] with the information below:

Email Subject:

Release Exception Audit for <your One PCC ITG request number> Go-Live < your intended go-live date> or

<Month + Year> Release Audit for <your One PCC ITG request number>

Email Body:

One PCC ITG Request number

Contact person for One PCC ITG request (for e.g. IT PM)

Intended go-live date

Back to slide #16: How to Request a Gatekeeper Audit


987654321

Kintana package description must be named as follows:

<Release Month + Release Year> + Release + <rest of your package description information>

<Release Exception> + <rest of your package description information>

December 2007 Release - This is a description of my package

Release Exception - This is a description of my package In the User Data area for the Kintana package, the developer must reference

One PCC ITG Request number in One PCC Request ID field

Referencing Change Request to Kintana Package

Back to slide #12: Gatekeeper Cheat Sheet


987654321

Activity Responsible Party

Cutoff Time

Complete testing in STAGE/TEST environment and request PCC audit

Project Manager Wednesday, 11AM PST

Complete PCC audit, inform Change Requestor of audit results Gatekeeper/ QC Manager

Wednesday, 5PM PST

Obtain CCB approval for deployment to production Project Manager Thursday, 2PM PST

Add approved change request number and associated Kintana package(s) to production deployment scope document- send document to SCM Build Team, SCM Operations, Gatekeeper, C3-Support alias

CCB Coordinator Thursday, 3PM PST

Have C3 downtime packages ready in Kintana for Gatekeeper approval

Developer Friday, 10AM PST

Provide Kintana migration approvals for C3 downtime packages Gatekeeper Friday, 11:30AM PST

Have C3 non-downtime or non-C3 packages ready in Kintana for Gatekeeper approval

Developer Friday, 3PM PST

Provide Kintana migration approvals for C3 non-downtime and non-C3 packages

Gatekeeper Friday, 4:30PM PST

Record Gatekeeper comments and decisions into production deployment scope document- archive document

Gatekeeper Friday, 5:00PM PST

SCM migrates package(s) to production SCM Friday, 6PM - 8PM PST

Saturday, 12AM - 4AM PST (C3 down)

Release Exception – Standard Production Approval Cutoff Time



987654321

Activity Responsible Party Cutoff Time

Complete testing in STAGE/TEST environment and request PCC audit

Project Manager 11 AM PST, 2 days prior to intended deployment

window

Complete PCC audit, inform Change Requestor of audit results Gatekeeper 5PM PST, same day review request received

Obtain CCB approval for deployment to production Project Manager 2PM PST, 1 day prior to intended deployment

window

Add approved change request number and associated Kintana package(s) to production deployment scope document- send document to SCM Operations, Gatekeeper, C3-Support alias

CCB Coordinator 3PM PST, 1 day prior to intended deployment

window

Have evening packages ready in Kintana for Gatekeeper approval Developer 2PM PST, on intended deployment day

Provide Kintana migration approvals for evening window packages Gatekeeper 4:30PM PST, on intended deployment day

Record Gatekeeper comments and decisions into production deployment scope document- archive document

Gatekeeper 5:00PM PST, on intended deployment day

SCM migrates package(s) to production SCM 6PM - 8PM PST

Release Exception – Non-Standard Production Approval Cutoff Time



987654321

1. Release Manager/Transition Manager downloads blank build list template:https://ework.cisco.com/Livelink/livelink.exe?func=ll&objId=12859941&objAction=Open

2. Release Manager/Transition Manager adds only CCB approved production deployment change requests to the the build list.

3. Release Manager/Transition Manager renames completed build list as follows:I2R_<Release Name>_PRD_Build_List_<Downtime/Non-Downtime>.xlsExample:

I2R_MAY2006_PRD_Build_List_Downtime.xls

I2R_MAY2006_PRD_Build_List_Non-Downtime.xls

4. Release Manager/Transition Manager uploads build list to the Release’s transition directory:

https://ework.cisco.com/Livelink/livelink.exe?func=ll&objId=12455305&objAction=browse&sort=name

5. Gatekeeper updates build list with audit results

6. Release Manager/Transition Manager emails final build list to SCM

Build List Instructions for RMO


987654321

Module 8: I2R Change Control 1.1 Gatekeeper Audit

Documents