December 6, 20041 Release Management – Ship It! Phyllis Kaiden Release Infrastructure Manager The Cobalt Group, Inc.

December 6, 2004 1

Release Management – Ship It!

Phyllis Kaiden

Release Infrastructure Manager

The Cobalt Group, Inc.

December 6, 2004 2

Outline

What is Release Management? Why does it matter? How does it work? Would you ship it? What did we learn?

December 6, 2004 3

What is Release Management?

Release Management Definition

Release Management Responsibilities

What is Configuration Management?

Product and Project Management

December 6, 2004 4

Release Management Definition

The process by which a product is made ready for distribution to customers. Release management is the coordination of all the activities leading to and including product release to customers.

The purpose of release management is to ensure products are ready when promised.

Focus on software release management but basics apply to all product releases.

December 6, 2004 5

It’s Like Baking

Recipe – the set of activities in order

Ingredients – quality components

Variations – Icing? Nuts?

Mixing – executing the recipe steps

Baking time – varies based on desired outcome

Is it done? – knowing when it’s just right

Final product – presentation, consumption

December 6, 2004 6

Baking & Releasing

Recipe – the set of activities in order (the plan)Ingredients – quality components (content, schedule,

resources)Variations – Icing? Nuts? (special considerations)Mixing – executing the recipe steps (do the plan)Baking time – varies based on desired outcome

(priority of time to market, features, quality)Is it done? – knowing when it’s just right (assess

readiness against measurable criteria)Final product – presentation, consumption (packaging

and distribution)

December 6, 2004 7

What is a release?

Delivery of a “system” from a supplier to a customer.

The “system” consists of a set of authorized and integrated components.

The supplier is usually the development organization.

The customer is the internal or external recipient of the system.

December 6, 2004 8

Key Practices

• Release Planning: Manage and publish release schedule

• Define deliverables included in a release• Build and Version the integrated release package• Manage dependencies across components• Coordinate activities across company • Control changes that would impact the release

schedule• Establish release criteria• Assess release readiness

December 6, 2004 9

Key Principle: Define “Done”

Deliverables Dependencies Avoid scope creep but allow change Release criteria

• Objective

• Quantifiable

• Cross-functional

December 6, 2004 10

Components of a release

Golden Build (software): CD-ROM, files Documentation and Help Training materials Marketing collateral Internal documents Customer Notification Release Notes

December 6, 2004 11

Types of Products Consumer: shrink-wrapped, commercial off-the-shelf

(COTS), e.g. MS Word Open-source: download as needed, e.g. MySQL Hosted: Application Service Provider (ASP), e.g. Yahoo

mail Corporate: Enterprise Resource Planning (ERP), e.g.

Oracle Financials Corporate desktop: enterprise distribution of desktop

applications, e.g. MS Office Operating system: core tools for IT, e.g. Sun Solaris

December 6, 2004 12

Attributes by Product Type

Consumer Open-Source

Hosted Corporate Corporate Desktop

Operating System

Installed by Self Self or IT

Vendor Vendor IT IT

Distribution Pull Pull Push Pull Push Push

Cost $ $$ $$$ $ $

Training No No Maybe Yes Maybe Maybe

Support Vendor No Vendor Vendor IT IT

Multiple supported versions

Yes Maybe No Yes Yes Yes

December 6, 2004 13

What is Configuration Management?

The process of building the product, tracking its components, and managing change.

The purpose of configuration management is to guarantee reproducibility of the product and traceability of all changes.

Release Management and Configuration Management are often the same group.

December 6, 2004 14

Common Responsibilities of CM

• Source Code and Version Control

• Reproducible – components of the release

• Traceable – what changed in this release

• Definitive Software Library: Baseline + delta

Version 1.0

Version 2.0

1.1 1.2delta deltabaseline

baseline branch merge

December 6, 2004 15

Product and Project Management

“ABCs of Release Management” by Mario Moreira. CM Crossroads, Aug 17 2004. http://www.cmcrossroads.com/ubbthreads/showflat.php?Cat=&Number=31323&Main=31323#Post31323

December 6, 2004 16

Why does Release Management matter?

Common Release Problems

Benefits of Release Management

Who Cares?

December 6, 2004 17

A Story – Timeliness, Quality

Three companies decide independently to develop appointment scheduling software. They all have the knowledge and funding to create a good product.

Company A releases on time with 100 known issues and is a success.

Company B releases on time with 50 known issues and fails.

Company C releases late with 10 known issues but fails because Company A had already captured the market.

December 6, 2004 18

Common Release Problems

Take too long – impacts revenue Unpredictable – impacts dependent plans Poor quality – impacts support Not useful – impacts revenue Contents unplanned – impacts integrity Contents uncontrolled – impacts reliability Overlooked distribution/installation – impacts

support

Cowham, Robert. “Release Management – Making it Lean and Agile” CM Crossroads, Aug 16, 2004 http://www.cmcrossroads.com/ubbthreads/showflat.php?Cat=&Number=31243

December 6, 2004 19

The Challenge

Time to Market vs Features vs Quality• Only one top priority• Balance – don’t ignore the other two• Release criteria reflect priorities• Product development methodology supports priorities:

concurrent engineering, test-first development, prototyping

Time to Market

Features Quality

December 6, 2004 20

Benefits of Release Management More successful releases Consistent release process Predictability Integration Completeness Quality Communication

December 6, 2004 21

Sales &Marketing

Support

IT

Pro*Management

QualityAssurance

Developers

Customers

Executives

Who Cares?

Worth installing?InstallableUsableSupportedBudgetTiming

Ready to sell?Demo updatedTrained sales forcePricingOrder FulfillmentContracts ready

Documentation readyTraining materialsTrained support staffRelease NotesKnown Problems FAQ, Knowledge BaseImplementation impact

Dependable InstallationResources to supportPredictable scheduleConfiguration OptionsTechnical DocumentationStability, Scalability, Reliability

Well-defined scopeResources to supportPredictable scheduleSufficient test timeMeets requirements

Product ManagementMarket positionProduct RoadmapProject ManagementProject delivery on-timeProgram ManagementProgram meets expectations

Dependencies managedMeets requirementsMinimize mergingNo “death march”Next project

Commitments toCustomers, Stockholders, EmployeesLegal ConsiderationsReturn on Investment

December 6, 2004 22

How does it work?

Types of Releases

Software Development Life Cycle

Release Process

December 6, 2004 23

Types of Releases

• Major = significant new features, product launch, planned, pull, annual

• Minor = small new features, backwards compatible, planned, pull, semi-annual

• Patch = collection of bug fixes, planned, pull, quarterly

• Emergency = Urgent bug fix, harmful if not fixed, unplanned, push, as needed

December 6, 2004 24

Attributes by Release Type

Features/Functions

Bugs Cost Urgent Scheduled Pull Backwards Compatible

Major Y (new)

Y Y Y Y

Minor Y (upgrade)

Y Y Y Y Y

Patch Y Y Y Y

Fix Y Y Y

December 6, 2004 25

Release Management in the Software Development Life Cycle

December 6, 2004 26

Activity DistributionBy Effort

10%

6%

8%

16%

40%

20%

Release

Reqts

Architecture

Detail Design

Construction

System Test

McConnell, Steve. Software Project Survival Guide. Microsoft Press, 1998.

Release activities account for

10% of effort and 15% of schedule.

December 6, 2004 27

Emergency?

BASELINE

RELEASE CYCLE

BASELINEDEVELOPMENT

LIFE CYCLE

UPGRADERELEASE

MAINTENANCERELEASE

PATCH RELEASE

ResolveWith

Initiator

FIX LATER OR

NEW FEATURE

SCHEDULED

MAINTENANCE

SCHEDULED

NEW FEATURES

BECOMES NEW

BASELINE

ROLL UP

ROLL UP

APPROVE

REJECT

DEVRM

RM

RM

RM

OWNERSCCB Change Control BoardDEV DevelopmentIS Information ServicesMRK MarketingPM Product ManagementQA Quality AssuranceRM Release ManagementSAL SalesSUP Support

3.1.2

3.1.3

3.1.1

3.1.4

3.2.0

3.1.0

3.2.0

DEFER

MOREINFORMATION

DEV

CHANGECONTROL

BOARD (CCB)PM

FIX NOW

UNSCHEDULED

MAINTENANCE

SUP

NOTIFY CCB

NEW CUSTOMERRELEASE

RM

TRACKINCIDENTS

SUP

TRACKCHANGES &

DEFECTSRM

Sources of IncidentsCustomersSystems Integrators

Sources of Changes &DefectsProduct ManagementMarketingSalesProduct SupportDevelopmentQAUser GroupStaffetc.

December 6, 2004 28

Release Process

1. Define Content2. Assign Version 3. Plan Release4. Build 5. Package6. Deploy7. Test8. Control Change9. Assess Readiness10. Release

Iterate

December 6, 2004 29

1. Define Content

Golden Build (software) Documentation Training materials Marketing collateral Internal documents Customer Notification Release Notes

December 6, 2004 30

1. Define Content: Must and NiceDisp ID Severity Pri Description

Nice 41928enhanceme

nt P3 Performance - Publish: loadSiteLite() to replace loadSite()

Nice 59439 trivial P4Misalignment of Editable site and Proof site sections on Manage site screen once the

actor checks Lock-site checkbox.

Nice 59929 minor P5Error message:Get A Quote form:The listed items in error message may need some

order; by logical or by alphabetic

Nice 57954 major P2 Only Ford Make is displayed in 'Quick Quote Form' : GM Design

Nice 59368 minor P2 Masthead descenders are being chopped off in Klamath_B_Dst design

Nice 57019 minor P3 Can't tell mouse over selection from current page selection in nav

Nice 59726 minor P3Design Celandine: Titles (Header) for static pages are not bold as those for dynamic

pages

Nice 60108 major P3 Misalignment of footer in Hours and directions page-Celandine designs

Must 64793enhanceme

nt P3 create new Saab luxury design and homepage

Must 64796enhanceme

nt P3 create new Land Rover luxury design and homepage

December 6, 2004 31

2. Assign version

Version is a unique identifier for the product state as of a certain point in time. Not all versions are released. Versions are sequential.

Versioning Schemes relate to release types, product management, and configuration management:

Major.Minor.Patch.Fix, e.g. 1.0, 1.1, 1.1.2, 1.1.2.1 Major.Feature.Patch, e.g. 1.0, 1.1, 1.1.2 Major.Feature.Patch.Build, e.g. 1.0.0.301

December 6, 2004 32

3. Plan Release

Dependencies Customer commitments Revenue recognition Resource availability: people, environments Introducing too much change at once Stabilizing between releases Organizational capability

December 6, 2004 33

3. Plan Release: MilestonesScope Freeze 9/7 COMPLETE

Functional Freeze 9/17 COMPLETE

Functional Test Complete 9/30 COMPLETE

Code Freeze 10/1 COMPLETE

Regression Test Complete 10/13 COMPLETE

Golden Build 10/8 COMPLETE

Performance Test Complete 10/13 COMPLETE

Mock Deployment 10/12 FAILED

Mock Deployment 10/14 COMPLETE

Deployment Plan Review 10/15 COMPLETE

Go/No-Go 10/15 COMPLETE – GO

Executive Approval 10/15 COMPLETE-APPROVED

RELEASE 10/15 COMPLETE

December 6, 2004 34

4. BuildBuild (What) Visibility (Who) Frequency (When) Purpose (Why)

Private System Build

Individual Developer

One or more times per change task

Provide feedback to the developer of the changes just implemented in the private workspace.

Integration Build

Whole Development Team

After each change task and/or one or more times per day (e.g. Nightly)

Integrates the latest changes in the repository and runs automated tests against the release providing feedback to the development team.

Release Build

Independent Test (QA or V&V) Team and Customers

At the end of each iteration/release

Packages release for distribution to the development team’s customer.

Appleton, Brad. “Agile Build Promotion: Navigating the Ocean of Promotion Notions”, CM Crossroads, September 2004. http://www.cmcrossroads.com/ubbthreads/showflat.php?Cat=&Number=32900

December 6, 2004 35

5. Package

Create the release package containing all release deliverables

File: tar, .zip, .exe, .ear Medium: CD-ROM, tape, file server Bill of Materials lists what’s in the

package Final release package is “golden build”

December 6, 2004 36

6. Deploy

Deployment Plan Install Team Deployment Plan Review Mock Deploy Mock Rollback Contact List

December 6, 2004 37

6. Deploy: Sample Steps1 Nitra "Version 2.3.3" Deployment            

2 WLS 8.1 on SMTP servers           Thomas

4 Verify email router updates in env file           Thomas

5 Verify UPSShip updates in Env File           Thomas

9Validate receipt of Nitra, LMCC Tar Balls and

Mapping Loader script           Claus

10 Add Scion Altercast fonts (Bug 61506)           Thomas

11 Prep-Outage Tasks 1 hr         Steven

12 Hot Backup Nitra DB 55 min 9:00 PM 9:55 PM     Claus

13Check 3rd Party Services (Chrome, AIC,

Vicinity) 10 mins 9:30 PM 9:45 PM     Jeff M

14 Team Check-In 5 mins 9:45 PM 9:50 PM     Team

15 Check CDC design flag to true 5 mins 9:45 PM 9:50 PM     Thomas

16 Outage            

17 NO DEALER ACCESS 5 mins 9:50 AM 9:55 AM     Thomas

18 NO CONSUMER ACCESS 5 mins 9:55 AM 10:00 PM     Thomas

22 Shutdown Servers 40 mins          

23 Disable Nitra Monitoring 3 mins 10:05 PM 10:08 PM     Thomas

24 Disable Incoming Leads (no consumer access) 2 mins 10:08 PM 10:10 PM     Thomas

25 Allow Existing leads to pass (Big IP) 2 mins 10:10 PM 10:12 PM     Thomas

26 Bring down WLS (Nitra and LM) 30 mins 10:12 PM 10:42 PM     Thomas

27 Snapshot net-app (back up file system) 2 mins 10:42 PM 10:44 PM     Thomas

28 Deployment time noted (for rollback purposes) 1 mins 10:44 PM 10:45 PM     Thomas

29 Database Modifications 5 mins          

31 LMCC DB Script 10 min 10:55 PM 11:05 PM     Claus

32 Run script for Mapping loader 10 min 11:05 PM 11:15 PM     Claus

33 Code Deployment (LM and Nitra Clusters) 40 mins          

December 6, 2004 38

7. Test

Plan ahead for test environments Execute Test Plan

• Functional test: does it function as required?

• Regression test: did old bugs reappear?

• Performance test: does it perform within acceptable limits?

• Reliability test: does it function consistently over time?

• Stress test: does it function consistently under volume?

December 6, 2004 39

8. Control Change

• Baseline scope, requirements, design• Accommodate controlled change

• Avoid scope creep• Allow change, it’s reality

• Change requests• Change request decision (approve, reject, defer)

• Change Control Board (CCB)• Cross-functional representation• Impact analysis• Business drivers

December 6, 2004 40

9. Assess Readiness

• Release Criteria define “done”• Defined Deliverables

• Features: Must Have vs Nice to Have

• Quality: test results, metrics

• Schedule: time-to-market criteria

• Objective, quantifiable criteria preferred

• Organizational Readiness• Support, Training, Marketing, Sales

• Acceptable Risk

December 6, 2004 41

9. Assess Readiness: Release CriteriaQuality Testing Complete (functional, regression, performance, reliability, stress)

Quality Successful Mock Deployment

Quality Successful Mock Rollback

Quality "Golden" Build in Hand

Features All “must have” features complete

Features Documentation and Help updated

Features Release Notes Accepted by Customer Support

Schedule Schedule Clearance (no conflict)

Schedule Outage Notification Processed

Schedule Release Package Delivered to IT

Schedule Acceptable Deployment Night Coverage

Schedule Acceptable Post-Deployment Coverage

Quality Go/No Go Meeting yields Go decision. Punch List completed.

December 6, 2004 42

Assess Readiness: MetricsCommon metrics: fix rate, find rate, defect

density, rate of changeExample: find rate vs fix rate

Stability reached at week 35 of testing, when defects fixed exceeds defects opened.

December 6, 2004 43

9. Assess Readiness: Go/No-Go

Release Readiness Assessment• Are all release criteria equal?

• Can any criteria be waived?

Go/No-Go• Formal sign-off by stakeholders

• Meeting, email vote

• Recommendation by Release Manager

• Go with Punch List

Executive Approval

December 6, 2004 44

10. Release

Formal release by Configuration Mgmt Deploy to Production or Distribute to

Customer

December 6, 2004 45

10. Release Programs

Alpha = test by in-house department Beta = test by small group of customers

• Pilot

• Early Adopters

General Availability (GA) = all customers

December 6, 2004 46

Would you ship it?

Situations

Is it ready?

Would you ship it?

December 6, 2004 47

Situation #1: Class registration Criteria

• Zero priority-one defects

• Ready in time for class registration on April 1 Data

• One priority-one defect affecting 100 music majors out of 3,000 students.

• Fix estimate is 3 days.

• Today is March 15. Ship it? More data?

December 6, 2004 48

Situation #1: Other Factors Cost: Penalty for late delivery? Confidence: accurate estimate? Complexity? Quality: Danger of breaking something else? How

much testing is needed? Schedule: Holidays? Vacation? Customer

availability? Hard deadline? Relationship: Unhappy customer? Reference

site? Workaround: Manual? Send staff on site?

December 6, 2004 49

Situation #2: Java web search tool Criteria

• Acceptable performance: + 10% current

• Backwards compatibility Data

• Performance on page load is 5% better

• Performance on search is 20% worse

• Fix will require customer to download new Java Virtual Machine (JVM)

Ship it? More data?

December 6, 2004 50

Situation #2: Other Factors Quantify: How much loss? < second? Accuracy: performance test environment? Alternatives: Optional download?

Automatic download? Support: Ease of download? Increased

support load? Competition: Market leader? Slipping? Usability: lose customers if slow?

December 6, 2004 51

What did we learn?

Release Management Definition

Key Practices

Key Principle

Release Process

Features, Quality, Time to Market

December 6, 2004 52

Release Management Definition

The process by which a product is made ready for distribution to customers. Release management is the coordination of all the activities leading to and including product release to customers.

The purpose of release management is to ensure products are ready when promised.

December 6, 2004 53

Key Practices

• Release Planning: Manage and publish release schedule

• Define deliverables included in a release• Manage dependencies across components

and activities in the release • Coordinate activities across company • Control changes that would impact the

release schedule• Establish release criteria• Assess release readiness

December 6, 2004 54

Key Principle: Define “Done”

Deliverables Dependencies Avoid scope creep but allow change Release criteria

• Objective

• Quantifiable

• Cross-functional

December 6, 2004 55

Release Process

1. Define Content2. Assign Version 3. Plan Release4. Build 5. Package6. Deploy7. Test8. Control Change9. Assess Readiness10. Release

Iterate

December 6, 2004 56

The Challenge

Time to Market vs Features vs Quality• Only one top priority• Balance – don’t ignore the other two• Release criteria reflect priorities• Product development methodology supports priorities:

concurrent engineering, test-first development, prototyping

Time to Market

Features Quality

December 6, 2004 57

Q & A

THANK YOU!