® IBM Software Group © 2006 IBM Corporation Agility in the Database: Data Doesnt Have to be a Four-Letter Word Any More Scott W. Ambler Practice Leader.

®

IBM Software Group

© 2006 IBM Corporation

Agility in the Database:Data Doesn’t Have to be a Four-Letter Word Any More

Scott W. AmblerPractice Leader Agile Development

IBM Software Group | Rational software

Scott Ambler - Background

Practice Leader Agile Development

Fellow – International Association of Software Architects

www-306.ibm.com/software/rational/bios/ambler.html


Presentation Overview

Warning!

Critical Observations

Adoption Rate

Evolutionary/Agile Data Modeling

Database Refactoring

Database Testing

Other Important Techniques

Conclusion


Warning!

I’m spectacularly blunt at times

Many new ideas will be presented

Some may not fit well into your existing environment

Some will challenge your existing notions about software development

Some will confirm your unvoiced suspicions

Don’t make any “career-ending moves”

Be skeptical but open minded


Critical ObservationsModern Development Leading development lifecycles are

iterative and incremental

New applications typically use a combination of object and data technologies – the world isn’t pure

There is growing proof that serial/traditional approaches are ineffective

Data in only one of many important aspects, and it is rarely the primary one

There is growing evidence that craftspeople, or “generalizing specialists”, are significantly more effective than specialists

Traditional IT organizations appear to be vulnerable to offshore outsourcers

Cultural There is a serious rift between the data and

development communities Each community is relatively ignorant of

what the other one is doing There are few books within either

community that do the other justice The agile movement is coming from the

development community Many developers are prepared to ignore

data professionals, and do so quite often


66% of Development Teams Go Around Data Groups: Why?

8%

17%

20%

36%

19%Don't Know it Exists

Don't Know TheyShould Work WithThem

Data Group TooDifficult

Data Group Too Slow

Data Group OffersLittle Value

Source: www.ambysoft.com/surveys/


Adoption Rate of Agile Database Techniques

Innovators Early Adopters Early Majority Late Majority Laggards

The“Chasm”

®

IBM Software Group


Evolutionary/Agile Data Modeling

What is Evolutionary/Agile Data Modeling

An Example

Why This Works


Agile Data Modeling

Data modeling is the act of exploring data-oriented structures.

Evolutionary data modeling is data modeling performed in an iterative and incremental manner.

Agile data modeling is evolutionary data modeling done in a collaborative manner.

Data modeling is only one of many aspects of modeling, see www.agilemodeling.com/artifacts

See Agile Model Driven Development (AMDD) www.agilemodeling.com/essays/amdd.htm


Karate School Management System Example

Based on an actual organization

Did the same problem two ways:

Without an initial conceptual model www.ddj.com – A three-issue series

With an initial conceptual modelwww.agiledata.org/essays/agileDataModeling.html


The Initial Requirements(A Karate School Management System)

Iteration User Stories

1 Maintain student contact information

Enroll student

Drop student

Record payment

2 Promote student to higher belt

Invite student to grading

Email membership to student

Print membership for student

3 Schedule gradings

Print certificate

Put membership on hold

4 Maintain product information

Sell product

5 Print catalog of products

Order product for inventory

Order product for student

6 Organize tournament

Enroll participant in tournament

Send out tournament announcement email to past participants

Print tournament announcement letters to past participants


Initial Domain Model


Iteration 1 PDM


Iteration 2 PDM


Iteration 3 PDM


Updated Requirements

Iter. User Stories

4 Enroll child student

Offer family membership plan

Support child belt system

5 Enroll student in Tai Chi

Support Tai Chi belt system

Enroll student in cardio kick boxing

Support Tai Chi belt system

Support the belt order for each style

6 Maintain product information

Sell products

7 Print catalog of products

Order product for inventory

Order product for student

8 Organize internal special event (special classes, internal tournaments, …)

Enroll student in special event

Print special event certificate for student


Updated Domain Model


Iteration 4 PDM


Iteration 5 PDM


Iteration 6PDM


Why This Works

Up front, high-level modeling defines the landscape

JIT model storming enables:You to ask more intelligent questions because you know more about the

domain the longer you work in itStakeholders to answer more intelligently because they understand the

solution space betterFocus on functionality that is actually needed, not what might be needed

®

IBM Software Group


Refactoring Databases

Database RefactoringWhy it’s HardThe ProcessExamplesWhy This Works


Uncomfortable Question

How long would it take you to safely rename a column in a production database table?

Could you do it in a single day?


Survey: How Long to Rename a Column in a Production Database?

25%

28%

21%

11%

7%

8%

Single Day

Within a Week

Within a Month

Within 3 Months

> 3 Months

Too Risky To Try



0

5

10

15

20

25

30

35

40

1 to 10 11 to50

51 to100

101 to500

501-1000

1000+

One Day

One Week

One Month

Three Months

> Three Months

Too Risky

Length of Time to Rename Production Column (% Respondents)


61.9% of Respondents Have Production Data Problems:Their Strategy to Address Data Source Problems

18%

33%

8%

33%

8%

No Strategy

Don't Make it Worse

Big Bang

Evolutionary

Other



Database Refactoring

A database refactoring is a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics.

A database schema includes both structural aspects such as table and view definitions as well as functional aspects such as stored procedures and triggers.

Important: Database refactorings are a subset of schema transformations, but they do not add functionality.

www.agiledata.org/essays/databaseRefactoring.html


Why DB Refactoring is Hard

YourDatabase

YourApplication

OtherDatabases

OtherApplications

You Know About

OtherApplicationsYou Don't

Know About

DataFile

DataExtracts

DataFile

DataImports

PersistenceFrameworks

TestCode


The Process of Database Refactoring

Verify that a refactoring is

needed

Run the tests

Change your schema

MigrateData

(optional)

[Fail]

[Pass]

[Not Needed]

[Finished]

[Work continues]

Choose theRight

Refactoring

Write aUnit Test

Deprecate the OriginalSchema

(optional)

Update ExternalAccess

Programs

Run the tests

AnnounceThe Refactoring

VersionControl Your

Work

[Fail]


Rename Column

Customer <<Table>>

CustomerID <<PK>>Fname

Customer <<Table>>

CustomerID <<PK>>Fname { drop date = November 14 2006 }FirstName

SynchronizeFirstName { event = update | insert, drop date = November 14 2006 }

Customer <<Table>>

CustomerID <<PK>>FirstName

Original Schema

Transition Period

Resulting Schema


Example:Replace Column

Original Schema

Transition Period

Resulting Schema

Customer

CustomerPOID <<PK>>CustomerNumber: integerFirstNameLastName

Customer

CustomerPOID <<PK>>CustomerNumber: integer { drop date = June 14 2006 }CustomerID: char(12)FirstNameLastName

SynchronizeCustomerIDNumber { event = update | insert, drop date = June 14 2006 }

Customer

CustomerPOID <<PK>>CustomerID: char(12)FirstNameLastName


Example:RemoveView

Account

Original Schema

Transition Period

Resulting Schema

Customer

AccountDetails <<View>>

Account

Customer

AccountDetails <<View>>{ drop date = June 14 2006 }

Account

Customer


Example:Merge Columns

Original Schema

Transition Period

Resulting Schema

Customer

PhoneCountryCodePhoneAreaCode { drop date = December 14 2006 }PhoneLocal { drop date = December 14 2006 }PhoneNumber

SynchronizePhoneNumber { event = update | insert, drop date = December 14 2006 }

Customer

PhoneCountryCodePhoneNumber

Customer

PhoneCountryCodePhoneAreaCodePhoneLocal


Why This Works

The database schema is evolved in very small steps in a low-risk manner

Each refactoring is simple to implement

It enables an evolutionary approach while supporting the existing infrastructure

®

IBM Software Group


Database Regression Testing


Possible Things to Test

Current Database Testing

Add a test

Run the tests

Make a littlechange

Run the tests

[Fail]

[Fail]

[Pass]

[Developmentstops]

[Developmentcontinues]


Current State of Database Regression Testing (July 2006)

95.7% of organizations considered data to be a corporate asset Of those, only 40.3% had a regression test suite in place

Of those, Only 63.3% allowed developers to run this test suite when they needed to

Of the organizations that didn’t have a test suite, only 31.6% had discussed putting one in place

63.7% of organizations implemented mission-critical functionality in the database Of those, only 46% had a regression test suite in place

Of those, Only 66.3% allowed developers to run this test suite when they needed to

Of the organizations that didn’t have a test suite, only 38.6% had discussed putting one in place



Agilists develop regression unit and acceptance test suites for their applications, why not for databases too?

You must test:Within the database (clear box)At the interface to the database (black box)

You must be able to put the database into a known state, therefore you need test data (generation)


Things to Test For


Do You Do Database Testing? (September 2006)

Yes66%

Inadequate Resources

20%

Never Considered

7%

Don't Know7%


35.3

12.9

43.3

43.5

29.9

19.8

App Dev (TDD)

Data Warehouse(TDD)

End of DevIteration

End of DevLifecycle

During DataExtract

Don't Know

When Are You Testing the Database? (% Respondents)


Types of Database Testing (% Respondents)

73.5

52.3

25.7

51.7

33.8

39.5

42.7

39.5

DB Input

DB Output

O/R Mappings

Stored Procs

View Definitions

RI Constraints

Column Constraints

Default Column Values


Other Important Techniques

Configuration management of all project artifacts

Continuous integration

Encapsulate database access

Train developers in data-oriented skills

Train data professionals in development skills

Promote the concept of generalizing specialists www.agilemodeling.com/essays/generalizingSpecialists.htm


Conclusion

Evolutionary development is the norm, not the exception

Agile development techniques are quickly being adopted

There is no reason why data professionals cannot work in an evolutionary, or even agile, manner

The software world has changed. Have you?

Visit www.agiledata.org

®

IBM Software Group


Keep In Touch!

Scott W. Amblerwww-306.ibm.com/software/rational/bios/ambler.html


References and Recommended Reading www.agilealliance.com www.agilemodeling.com www.agiledata.org www.ambysoft.com www.databaserefactoring.com www.enterpriseunifiedprocess.com Ambler, S.W. (2002). Agile Modeling: Effective Practices for XP and the UP.

New York: John Wiley & Sons. Ambler, S.W. (2003). Agile Database Techniques. New York: John Wiley &

Sons. Ambler, S.W. (2004). The Object Primer 3rd Edition: AMDD with UML 2. New

York: Cambridge University Press. Ambler, S.W. and Sadalage, P.J. (2006). Refactoring Databases: Evolutionary

Database Design. Reading, MA: Addison Wesley Longman, Inc. Larman, C. (2004). Agile and Iterative Development: A Manager’s Guide.

Reading, MA: Addison Wesley McGovern, J., Ambler, S.W., Stevens, M., Linn, J., Sharan, V., & Jo, E. (2003).

The Practical Guide to Enterprise Architecture. Prentice Hall PTR.

® IBM Software Group © 2006 IBM Corporation Agility in the Database: Data Doesnt Have to be a Four-Letter Word Any More Scott W. Ambler Practice Leader.

Documents

software development

student slide

pdm slide

evolutionary data modeling

data technologies

data doesnt

ineffective data

data groups