András Belokosztolszki 6/10/[email protected] Managing change in the database world.

András BelokosztolszkiAndrás Belokosztolszki6/10/20076/10/2007

[email protected]@red-gate.com

Managing change in Managing change in the database worldthe database world

www.sqlbits.com

Group BY: [Food and Drink at Reading Bowl, see you there!]

Feedback Forms: [Voucher for £30 book on return of Form]

Lunch Time Sessions: [Idera in Everest, Quest in Memphis, Grok in Chic 1 and 2]

Learn & Enjoy [Put your phone on Vibrate!]

Ask The Experts[Sessions need to finish on time, take questions to the ATE area]

BackgroundBackground

Working for Red Gate SoftwareWorking for Red Gate Software Software Architect/PMSoftware Architect/PM

– SQL Log RescueSQL Log Rescue (reads the transaction log, and allows fine (reads the transaction log, and allows fine

grained recovery)grained recovery)

– SQL CompareSQL Compare Compares and synchronizes databasesCompares and synchronizes databases

– SQL RefactorSQL Refactor Productivity tool for DBAs and developersProductivity tool for DBAs and developers

AgendaAgenda

MotivationMotivation Where is the schema storedWhere is the schema stored

– Database vs. Creation scriptsDatabase vs. Creation scripts– Advantages/disadvantagesAdvantages/disadvantages

Explore the database/make it Explore the database/make it explorableexplorable

Compare (scripts/db, db/db)Compare (scripts/db, db/db) Synchronize (scripts/db, db/db)Synchronize (scripts/db, db/db)

The Ideal WorldThe Ideal World

Requirements ChangeRequirements Change Natural growth of the databaseNatural growth of the database

– More data and physical limitationsMore data and physical limitations– Expansion: more information is storedExpansion: more information is stored– Access control restrictionsAccess control restrictions– Database merges (two perfect Database merges (two perfect

databases need to be consolidated)databases need to be consolidated) Turnover, new people? New ideas?Turnover, new people? New ideas? Change for the sake of change!Change for the sake of change! Poor documentation -> Re-Poor documentation -> Re-

implementing existing functionalityimplementing existing functionality

Changes Lead to Database Changes Lead to Database EvolutionEvolution

Database schemata change, i.e. Database schemata change, i.e. existing schemata need to be existing schemata need to be extended and changedextended and changed

Generally there is a deployed Generally there is a deployed production database and one or many production database and one or many “freely modifiable” development “freely modifiable” development databasesdatabases

There is a need to identify or track There is a need to identify or track changes!changes!

Database DevelopmentDatabase Development

DB Development Problems:DB Development Problems: It is NOT compiled code with public/private qualifiersIt is NOT compiled code with public/private qualifiers

– Dependencies can be brokenDependencies can be broken Where are these? What is using this object? Public APIWhere are these? What is using this object? Public API Can I modify this?Can I modify this? What is in this object?What is in this object? Who changed this the last time?Who changed this the last time?

– Problems are detected during production?Problems are detected during production? Garbage (unused, possibly unusable code, possibly Garbage (unused, possibly unusable code, possibly

maintained)maintained) Documentation: Where to store? How to retrieve?Documentation: Where to store? How to retrieve? Legacy codeLegacy code No versioningNo versioning No auditNo audit DataData

Database Development Database Development With ProblemsWith Problems

Where Is the Schemata Where Is the Schemata Stored?Stored?

??

Source Control For Source Control For DatabasesDatabases

Source control works with files, the Source control works with files, the database schema is not in filesdatabase schema is not in files

Problem: identifying the difference Problem: identifying the difference between the files and the database between the files and the database schema itselfschema itself

Problems: keeping the files in sync Problems: keeping the files in sync with the databasewith the database

What Is In the Database?What Is In the Database?

Processed objects Processed objects [tables]: [tables]: – syntax not preservedsyntax not preserved

Textual objects Textual objects [views, triggers]:[views, triggers]:– Syntax preservedSyntax preserved– Meta data not perfectly preserved -> Meta data not perfectly preserved ->

runtime problemsruntime problems Generated objects Generated objects [symmetric keys][symmetric keys] CLR:CLR:

– hard to explore, just a DLLhard to explore, just a DLL

Explore: TablesExplore: Tables

Tables are not textual objectsTables are not textual objects Comments and formatting in the creation Comments and formatting in the creation

scripts are lostscripts are lost The information is stored in several system The information is stored in several system

tables and are accessible via these or via tables and are accessible via these or via system viewssystem views

Compare: TablesCompare: Tables

Scripts: Textual comparisonScripts: Textual comparison Documentation: Scripts or extended Documentation: Scripts or extended

propertiesproperties Generated constraint names Generated constraint names

(defaults, foreign keys, check (defaults, foreign keys, check constraints)constraints)

2000 vs. 2005: users vs. schema2000 vs. 2005: users vs. schema

Synchronization: TablesSynchronization: Tables

Tables contain data, so a drop/create Tables contain data, so a drop/create is not an optionis not an option

Certain operations cannot be Certain operations cannot be performed using an alter statement:performed using an alter statement:– Change filegroupChange filegroup– Change identity propertyChange identity property– Certain data type changes (image)Certain data type changes (image)

Sync Problem: Alter ObjectSync Problem: Alter Object

In certain cases you cannot alter an objectIn certain cases you cannot alter an object– Tables need to be rebuilt when identity Tables need to be rebuilt when identity

columns need changecolumns need change– Table returning functions when the returning Table returning functions when the returning

table schema changestable schema changes– Underlying CLR assembly needs to be rebuiltUnderlying CLR assembly needs to be rebuilt

Be careful, because during the rebuild you Be careful, because during the rebuild you may lose your:may lose your:– PermissionsPermissions– Extended propertiesExtended properties

Explore: Stored Procedures, Explore: Stored Procedures, Views, …Views, …

Non-CLR stored procedures, functions, views and triggers Non-CLR stored procedures, functions, views and triggers are textual objects, i.e. they are stored as textare textual objects, i.e. they are stored as text

Comments and formatting in the creation scripts are Comments and formatting in the creation scripts are preservedpreserved

The information is stored in several system tables and are The information is stored in several system tables and are accessible via these or via system views, but meta accessible via these or via system views, but meta information may be inconsistent!information may be inconsistent!

This is a This is a problemproblem

Problem: Stored Procedures Problem: Stored Procedures (1)(1)

They are stored as textThey are stored as text Dependencies may change:Dependencies may change:

– Stored procedures can return bad resultStored procedures can return bad result– Stored procedures can break (and we learn Stored procedures can break (and we learn

this only when we try to execute them)this only when we try to execute them)– This is very different from compiled This is very different from compiled

software!software!

TableA

Stored procedure

B

Schema binding

Schema binding

Problem: Stored Procedures Problem: Stored Procedures (2)(2)

Sysdepends and Sysdepends and sys.sql_dependencies track sys.sql_dependencies track dependencies dependencies (DEMO)(DEMO)

– No, they do No, they do NOTNOT– Alters are not updating sysdepends Alters are not updating sysdepends

(2000)(2000) Stored procedures can reference Stored procedures can reference

nonexistent stored proceduresnonexistent stored procedures

Understanding Stored Understanding Stored Procedures & Textual Procedures & Textual

ObjectsObjects Legacy code “someone” wrote sometimeLegacy code “someone” wrote sometime Task: understand it fast and modify itTask: understand it fast and modify it

– Formatted code easier to readFormatted code easier to read– Documentation can be inlined unlike in tablesDocumentation can be inlined unlike in tables

Modification is by a simple alter Modification is by a simple alter statement?statement?– This is often sufficient, but there are This is often sufficient, but there are

problems with CLR and object dependenciesproblems with CLR and object dependencies

Problems: ViewsProblems: Views

Views are also stored as textual objectsViews are also stored as textual objects The owner (user or schema) if not The owner (user or schema) if not

specified is resolved differently for various specified is resolved differently for various users! users! – Solution: fully qualified names Solution: fully qualified names (DEMO)(DEMO)

Underlying tables may change!Underlying tables may change!select * from dbo.tableA select * from dbo.tableA – Columns can change -> sp_refresh!!Columns can change -> sp_refresh!!– Columns may be droppedColumns may be dropped– Solution: Expand * and schemabindingSolution: Expand * and schemabinding

Renaming ObjectsRenaming Objects

In scripts: Search and replaceIn scripts: Search and replace In database:In database:

– Need to identify dependencies (but Need to identify dependencies (but sysdepends is wrong)sysdepends is wrong)

– Rename in Management Studio uses Rename in Management Studio uses sp_rename sp_rename Sp_rename is evil! It does update only the Sp_rename is evil! It does update only the

sysobjects/sys.objects tables/views.sysobjects/sys.objects tables/views. Do not use it, or use it with care!Do not use it, or use it with care!

– Drop and create the textual objectDrop and create the textual object

Roles, Users, PermissionsRoles, Users, Permissions

The syntax for creating roles is different for The syntax for creating roles is different for 2000 and 2005 (sp_addrole vs. CREATE 2000 and 2005 (sp_addrole vs. CREATE ROLE)ROLE)– Which one should be in the scripts?Which one should be in the scripts?– Sp_adduser in 2005 creates a schemaSp_adduser in 2005 creates a schema

Good policy: grant permissions to roles onlyGood policy: grant permissions to roles only– Do not compare user membersDo not compare user members

Problem: create role transactional, but Problem: create role transactional, but sp_addrole and sp_addrolemember are NOTsp_addrole and sp_addrolemember are NOT

Explore: DefaultsExplore: Defaults

Two types of defaults, and they must be Two types of defaults, and they must be handled differently during synchronizationhandled differently during synchronization– DRI default (in the column)DRI default (in the column)– Bound default – can be reusedBound default – can be reused

2000: syscomments, sysobjects, 2000: syscomments, sysobjects, syscolumnssyscolumns

2005: sys.columns, 2005: sys.columns, sys.default_constraintssys.default_constraints

Processed Objects: DefaultsProcessed Objects: Defaults Default values for tables are parsed and processed by Default values for tables are parsed and processed by

SQL Server:SQL Server:

Note that there are:Note that there are:– Different casesDifferent cases– Different function callsDifferent function calls– Extra parametersExtra parameters– Extra parenthesesExtra parentheses

See my blog: See my blog: http://www.simple-talk.com/community/blogs/andras/default.aspx

Input SQL Server 2000 SQL Server 2005

2 (2) ((2))

1+2+3 (1+2+3) (((1)+(2))+3)

cast(3 as int) (convert(int,3)) (CONVERT([int],(3),0))

Compare and Sync:Compare and Sync:XML Schema CollectionsXML Schema Collections

SQL Server 2005 supports XML with SQL Server 2005 supports XML with schema validationschema validation

XML schema collection is a set of XML schema collection is a set of XML schemataXML schemata

Problem: it is difficult to alter this Problem: it is difficult to alter this object, i.e. you can only add to it!object, i.e. you can only add to it!

If you want to modify it, you must If you want to modify it, you must unbind it from all dependent objects, unbind it from all dependent objects, like table columns, functions, etclike table columns, functions, etc

CLRCLR

One can write .Net code and execute One can write .Net code and execute it from the database it from the database

It is compiled code It is compiled code Well, it can be very fastWell, it can be very fast

Where are the CLR objects?Where are the CLR objects?

The CLR assemblies are stored in the The CLR assemblies are stored in the database as binary files. See database as binary files. See sys.assemblies and sys.assembly_filessys.assemblies and sys.assembly_files

Strongly named assemblies can be Strongly named assemblies can be easier identified based on their easier identified based on their version numberversion number

Dependencies are much better Dependencies are much better tracked for CLR objectstracked for CLR objects

Problem: CLR AssemblyProblem: CLR Assembly

My database has assemblyXYZ, what My database has assemblyXYZ, what is in it?is in it?– Documentation?Documentation?– Get it out of the database and use Get it out of the database and use

Reflector?Reflector? What about the dependencies (Which What about the dependencies (Which

tables does it reference? It is like tables does it reference? It is like dynamic SQL)dynamic SQL)

CLR UDTCLR UDT

One can create custom user defined One can create custom user defined types using .Nettypes using .Net

These types can be used in a table as These types can be used in a table as column typescolumn types

Problem: How to update an assembly?Problem: How to update an assembly?– Alter assembly works sometimes!Alter assembly works sometimes!– What is the alternative? (See SQL What is the alternative? (See SQL

Compare)Compare)AssemblyAssembly

AssemblyAssembly

UDT1UDT1

CLR Procedure

CLR Procedure

Table with DATA

Table with DATA

Synchronize: RoutesSynchronize: Routes

DECLARE @RouteLifetime INTDECLARE @RouteLifetime INTDECLARE @CreateRouteCommand Nvarchar(4000)DECLARE @CreateRouteCommand Nvarchar(4000)SET @RouteLifetime = CASE WHEN (DATEDIFF(s, GETUTCDATE(), CAST('20070110 SET @RouteLifetime = CASE WHEN (DATEDIFF(s, GETUTCDATE(), CAST('20070110

09:23:45.823' AS DATETIME)) < 0)09:23:45.823' AS DATETIME)) < 0)THEN 1 ELSE DATEDIFF(s, GETUTCDATE(), CAST('20070110 09:23:45.823' AS DATETIME)) THEN 1 ELSE DATEDIFF(s, GETUTCDATE(), CAST('20070110 09:23:45.823' AS DATETIME))

ENDENDset @CreateRouteCommand = N'CREATE ROUTE [routeA]set @CreateRouteCommand = N'CREATE ROUTE [routeA]AUTHORIZATION [routeA_User] AUTHORIZATION [routeA_User] WITH ADDRESS=N''TCP://localhost'',WITH ADDRESS=N''TCP://localhost'',SERVICE_NAME=N''serviceA'',SERVICE_NAME=N''serviceA'',BROKER_INSTANCE=N''broker_instance_identifier'',BROKER_INSTANCE=N''broker_instance_identifier'',LIFETIME='+ cast(@RouteLifetime as nvarchar(12))+''LIFETIME='+ cast(@RouteLifetime as nvarchar(12))+''exec sp_executesql @CreateRouteCommandexec sp_executesql @CreateRouteCommand

Routes are used by the Service BrokerRoutes are used by the Service Broker Routes have a lifetime which is:Routes have a lifetime which is:

Specified in seconds that indicate time to Specified in seconds that indicate time to livelive

Stored as an absolute valueStored as an absolute value

Explore: Certificates And Explore: Certificates And KeysKeys

SQL Server 2005 supports Certificates SQL Server 2005 supports Certificates and (A)symmetric keysand (A)symmetric keys

These are mostly generated by their These are mostly generated by their create statementscreate statements

Can be used for encryption and Can be used for encryption and authorizationauthorization

Note that there is nothing about key Note that there is nothing about key rotation!!!rotation!!!

The system tables do not contain The system tables do not contain enough information to recreate theseenough information to recreate these

Sync: PartitionsSync: Partitions

A table that is stored on A table that is stored on several filegroups based several filegroups based on a partitioning columnon a partitioning column

Indexes are also Indexes are also partitionedpartitioned

Some filegroups can be Some filegroups can be read onlyread only

Two objects that control Two objects that control this: partition schemes this: partition schemes and partition functionsand partition functions

Sync: Partitions (2)Sync: Partitions (2)

Partition functions specify the Partition functions specify the intervalsintervals

Partition schemes specify the Partition schemes specify the filegroups for these intervalsfilegroups for these intervals

E.g. E.g. CREATE PARTITION FUNCTION pf1 ( INT ) CREATE PARTITION FUNCTION pf1 ( INT )

AS RANGE LEFT FOR VALUES ( 2000, 2001 )AS RANGE LEFT FOR VALUES ( 2000, 2001 )

CREATE PARTITION SCHEME ps1 AS PARTITION pf1 CREATE PARTITION SCHEME ps1 AS PARTITION pf1

TO ( [PRIMARY], [PRIMARY], [PRIMARY] )TO ( [PRIMARY], [PRIMARY], [PRIMARY] )

Sync Problem: PartitionsSync Problem: Partitions

The alter operation splits and merges The alter operation splits and merges a partition scheme, but this affects a partition scheme, but this affects both the partition function and the both the partition function and the partition schemepartition scheme

A split and a merge is a very resource A split and a merge is a very resource intensive operation because data is intensive operation because data is involved (moving rows from one involved (moving rows from one filegroup to another)filegroup to another)

Space limitationsSpace limitations

QuestionsQuestions

?? [email protected]@red-gate.com http://www.red-gate.com http://www.simple-talk.com/community/blogs/andras/defaul

t.aspx http://www.simple-talk.com/author/andr%c3%a1s-belokoszt

olszki/

www.sqlbits.comwww.SQLBits.com[Conference Web site]

www.SQLBlogCasts.com[Becoming the premier Blogging site for SQL professionals]

www.SQLServerFAQ.com[UK SQL Server Community Website]

UK SQL Bloggerscwebbbi.spaces.live.comsqlblogcasts.com/blogs/simonssqlblogcasts.com/blogs/tonyrogerson

[email protected]@red-gate.comhttp://www.red-gate.comhttp://www.red-gate.comhttp://www.simple-talk.com/community/blogs/andras/http://www.simple-talk.com/community/blogs/andras/default.aspxdefault.aspxhttp://www.simple-talk.com/author/andr%c3%a1s-http://www.simple-talk.com/author/andr%c3%a1s-belokosztolszki/belokosztolszki/ Feedback Forms!!