04 SQL Summit DBA Internals

SQL Server Summit 2005

SQL Server 2005 What is new for the DBA?

Cheryl Stepney3/31/05

Microsoft SQL Server Summit 2005

Minneapolis

What is new for the DBA?What is new for the DBA?

• High availability for enterprise applications• Key security and performance features• Focus on self manageability and optimization

• Integration with Visual Studio and .NET• Native XML technology • Interoperability via Web Services

• Real time decision making: reporting, data mining• Rich client and SharePoint solutions• Comprehensive ETL solution: up to 600% faster

Availability and security advancements

• Database mirroring, online operations for increased availability

• Data security and privacy: encryption, enhanced auditing

Performance and scalability

• Scales as the business grows: partitioning, snapshots

• From mobile devices to 64-bit data centers

Continued focus on manageability

• Integrated management toolset for relational and OLAP

• Expanded self tuning

This sessio

n

Topics for the DBA at the Summit

• Security – 9:40 am Sogeti• Service Broker – 2:15 pm Born

– Developer Track

• Reporting Services – 1:15 pmMagenic – Developer Track

• High Availability/Scalability – 1:15 pm Unisys• This Session – 2:15 pm

Microsoft

Agenda: SQL Server 2005 Internals and More

• SQL Packaging – Editions clarified • Integrated Data Management• Minimizing the Maintenance Window• Deployment Tools• Management and Performance Tools• New Datatypes• SQL and CLR Internals• Notification Services• Full-Text Engine

Introducing SQL Server 2005 PackagingNew for SQL Server 2005

Note: All higher editions include same functionality as the edition below it.

ExpressExpress WorkgroupWorkgroup StandardStandard EnterpriseEnterpriseFastest way for

developers to learn, build & deploy simple

data driven applications

Easiest to use & most affordable database solution for smaller

departments & growing businesses

Complete data management &

analysis platform for medium businesses

and large departments

Fully integrated data management and

analysis platform for business critical

enterprise applications

2 CPU2 CPU

3 GB RAM3 GB RAM

Management Management StudioStudio

Import/ExportImport/Export

Limited Replication Limited Replication PublishingPublishing

Back-up Log-Back-up Log-shippingshipping

4 CPU4 CPU

Unlimited RAMUnlimited RAM(64-bit)(64-bit)

Database MirroringDatabase Mirroring

OLAP ServerOLAP Server

Reporting ServerReporting Server

New Integration New Integration ServicesServices

Data MiningData Mining

Full Replication & Full Replication & SSB PublishingSSB Publishing

Unlimited Scale Unlimited Scale + Partitioning+ Partitioning

Adv. DB mirroring, Adv. DB mirroring, Complete online & Complete online &

parallel operations, parallel operations,

DB snapshotDB snapshot

Advanced Analysis Advanced Analysis Tools including full Tools including full

OLAP & Data OLAP & Data MiningMining

Report Builder + Report Builder + Customized & High Customized & High

Scale ReportingScale Reporting

Adv SSISAdv SSIS

1 CPU1 CPU1 GB RAM1 GB RAM

4GB DB Size4GB DB Size

Simple Simple Management ToolManagement Tool

Report Wizard & Report Wizard & Report ControlsReport Controls

Replication & Replication & SSB ClientSSB Client

Integration ServicesIntegration Services

What’s New for the DBA SQL Server 2005 Management Tools

• Computer Manager • Management Studio• Profiler and Replay• Database Tuning Advisor• SQL Agent• SQLCMD• Express Manager

SQL Computer Manager

• Combines the functionality of the following SQL Server 2000 tools:– Server Network Utility– Client Network Utility– Services Manager

• Ability to set service properties for:– SQL Server – SQL Server Agent – Analysis Server – Report Server – Microsoft Search – Microsoft Distributed Transaction Coordinator (MS DTC) – Full-Text Search

New Management Tools

• SQL Server Management Studio– New Rich GUI– Integrated Authoring, Management– Support all SQL Server components

• Management Objects (SMO)– Managed code, new design patterns– Improved Scalability, Scripting

• SQLiMail Replacement– SMTP Support, Cluster Support, No Outlook require

Management Studio

SQL Server Management Studio

Query Analyzer

Enterprise Manager

OLAP Analysis Manager

SQL Mobile

Report Services Notification Server

Integration Services

SQL Server Management Studio

• Enhanced Query Authoring – T-SQL– Analysis Services – MDX and XML/A

• Dynamic Help• SQLCMD Mode• Improved display for XML results• Integrated execution plan graphing tool for • Enhanced Project Management • Script Projects• Enhanced Templates

Database Management Infrastructure

• Monitoring & Performance Tuning– Dynamic Management Views– Catalog Views– Default Trace– Event Infrastructure Enhancements

• Problem Resolution– Dedicated Admin Connection (DAC)– Enhanced Mini Dumps– Plan Guides & Plan Forcing

• Very Large Databases– Partitioning– OnLine Index Operations– Database Snapshots– Page Checksums/Restore

Database Problem Resolution

• Dedicated Admin Connection– Connect even to a “hung” server – Uses reserved scheduler, port & resources

• Enhanced Mini-Dumps– All interesting internal structures now in mini-dump– No more full dumps!

• Plan Stability– Plan Guides

• controlling optimization with query hints – even when you can’t change the application

– Plan Forcing• Provide a full XML Showplan as a query hint

• Database Tuning Advisor– Time Bound Tuning

– What-If Analysis

Manageability - Diagnostics

• Dynamic Management Views– Server Transparency– “sys.memory_clerks” – how much memory is each component

consuming ?– “sys.schedulers” – Is a scheduler hung ?

• SQL Trace– Query data from live traces

• SQL Profiler– Deadlock and ShowPlan Visualization

Dynamic Management Views & Functions

• Server Level– dm_exec_*

• Execution of user code and associated connections

– dm_os_*• Memory, locking &

scheduling

– dm_tran_*• Transactions & isolation

– dm_io_*• Input/Output on network and

disks

– dm_db_*• Databases and database

objects

• Component Level– dm_repl_*

• Replication

– dm_broker_*• SQL Service Broker

– dm_fts_*• Full Text Search

– dm_qn_*• Query Notifications

– dm_clr_*• Common Language

Runtime

Data Base Encryption• Certificates used by

– SQL Server to create SSL connections– SQL Service Broker

• Asymmetric Keys– consists of a private key and the

corresponding public key– Each key can decrypt data encrypted by

the other– Asymmetric encryption and decryption

are relatively resource-intensive, but they provide a higher level of security than symmetric encryption

– An asymmetric key can be used to encrypt a symmetric key for storage in a database

• Symmetric Keys– A single key that is used for both

encryption and decryption – Encryption and decryption with a

symmetric key is fast– suitable for routine use with sensitive

data in the database

The certificates created by SQL Server comply with the X.509v3 certificate standard.

SQL Server 2005 Administration Enhancements• Dynamic AWE Memory Management

• ATTACH_REBUILD_LOG • Instant File Initialization • Programming: New System Management Objects (SMO)

API – extends and supersedes the Distributed Management Objects

(DMO) object model – provides classes for configuring and managing instances of the

Database Engine

• Programming: New SQL Server WMI Provider – The WMI Provider is an instance provider

Transact-SQL New Constructs

• Improved Error Handling – use familiar TRY-CATCH logic to trap and handle error conditions

• New Metadata Views – System catalog views provide information about the objects

• Data Definition Language (DDL) Triggers – DDL triggers can be used to audit or regulate changes to a database

schema• Event Notifications

– Send information about Transact-SQL data definition language (DDL) statements and SQL Trace events to a service broker service

– Can be used to log and review changes or activity occurring on the database

• Queue Processing Extensions – The INSERT, UPDATE, and DELETE now support an OUTPUT

clause that return data based on the rows modified by the statement– READPAST can now be specified on UPDATE and DELETE

statements

Transact-SQL Enhancements• Bulk Operations on OPENROWSET

– OPENROWSET now supports a BULK operation– Controlled using new table hints & BULK_BATCHSIZE and FORMATFILE

• TOP Enhancements – The TOP operator has been enhanced to take any numeric expression

• Distributed Query – The EXECUTE statement now supports an AT LinkedServer clause that

specifies executing a stored procedure on a linked server– Distributed queries also support the new large object data types

nvarchar(max), varchar(max), and varbinary(max)• TABLESAMPLE

– New TABLESAMPLE clause limits the number of rows processed by a query– Unlike TOP, which returns only the first rows from a result set

• New CASCADE Integrity Constraints – REFERENCES clause now supports the SET NULL and SET DEFAULT

cascading referential integrity actions • Overflow Data Can Exceed Page Size

Transact-SQL New Constructs

• Unified Large Object Programming Model – Now supports a MAX length specification for the shorter data types – When MAX is specified, the data types can store the same size strings

as text, ntext, and image values (up to 2 GB), • but are processed in the same way as when they store shorter strings

• Recursive Queries – Stores a Common Table Expressions (CTE) is a results set – that is stored temporarily during the execution of a SELECT, INSERT,

UPDATE, or DELETE statement

• New APPLY Operator – Invokes a table-valued expression (function or subquery) for each row

returned by an outer table reference

• New PIVOT and UNPIVOT Operators – Rotate result sets, where columns become rows and rows become

columns

• Query Notifications – Allow an application to cache a result set and request the Database

Engine to notify it if any of the underlying data is modified by another application

Index Enhancements

• Disabling Indexes – Indexes can be disabled to aid in administration and diagnostics – When rebuilding a non-clustered index

• disabling the index before rebuilding the index significantly reduces the amount of disk space required

• Reorganizing and Rebuilding Indexes – SQL Server 2005 introduces the ALTER INDEX statement that can

reorganize or rebuild indexes, or set index options

• Online index maintenance– Create, Rebuild, Reorganize, Drop, Add or drop constraint

• Fully parallel– Indexes can be rebuilt online, pages containing LOB data can be

compacted, and partitioned indexes can be reorganized or rebuilt on a single partition basis

• Online/offline are both supported

New XML Support

• New XML Data Type • XQuery Support

– based on working drafts of the W3C XQuery language definitions• XML Data Manipulation Language (XML DML)

– The current W3C definition of XQuery does not include the ability to modify XML documents

– Microsoft has extended the XQuery implementation in SQL Server 2005 to include the ability to insert, update, and delete XML documents or fragments

• Transact-SQL Enhancements: FOR XML Enhancements – can nest FOR XML statements to create a hierarchy of XML

documents • Indexes on XML Type Columns • Bulk Copy Format Files in XML

– an XML schema that can be used to specify bulk copy format files as XML documents

New Utilities

• New sqlcmd Utility – The sqlcmd utility replaces the osql and isql utilities – sqlcmd uses the OLE DB API to communicate with an instance of the SQL

Server Database Engine • while the earlier utilities used the ODBC or DB-Library APIs.

– sqlcmd also introduces a richer set of commands that allow it to operate better in application scripts,

• such as Visual Basic for Applications (VBA) scripts

• New sqlwb Utility – The sqlwb utility can be used to launch the new SQL Server

Management Studio from the command prompt

• New Profiler90 Utility – The Profiler90 utility can be used to launch the SQL Profiler tool

from the command prompt

• New dta Utility – used to launch the Database Tuning Advisor from the command

prompt

Notification Services

Rule Types • The operation of the generator is

controlled by the Transact-SQL rules defined for the application– Event chronicle rules store or

update the history of events in chronicle tables defined by the application developer

– Subscription event rules generate notifications for event-triggered subscriptions

– Subscription scheduled rules generate notifications for scheduled subscriptions.

Full Text Search

Snapshot Isolation

• Increased data availability for read applications– Allows non-blocking consistent reads in an OLTP environment– Copy on write mechanism– Writers don’t block readers– Readers don’t block writers

• Scenarios– Read-mostly database with relatively few writes – Reporting– Migration from Oracle to SQL Server

Database Snapshots

• Snapshot of a database at a point in time– Created instantly

– Read only

• Does not require a complete copy of the data– Shares unchanged pages of the database

– Requires extra storage only for changed pages

– Uses a “copy-on-write” mechanism

• Recover from User, Application or DBA error– Rewind database to Database Snapshot

Other replication changes

• Stand alone replication monitor• DDL on published objects supported seamlessly while

replicating• Merge performance and scalability improved 2-4x• New security architecture• New data types supported

– XML, CLR, varchar/varbinary(max) etc.

• New .Net API (RMO) for setup, admin and synchronization

Partitioning

• Range Partitioning breaks a single object into multiple

manageable pieces

• Partitioning is transparent to the application

• Partitioning applies to

– Tables

– Indexes

– Indexed views

• The row is the unit of partitioning

• All partitions reside in a single database

Partition Manageability

• Common operations– Split, merge, add, and drop partitions– Bulk data loading, Index maintenance

– Switch data instantly between• Table and partition of another table• Two partitions of different tables

TimeTime

Sliding WindowSliding Window

ArchivedArchived

HotRead-OnlyNear-Line Read-MostlyObsolete

SQL Server Data Access API Layer• SQL Native Client replaces Microsoft Data Access Components (MDAC)

– a stand alone data access API that is used for both OLE DB and ODBC – It combines the SQL OLE DB Provider and the SQL ODBC Driver into one native

dynamic link library (DLL) while also providing new functionality that is separate and distinct from MDAC

– create new applications to take advantage of new SQL Server features such as Multiple Active Result Sets (MARS), User-Defined Types (UDT), and XML data type support

• SqlClient Namespace and SQL Native Client- Multiple Active Result Sets (MARS)

– introduces the ability for multiple statements to return result sets at the same time on the same connection

• SqlClient Namespace and SQL Native Client: New Data Types – A new xml data type that supports storing XML documents and fragments in databases– ability of the char, nchar, varchar, nvarchar, binary, and varbinary data types to support

strings longer than 8,000 bytes

• Web Access: Native HTTP SOAP Access – Instances of the Database Engine can be configured with HTTP endpoints, or addresses

to which SOAP requests can be sent from applications– NO Need for IIS

HTTPS SynchronizationMerge Replication

-InternetUrl https://myserver.mydomain.com

Publisher/DistributorPublisher/Distributor

FirewallFirewall

IISIISSubscriberSubscriber

Replisapi.dllReplisapi.dll

•No VPN or port required for SQL Server•Eases deployment of mobile applications

•Off by default•Secure by default

•Only authenticated users can send request to web server•Requires HTTPS •Wizard turns off anonymous access

Programmability Enhancements - .Net

• Common Language Runtime Integration – The Microsoft .NET Framework Common Language Runtime V2.0 (CLR) is

now hosted in the SQL Server Database Engine– .NET languages such as C# and VB.NET– User-defined types and aggregates can also be written in .NET languages

• User-defined Aggregates – User-defined aggregate functions written in a .NET language extend the

basic aggregate functions, such as SUM, MIN, and MAX– An aggregate function is one that processes several data values and returns

a single value

User-defined Data Types – CLR user-defined data types (UDTs) allow you to implement database data

types in any of the languages supported by the .NET CLR– CLR UDTs are integrated with the Database Engine type system and can be

used in all the contexts where the built-in types are used

• Stored Procedures, Triggers, and User-defined Functions – Database procedural objects can be written in .NET languages to

incorporate more complex logic than is supported by the Transact-SQL language

Common Language Runtime Internals

SQL EngineSQL Engine

WindowsWindows

SQL OSSQL OS

CLRCLR

HostingHostingLayerLayer

• CLR goes through SQL OS for• Memory• Threads/fibers• Synchronization

• SQL-CLR host co-ordinates• Assembly Loading• Security management• App domains• Escalation policy for exceptions

CLR Memory Management

• All memory allocation from CLR through SQL Server• Most of CLR memory (GC Heap) comes from multi-page

memory allocator (outside Buffer-pool or “MemToLeave”)– Max server memory setting does not cover CLR memory

• On memory pressure– SQL informs CLR in order to initiate a GC

• DMVs for monitoring CLR memory usage– Sys.dm_os_memory_clerks– Sys.dm_os_memory_objects

• Perf counters: # GCs, amount of memory allocated• Memory clerks of interest: SQLCLR, SQLCLRASSEMBLY

CPU: threads / synchronization

• All managed “threads” are mapped to tasks managed by the SQL Server scheduler– Co-operative scheduling– Exception: GC threads

• SQL Scheduler detects non-yielding tasks (managed or T-SQL) and “punishes” them– Force the task to stop, put back in the queue and miss some turns

• Callback to SQL Server on P/Invoke – (when calling native code)– Switched to pre-emptive mode

• CLR lock requests go through SQL Server lock manager– Unified deadlock detection between locks held by CLR and native

SQL

Application Domain Management

• App-domains– CLR mechanism for code isolation, code unloading

• SQL Server uses two kinds of app-domains – DDL-time: temporary use during assembly DDL for verification– Execution time: actual code execution

• Granularity– One per assembly owner in each database– All assemblies owned by same user are in same app domain

• Cross-app domain mgd → mgd calls not allowed• DMVs of interest:

Sys.dm_clr_appdomains

Summary: DBA Productivity

• Reliable and secure

• Ease-of-use and TCO leader for database management

• Achieve high availability without added complexity, cost

• Scales from mobile device to the datacenter

SQL Server Summit 2005