Uk user group biz talk performance deepdive

Ricardo TorreProduct Strategist – BizTalk360

BizTalk Performance Deepdive

Understand architectural, design, and development patterns to improve BizTalk Server performance

Answer BizTalk performance related questions

Session Objectives and Takeaways

And a few other things

Development Optimizations

Optimize what?

Agenda

How to performance test, collect and analyse?

Architecture Optimizations

Optimize what?

Look at your architecture and identify optimization targets• BizTalk Applications• BizTalk Server• SQL Server• IIS Server• Windows Servers

How should I start?

Develop your applications with performance in mindDefine your testing strategy, tools and goalsCollect your baselineIdentify your current bottleneckChange somethingTestAnalyse

Use Visual Studio Load testing

Analyse during testing

Don’t be happy until you find a physical bottleneck

Development Optimizations

Adhere to BizTalk way of doing thingsDon’t work against it

Pipeline components

• Stream-based approach to pipeline components• Take advantage of BizTalk’s Streams

(Microsoft.BizTalk.Stream.dll)• VirtualStream and Event Streams• XPathMutatorStream• NamespaceTranslatorStream and XmlTranslatorStream

• Use PassThruReceive and PassThruTransmit pipelines whenever possible

• Promote items to the message context only if you need them for:• Message Routing (Orchestrations, Send Ports)• Demotion of message context properties (Send Ports)

• Instrument your source code to make your components simple to debug

• Avoid ordered delivery and transaction support whenever possible

Orchestrations

Tune the orchestration dehydration settings per the BizTalk Operations / Performance GuidesEliminate persistence points when possibleKeep orchestration state as small as possibleEliminate unnecessary context properties and distinguished fields

Use BizTalk Server 2013

XslTransform vs XslCompiledTransformA separate .NET class is generated for each transformation map.In most cases, the XslCompiledTransform class significantly outperforms the XslTransform class in terms of performance.Caveat with XslCompiledTransform: because the XSLT is compiled to MSIL, the first time the transform is run there is a performance hit, but subsequent executions are much faster.XslCompiledTransform class is the best choice in a "Load once, Cache, and Transform many times" scenario as the initial cost for map-compilation is highly compensated by the fact that subsequent calls are much faster.

BizTalk map (XslTransform) Test Case

1. One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder2. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb)3. The inbound request starts a new instance of the DefaultStaticLoop orchestration4. The orchestration executes a loop (1000 iterations) in which it uses a Transform Shape to apply a map to the inbound CalculatorRequest message (80KB)

XslCompiledTransform Test Case

1. One-Way FILE Receive Location receives a new CalculatorRequest xml document from the IN folder2. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb) 3. The inbound request starts a new instance of the CustomDynamicLoop orchestration4. The orchestration executes a loop (1000 iterations) in which it uses the XslCompiledTransformHelper static class to apply a map to the inbound CalculatorRequest message (80KB)

Test Results

BizTalk map (XslTransform) Test Case• Total Elapsed Time = ~57.3 seconds • Average Elapsed Time/Transformation = ~57 ms

XslCompiledTransform Test Case • Total Elapsed Time = ~3.6 seconds• Average Elapsed Time/Transformation = ~3.6 ms

Architecture Optimizations

Work with SME for each product in the stack

Optimize your architecture:• BizTalk Configuration Best practices• SQL Server/Disk Subsystem• IIS

Take the easy wins first

Apply all the BizTalk best practices

You will find the biggest performance improvements by removing logical bottlenecks

Be happy when you are CPU, IO or Memory bound

SQL Servers

The most common problem with SQL Server instances with BizTalk is PFS Contention

Don’t be afraid of using Multiple Message Boxes in a single SQL Instance

Invest on a good SAN

Multiple files and filegroups for the BizTalk MsgBoxDb

Primary (default file group)

Misc Data Misc IndexesPredicate

DataPredicate Indexes

Message Data

Message Indexes

For more information:BizTalk Server MessageBox Database Filegroups SQL Script

‘text in row’ in the MessageBox

Causes text, ntext, and image fields to be stored “in-row” instead of on a separate page in the tableMight be beneficial to enable this on the following tables in the MessageBox:• Parts• Spool• DynamicStateInfo_HostName

EXEC sp_tableoption N'Parts', 'text in row', '4000'

IIS Optimizations

Use Webgarden for IIS scalability

You can also tune the internal IIS engine for very specific

“UK Retailer” Compare Message Flow

“UK Retailer” Compare Hardware

2 BizTalk Servers: • 0.5 Business Transactions / second• 6.3 seconds avg. latency (5 sec wait time)

4 BizTalk Servers: • 0.69 Business Transactions / second• 11.2 seconds avg. latency

BUT…

“UK Retailer” Compare Results

Throughput will not increase, but we are not CPU or I/O bound… Why?

When a new row is inserted into an index, SQL Server will use the following algorithm to execute the modification: 1.Record a log entry that row has been modified. 2.Traverse the B-tree to locate the correct page to hold the new record. 3.“Latch” (lock) the page with PAGELATCH_EX, preventing others from modifying it. 4.Add the row to the page and, if needed, mark the page as dirty. 5.“Unlatch” the page

Eventually this can lead to “latch convoys” in highly concurrent workloads

Background – what is latch

Session ID Latch Type Database

Schema Object Name Index Name

Wait Type

241PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 109

273PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 63

348PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 63

177PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 62

302PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 62

311PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 62

108PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 47

110PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 47

122PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 47

166PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 47

234PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 47

300PAGELATCH_EX

BizTalkMsgBoxDb dbo

TrackingData_0_1

PK_TrackingData_0_1 47

294PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 46

107PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 31

111PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 31

121PAGELATCH_EX

BizTalkMsgBoxDb dbo Spool IX_Spool 31

Latch Convoy On Spool table

Contention on Small Number of Pages in Heavily Accessed Tables(Select, Update, Delete, Insert all running concurrently on multiple threads)

High rate of Insert/Select/Update/Delete access patterns against very small tables We have observed latch contention in MessageBox schema on 16 Cores and aboveThis problem is now being reported by production deployments, waits 100+msImpact: Latch waits > 50-100ms+Observed waits on internal SQL Structure• ACCESS_METHODS_HOBT_VIRTUAL_ROOT

WorkaroundMinimize load on MessageBox • App design to avoid persistence points• Use inline sends if necessary

Multi-MessageBox may alleviate this problem somewhat

BizTalk Workload Characteristics

Resources

Visual Studio Load TestingOrchestration ProfilerBizTalk Performance Optimization GuideBizTalk Benchmark Wizard

Q & A