Business Activity Monitoring in Depth for Developers Published: January 2009 Authors: Jesus Rodriguez and Joe Klug, Tellago, Inc. Technical Reviewers: Ofer Ashkenazi, Microsoft Marcel Fernee, Microsoft Stephen Kaufman, Microsoft Brian Loesgen, Neudesic LLC Jonathan Moons, Microsoft Allan Naim, Microsoft Paolo Salvatori, Microsoft Andy Shen, Microsoft Applies to: BizTalk Server 2006 R2 and BizTalk Server 2009 To download a copy of this document and the sample code, go to http://go.microsoft.com/fwlink/?LinkId=139086 . Summary Microsoft added Business Activity Monitoring (BAM) to Microsoft® BizTalk® Server as a way for business analysts to track information that is important to them. This paper provides a deep, low-level review of how BAM works and how BAM can be extended. It reviews BAM basics, shows ways in which BAM data can be collected, discusses the BAM infrastructure, demonstrates methods for collecting BAM data, and shows how BAM can be extended to non-Microsoft technologies.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Business Activity Monitoring in Depth for Developers
Published: January 2009
Authors: Jesus Rodriguez and Joe Klug, Tellago, Inc.
Technical Reviewers: Ofer Ashkenazi, Microsoft Marcel Fernee, Microsoft Stephen Kaufman, Microsoft Brian Loesgen, Neudesic LLC Jonathan Moons, Microsoft Allan Naim, Microsoft Paolo Salvatori, Microsoft Andy Shen, Microsoft
Applies to: BizTalk Server 2006 R2 and BizTalk Server 2009
To download a copy of this document and the sample code, go to
http://go.microsoft.com/fwlink/?LinkId=139086.
Summary
Microsoft added Business Activity Monitoring (BAM) to Microsoft® BizTalk® Server as a way for business analysts to track information that is important to them. This paper provides a deep, low-level review of how BAM works and how BAM can be extended. It reviews BAM basics, shows ways in which BAM data can be collected, discusses the BAM infrastructure, demonstrates methods for collecting BAM data, and shows how BAM can be extended to non-Microsoft technologies.
CopyrightThe information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Microsoft, BizTalk, Excel, Internet Explorer, MSDN, Outlook, PerformancePoint, PivotTable, SharePoint, and Windows are trademarks of the Microsoft group of companies.
All other trademarks are property of their respective owners.
Executive SummaryMicrosoft added Business Activity Monitoring (BAM) to Microsoft® BizTalk® Server as a means for business analysts to track information that is important to them. One common example of this is tracking the dollar amounts of every transaction processed (i.e., total sales from a particular region). Since BAM was first introduced in 2004, there have been a number of white papers, webcasts, and examples about the basics of BAM. Unfortunately, what’s been missing is a deep, low-level review of how BAM works and how BAM can be extended.
In this white paper we will take you beneath the covers of BizTalk Server Business Activity Monitoring. After reviewing the basics, we will show you the various methods by which BAM data can be collected, from the pre-built interceptors to the APIs that can be used in custom solutions. The second part of the paper digs into the BAM infrastructure, explaining the BAM databases and what happens to them when BAM models are deployed, and the role of SQL Server® Integration Services in BAM. In the third part of this paper we will demonstrate a number of methods for querying BAM data, covering the use of data access libraries, SQL Server Reporting Services, Microsoft Office PerformancePoint® Server and Microsoft Office SharePoint® Server, and Web services. Finally, this paper covers how BAM can be extended to non-Microsoft technologies through the use of Web services and programming models such as Representational State Transfer (REST).
The four sections of this paper are relatively independent of each other. You don’t have to read the section on the BAM infrastructure to understand how to extend BAM with REST. Our only suggestion is that you cover the basics at the beginning of this paper. We have also included with the white paper the samples that contain the code that is referenced in this paper. These examples can be compiled and deployed against a basic BizTalk Server installation.
Business Activity Monitoring OverviewBusiness Activity Monitoring (BAM) is one of the rapidly evolving areas of Business Process
Management (BPM). Conceptually, BAM focuses on providing nearly real-time monitoring
and analysis of business process activities. We can think of BAM as a bridge between
integration and business intelligence (BI) techniques. Ultimately, the goal of BAM is to
enable sophisticated BI techniques based on the information collected from business
processes.
In order to achieve this goal, BAM uses BI models based on the information from different
business processes. These models are typically known as event or activity models and are
normally translated into relational and multidimensional databases. BAM technologies
populate those models using events that describe important milestones of a business
process. BAM technologies are responsible for triggering those business events and mapping
them to the activity models. The following figure conceptually illustrates a typical BAM
architecture.
Figure: BAM concepts
As a technology, BAM technology stacks have emerged as a fundamental component of
integration and BPM suites such as Microsoft BizTalk Server, IBM WebSphere Process Server,
Oracle SOA Suite, etc. This is mostly a strategic decision given that many of the business
processes in the industry today are implemented by using those technology suites.
In the case of BizTalk Server, the first version of BAM was introduced with the 2004 release
and followed by subsequent versions in 2006 and 2006/R2.
BizTalk Server BAM: Exploring the activity modelAlthough BAM is released as part of BizTalk Server, it provides a flexible framework that can
be used to instrument any .NET Framework-based application. Additionally, as we will
explore in the last section of this paper, BizTalk Server BAM can be extended to interoperate
with non-.NET technologies such as J2EE or dynamic languages.
BizTalk Server BAM is built around the concept of an activity model. This type of model
defines relevant business information about a business process. The fundamental element of
a BizTalk Server BAM activity model is an activity. Activities represent atomic information
units that need to be monitored. As an analogy in the database world, we can think of
activities as entities of an entity relationship model. In that sense, activities are described by
a collection of fields that represent business data or important business milestones. For
instance, a banking transaction activity can contain a data field that indicates the monetary
amount of the transaction and another milestone field that represents the time in which the
transaction was processed.
Milestones are a key concept of BAM that, as explained in the previous example, capture
important states of a business process. In addition to business milestones (DateTime),
activity fields can be of type Text, Float, and Integer. Activities can have relationships with
other activities. For instance, the transaction activity can have a relationship with the
customer activity.
The data included in a set of activities can serve to provide relevant information to different
domains. For instance, a bank executive might use a BAM activity set to obtain some
financial analysis metrics while an IT manager might be more interested in seeing the
average number of failed and successful transactions. To address these scenarios BizTalk
Server BAM introduces the concept of an activity view, which projects a specific
representation of a set of BAM activities.
One of the main advantages of activity views is that they can combine the information of
various BAM activities, providing different real-time perspectives of a business process.
Additionally, activity views can capture historical metrics incorporating multidimensional
elements such as measures and dimensions. These concepts are the equivalent of the
measures and dimensions included in most OLAP engines.
Measures are numeric aggregations of an activity field; for instance, a measure
AVG(Transaction Amount) will calculate the average of the financial amount of the
transactions represented by the banking transaction activity. Measures are typically
segmented by dimensions, which are information hierarchies that represent areas of interest
in the information; for instance, we can use a Time dimension with three levels (year,
month, and day) to analyze the information provided by the measure AVG(Transaction
Amount) across different time intervals. The current release of BizTalk Server BAM includes
the concept of time, data, numeric range, and progress dimensions.
Progress dimension: Represents the creation of aggregations with respect to the
progress of activities that are still in process. For instance, a progress dimension can
represent the status of a financial transaction with the levels (RECEIVED,
PROCESSING, APPROVED, DENIED).
Data dimension: Used to categorize an aggregation. Data dimensions are based on
the value of string-formatted data items in the BAM activity.
Time dimension: Used to create aggregations across defined time segments. For
instance, a time dimension with levels Year, Month, Day can be used to segment the
measures of the banking transaction activity.
Numeric range dimension: Used to categorize aggregations based on friendly
names of given numeric ranges. For instance, a numeric range dimension can
categorize the amount of a banking transaction into levels Small ($1-$10000),
Medium ($10001-$100000), and Large ($100001-$1000000).
Dimensions and measures can be used to define real-time or scheduled aggregations. A
scheduled aggregation represents a time snapshot of an activity view. In order to execute all
the calculations required for the measures and dimensions, this type of aggregation is
scheduled to run based on specific time intervals. There are some scenarios in which
different parts of an aggregation need to be available nearly in real time. For those
scenarios, BizTalk Server BAM introduces the concept of a real-time aggregation (RTA),
which executes some of the required calculations as the activity data is being populated.
The “Inside the BAM Infrastructure” section drills down into the aggregation model.
BizTalk Server BAM represents the activity model using an XML-based language known as
the BAM definition language. This language is described by the BAM definition schema. The
BAM definition schema defines the constraints of the elements of a BAM activity model
including activities, views, cubes, etc. Developers can create BAM activity models by using
their favorite XML editor or specialized tools like the BAM Add-in for Microsoft Office Excel®
illustrated in the following section.
It is important to notice that the process of defining a BAM activity model is completely
independent of BizTalk Server. Developers or analysts can define activity models to
instrument applications implemented using heterogeneous technologies. Once created, an
activity model is deployed to BizTalk Server so that it can be used at run time. In that sense,
the elements of the BAM activity model are translated into relational artifacts deployed in
different BizTalk Server databases.
Throughout this paper we are going to explore in detail the intricacies of BAM activity
models. Specifically, the “Inside the BAM Infrastructure” section of this paper drills down into
the different components of a BAM activity model and how they are represented in the BAM
infrastructure.
In order to get a better understanding of the process of creating a BAM activity model, the
next section will walk us through the process of creating a sample activity model that we will
use in the remaining sections of this paper.
Defining a BAM sample activity modelA traditional process of some financial institutions like banks is to process credit card
transactions. As part of this process a transaction needs to be approved or denied based on
certain criteria like the debt or credit limits associated with the account. Additionally, if the
characteristic of a specific transaction matches some well-defined fraud pattern, an alert is
triggered in order to take the appropriate actions. In this section, we are going to define an
oversimplified transaction activity model that will serve as the basis for all the examples
included in this paper.
Conceptually, the fundamental elements of our model are the Transaction and Alert
activities detailed in the following tables.
Field name DescriptionAmount Decimal: Amount of the intended transaction
Transaction_Approved Milestone: Time at which the transaction was
approved
Transaction_Created Milestone: Time at which the transaction was
created
Transaction_Failed Milestone: Time at which the transaction
failed
Transaction_Failed_Alert_Triggered Milestone: Time at which the transaction
failed and an alert was fired based on a fraud
criteria
Customer_Age Integer: The age of the customer initiating
the transaction
Customer_Name Text: The name of the customer initiating the
transaction
Figure: Transaction activity
Field name DescriptionCode Text: Code associated with the alert
Description Text: Description of the alert
Alert_Created Milestone: Time at which the alert was
created
Alert_Processed Milestone: Time at which the alert was
processed
Figure: Alert activity
Using the BAM Excel Add-in we can create the activity definition that contains the previous
elements as illustrated in the following figure.
Figure: BAM activity definition
At this point, we can extend the information of the TRANSACTION activity with some
multidimensional elements that facilitate more sophisticated historical analysis over the
activity data. To achieve that, we will create the following dimensions and measures.
Element Type Description
Count_Transaction Measure Number of processed
transactions
Avg_Transaction Measure Transaction amount average
Total_Transaction Measure Total amount of the
processed transactions
Count_Alert Measure Number of triggered alerts
Transaction_Time_Dim Dimension A time dimension to segment
the transaction information
based on Year, Month, Day,
Hour levels
Transaction_Stg_Dim Dimension A process dimension to
segment the transaction
information based on its
different milestones
Amount_Dim Dimension A numeric range dimension
categorizing the amount of a
transaction into Small,
Medium, or Large
Alert_Time_Dim Dimension A time dimension
segmenting the alert time
into Year, Month, Day, Hour
levels
Figure: Transaction dimensions and measures
Using the Business Activity Monitoring View Wizard of the Excel Add-in we can define those
elements as illustrated in the following figure.
Figure: BAM activity view aggregation elements
Each of the multidimensional elements is defined separately.
Figure: Defining measures and dimensions
After defining the multidimensional elements, we can add the final touches to our BAM
activity view by creating specific PivotTable® reportsusing the multidimensional elements of
the activity model. The following figure illustrates this step.
Figure: Customizing a BAM activity view
After exporting the Excel definition to XML we obtain a representation of the TRANSACTION
activity model compatible with the BAM definition schema. Although the BAM Excel Add-in
is, undoubtedly, the preferred tool for creating BAM activity models, developers can build
custom tools that provide a more sophisticated user experience to help domain experts to
create specific activity models. Similarly to the Excel Add-in, these would translate the
graphical representation into the XML that can be processed by BizTalk Server. The following
code illustrates sections of the XML representation of our sample activity model.
<BAMDefinition xmlns="http://schemas.microsoft.com/BizTalkServer/2004/10/BAM"> <Activity Name="TRANSACTION" ID="IDEAA8FD2CA37D4C82A5F7BB7966D4E4EC">…Sections have been omitted for brevity.. </Activity>
…Sections have been omitted for brevity..<View Name="TRANSACTION" ID="ID12CBC52A28684F3BAE8F742DCF545A18"> <ActivityView Name="ViewTRANSACTION"
uri()='']" XPath="/*[local-name()='Request' and namespace-uri()='http://BAM_BTS.TransactionRequest']/*[local-name()='Created' and namespace-uri()='']" /> </Dimension> Other elements...
</TrackingProfile>
Looking at the code we can notice that it describes how the TRANSACTION_CREATED activity
field is populated from a BizTalk message payload. The highlighted sections of the tracking
profile XML detail the orchestration shape, message, and XPath sentence that are used to
populate the TRANSATION_CREATED activity.
At this point, the TPE profile can be deployed instructing the BAM runtime where and how to
populate the different elements of the BAM activity model.
Inside the BAM BizTalk interceptorThe architecture of the BizTalk Server BAM runtime is complex enough to require a separate
paper. We will try to cover some of the most important components in this section. After it’s
deployed, a BAM tracking profile is stored in the bam_Metadata_TrackingProfiles table in the
BAMPrimaryImport database, with a reference to it in the bam_TrackingProfiles table of the
BizTalk Management database (BizTalkMgmtDb). By default the BizTalk Server runtime uses
the different interceptors that can be found in the Trackinginterceptor table of the BizTalk
Management database.
Fundamentally, the BizTalk engine uses the Microsoft.XLANGs.Mozart.NativeInterceptor and
Microsoft.XLANGs.Mozart.BIWrapperInterceptor, which inherit from the
Microsoft.XLANGs.RuntimeTypes.BaseInterceptor class. The BaseInterceptor class is the
main interface that the BizTalk Server runtime components use to interact with the BAM
infrastructure.
Figure: BizTalk Server BAM interceptor
The complete architecture of the BizTalk Server BAM runtime is far more complex than what
has been explained in this section, but we intended to illustrate some of its fundamental
components.
Using BAM from Windows Communication Foundation and Windows Workflow FoundationWith the release of BizTalk Server 2006 R2, the use of BAM has been extended to Windows
Communication Foundation (WCF) and Windows Workflow Foundation (WF). Both
technologies introduced a declarative mechanism to express the element that should be
monitored by BAM. This mechanism is based on an interceptor configuration file that
declares how the BAM activities are populated from WCF or WF elements. The interceptor
configuration files for both WCF and WF share the same structure based on the
CommonInterceptorConfiguration.xsd XML schema, which describes generic events that
need to be intercepted and mapped to BAM activities.
The common configuration file provides a schema that can be reused across the WCF and
WF BAM interceptors. However, given the differences between the WCF and WF
programming models, that common schema needs to be extended to incorporate the
specific interception mechanisms for collecting the BAM data. For instance, developers using
the BAM WCF interceptor will, most likely, extract the data from a SOAP or XML message
intercepted at different stages of the client or dispatcher runtimes. On the other hand,
developers using the WF interceptor will extract the activity data from WF-specific
components such as workflow properties or user data types. Essentially, in order to provide
a complete declarative mechanism for populating a BAM activity model, the WCF and WF
interceptors need to extend the common configuration schema with specific elements of
their programming model.
BAM WCF interceptor The BAM WCF interceptor is a flexible technology that allows developers to populate BAM
activity models based on the information exchanged by WCF services and clients. In order to
model the interactions with the BAM model, the BAM WCF interceptor extends the basic
configuration model with WCF-specific elements. Essentially, the interceptor configuration
file models how WCF messages are mapped to BAM activities throughout the different
stages of the client or dispatcher runtimes.
Let’s take the following WCF service that processes a financial transaction.
public class FinancialTransactionService: IFinancialTransactionService { public TransactionResponse ProcessTransaction(TransactionRequest transactioninfo) { //Implementation omitted for brevity.... } }
[ServiceContract] public interface IFinancialTransactionService { [OperationContract] TransactionResponse ProcessTransaction(TransactionRequest transactioninfo); }
[DataContract] public class TransactionRequest { private string transactionId; private double amount; private string customerName; private int customerAge;
[DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; }
}
[DataMember] public double Amount { get { return amount; } set { amount= value; } }
[DataMember] public string CustomerName { get { return customerName; } set { customerName= value; } }
[DataMember] public int CustomerAge { get { return customerAge; } set { customerAge = value; } } }
[DataContract] public class TransactionResponse { private string transactionId; private bool success; private bool alertFired; private string alertDescription;
[DataMember] public string TransactionId { get { return transactionId; } set { transactionId = value; } }
[DataMember] public bool Success { get { return success; } set { success= value; } }
[DataMember] public bool AlertFired { get { return alertFired; } set { alertFired= value; } }
[DataMember]
public string AlertDescription { get { return alertDescription; } set { alertDescription= value; } }
}Figure: Sample WCF service (the complete implementation can be found in the
samples included with this paper)
At this point we can create a WCF interceptor configuration file that maps the WCF
messages to the sample activity model created in the previous section. The following figure
highlights sections of the interceptor configuration file that indicate how the
TRANSACTIONID element is populated based on the contents of the upcoming message.
Like any other WCF behavior, the BAMEndpointBehavior is configured by a behavior
extension element (BAMBehaviorExtension). Once instantiated, the BAMEndpointBehavior
behavior inserts a WCF message inspector (BAMMessageInspector, which inherits from the
BAMInspector class) into the client or dispatcher runtime. At run time, the message
inspector intercepts a WCF message and creates a BAM interceptor (BAMWcfInterceptor),
which interprets the interception configuration file in order to log the data to the
BAMPrimaryImport database. As you might imagine, there are other components involved in
the BAM WCF interceptor architecture, and it is not our goal to cover them all in detail. From
a developer standpoint, the complexity of this architecture is completely abstracted behind
the BAMBehaviorExtension behavior extension element.
BAM WF interceptorSimilar to the BAM WCF interceptor, the BAM WF interceptor extends the BAM programming
model into WF. Using this interceptor, developers can populate BAM activity models based
on the information processed by WF workflows. The mechanism to declare how a BAM
activity model will be populated is also based on an interceptor configuration file, which
extends the basic interceptor configuration model with specific WF elements.
To better understand how to use the BAM WF interceptor let’s start with the following
workflow that processes a financial transaction.
Figure: Sample transaction workflow
This workflow receives the financial transaction information and executes a very simple logic
to approve or reject the transaction. For this example, the fields that we would like to
monitor are declared as properties of the workflow class as illustrated in the following code.
public sealed partial class FinancialTransactionWorkflow: SequentialWorkflowActivity{ private double transactionID; ...rest of the implementation has been omitted for brevity public double TransactionID { get { return transactionID; } set { transactionID = value; } }
private void TransactionApproved_ExecuteCode(object sender, EventArgs e) { transactionID= sample transaction id….; ...rest of the implementation has been omitted for brevity }}
Figure: Transaction workflow implementation (you can find the complete implementation as part of the samples included with this paper)
In order to populate a BAM activity model based on the state of the WF workflow we need to
create a BAM WF interceptor configuration file like the one in the following figure.
…Rest of the elements have been omitted for brevity….
</LobSystem>
Figure: BDC model for the Transaction BAM activity
Once created, the BAM metadata model can be deployed by using the MOSS administration
console. After the BDC model is deployed the BAM entities can be used from any MOSS Web
site with the proper privileges. To render the BAM activity records we can use some of the
BDC Web parts included in MOSS. In our scenario we will use the Business Data List Web
Part to render all the instances of the transaction activity and the Business Data Item Web
Part to render the details of a specific activity instance (see the following figure).
Figure: Configuring BAM-BDC Web parts
Once configured, the BDC Web part will provide real-time access to the information stored in
the BAMPrimaryImport database activity views.
Figure: Accessing BAM data by using BDC
As we explained at the beginning of this section, one of the benefits of integrating BAM with the BDC is the capability of exposing BAM data to other MOSS technologies. For instance, we can associate BAM activity instances with Office documents that provide additional information about the business processes.
Figure: Using BAM activities in a MOSS document library
Accessing BAM by using Web servicesIn the first part of this section we explored different Microsoft products and technologies that
can complement BAM with enterprise capabilities such as reporting, collaboration, or KPI
analysis. Undoubtedly, the combination of BAM with technologies like SSRS,
PerformancePoint Server, and MOSS is the correct answer to a large variety of the most
common BAM scenarios in the enterprise. However, these technologies provide the most
dividends in scenarios like end-user reports that have certain design-time knowledge of how
the BAM activity data needs to be processed. Looking back to our previous examples, all of
them consume the BAM data by using artifacts created at design time like reports or KPIs.
However, these types of techniques are not always the best fit in scenarios that require a
more dynamic interaction with the BAM activities.
Querying the BAMAnalysis database by using the XML for Analysis Web serviceXML for Analysis (XMLA) is an open standard for querying multidimensional data sources by
using Web services. The last two editions of SQL Server Analysis Services (SSAS 2005 and
SSAS 2008) support XMLA 1.1 as one of the programming models used by client applications
to interact with an SSAS instance. Even more importantly, XMLA is the fundamental protocol
underneath other multidimensional programming technologies like ADOMD.NET and Analysis
Management Objects (AMO). Essentially, XMLA proposes a model for encoding commands
such as MDX queries in a SOAP envelope that can be processed by an XMLA-compatible Web
service. In its current version, the XMLA specification describes two generally accessible
operations: Discover and Execute. The Discover operation is used to query metadata about
multidimensional data sources like cubes, dimensions, etc. For instance, by using that
operation an application can browse the list of BAM cubes, dimensions, and measures
Figure: Using the XMLA Execute operation to query the BAMAnalysis database
In SSAS 2005 and SSAS 2008, developers can configure HTTP-SOAP XMLA access by
configuring the ISAPI extension deployed in the Web directory. Then, developers can
generate an XMLA proxy using the technology of choice. The following code illustrates a
WCF client that invokes the Execute operation to run an MDX query against the
TRANSACTIONS OLAP cube deployed as part of our sample BAM activity model.
MsXmlAnalysisSoapClient xmlaService = new MsXmlAnalysisSoapClient(new BasicHttpBinding(), new EndpointAddress("http://localhost:8080/olap/msmdpump.dll"));XmlDocument xdoc= new XmlDocument();
Figure: Using the XMLA Web service to query the BAMAnalysis database
Although powerful, the use of the XMLA Web service is restricted to the BAMAnalysis
database. If an application is required to use Web services to access the activity fact data
stored in the BAMPrimaryImport database there is another option available: the BAM Query
Web service.
Querying the BAMPrimaryImport database by using the BAM Query Web serviceAlthough not officially supported, the BAM Query Web service provides an interesting
solution to applications that need to query the BAM activity data stored in the
BAMPrimaryImport database. The BAM Query Web service provides a single operation,
GetInstanceData, that abstracts a T-SQL query to a BAM activity view. The following code
shows a sample client that uses the BAM Query Web service to query our sample activity
model.
BamQueryServiceSoapClient queryService = new BamQueryServiceSoapClient( new BasicHttpBinding(), new EndpointAddress( "<server url>/BamQueryService.asmx"));queryService.ClientCredentials.Windows.ClientCredential = new System.Net.NetworkCredential(<username>, <password>);InstanceQuery query= new InstanceQuery();query.SelectClauses = new string[] { "ActivityID, TRANSACTIONID, AMOUNT, TRANSACTION_CREATED, CREDITCARD" };Column[][] columns= queryService.GetInstanceData("dbo.bam_TRANSACTION_AllInstances", "TRANSACTION", query, 0);
Figure: Using the BAM Query Web service
The use of both XMLA and the BAM Query Web service considerably improves the BAM
implementations in distributed environments and also facilitates the interoperability of the
BAM infrastructure with non-Microsoft technologies such as J2EE or dynamic languages.
However, it is important to highlight that developers should use these Web services only as
a very specific mechanism to query BAM data and never as a general-purpose API to
abstract the interactions with the BAM infrastructure. Neither the BAM Query Web service
nor XMLA provide any abstraction of the BAM data model. Instead, both of them require the
consumer application to have intrinsic knowledge of the schema of the BAM databases.
As we’ve explored in the different sections of this paper, BizTalk Server BAM provides a
generic infrastructure that does not impose any restrictions on the type of client applications
that can interact with an activity model. However, from an API standpoint, the use of BAM is
currently reduced to .NET Framework-based applications such as the ones developed in
BizTalk Server, WCF, or WF. Certainly, abstracting the BAM infrastructure with an open-
standard API can enable the adoption of BAM across heterogeneous technologies. Although
currently there is no “supported” API that serves this purpose, developers can leverage
technologies like WCF and emerging architecture styles like Representational State Transfer
(REST) as a vehicle to service enable their BAM infrastructure.
Building a BAM RESTful API
Why do we need a new API?Throughout this paper we have explored different mechanisms developers can use to
interact with BizTalk Server BAM. Although the current technology provides developers with
powerful alternatives to instrument their applications using BAM, there are a few
outstanding challenges in order to ubiquitously adopt BAM in heterogeneous enterprise
environments.
One of the most attractive capabilities of BizTalk Server BAM is that its infrastructure is not
dependent on BizTalk Server. This means that, theoretically, non-.NET Framework-based
applications can benefit from the use of BAM. However, even though this is true from the
BAM infrastructure standpoint, the APIs required to interact with BAM are only available
to .NET Framework-based applications.
Another challenge that developers face when implementing BAM applications is the lack of a
consistent programming model for querying and executing CRUD (Create-Update-Delete)
operations on BAM activities. In that sense, developers need to interact with BAM stream-
based APIs to populate elements of an activity model and with a completely different model
when it comes to querying BAM data. This contrasts with other data-driven programming
models such as ADO.NET that abstract both functionalities through a consistent API.
To improve the consistency and interoperability of the BAM programming model, developers
often find themselves creating Web service interfaces that abstract the BAM stream APIs.
Typically, those Web service interfaces are based on the SOAP-WSDL-WS-* model and
expose the specific BAM capability that is initially required by the applications using them.
Extending these APIs to include the broad set of BAM capabilities will introduce the
traditional versioning challenges that large SOAP-WSDL APIs face in the industry today.
Another limitation of this model is the constraint on adoption to technologies that don’t have
a powerful SOAP-based engine. That list includes some of the most powerful technologies for
building rich Web user interfaces that consume BAM data, like dynamic or scripting
languages and AJAX toolkits.
Being aware of all these challenges, this section will propose a slightly different model that
leverages the principles of a set of emerging service-oriented architecture styles.
Specifically, this section will explore the design of BAM Web service APIs based on the
Representational State Transfer (REST) model.
Embracing the Web programming principlesAlthough this section is not intended to discuss the pros and cons of Representational State
Transfer (REST) vs. traditional SOAP/WS-* services, we think is important to highlight why
REST is a better choice for designing a consistent BAM Web services API.
The term REST was first enunciated by Roy Thomas Fielding as part of his PhD dissertation.
At a high level, REST intended to formulate the principles that make the Web work in a
scalable and performant way. Fielding’s arguments highly resonated in the Web service
community in which REST started to be positioned as an alternative to the traditional
SOAP/WS-* model for Web services, firing off one of the most fascinating debates in the
history of distributed computing. In spite of all those arguments, the fact is that REST, as an
architecture style, is a great solution for implementing resource/entity-based services like
the ones required for a BAM API. Essentially, RESTful services follow a set of well-established
principles that differentiate them from other alternatives. The following list summarizes
some of the most important characteristics of RESTful services:
Give every resource an ID: Fundamentally, RESTful services abstract interactions
with resources (BAM activities, database entities, documents, etc). This principle
states that those resources should be addressable and uniquely identified. The
typical way to satisfy both requirements is to use URIs as the unique resource
identifier. For instance, a BAM activity instance can be identified by the URI, where
activity ID is the unique BAM identifier for that activity instance.
Link things together: In a typical REST system, resources can be related to each
other. For instance, two BAM activities can have a reference relationship or a
customer resource can have a relationship with the account resource. As resources
are identified by URI, one resource will include the links to its related resources as
part of its representation so that they can be individually addressed.
Use standard methods: Web architectures are based on HTTP. REST enunciates
how the different HTTP verbs should be associated with resource operations. In
essence, resources can be created using an HTTP POST, retrieved using a HTTP
GET, updated by an HTTP PUT, and deleted using an HTTP DELETE. Other
operations like metadata retrieval can be abstracted by other HTTP verbs such as
HEAD or OPTIONS.
Resources with multiple representations: When we design Web services using
the traditional SOAP/WS-* techniques, we are using SOAP as the single
representation format. In RESTful services, resources are conceptual entities that are
independent of the representation format used on the wire. For instance, a BAM
activity instance can be represented by using XML, JavaScript Object Notation (JSON),
or Atom formats.
Communicate statelessly: Web architectures are stateless by definition and this is
one of the characteristics that makes them highly scalable. By stateless, we mean
that any state of the interaction should be carried in the resource itself and not in a
“server session.”
Designing RESTful services in the Microsoft platform has come a long way and it is safe to
say that the current release of Windows Communication Foundation (WCF) can be
considered one of the most solid REST development platforms in the industry today. The use
of WCF to build a BAM RESTful API will be illustrated in the following sections of this paper.
Following the REST principles, we can start formulating what a BAM RESTful API looks like.
The following section will explore some of the design alternatives to expose BAM capabilities
using RESTful services.
Brainstorming a BAM RESTful APIThe first aspect to consider is that the BAM conceptual model can naturally be modeled by
using the principles of Resource Oriented Architecture (ROA). We emphasize the term
natural because, even though we can also model a generic BAM API by using RPC-based
concepts like operations or parameters, ROA transparently reflects the concepts behind the
BAM programming model. In that sense, the entire BAM infrastructure can be atomically
represented as a set of resources such as activities, activity views, cubes, or activity models.
These BAM resources can be related to each other; for instance, a BAM activity can have a
reference relationship with other BAM activities. When adopting this architecture style, we
can abstract the entire BAM development process as a series of query and CRUD operations
against BAM resources.
Having represented the BAM programming model as resources, it is time to think of how to
uniquely identify instances of BAM resources. To embrace the addressability principle of
REST we can use URI as the unique identifier of a BAM resource. For instance, an activity
instance can be identified by the URI http://<my BAM server>/activities/{activity
name}/{activityid}.
At this point, our next challenge is how to adapt the different HTTP verbs to represent
queries (HTTP GET) or CRUD (HTTP POST, PUT, and DELETE respectively) operations against
BAM resources. This process is highly dependent on the aforementioned URI model and
should correctly represent the BAM conceptual model. The following list illustrates some
examples of how to use the HTTP verbs to model BAM operations following the REST
principles:
An HTTP GET against http://<my BAM server>/activities/{activity name} will
return the list of instances of the activity identified by {activity name}. Each instance
will be identified by a URI in the form of http://<my BAM
server>/activities/{activity name}/{activity id}.
An HTTP POST against http://<my BAM server>/activities/{activity name} will
create a new instance of the {activity name} BAM activity. The new instance will be
identified by a URI in the form of http://<my BAM server>/activities/{activity
name}/{activity id}.
An HTTP GET against http://<my BAM server>/activities/{parent activity
name}/{activity id}/{child activity name} will return the instances of the {child
activity name} BAM activity related to the {activity id} instance of the {parent
activity name} BAM activity.
An HTTP PUT against http://<my BAM server>/activities/{activity
name}/{activity id} will update the {activity id} instance of the {activity name}
BAM activity.
Following the same RESTful principles, you can think of similar models to apply to other BAM
elements such as views, alerts, or cubes.
An aspect to consider when designing a BAM RESTful API is that some BAM resources
represent metadata associated with other BAM resources. For instance, an activity model
resource will contain the definition of the different activities and views included in that
model. In that sense, the activity model resource contains metadata associated with the
activity resource.
At this point, we already know how to model the BAM elements as resources and how to
associate the HTTP verbs with operations against those resources. That brings us to one of
the most important aspects of a RESTful API, which is to select a resource representation
format. This is a decision that we don’t normally face when designing SOAP/WS-* APIs
because, in that case, SOAP is the only representation format. Going back to the previous
section, one of the principles of REST states that resources are representation agnostic. In
that sense, we could represent the same resource by using an XML dialect, JSON, or a
syndication format such as RSS or Atom. Although, at first glance, plain XML might seem the
perfect choice, the fact is that creating a resource representation language entails a lot of
details that ultimately can result in interoperability challenges. This is the reason why many
successful RESTful APIs use well-established languages like RDF or Atom. The case of
syndication formats such as Atom or RSS is particularly popular in scenarios such as event
processing in which resources are modified at a high frequency. Additionally, AtomPub is a
well-established standard with a large support on heterogeneous platforms. Finally, the
extensibility of AtomPub makes it an ideal language for representing untyped structures
such as BAM activities. Considering all those factors, we decided to leverage Atom and the
Atom Publishing Protocol (AtomPub) as the resource representation language for our sample
BAM API. As you might already suspect, a complete BAM RESTful API will include a large
variety of operations that will be very difficult to cover in this paper. However, in order to
give a pragmatic view of how BAM can take advantage of the REST architecture style to
extend its functionality, the following section will explore the implementation of some of the
operations of a BAM RESTFul API using WCF 3.5.
A sample BAM RESTful APIAs we explained previously, this section does not intend to describe an entire BAM RESTful
API. Instead, we intend to explore the implementation of a few operations highlighting the
fundamental principles that can be applied to the implementation of other BAM RESTful
operations. For the simplicity of the examples, let’s start with the following WCF contract.
The first two operations are intended to navigate through the BAM activities deployed in a
BAMPrimaryImport database while the last three operations expose the capabilities required
for creating and updating BAM activity instances. As you can see all the operations return an
instance of Atom10FeedFormatter, which represents an Atom feed.
GetActivities
This operation executes based on an HTTP GET request and returns an Atom feed with the
definition of the BAM activities deployed in a specific BAMPrimaryImport database. The
following code highlights a section of the implementation of this operation (the entire
implementation can be found with the samples that accompany this paper).
public class SampleBAMService: ISampleBAMService{ [WebGet(UriTemplate="/activities")] public Atom10FeedFormatter GetActivities() { /*Retrieves the list of activities from the BAMPrimaryImportDB database. Details omitted for brevity…see the attached samples for the details */
Notice that, following the principles of REST, each activity is individually addressable by a
URI in the form of http://<service endpoint>/activities/<activity name>. Given that the
response is encoded using Atom it can be rendered in a Web browser as illustrated in the
following figure.
Figure: GetActivities response rendered in Internet Explorer®
Selecting a specific activity will invoke the GetActivity operation to retrieve its instances.
GetActivity
This operation returns an Atom feed that includes either all instances or a specific instance
of a BAM activity and is activated by executing an HTTP GET to a URL in the form
http://<service endpoint>/activities/<activity name>/*|<activity instance id>. The following
code illustrates fragments of the implementation of the GetActivity operation.
[WebGet(UriTemplate="/activities/{activityname}/{instanceid}")]public Atom10FeedFormatter GetActivityInstances(string activityname, string instanceid){ /*Retrieves the list of activities instances from a specific BAM activity Details omitted for brevity…see the attached samples for the details */
BAMActivityInstance[] instances=
bamdb.GetBAMActivityInstances(activityname, instanceid); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); foreach (BAMActivityInstance instance in instances) items.Add(instance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed);}
Figure: GetActivity implementation
If the instanceid parameter is * this operation returns all the instances of a BAM activity. The
following code shows a response that lists the instances of the TRANSACTION activity used
Figure: GetActivity Atom response for an individual activity
Combining these two operations we can navigate through the BAM activity using a
syndication client. To create new instances of a BAM activity we need to execute the
CreateActivityInstance operation via an HTTP POST.
CreateActivityInstance
This operation creates an instance of a BAM activity based on an AtomPub request. The
following code illustrates sections of the implementation of this operation.
[WebInvoke(UriTemplate= "/activities/{activityname}?end={isend}", Method="POST")] public Atom10FeedFormatter CreateActivityInstace(string activityname, bool isend, Stream activityfeed) { /*Creates a new instance of a BAM activity using the Event Stream API. Details omitted for brevity…see the attached samples for the details */
BAMActivityInstance activityInstance= bamdb.CreateActivityInstance(activityname, activityFields, true, isend, activityContent); SyndicationFeed feed = new SyndicationFeed(); List<SyndicationItem> items = new List<SyndicationItem>(); items.Add(activityInstance.SerializeAsSyndicationItem()); feed.Items = items; return new Atom10FeedFormatter(feed); }
As you can see in the code, this operation is activated by an AtomPub request to a URI in the
form http://<service endpoint>/activities/<activity name>?end=true|false. The following
code shows a sample AtomPub request that creates an instance of the Transaction activity