53317246 AIA Error Handling and Recovery

<Insert Picture Here>

AIA Error Handling and Recovery

Varsha Bhatia, Principal Product Manager

Rohit Agarwal, Principal Solution Architect

AIA Deep Dive Series

Copyright © 2008, Oracle. All rights reserved.

Agenda

• Key features

• Architecture

• Fault policy

• Notification setup

• Error notification and details

• Extending Error Handling

• Logging

• Enabling Error Handling and Logging

• Targeted new features: 2.x – 3.0

• Guaranteed Delivery and Message Resubmission


Key Features

Unified Approach

• Across technologies (BPEL, ESB)

• Across integration patterns

Error Notifications

• Issues error notifications to suitable Actor roles, and FYI roles

• Links to Error Console (BPM Worklist) for error details

Error Console

• Centralized UI to access error details that are assigned for resolution or for

informational purposes

• Accessible to administrators, and end-users

Error Logging

• Logs messages non-intrusively in a consistent schema

• Logs can be searched, sorted, and filtered using Oracle Enterprise Manager


AIA Error Handling Architecture

ESB

AIA Common Error Handler

Async BPEL Process

AIA Error Topic

ESB Error Handler

AIA Logger

ESB Error Topic Catch Block

BPEL BPEL

BPEL Java Action

Error ListenerBPEL Fault

Policy

Error Scenarios

BPEL partner link errors

BPEL non-partner link errors

Enterprise Service Bus (ESB)

errors

3rd Party Errors (e.g. ODI)

http://images.google.co.in/imgres?imgurl=http://www.ruf.rice.edu/~evolve/images/emailIcon.png&imgrefurl=http://www.ruf.rice.edu/~evolve/&h=524&w=500&sz=125&hl=en&start=8&tbnid=-fD9XNlEmUJHqM:&tbnh=132&tbnw=126&prev=/images%3Fq%3Demail%26gbv%3D2%26svnum%3D10%26hl%3Den


<Action id="aia-ora-retry">

<retry>

<retryCount>3</retryCount>

<retryInterval>2</retryInterval>

<exponentialBackoff/>

<retryFailureAction ref="aia-ora-java"/>

<retrySuccessAction ref="aia-ora-java"/>

</retry>

</Action>

<Action id="aia-ora-java">

<javaAction className="oracle.apps.aia.core.eh.BPELJavaAction"

defaultAction="ora-rethrow-fault">

<returnValue value="REPLAY" ref="ora-terminate"/>

<returnValue value="RETHROW" ref="ora-rethrow-fault"/>

<returnValue value="ABORT" ref="ora-terminate"/>

<returnValue value="RETRY" ref="aia-ora-retry"/>

<returnValue value="MANUAL" ref="ora-human-intervention"/>

Fault Policies for BPEL/ESB Errors

• BPEL Fault Policy files are located at

<SOA_HOME>/bpel/domains/default/config/fault-

policies/AIADefaultPolicy.xml

• ESB Fault Policy file is located at

$AIA_HOME/config


Fault Message Schema

Enterprise Message Elements

• EBMID

• EBM Name

• EBO Name

• EBOID

• Verb Code

• Business Scope Reference ID

• Business Scope Reference Instance ID

• Enterprise Service Name

• Enterprise Service Operation Name

• Sender Reference ID

• Sender Message ID

• Sender Reference Transaction Code

• Sender Object Identification ID

Fault Specific Elements

• Fault Message Code

• Fault Message Text

• Severity

• Fault Stack

• Faulting Service ID

• Faulting Service Implementation Code

• Faulting Service Instance ID

• Corrective Action

• Reporting Date Time


Notification Setup

Notification Setup

Error routing mapped to parameters

Preference-based notification to Actor

Role, FYI Role


Notification Setup Parameters

• Error Code

• Fault code in BPEL, ESBString string for ESB. ESB has a

concept of error codes. Examples: bpelx:bindingFault,

bpelx:remoteFault

• System Code

• Value is obtained from EBM Header. Examples: SEBL_01,

PORTAL_01

• Service Name

• Name of the BPEL process or ESB service that encounters

error. Examples: CreateCustomerPartyProvABCSImpl,

CustomerPartyEBS

• Process Name

• Value is obtained from EBM Header. Examples: Siebel-

Portal-Get-Account-Balance


Error Notification Role Determination

For every error that occurs, the error handler checks for

matches in the following order:

1. SYSTEM_CODE, ERROR_CODE, SERVICE_NAME,

PROCESS_NAME

2. ERROR_CODE, SERVICE_NAME, PROCESS_NAME

3. SERVICE_NAME, PROCESS_NAME

4. SERVICE_NAME

If none of these values are available, the default roles

are fetched from the AIAConfigurationProperties.xml file.

<Property

name="EH.DEFAULT.ACTOR.ROLE">AIAIntegrationAdmin</Property>

<Property name="EH.DEFAULT.FYI.ROLE"></Property>


Error Notification and Resolution

Error Notification Example:


Error Console

Leverages SOA Suite Worklist application

• Error tracking

• Starting point for

further error resolution

• Group assignment:

Actor Role, FYI Role

• Re-assignment

• Identify responsible

persons / groups by

querying AIA setup

Note: 791273.1

Sample script to close out all the tasks

that are pending


Error Details in the Worklist App

Copyright © 2008, Oracle. All rights reserved.13

Extending Default Fault Message Content

Enriched Fault Content- Add custom content to base fault

message to give more complete

error context: Use xsd:anyType

- Functionally rich information

- More effective Error Resolution

- Applicable for BPEL and Mediator

errors

Depth of Content- Add one, two custom fields or entire

message payload

- View logged content in AIA Log Files

via Enterprise Manager

Enriching fault content

Step1- Implement Custom extension

handler. Implement AIA Interface

Step2- Register class with Error

Handling Framework

Step3- Identify error scenario for which

handler needs to be called within

Setup UI





Example

15

AIAConfigurationProperties.xml

Copyright © 2008, Oracle. All rights reserved.16

Custom Handling of Errors

Example:- Step 1:

• Indicate custom error handling

for Order Fallout functionality:

ORDER_FO

- Step 2:

• Indicate custom plus default AIA

Error Handling behavior: Error Type

= ORDER_FO, AIA_EH_DEFAULT

- Step 3:

• Implement extension listener to

subscribe to messages with JMS

Header stamp = ORDER_FO. AIA

will ensure default behavior in

additionEnabling Custom Handling of Errors

Step1- Define a unique custom filter

code

Step2- Identify error scenario for which

fault message should be

stamped with custom filter code

in Setup UI

Step3- Implement extension listener

to subscribe to AIA Error topic

for messages with custom

filter code


Custom Handling of Errors


Logging Framework

Siebel CRM On Demand

• Trace Logging –

Recording chronological

sequence of events.

- Custom XPath

- Java API

• Error Logging –

Recording of errors. No

specific configuration

required. Logging done

non-intrusively.

• Use Oracle Enterprise

Manager (OEM) as user

interface to search, sort,

and filter logs.


Log Search Including AIA Attributes


To set up error handling and logging:

• Set up Error Notifications and Roles

• Set up Trace Log Levels

• Enable Trace Logging in AIAConfigurationProperties.xml

Enabling Error Handling and Logging


Enable Trace Logging

• Can be done on system level or on service level

• Service level overrides system level

• To be done in AIAConfigurationProperties.xml in

$AIA_HOME/config:

-bash-2.05b$ more AIAConfigurationProperties.xml

<?xml version="1.0" encoding="UTF-8"?>

<AIAConfiguration xmlns="http://xmlns.oracle.com/aia/core/config/V1"

version="1.0">



<SystemConfiguration>



<Property name="TRACE.LOG.ENABLED">true</Property>

[Snip]

• Reload the configuration file afterwards through the admin page


Targeted New Features 2.x – 3.0

2.x • SOA platform 10.1.3.x

1. Support OC4J JMS

2. BPEL, ESB Fault Policies

3. Error Log + Trace Log

framework

• No B2B Support

• Worklist an integral part of framework

• Log onto BPEL and ESB dashboards

externally

• No customization of notification

• Multiple notifications based on number of

errors

• Manual error resubmission

3.0• SOA platform 11g

1. Support Weblogic JMS

2. Mediator Error Handling

3. Error Log + Trace Log

framework

• Capture Errors from B2B

• Option to decouple or switch off Worklist

• Drill down into EM Control Flow Trace

from email notification

• Customization of email notification content

• Notification aggregation

• Automatic error resubmission


Guaranteed Delivery and Error Recovery


Topics

Concepts

AIA Approach

Source Responsibilities

Sample Flows

Configuration Settings

Error Recovery

AIA Recommends


Guaranteed Delivery - Concepts

What is Guaranteed Delivery

• A message initiated from sender system is persisted until it is successfully delivered to and acknowledged by the receiver

Sender and Receiver are not necessarily the participating applications, they are the logical milestones in an integration scenario

• There could be multiple milestones in a given integration scenario

The work done between 2 milestones is considered as one logical unit of work

• This should be idempotent and could use appropriate compensation handlers to achieve this.


Guaranteed Delivery – Concepts …

Source

ApplicationTarget

Application

Integration

Milestone

Req

ABCS

EBS Prov

ABCS

Target

Milestone

Source

Milestone

Global Transaction


Target Milestone is with in Target Application transaction space

Sample Flow 2

Source

ApplicationTarget

ApplicationIntegration

Milestone

Req

ABCS

Prov

ABCSEBS

Target

Milestone

Source

Milestone

Global Transaction

Producer Process


AIA Approach

AQ-JMS for message persistence

Global Transaction

• Enable all services in one single transaction

• No commit point between two milestones

Error Handling

• Any system error will generate a rollback of all preceding

services

• One notification will be sent to Integration Administrator

Error Recovery

• Manual scripts to recover message from queue and resubmit


AIA Approach …

Each message with a unique identifier

EBM Header populated for source milestone identifier

Fault Notification shall contain the message identifier and

source milestone identifier of the faulted message

The source milestone identifier will be resolved to a specific

JMS destination (or source application) in AIA

Configuration Properties File

Error Recovery scripts shall make use of these parameters

to recover and resubmit a faulted message


Source Milestone Responsibilities

It could be Source Participating Application

Publish outbound messages to an AQ-JMS

• One queue for each Entity

Publish a unique message identifier on the JMS header of each

message

• Default MSGID generated in the Header Variable can also be

used for correlation


Fault Flow

Source

ApplicationTarget


Milestone

Req

ABCS

EBS Prov

ABCS

Target MilestoneSource

Milestone Fault Notification

Create Task

Global Transaction


Configuration Settings

Global Transaction

• IDG – Chapter 11 - Enabling Transactions for an Integration Scenario

in Asynchronous Patterns of the Application Integration Architecture

• No commit point between milestones

Error Handling

• IDG - Chapter 14: Implementing Error Handling and Recovery for the

Asynchronous Message Exchange Pattern to Ensure Guaranteed

Message Delivery

• Use bpelx:rollback instead of throw in the catch blocks.

• Java snippet to send notifications for AIA / User Faults.

• Fault policies to contain a filter expression and perform “No Action” in

case of a roll back fault(ORABPEL-02180).

– Filter Expression - <test>$fault.summary/summary[contains(.,

"ORABPEL-02180")]</test>


Error Recovery – Schema parameters

– EBM Header / EBMTracking

– Fault / FaultNotification / FaultMessage

• SenderResourceTypeCode

– Indicates the type of resource or system

where the rolled back message is stored,

whether Queue, Topic or Resequence

Store

• SenderResourceID

– Identification of the resource/system of

type SenderResourceTypeCode

• SenderMessageID

– Identification of the message persisted in

the resource/system associated with

type SenderResourceTypeCode


Manual Error Recovery script – FP2.3

Manual ant script at $AIA_HOME/Infrastructure/ErrorHandling/util

• ant –buildfile MessageResubmit.xml

Resubmit Utility API,

• oracle.apps.aia.core.eh.resubmit.util.AIAEHResubmissionUtil class.

• For persistence in a topic:

– public boolean resubmitFailedMessageTopic( String topicName, String

topicTableName, String messageID );

• For persistence in a queue:

– public boolean resubmitFailedMessageQueue( String queueName, String

queueTableName, String messageID);

• For persistence in a Resequencer store:

– public boolean resubmitFailedMessageResequencer( String

routingServiceName, String groupID );


Auto Error Recovery using EM

EM Targets

• Register all services and Edge applications as “Targets”

AIA Auto Recovery Notification Agent

• Notified when a Target comes back to life (SEBL_01)

• Query AIA_FAULT_TABLE and resubmit all messages

faulted due to this Target

AIA_FAULT_TABLE

• Mapping table for JMS Queue Name, JMS Message ID and

Target instance id


Auto Error Recovery – auto update Tasks

AIA Auto Recovery Notification Agent

• update the respective Task after message resubmit

• Comments and status as completed

AIA_FAULT_TABLE updated for WF Task ID

• At Task creation time, map a task Id with JMS Message Id


Source

ApplicationTarget


Milestone

Req

ABCS

EBS Prov

ABCS

Target MilestoneSource

Milestone

Global Transaction

Update AIA_FAULT_TABLE

Fault Notification

Create Task

Update AIA_FAULT_TABLE

with Task ID

Fault Flow – auto recoveryEM

AIA Auto

Recovery

Notification

Agent

WF


Some Do’s and Don’ts

Fault Policy Changes Do Not Require a Restart of Oracle BPEL

Server

• http://host:port/BPELConsole/domain_name/doReloadFaultP

olicy.jsp

Fault policies should not have any retry configured in

Guaranteed Delivery Global Transaction scenario

AQ JMS should be configured for max_retries=>0

BPEL Fault policy details

http://www.oracle.com/technology/products/ias/bpel/pdf/1013

3technotes.pdf

Taxonomy of BPEL Faults

http://www.oracle.com/technology/products/ias/bpel/htdocs/o

rabpel_technotes.tn007.html

http://www.oracle.com/technology/products/ias/bpel/pdf/10133technotes.pdf

http://www.oracle.com/technology/products/ias/bpel/pdf/10133technotes.pdf

http://www.oracle.com/technology/products/ias/bpel/htdocs/orabpel_technotes.tn007.html

http://www.oracle.com/technology/products/ias/bpel/htdocs/orabpel_technotes.tn007.html


AIA Recommends

• Application Services being XA-Compliant is the best

option

• Non idempotent Application Services (non-retryable)

need to have corresponding compensatory services

• ABCS invoking non-idempotent application services

should invoke compensatory services in case of roll back

• Application Services implemented via JCA adapters can

leverage session management to facilitate transaction

coordination

• Requestor, EBS and Provider ABCS have to participate in

same transaction

• Avoid log running transactions by setting the transaction

boundaries based on the business requirements.


Common Use cases

Integration of AIA Error Handling Framework with

• Trouble Ticket Application

• Monitoring Systems (BAM, OpenView)

• Legacy / 3rd Party application/Custom Integrations

Extending AIA Error Handling

• Custom notifications

• Extending Fault Message

• Turn off notifications only

• Turn off Task Assignments only




Q&A

53317246 AIA Error Handling and Recovery

Documents

implement

source milestone

identify error

custom filter

jms message

idempotent

oracle enterprise

custom handling