Definición y análisis de Indicadores de rendimiento de ...

Applied Software Engineering

Research Group

Definición y análisis de Indicadores de rendimiento de procesos

Adela del Río Ortega Universidad de Sevilla

www.isa.us.es

University of Sevilla

•  65.000 students •  4.600 teaching staff •  500 years

Computer Engineering School

Amador Durán Joaquín Peña David Benavides Beatriz Bernárdez Octavio Martín Manuel Resinas

Pablo Fernández Sergio Segura Carlos Müller Jose María García Pablo Trinidad Adela del Río

Javier Troya Jose A. Galindo Ana B. Sánchez

Antonio Ruiz

Jose Antonio Parejo Antonio Gámez

Applied Software Engineering Research Group www.isa.us.es

Alfonso Márquez Antonio M. Gutiérrez Bedilia Estrada

ISA in numbers

•  > 200 Publica3ons in Journals and Conferences

•  3 Interna3onal Patents •  16 So:ware Tools developed •  5 European Research projects •  10 Na3onal Research projects •  5 Research Networks •  35 Public-‐Private Transfer

Research Contracts •  2 Spin-‐offs

25 members 14 Senior Research Staff 4 Research Assistants, 3 PhD students, 4 Software Developers

(>20 former members)

ISA Research Areas

Business Process

Management Cloud and Services

Variability Management

Software Testing

Experimentation

Creating rich models

Finding techniques to analyse them

•  Constraint Satisfaction Problems

•  Description Logics and Ontologies

•  Metaheuristics

ISA Research Areas

Business Process

Management Cloud and Services

Variability Management

Software Testing

Experimentation

Introduction

Business Processes (BPs) Conference travel management (THESIS)IS

A Re

sear

ch G

roup

ISA Research Group

SubmitPaper

Fill TravelAuthorizat ion

Send TravelAuthorizat ion

Registerat Conference

Any problem?

MakeReservat ions

Accommodation Transport

TravelAuthorizat ion< < filled> >

Sign TravelAuthorizat ion

TravelAuthorizat ion< < signed> >

TravelAuthorizat ion< < stored> >

CheckResponse

TravelAuthorizat ion

< < approved> >

TravelAuthorizat ion< < rejected> >

Research Vice- chancellorship

NoYes

CRISTINA CABANILLAS MACÍAS 1 of 1 21.11.2012

Conference travel management (THESIS)IS

A Re

sear

ch G

roup

ISA Research Group

SubmitPaper

Fill TravelAuthorizat ion

Send TravelAuthorizat ion

Registerat Conference

Any problem?

MakeReservat ions

Accommodation Transport

TravelAuthorizat ion< < filled> >

Sign TravelAuthorizat ion

TravelAuthorizat ion< < signed> >

TravelAuthorizat ion< < stored> >

CheckResponse

TravelAuthorizat ion

< < approved> >

TravelAuthorizat ion< < rejected> >

Research Vice- chancellorship

NoYes

CRISTINA CABANILLAS MACÍAS 1 of 1 21.11.2012

Perspectives in Business Processes

Human resources

Control flow Access Types Read (R) Write (W) Read/Write (R/W)

Data

Data

Business Process Perspectives

Business Process

Resources/ Organisational

IT systems Data/ Informational

Functional

Performance

Control flow/Behavioural

Introduction Performance perspective

Need to Measure

RFC Management BP Diagram

delays caused by committee

number of RFCs per project

percentage of perfective changes

out of approved RFCs

Examples

Process Performance Indicator (PPI)

Quantifiable metrics that allow the evaluation of the efficiency and effectiveness of business processes. They can be measured directly by data that is generated within the process flow and are aimed at the process controlling and continuous optimization.

[G. Chase et al., 2011]

PPI vs KPIs

KPIs

PPIs number of RFCs per project

percentage of

corrective

changes from RFC

approved

percentage of

satisfied

costumers

Design

Instrumen-tation Computation

Evaluation

Design and Analysis

Configuration Enactment

Evaluation

BPM lifecycle PPIM lifecycle

Define PPIs, Connect with BP,

design-time analysis

Implement measurement

points

Calculate PPIs’ values and

monitor PPIs

Identify PPI correlations,

conflicts and predict future behaviour

The PPI Management Lifecycle

How to define PPIs to support the PPI management lifecycle?

How to extract valuable information?

Problem statement

How to define PPIs to support the PPI management lifecycle?

How to extract valuable information?

Problem statement

How are PPIs defined nowadays?

Low level -

implementation perspective

Informal -

natural language

This leads to several problems

Duration of the analysis activity

Which analysis activity

When

Ambiguity and Incompleteness

Way of defining a PPI

??

? ?

Traceability

? ?

Processable vs understandable

Business manager System architect

Understandability

Way of defining a PPI

Partial views Comprehensive views

Visual Gap

PPINOT Metamodel

3 1 2 ✔ ??✔

BP Instances

Scope ?

SMART

Specific

Achievable Relevant

Timed Measurable

1

2

3

•  High expressiveness

•  Automation

•  Traceability with BP elements

•  Extensible

Features

More details at: Adela del-Río-Ortega et al. “On the definition and design-time analysis of process performance indicators”. Information Systems 38(4): 470-490 (2013)

PPINOT Metamodel PPINOT Graphical Notation

Visual PPINOT

PPI

Measures

Aggregated Measure Derived Single-Instance Measure

Derived Multi-Instance Measure

Base Measure

What to measure Time

SUM

SUM

Count

State Condition

Data Property Condition

Data content

Connectors

Aggregates Time connectors

Applies to isGroupedBy Uses

IT D

epar

tmen

t

PLan

ning

& Q

ualit

y M

anag

er

RFCReceived

Analyse RFC

Cancel RFC

Approve RFC

Elevatedecision tocommittee

ReportRFC

cancelled

ReportRFC

approved

RFCregistered

RFCcancelled

RFCapproved

Com

mitt

ee

Analyse incommittee

Requester

PPI5

AVG

Avg time of RFC analysis

PPI4

(a/b)*100

Perfective RFCs out of approved

1 2 3

SUMSUM

from

bape

rfec

tive

chan

ges

to

More details under request (BISE journal article under review)

?

Scalability

Business manager

Learning curve

Problems

PPINOT Metamodel PPINOT Graphical Notation

PPINOT Templates

Helps to structure information

Serves as a guide

Uses (structured) natural language

Templates

The PPI value must be greater than [or equal to] <lower bound>

Easier and faster than writing whole paragraphs from scratch

Fills placeholders in prewritten sentences

Sucessfully used in RE

Linguistic Patterns

Mapping to the metamodel

PPI-005 Average time of RFC analysis Process Request for change (RFC) Goals •  BG-002: Improve customer satisfaction

•  BG-014: Reduce RFC response time MeasureDefinition The PPI is calculated as the average of the duration between the time

instants when activity RFC analysis becomes active and when activity RFC analysis becomes completed

Target The PPI value must be lower than or equal to 1 working day Scope The process instances considered for this PPI are those in Last 100

instances scope Source Event logs of BPMS Responsible Planning and quality manager Informed Chief Information Officer (CIO) Comments Most RFCs are created after 12:00

PPI Template Example

More details at: Adela del-Río-Ortega et al. “Using templates and linguistic patterns to define process performance indicators”. Enterprise Information Systems 10(2): 159-192 (2016)

•  How to define PPIs to support the PPI management lifecycle?

•  How to extract valuable information?

Problem statement

How do I instrument

this process? If I change this

activity, which PPIs might be affected?

Which process elements are measured by this PPI?

Average lifetime of approved

RFCs

How do I instrument this

indicator?

Measured By Relationship

Which process elements are involved in this PPI?

Average lifetime of approved

RFCs

This PPI is too costly. We need

to replace it

Involved In Relationship

Rela9onship Opera9ons

Measured By

MeasuredBPElement: Which process elements are measured by this PPI?

MeasuredPPI: Which PPIs are defined over this process element?

Involved In

InvolvedBPElements: Which process elements are involved in a given PPI?

NotInvolvedBPElements: Which process elements are not involved in any PPI?

InvolvedInAllBPElements: Which process elements are involved in all PPIs?

Associated PPI: Which PPIs are involved in a given process element?

Formulate operations in terms of DL reasoning operations (e.g. satisfiability, subsumption, realization, etc.)

DL reasoners •  Hermit •  Pellet

•  Racer •  Etc.

DL-based implementation

DL Knowledge Base

More details at: Adela del-Río-Ortega et al. “On the definition and design-time analysis of process performance indicators”. Information Systems 38(4): 470-490 (2013)

Applied to several use cases

•  IT department of the Andalusian Health Service •  Company for training health professionals •  Information and Communication service of the University of

Seville •  A part of the administration of the Andalusian Regional

Government


Tooling support

PRspectives

Available at: http://labs.isa.us.es:8080/prspectives

PRspectives

PRspectives


Tooling support Ongoing work

Ongoing work

•  SLA modelling for BPO services

•  Automatic establishment of links between natural language PPIs and their implementation

•  PPI variability management

•  PPI Thresholds determination based on execution data

Ongoing work





SLA modelling for BPO services 5

Table 2. Penalties definition (in monthly billing percentage) for the FI Service SLA

AFIP Penalty94% AFIP < 95% -1%93% AFIP < 94% -2%92% AFIP < 93% -3%91% AFIP < 92% -4%90% AFIP < 91% -5%

AFIP < 90% -10%

Cont

ract

or

APC

FI requested

FI request

Plan FI Perform FI

FI documentation

required?Create and submit FI

documentationDocumentation

Accepted

Correctionrequired

FI closed

no

FI documentation Correction request

FI documentationacceptation

Fig. 1. BPMN model of Field Intervention (FI) service

have two main features. On the one hand, it must be expressive, i.e. it must allow the137

definition of a wide variety of metrics. On the other hand, it must be traceable with138

the business process so that it enables their automated computation. In addition, it is139

convenient that the metrics are defined in a declarative way because it reduces the gap140

between the SLA defined in natural language and the formalised SLA and decouples141

the definition of the metric from its computation.142

3.3 Service Level Objectives (SLOs)143

These are the assertions over the aforementioned metrics that are guaranteed by the144

SLA and, hence, must be fulfilled during the execution of the service. For instance, the145

running example defines AFIP > 95% as an SLO for AFIP metric of the FI service.146

In general, SLOs can be defined as mathematical constraints over one or more SLA147

metrics.148

3.4 Penalties and rewards149

They are compensations that are applied when the SLO is not fulfilled or is improved,150

respectively. An example is shown in Table 2, which depicts the penalties that apply for151

4

Resolution Time Elapsed time between the technician arrival and the end and closureof the FI.

Documentation Time If documentation, i.e. reports, is required, it is defined as theelapsed time between the end and closure of the FI and documentation submission.If the APC considers such documentation as incomplete or inadequate, it will bereturned to the contractor and documentation time is again activated and computed.

3 Requirements for Modelling SLAs of BPO Services

The requirements for modelling BP SLAs in the context of SLA–aware PAIS have beenidentified after a study of the state of the art in SLAs for both computational and non–computational services, and the analysis of more than 20 different BPO SLAs developedby 4 different organisations in 2 different countries. The conclusion is that four elementsmust be formalized in SLAs for BPO services, namely: 1) the business process; 2) themetrics used in the SLA; 3) the SLOs guaranteed by the SLA; and 4) the penalties andrewards that apply if guarantees are not fulfilled. Next we describe each of them.

3.1 Business process

An SLA is always related to one or more specific services. The way such services mustbe provided is usually defined by describing the underpinning business process, andthis is often done in natural language. Consequently, the formalization of SLAs forBPO services requires the formalization of the business process itself. Note that it is notrequired for the SLA to detail the low level business process that will be enacted by theprovider’s PAIS since most SLAs do not delve into that level of detail and just focuson main activities and the consumer–provider interaction (cf. Fig 1 for the high–levelbusiness process of the running example). However, it should be possible to link thishigher level business process to the lower level business process enacted by the PAIS.

3.2 SLA metrics

These are the metrics that need to be computed so that the fulfilment of the SLA canbe evaluated. For instance, in the running example, response time, presence time, orAFIP are examples of such metrics. The mechanism used to define these metrics must

Table 1. Committed times by the contractor (in hours) for the FI Service SLA

CriticalityLevel

ResponseTime

PresenceTime

ResolutionTime

Document.Time

Timetable Calendar

Critical 0.5 4 2 4 8:00 – 20:00 LocalHigh 2 8 4 12 8:00 – 20:00 LocalMild 5 30 6 24 8:00 – 20:00 LocalLow 5 60 8 48 8:00 – 20:00 Local

3

reports on work related to the definition of SLAs for BPO services. Finally, conclusionsare detailed in Section 7.

2 Running Example

Let us take one of the BPO SLAs to which our approach has been applied as runningexample throughout this paper. The SLA takes place in the context of the definitionof statements of technical requirements of a public company of the Andalusian Au-tonoumous Government, from now on Andalusian Public Company, APC for short.Statements of technical requirements are described in natural language and include in-formation about the services required as well as their SLA. Although the running exam-ple includes one service only, further information on this or the rest of services, as wellas on other application scenarios, is available at http://www.isa.us.es/ppinot/caise2015.

The statement of technical requirements document of this example is defined forthe Technical Field Support for the Deployment of the Corporative TelecommunicationNetwork of the Andalusian Autonomous Government. It is presented in a 72–page doc-ument written in natural language including the SLAs defined for five of the requiredservices, namely: 1) field interventions; 2) incidents; 3) network maintenance; 4) instal-lations and wiring; and 5) logistics. In particular, we focus on the field interventions (FI)service. The term field intervention makes reference to the fact of requiring the pres-ence of a technician at any headquarter of the APC for different reasons: troubleshootingtechnical assistance, installations supervision or restructure, for instance.

From a high–level perspective, the FI service can be defined as follows: the APC re-quires an FI, which can have different levels of severity, from the contractor staff. Then,the contractor plans the FI and performs it at headquarters. In some cases, it is necessaryfor the contractor to provide some required documentation and, if such documentationis considered incomplete or inadequate by the APC, it needs to be resubmitted by thecontractor until it fulfils the APC’s quality requirements.

For this service, the statement of technical requirements document presents the fol-lowing information: 1) the committed times by the contractor (see Table 1); 2) the gen-eral objective defined for FIs —the SLO of the SLA— represented as AFIP > 95%,where the AFIP (accomplished FIs percentage) metric is defined as:

AFIP =# accomplished FIs

# FIs⇥ 100

and 3), the penalties applied in case the SLO is not accomplished (see Table 2). Thesepenalties are defined over the monthly billing by the contractor for the FI service. Inaddition, the statement of technical requirements document presents the following def-initions for the referred times in Table 1:

Response Time Elapsed time between the notification of the FI request to the contrac-tor and its planning, including resources assignment, i.e. technicians.

Presence Time Elapsed time between resource (technician) assignment and the begin-ning of the FI, i.e. technician arrival.

>95%

5

Table 2. Penalties definition (in monthly billing percentage) for the FI Service SLA

AFIP Penalty94% AFIP < 95% -1%93% AFIP < 94% -2%92% AFIP < 93% -3%91% AFIP < 92% -4%90% AFIP < 91% -5%

AFIP < 90% -10%

Cont

ract

or

APC

FI requested

FI request

Plan FI Perform FI

FI documentation

required?Create and submit FI

documentationDocumentation

Accepted

Correctionrequired

FI closed

no

FI documentation Correction request

FI documentationacceptation

Fig. 1. BPMN model of Field Intervention (FI) service

have two main features. On the one hand, it must be expressive, i.e. it must allow thedefinition of a wide variety of metrics. On the other hand, it must be traceable withthe business process so that it enables their automated computation. In addition, it isconvenient that the metrics are defined in a declarative way because it reduces the gapbetween the SLA defined in natural language and the formalised SLA and decouplesthe definition of the metric from its computation.

3.3 Service Level Objectives (SLOs)

These are the assertions over the aforementioned metrics that are guaranteed by theSLA and, hence, must be fulfilled during the execution of the service. For instance, therunning example defines AFIP > 95% as an SLO for AFIP metric of the FI service.In general, SLOs can be defined as mathematical constraints over one or more SLAmetrics.

3.4 Penalties and rewards

They are compensations that are applied when the SLO is not fulfilled or is improved,respectively. An example is shown in Table 2, which depicts the penalties that apply for

Metrics

Penalties & Rewards

Service Level Objective (SLOs)

SLA modelling for BPO services

Agreement

Terms

Name

Context

Service Terms

Guarantee Terms

Service description terms

Service properties

Service references

Business Process

SLA Metrics

SLO, Penalties &

Rewards

Structure: WS-Agreement

From Computational SLAs

Published at: Adela del-Río-Ortega et al. “Modelling Service Level Agreement for Business Process Outsourcing Services”. CAiSE 2015: 485-500

Ongoing work





Automatic establishment of links between natural language PPIs and their implementation

•  The formulation of PPIs is typically a managerial concern •  Monitoring PPIs requires a technical perspective on a

process

Natural language PPI description

Process model

Alignment

Automatic establishment of links between natural language PPIs and their implementation

Process model

Alignment Approach

Decision tree

1. Type classification

2. PPI parsing

3. Alignment to model

Classification

Typeindicators

Extractedphrases

Measuretype Alignment

Unstructured PPI description

Published at: Han van der Aa et al. “Narrowing the Business-IT Gap in Process Performance Measurement”. CAiSE 2016: 543-557

Ongoing work





PPI variability management

DIM-2

DIM-1

Dimensions: result of analysis Performance perspective subject to variation

VAR

Needs of models and tools

VAR

VAR

First Step

•  Avoid redundancy •  Reduce efforts •  Increase clarity

Published at: Bedilia Estrada-Torres et al. “Identifying Variability in Process Performance Indicators”. BPM (Forum) 2016: 91-107

Ongoing work





PPI Thresholds determination based on execution data

Lagging Indicators Measures goal

accomplishment, easy to measure but hard to

influence.

Leading Indicators Predict goal achievement,

you can influence them but can be more difficult to

measure.

Method for PPI Threshold Determination

Data Preprocessing

Relationship checking

Threshold extraction: ROC Curve

Error probabilities:

Bender method

Thresholds validation

Lag indicator

Lead indicators

Threshold

To be published: Adela del-Río-Ortega et al. “Enriching decision making with Data-Based Thresholds of Process-Related KPIs”. CAiSE 2017: TBD

Data

Ongoing work

•  And much more…

Thanks

Adela del Río Ortega [email protected] www.isa.us.es/ppinot

Dept. of Computer Languages and Systems

ETSI Informática, University of Seville, Spain

Definición y análisis de Indicadores de rendimiento de ...

Documents

Definición y análisis de Indicadores de rendimiento de ...