Cmsc 1215 ensuring a quality order management (3)

Ensuring a Quality Order Management

Implementation

Subroto Majumdar, OMS Director

[email protected]

Agenda

2

Introduction to Quality

and Testing

Why is Quality important

to Sterling OMS

QA in OMS

Journey to QA Center of

Excellence

Conclusion

Software testing is a process, or a series of processes, designed to make sure computer code does what it was designed to do and that it does not do anything unintended.

The program should be predictable and consistent, offering no surprises to users.

Test should start with the assumption that the program contains defects and the goal is to find as many defects as possible.

Introduction

3

Why is quality important for Sterling OMS?

• OMS – A Major Enterprise Initiative

• QA - Airplanes vs. Order Management Implementation

– Hopefully, no one gets hurt

– Impacts project timelines

– Companies lose money and delay their ROI

– It can mean the difference between business accepting or rejecting

the solution

– It hurts the product and vendor reputation

• Bill Lear an engineer and pioneer in executive jets Grounded “All”

customer planes till he could triage and resolve a mysterious issue with

his jets.

– Quality cost money but is the difference between success and

failure

• OMS Headless Application - Quality that cannot always be seen but felt

4

Importance of Testing and Validation

Testing should be introduced in the early

stage of the Software Development Life

Cycle (SDLC)

– Cost of fixing the defect is higher if

testing is not done in early stage &

defects found in later critical stages

Retesting and Regression Testing is also

important

– Ensure that no previously working

functions have failed as a result of the

fix

– New features have not created

problems with previous software

versions 5

Typical Testing phases in OMS testing projects

Testing Phases

6

UAT/Performance

E2E

FT

CIT

UT

UT – Unit Testing

CIT – Component

Integration Testing

FT – Functional

Testing

E2E – End To End or

System Testing

UAT – User

Acceptance Testing

Performance Testing

OMS - What can be Tested?

Order flow across different status

Holds resolution

Credit card authorization

Fraud check validation

Sourcing and scheduling

Release

Ship confirmation, ship cancellation,

status tracking

Settlement and invoicing

Data purge

Order load (data conversion)

Inventory load, ATP and sync

Item load and sync

Customer data management

Alerts, emails and exceptions

7

OMS - What can be Tested?

8

UI and Database validations have their own importance

UI validations recommended when the tester needs to validate the complete life cycle of a single order– Eg: Validating the ship confirmation from the fulfillment center. For

this scenario, the tester can validate the order creation in OMS, order processing till release and once the shipment confirmation arrives, the order status is updated to Shipped.

Database validations beneficial when the tester has to validate from a range of data – Eg: To validate if there is any order where tax is $0 or shipping

discount of more than $5 can be done by creating a simple query to fetch the above results. Other DB validations include data purge, to check when the order reached a specific status, to confirm if the order was picked up by the agent or not, verify the bulk inventory and item feed coming from different systems to OMS.

Performance Testing Scope - OMS

Performance testing is very

important since OMS always

deals with a huge volume of

order, item and inventory

related data.

At times, it also processes

bulk orders in a single batch

and hence performance

testing plays a key role in any

OMS implementation.

9

Automation Testing Scope - OMS

Some of the many scenarios which can be

automated:

Order flow across various status

Shipment charges

Tax calculation

Discounts and promotional charges

Payment tenders and tokens

Item validation

Inventory reservation pre/post order creation

Shipment and settlement process

10

Sample Waterfall OMS Testing Methodology

CALENDAR

Requirement Analysis

Strategy

Formulation

Test Planning

Test Case

Creation

Test

Execution

Scripting

Post Deployment

Evaluation

Release

Test Summary

Report

Benefits

Onsite Onsite - Offshore

1

2

3

4

5

6

7

8

9

11

Test Strategy/

Planning

Test Case

Development

– Vendor assumes full responsibility for all software testing activities for aspecific project, release, or enhancement, Manages delivery with guidanceand oversight from client testing/SQA leadership.

– Your team creates the test plan and manages overall testing delivery.

– Vendor executes full set of testing activities defined within the client test planunder client’s direct management.

– Client manages overall testing delivery and creates all testing artifacts.

– Vendor is responsible only for executing test cases provided by the client,logging defects, and providing necessary reporting.

Environment

Preparation

Test

ExecutionAnalysis

Mgmt

Reporting

Sign

Off

Deli

very

Mo

del

Full Ownership

Combined Ownership

Testing Execution Only

Typical Delivery Models

12

Typical QA Team Responsibility

13

Testing Strategy

Test Planning

Test Case Creation

Test Execution

Defects Verification

Reporting

Automation

Positive Sign Off

Successful Go Live

Sample QA Artifacts

Detailed Scorecards - Integrated into Dashboard

Executive Dashboard

Partners Stakeholders

ID RID Opened Description and Impact Mitigation Activities Severity Probability Due Date Owner Status Comments / Resolution

001 D 2/10/04

The project described in this

Statement of Work is based on a

projected start date of 2/9/2004.

Project will start on 2/9 based on

good faith.High Unlikely 2/9/04 F. Schutrumpf Closed

CLOSED (2/9/04): Project started on schedule

on good faith based on verbal ok from Bruce

relayed from Warren. Issue 009 is tracking non-

signed SOW.

001 D 4/12/04

Delivery of Pan and transition of code

base to the Billing Implementation

Team on 4/23/2004.

Pan slips can jeopardize planned

delivery of Billing functionality

components that have Pan

dependencies.

In addition, given the finite number of

iterations prior to year end,

implementation of Pan functionality

could limit scope of Billing, Payments,

Fraud or Streamline functionality

implemented by Year end

First iteration designed to

transfer, organize and cleanup

existing Siebel code base.

Schedule work for subsequent

iterations that must have

Requirements fully defined up

front prior to iteration start to

minimize developer churn

Blending of Pan and Billing

functionality on second and third

iterations to continue working Pan

functionality, but also include new

billing functionality in each

release.

Scope impact evaluation at the

end of the second iteration to

assess the open issues

surrounding Pan.

Medium Certain On-going K. Sheen Open

UPDATE 4/12/04: On-going dependency until

all Pan dependencies for planned Billing,

Payments and Fraud functionality have been

resolved.

UPDATE 4/26/04: Current first iteration

contains multiple PAN user stories still. The

first iteration will contain mostly Pan

functionality (rather than billing functionality).

This may require a change to scope (Payments

or Streamline Integration) to maintain schedule.

This will be evaluated at the completion of the

first iteration.

UPDATE (5/3/04): Second iteration will most

likely still include mostly Pan functionality. In

addition, specific BAs will need to be assigned

to continue the Pan requirements wrap-up. This

is expected to take 2 full time BAs through the

end of the second iteration (assumes that Pan

functionality can be wrapped up in the third

iteration). This will definitely affect the amount

of billing work that can be completed by

10/22/04. A complete impact evaluation will be

completed at the end of the second iteration

(see also issue #010)

002 I 4/12/04

The project described in this

Statement of Work is based on a

Projected Start Date of 4/12/04.

Team members can ramp up

more slowly to effect transition. Medium Unlikely 2/10/04 K. Sheen Closed

CLOSED (4/12/04): Project started on

schedule.

Page 1Technology Triage Center – Severity 1 Defect Summary

Sunday June 4th, 2006 (Data Pulled a 7:45pm)Overall Status

Defect ID Status Partner Project Summary Due Date Action Item Detected on

Date

2299 New Visage Web Portal Production: Incorrect shipping cost shown in Siebel for Web Portal Order Per Darren, it appears that this is just a display item (totals & amt charged is correct) 5/27/2006

2382 New Visage CRM PROD - Adding the Shared Feature 'Guest Services Offer - Courtesy 100,200,50' throwing a

System Failure Error in Siebel

6/1/2006

2435 New Visage Activation Activating a second suscriber under a demo account fails in Siebel 6/4/2006

2437 New Patni Web Portal Mecury fails to send emails created within test director 6/4/2006

2436 New Patni Web Portal Production: NSP: Entered LTDCASE1 into shopping cart - now when the page refreshes the

sku is not there at all

6/4/2006

2151 Open Visage ROE ROE PROD: After ROE ESN activation, Welcome Email from Siebel is not sent to the ROE

customer

Rick confirmed all channels require this & per Scott, Visage is working this. Needs to

provide due date.

5/22/2006

2404 Open Convergys Web Portal Order Handset Process - Calculate Sales Tax error in Production with concurrent users New duedat 6/6 - need to confirm before placing it in the ticket - 6/4 6/1/2006

2354 Reopen Visage Web Portal Production: Shared Features added or removed on Web Portal do not reflect despite giving an

Order Confirmation

Sent back to webportal dev team as it failed retest - 6/4 5/31/2006

2355 Reopen Visage Web Portal Production: Non-Shared Features added or removed on Web Portal do not reflect despite giving

an Order Confirmation

Sent back to webportal dev team as it failed retest - 6/4 5/31/2006

2416 Retest Telcordia Service

Control

Call setup latency - mobile-terminated calls are taking more than 10 seconds to ring through 11:30 fix tonight - retest tomorow - 6/3 5/31/2006

2369 Retest Disney

Mobile

CRM Production: Activation failure for web sales order in Data Platform Sharique to do sales order and see if it passes - 6/4 6/1/2006

1147 Reopen Patni Web Portal EID Verification needs to be integrated in the Checkout Process 5/31/2006 Equifax to push their side into prod tomorrow (6/1) at 6am - we can follow our prod

push after this executes. Update tomorrow afternoon. 5/31

4/27/2006

2047 Retest mPortal Family Alerts Production: Config file incorrect for non-MDN accounts, preventing Family Alert! from working 6/3/2006 Into prod last night - ready for retest. - Ian to retest - 6/4 5/18/2006

2427 New Patni Web Portal Production: Site Wide: Hitbox tracking not working properly - talk to Todd Sheive at Hitbox 6/4/2006 Going in tomorrow morning - 6/4 6/3/2006

2429 New Patni Web Portal Production: SMS Messaging: need to set the originating address to 1002 in the XML in order to

support the billing reference data agreed with Convergys

6/4/2006 Working on, pushed by tuesday night - 6/4 6/3/2006

2207 Open Patni ROE ROE PROD: Family Account with SA Portability failed at AO & FM level ...Record unusually

stored in Siebel

6/4/2006 All ROE CR's will go in at end of next week (6/9) - 6/3 5/24/2006

2274 Open Patni ROE ROE UAT: The Auto Bill Pay Page does not allow to proceed to Order Confirmation 6/4/2006 5/26/2006

1793 Reopen Patni Web Portal BOGO Promotion SKUs are not uploaded in Visage Product Catalog 6/4/2006 Need to understand what actions Patni needs to take to resolve defect - Bobby,

Dave, and Ashish to discuss real time - 6/3

5/10/2006

1925 Hold InPhonic Web Portal Production: Entertainment - Ringtones Preview hyperlink not working 6/6/2006 New due-date is Tuesday COB - 6/4 5/16/2006

2380 Open Autodesk LBS Production: Family Locator not functioning on port-in-swapped handset 6/6/2006 Pushing for tomorrow evening - 6/4 6/1/2006

Risk, Issue, Dependency Log

Defect Tracking Log

Connectivity

Network Overall Status

C-WAP

Test Prod

Billing Provider/CRM Provider:

AVS Product

Network and Operations

Finance

Legal

Test Cases

Defined

Test Case

Sign-OffStarted

IT / Systems / Data Center

Overall Status

Page 1Integrated Application Testing Dashboard

Tuesday, November 6th, 2007

Overall Integration Test Status

Milestones:

Milestone one – COMPLETE

Milestone two – COMPLETE

Milestone three – 10/17/07

Milestone four – 10/22/07

Milestone five – 10/24/07

· High level status messages go here

· Additional thoughts go here.

Days until Launch*

4 4Soft Launch 1/4/2008

Key Issues:

ØIssue one - ………………………………………...

ØIssue two - ……………………………………….

ØIssue three - ……………………………………..

Green

ATG Link

Connectivity Overall Status

Martin-Dawes / AAA

Test Prod



Martin-Dawes / AirCell

Agile (Louisville / Itasca)

Passur / Itasca

Yellow

Yellow

CompletedOpen Defects

Sev 1 Sev 2 Sev 3

Overall Status and Issue


· Additional thoughts go here. Lkasjdlkjflkjsdflkajskldjflkjsdlkjfkjsdlfjlakjdflkajsdlkjflkadlskjflajslkdjfslkdjf

· Alkjdlkfjlaksdjfslkdjflkjskldjf

· Alkjdlkfjalksjdflkjalkjlkj

· ajlksdjflkjasldkjfljaslkdjflasjdlkfjlkajsdflajsdlfjljdflkjalkjdkljlkdjflakdlfkjalkdfjlkjfdljaslkdjflkajsdlkfjlksdjflkajsdlkfjkajsdfkljaldkjf

Systems Overall Status

Martin-Dawes

Dev Test



AAA BridgePort

Agile

Passur

Yellow

Infor

Martin-Dawes

AAA BridgePort

Agile

Passur

ESB (TIBCO)

AAA BridgePort

Agile

Passur

ESB (TIBCO)

Prod

Systems Integration Overall Status

Martin-Dawes / AAA

Dev Test



Infor / Agile

etc / etc

etc / etc

Yellow

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

etc / etc

Prod

Critical AirCell Launch Milestones

• Integration Testing Environment Build-out

• Integration Testing Environment Ready

• System Integration Test Planning & Definition

• Vendor Q-Gate Sign-off

Vendor testing completed and verified

CM process in place

• Install / Configure / Remote Admin Testing

• Neighborhood Testing

• Formal Integration Testing

End to End Functional Testing

Non-Functional Requirements Testing

Performance / Load Testing

Disaster Recovery Testing

• Production Environment Build-out

• Production Environment Ready

Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar

Soft Launch

Commercial Launch







CM process in place











Soft Launch

Commercial Launch

Days until Launch*

4 4Soft Launch 1/4/2008







CM process in place











Soft Launch

Commercial Launch







CM process in place











Soft Launch

Commercial Launch

Test Case ID Priority Sequence Priority Sequence Applications RequiredApplications

ReadyTest Phase Test Date

PASS-001 A 1 A 1 Portal, IMS, AAA 9/15/07 Neighborhood 9/15/07

PASS-007 A 2 A 2 Portal, IMS, AAA 9/15/07 Neighborhood 9/15/07

NET-002 A 3 A 1 Portal, IMS, AAA, MDS 9/22/07 Neighborhood 9/22/07

NET-005 A 4 A 2 NMS, Portal 9/22/07 Neighborhood 9/22/07

LEG-001 A 5 A 1 Passur, MDS 9/1/07 Neighborhood 9/1/07

FIN-028 A 6 A 2 app1, app2, app3.component 9/2/07 Neighborhood 9/2/07

xxx-nnn A 7 A 2 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07

xxx-nnn B 1 A 2 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07




xxx-nnn B 5 B 3 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07





xxx-nnn C 1 B 3 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07

xxx-nnn C 2 B 5 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07

xxx-nnn C 3 C 5 app1, app2, app3.component 10/1/07 Formal E2E 10/1/07

Planned ExecutionOverall Business Unit

Test Case Execution Tracking

Schedule

0

10

20

30

40

50

60

70

80

4/14

4/16

4/18

4/20

4/22

4/24

4/26

4/28

4/30 5/

25/

45/

65/

85/

105/

125/

145/

165/

185/

205/

225/

245/

265/

285/

30 6/1

6/3

Defe

cts

Daily Opened Daily Closed Total Currently Open

Daily Opened 3 6 8 14 13 5 5 11 13 12 7 11 11 4 6 1 0 6 5 5 3 4 2 4 11 12 2 4 3 4 5 3 7 4 6 2 1 2 3 2 4 3 4 3 1 0 4 12 11 3 2 3

Daily Closed 0 0 1 0 3 5 6 7 6 7 4 8 2 7 9 3 2 5 8 12 10 2 3 1 1 4 5 8 5 6 3 11 12 9 8 3 4 5 5 4 2 2 4 1 4 1 7 9 14 4 5 3

Total Currently Open 3 9 16 30 40 40 39 43 50 55 58 61 70 67 64 62 60 61 58 51 44 46 45 48 58 66 63 59 57 55 57 49 44 39 37 36 33 30 28 26 28 29 29 31 28 27 24 27 24 23 20 20

4/

1

4

4/

1

5

4/

1

6

4/

1

7

4/

1

8

4/

1

9

4/

2

0

4/

2

1

4/

2

2

4/

2

3

4/

2

4

4/

2

5

4/

2

6

4/

2

7

4/

2

8

4/

2

9

4/

3

0

5/

1

5/

2

5/

3

5/

4

5/

5

5/

6

5/

7

5/

8

5/

9

5/

1

0

5/

1

1

5/

1

2

5/

1

3

5/

1

4

5/

1

5

5/

1

6

5/

1

7

5/

1

8

5/

1

9

5/

2

0

5/

2

1

5/

2

2

5/

2

3

5/

2

4

5/

2

5

5/

2

6

5/

2

7

5/

2

8

5/

2

9

5/

3

0

5/

3

1

6/

1

6/

2

6/

3

6/

4

Defect Tracking Heuristics

14

Sterling OMS: Driver for QA CoE

We talked about Importance of Quality and QA in the

context of Sterling OMS

Journey beyond OMS QA to a QA Center of Excellence

15

Ensuring a Quality OM Implementation

Who is responsible for Quality?

– QA Team?

• Quality is everyone’s responsibility

and it’s a continuous effort

– Why and Where do defects exists?

• So many reasons and so many

places

– Requirements Validation

– Scope Validation

– Design Validation

– Planning Validation

– Experience of the team (Have you

done this before?)

– Reporting

– Communication

– Change Management

16

Quality is a journey, not something

that happens towards the end…

Components of a Quality Center of

Excellence

1. Engagement or methodology of QA• Create engagement and a framework to show the value of QA

• If I spend x on QA I can expect y savings

• Establish credibility and a cost model that focuses on quality, value and stability

2. Creates a metrics or KPI driven QA function• Show how QA improves quality, drives down time to market, increases stability

3. Creates value and ownership much like that of development teams• QA owns the applications they support as much as development does

• QA owns strategy, testing plan, automation, regression scripts,

• Show this alignment with development

4. Continuous improvement• Find innovative ways to drive down the cost of QA, and deliver higher quality.

• Always look for ways to push QA further up stream in the SDLC cycle

• Show how the overall quality of an application or project is not the sum of QA but is

the sum of all project parts (requirements, designs, development etc.)

17

What does a QA COE do?

QA (CoE) should consist of: A team of people that promote collaboration and

using best practices around QA to drive business results.

Responsibilities:

– Support: CoEs should offer support to the business lines. This may be

through services needed, or providing subject matter experts.

– Guidance: Standards, methodologies, tools and knowledge repositories

are typical approaches to filling this need.

– Shared Learning: Training and certifications, skill assessments, team

building and formalized roles.

– Measurements: CoEs should be able to demonstrate they are delivering

the valued results that justified their creation through the use of output

metrics.

– Governance: Allocating limited resources (money, people, etc.) across all

their possible use is an important function of CoEs. They should ensure

organizations invest in the most valuable projects and create economies

of scale for their service offering.

18

What to Measure?

Here are just a few of the measurements that should be considered:

– Defect density (defects per lines of code)

– Defect by application area

– Defects by project phase (requirement, design, development)

– Defect re-open rate

• And why? (design issue, dev issue)

– Performance metrics

• Number of critical issues resolved through performance testing

• Business value surrounding resolution of critical issues

– Automation metrics

• Efficiencies gained through automation (regression cycle time)

• Improvements in quality as a % of regression

– Trending

• Number of open critical and priority defects by area

• Number of critical or priority defects by phase (design/dev)

– Production P1-P4 counts by release

• You should see this drop as quality increases and problems are found earlier

19

Trends and Patterns Derived

With the right metrics an IT organization can move various SDLC phases to

become more quality focused by derived patters found in the QA cycle.

Application area’s which struggle with quality of design or development code

can be pin pointed and as such can be provided assistance to become more

effective.

Processes within the development and design lifecycles can be brought to the

forefront based on how many defects are being derived.

Innovations in the design, development and QA lifecycles can emerge

through continuous improvement discussions which can be tied to QA

metrics.

A true sense of the overall maturity of a Sterling OMS eco system, or any

development domain can be established, benchmarked and then monitored

for improvements.

Look for mature development deployments to production that are yielding low

production issue counts post launch, compare release over release and

improve. But also compare against other application areas

20

Conclusion

Quality is very important

Think Total Quality of the Process not just

software Quality

Quality is an investment with favorable ROI

Choose the right OMS QA model for your

company

Everyone is responsible for Quality

A metrics based QA organization will

create positive momentum and spread

beyond one or two applications

Quality Center of Excellence. QA is

something that can spread beyond Sterling

OMS21

Thank you! For more information

Listen to our webinar:

Learn How to Create a Seamless Omni-Channel Retail Experience

22

Subroto Majumdar, OMS [email protected]

http://www.perficient.com/Thought-Leadership/On-Demand-Webinars/2014/Learn-How-to-Create-a-Seamless-Omni-Channel-Retail-Experience

Cmsc 1215 ensuring a quality order management (3)

Documents

disaster recovery testing

production environment build

formal integration testing

functional requirements testing

production environment ready

shared features added

dawes aaa

webportal dev team