Continuous Testing 2016

Continuous Testing2016Karim Fanadka, DevTest Manager

About me: Karim Fanadka

Job: DevTest Manager at HPE Software, 7+

Product: StormRunner Load

Favorite activates: Traveling, Ski, Blogging

Ski – France 2013

Road Trip – Iceland 2015

We are going to talk about:

AgileModern QA

SaaS

Automation

Environments

Hotfixes Continuous Deployment

SaaS – Software as a Service

Before SaaS – On Premise

Software

Users

SaaS – Software as a Service

Data Base

Servers

File System

Firewall

Users SaaS

Which product my team is testing :

StormRunner Load - SaaS product

Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.

9

1 Create and run a load test using your favorite scripting tool

2 Load Generators spin up in AWS EC2 or MS Azure

3 Results and app insights are available in real time

The product my team is testing: StormRunner Load

New

Soon

Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.

StormRunner Load as a Service

Data Base

Servers

File System

Firewall

Users SaaS Cloud

Agile – Do It Fast

Before Agile - Delivery Timeline

1 or 2 Major releases to production in 1 year

Agile StormRunner Load Delivery Timeline

March April AugustJune Nov Dec

Whats the CodeName of the next release?If you know the answer, I will invite you

to a AlleyCat BubbleBobble Elevator Action Frogger Golden Axe Hang-On

FebruaryDec

CommanderKeen Digger

6 Major releases to production in 1 yearYear: 2015

Agile – The Challenges• Test new features

• Test Regression

• Automation

• Environments

• Continues Integration / Continues Delivery

• Hotfixes

Do it in 1.5 months

• Data Migration (When update SaaS with the newer version)

• Performance Testing

• Security Testing

• Globalization

• Documentation

• And more …

Modern QA Team (DevTest)

What is DevTest? Using top end QA/DEV practices combined with DevOps techniques

QA & DEV Working very close, on the same IDE, code base, and in united process

Automation is a MUST

Manual is for new features / exception flows that are difficult to automate

NO MORE “Deliveries for QA” YES for “Push to Production”

Training“we are testing, and breaking code constantly but we help rebuild it better…….”

Continues Testing

Push new code

Master

Release branch (1.5 months)

Test new codeShort Regression

Push to production

Automation

X Many times a day

Developer Engineer

Test Engineer DevTest Team

Builds in 1 Release (1.5 month): 1,400• Full: 236• Commit: 1,164

Modern QA

DevTest

DevOpsSaaS

Dev

Product

Customer

Automation

Automation

AlleyC

at

Bubble

Comman

der...

Digger

Elevato

r

Frogge

r

Golden

Axe

Hang-O

n

IceClim

ber

JumpM

an0

100

200

300

400

500

600

700

030

80 100 120150 170

200250

300

0 0

75120

147

220

390

450

600650

0 0 0 0

100

200

270 270 270 270

Automation Active Tests - Trend

Client Backend Cloud

85% Coverage

100% Coverage

65% Coverage X 2 Browsers

Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190

23

Labels automation coverage SPEC – 66 pages

24

Product Tree – Regression plan

Frameworks:

Frameworks Tests

Client Testing Framework

API Testing FrameworkCloud Framework

Results

28

Bot

29

Testing based analytics from production

API discovery tool

Cost cloud tool

Cloud Testing

Example for Automation challenge17 Regions in the Cloud 15 different metrics

255 Tests

Automation – Cloud Framework - Report

Test Environments

Test Environments Farm

QA Security Performance IntegrationLocal (Linux)

Staging

Hotfixes in 1 day

Hotfixes Lifecycle

07:00Production bug

08:00Research by QA and Dev

10:00Fixing bug

16:00Testing (Staging)

14:00Testing (Local)

18:00 Push to production

Articles • 10 best practices for QA teams to deliver quality software, fast

• Blurring the lines: How the development tester role changes on agile teams

• 3 steps to building test databases for the real world

• Plan your tests and release criteria the right way

Tip

“Changing our mindset takes precedence over technology. When we work together, we will succeed.”

Thank you

@kim0z85

Question?

https://www.facebook.com/karim.f.as

https://il.linkedin.com/in/kareemfa [email protected]@HPE.com

Karim FanadkaDevTest ManagerHPE Software

46

47

12/27/2015 3/6/2016

3/6/2016Go liv e

3/2/2016Staging

2/28/20 16CF

2/14/20 16FF

1/3/2016DevTest Start

Ice Climber V1.9 (Demo T imeline)

• Production

• Staging

• CF

• FF

• Dev

48

CIProductionFFCF StagingSecurity + SAASFeature discovery Defects -> ER’s

In Progress

49

12/27/2015 3/6/2016

3/6/20 16Go liv e

3/2/2016Staging

2/28/20 16CF

2/14/2016FF

1/3/2016DevTest Start

Ice Climber V1.9 (Demo T imeline)

Preparations: Feature discovery (meetings) Defects review (re prioritize, clean) Writing automation spec - example Automation design for exist and new frameworks

Testing: Test new features (Top priority, to make sure no delay in the timeline) Write automation tests to increase coverage (in the first 2 weeks, content is very low)

Feature Freeze

50

12/27/2015 3/6/2016

3/6/20 16Go liv e

3/2/2016Staging

2/28/20 16CF

2/14/2016FF

1/3/2016DevTest Start

Ice Climber V1.9 (Demo T imeline)

10 days for stabilization : Test un finished features (2 days in the FF) Prepare [Cloud deployment, move to Release branch, prepare regression plan, update AGM] Regression ~7 days – Product matrix Migration – Copy DB from production + File system

In addition: Prepare Security Env Demo all the features to Security / SaaS Discover next release features

Code Freeze

51

12/27/2015 3/6/2016

3/6/20 16Go liv e

3/2/2016Staging

2/28/20 16CF

2/14/2016FF

1/3/2016DevTest Start

Ice Climber V1.9 (Demo T imeline)

2 days for final validations : Verify defects that was fixed in the end of FF Validate Security + Performance + Globalization have no holders

Staging ~1 day sanity

Production ~1 day sanity

CI/CD

52

Follow all Jenkins failures Test issue: Fix test Code issue: Contact Dev and fix directly / Open defect CI/CD Env issue: Contact DevOps

Deployment SaaS QA1 / SaaS Q2 / Migration / Security Cloud images

On Premise

Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190

Defects

54

Coverage : 65% - example Tests: 700 active tests UI product tree covergae

AutomationFrameworks

Client Backend Coverage: 85% - example API discover

Cloud Coverage: All regions – 20 Regions

Protocols –

Metrics

NV

Save months of WD Cover new LG’s very fast Duration: 1.5 hour

~13 min (all tests)

Commit

Full

~2.5 min (main flows)

Automation

AlleyC

at

Bubble

Comman

der...

Digger

Elevato

r

Frogge

r

Golden

Axe

Hang-O

n

IceClim

ber

JumpM

an0

100

200

300

400

500

600

700

030

80 100 120150 170

200250

300

0 0

75120

147

220

390

450

600650

0 0 0 0

100

200

270 270 270 270

Automation Active Tests - Trend

Client Backend Cloud

85% Coverage

100% Coverage

65% Coverage X 2 Browsers

Builds Number (JumpMan 1.95): 1,617• Number of Full : 427• Number of Commits: 1,190

56

Next slides are reference – DON’T PRESENT

57

Labels automation coverage SPEC – 66 pages

58

Product Tree – Regression plan

59

API Coverage

60

API Coverage