ReportPortal.io - Open Source experience. Showcase, benefits

1CONFIDENTIAL

Дмитрий Гуменюк

ReportPortal.ioOpen Source experience.

Showcase, benefits

2CONFIDENTIAL

SPEAKER NAME

Проект Менеджер

10 лет с компанией EPAM Systems. Разработчик в прошломВозглавляю разработку инициатив внутри Центра Компетенций Тестирования.

Report Portal – Product Owner.

DZMITRY HUMIANIUK

EPAM Systems, Project Manager

ДМИТРИЙ ГУМЕНЮК

3CONFIDENTIAL

SPEAKER NAME

DZMITRY HUMIANIUK






4CONFIDENTIAL

5CONFIDENTIAL

OPEN SOURCEDGPL V3.0

GITHUB.COM/ REPORTPORTAL

6CONFIDENTIAL

SPEAKER NAME




DZMITRY HUMIANIUK



7CONFIDENTIAL

SPEAKER NAME



Report Portal – Product Owner.DZMITRY HUMIANIUK


8CONFIDENTIAL

9CONFIDENTIAL

АВТОМАТИЗАЦИЯ

10CONFIDENTIAL

C H A O S

Х А О С

11CONFIDENTIAL

СЕРЕБРЯНАЯ ПУЛЯ

12CONFIDENTIAL

ТЕСТОВЫЕ ПРОЦЕССЫТЕСТОВАЯ СТРАТЕГИЯ

АВТОМАТИЗАЦИЯ

13CONFIDENTIAL

14CONFIDENTIAL

15CONFIDENTIAL

ЦИКЛ ПОСТАВКИ

16CONFIDENTIAL

ДАВАЙТЕ ВВЕДЕМ МЕТРИКИ

17CONFIDENTIAL

МЕТРИКИ

НУЖНЫ ДЛЯ ПОНИМАНИЯЧТО ВЫ ДЕЛАЕТЕВЕЩИПРАВИЛЬНО

18CONFIDENTIAL

КАК ЧАСТО ВЫ ЗАПУСКАЕТЕ АВТОМАТИЗАЦИЮ?КАКОЙ ПРОЦЕНТ ПРОХОЖДЕНИЯ

ВАШИХ ТЕСТОВ?ПРОСМАТРИВАЕТ ЛИ КОМАНДА ВСЕ РЕЗУЛЬТАТЫ?КАК МНОГО ТЕСТОВ УПАЛО ИЗ-ЗА

ПРОДУКТОВЫХ БАГОВ?НА СКОЛЬКО ВАША АВТОМАТИЗАЦИЯ БЛОКИРУЕТ ПОСТАВКУ?

19CONFIDENTIAL

20CONFIDENTIAL

ПОЧЕМУ АВТОМАТИЗАЦИЯ ПОПУЛЯРНА?CI/CD

21CONFIDENTIAL

ИДЕИ В ОСНОВЕ РЕПОРТ ПОРТАЛАLIVE VISIBILITY В СОСТОЯНИЕ

АВТОМАТИЗАЦИИ

СОКРАЩЕНИЕ ЗАТРАТ НА

РАЗБОР РЕЗУЛЬТАТОВ

ПОДДЕРЖКА ПОПУЛЯРНЫХ

ТУЛОВ В АВТОМАТИЗАЦИ

И

КРАСИВЫЕ ГРАФИКИ ПРЯМО ИЗ КОРОБКИ

МОЖЕТ РАБОТАТЬ С

СУЩЕСТВУЮЩЕЙ АВТОМАТИЗАЦИЕ

Й

СОЛЮШЕНЫ ПОМОГАЮТ

ДЕЛАТЬ ДЕНЬГИ

22CONFIDENTIAL

УСПЕХ РЕПОРТ ПОРТАЛА8ВНЕШНИХ

УСТАНОВОКУ

ЗАКАЗЧИКОВ

110+ ПРОЕКТОВ

ИСПОЛЬЗОВАЛИREPORTPORTAL

23CONFIDENTIAL

OPEN SOURCEDGPL V3.0


24CONFIDENTIAL

REPORT PORTAL В ДЕЙСТВИИ

25CONFIDENTIAL

REPORT PORTAL: ПРОСТАЯ ИНТЕГРАЦИЯ

<4 ЧАСОВ

Без рефакторинга тестов!

TestNGJUnitNunitSoapUICucumber Cucumber-JVMProtractor Node.JSSpecFlowjBehave…

26CONFIDENTIAL

v

REPORT PORTAL IS NOT A

27CONFIDENTIAL

REPORT PORTAL: ПРОСТАЯ ИНТЕГРАЦИЯ

CI tool (Jenkins)

TestNGRP adapter

Report Portal Server

28CONFIDENTIAL

HOW IT WORKS?

EPAM Private Cloud

REST Web Service

Mongo DBNoSQL

UIJava Script

Agents

HTTPs

Domain Credentials,UPSA profiles

29CONFIDENTIAL

30CONFIDENTIAL

5

31CONFIDENTIAL

“FIRE DETECTOR” METRICSName Why it is important to

monitor thisWhat decision can be

madePeriodicity

Calculation

Execution Frequency

To understand and evaluate how often test cases executed.Goal: Increase execution ratio per day/build.Configure automated execution of TA after commits (smoke, acceptance) and regressions right after nightly builds.

Triggers1 execution per week or less (???)

Target1 per commitInterpretationcode change should be verified not to affect functionality or other code base.

Each day

Measures1. test executions countFormula Unit of measureNumber

% of results analyzed

To understand and evaluate percentage of reviewed fail reports. If fail report not reviewed - there is not any value

Triggers<60%

Target100%Interpretationfailed report(test cases) should be review

Each day

Measures1. No of not analyzed failed test

cases (TI)2. No of failed due to product bug

(PB)3. No of failed due to automation

issues (AB)4. No of failed due to system issue

(SI)

Formula100% - To_Investigate / (PB+AB+SI+TI)

Unit of measure%

32CONFIDENTIAL

“FIRE DETECTOR” METRICSName Why it is

important to monitor this

What decision can be made Periodicity

Calculation

% of product issues (stability of automation)

To understand and evaluate percentage of failed test which corresponds to production issues

Triggerstrend not being positive

Target>90%Interpretationtrend of fails due to test automation instability should decrease (automation issues, system issue, not analyzed failed test cases)

Measures1. No of not analyzed failed test cases

(TI)2. No of failed due to product bug (PB)3. No of failed due to automation

issues (AB)4. No of failed due to system issue (SI)

FormulaPB / (PB+AB+SI+TI)

Unit of measure%

% of automation issues

To understand and evaluate percentage of failed test which corresponds to test automation issues (code issues, test case invalid)


Target<5%Interpretationtrend of fails due to test automation instability should decrease

Measures1. No of not analyzed failed test cases

(TI)2. No of failed due to product bug (PB)3. No of failed due to automation

issues (AB)4. No of failed due to system issue (SI)

FormulaAB / (PB+AB+SI+TI)

Unit of measure%

33CONFIDENTIAL

“FIRE DETECTOR” METRICSName Why it is important

to monitor thisWhat decision can be

madePeriodicit

y

Calculation

% of system issues

To understand and evaluate percentage of failed test which corresponds to system issues (no connectivity, lack of resources and etc.)


Target<5%Interpretationtrend of fails due to system issue instability should decrease

Measures1. No of not analyzed failed test cases (TI)2. No of failed due to product bug (PB)3. No of failed due to automation issues

(AB)4. No of failed due to system issue (SI)

FormulaSI / (PB+AB+SI+TI)

Unit of measure%

Execution Time

To understand and evaluate time of each execution cycle.

TriggersRegression cycles > 15hacceptance cycles >1h

Targetregression <8hacceptance <1hInterpretationtime for execution should not go over described limits

Measures1. Execution duration of regression, hours

(Reg)2. Execution duration of acceptance, hours

(Acp)

FormulaReg < 15hAcp < 1h

Unit of measureHours

34CONFIDENTIAL

SPECIFIC FOR QA AUTOMATION METRICSMetric Name Description Category

Percent Automatable (PA) # of test cases automatable (ATC) / # of total test cases (TC)

Quality

Regression frequency How often does automated regression run? QualityAutomation Progress (AP) # of actual test cases automated (AA) / # of

test cases automatable (ATC)Progress

Percent of Automated Testing Coverage (PTC for Automation)

Automation coverage (AC) / Total Coverage (C) (KLOC, FP, etc)

Coverage

Automated Testing Window

1) How much time does it take to run QA Automation?

2) How much system time/lab time is required to run the test(s) selected?

Cost / time

Testing Results Analyses Windows

How much time does it take to do data analysis?

Cost / time

Auto-Test Development Time

# of automated tests/ time to develop Cost / time

Auto-Test Support Time # of automated tests/ time to support Cost / time

35CONFIDENTIAL

reportportal.io

[email protected]

[email protected]


ReportPortal.io - Open Source experience. Showcase, benefits

Technology