Artificial Intelligence in testing - A STeP-IN / NASSCOM Evening Talk Session Speech by Kalilur Rahman

1

• Progress of AI and Robotics

• What’s the need for Artificial Intelligence?

• What will happen at singularity?

• Some AI Concepts

High level Intro to AI

• Is it an Intelligent Activity?

• Are we testing at the heights of

Augmented General Intelligence?

• AI in Testing - Is it augmented or Artificial

/ Is anything artificial about it?

• How will AI evolve Testing?

• Some Examples of AI Testing

AI in Testing

Agenda

- ANDREW NG Founder of Coursera, Stanford Adjunct Professor

Ex. Chief AI Scientist of BAIDU

4

Current Understanding of AI -

Source : https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/

A Good Explanation of Progress of AI

E a r l y A I

Basic Turing Test Style

Use of Memory and Knowledge

Post John McCarthy’s Conceptualization

Basic Robotics and Degrees of Freedom

D e e p L e a r n i n g

Rapid Infrastructure Growth

Advanced Algorithms

Big Data Explosion

Quantum Computing

M a c h i n e L e a r n i n g

Algorithms Centric

Statistics Driven

Supervised and Unsupervised Learning

Perhaps the greatest

Computer Scientist ever

predicting on Machine

Intelligence

We have clearly passed

the TURING TEST

We are seeing Leaps and

Bounds in advances of

Technology!

Let’s hear!

Neal Creative | click & Learn moreNeal Creative ©

Singularity or transcendence is around the corner – Aided by AI

https://youtu.be/zatL4uFRpC0

Fast Learning – Download and Fly an helicopter

Can AI Take us to this stage?

How about Fast Testing – Hey – Can I test this brand new “thing” in 2 minutes?

From a Leader in AI – AI or an Algorithm Writing itself

Top Human World Champions Royally defeated by AI!

2011

2016

1996/97

IBM’s

Deep Blue

IBM

Watson

Google

Deepmind

AlphaGo won 60–0 rounds on two public Go websites

including 3 wins against World Go champion Ke Jie.

But... AI is not without controversies though!

Facebook Researchers shut down an AI engine at the Facebook AI Research Lab (FAIR), discovering that the AI created its own unique language undecipherable by humans - Simultaneous glimpse of both the awesome and horrifying potential of AI

Elon Musk - “AI isotentially more Dangerous than Nukes” sets up a $1 billion (£770M) OpenAI.org to try and promote safe development of AI

Vladimir Putin -“Whoever masters AI will rule the world!”

ISAAC ASIMOV’s Laws of RoboticsLaw 1: A tool must not be unsafe to use.Law 2: A tool must perform its function efficiently unless this would harm the user. The safety of the user is paramount.Law 3: A tool must remain intact during its use unless its destruction is required for its use or for safety.

Japan is using AI and Robots in Multiple ways

https://quickdraw.withgoogle.com/ , https://www.autodraw.com/ & https://g.co/aiexperiments

Google is using Deep Learning to make Simple things –Better!

https://oxism.com/thing-translator/

Take it to next levelHow can you easily integrate globally. Language

will no longer be a barrier.

AI Startups are taking it to next Level – in all areas

Source: https://www.cbinsights.com/research-ai-100

Bots AutomobileComputer

Vision

Core /

Functional AICommerce IIoT/IOT

Healthcare Fintech Robotics

AnalyticsCyber

Security

Sales &

Marketing

Let’s get to TESTING!

“ Let’s get to TESTING

• How is AI helping

Testing?

• How can we test

better with AI?

• How can we test AI

systems Better?

17

“Be a yardstick of QUALITY. Some

people aren't used to an environment where EXCELLENCE

is expected”

18

Steve Jobs

Business Agility - Some Statistics

19

Google - refactors code by

50% each month*

Netflix - 5 Billion+ API

Calls per Day (and

increasing daily)

~75% of Corporates to

have bi-modal IT

~63% all projects are not

aligned to Business

Strategy

~79% organizations using

CI/CD/DevOps practices in

one form or the other

52% of Fortune 500

companies have

disappeared from the list

& Average S&P500 span

reduces from 61 Years to

17 Years in 60 years

In 2020, 100 million

consumers will shop via

augmented reality

By 2020, 30% web

browsing will be done

without a screen

by 2022 - $1 Trillion a year

to be saved through IoT

Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016

* - Google runs on ~2 Billion LOC Source: CA Workshop on Modern Software Factory

Source: CB Insights* AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI

Is the Testing Industry ready for testing the following innovations?

21

Tip of the Iceberg seen in 2016

2016 A Year in Review – Software Failures

22Source: Tricentis Software Fails 2016 Report - https://www.tricentis.com/wp-content/uploads/2017/01/20161231SoftwareFails2016.pdf

Over 4.4 Billion people got affected by

a Software Fail (Up

from 4.3 Billion in

2015) > 50% Global

Population

$1,062,106,142,949- Assets Affected (Up

from $4.2 Billion in

2015)

315 years, 6 months,

2 weeks, 6 days, 16

hours, & 26 minutes -

Accumulated time-

lost due to Bugs

2.66 Billion Mobile

Phones impacted with Malware

12% Year on Year Increase in impactful

Software Bugs

British Airways lost

$20 Billion (3%) in

Market Cap within a

few days after a failed

software upgrade

More Than 21

Million Automobile

recalls as a result of

Glitches / Bugs

$5.7 Billion Impact

in Failed Government

Software Projects due

to Bugs

2.2 Billion people live on less than $2 a day

One School of Thought on Testing – By Tricentis

Source: TRICENTIS webinar on Future of Testing

Wh

ere

AI

can

help

Legacy

Firms

Bi-model Firms

Technology Leaders

Test

Coverage

GAP

Years

Months

Weeks

Days

Hours

Seconds

Test

ing

Du

rati

on

Challenge of Complexity, Less time and more Tests

Where we are

in time

Testing Complexity

Tim

e

26

Resources

Effort

CostsCoverage

#of Test

Cases

When you have less time overall

Some Algorithms making Machine Think!

Source: https://futurism.com/predicting-2017-the-rise-of-synthetic-intelligence/ - Some of the artificial intelligence (AI) algorithms currently helping machines think. Credit: CIO Journal/Narrative Science

Approaches used for AI, Machine Learning and Deep LearningReinforcement Learning

• Passive Reinforcement

Regression Algorithms

• Linear Regression

• Gaussian Process

Supervised Learning

• Neural Networks

Unsupervised Learning

• Independent Component

Analysis

• Principle Component

Analysis

Natural Language

Understanding

• Morphological, , semantic,

syntactic , Discourse

analysis

Natural Language

Generation

• Deep planning

• Syntactic generation

Clustering Algorithms

• K-Means Clustering

• KPCA – Kernel Analysis

Statistical Algorithms

• Support Vector Machines

• K-Nearest Neighbor

• Native Bayes Classifier

• Maximum Entropy Classifier

Pattern Recognition

• Statistical , Syntactic

approach

• Template Matching

• Neural Networks

Other Techniques

• Spanning Trees and Graphs

• Neural Network – Multi-

Level Perceptron's

Other Techniques

• Labeling

• Hidden Markov Model

• Maximum Entropy MM

Other Techniques

• Conditional Random Fields

• Parsing Algorithms

29

Some methods to build AI Data Models for Testing

1 2 3 4 5

What are the feasibilities with AI Driven Testing?

30

Automated Defect

Detection

Automated

Exploratory

Testing

Test Coverage

Heat map

Self Healing

Automation

Predictive

Modeling

Self Adjusting

Regression

Pattern

Recognition

Risk & Coverage

Optimization

Diagnostic,

Prescriptive and

Predictive Analysis

Deep LearningRoot-Cause

AnalysisSentiment Analysis

31

AI Models Algorithms

Application Under

Test

Designer Developer Business UserTesterBots / Agents

AI Engine

Testing Outcomes

Test CasesProduction

LogsRequirements

Defect Logs Source CodeTraceability

Matrix

Root Cause

Analysis

Test Data

Specifications

Functional

Logic

Sample AI Model for TestingHistorical & Real-time Data

Let’s see an example

Example: Candy Crush Saga’s AI Strategy

https://www.youtube.com/watch?v=wHlD99vDy0s

• Use of AI engine for continuous Feedback Loop

• Use of BOTS to perform Testing

• Continuous Feedback Loop

• Deep Artificial Neural Network

• Use of Monte Carlo Tree Simulation

• Use of Advanced Automation by BOTS

• Hybrid Test team (150-200+ Testers) with unique skills

• Use of Data Scientists for Domain Knowledge, Fun (using

historic info and user behavior, Game Balancing)

• Regular Crash Testing, Performance Testing, Regression

Testing

• Regular Upgrade of AI Bot for Testing

v

Recommendations / Summary

AI Testing Recommendations

Summary