Top Banner
Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann
162

Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Mar 30, 2015

Download

Documents

Cael Tomson
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Automating Coded UI Testing using Microsoft Testing Tools

in Visual Studio 2012

Clive G Hermann

Page 2: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

• Name

• Title/Function

• Job Responsibility

• Testing Experience

• C# Experience

• Expectations for the Course

Introductions

Page 3: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Course Materials

Page 4: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

1. Overview of Automated Code UI Testing

2. Walkthrough steps from beginning-to-end

3. Creating Manual Tests

4. Creating a coded UI test Using Coded UI Test Builder

5. Automating Coded UI Test Cases

6. Introduction to Application Lifecycle Management (ALM)

7. Anatomy of Coded UI Tests

8. New Visual Studio 2012 ALM Features

9. Automating Coded UI Tests

Course Contents

Page 5: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

6) Data Driven Coded UI Tests

7) Anatomy of Code UI Tests

8) Introduction to Lifecycle Management

9) Extending Coded UI Tests and Action Recordings

10)Testing Large Applications with Multiple UI Maps

11)Using HTML5 Controls in Coded UI Tests

12)Visual Studio 2012 ALM Features

13)Upgrading Coded UI Tests from Visual Studio 2010

Course Contents2

Page 6: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

14)Using WPF with Coded UI Tests

15)Best Practices for Coded UI Tests

16)Using the Coded UI Test Logs

17)Using SharePoint with Coded UI Tests

18)ExcelCodedUIHelper

Course Contents3

Page 7: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Overview of Automated Code UI Testing

Page 8: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

70% of all Testing is manual

Reality of Automated Testing

Page 9: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Replacing a Manual User Interface Test

With a Automated User Interface Testo To Allow:

Replication Consistency Regression Testing For:

• Multiple Environments

• Multiple Versions

Code UI Tsting

Page 10: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

UI Testingo Functional Testing

Controls User Interface Whole Application

The earlier we catch a “bug” or potential problem the less it costs!o Image

o Manpower

1) Overview of Automated Code UI Testing

Page 11: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

IT Has a Problem

Page 12: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Catching defects as early as possible is the least expensive way to ensure software quality.

Kent Beck and Cynthia Andres wrote

"Here is the dilemma in software development: defects are expensive, but eliminating defects is also expensive”.

“However, most defects end up costing more than it would have cost to prevent them."

Test Early and Often

Page 13: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Test-Driven Developmento Design the Test

o Then Write the Code

The Testing TrendsForwards

Page 14: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Linkingo Help-Desk

o Test

o Story Board

o Task

o Unit

o Programmer

The Testing TrendsBackwards - Circular

Page 15: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

To find the bugs before your users do!

UI Testing is END USER TESTING

OR TOTAL SYSTEM TESTING

OR INFRASTRUCTURE TESTING

The Ultimate Goal of Software Testing

Page 16: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Manual to Coded UI

Page 17: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Guarantee SOFTWARE QUALITY

Regression Test each Version or build

Test all Environments

Provide Developers with all the information required:o To REPRODUCE THE “BUG”

o Trace the steps to get the BUG

o Screen shots, IntelliTrace, Logs and Stack Traces

NO-REPRO must be ELIMINATED

Testers Multiple Roles

Page 18: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Clear Separation of Functions

Clear Definition of Communication and Collaboration

TEAM of Software Quality

Tiers of Software Testing

Page 19: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

2)Walkthrough steps from beginning-to-end

Page 20: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Lesson 1 Manual Testingo Using Microsoft Test Manger to create and manage Test Plans

o Running Test Cases and Publishing Results

o Conducting Exploratory Testing

o Fast Forward for Manual Testing

Lesson 2 Coded User Interface Testingo Automated Functional Tests – Using CUIT

o Create a CUIT from Scratch

o Create a CUIT from action recording

o Making CUIT more Robust and Flexible

2)Walkthrough steps from beginning-to-end

Page 21: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

A manual testing process

Page 22: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo of Simple Calculator

Building an AppFor Manual Testing

Page 23: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

Automating the Calculator

Page 24: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

Adding Data

Page 25: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

Introducing an Error

Page 26: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The Calculator

Lab 1

Page 27: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

3 Creating Manual Tests

Page 28: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Visual Studio 2012Supported Test Types

Page 29: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Provides a single environment to author and manage Tests

Microsoft Test Manager

Page 30: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 31: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 32: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

http://msdn.microsoft.com/en-us/library/dd380763.aspx

Quick Start Guide for Manual Testing using Microsoft Test Manager

Page 34: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

6 Data DrivenTests

Page 35: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Any Files Used in The Tests

Example XML to load Variable and Expected Data

Could be:o SQL

o Business Connectivity Services

o SharePoint Lists

o SQL Reports

o Excel / PowerPivot

o SSAS

6 Data Driven Tests

Page 36: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

DEMO and Lab

Page 37: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

4) Creating a coded UI test Using Coded UI Test Builder

Page 38: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Test Types

Microsoft Test Manager

Exploratory Testing

Easily reproducing issues through manual testing

4) Creating a coded UI test Using Coded UI Test Builder

Page 39: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

Lab

Authoring a TEST case

Page 40: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The first time you run MTM it will prompt o Add Team Foundation Manager

“If you don’t know the Name contact your administrator”

It might require “ServerName:Port Number”

Microsoft Test Manager

Page 41: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

TFS and MTM

Page 42: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

MTM is always connected to TFS

Connecting Microsoft Test Manager to TFS

Page 43: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

A user story can be stated as simply as

As an Employee I want to have an efficient way to manage my expenses.

The conversation around this statement between the product owner and the development team can raise questions such as:

Who can submit expense reports?

What states can an expense report be in?

When is it possible to change or remove an expense report?

What data is required in an expense report to register it correctly?

Where are the expense reports stored? For how long?

Story Boards

Page 44: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

An expense report has the following state model:

New when created

Pending after submitted for approval

Approved

Rejected

Adding Detail to User Stories

Page 45: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Allows the Tester to Verify what should happen

If this does not Happeno See the Tasks that were assigned to this requirement

o See the Programmers assigned to Each Task

o Notify the Developer(s) Provide the necessary info to avoid “NO REPRO”

Add this Test to the “Suite”

Story Boards are Vital to Testing

Page 46: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Test Plan Properties

Figure 21-5. Test plan properties

Page 47: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 48: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Static suite: The content of this suite is manually added test cases.

Query-based suite: A query-based suite lists all test cases matching a given work item filter.

Requirements-based suite: This suite shows the test cases associated with a selected TFS requirement.

Test Suites

Page 49: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 50: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 51: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

• Static suite: The content of this suite is manually added test cases.

• Query-based suite: A query-based suite lists all test cases matching a given work item filter.

• Requirements-based suite: This suite shows the test cases associated with a selected TFS requirement.

Page 52: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Establish Environments

Diagnostic Connectors

Best Practiceo Work with Development Teams

o The Developers need to have the same tools to view the test data you collected for them

Microsoft released a “Test Attachment Cleaner” available ato “Aka.ms./TFPowerTools”

Configuring Test Settings

Page 53: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Scouto Installing Software and running some initial tests

o After the SCOUT is successful it can become a “template for other Testers”

There is an available builds DROPDOWN

Click “ASSIGN TO PLAN” to add a new build to a plan

Using Builds

Page 54: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Quickly identify tests that need to be run based on changes to code

Can Run in the background

“Track” menu => Recommended Testso Does not recognize “DATA” changes

o Does not recognize other library changes

o Does not recognize other Application changes

Best Practice - Periodicaly run all Tests

Analyzing Impacted Tests

Page 55: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Environments may differ:o IE 8,9,10

o Ipad Horizontal/Vertical

o BYOD

o HTML5

o ETC.

Defining Multiple Test Configurations

Page 56: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Requirements-Based test suite

Query-Based test suite

Static Test Suite

o Status of tests in TEST SUITE In planning In Progress Completed

Test Cases are organized in to Suites

Page 57: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 58: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

1 – Unit and Component Tests2 – Functional Tests3 – Exploratory Testing4 – Capability Testing

Types of Tests

Page 59: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Unit and component tests are automated tests written to help the team develop software effectively.

With good suites of unit and component level tests we have the safety net that helps us develop software incrementally in short iterations without breaking existing functionality.

Invaluable when refactoring code. With good test coverage a developer should feel confident to make a change without knowing all about every dependency.

The tests should tell us if we did wrong

Unit and Component Tests

Page 60: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Functional Tests

Functional tests are mainly our traditional scripted system tests in different flavors.

It is hard to avoid running these tests manually at first, but we should try to find ways to automate them as we learn more about our product and how it needs to be tested.

Many functional tests can be automated and then we can focus on early testing for the manual tester

Functional Tests

Page 61: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Exploratory testing is a form of software testing in which the individual tester can design and run tests in a freer form.

Instead of following detailed test scripts, the tester explores the system under tests based on the user stories

As the tester learns how the system behaves the tester can optimize the testing work and focus more on testing than documenting the test process.

Exploratory Testing

Page 62: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Lastly we have the capability tests.

These tests are run against the behavior of the system;o we test non-functional requirements

o Performance

o Security

These tests are generally automated and run using special purpose tools, such as load test frameworks and security analyzers.

Capability Testing

Page 63: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

We should leave this category of tests as manual tests.

The focus should be to catch bugs that would fall through the net of automated tests.

A key motivation for automated testing is to let do more of exploratory and usability testing because these tests validate how the end-user feels when using the product.

Exploratory Testing

Page 64: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 65: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Testing Pyramid

Page 66: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Testing Structures

Page 67: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

LAB

Using Shared Steps

Page 68: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Demo

Assigning Configurations

Page 69: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

When you create a Coded UI Test, the Coded UI Test Builder creates a map of the user interface under test, and also the test methods, parameters, and assertions for all tests. It also creates a class file for each test

http://msdn.microsoft.com/en-us/library/ff398062.aspx

7) Anatomy of Coded UI Tests

Page 70: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

In-Line Programs

Sub-Routines

Functions

Time Sharing

On-Line Terminals

Client Sever

3 Tier Architecture

Thin Client

MVC

SOA

Evolution of Decoupling

Page 71: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Developers – View Individual Units of Code

Administrators Build Environments and Versions

Testers Test Complete Applications

De Coupling of Software Testing

Page 72: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Inability of TESTERS to test BlackBox Codeo 10,000 lines of code could result in 1 Million Routes

o Impossible to Test

Interface – Contract testing – is practical

CLOUD increases the environment exponentiallyo Interfaces are constant

Primary Drivers of SOA

Page 73: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Microsoft Windows 2012

Windows 8

Office 2013

Office365

SQL 2012

SharePoint 2012

LINQ 2012o are all Cloud – SOA Oriented.

Microsoft Commitment to the Cloud

Page 74: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The platform includes five services:o Live Services for communication and collaboration services

o SQL Azure (formerly SQL Services) for data-storage services

o AppFabric (formerly .NET Services) for cloud-computing services

o SharePoint Services for web-content-management and document-management services

o Dynamics CRM Services for customer-relationship-management services

Azure Services

Page 75: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Microsoft TFS is now Cloud Based

Testing is Moving to the CLOUD

Page 76: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 77: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Collaboration is the Key to TFSo Developers

o Testers – Creating Reproducible Automated Tests

o Administrators – Builders of (Virtual Machines and Versions)

Using TFS in the cloud can help us enhance collaboration

It can also enhance collaboration in an organization with geographically spread locations.

Benefits of TFS in the Cloud

Page 78: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

With TFS in the cloud is that we can neither add our own process template nor make adjustments to the existing templates.

Concerns with TFS in the Cloud

Page 79: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Interface Testing

Viewing Objects from the outside

Leads to Unit Testing

Test-Driven Design

Service Oriented Architecture

Page 80: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

IT has a Problem!

ACCELERATING RATE OF CHANGE

Page 81: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Reasons for SOARapid Adoption

Page 82: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The short answer is:

Cloud is an initial application of SOA to infrastructure

--its setup, configuration, monitoring , management , with the essential elements of metering and billing added to satisfy the goal of a charge model of e.g., pay-as-you-go

-- the specific requirements for XaaS: resource pooling and virtualization , elasticity and multi-tenancy, dynamic configuration and provisioning

Cloud paradigm leverages SOA to deliver a charge-based model for non-business specific services;

-- infrastructure/hardware

-- software

--platform

https://www.ibm.com/developerworks/mydeveloperworks/blogs/AliArsanjani/?order=asc&maxresults=15&sortby=0&lang=en

What is the Relationship Between Cloud and SOA

Page 83: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SIMMhttps://www2.opengroup.org/ogsys/publications/viewDocument.html?publicationid=12450&documentid=11340

Page 84: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Vulnerabilities

Security

Testing

DOD is Insisting on SOA

Page 85: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Discoverable

Self-Describing

Composable

Loose Coupling

Governed by Policy

Independent of Location, Language and Protocol

Definition of a Service

Page 86: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SOA Maturity Model

Page 87: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

IT Benefits Of SOA

Page 88: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SOA

Page 89: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SOA

Page 90: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SOA

Page 91: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 92: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

SOA

Page 93: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Clear Separation of Functions

Clear Definition of Communication and Collaboration

TEAM of Software Quality

Tiers of Software Testing

Page 94: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The number of versions will increase

The number of Environments will increase

The need for Automation and Regression Testing will increase

UI Testing will still be “Complete Application” or “END-USER Testing”

The interfaces may Change

Page 95: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

70% of all Application Testing is MANUAL

The UNFORTUNATE REALITY

Page 96: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Visual Studio Ultimate, Visual Studio Premium and Test Professional include a Microsoft Test Managero to help you define and manage your testing effort

by using test plans.

o You create a test plan and add any test suites, test cases, or configurations that you need

Visual Studio 2012

Page 97: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

In 2012 Visual Studio Improves:o Integration between testing and the rest of the

Software Development lifecycle

o Example – Better handoff from tester to developer

o Less NO REPRO

o Improved build-deploy-test workflow

o Improved Virtualization

Visual Studio 2010 to 2012Completely Restructured

Software Testing

Page 98: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Manual Tests

Coded UI Tests

Unit Tests

Web Performance Tests

Load Tests

Generic Tests

Ordered Tests

Types of TestSupported by VS 2012

Page 99: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Require a Human to Interact with an application

Verify some expected result

Report on the success of the test

Manual Tests

Page 100: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Provides the Capability to author tests that automatically interact with user interface

Verify Results

File Bug Report

It can be run Frequently without human intervention

Typically more expensive to author and maintain than manual test

Coded UI Tests

Page 101: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Used by Programmers for Programmers

Quickly becoming the standard practice

A vital part of Test-Driven Development

Generally focus on a lower level than other Testing

Ideally a unit is the smallest atomic part - One method of One object

Test written before the object

It will initially failo Correct

o Re-test

o Refactor

o Repeat

Essential component of Regression Testing

Unit Tests

Page 102: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Verify that the web site is behaving as expectedo Add load to see if the application is still capable of

Scaling

Web Performance Tests

Page 103: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Tests that Application Scales and Performs as Necessary

Load Tests

Page 104: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Calling external testing systems

Results can be automatically parsed to determine success

Could be simple console application to XML document exported from LOB

Generic Tests

Page 105: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Container for other tests

Specific order of tests

Some run more than once

Ordered Tests

Page 106: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The Key challenge of Testers

Provide Developers with sufficient information to reproduce the problem

Most Bugs result in No Repro

“It Works on my machine”

Diagnostic Data Adapters

Page 107: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

A Diagnostic Data Collector is responsible for collecting data about the machine under testo Action Log

o ASP.NET Profiler

o Code Coverage

o IntelliTrace

o Event Log

o System Information

o Test Impact

o Video Recorder

o Network Emulation

Diagnostic Data Adapters

Page 108: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Test-Driven Development

Unit Testing

Developers view

Page 109: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Is a software development process that relies on the repetition of a very short development cycle:

First the developer writes an (initially failing) automated test caseo That defines a desired improvement or new function

o Then produces the minimum amount of code to pass that test

o Refactors the new code to acceptable standards

Test-driven development (TDD)WikiPedia

Page 110: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Not Unit Testing

Not Integration Testing

Not Regression Testing

Whole Application Testing

Tester’s View

Page 111: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

When requirements or user stories, or features are ready to be tested, you can run your tests for each configuration that you specified

UI – User Interface Testing

Usability - Testing

Requirements-Driven Testing

Page 112: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

• Test Runner: The Test Runner from MTM can create action recordings from a manual test. The action recording can later be used to play back the test session in MTM or to generate a Coded UI test from the recorded steps.

• Coded UI Test: The implementation of the actual test written in C# or VB.NET. The Coded UI Test is either generated by the Coded UI Test builder code generator or coded directly against the Coded UI Test API.

• Test Recorder: Records actions from a test session, which then can be used by the Test Runner to automate a generated test or by the coded enerator to create a Coded UI test. Coded UI Test API: The Coded UI Test API can be used to create UI tests which are more tightly connected to the UI elements

• Technology Managers. These are the core components responsible for mapping UI elements to the underlying technology (i.e. web browser or a rich client). We can implement custom plug-ins to support platforms which the native plug-ins does not support.

Page 113: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 114: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

These tests are critical to project success

The number keeps growing

Regression testing is vital

Sometimes hundreds or even thousand of tests

Managing Automated Tests

Page 115: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Supported Platforms

As we can see from the architecture overview, the Coded UI tests require support from the platform being tested. For Visual Studio 2012 the following platforms are supported for Coded UI tests as well as action recordings:

•Internet Explorer 8, 9, 10.

•Windows Forms 2.0 and later

•WPF 3.5 and later

•Windows Win32 (not officially supported but may work)

•MFC (partially)

•SharePoint (partially)

•Dynamics CRM

•Citrix/Terminal Services (partially)

•PowerBuilder (partially)

Page 116: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 117: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

A common requirement of automated tests is to be able to run the test a number of times with different parameters. In Chapter 16 we looked at how we can work with data-driven test in details. The same techniques are valid for coded UI tests as well except that the data source for a UI test often points to the data set managed in the associated test case. Using the test case to manage test data for the automated test is nice because then the tester can update the data directly from inside MTM.

Page 118: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Coded UI tests are probably the ideal type of test for regression testing because they give us the opportunity to automatically test the system just like a real user would. But there are a number of issues that make UI tests difficult as a general purpose solution for automation solution, including

•Requires a UI.

•Requires a desktop session to run.

•UI tests are more complex to design and maintain.

•Difficult to scale.

To work around these challenges we generally recommend considering “under-the-UI” tests as a complement to the UI tests. These tests could be written against the APIs in the system or against the controller layer in an MVC style of application.

Page 119: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Another option for running tests from the command-line is to use the Test Case Management command-line tool, TCM.exe. This tool is installed under

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\TCM.exe

There are several options for us to start a test run, for instance:

•Run all tests in a test suite. Locate the Suite Id from the test suite in MTM. tcm run /create /title:"UI Test Run" /planid:5 /suiteid:11 /configid:2 /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

You can use tcm.exe to figure out the Ids for plans, suites, and configurations as well:

•List plans tcm plans /list /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

•List suites Tcm suites /list /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

•List configurations tcm configs /list /collection:http://tfs.kangreen.com:8080/tfs /teamproject:"Expense Reporting"

Page 121: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Components of a Test Plan

Page 122: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Automated tests can quickly test whether your application is still working correctly after code changes have been made to your application

Automating Test should not take more than about 40% of team members' time

Benefits of Automated Testing

Page 123: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The problem with user interface tests is not in how to execute them, but what to test.

Since human testers would often check the whole round-trip of information, from the GUI to the back-end report

First instinct is to replicate that and try to verify business domain rules. This is the Achilles’ heel of UI tests, and it ultimately leads to enormous waste of time.

The Achilles’ heel of automated tests

Page 124: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

UI Testing without shooting yourself in the foot

http://gojko.net/2010/04/13/how-to-implement-ui-testing-without-shooting-yourself-in-the-foot-2/

Page 125: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The idea of thinking about these different levels is good because it allows us to write

UI-level tests that are easy to understand,

efficient to write

relatively inexpensive to maintain

Checking that delivery is available for two books involves putting a book in a shopping cart. Putting a book in a shopping cart involves a sequence of technical steps. Entering address details does as well. Breaking things down like that and combining lower level concepts into higher level concepts reduces the cognitive load and promotes reuse.

Thinking About These Different Levels

Page 126: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

You can plan, manage and execute both manual and exploratory tests.

You can also automate your manual tests after they are stabilized.

While you are performing manual and exploratory tests, you can log bugs. o The bug will contain a trace of your recent actions, a snapshot

of the state of the system, and a copy of any notes you made while exploring the system. You can record your actions in the test case, so that they can be played back on later occasions.

Using Microsoft Test Manager

Page 127: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Manual to Coded UI

Page 128: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 129: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

With Visual Studio 2012 Lab Management it is possible to create an environment based on existing infrastructure without installing SCVMM. This is great because we can now start using Lab Management without any additional installation required. The lab machines in a standard environment can be physical or virtual (or both) and the virtual machine does not have to be managed by SCVMM or Hyper-V.

To create a standard environment you only need to know the names of the machine you want to add to the environment and a user account to be used to log in and deploy the test agent service onto the machine.

A standard environment can be connected to directly from MTM and the Test Runner making the integration with lab environments really smooth.

Page 130: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The alternative to standard environments is to use a SCVMM environment. The key features of a SCVMM environment include

•Create new machines from templates. New lab machines can automatically be created by anyone (having permissions) running MTM. It is no longer required to send a request to the operations department to get machines created for a new environment or to do other maintenance operations such as starting, stopping, snapshotting, or restoring machines.

•Support for snapshots. It is possible to work with snapshots from MTM, which enable testers to save state during testing. This is a powerful feature both to unlock the tester if a test case fails (no need to block an environment while waiting for a resolution) but also to give a developer a fixed point in time to reproduce a bug.

•Connection inside MTM. A SCVMM environment can be connected to MTM and the Test Runner just like with the standard environment.

Page 131: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

There are a number of metrics to can use as KPIs for software testing.

•Number of bugs per state. How many bugs do we have that are active, resolved, or closed? Are the number of active bugs increasing and number of resolved and closed bugs constant? Then we need to look into how we perform our testing.

•Number of bugs sent back from testers for more information. A large number might indicate that communication between developers and testers must improve.

•Code coverage. This shows us how much of the code has been covered by our automated unit tests. We get the value as a percentage of the whole code.

•Tests run results. How are our tests performing? Do we have many failed tests? If so, what can we do to improve this?

•Percent Requirements covered by test cases. Do we write test cases for all our requirements? If not, what is the reason?

•Percent Requirements covered by testing. Do we actually run the tests we have test cases for? If this figure is low and the figure for Percent Requirements covered by test cases is high we might have an issue we need to deal with.

Page 132: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The application tier is the core of TFS. Here we find all services TFS offers.

First we find all web services for team project collections. We all recognize this so no further explanation is necessary:

•Version control

•Build service

•Lab management

•Work Item tracking

•Team Foundation Framework services.

This set of services is running on the Application Tier of TFS and allows us to access the different aspects of Team Foundation Server, including administration, security, and events. They enable us to extend Team Foundation functionality by developing TFS extensions and adapters to integrate third-party tools with TFS 2012.

We also find the Application Web services:

Page 133: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 134: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 135: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 136: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

After you have run your tests, you can report on your progress. When you use test plans to structure your testing approach, you can answer the following questions:

How many tests have passed or failed?

How many tests still have to be run for this iteration?

When will the testing be complete?

Which areas of the product have high test failure rates?

Which test configurations have high test failure rates?

Who has the most tests left to be run?

Can resources be reallocated to better balance the remaining testing?

Which build should the testers be using based on code changes and bug fixes?

The Tester’s Benefits

Page 137: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Essential Guide for Running Automated Tests from a Test Plan

Page 138: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Coded UI Test

You perform the test manually while CUIT Test Builder runs in the Background

You can Specify the Values that should appear in specific fields

The CUIT Test Builder Records actions and generates Code

Your Main Skills are in Testing not Coding

Creating a CUIT

Page 139: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

You can extend the CUIT code

The CUIT Test Builder builds Structured Code

Straightforward to copy and adapt

If you are a Developer

Page 140: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Visual Studio Ultimate, Visual Studio Premium

Required Software

Page 141: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Planned

Exploratory

Load and Muli-user

The 3 Major UI Test Types

Page 142: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Defining Your Testing Effort Using Test Plans

Microsoft Test Manager enables you to do three important tasks, as shown in the following illustration:

Page 143: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester's skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.

Exploratory testing

Page 144: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

The main advantage of exploratory testing is that less preparation is needed, important bugs are found quickly, and at execution time, the approach tends to be more intellectually stimulating than execution of scripted tests.

Another major benefit is that testers can use deductive reasoning based on the results of previous results to guide their future testing on the fly

Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored.“

WikiPedia

Benefits of

Exploratory Testing

Page 145: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Disadvantages are that tests invented and performed on the fly can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.

Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.

DrawBacks of

Exploratory Testing

Page 146: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Exploratory testing is particularly suitable if reuirements and specifications are incomplete, or if there is lack of time.[7][8] The approach can also be used to verify that previous testing has found the most important defects.[7]

Exploratory Testing

Page 147: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.
Page 148: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Performing Exploratory Testing Using Microsoft Test Manager

Visual Studio and Exploratory Testing

In Microsoft Test Manager open Testing Center TestDo Exploratory Testing I don't see “Do Exploratory Testing”

Your team is using a version of TFS prior to 2012.

Page 149: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

o Automated Functional Tests – Using CUIT

o Create a CUIT from Scratch

o Create a CUIT from action recording

o Making CUIT more Robust and Flexible

Lesson 2 Coded User Interface Testing

Page 150: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Contents of a Coded UI Test

When you create a Coded UI Test, the Coded UI Test Builder creates a map of the user interface under test, and also the test methods, parameters, and assertions for all tests. It also creates a class file for each test.

Page 151: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Automating the Calculator

5) Automating Test Cases

Page 153: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

Conduct exploratory test sessions.

Locate, view, and edit your test methods, UI actions and their associated controls in the UI control map.

Quickly track your test plan progress using the Results tab in Microsoft Test Manager.

Clone your test plan to work on different releases in parallel.

Test Windows Store apps running on a remote device using manual tests or exploratory test sessions.

Install and configure agents more easily for Lab Management for Visual Studio 2012.

Include SCVMM 2012 and clustered Hyper-V hosts in Lab Management for Visual Studio 2012.

TestingNew in VS 2012

Page 154: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

6 Automating Coded UI Tests

Page 156: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

10) Upgrading Coded UI Tests from Visual Studio 2010

Page 157: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

11) Extending Coded UI Tests and Action Recordings

Page 158: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

12) Testing Large Applications with Multiple UI Maps

Page 159: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

13) Using HTML5 Controls in Coded UI Tests

Page 160: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

14) Using WPF with Coded UI Tests

Page 161: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

15) Best Practices for Coded UI Tests

Page 162: Automating Coded UI Testing using Microsoft Testing Tools in Visual Studio 2012 Clive G Hermann.

16) Using the Coded UI Test Logs