BETA work in progress, I add every training new material and tune current material. Sogeti Netherlands Microsoft Test and Lab Manager Training Guide A description and short reference guide covering Microsoft Test and Lab Manager training course “Testing Practices with ALM tools”. Clemens Reijnen -- VERSION 0.4 8/8/2011
This guide is an extract from the two and three day course provided by me. It spans the complete testing lifecycle and the tool usages. It will look at the infrastructure implications and testing practices in formal and in agile teams. But, the main focus stays on the usages of the Microsoft Visual Studio testing tools, the knowledge you need to get starting with it, the practices you must have to work with it in real live and how you can bend the tools, with extensibility and normal use to your team needs. The course and this guide is work in progress. It is not a testing training (I expect you already have testing knowledge), if you need that test process information I refer to the TMap website from Sogeti where you can find tons of information. This training guide follows the TMap testing lifecycle.
BETA work in progress, I add every training new material and tune current material.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
BETA work in progress I add every training new material and tune current
material
Sogeti Netherlands
Microsoft Test and
Lab
Manager Training
Guide A description and short reference guide covering Microsoft Test and Lab
Manager training course ldquoTesting Practices with ALM toolsrdquo
Clemens Reijnen -- VERSION 04 882011
The training course This guide is an extract from the two and three day course provided by me It spans the
complete testing lifecycle and the tool usages It will look at the infrastructure
implications and testing practices in formal and in agile teams But the main focus stays
on the usages of the Microsoft Visual Studio testing tools the knowledge you need to get
starting with it the practices you must have to work with it in real live and how you can
bend the tools with extensibility and normal use to your team needs
The course and this guide is work in progress It is not a testing training (I expect you
already have testing knowledge) if you need that test process information I refer to the
TMap website from Sogeti where you can find tons of information This training guide
follows the TMap testing lifecycle
Contents The training course 1
Application Lifecycle Management 4
Test Lifecycle - Planning Phase 8
Work items 9
Lab 1 Create team projects 10
Lab 2 Create user stories and tasks for the employee application 20
Work Item Queries 21
Reports Basic 24
Microsoft Test Manager 28
Test Lifecycle ndash The Control Phase I 29
Test Plans 29
Test Configurations 32
Test Suites 32
Lab 3 Create and configure test plan 34
TCMEXE 36
Test Lifecycle ndash Specification Phase 37
Test Cases 38
Shared Steps 41
Test Data Iterations 42
Lab 4 Create test cases 43
Test Lifecycle ndash Execution Phase 46
Test Run 47
Test Runner 48
Lab 5 Execute the created test cases 50
Test Case Execution Results 51
Test Lifecycle ndash Control Phase II 56
File a Bug 57
Verify a bug 60
Lab 6 file a bug 61
Bug Reports 67
Test Impact 67
Lab 7 test impact 69
Create test from bug 70
Lab 8 Create an exploratory bug and create a test case from it 71
Test Lifecycle - Infrastructure Phase 72
On Physical Environments with Microsoft Test Manager 74
Lab 9 Install and configure the test controller and test agent 79
Lab 10 Setup Flavor D 79
Test Lifecycle - Completion Phase 80
Test automation with Visual Studio 2010 81
When is test automation the investment worth 83
The investment levels explained 84
Lab 13 Create CodedUI 88
The UIMap 89
Data driven CodedUI tests 90
Lab 14 Execute the CodedUI during the build from Visual Studio 91
Setting up Flavor C 91
Lab 16 Execute the codedUI from MTM 93
Setting up Flavor D 93
Lab 17 Execute the codedUI during the build from MTM 95
Setting up Flavor E 95
On Virtualized Environments with Lab Management and SCVMM 96
Extensibility 99
Tools 99
Custome Data Adapter 100
Azure and test automation 101
Test Manager and an agile process 101
Intelitrace 101
Appendix 106
proposed training agenda 2 day training 106
Student and trainer Environment 107
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip