Top Banner
QA Lab Management Meeting new Business Challenges with Dynamic Quality Assurance Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation [email protected]
28

Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation [email protected].

Dec 19, 2015

Download

Documents

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

QA Lab ManagementMeeting new Business Challenges with Dynamic Quality Assurance

Mike AzocarSr. Developer Technical SpecialistMicrosoft [email protected]

Page 2: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

A little about me…

Page 3: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

So who are you?

Developer? Testers? Coachers/PMs? Analyst? Check signers?

Page 4: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

“we have a changed requirement, what do I need to test?”

Have you heard any of these?

“my testers are spending too long testing the same thing”

“tooling is expensive (time, licenses, people)”

“developers and testers work in silos and don’t communicate/speak the same language”

“when is my software ready to ship?”

“the developers say the defects are useless”

Page 5: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Manager

Page 6: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

68%

Page 7: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

What if you could…

reduce the time it takes to determine the root cause of a bug

enable users to easily run acceptance tests and track their results

reduce the time it takes to verify the status of a reported bug fix

reduce the time it takes to replicate a bug uncovered by user actions

reduce the time it takes to isolate differences between the test and production environment

Page 8: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

8

Welcome to the revolution

Page 9: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

rele

ase

pla

nnin

g set test strategydefine done, doneestablish environmentsreview configurationscreate plansgenerate data

multiple iterations

What tester do

inception construction release

rele

ase

ite

rati

on regression

testingrelease doneness testingrelease sign off

itera

tion p

lannin

g add stories to plandefine acceptanceselect regression tests

itera

tion e

xecu

tion author tests

run testsfile bugsverify fixesautomate tests

itera

tion

retr

osp

ect

iveupdate master

planidentify product debtidentify test debtselect test for automation

Page 10: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Generalist SpecialistManual Testing

Some scripting

Creates scripts to set up lab, create data

Strong scripting skills

Some coding skills

Strong coding

Develops fully automated testing procedures

Expert coding skills

Where does testing happen?

Black Box Testing

White Box Testing

API Testing

70% of testing happens here

Majority of test tools target here

Page 11: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Visual Studio 2010 Test Capabilities

coded ui test

unit testingweb performance test

load test

test runner

test case management

lab management

data collectors (video, action log, event log etc)

team foundation server with reporting(bugs, test cases, requirements, user stories, source control, build)

generalist

specialist

Page 12: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

12

Let’s drill down!

Page 13: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

rele

ase

pla

nnin

g set test strategydefine done, doneestablish environmentsreview configurationscreate plansgenerate data

itera

tion p

lannin

g add stories to plandefine acceptanceselect regression tests

itera

tion e

xecu

tion author tests

run testsfile bugsverify fixesautomate tests

itera

tion

retr

osp

ect

iveupdate master

planidentify product debtidentify test debtselect test for automation re

lease

ite

rati

on regression

testingrelease doneness testingrelease sign off

What testers do

inception construction release

Page 14: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Virtual Machines

Hyper-V Host

Visual Studio 2010 Lab Management

Automated Build-

Deploy-Test Workflow

Reduces Hardware Utilization

Remote Diagnostic Collection

Virtual Environment Setup/Tear

down

• Built on Hyper-V• Integrated with TFS

2010• Enabled by SCVMM

• Integrated management• Multi-hypervisor

management• P2V and V2V• Template deployment• Optimize resource usage

• Managed by Microsoft Test Manager

Page 15: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Library Share 1

Stored VM 1

Stored VM 2

Stored VM 3

Build Controller

Test Controller

Visual Studio Lab Management 2010 Leveraging virtualization for ALM

Physical Environment / ESX

Build Agent

Test Agent

Machine 1

Machine 2

Machine 3

• Lab Management

• Build Management

• Test Case Management

• Project Management

• Source Control

• Work Item Tracking

• Reporting

TFS 2010

Visual Studio 2010Microsoft Test Manager

Hyper-V Host 1

Build Agent

Lab AgentTest

Agent

Virtual Machine 1

Virtual Machine 2

Virtual Machine 3

Page 16: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Windows Vista

SCVMMVM Template Library

Sysprepped “gold images”• Windows Server 2003• Windows Server 2008•…etc.• Stored in SCVMM Library

Templates and Virtual Machines

Windows 7

Windows Server 2008

Test AgentBuild

AgentLab Agent

Windows Server 2003

Page 17: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

17

Let’s take a look:Managing Test Environment Complexity

Microsoft Confidential

Page 18: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Reporting on Testing Progress

Build progress overtime is tracked

Recent build status is reported to the

team

Page 19: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Reporting on Testing Progress

Are we getting enough coverage?

How many lines of code are we churning?

Page 20: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Reporting on Testing Progress

Workflow run for Deployment and

Lab Restore snapshots create new ones

after build

Test against Lab Environments Automatically

Page 21: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Failing Tests are Flagged for further

review

Amount of Code Covered by Test is

show

Amount of code checked in per

build

Page 22: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

22

Wrap-Up

Microsoft Confidential

Page 23: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Your Creativity Unleashed

− Save time, focus on high value tasks− Fast Forward for Manual Testing− High quality bugs with single click

− Embrace Change− Rapid Setup/Teardown of Environments− Test prioritization to assess impact of code

check ins− Traceability and Visibility

− Rich Modern Experience− Elegant Task focused UI− Combined manual & fast forward for manual testing− Plan, Test, Track flow

Page 24: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Simplicity through Integration− TFS is THE collaboration hub

− Break silos between roles− Share Virtual environments between

team members

− Go Live Readiness− Performance Testing Early and Often− Highly scalable (Also 64 bit support) − Standard languages for scripts

− Quality Insight− Cross subject area reporting− Real time feedback

Page 25: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Quality code ensured− No more “no repro”

− Integrated, Extensible Diagnostics− Rich Bugs, fix first time, every time− Intelli-trace

− Automation to reduce Regression− Coded UI Test− Familiar language & toolset support

− Eliminate wasted cycles − Test in production like environments− Efficient, predictable build hand off

Page 26: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Call to Action

Download Visual Studio and Team Foundation Server RTM virtual machine for Windows Virtual PC and try Test Professional today! http://bit.ly/VSTFSVPC

Check out the Online MSDN Documentation – Testing http://bit.ly/TestApp

Page 27: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

Questions and Answers

27

Page 28: Mike Azocar Sr. Developer Technical Specialist Microsoft Corporation mazocar@microsoft.com.

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Visual Studio, the Visual Studio logo, and [list other trademarks referenced] are trademarks of the Microsoft group of companies.

 The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.  Because Microsoft must respond

to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. 

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.