Advances in Unit Testing: Theory and Practice Tao Xie 1 Nikolai Tillmann 2 Pratap Lakshman 2 1 University of Illinois at Urbana-Champaign 2 Microsoft Tools for Software Engineers/Developer Division Materials: http://taoxie.cs.illinois.edu/courses/testing /
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Advances in Unit Testing: Theory and Practice
Tao Xie1 Nikolai Tillmann2 Pratap Lakshman 2
1 University of Illinois at Urbana-Champaign2 Microsoft Tools for Software Engineers/Developer
@RunWith(Parameterized.class)public class ParamTest { public int sum, a, b; public ParamTest (int sum, int a, int b) { this.sum = sum; this.a = a; this.b = b; }
@Parameters public static Collection<Object[]> parameters() { return Arrays.asList (new Object [][] {{0, 0, 0}, {2, 1, 1}}); }
@Test public void additionTest() { assertEquals(sum, a+b); }}
Answer: All Combinations of Values from @DataPoint
Annotations Where Assume Clause is True Four (of Nine) Combinations in This Particular
Case Note: @DataPoint Format is an Array.
@DataPoints public static String[] string = {"ant", "bat", "cat"};
@DataPoints public static Set[] sets = { new HashSet(Arrays.asList("ant", "bat")), new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)), new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop")) };
Successful Case of MSR Testing Tool: Pex & Relatives
Pex (released on May 2008) Shipped with Visual Studio 15 as IntelliTest 30,388 download# (20 months, Feb 08-Oct 09) 22,466 download# (10 months, Apr 13-Jan 14):
Code Digger Active user community: 1,436 forum posts
during ~3 years (Oct 08- Nov 11) Moles (released on Sept 2009)
Shipped with Visual Studio 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio
editions” got 1,457 community votes
A Journey of Bringing Automated Unit Test Generation to Practice
Lesson 1. Started as (Evolved) Dream
void TestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o);}
Pex team: “Do you want to be the first?” Developer/manager: “I love your tool but
no.”
Tool Adoption by (Mass) Target Users
Tool Shipping with Visual Studio
Macro Perspective
Micro Perspective
Lesson 3. Human Factors – Generated Data Consumed by Human
Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”
Pex team: “In C#, a \0 in a string does not mean zero-termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”
Developer: “Your tool generated “\0”” Pex team: “What did you expect?” Developer: “Marc.”
Lesson 3. Human Factors – Generated Name Consumed by Human
Developer: “Your tool generated a test called Foo001. I don’t like it.”
Pex team: “What did you expect?” Developer:“Foo_Should_Fail_When_Bar_Is_Ne
gative.”
Lesson 3. Human Factors – Generated Results Consumed by Human
Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11])
Lesson 5. Tool Users’ Stereotypical Mindset or Habits
“Simply one mouse click and then everything would work just perfectly” Often need environment isolation w/ Moles/Fakes or
factory methods, … “One mouse click, a test generation tool would
detect all or most kinds of faults in the code under test” Developer: “Your tool only finds null references.” Pex team: “Did you write any assertions?” Developer: “Assertion???”
“I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”
Lesson 6. Practitioners’ Voice
Gathered feedback from target tool users Directly, e.g., via
MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups
Indirectly, e.g., via interactions with MS Visual Studio team (a tool
vendor to its huge user base) Motivations of Moles
Refactoring testability issue faced resistance in practice
Observation at Agile 2008: high attention on mock objects and tool supports
Lesson 7. Collaboration w/ Academia Win-win collaboration model
Win (Ind Lab): longer-term research innovation, man power, research impacts, …
Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …
Reggae [ASE’09s] Rex MSeqGen [FSE’09] DyGen Guided Cov [ICSM’10] state coverage
Long-term indirect impacts, e.g., DySy by Csallner et al. [ICSE’08] Seeker [OOPSLA’11] Covana [ICSE’11]
Summary of Lessons Learned
Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code
Hunt Lessons in transferring tools
Started as (Evolved) Dream Chicken and Egg Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia
Experience Reports on Successful Tool Transfer
Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf
Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, and Tao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013. http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf
Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu, Yingjun Qiu, and Tao Xie. XIAO: Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012. http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf
Parameterized Unit Tests Supported by Pex/Pex4Funusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;
[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}
40
Interface for IntSetClass IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}
sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet
http://www.cs.unc.edu/~stotts/723/adt.html
41
(Buggy) Implementation for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs
Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”
if i = j then remove(s, i) else insert(remove(s, i), j)
Are we done yet?The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.
46
Guidelines for Completeness
But does this really specify sets? Do the following properties hold?
Order of insertion is irrelevant. insert(insert(s, i), j) = insert(insert(s, j), i)
Multiple insertion is irrelevant. insert(insert(s, i), i) = insert(s, i)
47
Interface (Implementation) for UIntStackClass UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}
See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs
Take-Home Exercise: Write Parameterized Unit Tests (PUTs)Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}
Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”
Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)
See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs
Recall: Parameterized Unit Tests Supported by Pex/Pex4Funusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;
[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}
50
Force Pex/Pex4Fun to Display All Explored Test Inputs/Pathsusing System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;
[PexClass]public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}
51
Factory Method: Help Pex Generate Desirable Object StatesIn class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states.
[PexFactoryMethod(typeof(UIntStack))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i); return uIntStack;
// TODO: Edit factory method of UIntStack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. }
52
Factory Method: Help Pex Generate Desirable Object StatesBelow is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.
[PexFactoryMethod(typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));
for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; }
53
One Sample PUTBelow is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.
[PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }
54
Pex4Fun Not Supporting Factory Method - WorkaroundIf you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below
[PexMethod] public void TestPush(int[] elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]);
//UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }
Guideline of Writing PUT• Setup: basic set up for invoking the
method under test• Checkpoint: Run Pex to make sure that
you don't miss any Pex assumptions (preconditions) for the PUT
• Assert: add assertions for asserting behavior of the method under test, involving
• Adding Pex assertions• Adding Pex assumptions for helping assert• Adding method sequences for helping
assert
Setup• Select your method under test m• Put its method call in your PUT• Create a parameter for your PUT as
the class under test c (annotated it with [PexAssumeUnderTest])
• Create other parameters for your PUT for parameters of m if any
• Add Pex assumptions for preconditions for all these parameters of PUT if any
Setup - Example[PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { s.Push(i); }
You may write your factory method to help Pex in test generationIf you get exceptions thrown • if indicating program faults, fix them• If indicating lack of PUT assumptions, add PUT assumptions• If indicating insufficient factory method assumptions or
inappropriate scenarios, add PUT assumptions or improve factory method.
Assert• Think about how you can assert the
behavior• Do you need to invoke other (observer)
helper methods in your assertions (besides asserting return values)?
• Do you need to add assumptions so that your assertions can be valid?
• Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?
Targets for Asserting
• Return value of the method under test (MUT)
• Argument object of MUT• Receiver object properties being
modified by MUT (if public fields, directly assertable)
• How to assert them? • Think about the intended behavior!• If you couldn't do so easily, follow the
guidelines discussed next
Cached Public Property Value
• A property value before invoking MUT may need to be cached and later used.
Observer Methods• Invoking observer methods on the
modified object statePattern 2.6: State Relation[PexMethod]void InsertContains(string value) {
var list = new List<string>();list.Add(value);Assert.IsTrue(list.Contains(value));
}
Each modified object property should be read by at least one observer method.
Observer Methods cont.• Forcing observer methods to return
specific values (e.g., true or false) can force you to add specific assumptions or scenarios
[PexMethod]void PushIsFull([PexAssumeUnderTest]UIntStack s, int value) { PexAssume.IsTrue(s.GetSize() == (s.GetMaxSize()-1));
s.Push (value);Assert.IsTrue(s.IsFull ());
}
Alternative Computation• Invoking another method/method
sequence to produce a value to be used
Pattern 2.7: Commutative Diagram[PexMethod]void CommutativeDiagram1(int x, int y) { // compute result in one way string z1 = Multiply(x, y).ToString(); // compute result in another way string z2 = Multiply(x.ToString(), y.ToString()); // assert equality if we get here PexAssert.AreEqual(z1, z2);}
Divide and Conquer• Split possible outcomes into cases
(each with pre and post condition)Pattern 2.8: Cases[PexMethod]void BusinessRules(int age, Job job) {
var salary = SalaryManager.ComputeSalary(age, job);PexAssert