Supervisor: Prof. Jyri Hämäläinen Instructor: Jari Simolin (M.Sc), Nokia Siemens Networks Espoo, 15.12.2009 Jyri Ilama.

Supervisor: Prof. Jyri Hämäläinen

Instructor: Jari Simolin (M.Sc),

Nokia Siemens Networks

Espoo,15.12.2009

Jyri Ilama

Testing terminology Test automation process & Benefits of TA UMTS architecture IPA2800 platform

◦ Call Management domain Continuous Integration The CI project of Call Management

◦ CI FRT pilot project Lessons learned

◦ What went wrong? Results and analysis

2

(our) Functional testing”Using” the ”system”• Actually, a script uses a sub system of an individual

entity (one, separate network element)• The actual, acceptance / system testing is done later

in a System Verification phase

Regression testing◦ Testing that the software still works after changes

are made in some unit

3

4

How can this be achieved? What’s inside the black boxes?

•Pitfalls to avoid•Good practices

The required effort for selecting a set of test cases and executing them takes a few minutes

More tests can be executed more often Some cases are even impossible to perform

by manual testing Use of resources Repeatability Reusability Simply: Saving resources and time!

5

6

We are interested in RNC’s and MGW’S

Network elements are complex networks of interconnected computer units, communicating via message connections

Network elements have very strict requirements on fault tolerance◦ ”the five nines availability performance”

3G network element platformbased on DX200, running in RNC’sand MGW’s◦ But much more efficient and clearer

of its architecture

7

Responsible of all call related operations◦ Setup, release, and connecting of calls◦ Capacity and performance◦ Remaining calls in case of recovery actions

E.g. Unit failures

8

A practice of software development, where the members of a team integrate their work very frequently, usually at least daily

Whenever an integration is made, it is verified by an automated build to detect possible integration errors as quickly as possible

A complex system that involves lots of people, servers, tools and connections working tightly together

9

10

A computer that runs some CI software and executes integration builds whenever software changes are made

Tools◦ CruiseControl & Hudson

Java-based free software, e.g. Ant, Maven and Rake builds supported

Modularity: plug-in support

11

Started in May 2007: only PRB compilation builds in the beginning, unit tests added later

No decent FRT software available◦ Hipla: an output report of even hundreds of

thousands of lines of plain text, not so nice Summer of 2008: HiBot was designed for

running old (Hipla) test cases in a new way◦ HIT2Python translator, all old Hipla code -> HiBot◦ Telnet functions implemented manually, in a

hurry, by a person who left the company◦ This was the point where this tool was left in the

hands of a summer trainee; me

12

I got HiBot up and running quickly, against all estimates

Started implementing and/or debugging all the functionalities

Lots of occasional problems with the Telnet connections

13

MGW test cases of Call Resource Handling Other pilots by other teams in Call

Management◦ Even more work with correcting all the commands◦ Still random crashes and annoying connection

and prompt detection problems External problems

◦ IPA Reporting Server◦ Other tools, such as CruiseControl and SVN

14

Test automation problems leading to jams and crashes◦ The old, manual test cases had to be automated

properly June 2009: The "Telnet guy" came back

◦ The problems were found and defects corrected◦ Time for me to focus on the essential: to find and

fix defects in IPA2800 platform◦ I implemented a very useful script

15

A very useful script for investigating results

16

Old manual cases are the most difficult ones

Changed outputs of functions: scanning strings◦ If the printout changes a little, does your macro

still work? The principle of modularity

◦ No hard-coded values◦ Reusability of test cases

a Forever-loop is definitely not your friend

17

Consecutive execution◦ A test case should be possible to run in any kind

of situation, no matter if there are e.g. other calls already, hanging, or whatever

◦ Proper teardowns At the end of test suites, so the execution of a new

suite can be started from scratch

Execution order◦ Problematic suites: Which first?

Possible system breakers as last ones

18

New software: Focus on the essential◦ Critical features/functionality

R&D&T should be done instead of R&D◦ And regression testing should be as important as

the testing of new features Scrum mode: we’re too deep inside it. Tasks

that are not backlog items won’t get enough priority, at least easily◦ Needed: RT sprints or a team dedicated to RT!

Close all external problems out of sight Write totally new test cases if possible

19

A stabile, working CI environment for our FRT◦ Real defects are found all the time

Also Backlog Item Testing can be done efficiently The "output2report" script was found very

useful Good lessons about test automation

◦ As well as writing modular, intelligent test cases

A very good quality level of the product release!

20

21