P R E S E N T A T I O N International Conference On Software Testing, Analysis & Review DEC 4-8, 2000 • COPENHAGEN, DENMARK Presentation Bio Return to Main Menu W6 Wednesday, Dec 6, 2000 e-Business Testing: Rainer Pirker & Andreas Rudolf Focus on User Perception
17
Embed
Presentation Bio Return to Main Menu W6 · (IMS TCP/IP OTMA Connector) Web/Application Server Authorisation Database IMS ITOC (backup) (IMS TCP/IP OTMA Connector) IMS (backup) Firewall
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
We have had experience of conducting large scale testing projects in many different industries
Our Department is strongly focused on:Test consulting (e-business, EURO, Y2k, new product developments etc.)Application Development Effectiveness consulting (supporting the customer to assess and improve their application development processes)
Our working experience is based on:a dozen large e-business/Y2k/EURO test projects since 1996 where we worked as consultants and as test managersdozens of projects for marketing and workshops for customerprojects in many industries: finance, insurance, transport, utilities
Only if you address the e-business risks from the beginning is a high return on investment in sight
Traditional solution:(+) creative, experienced human being, good exception handling(-) expensive workforce, processing is mainly done by humans (-) medium data quality, frequent 'media breaks'(+) "Human" firewall, resistant against "hacking"(-) service only during business hours
e-business solution:(RISK) single path, weak exception handling without human support(+) Processing is done by computers(+) high data quality and no media breaks (single electronic process)(RISK) hacking and spoofing are possible without a good security solution(RISK) customer expectations are 24h x 7 days service
Key is how users perceive the performance of your site and service
Scalability and Performance:Permit a maximum of 8 seconds for loading a web page. When the loading time increases just 10% the bailout rate goes from 6% to 40% Knowing your requirements is key (number of concurrent users, user profiles, desired response time, throughput, type of sessions, transaction mix, modem speed etc.)Focus on the user's perception of performance:
loading the whole java applet vs. loading JIT the classes needed for each transactionloading large HTML tables vs. loading smaller ones as required by the userthe user will accept longer download times when the site and form are perceived as secure
Taking care of your customer's security fears is a further key to success
User security fearsSecurity perception is key!Good: inform the user of your security solution (SSL, length of key used for encryption etc.)Bad: your site's security problems are made public
Only a customer who trusts your e-business will continue to do business with you!
Company securitypassive Web attacks: wire tapping of password and data, analysis of the structure of the data streamactive Web attacks: spoof, mutilate and delete data, password guessingUse a secure system architecture (i.e. firewalls)For sensitive data (credit card numbers) use a secure protocol like SSL, a secure protocol like SSL can have a big performance impact if not handled by cypto hardwareDo not allow access your companies internal functions. Do not test call center interfaces over the internet
Usability is also key as users need to quickly and easily find the information they seek
Usability aspects:Navigation: usability is improved when the site supports a search facility, a site map and buttons for navigating to and from each page. The site should never contain any broken links.User interactions: the user's ability to interact with the system must be kept very simple and easy i.e. "one click" shopping.Help messages: especially business to customer (B2C) users cannot be trained in the normal way. Extensive help messages and FAQs are usefulComputer literacy: users find it important to be able to quickly locate the information they require on a Web page - graphical design is not the crucial point for most e-business users. Printing: users tend to print Web pages. Reading a page on the screen is about 25% slower than reading a hardcopy of the same textSupport channel: however good your usability is, strong e-mail and telephone support remains a necessity
24x7 operation and the high visibility of e-business solutions drives reliability requirements ever higher
Reliability:B2C: customers from across the world expect 24h x 7days operations which leaves little time for even short service windowsCustomers expect short down times and high mean time between failure rates (MTBF)Outages have a serious business impact:American Express outage cost: 167,000 USD / min Charles Schwab outage cost: 1M USD / min
First we assessed the existing base in terms of test cases, documentation, environment and tools
Assessing the base situation:Test cases: there were none documentedUser documentation: an user handbook was not available however, the functional specification was in good shapeRequirements, design and product documentation: The functional specifications were sufficient however, the non-functional requirements, like performance, transaction rate, reliability etc. were missingTest environment: we agreed with the customer that an isolated test environment should be built. Only the mainframe connection was to the production environment.
Then we defined the test strategy, the heart of any testing project
Defining the test strategyDraw a system structure diagram - a diagram of the architecture was not available so we draw the picture to obtain an insight into possible performance bottlenecks and security implicationsFast start: a very tight time schedule of about 4 weeks forced us to do a really fast start in order to finish the engagement in timePerformance tests first: due to the customer's priorities we had to swap around the traditional approach and do the performance testing before functional testingBlack box end to end test: the customer was interested in the end to end results and how the end users perceived the system's performance
Defining the test conceptDefine the What and HowHighlight 1: Check functional and non functional requirements - for the functional requirements the application and it's specifications were available and sufficient. Non functional requirements like performance, response time, transaction mix, reliability etc. were only partially available and had to be detailed out by us.Highlight 2: Define test cases for measuring end to end performance and for checking system security
Assessingthe
base
Defining thetest strategy
Definingthe testconcept
Start End
Building upthe test
environment
Executing thetest cases
Definingthe testcases
Defining the test casesThe test cases were for business transactions which we found documented in the functional specifications and by exploring the live applicationHeadlines for the test cases were documented in MS Excel
The test environment has to be isolated from the production environment to get reproducible results
Building the test environmentTest environment: a dedicated and isolated test environment was built quite separate from production except for the connection to the mainframeFor defect tracking we used a Web based tool as the developers, customers and testers all worked at different locations.
Quality testing had shown up a very large gap between actual performance and the requirements
The following parameters were measured on the web application server after it was identified early as the performance critical system in the architecture:
CPU LoadVirtual Memory Size
The performance results were about a factor of 50 worse than requirements. The development team was unable to quicklydetermine what needed to be done to meet requirements. The projectwas therefore suspended and restarted with a new development team, project management and quality assurance brief.
Metrics and reports delivered
Response times per transaction type and the number of concurrent users
Response times per transaction type for different user profiles - private customers, small- , medium businesses
To obtain a measure for user perceived performance, we measured the time before the first HTTP data packet arrived. This was interesting for transactions such as bill display were a large amount of data was being transferred. The user himself can see the first page of the bill before the download has finished. This depends on the HTML data stream.
Focusing on quality is key for e-business solutions
Summary and Critical Success Factors (1): Security issues can kill your e-business. Address these issues carefully in advance - use secure protocols, disclosures for the testers, etc..Non functional requirements: define then early in the projectUse test cases based on business transactionsUse an independent test team focused only on the requirements - testers who work also as developers do not usually have that "black box" view of the systemE-business test tool skills: it is necessary to have experience of Web based problem management and browser simulator tools if you need to deploy them fastQuality cannot be tested into a product. Plan for quality from the beginning using, for example, IBM's Lifecycle Testing approach
Focusing on quality is key for e-business solutions
Summary and Critical Success Factors (2): Web content development and e-business solution application development are quite different. Application development requires a strong quality, requirements and project oriented approach Test the performance of your e-business solution under real world conditions (end to end tests, load the server with the right transaction mix, SSL) and in a near real world test environmentUse Web based tools for problem management and test case documentationUser profiles: always keep in mind that there are a lot of different types of users out on the Web who are accessing your e-businessPlan and execute performance tests during all of the solution development phases
W 6 e-Business Testing: Focus on User Perception Rainer Pirker
Rainer Pirker is a consultant with IBM Application Development Effectiveness Practice. He specializes in test concepts and strategies to improve software quality. He has 15 years of experience in information system and application development, and has in-depth knowledge of banking, insurance and public institutions in the fields of executing, developing and improving tests.
Within IBM Rainer Pirker started as a tester and system specialist for a workflow product, became a test leader for a business application and is a test consultant, especially for application development process improvement in testing, e-business, EURO and Y2K problems.
From June 2000 he is test manager in a new development e-business project – this customer project is completely outsource to IBM and the topic is internet payment. The development team consist of about 20 people and the environment is WebSphere. Rainer is responsible for Unit tests up to extensive Performance- and Security tests, including multi platforms and integration of SAP.
Rainer has developed and presented several testing workshops at customers and IBM internal, creating some follow up business at customer sites.
He designed and implemented a test-management tool suite in Lotus Notes and MS Access. This tool suite covers business transaction-, test case-, equivalence classes- and scenario-definition, test case execution and problem-management and is in use at several customers and IBM internal in the central European region.
Before joining IBM, Rainer worked several years for software companies and for public institutions, where he designed and implemented accounting and education software.
Andreas Rudolf Professional Profile
Andreas Rudolf is certified consultant with IBM Application Development Effectiveness Practice. He specializes in concepts and strategies to improving the software application development process, in the last two years his main focus was on test concepts and quality assurance. Mr. Rudolf has over 10 years experience in workflow, application development and -testing. He worked for a variety of industries including telecommunication, banking and insurance. Professional Experience
Currently Mr. Rudolf assesses the application development process for an IT provider with the target to find and to improve the weak points in the requirements and solution validation process.
He was responsible establishing measurements and conducting performance tests for an e-billing project in the telecommunication industry.
He supported a year 2000 test project at a large bank in Germany as test consultant with is expertise in test case creation and execution. Due to the sound and proven methodology used in similar projects, the project finished 6 weeks before schedule.
For a large bank in Austria, Mr. Rudolf has written a concept for testing the whole software portfolio for year 2000 compliance. He built up and led a team of 20 people dedicated to this project. Due to high customer satisfaction with the project, a subsequent contract at this customer led to an assessment of their AD change and problem management processes.
For an insurance company, he built up a performance laboratory for client/server and e-business applications. It will help the client to achieve the performance target and test performance relevant software changes before going to production.
He managed the acceptance test of a central directory access service for a large telecommunication provider. By replacing the old service the customer could reduce his operating costs about 25% and could improve the response time about 20%.
For a telecommunication provider Mr. Rudolf designed the workflow structure of a tracking system for the manual laying of telephone wires. This project lead to an increase of customer satisfaction, since the process cycle time was shortened significantly.
While working for IBMs Vienna Software Development Laboratory he was responsible for the development of a platform and protocol transparent communication layer. His prototype, which integrated IBMs Workflow Manager with the Web, shaped IBMs Workflow strategy and generated customer leads.
Before joining IBM, he worked for a small software development firm, where he designed and implemented a low-cost network for educational use. At a subsequent release he reduced the initial hardware costs about 40%. Education and Personal History
Mr. Rudolf was co-author of the IBM redbook "cooperative processing in an object oriented" environment and he was speaker at international conferences including “Software Quality Week” and “EuroSTAR” with topics in the area of software testing. At the "International Workflow Conference" in Vienna he presented the topic "Workflow meets Cyberspace".
Mr. Rudolf attended the engineering school in Moedling, Austria, where he earned a bachelor degree in telecommunication and electronics. His native language is German and he speaks English fluently.