CSTE Subjective Questions
CSTE Subjective Questions
The following questions are gathered from different place (from
Internet, Yahoo Groups, MSN groups, from o0ther posts, etc). This
is just for understanding to CSTE aspirants that how the subjective
questions will come in CSTE. Q1) The top management was feeling
that when there are any changes in the technology being used,
development schedules etc, it is a waste of time to update the Test
Plan also from time to time. Instead, they are emphasizing that you
should put your time into testing than working on the test plan.
Your Project Manager asked for your opinion. You have argued that
Test Plan is very important and you need to update your test plan
from time to time. It is not at all a waste of time and testing
activities would be more effective when you have your plan clear.
Explain with some metrics, how you would support your argument to
have the test plan in place all the time.Q2) The QAI is starting a
project to put the CSTE certification online. They will use an
automated process for recording candidate information, scheduling
candidates for exams, keeping track of results and sending out
certificates. Write a brief test plan for this new project.Q3) It
is being observed that in a project cost of testing is very high.
After going in detail, it was found that the testers are testing
the software that is not left with many defects. How will you make
sure that this is correct? Any three ways? What are the
disadvantages of Over Testing? (IMP question)Q4) What if the
application has a functionality that was not mentioned in the
requirements?Q5) You are given two scenarios to test. Scenario 1
has only one terminal for entry and processing whereas scenario 2
has several terminals where the data input can be made. Assuming
that the processing work is the same, what would be the specific
tests in Scenario 2 that you would do, which you would not carry on
Scenario 1? (V. IMP question)Q6) Your customer does not have
experience in writing Acceptance Test Plan. How will you do that in
coordination with customer? What will be the contents of Acceptance
Test Plan? (V. IMP question)Q7) How can it be known when to stop
testing?
Q8) What can be done if requirements are changing
continuously?Q9) What is the need for Test Planning?
Q10) What are the various status reports you will generate to
Developers and Senior Management?Q11) Define and explain any three
aspects of Code Review? (15 points) ?Q12) What is the need for Test
Planning?
Q13) Explain 5 risks in an e-Commerce project. Identify those
personnel who must be involved in the risk analysis of a project
and describe what they need to do. How will you prioritize the
risks?Q14) What are the various status reports you will generate to
Developers and Senior Management?Q15) Write 3 conditions that
influence one to stop testing other than the Project
Schedules?.
Q16) You have been asked to design a Defect Tracking system.
Explain what all fields you would put in the defect tracking
system?Q17) A credit limit has been fixed for $5,000, $10,000 and
$15,000. Give the 3 lower boundary values and 3 upper boundary
values for the credits?.Q18) Write a sample Test Policy?Q19)
Explain the various types of testing after arranging them in a
chronological order?Q20) Explain what test tools you will buy for
Client Server testing and why?Q21) Explain what test tools you will
buy for Web testing and why?Q22) Explain points in favor of testing
by Development team themselves and testing by an Independent
team?Q23) Differentiate Validation and Verification?Q24) Explain
Stress, Load and Performance testing?Q25) Describe automated
capture/playback tools and list the benefits of using them?Q26) How
can software QA processes be implemented without stifling
productivity?Q27) How is testing affected by object-oriented
designs?Q28) What is extreme programming and what does it have to
do with testing?Q29) Suppose you have written a project status
report for expected defects vs. Actual defects, using testing
metrics. If you PM ask, you to explain how to read this report,
what recommendations would you give her?Q30) Write a Test
transaction for a scenario where 6.2% of tax deduction for the
first $62,000 of income has to be done?Q31) What would be the Test
Objective for Unit Testing? What would be the quality measurements
to assure that unit testing is complete? (15 points)Q32) Prepare a
checklist for the developers on Unit Testing before the application
comes to testing department?Q33) What fields would you include in
creating a new defect tracking program (used by QA, developers,
etc)? (25 points)Q34) Draw a pictorial diagram of a report you
would create for developers to determine project status.Q35) Draw a
pictorial diagram of a report you would create for users and
management to show project status.Q36) What 3 tools would you
purchase for your company for use in testing and justify why you
would want them? (this question is in both essay parts, only
rephrased. I think 10 points each time)Q37) Describe the difference
between validation and verification. (5 points)Q38) Put the
following testing types in order and give a brief description of
each. System testing, acceptance testing, unit testing, integration
testing, benefits realization testing. (10 points)Q39) List what
you think are the two primary goals of testing. (5 or 10
points)Q40) If you company is going to conduct a review meeting,
what position would u select in the review committee and why? Q41)
Write any three attributes which will impact the Testing Process?
Q42) What activity is done in Acceptance Testing, which is not done
in System testing to ensure the Customer requirements?Q43) You are
a tester for testing a large system. The system data model is very
large with many attributes and there are a lot of interdependencies
within the fields. What steps would you use to test the system and
also what are the effects of the steps you have taken on the test
plan?Q44) Explain give examples of the following black box
techniques?Boundary Value testingEquivalence testingError
GuessingQ45)What are the product standards for? Test PlanTest
Script and Test ReportQ46) You are the test manager for about to
the system testing. The development team says that due to change in
requirements, they will be able to deliver the system to you for
testing 5 working days after the due date. You cannot change the
resources (Work hours, days, test tools). What steps will you take
to be able to finish the testing in time?Q47) Your company is about
to roll out an E-Commerce application. It is not possible to test
the application on all types of browsers on all platforms and
operating systems. What steps would you take in the testing
environment to reduce the business risks and commercial risks?Q48)
In your organization, testers are delivering code for system
testing without performing unit testing. Give an example of test
policy in this matter.Policy statementMethodologyMeasurementQ49) It
has been observed that testers in your organization are performing
tests on the deliverables even after significant defects have been
found. This has resulted in unnecessary testing of little value
because re-testing needs to be done after defects have been
rectified. You as the test manager are going to update the test
plan with recommendations on when to stop testing. List what
recommendations you are going to make?Q50) How do you measure?Test
EffectivenessTest EfficiencyQ51) You find that the senior testers
are making more mistakes then junior testers; you need to
communicate this aspect to the senior tester. Also, you don't want
to loose this tester. How should one goabout the constructive
criticism? Q52) You are assigned as the test lead for a new program
that will automate takeoffs and landing at an airport. How would
you write a est strategy for this new program?
software qa/testing faqs for test/verification engineer, tester,
text resume
Software QA/Testing
Glossary and Technical FAQs
Q1. What is verification?
A: Verification ensures the product is designed to deliver all
functionality to the customer; it typically involves reviews and
meetings to evaluate documents, plans, code, requirements and
specifications; this can be done with checklists, issues lists,
walkthroughs and inspection meetings.
Q2. What is validation?
A: Validation ensures that functionality, as defined in
requirements, is the intended behavior of the product; validation
typically involves actual testing and takes place after
verifications are completed.
Q3. What is a walk-through?
A: A walk-through is an informal meeting for evaluation or
informational purposes.
Q4. What is an inspection?
A: An inspection is a formal meeting, more formalized than a
walk-through and typically consists of 3-10 people including a
moderator, reader (the author of whatever is being reviewed) and a
recorder (to make notes in the document). The subject of the
inspection is typically a document, such as a requirements document
or a test plan. The purpose of an inspection is to find problems
and see what is missing, not to fix anything. The result of the
meeting should be documented in a written report. Attendees should
prepare for this type of meeting by reading through the document,
before the meeting starts; most problems are found during this
preparation. Preparation for inspections is difficult, but is one
of the most cost-effective methods of ensuring quality, since bug
prevention is more cost effective than bug detection.
Q5. What is quality?
A: Quality software is software that is reasonably bug-free,
delivered on time and within budget, meets requirements and
expectations and is maintainable. However, quality is a subjective
term. Quality depends on who the customer is and their overall
influence in the scheme of things. Customers of a software
development project include end-users, customer acceptance test
engineers, testers, customer contract officers, customer
management, the development organization's management, test
engineers, testers, salespeople, software engineers, stockholders
and accountants. Each type of customer will have his or her own
slant on quality. The accounting department might define quality in
terms of profits, while an end-user might define quality as user
friendly and bug free.
Q6. What is good code?
A: A good code is code that works, is free of bugs and is
readable and maintainable. Organizations usually have coding
standards all developers should adhere to, but every programmer and
software engineer has different ideas about what is best and what
are too many or too few rules. We need to keep in mind that
excessive use of rules can stifle both productivity and creativity.
Peer reviews and code analysis tools can be used to check for
problems and enforce standards.
Q7. What is good design?
A: Design could mean to many things, but often refers to
functional design or internal design. Good functional design is
indicated by software functionality can be traced back to customer
and end-user requirements. Good internal design is indicated by
software code whose overall structure is clear, understandable,
easily modifiable and maintainable; is robust with sufficient error
handling and status logging capability; and works correctly when
implemented.
Q8. What is software life cycle?
A: Software life cycle begins when a software product is first
conceived and ends when it is no longer in use. It includes phases
like initial concept, requirements analysis, functional design,
internal design, documentation planning, test planning, coding,
document preparation, integration, testing, maintenance, updates,
re-testing and phase-out.
Q9. Why are there so many software bugs?
A: Generally speaking, there are bugs in software because of
unclear requirements, software complexity, programming errors,
changes in requirements, errors made in bug tracking, time
pressure, poorly documented code and/or bugs in tools used in
software development.
There are unclear software requirements because there is
miscommunication as to what the software should or shouldn't
do.
Software complexity. All of the followings contribute to the
exponential growth in software and system complexity: Windows
interfaces, client-server and distributed applications, data
communications, enormous relational databases and the sheer size of
applications.
Programming errors occur because programmers and software
engineers, like everyone else, can make mistakes.
As to changing requirements, in some fast-changing business
environments, continuously modified requirements are a fact of
life. Sometimes customers do not understand the effects of changes,
or understand them but request them anyway. And the changes require
redesign of the software, rescheduling of resources and some of the
work already completed have to be redone or discarded and hardware
requirements can be effected, too.
Bug tracking can result in errors because the complexity of
keeping track of changes can result in errors, too.
Time pressures can cause problems, because scheduling of
software projects is not easy and it often requires a lot of
guesswork and when deadlines loom and the crunch comes, mistakes
will be made.
Code documentation is tough to maintain and it is also tough to
modify code that is poorly documented. The result is bugs.
Sometimes there is no incentive for programmers and software
engineers to document their code and write clearly documented,
understandable code. Sometimes developers get kudos for quickly
turning out code, or programmers and software engineers feel they
have job security if everyone can understand the code they write,
or they believe if the code was hard to write, it should be hard to
read.
Software development tools , including visual tools, class
libraries, compilers, scripting tools, can introduce their own
bugs. Other times the tools are poorly documented, which can create
additional bugs.
Q10. How do you introduce a new software QA process?
A: It depends on the size of the organization and the risks
involved. For large organizations with high-risk projects, a
serious management buy-in is required and a formalized QA process
is necessary. For medium size organizations with lower risk
projects, management and organizational buy-in and a slower,
step-by-step process is required. Generally speaking, QA processes
should be balanced with productivity, in order to keep any
bureaucracy from getting out of hand. For smaller groups or
projects, an ad-hoc process is more appropriate. A lot depends on
team leads and managers, feedback to developers and good
communication is essential among customers, managers, developers,
test engineers and testers. Regardless the size of the company, the
greatest value for effort is in managing requirement processes,
where the goal is requirements that are clear, complete and
testable.
Q11. Give me five common problems that occur during software
development.
A: Poorly written requirements, unrealistic schedules,
inadequate testing, adding new features after development is
underway and poor communication.
Requirements are poorly written when requirements are unclear,
incomplete, too general, or not testable; therefore there will be
problems.
The schedule is unrealistic if too much work is crammed in too
little time.
Software testing is inadequate if none knows whether or not the
software is any good until customers complain or the system
crashes.
It's extremely common that new features are added after
development is underway.
Miscommunication either means the developers don't know what is
needed, or customers have unrealistic expectations and therefore
problems are guaranteed.
Q12. Give me five solutions to problems that occur during
software development.
A: Solid requirements, realistic schedules, adequate testing,
firm requirements and good communication.
Ensure the requirements are solid, clear, complete, detailed,
cohesive, attainable and testable. All players should agree to
requirements. Use prototypes to help nail down requirements.
Have schedules that are realistic. Allow adequate time for
planning, design, testing, bug fixing, re-testing, changes and
documentation. Personnel should be able to complete the project
without burning out.
Do testing that is adequate. Start testing early on, re-test
after fixes or changes, and plan for sufficient time for both
testing and bug fixing.
Avoid new features. Stick to initial requirements as much as
possible. Be prepared to defend design against changes and
additions, once development has begun and be prepared to explain
consequences. If changes are necessary, ensure they're adequately
reflected in related schedule changes. Use prototypes early on so
customers' expectations are clarified and customers can see what to
expect; this will minimize changes later on.
Communicate. Require walk-throughs and inspections when
appropriate; make extensive use of e-mail, networked bug-tracking
tools, tools of change management. Ensure documentation is
available and up-to-date. Use documentation that is electronic, not
paper. Promote teamwork and cooperation.
Q13. Do automated testing tools make testing easier?
A: Yes and no. For larger projects, or ongoing long-term
projects, they can be valuable. But for small projects, the time
needed to learn and implement them is usually not worthwhile. A
common type of automated tool is the record/playback type. For
example, a test engineer clicks through all combinations of menu
choices, dialog box choices, buttons, etc. in a GUI and has an
automated testing tool record and log the results. The recording is
typically in the form of text, based on a scripting language that
the testing tool can interpret. If a change is made (e.g. new
buttons are added, or some underlying code in the application is
changed), the application is then re-tested by just playing back
the recorded actions and compared to the logged results in order to
check effects of the change. One problem with such tools is that if
there are continual changes to the product being tested, the
recordings have to be changed so often that it becomes a very
time-consuming task to continuously update the scripts. Another
problem with such tools is the interpretation of the results
(screens, data, logs, etc.) that can be a time-consuming task.
Q14. What makes a good test engineer?
A: Rob Davis is a good test engineer because he
Has a "test to break" attitude,
Takes the point of view of the customer,
Has a strong desire for quality,
Has an attention to detail, He's also
Tactful and diplomatic and
Has good a communication skill, both oral and written. And
he
Has previous software development experience, too.
Good test engineers have a "test to break" attitude, they take
the point of view of the customer, have a strong desire for quality
and an attention to detail. Tact and diplomacy are useful in
maintaining a cooperative relationship with developers and an
ability to communicate with both technical and non-technical
people. Previous software development experience is also helpful as
it provides a deeper understanding of the software development
process, gives the test engineer an appreciation for the
developers' point of view and reduces the learning curve in
automated test tool programming.
Q15. What makes a good QA engineer?
A: The same qualities a good test engineer has are useful for a
QA engineer. Additionally, Rob Davis understands the entire
software development process and how it fits into the business
approach and the goals of the organization. Rob Davis'
communication skills and the ability to understand various sides of
issues are important.
Good QA engineers understand the entire software development
process and how it fits into the business approach and the goals of
the organization. Communication skills and the ability to
understand various sides of issues are important.
Q16. What makes a good resume?
A: On the subject of resumes, there seems to be an unending
discussion of whether you should or shouldn't have a one-page
resume. The followings are some of the comments I have personally
heard: "Well, Joe Blow (car salesman) said I should have a one-page
resume." "Well, I read a book and it said you should have a one
page resume." "I can't really go into what I really did because if
I did, it'd take more than one page on my resume." "Gosh, I wish I
could put my job at IBM on my resume but if I did it'd make my
resume more than one page, and I was told to never make the resume
more than one page long." "I'm confused, should my resume be more
than one page? I feel like it should, but I don't want to break the
rules." Or, here's another comment, "People just don't read resumes
that are longer than one page." I have heard some more, but we can
start with these. So what's the answer? There is no scientific
answer about whether a one-page resume is right or wrong. It all
depends on who you are and how much experience you have. The first
thing to look at here is the purpose of a resume. The purpose of a
resume is to get you an interview. If the resume is getting you
interviews, then it is considered to be a good resume. If the
resume isn't getting you interviews, then you should change it. The
biggest mistake you can make on your resume is to make it hard to
read. Why? Because, for one, scanners don't like odd resumes. Small
fonts can make your resume harder to read. Some candidates use a
7-point font so they can get the resume onto one page. Big mistake.
Two, resume readers do not like eye strain either. If the resume is
mechanically challenging, they just throw it aside for one that is
easier on the eyes. Three, there are lots of resumes out there
these days, and that is also part of the problem. Four, in light of
the current scanning scenario, more than one page is not a
deterrent because many will scan your resume into their database.
Once the resume is in there and searchable, you have accomplished
one of the goals of resume distribution. Five, resume readers don't
like to guess and most won't call you to clarify what is on your
resume. Generally speaking, your resume should tell your story. If
you're a college graduate looking for your first job, a one-page
resume is just fine. If you have a longer story, the resume needs
to be longer. Please put your experience on the resume so resume
readers can tell when and for whom you did what. Short resumes --
for people long on experience -- are not appropriate. The real
audience for these short resumes is people with short attention
spans and low IQs. I assure you that when your resume gets into the
right hands, it will be read thoroughly.
Q17. What makes a good QA/Test Manager?
A: QA/Test Managers are familiar with the software development
process; able to maintain enthusiasm of their team and promote a
positive atmosphere; able to promote teamwork to increase
productivity; able to promote cooperation between Software and
Test/QA Engineers, have the people skills needed to promote
improvements in QA processes, have the ability to withstand
pressures and say *no* to other managers when quality is
insufficient or QA processes are not being adhered to; able to
communicate with technical and non-technical people; as well as
able to run meetings and keep them focused.
Q18. What is the role of documentation in QA?
A: Documentation plays a critical role in QA. QA practices
should be documented, so that they are repeatable. Specifications,
designs, business rules, inspection reports, configurations, code
changes, test plans, test cases, bug reports, user manuals should
all be documented. Ideally, there should be a system for easily
finding and obtaining of documents and determining what document
will have a particular piece of information. Use documentation
change management, if possible.
Q19. What about requirements?
A: Requirement specifications are important and one of the most
reliable methods of insuring problems in a complex software project
is to have poorly documented requirement specifications.
Requirements are the details describing an application's externally
perceived functionality and properties. Requirements should be
clear, complete, reasonably detailed, cohesive, attainable and
testable. A non-testable requirement would be, for example,
"user-friendly", which is too subjective. A testable requirement
would be something such as, "the product shall allow the user to
enter their previously-assigned password to access the
application". Care should be taken to involve all of a project's
significant customers in the requirements process. Customers could
be in-house or external and could include end-users, customer
acceptance test engineers, testers, customer contract officers,
customer management, future software maintenance engineers,
salespeople and anyone who could later derail the project. If
his/her expectations aren't met, they should be included as a
customer, if possible. In some organizations, requirements may end
up in high-level project plans, functional specification documents,
design documents, or other documents at various levels of detail.
No matter what they are called, some type of documentation with
detailed requirements will be needed by test engineers in order to
properly plan and execute tests. Without such documentation there
will be no clear-cut way to determine if a software application is
performing correctly.
Q20. What is a test plan?
A: A software project test plan is a document that describes the
objectives, scope, approach and focus of a software testing effort.
The process of preparing a test plan is a useful way to think
through the efforts needed to validate the acceptability of a
software product. The completed document will help people outside
the test group understand the why and how of product validation. It
should be thorough enough to be useful, but not so thorough that
none outside the test group will be able to read it.
Q21. What is a test case?
A: A test case is a document that describes an input, action, or
event and its expected result, in order to determine if a feature
of an application is working correctly. A test case should contain
particulars such as a...
Test case identifier;
Test case name;
Objective;
Test conditions/setup;
Input data requirements/steps, and
Expected results.
Please note, the process of developing test cases can help find
problems in the requirements or design of an application, since it
requires you to completely think through the operation of the
application. For this reason, it is useful to prepare test cases
early in the development cycle, if possible.
Q22. What should be done after a bug is found?
A: When a bug is found, it needs to be communicated and assigned
to developers that can fix it. After the problem is resolved, fixes
should be re-tested. Additionally, determinations should be made
regarding requirements, software, hardware, safety impact, etc.,
for regression testing to check the fixes didn't create other
problems elsewhere. If a problem-tracking system is in place, it
should encapsulate these determinations. A variety of commercial,
problem-tracking/management software tools are available. These
tools, with the detailed input of software test engineers, will
give the team complete information so developers can understand the
bug, get an idea of its severity, reproduce it and fix it.
Q23. What is configuration management?
A: Configuration management (CM) covers the tools and processes
used to control, coordinate and track code, requirements,
documentation, problems, change requests, designs, tools,
compilers, libraries, patches, changes made to them and who makes
the changes. Rob Davis has had experience with a full range of CM
tools and concepts. Rob Davis can easily adapt to your software
tool and process needs.
Q24. What if the software is so buggy it can't be tested at
all?
A: In this situation the best bet is to have test engineers go
through the process of reporting whatever bugs or problems
initially show up, with the focus being on critical bugs. Since
this type of problem can severely affect schedules and indicates
deeper problems in the software development process, such as
insufficient unit testing, insufficient integration testing, poor
design, improper build or release procedures, managers should be
notified and provided with some documentation as evidence of the
problem.
Q25. How do you know when to stop testing?
A: This can be difficult to determine. Many modern software
applications are so complex and run in such an interdependent
environment, that complete testing can never be done. Common
factors in deciding when to stop are...
Deadlines, e.g. release deadlines, testing deadlines;
Test cases completed with certain percentage passed;
Test budget has been depleted;
Coverage of code, functionality, or requirements reaches a
specified point;
Bug rate falls below a certain level; or
Beta or alpha testing period ends.
Q26. What if there isn't enough time for thorough testing?
A: Since it's rarely possible to test every possible aspect of
an application, every possible combination of events, every
dependency, or everything that could go wrong, risk analysis is
appropriate to most software development projects. Use risk
analysis to determine where testing should be focused. This
requires judgment skills, common sense and experience. The
checklist should include answers to the following questions:
Which functionality is most important to the project's intended
purpose?
Which functionality is most visible to the user?
Which functionality has the largest safety impact?
Which functionality has the largest financial impact on
users?
Which aspects of the application are most important to the
customer?
Which aspects of the application can be tested early in the
development cycle?
Which parts of the code are most complex and thus most subject
to errors?
Which parts of the application were developed in rush or panic
mode?
Which aspects of similar/related previous projects caused
problems?
Which aspects of similar/related previous projects had large
maintenance expenses?
Which parts of the requirements and design are unclear or poorly
thought out?
What do the developers think are the highest-risk aspects of the
application?
What kinds of problems would cause the worst publicity?
What kinds of problems would cause the most customer service
complaints?
What kinds of tests could easily cover multiple
functionalities?
Which tests will have the best high-risk-coverage to
time-required ratio?
Q27. What if the project isn't big enough to justify extensive
testing?
A: Consider the impact of project errors, not the size of the
project. However, if extensive testing is still not justified, risk
analysis is again needed and the considerations listed under "What
if there isn't enough time for thorough testing?" do apply. The
test engineer then should do "ad hoc" testing, or write up a
limited test plan based on the risk analysis.
Q28. What can be done if requirements are changing
continuously?
A: Work with management early on to understand how requirements
might change, so that alternate test plans and strategies can be
worked out in advance. It is helpful if the application's initial
design allows for some adaptability, so that later changes do not
require redoing the application from scratch. Additionally, try
to...
Ensure the code is well commented and well documented; this
makes changes easier for the developers.
Use rapid prototyping whenever possible; this will help
customers feel sure of their requirements and minimize changes.
In the project's initial schedule, allow for some extra time to
commensurate with probable changes.
Move new requirements to a 'Phase 2' version of an application
and use the original requirements for the 'Phase 1' version.
Negotiate to allow only easily implemented new requirements into
the project; move more difficult, new requirements into future
versions of the application.
Ensure customers and management understand scheduling impacts,
inherent risks and costs of significant requirements changes. Then
let management or the customers decide if the changes are
warranted; after all, that's their job.
Balance the effort put into setting up automated testing with
the expected effort required to redo them to deal with changes.
Design some flexibility into automated test scripts;
Focus initial automated testing on application aspects that are
most likely to remain unchanged;
Devote appropriate effort to risk analysis of changes, in order
to minimize regression-testing needs;
Design some flexibility into test cases; this is not easily
done; the best bet is to minimize the detail in the test cases, or
set up only higher-level generic-type test plans;
Focus less on detailed test plans and test cases and more on
ad-hoc testing with an understanding of the added risk this
entails. Q29. What if the application has functionality that wasn't
in the requirements?
A: It may take serious effort to determine if an application has
significant unexpected or hidden functionality, which it would
indicate deeper problems in the software development process. If
the functionality isn't necessary to the purpose of the
application, it should be removed, as it may have unknown impacts
or dependencies that were not taken into account by the designer or
the customer.
If not removed, design information will be needed to determine
added testing needs or regression testing needs. Management should
be made aware of any significant added risks as a result of the
unexpected functionality. If the functionality only affects areas,
such as minor improvements in the user interface, it may not be a
significant risk.
Q30. How can software QA processes be implemented without
stifling productivity?
A: Implement QA processes slowly over time. Use consensus to
reach agreement on processes and adjust and experiment as an
organization grows and matures. Productivity will be improved
instead of stifled. Problem prevention will lessen the need for
problem detection. Panics and burnout will decrease and there will
be improved focus and less wasted effort. At the same time,
attempts should be made to keep processes simple and efficient,
minimize paperwork, promote computer-based processes and automated
tracking and reporting, minimize time required in meetings and
promote training as part of the QA process. However, no one,
especially talented technical types, like bureaucracy and in the
short run things may slow down a bit. A typical scenario would be
that more days of planning and development will be needed, but less
time will be required for late-night bug fixing and calming of
irate customers.
Q31. What if organization is growing so fast that fixed QA
processes are impossible?
A: This is a common problem in the software industry, especially
in new technology areas. There is no easy solution in this
situation, other than...
Hire good people (i.e. hire Rob Davis)
Ruthlessly prioritize quality issues and maintain focus on the
customer;
Everyone in the organization should be clear on what quality
means to the customer.
Q32. How is testing affected by object-oriented designs?
A: A well-engineered object-oriented design can make it easier
to trace from code to internal design to functional design to
requirements. While there will be little affect on black box
testing (where an understanding of the internal design of the
application is unnecessary), white-box testing can be oriented to
the application's objects. If the application was well designed
this can simplify test design.
Q33. Why do you recommended that we test during the design
phase?
A: Because testing during the design phase can prevent defects
later on. We recommend verifying three things...
Verify the design is good, efficient, compact, testable and
maintainable.
Verify the design meets the requirements and is complete
(specifies all relationships between modules, how to pass data,
what happens in exceptional circumstances, starting state of each
module and how to guarantee the state of each module).
Verify the design incorporates enough memory, I/O devices and
quick enough runtime for the final product.
Q34. What is software quality assurance?
A: Software Quality Assurance (SWQA) when Rob Davis does it is
oriented to *prevention*. It involves the entire software
development process. Prevention is monitoring and improving the
process, making sure any agreed-upon standards and procedures are
followed and ensuring problems are found and dealt with. Software
Testing, when performed by Rob Davis, is also oriented to
*detection*. Testing involves the operation of a system or
application under controlled conditions and evaluating the results.
Organizations vary considerably in how they assign responsibility
for QA and testing. Sometimes they are the combined responsibility
of one group or individual. Also common are project teams, which
include a mix of test engineers, testers and developers who work
closely together, with overall QA processes monitored by project
managers. It depends on what best fits your organization's size and
business structure. Rob Davis can provide QA and/or SWQA. This
document details some aspects of how he can provide software
testing/QA service. For more information, e-mail
[email protected]
Q35. What is quality assurance?
A: Quality Assurance ensures all parties concerned with the
project adhere to the process and procedures, standards and
templates and test readiness reviews.
Rob Davis' QA service depends on the customers and projects. A
lot will depend on team leads or managers, feedback to developers
and communications among customers, managers, developers' test
engineers and testers.
Q36. Process and procedures - why follow them?
A: Detailed and well-written processes and procedures ensure the
correct steps are being executed to facilitate a successful
completion of a task. They also ensure a process is repeatable.
Once Rob Davis has learned and reviewed customer's business
processes and procedures, he will follow them. He will also
recommend improvements and/or additions.
Q37. Standards and templates - what is supposed to be in a
document?
A: All documents should be written to a certain standard and
template. Standards and templates maintain document uniformity. It
also helps in learning where information is located, making it
easier for a user to find what they want. Lastly, with standards
and templates, information will not be accidentally omitted from a
document. Once Rob Davis has learned and reviewed your standards
and templates, he will use them. He will also recommend
improvements and/or additions.
Q38. What are the different levels of testing?
A: Rob Davis has expertise in testing at all testing levels
listed below. At each test level, he documents the results. Each
level of testing is either considered black or white box
testing.
Q39. What is black box testing?
A: Black box testing is functional testing, not based on any
knowledge of internal software design or code. Black box testing
are based on requirements and functionality.
Q40. What is white box testing?
A: White box testing is based on knowledge of the internal logic
of an application's code. Tests are based on coverage of code
statements, branches, paths and conditions.
Q41. What is unit testing?
A: Unit testing is the first level of dynamic testing and is
first the responsibility of developers and then that of the test
engineers. Unit testing is performed after the expected test
results are met or differences are explainable/acceptable.
Q42. What is parallel/audit testing?
A: Parallel/audit testing is testing where the user reconciles
the output of the new system to the output of the current system to
verify the new system performs the operations correctly.
Q43. What is functional testing?
A: Functional testing is black-box type of testing geared to
functional requirements of an application. Test engineers *should*
perform functional testing.
Q44. What is usability testing?
A: Usability testing is testing for 'user-friendliness'. Clearly
this is subjective and depends on the targeted end-user or
customer. User interviews, surveys, video recording of user
sessions and other techniques can be used. Programmers and
developers are usually not appropriate as usability testers.
Q45. What is incremental integration testing?
A: Incremental integration testing is continuous testing of an
application as new functionality is recommended. This may require
that various aspects of an application's functionality are
independent enough to work separately, before all parts of the
program are completed, or that test drivers are developed as
needed. This type of testing may be performed by programmers,
software engineers, or test engineers.
Q46. What is integration testing?
A: Upon completion of unit testing, integration testing begins.
Integration testing is black box testing. The purpose of
integration testing is to ensure distinct components of the
application still work in accordance to customer requirements. Test
cases are developed with the express purpose of exercising the
interfaces between the components. This activity is carried out by
the test team.
Integration testing is considered complete, when actual results
and expected results are either in line or differences are
explainable/acceptable based on client input.
Q47. What is system testing?
A: System testing is black box testing, performed by the Test
Team, and at the start of the system testing the complete system is
configured in a controlled environment. The purpose of system
testing is to validate an application's accuracy and completeness
in performing the functions as designed. System testing simulates
real life scenarios that occur in a "simulated real life" test
environment and test all functions of the system that are required
in real life. System testing is deemed complete when actual results
and expected results are either in line or differences are
explainable or acceptable, based on client input.
Upon completion of integration testing, system testing is
started. Before system testing, all unit and integration test
results are reviewed by SWQA to ensure all problems have been
resolved. For a higher level of testing it is important to
understand unresolved problems that originate at unit and
integration test levels.
Q48. What is end-to-end testing?
A: Similar to system testing, the *macro* end of the test scale
is testing a complete application in a situation that mimics real
world use, such as interacting with a database, using network
communication, or interacting with other hardware, application, or
system.
Q49. What is regression testing?
A: The objective of regression testing is to ensure the software
remains intact. A baseline set of data and scripts is maintained
and executed to verify changes introduced during the release have
not "undone" any previous code. Expected results from the baseline
are compared to results of the software under test. All
discrepancies are highlighted and accounted for, before testing
proceeds to the next level.
Q50. What is sanity testing?
A: Sanity testing is performed whenever cursory testing is
sufficient to prove the application is functioning according to
specifications. This level of testing is a subset of regression
testing. It normally includes a set of core tests of basic GUI
functionality to demonstrate connectivity to the database,
application servers, printers, etc.
Q51. What is performance testing?
A: Although performance testing is described as a part of system
testing, it can be regarded as a distinct level of testing.
Performance testing verifies loads, volumes and response times, as
defined by requirements.
Q52. What is load testing?
A: Load testing is testing an application under heavy loads,
such as the testing of a web site under a range of loads to
determine at what point the system response time will degrade or
fail. Q53. What is installation testing?
A: Installation testing is testing full, partial, upgrade, or
install/un-install processes. The installation test for a release
is conducted with the objective of demonstrating production
readiness. This test includes the inventory of configuration items,
performed by the application's System Administration, the
evaluation of data readiness, and dynamic tests focused on basic
system functionality. When necessary, a sanity test is performed,
following installation testing.
Q54. What is security/penetration testing?
A: Security/penetration testing is testing how well the system
is protected against unauthorized internal or external access, or
willful damage. This type of testing usually requires sophisticated
testing techniques.
Q55. What is recovery/error testing?
A: Recovery/error testing is testing how well a system recovers
from crashes, hardware failures, or other catastrophic
problems.
Q56. What is compatibility testing?
A: Compatibility testing is testing how well software performs
in a particular hardware, software, operating system, or network
environment.
Q57. What is comparison testing?
A: Comparison testing is testing that compares software
weaknesses and strengths to those of competitors' products.
Q58. What is acceptance testing?
A: Acceptance testing is black box testing that gives the
client/customer/project manager the opportunity to verify the
system functionality and usability prior to the system being
released to production. The acceptance test is the responsibility
of the client/customer or project manager, however, it is conducted
with the full support of the project team. The test team also works
with the client/customer/project manager to develop the acceptance
criteria.
Q59. What is alpha testing?
A: Alpha testing is testing of an application when development
is nearing completion. Minor design changes can still be made as a
result of alpha testing. Alpha testing is typically performed by
end-users or others, not programmers, software engineers, or test
engineers.
Q60. What is beta testing?
A: Beta testing is testing an application when development and
testing are essentially completed and final bugs and problems need
to be found before the final release. Beta testing is typically
performed by end-users or others, not programmers, software
engineers, or test engineers.
Q61. What testing roles are standard on most testing
projects?
A: Depending on the organization, the following roles are more
or less standard on most testing projects: Testers, Test Engineers,
Test/QA Team Lead, Test/QA Manager, System Administrator, Database
Administrator, Technical Analyst, Test Build Manager and Test
Configuration Manager. Depending on the project, one person may
wear more than one hat. For instance, Test Engineers may also wear
the hat of Technical Analyst, Test Build Manager and Test
Configuration Manager.
Q62. What is a Test/QA Team Lead?
A: The Test/QA Team Lead coordinates the testing activity,
communicates testing status to management and manages the test
team.
Q63. What is a Test Engineer?
A: Test Engineers are engineers who specialize in testing. They
create test cases, procedures, scripts and generate data. They
execute test procedures and scripts, analyze standards of
measurements, evaluate results of system/integration/regression
testing. They also...
Speed up the work of your development staff;
Reduce your risk of legal liability;
Give you the evidence that your software is correct and operates
properly;
Improve problem tracking and reporting;
Maximize the value of your software;
Maximize the value of the devices that use it;
Assure the successful launch of your product by discovering bugs
and design flaws, before users get discouraged, before shareholders
loose their cool and before employees get bogged down;
Help the work of your development staff, so the development team
can devote its time to build up your product;
Promote continual improvement;
Provide documentation required by FDA, FAA, other regulatory
agencies and your customers;
Save money by discovering defects 'early' in the design process,
before failures occur in production, or in the field;
Save the reputation of your company by discovering bugs and
design flaws; before bugs and design flaws damage the reputation of
your company.
Q64. What is a Test Build Manager?
A: Test Build Managers deliver current software versions to the
test environment, install the application's software and apply
software patches, to both the application and the operating system,
set-up, maintain and back up test environment hardware. Depending
on the project, one person may wear more than one hat. For
instance, a Test Engineer may also wear the hat of a Test Build
Manager.
Q65. What is a System Administrator?
A: Test Build Managers, System Administrators, Database
Administrators deliver current software versions to the test
environment, install the application's software and apply software
patches, to both the application and the operating system, set-up,
maintain and back up test environment hardware. Depending on the
project, one person may wear more than one hat. For instance, a
Test Engineer may also wear the hat of a System Administrator.
Q66. What is a Database Administrator?
A: Test Build Managers, System Administrators and Database
Administrators deliver current software versions to the test
environment, install the application's software and apply software
patches, to both the application and the operating system, set-up,
maintain and back up test environment hardware. Depending on the
project, one person may wear more than one hat. For instance, a
Test Engineer may also wear the hat of a Database
Administrator.
Q67. What is a Technical Analyst?
A: Technical Analysts perform test assessments and validate
system/functional test requirements. Depending on the project, one
person may wear more than one hat. For instance, Test Engineers may
also wear the hat of a Technical Analyst.
Q68. What is a Test Configuration Manager?
A: Test Configuration Managers maintain test environments,
scripts, software and test data. Depending on the project, one
person may wear more than one hat. For instance, Test Engineers may
also wear the hat of a Test Configuration Manager.
Q69. What is a test schedule?
A: The test schedule is a schedule that identifies all tasks
required for a successful testing effort, a schedule of all test
activities and resource requirements.
Q70. What is software testing methodology?
A: One software testing methodology is the use a three step
process of...
Creating a test strategy;
Creating a test plan/design; and
Executing tests.
This methodology can be used and molded to your organization's
needs. Rob Davis believes that using this methodology is important
in the development and in ongoing maintenance of his customers'
applications.
Q71. What is the general testing process?
A: The general testing process is the creation of a test
strategy (which sometimes includes the creation of test cases),
creation of a test plan/design (which usually includes test cases
and test procedures) and the execution of tests.
Q72. How do you create a test strategy?
A: The test strategy is a formal description of how a software
product will be tested. A test strategy is developed for all levels
of testing, as required. The test team analyzes the requirements,
writes the test strategy and reviews the plan with the project
team. The test plan may include test cases, conditions, the test
environment, a list of related tasks, pass/fail criteria and risk
assessment.
Inputs for this process:
A description of the required hardware and software components,
including test tools. This information comes from the test
environment, including test tool data.
A description of roles and responsibilities of the resources
required for the test and schedule constraints. This information
comes from man-hours and schedules.
Testing methodology. This is based on known standards.
Functional and technical requirements of the application. This
information comes from requirements, change request, technical and
functional design documents.
Requirements that the system can not provide, e.g. system
limitations.
Outputs for this process:
An approved and signed off test strategy document, test plan,
including test cases.
Testing issues requiring resolution. Usually this requires
additional negotiation at the project management level.
Q73. How do you create a test plan/design?
A: Test scenarios and/or cases are prepared by reviewing
functional requirements of the release and preparing logical groups
of functions that can be further broken into test procedures. Test
procedures define test conditions, data to be used for testing and
expected results, including database updates, file outputs, report
results. Generally speaking...
Test cases and scenarios are designed to represent both typical
and unusual situations that may occur in the application.
Test engineers define unit test requirements and unit test
cases. Test engineers also execute unit test cases.
It is the test team who, with assistance of developers and
clients, develops test cases and scenarios for integration and
system testing.
Test scenarios are executed through the use of test procedures
or scripts.
Test procedures or scripts define a series of steps necessary to
perform one or more test scenarios.
Test procedures or scripts include the specific data that will
be used for testing the process or transaction.
Test procedures or scripts may cover multiple test
scenarios.
Test scripts are mapped back to the requirements and
traceability matrices are used to ensure each test is within
scope.
Test data is captured and base lined, prior to testing. This
data serves as the foundation for unit and system testing and used
to exercise system functionality in a controlled environment.
Some output data is also base-lined for future comparison.
Base-lined data is used to support future application maintenance
via regression testing.
A pre-test meeting is held to assess the readiness of the
application and the environment and data to be tested. A test
readiness document is created to indicate the status of the
entrance criteria of the release.
Inputs for this process:
Approved Test Strategy Document.
Test tools, or automated test tools, if applicable.
Previously developed scripts, if applicable.
Test documentation problems uncovered as a result of
testing.
A good understanding of software complexity and module path
coverage, derived from general and detailed design documents, e.g.
software design document, source code and software complexity
data.
Outputs for this process:
Approved documents of test scenarios, test cases, test
conditions and test data.
Reports of software design issues, given to software developers
for correction.
Q74. How do you execute tests?
A: Execution of tests is completed by following the test
documents in a methodical manner. As each test procedure is
performed, an entry is recorded in a test execution log to note the
execution of the procedure and whether or not the test procedure
uncovered any defects. Checkpoint meetings are held throughout the
execution phase. Checkpoint meetings are held daily, if required,
to address and discuss testing issues, status and activities.
The output from the execution of test procedures is known as
test results. Test results are evaluated by test engineers to
determine whether the expected results have been obtained. All
discrepancies/anomalies are logged and discussed with the software
team lead, hardware test lead, programmers, software engineers and
documented for further investigation and resolution. Every company
has a different process for logging and reporting bugs/defects
uncovered during testing.
A pass/fail criteria is used to determine the severity of a
problem, and results are recorded in a test summary report. The
severity of a problem, found during system testing, is defined in
accordance to the customer's risk assessment and recorded in their
selected tracking tool.
Proposed fixes are delivered to the testing environment, based
on the severity of the problem. Fixes are regression tested and
flawless fixes are migrated to a new baseline. Following completion
of the test, members of the test team prepare a summary report. The
summary report is reviewed by the Project Manager, Software QA
(SWQA) Manager and/or Test Team Lead.
After a particular level of testing has been certified, it is
the responsibility of the Configuration Manager to coordinate the
migration of the release software components to the next test
level, as documented in the Configuration Management Plan. The
software is only migrated to the production environment after the
Project Manager's formal acceptance.
The test team reviews test document problems identified during
testing, and update documents where appropriate.
Inputs for this process:
Approved test documents, e.g. Test Plan, Test Cases, Test
Procedures.
Test tools, including automated test tools, if applicable.
Developed scripts.
Changes to the design, i.e. Change Request Documents.
Test data.
Availability of the test team and project team.
General and Detailed Design Documents, i.e. Requirements
Document, Software Design Document.
A software that has been migrated to the test environment, i.e.
unit tested code, via the Configuration/Build Manager.
Test Readiness Document.
Document Updates.
Outputs for this process:
Log and summary of the test results. Usually this is part of the
Test Report. This needs to be approved and signed-off with revised
testing deliverables.
Changes to the code, also known as test fixes.
Test document problems uncovered as a result of testing.
Examples are Requirements document and Design Document
problems.
Reports on software design issues, given to software developers
for correction. Examples are bug reports on code issues.
Formal record of test incidents, usually part of problem
tracking.
Base-lined package, also known as tested source and object code,
ready for migration to the next level.
______________________________________________________
http://www.softwaretestengineer.com
http://www.robdavispe.comDifference between Quality Assurance,
Quality Control, and Testing?
Many people and organizations are confused about the difference
between quality assurance (QA), quality control (QC), and testing.
They are closely related, but they are different concepts.
But all these three are useful to manage risks of developing and
managing software.
Quality Assurance: A set of activities designed to ensure that
the development and/or maintenance process is adequate to ensure a
system will meet its objectives.
Quality Control: A set of activities designed to evaluate a
developed work product.
Testing: The process of executing a system with the intent of
finding defects. (Note that the "process of executing a system"
includes test planning prior to the execution of the test
cases.)
QA activities ensure that the process is defined and
appropriate. Methodology and standards development are examples of
QA activities. A QA review would focus on the process elements of a
project - e.g., are requirements being defined at the proper level
of detail.
QC activities focus on finding defects in specific deliverables
- e.g., are the defined requirements the right requirements
Testing is one example of a QC activity, but there are others
such as inspections
The difference is that QA is process oriented and QC is product
oriented.
Testing therefore is product oriented and thus is in the QC
domain. Testing for quality isn't assuring quality, it's
controlling it.
Quality Assurance makes sure you are doing the right things, the
right way.
Quality Control makes sure the results of what you've done are
what you expected.
What Is The Difference Between Quality Assurance, Quality
Control, And Testing? Many people and organizations are confused
about the difference between quality assurance (QA), quality
control (QC), and testing. They are closely related, but they are
different concepts. Since all three are necessary to effectively
manage the risks of developing and maintaining software, it is
important for software managers to understand the differences. They
are defined below:
Quality Assurance: A set of activities designed to ensure that
the development and/or maintenance process is adequate to ensure a
system will meet its objectives.
Quality Control: A set of activities designed to evaluate a
developed work product.
Testing: The process of executing a system with the intent of
finding defects. (Note that the "process of executing a system"
includes test planning prior to the execution of the test
cases.)
QA activities ensure that the process is defined and
appropriate. Methodology and standards development are examples of
QA activities. A QA review would focus on the process elements of a
project - e.g., are requirements being defined at the proper level
of detail. In contrast, QC activities focus on finding defects in
specific deliverables - e.g., are the defined requirements the
right requirements. Testing is one example of a QC activity, but
there are others such as inspections. Both QA and QC activities are
generally required for successful software development.
Controversy can arise around who should be responsible for QA
and QC activities -- i.e., whether a group external to the project
management structure should have responsibility for either QA or
QC. The correct answer will vary depending on the situation, but
Mosaic's experience suggests that:
While line management should have the primary responsibility for
implementing the appropriate QA, QC and testing activities on a
project, an external QA function can provide valuable expertise and
perspective.
The amount of external QA/QC should be a function of the project
risk and the process maturity of an organization. As organizations
mature, management and staff will implement the proper QA and QC
approaches as a matter of habit. When this happens only minimal
external guidance and review are needed.
Introduction
The world has started to rely increasingly on software that is
becoming more and more complex. Today software is everywhere - part
of every system or machine that we use in daily life. Software
Quality and Reliability have become an important concern of the
software industry. While improved and formal software development
processes help avoid more and more defects, a large number of
defects are still left in software. In one of history's well-known
bugs, the entire long distance of AT&T was down for nine hours.
The melt down was finally traced to a single line of code that
could have been best detected by some effective unit testing.
Beizer argues that the number of bugs left behind after a software
engineer "delivers" his code to others is still too high for
comfort. While improved design techniques try to avoid defects,
effective testing continues to play an important role in removing
defects left behind and is the cornerstone of the Software Quality
Assurance activity.
Testing takes significant effort
Testing accounts for significant part of the software
development lifecycle. According to Boris Beizer "Testing and
debugging costs range from 50% to 80% of the cost of producing the
first working version of a software package. If the lifecycle of
software is considered from inception to retirement, then test and
quality assurance related costs are an even larger part of the
total cost." [1]
Unit Testing plays a major role in the total testing efforts.
Studies by Barry Boehm reveal that coding and unit test takes 36 to
42% of the total resources consumed in software projects of various
sizes [2]. In projects that demand high quality and reliability,
unit testing is a key phase of testing.
Why Unit Testing ?
Units are the smallest building blocks of software. In a
language like C, individual functions make up the units. Unit
testing is the process of validating such small building blocks of
a complex system much before testing an integrated large module or
the system as a whole. Some of the major benefits are:
Be able to test parts of a project with out waiting for the
other parts to be available
Achieve parallelism in testing by being able to test and fix
problems simultaneously by many engineers
Be able to detect and remove defects at a much less cost
compared to other later stages of testing
Be able to take advantage of a number of formal testing
techniques available for unit testing
Simplify debugging by limiting to a small unit the possible code
areas in which to search for bugs
Be able to test internal conditions that are not easily reached
by external inputs in the larger integrated systems (for example,
exception conditions not easily reached in normal operation)
Be able to achieve a high level of structural coverage of the
code
Avoid lengthy compile-build-debug cycles when debugging
difficult problems
Studies have shown that Unit testing is more cost effective
compared to the other stages of testing. The following chart by
Capers Jones shows the relative costs of defects found in different
testing stages
fig 1: Relative Cost per defect for different Testing stages
The cost includes the effort taken for preparing the test cases,
executing the test cases, analysing the results, and fixing the
defects [3] This data proves the value of Early phase testing -
Unit and Integration testing. This leads to the conclusion that
better testing at the early phases is a smarter way to detect and
fix defects.
Unit testing can detect and remove a significant portion of the
defects. A study by Thayer and Lipow shows that comprehensive path
and parameter testing can remove 72.9% of the defects.
Unit Testing - some typical problems
Testing is monotonous, boring and repetitive: Unlike "debugging"
which is like solving a puzzle, testing is an activity that does
not have a predictable end. A Test case may or may not reveal a
defect. Testing gets repeated many times. There is no other
activity in the software engineering life cycle, which is as
repetitive as testing. Each time a change is made on any software,
as an enhancement or for fixing a problem it is desirable that all
the test cases are repeated. In the absence of any reasonable
automation, this is an activity no one enjoys and hence never is
done sufficiently well. Testing, unlike other software development
activities, suffers from a very poor degree of automation.
Surprisingly, while software engineers have gone about automating
practically the whole world around them, they have ignored all
along one of the most manual and disliked activity in their own
backyard. Automation of as many of the routine activities as
possible will help a long way in reducing the monotony of testing.
Defining concrete completeness criteria for the testing activity
also brings some predictability to testing in the sense that now
there is a concrete goal to be achieved. Simply exhorting the
engineers to do better and more complete testing without giving
them good tools to reduce the monotony will continue to fail to
produce results.
Poor Documentation of Test cases: According to G.J.Myers, "throw
away" the test cases only if it is a "throw away" program [4]. Test
cases are as valuable an asset as the program itself. Test cases
are modified and rerun many times in the future. So, documenting
the test cases is very essential for effective future usage. In
practice, test cases are not documented adequately. When the code
gets changed, the affected test cases may have to be modified and
the documentation must be updated. Testing often includes
on-the-fly test cases that are invented while executing a pre
defined set of test cases. Such test cases are very rarely
documented. It is very useful to ensure that test case
documentation gets automated as part of the testing process. This
way while testing gets done the documentation continues to get
generated.Coding Drivers and Stubs: Unit Testing involves writing
code for drivers and stubs which together often add up much more
code than the unit under test. Testers feel reluctant to write such
code that do not go into the final system. The drivers and stubs
may have bugs themselves that result in a lot of additional
debugging effort. Automation of code generation for drivers and
stubs can result in an useful saving of effort for the tester. It
also will ensure that there are no defects in the stubs or drivers
that results in avoidable loss of time.Informal testing process:
While the early part of the software development life-cycle like
Analysis and are well defined and widely practiced, the testing
process is defined much less formally. Well known and effective
testing techniques are often not practiced. While testing accounts
for 50% of the total software development efforts, the software
engineers get very little formal training on testing. In computer
science courses, testing is hardly dealt as an independent
discipline. As a result, testing continues to be an informal and
ill-understood process. Combining Functional (Black box testing
based on the Specifications), Structural (White box testing based
on the structure of the code) and Heuristic (based on human
intuition) testing techniques provide much better results than
simply using an intuitive approach to testing. Many software
engineers run a few intuitive test cases - just enough to unearth
some defects, then they go into "debugging" these defects. Testing
must be mostly systematic and partly intuitive instead of the
general practice of mostly intuitive and partly systematic approach
to testing.
Poor Regression Testing: Testing is a very repetitive activity
that needs to be repeated whenever an enhancement or change is made
to an existing code. For example, if the Test case No. 15 in a
large test suite fails, the problem gets analysed, fixed and the
failed test case alone is rerun to validate the fix. Then the
tester moves on to run Test case no. 16. During the maintenance and
upgradation phase, it is extremely desirable to rerun all the test
cases that were successfully run earlier on the program being
modified. However, for every fix it is safer to rerun all the
affected test cases (better still, all test cases) which have
already been successfully run. Testing today is an essentially
manual process and regression testing is not normally practiced to
the desired degree. It is very useful to build a capability of
retaining automated test cases as a useful resource along with the
code. Automation is the only solution to regression testing.
Lack of Complete Testing tools: While a large number of CASE
tools have emerged over the last decade that address the early
phases of software development lifecycle, Testing has not been so
fortunate. Given that Unit testing takes significant portion of the
total effort, very few tools are available to help in unit testing.
Only coverage analysis tools are available but they address a part
of the whole unit testing activity. Computer Aided Software Testing
(CAST) Tools are a fast growing discipline. Good unit test
automation tools are beginning to become available. Evolution of
such tools achieving a more comprehensive automation of Unit
testing activities are likely to help in a big way in solving many
of the problems currently faced in Unit testing.
Unit Testing Techniques
A number of effective testing techniques are usable in unit
testing stage. The testing techniques may be broadly divided into
three types:
Functional Testing
Structural Testing
Heuristic or Intuitive Testing
The defects in software can in general be classified as
Omissions, Surprises and Wrong Implementations. Omissions are
requirements that are missed in the implementation, Surprises are
implementation that are not found in the requirements and wrong
implementation is incorrect implementation of a requirement.
fig 2: Testing Techniques and Types of Defects
Fig 2 shows the major categories of testing techniques and what
types of defects they are effective against. While Functional
Testing techniques help catch omissions and wrong implementations,
Structural Testing techniques help catch surprises and wrong
implementations. Heuristic or intuitive testing techniques help
catch all types of defects. But Intuitive testing is effective only
when complementing the systematic types of Functional and
Structural testing techniques.
Functional testing Techniques (Some examples)
Boundary Value Analysis: Testing the edge conditions of
boundaries
Equivalence Partitioning: Grouping test cases into classes in
which executing one test cases is equivalent to executing any other
test cases in the same group
Cause Effect Graphing: When the behaviour of the unit under test
is specified as cause and effect. Design test cases that validate
this relationship.
Structural test Cases Techniques (Some examples)
Statement Coverage: Identify Test cases such that every line of
code is executed in one test case or other.
Branch Coverage: Identify Test cases such that every branch of
code is executed in one test case or other. 100% Branch Coverage
automatically assures 100% Statement Coverage.
Condition Coverage: Identify Test cases such that condition in
each predicate expression is evaluated in all possible ways.
Modified Condition-Decision Coverage: Identify Test cases such
that each Boolean operand can independently affect the outcome of a
decision.
Tools for Unit Testing
In the recent times there is an increasing awareness of Test
automation. While many test automation tools are available for
system testing at the GUI level, few complete tools are available
at the Unit and Integration testing stages. The tools available are
mainly coverage analysis tools or script based test case
specification tools.
Fig 3: Stages in Unit Testing
Conclusion
Unit Testing is the earliest stage of testing and is most cost
effective testing stage in removing defects. In later stages of
testing, detecting and fixing defects is more difficult involving
increased effort and time. Unit testing takes significant time and
effort. Unit testing must employ Functional, Structural and
Heuristic testing techniques to be effective against the different
types of defects. Though effective tools have not supported testing
so far, promising test automation tools are beginning to be
available. Unit testing can be very effective and affordable. It
will result in reduction of total efforts while simultaneously
increasing the quality of the product significantly also reducing
in the long-term maintenance cost and the total life cycle cost.
CAST Tools are becoming the key to solve the major problems faced
by Unit testers.
References:
[1] Boris Beizer, System Testing and Software Quality Assurance,
Boris Beizer, International Thomson Computer Press, 1996 [2] Barry
W. Boehm, Software Engineering Economics, Prentice Hall, Inc,
1991[3] Capers Jones, Applied Software Measurements, McGraw Hill,
1991[4] G.J.Myers, The Art of Software Testing, John Wiley &
Sons, 1979
Manual or Automated?
Summary:Automated test tools are powerful aids to improving the
return on the testing investment when used wisely. Some tests
inherently require an automated approach to be effective, but
others must be manual. In addition, automated testing projects that
fail are expensive and politically dangerous. How can we recognize
whether to automate a test or run it manually, and how much money
should we spend on a test?
When Test Automation Makes SenseLets start with the tests that
ideally are automated. These include:
Regression and confirmation. Rerunning a test against a new
release to ensure that behavior remains unbrokenor to confirm that
a bug fix did indeed fix the underlying problemis a perfect fit for
automated testing. The business case for test automation outlined
in Software Test Automation by Mark Fewster and Dorothy Graham is
built around this kind of testing.
Monkey (or random). Tests that fire large amounts or long
sequences of data, transactions, or other inputs at a system in a
random search for errors are easily and profitably automated
Load, volume, and capacity. Sometimes, systems must support
tremendous loads. On one project, we had to test how the system
would respond to 50,000 simultaneous users, which ruled out manual
testing! Two Linux systems running custom load-generating programs
filled the bill.
Performance and reliability. With the rise of Web-based systems,
more and more automated testing is aimed at looking for slow or
flaky behavior on Web systems.
Structural, especially API-based unit, component, and
integration. Most structural testing involves harnesses of some
sort, which brings you most of the way into automation. Again, the
article I wrote with Greg Kubaczkowski, "Mission Made Possible"
(STQE magazine, July/Aug. 2002), provides an example.
Other tests that are well-suited for automation exist, such as
the static testing of complexity and code standards compliance that
I mentioned in the previous article. In general, automated tests
have higher upfront coststools, test development, environments, and
so forthand lower costs to repeat the test.
When to Focus on Manual Testing High per-test or maintenance
costs are one indicator that a test should be done manually.
Another is the need for human judgment to assess the correctness of
the result or extensive, ongoing human intervention to keep the
test running. For these reasons, the following tests are a good fit
for manual testing:
Installation, setup, operations, and maintenance. In many cases,
these tests involve loading CD-ROMs and tapes, changing hardware,
and other ongoing hand-holding by the tester.
Configuration and compatibility. Like operations and maintenance
testing, these tests require reconfiguring systems and networks,
installing software and hardware, and so forth, all requiring human
intervention.
Error handling and recovery. Again, the need to force errorsby
powering off a server, for examplemeans that people must stay
engaged during test execution.
Localization. Only a human tester with appropriate skills can
decide whether a translation makes no sense, is culturally
offensive, or is otherwise inappropriate. (Currency, date, and time
testing can be automated, but the need to rerun these tests for
regression is limited.)
Usability. As with localization, human judgment is needed to
check for problems with the facility, simplicity, and elegance of
the user interface and workflows.
Documentation and help. Like usability and localization,
checking documentation requires human judgment.
WildcardsIn some cases, tests can be done manually, be
automated, or both.
Functional. Functionality testing can often be automated, and
automated functional testing is often part of an effort to create a
regression test suite or smoke test. However, it makes sense to get
the testing process under control manually before trying to
automate functional testing. In addition, youll want to keep some
of the testing manual.
Use cases (user scenarios). By stringing together functional
tests into workflows, you can create realistic user scenarios,
whether manual or automated. The trick here is to avoid automation
if many workflows involve human intervention.
User interface. Basic testing of the user interface can be
automated, but beware of frequent or extensive changes to the user
interface that can incur high maintenance costs for your automated
suite.
Date and time handling. If the test system can reset the
computers clocks automatically, then you can automate these
tests.
Higher per-test costs and needs for human skills, judgment, and
interaction push towards manual testing. A need to repeat tests
many times or reduce the cycle time for test execution pushes
towards automated testing.
Reasons to Be Careful with AutomationAutomated testing is a huge
investment, one of the biggest that organizations make in testing.
Tool licenses can easily hit six or seven figures. Neophytes cant
use most of these toolsregardless of what any glossy test tool
brochure saysso training, consulting, and expert contractors can
cost more than the tools themselves. Then theres maintenance of the
test scripts, which generally is more difficult and time consuming
than maintaining manual test cases.
Acceptance Test Plan
[Download Sample Chapters]What is it?An The Acceptance Test Plan
verifies that the final product meets the clients business
requirements. This template outlines the steps required to prepare
an Acceptance Test Plan. It also ensures that all components of the
system are tested. An Acceptance Test Plan describes the acceptance
testing process, such as the features to be tested, pass/fail
criteria, approach to testing, roles and responsibilities, resource
requirements and schedules. It also defines the functionality to be
tested, the requirements verified by the test, test preconditions,
test steps and test post-conditions.
Who uses it?Test Manager, QA Manager, Project Manager,
Development Manager, QA Engineer, Documentation Manager, IT
Manager, System Administrator.
When is it used?An Acceptance Test Plan is required during the
software testing process to ensure that all features and
functionality are correctly tested and that the system meets the
technical requirements.
Table of Contents1 Project Information1.1 Objectives1.2 Document
Overview1.3 System Description1.4 Key Stakeholders1.5 Relationship
to Other Plans1.6 Points of Contact1.7 References1.8 Methodology,
Tools, and Techniques1.9 Policies, Directives and Procedures2
Acceptance Test Plan2.1 Scope2.2 Business Processes Testing2.3
Testing Approach2.4 Test Schedule2.5 Problem Reporting2.6 Resource
Requirements2.7 Test Environment2.8 Test Identification2.9
Acceptance Test Report2.10 Corrective Action2.11 Summary of
Results2.12 Conclusion3 Project Management3.1 Test Deliverables3.2
Testing Tasks3.3 Schedule3.4 Risk Assessment3.5 Constraints3.6
Issues3.7 Assumptions3.8 Dependencies3.9 Sign-Off Criteria4 Project
Team4.1 Roles4.2 Resources4.3 Software Tools4.4 Training5 Appendix
A5.1 Glossary of Terms5.2 Acronyms and Abbreviations6 Appendix B
Sample Test CaseIndex of Tables
Table 1 Summary of ResultsTable 2 ScheduleTable 3 RisksTable 4
ConstraintsTable 5 IssuesTable 6 AssumptionsTable 7
DependenciesTable 8 Sign-off CriteriaTable 9 Roles and
ResponsibilitiesTable 10 Glossary of TermsTable 11 Acronyms and
AbbreviationsCode Review ChecklistTable of Contents
General Code Smoke TestComments and Coding ConventionsError
HandlingResource LeaksControl StructuresPerformanceFunctionsBug
FixesMath
A word from our sponsor:Visual Studio .NET users can
dramatically simplify code reviews with our CodeReview Add-In for
Visual Studio. With CodeReview you can attach comments, and even
code suggestions, to any source code file supported in Visual
Studio. Download a free 30-day trial
General Code Smoke TestDoes the code build correctly?No errors
should occur when building the source code. No warnings should be
introduced by changes made to the code.Does the code execute as
expected?When executed, the code does what it is supposed to.Do you
understand the code you are reviewing?As a reviewer, you should
understand the code. If you don't, the review may not be complete,
or the code may not be well commented.Has the developer tested the
code?Insure the developer has unit tested the code before sending
it for review. All the limit cases should have been tested.Comments
and Coding ConventionsDoes the code respect the project coding
conventions?Check that the coding conventions have been followed.
Variable naming, indentation, and bracket style should be used.
Does the source file start with an appropriate header and copyright
information?Each source file should start with an appropriate
header and copyright information. All source files should have a
comment block describing the functionality provided by the
file.
Are variable declarations properly commented?Comments are
required for aspects of variables that the name doesn't describe.
Each global variable should indicate its purpose and why it needs
to be global.Are units of numeric data clearly stated?Comment the
units of numeric data. For example, if a number represents length,
indicate if it is in feet or meters.Are all functions, methods and
classes documented?Describe each routine, method, and class in one
or two sentences at the top of its definition. If you can't
describe it in a short sentence or two, you may need to reassess
its purpose. It might be a sign that the design needs to be
improved.Are function parameters used for input or output clearly
identified as such?Make it clear which parameters are used for
input and output.Are complex algorithms and code optimizations
adequately commented?Complex areas, algorithms, and code
optimizations should be sufficiently commented, so other developers
can understand the code and walk through it.Does code that has been
commented out have an explanation?There should be an explanation
for any code that is commented out. "Dead Code" should be removed.
If it is a temporary hack, it should be identified as such.Are
comments used to identify missing functionality or unresolved
issues in the code?A comment is required for all code not
completely implemented. The comment should describe what's left to
do or is missing. You should also use a distinctive marker that you
can search for later (For example: "TODO:francis").
Error HandlingAre assertions used everywhere data is expected to
have a valid value or range?Assertions make it easier to identify
potential problems. For example, test if pointers or references are
valid.Are errors properly handled each time a function returns?An
error should be detected and handled if it affects the execution of
the rest of a routine. For example, if a resource allocation fails,
this affects the rest of the routine if it uses that resource. This
should be detected and proper action taken. In some cases, the
"proper action" may simply be to log the error.Are resources and
memory released in all error paths?Make sure all resources and
memory allocated are released in the error paths.Are all thrown
exceptions handled properly?If the source code uses a routine that
throws an exception, there should be a function in the call stack
that catches it and handles it properly.Is the function caller
notified when an error is detected?Consider notifying your caller
when an error is detected. If the error might affect your caller,
the caller should be notified. For example, the "Open" methods of a
file class should return error conditions. Even if the class stays
in a valid state and other calls to the class will be handled
properly, the caller might be interested in doing some error
handling of its own.Has error handling code been tested?Don't
forget that error handling code that can be defective. It is
important to write test cases that exercise it.Resource LeaksIs
allocated memory (non-garbage collected) freed?All allocated memory
needs to be freed when no longer needed. Make sure memory