Software Quality Analyst Certification Exam · PDF fileReview of Skill Categories in the Common Body of Knowledge (CBOK) for the Certified Software Quality Analyst: • Quality Principles
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
No part of this publication, or translations of it, may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopy-ing, recording, or any other media embodiments now known or hereafter to become known, without the prior written permission of the Quality Assurance Institute.
• Help the quality professional study and prepare for the CSQA examination.• Reinforce current knowledge• Re-introduce concepts that may not be used everyday• Explain the rationale for use within the IT Industry• Highlight sample questions, both multiple choice and essay type responses.• Provide you with a background of the IT quality fundamentals
- Due to time limitations of this course, it is not intended to cover all the components of the common body of knowledge in great depth.
• Who are you and your company• Why and when are you taking the exam• Have you taken an exam before – where/when• Time you spend studying now• Study group experience
Why Become Certified?KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS• The IT and software testing industry is competitive• Recognition by peers• Proof of you profession and skills in the IT field• Increased confidence in personal capabilities • Indicates a professional level of competence • Potentially more rapid career advancement • Greater acceptance in the role as advisor to management
Course Table of ContentsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSReview of Skill Categories in the Common Body of Knowledge (CBOK) for the Certified Software Quality Analyst:• Quality Principles• Quality Leadership• Quality Baselines• Quality Assurance• Quality Planning• Define, Build, Implement, and Improve Work Processes• Quality Control Practices• Metrics and Measurement• Internal Control and Security• Outsourcing, COTS, and Contracting Quality
Skill Category OneKEY CONCEPT PRESENTATION GUIDEQuality Principles and Concepts
SUPPORTING / EXPLANATORY MATERIALS• Vocabulary• Different Views of Quality• Quality Concepts and Practices• Quality Control and Quality Assurance• Quality Pioneers Approach to Quality
SUPPORTING / EXPLANATORY MATERIALSUnderstand the technical terms used to describe various testing techniques, tools, principles, concepts, and activities
Quality Vocabulary (cont.)KEY CONCEPT PRESENTATION GUIDEThe Quality Assurance function is charged with implementing the quality policy established by executive management. QAI recommends that the QA director position be a strong leadership position, and emphasizes the strong interpersonal skills necessary to successfully execute activities involved in making improvement occur.
SUPPORTING / EXPLANATORY MATERIALS• Activities that modify the development process to prevent the introduction of flaws
Quality Vocabulary (cont.)KEY CONCEPT PRESENTATION GUIDEThe Quality Assurance function is charged with implementing the quality policy established by executive management. QAI recommends that the QA director position be a strong leadership position, and emphasizes the strong interpersonal skills necessary to successfully execute activities involved in making improvement occur.
SUPPORTING / EXPLANATORY MATERIALS• Activities within the development process to detect the introduction of flaws
- Test planning and execution- Quality control measures a product against the existence of an attribute - Determines whether the product conforms to a standard or procedure (also known as
compliance checking).• Proactive approach focused on defect detection• Examples:
- Writing and executing test cases and scripts- Participating in verification and validation activities- Reporting defects to identify opportunities for process improvement
Quality Vocabulary (cont.)KEY CONCEPT PRESENTATION GUIDEMeeting requirements is a producer’s view of quality. This is the view of the organization responsible for the project and processes, and the products and services acquired, developed, and maintained by those processes. Meeting requirements means that the person building the product does so in accordance with the requirements.
SUPPORTING / EXPLANATORY MATERIALSProducer’s view of Quality:Per Philip Crosby:Conformance to requirements• Doing the right thing• Doing it the right way• Doing it right the first time• Doing it on time without exceeding cost
Quality Vocabulary (cont.)KEY CONCEPT PRESENTATION GUIDEBeing fit for use is the customer’s definition. The customer is the end user of the products or services. Fit for use means that the product or service meets the customer’s needs regardless of the product requirements.
SUPPORTING / EXPLANATORY MATERIALSCustomer’s view of Quality: From Joseph Juran and W. Edwards Deming
Fit for use• Receiving the right product for use• Being satisfied that needs have been met• Expectations have been met• Treated with integrity, courtesy, and respect
SUPPORTING / EXPLANATORY MATERIALSProvider’s view of Quality – This is the perspective of the organization that delivers the products and services to the customer.
Quality Vocabulary (cont.)KEY CONCEPT PRESENTATION GUIDEThis is the perspective of the organization (that may be external to the producer’s company, such as an independent vendor) that provides either the producer and/or the provider with products and services needed to meet the requirements of the customer.
SUPPORTING / EXPLANATORY MATERIALSThe perspective of the organization (which may be external to the producer’s company, such as an independent vendor) that provides either the producer and/or the provider with products and services needed to meet the requirements of the customer.
Infrastructure for Software Quality Products and ServicesKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThe figure shows the requirements coming from the customer to the producer/provider, who uses them to create the products and services needed by the customer. This process works because of the two-way measurement process established between the involved parties.
The Two Quality GapsKEY CONCEPT PRESENTATION GUIDEClosing these two gaps is the responsibility of the quality function
SUPPORTING / EXPLANATORY MATERIALSThe producer’s gap is the difference between what is specified (the documented requirements and internal standards) versus what is delivered (what is actually built). The customer’s gap is the difference between what the producers actually delivered versus what the customer wanted.
Quality Attributes for Information SystemsKEY CONCEPT PRESENTATION GUIDEQuality is a multifaceted concept driven by customer requirements
SUPPORTING / EXPLANATORY MATERIALSManagement needs to develop quantitative, measurable “standards” for each of these quality criteria for their development projects.
NOTES:
Attributes Definition
Correctness Extent to which a program satisfies its specifications and fulfills the user’s mission objectives.
Reliability Extent to which a program can be expected to perform its intended function with required precision.
Efficiency The amount of computing resources and code required by a program to perform a function.
Integrity Extent to which access to software or data by unauthorized persons can be controlled.
Usability Effort required learning, operating, preparing input, and interpreting output of a program.
Maintainability Effort required locating and fixing an error in an operational program.
Testability Effort required testing a program to ensure that it performs its intended function.
Flexibility Effort required modifying an operational program.
Reusability Extent to which a program can be used in other applications – related to the packaging and scope of the functions that programs perform.
Interoperability Effort required to couple one system with another.
Understanding the Quality ChallengeKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSAQL - Many think that defect-free products and services are not practical or
economical, and thus believe some level of defects is normal and acceptable.
Cost - Quality is frequently associated with cost, meaning that high quality is synonymous with high cost.
Detail - Quality by definition calls for requirements/specifications in enough detail so that the products produced can be quantitatively measured against those specifications.
Standards - Many technical personnel believe that standards inhibit their creativity, and thus do not strive for compliance to standards.
SUPPORTING / EXPLANATORY MATERIALSThe extent to which a program satisfies its specifications and fulfills the user’s mission objectives.a. Reliabilityb. Maintainabilityc. Correctnessd. Testabilitye. All of the AboveThe effort required to ensure the program performs its intended function.a. Reliabilityb. Maintainabilityc. Correctnessd. Testabilitye. All of the Above
SUPPORTING / EXPLANATORY MATERIALSQuality factors have frequently been used as a basis for measuring the quality of an information system. Listed below are two of those quality factors.Define each of the quality factors and provide an effective measure or metric for each of these quality factors? Ease of Use:• Effort required to learn, operate, prepare input and interpret output of the program/system. • Customer surveys targeted to a specific customer control group.• How many clicks it takes the user to find the objectMaintainability:• The effort required to locate and fix errors in the program/system. • The recording of how long it takes to make a change to code that has already been
released. • The recording of how long it takes to retest a change.
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDEPlan (P): Devise a plan
SUPPORTING / EXPLANATORY MATERIALSDefine the objective, expressing it numerically, if possible.Clearly describe the goals and policies needed to attain the objective at this stage. Determine the procedures and conditions for the means and methods that will be used to achieve the objective. Plan (P): Devise a plan• Objectives are quantitatively defined• Policies are defined• Practices/processes used to achieve the goals• Objectives are Identified
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDEDo (D): Execute the plan
SUPPORTING / EXPLANATORY MATERIALSExecute the plan - Create the conditions and perform the necessary teaching and training to ensure everyone understands the objectives and the plan. Teach workers the procedures and skills they need to fulfill the plan and thoroughly understand the job. Then perform the work according to these procedures.Do (D): Execute the plan• Create the practices (define them)• Provide training• Perform the work according to the defined practice
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDECheck (C): Check the results
SUPPORTING / EXPLANATORY MATERIALSCheck the results – As often as possible, check to determine whether work is progressing according to the plan and whether the expected results are obtained. Check for performance of the procedures, changes in conditions, or abnormalities that may appear.Check (C): Check the Results• Determine if work is progressing to plan.• Will anticipated results be realized? • Are goals and objectives satisfied?
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDEAct (A): Take the necessary action
SUPPORTING / EXPLANATORY MATERIALSTake the necessary action - If the check reveals that the work is not being performed according to plan, or if results are not what were anticipated, devise measures for appropriate action. Look for the cause of the abnormality to prevent its recurrence. Sometimes workers may need to be retrained and procedures revised. The next plan should reflect these changes and define them in more detail. Act (A): Take the Necessary Action• Devise measures and appropriate actions if work is not progressing to plan.• Look for the cause of abnormalities.• Update the plan.• Update training materials if needed.
The cost-of-quality components are the accounts for which cost information will be collected. In companies, the cost of quality is currently buried in the cost of sales. Within the information services organization, the cost of quality is buried in the cost of operations. This step involves the following two tasks:1. Determine the cost components that management wants to control. This will identify the
cost-of-quality components.2. Determine the cost for these components is collectible. The nature of the business, or the
methodologies used, may prohibit certain types of costs from being collected, or collected with enough consistency and reliability to be of value.
The Cost of Quality (cont.)KEY CONCEPT PRESENTATION GUIDEQuality is an attribute of a product or service. Productivity is an attribute of a process.
SUPPORTING / EXPLANATORY MATERIALSThe cost of quality (COQ) is the money spent beyond what it would cost to build a product right the first time. If every worker could produce defect-free products the first time, the COQ would be zero. Since this situation does not occur, there are costs associated with getting a defect-free product produced.The total production costs associated with the delivery of software applications include both the actual production costs and the cost of quality. The first component - production costs - consists of the costs associated with producing the product “right the first time,” or RTF costs. These costs include labor, materials, and equipment (hardware, software, tools) associated with the actual development of the application. The second component includes the additional costs associated with assuring that the product delivered meets the quality goals established for the product. This cost component is called the Cost of Quality, and includes all costs associated with the prevention, identification, and correction of product defects (includes repair and damage costs.)The cost of quality will vary from one organization to the next. The majority of costs associated with the Cost of Quality are associated with the identification and correction of defects.
The Cost of Quality (cont.)KEY CONCEPT PRESENTATION GUIDEThree cost of quality categories:1. Prevention2. Appraisal3. Failure
SUPPORTING / EXPLANATORY MATERIALSThere are three COQ categories:1. Prevention - Money required preventing errors and to do the job right the first time is
considered prevention cost. This category includes money spent on establishing methods and procedures, training workers and planning for quality. Prevention money is all spent before the product is actually built.
2. Appraisal – Appraisal costs cover money spent to review completed products against requirements. Appraisal includes the cost of inspections, testing and reviews. This money is spent after the product or subcomponents are built but before it is shipped to the user.
3. Failure – Failure costs are all costs associated with defective products. Some failure costs involve repairing products to make them meet requirements. Others are costs generated by failures, such as the cost of operating faulty products, damage incurred by using them and the costs incurred because the product is not available. The user or customer of the organization may also experience failure costs.
The Cost of Quality (cont.)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThe information services staff must be as committed to the cost of quality as is management. One of the quality principles is the elimination of fear. This is generally a prerequisite to an effective cost-of-quality program. If individuals fear management reprisals, they will be particularly reluctant to accurately record failure costs. Since this is an extremely important part of the cost-of-quality program, it must be recorded accurately and completely.Studies show that the Cost of Quality is approximately 50% of the cost to build a product or service:
SUPPORTING / EXPLANATORY MATERIALSWhich of the following is NOT a category of the Cost of Quality?a. Failure Costb. Appraisal Costc. Testing Costd. Preventive Cost
In defining the cost of quality, appraisal costs are BEST described as:a. Costs incurred to review completed products against requirementsb. Costs which can not be recoupedc. All costs associated with defective productsd. None of the above
SUPPORTING / EXPLANATORY MATERIALSA “best practice” is: • A most effective methodology for performing a specific process. • Normally identified by benchmarking or by an independent assessment. • Also identified through winners of quality competitions such as the MBNQA, Deming Prize,
Six SigmaKEY CONCEPT PRESENTATION GUIDEThe key focus of a Six Sigma program is to develop a good business strategy that balances the cost, quality, features and availability considerations for products.
SUPPORTING / EXPLANATORY MATERIALSWhen considering a project to improve using Six Sigma, the following characteristics are desirable. If one or more of these characteristics is missing, there will likely be barriers to success.• The project should clearly connect to business priorities.• The problem being solved should be very important, such as a 50% process improvement.• The importance of the project should be clear to the organization.• The project should have a limited scope that can be completed in less than six months.• The project should have clear, quantitative measures that define success.Management should support and approve the project to ensure resources are available, barriers are removed, and that the project continues over time.
What is a Defective Process?KEY CONCEPT PRESENTATION GUIDEA defective process is a process that will produce that will produce defects because the process is flawed.
SUPPORTING / EXPLANATORY MATERIALSFor example, in cutting wood, you cannot get a clean cut from a saw that has broken teeth. Likewise, if a programmer is not properly trained in how to save information on a PC under certain circumstances, that information will be lost, and the programmer will have to repeat the steps to recreate the information.Sigma is a statistical term dealing with standard deviation. What is states is, that at certain Sigma levels, in other words, the amount of correct processing under the curve, results in a certain percentage of correct products and a certain percentage of incorrect products.Motorola began using the Sigma system because their workers were conditioned in academia to believe that 99% was close to perfections. However, if only 99 out of 100 airplanes that took off landed correctly, in other words, one crashed, that would not be viewed as a near perfect process. Note that four Sigma is typical of most job shop processes such as physicians writing prescriptions, wait staff creating bills for restaurant customers, and IT professionals building software.A defective process is a process that will produce that will produce defects because the process is flawed. For example, in cutting wood, you cannot get a clean cut from a saw that has broken teeth. Likewise, if a programmer is not properly trained in how to save information on a PC under certain circumstances, that information will be lost, and the programmer will have to repeat the steps to recreate the information.
Baselining and BenchmarkingKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS• Baseline – Current level of performance• Benchmark - A comparison of one organization’s process to another organization’s process
or a comparison of one organization’s process to an industry model or standard
SUPPORTING / EXPLANATORY MATERIALSThe concept of Six Sigma measures defects per million parts. The number of defects for four Sigma is approximately:a. 3.4b. 233c. 6200d. 66,803The quality concept that focuses on defect rates, as opposed to percent performed correctly is called:a. TQMb. Six Sigmac. Benchmarkingd. Quality Control
SUPPORTING / EXPLANATORY MATERIALSVery few individuals can differentiate between quality control and quality assurance. Most quality assurance groups, in fact, practice quality control. You will learn the critical difference between control and assurance and how to recognize a control practice from an assurance practice.QA is activities that:• Assists in the development of processes• Fosters the concept of continuous process improvement through defect prevention• Is a staff function
Quality Control (QC) (Product Focus)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSQuality Control (QC) is defined as the processes and methods used to compare product quality to requirements and applicable standards, and the action taken when a nonconformance is detected. QC uses reviews and testing to focus on the detection and correction of defects before shipment of products. QC is the activity that:• Compares products to requirements• Compares work products to applicable standards• Is the responsibility of the organization and individuals who built the product or service• Is a line functionQC is the responsibility of the organization or individuals who builds the product or service
It is the responsibility of every project team member, not just the test or QA team, responsibility to assure the quality of the products. Developers, managers, and support teams all have a role in assuring quality.Participation in verification techniques, such as walkthroughs and inspections, is critical to the delivery of quality applications. These activities can be supported, but not replaced, by automation.For example, automation can be used to verify programming syntax but can not assure the developer is following the design strategy outlined for the project. Subjective criteria requires expert opinion which only humans can provide.• Techniques for quantifiably assuring that a work product meets its stated objectives:
- Verification: Performed during development on key artifacts• Walkthroughs, reviews, and inspections• Mentor feedback, training, checklists, and standards
- Validation: Performed after a work product is produced• Validating against established criteria • Ensuring product integrates correctly into the environment
• Reactive approach focused on defect detection and removal
Differentiating Between QA and QCKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSQC is an activity that verifies whether or not the product produced meets standards. QA is an activity that establishes and evaluates the processes that produce the products. If there is no process, there is no role for QA The following statements help differentiate QC from QA:• QC relates to a specific product or service.• QC verifies whether particular attributes exist, or do not exist, in a specific product or
service.• QC identifies defects for the primary purpose of correcting defects.• QC is the responsibility of the worker.• QA helps establish processes.• QA sets up measurement programs to evaluate processes.• QA identifies weaknesses in processes and improves them.• QA is a management responsibility, frequently performed by a staff function.• QA evaluates whether or not quality control is working for the primary purpose of
determining whether or not there is a weakness in the process.• QA is concerned with all of the products that will ever be produced by a process.• QA is sometimes called quality control over quality control because it evaluates whether
quality control is working.• QA personnel should never perform quality control unless doing it to validate quality control
• Developed by Toyota• Ultimate goal is to supply each process with exactly the required items• Must have large amounts of production flexibility and very short lead times• The key is the ability of the production area to quickly switch from job to job
• JIT - The ultimate goal of JIT production is to supply each process with exactly the required items, in exactly the required quantity, at exactly the required time. There are two conditions necessary to reach this situation: large amounts of production flexibility, and very short lead times.
• TQM - Total quality management (TQM) is the term used by many to indicate an organization-wide effort of continuous process improvement. The Federal Quality Institute defines TQM as a strategic, integrated management system for achieving customer satisfaction, which involves all managers and employees and uses quantitative methods to continuously improve an organization's processes.
• Each process gets supplied the required items• Each process gets the exactly required quantity• Each process gets it delivered at exactly the required time• Requires a high level of quality
SUPPORTING / EXPLANATORY MATERIALSThe responsibilities of this job include facilitation, process configuration, measurement, and risk analysis.a. Quality Controlb. Quality AssuranceThe responsibilities of this job include conducting inspections, reviews, testing, and focusing on the product.a. Quality Controlb. Quality Assurance
SUPPORTING / EXPLANATORY MATERIALSThe primary purpose of quality control is to:a. Prevent defects from occurringb. Assign quality responsibilitiesc. Uncover defectsd. Conduct testingThe most effective test approach is to begin testing when:a. A new project beginsb. After requirementsc. After external designd. After coding
SUPPORTING / EXPLANATORY MATERIALSAnswers:QC - Inspection of source code.QC - Verify the jobs you submitted to be run that day have been run.QA - Select and implement a development methodology.QC - Unit testing to validate that the program works.QA - Analysis of defects to determine the stage of origin.QA - Metrics collected to measure the effectiveness of system and unit testing
SUPPORTING / EXPLANATORY MATERIALSDr. Deming defined 14 principles for quality, which formed the basis for the turnaround of the Japanese manufacturing industry. He believed that all 14 principles must be used concurrently to make quality happen. Philip Crosby has developed 14 steps for an organization to follow in building an effective quality program. Dr. Juran believed that managing for quality required the same attention that other functions typically receive. To ensure that adequate attention was given, he developed a trilogy consisting of three interrelated, basic managerial phases/processes: quality planning, quality control and quality improvement. These are known as “The Juran Trilogy” or “The Quality Trilogy.”
Dr. W. Edwards Deming’s Quality Principles:1. Create a constancy of purpose2. Adopt a new philosophy3. Cease dependence on mass inspection4. End the practice of awarding business on the basis of price tag alone
Dr. W. Edwards Deming’s Quality Principles:5. Improve constantly & forever the system of production & service6. Institute training7. Adopt and institute leadership8. Drive out fear9. Break down barriers between staff areas
Dr. W. Edwards Deming’s Quality Principles:10. Eliminate slogans, exhortations, and targets for the work force11. Eliminate numerical quotas for the work force12. Remove barriers that rob people of pride of workmanship13. Encourage education and self-improvement14. Take action to accomplish the transformation
Philip Crosby has developed 14 steps for an organization to follow in building an effective quality program. 1. Management commitment2. The quality improvement team3. Quality measurement’4. The cost of quality
SUPPORTING / EXPLANATORY MATERIALSPhilip Crosby has developed 14 steps for an organization to follow in building an effective quality program. 5. Quality awareness6. Corrective action7. Zero Defect planning8. Supervisor training9. ZD (Zero Defect) Day
SUPPORTING / EXPLANATORY MATERIALSPhilip Crosby has developed 14 steps for an organization to follow in building an effective quality program. 10. Goal setting11. Error-cause removal12. Recognition13. Quality councils14. Do it over again
SUPPORTING / EXPLANATORY MATERIALSDr. Juran believed that managing for quality required the same attention that other functions typically receive. To ensure that adequate attention was given, he developed a trilogy consisting of three interrelated, basic managerial phases/processes: quality planning, quality control and quality improvement. These are known as “The Juran Trilogy” or “The Quality Trilogy.”Managing for quality requires the same attention as other projectsDeveloped a quality trilogy of basic managerial phases:• Quality Planning• Quality Control• Quality improvement
SUPPORTING / EXPLANATORY MATERIALSTQM - Total quality management (TQM) is the term used by many to indicate an
organization-wide effort of continuous process improvement. The Federal Quality Institute defines TQM as a strategic, integrated management system for achieving customer satisfaction, which involves all managers and employees and uses quantitative methods to continuously improve an organization's processes.
JIT - The ultimate goal of JIT production is to supply each process with exactly the required items, in exactly the required quantity, at exactly the required time. There are two conditions necessary to reach this situation: large amounts of production flexibility, and very short lead times.
A managerial philosophy based on the work of the Pioneers:• Cross functional goals• A process of controlled change• Requires a team effort• Communications must be encouraged throughout the organization• Implementation is not a one-size fits all
SUPPORTING / EXPLANATORY MATERIALSWhich of the following is NOT one of Dr. W. Edward Deming’s fourteen quality principles?a. Improve supervisionb. Drive out fearc. Learn the new philosophyd. Zero defectsTotal Quality Management means:a. Managing the quality of entire systemb. Emphasizing the importance of managing qualityc. Implementing company-wide quality attitudes and methodologiesd. Having the Quality Manager included in senior management
SUPPORTING / EXPLANATORY MATERIALSList 3 items that you consider important to having quality control in systems development. Explain each answer:• To reduce risks inherent in computer systems. • To detect variation from specification/expectation. • Establishing confidence that a program does what it is suppose to do. • Prove that the program is no good. • Detecting specification errors and deviations from specifications.
SUPPORTING / EXPLANATORY MATERIALSThe most important prerequisite for successful implementation of any major quality initiative is commitment from executive management. It is management’s responsibility to establish strategic objectives and build an infrastructure that is strategically aligned to those objectives. This category describes the management processes used to establish the foundation of a quality-managed environmentQuality Leadership• Leadership Concepts• Quality Management Infrastructure• Quality Environment
What Commitment MeansKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSManagement commitment is the single most important requirement for successful implementation of quality management. There is no precedent of successful quality improvement without executive management and the management team leading the effort. Having management commitment does not guarantee quality management success; it only improves the odds for successful implementation. The entire organization must eventually become committed to quality management. • Understand the concepts of quality management• Adopt behaviors required to show commitment• Accept the need to change to a participative management style• Lead in the development of a quality management implementation plan• Lead the formation of the implementation organization• Provide funds for training• Provide time for training and meetings• Publicize and reward results• Monitor and measure progress• Provide personnel and other resources
New Behaviors for ManagementKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSMost managers practice traditional management. They have been taught to control their organization and employees, using an “I’ll tell you what to do, and you’ll do it” mentality. Many managers look at the short-term because their commitment to the organization is short range. The key differences in philosophy between traditional management and quality management environments are illustrated in the table below.
Traditional Management Philosophy• Controls each result• Who made the error?• Correct the error• Employees are the problem• Management accountable to their
manager• Competition between organizations• Motivation from fear of failure• Management of outputs (results) -
focusing on detection of defects• Fire fighting• Accomplishment from meeting quotas,
the monthly or quarterly bottom line
Quality Management Philosophy• Use the process• What allowed the error?• Reduce variation and prevent the error• Refine the process• Management accountable to the customer• Teamwork• Motivation from within (self)• Management of process inputs - methods or
sources of variation that focus on preventing defects
• Continuous process improvement• Accomplishment from long-term impact of
New Behaviors for Management (cont.)KEY CONCEPT PRESENTATION GUIDEThe programs needed to change from a traditional to quality management culture must be customized for an organization and its current culture.
SUPPORTING / EXPLANATORY MATERIALS
NOTES:
Category Traditional Culture Quality Management Culture Mission Maximum return on investment
(ROI), management by objec-tives (MBO)
Ethical behavior and customer satisfaction, climate for continuous improvement, ROI as a measure of performance
Customer Requirements
Incomplete or ambiguous understanding of customer requirements
Uses a systematic approach to seek out, understand, and satisfy both internal and external customer requirements
Suppliers Undirected relationship Partnership
Objectives Orientation to short-term objec-tives and actions with limited long-term perspective
Deliberate balance of long-term goals with successive short-term objectives
Improvement Acceptance of process variability and subsequent corrective action as the norm
Understanding and continually improving the process
Problem-Solving
Unstructured individualistic problem-solving and decision-making
Predominantly participative and interdisciplin-ary problem-solving and decision-making based on substantive data
Jobs and People
Functional, narrow scope, management controlled
Management and employee involvement, work teams, integrated functions
Management Style
Management style with uncertain objectives that instills fear of failure
Open style with clear and consistent objectives, encouraging group-derived continuous improvement
Role of Manager
Plan, organize, assign, control and enforce
Communicate, consult, delegate, coach, mentor, remove barriers, and establish trust
Rewards & Recognition
Pay by job, few team incentives Individual and group recognition and rewards, negotiated criteria
Measurement Orientation toward data gathering for problem identification
Data used to understand and continuously improve processes
SUPPORTING / EXPLANATORY MATERIALSManagement commitment is the single most important requirement for successful implementation of quality management. There is no precedent of successful quality improvement without executive management and the management team leading the effort. Having management commitment does not guarantee quality management success; it only improves the odds for successful implementation. The entire organization must eventually become committed to quality management. • Manager – Works within the system by following accepted practices• Leader – Determines where the organization needs to be; then does what is necessary to
LeadershipKEY CONCEPT PRESENTATION GUIDELeadership and management are two different things. While a manager works within the system following the accepted practices of the system, a leader determines where the organization needs to be, and then does what is necessary to get there.
SUPPORTING / EXPLANATORY MATERIALSIn a business context, leadership is the ability to build the commitment of employees, to endow an organization with a positive perception of itself, and to give employees a positive perception of their role within the business.
NOTES:
Characteristic Behavior Demonstrated Substance Helps others achieve needed substance. Growth Helps others achieve personal and career growth. Opportunities Creates opportunities for others to make uninhibited
contributions to the enterprise. Environment Creates an environment conducive to performance. Empowerment Empowers others. Obstacles Removes obstacles to performance. Support Helps others do what they decide is in their own best interest. Coaching, training, education
Coaches, trains and educates others.
Coordination Helps coordinate the work of others. Market, Outlets Creates a market and outlet for the talents of others. Resources for others Acquires the resources others need. Uniquely Equipped Does what is necessary for success, which others are not
capable of doing. Strategies Creates a vision, communication and trust through positioning
and deployment of self. Persistent Pursues tirelessly the mission of the organization through
linkage with other leaders on strategic issues. Ethical, open, honest Maintains a totally open and honest state with others.
Fundamental MistakesKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Isolation – Lack of maintaining regular contact with people they manage• Inability to reward – Not taking the time to reward the people they manage• Lack of business perspective – Inability to take advantage of opportunities
Prepare for Awareness Training• Select awareness topic• Identify the topics customers• Define objective for awareness training• Define customer benefits• Develop administrative training plan
SUPPORTING / EXPLANATORY MATERIALSThe Bottom-up Approach - This approach sends the message that quality management is something for the employees, but not necessarily for management. This approach is like swimming against the current, and leads to frustration because resources are not readily provided to teams when required. The Middle-out Approach - Starting in the middle of the organization and then progressing simultaneously to the top and bottom of the organization can be successful.The Top-down Approach - Top-down has the highest probability for success, although success is not guaranteed. This model fosters management involvement - the single most important requirement for quality management success.
SUPPORTING / EXPLANATORY MATERIALSEach level is linked to the other by the common objective of making people capable of combined performance. The figure above shows the three levels of infrastructure normally needed.
SUPPORTING / EXPLANATORY MATERIALSA Quality Council is composed of the organization’s top executive and his or her direct reports.It may also be referred to as an Executive Council. The Quality Council acts as the steering group to develop the organization’s mission, vision, goals, values, and quality policy. Quality Council:• Initiates and commits to quality management• Incorporates quality management into strategic planning• Allocates resources: budget, people, time• Establishes lower level committees• Defines and deploys policies• Process approval authority• Acts on unresolved items• Provides review and oversight of progress
SUPPORTING / EXPLANATORY MATERIALSManagement committees (also called Process Management Committees) are composed of middle managers and/or key staff personnel, and are responsible for deploying quality management practices throughout the organization. One or more committees may be needed depending on the organization’s size and functional diversity. Committees should represent all the skills and functions needed to work on the specific processes or activities.Management Committees:• Gains an understanding of the organization’s mission, goals, and priorities• Commission the development of a process inventory, and process maps• Establish work groups• Monitor progress• Review and approve processes• Quality planning
SUPPORTING / EXPLANATORY MATERIALSTeams are formed under any number of names depending on their purpose. Common names and functions are:• Process Development Teams: They develop processes, standards, etc.• Process Improvement Teams: They improve existing processes, standards, etc.• Work Groups: They perform specific tasks such as JAD, inspections, or testing
- Subject matter experts- Study, define, and improve processes- Pilot newly defined processes- Provide process training- Assist in deployment- Serve as process consultants
Guidelines for TeamsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• The committee selects the teams• Each team has a chairperson• Keep is small (3-5 people)• Develop a work plan• Meet regularly to review the work• Different teams members participate• Team must reach consensus
Good ideas are of little value unless they are accepted and implemented. The QA report is designed to convey information and to change behavior. QA analysts write a report, distribute it, and follow up on the recommendations. The value of the quality function can be rated on whether management accepts the report. • Establish report objectives• Gather factual data and recommendations• Develop a report outline• Draft the report• Review the draft for reasonableness• Have the report reviewed for readability• Review the report with involved parties• Review the report with management• Distribute the report and follow up
The Six Attributes of an Effective Quality EnvironmentKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Five major accounting associations (i.e., Financial Executives International, American Institute of Public Accountants, American Accounting Association, The Institute of Internal Auditors, and the Institute of Management Accountants), which is referred to as COSO (Committee of Sponsoring Organizations), was organized to provide guidance on evaluating internal control. They issued this guidance as the COSO Internal Control Framework. The COSO Framework identified the six quality attributes. For each attribute, they listed several control objectives that if implemented would define each of the six attributes.1. Integrity and Ethical Values2. Commitment to Competence3. Management’s Philosophy and Operating Style4. Organizational Structure5. Assignment of Authority and Responsibility6. Human Resource Policies and PracticesImplement Mission, Vision, Goals, Values, and Quality Policy
Integrity and Ethical ValuesKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Key Questions:• Do you do what you say you will?• Do you treat everyone equally?• Are you honest with your suppliers, colleagues and customers?• Are a team member?• Do you work hard every day?
SUPPORTING / EXPLANATORY MATERIALSHearing the speaker requires an understanding of the five channels of communication incorporated into speech. Much of listening occurs beyond merely hearing the words. Attending to the speaker is sometimes referred to as being an active listener. Devote your full attention to the speaker to confirm that what you heard is what the speaker intended you to hear. When one has deciphered the information channel (i.e., what the subject is) and related the importance of that subject to the audience, listening must be adjusted to ensure that we get the message we need.• Provide Constructive Criticism• Achieving Effective Listening
- Hearing- Attending- Understanding
• Personal Persuasion• Resolving Customer Complaints• Written Reports
Oral communication (which includes listening) is rated as the number-one skill for the tester.Some facts about listening include:• Listening is the first language skill that we develop as children; however, it is rarely taught as
a skill. • Listening is the most frequently used form of communication.• Sales people often lose sales because the believe talking is more important than listening.
3-Step Listening Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Step 1. Hearing the speaker.Hearing the speaker requires an understanding of the five channels of communication incorporated into speech:• Information Channel - The speaker’s subject.• Verbal Channel - The words used by the speaker.• Vocal Channel - The tone of voice associated with the various words.• Body Channel - The body movements and gestures associated with the information being
conveyed.• Graphic Channel - The pictures, charts, etc. that the speaker uses to emphasize or illustrate
3-Step Listening Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Step 2. Attending to the Speaker (sometimes referred to as being an active listener).Some suggestions to help in attending to the speaker are:• Concentrate on the speaker.• Maintain eye contact (80%).• Provide feedback.• Restate what you heard.
3-Step Listening Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Step 3. Understanding the SpeakerThere are five types of listening. The type chosen will have an impact on the ability to understand what the speaker is saying.• Type 1: Discriminative Listening• Type 2: Comprehensive Listening• Type 3: Therapeutic Listening• Type 4: Critical Listening• Type 5: Appreciative or Enjoyment Listening
• Explains “WHY” a company, organization, or activity exists and what is done to accomplish it.
• The mission should focus on products and services, and it should be customer-oriented.• “Our mission is to improve continually our products and services to meet our customers’
• Establishes where the organization desires to move from its current state.• Senior management should establish the vision, ensuring how it contributes to the business
is clear.• “Our vision is to be a company with a strong customer base, known for reliability, trust and
What is an Effective Work EnvironmentKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
These characteristics of a quality work environment are the same items we emphasized in part 1 of this course on quality principles and concepts. The students might want to use this as a quick checklist to evaluate their quality work environment. If they got 3 yeses, meaning this is how their organization operates, they are well underway to a quality work environment.• Listening to customers to determine their requirements• Identifying costs of quality and focusing on prevention• Doing the right thing right the first time• Continuing process improvement• Taking ownership at all levels of the organization• Demonstrating executive leadership and commitment
What is Not an Effective Work EnvironmentKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
These items, which represent the lack of a quality work environment, are basically the negative statements regarding what a good work environment is. Again, a student could use this as a quick checklist, and if they answer yes to many or all of these items, they have a lot of work to do to improve their quality environment.• Assuming you know your customer’s requirements• Overlooking the hidden costs of poor quality• Doing it over to make it right• One-time fixes• Assigning responsibility for quality to one department (e.g., QA)• Assigning responsibility for product quality to workers
The Environment is Essential Because Quality:KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Emphasize that most of the principles of quality are just plain common sense. For example, your children will be much more likely to do good school work if you motivate them to do good work, describe the vision for them of the opportunities for a good student and reward them on good work, than if you attempt to get good school work from fear and punishment.Emphasize that success, meaning quality from the customer’s view, quality from the producer’s view and alignment to goals and objectives is not a technical problem. It cannot be solved with more hardware and software. It’s a people problem that can only occur through management initiatives. A culture is defined as the ideas, customs, skills, arts, etc. of a given people in a given period. The IT culture is defined by its vision, values, mission, approaches to work, rewards, punishments, excitement one experiences in the work environment, the way in which people view management and themselves as well as management's view of the way they want the IT organization to operate.Explain that if the students had an opportunity to go from like organization to like organization they would easily experience the difference in culture. In their personal lives, they probably experience a different culture in their home, their church, sporting events, clubs and so forth. Most people know the type of culture that they want to be in, but have difficulty defining that culture. We will attempt in the next few minutes to describe the different cultures.
Manage to Facilitate SuccessKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThis course will explain what is meant by vision, value and mission, give examples, and explain the importance. It also will provide guidelines on building a quality culture and a quality infrastructure.Managing to Facilitate Success means doing the following:• Defining the IT vision, value & mission• Building a quality infrastructure• Integrating quality planning into IT planning• Encouraging a quality culture by monitoring and enforcing organizational policies and
SUPPORTING / EXPLANATORY MATERIALSExplain the major difference between a leader and a manager?Leader:• Forms a Partnership• Seeks out, understands, and satisfies both internal and external customer requirements• Understands and continually improves the processManager:• Undirected relationship• Incomplete or ambiguous understanding of customer requirements• Acceptance of process variability and subsequent corrective action as the norm
Skill Category 3KEY CONCEPT PRESENTATION GUIDEOrganizations need to establish baselines of performance for quality, productivity and customer satisfaction.
SUPPORTING / EXPLANATORY MATERIALS
These baselines are used to document current performance and document improvements by showing changes from a baseline. In order to establish a baseline, a model and/or goal must be established for use in measuring against, to determine the baseline. • Quality Baseline Concepts• Methods used for Establishing Baselines• Model and Assessment Fundamentals• Industry Quality Models
SUPPORTING / EXPLANATORY MATERIALSIT organizations have established many different baselines to evaluate current performance and to measure improvement. To help understand the type of baselines that are used in IT, QAI has categorized four most commonly used baselines as listed below. Each one of these will be discussed individually.• Customer surveys• Benchmarking• Management established criteria• Industry models
• A baseline is a current level of performance.• An assessment determines the baseline against the model. (Goals to be accomplished)
Steps to Perform a Baseline StudyKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Identify products/services to be surveyed• Define conformance and nonconformance• Identify survey population• Identify size of population to be surveyed• Develop survey instrument• Conduct survey• Follow up on incomplete surveys• Accumulate and present survey results• Take action and notify participants of that action
Objective baseline studies are ones which are viewed as factual and non-argumentative.Examples of objective baseline measures include:• Project completed on schedule• Lines of code• Number of programs• Number of people assigned to a project• Number of abnormal terminations
Subjective baseline studies will be the most commonly conducted studies in measuring quality and productivity.Examples of Subjective baseline measures:• Customer satisfaction• Effectiveness of standards/manuals• Helpfulness of methodologies to solve problems• Areas/activities causing the greatest impediments to quality/productivity• Causes for missed schedules/over-budget conditions• Understandability of training materials• Value of tools• Importance of activities/standards/methods/tools to individual activity
Purpose of a ModelKEY CONCEPT PRESENTATION GUIDEA specific model may or may not fully apply to an IT organization, and selecting a model does not have to be an all or nothing decision.
SUPPORTING / EXPLANATORY MATERIALS
• A model is a goal to be accomplished• Most models define the minimum that has to be accomplished• Models are normally developed under the auspices of a national or international standards
organization• Organizations choose to use models to:
- Satisfy business goals and objectives- Satisfy requirements imposed by customers- Leapfrog the competition- Provide guidance (road map) for continuous improvement
Staged Versus Continuous ModelsKEY CONCEPT PRESENTATION GUIDEA specific model may or may not fully apply to an IT organization, and selecting a model does not have to be an all or nothing decision.
SUPPORTING / EXPLANATORY MATERIALSStaged models:• Composed of a number of distinct levels of maturity• Each level of maturity is decomposed into specific processes.• Each level of maturity serves as the foundation for the next level of maturity.Continuous models:• Processes are improved individually along a maturity scale that is independent of each
Model Selection ProcessKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThere are a number of improvement models:• Software Engineering Institute’s Capability Maturity Model (CMM) and the CMMI• Malcolm Baldrige• ISO 9001: 2000• ISO 12207• ISO 15504 (SPICE)An organization should consider the following when selecting a model:1. Applicability of the model to the organization’s goals and objectives2. Management commitment3. Need for baseline assessments4. Need for measurable goals and objectives
Using Models For Assessment and BaselinesKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSUsing Models For Assessment and Baselines• A baseline is a current level of performance.• An assessment determines the baseline against the model. (Goals to be accomplished)
SUPPORTING / EXPLANATORY MATERIALSAs shown in the above figure, the CMMI® framework is a method for organizing these steps into five levels of maturity that lay successive foundations to support short and long-term process improvement initiatives.The five maturity levels define an ordinal scale that enables an organization to determine its level of process capability. The framework is also an aid to quality planning as it affords organizations the opportunity to prioritize improvement efforts.Software Engineering Institute’s Capability Maturity Model levels include:
SEI CMMI - Level 1KEY CONCEPT PRESENTATION GUIDEWhen an organization lacks sound management practices, the benefits of good software engineering practices are undermined by reaction-driven commitments.
SUPPORTING / EXPLANATORY MATERIALSThe process capability at Level 1 is considered ad hoc because the software development process constantly changes as the work progresses. Schedules, budgets, functionality, and product quality are generally unpredictable.Level 1 characteristics include:• Unstable work environment• Organization lacks sound management practices• Code and fix methodology• Development processes constantly change• Cannot be repeated from project-to-project
SEI CMMI - Level 2KEY CONCEPT PRESENTATION GUIDEPolicies for managing a software project and procedures to implement those policies are established.
SUPPORTING / EXPLANATORY MATERIALSLevel 2 – Repeatable Key Process Areas• Requirements Management• Software Project Planning• Software Project Tracking and Oversight• Software Subcontract Management• Software Quality Assurance• Software Configuration ManagementLevel 2 characteristics include:• Basic management controls are installed• Planning and managing is based on experience• Realistic commitments• Budget, schedules, and functionality are tracked• Has the ability to repeat past successes
SEI CMMI - Level 3KEY CONCEPT PRESENTATION GUIDEThe standard engineering and management processes for developing and maintaining software across an organization are documented, and these processes are integrated as a whole
SUPPORTING / EXPLANATORY MATERIALSLevel 3 – Defined Key Process Areas• Organization Process Focus• Organization Process Definition• Training• Integrated Software Management• Software Product Engineering• Intergroup Coordination• Peer ReviewsLevel 3 characteristics include:• Organization uses a standard SDLC• Documented and integrated as a whole• A group is assigned ownership of the organization’s process activities• Organization-wide training program• Engineering and project management activities are integrated
SEI CMMI - Level 4KEY CONCEPT PRESENTATION GUIDEThe capability of Level 4 organizations is summarized as predictable because the process is measured and operates within measurable limits.
SUPPORTING / EXPLANATORY MATERIALSLevel 4 – Managed Key Process Areas• Software Quality Management• Quantitative Quality ManagementLevel 4 characteristics include:• Quantitative goals and objectives are set• Productivity and quality are measured• Process control is achieved by narrowing variability• Is predictable because processes are measured and operate within measurable limits
SEI CMMI - Level 5KEY CONCEPT PRESENTATION GUIDEThe organization has the means to identify weaknesses and strengthen the process proactively, with the goal of preventing the occurrence of defects.
SUPPORTING / EXPLANATORY MATERIALSLevel 5 – Optimizing Key Process Areas• Defect Prevention• Technology Change Management• Process Change ManagementLevel 5 characteristics include:• Entire organizational focus is on continuous process improvement• Root cause analysis• Lessons learned
Malcolm Baldrige National Quality Award KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThe United States national quality award originated from the Malcolm Baldrige National Quality Improvement Act (Public Law 100-107), signed by President Ronald Reagan on August 20, 1987. That act, named after a former Secretary of Commerce, called for the creation of a national quality award and the development of guidelines and criteria that organizations could use to evaluate their quality improvement efforts. • The United State’s national quality award• Enacted through Public Law 100-107• Named for the then Secretary of Commerce• Awards are given annually• Awards are given in five categories:
- Manufacturing- Service Organizations- Small Business- Education Organizations- Health Care Organizations
National Institute of Standards and TechnologyKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThe U.S. Department of Commerce is responsible for the MBNQA and Program. The National Institute of Standards (NIST), an agency of the Department’s Technology Administration manages the Baldrige Program.• Responsible for the MBNQA • Agency of the Department of Technology Administration• Manages the Baldrige Program by governing:
- A board of overseers- A board of examiners- The award recipients
SUPPORTING / EXPLANATORY MATERIALSAwards Criteria:1. Leadership2. Strategic Planning3. Customer and Market Focus4. Information and Analysis5. Human Resources6. Process Management7. Business Results
SUPPORTING / EXPLANATORY MATERIALSProcess refers to the methods your organization uses and improves to address the Item requirements in Categories 1-6. The four factors used to evaluate process are Approach, Deployment, Learning, and Integration (A-D-L-I).“Approach” refers to:• The methods used to accomplish the process• The appropriateness of the methods to the Item requirements• The effectiveness of your use of the methods• The degree to which the approach is repeatable and based on reliable data and information
(i.e., systematic)Applicant responses are scored on two evaluation dimensions:• Process
- Approach- Deployment- Learning and Integration
• Results - Current Level of Performance- Rate and Breadth- Benchmarks- Result measures
ISO 9001: 2000KEY CONCEPT PRESENTATION GUIDEThe ISO 9000 family of standards contains 3 standards and many supporting documents.
SUPPORTING / EXPLANATORY MATERIALSAn International quality model developed under the auspices of the International Organization for Standardization. (ISO)The Four primary standards are:• ISO 9000 – Introduction and guide• ISO 9001 – The quality model• ISO 9004 – Guide to help with installation• ISO 10011- Guidelines for auditing
ISO 9001: 2000 Model OverviewKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThe standard is based upon eight quality principles:1. Customer focus2. Leadership3. Involvement of people4. Process approach5. System approach to management6. Continuous improvement7. Factual approach to decision making8. Mutually beneficial supplier relationships
Model Overview - IS0/IEC 12207KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSISO/IEC 12207, which was published in 1995, is the international standard that covers the software life cycle from concept through retirement. It contains a framework for managing, controlling, and improving the software life cycle activities. For each process, the standard also describes the activities and tasks involved, defining specific responsibilities and identifying outputs of activities and tasks. Since it is a high-level standard, it does not detail how to perform the activities and tasks. • An international standard• Covers the software life cycle from concept through retirement• Describes the major life cycle phases:
- Activities and tasks involved- How phases interface with one another
Model Overview - IS0/IEC 15504KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSSoftware Process Improvement Capability DeterminationAn international standard:• Provides the framework for the assessment of processes• It produces a process profile that represents the maturity of the process rather than a pass/
SUPPORTING / EXPLANATORY MATERIALSThe Malcolm Baldrige National Quality Award was created by Public Law 100-107 and is named for Malcolm Baldrige, who served as the Secretary of Defense.a. Trueb. FalseObjective measures are those, which can be accomplished by counting. Which of the following is NOT an objective baseline measurement that can be used to establish baselines?a. Project completed on scheduleb. Number of programsc. Guessing at estimationsd. Number of abnormal terminations
SUPPORTING / EXPLANATORY MATERIALSBriefly explain a major similarity and major difference between the SEI’s CMM model and the ISO’s SPICE model.Similarity:• Maturity LevelsDifferences:• CMM is audited by external company• SPICE is a self-assessment
Skill Category 4KEY CONCEPT PRESENTATION GUIDEQuality Assurance is a professional competency whose focus is directed at critical processes used to build products and services.
SUPPORTING / EXPLANATORY MATERIALSThe profession is charged with the responsibility for tactical process improvement initiatives that are strategically aligned to the goals of the organization. This category describes the management processes used to establish the foundation of a quality-managed environment:Quality Assurance• Establishing a Function to Promote and Manage Quality• Quality Tools• Process Deployment• Internal Auditing and Quality Assurance
Establishing a Function to Promote and Manage QualityKEY CONCEPT PRESENTATION GUIDEA quality function exists when a specific individual/group is assigned the responsibility to assist in improving quality.
SUPPORTING / EXPLANATORY MATERIALSWhile individual workers have responsibility for the quality of their products and services, it is management's responsibility to ensure that the environment is one in which quality can flourish. The ultimate responsibility for quality rests with senior management.Many people argue that because everyone has some quality responsibility, a staff function for quality is unnecessary. That argument is theoretically correct, but in practice unless there is a group charged with responsibility for ensuring quality, the pressures of other priorities such as meeting schedules and budgets frequently takes precedence over quality. A quality function exists when a specific individual/group is assigned the responsibility to assist in improving quality.
Four Variables in a System Development Project KEY CONCEPT PRESENTATION GUIDEThe four project variables are:1. Scope2. Schedule3. Resources4. Quality.
SUPPORTING / EXPLANATORY MATERIALS
The management challenge in completing the project can be illustrated as a dashboard of four system attribute dials, which get set according to the project criteria as illustrated in the figure above. The four dials are interconnected, so movement of one dial affects one or more of the other dials.
The Challenges of Implementing a Quality FunctionKEY CONCEPT PRESENTATION GUIDETwo basic challenges emerge, and either or both may be present in any particular organization.
SUPPORTING / EXPLANATORY MATERIALSIf an organization is not ready, and also lacks the sales skills to convince the management group, the probability of success is so low that it is best not to bother with implementation at this time. A ready organization with good salesmanship can take care of itself. The other scenarios present different challenges. • Organizations that are not ready, but have the salesmanship, are referred to as “Challenge
1.” • Organizations that are ready, but lack the salesmanship, are referred to as “Challenge 2.”
How the Quality Function Matures Over TimeKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSIT management involved in a corporate quality management program should undertake the following four actions in an effort to rethink and assign their quality responsibilities:1. Determine what's new, changed, and continued quality responsibilities will be assigned to
the IT group as a result of the corporate quality program.2. Evaluate the capability of the IT group to deploy quality initiatives within and outside the
group. Define the skills, commitment, and quality approaches currently available to line management.
3. Determine the value added by utilizing IT QA analysts to support corporate quality management initiatives.
4. Develop a plan to assimilate and leverage the current function into the quality support activity needed to support both the corporate quality management program and IT quality responsibilities.
• Develop a charter• Identify the quality manager• Locate organizationally the quality function• Build support for quality• Staff and train the quality function• Build and deploy the quality tool box• Drive the implementation
Implementing an IT Quality FunctionKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSIT management involved in a corporate quality management program should undertake the following four actions in an effort to rethink and assign their quality responsibilities:• Determine what's new, changed, and continued quality responsibilities will be assigned to
the IT group as a result of the corporate quality program.• Evaluate the capability of the IT group to deploy quality initiatives within and outside the
group. Define the skills, commitment, and quality approaches currently available to line management.
• Determine the value added by utilizing IT QA analysts to support corporate quality management initiatives.
• Develop a plan to assimilate and leverage the current function into the quality support activity needed to support both the corporate quality management program and IT quality responsibilities.
1. Develop a charter2. Identify the quality manager3. Locate organizationally the quality function4. Build support for quality5. Staff and train the quality function6. Build and deploy the quality tool box7. Drive the implementation
SUPPORTING / EXPLANATORY MATERIALSAn IT Quality Plan Has Two Objectives:• Support the organization’s quality policy• Ensure the quality of the strategic planAn IT Quality Plan should include the following:1. A reference to the organization’s quality policy2. Current baseline data3. Long-term quality goals4. Short-term quality goals5. The means to implement quality objectives6. Resources required
SUPPORTING / EXPLANATORY MATERIALSA quality toolbox is a set of tools, which assists in defining, controlling, and improving quality. The starter toolbox for a QA analyst contains the seven tools listed below. These tools and others are discussed later:• Brainstorming• Flowcharts• Cause-and-effect diagrams• Histograms• Pareto charts• Control charts• Scatter diagrams
Tool CategoriesKEY CONCEPT PRESENTATION GUIDEA quality toolbox is a set of tools, which assists in defining, controlling, and improving quality.
SUPPORTING / EXPLANATORY MATERIALSThe starter toolbox for a QA analyst contains the seven tools listed below. These tools and others are discussed later:• Brainstorming• Flowcharts• Cause-and-effect diagrams• Histograms• Pareto charts• Control charts• Scatter diagramsTool Categories• Management Tools• Statistical Tools• Presentation ToolsEach tool must be:• Selected• Learned• Used in practice
• Brainstorming• Affinity Diagram• Nominal group technique• Cause and effect diagram• Force field analysis• Flowchart and process map• Benchmarking• Matrix• Quality Function Deployment
Tools for PresentationKEY CONCEPT PRESENTATION GUIDEPresentations are an integral part of the change process.
SUPPORTING / EXPLANATORY MATERIALSThe involved parties, sometimes called stakeholders, need to be convinced that a proposed change is beneficial, or want to see reports during and after implementation. Stakeholders include management, the individuals that will use the changed process, and the individuals impacted by the changed process. Tools for Presentations include:• Tables• Line Charts• Bar Charts• Pie Charts• Stem and Leaf Charts• Force-Field Analysis
SUPPORTING / EXPLANATORY MATERIALSOne of the most difficult tasks facing any IT function is changing the way that function operates. In most organizations, stability is the norm and change is abnormal. That cycle needs to be reversed, if quality and productivity are to be constantly improved.People resist change because:• It is more difficult to implement change than to develop the approach• People do not like change imposed on them• Personal risk: can be successful with current processes• Making routine mistakes as a result of change• Management may be more committed to meeting schedules and budgets
The Deployment Process PhasesKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSThere are three deployment phases - assessment, strategic, and tactical. The assessment and strategic deployment phases represent the Planning component of the PDCA cycle. The tactical deployment phase represents the Do, Check, and Act components of the PDCA cycle.• Phase One: Assessment
Critical Success Factors of DeploymentKEY CONCEPT PRESENTATION GUIDEDeployment is much harder than defining an approach.
SUPPORTING / EXPLANATORY MATERIALS
Approach is an intellectual exercise; deployment is a people-intensive process. There are five intangible attributes called critical success factors that help make deployment work.Five intangible attributes called critical success factors that help make deployment work:1. Deployment is a series of integrated tasks.2. Deployment champions are needed.3. Deployment is a team effort.4. Buy-in is required by the affected parties.5. Deployment / responsibilities must be effectively passed between individuals and between
Internal Auditing and Quality AssuranceKEY CONCEPT PRESENTATION GUIDEBoth internal auditing and QA are professions.
SUPPORTING / EXPLANATORY MATERIALSIt is generally recognized that a profession has the following criteria:• Code of ethics• Common body of knowledge • Statement of responsibilities• Certification program (including continuing education)
SUPPORTING / EXPLANATORY MATERIALSInternal auditing is a management control directed at measuring and evaluating an activity to determine if it is performed in accordance with the policies and procedures of an organization (i.e., meets the intent of management). It is an independent appraisal activity. The specific types of auditing are:
Financial Auditing - Financial auditing is performed in accordance with generally accepted accounting procedures and other applicable laws and regulations to determine that the accounting records are reasonable.
Operational Auditing - Operational auditing is performed to determine that operations are performed in an efficient, effective and economical manner.
Program Auditing - Program auditing is performed to determine that the objectives of specific business activities are being properly fulfilled.
Auditing:• Measures and evaluates activities to determine if they are performed in accordance with
policies and procedures.• Auditing is an independent appraisal activity• An auditor’s work is detached from day-to-day operations• Auditors cannot get involved in developing procedures/standards
SUPPORTING / EXPLANATORY MATERIALSQA should be a leadership position, emphasizing the strong interpersonal activities involved in making improvement occur. While QA performs many appraisals, it strives to be independent of the activities being appraised. Auditing, by nature, has a negative role; QA, by practice, should have a positive role.Quality Assurance:• A leadership position• Emphasis on strong interpersonal skills• Facilitate the development of solution• Development of policies, standards, and procedures• Measurement and analysis responsibility• Statistical process control overview
SUPPORTING / EXPLANATORY MATERIALSQuality principles dictate that the strategic quality plan should:a. Be written for the quality assurance functionb. Identify individual quality programsc. Be incorporated as a component of the strategic business plan. d. Be developed by the quality assurance functione. Define the size of the quality assurance function Internal auditing is a management control directed at measuring and evaluating an activity. Which statement is NOT considered part of a specific type of auditing activity?a. Financial Auditingb. Operational Auditingc. People Auditingd. Program Auditing
SUPPORTING / EXPLANATORY MATERIALSMany newly appointed quality managers are challenged to implement a quality function. Briefly explain three of the more important factors to be considered that the quality manager must integrate into the operation.• Deployment is a series of integrated tasks.• Deployment champions are needed.• Deployment is a team effort.• Buy-in is required by the affected parties.• Deployment / responsibilities must be effectively passed between individuals and between
Skill Category 5KEY CONCEPT PRESENTATION GUIDEExecutive management establishes the vision and strategic goals.
SUPPORTING / EXPLANATORY MATERIALSPlanning is the process that describes how those strategic goals will be accomplished. Quality planning should be integrated into the IT plan so that they become a single plan. In simplistic terms, the IT plan represents the producer, and the quality plan represents the customer.Quality Planning• Planning Concepts• Integrating Business and Quality Planning• Prerequisites to Quality Planning• The Planning Process• Planning to Mature IT Work Processes
Define the objective, expressing it numerically, if possible.Clearly describe the goals and policies needed to attain the objective at this stage. Determine the procedures and conditions for the means and methods that will be used to achieve the objective. • If you do not know where you are going, all roads will lead you there.• If you fail to plan – plan to fail
The PDCA CycleKEY CONCEPT PRESENTATION GUIDEThe cycle comprises the four steps of: 1. Plan2. Do3. Check4. Act
SUPPORTING / EXPLANATORY MATERIALS
Plan (P): Devise a plan• Objectives are quantitatively defined• Policies are defined• Practices/processes used to achieve the goals• Objectives are Identified
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDEExecute the plan
SUPPORTING / EXPLANATORY MATERIALSCreate the conditions and perform the necessary teaching and training to ensure everyone understands the objectives and the plan. Teach workers the procedures and skills they need to fulfill the plan and thoroughly understand the job. Then perform the work according to these procedures.Do (D): Execute the plan• Create the practices (define them)• Provide training• Perform the work according to the defined practice
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDECheck the results
SUPPORTING / EXPLANATORY MATERIALSAs often as possible, check to determine whether work is progressing according to the plan and whether the expected results are obtained. Check for performance of the procedures, changes in conditions, or abnormalities that may appear.Check (C): Check the Results• Determine if work is progressing to plan.• Will anticipated results be realized? • Are goals and objectives satisfied?
The PDCA Cycle (cont.)KEY CONCEPT PRESENTATION GUIDETake the necessary action
SUPPORTING / EXPLANATORY MATERIALSIf the check reveals that the work is not being performed according to plan or if results are not what were anticipated, devise measures for appropriate action. Look for the cause of the abnormality to prevent its recurrence. Sometimes workers may need to be retrained and procedures revised. The next plan should reflect these changes and define them in more detail. Act (A): Take the Necessary Action• If work is not progressing to plan, devise measures and appropriate actions.• Look for the cause of abnormalities.• Update the plan.• Update training materials, if needed.
• Change the schedule• Change the budget• Change the number of resources allocated• Change how one component of software will affect other components• Change in work priorities• Addition or deletion of work activities
Prerequisites to Quality PlanningKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• IT vision, mission, and goals documented• Defined planning process• Management support for planning• Planners competent in the planning process• Compliance to the plan• Maintenance of the planning process• Reliable information required.
Common Activities in the Planning ProcessKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Business or Activity Planning• Environment Planning• Capabilities and Opportunities Planning• Assumptions/Potential Planning• Objectives/Goals Planning• Strengths, Weaknesses and Tactics Planning• Priorities and Schedules Planning• Organization and Delegation Planning• Budgets and Resources Planning
Strategic Planning QuestionsKEY CONCEPT PRESENTATION GUIDEThe planning activities described were designed to answers six basic planning questions as listed below.
SUPPORTING / EXPLANATORY MATERIALSThe planning process then documents the answers to these six questions:1. Where are we?2. Where do we want to go?3. How are we going to get there?4. When will it be done?5. Who is responsible for what?6. How much will it cost?
Relationships that Can Affect Process MaturityKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• People skills and process definitions• Do and check procedures• Individuals' assessment of how they are evaluated to work performed• What management relies on for success• Maturity level to cost to do work• Process maturity to defect rates
Relationships that Can Affect Process MaturityKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Process maturity and cycle time• Process maturity and end user satisfaction• Process maturity and staff job satisfaction• Process maturity to an organization's willingness to embrace change• Tools to process maturity• Control/test process category and quick pay backs
Skipping Levels and Reverting Back to Lower LevelsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Skipping a level is not practical because each level builds the base for moving to a higher level.
Reversion back to lower levels will occur constantly when there is either significant business or technology change. This does not mean that the information organization drops to a lower level, but for specific technology or business change those changes will have to be begun at a lower level and move rapidly up to the organization's current level.
Skill Category 6KEY CONCEPT PRESENTATION GUIDEDefining and continuously improving work processes allows the pace of change to be maintained without negatively impacting the quality of products and services.
SUPPORTING / EXPLANATORY MATERIALS
Define, Build, Implement, and Improve Work ProcessesProcess management is a term used by many IT organizations to represent the totality of activities involved in defining, building, deploying and maintaining the work processes used to achieve the IT mission. • Process Management Concepts• Process Management Processes
Definition of a ProcessKEY CONCEPT PRESENTATION GUIDEA process is a vehicle of communication, specifying the methods used to produce a product or service.
SUPPORTING / EXPLANATORY MATERIALSIt is the set of activities that represent the way work is to be performed. The level of communication (detail of the process) is normally commensurate with the skill level associated with the job. A process is a:• Communication vehicle or model• Method used to produce a product or service. • Set of work performance activities
Why Processes are NeededKEY CONCEPT PRESENTATION GUIDEProcesses add value to both management and the workers, although the reasons differ.
SUPPORTING / EXPLANATORY MATERIALSFrom a management perspective, processes are needed to:• Explain to workers how to perform work tasks• Transfer knowledge from more experienced to less experienced workers• Assure predictability of work activities so that approximately the same deliverables will be
produced with the same resources each time the process is followed• Establish a basic set of work tasks that can be continuously improved• Provide a means for involving workers in improving quality, productivity, and customer
satisfaction by having workers define and improve their own work processes• Frees management from their activities associated with “expediting work products” to spend
more time on activities such as planning, and customer and vendor interaction
Why Processes are Needed (cont.)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSFrom a worker’s perspective, work processes are important to:• Increase the probability that the deliverables produced will be the desired deliverables• Put workers in charge of their own destiny because they know the standards by which their
work products will be evaluated• Enable workers to devote their creativity to improving the business instead of having to
develop work processes to build products• Enable workers to better plan their workday because of the predictability resulting from work
Process Workbench and ComponentsKEY CONCEPT PRESENTATION GUIDEThe workbench is a graphic illustration of a process, documenting how a specific activity is to be performed.
SUPPORTING / EXPLANATORY MATERIALSWorkbenches are also called phases, steps, or tasks. A process can be viewed as one or more workbenches. Depending on the maturity of the organization, process workbenches may be defined by process management (or standards) committees, QA analysts, or work teams.
SUPPORTING / EXPLANATORY MATERIALSApproaches for controlling businesses have evolved over many decades. Many of these business control models include these three general areas of control:1. Management processes2. Work processes3. Check processes
How Processes are ManagedKEY CONCEPT PRESENTATION GUIDEThe infrastructure in a quality management environment should support process management.
SUPPORTING / EXPLANATORY MATERIALSIn a quality management environment: • The infrastructure must supports process management. • Process management is primarily a line responsibility. • All levels of the organization should be involved in both establishing and using work
Planning ProcessesKEY CONCEPT PRESENTATION GUIDEProcess management is a PDCA cycle.
SUPPORTING / EXPLANATORY MATERIALSProcess management processes provide the framework from within which an organization can implement process management on a daily basis. The figure shows how this set of practices can be viewed as a continuous improvement cycle.
• Process planning allows priorities to be set for process management projects. • Priorities are set based on the relative importance of the process to accomplishing the
organization’s missions/goals, organizational constraints or readiness, and an assessment of the project’s status.
The Check cycle:• Includes process measurement • Determines what strategic and tactical measures and metrics are needed to manage by fact• Incorporates measurement into the appropriate processes.
SUPPORTING / EXPLANATORY MATERIALSGood quality practices require that work standards should be developed by:a. IT Managementb. Quality assurancec. Subject matter expertsd. Users of the standarde. Quality controlThree parts of the PDCA cycle are act, plan, and do. The fourth part of the PDCA cycle is:a. Contributeb. Calibratec. Checkd. Create
SUPPORTING / EXPLANATORY MATERIALSApproximately 10 years ago your IT department organized a Standards Committee. The committee determined that standards were needed, wrote those standards, incorporated them into a standards manual and delivered them to the appropriate staff members for implementation. After 10 years, it is generally agreed that the standards program is not working. The IT staff does not follow many of the standards, and management rarely enforces compliance. You, as the Quality Assurance Manager, have been asked to develop a new standards program. Indicate what you believe is the responsibility of the following groups in implementing an effective standards program.
Skill Category 7KEY CONCEPT PRESENTATION GUIDEQuality control practices should occur during product development and throughout product change and operation.
SUPPORTING / EXPLANATORY MATERIALSDuring development, the quality control process is frequently called verification and at the conclusion of development, it is called validation. The quality practitioner should also be familiar with verification and validation techniques, the framework for developing testing tactics, change control and configuration management. • Testing Concepts• Developing Testing Methodologies• Verification and Validation Methods• Software Change Control• Defect Management• Process Management Processes
Quality Control BasicsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSQuality Control Basics include:• Quality control measures a product against the existence of an attribute • Determines whether the product conforms to a standard or procedure (also known as
compliance checking).• Management is responsible for enforcing compliance to standards and procedures.
Process Workbench and ComponentsKEY CONCEPT PRESENTATION GUIDEThe workbench is a graphic illustration of a process, documenting how a specific activity is to be performed.
SUPPORTING / EXPLANATORY MATERIALSWorkbenches are also called phases, steps, or tasks. A process can be viewed as one or more workbenches. Depending on the maturity of the organization, process workbenches may be defined by process management (or standards) committees, QA analysts, or work teams.
Independent test team responsibilities: • System testing• Oversight of user acceptance testing• Provide an unbiased assessment of the quality of an application• Support or participate in other phases of testing including executing special test types such
Independent test manager responsibilities: • Planning and estimating tests• Designing the test strategy• Ensuring tests are created and executed in a timely and productive manner• Reviewing analysis and design artifacts• Chairing the test readiness review• Managing the test effort• Overseeing acceptance tests
Independent tester responsibilities: • Developing test cases and procedures• Planning, capturing, and conditioning test data• Reviewing analysis and design artifacts• Executing tests• Utilizing automated test tools for regression testing• Preparing test documentation• Tracking and reporting defects
Reviews and InspectionsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Are Performed To:• Maximize team talent• Identify defects.• Educate a large number of people quicklyRules• Review the product not the producer• Identify defects and issue, don’t resolve them• Every member is responsible for success
SUPPORTING / EXPLANATORY MATERIALSProject teams will use multiple review techniques for work products throughout the development lifecycle. The most effective verification and validation (V&V) programs utilize a combination of techniques to maximize the benefit received from reviews while minimizing impact to project timelines. When creating the Quality Plan (or Software Verification and Validation Plan) teams should determine the review technique for each work product based upon the complexity and risk associated with the deliverable, as well as the number of downstream work products that will utilize the artifact under review as input. For example, requirements feed almost every other work product in the software development process, and therefore should be reviewed using more formal techniques.Informal Techniques • Little/no preparation• No formal documentation • Little measurementSemi-Formal Techniques• Author presents material• Wide range of discussion• Metrics should be captured
NOTES:
Formal Techniques• Formal process with well-defined roles• Moderator leads review• Checklists and formal documentation• Metrics captured
SUPPORTING / EXPLANATORY MATERIALSA testing methodology forms the basis for the entire testing effort and should be developed for each project.A testing methodology should be developed for each development, maintenance, purchase, or upgrade project. The testing methodology defines the scope and general direction for testing. It is a high-level plan and should be developed as soon as the project is initiated. A test methodology must answer the following questions: • What type of project?• What is project’s scope?• What type of software?• When will testing occur?• Who will conduct testing?• What are the trade-offs?• What are the Critical Success Factors?
What Type of Project?KEY CONCEPT PRESENTATION GUIDEWe must use different testing approaches for different types of projects, just as we use different development approaches
SUPPORTING / EXPLANATORY MATERIALS
• Traditional Development System (Waterfall)• Client/Server• Interactive Development/ Prototyping/CASE• Object-Oriented• System Maintenance / Legacy Systems• Purchased / Contracted Software
Management of Verification and ValidationKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Management of software development verification and validation activities: • Begins at the start of a project• Performed for all SDLC processes and activities.
Reviews and InspectionsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Are Performed To:• Maximize team talent• Identify defects• Educate a large number of people quicklyReview Formats• Informal (peer review and desk checks) • Semi formal (JAD joint application development)• Formal (Inspection)
White Box Testing• Structural testing based on knowledge of internal code structure and usually logic driven• Statement coverage• Decision Coverage• Condition Coverage• Decision/Condition Coverage
Structural and Functional TestingKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSStructural and Functional Testing:Structural testing is considered white box testing because knowledge of the internal logic of the system is used to develop test cases.• Advantages:
- The logic of the software’s structure can be tested- Parts of the software will be tested which might have been forgotten if only functional
testing was performed• Disadvantages
- Its tests do not ensure that user requirements have been met- Its tests may not mimic real-world situations
Structural and Functional Testing (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Functional testing addresses the overall behavior of the program by testing transacting flows, input validation, and functional completeness but no knowledge of the internal logic system is used (black box).• Advantages
- Simulates actual system usage- Makes no system structure assumptions
• Disadvantages- Potential of missing logical errors in software- Possibility of redundant testing
Software Configuration ManagementKEY CONCEPT PRESENTATION GUIDEDynamic nature of business activities requires this to be in place
SUPPORTING / EXPLANATORY MATERIALS
• Constant program changes• Well-formulated and well-documented procedures• Program manipulation prevention • No unauthorized use. • Primary objective of SCM:
“Get the right changes installed at the right time”
Change Control ProceduresKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• All proposed changes should be in writing • Major changes should be approved by the Configuration Control Board• Developers should make and document the program changes, not the operations group• Someone independent of the person who designed and made the change should be
responsible for testing the final revised program• The documentation system should be updated with all change sheets or change registers
The primary goal is to prevent defects• The process should be risk driven.• Defect measurement should be integrated into the development process.• The capture and analysis of the information should be automated.• Defect information should be used for process improvement.
Defects are recorded for four major reasons:• Ensure the defect is corrected• Report status of the application• Gather statistics used to develop defect expectations in future applications• Improve the software development process.
• Requires a communication mechanism, either manual or automated• Facilitates communication between test and development teams • What to report?
- Defect name and type- Severity and priority- Status- Date and time of detection- Location identified (e.g. component, GUI, etc.)- Detailed description- Component or program where defect was found- Screen prints, etc.- Stage of origination (may be added later)- Person assigned to correct- Correction effort in hours
Using Defects for Process ImprovementKEY CONCEPT PRESENTATION GUIDEThe primary goal is to prevent defects.
SUPPORTING / EXPLANATORY MATERIALS
Based on the team’s findings, the process improvement activities should include the following:• Go back to the process that originated the defect to understand the root cause• Go back to the verification and validation processes which should have caught the defect
SUPPORTING / EXPLANATORY MATERIALSWhich of the following is NOT a verification technique:a. Feasibility Reviewb. Design Reviewc. Requirements Reviewd. Code Inspectione. Unit TestWhich of the following is NOT a validation techniquea. Executing all statements at least onceb. Executing each decision with all possible outcomes at least oncec. Executing test cases based on classes of datad. Executing each decision direction at least once
Skill Category 8KEY CONCEPT PRESENTATION GUIDETo effectively measure, one needs to know the basic concepts of measurement. This section provides those basic measurement concepts.
SUPPORTING / EXPLANATORY MATERIALSThere is no clearly defined, commonly accepted method of measuring software products and services. A large number of measures and metrics exist, but only a few have had widespread acceptance.
• Measurement Concepts• Measurement in Software• Variation and Process Capability• Risk Management• Implementing a Measurement Program
Standard Units of MeasureKEY CONCEPT PRESENTATION GUIDEA measure is a single quantitative attribute of an entity.
SUPPORTING / EXPLANATORY MATERIALSIt is the basic building block for a measurement program. Examples of measures are lines of code (LOC), work effort, or number of defects. Measurement cannot be used effectively until the standard units of measure have been defined. For example, talking about lines of code does not make sense until the measure LOC has been defined. • A measure is a single quantitative attribute of an entity. It is the basic building block for a
measurement program.• Since quantitative measures can be compared, measures should be expressed in numerical
MetricsKEY CONCEPT PRESENTATION GUIDEA metric normalizes data so that comparison is possible.
SUPPORTING / EXPLANATORY MATERIALS• A metric is a derived (calculated or composite) unit of measurement that cannot be directly
observed, but is created by combining or relating two or more measures. Since metrics are combinations of measures, they can add more value in understanding or evaluating a process than plain measures. Examples of metrics are mean time to failure and actual effort compared to estimated effort.
• A metric normalizes data so that comparison is possible.
Objective and Subjective MeasurementKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSObjective measurement uses hard data that can be obtained by counting, stacking, weighing, timing, etc. Examples include:• Number of defects• Hours worked• Completed deliverables.An objective measurement should result in identical values for a given measure, when measured by two or more qualified observers.Subjective data is normally observed or perceived. It is a person's perception of a product or activity, and includes personal attitudes, feelings and opinions, such as how easy a system is to use, or the skill level needed to execute the system. Objective measurement uses hard data that can be obtained by counting, stacking, weighing, timing, etc.
Types of Measurement DataKEY CONCEPT PRESENTATION GUIDEIdeally, models should be developed that are capable of predicting process or product parameters, not just describing them.
SUPPORTING / EXPLANATORY MATERIALS
Four uses of measurement• Manage and control processes• Manage and control products.• Improve processes.• Manage risks.Four types of measured data:• Nominal• Ordinal Data• Interval Data• Ratio Data
How to Know a Measure Is GoodKEY CONCEPT PRESENTATION GUIDEIdeally models should be developed that are capable of predicting process or product parameters, not just describing them.
SUPPORTING / EXPLANATORY MATERIALS
Measures and resulting metrics should be:Measures and resulting metrics should be:• Reliable• Valid• Easy to Use• Simple• Timely• Easy to Calibrate• Objective• Easily obtainable
Quantitative Management in an IT FunctionKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSAn integral part of an IT function is quantitative management, which contains three aspects:• Measurement Dashboards• Function Points• Statistical Process Control (SPC)Measurement dashboards (also called key indicators) are used to monitor progress and initiate change. Statistical process control is used to ensure that the process behaves in a consistent manner. Line managers use the principles of statistical process control to assess consistency of products and services, and as a basis for continuous process improvement.
Product MeasurementKEY CONCEPT PRESENTATION GUIDEA product can be measured at any stage of its development.
SUPPORTING / EXPLANATORY MATERIALSFor a software product, the requirements, the complexity of the software design, the size of the final program’s source or object code, or the number of pages of documentation produced for the installed systems can be measured.Examples of product measurement:• Size
Process MeasurementKEY CONCEPT PRESENTATION GUIDEA process can be measured by any of the following:• Attributes of the process, such as
overall development time• Type of methodology used• The average level of experience of the
development staff.
SUPPORTING / EXPLANATORY MATERIALSAccumulating product measures into a metric so that meaningful information about the process can be provided. For example, function points per person-month or LOC per person-month can measure productivity (which is product per resources), the number of failures per month can indicate the effectiveness of computer operations, and the number of help desk calls per LOC can indicate the effectiveness of a system design methodology.A process can be measured by any of the following:• Number of deliverables completed on time• Estimated cost vs. actual costs• Time spent fixing errors• Idle time• Number of modifications
Common Causes of VariationKEY CONCEPT PRESENTATION GUIDEStatistics that took into account common and special causes of variation.
SUPPORTING / EXPLANATORY MATERIALS
Common causes are those that can be controlled by improving the work processes. All processes contain some inherent variation, or common causes of variation. The amount of variation in a process is quantified with summary statistics.In a computer operation, abnormal terminations cause variation. Typical common causes of abnormal terminations include:• Invalid Data• No Disk Space• Errors in Operating• Job Control Instructions, etc.
Special Causes of VariationKEY CONCEPT PRESENTATION GUIDEStatistics that took into account common and special causes of variation.
SUPPORTING / EXPLANATORY MATERIALSSpecial causes of variation are not present in a process. Special causes are those that must be controlled outside the process; typically they need to be dealt with individually. • Special causes of variation are not present in a process. • They occur because of special or unique circumstances. • In the IT example of abnormal terminations in a computer operation, special causes might
include:- Operator Strikes- Citywide Power Outages- Natural Disasters, such as earthquakes or hurricanes
Defining RiskKEY CONCEPT PRESENTATION GUIDERisk is the possibility that an unfavorable event will occur.
SUPPORTING / EXPLANATORY MATERIALSRisk management involves the activities of defining, measuring, prioritizing, and managing risk in order to eliminate or minimize any potential negative effect associated with risk.• Risk is the possibility that an unfavorable event will occur.• It may be predictable or unpredictable.• Risk has three components, each of which must be considered separately when deterring
how to manage risk:- What can happen?- How likely is it that it will?- What do we do if it does?
SUPPORTING / EXPLANATORY MATERIALSChanges in a situation can result in new risks. Examples include:• Replacing a team member• Undergoing reorganization• Changing a project's scope.The probability of risk occurring at the beginning of the project is very high (due to the unknowns), whereas at the end of the project the probability is very low. Within a project, many tasks and deliverables are intertwined. The relationship of probability and impact are not linear, and the magnitude of the risk typically makes a difference. Risk has five distinguishing characteristics:• Situational• Time-based• Interdependent• Magnitude dependent• Value-based
SUPPORTING / EXPLANATORY MATERIALSChanges in a situation can result in new risks. Examples include, replacing a team member, undergoing reorganization, or changing a project's scope.The probability of risk occurring at the beginning of the project is very high (due to the unknowns), whereas at the end of the project the probability is very low. Within a project, many tasks and deliverables are intertwined. The relationship of probability and impact are not linear, and the magnitude of the risk typically makes a difference. Processes of risk:• Risk Identification• Risk Quantification• Risk Analysis• Risk Prioritization• Risk Response Development• Risk Response Planning• Risk Response Control• Risk Resolution• Risk Monitoring
Incorporating risk management info the software development life cycle includes planning at the following levels:• Long-Term or High-Level• Medium-Term or Medium Level• Short-Term or Low-Level
Risk of Integrating New TechnologyKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
The QA Analyst has three roles in integrating new technology:• Determining the Risk• Assuring that the Controls are adequate to reduce the risk• Assuring that existing processes are appropriately modified to incorporate the Use of New
Prerequisites to Implementing a Measurement ProgramKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSCurrent IT management is often ineffective because the IT function is extremely complex and has few well-defined, reliable processes or product measures to guide and evaluate results. Due to:• Operations complexity• Few well-defined, reliable process or product measures for resultsAs a result:• Accurate and effective estimating, planning, and control are nearly impossible to achieve.Implementing a measurement program requires four prerequisite steps:1. Perceive the need for a measurement program.2. Identify a champion and change agent, and assign organizational responsibility.3. Establish tangible objectives and meaningful measurement program activities.4. Facilitate management buy-in at all levels for the measurement program.
SUPPORTING / EXPLANATORY MATERIALSThe most frequent measure used to quantify efficiency in the use of resources is:a. Qualityb. Customer satisfactionc. Return on Investmentd. Productivitye. PerformanceThese causes of variation are NOT present in a process:a. Commonb. Codingc. Speciald. Sequence
SUPPORTING / EXPLANATORY MATERIALSYour management has defined what they believe are important attributes of a software system. Five attributes are listed below. Develop a metric that will measure each of these five attributes.
Software Attribute Proposed Metric to Measure the Attribute
1. Usability2. Maintainability3. Reliability4. Security5. Service Level
Skill Category 9KEY CONCEPT PRESENTATION GUIDETwo key issues for quality assurance are internal control and security.
SUPPORTING / EXPLANATORY MATERIALS
Interest in internal control has been highlighted by the passage of the Sarbanes-Oxley Act. Interest in internal control and security has been highlighted by publicized penetrations of security and the increased importance of information systems and the data contained by those systems.• Principles and Concepts of Internal Control• Environmental or General Controls• Transaction Processing Controls• The Quality Professionals Responsibilities• Risk and Internal Control Methods• Building Internal Controls• Building Adequate Security
SUPPORTING / EXPLANATORY MATERIALSFour key terms are understood:Risk – The probability that an undesirable event will occur.Exposure – The amount of loss that might occur if an undesirable event occurs.Threat – A specific event that might cause an undesirable event to occur.Control – Anything that will reduce the impact of risk.A process, effected by an organization’s Board of Directors, management, and other personnel, designed to provide reasonable assurance regarding the achievement of objectives in the following categories:• Effectiveness and efficiency of operations• Reliability of financial reporting• Compliance with applicable laws and regulations
Internal Control TermsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALSInternal Control• The interaction and/or interactivity within an organization that allows the entity to provide for
information accuracy, the safeguard of assets, reliability and efficiency of operations, and adherence to management policies, procedures and processes.
Process• An established methodology that has certain defined components which enable the activity
to be consistently repeated without variability.Testing• The activity performed to reduce the risk and probability of an unfavorable event occurring.
Internal Control Terms (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Risk• The probability that an undesirable event will occur.Exposure• The amount of loss that might occur if an undesirable event occurs.Threat• A specific event that might cause an undesirable event to occur.Control• Any activity, technique, method, or approach that minimizes or eliminates risk.
The International Standards for the Professional Practice of Internal Auditing, established by the Institute of Internal Auditors, specify that internal auditors should:• identify and evaluate significant exposures to risk and contribute to the improvement of risk
management and control systems• Monitor and evaluate the effectiveness of the organization’s risk management system• Evaluate risk exposures relating to the organization’s governance, operations, and
information systems • Assist the organization in maintaining effective controls by evaluating their effectiveness and
efficiency and by promoting continuous improvementIt should be recognized that the internal audit function does not have primary responsibility for establishing or maintaining the internal control system.
Internal Control ModelsKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
COSO Enterprise Risk Model (ERM)The COSO ERM framework defines risk and enterprise risk management, and provides a foundational definition, conceptualizations, objective categories, components, principles and other elements of a comprehensive risk management framework. ERM consists of eight interrelated categories:1. Internal Environment2. Objective Setting3. Event Identification4. Risk Assessment5. Risk Response6. Control Activities7. Information and Communications8. Monitoring
Building Adequate SecurityKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Vulnerabilities in SecurityFunctional Vulnerabilities• Physical Access• IT Operations• Test Processes• Computer Programs• Operating Systems Access and Integrity• Impersonation• Media
Building Adequate Security (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
IT Areas Where Security is Penetrated• Data and Report Preparation Facilities• Computer Operations• Non-IT Areas• Online Terminal Systems• Program Terminal Systems• Online Data Preparation and Report Generation• Digital Media Storage Facilities• Online Operations• Central Processors
Establishing a Security BaselineKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Creating Baselines• What to collect• From who will the information be collected?• The Precision of the information collected
- Establish a baseline team- Set baseline requirements and objective- Design baseline data collection methods- Train baseline participants- Collect baseline data- Analyze and report computer security status
SUPPORTING / EXPLANATORY MATERIALSMany people believe there is an overlap between internal auditors and quality assurance, however the main role of internal auditing is to:i. Identify and report problemsj. Define and implement solutions to a problemk. Build processes that prevent problems from occurringl. Check all products to make sure they are defect freem. Conduct acceptance testing before usageOne of the primary objectives of process control is to:a. Reduce variabilityb. Increase competitionc. Improve metricsd. Find root causes
Skill Category 10KEY CONCEPT PRESENTATION GUIDEQuality of software remains an internal IT responsibility regardless of who builds the software.
SUPPORTING / EXPLANATORY MATERIALSOrganizations can assign software development work responsibilities to outside organizations through purchasing software or contracting services; but they cannot assign the responsibility for quality.The quality professionals need to assure that those quality responsibilities are fulfilled through appropriate processes for acquiring purchased software and contracting for software services.• Quality and Outside Software• Selecting COTS Software• Selecting Software Developed by Outside Organizations• Contracting for Software Developed by Outside Organizations• Operating for Software Developed by Outside Organizations
Challenges in Selecting Acquired SoftwareKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Task or items missing• Software fails to perform• Extra features• Does not meet business needs• Does not meet operational needs• Does not meet people needs
Challenges in Selecting Acquired SoftwareKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Quality factors may not be specified• Non-testable requirements and criteria• Customer’s standards may not be met• Missing requirements• Overlooked changes in standards or technology• Training and deployment may be difficult
• Responsibility for the quality COTS or contracted software cannot be contracted• There must be a process to monitor the development and validate the correct functioning of
contracted software• The quality professional needs to look at how the contractor tests in relation to the SEI
COTS Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Assure completeness of needs specification• Define critical success factor• Determine compatibility with your computer environment• Assure the software can be integrated into your business system work flow• Demonstrate the software in operation• Evaluate the people fit• Acceptance test the COTS software
COTS Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Assure the software can be integrated into your business system work flow• Current system based on certain assumptions• Existing forms, existing data, existing procedures• COTS based on certain assumptions• COTS uses a predetermined set of forms & proc.• Current system & COTS may be incompatible• COTS is not going to change
COTS Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Demonstrate the software in operation• Observe during demonstration:• Understandability• Clarity of communications• Ease of use of manuals• Functionality of software• Knowledge to execute• Effectiveness of help routines• Evaluate program compatibility• Data compatibility• Smell Test
COTS Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Evaluate the people fit• The software can be used as is• Additional training and support is necessary• The software is not usable with the skill sets of the proposed users
Contracted Software Selection Process KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Assure process for contracting software is adequate• Assure requirements & contract criteria are testable• Review adequacy of the contractor’s development process• Review adequacy of the contractor’s test process• Define acceptance testing criteria• Assure the contractor’s report schedule is adequate• Ensure knowledge transfer & intellectual rights protected• Assure ongoing operation and maintenance • Assure the effectiveness of contractual relations
Contracted Software Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Assure process for contracting software is adequateContracts should include:• What is done.• Who does it• When it is done• How it is done• Where it is done• Penalties for nonperformance
Contracted Software Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Review adequacy of the contractor’s test process• Evaluate the adequacy and completeness of testing that will be performed.• May perform a comparison against internationally accepted test plan standards (e.g., IEEE)
Contracted Software Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Define acceptance testing criteriaMinimum the acceptance testing should validate:• The documentation is consistent with execution• Documentation is understandable• User will be adequately trained• It is operable within the operational constraints
Contracted Software Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Ensure knowledge transfer & intellectual rights protected• Training programs for staff• Being advised of defects uncovered by other organizations using software• Ability to contact contractor’s help desk• Use care to protect contractor property rights
Contracted Software Selection Process (cont)KEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
Assure ongoing operation and maintenance Major concerns:• Adequacy of control• Adequacy of documentation• Speed of service• Nearness of service• Competency of service• Adequacy of hardware• Skilled personnel• Multi-contractor problem resolution• Cost of services• Cost of operations• Error diagnosis• Error documentation
Assume that your organization was asked to build an important software project, but your IT organization was currently overcommitted, so it was decided to outsource the development of the software to a contractor. Your contracting officer has asked you as a test manager to write the section of the contract that specifies how the contractor is to test the software. Rather than write that section, list what you think are the five (5) most important contractor test responsibilities that you think should be included in the contract and then describe those responsibilities in enough detail that if the contractor met those testing responsibilities you would be comfortable with the contractor’s ability to effectively test the software.
SUPPORTING / EXPLANATORY MATERIALSSoftware can be developed by an organization within the country of use, or developed in another country. Which of the following is a difference associated with software developed by an organization in another country?a. Missing requirementsb. Customer standards may not be metc. Non-testable requirementsd. Training and deployment may be difficulte. Cultural differencesAt a minimum, the acceptance testing for contracted software should validate:a. The documentation is consistent with the software executionb. The documentation is understandablec. Users will adequately trained in the softwared. It is operable within the operation constraints of the organizatione. All of the above
On the Day of the ExamKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
• Arrive at the examination location at least 30 minutes before the scheduled start time of the examination
• You must have the following on the day of the exam:• Confirmation Letter from Software Certifications• Photo identification • Two # 2 pencils• Don’t bring personal items into the room• Once exam starts no one will be admitted
• Allow up to 12 weeks for results• Completed results are mailed the same day• Do not discuss the contents of the exam – The code of ethics• Do not call QAI or Software Certifications within those 12 weeks• Results will not be given over the phone.
• Remember one question may only be worth one point, if you are not able to answer questions later because you have lost time you could lose more points!
• Becoming nervous can make you lose your concentration and that leads to worse results. • Do not insist on completing each question before going to the next.
This is extremely important. If you spend a lot of time on one question that you don't understand there can be two negative outcomes: - Losing time - Becoming nervous
• For multiple choice parts, read each question’s stem twice, then read ALL the responses• For essay parts, read each of the essay questions thoroughly • As you read through each question (multiple choice and essay) determine whether:
- You absolutely know the answer to this question.- You believe you know the answer to this question. - You are not sure you know the answer, or it would take time to develop an answer.
• For both multiple choice and essay questions, first answer the questions to which you know the answers
If possible, go through the entire test, answering the questions you are sure you know. This results in your being more relaxed and feeling more confident.
Go through the test a second time, working out the answers to more difficult questions. Now you will feel more confident and this will improve your test taking.
Key Tip: Do not waste too much time on any one question.
Once you have gone through the test twice, see if any of the questions asked can help you answer those really difficult questions.This is a little used trick. Sometimes questions asked are answered in later questions asking for different things
Going back to think about it usually makes you unsure and often causes an error. This is very common so be very careful!
If you have a strong feeling about a question when you first answer it, don't go back and change it later. Usually (but not always) a strong first impulse means we know the answer and we don't really have to think about it too much.
Play the odds • If you don't know the answer, write something. • If you are answering a 4 possibility multiple choice question, you will still have a 25% chance
As a test manager for a large IT department, you’ve been asked what information should be recorded about each defect that is uncovered. Identify and explain what you believe are the four most important characteristics that should be recorded about a defect.
• Throw out the ridiculous answers in a multiple choice question • Many tests often include one or two completely ridiculous possibilities in any multiple choice
question. • Find these first and get rid if them. This will usually leave you with a choice between two.
Test Taking Tip SummaryKEY CONCEPT PRESENTATION GUIDE
SUPPORTING / EXPLANATORY MATERIALS
1. Answer the questions that you believe you know the answer 2. Answer the questions that you think you know the answer3. Answer the questions that you don’t know the answer
he 10 CSQA CBOK Skill Categor ies are common to all quality-related assignments andtherefore, the certification examination focuses equally on all of them. Pleasse note that SkillCategories 9 and 10 are new to the 2006 CSQA Common Body of Knowledge
The 2006 Common Body of Knowledge for the software quality analyst certificate includes these tenskill categories:
Skill Category 1 – Quality Principles and Concepts
Skill Category 2 – Quality Leadership
Skill Category 3 – Quality Baselines
Skill Category 4 – Quality Assurance
Skill Category 5 – Quality Planning
Skill Category 6 – Define, Build, Implement and Improve Work Processes
Skill Category 7 – Quality Control Practices
Skill Category 8 – Metrics and Measurement
Skill Category 9 – Internal Control and Security
Skill Category 10 – Outsourcing, COTS, and Contracting Quality
The 10 CSQA CBOK Skill Categories are common to all quality-relat ed assignments and therefore,the certification examination focuses equally on all of them.
To assess your competency of the CSQA CB OK, complete the “CSQA 2006 Skill AssessmentWorksheets” starting on page 308. Follow these guide lines on how to use the worksheet to rate yourcompetency and identify those areas that you need to better understand to successfully pass the CSQAexamination:
1. Assess your competency of ea ch skill listed on the w orksheet. Carefully read each skil lwithin the skill category. Based on your reading of the skill, assess your competency in oneof the following three categories and place a check mark (“ ”) in the appropriate columnon the CSQA 2006 CBOK Competency Rating Table:
Not Competent – “None”
Either you do not understand this skill, or if you do understand it you do not know “what”is required to perform this skill. For exampl e, you may know that an IT quality plan isneeded, but you do not know what is included in an IT quality plan.
Some Competency – “Some”
This assessment means that you know “what” is needed to accomplish a specific skill. Forexample, you may know what is to be included within an IT quality plan, but you havenever actually prepared an IT quality plan. In othe r words, you have book knowledge, butnot how-to knowledge.
Fully Competent – “Full”
This assessment means that you not only know what is required to perform a specific skill,but you have actually used that skill in pe rforming day-to-day work tasks. For example,you have written an IT quality plan.
Note that Skill Category 1 focuses on the voc abulary of IT quality assurance and the basicconcepts on which the quality assurance profession is built. In assessing this category for aquality term such as reliability a “not competent” response means you could not define theterm; a “some competency” response means you could defi ne the term; and a “fullycompetent” response means that you use the term in the pe rformance of your day-to-daywork.
Follow these steps to calculate your competency rating for the CSQA 2006 CBOK. This rating willhelp you determine if you are rea dy to submit your application fo r the CSQA examination or if youneed to study any Skill Categories further in order to pass the examin ation. Use the CBOK SkillCategory Competency Rating Table on page 318 to perform each step below.
1. Total the number of skills you have checked in each of the three co lumns for each skillcategory. Write your number s in the sp ace provided for each skill category on theworksheet. These are your competency rating totals for that skill category.
2. Transfer the three comp etency rating totals for each skill categor y to the correspondingcolumn (“Full,” “Some,” and “None”) in the CSQA Skill Category Competency Ratingstable provided.
3. Tally each column in the table to determine each Ratings Total.
4. Multiply each column by the indicated number to determine the Column Total.
5. Add the Column Totals together to determine the Sum of the Rows Total.
6. Divide the Sum of the Rows Total by 155 (the number of total skills in the CSQA 2006CBOK) to determine your CSQA CBOK Competency Rati ng. This number will bebetween 1 and 3.
Now you are able to determ ine if you are ready to submit your application and take the certificationexamination or if you need further study. Use your CSQA 2006 CBOK Competency Rating from step6 above and the following key to interpret your competency rating:
The closer your score is to “3,” the more competent you are in software quality assurance.
• If your score is a “3,” you are a world-class software quality analyst and ready to sub-mit your application. See the “Introduction” for information on submitting your appli-cation for the CSQA 2006 certification examination.
• If your score is between “2” and “3”, you are a competent quality analyst and ready to submit your application. See the “Introduction” for information on submitting your application for the CSQA 2006 certification examination.
• If your score is between “1” and “2”, you do not have the basic skills necessary to per-form software quality assurance. Study those skills that you rated “None” and then reassess your skills.
• If your score is a “1”, you are not competent in the CBOK. Study those skills that you rated “None” and then reassess your skills.
Skill Category 1 – Quality Principles and Concepts
Before an organization can begin to assess the quality of its pr oducts and services, and identifyopportunities for improvement, it first must have a working knowledge of quality principles and basicconcepts. This category tests th e CSQA candidate’s abil ity to understand and ap ply these principles,which include the quality vocabulary, various ways of defining qualit y, key concepts, distinguishingbetween quality control and quality assurance, and the contributions of quality pioneers.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 1 – Quality Principles and Concepts Competency RatingSkill # Skill Description Full Some None
1.1Vocabulary of Quality
Understand the vocabulary of quality
1.2The Different Views of Quality
The two quality gaps1.3 Quality attributes for an information system
1.4Quality Concepts and Practices
PDCA Cycle1.5 Cost of quality1.6 Six sigma quality1.7 Baselining and benchmarking1.8 Earned value
1.9Quality Control and Quality Assurance
Understand quality control and quality assurance1.10 Understanding and using the Just-in-Time (JIT) Technique1.11 Differentiating between Quality Control and Quality Assurance
1.12Quality Pioneers Approach to Quality
Includes Dr. W. Edwards Deming, Philip Crosby, and Dr. Joseph Juran
The most important prerequisites for suc cessful implementation of any major quality initiative areleadership and commit ment from executive management . Management must create a workenvironment supportive of qua lity initiatives. It is management’s responsibility to establish strategicobjectives and build an infrastructure that is strategically aligned to those obj ectives. This categorycovers the management processes used to establish the foundation of a quality-managed environment,as well as commitment, new behaviors, building the infrastru cture, techniques, approaches andcommunications.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 2 – Quality Leadership Competency RatingSkill # Skill Description Full Some None
2.14Leadership Concepts
Executive and middle management commitment2.15 Quality Champion2.16 New Behaviors for Management - traditional management versus
quality management, leadership, the importance of establishing mentoring relationships, and establishing trust
2.17 Empowerment of employees
2.18Quality Management Infrastructure
Quality council2.19 Management committees2.20 Teams and work groups2.21 Process improvement teams
2.22Quality Environment
The six attributes of an effective quality environment2.23 Setting the proper “tone” at the top2.24 Code of ethics and conduct2.25 Open communication2.26 Implementing a mission, a vision, goals, values and a quality
policy2.27 Monitoring compliance to organizational policies and procedures2.28 Enforcement of organizational policies and procedures
Organizations need to establis h baselines of performance for quality, productivity and customersatisfaction. These baselines are used to documen t current performance and document improvementsby showing changes from a baseli ne. In order to establish a base line, a model and/ or goal must beestablished for use in measuring against to determine the baseline.
Competency Rating Totals (total each “ ” in each column): ______ ______ _____
Skill Category 3 – Quality Baselines Competency RatingSkill # Skill Description Full Some None
3.29Quality Baseline Concepts
Baselines defined3.30 Types of baselines3.31 Conducting baseline studies
3.32Methods Used for Establishing Baselines
Customer surveys3.33 Benchmarking to establish a baseline goal3.34 Assessments against management established criteria 3.35 Assessments against industry models
3.36Model and Assessment Fundamentals
Purpose of a model3.37 Types of models (staged and continuous)3.38 Model selection process3.39 Using models for assessment and baselines
3.40Industry Quality Models
Software Engineering Institute Capability Maturity Model/CMMI3.41 Malcolm Baldrige National Quality Award3.42 ISO 9001:20003.43 ISO/IEC 122073.44 ISO/IEC TR 155043.45 Post-implementation audits
Quality Assurance is a professional competency whose focus is directed at the critical processes used tobuild products and services. The profession is charged with the respon sibility for tact ical processimprovement initiatives that are strategically alig ned to the goals of the organization. This categoryaddresses the understanding and application of quality assurance practices in support of the strategicquality direction of the organizati on. The quality practitioner should understand the importance of aquality function, how to implement a quality function and how it matures over time, as well as how tocreate a quality plan, the use of quality tools, process deployment, and differentiating between internalauditing and quality assurance.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 4 – Quality Assurance Competency RatingSkill # Skill Description Full Some None
4.46Establishing a Function to Promote and Manage Quality
The challenges of implementing a quality function4.47 How the quality function matures over time4.48 Support in corporate quality management environment4.49 Implementing an IT quality function
Getting buy-in for change through marketing4.54 The formula for effective behavior change4.55 The deployment process4.56 Critical success factors for deployment
4.57Internal Auditing and Quality Assurance
Types of internal audits4.58 Differences in responsibilities
Executive management establishes th e vision and strategic goals of an organization. Planning is theprocess that describes ho w those strategic go als will be acco mplished. Quality planning should beintegrated into the IT plan so that they become a single plan. In simplistic terms, the IT plan representsthe producer and the quality plan represents the customer.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 5 – Quality Planning Competency RatingSkill # Skill Description Full Some None
5.59Planning Concepts
The management cycle5.60 The planning cycle
5.61Integrating Business and Quality Planning
The fallacy of having two separate planning processes5.62 Planning should be a single IT activity5.63 Prerequisites to Quality Planning
5.64The Planning Process
Planning process overview5.65 The six basic planning questions5.66 The common activities in the planning process
5.67Planning to Mature IT Work Processes
QAI model and approach to mature IT work processes5.68 How to plan the sequence for implementing process maturity
Skill Category 6 – Define, Build, Implement and Improve Work Processes
The world is constantly changi ng. Customers are more knowledgeab le and demandin g, therefore,quality and speed of delivery are now critical need s. Companies must constantly improve their abilityto produce quality products th at add value to thei r customer base. Defi ning and continuouslyimproving work processes allows th e pace of change to be maintained without negatively impactingthe quality of products and services. This category addresses process management concepts, includingthe definition of a process, th e workbench conc ept and components of a process. Additionally, itaddresses the understand ing of definitions and continuous im provement of a pr ocess through theprocess management PDCA cycle.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 6 – Define, Build, Implement and Improve Work Processes
Competency Rating
Skill # Skill Description Full Some None
6.69Process Management Concepts
Definition of a process6.70 Why processes are needed6.71 Process workbench and components6.72 Process categories6.73 The process maturity continuum6.74 How processes are managed6.75 Process template
6.76
Process Management ProcessesPlanning processes:Process inventory
Quality control practices should occur during pr oduct development, pro duct acquisition, productconstruction at the end of de velopment/acquisition and throu ghout product change and operation.During development, the quality control process is frequently called verification and at the conclusionof development, it is called validation. This category addresses the various types of controls and whenthey are best used in the process. The quality practitioner should also be familiar with verification andvalidation techniques, the framework for developing testing tactics, change control and configurationmanagement.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 7 – Quality Control Practices Competency RatingSkill # Skill Description Full Some None
7.85Testing Concepts
The tester’s workbench7.86 Test stages7.87 Independent testing7.88 Static versus dynamic testing7.89 Verification versus validation7.90 Stress versus volume versus performance7.91 Test objectives7.92 Reviews and inspections
7.93Developing Testing Methodologies
Acquire and study the test strategy7.94 Determine the type of development project7.95 Determine the type of software system7.96 Determine the project scope7.97 Identify the tactical risks7.98 Determine when testing should occur7.99 Build the system test plan7.100 Build the unit test plans
7.101Verification and Validation Methods
Management of verification and validation7.102 Verification techniques7.103 Validation techniques7.104 Structural and functional testing
7.105Software Change Control
Software configuration management7.106 Change control procedures
7.107Defect Management
Defect management process7.108 Defect reporting 7.109 Severity versus priority7.110 Using defects for process improvement
A properly established measurement system is used to help achieve mi ssions, visions, goals, andobjectives. Measurement da ta is most reliable wh en it is generated as a by-product of producing aproduct or service. The QA analys t must ensure that quantitative da ta is valued and reliable, andpresented to management in a tim ely and easy-to-use manner. Measurement can be used to gauge thestatus, effectiveness and efficiency of processe s, customer satisfaction, product quality, and as a toolfor management to use in their decision-making processes. This categor y addresses measurementconcepts, the use of measuremen t in a software development e nvironment, variation, processcapability, risk management, the ways measurement can be use d, and how to implement an effectivemeasurement program.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 8 – Metrics and Measurement Competency RatingSkill # Skill Description Full Some None
8.111Measurement Concepts
Standard units of measure 8.112 Metrics8.113 Objective and subjective measurement8.114 Types of measurement data8.115 Measures of central tendency8.116 Attributes of good measurement8.117 Using quantitative data to manage an IT function8.118 Key indicators
8.119Measurement in Software
Product measurement8.120 Process measurement
8.121Variation and Process Capability
The measurement program8.122 Installing the measurement program8.123 Common and special causes of variation8.124 Variation and process improvement8.125 Process capability
8.126Risk Management
Defining risk8.127 Characterizing risk8.128 Managing risk8.129 Software risk management8.130 Risks of integrating new technology
Privacy laws and increase d accessibility to data have necessi tated increased se curity. Accountingscandals and governmental regu lation such as the Sarbanes-Oxley Act have pl aced increasedimportance on building and mainta ining adequate systems of intern al control. The quality assurancefunction can contribu te to meeting those objec tives by assuring that IT h as adequate processesgoverning internal control and security.
Competency Rating Totals (total each “ ” in each column): ______ ______ ______
Skill Category 9 – Internal Control and Security Competency RatingSkill # Skill Description Full Some None
9.133Principles and Concepts of Internal Control
Internal control and security vocabulary and concepts9.134 Preventive, detective, and corrective controls
9.135Risk and Internal Control Models
COSO enterprise risk management (ERM) model9.136 COSO internal control framework model9.137 CobiT model
9.138Building Internal Controls
Perform risk assessment
9.139Building Adequate Security
Where vulnerabilities in security occur9.140 Establishing a security baseline9.141 Security awareness training9.142 Security practices
Skill Category 10 – Outsourcing, COTS, and Contracting Quality
Organizations can a ssign software developmen t work responsibilities to outside organizations bypurchasing software or contracting services; but they cannot assign the responsibility for quality.Quality of software remains an internal IT responsibility regardless of w ho builds the software. Thequality professionals need to assure that those quality responsibil ities are fulfilled through appropriateprocesses for acquiring purchased software and contracting for software services.
Competency Rating Totals (total the “ ” in each column): ______ ______ ______
Skill Category 10 – Outsourcing, COTS, and Contracting Quality Competency RatingSkill # Skill Description Full Some None
Assure completeness of needs requirements10.146 Define critical success factor10.147 Determine compatibility with hardware, operating system, and other
COTS software10.148 Assure the Software can be Integrated into Your Business System
Work Flow 10.149 Demonstrate the Software in Operation10.150 Evaluate People Fit10.151 Acceptance Test the Software Process
10.152Selecting Software Developed by Outside Organizations
Contracting life cycle10.153 Developing selection criteria
10.154Contracting for Software Developed by Outside Organizations
Contract negotiations
10.155Operating for Software Developed by Outside Organizations
Your personal plan of action should be to improve those skills included in the CSTE Common Body of Knowledge for which you were not fully proficient. You should make two lists of skills by Skill Category. The first list would contain those skills which you rated yourself as "not competent." The second list should be those skills which you rated yourself "some competency." Merge the lists by Skill Category then competency level. The end product would be for each Skill Category you will have a list of skills which you are not com-petent followed by a list of skills for which you have some competency. As you study for the les-sons devoted to those skill categories, continually refer to your list of skills for which you are not competent or have some competency so that you can focus your study effort on the skills in which you need improvement.
My Personal Plan of Action to Improve These Skills
Appendix B - Topics for Quality AnalystsThe answers to these essay questions can be found starting on page 321. 1. What is COQ (Cost of Quality) and how it relates to software quality and testing?2. What is the PDCA (Plan, Do, Check, Act) and how it affects test activities.3. What is a risk matrix and be familiar with CSF's (Critical Success Factors)4. What is a test strategy and who participates in the development of it?5. Define the differences between tactical and strategic testing activities.6. Define root cause analysis and also explain the difference between common and special
causes of variation.7. Define a test policy and why it is important.8. Define "structured" approach to testing and explain the V-model concept.9. Define "how much testing is enough".10. Define the difference between verification and validation test activities.11. Name two structural and two functional test activities.12. Define a workbench and explain the four main components.13. How is maintenance testing different from new development testing and how is regression
testing different?14. What is the single most valuable static test technique available to testers?15. How do you differentiate a spec from a requirement?16. What are 6 of the main components of a test script?17. What are the 3 components of a test case? 18. What is the difference between a test case and a test script?19. How are internal controls built into software systems and subsequently how are these con-
trols tested?20. Name at least 3 types of test tools (not vendors).21. Explain the difference between QA and QC.22. Give an example of a test standard.23. Define 5 major components of a system test plan.24. What are the drawbacks of using production data to test?25. What are 5 major components of a good test report?26. What are defects and how are they managed and eliminated?27. What is a Pareto Chart and how is it used in testing?28. What is an Ishakawa diagram and how is it used in testing?29. Give a brief summary of how you would justify the expense of continuing to test versus stop-
1. COQ – Cost of Quality is all costs OTHER than the actual cost to build the software itself. It includes the cost of prevention, (QA activities), appraisal (testing or QC) and failure costs. It is usually as large or larger than the cost to build the software in the first place. Why it’s important to quality and testing is that most organizations spend far too much money in test-ing and not enough in preventive activates (reviews, walkthroughs, etc.)
2. PDCA is attributed to Dr. Edwards Deming, a quality guru from the 1940’s. First you plan your work, then you do the work, then you check to see that you did the work according to your plan, and then you act on what you found (measure and improve). It is a concept well suited to test planning and execution and finally, the follow-up and reporting.
3. A risk matrix is developed by determining which critical success factors are the most impor-tant to the customers during the project being tested. For example, if usability is critical to the client then usability testing should be the primary concern in the test effort in order to ensure a successful project implementation for that client. These usability tests are then found first on the traceability matrix. Other CSF’s include portability, performance, main-tainability, scalability, ease of use, conformance to standards or regulations, etc.
4. The client, the tester, the developer (or application manager) and project manager should par-ticipate in the development of a test strategy. The basic components include the type of soft-ware being testing, the type of project (mainframe, OO etc.), who is to be involved in the testing, what levels of testing are to be conducted, where the testing will be conducted, the environment in which the testing will occur, the risks, the assumptions, the critical success factors for the project, any tradeoffs that might be negotiable (time, scope, etc.),
5. Strategic activities are the high-level planning (where are we headed) like the master test plan and strategy documents. Tactical testing actives would include test plans at the developer level, system specific test plans, etc. (the how are we going to accomplish this activities.)
6. Root cause analysis involves getting to the first place the defect was injected. For example, many defects found in testing are really attributable to faulty requirements. Common causes of variation in testing are things like forgetting to run a series of test cases. Once identified this type of variation can be eliminated by instituting good standards for the test effort. Spe-cial causes are things that happen unexpectedly and infrequently and are hard to plan for. For example, if the operating system changes and no one tells the testers then the tests may not work. This situation is not one that would usually happen and as such is not usually part of the test plan process.
7. A policy is management’s articulation of behaviors they expect employees to follow. Most companies have policies on vacation and casual day attire but not for things like testing, or requirements gathering so people do whatever they think is correct with no guidance from management on the issue. A policy is high-level and strategic. It doesn’t state exactly WHAT to do, but says WHAT should be done. It does show that management is concerned about the subject enough to write a policy and expect compliance. In testing a simple test policy might be stated like this:
“Depending upon the risk score of each project, specific levels of testing must be con-ducted.” What this assumes is that there is a risk assessment process and that once it has been conducted then there are levels of testing that must be undertaken. When you get to the standards that support this policy, then one of the standards might say:
“If the risk score determination is high, then unit, integration, system, performance, and user acceptance testing must be conducted. If the risk score is medium, then unit, system, and user-acceptance testing must be conducted and if the risk score is low then only system and user- acceptance testing must be completed.” What this does is support the policy with “how” to enforce the policy. The procedures would then go on to define how to write a unit test plan, etc.
8. A structured approach means defining the levels of testing and also the accountabilities for each level. I will send you a copy of the V-model if you send me your fax number.
9. Testing is complete when it has been done according to a documented test plan and no addi-tional defects can be found. In other words, not just when the time is up. It means that the critical success factors have been defined and included in the test plan and once they have been adequately tested and are working then the testing can be assumed to be complete. Also, this statement assumes a level of competency in testing that can “predict” the number of defects that will be found and even where they will be found. Only very mature test organizations are actually able to do this.
10. Verification tests processes; validation tests products. This is easily seen on the V-model.11. Two structural test activities:executing branch coverage analysis, or simply running unit tests
to test logic. Functional test activities: conducting a walkthrough and running functional requirements tests (not logic-based).
12. Workbench is another name for a process. The four main components are: input, output, do and check.
13. Maintenance is every bit as important as new development testing in that there is a produc-tion system that is being used by perhaps thousands of clients so fixing problems is of the utmost concern. Regression testing for maintenance systems requires that any defects found previously are retested each and every time the regression test is run. Sound regression testing requires good tools to determine the paths taken through the code so that at a mini-mum these paths can be covered by test cases in the event that test time is limited.
14. Reviews 15. A requirement comes directly from the customer and is the “what” the client wants in the
project. A specification is the restatement of the functional requirement into “pseudo-code” or language that is used by the programmers to develop or change code. It is much more detailed than are usually a requirements document.
16. Log on, log off, security, the test cases, restart procedure, and how to handle an abnormal occurrence.
17. Condition to be tested, the procedure to run the test, and the expected result.18. A test case is usually very simple and is run manually. A script is done in an automated test
environment and can contain hundreds of test cases.19. Internal controls are developed to address the riskiest areas of a software process. For exam-
ple, input and output criteria are good sources of controls over what goes into a process before the process itself even starts. In very critical systems like financial or human-life sys-tems, management should determine which systems and which processes should have addi-tional controls. The Sarbanes-Oxley Act puts heavy burden now on a company’s system on internal controls.
20. Test case generator, code coverage analyzer, inspection.
21. Quality Assurance means defect prevention; Quality Control is defect detection.22. A test standard might state that “ At least 90% of all test cases must execute correctly before
moving to the next level of test”.23. Functions to be tested, test environment, tools to be used, constraints, how defects will be
handled.24. Production data contains errors. Sometimes we know how production data got “dirty” and
sometimes we don’t. Using it to test can be ok if we understand the extent to which it is dirty and add / delete/ or change data to make the testing most effective.
25. Test report would contain, at a minimum: the results of the testing (which functions work and which do not), what risks to implementation remain, the recommendation of the test team, outstanding defects, and recommendations for improvement as noted by the test team (usually become enhancements).
26. A defect is any nonconformance to requirements. The problem is that requirements are not well defined and therefore just about anything (or nothing) could honestly be considered a defect. A problem is just the word itself. Many organizations won’t use the term “defect”.
27. Pareto is based on the 80/20 rule: 80% of the problems will come from 20% of the work. Using root cause analysis for defect identification, for example, you could show on a Pareto Chart which of the defects occur most frequently and in which are of testing do they show up.
28. An Ishakawa diagram is another name for a fish-bone diagram.29. By conducting a requirements review and identifying an classifying the defects found during
the review you could easily see that finding just one fatal defect in the requirements review would have prevented that same defect from finding its way into the testing where it is most expensive to find (or worse, into production to affect the customers). Put a dollar amount to this fatal defect and it usually gets managements attention since the cost of conducting the review is usually far less than the estimated impact of the defect.
Appendix C - Critical Success FactorsAll systems have processes they perform and data they use and maintain. All systems also have certain attributes they must possess if the system is to be considered a success.Critical success factors are usually a combination of what the system must do (processes on data) and what the system must be (attributes) to be successful. Testing needs to identify and focus on verifying and validating that these critical success factors are met.We can identify general success factors as a basis for our test strategy as well as specific concerns testing should address associated with each factor.Critical Success Factors
Shown here the fifteen success factors you should consider during your test strategy plan-ning:
² Correctness - Assurance that the data entered, processed, and outputted by the application system is accurate and complete. Accuracy and completeness are achieved through controls over transactions and data element, which should commence when a transaction is originated and conclude when the transaction data has been used for its intended purpose.
² File integrity - Assurance that the data entered into the application system will be returned unaltered. The file integrity procedures ensure that the right file is used and that the data on the file and the sequence in which the data is stored and retrieved is correct.
² Authorization - Assurance that data is processed in accordance with the intents of manage-ment. In an application system, there is both general and specific authorization for the pro-cessing of transactions. General authorization governs the authority to conduct different types of business, while specific authorization provides the authority to perform a specific act.
² Audit trail - The capability to substantiate the processing that has occurred. The processing of data can be supported through the retention of sufficient evidential matter to substantiate the accuracy, completeness, timeliness, and authorization of data. The process of saving the supporting evidential matter is frequently called an audit trail.
² Continuity of processing - The ability to sustain processing in the event problems occur. Continuity of processing assures that the necessary procedures and backup information are available to recover operations should integrity be lost due to problems. Continuity of pro-cessing includes the timeliness of recovery operations and the ability to maintain processing periods when the computer is inoperable.
² Service levels - Assurance that the desired results will be available within a time frame acceptable to the user. To achieve the desired service level, it is necessary to match user requirements with available resources. Resources include input/output capabilities, commu-nication facilities, processing, and systems software capabilities.
² Access control - Assurance that the application system resources will be protected against accidental and intentional modification, destruction, misuse, and disclosure. The security procedure is the totality of the steps taken to ensure the integrity of application data and pro-grams from unintentional and unauthorized acts.
² Compliance - Assurance that the system is designed in accordance with organizational strat-egy, policies, procedures, and standards. These requirements need to be identified, imple-mented, and maintained in conjunction with other application requirements.
² Reliability - Assurance that the application will perform its intended function with the required precision over an extended period of time. The correctness of processing deals with the ability of the system to process valid transactions correctly, while reliability relates to the system’s being able to perform correctly over an extended period of time when placed into production.
² Ease of use - The extent of effort required to learn, operate, prepare input for, and interpret output from the system. This test factor deals with the usability of the system to the people interfacing with the application system.
² Maintainability - The effort required to locate and fix an error in an operational system. Error is used in the broad context to mean both a defect in the system and a misinterpretation of user requirements.
² Portability - The effort required to transfer a program from one hardware configuration and/or software system environment to another. The effort includes data conversion, program changes, operating system, and documentation changes.
² Coupling - The effort required to interconnect components within an application system and with all other application systems in their processing environment.
² Performance - The amount of computing resources and code required by a system to per-form its stated functions. Performance includes both the manual and automated segments involved in fulfilling system functions.
² Ease of operation - The amount of effort required to integrate the system into the operating environment and then to operate the application system. The procedures can be both manual and automated.
Test Concerns Matrix The table below identifies specific concerns testing should address associated with each factor.
Appendix D - Critical Skills QuestionsAnswers to these questions can be found starting on page 338.
COMMUNICATION
Giving/Receiving Information1. List five suggestions for relaying defect information and/or performing criticism effectively.
2. What are three reasons why people don’t listen well?
3. Give one example of an effective audience evaluation technique that can be used to level-set a testing presentation or design a course for test or development staff.
4. How do you know if the information you provided to an audience has been understood and communicated effectively?
5. Give three examples of pertinent information that should always be included with defect reports.
Personal Effectiveness1. You are the quality assurance manager at the APEX company. One of your quality assurance
analysts is engaged in a dispute with a developer over the cause of a defect reported by the analyst. What can/should you do?
5. What are the three most common phase-end reviews?
6. Post-implementation reviews are held when:a. The product is certified for releaseb. The product is replacedc. The product has been releasedd. There is a major failure
Test Management1. Draw a test process workbench.
2. List five skills a competent tester should have.
3. Give an example of a system objective and its associated test objective.
Test Standards1. List three industry test standard organizations.
Have the facts.Be prepared to help the person improve.Follow a process when giving criticism.Be specific on expectations
2. The speaker may not be talking about a topic of interest to them.
They are impatient and have a lot of other stimuli going through their minds.
hey are too busy rehearsing what they will say next.
3. Do an internal self-assessment (for a training course) or ask questions of the audience before and after a presentation
4. You ask questions, and conduct an audience evaluation immediately.
5. What type of defect was found?Where was the defect found?What is the severity level of the defect?
Personal Effectiveness1. You should discuss the defect report with the QA analyst to ensure that the information is
correct and that a root cause analysis was conducted. If it is clearly a case of developer error then have both people discuss and come to an effective resolution. Record the defect and sta-tus.
PROFESSIONAL DEVELOPMENT
Continuing Education1. Prepare for the CSTE exam.
Join a local professional association and actively participate.Read journals and magazines about the testing profession.
2. A tester can induce changes into the development process by providing both qualitative and quantitative data derived from the testing process. With this information and metrics, changes can be identified and introduced into the development process that will ultimately benefit the organization. Information like types and frequency of defects found, where in the develop-ment process they were found, the on-time delivery capability, etc. can assist in improving the overall quality of the delivered products.
Leadership and Recognition1. The main purpose of a task force is to develop options—not to solve a problem.
2. To provide an unbiased focal point to a discussion and keep the discussion “on track.”
3. A tester should play a major role in test process definition since the test process is his/her focus. The tester should review the current test process with an objective of inserting improvements and then should pilot the process and provide feedback and suggestions and/or recommendations. If a new test process is being developed then the tester is the best person to ensure that best testing practices are used.
QUALITY PRINCIPLES AND CONCEPTS
Quality Principles/Quality Management1. COQ is everything other than the true cost to build a product. It includes failure and
appraisal/prevention costs.2. MBF means that the specified results should be qualitative and the processes should provide
adequate quantitative data as a by-product of the process to determine whether or not the specified results are being achieved.
3. 14
4. 5
Quality Assurance/Quality Control1. QA activities are used to prevent the introduction of flaws; activities that find and correct
flaws are QA activities.
2. AccuracyEase of useService levelsPortabilityFile integrity
METHODS FOR SOFTWARE DEVELOPMENT AND MAINTENANCE
Process Knowledge1. Manual tools do not execute program code and do not require executing software; automated
tools do.
2. Manual – Reviews and checklistsAutomated – Code coverage analyzers and file comp
2. Test process knowledge;Excellent written and oral communication skills;Analytical ability;Knowledge of test tools; andUnderstanding of defect management.
3. System objective – System must be portable to both unit and NT operating systems.Test objective – Parallel test with some results on both platforms.
Test Standards1. IEEE
ISODoD
2. All levels of testing will prepare a test plan.Complete test documentation must be available before implement.All Category 1 and Category 2 defects will be resolved before implementation.
Test Environment1. Test beds are developed for a singly type of test; test scripts are an automated capture/play-
back tool and execute over and over.
2. Tools, plat forms, scripts, hardware, software, resources, and skills needed to support testing.
RISK ANALYSIS
Risk Identification1. Technology integration;
Size and complexity;System environment and stability;Criticality / mission impact; andReliability and integrity.
2. LU1 criticality / mission impact.LU2 size, system environment, reliability, and integration.
3. Risk analysis will identify high-risk applications in order to at them to more extensive testing and it can help focus testing on the critical components and/or quality dimensions that are most important to the project.
3. Yes, otherwise many test issues fall through the cracks.
4. Entrance criteria – Minimum standards to be met before product will be tested.Exit criteria – Minimum standards to be met before the product is released.
Test Plan1. Software description, milestones, schedule, test materials, test procedures, test cases, test
methods, and test assumptions.
2. True
TEST DESIGN
Design Preparation1. Assuring that the application has been covered by the test process.
2. Test beds are files/tables needed to test a system/unit whose contents have been predefined to meet the conditions defined in designing test cases.
Design Execution1. Scripting
Test bedsProduction data
2. Test caseFunctions to be testedTest performedSet-up proceduresValidation procedures
Execute tests1. False. Execution of a test can occur throughout the project lifecycle if a lifecycle approach to
testing is used.
Compare actual versus expected results.2. The condition, the expected result, and the procedure to run the test.
Test Logs1. As each part of the test plan is executed, control can be maintained by using a document cre-
ated to record test activities. A simple worksheet or spreadsheet can be used.
2. The test identifier, the test activity, who executed the test, the start and stop time of the test, and remarks or comments.
Record Discrepancies1. The test plan, the test cases, and the test logs.
2. Missing, wrong, or extra
DEFECT TRACKING AND MANAGEMENT
Testing Defect Corrections1. QA and developer and/or area which defect was found.
2. True. Regression testing revalidates that changes made to the system do not adversely influ-ence other system functions.
QUANTITATIVE MEASUREMENT
Test Completion Criteria/Metrics1. They are used to show the extent to which the logic in the program was executed during test-
ing.
2. User participation;Number of paths tested;Cost of all testing;Number of defects found; andNumber of requirements tested.
3. LOC Analysis
TEST REPORTING
Reporting tools and standards1. Description of test results and findings (defects);
Summary (environment and references); and Recommendation.
2. Can be used to compare actual number of defects found to the number of defects expected or to indicate the number of functional requirements actually tested.
1. The producer’s view of quality is:a. Meets requirements b. Fit for usec. Projects completed on-timed. Projects completed within budget
2. Quality assurance is:a. Product focusedb. Meant to prevent defects c. A line positiond. Software testing
3. Quality control is:a. Product focused b. Meant to prevent defectsc. A staff positiond. Software testing
4. Quality assurance is responsible for all of the following except:a. Quality planningb. Measurement and analysisc. Telling technicians how to perform their duties d. Facilitation
5. Quality control techniques may include all the following except:a. System testingb. Integration testingc. Inspectionsd. Defining standards and procedures
6. To calculate the Cost of Quality you must add all the following costs except:a. Cost to build the product b. Failure costsc. Appraisal costsd. Preventive costs
7. Software testing is _____________.a. A preventive cost of qualityb. An appraisal cost of quality c. Is not part of the cost of quality calculationd. The Cost to build the product
9. A policy statement answers the question:a. Whatb. Howc. Why d. When
10. We test software for all the following reasons except:a. To reduce riskb. To validate requirements workc. To establish confidence that a program worksd. To verify run times
11. Process improvement is most effective when it is performed by:a. Owners of the processb. Quality assurance personnelc. Independent consultantsd. Internal auditors
12. Functional tests validate:a. System quality characteristicsb. System requirements c. Systems structured. Performance requirements
13. White Box testing is used to validate:a. Business requirementsb. Interfaces to other systemsc. Program structure d. Usability
14. Which of the following would not be considered a critical success factor in acquired soft-ware?a. Ease of useb. Expandabilityc. Reputationd. Reliability
15. Acceptance testing is normally designed and performed by:a. The developerb. The quality control managerc. The product development managerd. The user
16. Black box testing is used to validate:a. Business requirements b. Interfaces to other systemsc. Program structure d. Usability
17. Black box testing is typically performed by:a. An independent test team and business users b. An independent test team and developersc. Developers and 3rd party testersd. Auditors and developers
18. The following are black box test techniques:a. Equivalence portioning and boundary analysis b. Error guessing and statement coveragec. Boundary analysis and path coveraged. Equivalence partitioning and decision point to decision point
19. How much testing is enough:a. When all scripts have been executedb. When the cost to detect defects exceeds the potential damage caused by the defect c. When the scheduled end date is reachedd. When testing detects no more defects
20. If a measure is considered valid it means that:a. It is easy to be obtainedb. It measures what it is intended to measurec. It is objectived. Two or more people can collect the same number
21. If you wanted to evaluate if the software system being acceptance testing fits the needs of the user; which components of fitting the users needs would you include in acceptance testing?a. Datab. Peoplec. Structure d. Rulese. All of the above
22. These model categories (leadership, strategic planning, customer and market focus, informa-tion and analysis, human resources, process management and business results) are part of which of the following quality models:a. Deming Prizeb. Malcolm Baldrige National Quality Awardsc. SEI CMMId. ISO 9000 Standards
23. Unit testing is:a. Performed by the developer and validates system requirementsb. Performed by the test team and validates system requirementsc. Performed by the developer and validates the structure of one program module or com-
ponent d. Performed by the test team and validates the structure of one program module or compo-
nent
24. Which recognized organization has developed the generally accepted model for internal con-trol:a. AICPA b. COSOc. CobiTd. ERME ISO
25. The “V Model” is a life cycle approach that integrates:a. White box and black box testingb. Functional and non-functional test techniquesc. Verification and validation techniques d. Boundary value analysis and equivalence partitioning
26. Reviews are conducted for all the following reasons except:a. Reduce time to marketb. Evaluate individual performance c. To detect defects in the phase where it was introduced into the productd. Knowledge transfer
27. Enforcement of meeting standards is the responsibility of:a. Managementb. Quality assurancec. Quality controld. Auditors
28. Which of the following is considered a subjective measure:a. Lines of codeb. Mean time to failurec. Hours to write a thousand lines of coded. Customer satisfaction
29. A loss associated with a customer leaving your website because it was too difficult to use was estimated to be$125 per occurrence and assume that happens 4 times per work day. If the site will only be active for one 5 day week, what is the maximum amount you could spend on controls if you could totally eliminate the problem of your customers leaving your website?a. $500b. $2,500c. $5,000 d. $13,000
30. Testing software developed by a third party company is best called:a. Microsoft Worm (MSW)b. Commercial-on-the-Shelf (COTS)c. Commercial-off-the-Shelf (COTS)d. Commercially Procured Software (CPS)
Mr. Mason is the IT director of your department. Mr. Mason has decided to make quality the number-one objective of the department. To accomplish this objective, Mr. Mason set some very specific quality goals, including improved customer satisfaction and reduced defect rates. To accomplish this objective, Mr. Mason established a documentation librarian to assure that all doc-umentation meets standards.
Mr. Mason requires computer operations to review all new systems and changes prior to turnover to ensure they meet standards. Mr. Mason has also established an independent test group to vali-date that all systems function correctly prior to turning them over to production. Based on what you read, evaluate how good a quality leader you believe Mr. Mason is, and state whether you believe his approach is based on quality principles, and why.
Appendix G - Quiz 21. This view of quality means that the requirement has not been met:
a. Provider viewb. Producer viewc. Customer viewd. Supplier view
2. This view means that the product or service has met the needs regardless of the requirements:a. Provider viewb. Producer viewb. Customer viewc. Supplier view
3. Most IT groups have two quality gaps. They are:a. Producer and providerb. Provider and customerc. Producer and customerd. Supplier and customer
4. Which statement BEST defines the producer gap of quality:a. The difference between what is specified versus what is specified to managementb. The difference between what is specified versus what is actually deliveredc. The difference between what is delivered versus what is specified to managementd. The difference between what is delivered versus what is specified by the customer
5. The extent to which a program satisfies its specifications and fulfills the user’s mission objec-tives is called:a. Correctnessb. Efficiencyc. Usabilityd. Flexibility
6. Effort required to locate and fix an error is a program is called:a. Integrityb. Efficiencyc. Usabilityd. Maintainability
7. This view of quality is based on that of the organization that provides the producer with prod-ucts or services:a. Provider viewb. Producer viewc. Customer viewd. Supplier view
8. Identifying defects in a program for the primary purpose of correcting defects.a. Quality Assuranceb. Quality Control
9. Testing is a:a. Processb. Craft or artc. Specific scienced. Cost-effective initiative
10. This view of quality is based on that of the organization that delivers the product:a. Provider viewb. Producer viewc. Customer viewd. Supplier view
Quality factors have frequently been used as a basis for measuring the quality of an information system. Listed below are four of those quality factors.
What measure or metric would you recommend as an effective measure or metric for each of these quality factors?
Appendix H - Quiz 31. In a cost of quality analysis, the costs required to avoid errors to do the job right the first time is called:
a. Appraisal costsb. Failure costsc. Prevention costsd. Direct costs
2. The P in the PDCA stands for:a. Preventb. Predictc. Pland. Pick
3. The cost of quality includes:a. Appraisal costsb. Testing costsc. Failure costsd. Both A and C
4. Three examples of appraisal activities are:a. Planning, organizing, and leadingb. Software testing, reviews, and inspectionsc. Budgeting, controlling expenses, and testingd. Cost controlling, leading, and coordinating
5. Which of the following is NOT a category of the Cost of Quality?a. Appraisal costsb. Failure costsc. Prevention costsd. Cost to Build
6. In defining the cost of quality, appraisal costs are BEST defined as:a. Costs incurred to review interim products against requirementsb. Costs which can not be recoupedc. All costs associated with defective productsd. None of the above
7. The cost of quality can be reduced by:a. Testing at specific points during developmentb. Testing mostly in the requirements stagec. Testing continuously throughout the SDLCd. Testing only at the User Acceptance Stage
8. Identifying defects in a program for the primary purpose of preventing defects.a. Quality Assuranceb. Quality Control
9. In the PDCA, training is provided during this piece:a. Planb. Doc. Checkd. Act
10. Retesting falls into which category of the Cost of Quality?a. Appraisal costsb. Failure costsc. Prevention costsd. Cost to Build
The computer operations department in your company established a help desk approximately two years ago. Since that time they have carefully recorded each user inquiry, what problems were reported, and the action taken by the help desk.
List and explain two ways that you could use that information to improve your software development pro-cess.
1. The leading authority and one who started the quality movement in Japan is:a. Thomas A. Eddisonb. W. Edwards Demingc. Ishia Hiaiakowad. Philip Crosby
2. In Deming’s first principle of “Constancy of Purpose” he is identifying the objective that:a. Once a process is established, don’t change itb. All employees must identify the purpose most meaningful to the companyc. Everyone must work toward a common goald. None of the above
3. Regarding Deming’s “constancy of purpose” principle, who is primarily responsible for setting this?
a. Subject Matter Experts (SME)b. Test Managerc. Executive Managementd. Project Manager
4. The reason Deming suggests a company to reduce the number of suppliers is to:a. Promote partnershipsb. Reduce vendor costsc. Develop long term relationshipsd. All of the abovee. None of the above
5. Quality is often referred to as being free. This is because:a. It’s difficult to quantify consumer satisfactionb. There is a never-ending goal to be achievedc. The payback is greater than its cost over timed. Of continuous improvement
6. Which of the following is NOT one of Dr. W. Edward Deming’s fourteen quality principles?a. Learn a new philosophyb. Drive out fearc. Zero defectsd. Look carefully at work standards
7. This quality guru is best known for his trilogy of quality planning, quality assurance, and qual-ity control.
a. Joseph Juranb. W. Edwards Demingc. Ishia Hiaiakowad. Philip Crosby
8. One of the most difficult aspects in installing an effective TQM program is that:a. It takes too much moneyb. It is difficult to get people to participatec. The company culture most often must changed. Managers do not understand the workers
9. Deming’s four activities of quality management are:a. Requirements, plans, execute, and checkb. Plan, check, do, and feedback loopc. Plan, do, check, and actd. Plan, organize, lead, and control
10. Total Quality Management (TQM) means:a. Managing the quality of entire systemb. Emphasizing the importance of managing qualityc. Implementing company-wide quality attitudes and methodologiesd. Having the Quality Manager included in senior management
2. Dr. W. Edward Deming stated 14 Quality Principles for organizations to use in becoming aworld class organization. Listed below are two of the principles. Explain what you believeis required to implement each of these two quality principles.
Appendix J - Quiz 51. What is normally NOT included on a problem identification worksheet that is prepared by
the tester?a. Name of the problem b. Discussion of the problemc. Screen prints of the problem d. Who caused the problem e. Severity of the problem
2. Which of the following is the correct sequence for the five phases of process maturity, as defined in the capability maturity model?a. Initial, repeatable, defined, managed, optimizingb. Initial, defined, repeatable, managed, optimizingc. Initial, managed, defined, repeatable, optimizing d. Initial, defined, optimizing, managed, repeatable
3. Experience has shown that over half of all defects originate in:a. Design b. Coding c. Testing d. Requirements e. Acceptance Testing
4. Some organizations have stated that testers make more defects in performing testing than developers make in building software. The most effective way to reduce tester's defects is to:a. Increase the importance of the testing processb. Apply continuous process improvement in the testing process c. Develop quality control checklists for testers to use d. Hire more competent testers e. Establish an inspection process to inspect test products
5. One way to minimize risk is to:a. Decrease acceptance testingb. Install Controls c. Increase the probability factord. None of the above
6. What type of testing is most closely associated with making system changes:a. Stress testing b. Regression testing c. Liability testing d. Correctness Testing
7. In-process reviews which are used to review the developmental work and identify defects, normally occur:a. Near the conclusion of a life cycle phase b. Preventive costc. At the end of the project d. When directed by test management e. After the project has been placed into production
8. Which of the following is not a component of a tester's workbench? (A workbench is a graph-ical representation of the testing risk.)a. Entrance criteria (inputs)b. Testing policies c. Exit criteria (deliverables) d. Work procedures e. Check procedures
9. Which statement best communicates the advantages of performing quality reviews?a. Reviews are conducted to prevent defects b. Results of the review process can be used to evaluate individual performance c. Reviews are conducted at various milestones to detect defects early in developmentd. Serving on a review team requires no special skills
10. Which is the correct sequence for the quality management infrastructure from lowest to high-est level?a. Teams, Management Committees, Quality Councilb. Quality Council, Management Committees, Teamsc. Management Committees, Teams, Quality Councild. Teams, Quality Council, Management Committeese. Quality Council, Teams, Management Committees
11. Which of the following types of testing is considered dynamic testing?a. Code inspections b. System reviewsc. Checklist d. Unit Testing e. Code reviews
12. Performance testing subjects a system to which of the following types of tests?a. Test scripts b. Large volumes of tests c. How fast the system responds d. Time spent fixing errorse. Error conditions
13. In a cost of quality analysis, the cost to run and maintain a help desk is called:a. Appraisal costb. Preventive costc. Failure cost d. Direct cost
14. Regression testing can be performed in which test stage(s)?a. Unit testingb. String/integration testingc. System testing d. Acceptance testing e. All of the above
15. Three parts of the PDCA cycle are act, plan and do. The fourth part of the PDCA cycle is:a. Contribute b. Calibrate c. Checkd. Create e. Condone
16. Which activity verifies whether a specific attribute(s) is, or is not, in a specific product or ser-vice?a. Quality Control b. Quality Assurance c. Computer Operationsd. Help Deske. Internal Auditor
17. Structural testing and functional testing typically require the same sets of experience, skills, and knowledge.a. Trueb. False
18. White-box testing is primarily:a. Logic driven b. Data drivenc. Bottom-up Drivend. Top Down Drivene. Defect Driven
19. The Quality Assurance is the function responsible for:a. Testing Software b. Writing Processes c. Managing Quality d. Conducting Reviews and Inspectionse. All of the Above
20. When we say that quality assurance must be a staff function, we mean it should:a. Take a direct position in detecting defects b. Report to the lowest executive c. Have direct supervisory responsibility for quality controld. Serve in an advisory capacity for quality, training, planning, measurement, and process