8th Sakai Conference 4-7 December 2007 Newport Beach What does Quality What does Quality Assurance Mean to Sakai? Assurance Mean to Sakai? Alan Berg Megan May Seth Theriault
Dec 30, 2015
8th Sakai Conference
4-7 December 2007Newport Beach
What does Quality Assurance What does Quality Assurance Mean to Sakai?Mean to Sakai?
Alan Berg
Megan May
Seth Theriault
TopicsTopics• What does a QA Director do?• QA Today
– Who is in the WG– Types of testing currently done and release activities– Strides forward
• Resources available – QA Network
• 2.5 Release Activities Update• QA Tomorrow• Questions
Role of QA DirectorRole of QA Director
• Coordination! – Help bring parties interested in the same testing
efforts together– Foster initiatives that support quality throughout
the community– Assist in bridging the gap between test and
design & development teams– Ensure release activities are transparent
QA Today QA Today
Development
Testing & Software Release
Requirement Gathering and
Design
Focus on Functional Testing
Who is the QA WG?Who is the QA WG?
2.4.0
93 People
37 Institutions
11 Countries
2.2.1
23 People
10 Institutions
4 Countries
2.3.0
44 People
17 Institutions
5 Countries
2.5.0
56 People
17 Institutions
4 Countries
. . . How do they get involved?. . . How do they get involved?
Strides ForwardStrides Forward
• Deployment testing on wider variety of platforms
• Integration of Accessibility WG activities in general QA cycle
• Expanding/updating library of test scripts• Reflection on Release process
– http://confluence.sakaiproject.org/confluence/x/fbU
QA NetworkQA Network
Amsterdam Boston U
Cambridge Cape Town
Columbia Indiana
rSmart Georgia Tech
Coming soon servers in Japan and Australia
CollabCollabHow to be in the know!How to be in the know!https://collab.sakaiproject.org/portal
JiraJirahttp://jira.sakaiproject.org/
ConfluenceConfluencehttp://confluence.sakaiproject.org/confluence/display/QA/Home
Perceived Quality LevelPerceived Quality LevelDevelopment
Management
QA
Support
Overall Perceived level of Quality
Quality, like beauty, is clearly in the eyes of the beholder.
Perceived Quality PollPerceived Quality Poll
User Perceptions
QA Tomorrow QA Tomorrow
• Sakai Community Concerns – Several prominent campuses experience serious
performance problems with 2.4
• 2.5 Release Update– Further refinement of process– Goals for 2.5
Paths ForwardPaths Forward
• Common set of testing vocabulary – Unit Testing– Integration testing– System Testing– Performance
• Load testing ([email protected]) • Integration testing
Development
Testing & Software Release
Requirement Gathering and Design
http://confluence.sakaiproject.org/confluence/x/M4B5
Roadmap for TomorrowRoadmap for Tomorrow
Metrics to Describe QualityMetrics to Describe Quality
2.3.0 – 85% Verification
2.2.0 – 85% Verification
(1,345 JIRA Tickets)
(611 JIRA Tickets)
(504 JIRA Tickets)
2.4.0 – 82% Verification
2.5.0 – Past bug verification
Metrics to Describe QualityMetrics to Describe Quality
Unit Integration System PerformanceKernelAssignmentsGradebookResources
Static code ReviewStatic code Review• Search for bug patterns• What to process: Source code, Binary• Two types of bug pattern
– Generic• Classic (buffer overflow, input validation)
– Contextual• Language specific• Framework specific• Configuration specific
• Proximity effect
Hinting nightlyHinting nightly
http://qa1-nl.sakaiproject.org
Random ExampleRandom Example
Duplicate CodeDuplicate Code
Do we wish to use the reports more Do we wish to use the reports more aggressively?aggressively?
• Are the developers taking note?– Bugs reported are not always correct– Bugs found not always important– But duplicate code, unit test coverage, and failing to deal properly
with exceptions hint strongly at quality.– Not enough Contextual searching for Sakai and spring
configuration specific issues– Reports do not explain enough– Answer probably not, but thankfully trends in the marketplace will improve
the quality of results overtime.
A year onA year on
• Update prototype– Less noise more explanation– Graphical tracking of change
• Recommend Eclipse Plugins
• Look at emerging products
• See as part of a series of QA processes.
• Agree on metrics to act on and then act.
A good book to readA good book to read
• Secure Programming with Static Analysis– Brian Chess, Jacob West
• Addison Wesley
• ISBN: 9780321424778
• Includes CD with software and examples.
What you can doWhat you can do• Pledge Resources
– What happens in the community has a direct local impact– Easy point of entry into Sakai– All skill sets are needed
• Discussion– Why hasn’t this been a priority?– What are obstacles in the way and how do we remove
them?
Questions
Other Sessions of InterestOther Sessions of Interest
• 2.5 QA Workshop (Thursday, 13:00 - 14:30 in Salon 5)
• The Automated Testing Conundrum in Sakai (Thursday, 15:10 - 16:10 in Salon 5)