#PhUSE
Standard Scripts Project 2
2014 - Proposal for Qualification of
Standard Scripts
#PhUSE
Main Sections
Summary of prior proposal, 2013• hidden slides, to preserve history
Updated proposal, July 2014
#PhUSE
Main Sections
Summary of prior proposalUpdated proposal– Objectives– Definitions: Qualification, States, Roles– Metadata and Test data– State Transitions
#PhUSE
Proposal 2014 - Objectives• For End-users
– Clear overview of resources available, incl. purpose & state of each– Inspire confidence from first user experience– Ease-of-use:
• clear messaging from scripts• Reproducible results in users’ own environments• Consistency of scripts, learning first one makes remaining familiar
– Ease of converting users to contributors
• For Contributors & Standard Scripts Team– Ease of contribution: Modularize & standardize workflows & checklists– Modularize core components (e.g., FUTS for dependency checking?)– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)– Centralize & consolidate information & results
#PhUSE
Qualification Proposalmeaningful terms in blue
http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx
• Qualification Instructions – See template on PhUSE Wiki, link to file "Qualification of CSS scripts.xlsx"– Certification applies to new scripts and functionality– Confirmation applies to updates of existing script
• States:
Contributed, Develop, Review, Qualified• Roles
– Contributor: Anyone with appropriate skills & interests– Developer: A volunteer in CSS Working Group 5, familiar with our objectives**– Tester: A volunteer in CSS WG 05 who is familiar with our objectives**– Environment Tester: Anyone in community who is able to set up automatic test
replication in their work environment– Reviewer: Author of white papers, designers of script targets**
** suggests a quick-start onboarding page in CSS Phusewiki
#PhUSE
Qualification Proposal• Metadata for scripts should indicate:
YML proposal– Whitepaper & output ID uniquely identify the target
<Script: Source, Title, Target>
– Programming language & version
<Language: name, version>
– Type of output:
<Script: Type>
– Study design:
<Script: StudyDesign>
– State of qualification <Stage>:
<Qualification: Stage>
– OS is not included, since should be indicated in OS compatibility report• Test Data requirements
– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each script– include expected results from these data for defined tests/checks
#PhUSE
Qualification Proposal• Transitions
"Contributed" is the original State of all scripts– to Develop, checklist includes
by Developer & Reviewer• R & D confer on suitability of contribution. Suitable starting point?
[ may require analysis details, specs, from contributor ]• D reviews components (checklist to be completed)• D works with Contributor to complete minimum components
[ including Test Data and Coverage of defined tests ]• D adds standard parameter, dependency checking• D writes Qualification of CSS scripts.xlsx, see PhUSE Wiki Qual Workbook
– to Review, checklist includes
by Tester• Review Qualification instructions, consider coverage of tests• Execute Qualification instructions• Work with Developer to complete execution successfully
#PhUSE
Qualification Proposal• Transitions continued
– to Qualified by Tester & Environment Tester & Reviewer• T updates reference test outputs from certification/confirmation• E updates & executes local tests (posting PASS/FAIL results)• R confirms script output matches intention• R confirms qualification process covers important elements and
considerations. • R also provides user (rather than technical) feedback?• Achieve "Qualified" state when all tests in all test environments PASS (i.e.,
match outputs that T has certified and/or confirmed) and that R agrees that target is achieved
#PhUSE
Qualification Proposal• Efforts Required
– Top priority• Finalize Qualification states, roles, workflow, checklists, and templates• Code criteria for "acceptable" (link to GPP, PhUSE)• Test definition, Test confirmation (also from White Paper team)• Test data
– Next priorities • Design test structure in google code• Develop scripts that will allow Environment Testing• Develop general components (e.g. parameter, dependency checking)• Identify Environment Testers based on
– Host environment– SAS or R version
• Identify opportunities to automate qualification. E.g.,– Environment Testers to post results back as machine readable– Script green-light/red-light qualification matrix on Phusewiki
#PhUSE