WG5 P02 Proposal 2014 Qualification of Standard Scripts

#PhUSE

Standard Scripts Project 2

2014 - Proposal for Qualification of

Standard Scripts

#PhUSE

Main Sections

Summary of prior proposal, 2013• hidden slides, to preserve history

Updated proposal, July 2014

#PhUSE

Main Sections

Summary of prior proposalUpdated proposal– Objectives– Definitions: Qualification, States, Roles– Metadata and Test data– State Transitions

#PhUSE

Proposal 2014 - Objectives• For End-users

– Clear overview of resources available, incl. purpose & state of each– Inspire confidence from first user experience– Ease-of-use:

• clear messaging from scripts• Reproducible results in users’ own environments• Consistency of scripts, learning first one makes remaining familiar

– Ease of converting users to contributors

• For Contributors & Standard Scripts Team– Ease of contribution: Modularize & standardize workflows & checklists– Modularize core components (e.g., FUTS for dependency checking?)– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)– Centralize & consolidate information & results

#PhUSE

Qualification Proposalmeaningful terms in blue

http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx

• Qualification Instructions – See template on PhUSE Wiki, link to file "Qualification of CSS scripts.xlsx"– Certification applies to new scripts and functionality– Confirmation applies to updates of existing script

• States:

Contributed, Develop, Review, Qualified• Roles

– Contributor: Anyone with appropriate skills & interests– Developer: A volunteer in CSS Working Group 5, familiar with our objectives**– Tester: A volunteer in CSS WG 05 who is familiar with our objectives**– Environment Tester: Anyone in community who is able to set up automatic test

replication in their work environment– Reviewer: Author of white papers, designers of script targets**

** suggests a quick-start onboarding page in CSS Phusewiki

#PhUSE

Qualification Proposal• Metadata for scripts should indicate:

YML proposal– Whitepaper & output ID uniquely identify the target

<Script: Source, Title, Target>

– Programming language & version

<Language: name, version>

– Type of output:

<Script: Type>

– Study design:

<Script: StudyDesign>

– State of qualification <Stage>:

<Qualification: Stage>

– OS is not included, since should be indicated in OS compatibility report• Test Data requirements

– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each script– include expected results from these data for defined tests/checks

#PhUSE

Qualification Proposal• Transitions

"Contributed" is the original State of all scripts– to Develop, checklist includes

by Developer & Reviewer• R & D confer on suitability of contribution. Suitable starting point?

[ may require analysis details, specs, from contributor ]• D reviews components (checklist to be completed)• D works with Contributor to complete minimum components

[ including Test Data and Coverage of defined tests ]• D adds standard parameter, dependency checking• D writes Qualification of CSS scripts.xlsx, see PhUSE Wiki Qual Workbook

– to Review, checklist includes

by Tester• Review Qualification instructions, consider coverage of tests• Execute Qualification instructions• Work with Developer to complete execution successfully

#PhUSE

Qualification Proposal• Transitions continued

– to Qualified by Tester & Environment Tester & Reviewer• T updates reference test outputs from certification/confirmation• E updates & executes local tests (posting PASS/FAIL results)• R confirms script output matches intention• R confirms qualification process covers important elements and

considerations. • R also provides user (rather than technical) feedback?• Achieve "Qualified" state when all tests in all test environments PASS (i.e.,

match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

#PhUSE

Qualification Proposal• Efforts Required

– Top priority• Finalize Qualification states, roles, workflow, checklists, and templates• Code criteria for "acceptable" (link to GPP, PhUSE)• Test definition, Test confirmation (also from White Paper team)• Test data

– Next priorities • Design test structure in google code• Develop scripts that will allow Environment Testing• Develop general components (e.g. parameter, dependency checking)• Identify Environment Testers based on

– Host environment– SAS or R version

• Identify opportunities to automate qualification. E.g.,– Environment Testers to post results back as machine readable– Script green-light/red-light qualification matrix on Phusewiki

#PhUSE