#PhUSE Standard Scripts Project 2 2014 - Proposal for Qualification of Standard Scripts.

#PhUSE

Standard Scripts Project 2

2014 - Proposal for Qualification of

Standard Scripts

#PhUSE

Main Sections

Summary of prior proposal, 2013Updated proposal, July 2014

#PhUSE

Main Sections

Summary of prior proposal– Concepts, definitions & metadata– Test data considerations– Heavy vs. Light qualification

Updated proposal

#PhUSE

Proposal from 2013

• Anyone can submit a script, according to a check list• Categorize scripts by complexity– Complexity:

low, medium, high, software

– Output:

tabulated data, analysis data, table, figure, listing

• Metadata for script should indicate– Type of output:

tabulated data, analysis data, table, figure, listing

– Study design:

parallel, crossover, etc

– State of qualification:

?

#PhUSE

Proposal from 2013

• Test data– Overall project should have minimum test data (SDTM & ADaM)– Scripts can propose new test data, must pass (Data fit? Open CDISC?)– Share program to produce test data, never binary test data

• 2 levels of qualification– Light vs. Heavy: according to complexity of script & output– Common elements include

• header• adhere to good programming practices• clear scope of script (e.g., study design(s))• test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?)• documentation available for: usage, inputs, outputs, dependencies

#PhUSE

Proposal from 2013

• Heavy qualification– Beta package includes:

minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC?)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced

– Draft• Write qualification plan, Review tests for completeness/suitability (e.g.,

Branch testing – are all conditional blocks/combos tested?)– Test

• Peer Review: Write qualification report, incl. log/output from tests– Final

#PhUSE

Proposal from 2013

• Light qualification– Beta package includes

skip if >1 yr production use without ERROR– Draft

minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC or other, as appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced• Write qualification plan, Review tests for completeness/suitability (e.g.,

Branch testing – are all conditional blocks/combos tested?)– Test

• Peer Review: Write qualification report, incl. log/output from tests– Final

#PhUSE

Proposal through CSS 2104Peer Review Checklist Heavy Light

Requirement specification X ?

Documented or perhaps only documented in header X

User Guide X X

SDTM/ADaM used in input/output X X

Open CDISC validator or Data Fit used to check input/output X X

GPP in source X X

Run according to Requirement specification X ?

Tested by qualification plan, tests & results all Peer reviewed X ?

Tested by End users X ?

Robust without red errors in contributor's production environment X X

Robust and used in FDA (other) scripts repository, ranked ****** X

#PhUSE

Main Sections

Summary of prior proposalUpdated proposal– Objectives– Definitions: Qualification, States, Roles– Metadata and Test data– State Transitions

#PhUSE

Proposal 2014 - Objectives• For End-users

– Clear overview of resources available, incl. purpose & state of each– Inspire confidence from first user experience– Ease-of-use:

• clear messaging from scripts• Reproducible results in users’ own environments• Consistency of scripts, learning first one makes remaining familiar

– Ease of converting users to contributors

• For Contributors & Standard Scripts Team– Ease of contribution: Modularize & standardize workflows & checklists– Modularize core components (e.g., FUTS for dependency checking?)– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)– Centralize & consolidate information & results

#PhUSE

Qualification Proposalmeaningful terms in blue

http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx

• Qualification Instructions (see embedded template ð) – Certification applies to new scripts and functionality– Confirmation applies to updates of existing script

• States:

Contributed, Develop, Review, Released• Roles

– Contributor: Anyone with appropriate skills & interests– Developer: A volunteer in CSS Working Group 5, familiar with our objectives**– Tester: A volunteer in CSS WG 05 who is familiar with our objectives**– Environment Tester: Anyone in community who is able to set up automatic test

replication in their work environment– Reviewer: Author of white papers, designers of script targets**

** suggests a quick-start onboarding page in CSS Phusewiki

Microsoft Excel Worksheet

#PhUSE

Qualification Proposal• Metadata for scripts should indicate:

YML proposal– Whitepaper & output ID uniquely identify the target

<Script: Source, Title, Target>

– Programming language & version

<Language: name, version>

– Type of output:

<Script: Type>

– Study design:

<Script: StudyDesign>

– State of qualification <Stage>:

<Qualification: Stage>

– OS is not included, since should be indicated in OS compatibility report• Test Data requirements

– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each script– include expected results from these data for defined tests/checks

#PhUSE

Qualification Proposal

• Transitions

"Contributed" is the original State of all scripts– to Develop, checklist includes

by Developer & Reviewer• R & D confer on suitability of contribution. Suitable starting point?

[ may require analysis details, specs, from contributor ]• D reviews components (checklist to be completed)• D works with Contributor to complete minimum components

[ including Test Data and Coverage of defined tests ]• D adds standard parameter, dependency checking• D writes Qualification of CSS scripts.xlsx (see template embedded above)

– to Review, checklist includes

by Tester• Review Qualification instructions, consider coverage of tests• Execute Qualification instructions• Work with Developer to complete execution successfully

#PhUSE

Qualification Proposal

• Transitions continued– to Released by Tester & Environment Tester & Reviewer

• T updates reference test outputs from certification/confirmation• E updates & executes local tests (posting PASS/FAIL results)• R confirms script output matches intention• R confirms qualification process covers important elements and

considerations. • R also provides user (rather than technical) feedback?• Achieve “Released" state when all tests in all test environments PASS (i.e.,

match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

#PhUSE

Qualification Proposal• Efforts Required

– Top priority• Finalize Qualification states, roles, workflow, checklists, and templates• Code criteria for "acceptable" (link to GPP, PhUSE)• Test definition, Test confirmation (also from White Paper team)• Test data

– Next priorities • Design test structure in google code• Develop scripts that will allow Environment Testing• Develop general components (e.g. parameter, dependency checking)• Identify Environment Testers based on

– Host environment– SAS or R version

• Identify opportunities to automate qualification. E.g.,– Environment Testers to post results back as machine readable– Script green-light/red-light qualification matrix on Phusewiki

#PhUSE