Appraisal Assistant workshop Evidence collection (ISO15504-5 assessments)

Appraisal Assistant Appraisal Assistant workshopworkshop

Evidence collection (ISO15504-5 Evidence collection (ISO15504-5 assessments)assessments)

OverviewOverview• Evidence collection overviewEvidence collection overview• Evidence related screensEvidence related screens• Getting evidences into the databaseGetting evidences into the database

– Importing a list of documentsImporting a list of documents– Using spreadsheet to collect Using spreadsheet to collect initialinitial evidences evidences– Using evidence registry to collect evidencesUsing evidence registry to collect evidences

• Manage evidences of process Manage evidences of process outcomes/attribute achievementsoutcomes/attribute achievements

• Transfer evidence data between Transfer evidence data between assessment projectsassessment projectsWorkshop exercise setup: Workshop exercise setup: project : ISO15504, with ACQ.1, ACQ.2 on target CL5, with instantiation: project : ISO15504, with ACQ.1, ACQ.2 on target CL5, with instantiation: Org.ProcessOrg.Process

Evidence collection- Evidence collection- OverviewOverview• Evidence collection is guided by the assessment Evidence collection is guided by the assessment

indicatorsindicators• Map the evidence to the indicator whenMap the evidence to the indicator when

– (Part of) the evidence exhibits the same characteristic as a work (Part of) the evidence exhibits the same characteristic as a work product indicatorproduct indicator

– (Part of ) the evidence is considered the input/output of (Part of ) the evidence is considered the input/output of implementing a practiceimplementing a practice

– (Part of) the evidence implies the non/existence of the required (Part of) the evidence implies the non/existence of the required infrastructure/resource.infrastructure/resource.

• Evidence indicates the strength/weakness of the Evidence indicates the strength/weakness of the implementationimplementation

• Do NOT need to find evidences for all assessment Do NOT need to find evidences for all assessment indicatorsindicators– Ratings are based on the satisfaction of the process outcomes / Ratings are based on the satisfaction of the process outcomes /

process attribute achievements, not the fulfilment of indicatorsprocess attribute achievements, not the fulfilment of indicators• Evidences are automatic mapped to process outcomes / Evidences are automatic mapped to process outcomes /

attribute achievementsattribute achievements• Evidences can NOT be mapped to outcomes / Evidences can NOT be mapped to outcomes /

achievements without first being mapped to related achievements without first being mapped to related assessment indicatorsassessment indicators

Evidence related screens – Evidence related screens – Evidence registry panelEvidence registry panel

• Map evidence to Map evidence to assessment indicatorsassessment indicators

• Evidences is Evidences is automatic mapped to automatic mapped to generic work product generic work product when it is mapped to when it is mapped to specific work productspecific work product

• Comment how/why Comment how/why evidence is mapped evidence is mapped to the indicatorto the indicator

• Evidence detail Evidence detail informationinformation

• Evidence coverage of Evidence coverage of process outcomes / process outcomes / attribute attribute achievementsachievements

Evidence related screen – Evidence related screen – Process Attribute rating tabProcess Attribute rating tab

• Form the bases for Form the bases for rating process rating process attributesattributes

• Evidences mapped to Evidences mapped to process outcomes / process outcomes / achievementsachievements

• Can Can NOTNOT add new add new evidencesevidences

• Can remove irrelevant Can remove irrelevant evidence mappingsevidence mappings

• Characterize the Characterize the nature of the evidence-nature of the evidence-outcome mappingoutcome mapping

• Comment on how/why Comment on how/why the evidence indicates the evidence indicates the implementation of the implementation of process outcomes / process outcomes / attribute achievementsattribute achievements

Importing a list of Importing a list of documents - 1documents - 1

• Documents are Documents are NOTNOT mapped to assessment mapped to assessment indicators (unlike the PII spreadsheets)indicators (unlike the PII spreadsheets)

• Can be extracted from the OU’s document Can be extracted from the OU’s document databasedatabase

• Instantiation name must be the same as Instantiation name must be the same as defined in assessment scopedefined in assessment scope– The row will be ignored if the Appraisal Assistant The row will be ignored if the Appraisal Assistant

can not find a matched instantiation in the can not find a matched instantiation in the assessment scopeassessment scope

• Use the included Use the included DocumentList.XLSDocumentList.XLS to to ensure correct file formatensure correct file format

• Save the Save the Evidence listEvidence list tab as TAB delimited tab as TAB delimited Text before importingText before importing

Importing a list of Importing a list of documents - 2documents - 2

• 3 documents3 documents• Belongs to 2 instantiationsBelongs to 2 instantiations• Not mapped to any practiceNot mapped to any practice• Check error log for rows that are Check error log for rows that are

ignoredignored

List of Document Evidences

Instantiation Evidence Name Document Link Source/Affirmation Description

org. processes evidence bah! www.gu.edu.au moo moo's siteorg. processes Evd 1 must import

Test project 1 EVD 2 import once

Importing a list of Importing a list of documents - 3documents - 3

Result of importing the previous document list

Using spreadsheet to Using spreadsheet to collect collect initialinitial evidences - 1 evidences - 1

• One row per evidence, insert rows for more evidencesOne row per evidence, insert rows for more evidences• Leave the row EMPTY if there is NO evidence Leave the row EMPTY if there is NO evidence • Each evidence can be mapped to multiple assessment Each evidence can be mapped to multiple assessment

indicatorsindicators• Each evidence can NOT be mapped to the same Each evidence can NOT be mapped to the same

assessment indicator more than onceassessment indicator more than once• Do Do NOTNOT need to find evidence for ALL indicators need to find evidence for ALL indicators• Consists of all indicators of the entire assessment scopeConsists of all indicators of the entire assessment scope• User ensures data accuracy and consistencyUser ensures data accuracy and consistency• XML spreadsheets can be opened by Microsoft Excel XML spreadsheets can be opened by Microsoft Excel

2002 or newer2002 or newer• Save as ‘Save as ‘Tab Delimited TextTab Delimited Text’ file before being imported’ file before being imported• Program merges evidences only if they have the same Program merges evidences only if they have the same

name name ANDAND the same document link strings the same document link strings

Using spreadsheet to Using spreadsheet to collect collect initialinitial evidences -2 evidences -2

• 2 evidences here, mapped to 4 indicators2 evidences here, mapped to 4 indicators• Process Instantiation name in the top most cellProcess Instantiation name in the top most cell• Detail descriptions of process / process Detail descriptions of process / process

attribute / assessment indicator in cell commentsattribute / assessment indicator in cell comments• ACQ.1/18-08 has 2 evidencesACQ.1/18-08 has 2 evidences• ACQ.1/BP1 has no evidenceACQ.1/BP1 has no evidence

Org. Processes

Pro

cess A

ttribu

te

Indic. Abbr Indicator Name

Evidence Name Document Link Comment

ACQ. 1 PA1.1 18-08 Supplier selection criteria Test evidence 1 www.cit.gu.edu.au evidence exhibits the same characteristics

        Test evidence 2 www.sqi.gu.edu.au bah!

ACQ. 1 PA1.1 ACQ.1.BP1 Establish the need.    

ACQ. 1 PA1.1 ACQ.1.BP2 Define the requirements. Test evidence 1 www.cit.gu.edu.au evidence defines the criteria of the supplier selection

ACQ. 1 PA1.1 ACQ.1.BP3 Review requirements. Test evidence 2 www.sqi.gu.edu.au doh!

Using spreadsheet to Using spreadsheet to collect collect initialinitial evidences - evidences -

ImportedImported• 2 evidences 2 evidences

imported from imported from the previous the previous spreadsheetspreadsheet

• All 2 evidences All 2 evidences belong to the belong to the same same instantiationinstantiation

• Evidences Evidences mapped to mapped to specific work specific work products are products are automatic automatic mapped to the mapped to the generic products generic products of the same of the same processprocess

Using spreadsheet to Using spreadsheet to collect collect initialinitial evidences - 4 evidences - 4

• 4 evidences mapped to 4 specific work 4 evidences mapped to 4 specific work productsproducts– The 2 ‘Test evidence A’ have different document The 2 ‘Test evidence A’ have different document

linkslinks– The 2 ‘Test evidence B’ do not have document The 2 ‘Test evidence B’ do not have document

linklink

Org. Processes

Pro

cess A

ttribu

te

Indic.Abbr Indicator Name Evidence Name

Document Link Comment

ACQ. 1 PA1.1 08-02 Acquisition plan Test evidence A www.sqi.gu.edu.au test evidence A, linked to SQI

ACQ. 1 PA1.1 12-01 Request for proposal

Test evidence A www.gu.edu.au another evidence named 'test evidence A', linked to Griffith main page

ACQ. 1 PA1.1 13-19 Review record Test evidence B    

ACQ. 1 PA1.1 15-01 Analysis report Test evidence B   another evidence named 'Test evidence B'

Using spreadsheet to Using spreadsheet to collect collect initialinitial evidences - 5 evidences - 5

• 4 evidences are 4 evidences are imported from the imported from the previous previous spreadsheetspreadsheet

• Evidences with Evidences with duplicated name duplicated name strings will cause strings will cause confusions when confusions when validating datavalidating data– Use unique name Use unique name

string for evidencestring for evidence– Import routine can Import routine can

merge evidences if merge evidences if they have the same they have the same name name ANDAND the the same document same document linklink

Using spreadsheet to Using spreadsheet to collect collect initial initial evidences – evidences –

Exercise Exercise

Using evidence registry – Using evidence registry – Collect evidence - 1Collect evidence - 1

• Organization provides evidences but Organization provides evidences but unsure how they are related to the unsure how they are related to the assessment modelassessment model

• Assessors Assessors familiarfamiliar with the with the ISO15504-5 modelISO15504-5 model

• Deleting an evidence record will also Deleting an evidence record will also remove it from related process remove it from related process outcomes, achievements, and outcomes, achievements, and indicatorsindicators

Using evidence registry – Using evidence registry – Collect evidence - 2Collect evidence - 2

• Click “Add” button to Click “Add” button to add a new evidence add a new evidence recordrecord

• Enter the Name of the Enter the Name of the evidenceevidence

• Specify the Specify the instantiation (optional)instantiation (optional)

• Map the evidence to Map the evidence to assessment indicator(s)assessment indicator(s)– If the evidence is If the evidence is

mapped to a specific mapped to a specific work product, the work product, the program will also map program will also map the evidence to its the evidence to its generic work productgeneric work product

• Comment on why/how Comment on why/how the evidence is mapped the evidence is mapped to the assessment to the assessment indicator(s)indicator(s)

Using evidence registry – Using evidence registry – Collect evidence exerciseCollect evidence exercise

Using evidence registry - Using evidence registry - Collect evidence by Collect evidence by

assessment indicators - 1assessment indicators - 1• Organization provides evidences but unsure Organization provides evidences but unsure

how they are related to the assessment modelhow they are related to the assessment model• Assessors Assessors unfamiliarunfamiliar with the ISO15504-5 with the ISO15504-5

modelmodel– Use indicator description in the tool to help Use indicator description in the tool to help

identify what evidences to look foridentify what evidences to look for

• No need to find evidence for ALL indicatorsNo need to find evidence for ALL indicators• Deleting an evidence record will also remove Deleting an evidence record will also remove

it from related process outcomes, it from related process outcomes, achievements, and indicatorsachievements, and indicators

Using evidence registry – Using evidence registry – Collect evidence by Collect evidence by

assessment indicators - 2assessment indicators - 2• Select the assessment Select the assessment

indicator of interest, indicator of interest, use the indicator use the indicator description as a description as a guide, look for guide, look for evidenceevidence

• Click “Add” button to Click “Add” button to add a new evidenceadd a new evidence– The evidence is The evidence is

automatic mapped to automatic mapped to the selected indicatorthe selected indicator

• Name the new Name the new evidenceevidence

• Specify the Specify the instantiation of the instantiation of the evidence (optional)evidence (optional)

• Comment on why/how Comment on why/how the evidence is the evidence is mapped to the mapped to the indicatorindicator

Using evidence registry – Using evidence registry – Collect evidence by Collect evidence by indicators - Exerciseindicators - Exercise

Using evidence registry – Using evidence registry – Evidence detail informationEvidence detail information

• Only evidence name Only evidence name is mandatoryis mandatory

• Evidence status, Evidence status, document link can document link can also be changed also be changed through context through context menu of the evidence menu of the evidence listlist

• Evidence name and Evidence name and instantiation can be instantiation can be changed on the changed on the evidence listevidence list

• LabelLabel: a short string : a short string assigned by the assigned by the assessment team to assessment team to identify the identify the documents.documents.

• Source/AffirmationSource/Affirmation: : the source/owner of the source/owner of the evidencethe evidence

Using evidence registry – Using evidence registry – Evidence coverage reportEvidence coverage report

• Show evidence Show evidence count for process count for process outcomes /attribute outcomes /attribute achievementsachievements

• Red cells highlight Red cells highlight the model elements the model elements with with 00 evidence evidence

• User needs to ensure User needs to ensure the data gathered is the data gathered is sufficient to cover sufficient to cover the assessment the assessment scope (scope (ISO15504-2, ISO15504-2, 4.2.2 b/c4.2.2 b/c))

• Select the process Select the process outcome /attribute outcome /attribute achievement to achievement to display all related display all related indicatorsindicators

Process attribute tab – Process attribute tab – Remove irrelevant evidences Remove irrelevant evidences

mappings - 1mappings - 1• Evidences are automatic mapped to Evidences are automatic mapped to

process outcomes/attribute process outcomes/attribute achievementsachievements– The mapping is done by looking up the The mapping is done by looking up the

mapping table defined in ISO15504-5mapping table defined in ISO15504-5– No guaranty all mapped evidences are No guaranty all mapped evidences are

meaningful for the process meaningful for the process outcome/attribute achievementoutcome/attribute achievement

• Same evidence can be mapped to the Same evidence can be mapped to the same process outcome more than once same process outcome more than once through different indicatorsthrough different indicators

Process attribute tab – Process attribute tab – Remove irrelevant Remove irrelevant

evidences mappings - 2evidences mappings - 2

• Clear the check box to remove an evidence Clear the check box to remove an evidence mapping.mapping.

• Click ‘Delete Unchecked’ will deleted it from the Click ‘Delete Unchecked’ will deleted it from the listlist

• Click ‘Reset All evidence’ to reset the list back to Click ‘Reset All evidence’ to reset the list back to the initial statethe initial state

Process attribute tab – Process attribute tab – characterize and comment characterize and comment

evidences mappingsevidences mappings

• An evidence can be characterized as strength or An evidence can be characterized as strength or weaknessweakness

• Comment on how the evidence contribute to the Comment on how the evidence contribute to the rating of process outcome/attribute achievementrating of process outcome/attribute achievement– Not the same as the evidence-indicator commentNot the same as the evidence-indicator comment– Evidence-indicator comments are shown at the Evidence-indicator comments are shown at the

bottom as read onlybottom as read only

Transfer evidences between Transfer evidences between assessment projectsassessment projects

• Evidences that are out of destination Evidences that are out of destination scope are scope are NOTNOT imported imported

• Import routine does Import routine does NOTNOT check for check for duplication with the existing duplication with the existing evidencesevidences– Import the database once and ONLY Import the database once and ONLY

once.once.– Import the database before generating Import the database before generating

any mini teamsany mini teams

Workshop revision Workshop revision historyhistory

VeVerr

AuthorAuthor DateDate CommentComment

0.10.1 Fred LiangFred Liang 19/July/0519/July/05 Initial versionInitial version

0.20.2 Fred LiangFred Liang 7/Dec/057/Dec/05 Add import list of documentsAdd import list of documents