A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 1 A Structured Approach to Systems Projects Using Design Checklists By Adam Wilson DePaul University Author Note This paper was prepared for the Advanced Project course taught by Ed Paulson, Winter Quarter 2015.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 1
A Structured Approach to Systems Projects
Using Design Checklists
By Adam Wilson
DePaul University
Author Note
This paper was prepared for the Advanced Project course taught by Ed Paulson, Winter Quarter 2015.
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 2
Introduction
This paper addresses the problem of success in designing information systems. The best system
design occurs when a designer is familiar with good design approaches and can apply them appropriately
given the particular situation. Outcomes are less successful when designers do not choose intentionally
or skillfully but instead use approaches that are familiar or comfortable for them (Saffer, 2010). The
solution to this problem I will explore in this paper is the use of design checklists. The checklists will
consist of critical and easy to miss tasks taken from the Systems Development Lifecycle methodology as
well as other scholarly sources and my own personal experience. I hope to show in this paper that
checklists can help professionals in real situations to “make sense of… complex issues… in decision making
…” (McKeown, 2013). I will also address a number of obstacles to the successful use of checklists.
Obstacles include production pressures, personal inclinations, overloaded or unclear checklist design and
imposition of overly restrictive requirements (Katherine Radeka, 2015). There are three outcomes I hope
to achieve through my efforts. I would like my checklists to get used in actual design work, I would like to
achieve more consistent success in creating information systems and I would like to use the checklists in
project evaluations to better understand the influence of particular design tasks on system outcomes.
Advanced Project Competency Statements
F-11: Can design and produce a significant product that gives evidence of advanced competence.
F-12: Can design practical, holistic design checklists that allow flexibility while controlling for coverage of
primary value adding, critical and easy to miss project tasks.
1. Can describe primary value adding, critical and easy to miss project tasks in computer systems design.
2. Can define strategic pause points for checklist review amidst production pressures.
3. Can describe how checklists help individuals or teams take complex knowledge and consistently, correctly
and safely apply it under pressures and constraints.
The Problem
In today’s knowledge workforce end-users “design and develop an increasing number of their
own applications” (Turban, 2009). The discipline of design is therefore of increasingly broad relevance.
The best design occurs when the designer can apply the best approaches for the particular situation
however many designers include only those with which they are familiar or most comfortable (Saffer,
2010). The many phases and tasks of design, often with sequential dependencies present a problem of
complexity. These challenges can result in delays, abandoned projects, lost resources, communication
failures and tensions between stakeholders, poor usability, unmet system requirements, reduced internal
and external efficiency and effectiveness, poor usage rates and lost organizational knowledge (Plessis,
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 3
2007) and the dissolution of relationships. Even when we have years of experience and training,
“avoidable failures continue to plague us… the volume and complexity of knowledge… [Can] exceed our
ability as individuals to properly deliver it…” (Gawande, 2009). I’ve had varied success with system
projects, facing many of the challenges described above not always understanding the design tasks that
might help or how to consistently integrate them under constraints of time, budget, expertise, etc..
My Background
From 1999 to 2009 I worked as an office manager in a small private school. I oversaw customer
service and developed workflows for staff-client interactions. In 2009 I left this position and returned to
school at DePaul to complete my BA in Computer Information Systems. I hoped to bring my interest in
administrative process to the digital space as administrative work was becoming increasingly digital. The
Information Systems Focus Area courses have covered technical aspects of computing such as
programming and administering systems as well as people and process oriented aspects like systems
analysis and design, product and project management and organizational dynamics. I was in school for 2
years before I took another full time job as office manager at a church I admired. The church hoped that I
could bring some of their office processes “into the digital age”. Over that time I partnered with various
staff and others to design, built and support new systems for a number of core processes. These projects
included a public website redesign, implementation of Google docs and sites for real time document co-
authoring, an online room reservation system that has handled thousands of transactions and an online
staff resource portal among others.
Solution Overview: The Design Checklist
My solution to the problem of achieving consistent system design success is a design checklist.
Research reveals that checklists are powerful but they need to be created and implemented well
(Katherine Radeka, 2015). Checklists are everywhere in various forms from aircraft flight checklists to
construction site work schedules to rubrics used by teachers to grade assignments. Checklists are used to
support work by individuals and teams in complex, fast paced environments full of production pressures,
cultural factors, personal inclinations and regulatory issues (Asaf Degani, 1991). They can help
professionals “make sense of… complex issues… in decision making situations…” (McKeown, 2013).
Important checklist best practices I will incorporate include customization to individual needs, inclusion of
only critical or easy to miss items so the lists are non-cumbersome, a series of short lists instead of one
long list and usage guidelines for when and by whom each list should be reviewed (Katherine Radeka,
2015). The lists will be populated by design tasks taken from the Software Development Lifecycle (SDLC)
(Turban, 2009) and other methodologies including Interaction Design and User-Centered Design (UCD)
among others.
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 4
My intended outcome from the design checklists is consistent system design success. I would
also like to better understand the linkages between design tasks and outcomes and be more intentional
in choosing how to spend my time and energy. The next section on evaluation will discuss ways of
measuring success. I would like to replicate the outcomes I’ve achieved using checklists for my tasks as an
office manager. The partial image of my office management weekly checklist below demonstrates that
my prior work in administration has been highly checklist driven. Although I had not yet studied checklist
best practices the lists I created for various aspects of my job follow some of them. They include easy to
miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or logically,
were used collaboratively by administrative staff to track and communicate among team members and
were online to facilitate sharing and frequent updates (Katherine Radeka, 2015). Comprehensive
instructions were hyperlinked to items to facilitate access without overloading the checklist itself. The
lists were built in Google Sheets and the Design Checklists at the end of this paper may eventually be
migrated there as well. My office management checklists helped make me a very reliable and consistent
administrator and I would like to expand that kind of competency into the slightly less routinized and
more creative space of design.
Office management checklist used by the author. Checklists helped make me a very reliable and consistent administrator and I’d like to generalize this to systems design.
Criteria and Methods of Checklist Evaluation
In the next few pages I’ll describe some criteria that makeup successful system outcomes as well
as how to evaluate them and at what stages of design projects. The activities will become part of my
design checklists. Joel Palmius states in the article Criteria for measuring and comparing information
systems (Palmius, 2007) that “With operational criteria… a foundation can be laid for improving the
system… A designer needs to know what the goal is, and a buyer… needs to know whether the goals have
been fulfilled. With criteria, there could be a checklist before, during and after a design phase… (Palmius,
2007).” Evaluations are conducted by designers looking independently at analytics, transactions and
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 5
other data but the sections below focus mostly on usability factors and subjective stakeholder feedback
via questionnaires and observation.
Evaluating the Existing System (Pre-Design)
The first half of the items on a Checklist for User-Centered Design (Oregon State Computer
Engineering Dept., 2000) address “Initial motivation” and “pre-Design Activities”. These activities involve
user’s right from the start with evaluation of the current system and other existing competitive systems.
This is crucial for establishing baseline measures against which to compare any new systems designed. It
is also crucial to put users at the center of the design process from the outset. User concerns involve
work goals, tasks, mental models, needs and desires. This pre-design evaluation also investigates
whether the system is actually the main problem as opposed to factors like user awareness, training or
trust in the current system.
Multiple Evaluations during the Development Lifecycle
Evaluation should occur at various points during design. Methods that involve multiple iterations
of a solution in the design, prototype and eventually the final system need to evaluate repeatedly.
Various methodologies all count on feedback at points (Turban, 2009). Examples include pre-design,
formal design proposal (design brief/system requirements document), low-fidelity prototypes like
wireframes, initial system rollouts, and ongoing during the operation of the system.
Project Management and Collaborative Criteria for Success
Communication is a prerequisite for design success. In the building construction industry experts
have said that communication and tracking advances are the biggest improvements to the trade in recent
decades (Gawande, 2009). Consultant Scott Vaughn (Vaughan) goes so far as to recommend signed
agreements between parties in settings that are usually more informal such as between colleagues. In
today’s workplace where “rapid expert design” is going on all the time (Saffer, 2010) the “designer” is
often a member of the team. The table below lists some metrics and ways to control for them.
1. The system problem has been framed to clarify where the root of the actual problem lies.
2. This project has been ranked a top priority against others based on mission goals, beneficiaries and scope of impact.
3a. This project has been deemed technically feasible.
3b. Resources are available which make this project economically feasible.
3c. This project aligns with organizational and IT strategy based on documentation and confirmation with appropriate personnel.
3di. The initial concept or motivation for this project came from users or users have voiced a willingness to make changes to support this project.
3dii. The organizational staff chart has been obtained and reviewed to identify various project stakeholders.
3diii. A top leader with adequate authority is passionate about the project and has agreed formally to shepherd it through all phases including research, design, build, test, iterate, launch, training, operation. Who?
Who is passionate about this? How to engage them?
3div. Stakeholders have definitely been able to see potential value to their own everyday work life from this project.
3dv. Users trust of information systems and the leaders of this project enough that they will trust the system with their valuable information resources.
3dvi. A dedicated, qualified staff member has been allocated by top leadership to take ownership of this project.
3dvii. Formal training processes and personnel exist and have been authorized to insure proper training on the new system.
3dviii. Past failures and successes have been shared by users and stakeholders and this project involves factors that predict it will succeed and can mitigate risks.
DELIVERABLE: GO OR NO-GO DECISION (please circle) 1. Will not proceed based on these criteria (list): 2. Looks promising, will be deferred for Review on: 3. May proceed, given the following additional criteria are met: 4. Will proceed beginning:
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 24
SYSTEMS DESIGN CHECKLIST Revised 3/16/15 by Adam Wilson
> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 of 2 > ANALYSIS 2 > DESIGN >
PROTOTYPE-TEST > REFINE > IMPLEMENT > OPERATE >
NOTES
TRADITIONAL RESEARCH
1. Have a design approach been consciously chosen?
2. The system problem has been framed to clarify where the root of the actual problem lies.
3. Tentative Key dates and deliverables have been agreed to by the team and the tentative timeline has been posted.
4a. Key documents have been identified, collected and analyzed.
4b. Have you reviewed relevant quantitative information such as analytics, user logins or transaction statistics?
4c-d. A hypothetical workflow analysis chart and/or table of information inputs, people, process, and outputs has been created to go over with stakeholders.
5. Have you reviewed trade journals, industry analysis and competing businesses and identified best practices?
USABILITY and DESIGN
6a. Have designers conducted a personal heuristic evaluation of the existing system?
6b. Designers have evaluated the existing system using design criteria, identified issues and recorded ideas for their correction.
DELIVERABLE: System Requirements Hypothesis
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 25
SYSTEMS DESIGN CHECKLIST Revised 3/16/15 by Adam Wilson
> INITIATION > INV./FEASIBILITY > SYSTEMS ANALYSIS 1 > SYSTEMS ANALYSIS 2 of 2 >
1a. Have you discussed the organizational chart or similar
document to identify key stakeholders?
1b. A second researcher has been recruited for a second
pair of eyes and discussion of findings.
1c. A Test Plan has been devised containing neutral
questions for stakeholders and planned research
activities.
See stakeholder interview questionnaire.
1c. A test route through the product has been designed
“through the product in order to test functionality.
1d. Users have been observed in their own environments
with neutral questioning and recording with text, photos,
etc. The following activities have been conducted (please
conduct at least 2 and circle them at right).
Fly on the wall, shadowing, directed storytelling, role play, interviews, tour of computer and workflow etc., user workflow drawings explained afterward.
1f. A summative “Opportunity Report” has been created
from observations and identified patterns.
2a. The System Usability Scale (SUS) or alternate usability
questionnaire has been administered and scored.
SUS Supplementary Questionnaire available.
2c. Questionnaire results have been analyzed.
3. DELIVERABLE (OF ANALYSIS 1 & 2): Design Brief with System Requirements Stakeholder Research analysis has informed the Design Brief.
Traditional Research hypothesis have informed the Design Brief.
Information has been structured in a visual and easy to understand way in the Brief.
Strengths and weakness findings about the existing system are contained in the brief.
Functions the new system must have are included.
User information requirements are included.
Has a meeting been scheduled with appropriate stakeholders including real users to go over the Brief
Did designers clarify user and stakeholder feedback about the Brief to arrive at a significant majority opinion,
revise and agree on the system requirements and other elements of the Brief.
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 26
SYSTEMS DESIGN CHECKLIST Revised 3/16/15 by Adam Wilson
1. All touch points at which users will interact with the system including related services have been mapped holistically.
2. Usability Heuristics and Design Principles have been applied to each interface and aspect of the system.
Information Systems and other personnel involved in the system operation have been defined.
Appropriate user permission profiles have been created.
Security measures both behavioral and technical have been designed for information backup and access.
Collaborative issues such as precise role definitions, face time and accountability have been designed.
PHYSICAL DESIGN
System Requirements from the Analysis phase have been added to the general system requirements table.
See table in Appendix.
Market research has identified software alternatives.
Potential vendors have been rated using Requirements table Consider using the Vendor Rating Tool worksheet.
DELIVERABLE: FROZEN TECHNICAL DESIGN SPECIFICATIONS Both logical and physical design have been reviewed with all appropriate stakeholders including users.
Did users specifically notice that their suggestions had been incorporated?
Did users specifically say the new design was better?
Specification have been revised as needed.
All appropriate stakeholders have approved the final specifications and formally acknowledged they are being
“frozen” to prevent scope creep.
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 27
Generic System Requirements
Checklist Taken from Introduction to Information Systems: Supporting and Transforming Business (Turban, 2009)
Software Under Consideration: _____________________
Project specific requirements have been added to the generic requirements below.
1. Functionality
2. IT Strategy Alignment
Usability
Availability of Quality Documentation and Training Resources
Vendor Reputation, Industry Analysis
What do competitors use?
Cost. Free Trial?
Integration with existing applications
Necessary Hardware, Software and Networking Resources and compatibility:
Implementation, Updates
Security
usable for other processes
Long term product availability
Revised 3/16/15 by Adam Wilson
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 28
Reflection
Surprising Finding
The single most surprising thing I begin to see in the 12 sources and other research on this
project is the power of simple, precise interventions to dramatically impact extremely complex situations.
I was pleasantly surprised About the Checklists I’ve been using for Administrative Work
My prior work in administration has been highly checklist driven. Although I had not yet studied
checklist best practices the lists I created for various aspects of my job follow some of them. They include
easy to miss and critical items, fit on one page, are quickly reviewable and ordered sequentially and/or
logically, were used collaboratively by administrative staff to track and communicate among team
members and were online to facilitate sharing and frequent updates (Katherine Radeka, 2015).
Comprehensive instructions were hyperlinked to items to facilitate access without overloading the
checklist itself. The lists were built in Google Sheets and the Design Checklists at the end of this paper
may eventually be migrated there as well. My office management checklists helped make me a very
reliable and consistent administrator and I would like to expand that kind of competency into the slightly
less routinized and more creative space of design.
The Checklists in the Paper are Prototypes
These checklists have not been trialed and validated with front line users either in a simulation or
in a real situation and revised as needed. They are merely prototypes.
Submitting the Checklists to Design Scrutiny
While the design checklist should be subjected to the best practice standards of creating checklist
they should then again be subjected to the best practice standards of design. Under such analysis I
observe that I prefer online checklists which can be shared and accessed remotely. I suspect the next
iteration of them will involve migrating them to Google Sheets.
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 29
References
Aline Chevalier, M. Y. (2002). We site designs: Influences of designer's expertise and design constraints.
International Journal of Human-Computer Studies, 57-87.
Allen, D. (2002). Getting Things Done: The Art of Stress-Free Productivity. Penguin Books.
Asaf Degani, E. L. (1991). Human factors of flight-deck checklists: The normal checklist. Retrieved from
NASA Technical Reports Server (NTRS): http://ntrs.nasa.gov/search.jsp?R=19910017830
Axelrod, G. (2013, October 23). 47 Stats You Need to Know About the Google Apps Ecosystem. Retrieved
from bettercloud.com: http://blog.bettercloud.com/google-apps-stats/
Brooke, J. (1986). SUS - A quick and dirty usability scale. Farley, UK: Redhatch Consulting, Ltd.
Coleman, M. (2015). Process. Retrieved from megancoleman.com:
http://www.megancoleman.com/process/
davidallengtd.com. (2014). Getting Things Done Daily Planner. ataglance.
Erickson, L. G. (2007). Eight Ways to Build Collaborative Teams. Retrieved from Harvard Business Review:
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 32
Appendix A
System Usability Scale
Instructions: Use after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time. All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale. Strongly Strongly Disagree agree
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 33
Appendix B
Stakeholder Interview Sample Questions Taken from Interaction Design (Saffer, 2010)
1. Who are you and what is your role in the org
2. Why is this project important to you? To the org? (internal context)(look for unstated
goals/agendas)
3. What would make a successful project? (metrics)
4. Has anyone ever tried to address this problem before? (Why?)
5. What doesn’t this project cover? (Why?)(scope)(What are constraints, boundaries can’t cross?)
6. If we could only do only one thing with this project, what would that be? (Why?)
7. How could this project affect your day-to-day life? (Organizational expectations, part of life at…)
8. Are there any issues about this project I should be aware of?
9. What are the risks in doing this project? What could make it fail?
10. What are your competitors doing in this space?
11. “What would customers do (instead) if all the traditional competitors went away?
12. Who else should I talk to about this project?
I’ve added these questions taken from a Checklist for User-Centered Design (Oregon State Computer
Engineering Dept., 2000)
1. What are your work goals related to the project?
2. How do you currently structure those goals into tasks?
3. Why do you do it that way?
4. How do you wish you could do it?
A STRUCTURED APPROACH TO SYSTEMS PROJECTS USING DESIGN CHECKLISTS 34
Appendix C
Stakeholder Questionnaire Items Supplemental to the SUS
The following questions are not directly addressed by the SUS. They could be part of a separate evaluation questionnaire with follow-up interactions. I have limited them to 25 per the recommendations by SUS article author John Brooke (Brooke, 1986). They are designed to be answered using a Likert scale from 1-5 strongly disagree to strongly agree.
Regarding expanding software choices for users and use on multiple devices.
1. The system is my preferred way to perform the tasks it is designed for.
2. The system has worked well on all my devices.
From A 1995 Computer System Usability Questionnaire developed by IBM (Lewis, 1995).
1. I can effectively (and efficiently) complete my work
2. Whenever I make a mistake I recover easily and quickly.
3. The system was intuitive or Instructions were visible or easily retrievable
4. List the most negative aspects: (3 open ended fields to complete)
5. List the most positive aspects: (3 open ended fields to complete)
The Experience Design Framework “includes attitude measurements and emotional response” as well as social and pleasurable elements at the top of a “hierarchy of user needs” (Jefsioutine, 2006).
1. The system can help divide workload to the right people
2. I would definitely recommend this system to others 3. Working with this system would support positive relationships with others. 4. This system could help users work together well as a team. 5. This system could support improved relationships between different departments.
The article Criteria for Measuring and Comparing Information Systems (Palmius, 2007) contributes these questions related to organizational and knowledge management impacts of the system.
1. The system will help the organization capture significant knowledge embedded in individuals.
2. The system makes this knowledge easy to access and use by the appropriate people.
3. Processes are in place for leveraging this knowledge to create value. 4. People are regularly using the processes provided by the system for knowledge capture. 5. The system provides strong ROI. 6. The system significantly enhances the competitive advantage of the organization. 7. The system significantly enhances the satisfaction of external customers.
Additional Questions regarding trust and sustainable operation and support.
1. I am very comfortable that critical information in the system is safe.
2. Updating the system with information to keep it very current will be quite feasible. 3. We have the right people to maintain the system in good working order long term. 4. Problems with the system are addressed rapidly to my satisfaction. 5. My satisfaction and/or effectiveness with the system would be very much enhanced by these