Analytic Design Group Design Research Qualifications

QualificationsUser Testing & Design Research

January 24, 2013

Who we are

Founded in 2005 on the principle that evidence-based design will always be more powerful than design driven by best practices, we have grown from a single practitioner to a vibrant, collaborative team.

Analytic Design Group Inc (ADGi) is a visionary user experience strategy and design firm that specializes in innovating in digital environments by leveraging in-depth primary research to find expose unexamined assumptions. Our work not only withstands the complexity of multiple agendas and intricate implementation but also the scrutiny of the public.

Some of our clients include:Samsung, Sony, AT&T, Adobe, Nokia, LG, Motorola

Our ServicesKey service areas include: design research, user experience strategy development, interaction design, communication design, and usability testing. Lately some of our work has also included service design considerations as well. Our projects can include the full sweep of user experience services (i.e. user research through strategy and design) or just one element. Our aim is to always fit the work required to the need, and we’ll work with you to ensure you are getting the best value from our efforts.

This presentation focuses on our design research and user testing services.

/ Usability Testing / / Communication Design / / Interaction Design / / User Experience Strategy Development // Design Research /

Design Research

Design ResearchWe use a diverse set of design research methodologies:

Surveys — we have used surveys to establish baseline data (largely attitudinal), help to segment audiences, and in some cases, help to identify core issues that can be further explored by other research.

Context-rich group interviews (like marketing focus groups but much richer) — the focus groups we do, are typically very rich and usually drive out a great deal of contextual as well as attitudinal data. We usually ask participants to complete homework prior to the session (aids in grounding the user and supports contextual data gathering) as well as have some form of participatory design exercise to allow participants to tap into their feelings and attitudes quickly.

Design ResearchOn-site observation (w/o) interviews — this is useful when we are looking for issues that are process related.

Task analysis — this is usually both an expert review and then a walkthrough with participants to identify particular pain points with certain tasks. This often involves both offline and online elements.

Expert review/Heuristic analysis — this can be a quick and cost-effective means of identifying user experience and usability issues. We typically rank severity of issues identified and can include an accessibility review in this process.

Card sorting — we have done card sorting exercises in both one on one as well as group sessions. We’ve used both open and closed card sorts and typically use the findings to develop information architectures.

Diary Studies — are useful when we are looking at processes that occur over a longer period of time or are looking at the impact of certain things over time.

TASK 1

80%50%70%

Design ResearchIn-Situ & Ethnographic

We have conducted numerous ethnographic or in-situ studies on a wide range of physical and digital products. These typically are very data rich and result in in-depth, tactical, near-term findings as well as robust, strategic, longer-term, insights. Our clients report that the ROI on these studies is that along with finding solutions to nagging problems, it can help them focus their product management for a year or more.

For example, last year ADGi conducted an ethnographic study for a mobile carrier on a device experiencing high returns. We were able to identify key usability issues, service design issues, and deliver insights about how their customers currently perceived these devices and were likely to for the foreseeable future.

� �2SHQ�%R[

� �&RQQHFW�WR�+RW6SRW � �&RQÀUP�6SHHG��&RQQHFWLRQ

� �$GPLQLVWUDWLRQ�3DJH

� �'HYLFH�$VVHPEO\ � �3RZHU�8S��'HYLFH�2ULHQWDWLRQ

PE�V$GPLQLVWUDWRU�/RJLQ

0WFSBMM�'MPX

$35,/��������

� �2SHQ�%R[

� �&RQQHFW�WR�+RW6SRW � �&RQÀUP�6SHHG��&RQQHFWLRQ

� �$GPLQLVWUDWLRQ�3DJH

� �'HYLFH�$VVHPEO\ � �3RZHU�8S��'HYLFH�2ULHQWDWLRQ

PE�V$GPLQLVWUDWRU�/RJLQ

0WFSBMM�'MPX

$35,/��������

� �2SHQ�%R[

� �&RQQHFW�WR�+RW6SRW � �&RQÀUP�6SHHG��&RQQHFWLRQ

� �$GPLQLVWUDWLRQ�3DJH

� �'HYLFH�$VVHPEO\ � �3RZHU�8S��'HYLFH�2ULHQWDWLRQ

PE�V$GPLQLVWUDWRU�/RJLQ

0WFSBMM�'MPX

$35,/��������

Design ResearchSample Report

User Testing

User TestingThe range of user testing methods we use include:

Metrics-based usability studies — the usability studies we do are quite rich with quantitative (metrics) data as well as qualitative data. We typically collect task time, performance, SUS, satisfaction, and hedonic scores

Remote-moderated usability studies — through the use of such tools as WebEx (or other screen sharing tools) we have successfully conducted remote moderated testing, collecting similar (or the same) metrics as we do for in person tests – this is particularly useful when testing with participants who are geographically dispersed or where the user’s context heavily influences their interaction and on-site observation is not possible/feasible.

User Testing‘Listening-lab’ style user testing — this is essentially user testing without a set task list. There is some hard data we draw out of these sessions, but mainly this is focused on qualitative data.

Un-moderated usability testing — this is user testing where the user is in the lab and observed and recorded but completing the tasks on their own.

ADGi Field Test — this is a web-based tool we developed in house that automates a field test: participants are asked via email whether they wish to participate. If they indicate yes, they are sent a set of instructions or tasks to complete along with an NDA reminder. After a set period of days participants are then sent a survey to fill out. From a test administration point of view we can track all the participants, where they are in the study and get a graphical view on how they responded to each question, as well as download a CSV of the results for additional manipulation. We’ve used this tool to test devices and apps.

User TestingNavigation testing — this is another tool we developed in house to test navigation structures. Users are asked a series of questions about under what categories and labels they would expect to find certain pieces of information. They are shown the tree structure for the site and can navigate through it to the spot where they would expect to find the content. This testing has been very effective for us in establishing how findable content on very large sites will be and in determining the effectiveness of categorization and labeling schemes.

Concept acceptance testing — this is useful for trying out a new concept, typically while comparing it to other more familiar ones. We’ve used this on devices when a client wants to evaluate new way of navigating or different form factor

Competitive benchmark testing — this is useful when comparing a product (interface, device, site) against one or more others – we have used this to set benchmarks for future comparison as well as just comparisons

Iterative testing — this is where we test one or at most two discreet elements with a very small set of users (2 or 3) make recommendations on that testing, the development team makes those changes and we test again until we do not see the need for any more changes. We use this method primarily for games research looking at a particular interaction. While other clients have asked about this, after discussing it we have so far determined that the value of this approach does not warrant the effort and cost for the project at hand.

User Testing

User TestingRemote user testing — Ability and experience in executing remote usability testing — inclusive of screen sharing, audio and video recording.

We have experience conducting remote-moderated usability as well as focus group sessions. We screen share and capture (audio and video record) the sessions. We have found that this type of research can be very cost effective and is especially useful when we are asking participants to log in to their own accounts, or are geographically dispersed. On occasion we’ve also found that by having the user located in their own environment, we are able to glean more contextual information than we are typically able to in the lab.

User TestingSample Report

6TBCJMJUZ�4DPSFDBSE

���� �����&855(17�� �����3,/27�

� ��� ����:(,*+7�

� ��� ����:(,*+7�

�686�

�5$7,1*��)$,5:HLJKWHG�8;�6FRUH����

7DVN�3HUIRUPDQFH�5DWLQJ 2YHUDOO�5DWLQJ $UHD�$SSHDO�5DWLQJ6\VWHP�8VDELOLW\�6FDOH

5$7,1*��)$,5 5$7,1*��)$,5 5$7,1*��)$,5 5$7,1*��)$,5

$�IDLU�WDVN�SHUIRUPDQFH�VFRUH�LQGLFDWHV�LVVXHV�ZLWK�XVDELOLW\�RI�VRPH�PRGXOHV��VLWH�QDYLJDWLRQ�DQG�SUHVHQWDWLRQ�RI�FRQWHQW��3DUWLFLSDQWV�VWUXJJOHG�ZLWK�XVLQJ�WKH�$UHDV�IDW�IRRWHU�WR�ÀQG�QHZV�DERXW�/LWWOH�0RXQWDLQ��7KH\�DOVR�SHUIRUPHG�SRRUO\�RQ�ÀQGLQJ�LQIRUPDWLRQ�DERXW�IRRG�VFUDSV�DQG�GLUHFWLRQV�WR�D�QHDUE\�ÀUH�VWDWLRQ�

$�IDLU�686�UDWLQJ�UHÁHFWV�WKH�SHUFHLYHG�FRPSOH[LW\�RI�WKH�SLORW�ZHEVLWH��3DUWLFLSDQWV�FLWHG�D�QXPEHU�RI�LVVXHV�ZLWK�EXULHG�OLQNV��FRQIXVLQJ�JOREDO�QDYLJDWLRQ�KHDGLQJV�DQG�WH[W�KHDY\�SDJHV��7KHVH�SUREOHPV�FUHDWHG�VRPH�FRQIXVLRQ�RYHU�ZKHUH�WR�ÀQG�WKH�GHVLUHG�FRQWHQW�DQG�H[WHQGHG�WDVN�FRPSOHWLRQ�WLPH���

:HEVLWH�DSSHDO�LQFUHDVHG�IURP�ORZ�´IDLUµ�UDWLQJ�RI�WKH�FXUUHQW�ZHEVLWH�WR�D�KLJK�´IDLUµ�UDWLQJ�RI�WKH�3LORW�VLWH��7KLV�LQFUHDVH�UHÁHFWV�SDUWLFLSDQWVҋ�SRVLWLYH�UHDFWLRQ�WR�WKH�3LORWҋV�ORRN�DQG�IHHO���WKH\�DSSUHFLDWHG�LPSURYHG�QDYLJDWLRQ�RSWLRQV��FRORU�FRGLQJ�RI�VHFWLRQV�DQG�KLJK�TXDOLW\�SLFWXUHV��5HPDLQLQJ�LVVXHV�UHODWHG�WR�3LORW�FRPSOH[LW\�NHSW�WKH�RYHUDOO�UDWLQJ�ZLWKLQ�WKH�IDLU�UDQJH�

3DUWLFLSDQWV�UDWHG�WKH�DUHDV�IHDWXUH�DV�IDLU��7KH\�OLNHG�WKH�LQWHUDFWLYH�PDS�DQG�DFNQRZOHGJHG�WKDW�WKLV�NLQG�RI�LQIRUPDWLRQ�LV�GLIÀFXOW�WR�SUHVHQW��HVSHFLDOO\�FRQVLGHULQJ�WKH�FRPSOH[LW\�RI�9DQFRXYHUҋV�QHEXORXV�QHLJKERXUKRRG�ERXQGDULHV��7KH\�DGGHG�WKDW�WKH�IHDWXUH�FRXOG�EH�LPSURYHG�E\�KDYLQJ�D�PRUH�ORJLFDO�OLVW�RI�DUHDV�DQG�DQ�H[SODQDWLRQ�RI�QHLJKERXUKRRGV�ZLWKLQ�HDFK�DUHD��

1RWH��$�7DVN�3HUIRUPDQFH��5DWLQJ�VFRUH�DQG�6\VWHP�8VDELOLW\�6FDOH��686��RI����DQG�XS�LV�´([FHOOHQWµ��������LV�´)DLUµ��DQG�EHORZ����LV�´3RRUµ

$�:HEVLWH�$SSHDO�VFRUH�DQG�$UHD�$SSHDO�5DWLQJ�RI�����DQG�XS�LV�´([FHOOHQWµ����������LV�´0HGLRFUHµ��DQG�EHORZ�����LV�´3RRUµ

,QVWUXFWLRQV<RXҋYH�GHFLGHG�LWҋV�WLPH�WR�KDYH�DQ�HPHUJHQF\�NLW�UHDG\�LQ�FDVH�RI�QDWXUDO�GLVDVWHUV��)LQG�WKH�OLVW�RI�WKLQJV�WKH\�UHFRPPHQG�\RX�KDYH�LQ�\RXU�VDIHW\�NLW�

&DXVHV�IRU�2YHU�7DUJHW�7LPH

$SFBUF�&NFSHFODZ�,JU

6OEFS�*EFBM�5BSHFU�5JNF

8JUIJO�.BYJNVN�5BSHFU�5JNF

'BJMFE�5BTL

,GHDO�7DUJHW�7LPH������0D[LPXP�7LPH�$OORWWHG������

�����

������

7DVN�5HVXOWV��7LPLQJ

3$57,&,3$176 &$86( ),1',1*3���3�� LQWHUDFWHG�ZLWK�

HPHUJHQF\�NLW�LPDJH����

3���3���3���3���3�� FRQIXVHG�E\�SGI�LQVWUXFWLRQV

���

3���3���3�� WRR�PDQ\�VWHSV ���3���3���3����3�� WH[W�KHDY\ ���3�� OHIW�QDYLJDWLRQ�GLIÀ�FXOW\ ���

7KHUH�ZHUH�FURVV�WDVN�LVVXHV�REVHUYHG�LQ�WKLV�WDVN1RWH

i*OJUJBMMZ�JU�XBT�EJGm�DVMU�UP�EFUFSNJOF�XIJDI�MJOL�XBT�UIF�DPSSFDU�MJOL�UP�m�OE�UIF�DIFDLMJTU�w�'JOEJOH����� ����

i5IF�UFYU�VTFE�XBT�OPU�UIF�MBOHVBHF�*�XPVME�FYQFDU�XIJMF�MPPLJOH�GPS�B�TBGFUZ�LJUw�w�'JOEJOH�����

$QDO\WLF�'HVLJQ�*URXS��

User TestingSample Report

Mobile Test Lab

Mobile Test Lab

Mobile test lab — we conduct a great deal of testing on mobile devices and our lab set up is both flexible and powerful:

Our testing equipment is deliberately flexible so that we can set up in a lab environment, a coffee shop, a person’s home or office. We have designed a very stable, yet flexible camera mount that allows us to capture a variety of interactions. Assuming we can connect to a stable WiFi, we can also live stream (to allow remote viewing) outside of a lab environment.

For more information view this presentation:

Mobile Usability: What's Your Strategy

Karyn ZuidingaPrincipal & Director of User Experience

[email protected]

www.analyticdesigngroup.com@analytic_design