Evaluating Web Site Performance in Internet-Based Selling from a ...

EVALUATING WEB SITE PERFORMANCE IN INTERNET-BASED SELLING FROM A BUSINESS VALUE PERSPECTIVE

Jungpil Hahn (contact author)

Doctoral Program

Robert J. Kauffman Professor and Chair

Information and Decision Sciences

Carlson School of Management University of Minnesota Minneapolis, MN 55455

Email: {jhahn, rkauffman}@csom.umn.edu

Last revised: June 1, 2001

Submitted to 2001 International Conference on Electronic Commerce, October 2001, Vienna, Austria. _____________________________________________________________________________________

ABSTRACT

Current evaluative approaches for the performance of e-commerce Web sites fail to adequately address senior manager concerns about the returns on investment (ROI) of corporate efforts to develop and deploy software applications that support Internet-based selling. In this article, we explore the basis for developing a new approach to the evaluation and the prioritization of software development activities that modify the capabilities of Web sites in terms of the quality of services that they can deliver to a firm’s customer-users. Our key insight is that the design of Web-based software applications must be business value-driven. We also argue that decisions about the kinds of changes and adjustments that are appropriate should be based upon measurement approaches that emphasize the managerial actions that are possible as a result of new ways of thinking about Web-based performance assessment and the application of Web-based data mining techniques. To illustrate this perspective, we examine the qualities of currently available evaluative approaches in the context of the Internet-based sales activities of OnlineGrocery.com, and propose a new framework to guide research that aims to formulate a new value-driven metrics suite for e-commerce Web site performance that emphasizes the ROI outcomes. At ICEC 2001, we will present the background of this work, and additional theoretical and empirical extensions that will show the efficacy of our approach for guiding Web design adjustments to support Internet-based selling.

_____________________________________________________________________________________ KEYWORDS: Business value, electronic commerce, e-tailing, evaluation methodologies, Internet,

information technology, site design, value-driven design, World Wide Web. _____________________________________________________________________________________

1. INTRODUCTION

Since the crash of the DotComs in the American stock market in May and June of 2000, the

evaluation of electronic commerce Web sites has become increasingly important (Varianini and Vaturi,

2000). No longer are the venture capital firms willing to make portfolio investments in e-commerce

properties that only offer future ROI (return on investment) opportunities. Instead, they seek more

immediate opportunities, especially firms that can demonstrate an increasingly well-developed discipline

for evaluating investments in Web-related software development and e-commerce business models, so

that the risks and uncertainties of investing in this emerging market are better balanced with the rewards.

Recent industry analyses, however, point out that e-commerce retailers are earning low scores on

ROI, by failing to meet consumers’ purchase needs with the poor usability and errant designs of their

Web-based storefronts. For example, a study by Zona Research recently reported that 60% of Web-savvy

users dropped out of the purchasing process because they could not find the products in the e-tailers’ Web

sites (Zona Research, 1999). Another study conducted by A.T. Kearney showed that 80% of experienced

online shoppers gave up shopping on e-commerce Web site sites due to problems that encountered while

interacting with the Web site (Rizzuti and Dickinson, 2000). Yet another study conducted by Creative

Good showed that 43% of purchase attempts ended in failure due to poor usability of the Web sites

(Rehman, 2000). This shortfall in realized value compared to the potential value that Web-based selling

approaches off is dramatic. The Creative Good study points out that this level of failed purchase attempts

is consistent with an estimated loss of $14 billion in sales for e-tailers in the 2000 Christmas-New Year’s

holiday shopping season alone. Recent academic research reinforces the picture that emerges.

Apparently the quality of the online customer experience that effectively-design Web sites create not only

have a positive effect on the financial performance of a firm, but they also possess the potential to create

unique and sustainable competitive advantage for Internet-based sellers and other e-commerce firms

(Rajgopal, Venkatachalam, and Kotha, 2001).

Indeed, the industry has progressed to the second phase of electronic commerce. During the first

phase of electronic commerce, the goal for most companies was to secure a share of the virtual market

space through an online presence by attracting as many visitors as possible to their Web site, whereas now

– as electronic commerce begins to “grow up” – the ability to conduct online operations justified by ROI

is the only way an e-business can survive. This situation brings to the foreground the importance of

value-driven evaluation and management of e-commerce and Web site performance. However, e-

businesses are facing difficulties due to the lack of value-driven evaluation methodologies for online

operations.

The purpose of this paper is to present a value-driven framework for e-commerce Web site

evaluation. Toward this end, we review the key reference disciplines that motivate and guide the

1

development of the framework. We also conduct an in-depth exploratory case study of an online business

to understand how Web site performance evaluation is being conducted by e-businesses. This enables us

to generate insights into how Web site performance evaluation should be conducted and what tools and

techniques need to be developed to guide effective evaluation and management of business performance.

This exploratory investigation yielded a new, but still preliminary framework that we hope to refine and

expand, and eventually empirically test, to provide better management science-based performance

evaluation methods that will promote the appropriate managerial actions related to Web site design and

development priorities.

2. LITERATURE REVIEW

In order to develop a value-driven framework for Web site performance evaluation, we first turn to

the literature from key reference disciplines that have investigated the evaluation of Web sites. We will

discuss and critique approaches to general Web site evaluation, followed by approaches and tools

proposed specifically to evaluate e-commerce Web sites.

2.1. Web Site Evaluation

Recent reports concerning the poor quality and usability of Web sites have led researchers and

practitioners to express increasing interests in the methodologies and approaches that are used to conduct

the evaluations. Traditional approaches to Web site evaluation fall into three different categories: (Ivory

and Hearst, 2000).

Testing: Users perform representative tasks with a given Web site and usability problems are

determined based on the range of observed user interactions (e.g., Spool, Scanlon, Schroeder,

Synder, and DeAngelo, 1999).

Inspection: Usability experts use a set of criteria (e.g., Web usability heuristics, such as those

suggested by (Nielsen, 1994)) to identify potential usability problems in the Web site design

(Nielsen and Mack, 1994).

Inquiry: Users provide feedback on the Web site via structured interviews, participation in focus

groups, responding to surveys, etc. (Schubert and Selz, 1999).

The methods we mentioned above have been adopted from the discipline of user interface (UI) testing

within the broader realm of human-computer interaction (HCI). However, even though these approaches

have been applied for the evaluation of user interfaces for traditional IS applications, they are far from

perfectly-suited for the task at hand. In fact, the Internet and World Wide Web introduce several issues

that impede with the wide applicability and effectiveness of these methods. For example, Web sites are

typically updated and redesigned very frequently, which makes the cost of recruiting users and experts for

each redesign initiative excessive for most organizations. Furthermore, it is difficult to reconstruct a

2

representative environment for user testing. Most developers and investigators do not have control over

the technologies that users employ when they visit a given Web site. For example, they are unable to

control for the various Web browsers that users select, and these often have the impact of changing the

appearance of Web pages. Moreover, they are unable to pre-specify the connection speed with which a

user accesses Web pages at a site, affecting screen presentation and download times, and the overall

quality of the user experience. It is also important to point out that in these kinds of contexts, Web-based

application users are most often customers, rather than users, as might be typical of the IS applications of

a firm within its own boundaries. As a result, even greater constraints are placed on what a designer can

do to create a setting for system use on the part of a user/customer that maintains the desirable features of

consistency.

These limitations have led researchers and practitioners to devise automated methods for Web site

evaluation (Brajnik, 2000). According to Ivory and Hearst (2000), the automation may occur at several

different levels:

capture of user testing data;

analysis of the actions of a user on a Web site; and,

automated critique of the code that constitutes the user interface.

Many such tools are now available, and some are quite innovative in the range of technical approaches

that they use. For example, WebLogger automatically records users’ actions while they interact with a

Web site (Reeder, Pirolli, and Card, 2001). The Web Static Analyzer Tool, WebSAT, (Scholtz, Laskowski,

and Downey, 1998) and WebTANGO (Ivory, Sinha, and Hearst, 2001) automatically check the HTML

code of Web pages against typical usability guidelines. Finally, WebCriteria’s SiteProfile dispatches

software agents that browse through a Web site to log download times of web pages, which are used to

evaluate the accessibility and usability of the site (WebCriteria, 1999).

We conclude from our brief review of the literature on general Web site evaluation that the major

focus has been on evaluating the usability of Web sites. Even though ease of use of Web sites may be an

important and necessary condition for the success of a Web site (i.e., people should be able to perform

tasks), however, the restricted concentration on usability that we observe only limits the applicability of

these methods in evaluating the performance of e-commerce Web sites.

2.2. E-Commerce Web Site Evaluation

The methods for Web site evaluation that we described above also can be employed for e-commerce

Web site evaluation. For example, in terms of user testing, several authors point out that it is possible to

simulate the use of an Internet-based seller’s Web site by engaging users who are given the task of finding

and purchasing items from an e-commerce storefront (e.g., Rizzuti and Dickinson, 2000; Spool et al.,

1999). In terms of inspection, Gomez Advisors (www.gomez.com) rates e-commerce Web sites based on

3

multiple criteria. The criteria typically include ease of use, customer confidence, on-site resources, and

relationship services, among others, depending on the business category of the firm’s Web site that is

evaluated. Finally, Schubert and Selz (1999) have proposed the Web Assessment Tool, a survey-based

inquiry method that is applied to evaluate the effectiveness of different phases of market transactions for

e-commerce Web sites. The phases include customer information gathering on potential products and

services, reaching agreement when negotiations between customers and suppliers take place, and settling

on the payments and logistics when the products and services are actually delivered.

Even though some of these approaches to e-commerce Web site evaluation are useful, the

fundamental limitations of each approach still hold. User testing often is not cost-effective, and in many

situations, it fails to properly measure the outcomes associated with a range of users’ or customers’

experience with a Web site. Inspection and inquiry, on the other hand, may generate useful insights about

where to focus the firm’s efforts for Web site maintenance and future additions to the software

functionality. But these approaches may not be as effective in showing the details of what actually needs

to be done to improve the Web site to increase ROI.

Furthermore, the narrow focus on usability only also limits their relevance. So, even though the

quality of the online customer experience (especially in terms of usability) has been shown to positively

affect the overall firm performance of e-commerce firms (i.e., financial performance), it still is unclear

how the usability of e-commerce Web sites translates into increased ROI and organizational performance.

Thus, we believe that it is imperative for e-commerce firms to understand how their site is performing

against business performance metrics (e.g., customer acquisition, online sales and revenue, merchandizing

effectiveness, banner advertisement return on investment etc.). However, usability evaluations for most

e-commerce Web sites fail to provide the missing link between usability and business performance, and

as a result, do not provide management decision makers with actionable information through which they

can leverage ROI.

Recent research in Web usage mining is showing promise in providing this missing link. We define

Web usage mining as the application of data mining techniques to discover patterns from Web data (e.g.,

Web site server logs) in order to understand Web usage behaviors and better serve the design and

maintenance needs of managers who are responsible for Web-based applications (Srivastava, Cooley,

Deshpande, and Tan, 2000). Early web usage mining techniques (e.g., Open Market’s Web Reporter

(www.opemmarket.com) and NetIQ’s WebTrends (www.webtrends.com) have focused on generating

insights related to general site usage. For example, Web server logs can be compiled to generate

descriptive statistics on Web site usage for questions such as: What are the most requested web pages

within the site? Where are our users coming from? And how many times was a particular banner

advertisement clicked on?

4

Current directions in research on Web usage mining have broadened the scope of analysis to perform

even more sophisticated pattern extraction related to site usage. Examples of such applications include:

Association rules: These are rules implemented by a Web usage mining tool that enable an

analyst to correlate the set of Web pages within a site that are accessed within single or

multiple specific user or customer sessions (e.g., Borges and Levene, 1998).

User clustering capabilities: Enables inferences to be made about customer demographics

for market segmentation based on observed click stream behavior on a Web site (e.g., Fu,

Sandhu, and Shih, 1999; Heer and Chi, 2001).

Classification gauges: Supports the development of individual and group-member user

profiles (e.g., Murray and Durrell, 1999).

Sequential pattern identification: This kind of application has the power to predict future

site navigation patterns based on current and prior observed click stream data and user

patterns (e.g., Pitkow and Pirolli, 1999; Schechter, Krishnan, and Smith, 1998).

Dependency modeling: Finally, the capabilities of Web usage mining software now extend

to the use of the tools to develop preliminary descriptive models representing dependencies

that occur among variables describing Web site use and the Web site user (e.g., Büchner,

Baumgarten, Anand, Mulvenna, and Hughes, 1999).

More specific to Web usage mining for e-commerce, Berry and Linoff (1997) propose market basket

analysis. This technique examines the content of Web shopping carts to infer patterns of product co-

occurrence so that cross-sells and up-sells may be targeted on the fly via recommendation systems

(Ansari, Kohavi, Mason, and Zheng, 2000).

Even though Web usage mining techniques may be capable of extracting interesting Web site usage

patterns to deepen our understanding of how customers are actually using the Web site, the link to

business performance is still lacking. Likewise, market basket analysis enables the evaluation of business

performance (i.e., sales, marketing and merchandising performance), but is incapable of linking the

measures of business performance to actual usage patterns of the website. In other words, either we can

understand how customers interact with the Web site but cannot grasp the value produced by such

behaviors or we can measure the resulting performance but cannot understand how the performance was

brought about.

Combined with our earlier critique of the automated Web site evaluation tools, our assessment of the

strengths and weaknesses of Web usage mining tools suggests to us the importance of finding a new

“common ground” for the tools and techniques of Web assessment that will better align with them with

the corporate need of justifying investments in Web-based applications and achieving higher levels of

ROI. We next turn to the presentation of an exploratory case study of an Internet-based seller, to see if

5

we can discover some new ways to think about Web site evaluation and usage mining evaluation that will

inform our efforts to better align measurement practices with management practices in electronic

commerce.

3. AN EXPLORATORY CASE STUDY OF AN INTERNET-BASED GROCERIES SELLER

For our exploratory case study, we chose to examine the operations of an Internet-based seller of

groceries. We collected rich background information and detailed descriptions of the design, operation

and current approaches to the evaluation of the company’s e-business. Our primary data sources were

unstructured face-to-face interviews with key informants who were able to offer useful insights into the

evaluation of the firm’s e-commerce Web site. To increase the validity of the findings, we used multiple

data sources and multiple informants whenever possible (Eisenhardt, 1989). We interviewed the CEO,

the CIO, an operations manager, an IT manager, a technology architect, a web content specialist and also

a web developer. We also used additional sources of data, including company documentation and reports,

which were used to complement the interviews. Finally, we built a stronger understanding of the

operations of this firm in the context of the e-commerce market-at-large, and such firms as Peapod.com

and Webvan.com in the online groceries market, by scanning the popular business press and the e-

commerce consulting company’s reports for topical studies and background information on the

performance of online grocery sellers Web sites.

3.1. Online Grocery Industry Overview

The online grocery market is a fast-growing retail sector with estimated revenues of $500 million in

1998 expanding to $70 billion by 2007, representing 12% to 15% of all grocery purchases (Andersen

Consulting, 1998). Even though groceries are not associated with the typical product mix for e-commerce

(e.g., near-commodity books and music CDs, low cost-of-delivery information goods, and hard-to-find

and niche trade goods, etc.), the online grocery business has the potential to become a highly profitable

business for e-tailers. Barsh, Crawford and Grosso (2000) report that average order sizes in Internet-

based grocery selling often are as high as $100 or more, and customers frequently make repeat orders

once a week or each fortnight to replenish their stocks. Apparently, the key to success for online grocers

is to generate sufficient volume while keeping delivery costs low (Palmer, Kallio, Saarinen, Tinnilä,

Tuunainen, and van Heck, 2000).

Despite the opportunities that we describe, firms that compete in the online grocery market also need

to overcome significant challenges in order to reach the necessary operational scale size. The premier

reason consumers do not adopt online grocery delivery is that they want to see and touch what they are

buying (Food Marketing Institute, 2000). However, despite this potential threat to initial adoption, 90%

of customers rate their initial online grocery purchasing experience as good or better than their in-store

6

shopping experience, and 85% say that they are likely to repeat the online grocery purchase. Hence, it is

essential for online grocers to offer attractive promotions backed with high quality products and services

in order to convert potential customers to actual shoppers. Once potential customers are converted, they

are likely to become loyal repeated customers.

Another potential obstacle to the success of online grocers relates to the complexity of the purchase

process. Typical supermarkets carry on the order of 40,000 to 45,000 different sale items. Grocery

shopping typically involves the purchase of numerous items in multiple product categories on a regular

basis, in contrast to bookstore and music store purchases, where the customer only seeks out a small

number of items on an infrequent basis. Hence, compared to other types of online retail, customers will

need to spend much more time exploring the Web site in order to find all the items in their grocery list.

This may be a daunting task considering the fact that usability problems often will intervene. The typical

problems that arise are difficulties with site navigation and the customer’s inability to find the products

she seeks to buy (Zona Research, 1999). Furthermore, the target segment for online grocery shopping

adoption is the common grocery shopper, who has less and less time to conduct this tedious, yet essential

task. Hence, online grocers need to make their Web sites extremely easy to use, since convenience and

saving time are the major thrusts for online grocery shopping.

3.2. OnlineGrocery.com: A Company Overview

Founded in October 1997, OnlineGrocery.com is a pure-play Internet-based retailer that delivers

groceries directly to the customer’s doorsteps with the mission of “taking the dread out of grocery

shopping.” The company made its first delivery in April 1999, and by-mid July 2000, it had over 9000

customers generating more than $16 million in revenue. Currently, OnlineGrocery.com operates only in

one metropolitan area in the upper Midwest, where it is the only online service within its regional market.

OnlineGrocery.com employs a route-based system for delivering groceries. Its registered customers

select a delivery time and shop for groceries online before a specified cut-off time. This is typically one

day prior to an assigned once-per-week delivery date. For example, if a customer’s scheduled delivery

time is 2:00 each Wednesday afternoon, then she must submit an order via the OnlineGrocery.com Web

site before 11:00 Tuesday night.

The current emphasis of the firm’s business strategy centers on growth in its customer base and

critical mass scale size to achieve operational efficiencies. The company is also expanding its territorial

scope by adding new delivery routes in different zip code areas of the metropolitan area that it serves with

the most advantageous consumer demographics. At the same time, OnlineGrocery.com strives to

distinguish itself through its uniquely high quality, individualized service so that existing customers

return to the OnlineGrocery.com’s Web site for their regular grocery shopping. Its primary mechanism is

7

with a marketing approach that positions the company’s truck-based grocery delivery staff as friendly and

accessible “delivery boys” who come to know the customer and can provide more individualized services.

3.3. The OnlineGrocery.com Web site

OnlineGrocery.com’s Web site is a typical hierarchically structured catalog-based dynamic Web

application in which customers can drill down pre-defined product categories (e.g., Beverages >

Soda/Pop > Cola) to find the products they wish to order. The product category hierarchy has four levels

with individual products descriptions at the fourth leaf node. The site also offers a search facility and a

site map showing the entire information architecture of the site. Since OnlineGrocery.com employs a

route-based system with weekly order capture deadlines, its Web site also supports multi-session

shopping carts. In other words, a customer’s shopping cart preserves its state even when she logs out or

her HTTP session times out. This permits a customer to add to her cart as the week progresses, in much

the same way that she would mark a white board with a magic marker on her refrigerator as the week

progresses, to remind herself of what she will need to buy when she makes her next shopping trip. As a

consequence of this, OnlineGrocery.com customers may shop at multiple intervals and finalize their

orders right before the cut-off deadline. Indeed, the firm promotes its service by stating that so long as an

order is received by the cut-off time – even a minute before – the customer’s order will ship the next day.

In addition to the basic information architecture of the Web site, OnlineGrocery.com features several

sections to aid customers in filling their shopping carts with desirable items and to shift attention to higher

margin items for the grocer. These include weekly specials, recipe suggestions, personalized order

history information and new items. The specials section is updated weekly with promotional items (i.e.,

branded food items with special discounts) and seasonal items (e.g., all items related to barbeque

cookouts in time for Memorial Day). The recipes section features ideas for meals with a list of

ingredients and the recipe for each meal. Customers may print the recipe and buy any or all the

ingredients required to prepare the dish. They can also populate their order basket with the required

ingredients at the click of a button on recipe item’s screen display. The order history section lists a

customer’s last eight orders so that she can keep track of grocery budget information or use it as a

personalized shopping list to reorder any frequently-purchased items. Finally, since OnlineGrocery.com

is continuously expanding its product selection, the new items section shows the recent additions to the

site’s list of products.

The OnlineGrocery.com Web site has gone through significant changes, upgrades and redesigns since

its launch in April 1999. In order to better manage its development process, OnlineGrocery.com has

adopted the Unified Modeling Language (UML) (Booch, Rumbaugh, and Jacobson, 1999) and Rational

Software’s unified process approach (Jacobson, Booch, and Rumbaugh, 1999), the new de facto standard

methodology for object-oriented software development. However, this does not mean that the firms’ Web

8

site is without problems as a result of the development process OnlineGrocery.com has selected. Instead,

a major challenge for managing the Web site arises due to the abundance of ideas for redesigns that have

been offered, and the new capabilities that the rapid software development tools offer.

A significant source of ideas for redesign is OnlineGrocery.com’s personnel. All employees are also

regular customers of the online service and, hence, are intimate with the strengths and weaknesses of the

Web site’s design. The employees identify and suggest areas where the site may be redesigned to improve

the overall shopping experience for OnlineGrocery.com’s shoppers. Other sources of ideas to improve

the Web site include the online feedback forms that permit customers to post questions and suggestions,

the customer service call center, and occasional customer focus groups. These sources produce many

more proposals for development projects than the software development staff and the IS budget at

OnlineGrocery.com can handle.

3.4. Web Site Performance Evaluation at OnlineGrocery.com

Performance evaluation of the firm’s Web site at OnlineGrocery.com has multiple purposes. First,

performance evaluation is carried out to assess and manage the business, and to assure investors that the

their invested funds are deployed in a manner that has the potential to create significant returns. Second,

performance evaluation of the Web site is employed to find ways to improve the business process that

customers participate in when they shop, and, as a result, firm performance. Similar to many other Web-

based businesses, OnlineGrocery.com has adopted the attitude that competent measurement is a precursor

to formulation of effective management policy for the firm’s Web operations. With this goal in mind,

management spends time to do Web site performance evaluation so that it can generate insights into how

the Web site is operating, what changes are required to improve service quality, and why one change

might be given a greater priority another, due to the relative leverage on ROI that each may provide.

Toward this end, OnlineGrocery.com has defined and tracks several key business metrics: conversion (i.e.,

new customer acquisition), dollar ring (i.e., actual sales), margin (i.e., profitability of sales items) and

frequency (i.e., loyalty).

Currently, the data for estimating these business metrics are derived from two separate systems. One

is the customer data warehouse and the other is a Web site analysis tool that is provided as a bundled

service by the Texas-based application service provider that hosts the firm’s Web site. The data

warehouse, which contains customer and sales data, is used to conduct market basket analysis (Berry and

Linoff, 1997). For example, final sales statistics are used to answer questions such as: “What are our best

selling products?” “What are the demographics of our customer segments?” And “What is the average

profitability for each customer segment?” This analysis is valuable for assessing the overall performance

of the online service. However, it provides very little managerially-actionable information about how to

improve OnlineGrocery.com’s Web site.

9

The Web site analysis tool is employed towards this second goal. The Web site analysis tool,

WebTrends, is a first-generation Web usage mining tool that compiles Web server logs to generate Web

site hit statistics. The analysis tool offers a series of pre-packaged reports that show various aspects of

online activity. For example, some reports list the most requested pages, whether the page hits come to

OnlineGrocery.com through external “referring” Web sites, the browsers that are used by people who

visit the site, the number of hits and visits for a given date range on different parts of the Web site, and the

most frequently occurring HTTP errors. These reports are used to answer very questions such as “What

are the most popular product categories?” “What are the most popular products?” “When do customers

shop?” “How long are the typical shopping sessions?” and “What is the ratio of customers who shop or

browse vs. customers who purchase?” A shortcoming of the ready-made reports is that they are designed

to only list a set of 200 statistics, constraining the extent to which the tool can extract useful data to

support a variety of managerial decision and evaluation tasks. For example, like a typical grocery store,

the number of products offered by OnlineGrocery.com is much greater than 200, so it is impossible for

the firm’s management to acquire a complete and accurate view of site usage, if they wish to track more

than this number of products. As the reader can imagine, this has become a major source of frustration

for the firm’s managers. They often are more interested in identifying the least visited areas of the Web

site and the pre-packaged statistics tend to focus more on the most frequently-visited pages (e.g., the

home page, the check out page, the sale items page, and so on).

Thus, we see that the analysis tools and techniques that OnlineGrocery.com currently can use are

limited. They provide two extreme views of performance. The analysis performed with the customer

data warehouse is only able to convey information about high-level business performance (e.g.,

merchandising and marketing effectiveness). In contrast, the Web site analysis tools are only capable of

depicting inflexible, low-level raw statistics of site usage. It is difficult to bridge the gap between high-

level business performance and low-level site usage.

Table 1 presents OnlineGrocery.com’s “wish list” of analysis capabilities that we distilled from our

field interviews with the firm’s senior management team and its Web operations staff. The reader should

note that in contrast to the kinds of data that the tools we discussed can deliver, the “wish list” requires

data that provide a richer representation of detailed online consumer behavior. This data would bridge

the gap between high-level business performance metrics and low-level site usage. Moreover, it is

important to recognize the critical issue. Substantially all of the most actionable information that Web

site evaluation and usage mining tools need to deliver to improve management’s is in this unmapped

“middle ground.” It simply is not being made available at OnlineGrocery.com, and at many other firms

that we have adopted similar performance evaluation approaches based on the tools and techniques that

currently exist in the marketplace.

10

Table 1. OnlineGrocery.com's “Wish List” of Site Performance Analysis Capabilities

MANAGEMENT’S QUESTIONS ROI-ENHANCING ANSWERS APPLICABLE BUSINESS LEVER

Where are people hitting the buy button: order history, specials, categories, search results, etc.?

Understand where sales are generated to estimate value of screen estate, and to guide which areas of the site to improve

Dollar ring Margin

How many clicks does it take people to get items in their carts?

Guide design to reduce clicks Frequency

What is the conversion rate of promotions?

How many people saw / clicked on / bought promotion items?

Understand the best way to promote Dollar ring Margin

What are the top X searches with results?

Infer customer needs Frequency Dollar ring

What are the top X searches that yield 0 results?

Populate product synonym list Improve product descriptions Infer customer needs

Frequency Dollar ring

Where are people aborting? How many abandoned carts are

there by age of cart?

Identify technical problems Guide design to reduce technical

problems

Frequency

Do people shop in one trip? Or several?

Does this vary by market / customer segment?

What are average times per trip? What is average time to fill a cart,

by combining multiple trips?

Understand how people shop to redesign Web site for ease of use

“Dial in” ROI in use-specific contexts Find a basis for identifying how to

redesign to create greatest leverage for maximizing ROI by market / customer segment

Frequency

Are people browsing / looking for specific items when they shop?

Improve information architecture Frequency Dollar ring

What features are customers asking for on our site?

Prioritize development projects Frequency

Web availability problems? - System up 24 x 7? - Peak hours?

Identify technical problems Explain variance in order volume Design to service maximum usage

Frequency

Web response time? - During shopping? - During checkout?

Identify technical problems Identify situations where unacceptable

site performance is most likely to create conditions for lost revenues

Frequency

State of the “digital shelves”? - number of items in the store - number of items in stock - number of missing photos

Number of products out of stock during check out

Understand customer shopping experience

Identify dollar ring issues to determine product mix inefficiencies that may diminish site potential for generating revenues

Frequency Dollar ring

Customer’s client technology? - What browsers are people using? - What platforms? - How many AOL users?

Develop style guides Understand technology design space Define the range of “necessary

standards” with which to comply

Conversion

Are credit cards being rejected? Understand customer shopping experience

Conversion

11

Even though our criticism of the limited set of capabilities that OnlineGrocery.com has for identifying

the key drivers that will enhance the ROI of its Web site has been harsh, we should point out as we close

our case discussion that the technical state-of-the-art in this area is still somewhat lacking. For example,

to more fully understand the “inner workings” of online consumer behavior at OnlineGrocery.com’s Web

site would require multi-session pattern mining. This differs from the approaches that we have discussed

in that data which might enable an analyst to make sense of the observed click stream behavior would

need to come from a time-series of sessions in which usage data are mined by a tool. Understanding

human behavior in this context, and being able to take the next step – to discerning what it means in terms

of the ROI consequences for the Web site – is a more challenging problem.

As a result, determining exactly what to do in order to optimize service quality, enhance the product

selection, improve the usability characteristics and maximize ROI for the firm’s Web site requires debate

and discussion, as well as a “gut feel” for what the next steps ought to be. To prioritize and decide on

what development projects to pursue, OnlineGrocery.com holds monthly “Web Board” meetings where

senior executives discuss potential development projects and set high-level priorities and business goals.

A “Web Team” meeting is also held weekly, and the staff member discuss details of the development

projects that are planned and underway. This process results in development projects being prioritized

based on estimated impact on key business performance metrics (e.g., customer acquisition, customer

conversion, dollar ring etc.), which act as a loose proxy for ROI. However, the estimation is rather ad

hoc, and intuition plays a more significant role than it should, given the critical importance of ensuring

that the firm has a high performance Web site as the front-end presence for interacting with its customers.

Our exploratory case study of OnlineGrocery.com reveals several interesting insights into the

requirements for value-driven Web site performance evaluation metrics. The insights may be

summarized into the following:

Metrics should reflect business value. OnlineGrocery.com’s Web site analysis tool provides

useful information to infer customer interaction with the Web site. However, it is difficult to

associate behavior with ROI-related outcome. Instead, the appropriate measures are those that

can show more directly how the tie-in works.

Metrics need to be managerially-actionable. The goal of performance evaluation is ultimately

to devise ways to improve business performance. The metrics that are obtained in Web site

evaluation should guide the generation of ideas for improving the Web site. For example,

knowing that product X is the best selling product for this week is good to know, but there is

nothing that can be done about it. Instead, it would be more appropriate to have specific

information that indicates the effectiveness of the placement of product promotions, and the

extent to which different kinds of placement create marginal impacts in sales. With this kind of

12

information in hand, management would have more information about how to place information

so as to maximize the ROI associated with the screen real estate that is being used.

Measurement should be at appropriate operational level. OnlineGrocery.com’s technology

support for performance evaluation was either too high-level (i.e., customer data warehouse) or

too low-level (i.e., log analysis). However, the most pressing questions were at the middle

ground where low-level click stream information is mapped with high-level site performance.

The appropriate metrics are those that make “light up” the dark pockets of opportunity to enhance

selling effectiveness and increase revenues through the Web site.

Metrics should be flexible. The maintenance and upgrade of an e-commerce Web site is

performed at increasingly fast rates. Accordingly, the goals of, and questions related to site

redesign will change continuously. Hence, the metrics that will be employed need to be flexible

enough so that they may adapt to changing goals and questions on the part of senior management.

4. A FRAMEWORK FOR WEB SITE EVALUATION

The exploratory case study of OnlineGrocery.com presents a useful basis for theorizing about what a

value-based framework for e-commerce Web site evaluation should consist of. The major emphasis in

our framework is on business value and ROI. We believe that prior approaches to Web site evaluation

lack this emphasis and hence do not provide adequate guidance in the management of e-businesses. This

section presents our proposed framework for Web site evaluation.

4.1. Extrapolating from OnlineGrocery.com: Making Sense of Web Performance Metrics

Our findings from the prior discussion of OnlineGrocery.com’s efforts to measure its Web site

performance suggest to us that there are a number of important considerations that will help us build

towards a theoretical basis for a new Web site metrics suite proposal. They include: customer

participation in different phases of a transaction at a Web site, the extent to which different customer

segments use the same Web site, the time-series nature of customer interactions and experiences with a

Web site, and the tension between observable, measurable aspects and unobservable, more difficulty to

measure aspects of customer behavior on a Web site. Taken together, these things help of to make sense

of what Web performance metrics should be like, because it informs us about the situations and issues

that they must treat.

First, in order to better understand how and where value is created via the Web site, we need to break

the online customer interaction with the site based on different phases of the transaction. Unlike

informational Web sites, the e-commerce Web site is composed of various functions to facilitate the

consumer purchase process (Miles, Howes, and Davies, 2000). Schubert and Selz (1999) characterize the

three transaction phases of information, negotiation and settlement. In all three phases of the transaction

13

process are not effectively supported by the Web site, then value leakages may occur. Value leakages

occur when the realized value of IT in some applied setting compares unfavorably with the potential value

that was envisioned when an investment was made (Davern and Kauffman, 2000).

Web performance metrics need to identify the sources of value leakages so that senior management

may plan for and implement revamped and refined designs that rectify the situation and restore the

necessary qualities of the Web site to achieve an acceptable level of ROI. For example, one of the major

questions that management at OnlineGrocery.com has asked is where customers most often “abort” the

shopping process. Their concern is that customers have every intention to actually place an order with the

firm, but various aspects of the business process, Web site service quality and customer information

requirements “defeat” the customer’s best efforts. As a result, customers may become dissatisfied at

various stages that include:

during the initial sign-up stage, when concerns about security may hinder the adoption of the

online service;

during the browsing phase, when customers may not find their preferred products, either because

the product is not offered by the company or because of poor navigation and information

architecture design that prevents customers from finding a product even if it is offered at the site;

during checkout, when the tedious process of specifying delivery and payment methods causes

frustrations, or where delays involving with late response times occur due to ineffective

transaction processing through a separate back-end ERP system; and,

when customers finally receive their products but are unsatisfied with their quality (e.g., not-so-

fresh vegetables, cracked eggs, the wrong brand of morning cereal, etc.)

Second, in order to better assess the potential for value creation, Web performance metrics should

take into consideration the heterogeneous nature of different customer segments. Customers at e-

commerce sites are not homogeneous. Different customer segments have different preferences, spending

budgets and habits. For example, when considering what new products to offer, OnlineGrocery.com

frequently makes use of feedback from customers requesting that new items be offered, and also

indirectly infers this need by analyzing the search terms that customers use for exploring the products

space. Even though this is valuable information in identifying the needs of customers, it is difficult to

prioritize what products are essential and what products can wait. To properly support a value-driven

discipline for Web site design, however, this information must be linked in some way to the customer

demographic profiles, so that we may estimate which new additions will generate the greatest impact.

Third, Web performance metrics should be sensitive to changes in the customer life-cycle of

interaction at a Web site, and hence, must be time-series in nature. Customer segments are not static;

instead they progress through an observable (or, at least, a discoverable) life-cycle. For example,

14

customers start as “aware non-tryers” when they become aware of the site through advertising or word-of-

mouth. Later they become “tryer non-buyers” when they come to the Web site to take a look around to

assess whether the online service might be of value. Still later, they may become “converted customers,”

when they actually submit an order for the first time. Thereafter, they may progress to achieve a highly

desirable status from the point of view of the Internet-based seller: “repeat customers” who come back for

additional purchases when the need arises. Finally, customers may attain the highest level and become

“loyal customers,” when their Web site use becomes habitual (e.g., one a week or even once each day,

when information goods purchases are involved).

Not only do the different stages of the customer life-cycle offer different levels of potential for value

creation as we know is the case for different customer segments, important behavioral changes occur as

customers progress through the life-cycle, too. For example when a customer returns to a seller’s Web

site that she has used before, or if she begins to use a specific site on a more frequent basis, then her

familiarity with the site increases accordingly. In terms of user interaction, the customer will start to

develop expertise and find short cuts for attaining their shopping goals. If the Web performance metric

were not sensitive to this time-series behavioral change, it would be extremely difficult to make sense of

changes in site usage patterns.

A second example illustrates this assertion. A major question that managers at OnlineGrocery.com

ask is at what point in the sequence of shopping-related activities customers actually click on the “BUY”

button. This event may occur at the level of the general product category pages (where few specific

details about a product are shown), at the customer’s order history page (where no product information

other than price is shown), on the specials and discounted products pages (where price is emphasized, but

the customer can still click another level down for details), or as a result of search (where multiple

branded instances of the same purchase item are returned), and so on. We would like to understand how

behavior changes over time so that we may predict similar behavioral changes for other customers, as

well as use this information when evaluating how much leverage a new feature on the Web site may have

to induce specific kinds of behavior that are not currently elicited.

Finally, Web performance metrics need to be measurable from observations of online customer

behavior. We remind the reader that the major shortcoming of the analysis tools employed at

OnlineGrocery.com was that the observed behavior for the analysis was either too high (e.g., final sales

data) or too low (e.g., page views). In order to precisely attribute online consumer behavior to the sources

of value creation and leakages, data at multiple levels of operation must be integrated. Only then can we

accurately measure the value of the Web site that brings about the resulting consumer behavior. Table 2

presents a summary of the key metrics characteristics for value-driven Web site performance evaluation.

15

Table 2. Key Metrics Characteristics to Support Value-Driven Web Site Design

DIMENSIONS RATIONALE Phase-focused Infer where value creation and leakages occur Customer-specific Assess the relative potential for value creation Time-series in nature Consider changes in customer behavior Measurable from observations Attribute consumer behavior to value sources

4.2. A New Framework for Value-Driven E-Commerce Web Site Design

Now that we have established what characteristics the Web performance metrics should posses, we

may start to think about a more general framework for value-driven performance evaluation and design

for Internet-based selling Web sites. We start by defining the major elements of the framework, and

follow that with a discussion of its logic and inner workings.

Framework Elements. Our framework is based upon a model of online consumer buying behavior.

The first element of the framework is the customer coming to the Web site to address some unmet need,

which may be satisfied by the acquisition of a product. Next, in order for the customer to acquire the

product, she must go through a transaction process which consists of three phases: information,

negotiation and settlement. This transaction process is achieved by virtue of some observable behaviors –

interaction with the Web site. Finally, this process may repeat itself over time, thereby changing the level

of expertise the customer has related to the site design or changing the level of knowledge the site has

related to the customer, so as to be able to provide more customized services to each customer. Hence,

our framework consists of five top-level elements as depicted in Figure 1 – customer, product, transaction

process, behavior and time – which may be further broken down into lower-level components. Let’s

consider each one in a little more depth.

Figure 1. Framework Elements for Value-Driven Web Site Design

Time

Customer Products

Transaction Phases

Information Negotiation Settlement

BehaviorCustomer Products

Transaction Phases

Information Negotiation SettlementSettlement

Behavior

Time

Customer Products

Transaction Phases

Information Negotiation Settlement

BehaviorCustomer Products

Transaction Phases

Information Negotiation SettlementSettlement

Behavior

16

Customers. In order to achieve high levels of ROI, it is important to understand in detail

characteristics of the customers. Customers have demographic particularities, such as

preferences for items, spending budgets and spending habits, which make some more profitable

to target than others for a firm. In addition, we need to consider the dynamic nature of customer

relationships. Customers progress through a life-cycle from potential customers, to acquired

customers, to converted buyers, to repeat purchasers, finally to loyal customers.

Products. Merchandising is important for the success of e-commerce Web sites. Customers

typically are looking for wide assortment of items that a Web site should offer, but it is the role of

management to selecting the value-maximizing breadth of the product line that is available. In

addition, products also have varying levels of margins, and this makes some items more

profitable to sell than others for the firm. More importantly, making the right decision about the

manner in which higher and lower margin products are presented and promoted to customers play

an important role in profitability. Also, the online seller needs to think about the circumstances of

the product purchase and usage. For example, an online grocery may think of products as

individual items that customers need. However, grocery shopping is not merely a matter of the

customer stacking all of the selected items in the shopping list into a shopping cart. Products may

be aggregated into larger groupings that are more meaningful to the customers. For example,

customers may buy sausages and buns, not because they need sausages and buns, but to prepare

hotdogs for a picnic. In a broader context, picnics may be planned for a seasonal occasion (e.g.,

Memorial Day holiday), which suggests that products purposefully grouped in bundles and sets,

matching the varied needs of customers.

Transaction Process. The customer needs to go through a transaction process of information,

negotiation and settlement in order to satisfy her need of acquiring some product. As a result, the

effectiveness of a Web site must be evaluated based on the separate and individual efficacy of the

component phases. Each component phase may have different requirements for success. For

example, in the information phase (which consists of need recognition, information search and

evaluation), the quality of product description and the ease of navigation within the Web site may

be the most important factors for the customers. In the negotiation phase where the trade terms

are agreed upon, however, the key elements may shift; security concerns, for example, may take

on an increased importance. Convenience and efficiency may also play significant roles. Most

customers are unwilling to tolerate processing delays. They typically think that the shopping

process is over when all items are in the shopping cart and the checkout button is pressed. Too

often, however, customers fail to recognize that there is more to transaction completion than just

the checkout. Finally, for settlement processes involving instances in which the product is

17

delivered to the customers, flexibility may be the critical factor. Customers may wish to change

the delivery time due to unforeseen circumstances, they may wish to adjust an order (even after it

has been submitted and logged), or they may wish to cancel an order.

Behavior. In order to support transaction processes that bring together customers and products,

the Web site should be designed to provide affordances for the types of behaviors that faithfully

lead to the success of the transaction processes. Users are heterogeneous in terms of the patterns

of behaviors they exhibit when interacting with a Web site. In addition, patterns of usage may

differ across customer segments and also within individual customer segments. Hence we need

to think about how the design of the Web site will affect user-site interaction, and link this

information back to customer profiles so as to assess what types of behavior will increase the ROI

of the company and what types of Web site design will generate such behaviors. Consider a

hypothetical situation where the online seller is planning to improve the functionality of browse

or search actions on the Web site. These are typically-used navigation mechanisms for finding

information within the site. Imagine that an analysis of previous Web logs reveal that customers

in the repeat purchase stage of the life-cycle tend to browse through the site to fill their carts.

However, new customers, who are exploring the site to see whether they will adopt the service,

most often use the search facility to see if their preferred items are offered and in stock. If the

goal of the company were focused on growth (i.e., to attract more customers and convert them

into shoppers), then focusing on the search facility would be more beneficial. On the other hand,

if the firm’s goal were to provide better service to its customers and increase the cart size –

enhancing revenue per customer shopping trip -- focusing on improving the browse capabilities

would be more beneficial.

Time. Customer interactions with a Web site are not static; they change over time in various

(and, we imagine), predictable ways. As customers increasingly interact with the site, their

familiarity with the site increases. As a result, the customers build expertise related to their site

use and develop shortcuts for attaining shopping goals. Customers also progress through the life-

cycle. Consequently, patterns of site usage evolve. These can be discovered, catalogued and

characterized, both descriptively and in conceptual terms, and using Web mining techniques that

drill out patterns of observed Web site interaction and consumer behavior. At the same time as

patterns of site usage evolve on the customer’s part, the online seller becomes more

knowledgeable about the customers’ needs and patterns of site usage, enabling the seller to

provide more customized services. The provision of customized services through improved site

redesigns that are responsive to customer interaction and behavior patterns also have the potential

to change how customers interact with the site. The product dimension changes as time passes as

18

well. Over time, the assortment and aggregation of items offered at the site will vary based on

merchandising activities. The change in product selection will invariably alter customers’

purchasing decisions as well as profits the site may generate. Finally, the transaction processes

can also change over time. For example, the online seller may opt to provide other ways of

getting the products to the customers (e.g., distributed pick-up centers where customers drive

through and pick up their groceries on the way home), or other means of price negotiation (e.g.,

dynamic pricing instead of fixed pricing). All these changes will alter the performance of the site,

and the online seller needs to be sensitive to them all.

Table 3 summarizes the major elements of our proposed framework.

Table 3. Framework Elements

ELEMENTS COMPONENTS Customers Customer demographics

Customer life-cycle Products Assortment

Margins Circumstances of purchase and usage

Transaction process Information quality Usability Security Convenience Efficiency Flexibility

Behavior Patterns of usage Site Design

Time Change in patterns of usage Change in customer life cycle stage Increasing knowledge about customers

The Framework. Figure 2 presents a high-level representation of our proposed framework for value-

driven Web site design. The process of value-driven Web site design is an iterative process of evaluation

and design, which is driven by value-oriented questions. They include some of the following:

How are our customers using our site?

How can we redesign to increase business performance?

How do we want customers to use our site?

Did the customers use the site as we intended?

19

Figure 2. Framework Elements for Value-Driven Web Site Design

EC Web SiteEvaluation

EC Web SiteDesign

How are customers using our site?

How can we redesignthe site to increase

business performance?

How do we wantcustomers to use

our site?Did customersuse the site as

intended?

EC Web SiteEvaluation

EC Web SiteDesign

How are customers using our site?

How can we redesignthe site to increase

business performance?

How do we wantcustomers to use

our site?Did customersuse the site as

intended?

At the heart of this framework lies the ability to bridge the gap between high-level business

performance metrics and low-level site usage metrics. On the low-level end, Web usage mining is

conducted to identify recurring site usage patterns. For example, site usage patterns are used to infer

where in the transaction process customers may be aborting the shopping process, or what products are

most requested, viewed and purchased. These patterns of site usage are used to identify problems or

potential enhancement ideas for site redesign. On the high-level end, data mining and profiling can be

performed on the customer data warehouse to identify multi-dimensional customer profiles. The resulting

profiles express the customers’ life-cycle stages, shopping habits (e.g., shopping frequency), profitability

(e.g., average shopping cart sizes). The two analyses are integrated to identify site usage patterns by

different customer profiles, which we use to estimate the ROI of site redesign initiatives identified

through Web usage mining by asking questions such as:

Who (which customer segment) will be most affected by the redesign?

How many customers will be affected?

What will be the behavioral / economical changes as a result of the redesign? For example, will

the redesign reduce shopping times? Or will it lead to larger shopping carts?

We acknowledge that the proposed framework is still in its preliminary stages and much refinement

and expansion is needed. However, we believe that our framework provides a sound basis for value-

driven Web site performance evaluation that may be generalized to a broader population of Internet-based

retailing businesses, since the same kinds of concerns for ROI outcomes arise with other Web sites.

5. CONCLUSIONS AND LIMITATIONS

The case study of OnlineGrocery.com presented in this paper offers several interesting insights into

the evaluation of Web sites and web-based services. The secret to success in the online world is not

radically different to succeeding in the physical brick-and-mortar world. Effective performance

20

evaluation is imperative in order for effective management of the business. The key to winning is

acquiring the knowledge about the needs of potential and existing customers and then taking advantage of

the ability to cost-effectively establish products and services that satisfy those needs. The only difference

is that e-businesses need to micro-manage the delivery of their products and services. They especially

need to understand and manage the way their customers behave online, so that they come to appreciate

how to respond effectively and satisfy customer needs. To progress toward this goal, firms that are

focused on Internet-based selling should not just track and evaluate business performance. They should

also work towards linking ROI performance to a detailed analysis of online consumer behavior, so that

they may understand why the business performance resulted as such and design improved ways to

increase performance.

The limitations of this paper arise from the case study methodology, which makes it difficult to

generalize to the broader population of e-commerce companies. However, we intended the purpose of the

case study to be a rich illustration of current practices to generate insights and future research directions

and not to act as a means to test hypotheses. Further research is required to further refine and test the

efficacy and validity of the framework that we propose for Web site performance evaluation. Even

though it is clear that the current state-of-the-art in e-commerce performance evaluation is still limited, we

have made a significant effort in this article to argue that the time has now come to shift the focus towards

better measurement and management of Web-driven e-commerce investment ROI. For example, current

techniques for web usage mining need to incorporate more accurate models of consumer behavior (e.g.,

multi-session pattern discovery). Furthermore, web usage mining for e-commerce needs to integrate data

across heterogeneous information systems (i.e., Web server logs with customer data warehouse). We are

currently conducting a multi-year longitudinal research project with OnlineGrocery.com to develop a

framework for the performance evaluation of Internet-based seller Web sites. We also hoping to extend

our efforts to develop a formal basis for value-driven discipline for Web-based systems design and to

extend Web usage mining techniques to better fit various kinds of selling environments on the World

Wide Web.

REFERENCES

Andersen Consulting. (1998). "On-line Grocery Shopping on Track for Rapid Growth" (Report): Andersen Consulting/Accenture, January 20.

Ansari, S., Kohavi, R., Mason, L. and Zheng, Z. (2000). "Integrating E-Commerce and Data Mining: Architecture and Challenges". In R. Kohavi, M. Spiliopoulou and J. Srivastava (Eds.), Proceedings of the 6th ACM International Conference on Knowledge Discovery and Data Mining (WEBKDD 2000 Workshop) (SIGKDD 2000), Boston, MA, August 20-23, New York, NY: ACM Press, available on the Internet at http://robotics.stanford.edu/~ronnyk/WEBKDD2000/papers/.

21

Barsh, J., Crawford, B. and Grosso, C. (Fall 2000). "How E-Tailing Can Rise from the Ashes". The McKinsey Quarterly, 2000(3), pp. 98-109.

Berry, M. J. A. and Linoff, G. (1997). Data Mining Techniques for Marketing, Sales, and Customer Support. New York, NY: John Wiley and Sons.

Booch, G., Rumbaugh, J. and Jacobson, I. (1999). The Unified Modeling Language User Guide. Reading, MA: Addison Wesley.

Borges, J. and Levene, M. (1998). "Mining Association Rules in Hypertext Databases". In G. Piatetsky-Shapiro (Ed.), Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining (KDD '98), New York, NY, August 27-31, Menlo Park: CA: AAAI Press, pp. 149-153.

Brajnik, G. (2000). "Automatic Web Usability Evaluation: What Needs to Be Done?" In P. Kortum and E. Kunzinger (Eds.), Proceedings of the 6th Conference on Human Factors and the Web (HF&W 2000), Austin, TX, June 19, available on the Internet at http://www.tri.sbc.com/hfweb/.

Büchner, A. G., Baumgarten, M., Anand, S. S., Mulvenna, M. D. and Hughes, J. G. (1999). "Navigation Pattern Discovery from Internet Data". In B. M. Masand and M. Spiliopoulou (Eds.), Proceedings of the 5th ACM International Conference on Knowledge Discovery and Data Mining (WEBKDD '99 Workshop) (SIGKDD '99), San Diego, CA, August 15, New York, NY: Springer, pp. 25-30.

Davern, M. J. and Kauffman, R. J. (Spring 2000). "Discovering Potential and Realized Value from Information Technology Investments". Journal of Management Information Systems, 16(4), pp. 121-143.

Eisenhardt, K. M. (1989). "Building Theories from Case Study Research". Academy of Management Review, 14(4), pp. 532-550.

Food Marketing Institute. (2000). "The e-Tail Experience: What Grocery Shoppers Think About Online Shopping" (Report #1349). Washington DC: Food Marketing Institute.

Fu, Y., Sandhu, K. and Shih, M.-Y. (1999). "Clustering of Web Users Based on Access Patterns". In B. M. Masand and M. Spiliopoulou (Eds.), Proceedings of the 5th ACM International Conference on Knowledge Discovery and Data Mining (WEBKDD '99 Workshop) (SIGKDD '99), San Diego, CA, August 15, New York, NY: Springer, available on the Internet at http://www.umr.edu/~yongjian/pub/pub.html.

Heer, J. and Chi, E. H. (2001). "Identification of Web User Traffic Composition using Multi-Modal Clustering and Information Scent". In J. Srivastava and J. Ghosh (Eds.), Proceedings of the 2001 SIAM Conference on Data Mining (Workshop on Web Mining) (SDM 2001), Chicago, IL, April 7, Philadelphia, PA: SIAM Press, pp. 51-58.

Ivory, M. Y. and Hearst, M. A. (2000). "The State of the Art in Automated Usability Evaluation of User Interfaces" (Working Paper UCB/CSD-00-1105). Berkeley, CA: School of Information Management and Systems, UC Berkeley, June.

Ivory, M. Y., Sinha, R. R. and Hearst, M. A. (2001). "Empirically Validated Web Page Design Metrics". In J. A. Jacko, A. Sears, M. Beaudouin-Lafon and R. J. K. Jacob (Eds.), Proceedings of the 2001 ACM Conference on Human Factors in Computing Systems (CHI 2001), Seattle, WA, March 31-April 5, New York, NY: ACM Press, pp. 53-60.

Jacobson, I., Booch, G. and Rumbaugh, J. (1999). The Unified Software Development Process. Reading, MA: Addison Wesley.

22

Miles, G. E., Howes, A. and Davies, A. (2000). "A Framework for Understanding Human Factors in Web-Based Electronic Commerce". International Journal of Human-Computer Studies, 52(1), pp. 131-163.

Murray, D. and Durrell, K. (1999). "Inferring Demographic Attributes of Anonymus Internet Users". In B. M. Masand and M. Spiliopoulou (Eds.), Proceedings of the 5th ACM International Conference on Knowledge Discovery and Data Mining (WEBKDD '99 Workshop) (SIGKDD '99), San Diego, CA, August 15, New York, NY: Springer, pp. 7-20.

Nielsen, J. (1994). Top 10 Heuristics for Usability. UseIT: Jakob Nielsen's Website, Nielsen/Norman Group, Fremont, CA. available on the Internet at http://www.useit.com/papers/heuristic/heuristic_list.html.

Nielsen, J. and Mack, R. L. (Eds.). (1994). Usability Inspection Methods. New York, NY: John Wiley and Sons.

Palmer, J., Kallio, J., Saarinen, T., Tinnilä, M., Tuunainen, V. K. and van Heck, E. (September 2000). "Online Grocery Shopping Around the World: Examples of Business Models". Communications of the AIS, 4(Article 3), available on the Internet at http://cais.aisnet.org/contents.asp with subscription only.

Pitkow, J., E. and Pirolli, P. L. T. (1999). "Mining Longest Repeated Subsequences to Predict World Wide Web Surfing". In F. Douglis, E. A. Brewer, P. Honeyman, D. B. Johnson, P. Krishnan, G. H. Kuenning, Y. Maarek, U. Manber, J. Mogul and K. Obraczka (Eds.), Proceedings of the 2nd USENIX Symposium on Internet Technologies and Systems (USITS '99), Boulder, CO, October 11–14, Los Alamitos, CA: IEEE Computer Society, available on the Internet at http://www.usenix.org/events/usits99/.

Rajgopal, S., Venkatachalam, M. and Kotha, S. (2001). "Does the Quality of Online Customer Experience Create a Sustainable Competitive Advantage for E-commerce Firms?" (Working Paper). Seattle, WA: School of Business Administration, Washington University, April.

Reeder, R. W., Pirolli, P. L. T. and Card, S. K. (2001). "WebEyeMapper and WebLogger: Tools for Analyzing Eye Tracking Data Collected in Web-use Studies". In J. A. Jacko and A. Sears (Eds.), Proceedings of the 2001 ACM Conference on Human Factors in Computing Systems (CHI 2001-Extended Abstracts), Seattle, WA, New York, NY: ACM Press, pp. 19-20.

Rehman, A. (2000). "Holiday 2000 E-Commerce: Avoiding $14 Billion in "Silent Losses"" (Research Report). New York, NY: Creative Good, October.

Rizzuti, K. and Dickinson, J. (2000). "Satisfying the Experienced On-Line Shopper: Global E-Shopping Survey" (Research Report). London: A.T. Kearney.

Schechter, S., Krishnan, M. and Smith, M. D. (1998). "Using Path Profiles to Predict HTTP Requests". In P. Thistlewaite and H. Ashman (Eds.), Proceedings of the 7th International WWW Conference (WWW7), Brisbane, Australia, April 14-18, New York, NY: Elsevier Science Publishing, available on the Internet at http://www7.scu.edu.au/programme/fullprog.html.

Scholtz, J. C., Laskowski, S. and Downey, L. (1998). "Developing Usability Tools and Techniques for Designing and Testing Web Sites". In J. Cantor (Ed.), Proceedings of the 4th Conference on Human Factors and the Web (HF&W '98), Basking Ridge, NJ, June 5, available on the Internet at http://www.research.att.com/conf/hfweb/.

Schubert, P. and Selz, D. (1999). "Web Assessment: Measuring the Effectiveness of Electronic Commerce Sites Going Beyond Traditional Marketing Paradigms". In R. H. Sprague (Ed.), Proceedings of the 32nd Hawaii International Conference on System Sciences (HICSS 32), Maui, HI, January 5-8, available on CD-ROM only.

23

Spool, J. M., Scanlon, T., Schroeder, W., Synder, C. and DeAngelo, T. (1999). Web Site Usability: A Designer's Guide. San Francisco, CA: Morgan Kaufmann Publishers.

Srivastava, J., Cooley, R., Deshpande, M. and Tan, P.-N. (2000). "Web Usage Mining: Discovery and Applications of Usage Patterns from Web Data". SIGKDD Explorations, 1(2), pp. 12-23.

Varianini, V. and Vaturi, D. (Winter 2000). "Marketing Lessons from E-Failures". The McKinsey Quarterly, 2000(4), pp. 86-97.

WebCriteria. (1999). "Max, and the Objective Measurement of Web Sites" (White Paper). Portland, OR: WebCriteria.

Zona Research. (1999). "Shop Until You Drop? A Glimpse into Internet Shopping Success" (Zona Assessment Paper). Redwood City, CA: Zona Research.

24