Automation and Customization of Rendered Web Pages Rob Miller User Interface Design Group MIT Computer Science & Artificial Intelligence Lab [email protected].

Automation and Customization of Rendered Web Pages

Rob MillerUser Interface Design GroupMIT Computer Science & Artificial Intelligence [email protected]://uid.csail.mit.edu/

Joint work with Michael Bolin, Greg Little, Eric Lieberman, Marcos Ojeda, Matt Webber, Philip Rha, Tom Wilson, Darris Hupp, Ricarose Roque

mailto:[email protected]

NPUC July 24, 2006

MIT Computer Science and Artificial Intelligence Laboratory 2

Web Applications

• The Web is now an application platform– Many web sites are now rich applications

– As we spend more time using them, we want to customize them for our own needs

NPUC July 24, 2006


Web Automation & Customization

• Automate repetitive operations– Bookmark my latest bank statement

– Click on the nearest download mirror site

– Print or save a list of web pages

– Fill in defaults on a web form

NPUC July 24, 2006



• Automate repetitive operations

• Transform appearance– Change color schemes for better contrast

– Concatenate a sequence of pages into one page

– Simplify site to remove unnecessary or distracting features

NPUC July 24, 2006



• Automate repetitive operations

• Transform appearance

• Integrate multiple web sites (“mashups”)– Bookstore has links for New Books, Used Books, Auctions… but it

doesn’t know about books in my local library

– Real estate site has lots of data about houses for sale… but it can’t compute the length of my trip to work

NPUC July 24, 2006


Web Apps Are Inherently Open

• Web applications have built-in hooks for scripting and modification

• Desktop applications don’t have these hooks unless the developer intentionally provides an API

• So it’s possible for an end user to change a web application– Greasemonkey for Firefox, User Javascript for Opera

– But it’s not easy

HTTP requests

HTML/XML replies

HTML/CSS display

can be automated

can be parsed for data

can be modified to change appearance

NPUC July 24, 2006


Problem: It’s Hard to Customize

• Web applications are complex under the hood

• The user has to know how to programvar xpath = "//text()[(parent::" + allowedParents.join(" or parent::") + ") and " + "contains(translate(., 'HTTP', 'http'), 'http')]";var candidates = document.evaluate(xpath, document, null, XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null); for (var cand = null, i = 0; (cand = candidates.snapshotItem(i)); i++) { if (urlRegex.test(cand.nodeValue)) { var span = document.createElement("span"); var source = cand.nodeValue;

http://www.mit.edu/...http://www.mit.edu/...http://www.mit.edu/...http://www.mit.edu/...

NPUC July 24, 2006


Our Approach

• Avoid complexity by looking at the user interface, not the HTML code

• Avoid learning a programming language by using keywords rather than language syntax

click feeling lucky button

feeling luckypush “I’m Feeling Lucky”

NPUC July 24, 2006


Outline

Introduction• Web automation

– Demo– How it works

• Keyword commands– Motivation– Demo– How it works

• Discussion

NPUC July 24, 2006


Web Automation

• Chickenfoot is our new web automation system– User shouldn’t have to look at HTML source to customize the Web

– Runs in Mozilla Firefox web browser

– Available for download from http://uid.csail.mit.edu/chickenfoot

NPUC July 24, 2006


Web Automation Demo

DEMO

NPUC July 24, 2006


Web Automation Commands

• Page navigationgo(url) openTab(url) fetch(url)

• Clicking and form manipulationclick(button-or-link) check(checkbox)enter([textbox], value) pick([listbox], choice)

• Pattern matchingfind(pattern)

• Page modificationinsert(pattern, html) remove(pattern) replace(pattern, html)

• Widgets & input handlingButton(label, action) Link(label, action)onClick(pattern, action)

NPUC July 24, 2006


Keyword Patterns

• Keywords + component type

• Component type is optional if it can be inferred– For example, click() command assumes a button or a link

feeling lucky button depart textbox

search web form

NPUC July 24, 2006


Pattern Matching Algorithm

Web page

Keyword patterngoogle search button

1.0

0.5

Matcher

Weighted list ofcomponents

NPUC July 24, 2006



Steps

1. Find labels matching the keywords– Text in the page, button labels, menu items, image ALT attributes

– Matching ignores word ordering, punctuation, capitalization

Web page


1.0

0.5

Matcher


NPUC July 24, 2006



Steps

1. Find labels matching the keywords

2. Find components matching label

Web page



1.0

0.5

Matcher

NPUC July 24, 2006



Steps

1. Find labels matching the keywords

2. Find components matching label

3. Rank the components & pick the best match– How well do the keywords match the label? (# of matching keywords)

– How well does the label match the component? (pixel distance)

Web page



1.0

0.5

Matcher

NPUC July 24, 2006


Outline

IntroductionWeb automation• Keyword commands

– Motivation– Demo– How it works

• Discussion

NPUC July 24, 2006


Problem: How Do I Learn the Language?

tbox = find('textbox');text = tbox.element.value;click('advanced image search');enter('all of the words', text);pick('small');pick('GIF');click('search button');button = new Button('Icon Search', doIconSearch);insert(after('search images button'), button);

NPUC July 24, 2006


Keyword Commands• Save the user from learning details of syntax

– Don’t worry about precise command names, punctuation, parameter ordering

go to google.comtype “npuc 2006”click feeling lucky button

google.comnpuc 2006feeling lucky

go(“google.com”);enter(“npuc 2006”);click(“feeling lucky button”);

Chickenfoot Keyword Commands

• Divide command into keywords

• Search for keywords in context– Some keywords are found in user interface; others are command names

• Generate a ranked list of possible interpretations– Punctuation and keyword ordering affect ranking, but are not required

• Use the highest-ranked interpretation– If ambiguous, ask the user to choose an interpretation

NPUC July 24, 2006


Keyword Commands Demo

DEMO

NPUC July 24, 2006


Advantages of Keyword Commands

• Is this natural?– Yes: this is how my 2-year-old son talks, and I can understand him– Most users are at the same level when it comes to talking to a computer in a

formal language– Our computers should become smart enough to understand us when we talk

this way

• Similar to keyword matching– e.g., web search engines

• Benefits novice users and expert users differently– Novices may use verbose, almost natural language commands– But experts can be very concise

• Keyword commands vs. point & click interfaces– More appropriate in some contexts (especially if combined with speech)– May even be faster (no hunting through menus and dialog boxes)

• Keyword commands vs. macro recording– Macro recordings don’t capture the user’s intention, but keyword programs do– Keyword programs are text, which can be easily modified and shared with

others

NPUC July 24, 2006


How Keyword Commands Work

Translatorpress feeling lucky button

Keyword command Program code

FunctionDatabase

enter

textbox string

go

url

click

button

findTextbox

keywords

textbox

findButton

button

keywords

click, press, push,

…

NPUC July 24, 2006


How Keyword Commands Work

Translatorpress feeling lucky button

Keyword command Program code

click(“feeling lucky”)

FunctionDatabase

button

click

findButton

keywords

NPUC July 24, 2006


Outline

IntroductionWeb automationKeyword commands

• Discussion– Robustness

– Security

– Resistance from web application developers

NPUC July 24, 2006


Robustness

• When web applications change, user customizations and mashups break– Customizations need to be robust to change

• Open question: How does the abstraction level of a customization affect its robustness?

<SOAP-ENV:Body> <ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <key xsi:type="xsd:string">00000000000000000000000000000000</key> <q xsi:type="xsd:string">shrdlu winograd maclisp teletype</q>

Rendered web page

DOM or raw HTML

Web service

NPUC July 24, 2006


Security

• Can user customizations open security holes for malicious web sites to exploit?– Greasemonkey’s GM_xmlhttprequest was an unexpected hole

– Chickenfoot has to open carefully-controlled holes in the browser’s security model to make certain things possible

• Can users be fooled into installing malicious customizations?– Yes: spyware is a case in point

– Making customizations transparent (easy to read and understand) might help

file:// images

https:// page

localUrl(“file://...”) “chicken-bypass:4hXFDh:file://…”

NPUC July 24, 2006


Resistance from Web App Developers

• “Don’t muck around with my DOM!”– Design: user customizations may make the site ugly or less usable

– Financial: customizations that hide or remove ads

– Server load

– Copyright

• We’ve been down this road before– Ad blockers

– Putting frames around other sites’ content

– Deep linking

• Web sites that fight their users will lose their users

NPUC July 24, 2006


Conclusion

• Two major barriers to automation and customization– Complexity of programming system

– Difficulty of language syntax

• We can tackle these barriers with two new techniques– Programming by looking at the user interface

– Eliminating syntax from commands

• Chickenfoot is available for download from http://uid.csail.mit.edu/chickenfoot

click feeling lucky button

feeling luckypush “I’m Feeling Lucky”

go(“google.com”);enter(“npuc 2006”);click(“feeling lucky button”);

Automation and Customization of Rendered Web Pages Rob Miller User Interface Design Group MIT Computer Science & Artificial Intelligence Lab [email protected].

Documents

web form slide

open web applications

list of web pages

web application greasemonkey

appearance slide

lucky slide

needs slide

http var candidates