Automation and Customization of Rendered Web Pages
Rob MillerUser Interface Design GroupMIT Computer Science & Artificial Intelligence [email protected]://uid.csail.mit.edu/
Joint work with Michael Bolin, Greg Little, Eric Lieberman, Marcos Ojeda, Matt Webber, Philip Rha, Tom Wilson, Darris Hupp, Ricarose Roque
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 2
Web Applications
• The Web is now an application platform– Many web sites are now rich applications
– As we spend more time using them, we want to customize them for our own needs
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 3
Web Automation & Customization
• Automate repetitive operations– Bookmark my latest bank statement
– Click on the nearest download mirror site
– Print or save a list of web pages
– Fill in defaults on a web form
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 4
Web Automation & Customization
• Automate repetitive operations
• Transform appearance– Change color schemes for better contrast
– Concatenate a sequence of pages into one page
– Simplify site to remove unnecessary or distracting features
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 5
Web Automation & Customization
• Automate repetitive operations
• Transform appearance
• Integrate multiple web sites (“mashups”)– Bookstore has links for New Books, Used Books, Auctions… but it
doesn’t know about books in my local library
– Real estate site has lots of data about houses for sale… but it can’t compute the length of my trip to work
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 6
Web Apps Are Inherently Open
• Web applications have built-in hooks for scripting and modification
• Desktop applications don’t have these hooks unless the developer intentionally provides an API
• So it’s possible for an end user to change a web application– Greasemonkey for Firefox, User Javascript for Opera
– But it’s not easy
HTTP requests
HTML/XML replies
HTML/CSS display
can be automated
can be parsed for data
can be modified to change appearance
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 7
Problem: It’s Hard to Customize
• Web applications are complex under the hood
• The user has to know how to programvar xpath = "//text()[(parent::" + allowedParents.join(" or parent::") + ") and " + "contains(translate(., 'HTTP', 'http'), 'http')]";var candidates = document.evaluate(xpath, document, null, XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null); for (var cand = null, i = 0; (cand = candidates.snapshotItem(i)); i++) { if (urlRegex.test(cand.nodeValue)) { var span = document.createElement("span"); var source = cand.nodeValue;
http://www.mit.edu/...http://www.mit.edu/...http://www.mit.edu/...http://www.mit.edu/...
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 8
Our Approach
• Avoid complexity by looking at the user interface, not the HTML code
• Avoid learning a programming language by using keywords rather than language syntax
click feeling lucky button
feeling luckypush “I’m Feeling Lucky”
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 9
Outline
Introduction• Web automation
– Demo– How it works
• Keyword commands– Motivation– Demo– How it works
• Discussion
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 10
Web Automation
• Chickenfoot is our new web automation system– User shouldn’t have to look at HTML source to customize the Web
– Runs in Mozilla Firefox web browser
– Available for download from http://uid.csail.mit.edu/chickenfoot
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 11
Web Automation Demo
DEMO
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 12
Web Automation Commands
• Page navigationgo(url) openTab(url) fetch(url)
• Clicking and form manipulationclick(button-or-link) check(checkbox)enter([textbox], value) pick([listbox], choice)
• Pattern matchingfind(pattern)
• Page modificationinsert(pattern, html) remove(pattern) replace(pattern, html)
• Widgets & input handlingButton(label, action) Link(label, action)onClick(pattern, action)
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 13
Keyword Patterns
• Keywords + component type
• Component type is optional if it can be inferred– For example, click() command assumes a button or a link
feeling lucky button depart textbox
search web form
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 14
Pattern Matching Algorithm
Web page
Keyword patterngoogle search button
1.0
0.5
Matcher
Weighted list ofcomponents
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 15
Pattern Matching Algorithm
Steps
1. Find labels matching the keywords– Text in the page, button labels, menu items, image ALT attributes
– Matching ignores word ordering, punctuation, capitalization
Web page
Keyword patterngoogle search button
1.0
0.5
Matcher
Weighted list ofcomponents
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 16
Pattern Matching Algorithm
Steps
1. Find labels matching the keywords
2. Find components matching label
Web page
Keyword patterngoogle search button
Weighted list ofcomponents
1.0
0.5
Matcher
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 17
Pattern Matching Algorithm
Steps
1. Find labels matching the keywords
2. Find components matching label
3. Rank the components & pick the best match– How well do the keywords match the label? (# of matching keywords)
– How well does the label match the component? (pixel distance)
Web page
Keyword patterngoogle search button
Weighted list ofcomponents
1.0
0.5
Matcher
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 18
Outline
IntroductionWeb automation• Keyword commands
– Motivation– Demo– How it works
• Discussion
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 19
Problem: How Do I Learn the Language?
tbox = find('textbox');text = tbox.element.value;click('advanced image search');enter('all of the words', text);pick('small');pick('GIF');click('search button');button = new Button('Icon Search', doIconSearch);insert(after('search images button'), button);
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 20
Keyword Commands• Save the user from learning details of syntax
– Don’t worry about precise command names, punctuation, parameter ordering
go to google.comtype “npuc 2006”click feeling lucky button
google.comnpuc 2006feeling lucky
go(“google.com”);enter(“npuc 2006”);click(“feeling lucky button”);
Chickenfoot Keyword Commands
• Divide command into keywords
• Search for keywords in context– Some keywords are found in user interface; others are command names
• Generate a ranked list of possible interpretations– Punctuation and keyword ordering affect ranking, but are not required
• Use the highest-ranked interpretation– If ambiguous, ask the user to choose an interpretation
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 21
Keyword Commands Demo
DEMO
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 22
Advantages of Keyword Commands
• Is this natural?– Yes: this is how my 2-year-old son talks, and I can understand him– Most users are at the same level when it comes to talking to a computer in a
formal language– Our computers should become smart enough to understand us when we talk
this way
• Similar to keyword matching– e.g., web search engines
• Benefits novice users and expert users differently– Novices may use verbose, almost natural language commands– But experts can be very concise
• Keyword commands vs. point & click interfaces– More appropriate in some contexts (especially if combined with speech)– May even be faster (no hunting through menus and dialog boxes)
• Keyword commands vs. macro recording– Macro recordings don’t capture the user’s intention, but keyword programs do– Keyword programs are text, which can be easily modified and shared with
others
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 23
How Keyword Commands Work
Translatorpress feeling lucky button
Keyword command Program code
FunctionDatabase
enter
textbox string
go
url
click
button
findTextbox
keywords
textbox
findButton
button
keywords
click, press, push,
…
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 24
How Keyword Commands Work
Translatorpress feeling lucky button
Keyword command Program code
click(“feeling lucky”)
FunctionDatabase
button
click
findButton
keywords
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 25
Outline
IntroductionWeb automationKeyword commands
• Discussion– Robustness
– Security
– Resistance from web application developers
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 26
Robustness
• When web applications change, user customizations and mashups break– Customizations need to be robust to change
• Open question: How does the abstraction level of a customization affect its robustness?
<SOAP-ENV:Body> <ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <key xsi:type="xsd:string">00000000000000000000000000000000</key> <q xsi:type="xsd:string">shrdlu winograd maclisp teletype</q>
Rendered web page
DOM or raw HTML
Web service
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 27
Security
• Can user customizations open security holes for malicious web sites to exploit?– Greasemonkey’s GM_xmlhttprequest was an unexpected hole
– Chickenfoot has to open carefully-controlled holes in the browser’s security model to make certain things possible
• Can users be fooled into installing malicious customizations?– Yes: spyware is a case in point
– Making customizations transparent (easy to read and understand) might help
file:// images
https:// page
localUrl(“file://...”) “chicken-bypass:4hXFDh:file://…”
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 28
Resistance from Web App Developers
• “Don’t muck around with my DOM!”– Design: user customizations may make the site ugly or less usable
– Financial: customizations that hide or remove ads
– Server load
– Copyright
• We’ve been down this road before– Ad blockers
– Putting frames around other sites’ content
– Deep linking
• Web sites that fight their users will lose their users
NPUC July 24, 2006
MIT Computer Science and Artificial Intelligence Laboratory 29
Conclusion
• Two major barriers to automation and customization– Complexity of programming system
– Difficulty of language syntax
• We can tackle these barriers with two new techniques– Programming by looking at the user interface
– Eliminating syntax from commands
• Chickenfoot is available for download from http://uid.csail.mit.edu/chickenfoot
click feeling lucky button
feeling luckypush “I’m Feeling Lucky”
go(“google.com”);enter(“npuc 2006”);click(“feeling lucky button”);