A Look Under the Hood at a World-Class IBM Lotus Domino Web Application

®

BP203: A Look Under the Hood at a World-Class IBM Lotus Domino Web Application

Scott GoodPresidentTeamwork Solutions, Inc.

Henry NewberryDirector of Technical ServicesTeamwork Solutions, Inc.

Scott Good – Bio

President, Teamwork Solutions, one of Lotus’ earliest BPs

Started with Notes 2.1 (1992)

Extensive experience building workflow applications for both Notes and the web

Have published over 50 articles on Notes & Domino application development (LotusScript, JavaScript, CSS, AJAX, JSON, etc)

Contributing Editor, Lotus Advisor magazine

Regular speaker at Lotusphere and Advisor conferences

Henry Newberry – Bio

Director of Technical Services at Teamwork Solutions, Inc.

Formerly President of Newbs Consulting, Inc. (2000-2005)

Cofounder of Synergistics, Inc. (1993-2000)

Notes Pioneer at CBIS (1989-1993)

Lotus Advisor Contributor, Speaker, and Editorial Advisory Panelsince 1996

Speaker at IBM Lotus events since 2001

We were …

Asked by Kevin Cavanaugh and Bob Balaban to re-build the Lotus on-line forums to take advantage of Web 2.0 capabilities

And, to do it using Domino

The result is a really interesting Domino web application which we’re going to share with you today

Agenda

Introduction to the problem

System setup and definition

Handling of views and navigation

Handling individual threads

Creating new threads and responsesDojo Editor

SearchingWithin one forumAcross multiple forums

Most Popular Topics

Introduction to the problem Opportunity

Existing forums were Domino-based but…OldSlowUgly

Every click required a full page refreshView twistiesEach response in a threadEtc

Goal was to rebuild the forums to take advantage of AJAX, JSON, DHTML and CSS to make an attractive, fast, easy-to-use site

And, to NOT lose all information currently in the old forums

And, wherever possible, to take advantage of Domino’s strengths

------------------

Before and After

System Setup Overview

A hub-and-spoke arrangement of a single Central database and multiple external Forums (databases)

IBMDocuments

Forum

IBMSpreadsheets

Forum

IBMPresentations

Forum

SupportForums“Home”

System Setup: Forum Databases

One central database per “suite” (e.g., Lotus Symphony)

One database per “Forum”IBM Lotus Spreadsheets ForumIBM Lotus Documents ForumIBM Lotus Presentations Forum

In each Forum, any number of “sections”Feature RequestsFeedback (general discussion)Hints and TricksIssues and Troubleshooting

Sections do not need to match from Forum to Forum

Each section contains the actual documents and responses

System Setup: Central Database

Central databaseContains forum and area configuration documentsContains the main Home page with links to all forums and forum sectionsSearches done from here reach into all forum databasesCollects “Most Popular Topic” information from all forums

Support ForumsCentral

Database

ForumConfiguration

ForumsHomePage

Multi- Forum

Searching

MostPopularTopics

FeatureRequestsSection

FeedbackSection

Hints &Tips

Section

Issues &Troubleshooting

Section

System Setup: Forums

CentralDatabase

SpreadsheetsForum

Configuration

FeatureRequests

FeedbackSection

HintsSection

IssuesSection

IBMSpreadsheets

Forum

IBMPresentations

Forum

IBMDocuments

Forum

From the Home page, links to Forums

Each Forum reads configuration information in Central Db to get Sections

System Setup: Creating a New Forum

Create new Forum database from template

Create On-Line Forum Configuration form in Central database

Automatically creates a keyword doc in the Forum database to let Forum find the central database

Create one or more Forum Section documents to define that Forum’s sections

Close and reopen the browser (to clear the cache)

Done

Central Database Home Page

Our plan…Get something on-screen immediately (core page, header)Get most important content next (Forum links)Get less-important content last (Most Popular Topics)Give the user something to look at ASAP

AJAX, then AJAX, then more AJAX

1 AJAX lookup in the onLoad to get all the Forums and sections?ReadViewEntries&OutputFormat=JSONUses JavaScript to build the central page area’s HTML on the fly

2 Get the Most Popular TopicsPulls top 5 topics from each Forum (sequentially, with AJAX)

As a custom JSON format…more about this later

Collates, sorts, builds HTML of top Topics and puts them on the pageResults are cached during your session

Home Page Links to Each Forum…

Primary link to the database (“IBM Spreadsheets”)

Links to each individual Section (“Feature Requests,” etc)

Information for each Section on…Last post date/timeNumber of main topicsTotal number of postsNumber of views

Forum Views

AJAX content retrieval and pagination

Response hierarchy creation

UI Navigation approach (tracking via cookies, breadcrumbs)

User preferences options

Forum Views

Two optionsMain Forum viewForum Section

Both are the same view, same $$ViewTemplateView: AllDateThreadedWeb

Section “views” don’t display the list of other sectionsMain Forum view DOES have links to all sections

What you see is controlled by the URL, Cookies

When the View Opens…

Core HTML is loadedAnd JavaScriptAnd CSS

Main view documents (top-level documents) are retrieved

Then Forum Section information (if needed) from Central DbCached AJAX query…very fast

THEN, if needed, documents for open response hierarchies

Each component is built and displayed as it is available

How the Views Work

Initial URL determines what will be shownAllDateThreadedWeb?OpenView&RestrictToCategory=AllAllDateThreadedWeb?OpenView&RestrictToCategory=Feature%20Requests

Initial URL does not return any view information

$$ViewTemplateDefault does not have a $$ViewBody fieldAll view information is built on the fly from AJAX query returns and placed into the existing HTML

$$ViewTemplateDefault DOES have…HTML used for the rest of the pageMany computed JavaScript variablesNotes fields with lookupsJavaScript librariesCSS libraries

AJAX 1…Main Documents

onLoad event of $$ViewTemplateDefault calls an initView() function and starts the first of a series of AJAX calls

Gets the main View documents (using AJAX)AllDateThreadedWeb?ReadViewEntries&OutputFormat=JSON&count=10 &RestrictToCategory=All&CollapseView&nocache=1196698255351

Pulls main docs data from a multi-column view using Domino’s standard JSON format

Returned string needs converted to become “live” JSONeval("forumLinksJSON = " + forumLinksAjax.responseText);

JSON data is built into a ViewObj classWithin class, builds HTML for the “view,” places it on-screendocument.getElementById("ViewOutput").innerHTML = …;

How to Know What to Show

Since all data is retrieved with AJAX, URL never changesWithout tracking current user position, returning to the view (e.g., “Back”button) would always take you to the first page…not goodHave to keep track of where you are and rebuild accordingly

Current view location is managed with cookiesView position/lines per view (per Forum) 21~10Expanded row numbers (per Forum) 21~21.1~21.1.1~22

Whenever view is opened, this is read and used to control what part of the view is actually returned

AJAX 2…Forum Sections

Forum Section list is retrieved from the Central DatabaseRetrieve Forum Section names (a cached lookup)/MainDb/ForumAreaLookups?ReadViewEntries&OutputFormat=json&count=200

Pulls necessary information from multi-column view, again using Domino’s standard JSON format

If the view “category” is not “All,” this is ignoredBecause it’s cached…it’s incredibly fast

Otherwise, JSON is converted using eval(), processed with JavaScript, and added to the page

AJAX 3…Response Hierarchy in the View

Only top-level documents load when the view is first shown

Responses are loaded when any “twistie” is clickedALL descendants of a given topic are loaded when the top-level twistie is clicked, but only displayed one level at a timeNo descendants are loaded for any hierarchy not clicked

A delayed call in the onLoad event resets hierarchy when returningsetTimeout("reOpenViewToPriorSettings()", 100);Reads cookie to know which response sets to re-loadReads cookie to know which elements of the hierarchy should be expandedGives users the expected experience when hitting “back” or otherwise returning to the view from somewhere else

View Load Review

Initial HTML, including breadcrumbs

Main Documents

Forum Section links (if needed)

Response hierarchies as needed

Reset expanded response trees

Opening a Thread

Clicking on any document or any of its responses opens the top-level document with all responses rendered into it

HTML for the main document loads with the form, all responses are retrieved with AJAX

Response hierarchy is displayed with indentation

Every document and response has a “respond” link allowing users to respond to any point in the thread (as you would expect)

Lateral navigation to next and previous documents in the view

Creating a New Document

Click “Post a Topic” link from virtually anywhere in the site to create a new Topic (main document)

Click “respond” in any of the document hierarchies to respond to a part of an existing thread

Both the document and response are almost identical and kept as simple as possible

User name and e-mail address are stored in a cookie (user are not required to be authenticated)

Saving a New Post

Returns you to where you were in the viewWith the right things expanded

Traversing laterally through documents (using Next/Previous)View location keeps track of where the document you’re in (now) is locatedAs you move to documents in the next segment of the view, returning to the view puts you in the new location

Document “View” Counter

Each time a document is opened, a Profile document (used by all documents) is incremented

Guest Preferences

Guest Preferences allow users to set values for Number of rows displayed per pageTime Zone (which is ignored)

After changing settings, you’re returned where you came from with the new settings in effect

Adjusts view paging to put you in the “same” placeFrom 7th page on a 10/page setting to the 3rd page on a 25/page setting

Searching

There are two types of searchingJust within the current Forum (from anywhere in a Forum)Across all Forums (only available from the main Home page)

The two search types use two different search techniquesHTML-formatted search view (current Forum searching)Custom JSON-formatted search views (multi-Forum searching)

Single-Database Searching

Search query is built with JavaScript into a URLsearchview?searchview&query=replication%20and%20conflict

SearchView is a view formatted in HTML to display nicely in the browser

Uses $$SearchTemplateDefault form to display results$$ViewBody field controls placement of “view,” which is really just search results from the view

Advanced Search option (on the $$SearchTemplate) expands the parameters to further refine the search

searchview?searchview&query=replication%20and%20conflict&SearchOrder=1&SearchWv=TRUE&SearchFuzzy=TRUE&SearchMax=250These are all standard Domino capabilities/URL parametersAgain, constructed with JavaScript based on selections in the available fields

Requires database to be Full-Text Indexed

Cross-Database Searching

A little more complicated than single-database searching

Surprisingly—shockingly—fast

Is initiated from the Home page in the Central database

When started, makes an AJAX search query to each of the associated Forums, one at a time

As one set of results is returned, the next query is sentContinues until all Forums have been touched

Queries are made to custom JSON viewsNOT Domino’s JSON format

Query results are organized using a special $$ViewTemplate for the view

Custom JSON Search View

Used custom JSON to make it easier to get only the information we wanted back

Enough to build a results page with links to documents in the various Forums

Our format for each line in the view…{

sprid: "WEBB76TFX3",unid: "00EA8A8C84B104EC0525834F005AFC36",dbtitle: "IBM Spreadsheets",dbpath: "IBM\SpreadsheetForum.nsf",subject: "Full-text search not working?",date: "09/07/2007",author: "John Doe",numviews: "12",dbform: "Feedback"

},

NOTE the closing comma

Custom Search ViewTemplate

$$ViewTemplate for MultiDbSearchView

JSON object contained in a <div> for JavaScript parsing purposesMore about this in a moment

Blank set of values at the end because each live of the view ends with a comma

In the Home Page…

Starting a search puts query on the URL and calls SearchResults form

window.location = "SearchResults?Open&query=” + escape(document.forms[0].SearchField.value);

SearchResults form…

Is very much like $$ViewTemplateDefault in layout, formatIncludes fields for Advanced Search

Has onLoad event which starts the cascade of AJAX searchesdoSearch() builds search URL and calls makeSearch() which makes all AJAX calls

function makeSearch(){queryUrl = "/" + searchDbPaths[curDb] + "/multidbsearchview?searchview&searchmax=0" + queryVal;curDb++;createAJAXRequest(queryUrl, "handleMultiDbSearch");

}

handleMultiDbSearch() continues to call makeSearch() until all Forums have been searched

Parsing AJAX Return Values for JSON

$$ViewTemplate for MultiDbSearchView returns a big bunch of HTML around the part you actually want

Parsing AJAX Return Values for JSON

$$ViewTemplate for MultiDbSearchView returns a big bunch of HTML around the part you actually want

Can’t change the form to display as HTML or as “text/plain”because neither will work with a form being “edited”

Have to live with the mess and parse it outHence, the <div> around the JSON object…a handle to grab with JavaScript

Getting the JSON Out of that Mess

It’s as simple as finding the locations of the <div> and </div> tags in the string that is the HTML

var searchHTML = ajaxReq.responseText;var divStart = searchHTML.indexOf("<div>") + 5;var divEnd = searchHTML.indexOf("</div>");

var viewStr = searchHTML.substring(divStart, divEnd);eval("var viewObj = " + viewStr);

Of course, you have to know there are no <div> tags in the text

Why Not Use an Agent for Searching?

One reason: Speed

Using the Domino-standard searching against a Full-Text Indexed database is amazingly fast

Agents give you flexibility you can’t really get out of a viewThat flexibility comes with a cost…it’s slowerWe didn’t need the flexibility

Finishing the Search

Once JSON is back from all Forums, page is built

JavaScript iterates through all JSON objects building HTML

Search results are segregated by Forum

Most Popular Topics

Similar to multi-Forum searching

On the initial Home page load, all Forums are queried (via AJAX) for the Most Popular Topics

Based on the number of views

Query is made to a custom JSON view sorted by view count

Only five documents are returned from each database

Results from all Forums are combined and sorted

Top five (plus ties) are displayed

Results are cached during the user’s session

Cascading Style Sheets

Our design is based on that used for interim Jive-based site

Most of our application was done before we got access to actual CSS code

Built it from screen shotsThen re-built it to match, using CSS from IBM (ON IBM site) plus localized adjustments needed for our systemThen IBM’s CSS changed…a little

Now we store ALL our own CSS in the databases

Lessons Learned

Don’t count on others to never change their CSS

JavaScript compression options can make an enormous difference in file sizes (and corresponding page-load speeds)

Sequentially searching across multiple databases is amazingly fast if you leverage the Domino search mechanisms

FireBug ROCKS!

Thank you!Scott Good: [email protected]

614-457-7100 x200http://www.scottgood.com (blog)

Henry Newberry: [email protected] x206

http://www.henrynewberrry.com (blog)

http://www.teamsol.com

PLEASE FILL OUT YOUR EVALUATIONS

Teamwork Solutions