DB-14: Sometimes A Great Notion: Breaking The Logjam Glen West Software Architect, EarthLink.

DB-14: Sometimes A Great Notion:Breaking The Logjam

Glen WestSoftware Architect, EarthLink

2 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation

Sometimes A Great Notion

What is EarthLink? One of the world’s largest ISPs 5.4 million subscribers 1.4B annual revenue, 1.4B market cap Received the highest ranking for

customer satisfaction for both high-speed (3rd year) and dial-up Internet services by J.D. Power and Associates in its 2004 Internet Service Provider Residential Customer Satisfaction Study - Sept. 2004

PC Magazine’s Readers’ Choice Award for both high-speed and dial-up services - Sept. 2003

Breaking the EarthLink Logjam


Sometimes A Great Notion

Who is Glen West? Software Architect, EarthLink (Aug

2004) Worked with Progress since 1989,

V5 Old Movie Buff

Why “Sometimes A Great Notion”?

Imagery Logging Metaphor

Breaking the EarthLink Logjam


Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions




Why Are We Here?

Updating antique applications Improving infrastructure Providing developer tools Positive encouragement for standardization WITHOUT upgrading everything to the

bleeding edge

What’s in it for ME?




The EarthLink Environment True 24x7 (with a maintenance window) Production in Atlanta, DR in Pasadena Points of Presence (POP) everywhereeverywhere Employees and contractors worldwide

(8000 support calls per hour, 2 per second) 1600 servers, 6M customers, 1.6T of data Progress, Oracle, SQL Server, WebSpeed,

Java, JBOSS, Netcool, Foglight, ChUI, Web 50 developers writing code daily Frequent acquisitions, 10G of ASCII logs per DAY10G of ASCII logs per DAY




The Scope of the Problem 50 developers. How many ways of writing

log file information? 200+ No one can make use of 10G of log files a

DAY Heisenburg principle. Our logging is

choking the disks, 100% CPU and disk IO Log information comes from all over,

different systems and processes All of the applications have their own

logging locations and methods




The Objectives If it isn’t simple, they won’t come Has to support multiple destinations and

types (distribute/consolidate) 80/20 – go for the quick win, then extend Manageable at runtime

(destination/activation) Standardized Efficient Scaleable




The Architecture Persistent Session-Services procedure Simple Publish/Run commands AppServer™ for consolidation/distribution Database tables to control destination


The Architecture (ERD)


Destinations

App Procedure

Session Services

Procedure

AppServer Log Manager

Database

TablesEmail

Netcool

Foglight

Session Log

Central Logsyslogd




The Tools Foglight® (Quest) – monitors ASCII log files

http://www.quest.com/

Netcool® (Micromuse) – records system eventshttp://www.micromuse.com/products_sols/

Syslogd (unix std) logs to OS logs Progress® AppServer eMail




The Construction Persistent Session-Services procedure AppServer Log Event Manager WebSpeed® Plugin

– Web context management The Slight-of-Hand naming trick


Session Services Procedure Cache Destinations Cache status of/connection to AppServer Handle independent Public vs. Private procedures Difference between PUBLISH/SUBSCRIBE

and running a SUPER-PROCEDURE. Fault tolerant – AppServer may not be

available Structure to provide additional services to

developers


How Does Publish/Subscribe Work? I

Progress Session (“Agent”)

Program A(Session Svcs)

Program B(Entry)

Program C(Dialog Box)

Event Name Subscriber Target

Named Event Table


How Does Publish/Subscribe Work? II



Program B(Entry)



Named Event Table

Launch Persistent


How Does Publish/Subscribe Work? III



Get Preference Session Svc Anywhere

Set Preference Session Svc Anywhere

Named Event Table


Program B(Entry)


Publish “Get-Preference” (“background color”, output w-bg-color, output w-success).

Run get-preference in hSessionSvc (“background color”, output w-bg-color, output w-success).

Run

Pro

xy

Publish

Run ProxyP

ublish “Get P

reference”


How Do Super Procedures Work?

Propath

Progress Session

Procedure xxx:

End.

Procedure xxx:

End.

Procedure xxx:

End.

xxx.p

App.pApp-svc.p

Sess-Svc.p

Run App-Svc.p persistent set hAppSvc.

this-procedure:add-super-procedure(hAppSvc).

RUN xxx.

Session:add-super-procedure()


Super-Procedure - The Little Trick When you say “run xxx” where does Progress

look for x?– Internal procedure xxx IN THIS-PROCEDURE– Internal procedure xxx IN THIS-PROCEDURE:SUPER-

PROCEDURES– Internal procedure xxx in SESSION:SUPER-PROCEDURES– xxx.r in PROPATH.

Run log_manager (message)…– log_manager.p runs sessionsvc.p persistent– Sessionsvc.p defines internal procedure log_manager– Sessionsvc.p makes itself a session:super-procedure

The next Run log_manager will run the super-procedure


Pub/Sub vs. Super Procedure

Publish/Subscribe Pros

– Handle-less– Fire and Forget– Multiple Listeners– No errors in handoff

Cons– Multiple Listeners– No clue no-one is

listening See extras for 2003

Pub/Sub presentation

Super-procedures Pros

– Handle-less– Only one listener– Order of succession

Cons– Throws errors if no

listener– Can be “hard-error”

Pros and Cons


Other Tricks How do you ensure only one services

procedure is running?– PUBLISH/SUBSCRIBE heartbeat

PUBLISH returning output parameters


The AppServer Manager Why use the AppServer at all? It’s all about EfficiencyEfficiency Database Logging – Outside the

transaction Email – Asynchronous Cache messages bound for ASCII files Gatekeeper to ensure integrity of writes Security issues Stateless Future: multi-threaded TCP socket listener


WebSpeed Plugin Part of the startup of the agents:

pros and cons Why it is even more necessary in

WebSpeed? How do you keep messages separated by

end-user (context management)?




The Result ASCII log volume reduced by x%,

continuing to decrease daily Foglight enabled on critical logs to alert on

critical events Netcool Dashboards now receiving real-

time data on vendor polling results. Disk IO reduced x% WebHitLog can now be relied upon

Statistics not available at press time, will be provided at conference




The Lessons Learned Have accurate numbers on your

environment before starting When to get visibility Give them something to ‘fix’ Incrementalism is both good and bad Everyone wants the process improved, but

no-one has the time to improve it Master of the obvious: plan for the future


In Summary It IS possible to

change the infrastructure of an aging application without a rewrite

Large gains can be made with small, incremental efforts as long as you have a plan

Things flow better with a little bit of organization


Questions?


Thank you for your time!


DB-14: Sometimes A Great Notion: Breaking The Logjam Glen West Software Architect, EarthLink.

Documents

problem n

construction n

architecture n

objectives n

result n

infrastructure n

subscribers n

n employees