17th Large Installation Systems Administration Conference ......was distinguishing between the physical world and the software world but that a distinction made between mechanism and

law) for the approximately 60% non-lawyers attending the conference .

In discussion, someone commented onMr. Swire’s model, pointing out that hewas distinguishing between the physicalworld and the software world but that adistinction made between mechanismand instances would have been a betterapproach. Mr. Swire replied that whenconsidering instances, one must oftenconsider the first instance differentlythan others (since that will often educatethe defenders and change the effect ofsubsequent instances). This led to a dis-cussion of the ability of the law to oper-ate in this complex arena (and thelikelihood, or not, of lawyers staying outof the fray). There seemed to be someagreement that we will have some veryconfused judges, at least for a while.

PANELBRIEF CONCLUDING REMARKS

Jennifer Granick, Stanford CIS; LaurenGelman, Stanford CIS; Scott Blake,BindView; Greg Schaffer, Pricewater-houseCoopersNo one today has argued against theidea that the market has failed to pro-vide security. Instead of capitalism sav-ing us, we are beginning to concludethat there may be a role for government,a conclusion that many of us find bothinteresting and disturbing.

There are some interesting (legal) ques-tions to be answered with regard to dis-closure, nondisclosure, and liability.What if one can become liable for know-ing something and not disclosing it?

Security is about more than fixing “thisone bug.” It could be about democracy.We don’t know enough about security toknow that it ought to (or not) be con-sidered differently from other scientificenterprises.

Some people think that the disconnect isabout Republicans and Democrats, butit is really about the information-tech-nology and legal communities. Bothhave well-developed models of their

65February 2004 ;login: LISA ’03 l

l

CO

NFE

REN

CE

REP

ORT

Suniverses and like to be the masters oftheir respective domains. Neither likesthe discomfort of not having a handleon important things that apply to theirrealms. There are lots of people whohave not thought about these problemsand won’t until there is a crisis, and thenthe decisions are unlikely to be well-con-sidered and thoughtful. There is a seri-ous need for us to think about theseproblems in advance, as we have beendoing today.

17th Large Installation Systems Administration Conference (LISA ’03)San Diego, CaliforniaOctober 26–31, 2003KEYNOTE ADDRESS

INSIDE EBAY.COM: THE SYSTEM ADMINISTRA-TOR’S PERSPECTIVE

Paul Kilmartin, eBay, Inc.Summarized by Bryan ParnoKicking off the 17th annual LISA confer-ence, Paul Kilmartin, eBay’s director ofavailability and performance engineer-ing, gave a spirited and engaging tour ofthe development of eBay’s infrastruc-ture, from a single PC in eBay founderPierre Omidyar’s bedroom to the cur-rent SAN-based system composed ofhundreds of enterprise-level machines.Along the way, eBay’s user populationexploded from a few hundred in 1995 toover 85 million today.

Throughout the talk, Kilmartin stressedthe incredible importance of availability.Since eBay averages $738 of gross mer-chandise sales every second, the prospectof any prolonged outage is costly indeed.This intense usage also makes eBay theworld’s 75th largest economic market,falling somewhere between Uzbekistanand the Dominican Republic. Kilmartinrepeatedly emphasized how the magni-tude of eBay’s 85 million user-baseimpacts virtually every decision thecompany makes.

In the historical segment of his talk, Kil-martin highlighted eBay’s transitionfrom a system based on two-node Veri-tas clusters to a large-scale SAN. On theplus side, this cut down on the amountof idle hardware, always an importantconsideration for cost-conscious admin-istrators. It also provided a greaterdegree of fault minimization and isola-tion, since the two-node clusters suf-fered from electrical issues duringservicing. Unfortunately, shortly afterthe migration to the SAN, the co-loca-tion company hosting the siteannounced it would be going out ofbusiness. Kilmartin’s team of systemadministrators built an entirely newSAN in three weeks and made themigration with only two hours of down-time in September of 2001. The bank-ruptcy of the Exodus storage facility inNovember of 2001 forced yet anothermove.

Even though the public perceives eBay asan industry leader, Kilmartin repeatedlyemphasized his preference for remainingfirmly in the mainstream of technology.On several occasions, he urged the audi-ence to forge on ahead and aggressivelyreport problems, so that after a few yearsof maturation, eBay could adopt the“new” technology. He offered several tipsto the audience, encouraging systemadministrators to doubt everything, tomake the system work hundreds oftimes before trusting it, and to challenge“best procedures” by at least asking forreferences. He also emphasized theimportance of knowing one’s role on theteam, citing his initial resistance toeBay’s foray into the car market (now, hesays, a Corvette sells on eBay every 64minutes). Kilmartin also stressed theneed to constantly seek out a betterunderstanding of the customer and howthe customer uses the product. Com-menting on hiring decisions, hereminded the audience that neitherexperience nor certification necessarilyequates to competence. Concluding witha return to the theme of availability, Kil-

martin asserted the need for vendors torecognize eBay as an active customer,not a cadaver; in other words, the com-pany needs working solutions that canbe diagnosed and repaired on the fly, notsystems that need to be taken offline anddissected to provide information.

REFEREED PAPERS

ADMINISTERING ESSENTIAL SERVICESSummarized by Ari PollackRADMIND: THE INTEGRATION OF FILESYSTEMINTEGRITY CHECKING WITH FILESYSTEMMANAGEMENT

Wesley D. Craig and Patrick M.McNeal, University of MichiganWesley and Patrick introduced radmind,a filesystem management tool designedto replace similar tools, such as Tripwireand cfengine, and overcome some limi-tations with existing products. Tripwire,for instance, does not scale well or knowthe difference between unintendedchanges and OS updates.

Radmind is based on existing work frompeople in the sysadmin community suchas Evard and Anderson, and on featuresfrom tried-and-true software. Like Trip-wire, it includes integrity. Features inboth rsync and radmind include copyingof files and comparison to policy, not alive filesystem. Borrowing fromcfengine, radmind provides abstractconfiguration and abstraction of any fileset.

Radmind goes further than tripwire; inaddition to detecting unwanted changesto the filesystem, it can automaticallyrevert back to a known good state con-figured in the policy. It only generatesreports when something unusual hap-pens. It is easy to understand, has simplesetup and configuration, and requires noprogramming skills for successful use.Radmind is platform-independent; itworks on Windows, and it is already inuse on MacOS X laptops, Linux andSolaris servers, and supercomputingclusters.

66 Vol. 29, No. 1 ;login:

FURTHER TORTURE: MORE TESTING OFBACKUP AND ARCHIVE PROGRAMS

Elizabeth D. Zwicky, Great Circle AssociatesElizabeth presented the results of herfindings from torture-testing variousbackup tools for UNIX and UNIX-likesystems. This is a follow-up to her 1991paper, which was inspired by frustrationat conflicting rumors and vague docu-mentation. The term “backup program”is used loosely; there is no correct termfor something that’s intended to copyfiles to another medium for storage(rather than immediate usage).

What she found in 1991 can be summedup as, “don’t trust what you’ve heard, goout and verify.” She had heard reportsthat “cpio doesn’t handle too many hardlinks,” so she found out what “too many”meant.

Her latest paper presents a new round ofverification of old, out-of-date data.Some of the properties of backups shecovers are: file size, devices, strangenames, access permissions, holes(numerical representations of nulls onthe filesystem), long names, and links. In1991, every tool died at some pointexcept dump, resulting in core dumpsand/or data corruption. Now, nothinghandles paths over the maximum pathlength defined by the operating system,and nothing but restore handled holesabsolutely correctly.

Elizabeth says that while backups aredifficult, testing backup tools is fun andnot that hard. Also, backup programshave different targets and are not consis-tently useful to everyone. She also pre-sented some conclusions stemming fromher research:

n Don’t write your own backup pro-gram; there are more than enoughalready.

n Never use old file formats for back-ups.

n The name of your backup programdoes not predict its performance inyour configuration.

n Long pathnames are an unsolvedproblem.

n Trust, but verify.n Backup programs need time to

mature.

AN ANALYSIS OF DATABASE-DRIVEN MAILSERVERS

Nick Elprin and Bryan Parno, HarvardUniversityNick and Bryan took a look at the differ-ent kinds of common mail storage for-mats in use. The three most commonare: mbox format, where every email isconcatenated into a flat text file; maildirs,where every email is stored in a databasefile; and databases, where all mail isstored in some kind of structured data-base.

The two database formats used for test-ing were Cyrus, which uses Berkeley DB,and their own SQL model using MySQL.Here is what Nick and Bryan found:

n Mbox performs better than Cyrusfor a small account in a full-textsearch.

n Cyrus performs better than maildirand mbox for larger accounts.

n MySQL performs better than theothers overall.

n Maildir always performs the worst.

Databases allow better fine-tuning ofmail servers and better scalability. File-based solutions perform better on someoperations, such as expunging mail.However, performance is usually not theonly factor when deciding on a mail for-mat. Maildirs do not suffer from thesame locking problems as mbox, and astructured database may require moreoverhead than is acceptable in some sit-uations.

INFORMATION AND CONTENT MANAGEMENTSummarized by Kenytt Avery

A SECURE AND TRANSPARENT FIREWALL WEBPROXY

Roger Crandell, James Clifford, andAlexander Kent, Los Alamos NationalLaboratoryJames Clifford describes the LANL Webproxy as a “benevolent man in the mid-dle.” In contrast to ordinary Web proxieslike Squid, the LANL Web proxy pro-vides access control on incoming ratherthan outgoing connections. The purposeof the proxy is to allow access to internalWeb applications (e.g., Web mail, Nagiosnetwork monitoring) from public Inter-net sites outside the firewall.

The proxy consists of two pieces, theredirection daemon redird, which redi-rects HTTP requests for internal docu-ments to the equivalent request viaHTTPS, and the Web flow daemon wfd,which handles authentication and for-warding requests to the internal net-work. The external server contains awildcard SSL certificate for lanl.gov,allowing it to proxy for any internal sys-tem.

According to the authors, the chief bene-fit of the proxy solution is its simplicity,requiring no configuration changes orextra software to be installed on theclient beyond an ordinary Web browser.This is in contrast to VPN solutions,which require client software and usertraining, or to non-transparent proxyservers, which require browsers to beconfigured to use them.

An important question from the audi-ence concerned the security of potentialclients. An untrusted client machinemight be running keystroke logging orscreen capture software. Cliffordresponded that the solution has workedwell as a stopgap measure until a fullVPN can be implemented. In the mean-time, efforts have been underway toeducate users about the risks of usingunknown clients.

67February 2004 ;login:

l

CO

NFE

REN

CE

REP

ORT

SURL: http://www.lanl.gov/orgs/ccn/publications.shtml

DESIGNING, DEVELOPING, AND IMPLEMENT-ING A DOCUMENT REPOSITORY

Joshua S. Simon, Consultant; LizaWeissler, METIJosh Simon described a solution to aproblem faced by many large sysadminteams, that of finding documentation. Inorder to address the constant flow ofemail asking about various tasks withintheir consulting company, he and LizaWeissler built a Web-based documentmanagement system with the goal ofmaking it easier to find information.

The first problem the authors faced wasone of categorization: At one point theyidentified 52 different types of docu-ment. While it is clear that a documentmanagement system is considerablymore useful when items are separatedinto categories, users are often unwillingto make the effort to do so as each docu-ment is entered. A practical solution wasto define a small number of top-levelcategories (e.g., Customers, Internal,Marketing, Recruiting, Other), each witha small number of subcategories. Cate-gories were assigned single-letter codes,allowing each document to be classifiedwith a two-letter code (e.g., IC for Inter-nal Code).

The other major problem the authorsfaced was maintaining the metadataabout each document once it had beenstored in the system. While users sub-mitting documents were encouraged tosupply metadata, consultants who werenot currently assigned to billable proj-ects were recruited to serve as “librari-ans,” with the ability to edit and updateother users’ records.

Combining a coarsely grained catego-rization scheme with constant mainte-nance by librarians dramaticallyimproved the accessibility of informa-tion to employees. The system beganwith approximately 800 documents andgrew to 1200 in its first five months. By

that point, only two documentsremained in the “Other” class. The sys-tem is still in use, and the authors hopeto make the code publicly available.

DRYDOCK: A DOCUMENT FIREWALL

Deepak Giridharagopal, University ofTexas at AustinGiridharagopal works in a universityresearch lab, a relatively open environ-ment where many autonomous groupsshare responsibility for publishing con-tent to the Web. The lab’s managementneeded to enforce a policy on publishinginformation to the Web, ensuring thatsensitive or proprietary information isnot accidentally made available on thepublic Web server. Enforcing policyrequires oversight and accountability,both of which are addressed by the Dry-Dock system. Until the implementationof DryDock, policy was enforced onlywhen complaints were received.

The DryDock system uses a Web appli-cation to manage a Web site. Content isstored in CVS, and document metadataand approvals are stored in a MySQLdatabase. The approach requires twoWeb servers: an internal staging serverlocated behind the firewall and an exter-nal production server located on theDMZ. Authors are free to work with thecontent on the staging server, usingmethods such as FTP or WebDAV toaccess the document root. The produc-tion server, however, is stripped ofnon-essential programs and hardened.DryDock automatically propagates content from the staging server to theproduction server via SSH once theappropriate approvals have been obtained.

Giridharagopal suggests that one way tolook at DryDock is as a tool to shiftresponsibility for content oversight awayfrom sysadmins and back to manage-ment. Sysadmins are responsible forkeeping the system running, but in orderfor any content to appear on the publicWeb site, DryDock requires it to beapproved. Web authors are free to workdirectly on the staging server, and Dry-

LISA ’03 l

Dock will show the differences betweenthe current contents of the stagingserver and that of the public Web site.Users are informed when pages havechanged, and those with managementauthority are able to approve publica-tion. DryDock logs the time at whichfiles were approved and which usersapproved them, and allows content to berolled back to previous versions whennecessary. In use for over a year, the sys-tem has resulted in improved Web serversecurity and better management over-sight of the publication process.

URL: http://tools.arlut.utexas.edu/DryDock/

SYSTEM AND NETWORK MONITORINGSummarized by Venkata Phani KiranAchanta

RUNTIME DETECTION OF HEAP-BASEDOVERFLOWS

William Robertson, ChristopherKruegel, Darren Mutz, and FredrikValeur, University of California, SantaBarbara This paper is about a technique thatprotects the management information ofboundary-tag-based heap managersagainst malicious or accidental modifi-cation. William started out by describingthe motivation behind his work, whichhe mainly attributes to the increasinglycommon buffer overflow exploits result-ing from use of various insecure lan-guages for application development. Hereinforced his argument by citing therecent vulnerabilities in OpenSSH,MySQL, etc.

He explained how the buffer overflowexploit occurs and then discussed exist-ing approaches to detect and preventthem, pointing out flaws and describinglimitations in existing methods.

Then he introduced his approach, anadaptation of the canary-based stack-protection scheme, where the canariesare seeded with a random number,which a mechanism prevents the


intruder from seeing. This detectionscheme has been implemented as apatch to the GNU libc library.

William did some micro- and macro-benchmarking and stability evaluation.Later, he discussed techniques to beadopted to handle buffer overflowexploits.

The software can be downloaded fromhttp://www.cs.ucsb.edu/~rsg/heap.

DESIGNING A CONFIGURATION MONITORINGAND REPORTING ENVIRONMENT

Xev Gittler and Ken Beer, DeutscheBank The configuration monitoring andreporting environment (CMRE) is a tooldesigned to collect and report on themany configuration details of systemswithin an enterprise. Its goal is to pro-vide a single, complete, up-to-daterepository of all system configurationinformation regardless of platform oruse.

Gittler described their operating envi-ronment as a conglomeration of diversesystems with different standards andprocedures and discussed the potentialproblems posed by such an environ-ment.

CMRE needs few prerequisites in orderto do its job; in fact, the necessaryframework for CMRE already exists attheir shop. CMRE is modular, flexible,and runs on many different platforms. Itis written in a combination of Perl, Kornshell, and PHP and uses proprietary aswell as open source software. CMREcurrently collects data on thousands ofUNIX and Windows systems atDeutsche Bank worldwide.

Gittler showed us some GUIs of CMREand explained the usefulness of the datait collected. He then described the sce-narios where they ran into problemswhen designing and deploying this sys-tem.

Although most of the organizations havethis kind of monitoring tool already in

use, Gittler advocated the superiority ofCMRE, citing the simplicity and non-intrusive nature of the tool and the easein interpretation of the gathered data.

Contact information: [email protected];[email protected]

NEW NFS TRACING TOOLS AND TECHNIQUESFOR SYSTEM ANALYSIS

Daniel Ellard and Margo Seltzer, Harvard University Daniel opened with the background andmotivation for doing the paper. He thendiscussed the usefulness of looking atpassive NFS traces over a period of timeand talked about the work already donein this arena. He went on to cite someexamples of basic and advanced analysesof the gathered data and their relevanceto system administration.

The two main tools used for data gather-ing and analysis were nfsdump and nfs-scan. Several related utilities were usedin the analysis part. The data was gath-ered in a university environment, andmeasures were taken to anonymize thedata as much as possible. There is con-trol over anonymity of the data if some-one wants to use the tool for real datacollection and analysis.

The software and the results can befound at http://www.eecs.harvard.edu/sos/software/.

DIFFICULT TASKS MADE EASIERSummarized by Jarrod Millman

EASYVPN: IPSEC REMOTE ACCESS MADEEASY

Mark C. Benvenuto and Angelos D.Keromytis, Columbia UniversityAs a student at Columbia University,Mark developed EasyVPN to integratean unencrypted, untrusted wireless LANinto the Computer Science Depart-ment’s LAN and to the Internet. Hismain design goal was to create a simpleand easy-to-use VPN based on IPSec.Unfortunately, as anyone who has triedto do this in a heterogeneous environ-ment knows, setup varies with each

IPSec platform; furthermore, managingcertificates is too complicated for usersand too time-consuming for administra-tors. To address these issues, Mark cre-ated a solution that leverages the wideavailability of Web browsers withSSL/TLS support and the familiarity ofusers with Web-based interfaces. TheWeb interface allows the user to createand download the configurations andcertificates for their computer withoutfurther burdening the system adminis-trator or requiring the user to under-stand the technical minutiae.

EasyVPN is composed of three maincomponents: the client, the gateway, andthe VPN server. The client receives thecertificate from the gateway, whichserves as the certificate authority (CA).The VPN server trusts the client becauseit trusts the gateway. Thus, EasyVPN isbuilt on trust and the easy manageabilityof the CA. To demonstrate the feasibilityof such an approach, Mark implementedEasyVPN using Linux FreeS/WAN andWindows clients.

THE YEARLY REVIEW, OR HOW TO EVALUATEYOUR SYS ADMIN

Carrie Gates and Jason Rouse, Dalhousie University Many nontechnical managers andemployers do not fully understand whata system administrator is or what he orshe does. Only recently have there beenany publications on the hiring and firingof system administrators. Moreover, thereis no clear course of study or career pathfor becoming a system administrator.Consequently, it comes as no surprisethat there is no systematic approach forevaluating the performance or effective-ness of a system administrator. Carrieand Jason presented an approach toevaluating system administrators basedon three criteria: achievement of goals,achievement of specified service levels,and general competence. Using thesethree broad criteria, they developed aquantitative system for evaluating sys-


l

CO

NFE

REN

CE

REP

ORT

Stem administrators that is measurableand fair.

The first criterion, measuring theachievement of stated goals, requiresthat the manager and administratorwork together and provides the managerwith an objective assessment of perfor-mance. To better understand how anadministrator was achieving specifiedservice levels, Carrie and Jason refinedthis criterion to four components: avail-ability, usability, security, and customerservice. General competence was meas-ured by how often the administratorneeded to revisit the same problem.Breaking the evaluation into these threecriteria provides the manager with aneffective tool to isolate the systemadministrator’s strengths and weak-nesses. They concluded by describingfive different scenarios illustrating howyou might deploy this system, whattypes of scores you might get, and aninterpretation of those scores with sug-gestions for appropriate action. It wasemphasized that this system was meantto initiate a wider and more extensivediscussion on this important topic.

PEER CERTIFICATION: TECHNIQUES ANDTOOLS FOR REDUCING SYSTEM ADMINSUPPORT BURDENS WHILE IMPROVINGCUSTOMER SERVICE

Stacy Purcell, Sally Hambridge, DavidArmstrong, Tod Oace, Matt Baker, andJeff Sedayao, Intel Corp.Before peer certification, trouble ticketsat Intel Online Services (IOS) werereceived by help-desk technicians, whowould pass them on to the system andnetwork administrators to handle. Thiscaused constant interruptions for theadministrators, frustrated the techni-cians because they weren’t able to solvethe problems, and impeded customerservice due to the lack of direct contactbetween the customer and the problemsolver. IOS wanted a way to allow thetechnicians to handle the tickets them-selves, but needed to ensure that thetechnicians were qualified to do so. Tothis end, they created a peer certification

process to add qualified troubleshootingpersonnel.

The certification process divided trou-bleshooting personnel requirements intwo ways – specialty areas and specialtylevels. Certification for a specific areaand level requires previous-level certifi-cation, an oral test, and monitored com-pletion of tasks. Once implemented,peer certification resulted in an increasein the number of staff able to makechanges and a reduction in the numberof trouble tickets referred to the systemadministrators.

EMERGING THEORIES OF SYSTEMADMINISTRATIONSummarized by Kevin Sullivan

ISCONF: THEORY, PRACTICE, AND BEYOND

Luke Kanies, Reductive Consulting, LLCLuke describes his development experi-ences with a configuration managementtool, ISconf. Although ISconf has gonethrough significant rewrites since theinitial version, it still functions by pair-ing listings of commands with a list ofhosts for those commands to be run on.ISconf ’s use of make satisfies three com-ponents of deterministic ordering: statemaintenance, failure on error, and con-sistent ordering. The concept of atomic-ity is one which ISconf does not currentlypossess. In many processes, the lack ofsupport for atomicity requires humanintervention when an error is encoun-tered. Also, hidden preconditions of asystem create situations that ISconfwould have difficulty handling. The dis-cussion of these shortcomings will helpthe development of ISconf and tools likeit. ISconf is still a very useful tool andwhen combined with other configura-tion management tools these inherentproblems can be mitigated.

LISA ’03 l

SEEKING CLOSURE IN AN OPEN WORLD: A BEHAVIORAL AGENT APPROACH TOCONFIGURATION MANAGEMENT

Alva Couch, John Hart, Elizabeth G.Idhaw, and Dominic Kallas, Tufts UniversityAlva opened by describing a racebetween theory and practice in whichtheory always wins. The main goals ofhis work are portable validation, wherevalidation occurs once and the resultsare the same everywhere, and to producean algebraic model of configurationmanagement. Couch contends that thesegoals can be achieved through the use ofclosures and conduits. Closures are like ablack-box system that has well-definedinputs and outputs and functionsexactly as specified. Conduits are com-munication channels between closures.The first step in developing a closure isseparating internal and external parame-ters. If it were not for latent precondi-tions, the composition of closures wouldbe closures themselves. This essentiallycreates complex services with knownfunctionality and well-defined inputsand outputs. File editing was an initialprototype of this work. A file-editingclosure can define all permissible actionsto a file in an attempt to reduce errors.Many system administrators arewrapped up in the minutiae of the manysystems they manage and have less timeto do high-level coordination of ser-vices. When these low-level systems aretreated as closures and conduits, itbecomes easier to focus on moreadvanced system administration tasks.

ARCHIPELAGO: A NETWORK SECURITYANALYSIS TOOL

Tuva Stang, Fahimeh Pourbayat, MarkBurgess, Geoffrey Canright, KenthEngø, and Åsmund Weltzien, Oslo University CollegeTuva Stang presented a tool that wasintended to visually model intercon-nected networks. These networks can bephysical, social, or knowledge networks.Graph theory was used to show the con-nections that exist between groups of


people, hosts, or other informationsources. The most well-connected nodeswill become visually apparent. An inter-esting comparison was drawn betweenan organizational chart and the chartspresented here; in some cases they differ,and the truly connected people arerevealed. As a security tool, Archipelagocan reveal vulnerable points in a net-work or even the nodes that should bebest secured, due to their importance.The graphs produced by this tool showboth the importance and centrality ofthe nodes.

PRACTICUM: UNUSUAL TECHNIQUESFROM THE FRONT LINESSummarized by William Reading

THREE PRACTICAL WAYS TO IMPROVE YOURNETWORK

Kevin Miller, Carnegie Mellon UniversityFirst Idea: IP Anycast

IP anycast is the same as shared unicast,in which one IP address is assigned tomultiple hosts and the network routingis configured to deliver to one of themany machines that have that IP addressconfigured.

Migrating is not very difficult. Forservers that simply use DNS, only anupdate to DNS is required. In an IP any-cast environment, without requiring aconfiguration change, clients end upusing a server that is closer to them thanothers on the network.

Second Idea: Source Address Verification

Filtering is accomplished by performingsource address verification on edgerouters using unicast reverse path for-warding. This uses the unicast routingtable to make the filtering policy andrequires little work compared to tradi-tional filtering with ACLs.

Third Idea: Host Filtering

This builds on the topics mentioned ear-lier. Essentially, the problem is that thereare a large number of hosts that need to

be denied access to the network due toviruses and such.

Expect scripts are tedious and can causeproblems, so a host route is given, essen-tially pointing to a sinkhole – whichthen drops the packets. When the hosthas been cleaned up, the route is removed.

TOSSING PACKETS OVER THE WALL USINGTRANSMIT-ONLY ETHERNET CABLES

Jon Meek and Frank Colosimo, WyethProtecting an internal network whilemonitoring from remote sites consid-ered to be insecure poses a difficultproblem. The talk was loosely organizedinto the topics of hardware, software,and applications.

On the hardware side, simply snippingthe wires does not work, and it is hap-hazard to do things like soldering apaper clip to an Ethernet card if securityis concerned.

However, it is possible to create a circuitthat does not permit packets to returnover the line. By writing custom soft-ware which only relays packets to a spec-ified host on an internal network fromthe crippled line, security can be main-tained.

THE REALITIES OF DEPLOYING DESKTOPLINUX

Bevis King, Roger Webb, and GraemeWilford, University of Surrey Linux offers a number of benefits fordeploying on the desktop, yet a certaindegree of Windows compatibility is amust. However, using Linux on the cor-porate desktop reduces the support timerequired.

Running Microsoft Windows in a virtualmachine has a number of benefits forsupport because the Windows machinesdo not have direct access to the network,have abstracted hardware, and are notwritable by the end user.

The desktops themselves have greateraccess to scientific applications that onlyrun on UNIX, and there is a completely

supported X server running to hostthese applications remotely.

CONFIGURATION MANAGEMENT:TOOLS AND TECHNIQUESSummarized by Marko Bukovac

STRIDER: A BLACK-BOX, STATE-BASEDAPPROACH TO CHANGE AND CONFIGURA-TION MANAGEMENT AND SUPPORT

Yi-Min Wang, Chad Verbowski, JohnDunagan, Yu Chen, Helen J. Wang,Chun Yuan, and Zheng Zhang,Microsoft Research

In a dynamic talked welcomed byadministrators who have MicrosoftWindows machines on their network,Dr. Wang presented STRIDER, a Win-dows tool that helps to pinpoint the ori-gin of Windows registry problems.Windows XP has about 200,000 registryentries storing all configuration data, sofinding a source of evil is downrightimpossible without a proper tool. Byusing white-box data (from supportdocumentation) and black-box testing,STRIDER manages to narrow down thenumber of possible problems in the reg-istry, making identification fathomablefor a human administrator.

Starting with all the registry entries,STRIDER creates a smaller subset bymechanically eliminating entries that areirrelevant to the current problem. Itthen uses a statistical model to filter outthe entries that may be relevant but aremost likely not the root of the problem.


l

CO

NFE

REN

CE

REP

ORT

SEach entry in the smaller subset is thencompared to a computer genomics data-base, a data set obtained from trou-bleshooting experiences and black-boxtests, to potentially pinpoint the solu-tion.

In addition to the published paper, Dr.Wang has a Web page at http://research.microsoft.com/~ymwang whereone can find more information onSTRIDER.

CDSS: SECURE DISTRIBUTION OF SOFTWAREINSTALLATION MEDIA IMAGES IN A HETERO-GENEOUS ENVIRONMENT

Ted Cabeen, Impulse Internet Services;Job Bogan, ConsultantCDSS provides a framework for a distri-bution of software images over a num-ber of protocols. Software images arestored on an isolated server for everyuser who is trying to download animage. The user can communicate onlywith the designated server and canobtain only the requested files. The sys-tem does not require any additionalsetup on the user’s side, as CDSS usesstandard protocols (HTTP, FTP, SMB,etc.) and a set of shell scripts to accessthe desired information.

A user who visits a Web page that lists allavailable software images selects theones he or she’s interested in and pro-vides necessary passwords to accessthem. At that point, a directory is cre-ated for that user, containing only therequested images. At the same time, theservers necessary to allow the user toaccess the data over the desired protocolare configured and started. By usingLinux firewall rules, the user’s request isredirected to a non-standard port foreach protocol and the data is made avail-able.

CDSS is under a GPL license; moreinformation about it can be found athttp://cdss.sf.net.

VIRTUAL APPLIANCES FOR DEPLOYING ANDMAINTAINING SOFTWARE

Constantine Sapuntzakis, David Brum-ley, Ramesh Chandra, Nickolai Zeldovich, Jim Chow, Monica S. Lam,and Mendel Rosenblum, Stanford UniversityComputer Appliance is a device, likeTivo, for which the software is installedby the manufacturer (who also providesupdates) rather than by the user. Sapuntza-kis and fellow researchers took this con-cept and applied it to virtual appliances,which are just like the physical appli-ances but without the hardware. Ratherthan running the appliances on the barex86 hardware, the authors use theVMware GSX Server.

In the presentation and the demo thatfollowed, Sapuntzakis introduced thebasic concepts and presented a proto-type model that allows creation, publica-tion, execution, and update of virtualappliances. He argues that using virtualappliances reduces the amount of timeneeded to administer computers, byhaving a central management unit con-trol all the software for all the applianceusers.

Sapuntzakis et al. also developed aunique configuration language, CVL(collective virtual appliance language),whose syntax is used to describe VAPconfigurations. Their demo showed theaudience sample .cvl files and how toadminister the VAPs. More informationon Sapuntzakis and the project can befound at http://suif.stanford.edu/~csapuntz/.

CONFIGURATION MANAGEMENT:ANALYSIS AND THEORYSummarized by Aaron Teche

GENERATING CONFIGURATION FILES: THEDIRECTOR’S CUT

Jon Finke, Rensselaer Polytechnic InstituteAt LISA 2000, Jon Finke presented apaper about configuration generationfrom a relational database. At LISA ’03,

Yi-Min Wang and Chad Verbowskireceiving the Best Paper Award from

Æleen Frisch

LISA ’03 l

he shared his improvements using XMLand XSL, with data stored in the rela-tional database for configuration man-agement. While the original systemworked very well, it wasn’t flexibleenough. Any layout changes required aPL/SQL programmer, and the PL/SQLprogrammer needed presentation skills.In comes XML with XSL transforms.The relational database is still used, butthe data goes from the database to XMLthrough an XSL translation to the finaloutput. XML and XSL are platform-independent, which makes this solutionvendor-independent. And, finally, themove to an XML/XSL system providesbasic consistency checking along thetransformation path.

PREVENTING WHEEL REINVENTION: THE PSG-CONF SYSTEM CONFIGURATION FRAMEWORK

Mark D. Roth, University of Illinois atUrbana-ChampaignMost configuration management toolsare designed monolithically and can’tmix and match ideas and functionality.This results in lots of wheel reinvention.Mark Roth presented his solution to thisproblem, psgconf. While monolithicconfiguration management tools man-age file configs, not abstract ones, psg-conf solves this problem with modularity.The psgconf framework is a hierarchy ofsmall, write-once-use-often Perl mod-ules that manage the configuration at aconceptual level. It is intended to knowwhat the data is and to control manipu-lation of that data according the require-ments set by the admin.

SMARTFROG MEETS LCFG: AUTONOMOUSRECONFIGURATION WITH CENTRAL POLICYCONTROL

Paul Anderson, University of Edin-burgh; Patrick Goldsack, HP ResearchLaboratories;Jim Paterson, University ofEdinburgh.LCFG is a config tool that takes a high-level specification and generates amachine profile. LCFG can rebuild anentire site from bare metal, given a cen-tral source repository. SmartFrog pro-


vides a framework for configurationmanagement of distributed applications.It is a runtime environment whichorchestrates the workflow of computersaccording to configuration. SmartFrogin combination with LCFG can controland maintain a robust service that auto-matically reallocates machines and ser-vices based on demand, including theability to rebuild around failure.

NETWORK ADMINISTRATIONSummarized by Hernan Laffitte

DISTRIBUTED TARPITTING: IMPEDING SPAMACROSS MULTIPLE SERVERS

Tim Hunter, Paul Terry, and Alan Judge,eircom.net

The authors’ company, eircom.net, is thebiggest ISP in Ireland, with approxi-mately 500,000 users. For them, spam isa big problem: On several occasions theyhave seen their server outages reportedon by the media. To help alleviate thisproblem, they have configured a tarpit-ting mechanism.

The method known as “tarpitting” involvesinserting a time delay between themoment a message is received by theSMTP server and the moment when theserver returns its “250 OK” response.This time delay varies: The goal is for itto be zero for legitimate users and up to30 seconds per message for spammers.This solution is a reasonable middleground; there is no need to filter mes-sages based on content, which raises pri-

vacy concerns or risks dropping poten-tially valid messages.

The paper explains how eircom.netimplemented a centralized database ofmessages recently received from eachclient. A “Theory” section of the paperexplains how to set the right parametersso client addresses get tarpitted anduntarpitted over time, according to howmany messages they send. The “Data”section explains how the method wasimplemented across eircom.net’s variousmail servers, using qmail as SMTPserver, and IP multicast to share clientbehavior data, which each machinestores locally on a SQL database.

Finally, a “Tarpitting in Practice” sectiondescribes the political problems involvedin setting the right parameters for thetarpit and developing policies to followwhen a would-be spammer is found inthe tarpit. The authors also include datagathered from an actual spamming ses-sion, with the spammer trying to navi-gate around the restrictions posed by thetarpit.

This method has helped eircom.netsolve the problem of burst attacks, butsome work remains to be done regard-ing lower-level spamming. In conclu-sion, tarpitting is a useful addition to theanti-spam toolbox.

USING SERVICE GRAMMAR TO DIAGNOSEBGP CONFIGURATION ERRORS

Xiaohu Qie, Princeton University; SanjaiNarain, Telcordia TechnologiesIt is not uncommon for all routers on aBGP network to be operational and yetroute packets incorrectly. This happensbecause traditional network diagnostictools can only detect localized errors,such as bad cables or software failures.More automated tools are needed to sys-tematically search through the problemspace.

This paper analyzes the use of the Ser-vice Grammar technique for diagnosingBGP configuration errors. BGP presentsa number of challenges for its imple-

Tim Hunter and Paul Terry receivingthe Best Paper Award from

Æleen Frisch

mentation: At the low level, individualrouters have to be configured independ-ently, yet the high-level global routingpolicy of the (sometimes very large) net-work has to be kept consistent across allrouters.

Since BGP is a complex protocol, themanual configuration of routers is atime-consuming and error-prone task.This paper presents a Service Grammarfor configuring BGP networks. This Ser-vice Grammar consists of a “BGP Require-ments Language,” which expresses theBGP logical structures; a ConfigurationDatabase, which abstracts the differentvendor-specific configurations; and aDiagnosis Engine, which is a set of algo-rithms that validates the configurationdatabase and provides useful informa-tion for the debugging process.

The paper includes an example network,where Service Grammar was used todiagnose the configuration of nine Ciscoroutes, grouped in five ASes.

SPLAT: A NETWORK SWITCH/PORTCONFIGURATION MANAGEMENT TOOL

Cary Abrahamson, Michael Blodgett,Adam Kunen, Nathan Mueller, andDavid Parter, University of Wisconsin,MadisonThe old network infrastructure of theUniversity of Wisconsin Computer Sci-ence Department consisted of multipleunmanaged Ethernet switches, wherepeople would just plug in their worksta-tions. When the old network wasreplaced with 50 managed switchesusing VLANs, the need arose to imple-ment a solution to automate the man-agement of the network infrastructure.

After considering the existing solutions,the authors of the paper decided toimplement the Splat tool. This tool pro-vides an easy-to-use interface for config-uring the switch ports while enforcingsysadmin best practices.

Using Splat’s CLI interface is relativelystraightforward; the tool was designed toaccommodate relatively inexperienced


l

CO

NFE

REN

CE

REP

ORT

Sadministrators. For example, to connecta host to a switch port, the only requiredparameters are the hostname and thelabel of the data jack on the wall. Thetool does the rest: updates the database,computes the new VLAN configuration,and issues the required switch configu-ration command using the Rancidswitch configuration manager. The cur-rent configuration data is stored in aPostgreSQL database, which can also bequeried using Splat.

The use of the tool is enforced because,without it, the VLAN number is not cor-rectly configured for the switch port,which means the network connectionwon’t work. Also, the tool “locks” theswitch port to the MAC address of theworkstation. Thus, using Splat is easierthan changing all these parameters byhand.

This creates a virtuous cycle: the Splatdatabase is the definitive data source forhost/switch-port mapping. And since it’seasier to use Splat than to configure theswitches by hand, the Splat database iskept current. This way, the sysadminscan easily follow the best practices whenmanaging the switch port configuration.

GURU SESSIONS

IPSEC

Hugh Daniel, Linux FreeS/WAN Project Summarized by Siddharth AggarwalSince this was a guru session, it involveddirect questions to the speaker by theaudience. Hugh Daniel began by sayingthat IP networking is antithetical toIPSec. Most system administrators findimplementing IPSec problematicbecause the setup is not done correctly.So the speaker explained a test setup fora Web site in which all the machines arephysically kept together.

Daniel clarified some misconceptionsabout IPSec – for example, that it istechnically a transport mechanism andnot a technique for authentication orencryption. It is the job of Internet KeyExchange (IKE) to maintain pre-shared

secrets and RSA keys. Daniel introducedvarious ways of deciding if two hosts cantalk to each other: pre-shared secrets,RSA keys, X Auth, X.509, etc. Also, abrief introduction about a PDA thatruns Linux, called Zaurus, was given.

Daniel then introduced the Wavesectechnology, which uses a combination ofopportunistic encryption (OE), dynamicDNS, and DHCP. OE enables you to setup IPSec tunnels without coordinatingwith another site administrator andwithout hand-configuring each tunnel.He also explained the goal of FreeS/WAN, which is to provide a host-to-host or network-to-network privacyenvironment via a distributed databaseof DNS entries and keys. He explainedwhy FreeS/WAN emphasizes an anti-NAT (Network Address Translation).IPSec fails when packets go through aNAPT (network address and port trans-lation) box, because NAPT mangles thepackets.

The session concluded with some linksto useful resources:http://www.freeswan.cahttp://www.wavesec.orghttp://www.freeswan.org/talks/lisa-2003

AFS Esther Filderman, The OpenAFS Project;Garry Zacheiss, MIT Summarized by Venkata Phani KiranAchantaThe AFS guru session consisted of ques-tions about large file size support, read-write replication functionality, status ofdisconnected AFS, back-up strategies,and many other topics as well.

Some people asked whether there wereplans to make read-write replication ofvolumes. Esther said the Coda filesystemdoes RW replication of volumes (there isno notion of cell in Coda yet), but theywere not sure whether it would be avail-able in AFS or not. Garry added thatCoda is entirely a research project and isnot for use in a production environ-ment.

LISA ’03 l

Regarding disconnected AFS status,Garry said that there was an initial ver-bal commitment from the University ofMichigan to incorporate disconnectedAFS functionality into OpenAFS code,but they later backed out because theyare heavily into OpenBSD research.

Alf Wachsmann from Stanford LinearAccelerator Center made an announce-ment about an OpenAFS best-practicesworkshop being held at SLAC in Febru-ary.

People were curious to know how MITand PSC were doing backups. Garry saidthey were using butc with a bunch ofself-written Perl scripts, which need nohuman interaction. Esther said that theywould do a vos dump locally and, withHSM support, would migrate that dumpto a repository. She added that thereused to be an add-on to Legato a whileback. The most popular backup solutionfor AFS is TSM. Cornell University isworking to tie AFS into Amanda.

There were some people interested inusing OpenAFS in a grid computingenvironment, but lack of file support forfiles greater than 2GB seems to be a lim-itation for them.

A newbie asked about recovery when anRW volume is lost. Esther said they canalways do a vos dump of the existing ROcopy of the volume as an RW volumeand start using it as if nothing had hap-pened.

Somebody asked whether the 22-charac-ter limit in the naming size of volumeswould be increased in future releases ofOpenAFS. Garry said that there are noplans to increase it, but that there is aworkaround using MD5 hashing. Estheradded that if they did increase the limit,the old AFS clients would be confused.

Answering a question on ideal clientcache size, Esther said that it wouldmostly depend on the chunk size at theirsite. Someone asked whether to restartthe file server once a week if clients are


using it 24/7. Garry said there is nonecessity to restart.

There was discussion about MRAFS,which is heavily used by Naval ResearchLabs; different authentication tech-niques; and why AFS uses Kerberos.

Like any other open source project, Open-AFS also seems to suffer from lack of“more” volunteer time. The gurus wereoptimistic about the future of OpenAFSand said that if more volunteers werewilling to contribute to the OpenAFSproject, there would be much morefunctionality that could be incorporatedinto OpenAFS.

MBAS FOR SYSADMINS

Brent Chapman, Great Circle AssociatesSummarized by Carrie GatesWhy should a system administrator pur-sue an MBA? There are two answers tothis. The first is the marketing-typeanswer, which is that it will, on average,add 25–40% to your current salary. Thesecond answer is that it provides a betterunderstanding of the entire businessenvironment, such as finance and per-sonnel, which in turn will allow you tobetter relate to the concerns of thosewho work in these other departments.

There are three paths to an MBA: stan-dard full-time courses, part-timecourses, and the executive-level MBA.Although the full-time MBA allows astudent to complete the degree morequickly, the part-time MBA enables oneto keep working while obtaining thedegree. Unfortunately, the part-time stu-dents often miss out on many of theopportunities available to the full-timestudents. Conversely, the part-time stu-dents tend to be older and have morebusiness experience, and so the full-timestudents often miss out on learningfrom discussions with them. The execu-tive MBA is a combination of the twoapproaches, but is more expensive and isgeared toward senior managers (wheretheir company is paying for tuition).Typical courses consist largely of case

studies, with a single case study takingup 5–25 pages of scenario. These casestudies are used to generate and guidediscussion.

The bottom line is that you will get outof an MBA what you put into it. MBAsoffer a wealth of learning opportunities,both in the classroom and outside of it,as well as providing the opportunity forconsiderable networking within thebusiness field. For those who are inter-ested in pursuing a management path, itcan also provide an extra credentialwhen applying for management posi-tions. Beyond this, it can provide some-one who has a technical backgroundwith the confidence to pursue careerpaths such as CIO or CTO.

PKI/CRYPTOGRAPHY

Greg Rose, QUALCOMM, Inc. Summarized by der.hansRose mentioned that there are two typesof cyphers, symmetric and asymmetric.Symmetric cyphers, such as DES andRijndael (accepted as AES), are the tra-ditional type of cypher and there is evi-dence they were used as far back as 2000BC. Symmetric cyphers use the same keyboth ways. Asymmetric cyphers, a.k.a.public key cyphers, such as RSA, use dif-ferent keys for encryption and decryp-tion.

All the old cell phone cryptography wasbroken. Rose was first to break some ofthe algorithms. The new 3G cell net-works use different but equivalentciphers. All use 128-bit keys. One of theproblems with the old algorithms is thatthey were created behind closed doors.Review of the algorithm and the code isimportant to be certain an implementa-tion is secure.

Rose gave several examples of cryptogra-phy that was weak due to shortcomingsin the algorithms or errors in the imple-mentation. He mentioned that mostWeb server administrators know that-most of the CPU is used in putting thepadlock on the browser, not in transmit-

ting the data. For instance, small keystake constant time because they fit 32-bit CPUs, but large keys have to be bro-ken up and done “longhand.” Goingfrom 1024 bits to 2048 bits cubes thetime needed to generate the key. Com-putational time equals lost battery lifefor cell phones.

LINUX

Bdale Garbee, HP Linux and OpenSource Lab/DebianSummarized by Hernan LaffitteTopics such as the SCO lawsuit and theend-of-life announcement from RedHatfigured prominently in the first segmentof the talk. Mr. Garbee explained thatHP’s first concern is supporting its cus-tomers, many of whom run SCO andRedHat, and also promoting the use ofopen and free standards.

Another important issue facing Linuxdevelopers is that a number of indepen-dent software vendors (ISVs), such asOracle, and hardware manufacturers,such as HP, will only certify their prod-ucts against a small number of commer-cial Linux distributions. This is a resultof the economic realities of setting upQA and support, and the fact that notwo Linux distributions seem to use thesame kernel.

Setting up standards for Linux distribu-tions will help alleviate this problem,and Linux 2.6 will have a feature setcloser to what many ISVs want. Othercompanies, however, will want to adddifferent features to the kernel. Andthere is always the issue, even if every-body agrees on the current standard, ofnegotiating which features will go intothe next one.

Economic realities also conspire againstselling Linux to the general (read: non-techie) public. For example, putting aline of Linux-powered machines on theshelves of a computer store involves a lotof expenses: printing a different set ofmanuals, different packaging, tracking adifferent part/model number from the


l

CO

NFE

REN

CE

REP

ORT

Sfactory down to the store in Kalamazoo . . . it’s all expensive, even if the OS isfree.

Actually, the money involved in the OSlicensing is not as much as many believe.It’s simply a question of demand. Thedemand is growing, but it’s still notthere. The marketing people would say,“Come back next year if you have 10times the current volume.” Also, Mr.Garbee commented jokingly, whateverdistribution you choose to sell, the restof the Linux users will hate you.

The juggernaut is rolling in the rightdirection, though. For example, HPrecently released a BIOS patch for one ofits systems specifically to improve Linuxcompatibility, and is working to improveLinux compatibility in general.

Mr. Garbee also talked about his experi-ence in porting Linux to the Itaniumplatform. A big percentage of the Ita-nium 2 systems shipping in the firstquarter of production were Linux, andthe trend increased in the second quar-ter. Linux is also very popular in Itaniumworkstations, and HP-UX customers likehaving the possibility of replacing oldPA-RISC machines with Itanium with-out having to make any changes to thesoftware.

In addition to his work on UNIX andLinux, Mr. Garbee is a prominent mem-ber of the amateur satellite community.The talk touched briefly on the issues ofLinux in space (it was used in an experi-ment on the shuttle, and will also beused in the amateur radio experimenton the international space station). Mr.Garbee also discussed the technology ofamateur satellites. He stressed that thereis a constant need to simplify the hard-ware requirements. The 1802 processorused on many satellites, for example,runs at 100 KIPS (kilo instructions persecond). Things don’t happen very fastin space, so there is no need for lots ofprocessing power. And amateur satellitesare a fun hobby in part because thetypes of problems faced when working

on 8-bit micro-controllers are quite dif-ferent from the ones encountered whenworking on Linux for Itanium systemsat HP.

AUTOMATED SYSTEMADMINISTRATION/INFRASTRUCTURE

Paul Anderson, University of Edinburgh; Steve Traugott, Infrastructures.OrgSummarized by Kevin SullivanConfiguration management seemed tobe a central theme of this year’s confer-ence, and it took center stage at this gurusession. A packed room gathered to hearPaul Anderson and Steve Traugott givetheir opinions on the state of automatedsystem administration. The hour-and-a-half session was very informative, with agreat discussion of theory interspersedwith various tools administrators areusing today.

The discussion quickly turned to “push”vs. “pull” systems in configuration man-agement. Steve and Paul contended thatmany people who think they have a“push” system actually have a “pull” sys-tem. Steve said that a “pull” system isadvantageous because it reduces thethreat of divergence, since a machinewill properly configure itself before itoffers any services. Paul added that“pull” systems don’t require any knowl-edge about the state of the machine atconfiguration time, so offline hosts willnot be missed.

Paul went on to describe a configurationfabric consisting of hardware, software,specifications, and policies. Soon theroom was buzzing about the tools usedto build and maintain this fabric. Eachtool employed a different paradigm:Anderson’s tool, LCFG, tells a host whatit wants to look like, while Traugott’sISConf – originally a quick fix aimed atbuilding up an infrastructure – tells ahost what to do.

Also discussed was “The Test,” in whichyou imagine taking a random machinethat has never been backed up, destroy

LISA ’03 l

it, and then have its services recoveredwithin 10 minutes. Both Paul and Stevenote that their infrastructure manage-ment systems pass The Test.

PROFESSIONAL GROWTH AND DEVELOPMENT

David Parter, University of Wisconsin,MadisonSummarized by Marko BukovacDavid Parter led an excellent free-flow-ing discussion covering several topics ofinterest to system administrators inindustry and academia. The first topiccame from mid-level administratorswho were interested in knowing how tomentor their students and colleagues.Senior administrators recommendedthat students develop a range of techni-cal skills, including “people skills,” whichis a big part of the job. In addition, men-tors should always treat system adminis-tration as a legitimate profession (it isnot always seen as such by users). Stu-dents should be encouraged to commu-nicate with their mentors (who shouldset some time aside to work with stu-dents) and ask questions using SAGEonline resources, such as the Web site,IRC channels (irc.sage-members.org#sage-members), and the mailing list([email protected]).

Mid-level admins mentioned that logicalthinking and thorough knowledge of thefundamentals (though it can sometimesbe hard to define what fundamentalsreally are) are perhaps the most highlyvalued skills in the field. Some adminsmentioned that students’ fear of “break-ing things” slows their growth and thatthey should be encouraged to experi-ment, only not on the main servers. Adebate about the relative importance ofdepth cersus breadth concluded thatthey are equally important.

To keep their job fun and interesting,some administrators would like theirjobs to change with time and includemore research. While there is no overallsolution to this, as it is company-depend-ent, some senior admins recommendedbooks, such as O’Reilly’s Love Your Job,


and some recommended writing andsharing tools, which can then lead tomore communication between compa-nies and to more research on the subject.Many recommended getting books andtaking classes on time management,since this is a skill that many admins(especially younger ones) lack. Givingsmall group tutorials and then expand-ing might lead to giving a tutorial at aLISA conference.

Many of the admins wondered how totake control of their careers. Senioradmins saw themselves in the position ofhaving to join the management andabandon technical duties, to the dismayof most of them. The main suggestion inthis case was to check with HR (evenbefore getting hired) to ask about jobgrowth and possible future duties. Someadmins considered switching from uni-versity environments to the “real world”but feared that they were not ready for it(myth: work at the university is not asimportant and difficult as work at thecorporation). All of them were encour-aged by the corporate admins, who saidthat academia is not at all different fromthe corporate world.

The session concluded with a discussionabout personal career plans. Everyoneshould have a personal career plan andan idea of what their dream job wouldbe. One should not be afraid to ask theemployer about future plans and howthe job will evolve. In their work, adminsneed to manage users, systems, andmanagement, and many find it verytricky to manage all three successfully.Some senior admins suggested takingnonsystem administration courses, suchas management, as well as documentingall political decisions (resources, time,budget) made by their supervisors.Managing management is a vital part ofthe job, and senior admins recom-mended learning this skill.

INVITED TALKS

OUTSOURCING: COMMON PROBLEMS ANDCURRENT TRENDS IN THE OUTSOURCINGINDUSTRY

John Nicholson, Shaw Pittman LLPSummarized by Emma BuneciOutsourcing has been a hot topic overthe past few years, and John Nicholsonpresented an excellent overview of thetopic. Outsourcing is defined as thelong-term contracting of an informationsystem or business process to an externalservice provider in order to achievestrategic business results.

The top-tier providers are IBM, CSD,EDS, and ACS, while in the second tierwe find Perot Systems, Accentrue, CGI,Unisys, and Lockheed Martin Siemens,as well as other consulting firms. As aninteresting change, the hardwareproviders, such as Dell, Compaq, andHP, have all been moving into providingservices for their clients. As offshoreproviders, there are typically the largerIndian companies, such as the TataGroup. IBM is the dominant player inthe global market and was able to main-tain this position by drawing on its ownstrengths and taking advantage of theleadership and accounting problems atother companies.

After outlining the seven major trends inthe outsourcing industry – mid-sizedmarkets; outsourcing of IT, businessprocess, and business transformation;offshore outsourcing; shareholder influ-ence; renegotiation of existing agree-ments; piecemeal deals; and thechanging nature of IT departments –Nicholson discussed problems with out-sourcing. The three major issues seem tobe timing, customer perspective, andperceived poor customer service.

Rushed negotiations, differing expecta-tions, and poor communication withend users lead to a very unhappy rela-tionship. In order to minimize prob-lems, any outsourcing deal must betreated with the same care and planningas buying a used car. It makes sense to

talk to multiple vendors because talkingto only one vendor will undercut negoti-ating leverage. The customers must beclear about document scope, service lev-els, and cost. Pricing must be clearlyspecified before signing the deal. Assump-tions and dependencies must be avoided:If there is any assumption or depend-ency written in a deal, it must be speci-fied how it will imply a change in theprice.

Using an independent deal consultant ishighly recommended; in the same waythat a car mechanic is crucial to buying aused car, a consultant will know how tolook for and evaluate problems that youmight not see. The final piece of advice:“Communicate, communicate, commu-nicate!”

A CASE STUDY IN INTERNET PATHOLOGY:FLAWED ROUTERS FLOOD UNIVERSITY’SNETWORK

Dave Plonka, University of Wisconsin,MadisonSummarized by Jason RouseDave Plonka gave an enlightening talkon the story behind the flooding of theUniversity of Wisconsin’s public NTPserver. On May 14, 2002, Dave wasreviewing network logs. He was quitesurprised to find a nearly 90,000 packet-per-second forwarding rate through oneof the university’s public NTP servers.Seeing that the source port was fixedand IP addresses associated with theflows were random, Dave’s first guesswas a distributed denial of service. Tocombat this, he placed university-localblocks on the ingress routers.

A month later, however, Dave was sur-prised to find the access control listsdropping over 250,000 packets per sec-ond, all with the same IP profile! Thistime, Dave decided to escalate the inves-tigative procedure. He chose the two toptalkers and emailed them directly, receivedimmediate responses, and found thecommonality was a Netgear product.After searching for the model number,Dave located a few references to the


l

CO

NFE

REN

CE

REP

ORT

Sproduct, one such reference, to ICSALabs, mentioning that the Netgearrouter did not include a battery-backedclock.

Plonka’s next step was to examine thehardware and software directly. Hedownloaded the firmware available fromthe Netgear Web site. After a cursoryexamination, he found that the Netgearfirmware included the IP address of oneof the university’s NTP servers. As soonas he made this discovery, Plonka con-tacted Netgear directly via their helpdesk and customer service channels.After a number of days withoutresponse, Plonka phoned a Netgearexecutive directly.

Plonka then guided the formation of ateam consisting of Netgear employees,university employees, and independentexperts. This key step ensured that theproblem could be addressed in a waythat was fair to the university, the com-pany, and the Internet community as awhole. The initial response was to pointusers to an “Instant Code” update, avail-able from the Netgear Web site. Interest-ingly, this code had been available forsome time, but had not been widelyadvertised or adopted by the productcommunity.

Understanding the difficulties involvedin communicating to such a diverse usergroup, the review team pursued otheroptions in order to mediate the largeamount of incoming NTP traffic.Finally, the team concluded that theimplementation of an anycast NTP timeservice at the Wisconsin site could suc-cessfully handle such a traffic load. As ofthis writing, Netgear and the Universityof Wisconsin have undertaken a projectto provide this anycast deployment.

Plonka’s experiences were summed up intwo pieces of sage advice. First, involveall parties in any dialogue when search-ing for a solution. Second, recognize thatthe Internet is a shared resource basedon the good citizenship of many, manyusers, and act accordingly.

ORGANIZATIONAL MATURITY MODELS:ACHIEVING SUCCESS AND HAPPINESS INMODERN IT ENVIRONMENTS

Geoff Halprin, The SysAdmin GroupSummarized by Jason RouseGeoff Halprin has the courage to saywhat we’ve all been thinking: Sysadminshave a hard work life. What with theeconomic downturn since the dot-combomb, the reactionary posture we haveto assume in order to meet fluid busi-ness goals, and the organic nature ofsystem and software development,sysadmins truly have a difficult jugglingact in front of them.

Halprin described system administra-tion as a constant quest for reliability,availability, and serviceability. As a partof this quest, system administratorsmust combat the often organic growthof systems and software, engineeringfixes in order to maintain systemicimprovements. Halprin also mentionedthe distinct lack of recognition for sys-temic improvements, leading to a lack ofwork in this area. This cycle of lowreward and organic growth leads to sys-tems that age badly, requiring more andmore work to maintain them as timepasses.

Halprin also understands that systemadministrators must deal with constantchange. Systems creep toward states ofincreased entropy, and Halprin showshow system administrators can combatthis gradual degradation. By having anexact worst-case cost associated withdowntime, Halprin believes that systemadministrators can communicate moreeffectively with management, achievingmanagement buy-in. Management buy-in improves overall workflow manage-ment, thus lightening the workload onthe system administrator. Managementbuy-in also allows a larger measure ofroot-cause analysis, so often missing inhighly dynamic workplaces.

Finally, given that systems will break,how do system administrators minimizeor control failures? Halprin’s answer is to

LISA ’03 l

ensure that system administrators con-tinuously move toward a proactivestance, constantly re-evaluating theirworkflows and incident handling.

NETWORK TELESCOPES: TRACKING DENIAL-OF-SERVICE ATTACKS AND INTERNET WORMSAROUND THE GLOBE

David Moore, CAIDA (CooperativeAssociation for Internet Data Analysis)Summarized by Carrie GatesDavid Moore described network tele-scopes, what they are and how they canbe used. The basic premise is to take achunk of IP address space that receiveslittle or no legitimate traffic (or receivestraffic that can easily be filtered) andanalyze the traffic that it receives. All ofthe traffic seen by that space (other thanany known, legitimate traffic that hasbeen filtered) represents some unusualnetwork event.

For example, network telescopes can beused to examine the presence of spoofed-IP denial-of-service attacks on the Inter-net. Say you have a /8 network that youcan use as a network telescope. Thisaddress space represents 1/256 of theInternet. If an attacker is DoSing sometarget using spoofed IP addresses thathave been randomly chosen, then thetelescope should see approximately1/256 of the response traffic, as that isthe likelihood that an IP address in thetelescope address space has been chosen.By analyzing this information, we caninfer the number of DoS attacks occur-ring on the network, as well as informa-tion about the attack itself. Over the pasttwo years, for example, there have beenapproximately 40 DoS attacks against/24 networks per hour. The majority ofthese consisted of SYN floods againstHTTP services.

Network telescopes can also be used tostudy the spread of Internet worms.Assuming that there are no biases (orbugs!) in choosing the next IP address toinfect (that is, any target IP address hasbeen chosen randomly across the entireInternet address space), a network tele-


scope can expect to see 1/256 of thescanning traffic generated by any oneinstance of the worm. It was seen withCode Red that the majority of the infec-tions were ISPs providing home andsmall-business connectivity. Within 10hours, Code Red had infected 360,000hosts, indicating that there was no effec-tive patch response to the spreadinginfection. Additionally, Code Redremained inactive for 12 days and thenbecame active again. It was well knownthat the worm would reactivate onAugust 1, and so there was a lot of mediacoverage. Despite this, the majority ofpreviously infected machines were notpatched until August 2, after being rein-fected.

For users interested in building theirown network telescope, all that isrequired is a globally accessible networkaddress space that can be monitored.Suggested tools for analyzing the cap-tured data include FlowScan (for analyz-ing flows), CoralReef (for analyzingpackets), and AutoFocus (which analyzesboth flows and packets). The effective-ness of the network telescope willdepend largely on the amount of addressspace that can be monitored. The largerthe address space, the more traffic it willbe able to analyze. For example, a /8 net-work represents 1/256 of the Internet,but a /16 will only see 1/65536 of theInternet and so will have considerablyless chance of seeing any traffic that hasbeen randomly addressed.

Network telescopes, especially whendeployed across a large address space,can provide significant insight into non-local network events.

INTERNET GOVERNANCE RELOADED

Paul Vixie, Internet Software ConsortiumSummarized by der.hans[Note: Due to the fires in Southern Cali-fornia, Paul Vixie was unable to attendLISA ’03, so kc claffy substituted for himon short notice and used his slides.]

kc explained that governance is neededfor such shared resources as IP addresses,domain names, AS numbers, and proto-col numbers. Governance means thatthose who are affected by a decision getto help make that decision. Stakeholdersare those who hold/own/use/control theresources and those who allocate theresources.

The first example of shared resources kcmentioned is global routable IP. Demandappears to be higher than scale allows.ARIN/RIPE/APNIC/LACNIC are con-stantly searching for an equilibriumbetween routing table size and mini-mum allocation size.

The next example was Verisign’s typo-squatting with SiteFinder. While the talkwasn’t specifically about Verisign,SiteFinder became the primary topic,with lots of input from the audience.

Verisign doesn’t see itself as the stewardof public resources; it sees itself as theowner of those public resources. Unfor-tunately, the contract with Verisignapparently doesn’t specify which view iscorrect. Both kc and Vixie were in Wash-ington, D.C., for the first ICANN secu-rity meeting about the Verisigntyposquatting. kc pointed out thatICANN responded with impressivespeed and integrity with regard toVerisign’s typosquatting, which wasturned off 19 days after Verisign insti-tuted it.

Responding to customer requests, ISCcreated a patch for BIND9 to blockSiteFinder. China opted out of Site-

kc claffy

Finder by null-routing Verisign’s IP forSiteFinder. kc described SiteFinder, ISC’sBIND9 patches, and China’s blocking ofSiteFinder as examples of cyberneticwarlordism.

Several times, kc suggested gettinginvolved, emphasizing how close we areto the action. This is Internet policybeing made right before our eyes, andwe can participate. She reminded every-one to be courteous, mature, and profes-sional. We can help make the rules.

Vixie says SiteFinder’s losers are regis-trars, domain registrants, spam victims,Web surfers, other typosquatters, usersof non-Web protocols, and the Internetgovernance trust model. He challengesVerisign to provide diverse and specificexamples of entities other than Verisignthat benefit from SiteFinder.

Vixie predicts lawsuits and countersuitsbefore the SiteFinder and stewardshipvs. ownership issues are resolved.

Many members of the audience men-tioned that the governance organiza-tions need to be non-national andspecifically non-USA.

Resources:

http://www.icann.org/tlds/agreements/verisign/

http://www.icann.org/announcements/announcement-17sep03.htm

http://www.icann.org/correspondence/twomey-to-tonkin-20oct03.pdf

http://secsac.icann.org/http://www.icannwatch.org/http://www.isoc.org/http://www.ntia.doc.gov/http://www.stanford.edu/class/ee380/

Abstracts/031001.html

HIGH RISK INFORMATION: SAFE HANDLINGFOR SYSTEM ADMINISTRATORS

Lance Hayden, Advanced Services forNetwork Security (ASNS)Summarized by Jason RouseLance Hayden began by explaining thatmost information, if viewed in theproper context, could be damaging and,


l

CO

NFE

REN

CE

REP

ORT

Stherefore, high risk. Examples of suchinformation could be names, addresses,credit card numbers, or phone numbers.Since system administrators are oftentasked with securing and maintainingsystems on which this data is stored,Hayden believes that it is in the bestinterest of system administrators tomake themselves aware of the ongoingwork in regulatory legislation and prac-tices.

Hayden gave an excellent overview ofcurrent and future legislation and inter-pretations, focusing on their impact onsystem administrators. He produced aworld map, showing the increase in dataprivacy legislation across the globe, andthen outlined a six-step iterative processto enable system administrators to edu-cate themselves about the high-riskinformation they might handle and theninventory and build a strategy for deal-ing with that information. Review andalignment of IT with core business goalsis a key factor in this process.

Summing up, Hayden introduced the“true” OSI model – one where the“financial” and “political” layers heapupon the application layer. In this envi-ronment, Hayden argues, system admin-istrators must be aware not only of theirplace in the legal and social infrastruc-ture but of their potential liability andmethods to mitigate this risk.

PANEL: MYTH OR REALITY: STUDIESOF SYSTEM ADMINISTRATORSModerators: Jeff R. Allen, Tellme Net-works, Inc.; Eser Kandogan, IBMResearchPanelists: Nancy Mann, Sun Microsys-tems; Paul Maglio, IBM Research;Kristyn Greenwood, Oracle; CynthiaDuVal, IBM SoftwareSummarized by Kevin SullivanThis session assembled three researchersfrom major corporations, each of whomstudies the actions and responsibilitiesof system administrators. For some itwas surprising to learn that there is a lot

of research devoted to usability withinthe system administration community.It was quickly suggested that “systemadministration is a misunderstood pro-fession, both from inside and out.” Thesession focussed on how usability expertscan study what system administratorsdo, and how system administrators canemploy usability research tools to improvehow they do their jobs.

The panel suggested that there are fouraspects to system administration: psy-chological, technological, cognitive, andsocial. These aspects can be studied invarious ways, including diaries, lab stud-ies, questionnaires, and observation.

Kristyn Greenwood discussed how sheconducts usability studies known as“DBAs in the Wild.” This was a natura-listic observation of DBAs and SAswhere the researchers recorded everyaction of the user. The primary aim wasto provide this information to productdevelopment teams so that they couldimprove their products based on thefeedback from these sessions. Interest-ingly, Kristyn found that SAs spent 18%of their time on group coordinationcompared to 27% on actual trou-bleshooting.

Paul Maglio spoke on his study of inter-nal Web administrators at IBM. Hisfocus was on the methods of communi-cation used in problem solving, namely,phone or instant messaging. Paul alsonoted that large portions of time arespent on collaboration and communica-tion. He suggested that tool develop-ment focus on collaborating andallowing the user to shift effortlesslybetween systems. A particularly insight-ful comment was that command lineinterfaces do not provide the situationalawareness that is important to manycomplex tasks.

Nancy Mann spoke about her study,“Who Manages Sun Systems?” Thisstudy aimed to develop a profile of a sys-tem administrator, including experience,tasks, goals, motivators, and tools. Infor-

LISA ’03 l

mation gathered in the process will alsobe provided to software design teams toimprove the overall experience for sys-tem administrators.

It is quite apparent that system adminis-trators need well-designed tools just asmuch as novice users. This panel showedthat there are people devoted to improv-ing the computing experience for alltypes of users. Usability as it applies tosystem administrators is very different,but just as important.

SPAM MINI-SYMPOSIUM Summarized by Steve WormleyThe first part of the LISA ’03 SpamMini-Symposium consisted of two pre-sentations.

EMERGING SPAM-FIGHTING TECHNIQUES

Robert Haskins, Computer Net Worksand Rob Kolstad, SAGEThe authors started with a quick surveyof the audience which found that mostreceive over 30 spam messages per day.The first point mentioned was that oneof the problems with spam is the defini-tion. The end users know spam whenthey see it, the ISP knows it uses resources,and the spammer knows it makes themmoney. Yet, spam is hard to define. Asecond problem is that bulk email ischeap for the sender. Of course, thespammers say “Just hit delete,” but we allknow it’s not that easy for the recipient.Bandwidth costs continue to increaseand the consumer bears the cost of theemail. For one example, Rob Kolstadapparently receives 400 spam messagesper day.

One interesting point that was made isthat spam is fraud. Spam has misleadingsubject lines and advertises fraudulentproducts. Also, opt-out in spam isn’t away to escape, and opt-in is a joke. Andfinally, spam almost always hides its sitesand sources. More spam problemsinclude that spam is hard to winnow, itoverloads mailboxes, and the messagesthemselves are annoying. And sending


spam is easy there are fairly low barriersto entry.

The presenters believed that most spamis already covered by existing laws: fraudis already covered, as is trespass. Newlaws for other email will be expensiveand difficult to pursue. In addition, theissues of free versus regulated speechversus privacy will be difficult to balancegoing forward. And the root of the issueis that spammers spam because people

buy stuff from spam: at least one surveysaid 7% of recipients have ordered fromunsolicited e-mail.

How spammers are still sending mailvaries. There are still open relays spam-mers can use. More these days are alsohijacking PCs to send their spam. Someservice providers also allow spam via“pink contracts,” allowing them to avoidtypical terms of service. The presentersmentioned that even the smallest serviceproviders should be able to block mostoutgoing spam should they choose to.

Spam turns out to be an arms race.Spam is not easy to stop because mostspam comes from forged sources,hijacked systems, drive-by spammingfrom wireless, gypsy accounts (set up,spam, and leave), and the content (whatthe spam points to) is often not trace-able.

The practical solutions consist of edu-cation, technical solutions, legal solu-tions, or social solutions. Education issuch things as getting people to shutdown open relays, which is often anissue in developing countries, and hav-ing people secure their home PCs. One

of the better legal, social, and economicmethods is to enforce existing laws.

On the technical side, it is fairly easy tohandle outbound spam: simply requireauthentication of the user sending themail. The inbound side of spam iswhere the problem is. The first recom-mendation is to replace RFC 822. Otherideas are things like blacklists, whitelists,distributed collaborative filters, onetimeor limited-use addresses, challengeresponse, forcing the sender to computesomething, filtering services, scoring andrating products(SpamAssassin), enter-prise plug-ins, and Bayesian filtering.Bayesian filtering uses probability theoryto perform its spam checks; CRM 114looks at 16 observations for each wordand works fairly well. Blacklists are goodfor providers. Reporting spam is impor-tant so that things can get fixed wherepossible.

ADAPTIVE FILTERING: ONE YEAR ON

John Graham-Cumming, ActiveStateJohn’s presentation emphasized the factthat the best way to control spam was toincrease barriers to entry. One way to dothis is with filtering. Products such asPOP file use adaptive filtering to gaugethe level of spamminess of an email.

One of the reasons spam filtering is a bigissue is the “Grandma Problem”: nowthat Grandma is starting to get spams,filtering them is becoming more impor-tant. Many filters exist today both inopen source and commercial products.John expects that by 2004 every mailclient will have adaptive filtering.

The primary adaptive filtering issues arethe man-in-the-street usability issues,false positives, overtraining, onemanspam, and internationalization. Thingssuch as integration into the mail client,auto whitelisting, and the filter guardingagainst false positives help. However,overtraining needs to be handled by theuser, who may click the “spam” buttonon far too many messages, causing thesystem to think everything is spam. For

The Spam Mini-Symposium

internationalization the filter systemneeds to understand how languageswork and how punctuation and tok-enization should be handled.

Most spammers are trying to overwhelmfilters with good words which are thenhidden using various HTML tricks suchas comments and invisible ink. As thearms race progresses, the spammers trymore things, and the anti-spammerssometimes get more fingerprints. Thequestion is, do filters make spam moreeffective, since at least one spammer hasclaimed that filters helped him by reduc-ing complaints.

PANEL DISCUSSION: CURRENT BEST

PRACTICES AND FORTHCOMING ADVANCES

Part 2 of the Symposium was moder-ated by Dan Klein, with the presentersfrom the first spam session and threeadditional participants.

First there was a brief presentation byKen Schneider of Brightmail. Brightmailprovides a spam filtering package withservice and products. They estimate thatover 50% of email is spam now. Themajority of the spam messages advertiseproducts, and another large category ofspam is adult advertisement. Brightmailuses a set of decoy accounts on clientsystems to collect spam, which theiroperations center then classifies, andthey creates rules which are sent back tothe clients.

Other panel members were LauraAtkins, president of the Spamcon Foun-dation, which is working to keep mailusable, reduce false positives, assist withlegal fees for anti-spammers and filesuits against spammers; and DanielQuinland, the author of SpamAssassin.SpamAssassin is an open source productwhich uses anything that works to stopspam. He also encouraged everyone toimplement SPF, at http://spf.pobox.com/.

Who writes the software for spammers?The general consensus was that it wascommercial organizations, some soft-


l

CO

NFE

REN

CE

REP

ORT

Sware often shipped with anti-spam soft-ware to test the spam before it’s sent.

One of the more contentious issueswhich came up in the round table wasthe issue of challenge response. The con-sensus from the panel was that none ofthem thought it was a good idea. Someof the issues included fake challengesfrom spammers, spammers faking aknown good address, spammers using asweatshop to accept all the challenges,and the general annoyance to peoplewho send you email for legitimate rea-sons.

The panel then was asked about block-ing customers with viruses by ISPs. Theyfelt it was useful for customer ISPs butnot necessarily co-location facilities.There was also some concern that itcould affect the common carrier statusof an ISP.

How do people handle users who reportspam that is actually requested email?Brightmail in this case requires a mini-mum threshold for something to beclassified as spam.

What about the spam program writers?Apparently in many cases the programsare legitimate bulk mail tools for variouscompanies. Rob Kolstad pointed outthat programmers cannot be responsiblefor content.

Is spam legislation needed? Rob Kolstadfelt that the main things was that spam-mers should not be able to say what theyare doing is legal. Laura Atkins respondedthat the DMA (Direct Marketing Associ-ation) is in the pockets of the people onCapitol Hill. The DMA does not wantopt-in for email. They also don’t wantthis to become the requirement forfuture marketing.

COPING WITH THE DISAPPEARANCE OFNETWORK BOUNDARIES

Peyton Engel, BerbeeSummarized by Jason RousePeyton Engel highlighted the advance-ment of technologies such as VPNs, dis-

tributed computing, and load-balancingboxes and how the introduction of thesetechnologies has blurred the boundariesof traditional IT roles and networkdemarcation points.

When using these technologies, one hasto ask questions about liability and duediligence. If a distributed computingcluster is compromised and is used toscan or compromise other networks,who is responsible? Since VPN technol-ogy effectively extends network bound-aries to arbitrary limits, how do wehandle cybersecurity threats in this newenvironment? This, Engel argues, is theworld into which we will be heading inthe coming months.

As organizations begin to incorporatethese new technologies, Engel believesthat security is frequently overlooked, orexisting security solutions are trusted tooperate in environments for which theywere never designed. Engel dealt withthese questions and more, citing theneed for competent, well-rounded secu-rity practitioners and the defense-in-depth strategy of multi-level, multi-vector infrastructure and employee pro-tection. Engel also noted the growingfluidity of administrative domains, forexample merging two corporate net-works.

Engel believes that this new environ-ment will provide both challenges andinsights into tomorrow’s best practices,and that these issues will become thegroundwork for system, network, andsecurity administrator approaches in thecoming years.

SECURITY VS. SCIENCE: CHANGING THESECURITY CULTURE OF A NATIONAL LAB

Rémy Evard, Argonne National Laboratory Summarized by Carrie GatesRémy Evard gave a presentation onchanging the culture of a research sci-ence lab to incorporate secure practices.Such a change in culture requires severalstages, starting with reaction mode and

LISA ’03 l

then moving through project mode andinstitutionalize mode before achievingan ongoing program.

The reaction mode, in which they started,consisted of a climate where there wereno policies or support for security. Forexample, there were no policies restrict-ing the use of cleartext passwords. Theresult was a number of intrusions, andpoor results from security auditors. Theproblem was the culture – the belief wasthat effective security would keep usersfrom being able to do what they wantedto do, and so there was no support forsecurity, which translated into no fund-ing and no direction.

The catalyst for change, causing them toenter the project mode, was a new direc-tor who took security more seriouslyand asked for an internal report. Thereport’s recommendation was for thedevelopment of a security policy com-mittee. This committee was formed withthe goal of fixing everything (!), fol-lowed by passing another audit. A keypart of attaining this goal was the devel-opment of policies. And a key part ofdrafting acceptable policies was holdinggeneral discussions of the policy in townhall meetings with the entire lab. Thishelped to alleviate the fear that peoplewould not be able to perform theirwork, and helped to create the buy-inrequired to have the policies work. Bythe end of this stage, an internal riskassessment had been performed, ongo-ing internal scanning for vulnerabilitieswas being performed, and firewalls hadbeen deployed.

There was a gradual move into the insti-tutional mode after this. Here the goalswere to reduce the effort required toachieve effective security (while stillkeeping up the energy for it) and to pre-pare for the next audit. The technicalactivities consisted of improving bothconsistency and integration and deploy-ing practical solutions. During thisstage, an intrusion detection system wasalso deployed, which has been found to


be useful for detecting large-scale scansand viruses. By the end of this stage, theauditors returned and performed both amanagement review and a technicalreview. The resulting grade: “effective”(A).

There were three points Evard felt werekey factors in their success in deployingappropriate security policies and infra-structures. The first was that the highestlevel of management “got it,” and thatthey bought into the process and thenecessity of having security. The secondwas that audits work and provide valu-able motivation and feedback. The thirdfactor was that everyone helped andbecame involved.

TALKING TO THE WALLS (AGAIN)Mark Burgess, Oslo University CollegeSummarized by Siddharth AggarwalMark Burgess discussed the evolution ofpervasive computing and the challengesit could pose to system administrators inthe years to come.

He introduced the topic by looking atsmart houses and smart cities, whichwill make extensive use of pervasivecomputing in the future. According toBurgess, pervasive computing brings upnew challenges for a system administra-tor because of the diversity of devicesthat have to be managed, coupled withthe high density of communication.Because of limited consumer demand,the slow introduction of these deviceswill tend toward a non-standardized,heterogeneous computing environment.This also leads to a lot of security issues.

Burgess grouped the challenges posed bypervasive computing into three cate-gories: diversity, stability, and sociologyof interaction. When implementing per-vasive computing, a key decision to bemade is who should control the system.Who decides the policies and controlsthe resources? This leads to anotherquestion: Should humans and comput-ers cooperate with each other or com-pete against one another? Should a

device adapt to the environment, orshould the environment adapt to thedevice when it comes into a system?Burgess discussed various techniques,such as game theory, for modeling inter-action between such systems.

Burgess finished by introducing modernconcepts like the pull model of commu-nication between systems having anemergent behavior, human-computerswarms, and pseudo-hierarchical socialswarms. The emphasis is on systemshaving probable control, probable risk,and probable behavior rather thanabsolute control. He concluded by say-ing that the world is controlling us asmuch as we are controlling it. The chal-lenge lies with system administrators tofind stable points for equilibrium.

THROUGH THE LENS GEEKLY: HOWSYSADMINS ARE PORTRAYED IN POP CULTURE

David N. Blank-Edelman, NortheasternUniversitySummarized by Ari PollackDavid Blank-Edelman presented ahighly entertaining talk on the portray-als of sysadmins in US popular culture.In the minds of the public, sysadminstypically get lumped into a broader“computer person” category along withprogrammers and hackers/crackers, sothe examples in this talk included bothsysadmin and sysadmin-related charac-ters, mostly from the movies. Davidnoted that portrayals of sysadmins brokedown into three polarities: “competentor incompetent,”“good or evil,” or “hipor really uncool.” Examples were shownof each, much to the amusement of thecrowd.

After this demonstration, David sug-gested that these portrayals are closelytied to the public’s views on computingand technology in general (e.g., people’sviews of computers as being totally com-petent or incompetent get projectedonto sysadmins). Given that peopleaccept the stereotypes they see in popu-lar culture when they interact withsysadmins on a daily basis, David ended

with tips on ways to respond to thesestereotypes in the workplace.

HOW TO GET YOUR PAPERSACCEPTED AT LISATom Limoncelli, Lumeta Corporation;Adam Moskowitz, Menlo ConsultingSummarized by Carrie GatesLimoncelli and Moskowitz based theirtalk on their experiences as programcommittee paper referees. Their firstadvice to potential authors was to readand follow the instructions on the callfor papers.

The paper submission process for LISAconsists first of submitting an extendedabstract (not a full paper) and a paperoutline. An “extended abstract” is a shortversion of the full paper, consisting ofabout 4–5 pages (not 4–5 paragraphs!).It should not be a teaser but, rather,should provide enough details to allowthe committee to make a decision, with-out providing details of required back-ground knowledge.

Abstracts are then reviewed by the com-mittee members. Each paper is assignedto 4 or 5 readers, who rank the paper ona scale of 1 to 5 in various categories,such as the quality of writing and appro-priateness to the conference. The com-mittee meets as a whole and reviews therankings of the various papers, acceptingthe papers with obviously high scores,and rejecting papers with obviously lowscores. The committee then reviews eachof the remaining papers until a finalprogram has been designed.

The three main criteria for getting apaper accepted at LISA are:

1. Is the work worthwhile? (For workthat is publishable but not appro-priate for LISA, the reviewers willsuggest other forums for publica-tion.)

2. Has it been done before?3. Can the author write well?

What makes a good paper? First, thepotential author should note that the


l

CO

NFE

REN

CE

REP

ORT

Spurpose of the refereed-papers track atLISA is to advance the state of the art insystem administration. Otherwise goodpapers might be rejected if they do notmeet this criterion. Alternatively, anauthor can be asked to give an invitedtalk instead (ITs tend to be on hot topicsor by cool people). The author shouldrecognize that the audience is highlytechnical and write for this audience. Ifthere is any confusion about the level atwhich a paper should be written, reviewthe papers that have been published atprevious LISA conferences (available onthe USENIX Web site).

In terms of style, the author shouldintroduce the topic immediately, andthen proceed to explain the terms orprocess or arguments. This allows thereader to know immediately what thepaper is about, rather than needing toread several paragraphs before findingthe actual topic. Also, the author shouldexplain why the work is original, show-ing how his or her work is different from(or, hopefully, better than!) work thatothers have done in the same area. (Allauthors should list their references intheir extended abstracts – this is a petpeeve of some of the program commit-tee.)

In summary, a good paper is clearlywritten, concise, relevant to LISA, andadvances the current knowledge in thearea of system administration. It clearlyshows the data, methodology, andresults, and it discusses related work,showing how the current approach isdifferent from or better than previousapproaches.

SECURITY LESSONS FROM “BEST IN CLASS”ORGANIZATIONS

Gene Kim, Tripwire, Inc.Summarized by Carrie GatesGene Kim gave a presentation on someresearch he has been doing on the secu-rity practices of “best-in-class” organiza-tions, such as Verisign and the New YorkStock Exchange. His goal is to determinethe characteristics of a best-in-class organ-

ization, and how these can be achievedin other organizations.

Best-in-class operations and securityorganizations can be recognized by fourcriteria. First, they have the highestserver to system administrator ratio,often with 100+ servers per administra-tor. The second characteristic is that theyhave the lowest mean time to repair, aswell as the highest mean time betweenfailures. The final characteristic is theydemonstrate the earliest integration ofsecurity into operations (when com-pared with other organizations).

Many of the problems encountered byorganizations today are created by peo-ple. For example, the IT departmentoften does not know about changes thathave been made by the security depart-ment. This results in an adversarial rela-tionship between security and operationsinstead of a close working relationship.To further complicate matters, manydownsized companies have developersinstead of administrators maintainingproduction servers. Finally, documenta-tion is often not performed, resulting inonly a couple of people in the entireorganization who know how thingsreally work.

This situation affects how work is per-formed, resulting in constant firefightingrather than proactive server man-agement. This further results in situa-tions where no two servers are the same,complicating the system administrationpractice.

By comparison, best-in-class organiza-tions have controls embedded in secu-rity and operations to manage change.These organizations have identified whatthey consider to be the key issues (e.g.,outages with a long remediation time,inconsistent system footprints in 1000+servers running critical business pro-cesses), and have developed approachesto controlling these issues (e.g., integrityscans every 10 minutes for business con-tinuity, regular audits to determine

LISA ’03 l

whether system footprints across serversare identical).

The main observations are that best-in-class organizations have developed prac-tices that make it easy to understand,know, and recover to good states in thesystem. Additionally, they have devel-oped proper processes and proceduresfor managing change, rather than takingan ad hoc, firefighting approach to theprocess.

WHAT WASHINGTON STILL DOESN’T GET

Declan McCullagh, CNET News.comSummarized by William ReadingWhy do we need Washington? They pro-vide national defense and handle foreignaffairs and interstate commerce, amongother things.

However, Washington also wants to reg-ulate where it is actually difficult or impos-sible to do so without a number of verynegative implications.

Although it was struck down, the Com-munications Decency Act was one ofCongress’s first attempts at online cen-sorship. It banned “indecent” or “patentlyoffensive” words. As former Sen. JamesExon (D-Neb) said, “This is the time toput some restrictions or guidelines onit.”

Washington politicians, Bill Clintonamong them, also suggested having asort of “V-Chip” for Internet access.

Al Gore, who still claims that he “tookinitiative in creating the Internet,” sup-ported an equivalent to the “Clipperchip” for computer networks.

Some politicians do not even realize thatsome legislation is simply impossible,having indicated that they do not sup-port bills such as “602P,” which was ahoax that claimed the U.S. Postal Servicewould begin to charge for email.

The “Office of Cybersecurity” does notseem to gauge threats very well, withcybersecurity advisor to the White


House Richard Clarke resigning over theSapphire worm.

Rep. Howard Berman (D-Cal) proposedthat “a copyright owner shall not beliable in any criminal or civil action fordisabling, interfering with, blocking,diverting, or otherwise impairing theunauthorized distribution, display,performance, or reproduction of his orher copyrighted work on a publiclyaccessible peer-to-peer file trading net-work” (http://thomas.loc.gov/cgi-bin/bdquery/z?d107:h.r.05211).

Others advocate destroying computers:“If we can find some way to do thiswithout destroying their machines, we’dbe interested in hearing about that,” Sen.Orrin Hatch (R-Utah) said. “If that’s theonly way, then I’m all for destroyingtheir machines. If you have a few hun-dred thousand of those, I think peoplewould realize [the seriousness of theiractions]. There’s no excuse for anyoneviolating copyright laws,” Hatch said.

STICK, RUDDER, AND KEYBOARD: HOWFLYING MY AIRPLANE MAKES ME A BETTERSYSADMIN

Ross Oliver, Tech Mavens, Inc.Summarized by Robert W. GillRoss Oliver has been a sysadmin for 15years and a pilot for 13. He has loggedover 500 flying hours and is almostinstrument rated. His invited talkfocused on the lessons sysadmins cantake from aviation. The talk was relaxed,fun, and chockfull of useful ideas tomake the lives of sysadmins easier.

Despite new laws like HIPAA, IT is stillvery unregulated. Ross presented nineareas in which he thinks IT and sysad-mins can learn from aviation. Brieflysummarized, his points were:

1. Make use of checklists. Use them asmemory aids and as tools to avoid mis-steps. Checklists allow you to standard-ize tasks for multiple actors and can beused as a training tool.

2. Prepare for abnormal procedures.Anticipate what things can go wrongand prepare how to deal with thembefore there is a problem. Drilling isimportant to ensure that the steps you’veworked out are correct and to provideconfidence when you need to use theprocedures under fire.

3. Perform “pre-flight” planning. Plan-ning ahead reduces in-flight workloadand puts all variables on the table. Youwill save time and effort by making deci-sions in advance, adhering to a checklistformat, and allowing for peer review.

4. Know how things work. A checklistwill not cover everything, and instru-ments can lie. By understanding theunderlying technology, sysadmins canbetter cope with situations that fall out-side normal operations.

5. Learn to assess risk. Understand yourown biases so that they don’t distortyour viewpoint.

6. Identify chains of errors, in which sev-eral different factors combine to causean accident. Aviation has, for the mostpart, routed out most single-cause fail-ures; instead, crashes often result from aseries of missteps. Such tragedies oftenoccur after signs of a low-level problemhave been ignored.

7. Deal with crew resource management.Command and control structures are, attimes, too rigid for the environment.Sysadmins are often soloists, accustomedto working at their own pace. Each groupneeds to find the right amount of struc-ture (checklists, peer review, etc.).

8. Work toward continuous improve-ment. Strive to find little things you cando to make things better. Learn fromother industries (such as aviation withits 100 years of experience).

9. Beware automation. Automation isbest applied to frequently utilized andwell-understood functions, but is worstsuited to exception handling, since it is

difficult to account for all the possibleexceptions.

As technology becomes more involvedin public safety, the risks become greater.Ross’s talk offered excellent examples ofhow these steps have helped the aviationindustry improve its safety record andhow they can be applied to the work ofsysadmins.

SECURITY WITHOUT FIREWALLS

Abe Singer, San Diego SupercomputerCenterSummarized by Ari PollackAbe Singer presented a look at why fire-walls are so popular these days, why theyshould be used, and why they don’t needto be used. A common misconceptionamong technical and non-technical peo-ple alike is that you’re not secure unlessyou have a firewall. Firewall vendorswant to make you think installation willsolve all your problems; in reality, fire-walls fail all the time, and they dorequire a great deal of effort to be con-figured properly. Misconfigured firewallscan inhibit real productivity and donothing to enhance security. Addition-ally, there are no data or statistics aboutthe effectiveness of firewalls.

The SDSC currently takes many securityprecautions to ensure that their systemswill be secure against an attack, evenwithout a firewall. Some of these pre-cautions, such as using restricted sudoor patching early and often, may becommonplace in many organizations,but they provide an added level of secu-rity nonetheless and have little to noimpact on day-to-day usability. Inexpe-rienced users may do things by accident,and in many cases they do not careabout security; they just want to do theirwork, and will try to get around defensesthat make it harder for them to performtheir job.

There is a place for firewalls, but theymay not be worth the effort for all net-works. In some cases, 95 to 100% of thesecurity effort at an organization is


l

CO

NFE

REN

CE

REP

ORT

Sspent on firewalls. In reality, this shouldbe closer to 5%. Firewalls can be usefulfor hosts that can’t be secured on theirown, such as printers or embeddeddevices, and they can give an extra layerof protection, but firewalls should notbe used as the only line of defense.

WORKSHOP SERIES

AFSEsther Filderman, The OpenAFS Project;Garry Zacheiss, MIT; and Derrick Bras-hear, CMUThe AFS workshop covered many topics:Open AFS roadmap, Kerberos integra-tion, IBM’s Stonehenge project, APIs,and other AFS workshops.

Derrick Brashear presented the OpenAFS roadmap:

n 1.3 coming soon.n MacOS 10.3 support now (on Ope-

nAFS 1.2.10a).n large file support “coming soon”

(actually available, but only limitedtesting has been done).

n FreeBSD and OpenBSD ports arecoming along nicely.

n Linux 2.6 kernel is problematic withrespect to the interface used byPAGs (Process AuthenticationGroups). IBM Germany and SUSEhave been working together someon this as well.

The second theme was managing Ker-beros: MIT vs. Heimdal vs. OpenAFS (orArla or Transarc AFS). The consensus isthat most common configuration ques-tions and issues have solutions, and thatthose interested should consult the AFSWiki, as well as the OpenAFS mailinglist archives.

Next, we heard what IBM has beendoing with the Stonehenge project. In anutshell, the Stonehenge project is aboutputting together a turnkey storage man-agement system that uses AFS as its net-worked filesystem layer. IBM has beendeveloping the management interfacesand has released a Java API so that oth-

ers can build management tools for AFSas well.

Alf Wachsmann and Venkata Achantadiscussed the Perl API they have beenworking on under the direction of Nor-bert Gruner (which utilizes XS). TheAPI now containts vos and vldb inter-faces, so volume management programscan be written as well. For those inter-ested in a different Perl API, Phil Moorehas released his to CPAN. The primarydifference in the two APIs is that Phil’sforks off shell calls to the underlyingcommands, while Norbert’s uses XS andsaves the overhead of the fork/exec.Phil’s API is more complete, however.

Wolfgang Friebel gave an update on theGerman AFS workshop that took placeOctober 7–10, 2003. Alf Wachsmannand Randy Melen announced an AFSBest Practices workshop, to be hosted byStanford Linear Accelerator Center onMarch 24–26, 2004.

SYSADMIN EDUCATION

Curt Freeland, University of NotreDame; and John Sechrest, Peak InternetServicesThis workshop featured discussions ofcore topics for system administrationeducation programs, a roundtable pres-entation of participants’ courses, anddiscussions of future work in the area.

Participants assembled a list of core top-ics in system administration and dis-cussed how this hypothetical listcompared to the actual syllabi variousprograms offer. A consensus is that asingle system administration course isnot enough, and that programs need tobe more comprehensive. Various issuesand strategies for encouraging schoolsand departments to offer system admin-istration courses were discussed.

Curt Freeland and John Sechrest haveassembled a list of universities that offercourses and programs in system admin-istration. While there are many suchcourses and programs, of particular noteare two new programs in Europe:

LISA ’03 l

Netherlands Master in System and Net-work Engineering (http://www.os3.nl/).

Master’s degree in Network and SystemAdministration at Oslo University College (http://www.iu.hio.no/data/msc.html)

These two are of special note as theylead to Master’s degrees and are not sim-ply standalone courses.

Work on the theoretical foundations ofsystem administration is advancing ascan be seen in this year’s SAGE Achieve-ment awards. Participants discussedsome ways to help students join withfaculty to do further research in systemadministration.

ADVANCED TOPICS

Moderators: Adam Moskowitz, MenloComputing; Rob Kolstad, SAGETranscribed and summarized by JoshSimon and Rob Kolstad This meeting included experimental useof IRC as a backchannel for interper-sonal communications to keep the inter-ruptions down. It led to a few interest-ingly surreal conversations and mixedevaluations.

The meeting led off with introductionsand then the opening question: What’sthe most difficult challenge you haveright now? Or, What do you wish youhad to address challenges? Repliesincluded: Overcoming cultural andpolitical resistance to centralized systemadministration. Sales is a problem. Somehave succeeded (with templates and thelike). One participant said: “I can sell it.Only takes a 1–2 hour presentation tosell management . . . which is 50% of theproblem. Technical dudes MUST buyin!” Standard builds were advocated.

Linux was said to be a hard sell but usedanyway due to its affordability – lots ofmachines coming in under the radar.

Someone noted that heterogeneous clus-ter participants seem willing give upsome autonomy for functionality and its


darker side: “If you drive people to out-source their S.A., you’re screwed again.”

More draconian measures included the“network citizenship” notion: “We justunplug machines that aren’t in confor-mance with our standards.” Anotherparticipant disables ports when virusesare discovered.

But “Technical dictatorships don’t oftenwork well enough. Standardization isgood; innovation is good. There must becollaboration and accommodation.[Sharing] the goals helps.”

The next discussion is shown in fairlydeep detail in order to convey a sense ofthe workshop’s ebb and flow. It has beendramatically condensed even in thislengthy summary.

Cash flow was one participant’s #1 prob-lem. “We don’t have good structures fordoing things like collaborative adminis-tration, charge-backs, funny money(between departments). Industry-wise:Administrative toolsets that we havedon’t support sysadmins well enough(we end up using sneakernet, telephone,etc.). How do we create for the serviceindustry something like financial instru-ments in the financial community? We’llwant data-feeds between/among ourtoolsets. Consider carrying around a lit-tle micro-charging header on servicesbeing rendered (e.g., a virus elimina-tion). Millions of small businesses needthis!”

Discussion ensued: “Granularizing thesetasks hurts innovation. We are pureoverhead.”

“We’re on the tail end of the stick andget our budgets cut first when thingsaren’t great.”

“Of course, being a profit center doesn’thelp that much – everyone else is just asmessed up as we are.”

“. . . and this leads to bad local optimiza-tions.”

“People think they want detailed sum-maries of IT costs, but then they balkand refuse to buy certain services/prod-ucts. Bad global impact.”

“You must be in a very large companyfor market forces to work effectivelyamong divisions – otherwise you don’thave the proper efficiencies of scale.”

“It’s good to know costs. Sometimes,though, this perverts the problem solu-tion technique by pushing costs around.Monetary values on various servicessometimes thwart corporate missions.”

“People buy bandwidth, CPU, disk andwant to own it ‘forever’. They want topay once. They prefer to think of havinga computer, not the use of 100,000 CPUcycles to do an operation.”

“From whose point of view does onelook at costs [and value]? Customer, VP,Manager, CIO, CEO: different points ofview!”

“Don’t artificially granulate the cost. Ilike the all-you-can-eat approach. Tieredplans are fine, but try to avoid artificialsituations with costs over which youhave no control. Try to cost things sothat both sides of the arrangementarrive at mutual efficiency. I wonder ifmonthly billing is going to increase ourcustomers’ perceptions of us.”

“We should teach them what we’redoing! ‘I did a tuneup for you.’”

“Auto repair; you pay book rate,independent of how long it takes. Weneed to insert (deliberately) a noisy levelof suffering-causing failures so peopleunderstand what ‘good’ is.”

[General group muttering: It’s unethical.]

[Consider a] “popup [that] says ‘Net-work failed . . . we repaired it for you inbackground’”

“Valuation of services is the main prob-lem. Outsourcing has hidden costs (e.g.,cost of data access). ‘Flexibility’ is nevervaluated. Agility counts!”

“Must valuate the ‘cost’ or ‘value’ ofNOT doing something.”

“Management at our institution wanteddisaster recovery after a disaster, despiteour requests for years prior.”

Why can’t we describe ourselves/ourjob?

“J. Deming says, ‘You can’t fix [manage]what you can’t measure.’”

“We just got into metrics. We use RUM(‘resource utilization metric’): 10% oftime doing tickets, 10% training, etc.Management prefers this to ‘17 minutesto add a user’.”

“I am opposed to bad metrics and badcharges – these are worse than havingnothing at all. Metrics disincentize. [Forexample,] you promote or terminatepeople based on the number of ticketsclosed (thus punishing those who cansolve difficult problems).”

One group member told this story: “Itried to morph into an MBA; failed. I’mreally a consultant. I repeatedly encoun-tered a request for ‘a better way to dosystem administration’. Yet, lots oforganizations denied there was a prob-lem. I finally theorized: I think it’s ourfault. The knowledge we bubble up toour management is ‘good news’ intendedto make us look good. ‘We’re doing fine;everything is under control.’ Instead, weneed to send more details than the CIO/COO wants to hear. We need face-timewith management structure to makethem learn enough to understand thereal problems in their own infrastruc-ture. IT buttresses all people. We need tomake that clear!”

General discussion about actual use andsizes of LDAP scaling.

Challenge: Document management sys-tem. Anyone have any good solutions?

Xerox Docushare was mentioned repeat-edly. Webdav, twiki, Zope, and DCWork-flow were mentioned.

87February 2004 ;login: LISA ’03 l

l

CO

NFE

REN

CE

REP

ORT

SChallenge: How do we keep things fun(esp. for those with spouses and chil-dren)?

Comments included: “movie bucks,”general agreement, free coffee, thenotion that the job isn’t fun, the notionthat it shouldn’t be too much fun,engendering pride, development vs. fire-fighting, uninterrupted time for proj-ects, SWAT team assignments vs.development projects, project demos,fellowship, recognition, separating workfrom socialization/other-parts-of-life, involving others in pur-chases (e.g., peripherals), “developmentteams” to attack projects in a sprint, andtwo-way radios.

A short discussion of spam covered itsvolume and demoralization potential.Email size limits were discussed. Some-times email is the only way to share largefiles; this means that a new mechanismis required.

How does one evaluate value? Howmuch is RH10 really worth?

General discussion of integrationcosts/issues, pricing, etc., continued.One person noted: “There’s nothingRH’ish about this question. Senior levelsysadmin means understanding vendorspull the rug out at any time and we mustproactively deal with this. I don’t putcertain products in core services. Build-ing too many dependencies on some-thing you’re locked into can be bad.Recently Verisign changed their licensingterms to charge us a lot since we’re anunusual site. Plan for this! We use opensource when we can, open standards. Weneed to be agile.”

Complexity was raised as an issue: “Wesee increasing complexity: volume man-agers, grids, etc. etc. How do we keepthese different level of sysadmins cur-rent on these when we have 1,000machines, many of which change a lot?”

Comments included: the expense ofdiversity, the impossibility of having “allsenior sysadmins,” a disagreement about

that, and a list of different solutions forNAS, SAN, and other storage manage-ment.

One of the group had “a managementissue. I have a good fire-fighting sysad-min, short tasks, etc. This person wantsto do ‘more meaty’ things but can’t.”How to solve this?

Suggestions included: career counselingand a set of discussions about that,“spinning his own job to him,” encour-aging him to grow, his inability to recog-nize his own failure, training, theadrenaline and endorphins of firefight-ing, and a thought that maybe he shouldbe a firefighter/savior kinda of person.

What about lifecycle management forfiles? We get thousands of new files perday and we need to manage where theyreside, where the copies are, etc. Anyoneknow any software to do this?”

Suggestions included: Permabit andAlien Brain (though that is mostly in theaudio space).

One person had an interesting issue:“Availability is declining. I see threemanagement psychoses. First one: Everytime availability declines, they increase‘process’ to fix the problem. Currently,we have a 90-minute daily change man-agement meeting. They’re squeezing. #2:Ownership. They’re so afraid aboutsomeone dropping a problem, they cre-ate process to thwart moving the solu-tion to the best person for the job.Admins work on a per-machine basis,not on subsystems. Ownership is sticky– must stay on phone with people forhours to fix things. #3: We’re ‘xxx.com’,and we do things differently and no onecan teach us anything.”

Discussion included: hire a consultant(though the problem owner said thatthat would be impossible), a generalthrowing-up-of-hands that this problemwas unsolvable, the notion that ‘fear tofix problems’ is also part of the uptimeproblem in addition to ‘procedurizing,’‘philosophy of processes,’ be careful of

sensitivity to alarms, demonstration ofhow process hurts the metric, and admo-nitions to play by the (presumably defec-tive) rules until it’s clear they’re bad.

Finally the group made predictions for11/16/2004. A few of them were particu-larly interesting:

n Unemployment levels will still beabove 5% for the national average[100%]

n Context-aware services (those thatare location-dependent) will beginto be deployed (there’ll be some inmajor cities) [14/30]

n Sun’s market share will continue todecline [100%]

n Spam will force a sea change (dis-continuity) in either government, orbusiness, or both, such as major leg-islation or some major companydoing something really dramatic, orsomething [27/30]

n There’ll be a significant backlashagainst the RIAA in particular anddigital rights management in gen-eral, probably from a university orcollection of universities, with thepotential to completely change thelandscape [22/30]

n The SCO thing will still be going onand still nobody will give a $#!+[28/30]

n You will still not be able to usenative IPv6 end to end across theInternet in any useful way [28/30]

n No technical solution will stem thetide of spam on the Internet back-bone [100%]

n There still won’t be a widespreadmusic CD copy-protection system[100%]

n Most consumer PCs sold will nothave a floppy drive and will havewritable DVD drive [25/30]

n A Windows-basedvirus/worm/whatever will causewidespread data loss [26/30]

n SCO will lose the lawsuit [100%]


THE LISA GAME SHOWSummarized by Josh SimonThis year’s quiz show was more excitingthan in years past for a few reasons. OnMonday, Rob Kolstad’s laptop – theancient piece of crap with a brokenscreen – was stolen out of a lockedroom, which was supposedly guarded bysecurity as well. He didn’t have the mostcurrent version of the code or questionsbacked up to his home network. (Les-son: Back up your laptop frequently!) SoRob was more invisible than usual thisconference, rewriting the game showsoftware, writing new questions, choos-ing audio songs involving smoke andfire (because of the nearby wildfires andthe ashfall the first half of the week), andtrying not to go completely insane. Inaddition to the hardware and softwareissues, we’d changed the format slightly.We now had four rounds of four con-testants (involving 16 people) instead ofthe three rounds of three (nine people).Consensus after the fact was that it keptRob from spending time with the con-testants and in the banter that’s verypopular.

Things in the show itself were goingokay, modulo a “wrong answer” buzzereffect every time we exited a question togo back to the board, regardless of thecorrectness (or not) of the answer, untilfor no apparent reason the softwarecrashed just before the midpoint ofgame one. Luckily, we’d been keeping amanual transaction log at the judgingtable so we had it to recover from. Theshow resumed (after Rob did a code fixin real time with the main monitors offand Dan Klein did an improvisationalcomedy routine to keep folks enter-tained) only to have the buzzer systemfail spectacularly in the middle ofanother game. So Dan and Josh went tothe backup system of contestants raisingtheir hands. We had a couple ofinstances where the contestants didn’twait to be acknowleged and so thewrong person answered, but it didn’tseem to affect the final scoring much.

The first- and second-place finishers ineach round won one of the Linuxadapter kits for their Sony PlayStations;the third- and fourth-place contestantsin each round won a variety of booksfrom several publishers.

The final round (with the winners fromthe first four rounds) ended in a tie forfirst and second place, so we played atie-breaker catgeory. That caused us toend in a tie for second and third place,so we played another tie-breaker cate-gory. When all was said and done, wedeclared Ken Hornstein the winner, andhe walked away with his Linux adapterkit, a satellite photo of the smokeplumes from the San Diego fire (withthe Town & Country more or less cen-tered on the map), and an signed (byDan Klein) photo of the Sunday sun,with the visible sunspots. Final-roundwinners also received valuable cashprizes in the form of pictures of deadpresidents ($25 each for third andfourth place, $50 for second place, and$100 for the grand winner).

SAVE THE DATE!13th USENIX Security Symposium

August 9–13, 2004 u San Diego, California

The USENIX Security Symposium brings togetherresearchers, practitioners, system administrators, sys-tem programmers, and others interested in the latest

advances in security of computer systems.

–Steve Bellovin, AT&T Fellow, AT&T Labs Research; co-author of Firewalls and Internet Security: Repelling the

Wily Hacker (Addison-Wesley Professional, 2003)

http://www.usenix.org/sec04/

“This is the most important conference I go to.”

SAVE THE DATE!2004 USENIX AnnualTechnical Conference

June 27–July 2, 2004 u Boston, Massachusetts

http://www.usenix.org/usenix04/

NEW FO

RMAT!

The new-format Annual Tech ’04 will feature:u 2.5 days of General Sessions—original and inno-

vative papers about modern computing systemsu 2.5 days of FREENIX—a showcase for the latest

developments in and interesting applications offree and open source software

u 5 days of content from Special Interest Group Ses-sions, including UseLinux, Security, and more

u 6 days of training with up to 30 tutorial offeringsu Famous-name Plenary Sessions every dayu Special social events every eveningu Plus BoFs and Guru Is In Sessions

17th Large Installation Systems Administration Conference ......was distinguishing between the physical world and the software world but that a distinction made between mechanism and

Documents