Gary Kratkiewicz BBN Technologies Geoffrey Knauth BAE Systems

Taming Development and Data Chaos in Military Logistics:

The ICIS Experience With Lightweight Languages

Gary Kratkiewicz BBN Technologies

Geoffrey KnauthBAE Systems

LL2 - November 9, 2002 - Gary Kratkiewicz & Geoffrey Knauth

2

ICIS Background

• Web-based software system

• Analyzes future combat and humanitarian operations

• Measures ability of Defense Logistics Agency to satisfy wartime requirements for following items:• Fuel

• Food

• Replacement items (repair parts, clothing, medical items, etc.)

• Identifies potential problems• When will I run out of stock?

• What should I buy today for tomorrows needs?

• Do I have adequate sources & transportation?


3

Accomplishments

• Early Web-based DOD system

• Saved over $100 million • By computing reduced inventory needs for fuel and food

• Adopted as part of DOD doctrine

• Months of calculations reduced to hours• What-if capability leads to better decisions

• Short analyses expanded role to crisis action

• Used in military operations to:• Assess readiness

• Compute operational requirements


4

Development Notes

• Started in 1994

• Grew from small prototype to large system in 4 phases

• User interface moved to Web in stages from 1996 to 1999

• Approximately a third of code is in Lightweight Languages

• Significant data issues


5

Data, Data, Data

• Large amounts of data

• 800-1000 input files; 200 different formats

• Gigabytes of input and output per analysis

• Hundreds of analyses “active”

• Data management issues

• Received from many different agencies

• Regular and irregular updates

• Meandering formats

• Duplication between analyses: • Computations

• Intermediate & output files


6

1. Prototype ICIS – 1994/1995

• Original prototypes completely coded in Tcl/Tk

• Vague, rapidly changing, and conflicting requirements

• Less than 1 MB of data

• Ease of developing/tailoring lightweight prototypes greatly contributed to initial success


7

2. Production ICIS – 1995/1996

• Lots of data: approaching 1 GB

• Addition of (mostly) well-specified computational modules

• Real use with classified data in mid-1996

• Mix of languages:• Tcl/Tk for GUI and glue for module execution

• Perl for data preprocessing

• C/C++ for computational modules


8

3. Web ICIS – 1996-1999

• Address remote access issues

• Moved to Web in stages

• On server:• Tcl/Tk – glue for module execution

• Perl – data preprocessing

• C/C++ and Java – computational modules

• Perl and C++ – dynamic generation of HTML and JavaScript

• In browser:• HTML – static GUI elements

• JavaScript – form-based GUI elements

• Java AWT applets – interactive GUI elements


9

4. Redesigned Web ICIS – 2001/2002

• Improve GUI and better manage data• Data approaching 1 TB

• On server:• JScheme – managing data & module execution (sophisticated

“glue”)

• Perl – data preprocessing

• C/C++ and Java – computational modules

• In browser:• HTML – static GUI elements

• JavaScript (much less) – simple GUI elements

• Java AWT applets – more complex GUI elements

• Java Swing applets – most complex GUI elements

• JScheme for understanding scenario


10

Source Code Evolution

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

Prototype 1994 Production 1996 Some Web 1997 Full Web 1999 Redesign 2002

Lin

es o

f C

od

e

Tcl/Tk

C++

C

Perl

SQL

HTML

Java

Scheme


11

Summary of Lightweight Language Use in ICIS

• Rapid prototyping

• LL replaced by HL: completely for computations, partially for GUI

• Highly iterative, innovative design & development

• LL used in Data Set Manager, patent pending

• Processing of meandering data

• Gluing together heavyweight modules

• LL replaced with more sophisticated LL

• Building little languages

• Test code


12

JScheme — Application Flexibility

• Interpreter for debugging

• Scripts for component functionality

• Can be a server for execution management of multiple tasks

• Example implementation:

BBN’s Data Set Manager


13

User Asks ICIS Questions

• User interaction with ICIS is web-based

• User defines composite scenario

• User specifies results of interest

• ICIS runs the assessment

• Numerous modules might be run

• It might take a while…

• Results appear in browser

Example: “How much Fuel do I need?”


14

An Example Small Assessment


15

Problems

• Obvious approach of using shell scripts and files doesn’t cut it here

• Too much data

• Many computations recomputed needlessly


16

A Full ICIS AssessmentDoes Many Computations

The readable version of this graph takes up a large wall.


17

Data Set Manager

• Data is described by features, not file names

• A module spec describes

• Inputs and outputs of a module

• Process relationships with other modules

• Modules can be implemented in any language

• A module spec is applied to a composite scenario to produce an execution script

• The script is executed

• All computations are memoized


18

Execution Management Needs Dynamic Capabilities

Data Set Manager takes full advantage of Scheme


19

Implementing the Data Set Manager in JScheme was fairly easy because:

• Quasiquote was used to go from module spec to execution script

• A full language was required to write module specs

• Executing the script was just walking a tree structure


20

Murphy Strikes

Start

Goal

Mistrial!


21

MVC: JScheme = M+C, Java = V

Show DetailsTry Again

GenerateUser Options

Draw graph from knowledge of assessment state


22

Data Set Manager — Conclusion

• Core implementation is 3804 lines of JScheme code

• Supported by 287 lines of Perl, 102 lines of SQL

• ICIS Module specs add another 1853 lines of JScheme

• Current APIs for other languages are:

• 688 lines of C

• 496 lines of Perl

• 356 lines of Java (1 class)

• 120 lines of JScheme

• BBN has applied for two patents

Gary Kratkiewicz BBN Technologies Geoffrey Knauth BAE Systems

Documents