ns-3 Design Overview - Tom H · We will select between the the GNU build environment (autoconf, automake, libtool, gcc) and SCons. The default Windows build environment is Cygwin.

ns-3 Design Overview

Version 1.0 draft 1

ns-3 project

http://www.nsnam.org/

feedback: [email protected]

August 5, 2006

Introduction

This ns3 design document is intended to document the technical goals, software architecture, implementation choices, andinterfaces of the simulator. It provides a place to record design agreements or archive design decisions so that they arenotendlessly revisited. It is expected that this document can evolve or fork to become an “ns-3 Developers Guide” at some pointin the future.

This document is written in Latex and is to be maintained in revision control on thens3 server. Changes to the documentshould be discussed on the ns-developers mailing list.

Contents

1 Functional Overview 31.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 31.2 User experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 3

1.2.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 31.2.2 User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 31.2.3 Scenario definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 41.2.4 Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 4

1.3 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 41.3.1 Source code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 41.3.2 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 4

2 Technical Overview 62.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 62.2 Use of external libraries . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 62.3 C++ Coding Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 72.4 Build environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 7

2.4.1 Build targets and variables . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . 82.4.2 Backward compatibility of bug fixes . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . 92.4.3 High-level architecture . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . 10

3 ns-3 core 123.1 Executing simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . 12

3.1.1 main() program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . 123.1.2 Command-line arguments . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 133.1.3 Scripting interface . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 14

3.2 Simulator, Events, Scheduler, Timers, and Callbacks . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2.1 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 153.2.2 Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . 153.2.3 Events and callbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 153.2.4 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . 16

3.3 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 173.3.1 Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 173.3.2 Simulation staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 18

3.4 Random number generators . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . 193.5 Messages and Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . 19

3.5.1 thens2 packet: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 193.5.2 George Riley’s current proposal . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . 193.5.3 Summary of Mathieu’s proposal based on yans . . . . . . . . .. . . . . . . . . . . . . . . . . . . . 27

3.6 Simulation output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . 303.6.1 Tracing and Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . 313.6.2 Statistics objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . 32

1

3.7 Memory management and topology construction . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . 333.8 Animation and GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . 34

4 Acknowledgments 35

2

Chapter 1

Functional Overview

This chapter describes thens3 simulator from a functional or user’s perspective; i.e., without as much regard to internalimplementation.

1.1 Goals

This section describes the broad goals forns3.

• ns3 is a discrete-event networking simulator, written in C++,with an emphasis on layers 2-4 of the OSI stack, includingIPv4, IPv6, and future next-generation (non-IP) networks.

• ns3 is oriented towards supporting networking research and education via simulation.

• ns3 is free software developed using a community-oriented, open source development process, under GNU GPLv2compatible licensing.

1.2 User experience

1.2.1 Installation

ns3 should be buildable from source or binary formats on popular desktop and server platforms, includingi386, x86_64,andppc, and the Linux, OS X (Darwin), Windows (32-bit, build environment TBD), and FreeBSD operating systems.

1.2.2 User interface

• ns3 should continue to offer text/script-based (non-GUI) configuration. It should be possible to create GUI-basedconfigurators, but such configurators are outside the scope of thens3 project.

• ns3 should output trace (including pcap), log, statistics, and animation files

3

– trace and log files should be convertible to the existingnam format, via some external scripting technique, forbackward compatibility.

• Open issue: shouldns3 be directly integrated with an animator or should the animator just run on output files(post-process)?

1.2.3 Scenario definition

Users use the simulator by firstdefininga simulation scenario,compilingthe scenario if necessary,executingthe simulationscenario, andprocessingthe simulation output, either visually through an animator, or through other handling of the outputfiles generated. Often, users of the simulator will directlymodify the simulator source code as a starting point.

• ns3 scenarios can be written in C++ (as a main() function), withselected configuration options exposed as command-line arguments

• ns3 should provide a scripting environment or interface

– full backward compatibility withns-2scripts is a non-goal

– Open issue: Selection of scripting language(s) is TBD. Options discussed include python, perl, ruby, scheme,Lisp, Java, OTcl variations, SWIG.

• scenario execution is visible on a console standard output,or written to a log file.

• support for some stock topology constructs should be provided

1.2.4 Capabilities

• ns3 shall provide an emulation capability– ability to source/sink real packets and execute in real-time

• ns3 shall be designed to scale for parallel processor support and distributed federated simulations, in a manner mostlytransparent to users.

• ns3 shall provide interfaces that facilitate the porting of implementation code (user space and kernel TCP/IP stacks)

1.3 Documentation

1.3.1 Source code

Open issue:whether to use Doxygen or kdoc. I assume Doxygen.

1.3.2 Documentation

ns-2uses Latex, as do all of these specs. An alternative is Docbook. Since we don’t have much math equations to handle inthe documentation, Docbook may have an advantage in that there are XSLT stylesheets to generate Latex/XHTML outputanyway. Going from Docbook to latex (bib, tex) seems not too hard; the other way, I am not so sure.

4

The documentation should be checked out nightly, built intoPDF and HTML, and posted on the web. Documentation shouldbe stored in the source code repository in source form, with figures stored in eps or image format. Vector graphics should bedrawn in a commonly available vector graphics program such as dia, tgif, or xfig, and the sources stored with the eps.

5

Chapter 2

Technical Overview

2.1 Basics

ns3 is a user-space program that runs on Unix- and Linux-based systems and on Windows (support type, via Cygwin or vianative win32 APIs, is to be determined). It contains supportfor the following:

• construction of virtual networks (nodes, channels, applications) and support for items such as event schedulers, topologygenerators, timers, and the like to support discrete-eventnetwork simulation focused on Internet-based and possiblyother packet network systems.

• support for network emulation; the ability for simulator processes to emit and consume real network packets

• distributed simulation support; the ability for simulations to be distributed across multiple processors or machines

• support for animation of network simulations

• support for tracing, logging, and computing statistics on the simulation output

ns3 is written in C++, with TBD scripting interface(s) for users.

Contributors are expected to conform to the project coding style, licensing, and copyright requirements (licensing and copy-right requirements described in the project plan document).

2.2 Use of external libraries

Use of external libraries is a balance between reusing code and introducing additional build dependencies on your simulation.In the past,ns2 has avoided dependencies on outside libraries. Forns3, we leverage selected outside libraries in the core,and other developers should be able to introduce their own libraries to support their specific implementations (e.g., use oflibxorp for routing protocol support).

Use of outside libraries (including Boost and standard library features including TR1) in thens3 core is to be considered on acase by case basis, to ease portability.

6

2.3 C++ Coding Style

Action item: George Riley to draft coding standard based on GTNetS with 4 space indents

Background: The following guidelines are recommended: http://geosoft.no/development/cppstyle.html

Other:

• C++ source files are suffixed by .cc or .h

• first level indent is four spaces (no tabs)

• opening brace is included on next line following object definition (rationale: support vi "]]" function; seems to be mostcommon style)

• continuation lines indented one-half level

• first letter of identifier is in caps only for classes

• compile time constants are all uppercase

• order of header files runs from most specific to most general

• use of C++ namespaces

• include allcaps (underscore _H) guard words on all header files for preprocessor defines

• use of require(), assure(), assert() is supported for checking pre- and post-conditions

• no global “using” directives in header files; no “using” directives outside of function definitions (rationale: avoidnamespace pollution)

• use built-in types where possible (stdint)

• use C++ namespaces, typecasting, templates, smart pointers, and other facilities of the language where possible, withtwo caveats: do not use immature or narrowly supported features (hopefully, regression build testing will uncoverthese), and document things that may be obscure to the developer with only moderate C++ experience

• support Doxygen or other documentation support

It is expected that contributions to the core of ns-3 could bevalidated with a style-checking script before patch inclusion. Ofcourse, deviations from the above shall be accepted if thereis a good reason to accommodate.

2.4 Build environment

We will select between the the GNU build environment (autoconf, automake, libtool, gcc) and SCons. The default Windowsbuild environment is Cygwin. Other compilers and build environments are outside the scope of the project, but we welcomeanyone who wants to look at mingw, msvc, or Intel compilers and document how to use them.

7

2.4.1 Build targets and variables

Building ns-3 shall result in the following:

• one or more main executables for discrete-event simulation; these executables should be runnable as a C++-executableor there may be support for running from scripting environment

• a set of libraries that can be dynamically linked to provide different components.

The build process and software structure (header file inclusion) should be designed such that changes to a module only resultsin recompilation of that executable or library.

Variables

All code should always be built with debugging symbols; evenoptimized builds.

By default,-Wall and-Werror should be turned on in the development branch, and disabled in the released code.

The following should be controllable by flags (note: taken from yans):

- TOP_BUILD_DIR:all the files generated during the build of the project arestored there. This variable is set to "bin" by default.example: make TOP_BUILD_DIR=/tmp/build-dir

- OPTI_FLAGS:the flags to pass to the compiler for optimization purposes.This variable is set to -O0 by default. This variable isnot very useful since you can use the "opti" target.example: make OPTI_FLAGS=-O3

- VERBOSE:if set to a non-empty value, ("y" for example), each compilationcommand is output on the terminal. Otherwise, short build messagesare output on the terminal. This variable is not set by defaultwhich means that only the short build messages are output.example: make VERBOSE=y

Targets

A number of build targets should be defined, including:

- all:build a non-optimized debug build of the code located in TOP_BUILD_DIR.asserts are enabled.

- clean:remove the TOP_BUILD_DIR directory

- cleano:

8

remove all object files from TOP_BUILD_DIR

- opti:build an optimized version of the code located in TOP_BUILD_DIR/opti.asserts are disabled.

- opti-arc:build an optimized version of the code with arc profiling information,run the programs specified by the run-opti-arc-profile-hook target,rebuild an optimized version of the code for this specific load.The code is built in TOP_BUILD_DIR/opti-arc.

- gcov:build a non-optimized version of the code with arc profiling informationand coverage analysis code. You can then run various programs to generatethe coverage information and then use gcov itself to gather user outputfrom this coverage information.The code is built in TOP_BUILD_DIR/gcov

- dist:build a tarball of the project named PACKAGE_NAME-PACKAGE_VERSION.tar.gzwhich decompresses in PACKAGE_NAME-PACKAGE_VERSION by default and containsall the code we ship.

- distcheck:the same as target dist but also decompresses the generated tarball, andattempts to build its default target.

- fastdist:the same as dist except that we do not delete the temporary directoryused to generate the tarball once it has been generated.

- fastdistcheck:the same as distcheck but we do not delete the temporary directoryused to build the decompressed tarball before running distcheck.

2.4.2 Backward compatibility of bug fixes

In the past,ns-2developers commonly inserted knobs (flags) for enabling previous model behavior, even if it was buggy.Going forward, we recommend:

• fix bugs in new releases without adding a new knob to disable the bug fix

• document behavior-changing bugfixes (by definition, they are all behavior-changes)

• when a user asks for the ability to reproduce a given simulation result obtained with an older release, answer that theony way to obtain reproduceable results is to use the *exact*same version all the time

• be willing to tweak this policy for certain important bug fixes which might be considered really parameters to the modelor different versions of the same model. For example, classic TCP bugs.

9

Channel

App App

Node

Interface Interface

Stack(e.g.TCP/IP,PCBs,etc.)

...

... Unix-like APIs(sockets, netfilter,pfkey, etc.)

Net/3-likeinterface API

Packet

This classstill underdebate(packed vs.unpackedformat, type ofmemory mgmt.features,level ofexplicitnessin the APIs)

...Stack

Position, energy

Figure 2.1: High-level architecture forns3 project.

2.4.3 High-level architecture

ns3 should have a modular implementation containing a core library supporting generic aspects of the simulator (scheduler,events, packets, random number generators, tracing, logging, and statistics, etc.) and a few abstract base classes, just to getthe architecture and interfaces defined consistently.

ns3 uses a stack-based architecture (where stack refers to a data stack and not the data structure). The basic idea is to moreclosely mirror how typical Internet-based computers are structured, both to facilitate software reuse (e.g., NetworkSimulationCradle by Jansen, porting of application code) and for educational purposes. Inns2, this architecture probably most resemblesthat of the CMU Monarch extensions (wireless).

Figure 2.1 provides an overview. Aside from the objects thatmake up the core simulation environment (such as scheduler,events, etc.), the main objects in the simulator are theNode (which is a composite object),Channel, andPacket. Atthe highest level, Nodes are interconnected via Channels, and they exchange Packets. A Node is a composite object con-sisting of interface a number of other objects make up the guts of a Node, including primarilyInterface, Stack, andApplication objects. Nodes may also have other objects such asPosition andEnergy objects. There are variousinterfaces defined internally between these objects but there are two crucial ones. First, applications bind to the kernel objectvia APIs that closely resemble Unix/Linux APIs (sockets, resolver, netlink, etc.). This allows for easier portabilityof actualapplication code, and is useful from an educational standpoint. Second, the interface between the TCP/IP protocol stack andany network interfaces resembles the Net/3 interface.Note: The packet structure is still an open issue.

The simulator should allow combinations of different stacks, interfaces, channels, and applications; e.g., we shouldstrive toavoid problems such as exist inns2 (wireless stack incompatibility with multicast routing).. Different stacks could speak thesame protocol (e.g., a Linux stack talking to a BSD stack) or there could be different stacks (e.g., non-TCP-IP). We may beable to port existing code to anns2 stack (and make backward compatibility optional if you do not want to compile all of theOTcl). We can focus on a reference TCP/IP stack (for IPv4 and IPv6) as part of the corens3 distribution, designed for clarityand extensibility, and allow projects such as the Network Simulation Cradle to create stacks derived from real systems.

Not all such stacks need be included in the main distributionand some may need to be external due to licensing compatibility

10

issues anyway.

11

Chapter 3

ns-3 core

This chapter discusses core simulation environment ofns3 (scheduler, packets, random number generators, scripting andexecutable interface, tracing, logging, and statistics, etc.).

I would suggest that animation is not part of the core but we may need some hooks for it.

3.1 Executing simulations

In ns2, simulation scripts are written in OTcl. Inns3, simulation scripts are written in C++, with support for extensions thatallow simulation scripts to be written in TBD scripting language(s).

3.1.1 main() program

Simulations are executed as C++ programs that link compiledobject code and libraries, instantiate and create bindingsbe-tween objects, and create a simulator object that controls the running of the objects in simulation time.

A basic skeleton of what this will look like is below:

// pull in headers, defines (constant objects, enums), etc.#include ...

// pull in namespacesusing namespace ns3; ...

int main (int argc, char *argv[]){// create simulatorSimulator s; (or Simulator may be static object...)

// process cmd-line argumentsArg::ProcessArgs(argc, argv);

12

// set up any other support such as tracing, logging

// set up defaultsQueue::DefaultLength(50); // 50 packets...

// Create topology

// create and define node objects in the simulation// and bind them together

// define when simulation events happen (stop, start, etc.)s.StopAt(10.0);

// Start the simulation; this is a blocking calls.Run() or Simulator::run();

// Do any cleanup and closing/post processing of files}

Both yans and GTNetS are structured along these lines, and alsons2 although written in OTcl..

3.1.2 Command-line arguments

Since it is often the case that a given simulation scenario must be run multiple times with a variety of different parameters,ns3should provide a simple interface to handle command line arguments. GTNetS has a nice facility for this– an args-processingsupport class.

Suppose that one wants to define an argument such as "stoptime". The Arg class supports the following (overloaded) methods

Arg(const std::string& n, double&);Arg(const std::string& n, double&, double); // third argument is a default

This can be used in a program as follows:

Arg("stoptime", stoptime, 100.0); // provides 100 second default// if user declines to provide argument

Arg::ProcessArgs(argc, argv); // called after all lines such as above// (may overwrite "stoptime" default)

If arguments are specified or not can be checked in the program:

if (Arg::Specified("animate"){

// enable animation}

See SRC/args.cc,h in GTNetS.

13

3.1.3 Scripting interface

Open issue:How to support scripting.

It seems generally accepted thatns3 should have a scripting interface, not based on OTcl, but opinions differ on what is theright alternative.

There have been a few options suggested:

• SWIG (http://www.swig.org)

• boost.python

• Ruby

• reuse/generalize the GIMP PDB (http://www.advogato.org/article/550.html).

It may be likely that we get some novel contributions from thecommunity on how to do scripting in multiple ways.

Lee Begg and Daniel Mahrenholz have advocated SWIG; the below post is from Lee: “SWIG would be good as with *nochange* to the library, we could have Ruby bindings, python bindings, java bindings, etc, etc. Users could then use thelanguage they are most confortable with, while keeping exactly the same core library code. The library should be sharedbetween all of them (a .so file on unix and a dll on windows). Also, when someone adds a new module (protocol, nodetype, whatever), they wouldn’t need to recompile the main ns3 library, and they can easily generate the language bindings forwhatever language they wish to use. These modules would alsobe shared libraries.”

Mathieu Lacage has raised some points in favor of boost.python over SWIG: “I don’t like swig very much and this is why Itried to use boost::python. The main reason is that boost::python allows you to control very precisely how the python codeAPI looks like. i.e., it is possible to make the python API be very pythonish (follow classic python idioms), independently ofthe c++ API. Boost::python is a way to build a python scripting interface by hand and save you the pain of having to writethe bulk of the code. Swig, on the other hand, is a way to build ascripting interface automatically which saves you having towrite the code but doe snot allow you fine-grained control over the final API. Swig usually involves more constraints over theunderlying C++ API because you cannot really tweak the behavior of swig. Memory management is, for example, very hardto get right. Of course, if someone volunteers to do all the swig work and since I don’t plan on using the bindings myself andif it has no negative impact/requirements on the core c++ API, then, I don’t care much either way.”

Ilyes Gouta has advocated Python as the scripting language on the ns-developers list.

3.2 Simulator, Events, Scheduler, Timers, and Callbacks

This section is concerned with the general structure of the simulator for coordinating the execution of simulation events.

At high level,ns2, yans, and GTNetS all have the following concepts:

• a base Simulator class

• an Event class and some facility for handling events

• a base Scheduler class designed to hold Events

• some facility for callbacks

• support for timers

14

3.2.1 Simulator

At its most basic, a Simulator object provides a public interface that allows objects to insert Events into a Scheduler, andkeeps track of simulation time elapsed.

Open issue:how to define the Simulator object.

In ns2, the Simulator object is allocated on the heap; in GTNetS, Simulator is an object allocated on the stack; and in yans, itis a statically-defined object.

Time is maintained in a double called clock_ in ns-2 (with precision of1/2(52)), in a uint64 in yans (integer microseconds),

and a double in gtnets (floating point). Mathieu has argued that it is better to avoid using a floating point number to avoidrounding errors due to floating point arithmetic on different platforms.

3.2.2 Scheduler

The Scheduler is an object that dynamically stores Events insome type of ordered data structure. There are various structurespossible (linked list, heap, map, etc.) to hold the events; depending on the type and number of Event manipulations requiredin a simulation, one type of underlying scheduler may perform better than another.

The simulator should support easy redefinition of the underlying scheduler (all schedulers supporting the same public inter-face); this should be a non-controversial part of the designand can be adapted fromns2, GTNetS, and/or yans.

3.2.3 Events and callbacks

An Event is an object that tells the simulator to do somethingat a specific time. There are a few related concepts to discusshere: events, timers, and callbacks

A concern with Events is memory management (leaks). The Event class can be given a smart pointer to avoid memorymanagement problems.

Next, we discuss how Events are generated and consumed.

The ns2 way of doing things: In ns2, Events are consumed by a subclass derived from class Handler. This leads tothe convention that class Handler, which provides pure virtual handle() method that is called when the event executes, issubclassed when needed (and such classes are declared friend to classes that use the Handlers), and the pure virtual handle()method is tailored by the specific Handlers to consume specific events. Class Event is a wrapper around Handler, defined asa node in a linked list.

It seems to me that gtnets essentially adopts the framework used byns2 (Handler class), although George Riley may havemore to say here.

The yans callback design: A yans callback is a wrapper around a C++ method or function. Ayans Event is a wrapperaround a C++ method or function that also holds the value of the arguments passed to the method/function. The methodconsuming the event can be any method in scope (possibly witharguments). The Event subclass is simple, providing a purevirtual notify() that consumes the event (you need to invokedelete on an Event explicitly after calling notify).

15

At this point, the yans Event and ns-2 Handle provide similarfunctionality (with the yans Event list being maintained insteadby the Simulator object).

The departure from ns-2 way of doing things is that two layersof templates are defined on top of Event. Basically, yansdefines callback objects that implement a C++ template-based version of the Functor design pattern. In terms of Boost, ayans Event is a fully-bound functor. Events are scheduled with a make_event(&method) call, which instantiates a callbackevent. The callback event (also defined by template) calls the method referenced via its function pointer, and does garbagecollection on itself if needed. The net effect is an avoidance of needing to declare friend classes of objects for the purposeof consuming simulation events– arbitrary methods insteadcan be used for that purpose and scheduled by referencing theconsuming method() by function pointer.

The callback API is absolutely fundamental to yans : its purpose is to minimize the overall coupling between various piecesof yans by making each module depend on the callback API itself rather than depend on other modules. It acts as a sort ofthird-party to which work is delegated and which forwards this work to the proper target module. This callback API, beingbased on C++ templates, is type-safe; that is, it performs static type checks to enforce proper signature compatibilitybetweencallers and callees. The API is minimal, providing only two services:

• callback type declaration: a way to declare a type of callback with a given signature, and,

• callback instantiation: a way to instantiate a template-generated forwarding callback which can forward any calls toanother C++ class member method or C++ function.

This callback API is already used extensively in most modelscurrently available in yans . For example, the LLC/SNAPencapsulation module was made independent of the IPv4 and Arp layers with callbacks which are invoked whenever a payloadmust be forwarded to the higher layers.

We are leaning towards adopting this yans design.

Open issue:How to cancel an event. There are two basic options to cancel an event:

• Place a cancel method on the Event class, but do not remove theEvent from the Scheduler (simply ignore it when itexpires)

• Place a remove method on the Scheduler class

The Event::cancel() method has O(1) complexity (set a bit inthe Event data structure) but increases the size of the Eventqueue. The Scheduler::remove() method has O(log(n)) complexity but decreases the size of the Event queue (faster inserts).Although both methods have the same semantics, they have different complexity behaviors. We could support both, but thereis a danger in providing users with two ways to do the same thing. We could also provide support for both via a wrapperfunction, experiment with one or the other for a while, and later rip out one of them.

3.2.4 Timers

In ns-2, timers (class TimerHandler) are derived from Handlers. At a high-level, they wrap Events that are inserted intothescheduler, providing methods like cancel(), sched() and resched().

In GTNetS, timers also derive from Handlers, and carry a TimerEvent that derives from Event. the public interface is similarto but a bit richer than that of ns-2.

In yans, timers are Events; the method to call upon expiry is stored in the function pointer (discussed above in the context ofyans Events).

16

3.3 Scalability

One of the major concerns aboutns2 cited by its users is scalability.ns is a sequential execution simulator with a single eventprocessing loop running on a single processor. Although such a simulator can scale to hundreds of nodes when underlyingcommunications models are heavily abstracted, the memory and processing resources of a single machine become a bottleneckwhen more sophisticated channel models (e.g., wireless) orhigher-rate links (e.g., 10 Gbps) are included.

Scalability generally refers to running simulations with large numbers of events or packets in the smallest amount of time orwith the lowest memory footprint. This has impact on both thecore simulator (scheduler, packet design) and models builtontop of the core:

• the core (scheduler, packet) should be as fast as possible and permit parallelization, and

• selected models relating to large numbers of nodes, packets, or events can employ additional scalability techniques.Parallelization techniques should be added to some models.

Researchers have taken various approaches to improvens-2scalability, including the caching of redundant computationsand function calls (the “Staged NS (SNS)” project at Cornell), use of on-demand route computation (NixVector routing),and partitioning the simulation into wireless clusters (the ns-2 gridkeeperand similar structures developed by Naoumov andGross1.

3.3.1 Parallelization

We would like to enablens3 to use multiple processors in parallel, to speed simulation. One strong requirement is to beable to reuse the same simulation script in both the parallelcase and the non-parallel case (the difference is a rebuild). Toachieve this, GTNetS uses ghost nodes (and ghost links, etc): whenever a node is instantiated, it takes an computing-node-idwhich is used to chose whether to instantiate a real node or a ghost node. Basically, if you are on computing-node X andyou have mapped 3 simulation-nodes, A,B,C to this computing-node, you will create 3 real simulation-nodes and n - 3 ghostsimulation-nodes on computing-node X (where n is the numberof simulation-nodes in the global topology).

Ghost nodes are also used later during the simulation to allow each computing node to access the global simulation topology(each ghost node knows to which other ghost node it is connected to) which allows each computing node to perform whole-simulation routing decisions with nix vectors by parsing the ghost node topology graph.

The COMPASS, PADS, and MASCOTS research groups at Georgia Tech have developed aFederated Simulation DevelopersKit (FDK) and aghost–nodeapproach, used in GTNetS, which we plan to apply tons3. This approach creates a federatednetwork simulation consisting of a number of instances of the simulator tied together using a Runtime Infrastructure (RTI)middleware software layer, with the RTI-interconnected nodes communicating over Myrinet or Ethernet. With this approach,a small memory overhead is incurred at each of the distributed simulation processes, which obviates the need for complicatedinter–simulator routing decisions and obviates the need for a simulated routing protocol. This approach has been applied tons2 and is called parallel, distributed ns (PDNS).

PDNS uses a conservative (blocking based) approach to synchronization. No federate in the parallel simulation will everprocess an event that would later have to be undone due to receiving messages in the simulated past. This avoids the needto implement state saving in the existingns code. The PADS research group at Georgia Tech has previouslydeveloped anextensive library of support software for implementing parallel and distributed simulations. The sofware has supportforglobal virtual time management, group data communications, and message buffer management. It has support for a varietyofcommunication interconnects, including shared memory, Myrinet and TCP/IP networks, and runs on a variety of platforms.

1V. Naoumov and A. Gross, "Simulation of Large Ad Hoc Networks," Proceedings of the 6th ACM Workshop on Modeling, Analysis, and Simulation ofWireless and Mobile Systems, 2003.

17

By using this synchronization software for the parallelization of ns, we were able to rapidly modify the main event processingloop of ns to support the distributed time management functions needed to insure that no unsafe event is ever processed byany federate.

The modifications needed tonscan be broadly classified in two major categories, the modifications to thensevent processinginfrastructure, and extensions to thenssyntax for describing simulations. In particular,

1. Any reference to a remote endpoint of a connection must be by network elements such as IP address and port number,rather than memory address pointers to the remote object. Ina distributed simulation, there is no guarantee that aremote protocol object has a memory address or a representation on any given simulator instance.

2. Packet routing decisions may need topology information not present on a simulator instance, and thus some methodmust be designed to properly route packets in the presence ofincomplete topology information. The obvious solution tothis issue is the use of arouting protocol, such asOSPF, to compute routes. However, this approach reduces scalabilitydue to the need for potentially large routing tables at each simulated node.ns-2already provides a choice betweencentralized routing, the use of simple models of distributed routing (distance vector and link state) and NixVectorrouting– a form of source routing that can retain and use a complete routing path from a source to a destination in avery compact form.

3. Some method of packet serialization and reassembly must be provided, to allow for the flow of packets from onesimulator instance to another.

4. The distributed simulation features of the simulator should not hinder in any way a simple sequential simulation.

5. It should be no more difficult to construct a distributed simulation than to construct a sequential simulation.

Refer to the paper and contributed code (“pdns”)2 for more details.

3.3.2 Simulation staging

One enhancement we want to add tons3 is a capability called “staged simulation”3. This capability is of particular relevanceto wireless models but may not be as applicable elsewhere. This capability is already inns2 via non-integrated ContributedCode.

The main idea is to improve the run-time by eliminating redundant computations, via:

• function caching and reuse:cache arguments, outputs, and side-effects of computationally intensive functions, andlater reapply them if the event reoccurs;

• event restructuringimproves on function caching by modifying events such that their results are reusable even when achange of inputs would normally preclude reuse;

• time shiftingtakes advantage of the small, fine-grained events produced by event restructuring by reordering them intoan equivalent but more efficient schedule.

Refer to the paper and contributed code (“SNS”) for more details.

2G. Riley, R. Fujimoto, and M. Ammar, “Parallel Distributed ns,” Software on-line: http://www.cc.gatech.edu/computing/compass/pdns/index.html3K. Walsh and E. Gun Sirer, “Staged Simulation: A General Technique for Improving Simulation Scale and Performance,” ACMTOMACS, Vol. 14, No.

2, April 2004, pages 170-195.

18

3.4 Random number generators

ns-2, yans, and GTNetS all use Pierre L’Ecuyer’s MRG32k3a randomgenerator. There was talk on the list, started byMathieu, about moving to GNU Scientific Library, but it seemed that consensus was to stay with Pierre’s RNG due to itsbetter portability and also support for independent streams.

3.5 Messages and Packets

Packets (messages) are fundamental objects in the simulator and their design is important from a performance and resourcemanagement perspective. There are various ways to design the simulation packet, and tradeoffs among the different ap-proaches. In particular, there is a tension between ease-of-use, performance, and safe interface design.

There are a few requirements on this object design:

• Creation, management, and deletion of this object should beas simple as possible, while avoiding the chance formemory leaks and/or heap corruption;

• Packets should support serialization and deserializationso that network emulation is supported.

• It should be natural for packets to carry actual applicationdata, or if there is only an emulated application and thereis no need to carry dummy bytes, smaller packets could be usedwith just the headers and the data size, but not bytes,conveyed in the simulated packet.

• Packets should provide an interface to emulate BSD-like operations on mbufs.

• Additional side-information should be supported, such as atag for cross-layer information.

3.5.1 thens2 packet:

In ns2, packet headers are defined as individual data structures,and a Packet object contains the concatenation of all activeheaders in the simulator. New packet headers are introducedinto the simulator by defining a C++ structure with the neededfields, defining a static class to provide OTcl linkage, and then modifying some of the simulator initialization code to assigna byte offset in each packet where the new header is to be located relative to others.

When the simulator is initialized through OTcl, a user may choose to enable only a subset of the compiled-in packet formats,resulting in a modest savings of memory during the executionof the simulation. Presently, most configured-in packet formatsare enabled. Recently, a packet header manager class was created to more easily enable the headers in use. Application datacan be carried in these packets, but often is not.

3.5.2 George Riley’s current proposal

George and Mathieu have been discussing the GTNetS and yans packet designs, which has led George to this current proposal.Mathieu’s is summarized in the next subsection.

The design of thePacketobject uses the concepts ofsmart pointersand copy on writesemantics. With this design, themanagement of the dynamic memory associated with a packet iscompletely hidden from the users of packets, and it includesan efficient mechanism to copy the contents of a packet only when necessary. This design and the application programminginterface (API) for managing packets is discussed it the following paragraphs.

19

eb

Empty PDB

c

Figure 3.1: Empty Packet Data Buffer

802.3

eb

TCP

PDB with PDUs

IP

c

Application

Data

Figure 3.2: Non–Empty Packet Data Buffer

Logically, apacketis simply an ordered collection ofprotocol headersor (PHdrs). EachPHdr might be a protocol header,such as aTCP header, or a set of application data, such as an query requestin a peer-to-per simulation. ThepacketAPI,described in detail later, is primarily a set of methods to manipulate thePHdrsby adding, removing, inspecting, or changingthe variousPHdrsthat comprise the packet.

The Packet Data Buffer (PDB)

The collection ofPHdrsmanaged by a packet are stored in thePacket Data Buffer(PDB). ThePDB is a single, contiguousarea of memory that is allocated when aPHdr is added to an empty packet, and is deleted when all copies of the packetcontaining thePDBhave been destroyed. Details of this are given below in the smart pointer semantics section.

ThePDB is managed by three pointers,b which points to the beginning of the buffer,e which pointsone beyond the endofthe buffer, andc which points to thePHdr at the top of the stack. Ifc is equal toe, then thePDB is empty, and there is noPHdr at the top of the stack.

When a packet is constructed, thePDB is not allocated, and all three pointers arenil. ThePDBwill be allocated the first timea newPHdr is added. The reason for delaying the allocation of thePDB is to be able to make a reasonable guess as to howbig thePDBshould be. In thensdesign, there are two sizes ofPDBs, calledsmallandlarge. The reason for this will becomeclear in the subsection discussingApplication Databelow.

An emptyPDB is illustrated in Figure 3.1 and a non–emptyPDB is illustrated in Figure 3.2.

20

Protocol Headers

All protocol header objects innsare subclasses of the base classPHdr, which is an abstract base class. The base class definesthe following pure virtual functions which must be defined byall subclasses.

1. Layer - Return the layer number for thisPHdr. For example, aTCPheaderPHdr would return the value 4.

2. Protocol - Return the protocol number for thisPHdr. For example, aUDP headerPHdr would return the value 17.

3. Size - Return the size, in bytes, of thePHdr in the actual protocol implementation. For example, aTCP header isdefined by RFC761 to be 20 bytes (ignoring the presence of theOptions). Thus theTCPheaderPHdr would return thevalue 20. This is used when packets are being transmitted on amedium, and is used to determine how many bytes mustbe transmitted, and therefore how long it will take on a specific medium.

4. SizeOf - Return the size, in bytes, of thens object for thePHdr. This is unrelated to the value returned bySize,since thens PHdrobjects are not packed as are actualPHdrs.

Application Data PDUs

In general, the firstPHdr pushed onto thePDB for a new packet will be the application data. Inns, the application dataPHdrscome in two varieties,DatalessandDatafull. A Datalessapplication dataPHdr is used in the case where to amount of datais important (for example, 1000 bytes), but the actual contents of the data are not. This would be the case, for example, ifmeasuringTCP throughput under varying buffering scenarios. TheTCP throughput is in general independent of the contentsof the data. ADatafullapplication dataPHdr would be used in the case where the actual contents of the dataare meaningful tothe simulation. This would be the case when modeling routingprotocols, where the contents of routing tables are exchangedand used in the simulation.

If the first PHdr pushed isDataless, the asmall PDBis allocated. If the first isDatafull, then alarge PDBis allocated. Theactual value for the sizes of the small and largePDBs is determined heuristically, as described below.

Managing the Protocol Headers

The collection ofPHdrs in a PDB by a packet are managed by thePacketmember functions. These can be categorized asPush(add a newPHdr to thePDB), Pop (remove aPHdr from thePDB), Peek(get read-only access to the top of thePHdrstack), and querying withIsEmpty().

1. A newPHdr is added to a packet (and to thePDB) using thePushPHdr method, defined as follows:

void PushPHdr(const PHdr& pHdr)

This method performs the following:

(a) Insure that the packet has aprivatecopy of thePDB. If not, a copy of the existingPDB is made. As we showbelow, this should be a very rare condition.

(b) Decrement thec pointer bypdu.SizeOf(). If c is less thanb, then thePDB has overflowed. In this case, thedefault size of thesmall(if the PDB is small) orlarge (if the PDB is large) is doubled, thePDB is reallocated andcopied, and this step is repeated.

(c) Use thein–placecopy constructor and copyPHdr pHdr to thePDBat locationc.

21

2. A PHdr is removed from the packet and thePDBusing thePopPHdr method, defined as follows:

PDUSubclass PopPHdr()

This function returns a copy of thePHdr at the top of thePHdr stack. See the examples below for an illustration ofhow it is called efficiently.

(a) Abort with an error message if thePDB is empty.

(b) If running indebugcompilation mode, compares theSizeOf() value from the top of hePHdr stack with theSizeOf() a temporary variable of the same type as ther return type. If these are not the same, the call iserroneous andnswill abort with an appropriate error message. If running with theoptimizedcompilation mode,this check (and the checks in the next step) are not performed.

(c) Similarly, if in debugmode, compare theLayer(), Protocol(), andSize(), all of which must match.

(d) Save a copy of thec pointer and advance it by theSizeOf() thePHdr being removed.

(e) Return a copy of thePHdr pointed to by the saved copy of thec pointer.

(f) Note that a private copy of thePDB is not required for this call, as the contents of thePDBare not being modified.

As an example, suppose theTCPprotocol object is processing anupcall from the layer 3 processor for a packet. Thecode snippet would be similar to the following:

void TCP::DataIndication(Packet& p){

TCPHeader tchp = p.PopPHdr<TCPHeader>();... Process the tcp header here.

}

3. The head of thePHdr stack can be examined without removing it using thePeekPHdr()method, defined as follows:

PDUSubclass PeekPHdr()

(a) If running indebugcompilation mode, perform the same checks as thePopPHdr() method. If inoptimizedcompilation mode, the checks are not performed.

(b) Return a copy of thePHdr pointed to by thec pointer. Thec pointer is not modified.

A common expected use of thePeekPHdr() method would be in the case of layer 2 processing in a broadcastmedium, where many copies of the packet exist. All receiverswill need to inspect the layer 2PHdr to check thedestination address, and to process things such as thedurationvalue in an 802.11 header. Only the intended recipientwill need to remove thePHdr or change the packet in any way.

4. A packet can be queried to see if there are anyPHdrs in the packet using theIsEmpty() call, which returnstrue ifthere are noPHdrsin the packet, and true if there are. This is used primarily bya layer 4 processor to see if a DataPDUis present.

The PDB Smart Pointer Semantics

As mentioned previously, a primary design goal of thePacket object and the management of thePHdr stack is to efficientlymanage the potentially largePDB, and to hide from the users the need for the memory managementcalls for allocation (new)and de–allocation (delete) of packets,PHdrs, andPDBs. To accomplish these goals, thePDB is managed usingsmartpointer semanticsas follows.

1. When a newPDB is needed (in response to a call toPushPHdr() on an empty packet), an appropriate sizedPDBbuffer will be allocated from theheapusing thenew semantics. In addition, a separateint variable will also beallocated from the heap. This second variable is called therefCount and indicates now many packets have a pointerto thePDB just allocated. Clearly, when thePDB is initially allocated therefCount value should be 1.

22

2. Whenever a packet object is copied, either with theCopy Constructoror theAssignment Operator, the pointers forboth thePDBand therefCount variables are copied, but not the contents of either. This isknown as ashallow copy.When a shallow copy occurs, the content of therefCount variable is incremented by 1. Packet copying will occurwhenever a packet is passedby valueto a subroutine, or when initializing a structure member of typePacket fromanother packet.

3. Whenever a packet object is destroyed with thedestructorcall, the content of therefCount variable is decrementedby 1. If the result is zero, this indicates that there are no more copies of the correspondingPDB, and both thePDBandtherefCount should be de-allocated with adelete call.

The decision of how to pass packets (by value, by const reference, or by non–const reference) when calling subroutines shouldin general be as follows:

1. If the called subroutine is not expected to modify the packet, it should be passed byconst reference.

2. If the called subroutine is expected to modify the packet,it should be passed by reference.

3. Passing packets by value to subroutines should never be done.

Passing Packets in the Protocol Stack

To illustrate how the smart pointer semantics are to be used in protocol stack processing, we give a simple example showingthe flow of a packet starting from its creation at layer 4, until it arrives at the final destination at layer 4. To reduce or eliminateall copying of thePDB, all calls between protocol stack layers should be bynon–const reference. This will insure that, as thepacket is being modified either by addingPHdrsor removingPHdrs, therefCount value will usually be exactly 1, and nocopying is needed.

When a protocol stack layer processor (either aDataIndicationor aDataRequest) receives a packet as a reference parameter,it can do one of three things.

1. Modify the packet in some way, and pass the packet to another layer (either up or down the stack) by non–constreference. In this case therefCount value remains unchanged, and presumably is still exactly 1.

2. Decide that the packet is no longer of value and should be deleted. This might be the case if the IPV4 processordetermines it does not know how to route the packet, or if theTTL value has decremented to zero. In this case, theprotocol stack layer processor simply does nothing and returns. TherefCount value remains unchanged.

3. Copy the packet for later processing. This might be the case when the layer 2 processor for a point–to–point link haseither put the packet in a queue (by copying it to the queue), or scheduling aReceive Packetevent for the receivinginterface. In either case, a copy of the packet (but not thePDB) is made, and therefCount is incremented by 1 (mostlikely to the value two).

As an illustration, we start by assuming that aTCPprotocol object has determined that it should send some datato a remoteendpoint. This might be in response to aSend method call from an application, or the processing of a receivedAck openingup additional congestion window space. We point out that theexamples are illustrative, and the arguments to the variousprotocol stack layer calls are not necessarily shown exactly as implemented.

bool TCP::SendPendingData(){ // Send up to MSS data bytes

Packet p; // The packet to use to send this data

23

DataPDU& data = GetPendingData(); // The data to sendp.PushPHdr(data); // Add the data

TCPHeader tcph; // The corresponding TCP header// Set the appropriate tcp header values here (seq, ack, flags, etc.)p.PushPHdr(tcph); // Add the TCP header to the packet// At this point, we have a packet with a packet data buffer// and a refCount value of 1.// Call layer 3 data indication to continue down the stackIPV4::DataRequest(DestIP, p);

// At this point, one of two things has happened. Either the lower// layers have copied the packet somewhere (a queue or a receive packet// event), in which case the refCount is two. Or the lower layers// have determined the packet cannot be processed and simply returned,// in which case the refCount is 1.

// When this method exits, the packet p goes out of scope and the// destructor is called and the refCount is decremented.// If the resulting refCount is 0, there are no more copies of the// packet and the corresponding PDB is deleted. If not, this means// one of the lower layers has copied the packet, the refCount// is 1 (or more) and the corresponding PDB is retained.

}

The processing at the IPV4DataRequestmethod might be:

void IPV4::DataIndication(IPAddr destIP, Packet& p){

if (CantRoutePacket(destIP)){ // Unable to find a route// Make trace file entry indicating no route to destinationreturn; // Nothing else needed

}IPV4Header iphdr; // IP Header to add// Set fields in iphdr herep.PushPHdr(iphdr); // Add the layer 3 PDU// Find the layer 2 processor for apprpriate route (not shown)// Forward the packet to the layer 2 processor for forwarding to next hopl2->DataRequest(destMAC, p);

}

Finally, the processing at the layer 2DataRequestfor a point–to–point link might be:

void LinkP2P::DataRequest(MacAddr destMac, Packet& p){

if (LinkIsDown()){ // Can’t send on down link// Make trace file entry indicating packet lost due to link down

return; // Nothing else needed

24

}L28023Header l28023hdr; // Layer 2 header to add// Fill in appropriate header fields here.Time_t rxTime; // Calculate time packet is receivedTime_t txTime; // Calculate time packet transmission is completedSimulator::Schedule(txTime, TxHandler, MakeEvent(p)); // Schedule tx completeSimulator::Schedule(rxTime, RxHandler, MakeEvent(p)); // Schedule rx complete// Note the calls to MakeEvent above make a copy of the packet.// This results in a refCount value of 3 at this point.// However, when the TCP::SendPendingData() method above exits, the// destructor for this packet is called, and the refCount becomes 2.

The processing at theTxHandlermethod might be:

void LinkP2P::TxHandler(Packet& p){

// If another packet in queue, deque it and send.// Make a trace file entry indicating the transmission of p is completereturn;// Note that nothing is done with p, so the refcount remains at 2.// When the caller of TxHandler exits, the event structure with// the copy of the packet p goes out of scope, and the packet is// destroyed, reducing the refCount to 1.

}

The processing at theRxHandlermethod might be:

void LinkP2P::TxHandler(Packet& p){

if (p.bitError){ // Packet has a bit error on receipt. Make a trace file entry and returnreturn; // Nothing else needed// In this case, nothing is done to refCount, so when the caller// exits, the event is destroyed, along with the packet, and// the refCount goes to zero, indicating the PDB is no longer// needed.

}L28023Header l28023hdr = p.PopPHdr<L28023Header>(); // Remove the l2 headerif (l28023hdr.destMac != myMac)

{ // Not addressed to me, make trace file entry and return.return;// As above, refCount is unchanged

}// Find the protocol number from the l2 header and find the l3// processor from the protocol graph.L3Proto& l3 = FindProto(l28023hdr.proto, 3);l3.DataIndication(p); // Up the stack to layer 3

}

The processing at the layer 3 data indication might be:

25

void IPV4::DataIndication(Packet& p){

IPV4Header iphdr = p.PopPHdr<IPV4Header>(); // Get the ipv4 headerif (!IsLocalIP(iphdr.destIP))

{ // Not addressed here, re-route and forwardif (--iphdr.ttl == 0)

{ // TTL expired, make trace file entry and returnreturn;

}// Find the layer 2 processor for apprpriate route (not shown)// Forward the packet to the layer 2 processor for forwarding to next hopp.PushPHdr(iphdr); // Push the revised ip headerl2.DataRequest(destMAC, p);

}else

{ // Is addressed here, find l4 protocol object (not shown)l4.DataIndication(p); // Pass the packet up the stack

}}

The processing at the layer 4 data indication might be:

void TCP::DataIndication(Packet& p){ //

TCPHeader tcph = p.PopPHdr<TCPHeader>(); // Remove the tcp header// Send ack if neededif (!p.IsEmpty())

{ // If a data pdu existsDataPDU data = p.PopPHdr<DataPDU>();// Pass the data to any associated applicationapp.Receive(data);

}// Returns without forwarding the packet anywhere. When the// protocol stack unrolls back to layer 2, the original packet// will be destroyed with a refCount of 1, resulting in a deletion// of the PDB and the refCount field.

}

The primary issue of importance in these examples is to note that each time a packet needs to be modified, it will have apacket with arefCount value of 1, allowing modification without copying thePDB. The only likely exception to this ruleis for a broadcast media (ie. wireless) when the packet is replicated to all potential receivers. In this case, thedesignatedreceiver(ie. the receiver the packet is actually addressed to) will modify the packet withPopPHdr() and a copy of thePDBwill occur. All other receivers will simply inspect the packet without modification.

Packet Fragmentation

Nothing yet specified.

26

3.5.3 Summary of Mathieu’s proposal based on yans

Mathieu Lacage has defined the packet in yans a bit differently. yans stores a bunch of bytes, similar to how the Linux/BSDskbuff/mbuf structure is designed. In yans, it is thus up to the user to decide on how to interpret the content of a Packet, whilein George’s proposed approach, the packet itself knows whatprotocol headers were pushed into it. Unlikens2, there is nostatically defined offset value that allows a user to easily find the header in the packed data structure.

Each approach has pros and cons: the GTNetS approach makes itvery easy and natural to print the content of a packetin a string while the yans approach makes it really natural tohandle fragmentation and reassembly. Discussions betweenGeorge and Mathieu suggested that it possible technically to merge the two approaches and make the yans packet know aboutprotocol headers or the GTNetS packet know about the byte-representation of protocol headers. However, it is clear thatanysuch solution will be much more complex to implement than anyof the two solutions we have today.

George indicated that he strongly prefer the GTNetS approach, as it provides substantial more flexibility and enables experi-menting with new protocols or variations of existing without having to re-use bits or squeeze out bits from existing fields. Weunderstand of course that the C/C++ "sizeof" the PDU is not related to the computation of how many bits will be transmittedon a wire. A TCP header will be 20 bytes as expected, regardless of the size of the C structure.

However, Mathieu observed that the GTNetS implementation can convert easily a normal packet to a packed representationthrough serialization. The reverse operation, on the otherhand, requires external code to generate a GTNetS Packet from abuffer of bytes by parsing the buffer of bytes and reconstructing the protocol headers stored in the gtnets Packet. Special codeis not needed with the implementation in yans: it is “transparent.”

This discussion then boils down to where and upon who we want to put the burden for the “transparency” of emulation andreal- world-code. i.e. allowing the user to be lazy (which isnot bad in itself) means that you will need to put extra effortduring integration with the other emulation/real-world-code-integration projects. yans has tried to avoid this sortof extraeffort later at the cost of not allowing the user to be lazy at the beginning.

It should be noted also, that lazyness is still possible withthe approach choosen in yans: it is possible for a model author to notproduce a correct network-conformant serialization and just serialize/deserialize enough data for his or her needs (althoughwe do not encourage it). Of course, this is a poor man’s version of what GTNetS can do because, in this case, the number ofbytes you need for your serialization *must* be smaller or equal to the number of bytes of the simulated protocol header. Orit can be larger but in this case, you will just simulate a somewhat larger packet than expected which might be a problem ornot, depending on what you want to do.

Memory management issues:

Whichever approach is chosen, care must be taken to avoid memory leaks and properly handling constness issues.

George and Mathieu agreed that whatever the Packet API lookslike, it needs to provide transparent memory management,potentially through the use of smart pointers.

Mathieu suggested decoupling the Packet class itself from the PacketPtr and ConstPacketPtr classes as is done in yans. Inyans, Packets are always accessed though the PacketPtr or ConstPacketPtr classes which are pointers to Packets.

typedef RefPtr<Packet> yans::PacketPtr- smart-pointer for Packet- PacketPtr objects are pointers to Packet instances. They behave likeany normal pointer: you can assign 0 to them, compare them against 0 andpass them around by value. They manage the resources of the Packet theypoint to automatically.

27

typedef RefPtr<Packet const> yans::ConstPacketPtr- smart-pointer for const Packet- ConstPacketPtr objects are pointers to Packet instances, just likePacketPtr, except that they point to instances of _const_ Packets whichmeans you cannot invoke any non-const method on the underlying Packet.

George advocates using the Packet class described in the COWapproach described below.

“Copy-on-write” semantics:

There was some discussion of the copy-on-write semantics.i) copy-on-write semantics as suggested by George

class Packet {void PushPDU (PDU pdu) const;void PopPDU (PDU pdu) const;private:PacketImpl *m_impl;};

ii) non-copy-on-write as suggested by Mathieu

class PacketPtr {Packet *operator -> ();private:Packet *m_impl;};class Packet {static PacketPtr copy (void);static PacketPtr create (void);void PushPDU (PDU pdu) const;void PopPDU (PDU pdu) const;};

The latter approach has more problems with constness.

Here is an interesting use-case which modelizes a MAC protocol which can do broadcast or unicast transmission. The intdest variable is set to -1 for broadcast or to the address of the destination. When we are the target of a packet, we need to*remove* the PDU. Otherwise, we can just *look* at it with Packet::peek.

class Packet {void ref (void);

void unref (void);void add (PDU const &pdu);void remove (PDU const &pdu);Packet *copy (void) const;void copy_and_remove (PDU const &pdu) const;void peek (PDU const &pdu) const;

28

};

void receive (Packet const*a, bool dest){PDU pdu;if (dest) {#ifdef GEORGEa->copy_and_remove (pdu);#elsePacket *b = a->copy ();b->remove (pdu);#endif} else {a->peek (pdu);}}

Packet *b = /* XXX */;int dest = /* XXX */;for (i = 0; i < 10; i++) {if (i == dest || dest == -1) {receivers[i]->receive (b, true);} else {receivers[i]->receive (b, false);}}b->unref ();

Ifdef GEORGE, we invoke the copy_and_removemethod which removes the PDU but only after making a copy if the refcountof the packet is not 1 (if the refcount is 1, then we are the onlyowner the packet, so we can remove without a copy). It is notclear, however, how often this optimization will be triggered in the real world.

ifnot GEORGE, we first make a copy and then call remove on the copy. Of course, the Packet::copy method does the smart"copy only if refcount is not 1, remove const otherwise".

So, the difference between the two solutions is that we provide either two methods to the user or just one. In both cases, the per-formance is optimal and the code is const-safe (other than the magic const_cast in Packet::copy and in Packet::copy_and_remove).

Therefore, it is possible to do some optimization in the copymethod to avoid the copy altogether if one can. However, it isstill being debated whether it is nice to invoke copy from another method exposed to the user.

If we drop the optimization for avoiding a copy if refcount is1, then the two approaches are equivalent in performance andonly differ in whether the copies are hidden (ifdef GEORGE) or explicit.

Other differences

This section summarizes the main differences between the two approaches, as of the start of the kickoff meeting. Mathieu’sproposal is called “yans” below

1. Explicit vs Implicit issues

29

• yans has an explicit PacketPtr class which explicitly points to a Packet class with explicit pointer syntax whileGeorge’s proposal has a Packet class that is really a pointerto the data but that behaves as if it were the actualdata; the Packet does not have pointer syntax.

• yans has an explicit Packet::copy method which does a deep copy while George’s design has implicit copy con-structors and assignment operators.

• yans has an explicit Packet::create packet creation methodwhile George’s design has simply a default constructorwhich can be invoked on the stack

In Mathieu’s opinion, making these operations explicit rather than hide them behind standard c++ operators makes thecode easier to read.

2. Value/Pointer vs Reference/COW semanticsIn yans, PacketPtrs must be passed around by value. In George’s designPackets must be passed around by reference. yans thus alwaysmaintains an accurate track of its number of users whilein George’s design, there are the "direct" users and the "indirect users" and only the "direct" users are tracked. Thisallows George to implement the "deep copy only if refcount is1" optimization. This also allows George to avoid oneextra deep copy in typical PHY-level broadcast scenarios.

3. packed vs unpacked design

• yans requires every Header added to a Packet to be serializedat this moment. George’s approach does the equiv-alent of a memcpy which is inherently faster. memcpy is approximatively 10% faster than the full serialization.(tests in yans)

• yans requires every Header removed from a Packet to be deserialized at this moment. George’s does the equivalentof a memcpy which is, again, aproximatively 10% faster (tests in yans)

If you take a packed design, you take a hit in performance (as in yans); however, if packets are to be serialized/deserializedand fragmentation handled, performance should be better and the design cleaner with a packed design. These oper-ations are trivial to implement in yans and have obvious semantics: parts of the byte buffer are copied around. InGeorge’s design, these operations are clearly more complicated to implement and their semantics are less obvious sinceGeorge’s packet is fundamentally a list of protocol headersand we are trying to give this discrete list the semantics ofa continuous buffer of bytes. There is no obvious solution here; the answer depends on the expected use cases ofns3.

Similarly, yans design is expected to be easier to use in emulation. In yans, simulation packets are really networkpackets so we can convert from one to the other trivially. In George’s design, it is easy to convert from simulationpackets to network packets (all you have to do is parse the list of Headers and invoke their serialize methods. Althoughhow this would be implemented with Fragmentation/Defragmentation/Repacketization is not obvious) but convertingfrom network packets to simulation packets requires the user to write simulation-specific glue code: he needs to knowthe format of the network packets, parse them, build simulation headers and push them in a simulation packet.

Fundamentally, in George’s design, it is easier to write a simple Header class because you don’t need to provideany serialization and deserialization code if you don’t useemulation or real-world code integration but if you usethem, you need to write simple serialization code and more complex deserialization code (which parses the wholenetwork packet and constructs a simulation packet from the parsed headers). In yans, you always need to provideserialization/deserialization function pairs (these functions can be pretty simple to write) but you get emulation andreal-world code integration for free.

Here, George’s design is easier to use than yans if you don’t use emulation and real-world code but yans is easier to usethan George’s design if you use them.

3.6 Simulation output

This section describes tracing, logging, and statistics gathering. Data collection could be improved over what is inns2. Thebasic capabilities are for traces (out.tr, out.nam) and monitors. The tracing cannot be controlled at a fine granularity(e.g.trace only the foreground traffic). Monitors are primarily associated with queues.

30

Note that there are some third party contributed code frameworks for improving ns-2 in this regard. For example, Akaroa(commercial product). Also, a newer contribution from University of Pisa (http://info.iet.unipi.it/ cng/ns2measure/). Thislatter one might make a nice starting point for us.

3.6.1 Tracing and Logging

Tracing and logging refer to simulation output that shows (with a timestamp) that a particular event occurred. Tracinggenerally refers to packet events, while logging refers to e.g., process log events, although the two are somewhat related.

Some requirements on this have been expressed:

• Ability to generate pcap formatted tracefiles, for use in analyzers such as Ethereal.

• Ability to take (parse) pcap files as input to traffic generators (under discussion on ns mailing list).

• if backward compatibility for nam or ns-2 traces is desired,ability to post-process thens3 trace format to generate .trand .nam files.

yans has added support already for writing pcap files.

In yans, every object which wants to trace something createsone of:

• packet logger (can log packets)

• stream tracer (can log arbitrary strings and data to a stdc++ostream with operator«)

• variable tracer (logs every change of the value of the variable

Every object also needs to report about these trace objects in a TraceContainer whenever the user asks for it. Then, theuser connects callbacks to any of these trace objects, each of which is identified by a unique string. We provide helpercallbacks which automatically log the trace data to files. The pcap output support is implemented as a packet writer that canbe connected to any number of packet loggers. The granularity of this approach is very small, which allows the user to:

• choose only the trace sources it is interested in (for example, ipv4 drops from node A, mac drops from node B and802.11 rx from node C).

• connect arbitrary filtering code to these trace events

• connect default dumping code to these trace events

We are considering moving the yans framework for tracing tons3.

Open issue: Whether trace files have a unique object ID embedded in the trace line itself (e.g. a single trace file for thesimulation), or whether each object generates its own tracefile and the object name is part of the trace file name.

trace format for TraceGraph support

Jaroslaw Malek contributed the TraceGraph4 utility to ns2 and has contributed the following ideas for revising the tracing:

4http://www.tracegraph.com

31

For Matlab processing only, something like Tracegraph format is easy to load. For stream processing like I used in the Traceconverter, new trace format is very good (much easier to parse than old-wireless one). For the fastest processing, the newformat would have to contain the same tags (fields, number of columns) with different/empty values in each line becausechecking a line if it’s correct takes some time (like old wireless format is very messy, new trace format also is not consistentbut much better) and with the same tags there would be no need to check lines.

There should be a few formats:

1. binary - suitable for parsing using streams like in C, java, etc. there should be the same number of fields (also the samefield size for each event would be good), adding a new field witha new format version would be easy to support in theparser

2. text (lines, like new trace format) - suitable for parsingusing awk, perl, etc.

3. XML - object format (1 event have n objects, 1 object have n fields, n events in the file) slower but easier to read,convert and analyse using many tools, e.g. warehouse and statistical applications

4. multi-segment for multi-threaded processing with no need to load the whole file into memory, i.e. statistics could becalculated on the fly during parsing for each segment and finally gathered in one package.

There should be the same format for wired, wireless and otherevents (not like wired-cum-wireless which is too complicatedfor many people as I can see on Trace graph and ns-2 groups). Asfor above formats details I haven’t thought about them yet.:) New format would better require to rewrite Trace graph to obtain correct analysis results (actually it could be converted toTrace graph format but calculations could be wrong - it depends on the new format) so it could inspire somebody (maybe me)to write a new application. If there was a free analysis tool supplied with ns-2 everybody would be happy and there would beno need to create scripts and programs to analyse trace files.

A few more things have come to my mind. Each generated packet should have an unique ID (no distinction between e.g.wired or wireless event, ID should be unique for the whole trace file), so tracing it (e.g. for delays calculations) would beeasy. There should be packet type, e.g. wired (and subcategories), wireless (and subcategories), etc., for each event.Correctsequence numbers would also be good for RTT calculations (itshould be easy to find out which ACK packet corresponds toa sent packet). The new format should be easy not only to parsebut also easy to calculate statistics, so it’s good to think howeasy would be to calculate them (and if the results would be 100% correct) in a fastest way with the new format (and maybeput some example algorithms or just explanations inns3 documentation). Because I’ve made Tracegraph with MATLABIrepresent trace file as a matrix (the same number of columns) and all the calulations are simpler to implement in that way.

There should be no IP addresses in the ns-3 trace format. There should be only node numbers and a mapping file (for example.tr.ip extension), e.g. a line could be like: node_number IP_address.

Note: Global unique IDs break assumptions about emulation and maynot be supported. Having no IP addresses may alsocause emulation and real-world code integration problems.

3.6.2 Statistics objects

Presently,ns2 does not have much support for statistics; there are a few mathematical objects such as Integrator, but this hasnot been a priority forns2. Again, the work at university of Pisa can probably help here.

GTNetS has support for statistics printed out at the end of the simulation run; we should consider bringing in this frameworkto ns3. GTNetS also has capability of gathering statistics on thesimulation run itself (progress).

32

3.7 Memory management and topology construction

In topology construction, Nodes are often created and populated with similar objects. For instance, a newly created Nodeobject may get copies of some "Stuff" object, such as in the below.

int main (){Stuff s;s.set_param1 (0);s.set_param2 (0);

Node *a = new Node ();Stuff *a_s = a->addStuff (&s); // Return a newly-created Stuff

// that is held in the node.// The returned pointer does not// belong to the caller and cannot// be deleted by the caller.

a_s->set_param2 (100); // modify the a-version of stuff.

Node *b = new Node ();s.set_param1 (1);Stuff *b_s = b->addStuff (&s);b_s->set_param2 (2); // modify the b-version of stuff.

// end of simulationdelete a;delete b;}

Now, a possibly annoying feature of the above is that we are using Stuff objects in two different ways; a proto-Stuff thatis copy-constructed in the addStuff() method, and the Stuffobjects belonging to the Nodes. It might be cleaner instead todecouple the objects used to construct the topology from theobjects used in the topology itself, by creation of a StuffFactoryand NodeFactory class, as shown below. Further, the use of smart pointers for every object could avoid the manual deletionof all objects created earlier:

int main (){NodeFactory f_node;StuffFactory f_stuff;f_stuff.set_param1 (10);f_stuff.set_param2 (1);

ptr<Node *> node = f_node.create ();ptr<Stuff *> stuff = node->add_stuff (f_stuff);stuff->set_param1 (0);

node = f_node.create ();stuff = node->add_stuff (f_stuff);stuff->set_param2 (50);}

33

3.8 Animation and GUI

GTNetS has nicer animation capabilities (Qt-based) than doesns2, which just post-processes an out.nam file; gtnets integratessupport for a GUI into the simulator. yans does not have a GUI.

Open issue:How (and with what priority in our development schedule) to handle animation and GUI issues.

34

Chapter 4

Acknowledgments

The following people have contributed to this document, viadirect text contribution, comments, or mailing list posts:

• Kevin Fall

• Tom Henderson

• Mathieu Lacage

• Jaroslaw Malek

• Steve McCanne

• George Riley

35