Process Migration - Virginia Techcourses.cs.vt.edu/.../fall05-kafura/Papers/Migration/ProcessMigratio… · Process Migration 243 of the potentials migration can offer to different

Process Migration

DEJAN S. MILOJICIC

HP Labs

FRED DOUGLIS

AT&T Labs–Research

YVES PAINDAVEINE

TOG Research Institute

RICHARD WHEELER

EMC

AND

SONGNIAN ZHOU

University of Toronto and Platform Computing

Process migration is the act of transferring a process between two machines. It enablesdynamic load distribution, fault resilience, eased system administration, and dataaccess locality. Despite these goals and ongoing research efforts, migration has notachieved widespread use. With the increasing deployment of distributed systems ingeneral, and distributed operating systems in particular, process migration is againreceiving more attention in both research and product development. Ashigh-performance facilities shift from supercomputers to networks of workstations, andwith the ever-increasing role of the World Wide Web, we expect migration to play amore important role and eventually to be widely adopted.

This survey reviews the field of process migration by summarizing the key conceptsand giving an overview of the most important implementations. Design andimplementation issues of process migration are analyzed in general, and then revisitedfor each of the case studies described: MOSIX, Sprite, Mach, and Load Sharing Facility.The benefits and drawbacks of process migration depend on the details ofimplementation and, therefore, this paper focuses on practical matters. This survey willhelp in understanding the potentials of process migration and why it has not caught on.

Categories and Subject Descriptors: C.2.4 [Computer-Communication Networks]:Distributed Systems—network operating systems; D.4.7 [Operating Systems]:Organization and Design—distributed systems; D.4.8 [Operating Systems]:Performance—measurements; D.4.2 [Operating Systems]: Storage Management—distributed memories

General Terms: Design, Experimentation

Additional Key Words and Phrases: Process migration, distributed systems, distributedoperating systems, load distribution

Permission to make digital/hard copy of part or all of this work for personal or classroom use is grantedwithout fee provided that the copies are not made or distributed for profit or commercial advantage, thecopyright notice, the title of the publication, and its date appear, and notice is given that copying is bypermission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists,requires prior specific permission and/or a fee.c©2001 ACM 0360-0300/01/0900-0241 $5.00

ACM Computing Surveys, Vol. 32, No. 3, September 2000, pp. 241–299.

242 D. S. Milojicic et al.

1. INTRODUCTIONOrganization of the Paper

2. BACKGROUND2.1. Terminology2.2. Target Architectures2.3. Goals2.4. Application Taxonomy2.5. Migration Algorithm2.6. System Requirements for Migration2.7. Load Information Management2.8. Distributed Scheduling2.9. Alternatives to Process Migration

3. CHARACTERISTICS3.1. Complexity and Operating

System Support3.2. Performance3.3. Transparency3.4. Fault Resilience3.5. Scalability3.6. Heterogeneity3.7. Summary

4. EXAMPLES4.1. Early Work4.2. Transparent Migration in

UNIX-like Systems4.3. OS with Message-Passing Interface4.4. Microkernels4.5. User-space Migrations4.6. Application-specific Migration4.7. Mobile Objects4.8. Mobile Agents

5. CASE STUDIES5.1. MOSIX5.2. Sprite5.3. Mach5.4. LSF

6. COMPARISON7. WHY PROCESS MIGRATION

HAS NOT CAUGHT ON7.1. Case Analysis7.2. Misconceptions7.3. True Barriers to Migration Adoption7.4. How these Barriers Might be Overcome

8. SUMMARY AND FURTHER RESEARCHACKNOWLEDGMENTSREFERENCES

1. INTRODUCTION

A process is an operating system ab-straction representing an instance of arunning computer program. Process mi-gration is the act of transferring a pro-

cess between two machines during itsexecution. Several implementations havebeen built for different operating systems,including MOSIX [Barak and Litman,1985], V [Cheriton, 1988], Accent [Rashidand Robertson, 1981], Sprite [Ousterhoutet al., 1988], Mach [Accetta et al., 1986],and OSF/1 AD TNC [Zajcew et al., 1993].In addition, some systems provide mech-anisms that checkpoint active processesand resume their execution in essentiallythe same state on another machine, in-cluding Condor [Litzkow et al., 1988] andLoad Sharing Facility (LSF) [Zhou et al.,1994]. Process migration enables:r dynamic load distribution, by mi-

grating processes from overloaded nodesto less loaded ones,r fault resilience, by migrating pro-cesses from nodes that may have expe-rienced a partial failure,r improved system administration, bymigrating processes from the nodes thatare about to be shut down or otherwisemade unavailable, andr data access locality, by migrating pro-cesses closer to the source of some data.

Despite these goals and ongoing re-search efforts, migration has not achievedwidespread use. One reason for this is thecomplexity of adding transparent migra-tion to systems originally designed to runstand-alone, since designing new systemswith migration in mind from the begin-ning is not a realistic option anymore. An-other reason is that there has not been acompelling commercial argument for op-erating system vendors to support processmigration. Checkpoint-restart approachesoffer a compromise here, since they canrun on more loosely-coupled systems byrestricting the types of processes that canmigrate.

In spite of these barriers, process mi-gration continues to attract research. Webelieve that the main reason is the po-tentials offered by mobility as well asthe attraction to hard problems, so in-herent to the research community. Therehave been many different goals andapproaches to process migration because

ACM Computing Surveys, Vol. 32, No. 3, September 2000.

Process Migration 243

of the potentials migration can offer todifferent applications (see Section 2.3 ongoals, Section 4 on approaches, and Sec-tion 2.4 on applications).

With the increasing deployment of dis-tributed systems in general, and dis-tributed operating systems in particular,the interest in process migration is againon the rise both in research and in prod-uct development. As high-performance fa-cilities shift from supercomputers to Net-works of Workstations (NOW) [Andersonet al., 1995] and large-scale distributedsystems, we expect migration to play amore important role and eventually gainwider acceptance.

Operating systems developers in in-dustry have considered supporting pro-cess migration, for example Solaris MC[Khalidi et al., 1996], but thus far theavailability of process migration in com-mercial systems is non-existent as wedescribe below. Checkpoint-restart sys-tems are becoming increasingly deployedfor long-running jobs. Finally, techniquesoriginally developed for process migrationhave been employed in developing mobileagents on the World Wide Web. Recent in-terpreted programming languages, suchas Java [Gosling et al., 1996], Telescript[White, 1996] and Tcl/Tk [Ousterhout,1994] provide additional support for agentmobility.

There exist a few books that discussprocess migration [Goscinski, 1991; Baraket al., 1993; Singhal and Shivaratri, 1994;Milojicic et al., 1999]; a number of sur-veys [Smith, 1988; Eskicioglu, 1990; Nut-tal, 1994], though none as detailed asthis survey; and Ph.D. theses that dealdirectly with migration [Theimer et al.,1985; Zayas, 1987a; Lu, 1988; Douglis,1990; Philippe, 1993; Milojicic, 1993c; Zhu,1992; Roush, 1995], or that are relatedto migration [Dannenberg, 1982; Nichols,1990; Tracey, 1991; Chapin, 1993; Knabe,1995; Jacqmot, 1996].

This survey reviews the field of pro-cess migration by summarizing the keyconcepts and describing the most impor-tant implementations. Design and im-plementation issues of process migrationare analyzed in general and then re-

visited for each of the case studies de-scribed: MOSIX, Sprite, Mach, and LSF.The benefits and drawbacks of process mi-gration depend on the details of implemen-tation and therefore this paper focuseson practical matters. In this paper weaddress mainly process migration mech-anisms. Process migration policies, suchas load information management and dis-tributed scheduling, are mentioned to theextent that they affect the systems be-ing discussed. More detailed descriptionsof policies have been reported elsewhere(e.g., Chapin’s survey [1996]).

This survey will help in understand-ing the potential of process migration. Itattempts to demonstrate how and whymigration may be widely deployed. Weassume that the reader has a generalknowledge of operating systems.

Organization of the Paper

The paper is organized as follows. Sec-tion 2 provides background on process mi-gration. Section 3 describes the processmigration by surveying its main charac-teristics: complexity, performance, trans-parency, fault resilience, scalability andheterogeneity. Section 4 classifies vari-ous implementations of process migrationmechanisms and then describes a coupleof representatives for each class. Section 5describes four case studies of process mi-gration in more detail. In Section 6 wecompare the process migration implemen-tations presented earlier. In Section 7 wediscuss why we believe that process migra-tion has not caught on so far. In the lastsection we summarize the paper and de-scribe opportunities for further research.

2. BACKGROUND

This section gives some background onprocess migration by providing an over-view of process migration terminology,target architectures, goals, applicationtaxonomy, migration algorithms, systemrequirements, load information manage-ment, distributed scheduling, and alterna-tives to migration.



Fig. 1. High Level View of Process Migration.Process migration consists of extracting the state ofthe process on the source node, transferring it to thedestination node where a new instance of the processis created, and updating the connections with otherprocesses on communicating nodes.

2.1. Terminology

A process is a key concept in operatingsystems [Tanenbaum, 1992]. It consists ofdata, a stack, register contents, and thestate specific to the underlying OperatingSystem (OS), such as parameters relatedto process, memory, and file management.A process can have one or more threadsof control. Threads, also called lightweightprocesses, consist of their own stack andregister contents, but share a process’s ad-dress space and some of the operating-system-specific state, such as signals. Thetask concept was introduced as a gener-alization of the process concept, wherebya process is decoupled into a task and anumber of threads. A traditional processis represented by a task with one threadof control.

Process migration is the act of trans-ferring a process between two machines(the source and the destination node) dur-ing its execution. Some architectures alsodefine a host or home node, which is thenode where the process logically runs. Ahigh-level view of process migration isshown in Figure 1. The transferred stateincludes the process’s address space, exe-cution point (register contents), communi-cation state (e.g., open files and messagechannels) and other operating system de-pendent state. Task migration representstransferring a task between two machinesduring execution of its threads.

During migration, two instances of themigrating process exist: the source in-stance is the original process, and the

Fig. 2. Taxonomy of Mobility.

destination instance is the new processcreated on the destination node. After mi-gration, the destination instance becomesa migrated process. In systems with ahome node, a process that is running onother machines may be called a remoteprocess (from the perspective of the homenode) or a foreign process (from the per-spective of the hosting node).

Remote invocation is the creation of aprocess on a remote node. Remote invo-cation is usually a less “expensive” opera-tion than process migration. Although theoperation can involve the transfer of somestate, such as code or open files, the con-tents of the address space need not betransferred.

Generally speaking, mobility can beclassified into hardware and software mo-bility, as described in Figure 2. Hardwaremobility deals with mobile computing,such as with limitations on the connectiv-ity of mobile computers and mobile IP (see[Milojicic et al., 1999] for more details). Afew techniques in mobile computing havean analogy in software mobility, such assecurity, locating, naming, and communi-cation forwarding. Software mobility canbe classified into the mobility of passivedata and active data. Passive data rep-resents traditional means of transferringdata between computers; it has been em-ployed ever since the first two comput-ers were connected. Active data can befurther classified into mobile code, pro-cess migration and mobile agents. Thesethree classes represent incremental evo-lution of state transfer. Mobile code, suchas Java applets, transfers only code be-tween nodes. Process migration, which isthe main theme of this paper, deals pri-marily with code and data transfer. It also



deals with the transfer of authority, forinstance access to a shared file system,but in a limited way: authority is underthe control of a single administrative do-main. Finally, mobile agents transfer code,data, and especially authority to act onthe owner’s behalf on a wide scale, suchas within the entire Internet.

2.2. Target Architectures

Process migration research started withthe appearance of distributed processingamong multiple processors. Process mi-gration introduces opportunities for shar-ing processing power and other resources,such as memory and communication chan-nels. It is addressed in early multipro-cessor systems [Stone, 1978; Bokhari,1979]. Current multiprocessor systems,especially symmetric multiprocessors, arescheduled using traditional schedulingmethods. They are not used as an envi-ronment for process migration research.

Process migration in NUMA (Non-Uniform Memory Access) multiprocessorarchitectures is still an active area of re-search [Gait, 1990; Squillante and Nelson,1991; Vaswani and Zahorjan, 1991; Nelsonand Squillante, 1995]. The NUMA archi-tectures have a different access time to thememory of the local processor, comparedto the memory of a remote processor, or toa global memory. The access time to thememory of a remote processor can be vari-able, depending on the type of intercon-nect and the distance to the remote pro-cessor. Migration in NUMA architecturesis heavily dependent on the memory foot-print that processes have, both in memoryand in caches. Recent research on virtualmachines on scalable shared memory mul-tiprocessors [Bugnion, et al., 1997] rep-resents another potential for migration.Migration of whole virtual machines be-tween processors of a multiprocessor ab-stracts away most of the complexities ofoperating systems, reducing the migrate-able state only to memory and to statecontained in a virtual monitor [Teodosiu,2000]. Therefore, migration is easier to im-plement if there is a notion of a virtualmachine.

Massively Parallel Processors (MPP)are another type of architecture usedfor migration research [Tritscher andBemmerl, 1992; Zajcew et al., 1993]. MPPmachines have a large number of pro-cessors that are usually shared betweenmultiple users by providing each of themwith a subset, or partition, of the pro-cessors. After a user relinquishes a par-tition, it can be reused by another user.MPP computers are typically of a NORMA(NO Remote Memory Access) type, i.e.,there is no remote memory access. Inthat respect they are similar to net-work clusters, except they have a muchfaster interconnect. Migration representsa convenient tool to achieve repartition-ing. Since MPP machines have a largenumber of processors, the probability offailure is also larger. Migrating a runningprocess from a partially failed node, for ex-ample after a bank of memory unrelated tothe process fails, allows the process to con-tinue running safely. MPP machines alsouse migration for load distribution, suchas the psched daemon on Cray T3E, orLoadleveler on IBM SP2 machines.

Since its inception, a Local Area Net-work (LAN) of computers has been themost frequently used architecture for pro-cess migration. The bulk of the systems de-scribed in this paper, including all of thecase studies, are implemented on LANs.Systems such as NOW [Anderson et al.,1995] or Solaris [Khalidi et al., 1996] haverecently investigated process migrationusing clusters of workstations on LANs.It was observed that at any point in timemany autonomous workstations on a LANare unused, offering potential for otherusers based on process migration [Mutkaand Livny, 1987]. There is, however, a so-ciological aspect to the autonomous work-station model. Users are not willing toshare their computers with others if thismeans affecting their own performance[Douglis and Ousterhout, 1991]. The pri-ority of the incoming processes (process-ing, VM, IPC priorities) may be reducedin order to allow for minimal impacton the workstation’s owner [Douglis andOusterhout, 1991; Krueger and Chawla,1991].



Most recently, wide-area networks havepresented a huge potential for migration.The evolution of the Web has significantlyimproved the relevance and the opportu-nities for using a wide-area network fordistributed computing. This has resultedin the appearance of mobile agents, en-tities that freely roam the network andrepresent the user in conducting his tasks.Mobile agents can either appear on the In-ternet [Johansen et al., 1995] or in closednetworks, as in the original version ofTelescript [White, 1996].

2.3. Goals

The goals of process migration are closelytied with the type of applications that usemigration, as described in next section.The goals of process migration include:

Accessing more processing poweris a goal of migration when it is usedfor load distribution. Migration is partic-ularly important in the receiver-initiateddistributed scheduling algorithms, wherea lightly loaded node announces its avail-ability and initiates process migrationfrom an overloaded node. This was thegoal of many systems described in this sur-vey, such as Locus [Walker et al., 1983],MOSIX [Barak and Shiloh, 1985], andMach [Milojicic et al., 1993a]. Load distri-bution also depends on load informationmanagement and distributed scheduling(see Sections 2.7 and 2.8). A variationof this goal is harnessing the computingpower of temporarily free workstations inlarge clusters. In this case, process mi-gration is used to evict processes uponthe owner’s return, such as in the case ofSprite (see Section 5.2).

Exploitation of resource locality isa goal of migration in cases when it ismore efficient to access resources locallythan remotely. Moving a process to an-other end of a communication channeltransforms remote communication to lo-cal and thereby significantly improves per-formance. It is also possible that the re-source is not remotely accessible, as in thecase when there are different semanticsfor local and remote accesses. Examplesinclude work by Jul [1989], Milojicic et al.[1993], and Miller and Presotto [1981].

Resource sharing is enabled by mi-gration to a specific node with a specialhardware device, large amounts of freememory, or some other unique resource.Examples include NOW [Anderson et al.,1995] for utilizing memory of remotenodes, and the use of parallel make inSprite [Douglis and Ousterhout, 1991] andwork by Skordos [1995] for utilizing un-used workstations.

Fault resilience is improved by migra-tion from a partially failed node, or in thecase of long-running applications whenfailures of different kinds (network, de-vices) are probable [Chu et al., 1980]. Inthis context, migration can be used in com-bination with checkpointing, such as inCondor [Litzkow and Solomon, 1992] orUtopia [Zhou et al., 1994]. Large-scale sys-tems where there is a likelihood that someof the systems can fail can also benefitfrom migration, such as in Hive [Chapinet al., 1995] and OSF/1 AD TNC [Zajcewet al., 1993].

System administration is simplifiedif long-running computations can be tem-porarily transferred to other machines.For example, an application could mi-grate from a node that will be shutdown,and then migrate back after the node isbrought back up. Another example is therepartitioning of large machines, such asin the OSF/1 AD TNC Paragon configura-tion [Zajcew et al., 1993].

Mobile computing also increases thedemand for migration. Users may want tomigrate running applications from a hostto their mobile computer as they connectto a network at their current location orback again when they disconnect [Bharatand Cardelli, 1995].

2.4. Application Taxonomy

The type of applications that can benefitfrom process migration include:

Parallelizable applications can bestarted on certain nodes, and then mi-grated at the application level or by asystem-wide migration facility in responseto things like load balancing consider-ations. Parallel Virtual Machine (PVM)[Beguelin et al., 1993] is an exampleof application-level support for parallel



invocation and interprocess communi-cation, while Migratory PVM (MPVM)[Casas et al., 1995] extends PVM to al-low instances of a parallel applicationto migrate among nodes. Some otherapplications are inherently paralleliz-able, such as the make tool [Baalbergen,1988]. For example, Sprite provides amigration-aware parallel make utilitythat distributes a compilation acrossseveral nodes [Douglis and Ousterhout,1991]. Certain processor-bound applica-tions, such as scientific computations, canbe parallelized and executed on multi-ple nodes. An example includes work bySkordos [1995], where an acoustic appli-cation is parallelized and executed on acluster of workstations. Applications thatperform I/O and other nonidempotent op-erations are better suited to a system-wideremote execution facility that provides lo-cation transparency and, if possible, pre-emptive migration.

Long-running applications, whichcan run for days or even weeks, cansuffer various interruptions, for examplepartial node failures or administrativeshutdowns. Process migration can relo-cate these applications transparently toprevent interruption. Examples of suchsystems include work by Freedman [1991]and MPVM [Casas et al., 1995]. Migra-tion can also be supported at the appli-cation level [Zhou et al., 1994] by pro-viding a checkpoint/restart mechanismwhich the application can invoke periodi-cally or upon notification of an impendinginterruption.

Generic multiuser workloads, forexample the random job mix that anundergraduate computer laboratory pro-duces, can benefit greatly from process mi-gration. As users come and go, the load onindividual nodes varies widely. Dynamicprocess migration [Barak and Wheeler,1989, Douglis and Ousterhout, 1991] canautomatically spread processes across allnodes, including those applications thatare not enhanced to exploit the migrationmechanism.

An individual generic application,which is preemptable, can be used withvarious goals in mind (see Section 2.3).

Such an application can either migrate it-self, or it can be migrated by another au-thority. This type of application is mostcommon in various systems described inSection 4 and in the case studies describedin Section 5. Note that it is difficult toselect such applications without detailedknowledge of past behavior, since manyapplications are short-lived and do not ex-ecute long enough to justify the overheadof migration (see Section 2.7).

Migration-aware applications areapplications that have been coded toexplicitly take advantage of processmigration. Dynamic process migration canautomatically redistribute these relatedprocesses if the load becomes uneven ondifferent nodes, e.g. if processes are dy-namically created, or there are many moreprocesses than nodes. Work by Skordos[1995], Freedman [1991] and Cardelli[1995] represent this class of application.They are described in more detail in Sec-tion 4.6.

Network applications are the mostrecent example of the potential use of mi-gration: for instance, mobile agents andmobile objects (see Sections 4.7 and 4.8).These applications are designed with mo-bility in mind. Although this mobility dif-fers significantly from the kinds of “pro-cess migration” considered elsewhere inthis paper, it uses some of the same tech-niques: location policies, checkpointing,transparency, and locating and communi-cating with a mobile entity.

2.5. Migration Algorithm

Although there are many different migra-tion implementations and designs, most ofthem can be summarized in the followingsteps (see also Figure 3):

1. A migration request is issued to aremote node. After negotiation, mi-gration has been accepted.

2. A process is detached from itssource node by suspending its execu-tion, declaring it to be in a migratingstate, and temporarily redirecting com-munication as described in the follow-ing step.



Fig. 3. Migration Algorithm. Many details have been simplified, such as user v. kernel migration, whenis process actually suspended, when is the state transferred, how are message transferred, etc. These detailsvary subject to particular implementation.

3. Communication is temporarilyredirected by queuing up arrivingmessages directed to the migratedprocess, and by delivering them tothe process after migration. This stepcontinues in parallel with steps 4,5, and 6, as long as there are addi-tional incoming messages. Once thecommunication channels are enabledafter migration (as a result of step 7),the migrated process is known to theexternal world.

4. The process state is extracted,including memory contents; proces-sor state (register contents); commu-nication state (e.g., opened files and

message channels); and relevant ker-nel context. The communication stateand kernel context are OS-dependent.Some of the local OS internal state isnot transferable. The process state istypically retained on the source nodeuntil the end of migration, and in somesystems it remains there even after mi-gration completes. Processor dependen-cies, such as register and stack con-tents, have to be eliminated in the caseof heterogeneous migration.

5. A destination process instance iscreated into which the transferredstate will be imported. A destination in-stance is not activated until a sufficient



amount of state has been transferredfrom the source process instance. Af-ter that, the destination instance willbe promoted into a regular process.

6. State is transferred and importedinto a new instance on the remotenode. Not all of the state needs to betransferred; some of the state could belazily brought over after migration iscompleted (see lazy evaluation in Sec-tion 3.2).

7. Some means of forwarding refer-ences to the migrated process must bemaintained. This is required in orderto communicate with the process or tocontrol it. It can be achieved by regis-tering the current location at the homenode (e.g. in Sprite), by searching for themigrated process (e.g. in the V Kernel,at the communication protocol level),or by forwarding messages across allvisited nodes (e.g. in Charlotte). Thisstep also enables migrated communica-tion channels at the destination and itends step 3 as communication is perma-nently redirected.

8. The new instance is resumed whensufficient state has been transferredand imported. With this step, processmigration completes. Once all of thestate has been transferred from theoriginal instance, it may be deleted onthe source node.

2.6. System Requirements for Migration

To support migration effectively, a systemshould provide the following types of func-tionality:r Exporting/importing the process

state. The system must provide sometype of export/import interfaces thatallow the process migration mechanismto extract a process’s state from thesource node and import this state on thedestination node. These interfaces maybe provided by the underlying operatingsystem, the programming language,or other elements of the programmingenvironment that the process has accessto. State includes processor registers,process address space and communi-

cation state, such as open messagechannels in the case of message-basedsystems, or open files and signal masksin the case of UNIX-like systems.r Naming/accessing the process andits resources. After migration, the mi-grated process should be accessible bythe same name and mechanisms asif migration never occurred. The sameapplies to process’s resources, such asthreads, communication channels, filesand devices. During migration, access toa process and/or some of its resourcescan be temporarily suspended. Varyingdegrees of transparency are achieved innaming and accessing resources duringand after migration (see Section 3.3).r Cleaning up the process’s non-migratable state. Frequently, the mi-grated process has associated systemstate that is not migratable (examplesinclude a local process identifier, pid, andthe local time; a local pid is relevant onlyto the local OS, and every host may havea slightly different value for the localtime—something that may or may notmatter to a migrating process). Migra-tion must wait until the process finishesor aborts any pending system operation.If the operation can be arbitrarily long, itis typically aborted and restarted on thedestination node. For example, migra-tion can wait for the completion of localfile operations or local device requeststhat are guaranteed to return in a lim-ited time frame. Waiting for a message oraccessing a remote device are examplesof operations that need to be aborted andrestarted on the remote node. Processesthat cannot have their non-migrateablestate cleaned cannot be considered formigration.

2.7. Load Information Management

The local processes and the resources oflocal and remote nodes have to be char-acterized, in order to select a process formigration and a destination node, as wellas to justify migration. This task is com-monly known as load information manage-ment. Load information is collected andpassed to a distributed scheduling policy



Fig. 4. Load Information Management Mod-ule collects load information on the local nodeand disseminates it among the nodes. DistributedScheduling instructs the migration mechanismwhen, where, and which process to migrate.

(see Figure 4). Load information manage-ment is concerned with the following threequestions:

What is load information and howis it represented? The node load is typi-cally represented by one or more of the fol-lowing load indices: utilization of the CPU,the length of the queue of processes wait-ing to be executed, the stretch factor (ra-tio between turnaround- and execution-time-submission to completion v. start tocompletion) [Ferrari and Zhou 1986], thenumber of running processes, the num-ber of background processes, paging, com-munication [Milojicic, 1993c], disk uti-lization, and the interrupt rate [Hwanget al., 1982]. A process load is typicallycharacterized by process lifetime, CPU us-age, memory consumption (virtual andphysical), file usage [Hac, 1989a], commu-nication [Lo, 1989], and paging [Milojicic,1993c]. Kuntz uses a combination of work-load descriptions for distributed schedul-ing [Kunz, 1991]. The application type isconsidered in Cedar [Hagmann, 1986].

When are load information col-lection and dissemination activated?These can be periodic or event-based. Atypical period is in the range of 1 secondor longer, while typical events are pro-cess creation, termination, or migration.The frequency of information dissemina-tion is usually lower than the frequency ofinformation collection, i.e. it is averagedover time in order to prevent instability[Casavant and Kuhl, 1988b]. It also de-pends on the costs involved with dissemi-

nation and the costs of process migration.The lower the costs, the shorter the periodcan be; the higher the costs, less frequentlyload information is disseminated.

How much information should betransferred? It can be the entire state,but typically only a subset is transferredin order to minimize the transfer costs andhave a scalable solution. In large systems,approximations are applied. For example,only a subset of the information might betransferred, or it might be derived fromthe subset of all nodes [Barak and Shiloh,1985; Alon et al., 1987; Han and Finkel,1988; Chapin and Spafford, 1994].

There are two important observationsderived from the research in load infor-mation management. The first one is thatjust a small amount of information canlead to substantial performance improve-ments. This observation is related to loaddistribution in general, but it also appliesto process migration. Eager et al. wereamong the first to argue that load shar-ing using minimal load information cangain dramatic improvements in perfor-mance over the non-load-sharing case, andperform nearly as well as more complexpolicies using more information [Eageret al., 1986b]. The minimal load informa-tion they use consists of the process queuelength of a small number of successivelyprobed remote nodes. A small amount ofstate also reduces communication over-head. Kunz comes to the same conclusionusing the concept of stochastic learningautomata to implement a task scheduler[Kunz, 1991].

The second observation is that the cur-rent lifetime of a process can be usedfor load distribution purposes. The issueis to find how old the process needs tobe before it is worth to migrate it. Costsinvolved with migrating short-lived pro-cesses can outweigh the benefits. Lelandand Ott were the first to account for theprocess age in the balancing policy [1986].Cabrera finds that it is possible to predicta process’s expected lifetime from how longit has already lived [Cabrera, 1986]. Thisjustifies migrating processes that manageto live to a certain age. In particular, hefinds that over 40% of processes doubled



their age. He also finds that the mostUNIX processes are short-lived, more than78% of the observed processes have a life-time shorter than 1s and 97% shorterthan 4s.

Harchol-Balter and Downey explorethe correlation between process lifetimeand acceptable migration costs [Harchol-Balter and Downey, 1997]. They derivea more accurate form of the process life-time distribution that allows them to pre-dict the life-time correlated to the pro-cess age and to derive a cost criterionfor migration. Svensson filters out short-running processes by relying on statistics[Svensson, 1990], whereas Wang et al. de-ploy AI theory for the same purpose [Wanget al., 1993].

2.8. Distributed Scheduling

This section addresses distributed sched-uling closely related to process migrationmechanisms. General surveys are pre-sented elsewhere [Wang and Morris, 1985;Casavant and Kuhl, 1988a; Hac, 1989b;Goscinski, 1991; Chapin, 1996].

Distributed scheduling uses the infor-mation provided by the load informa-tion management module to make migra-tion decisions, as described in Figure 4.The main goal is to determine when tomigrate which process where. The acti-vation policy provides the answer to thequestion when to migrate. Scheduling isactivated periodically or it is event-driven.After activation, the load is inspected, andif it is above/below a threshold, actionsare undertaken according to the selectedstrategy. The selection policy answers thequestion which process to migrate. Theprocesses are inspected and some of themare selected for migration according to thespecified criteria. Where to migrate de-pends on the location policy algorithm,which chooses a remote node based on theavailable information.

There are a few well-known classes ofdistributed scheduling policies:r A sender-initiated policy is activated

on the node that is overloaded and thatwishes to off-load to other nodes. Asender-initiated policy is preferable for

low and medium loaded systems, whichhave a few overloaded nodes. This strat-egy is convenient for remote invocationstrategies [Eager et al., 1986a; Kruegerand Livny, 1987b; Agrawal and Ezzat,1987].r A receiver-initiated policy is acti-vated on underloaded nodes willing toaccept the load from overloaded ones.A receiver-initiated policy is preferablefor high load systems, with many over-loaded nodes and few underloaded ones.Process migration is particularly well-suited for this strategy, since only withmigration can one initiate process trans-fer at an arbitrary point in time [Bryantand Finkel, 1981; Eager et al., 1986a;Krueger and Livny, 1988].r A symmetric policy is the combina-tion of the previous two policies, in anattempt to take advantage of the goodcharacteristics of both of them. It is suit-able for a broader range of conditionsthan either receiver-initiated or sender-initiated strategies alone [Krueger andLivny, 1987b; Shivaratri et al., 1992].r A random policy chooses the destina-tion node randomly from all nodes in adistributed system. This simple strategycan result in a significant performanceimprovement [Alon et al., 1987; Eageret al., 1986b; Kunz, 1991].

The following are some of the issues indistributed scheduling related to the pro-cess migration mechanism:r Adaptability is concerned with the

scheduling impact on system behavior[Stankovic, 1984]. Based on the cur-rent host and network load, the rela-tive importance of load parameters maychange. The policy should adapt to thesechanges. Process migration is inherentlyadaptable because it allows processes torun prior to dispatching them to othernodes, giving them a chance to adapt. Mi-gration can happen at any time (therebyadapting to sudden load changes),whereas initial placement happens onlyprior to starting a process. Examples ofadaptive load distribution include workby Agrawal and Ezzat [1987], Krueger



and Livny [1988], Concepcion andEleazar [1988], Efe and Groselj [1989],Venkatesh and Dattatreya [1990], Shiv-aratri and Krueger [1990], and Mehraand Wah [1992].r Stability is defined as the ability to de-tect when the effects of further actions(e.g. load scheduling or paging) will notimprove the system state as defined bya user’s objective [Casavant and Kuhl,1988b]. Due to the distributed state,some instability is inevitable, since itis impossible to transfer state changesacross the system instantly. However,high levels of instability should beavoided. In some cases it is advisablenot to perform any action, e.g. under ex-tremely high loads it is better to aban-don load distribution entirely. Processmigration can negatively affect stabilityif processes are migrated back and forthamong the nodes, similar to the thrash-ing introduced by paging [Denning,1980]. To prevent such behavior a limiton the number of migrations can be im-posed. Bryant and Finkel demonstratehow process migration can improve sta-bility [Bryant and Finkel, 1981].r Approximate and heuristic sched-uling is necessary since optimal solu-tions are hard to achieve. Suboptimalsolutions are reached either by approx-imating the search space with its subsetor by using heuristics. Some of theexamples of approximate and heuristicscheduling include work by Efe [1982],Leland and Ott [1986], Lo [1988],Casavant and Kuhl [1988a], and Xu andHwang [1990]. Deploying process migra-tion introduces more determinism andrequires fewer heuristics than alterna-tive load distribution mechanisms. Evenwhen incorrect migration decisions aremade, they can be alleviated by subse-quent migrations, which is not the casewith initial process placement whereprocesses have to execute on the samenode until the end of its lifetime.r Hierarchical scheduling integratesdistributed and centralized scheduling.It supports distributed schedulingwithin a group of nodes and centralized

scheduling among the groups. This areahas attracted much research [Bowenet al., 1988; Bonomi and Kumar, 1988;Feitelson and Rudolph, 1990; Guptaand Gopinath, 1990; Gopinath andGupta, 1991; Chapin, 1995]. A processmigration mechanism is a good fit forhierarchical scheduling since processesare typically migrated within a LAN orother smaller domain. Only in the caseof large load discrepancies are processesmigrated between domains, i.e. betweenpeers at higher levels of the hierarchy.

The most important question that dis-tributed scheduling studies address re-lated to process migration is whether mi-gration pays off. The answer dependsheavily on the assumptions made. For ex-ample, Eager et al. compare the receiver-and sender-initiated policies [Eager et al.,1986a], and show that the sender-initiatedpolicies outperform the receiver-initiatedpolicies for light and moderate systemloads. The receiver-initiated policy is bet-ter for higher loads, assuming that trans-fer costs are same. They argue that thetransfer costs for the receiver policy, thatrequires some kind of migration, are muchhigher than the costs for mechanismsfor the sender-initiated strategies, whereinitial placement suffices. They finallyconclude that under no condition could mi-gration provide significantly better perfor-mance than initial placement [Eager et al.,1988].

Krueger and Livny investigate the rela-tionship between load balancing and loadsharing [Krueger and Livny, 1988]. Theyargue that load balancing and load shar-ing represent various points in a contin-uum defined by a set of goals and load con-ditions [Krueger and Livny, 1987]. Theyclaim that the work of Eager et al. [1988]is only valid for a part of the contin-uum, but it cannot be adopted generally.Based on better job distributions thanthose used by Eager et al., their simulationresults show that migration can improveperformance.

Harchol-Balter and Downey present themost recent results on the benefits of us-ing process migration [Harchol-Balter and



Downey, 1997]. They use the measureddistribution of process lifetimes for a va-riety of workloads in an academic envi-ronment. The crucial point of their workis understanding the correct lifetime dis-tribution, which they find to be Pareto(heavy-tailed). Based on the trace-drivensimulation, they demonstrate a 35-50%improvement in the mean delay when us-ing process migration instead of remoteexecution (preemptive v. non-preemptivescheduling) even when the costs of migra-tion are high.

Their work differs from Eager et al.[1988] in system model and workload de-scription. Eager et al. model server farms,where the benefits of remote executionare overestimated: there are no associatedcosts and no affinity toward a particularnode. Harchol-Balter and Downey model anetwork of workstations where remote ex-ecution entails costs, and there exists anaffinity toward some of the nodes in a dis-tributed system. The workload that Eageret al. use contains few jobs with non-zerolife-times, resulting in a system with lit-tle imbalance and little need for processmigration.

2.9. Alternatives to Process Migration

Given the relative complexity of im-plementation, and the expense incurredwhen process migration is invoked, re-searchers often choose to implement al-ternative mechanisms [Shivaratri et al.,1992; Kremien and Kramer, 1992].

Remote execution is the most fre-quently used alternative to process migra-tion. Remote execution can be as simpleas the invocation of some code on a re-mote node, or it can involve transferringthe code to the remote node and inherit-ing some of the process environment, suchas variables and opened files. Remote exe-cution is usually faster than migration be-cause it does not incur the cost of trans-ferring a potentially large process state(such as the address space, which is cre-ated anew in the case of remote execu-tion). For small address spaces, the costsfor remote execution and migration canbe similar. Remote execution is used inmany systems such as COCANET [Rowe

and Birman, 1982], Nest [Agrawal andEzzat, 1987], Sprite [Ousterhout et al.,1988], Plan 9 [Pike et al., 1990], Amoeba[Mullender et al., 1990], Drums [Bond,1993], Utopia [Zhou et al., 1994], and Hive[Chapin et al., 1995].

Remote execution has disadvantages aswell. It allows creation of the remote in-stance only at the time of process creation,as opposed to process migration which al-lows moving the process at an arbitrarytime. Allowing a process to run on thesource node for some period of time is ad-vantageous in some respects. This way,short-lived processes that are not worthmigrating are naturally filtered out. Also,the longer a process runs, the more in-formation about its behavior is available,such as whether and with whom it commu-nicates. Based on this additional informa-tion, scheduling policies can make moreappropriate decisions.

Cloning processes is useful in caseswhere the child process inherits state fromthe parent process. Cloning is typicallyachieved using a remote fork mechanism.A remote fork, followed by the termina-tion of the parent, resembles process mi-gration. The complexity of cloning pro-cesses is similar to migration, because thesame amount of the process state is in-herited (e.g. open files and address space).In the case of migration, the parent isterminated. In the case of cloning, bothparent and child may continue to accessthe same state, introducing distributedshared state, which is typically complexand costly to maintain. Many systems useremote forking [Goldberg and Jefferson,1987; Smith and Ioannidis, 1989; Zajcewet al., 1993].

Programming language support formobility enables a wide variety of options,since such systems have almost completecontrol over the runtime implementationof an application. Such systems can en-able self-checkpointing (and hence migrat-able) applications. They are suitable forentire processes, but also for objects assmall as a few bytes, such as in Emerald[Jul et al., 1988; Jul, 1989] or Ellie[Andersen, 1992]. Finer granularity in-curs lower transfer costs. The complexity



of maintaining communication channelsposes different kinds of problems. InEmerald, for example, the pointers haveto be updated to the source object. Pro-gramming language support allows a pro-grammer to introduce more informationon object behavior, such as hints aboutcommunication and concurrency patterns.

Object migration at the middlewarelevel is also possible. Because of the in-creasing costs of operating system devel-opment and the lack of standard solutionsfor distributed systems and heterogene-ity, middleware level solutions have be-come of more interest [Bernstein, 1996].Distributed objects are supported in mid-dleware systems such as DCE [Rosenberryet al., 1992] and CORBA [OMG, 1996]. Ob-ject migration at the middleware level hasnot attracted as much research as processmigration in operating systems. One ofthe reasons is that the early heterogeneityof these systems did not adequately sup-port mobility. Nevertheless, a couple ofsystems do support mobility at the mid-dleware level, such as DC++ [Schill andMock, 1993] and the OMG MASIF speci-fication for mobile agents [Milojicic et al.,1998b] based on OMG CORBA.

Mobile agents are becoming increas-ingly popular. The mobility of agents onthe Web emphasizes safety and security is-sues more than complexity, performance,transparency and heterogeneity. Mobileagents are implemented on top of safelanguages, such as Java [Gosling et al.,1996], Telescript [White, 1996] and Tcl/Tk[Ousterhout, 1994]. Compared to processmigration, mobile agents have reduced im-plementation complexity because they donot have to support OS semantics. Per-formance requirements are different dueto the wide-area network communicationcost, which is the dominant factor. Hetero-geneity is abstracted away at the languagelevel. The early results and opportunitiesfor deployment, as well as the wide inter-est in the area of mobile agents, indicatea promising future for this form of mobil-ity. However, the issues of security, socialacceptance, and commercializable appli-cations have been significantly increasedand they represent the main focus of re-

Fig. 5. Migration levels differ in implementationcomplexity, performance, transparency, and reusa-bility.

search in the mobile agent community.Mobile agents are described in more detailin Section 4.8.

3. CHARACTERISTICS

This section addresses issues in pro-cess migration, such as complexity, per-formance, transparency, fault resilience,scalability and heterogeneity. These char-acteristics have a major impact on theeffectiveness and deployment of processmigration.

3.1. Complexity and OperatingSystem Support

The complexity of implementation anddependency on an operating system areamong the obstacles to the wider useof process migration. This is especiallytrue for fully-transparent migration im-plementations. Migration can be classifiedaccording to the level at which it is applied.It can be applied as part of the operatingsystem kernel, in user space, as part of asystem environment, or as a part of the ap-plication (see Figure 5). Implementationsat different levels result in different per-formance, complexity, transparency andreusability.

User-level migration typically yieldssimpler implementations, but suffers toomuch from reduced performance andtransparency to be of general use for loaddistribution. User-space implementationsare usually provided for the support oflong-running computations [Litzkow andSolomon, 1992]. Migration implementedas part of an application can have poor



reusability if modifications are required tothe application, as was done in the work byFreedman [1991] and Skordos [1995]. Thisrequires familiarity with applications andduplicating some of the mechanisms foreach subsequent application, frequentlyinvolving effort beyond re-linking the mi-gration part with the application code.It could be somewhat improved if partsof migration support is organized in areusable run-time library. Lower-level mi-gration is more complex to implement, buthas better performance, transparency andreusability.

Despite high migration costs, user-levelimplementations have some benefits withregard to policy. The layers closer to anapplication typically have more knowl-edge about its behavior. This knowledgecan be used to derive better policies andhence, better overall performance. Sim-ilar motivations led to the developmentof microkernels, such as Mach [Accettaet al., 1986], Chorus [Rozier, 1992], andAmoeba [Tanenbaum, 1990], which havemoved much of their functionality fromthe kernel into user space. For example,file servers and networking may be imple-mented in user space, leaving only a min-imal subset of functionality provided inthe microkernel, such as virtual memorymanagement, scheduling and interprocesscommunication.

Extensible kernels, such as Spin[Bershad et al., 1995], Exokernel [Engleret al., 1995], and Synthetix [Pu et al.,1995], have taken an alternative approachby allowing user implemented parts tobe imported into the kernel. Both micro-kernels and extensible kernels provideopportunities for extracting a process’sstate from the operating system.

There have been many implementa-tions of migration for various operat-ing systems and hardware architectures;many of them required a significant im-plementation effort and modifications tothe underlying kernel [Barak and Shiloh,1985; Theimer et al., 1985; Zayas, 1987a;Douglis and Ousterhout, 1991]. This com-plexity is due to the underlying operat-ing system architecture and specificallyits lack of support for the complex

interactions resulting from process mi-gration. In the early days, migration re-quired additional OS support, such as ex-tensions for communications forwarding[Artsy et al., 1987], or for data transferstrategies [Theimer et al., 1985; Zayas,1987a]. In the case of some subsequent mi-gration implementations, this support al-ready existed in the OS, such as in the caseof Mach [Milojicic et al., 1993a].

In UNIX-like operating systems, sup-port for opened files and signals requiressignificant interaction with various kernelsubsystems [Douglis, 1989; Welch, 1990].Process migration in message-passingkernels requires significant effort to sup-port message handling [Theimer et al.,1985; Artsy et al., 1987; Artsy and Finkel,1989]. Recent operating systems providemuch of this support, such as transparentdistributed IPC with message forwarding,and external distributed pagers, whichallow easier optimizations and customiz-ing [Black et al., 1992; Rozier, 1992].Nevertheless, migration still challengesthese mechanisms and frequently breaksthem [Douglis and Ousterhout, 1991;Milojicic, 1993c].

3.2. Performance

Performance is the second importantfactor that affects the deployment of pro-cess migration. Migration performance de-pends on initial and run-time costs intro-duced by the act of migration. The initialcosts stem from state transfer. Insteadof at migration time, some of the statemay be transferred lazily (on-demand),thereby incurring run-time costs. Bothtypes of cost may be significant, depend-ing on the application characteristics, aswell as on the ratio of state transferredeagerly/lazily.

If only part of the task state is trans-ferred to another node, the task can startexecuting sooner, and the initial migra-tion costs are lower. This principle is calledlazy evaluation: actions are not taken be-fore they are really needed with the hopethat they will never be needed. However,when this is not true, penalties are paidfor postponed access. For example, it is



convenient to migrate a huge addressspace on demand instead of eagerly. In thelazy case, part of the space may never betrans-ferred if it is not accessed. However,the source node needs to retain lazily eval-uated state throughout the life-time of themigrated process.

A process’s address space usually con-stitutes by far the largest unit of processstate; not surprisingly, the performance ofprocess migration largely depends on theperformance of the address space transfer.Various data transfer strategies have beeninvented in order to avoid the high cost ofaddress space transfer.

r The eager (all) strategy copies all ofthe address space at the migration time.Initial costs may be in the range ofminutes. Checkpoint/restart implemen-tations typically use this strategy, suchas Condor [Litzkow and Solomon, 1992]or LSF [Zhou et al., 1994].r The eager (dirty) strategy can be de-ployed if there is remote paging sup-port. This is a variant of the eager(all) strategy that transfers only modi-fied (dirty) pages. Unmodified pages arepaged in on request from a backing store.Eager (dirty) significantly reduces theinitial transfer costs when a process hasa large address space. Systems support-ing eager (dirty) strategy include MOSIX[Barak and Litman, 1985] and Locus[Popek and Walker, 1985].r The Copy-On-Reference (COR) strat-egy is a network version of demand pag-ing: pages are transferred only upon ref-erence. While dirty pages are broughtfrom the source node, clean pages can bebrought either from the source node orfrom the backing store. The COR strat-egy has the lowest initial costs, rang-ing from a few tens to a few hundredmicroseconds. However, it increases therun-time costs, and it also requires sub-stantial changes to the underlying oper-ating system and to the paging support[Zayas, 1987a].r The flushing strategy consists of flush-ing dirty pages to disk and then access-ing them on demand from disk instead

of from memory on the source nodeas in copy-on-reference [Douglis andOusterhout, 1991]. The flushing strategyis like the eager (dirty) transfer strat-egy from the perspective of the source,and like copy-on-reference from the tar-get’s viewpoint. It leaves dependencieson the server, but not on the sourcenode.r The precopy strategy reduces the“freeze” time of the process, the time thatprocess is neither executed on the sourcenor on the destination node. While theprocess is executed on the source node,the address space is being transferredto the remote node until the numberof dirty pages is smaller than a fixedlimit. Pages dirtied during precopy haveto be copied a second time. The precopystrategy cuts down the freeze time belowthe costs of the COR technique [Theimeret al., 1985].

There are also variations of the abovestrategies. The most notable example ismigration in the Choices operating sys-tem [Roush and Campbell, 1996]. It usesa variation of the eager (dirty) strategywhich transfers minimal state to the re-mote node at the time of migration. The re-mote instance is started while the remain-der of the state is transferred in parallel.The initial migration time is reduced to13.9ms running on a SparcStation II con-nected by a 10Mb Ethernet, which is anorder of magnitude better than all otherreported results, even if results are nor-malized (see work by Rousch [1995] formore details on normalized performanceresults).

Leaving some part of the process stateon the source or intermediate nodes of themigrated instance results in a residual de-pendency. Residual dependencies typicallyoccur as a consequence of two implemen-tation techniques: either using lazy evalu-ation (see definition below), or as a meansfor achieving transparency in communica-tion, by forwarding subsequent messagesto a migrated process.

A particular case of residual depen-dency is the home dependency, which isa dependency on the (home) node where



a process was created [Douglis andOusterhout, 1991]. An example of a homedependency is redirecting systems calls tothe home node: for example, local host-dependent calls, calls related to the filesystem (in the absence of a distributedfile system), or operations on local devices.A home dependency can simplify migra-tion, because it is easier to redirect re-quests to the home node than to supportservices on all nodes. However, it also ad-versely affects reliability, because a mi-grated foreign process will always dependon its home node. The notion of the homedependency is further elaborated upon be-low in Section 5.1 (MOSIX) and Section 5.2(Sprite).

Redirecting communication through thepreviously established links representsanother kind of residual dependency. Ingeneral, dependencies left at multiplenodes should be avoided, since they re-quire complex support, and degrade per-formance and fault resilience. Therefore,some form of periodic or lazy removal ofresidual dependencies is desirable. For ex-ample, the system could flush remain-ing pages to the backing store, or updateresidual information on migrated commu-nication channels.

3.3. Transparency

Transparency requires that neither themigrated task nor other tasks in the sys-tem can notice migration, with the possi-ble exception of performance effects. Com-munication with a migrated process couldbe delayed during migration, but no mes-sage can be lost. After migration, theprocess should continue to communicatethrough previously opened I/O channels,for example printing to the same consoleor reading from the same files.

Transparency is supported in a varietyof ways, depending on the underlying op-erating system. Sprite and NOW MOSIXmaintain a notion of a home machine thatexecutes all host-specific code [Douglisand Ousterhout, 1991; Barak et al., 1995].Charlotte supports IPC through links,which provide for remapping after migra-tion [Finkel et al., 1989].

Transparency also assumes that the mi-grated instance can execute all systemcalls as if it were not migrated. Some user-space migrations do not allow system callsthat generate internode signals or file ac-cess [Mandelberg and Sunderam, 1988;Freedman, 1991].

Single System Image (SSI) represents acomplete form of transparency. It providesa unique view of a system composed of anumber of nodes as if there were just onenode. A process can be started and com-municated with without knowing whereit is physically executing. Resources canbe transparently accessed from any nodein the system as if they were attachedto the local node. The underlying systemtypically decides where to instantiate newprocesses or where to allocate and accessresources.

SSI can be applied at different levels ofthe system. At the user-level, SSI consistsof providing transparent access to objectsand resources that comprise a particu-lar programming environment. Examplesinclude Amber [Chase et al., 1989] andEmerald [Jul, 1989]. At the traditional op-erating system level, SSI typically con-sists of a distributed file system and dis-tributed process management, such as inMOSIX [Barak and Litman, 1985], Sprite[Ousterhout et al., 1988] and OSF/1 ADTNC [Zajcew et al., 1993]. At the microker-nel level, SSI is comprised of mechanisms,such as distributed IPC, distributed mem-ory management, and remote tasking. Anear-SSI is implemented for Mach [Blacket al., 1992] based on these transpar-ent mechanisms, but the policies are sup-ported at the OSF/1 AD server running ontop of it. At the microkernel level the pro-grammer needs to specify where to createremote tasks.

SSI supports transparent access to aprocess, as well as to its resources, whichsimplifies migration. On the other hand,the migration mechanism exercises func-tionality provided at the SSI level, posinga more stressful workload than normallyexperienced in systems without migra-tion [Milojicic et al., 1993a]. Therefore,although a migration implementation ontop of SSI may seem less complex, this



complexity is pushed down into the SSIimplementation.

Some location dependencies on anotherhost may be inevitable, such as access-ing local devices or accessing kernel-dependent state that is managed by theother host. It is not possible transparentlyto support such dependencies on the newlyvisited nodes, other than by forwardingthe calls back to the home node, as wasdone in Sprite [Douglis and Ousterhout,1991].

3.4. Fault Resilience

Fault resilience is frequently mentionedas a benefit of process migration. How-ever, this claim has never been substan-tiated with a practical implementation,although some projects have specificallyaddressed fault resilience [Chou andAbraham, 1983; Lu et al., 1987]. So farthe major contribution of process migra-tion for fault resilience is through com-bination with checkpointing, such as inCondor [Litzkow and Solomon, 1992], LSFZhou et al., 1994] and in work by Skordos[1995]. Migration was also suggested as ameans of fault containment [Chapin et al.,1995].

Failures play an important role inthe implementation of process migration.They can happen on a source or target ma-chine or on the communication medium.Various migration schemes are more orless sensitive to each type of failure. Resid-ual dependencies have a particularly neg-ative impact on fault resilience. Usingthem is a trade-off between efficiency andreliability.

Fault resilience can be improved in sev-eral ways. The impact of failures duringmigration can be reduced by maintainingprocess state on both the source anddestination sites until the destinationsite instance is successfully promoted toa regular process and the source node isinformed about this. A source node failurecan be overcome by completely detachingthe instance from the source node onceit is migrated, though this preventslazy evaluation techniques from beingemployed. One way to remove communi-cation residual dependencies is to deploy

locating techniques, such as multicasting(as used in V kernel [Theimer et al.,1985), reliance on the home node (as usedin Sprite [Douglis and Ousterhout, 1991],and MOSIX [Barak and Litman, 1985]),or on a forwarding name server (as usedin most distributed name services, suchas DCE, as well as in mobile agents, suchas MOA [Milojicic et al., 1999]). This waydependencies are singled out on dedicatednodes, as opposed to being scatteredthroughout all the nodes visited, as isthe case for Charlotte [Artsy et al., 1987].Shapiro, et al. [1992] propose so-calledSSP Chains for periodically collapsingforwarding pointers (and thereby reduc-ing residual dependencies) in the case ofgarbage collection.

3.5. Scalability

The scalability of a process migrationmechanism is related to the scalability ofits underlying environment. It can be mea-sured with respect to the number of nodesin the system, to the number of migra-tions a process can perform during its life-time, and to the type and complexity ofthe processes, such as the number of openchannels or files, and memory size or frag-mentation.

The number of nodes in the system af-fects the organization and managementof structures that maintain residual pro-cess state and the naming of migrated pro-cesses. If these structures are not part ofthe existing operating system, then theyneed to be added.

Depending on the migration algorithmand the techniques employed, some sys-tems are not scalable in the number ofmigrations a process may perform. As weshall see in the case study on Mach (seeSection 5.3), sometimes process state cangrow with the number of migrations. Thisis acceptable for a small number of mi-grations, but in other cases the additionalstate can dominate migration costs andrender the migration mechanism useless.

Migration algorithms should avoid lin-ear dependencies on the amount of stateto be transferred. For example, the eagerdata transfer strategy has costs propor-tional to the address space size, incurring



significant costs for large address spaces.The costs for a lazily copied process areindependent of the address space size, butthey can depend on the granularity andtype of the address space. For example, thetransfer of a large sparse address spacecan have costs proportional to the num-ber of contiguous address space regions,because each such region has metadata as-sociated with it that must be transferredat migration time. This overhead can beexacerbated if the meta-data for each re-gion is transferred as a separate opera-tion, as was done in the initial implemen-tation of Mach task migration [Milojicicet al., 1993b].

Communication channels can also affectscalability. Forwarding communication toa migrated process is acceptable after asmall number of sequential migrations,but after a large number of migrationsthe forwarding costs can be significant. Inthat case, some other technique, such asupdating communication links, must beemployed.

3.6. Heterogeneity

Heterogeneity has not been addressed inmost early migration implementations.Instead, homogeneity is considered as arequirement; migration is allowed onlyamong the nodes with a compatible archi-tecture and processor instruction set. Thiswas not a significant limitation at the timesince most of the work was conducted onclusters of homogeneous machines.

Some earlier work indicated the needas well as possible solutions for solvingthe heterogeneity problem, but no matureimplementations resulted [Maguire andSmith, 1988; Dubach, 1989; Shub, 1990;Theimer and Hayes, 1991].

The deployment of world-wide comput-ing has increased the interest in hetero-geneous migration. In order to achieveheterogeneity, process state needs to besaved in a machine-independent represen-tation. This permits the process to resumeon nodes with different architectures. Anapplication is usually compiled in advanceon each architecture, instrumenting thecode to know what procedures and vari-ables exist at any time, and identifying

points at which the application can besafely preempted and checkpointed. Thecheckpointing program sets a breakpointat each preemption point and examinesthe state of the process when a breakpointis encountered. Smith and Hutchinsonnote that not all programs can be safelycheckpointed in this fashion, largely de-pending on what features of the languageare used [Smith and Hutchinson, 1998].Emerald [Steensgaard and Jul, 1995]is another example of a heterogeneoussystem.

In the most recent systems, heterogene-ity is provided at the language level, asby using intermediate byte code repre-sentation in Java [Gosling et al., 1996],or by relying on scripting languages suchas Telescript [White, 1996] or Tcl/Tk[Ousterhout, 1994].

3.7. Summary

This subsection evaluates the trade-offsbetween various characteristics of processmigration, and who should be concernedwith it.

Complexity is much more of a concernto the implementors of a process migra-tion facility than to its users. Complexitydepends on the level where migrationis implemented. Kernel-level implemen-tations require significantly more com-plexity than user-level implementations.Users of process migration are impactedonly in the case of user-level implemen-tations where certain modifications of theapplication code are required or where mi-gration is not fully transparent.

Long-running applications are not con-cerned with performance as are thoseapplications whose lifetimes are com-parable to their migration time. Short-running applications are generally notgood candidates for migration. Migration-time performance can be traded off againstexecution-time (by leaving residual depen-dencies, or by lazily resolving communica-tion channels). Residual dependencies areof concern for long-running applicationsand for network applications. Applicationswith real-time requirements generally arenot suitable candidates for residual depen-dency because of the unpredictable costs



of bringing in additional state. On theother hand, real-time requirements can bemore easily fulfilled with strategies, suchas precopy.

Legacy applications are concerned withtransparency in order to avoid anychanges to existing code. Scientific ap-plications typically do not have trans-parency requirements. Frequently, one isallowed to make modifications to the codeof these applications, and even supportmigration at the application level (e.g.by checkpointing state at the applicationlevel). Transparency typically incurs com-plexity. However, transparency is not re-lated to migration exclusively, rather itis inherent to remote access. Transpar-ent remote execution can require supportthat is as complex as transparent pro-cess migration [Douglis and Ousterhout,1991].

Scientific applications (typically long-running), as well as network applicationsare concerned with failure tolerance. Inmost cases periodic checkpointing of thestate suffices.

Scalability requires additional complex-ity for efficient support. It is of concern forscientific applications because they mayrequire a large number of processes, largeaddress spaces, and a large number ofcommunication channels. It is also impor-tant for network applications, especiallythose at the Internet scale.

Heterogeneity introduces performancepenalties and additional complexity. It isof most concern to network applicationswhich typically run on inhomogeneoussystems.

4. EXAMPLES

This section classifies process migrationimplementations in the following cate-gories: early work; UNIX-like systemssupporting transparent migration; sys-tems with message-passing interfaces;microkernels; user-space migration; andapplication-specific migration. In addi-tion, we also give an overview of mobileobjects and mobile agents. These last twoclasses do not represent process migrationin the classic sense, but they are similar

in sufficiently many ways to warrant theirinclusion [Milojicic et al., 1998a]. For eachclass, an overview and some examples arepresented. Finally, in the last subsection,we draw some conclusions. The next sec-tion expands upon four of these systems insubstantial detail.

There are also other examples of pro-cess migration that can fit into one ormore classes presented in this section.Examples include object migration inEden [Lazowska, et al., 1981]; MINIX[Louboutin, 1991]; Galaxy [Sinha et al.,1991]; work by Dediu [1992]; EMPS [vanDijk and van Gils, 1992]; object migrationfor OSF DCE, DC++ [Schill and Mock,1993]; work by Petri and Langendorfer[1995]; MDX [Schrimpf, 1995]; and manymore. A description of these systems isbeyond the scope of this paper. In ad-dition to other surveys of process mi-gration already mentioned in the intro-duction [Smith, 1988; Eskicioglu, 1990;Nuttal, 1994], Borghoff provides a cat-alogue of distributed operating systemswith many examples of migration mech-anisms [Borghoff, 1991].

4.1. Early Work

Early work is characterized by specialized,ad hoc solutions, often optimized for theunderlying hardware architecture. In thissubsection we briefly mention XOS, Worm,DEMOS/MP and Butler.

Migration in XOS is intended as atool for minimizing the communication be-tween the nodes in an experimental mul-tiprocessor system, organized in a treefashion [Miller and Presotto, 1981]. Therepresentation of the process and its stateare designed in a such a way as to facili-tate migration. The Process Work Object(PWO) encapsulates process related stateincluding stack pointers and registers. Mi-gration is achieved by moving PWO ob-jects between the XOS nodes. The processlocation is treated as a hint, and the cur-rent location is found by following hints.

The Worm idea has its backgroundin the nature of real worms [Shoch andHupp, 1982]. A worm is a computationthat can live on one or more machines.Parts of the worm residing on a single



machine are called segments. If a segmentfails, other segments cooperatively rein-stantiate it by locating a free machine, re-booting it from the network, and migrat-ing the failed worm segment to it. A wormcan move from one machine to another, oc-cupying needed resources, and replicatingitself. As opposed to other migration sys-tems, a worm is aware of the underlyingnetwork topology. Communication amongworm segments is maintained throughmulticasting.

The original Butler system supportsremote execution and process migra-tion [Dannenberg, 1982]. Migration oc-curs when the guest process needs to be“deported” from the remote node, e.g. incase when it exceeds resources it nego-tiated before arrival. In such a case, thecomplete state of the guest process ispackaged and transferred to a new node.The state consists of the address space,registers, as well as the state containedin the servers collocated at the samenode. Migration does not break the com-munication paths because the underly-ing operating system (Accent [Rashid andRobertson, 1981]) allows for port migra-tion. The Butler design also deals withthe issues of protection, security, and au-tonomy [Dannenberg and Hibbard, 1985].In particular, the system protects theclient program, the Butler daemons on thesource and destination nodes, the visitingprocess, and the remote node. In its laterincarnation, Butler supports only remoteinvocation [Nichols, 1987].

DEMOS/MP [Miller et al., 1987] is asuccessor of the earlier version of theDEMOS operating system [Baskett et al.,1977]. Process migration is fully transpar-ent: a process can be migrated during exe-cution without limitations on resource ac-cess. The implementation of migration hasbeen simplified and its impact to otherservices limited by the message-passing,location-independent communication, andby the fact that the kernel can partici-pate in the communication in the samemanner as any process [Powell and Miller,1983]. Most of the support for processmigration already existed in the DEMOSkernel. Extending it with migration re-

quired mechanisms for forwarding mes-sages and updating links. The transferredstate includes program code and data(most of the state), swappable and non-swappable state, and messages in the in-coming queue of the process.

4.2. Transparent Migration inUNIX-like Systems

UNIX-like systems have proven to be rel-atively hard to extend for transparentmigration and have required significantmodifications and extensions to the under-lying kernel (see Subsections 4.3 and 4.4for comparisons with other types of OSes).There are two approaches to addressingdistribution and migration for these sys-tems. One is to provide for distribution atthe lower levels of a system, as in MOSIXor Sprite, and the other is by providingdistribution at a higher-level, as in Locusand its derivatives. In this section, weshall describe process migration for Locus,MOSIX and Sprite. All of these systemsalso happened to be RPC-based, as op-posed to the message-passing systems de-scribed in Section 4.3.

Locus is a UNIX-compatible operatingsystem that provides transparent accessto remote resources, and enhanced relia-bility and availability [Popek et al., 1981;Popek and Walker, 1985]. It supports pro-cess migration [Walker et al., 1983] andinitial placement [Butterfield and Popek,1984]. Locus is one of the rare systemsthat achieved product stage. It has beenported to the AIX operating system on theIBM 370 and PS/2 computers under thename of the Transparent Computing Fa-cility (TCF) [Walker and Mathews, 1989].Locus migration has a high level of func-tionality and transparency. However, thisrequired significant kernel modifications.

Locus has subsequently been ported tothe OSF/1 AD operating system, underthe name of TNC [Zajcew et al., 1993].OSF/1 AD is a distributed operating sys-tem running on top of the Mach microker-nel on Intel x86 and Paragon architectures(see Section 5.3). TNC is only partially con-cerned with task migration issues of theunderlying Mach microkernel, because in



the OSF/1 AD environment the Machinterface is not exposed to the user, andtherefore the atomicity of process migra-tion is not affected. Locus was also usedas a testbed for a distributed shared mem-ory implementation, Mirage [Fleisch andPopek, 1989]. Distributed shared mem-ory was not combined with process migra-tion as was done in the case of Mach (seeSection 5.3).

The MOSIX distributed operating sys-tem is an ongoing project that began in1981. It supports process migration ontop of a single system image base [Barakand Litman, 1985] and in a Network ofWorkstations environment [Barak et al.,1995]. The process migration mechanismis used to support dynamic load balancing.MOSIX employs a probabilistic algorithmin its load information management thatallows it to transmit partial load informa-tion between pairs of nodes [Barak andShiloh, 1985; Barak and Wheeler, 1989].A case study of the MOSIX system is pre-sented in Section 5.1.

The Sprite network operating system[Ousterhout et al., 1988] was developedfrom 1984–1994. Its process migration fa-cility [Douglis and Ousterhout, 1991] wastransparent both to users and to appli-cations, by making processes appear toexecute on one host throughout their ex-ecution. Processes could access remote re-sources, including files, devices, and net-work connections, from different locationsover time. When a user returned to a work-station onto which processes had been off-loaded, the processes were immediatelymigrated back to their home machines andcould execute there, migrate else-where,or suspend execution. A case study of theSprite system is presented in Section 5.2.

4.3. OS with Message-Passing Interface

Process migration for message-passing op-erating systems seems easier to designand implement. Message passing is con-venient for interposing, forwarding andencapsulating state. For example, a newreceiver may be interposed between theexisting receiver and the sender, with-out the knowledge of the latter, and mes-

sages sent to a migrated process can beforwarded after its migration to a newdestination. However, much of the sim-plicity that seems to be inherent formessage-passing systems is hidden in-side the complex message-passing mech-anisms [Douglis and Ousterhout, 1991].

In this section we describe Charlotte,Accent and the V kernel. The V kernel canbe classified both as a microker-nel andas a message passing kernel; we chose topresent it in the message-passing section.

Charlotte is a message-passing operat-ing system designed for the Crystal mul-ticomputer composed of 20 VAX-11/750computers [Artsy and Finkel, 1989]. TheCharlotte migration mechanism exten-sively relies on the underlying operatingsystem and its communication mecha-nisms which were modified in order to sup-port transparent network communication[Artsy et al., 1987]. Its process migration iswell insulated from other system modules.Migration is designed to be fault resilient:processes leave no residual dependency onthe source machine. The act of migrationis committed in the final phase of the statetransfer; it is possible to undo the migra-tion before committing it.

Accent is a distributed operating sys-tem developed at CMU [Rashid andRobertson, 1981; Rashid, 1986]. Its pro-cess migration scheme was the first one touse the “Copy-On-Reference” (COR) tech-nique to lazily copy pages [Zayas, 1987a].Instead of eagerly copying pages, virtualsegments are created on the destinationnode. When a page fault occurs, the vir-tual segment provides a link to the pageon the source node. The duration of the ini-tial address space transfer is independentof the address space size, but rather de-pends on the number of contiguous mem-ory regions. The subsequent costs for lazilycopied pages are proportional to the num-ber of pages referenced. The basic as-sumption is that the program would notaccess all of its address space, therebysaving the cost of a useless transfer. Be-sides failure vulnerability, the drawbackof lazy evaluation is the increased com-plexity of in-kernel memory management[Zayas, 1987b].



The V Kernel is a microkernel devel-oped at Stanford University [Cheriton,1988]. It introduces a “precopying” tech-nique for the process address space trans-fer [Theimer et al., 1985]. The addressspace of the process to be migrated iscopied to the remote node prior to itsmigration, while the process is still ex-ecuting on the source node. Dirty pagesreferenced during the precopying phaseare copied again. It has been shown thatonly two or three iterations generally suf-fice to reach an acceptably small numberof dirty pages. At that point of time theprocess is frozen and migrated. This tech-nique shortens the process freeze time, butotherwise negatively influences the exe-cution time, since overhead is incurredin iterative copying. Migration benefitsfrom a communications protocol that dy-namically rebinds to alternate destina-tion hosts as part of its implementationof reliable message delivery. Instead ofmaintaining process communication end-points after migration, V relies on multi-cast to find the new process location.

4.4. Microkernels

The microkernel approach separates theclassical notion of a monolithic kernelinto a microkernel and an operating sys-tem personality running on top of it ina separate module. A microkernel sup-ports tasks, threads, IPC and VM man-agement, while other functionality, suchas networking, file system and processmanagement, is imple-mented in the OSpersonality. Various OS personalities havebeen implemented, such as BSD UNIX[Golub et al., 1990], AT&T UNIX SystemV [Rozier, 1992; Cheriton, 1990], MS DOS[Malan et al., 1991], VMS [Wiecek, 1992],OS/2 [Phelan and Arendt, 1993] and Linux[Barbou des Places et al., 1996].

In the late eighties and early nineties,there was a flurry of research into mi-crokernels, including systems, such asMach [Accetta et al., 1986], Chorus[Rozier, 1992], Amoeba [Mullender et al.,1990], QNX [Hildebrand, 1992], Spring[Hamilton and Kougiouris, 1993] and L3[Liedtke, 1993], which eventually reached

commercial implementations, and manymore research microkernels, such asArcade [Cohn et al., 1989], Birlix [Haertiget al., 1993], KeyKOS [Bomberger et al.,1992] and RHODOS [Gerrity et al., 1991].

The microkernel approach, combinedwith message passing, allows for trans-parent, straightforward extensions to dis-tributed systems. Not surprisingly, micro-kernels are a suitable environment forvarious migration experiments. The taskmigration mechanism can be reused bydifferent OS personalities, as a commondenominator for different OS-specific pro-cess migration mechanisms. In this sub-section we describe process migrations forRHODOS, Arcade, Chorus, Amoeba, Birlixand Mach.

RHODOS consists of a nucleus thatsupports trap and interrupt handling, con-text switching, and local message pass-ing. The kernel runs on top of the nu-cleus and supports IPC, memory, process,and migration managers [Gerrity et al.,1991]. The migration mechanism is sim-ilar to that in Sprite, with some modifica-tions specific to the RHODOS kernel [Zhu,1992].

Arcade considers groups of tasks formigration [Cohn et al., 1989]. It is usedas a framework for investigating sharingpolicies related to task grouping [Tracey,1991]. The group management softwareensures that members of the group ex-ecute on different machines, thereby ex-ploiting parallelism.

The Chorus microkernel was extendedto support process migration [Philippe,1993]. The migration mechanism is sim-ilar to task migration on top of Mach (cf.Section 5.3), however it is applied at theprocess level, instead of the Actor level. Ac-tors in Chorus correspond to Mach tasks.Chorus migration is biased toward thehypercube implementation (fast and reli-able links). Some limitations were intro-duced because Chorus did not support portmigration.

Steketee et al. implemented process mi-gration for the Amoeba operating system[Steketee et al., 1994]. Communicationtransparency relies on the location inde-pendence of the FLIP protocol [Kaashoek



et al., 1993]. Since Amoeba does not sup-port virtual memory, the memory transferfor process migration is achieved by phys-ical copying [Zhu et al., 1995].

Birlix supports adaptable object migra-tion [Lux, 1995]. It is possible to spec-ify a migration policy on a per-object ba-sis. A meta-object encapsulates data forthe migration mechanism and informationcollection. An example of the use of anadaptable migration mechanism is to ex-tend migration for improved reliability orperformance [Lux et al., 1993].

Mach [Accetta et al., 1986] was usedas a base for supporting task migration[Milojicic et al., 1993b], developed at theUniversity of Kaiserslautern. The goalswere to demonstrate that microkernelsare a suitable substrate for migrationmechanisms and for load distribution ingeneral. The task migration implementa-tion significantly benefited from the nearSSI provided by Mach, in particular fromdistributed IPC and distributed memorymanagement. Process migration was builtfor the OSF/1 AD 1 server using Machtask migration [Paindaveine and Milojicic,1996]. Task and process migration on topof Mach are discussed in more detail inSection 5.3.

4.5. User-space Migrations

While it is relatively straightforward toprovide process migration for distributedoperating systems, such as the V kernel,Accent, or Sprite, it is much harder to sup-port transparent process migration on in-dustry standard operating systems, whichare typically non-distributed. Most work-stations in the 1980s and 1990s run pro-prietary versions of UNIX, which makesthem a more challenging base for processmigration than distributed operating sys-tems. Source code is not widely availablefor a proprietary OS; therefore, the onlyway to achieve a viable and widespreadmigration is to implement it in user space.

User-space migration is targeted tolong-running processes that do not posesignificant OS requirements, do not needtransparency, and use only a limited set ofsystem calls. The migration time is typi-

cally a function of the address space size,since the eager (all) data transfer schemeis deployed. This subsection presents afew such implementations: Condor, thework by Alonso and Kyrimis, the workby Mandelberg and Sunderam, the workby Petri and Langendoerfer, MPVM, andLSF.

Condor is a software package that sup-ports user-space checkpointing and pro-cess migration in locally distributed sys-tems [Litzkow, 1987; Litzkow et al., 1988;Litzkow and Solomon, 1992]. Its check-pointing support is particularly useful forlong-running computations, but is too ex-pensive for short processes. Migration in-volves generating a core file for a process,combining this file with the executableand then sending this on to the targetmachine. System calls are redirected to a“shadow” process on the source machine.This requires a special version of the Clibrary to be linked with the migratedprograms.

Condor does not support processes thatuse signals, memory mapped files, timers,shared libraries, or IPC. The scheduleractivation period is 10 minutes, whichdemonstrates the “heaviness” of migra-tion. Nevertheless, Condor is often usedfor long-running computations. It hasbeen ported to a variety of operating sys-tems. Condor was a starting point for a fewindustry products, such as LSF from Plat-form Computing [Zhou et al., 1994] andLoadleveler from IBM.

Alonso and Kyrimis perform minormodifications to the UNIX kernel in orderto support process migration in user space[Alonso and Kyrimis, 1988]. A new signalfor dumping process state and a new sys-tem call for restarting a process are intro-duced. This implementation is limited toprocesses that do not communicate andare not location- or process-dependent.The work by Alonso and Kyrimis was donein parallel with the early Condor system.

Mandelberg and Sunderam presenta process migration scheme for UNIX thatdoes not support tasks that perform I/Oon non-NFS files, spawn subprocesses,or utilize pipes and sockets [Mandelbergand Sunderam, 1988]. A new terminal



interface supports detaching a processfrom its terminal and monitors requestsfor I/O on the process migration port.

Migratory Parallel Virtual Machine(MPVM) extends the PVM system[Beguelin et al., 1993] to support processmigration among homogeneous machines[Casas et al., 1995]. Its primary goals aretransparency, compatibility with PVM,and portability. It is implemented entirelyas a user-level mechanism. It supportscommunication among migrating pro-cesses by limiting TCP communication toother MPVM processes.

Load Sharing Facility (LSF) sup-ports migration indirectly through pro-cess checkpointing and restart [PlatformComputing, 1996]. LSF can work withcheckpointing at three possible levels:kernel, user, and application. The tech-nique used for user-level check-pointing isbased on the Condor approach [Litzkowand Solomon, 1992], but no core fileis required, thereby improving perfor-mance, and signals can be used acrosscheckpoints, thereby improving trans-parency. LSF is described in more detail inSection 5.4.

4.6. Application-specific Migration

Migration can also be implemented as apart of an application. Such an approachdeliberately sacrifices transparency andreusability. A migrating process is typi-cally limited in functionality and migra-tion has to be adjusted for each new appli-cation. Nevertheless, the implementationcan be significantly simplified and opti-mized for one particular application. Inthis subsection we describe work by Freed-man, Skordos, and Bharat and Cardelli.

Freedman reports a process migrationscheme involving cooperation between themigrated process and the migration mod-ule [Freedman, 1991]. The author ob-serves that long-running computationstypically use operating system servicesin the beginning and ending phases ofexecution, while most of their time is spentin number-crunching. Therefore, little at-tention is paid to supporting files, sock-ets, and devices, since it is not expected

that they will be used in the predominantphase of execution. This ad hoc process mi-gration considers only memory contents.

Skordos integrates migration with par-allel simulation of subsonic fluid dy-namics on a cluster of workstations[Skordos, 1995]. Skordos statically allo-cates problem sizes and uses migrationwhen a workstation becomes overloaded.Upon migration, the process is restartedafter synchronization with processes par-ticipating in the application on othernodes. At the same time, it is possible toconduct multiple migrations. On a clusterof 20 HP-Apollo workstations connectedby 10 Mbps Ethernet, Skordos notices ap-proximately one migration every 45 min-utes. Each migration lasts 30 seconds onaverage. Despite the high costs, its rela-tive impact is very low. Migrations happeninfrequently, and do not last long relativeto the overall execution time.

Bharat and Cardelli describe Migra-tory Applications, an environment for mi-grating applications along with the userinterface and the application context,thereby retaining the same “look and feel”across different platforms [Bharat andCardelli, 1995]. This type of migration isparticularly suitable for mobile applica-tions, where a user may be travelling fromone environment to another. Migratory ap-plications are closely related to the un-derlying programming language Oblique[Cardelli, 1995].

4.7. Mobile Objects

In this paper we are primarily concernedwith process and task migration. Objectmigration and mobile agents are two otherforms of migration that we mention brieflyin this and the following subsection. Al-though used in different settings, theseforms of migration serve a similar purposeand solve some of the same problems asprocess migration does. In this subsection,we give an overview of object migration forEmerald, SOS and COOL.

Emerald is a programming languageand environment for the support of dis-tributed systems [Black et al., 1987]. Itsupports mobile objects as small as a



couple of bytes, or as large as a UNIX pro-cess [Jul, 1988; Jul et al., 1988]. Objectshave a global, single name space. In addi-tion to traditional process migration ben-efits, Emerald improves data movement,object invocation and garbage collection.

In Emerald, communication links arepointers to other objects. Upon each ob-ject migration, all object pointers need tobe updated. The Emerald compiler gen-erates the templates that are associatedwith the object data areas describing itslayout. The templates contain informa-tion about which objects are associatedwith the given object, including the point-ers to other objects. These pointers arechanged if the referenced object moves.Pointers are optimized for local invocationbecause mobility is a relatively infrequentcase compared to local invocation. Objectsthat become unreachable are garbage col-lected. Moving a small passive object on acluster of 4 MicroVax II workstations con-nected by a 10 megabit/second Ethernettakes about 12 ms while moving a smallprocess takes about 40 ms. Some mod-est experiments demonstrated the bene-fits of Emerald for load distribution [Jul,1989].

Shapiro investigates object migrationand persistence in SOS [Shapiro et al.,1989]. The objects under considerationare small to medium size (a few hun-dred bytes). Of particular concern areintra-object references and how they arepreserved across object migrations. Ref-erences are expressed through a newtype, called a permanent pointer. Aftermigration, permanent pointers are lazilyevaluated, based on the proxy principle[Shapiro, 1986]. A proxy is a new ob-ject that represents the original object,maintains a reference to it at the newlocation, and provides a way to access it.Proxies, and the term proxy principle de-scribing its use, are extensively used indistributed systems with or without mi-gration (e.g. for distributed IPC [Barrera,1991], distributed memory management[Black et al. 1998], and proxy servers onthe Web [Brooks, et al. 1995]). Functional-ity can be arbitrarily distributed betweena proxy and its principal object.

COOL provides an object-oriented layeron top of Chorus [Amaral et al., 1992]. Itsupports DSM-based object sharing, per-sistent store, and object clustering. Trans-parent remote invocation is achieved witha simple communication model using theCOOL base primitives. When re-mappedonto a new node, all internal referencesare updated depending on the new lo-cation by pointer swizzling [Lea et al.,1993], which is a technique for convert-ing the persistent pointers or object iden-tifiers into the main memory pointers(addresses). Conversion can be activatedupon an access to the object (swizzlingon discovery) or eagerly (all objects atonce upon the discovery of the first per-sistent pointer). Pointer swizzling can alsobe used for supporting large and persis-tent address spaces [Dearle, et al., 1994]and in very large data bases [Kemper andKossmann, 1995].

4.8. Mobile Agents

In the recent past, mobile agents have re-ceived significant attention. A number ofproducts have appeared and many suc-cessful research systems have been devel-oped (see description of these systems be-low). A patent has been approved for one ofthe first mobile agent systems, Telescript[White, et al. 1997] and a standard wasadopted by OMG [Milojicic et al., 1998b].

Mobile agents derive from two fields:agents, as defined in the artificial in-telligence community [Shoham, 1997],and distributed systems, including mobileobjects and process migration [Milojicicet al., 1999]. However, their popularitystarted with the appearance of the Weband Java. The former opened vast oppor-tunities for applications suited for mobileagents and the latter became a drivingprogramming language for mobile agents.

In a Web environment, programminglanguages focus on platform indepen-dence and safety. Innovations in OSservices take place at the middlewarelevel rather than in kernels [Bernstein,1996]. Research in distributed systemshas largely refocused from local to wide-area networks. Security is a dominant



requirement for applications and sys-tems connected to the Web. In this en-vironment, mobile agents are a verypromising mechanism. Typical uses in-clude electronic commerce and supportfor mobile, sporadically-connected com-puting for which agents overcome limita-tions posed by short on-line time, reducedbandwidth, and limited storage.

Java has proven to be a suitable pro-gramming language for mobile agents be-cause it supports mobile code and mobileobjects, remote object model and languageand run-time safety, and it is operatingsystem independent.

While a large amount of the OS-level support for migration concentratedon transparency issues, the agent ap-proach has demonstrated less concern fortransparency.

We provide an overview a few commer-cial mobile agent systems, such as Tele-script, IBM Aglets, and Concordia, and afew academic systems, such as Agent Tcl,TACOMA and Mole.

Telescript first introduced the mobileagent concepts [White, 1996]. It is tar-geted for the MagicCap, a small hand-held device. Telescript first introduced mo-bile agent concepts place and permit andmechanisms meet and go. IBM Aglets isone of the first commercially available mo-bile agent systems based on Java [Langeand Oshima, 1998]. It is developed byIBM Tokyo Research Lab IBM. Aglets hasa large community of users and applica-tions, even a few commercial ones. Con-cordia is a mobile agent system developedat the Mitsubishi Electric ITA Laboratory[Wong, et al., 1997]. It is a Java-based sys-tem that addresses security (by extend-ing the Java security manager) and reli-ability (using message queuing based ontwo-phase-commit protocol). Concordia isused for many in-house applications.

Agent Tcl started as a Tcl/Tk-basedtransportable agent, but it has been ex-tended to support Java, Scheme and C/C++ [Kotz, et al., 1997]. It is used forthe development of the DAIS system forinformation retrieval and disseminationin military intelligence [Hoffman, et al.,1998]. Agent Tcl is optimized for mobile

computers, e.g. by minimizing connectiontime and communication. The TACOMAproject is a joint effort by Tromso andCornell Universities [Johansen et al.,1995]. Compared to other mobile agentresearch, which addresses programminglanguages aspects, TACOMA addressesoperating system aspects. The main re-search topics include security and re-liability. Mole is one of the first aca-demic agent systems written in Java[Baumann, et al., 1998]. It has beenused by industry (Siemens, Tandem, andDaimler Benz), and academia (Univer-sity of Geneva). Mole addresses groups ofagents, agent termination, and securityfor protecting agents against malicioushosts.

There are also many other mobileagent systems, such as Ara [Peine andStolpmann, 1997], Messenger [Tschudin,1997], MOA [Milojicic et al., 1998a], andSumatra [Ranganathan, et al., 1997]. Alot of effort has been invested in securityof mobile agents, such as in the work byFarmer, et al. [1996], Hohl [1998], Tardoand Valente [1996], Vigna [1998], andVitek, et al. [1997]. A paper by Chess et al.[1995] is a good introduction to mobileagents.

5. CASE STUDIES

This section presents four case studies ofprocess migration: MOSIX, Sprite, Mach,and LSF. At least one of the authors ofthis survey directly participated in the de-sign and implementation of each of thesesystems. Because it is difficult to choose arepresentative set of case studies, the se-lection of systems was guided by the au-thors’ personal experience with the chosensystems.

5.1. MOSIX

MOSIX is a distributed operating systemfrom the Hebrew University of Jerusalem.MOSIX is an ongoing project which be-gan in 1981 and released its most recentversion in 1996. Automatic load balanc-ing between MOSIX nodes is done by pro-cess migration. Other interesting features



Fig. 6. The MOSIX Architecture.

include full autonomy of each node in thesystem, fully-decentralized control, singlesystem image, dynamic configuration andscalability.

Various versions of MOSIX have been inactive use at the Hebrew University since1983. The original version of MOSIX wasderived from UNIX Version 7 and ran ona cluster of PDP-11/45 nodes connected bya token passing ring [Barak and Litman,1985]. The version of MOSIX documentedin the MOSIX book is a cluster of multipro-cessor workstations which used a UNIXSystem V.2 code base [Barak and Wheeler,1989; Barak et al., 1993]. The most recentversion, developed in 1993, is called NOWMOSIX [Barak et al., 1995]. This versionenhances BSDI UNIX by providing pro-cess migration on a cluster of Intel Pen-tium processor based workstations.

Goals of the MOSIX system include:

r Dynamic process migration. At contextswitch time, a MOSIX node may elect tomigrate any process to another node. Themigrated process is not aware of the mi-gration.r Single system image. MOSIX presents aprocess with a uniform view of the filesystem, devices and networking facili-ties regardless of the process’s currentlocation.r Autonomy of each node. Each node in thesystem is independent of all other nodesand may selectively participate in theMOSIX cluster or deny services to other

nodes. Diskless nodes in MOSIX rely ona specific node for file services.r Dynamic configuration. MOSIX nodesmay join or leave a MOSIX cluster at anytime. Processes that are not running ona node or using some node specific re-source, are not affected by the loss of thatnode.r Scalability. System algorithms avoid us-ing any global state. By avoiding depen-dence on global state or centralized con-trol, the system enhances its ability toscale to a large number of nodes.

Design. The system architecture sepa-rates the UNIX kernel into a lower andan upper kernel. Each object in MOSIX,like an open file, has a universal objectpointer that is unique across the MOSIXdomain. Universal objects in MOSIX arekernel objects (e.g. a file descriptor entry)that can reference an object anywhere inthe cluster. For example, the upper kernelholds a universal object for an open file;the universal object migrates with the pro-cess while only the host of the file has thelocal, non-universal file information. Theupper kernel provides a traditional UNIXsystem interface. It runs on each node andhandles only universal objects. The lowerkernel provides normal services, such asdevice drivers, context switching, and soon without having any knowledge or de-pendence on other nodes. The third com-ponent of the MOSIX system is the linker,which maps universal objects into local ob-jects on a specific node, and which providesinternode communication, data transfer,process migration and load balancing al-gorithms. When the upper kernel needs toperform an operation on one of the univer-sal objects that it is handling, it uses thelinker to perform a remote kernel proce-dure call on the object’s host node.

MOSIX transfers only the dirty pagesand user area of the migrating processat the time of the migration, an eager(dirty) transfer strategy. Text and otherclean pages are faulted in as needed oncethe process resumes execution on the tar-get node.

Process migration in MOSIX is a com-mon activity. A process has no explicit



knowledge about what node it is actuallyrunning on or any guarantees that it willcontinue to run on its current node. Themigration algorithm is cooperative: for aprocess to migrate to a node, the targetnode must be willing to accept it. Thisallows individual nodes control over theextent of their own contribution to theMOSIX system. Individual nodes can alsoforce all active processes to migrate away,a procedure that is used when shuttingdown an individual node.

Process migration in MOSIX relies onthe fact that the upper kernel contextof each process is site-independent: re-gardless of where the process physicallyruns, its local upper kernel and linkerroute each system call to the appropri-ate node. If the process decides to migrateto a new node, the migration algorithmqueries the new node to ensure that itis willing to accept a new process. If so,the upper kernel invokes a series of re-mote kernel procedure calls that createan empty process frame on the new node,moves the upper kernel context and anydirty pages associated with the processand then resumes the process on the newnode.

Fault Resilience. Failed nodes inMOSIX affect only processes running onthe failed node or directly using resourcesprovided by the node. Nodes dynamicallyjoin and leave a MOSIX cluster at will.Detection of stale objects—those that sur-vive past the reboot of the object’s server—is done by maintaining per object versionnumbers. (As an example of a stale object,a universal pointer to a file object mustbe reclaimed after the home node for thefile reboots.) Migrated processes leave notraces on other nodes.

Transparency. Migration is com-pletely transparent in MOSIX, except forprocesses that use shared memory andare not eligible for migration. Full singlesystem image semantics are presentedby MOSIX, making processes unaware oftheir actual physical node. A new systemcall, migrate(), was added to allow pro-cesses to determine the current locationor to request migration to a specifiednode.

Scalability. MOSIX was designed asa scalable system. The system relies onno centralized servers and maintains noglobal information about the system state.Each MOSIX node is autonomous and candynamically join or withdraw from theMOSIX system. No remote system oper-ations involve more than two nodes: theinitiating node and the node providing theservice. The process migration and loadbalancing algorithms also support scala-bility: load information is totally decen-tralized. Currently, an 80-node MOSIXsystem is running at Hebrew University.

Load Information Management andDistributed Scheduling. Several typesof information are managed by MOSIX inorder to implement its dynamic load bal-ancing policy: the load at each node, indi-vidual process profiling, and load informa-tion about other nodes in the system.

Each node computes a local load esti-mate that reflects the average length ofits ready queue over a fixed time period.By selecting an appropriate interval, theimpact of temporary local load fluctua-tions is reduced without presenting obso-lete information.

For each process in the system, an exe-cution profile is maintained which reflectsits usage of remote resources like files orremote devices, communication patternswith other nodes, how long this processhas run and how often it has created newchild processes via the fork() system call.This information is useful in determin-ing where a process should migrate towhen selected for migration. For example,a small process that is making heavy useof a network interface or file on a specificnode would be considered for migrationto that node. This profiling information isdiscarded when a process terminates.

The MOSIX load balancing algorithmis decentralized. Each node in the systemmaintains a small load information vectorabout the load of a small subset of othernodes in the system [Barak et al., 1989].On each iteration of the algorithm, eachnode randomly selects two other nodes, ofwhich at least one node is known to havebeen recently alive. Each of the selectednodes is sent the most recent half of the



local load vector information. In addition,when a load information message is re-ceived, the receiving node acknowledgesreceipt of the message by returning itsown load information back to the sendingnode.

During each iteration of the algorithm,the local load vector is updated by incorpo-rating newly received information and byaging or replacing older load information.To discourage migration between nodeswith small load variations, each node ad-justs its exported local load informationby a stability factor. For migration to takeplace, the difference in load values be-tween two nodes must exceed this stabilityvalue.

The load balancing algorithm decides tomigrate processes when it finds anothernode with a significantly reduced load. Itselects a local process that has accumu-lated a certain minimum amount of run-time, giving preference to processes whichhave a history of forking off new subpro-cesses or have a history of communica-tion with the selected node. This preventsshort-lived processes from migrating.

Implementation and Performance.Porting the original version of MOSIX to anew operating system base required sub-stantial modifications to the OS kernel inorder to layer the code base into the threeMOSIX components (linker, lower and up-per kernels). Few changes took place atthe low level operating system code [Barakand Wheeler, 1989].

In order to reduce the invasiveness ofthe porting effort, a completely redesignedversion of NOW MOSIX was developed forthe BSDI version of UNIX [Barak et al.,1995]. The NOW MOSIX provides processmigration and load balancing. without asingle system image. As in Sprite, systemcalls that are location sensitive are for-warded to the home node of a migratedprocess as required (cf. Section 5.2).

The performance of a migrated pro-cess in MOSIX depends on the natureof the process. One measurement of theeffect that migration has on a processis the slower performance of remote sys-tem calls. Using the frequencies of systemcalls measured by Douglis and Ouster-

Table 1. MOSIX System Call Performance

System Call Local Remote Slowdown

read (1K) 0.34 1.36 4.00write (1K) 0.68 1.65 2.43open/close 2.06 4.31 2.09fork (256 Kb) 7.8 21.60 2.77exec (256 KB) 25.30 51.50 2.04

hout [1987], system calls were 2.8 timesslower when executed on a remote 33MHzMOSIX node [Barak et al., 1989]. Table 1shows the measured performance andslowdown of several commonly used sys-tem calls. Many system calls, for examplegetpid (), are always performed on the pro-cesse’s current node and have no remoteperformance degradation.

The performance of the MOSIX mi-gration algorithm depends directly onthe performance of the linker’s datatransfer mechanism on a given networkand the size of the dirty address spaceand user area of the migrating process.The measured performance of the VME-based MOSIX migration, from one nodeof the cluster to the bus master, was1.2 MB/second. The maximum data trans-fer speed of the system’s VME bus was3 MB/second.

Some applications benefit significantlyfrom executing in parallel on multiplenodes. In order to allow such applicationsto run on a system without negatively im-pacting everyone else, one needs processmigration in order to be able to rebalanceloads when necessary. Arguably the mostimportant performance measurement isthe measurement of an actual user-levelapplication. Specific applications, for ex-ample an implementation of a graph color-ing algorithm, show a near-linear speedupwith increasing number of nodes [Baraket al., 1993]. Of course, this speedup doesnot apply to other types of applications(non-CPU-bound, such as network or I/Obound jobs). These applications may ex-perience different speedups. No attempthas been conducted to measure an averagespeedup for such types of applications.

Lessons Learned. The MOSIX sys-tem demonstrated that dynamic load bal-ancing implemented via dynamic process



migration is a viable technology fora cluster of workstations. The earlierMOSIX implementations required toomany changes to the structure of the baseoperating system code in order to main-tain the single system image nature ofthe system. Giving up the single systemimage while preserving process migrationdelivers most of the benefits of the earlierMOSIX systems without requiring inva-sive kernel changes.

5.2. Sprite

The Sprite Network Operating Systemwas developed at U.C. Berkeley between1984 and 1994 [Ousterhout et al., 1988].Its primary goal was to treat a networkof personal workstations as a time-sharedcomputer, from the standpoint of shar-ing resources, but with the performanceguarantees of individual workstations. Itprovided a shared network file systemwith a single-system image and a fully-consistent cache that ensured that allmachines always read the most recentlywritten data [Nelson et al., 1988]. Thekernel implemented a UNIX-like proce-dural interface to applications; internally,kernels communicated with each other viaa kernel-to-kernel RPC. User-level IPCwas supported using the file system, witheither pipes or a more general mech-anism called pseudo-devices [Welch andOusterhout, 1988]. Virtual memory wassupported by paging a process’s heap andstack segments to a file on the local diskor a file server.

An early implementation of migrationin Sprite [Douglis and Ousterhout, 1987]suffered from some deficiencies [Douglis,1989]:r processes accessing some types of files,

such as pseudo-devices, could not be mi-grated;r there was no automatic host selection;andr there was no automatic failure recovery.

After substantial modifications to theshared file system to support increasedtransparency and failure recovery [Welch,1990], migration was ported to Sun-3

workstations, and later Sparcstation andDECstation machines. Automatic host se-lection went through multiple iterationsas well, moving from a shared file to aserver-based architecture. Migration wasused regularly starting in the fall of 1988.

Goals:r Workstation autonomy. Local users hadpriority over their workstation. Dynamicprocess migration, as opposed to merelyremote invocation, was viewed primarilyas a mechanism to evict other users’ pro-cesses from a personal workstation whenthe owner returned. In fact, withoutthe assurance of local autonomy throughprocess migration, many users would nothave allowed remote processes to starton their workstation in the first place.r Location transparency. A process wouldappear to run on a single workstationthroughout its lifetime.r Using idle cycles. Migration was meantto take advantage of idle workstations,but not to support full load balancing.r Simplicity. The migration system triedto reuse other support within the Spritekernel, such as demand paging, even atthe cost of some performance. For ex-ample, migrating an active process fromone workstation to another would re-quire modified pages in its address spaceto be written to a file server and faultedin on the destination, rather than sentdirectly to the destination.

Design. Transparent migration inSprite was based on the concept of a homemachine. A foreign process was one thatwas not executing on its home machine.Every process appeared to run on itshome machine throughout its lifetime,and that machine was inherited by de-scendants of a foreign process as well.Some location-dependent system calls bya foreign process would be forwarded au-tomatically, via kernel-to-kernel RPC, toits home; examples include calls dealingwith the time-of-day clock and processgroups. Numerous other calls, such asfork and exec, required cooperation be-tween the remote and home machines.Finally, location-independent calls, which



included file system operations, could behandled locally or sent directly to themachine responsible for them, such as afile server.

Foreign processes were subject toeviction—being migrated back to theirhome machine—should a local user re-turn to a previously idle machine. Whena foreign process migrated home, it leftno residual dependencies on its formerhost. When a process migrated away fromits home, it left a shadow process therewith some state that would be used tosupport transparency. This state includedsuch things as process identifiers and theparent-child relationships involved in theUNIX wait call.

As a performance optimization, Spritesupported both full process migration, inwhich an entire executing process wouldmigrate, and remote invocation, in whicha new process would be created on a dif-ferent host, as though a fork and execwere done together (like the Locus runcall [Walker et al., 1983]). In the lattercase, state that persists across an exec call,such as open files, would be encapsulatedand transferred, but other state such asvirtual memory would be created from anexecutable.

When migrating an active process,Sprite writes dirty pages and cached fileblocks to their respective file server(s). Theaddress space, including the executable,is paged in as necessary. Migration in theform of remote invocation would result indirty cached file blocks being written, butwould not require an address space to beflushed, since the old address space is be-ing discarded.

The migration algorithm consists of thefollowing steps [Douglis, 1989]:

1. The process is signaled, to cause it totrap into the kernel.

2. If the process is migrating away fromits home machine, the source contactsthe target to confirm its availabilityand suitability for migration.

3. A “pre-migration” procedure is invokedfor each kernel module. This returnsthe size of the state that will be trans-ferred and can also have side effects,

such as queuing VM pages to be flushedto the file system.

4. The source kernel allocates a bufferand calls encapsulation routines foreach module. These too can have sideeffects.

5. The source kernel sends the buffer viaRPC, and on the receiving machine eachmodule de-encapsulates its own state.The target may perform other opera-tions as a side effect, such as commu-nicating with file servers to arrange forthe transfer of open files.

6. Each kernel module can execute a “post-migration” procedure to clean up state,such as freeing page tables.

7. The source sends an RPC to tell the tar-get to resume the process, and frees thebuffer.

Fault Resilience. Sprite process mi-gration was rather intolerant of faults.During migration, the failure of the targetanytime after step 5 could result in the ter-mination of the migrating process, for ex-ample, once its open files have been movedto the target. After migration, the failureof either the home machine or the process’scurrent host would result in the termina-tion of the process. There was no facility tomigrate away from a home machine thatwas about to be shut down, since therewould always be some residual dependen-cies on that machine.

Transparency was achieved througha conspiracy between a foreign process’scurrent and home workstations. Opera-tions on the home machine that involved aforeign process, such as a ps listing of CPUtime consumed, would contact its currentmachine via RPC. Operations on the cur-rent host involving transparency, includ-ing all process creations and terminations,contacted the home machine. Waiting for achild, even one co-resident on the foreignmachine, would be handled on the homemachine for simplicity.

All IPC in Sprite was through the filesystem, even TCP connections. (TCP wasserved through user-level daemons con-tacted via pseudo-devices.) The sharednetwork file system provided transparent



access to files or processes from differentlocations over time.

As in MOSIX, processes that sharememory could not be migrated. Also, pro-cesses that map hardware devices directlyinto memory, such as the X server, couldnot migrate.

Scalability. Sprite was designed for acluster of workstations on a local area net-work and did not particularly address theissue of scalability. As a result, neitherdid the migration system. The central-ized load information management sys-tem, discussed next, could potentially bea bottleneck, although a variant based onthe MOSIX probabilistic load dissemina-tion algorithm was also implemented. Inpractice, the shared file servers proved tobe the bottleneck for file-intensive opera-tions such as kernel compilations with asfew as 4-5 hosts, while cpu-intensive simu-lations scaled linearly with over ten hosts[Douglis, 1990].

Load Information Management. Aseparate, user-level process (migd) wasresponsible for maintaining the state ofeach host and allocating idle hosts to ap-plications. This daemon would be startedon a new host if it, or its host, shouldcrash. It allocated idle hosts to request-ing processes, up to one foreign “job” peravailable processor. (A “job” consisted ofa foreign process and its descendants.)It supported a notion of fairness, in thatone application could use all idle hostsof the same architecture but would havesome of them reclaimed if another ap-plication requested hosts as well. Re-claiming due to fairness would look tothe application just like reclaiming dueto a workstation’s local user returning:the foreign processes would be migratedhome and either run locally, migratedelsewhere, or suspended, depending ontheir controlling task’s behavior and hostavailability.

Migration was typically performed bypmake, a parallel make program like manyothers that eventually became common-place (e.g., [Baalbergen, 1988]) Pmakewould use remote invocation and thenremigrate processes if migd notified itthat any of its children were evicted. It

would suspend any process that could notbe remigrated.

Implementation and Performance.Sprite ran on Sun (Sun 2, Sun 3,Sun 4, SPARCstation 1, SPARCstation 2)and Digital (DECstation 3100 and 5100)workstations. The entire kernel consistedof approximately 200,000 lines of heav-ily commented code, of which approximate10,000 dealt with migration.

The performance of migration in Spritecan be measured in three respects. Allmeasurements in this subsection weretaken on SPARCstation 1 workstationson a 10-Mbps Ethernet, as reported in[Douglis and Ousterhout, 1991].

1. The time to migrate a process was afunction of the overhead of host selec-tion (36ms to select a single host, amor-tized over multiple selections when mi-gration is performed in parallel); thestate for each open file (9.4ms/file);dirty file and VM blocks that mustbe flushed (480-660 Kbytes/second de-pending on whether they are flushedin parallel); process state such as execarguments and environment variablesduring remote invocation (also 480Kbytes/second); and a basic overheadof process creation and message traffic(76ms for the null process).

2. A process that had migrated away fromits home machine incurred run-timeoverhead from forwarding location-dependent system calls. Applications ofthe sort that were typically migrated inSprite, such as parallel compilation andLaTeX text processing, incurred only1-3% degradation from running re-motely, while other applications thatinvoked a higher fraction of location-dependent operations (such as access-ing the TCP daemon on the home ma-chine, or forking children repeatedly)incurred substantial overhead.

3. Since the purpose of migration in Spritewas to enable parallel use of manyworkstations, application speedup isan important metric. Speedup is af-fected by a number of factors, in-cluding the degree of parallelism, theload on central resources such as the



migd daemon, and inherently non-parallelizable operations. By compar-ing the parallel compilation of sev-eral source directories, ranging from24 to 276 files and 1 to 3 indepen-dent link steps, one found that thespeedup compared to the sequentialcase ranged from about 3 to 5.4 usingup to 12 hosts, considerably below lin-ear speedup. During a 12-way pmake,the processors on both the server stor-ing the files being read and written, andthe workstation running pmake, weresaturated. Network utilization was nota significant problem, however.

Lessons Learned. Here we summa-rize the two most important lessons andexperiences in Sprite process migration[Douglis, 1990; Douglis and Ousterhout,1991].r Migration provided a considerable

source of processor cycles to the Spritecommunity. Over a one-month period,30% of user processor activity came frommigrated (foreign) processes. The hostthat accounted for the greatest total us-age (nearly twice as many cpu-secondsas the next greatest) ran over 70% of itscycles on other hosts.r Evictions accounted for 6% of all migra-tions, with about half of these evictionsdue to fairness considerations and theother half due to users reclaiming theirmachines. About 1% of all host alloca-tions were revoked for one of these tworeasons. (Evictions counted for a rela-tively higher fraction of all migrationsbecause one host revocation could resultin many processes being migrated.)

5.3. Mach

Mach is a microkernel developed atthe Carnegie Mellon University [Accettaet al., 1986; Black et al., 1992], and laterat the OSF Research Institute [Bryant,1995]. A migration mechanism on top ofthe Mach microkernel was developed atthe University of Kaiserslautern, from1991 to 1993 [Milojicic et al., 1993b].

Task migration was used for exper-iments with load distribution. In this

Fig. 7. Task Migration Design.Only task abstraction is migrated,while process abstraction remains onthe source node.

phase, only tasks were addressed, whileUNIX processes were left on the sourcemachine, as described in Figure 7. Thismeans that only Mach task state was mi-grated, whereas the UNIX process statethat was not already migrated as a part ofthe Mach task state (e.g. state in the UNIX“personailty server” emulating UNIX ontop of the Mach microkernel) remained onthe source machine. Therefore, most of theUNIX system calls were forwarded back tothe source machine, while only Mach sys-tem calls were executed on the destinationmachine.

Process migration for the OSF/1 AD 1server [Paindaveine and Milojicic, 1996]was developed during 1994 at the Univer-site Catholique de Louvain, Belgium, as apart of a project on load-leveling policiesin a distributed system [Jacqmot, 1996].OSF/1 AD 1 is a version of the OSF/1operating system which provides a scal-able, high-performance single-system im-age version of UNIX. It is composed ofservers distributed across the differentnodes running the Mach microkernel. Pro-cess migration relies on the Mach task mi-gration to migrate microkernel-dependentprocess state between nodes.

Mach task migration was also used atthe University of Utah, for the Schizoproject [Swanson et al., 1993]. Task andprocess migration on top of Mach weredesigned and implemented for clusters ofworkstations.

Goals. The first goal was to provide atransparent task migration at user-level



with minimal changes to the microkernel.This was possible by relying on Mach OSmechanisms, such as (distributed) mem-ory management and (distributed) IPC.The second goal was to demonstrate that itis possible to perform load distribution atthe microkernel level, based on the threedistinct parameters that characterize mi-crokernels: processing, VM and IPC.

Design. The design of task migrationis affected by the underlying Mach mi-crokernel. Mach supported various pow-erful OS mechanisms for purposes otherthan task and process migration. Exam-ples include Distributed Memory Manage-ment (DMM) and Distributed IPC (DIPC).DIPC and DMM simplified the design andimplementation of task migration. DIPCtakes care of forwarding messages to mi-grated process, and DMM supports remotepaging and distributed shared memory.The underlying complexity of messageredirection and distributed memory man-agement are heavily exercised by taskmigration, exposing problems otherwisenot encountered. This is in accordancewith earlier observations about message-passing [Douglis and Ousterhout, 1991].

In order to improve robustness and per-formance of DIPC, it was subsequentlyredesigned and reimplemented [Milojicicet al., 1997]. Migration experiments havenot been performed with the improvedDIPC. However, extensive experimentshave been conducted with Concurrent Re-mote Task Creation (CRTC), an in-kernelservice for concurrent creation of remotetasks in a hierarchical fashion [Milojicicet al., 1997]. The CRTC experiments aresimilar to task migration, because a re-mote fork of a task address space is per-formed.

DMM enables cross-node transparencyat the Mach VM interface in supportof a distributed file system, distributedfile system, distributes processes, and dis-tributed shared memory [Black, et al.,1998]. The DMM support resulted in sim-plified design and implementation of thefunctionality built on top of it, such asSSI UNIX and remote tasking, and itavoided pager modifications by interpos-ing between the VM system and the

pager. However, the DMM became toocomplex, and had performance and scal-ability problems. The particular designmistakes include the interactions betweenDSM support and virtual copies in a dis-tributed system; transparent extension ofMach copy-on-write VM optimization todistributed systems; and limitations im-posed by Mach’s external memory man-agement while transparently extending itto distributed systems. (Copy-on-write isan optimization introduced to avoid copy-ing pages until it is absolutely needed,and otherwise sharing the same copy. Ithas also been used in Chorus [Rozier,1992] and Sprite [Nelson and Ousterhout,1988].)

DMM had too many goals to be success-ful; it failed on many general principles,such as “do one thing, but do it right,” and“optimize the common case” [Lampson,1983]. Some of the experiments with taskmigration reflect these problems. Varia-tions of forking an address space and mi-grating a task significantly suffered inperformance. While some of these casescould be improved by optimizing the al-gorithm (as was done in the case of CRTC[Milojicic et al., 1997]), it would only addto an already complex and fragile XMMdesign and implementation. Some of theDMM features are not useful for task mi-gration, even though they were motivatedby task migration support. Examples in-clude DSM and distributed copy-on-writeoptimizations. DSM is introduced in orderto support the transparent remote forkingof address spaces (as a consequence of re-mote fork or migration) that locally sharememory. Distributed copy-on-write is mo-tivated by transparently forking addressspaces that are already created as a con-sequence of local copy-on-write, as well asin order to support caching in distributedcase.

Even though the DIPC and DMM inter-faces support an implementation of user-level task migration, there are two excep-tions. Most of the task state is accessiblefrom user space except for the capabilitiesthat represent tasks and threads and ca-pabilities for internal memory state. Twonew interfaces are provided for exporting



the aforementioned capabilities into userspace.

A goal of one of the user-space migrationservers is to demonstrate different datatransfer strategies. An external memorymanager was used for implementation ofthis task migration server. The followingstrategies were implemented: eager copy,flushing, copy-on-reference, precopy andread-ahead [Milojicic et al., 1993b]. Formost of the experiments, a simplified mi-gration server was used that relied on thedefault in-kernel data transfer strategy,copy-on-reference.

The task migration algorithm stepsare:

1. Suspend the task and abort the threadsin order to clean the kernel state.1

2. Interpose task/thread kernel ports onthe source node.

3. Transfer the address space, capabili-ties, threads and the other task/threadstate.

4. Interpose back task/thread kernel portson the destination node.

5. Resume the task on the destinationnode.

Process state is divided into several cat-egories: the Mach task state; the UNIXprocess local state; and the process-relationship state. The local process statecorresponds to the typical UNIX proc anduser structures. Open file descriptors, al-though part of the UNIX process state, aremigrated as part of the Mach task state.

Fault Resilience of Mach task migra-tion was limited by the default transferstrategy, but even more by the DIPC andDMM modules. Both modules heavilyemploy the lazy evaluation principle,leaving residual dependencies throughoutthe nodes of a distributed system. Forexample, in the case of DIPC, proxies ofthe receive capabilities remain on thesource node after receive capability ismigrated to a remote node. In the case

1 Aborting is necessary for threads that can wait inthe kernel arbitrarily long, such as in the case of wait-ing for a message to arrive. The wait operation isrestartable on the destination node.

of DMM, the established paging pathsremain bound to the source node evenafter eager copying of pages is performedto the destination node.

Transparency was achieved by delay-ing access or providing concurrent accessto a migrating task and its state duringmigration. The other tasks in the sys-tem can access the migrating task eitherby sending messages to the task kernelport or by accessing its memory. Send-ing messages is delayed by interposingthe task kernel port with an interposeport. The messages sent to the interposeport are queued on the source node andthen restarted on the destination node.The messages sent to other task portsare transferred as a part of migration ofthe receive capabilities for these ports.Access to the task address space is sup-ported by DMM even during migration.Locally shared memory between two tasksbecomes distributed shared memory aftermigration of either task.

In OSF/1 AD, a virtual process (Vprocs)framework supports transparent opera-tions on the processes independently ofthe actual process’s location [Zajcew et al.,1993]. By analogy, vprocs are to processeswhat vnodes are to files, both providing lo-cation and heterogeneity transparency atthe system call interface. Distributed pro-cess management and the single systemimage of Mach and OSF/1 AD eased theprocess migration implementation.

A single system image is preserved byretaining the process identifier and byproviding transparent access to all UNIXresources. There are no forwarding stubprocesses or chains. No restrictions areimposed on the processes considered formigration: for example, using pipes or sig-nals does not prevent a process from beingmigrated.

Scalability. The largest system thatMach task migration ran on at Universityof Kaiserslautern consisted of five nodes.However, it would have been possible toscale it closer towards the limits of thescalability of the underlying Mach micro-kernel, which is up to a couple of thou-sand nodes on the Intel Paragon super-computer.



Table 2. Processing, IPC and VM Intensive Applications

User/Total IPC VMType Application Time (msg/s) ((pagin+ out)/s)

Processing Dhrystone 1.00 3.49 0.35+ 0IPC find 0.03 512.3 2.75+ 0VM WPI Jigsaw 0.09 2.46 28.5+38.2

Migration of the address space reliesheavily on the Mach copy-on-write VM op-timization, which linearly grows the inter-nal VM state for each migration [Milojicicet al., 1997]. In practice, when there arejust few migrations, this anomaly is notnoticeable. However for many consecutivemigrations it can reduce performance.

Load Information and DistributedScheduling. Mach was profiled to reflectremote IPC and remote paging activ-ity in addition to processing information.This information was used to improveload distribution decisions. [Milojicic,1993c]. Profiling was performed insideof the microkernel by collecting statis-tics for remote communication and for re-mote paging and in user space, by in-terposing application ports with profilerports.

A number of applications were profiledand classified in three categories: process-ing, communication and paging intensive.Table 2 gives representatives of each class.

Extended load information is used forapplying more appropriate distributedscheduling decisions [Milojicic et al.,1993a]. An application that causes asignificant amount of remote paging,or communicates with another node, isconsidered for migration to the appropri-ate node. CPU-bound applications haveno such preference and can be migratedbased only on the processing load criteria.For applications consisting of a number ofprocesses that communicate among them-selves, improvements achieved by con-sidering IPC/VM information in additionto CPU load is proportional to the loadand it can reach up to 20-50% for dis-tributed scheduling strategies [Milojicicet al., 1993a]. Improvements of the perfor-mance of a simple application due to local-ity of reference can be multifold [Milojicicet al., 1993b].

Implementation and Performance.Milojicic et al. built three implementa-tions: two user-level migrations (an op-timized and a simple migration server);and a kernel implementation. The size ofthe simplified migration server is approx-imately 400 lines of code that took about 3months to design and implement. A lot ofthis time was spent in debugging the DIPCparts of code that were never before exer-cised. Task migration, especially load dis-tribution experiments using task migra-tion, turned out to be a very stressful testfor DIPC.

The size of the in-kernel version is closeto the simplified migration server, fromwhich it was derived. These two imple-mentations relied on the in-kernel sup-port for address space transfer. However,the size of the DIPC and DMM moduleswas significantly higher. One of the lat-est versions of optimized DIPC (nmk21b1)consisted of over 32,000 lines of code. Ittook over 10 engineer-years to release thesecond version of DIPC. The DMM, whichwas never optimized, consists of 24,000lines of code.

The optimized migration server islargest in size with a few thousand linesof code. Most of this implemented a pagersupporting different data transfer strate-gies. The optimized migration server didnot rely on in-kernel data transfer strat-egy, except for the support of distributedshared memory.

Although there is an underlying dis-tributed state in the microkernel, no dis-tributed state is involved in the processmigration facility at the server level, ren-dering the design of the migration mech-anism simple. The process migration codeconsists of approximately 800 lines of code.However, adding distributed process man-agement requires about 5000 lines of addi-tional code. The main (initial and runtime)



Fig. 8. Task migration performance as a function of VM size: initialcosts are independent of task address space size (aside of variationsdue to other side effects).

costs of migration are due to task mi-gration. Process migration has very littleoverhead in addition to task migration.

Performance measurements were con-ducted on a testbed consisting of three In-tel 33MHz 80486 PCs with 8MB RAM. TheNORMA14 Mach and UNIX server UX28were used. Performance is independent ofthe address space size (see Figure 8), andis a linear function of the number of ca-pabilities. It was significantly improved insubsequent work [Milojicic et al., 1997].

Lessons Learned

r Relying on DIPC and DMM is crucialfor the easy design and implementationof transparent task migration, but thesemodules also entail most of the complex-ity and they limit performance and faultresilience.r Task migration is sufficient for mi-crokernel applications. In contrast, asmentioned above, UNIX applicationswould forward most system calls backto the source node, resulting in anorder-of-magnitude performance degra-dation. Migrating the full UNIX pro-

cess state would presumably havealleviated this overhead, similar tothe evolution in Sprite toward dis-tinguishing between location-dependentand location-independent calls [Douglis,1989].r Applications on microkernels can be pro-filed as a function of processing, IPC andVM and this information can be usedfor improved load distribution. Improve-ment ranges from 20-55% for collabora-tive types of applications.

5.4. LSF

LSF (Load Sharing Facility) is a load shar-ing and batch scheduling product fromPlatform Computing Corporation [Plat-form Computing, 1996]. LSF is based onthe Utopia system developed at the Uni-versity of Toronto [Zhou et al., 1994],which is in turn based on the earlier Ph.D.thesis work of Zhou at UC Berkeley [Zhou,1987; Zhou and Ferrari, 1988].

LSF provides some distributed op-erating system facilities, such as dis-tributed process scheduling and transpar-ent remote execution, on top of various



operating system kernels without change.LSF primarily relies on initial processplacement to achieve load balancing, butalso uses process migration via check-pointing as a complement. LSF currentlyruns on most UNIX-based operating sys-tems.

Checkpointing and MigrationMechanisms. LSF’s support for processuser-level process migration is based onCondor’s approach [Litzkow and Solomon,1992]. A checkpoint library is providedthat must be linked with application code.Part of this library is a signal handlerthat can create a checkpoint file of theprocess so that it can be restarted on amachine of compatible architecture andoperating system. Several improvementshave been made to the original Condorcheckpoint design, such as:

r No core dump is required in order to gen-erate the checkpoint file. The runningstate of the process is directly extractedand saved in the checkpoint file togetherwith the executable in a format that canbe used to restart the process. This notonly is more efficient, but also preservesthe original process and its ID across thecheckpoint.r UNIX signals can be used by the check-pointed process. The state informationconcerning the signals used by the pro-cess is recorded in the checkpoint file andrestored at restart time.

In addition to user-level transparentprocess checkpointing, LSF can also takeadvantage of checkpointing already sup-ported in the OS kernel (such as in CrayUnicos and ConvexOS), and application-level checkpointing. The latter is achiev-able in classes of applications by the pro-grammer writing additional code to savethe data structures and execution state in-formation in a file that can be interpretedby the user program at restart time inorder to restore its state. This approach,when feasible, often has the advantage ofa much smaller checkpoint file because itis often unnecessary to save all the dirtyvirtual memory pages as must be donein user-level transparent checkpointing.

Application-level checkpointing may alsoallow migration to work across heteroge-neous nodes.

The checkpoint file is stored in a user-specified directory and, if the directory isshared among the nodes, the process maybe restarted on another node by accessingthis file.

Load Information Exchange. Simi-lar to Sprite, LSF employs a centralizedalgorithm for collecting load information.One of the nodes acts as the master, andevery other node reports its local load tothe master periodically. If the master nodefails, another node immediately assumesthe role of the master. The scheduling re-quests are directed to the master node,which uses the load information of all thenodes to select the one that is likely to pro-vide the best performance.

Although many of the load informationupdates may be wasted if no process needto be scheduled between load informa-tion updates, this algorithm has the ad-vantage of making (reasonably up-to-date)load information of all nodes readily avail-able, thus reducing the scheduling delayand considering all nodes in scheduling.Zhou et al. [1994] argue that the networkand CPU overhead of this approach isnegligible in modern computers and net-works. Measurements and operational ex-perience in clusters of several hundredhosts confirm this observation. Such a cen-tralized algorithm also makes it possibleto coordinate all process placements - oncea process is scheduled to run on a node,this node is less likely to be considered forother processes for a while to avoid over-loading it. For systems with thousandsof nodes, several clusters can be formed,with selective load information exchangeamong them.

Scheduling Algorithms. LSF usescheckpoint and restart to achieve processmigration, which in turn is used to achieveload balancing. If a node is overloaded orneeded by some higher priority processes,a process running on it may be migratedto another node. The load conditions thattrigger process migration can be config-ured to be different for various types ofjobs. To avoid undesirable migration due



to temporary load spikes and to controlmigration frequency, LSF allows users tospecify a time period for which a processis suspended on its execution node. Onlyif the local load conditions remain unfa-vorable after this period would the sus-pended process be migrated to anothernode.

The target node is selected based on thedynamic load conditions and the resourcerequirements of the process. Recognizingthat different processes may require dif-ferent types of resources, LSF collects avariety of load information for each node,such as average CPU run queue length,available memory and swap space, diskpaging and I/O rate, and the duration ofidle period with no keyboard and mouseactivities. Correspondingly, a process maybe associated with resource requirementexpressions such as

select[sparc && swap >= 120 &&mem >= 64] order[cpu : mem]

which indicates that the selected nodeshould have a resource called “sparc,” andshould have at least 120 MB of swap spaceand 64 MB of main memory. Among theeligible nodes, the one with the fastest,lightly loaded CPU, as well as large mem-ory space, should be selected. A heuristicsorting algorithm is employed by LSF toconsider all the (potentially conflicting) re-source preferences and select a suitablehost. Clearly, good host allocation can onlybe achieved if the load condition of allnodes is known to the scheduler.

The resource requirements of a processmay be specified by the user when sub-mitting the process to LSF, or may beconfigured in a system process file alongwith the process name. This process fileis automatically consulted by LSF to de-termine the resource requirement of eachtype of process. This process file also storesinformation on the eligibility of each typeof process for remote execution and mi-gration. If the name of a process is notfound in this file, either it is excluded frommigration consideration, or only nodes ofthe same type as the local node would beconsidered.

Process Migration vs. Initial Place-ment. Although LSF makes use of pro-cess migration to balance the load, it isused more as an exception rather than therule, for three reasons. First, transparentuser-level checkpointing and migrationare usable by only those processes linkedwith the checkpoint library, unless the OSkernel can be modified; in either case,their applicability is limited. Secondly,intelligent initial process placement hasbeen found to be effective in balancing theload in many cases, reducing the need formigration [Eager et al., 1988]. Finally, andperhaps most importantly, the same loadbalancing effect can often be achieved byprocess placement with much less over-head. The remote process execution mech-anism in LSF maintains the connectionbetween the application and the RemoteExecution Server on the execution nodeand caches the application’s execution con-text for the duration of the application ex-ecution, so that repeated remote processexecutions would incur low overhead (0.1seconds as measured by Zhou et al. on anetwork of UNIX workstations [1994]).

In contrast, it is not desirable to main-tain per-application connections in a ker-nel implementation of process migrationto keep the kernel simple, thus every pro-cess migration to a remote node is “cold”.Per-application connections and cachedapplication state are rather “heavyweight”for kernel-level migration mechanisms,and the kernel-level systems surveyedin this paper treat each migration sepa-rately (though the underlying communi-cation systems, such as kernel-to-kernelRPC, may cache connection state). Thebenefits of optimizing remote executionare evident by comparing LSF to an ear-lier system such as Sprite. In the case ofSprite, the overhead of exec time migra-tion was measured to be approximately330ms on Sparcstation 1 workstationsover the course of one month [Douglisand Ousterhout, 1991]. Even taking dif-ferences in processor speed into accountas well as underlying overheads such asfile system cache flushing, LSF shows amarked improvement in remote invoca-tion performance.



6. COMPARISON

In this section, we compare the variousmigration implementations described inthe paper. We cover the case studies, aswell as some other systems mentioned inSection 4.

Table 3 summarizes the process migra-tion classification provided in Section 4.We mention examples of each class ofmigration, followed by the main charac-teristic of each class. These columns areself-explanatory. The OS v. ApplicationModification column describes wherethe majority of modifications to supportmigration are performed. Migration in theearly work, UNIX-like systems, messagepassing and microkernels require modifi-cations to the underlying OS. User-spaceand application-specific systems requiremodifications to the application, typicallyrelinking and in certain cases also recom-piling. Mobile objects and agents requiremodifications to the underlying program-ming environment. However, they alsohave the least transparency, as describedbelow.

The Migration Complexity columndescribes the amount of effort requiredto design and implement migration. Com-plexity is high for kernel implementations.Exceptions are message-passing kernels,which already provide much of the re-quired functionality in their support formessage passing. This results in a sim-pler migration mechanism. Microkernelsalso support migration more easily be-cause of simpler abstractions and reducedfunctionality (for example, no UNIX com-patibility). However, extensive complexityis introduced for supporting distributedIPC and distributed memory manage-ment. The least complex implementationsare those done at user level and those doneas part of an application.

The “other complexity” subfield de-scribes where the complexity in the systemexists. Early work incurred complexity ininfrastructure support for the underlyinghardware and software, such as Alto com-puters in the case of Worms, and theX-Tree architecture in the case of XOS.Transparent migration on UNIX-like sys-

tems incurs a lot of complexity for thesupport of Single System Image and ex-tending UNIX semantics to a distributedsystem. As already pointed out, messagepassing typically requires a lot of com-plexity; examples include Charlotte andthe V kernel, as well as some of themicrokernels, such as Mach and Chorus.In addition, some of the microkernels (e.g.Mach) also support distributed memorymanagement, which is even harder to sup-port. User-space migrations trade off thesimplicity of the underlying support forredirecting system calls or imposing limitson them. Application-specific migrationsrequire knowledge of the application se-mantics in order to integrate migrationcalls at appropriate places.

Extensibility describes how easy it isto extend a process migration implemen-tation. Examples include support for mul-tiple data transfer and location strategies.In most cases, extensibility is inverselyproportional to complexity. An exceptionto this rule are message-passing kernels,which have simple migration implementa-tions, but are not as extensible. Extensionsto a migration mechanism for performanceand improved fault resilience typically re-quire complex changes to the underlyingmechanism for message passing. Porta-bility describes how easy it is to port themigration mechanism to another operat-ing system or computer. User-space andapplication-specific implementations havesuperior portability. Condor and LSF runon numerous versions of operating sys-tems and computers. Kernel-level imple-mentations are typically closely related tothe underlying system and consequentlytheir portability is limited to the portabil-ity of the operating system. For exampleMach and MOSIX were ported to a num-ber of computer architectures.

It is hard to compare the performanceof various migration mechanisms becausethe implementations were done on anumber of different architectures. It isalso hard and inaccurate to normalizeperformance (some attempts toward nor-malizations were done by Roush [1995]).Therefore, we have not provided a columndescribing performance. Nevertheless,



Tabl

e3.

Sum

mar

yof

the

Diff

eren

tMig

ratio

nIm

plem

enta

tions

Mig

rati

on/

Exa

mpl

esM

ain

OS

v.A

ppl.

Mig

rati

onC

ompl

exit

yE

xten

sibi

lity

Tra

nsp

aren

cyC

har

acte

rist

ics

Ch

arac

teri

stic

sM

odifi

cati

on(O

ther

Com

plex

ity)

&P

orta

bili

ty

Ear

lyW

ork

XO

S,W

orm

,ad

-hoc

solu

tion

s,O

Slo

w(l

ack

ofpo

orli

mit

edD

EM

OS,

Bu

tler

HW

depe

nde

nt

infr

astr

uct

ure

)

Tra

nsp

.Mig

rati

onL

ocu

s,M

OS

IX,

maj

orch

ange

sto

the

OS

hig

hfa

irfu

llin

UN

IX-l

ike

OS

Spr

ite

un

derl

yin

gen

v.(S

upp

orti

ng

SS

I)(O

Sde

pen

d.)

Mes

sage

-Pas

sin

gO

SC

har

lott

e,A

ccen

t,co

mpl

exO

Ssu

ppor

tO

Slo

wfa

irfu

llV

Ker

nel

easy

PM

impl

emen

t.(M

essa

geP

assi

ng)

(OS

depe

nd.

)

Mic

roke

rnel

sA

moe

ba,A

rcad

e,B

irli

X,

no

UN

IXse

man

tics

OS

low

(DM

Mgo

odfu

llC

hor

us,

Mac

h,R

HO

DO

Sco

mpl

exO

Ssu

ppor

tan

dD

IPC

)(O

Sde

pen

d.)

Use

rS

pace

Con

dor,

Alo

nso

&K

yrim

is,

less

tran

spar

ency

appl

icat

ion

low

(for

war

din

gve

rygo

odli

mit

edM

ande

lber

g,L

SF

(rel

inke

d)sy

stem

call

s)(a

ppl.

dep.

)

App

lica

tion

Fre

edm

an,S

kord

os,

min

.tra

nsp

aren

cy,

appl

icat

ion

low

est

(app

mig

rati

onve

rygo

odm

inim

alB

har

at&

Car

dell

im

ore

appl

.kn

owle

dge

(rec

ompi

led)

awar

enes

s)

Mob

ile

obje

cts

Em

eral

d,S

OS,

obje

ctor

ien

ted

prog

ram

min

gm

oder

ate

good

full

CO

OL

envi

ron

men

t(c

omm

un

icat

ion

)

Mob

ile

Age

nts

Age

nt-

TC

L,A

glet

sh

eter

ogen

eity

prog

ram

min

glo

wes

t(s

ecu

rity

good

fair

TA

CO

MA

,Tel

escr

ipt

envi

ron

men

t&

safe

ty)



Table 4. Transparency “Checklist”

Migration/ Open Fork Communication Need to Relink Changes to SharedSupported Files Children Channels Application Kernel Memory

MOSIX yes yes yes no yes noSprite yes yes yes no yes noMach & OSF/1 AD yes yes yes no yes yesLSF some no no yes no no

we note that the performance of user-and application-level migrations typicallyfall in the range of seconds, even min-utes, when migrating processes with largeaddress spaces. The kernel supportedmigrations, especially the newer imple-mentations, fall in the range of tens ofmilliseconds. The most optimized ker-nel implemented migration (Choices) hasinitial costs of only 14ms [Roush andCampbell, 1996], and it is better even ifsome rough normalization is accounted for(see [Roush, 1995]).

As mentioned earlier, the dominant per-formance element is the cost to transferthe address space. Kernel-level optimiza-tions can cut down this cost, whereas user-level implementations do not have accessto the relevant data structures and cannotapply these optimizations.

Recently, trends are emerging thatallow users more access to kernel data,mechanism, and policies [Bombergeret al., 1992]. For example, microkernelsexport most of the kernel state needed foruser-level implementations of migration[Milojicic, 1993c]. Extensible kernels pro-vide even more support in this direction[Bershad et al., 1995; Engler et al., 1995].These trends decrease the relevance ofuser versus kernel implementations.

Transparency describes the extent towhich a migrated process can continueexecution after migration as if migra-tion has not happened. It also determineswhether a migrated process is allowed toinvoke all system functions. Many user- orapplication-level implementations do notallow a process to invoke all system calls.Migration that is implemented inside thekernel typically supports full functional-ity. In general, the higher the level ofthe implementation, the less transparencyis provided. User-space implementations

are aware of migration and they can in-voke migration only at predefined placesin the code. Kernel-supported imple-mentations typically have higher lev-els of transparency. Single system imagesupports transparent migration at anypoint of application code; migration cantransparently be initiated either by themigrating process or by another process.Most mobile agent implementations donot allow transparent migration invoca-tion by other applications; only the mi-grating agent can initiate it. Even thoughless transparent, this approach simplifiesimplementation.

More specifics on transparency in thecase studies are presented in Table 4.Migration for each case study is catego-rized by whether it transparently sup-ports open files, forking children, commu-nication channels, and shared memory. Ifmigration requires changes to the kernelor relinking the application, that is alsolisted.

Support for shared memory of migratedtasks in Mach is unique. In practice, itwas problematic due to a number of de-sign and implementation issues [Blacket al. 1998]. Other systems that supportedboth shared memory and migration ei-ther chose not to provide transparent ac-cess to shared memory after migration(e.g. Locus [Walker and Mathews, 1989;Fleisch and Popek, 1989]), or disallowedmigration of processes using shared mem-ory (e.g., Sprite [Ousterhout et al., 1988]).

Kernel-level migration typically sup-ports all features transparently, whereasuser-level migrations may limit access toNFS files and may not support communi-cation channels or interprocess communi-cation. In addition, a user-level migrationtypically requires relinking applicationswith special libraries. Migration done as



Table 5. Summary of Various Data Transfer Strategies

Data Transfer Example Freeze Freeze Residual Residual InitialStrategy Time Costs Time & Costs Dependency Migration Time

eager (all) most user-level and high high none none highearly migrations

eager (dirty) MOSIX, Locus moderate moderate none none moderate

precopy V kernel very low high none none high

copy on Accent, Mach low small high yes lowreference

flushing Sprite moderate moderate moderate none moderate

part of an application requires additionalre-compilation.

In Table 5, we compare different datatransfer strategies with respect to freezetime, freeze costs, residual time andcosts, residual dependencies, andinitial migration time (time passedsince request for migration until processstarted on remote node).

We can see that different strategieshave different goals and introduce differ-ent costs. At one end of the spectrum,systems that implement an eager (all)strategy in user space eliminate residualdependencies and residual costs, but suf-fer from high freeze time and freeze costs.

Modifying the operating system allowsan eager (dirty) strategy to reduce theamount of the address space that needs tobe copied to the subset of its dirty pages.This increases residual costs and depen-dencies while reducing freeze time andcosts.

Using a precopy strategy further im-proves freeze time, but has higher freezecosts than other strategies. Applicationswith real-time requirements can bene-fit from this. However, it has very highmigration time because it may requireadditional copying of already transferredpages.

Copy on reference requires the most ker-nel changes in order to provide sophis-ticated virtual mappings between nodes.It also has more residual dependenciesthan other strategies, but it has the lowestfreeze time and costs, and migration timeis low, because processes can promptlystart on the remote node.

Finally, the flushing strategy also re-quires some amount of change to thekernel, and has somewhat higher freezetime than copy-on-reference, but improvesresidual time and costs by leaving residualdependencies only on a server, but not onthe source node. Process migration in theChoices system, not listed in the table, rep-resents a highly optimized version of eager(dirty) strategy.

The data transfer strategy dominatesprocess migration characteristics such asperformance, complexity, and fault re-silience. The costs, implementation detailsand residual dependencies of other pro-cess elements (e.g. communication chan-nels, and naming) are also important buthave less impact on process migration.

In the Mach case study, we saw thatmost strategies can be implemented inuser space. However, this requires a pager-like architecture that increases the com-plexity of OS and migration design andimplementation.

Table 6 summarizes load informationdatabase characteristics. Database typeindicates whether the information ismaintained as a distributed or a cen-tralized database. Centralized databaseshave shown surprising scalability for somesystems, in particular LSF. Neverthe-less, achieving the highest level of scal-ability requires distributed informationmanagement.

Maximum nodes deployed is definedas the number of nodes that were ac-tually used. It is hard to make predic-tions about the scalability of migrationand load information management. An



Table 6. Load Information Database

Migration/ Maximum Nodes Database Fault KnowledgeCharact. Database type Deployed Scope Tolerance Relevance

MOSIX distributed 64 partial yes agingSprite centralized 30 global limited verification, update on

state change or periodicMach distributed 5 global no negotiationLSF centralized 500 global yes none

approximate prediction is that centralizedload information management could scaleup to 500 nodes without hierarchical or-ganization, such as in Sprite. With hier-archical organization, such as in LSF, itcould scale beyond 2000 nodes. Decentral-ized information management, such as inMOSIX, can scale to an even larger num-ber of nodes. Even though Mach task mi-gration has not been used on larger sys-tems than a 5-node Ethernet cluster, mostof its components that can impact scalabil-ity (distributed IPC, distributed memorymanagement, and remote address spacecreation) have been demonstrated to scalewell. The Intel Paragon computer, thelargest MMP machine that runs Mach,has over 2000 nodes [Zajcew et al., 1993].However, in order to use migration for loaddistribution some decentralized informa-tion management algorithm would haveto be deployed, similar to the one used forTNC.

Database scope defines the amountof information that is considered. Somesystems, like MOSIX, maintain partialsystem information in order to enhancescalability. Large systems need to addressfault tolerance. One drawback of cen-tralized databases is that storing the dataon one node introduces a single point offailure. This problem can be alleviated byreplicating the data.

Once knowledge about the state of asystem is collected and disseminated, itstarts to lose its relevance as the state ofthe system changes. This is an intrinsiccharacteristic of distributed systems. Thelast column, knowledge relevance, liststhe methods used by the load informationmanagement modules to account for this.

Table 7 describes the type of informa-tion managed by load information col-lection and dissemination. Load informa-

tion is maintained for each process ina system, as well as for each machinein a system. Process parameters liststhe information gathered about individ-ual processes while system parametersshows the information gathered per node.All systems use processor utilization infor-mation while some of them also considersystem communication patterns and ac-cess to off-node devices.

Disseminated parameters describeshow much of the information is passed onto other nodes in a system. In most cases,only system information is disseminated,and the average ready queue length is ofmost interest. In some systems, not allavailable information is retained, as de-scribed in the retained information col-umn. For example, MOSIX retains only asubset of collected information during dis-semination phase. Negotiation parame-ters details the information exchanged atthe time of an attempted migration. Pro-cess parameters are used during negoti-ation phase. Finally, the collection anddissemination columns detail the fre-quency of collection and dissemination infour case studies. In all cases both collec-tion and dissemination are periodic, withthe exception of Sprite—it also dissemi-nates upon a state change.

Table 8 summarizes the characteristicsof distributed scheduling. The migrationclass column indicates the type of migra-tion mechanism employed. The consid-ered costs column indicates whether andhow systems weigh the actual cost of mi-gration. The migration costs are consid-ered in the case of MOSIX and LSF, andnot in the case of Sprite and Mach. In ad-dition to CPU costs, MOSIX also accountsfor communication costs.

Migration trigger summarizesthe reasons for migration activation.



Tabl

e7.

Load

Info

rmat

ion

Col

lect

ion

and

Dis

sem

inat

ion

Col

lect

ion

Dis

sem

inat

ion

Mig

rati

on/

Per

Pro

cess

Sys

tem

Par

amet

ers

Ret

ain

edN

egot

iati

on-p

erio

dic

(fre

q.)

-per

iodi

c(f

req.

)C

har

act.

Par

amet

ers

(als

odi

ssem

inat

ed)

Info

rmat

ion

Par

amet

ers

-eve

nt

driv

.(ev

ent)

-eve

nt

driv

.(ev

ent)

MO

SIX

age,

I/O

patt

ern

s,av

erag

ere

ady

queu

epa

rtia

lm

igra

tin

gpr

oces

spe

riod

icpe

riod

ic(1

-60s

)fi

leac

cess

(ran

dom

subs

et)

may

bere

fuse

d(w

orm

-lik

e)

Spr

ite

non

eti

me

sin

cela

stlo

cal

alli

nfo

reta

ined

mig

rati

onpe

riod

ic(5

s)pe

riod

ic(1

min

)an

du

ser

inpu

t,re

ady

vers

ion

upo

na

stat

ech

ange

queu

ele

ngt

h

Mac

hag

e,re

mot

eIP

C,

aver

age

read

yqu

eue,

alli

nfo

reta

ined

dest

inat

ion

load

,pe

riod

ic(1

s)pe

riod

ic(1

s)an

dre

mot

ere

mot

eIP

C,r

emot

efr

eepa

gin

gsp

ace

pagi

ng

pagi

ng

LS

Fn

one

arbi

trar

yal

lin

fore

tain

edsy

stem

para

met

ers

peri

odic

peri

odic

con

figu

rabl

eof

alln

odes

Tabl

e8.

Dis

trib

uted

Sch

edul

ing

Sys

tem

/M

igra

tion

Con

side

red

Mig

rati

onA

Pri

ori

Lea

rnin

gfr

omS

tabi

lity

Ch

arac

teri

stic

sC

lass

Cos

tsT

rigg

erK

now

ledg

eth

eP

ast

MO

SIX

proc

ess

mig

rati

onC

PU

&th

resh

old

cros

s+

non

elig

ible

agin

glo

adve

ctor

min

imu

mre

side

ncy

(UN

IX-l

ike

OS

)co

mm

un

icat

ion

load

diff

eren

cepr

oces

ses

proc

ess

resi

den

cyn

ode

refu

sal

info

wei

ghti

ng

Spr

ite

proc

ess

mig

rati

onn

opm

ake,

mig

rato

rysh

ell,

non

elig

ible

proc

esse

sn

one

bias

tow

ard

(UN

IX-l

ike

OS

)ev

icti

on(d

ue

tou

ser

orli

stof

elig

ible

lon

g-id

leac

tivi

tyor

fair

nes

s)on

esm

ach

ines

Mac

hta

skm

igra

tion

no

thre

shol

dcr

oss

pred

efin

edn

on-

lim

itco

nse

cuti

veh

igh

thre

shol

d(m

icro

kern

el)

elig

ible

task

sm

igra

tion

LS

Fpr

oces

sm

igra

tion

CP

Uov

erh

ead

con

figu

rabl

eth

resh

olds

pred

efin

edn

on-

low

erin

gst

anda

rdh

igh

thre

shol

ds(u

ser-

leve

lmig

r.)

elig

ible

com

man

dsde

viat

ion



Examples include crossing a load thresh-old on a single node or on demand afteran application-specific request, or only forspecific events like process eviction. Spriteprocess migration can be initiated as apart of the pmake program or a migratoryshell, or as a consequence of the evictionof a remotely executed process.

Some of the systems use a prioriknowledge, typically in the form of spec-ifying which processes are not allowedto migrate. These are for example wellknown system processes, such as in thecase of MOSIX, Sprite and Mach, or com-mands in the case of LSF. The learningfrom the past column indicates how somesystems adapt to changing loads. Exam-ples include aging load vectors and pro-cess residency in MOSIX, and limitingconsecutive migrations in Mach. Stabil-ity is achieved by requiring a minimumresidency for migrated processes after amigration (such as in MOSIX), by intro-ducing a high threshold per node (such asin Mach and LSF), or by favoring long-idle machines (such as in Sprite). It canalso be achieved by manipulating load in-formation as was investigated in MOSIX.For example, dissemination policies can bechanged, information can be weighed sub-ject to current load, and processes can berefused.

7. WHY PROCESS MIGRATIONHAS NOT CAUGHT ON

In this section, we attempt to identifythe barriers that have prevented a wideradoption of process migration and to ex-plain how it may be possible to overcomethem. We start with an analysis of eachcase study; we identify misconceptions;we identify those barriers that we con-sider the true impediments to the adop-tion of migration; and we conclude by out-lining the likelihood of overcoming thesebarriers.

7.1. Case Analysis

MOSIX. The MOSIX distributed operat-ing system is an exception to most othersystems supporting transparent processmigration in that it is still in general

use. Several things worked against thewider adoption of the MOSIX system: theimplementation was done on a commer-cial operating system which preventedwide-spread distribution of the code. Onecommercial backer of MOSIX withdrewfrom the operating system business.

The current outlook is much brighter.The latest versions of MOSIX support pro-cess migration on BSDI’s version of UNIXand Linux. The Linux port eliminates thelegal barriers that prevented the distribu-tion of early versions of the system.

Sprite. Sprite as a whole did notachieve a long-lasting success, so its pro-cess migration facility suffered with it.Sprite’s failure to expand significantly be-yond U.C. Berkeley was due to a consciousdecision among its designers not to in-vest the enormous effort that would havebeen required to support a large externalcommunity. Instead, individual ideas fromSprite, particularly in the areas of file sys-tems and virtual memory, have found theirway into commercial systems over time.

The failure of Sprite’s process migrationfacility to similarly influence the commer-cial marketplace has come as a surprise.Ten years ago we would have predictedthat process migration in UNIX would becommonplace today, despite the difficul-ties in supporting it. Instead, user-levelload distribution is commonplace, but it iscommonly limited to applications that canrun on different hosts without ill effects,and relies either on explicit checkpointingor the ability to run to completion.

Mach and OSF/1. Compared to othersystems, Mach has gone the furthestin technology transfer. Digital UNIXhas been directly derived from OSF/1,NT internals resemble the Mach design,and a lot of research was impacted byMach. However, almost no distributedsupport was transferred elsewhere. Thedistributed memory management and dis-tributed IPC were extremely complex, re-quiring significant effort to develop and tomaintain. The redesign of its distributedIPC was accomplished within the OSFRI [Milojicic et al., 1997], but distributedmemory management has never beenredesigned and was instead abandoned



[Black et al. 1998]. Consequently, task andprocess migration have never been trans-ferred elsewhere except to Universitiesand Labs.

LSF. Platform Computing has not ag-gressively addressed process migrationbecause the broad market is still notready—partially due to an immature dis-tributed system structure, and partiallydue to a lack of cooperation from OSand application vendors. But most impor-tantly there was no significant customerdemand.

Since a vast majority of users run Unixand Windows NT, for which dynamic pro-cess migration is not supported by the OSkernel, Platform Computing has been us-ing user-level job checkpointing and mi-gration as an indirect way to achieveprocess migration for the users of LSF.A checkpoint library based on that ofCondor is provided that can be linkedwith Unix application programs to en-able transparent process migration. Thishas been integrated into a number of im-portant commercial applications. For ex-ample, a leading circuit simulation toolfrom Cadence, called Verilog, can be check-pointed on one workstation and resumedon another.

It is often advantageous to have check-pointing built into the applications andhave LSF manage the migration process.The checkpoint file is usually smallercompared to user-level, because only cer-tain data structures need to be saved,rather than all dirty memory pages. Withmore wide-spread use of workstations andservers on the network, Platform Comput-ing is experiencing a rapidly increasingdemand for process migration.

7.2. Misconceptions

Frequently, process migration has beendismissed as an academic exercise withlittle chance for wide deployment [Eageret al., 1988; Kremien and Kramer, 1992;Shivaratri et al., 1992]. Many rationaleshave been presented for this position,such as:r significant complexity,r unacceptable costs,

r the lack of support for transparency, andr the lack of support for heterogeneity.

Some implementations, even success-ful ones, indeed have reinforced such be-liefs. Despite the absence of wide spreaddeployment, work on process migrationhas persisted. In fact, recently we haveseen more and more attempts to providemigration and other forms of mobility[Steensgaard and Jul, 1995; Roush andCampbell, 1996; Smith and Hutchinson,1998]. Checkpoint/restart systems are be-ing deployed for the support of long-running processes [Platform Computing,1996]. Finally, mobile agents are being in-vestigated on the Web.

If we analyze implementations, we seethat technical solutions exist for eachof these problems (complexity, cost, non-transparency and homogeneity). Migra-tion has been supported with various de-grees of complexity: as part of kernelmechanisms; as user-level mechanisms;and even as a part of an application (seeSections 4.2–4.6). The time needed to mi-grate has been reduced from the rangeof seconds or minutes [Mandelberg andSunderam, 1988; Litzkow and Solomon,1992] to as low as 14ms [Roush andCampbell, 1996]. Various techniques havebeen introduced to optimize state transfer[Theimer et al., 1985; Zayas, 1987a; Roushand Campbell, 1996] (see Section 3.2).Transparency has been achieved to dif-ferent degrees, from limited to complete(see Section 3.3). Finally, recent workdemonstrates improvements in support-ing heterogeneous systems, as done inEmerald [Steensgaard and Jul, 1995],Tui [Smith and Hutchinson, 1998] andLegion [Grimshaw and Wulf, 1996] (seeSection 3.6).

7.3. True Barriers to Migration Adoption

We believe that the true impediments todeploying migration include the following:r A lack of applications. Scientific

applications and academic loads (e.g.pmake and simulations) represent asmall percentage of today’s applications.The largest percentage of applications



today represent standard PC applica-tions, such as word-processing, and desk-top publishing. Such applications do notsignificantly benefit from migration.r A lack of infrastructure. There hasnot been a widely-used distributed op-erating system. Few of the distributedfeatures of academically successful re-search operating systems, such as Mach,Sprite, or the V kernel, have been trans-ferred to the marketplace despite ini-tial enthusiasm. This lack increasesthe effort needed to implement processmigration.r Migration is not a requirement forusers. Viable alternatives, such as re-mote invocation and remote data access,might not perform as uniformly as pro-cess migration but they are able to meetuser expectations with a simpler andwell understood approach [Eager et al.,1986a, Kremien and Kramer, 1992].r Sociological factors have been im-portant in limiting the deployment ofprocess migration. In the workstationmodel, each node belongs to a user. Usersare not inclined to allow remote pro-cesses to visit their machines. A lotof research has addressed this prob-lem, such as process eviction in Sprite[Douglis and Ousterhout, 1991], or low-ering the priority of foreign processesin the Stealth scheduler [Krueger andChawla, 1991]. However, the psychologi-cal effects of workstation ownership stillplay a role today.

7.4. How these Barriers Might be Overcome

It often takes a long time for good re-search ideas to become widely adoptedin the commercial arena. Examples in-clude object-orientation, multi-threading,and the Internet. It may be the case thatprocess mobility is not ripe enough to beadopted by the commercial market.

We address each of the barriers iden-tified in previous section and try to pre-dict how migration might fit the futureneeds. The rest of the section is highlyspeculative because of the attempts to ex-trapolate market needs and technology.

Applications. To become popular inthe marketplace, migration needs a “killerapplication” that will provide a compellingreason for commercial operating systemvendors to devote the resources neededto implement and support process migra-tion. The types of application that arewell-suited for process migration includeprocessor-intensive tasks such as paral-lel compilation and simulation, and I/O-intensive tasks that would benefit fromthe movement of a process closer to somedata or another process (see also Section2.4). These applications are exceedinglyrare by comparison to the typical uses oftoday’s computers in the home and work-place, such as word processing, spread-sheets, and games. However, applicationsare becoming more distributed, modular,and dependent on external data. In thenear future, because of the exceeding dif-ference in network performance, it will bemore and more relevant to execute (mi-grate) applications close to the source ofdata. Modularity will make paralleliza-tion easier (e.g. various component mod-els, such as Java Beans and MicrosoftDCOM).

Infrastructure. The NT operatingsystem is becoming a de facto standard,leading to a common environment. UNIXis also consolidating into fewer ver-sions. All these systems start to addressthe needs for clustering, and large-scalemulticomputers. Both environments aresuitable for process migration. Theseoperating systems are becoming more andmore distributed. A lot of missing infras-tructure is becoming part of the standardcommercial operating systems or itsprogramming environments.

Convenience vs. requirement (im-pact of hardware technology). The fol-lowing hardware technology trends mayimpact process migration in the future:high speed networks, large scale systems,and the popularity of hardware mobilegadgets. With the increasing differencein network speeds (e.g. between a mobilecomputer and a fiber-channel), the differ-ence between remote execution and migra-tion becomes greater. Being able to moveprocesses during execution (e.g. because it



was realized that there is a lot of remotecommunication) can improve performancesignificantly. Secondly, with the largerscale of systems, the failures are more fre-quent, thereby increasing the relevanceof being able to continue program execu-tion at another node. For long-running orcritical applications (those that should notstop executing) migration becomes a moreattractive solution. Finally, the increas-ing popularity of hardware mobile gadgetswill require mobility support in software.Examples include migrating applicationsfrom a desktop, to a laptop, and eventuallyto a gadget (e.g. future versions of cellularphones or palmtops).

Sociology. There are a few factors re-lated to sociology. The meaning and rel-evance of someone’s own workstation isblurring. There are so many computers inuse today that the issue of computing cy-cles becomes less relevant. Many comput-ers are simply servers that do not belongto any single user, and at the same timethe processing power is becoming increas-ingly cheap. A second aspect is that as theworld becomes more and more connected,the idea of someone else’s code arriving onone’s workstation is not unfamiliar any-more. Many security issues remain, butthey are being actively addressed by themobile code and agents community.

In summary, we do not believe thatthere is a need for any revolutionary de-velopment in process migration to make itwidely used. We believe that it is a matterof time, technology development, and thechanging needs of users that will trigger awider use of process migration.

8. SUMMARY AND FURTHER RESEARCH

In this paper we have surveyed processmigration mechanisms. We have classi-fied and presented an overview of a num-ber of systems, and then discussed fourcase studies in more detail. Based on thismaterial, we have summarized variousmigration characteristics. Throughout thetext we tried to assess some misconcep-tions about process migration, as well asto discover the true reasons for the lack ofits wide acceptance.

We believe there is a future for processmigration. Different streams of develop-ment may well lead to a wider deploy-ment of process migration. Below we in-clude some possible paths.

One path is in the direction of LSF,a user-level facility that provides muchof the functionality of full-fledged pro-cess migration systems, but with fewerheadaches and complications. The check-point/restart model of process migrationhas already been relatively widely de-ployed. Packages such as Condor, LSFand Loadleveler are used for scientific andbatch applications in production environ-ments. Those environments have high de-mands on their computer resources andcan take advantage of load sharing in asimple manner.

A second path concerns clusters of work-stations. Recent advances in high speednetworking (e.g. ATM [Partridge, 1994]and Myrinet [Boden et al., 1995]) have re-duced the cost of migrating processes, al-lowing even costly migration implementa-tions to be deployed.

A third path, one closer to the con-sumers of the vast majority of today’scomputers (Windows systems on Intel-based platforms), would put process mi-gration right in the home or office. Sun re-cently announced their Jini architecturefor home electronics [Sun Microsystems,1998] and other similar systems are sureto follow. One can imagine a process start-ing on a personal computer, and migrat-ing its flow of control into another devicein the same domain. Such activity wouldbe similar to the migratory agents ap-proach currently being developed for theWeb [Rothermel and Hohl, 1998].

Still another possible argument for pro-cess migration, or another Worm-like fa-cility for using vast processing capabilityacross a wide range of machines, wouldbe any sort of national or internationalcomputational effort. Several years ago,Quisquater and Desmedt [1991] suggestedthat the Chinese government could solvecomplex problems (such as factoring largenumbers) by permitting people to use theprocessing power in their television sets,and offering a prize for a correct answer



as an incentive to encourage televisionowners to participate. In case of ex-traordinary need, process migration couldprovide the underlying mechanism forlarge-scale computation across an ever-changing set of computers.

Finally, the most promising new op-portunity is the use of mobile agents inthe Web. In this setting, both technicaland sociological conditions differ from thetypical distributed system where processmigration has been de-ployed (see theanalysis in Section 7.2). Instead of theprocessor pool and workstation models,the Web environment connects computersas interfaces to the “network-is-computer”model. The requirements for transparencyare relaxed, and user-specific solutions arepreferred. Performance is dominated bynetwork latency and therefore state trans-fer is not as dominant as it is on a lo-cal area network; remote access and re-mote invocation are not competitive withsolutions based on mobility. Users areready to allow foreign code to be down-loaded to their computer if this code isexecuted within a safe environment. Inaddition, there are plenty of dedicatedservers where foreign code can execute.Heterogeneity is supported at the lan-guage level. Generally speaking, the use ofmobile agents in a Web environment over-comes each of the real impediments to de-ploying process migration, and will be agrowing application of the technology (al-beit with new problems, such as security,that are currently being addressed by themobile agents community). Mobile agentsbear a lot of similarity and deploy similartechniques as process migration.

Process migration will continue to at-tract research independently of its successin market deployment. It is deemed aninteresting, hard, and unsolved problem,and as such is ideal for research. How-ever, reducing the amount of transparencyand the OS-level emphasis is common foreach scenario we outlined above. Eventu-ally this may result in a less transpar-ent OS support for migration, reflectingthe lack of transparency to the applicationlevel while still providing certain guaran-tees about connectivity.

ACKNOWLEDGMENTS

We would like to thank Amnon Barak, David Black,Don Bolinger, Bill Bryant, Luca Cardelli, SteveChapin, Alan Downey, Shai Guday, Mor Harchol-Balter, Dag Johansen, and Dan Teodosiu for pro-viding many useful suggestions that significantlyimproved the paper. The anonymous reviewers pro-vided an extensive list of general, as well as very de-tailed, suggestions that have strengthened our focus,presentation and correctness of the paper. We are in-debted to them and to the ACM Computing Surveyseditor, Fred Schneider.

REFERENCES

ACCETTA, M., BARON, R., BOLOSKY, W., GOLUB, D.,RASHID, R., TEVANIAN, A., AND YOUNG, M. 1986.Mach: A New Kernel Foundation for UNIX De-velopment. Proceedings of the Summer USENIXConference, 93–112.

AGRAWAL, R. AND EZZAT, A. 1987. Location Indepen-dent Remote Execution in NEST. IEEE Trans-actions on Software Engineering 13, 8, 905–912.

ALON, N., BARAK, A., AND MANBER, U. 1987. On Dis-seminating Information Reliably without Broad-casting. Proceedings of the 7th InternationalConference on Distributed Computing Systems,74–81.

ALONSO, R. AND KYRIMIS, K. 1988. A Process Migra-tion Implementation for a UNIX System. Pro-ceedings of the USENIX Winter Conference, 365–372.

AMARAL, P., JACQEMOT, C., JENSEN, P., LEA, R., AND

MIROWSKI, A. 1992. Transparent Object Migra-tion in COOL-2. Proceedings of the ECOOP.

ANDERSEN, B. 1992. Load Balancing in the Fine-Grained Object-Oriented Language Ellie. Pro-ceedings of the Workshop on Dynamic ObjectPlacement and Load Balancing in Parallel andDistributed Systems Programs, 97–102.

ANDERSON, T. E., CULLER, D. E., AND PATTERSON, D. A.1995. A Case for NOW (Networks of Worksta-tions). IEEE Micro 15, 1, 54–64.

ARTSY, Y., CHANG, Y., AND FINKEL, R. 1987. Interpro-cess Communication in Charlotte. IEEE Soft-ware, 22–28.

ARTSY, Y. AND FINKEL, R. 1989. Designing a ProcessMigration Facility: The Charlotte Experience.IEEE Computer, 47–56.

BAALBERGEN, E. H. 1988. Design and Implementa-tion of Parallel Make. Computing Systems 1,135–158.

BANAWAN, S. A. AND ZAHORJAN, J. 1989. Load Sharingin Hierarchical Distributed Systems. Proceed-ings of the 1989 Winter Simulation Conference,963–970.

BARAK, A. AND LITMAN, A. 1985. MOS: a Multicom-puter Distributed Operating System. Software-Practice and Experience 15, 8, 725–737.



BARAK, A. AND SHILOH, A. 1985. A Distributed Load-Balancing Policy for a Multicomputer. Software-Practice and Experience 15, 9, 901–913.

BARAK, A. AND WHEELER, R. 1989. MOSIX: AnInte-grated Multiprocessor UNIX. Proceedingsof the Winter 1989 USENIX Conference, 101–112.

BARAK, A., SHILOH, A., AND WHEELER, R. 1989.Flood Prevention in the MOSIX Load-BalancingScheme. IEEE Technical Committee on Operat-ing Systems Newsletter 3, 1, 24–27.

BARAK, A., GUDAY, S., AND WHEELER, R. G. 1993. TheMOSIX Distributed Operating System. SpringerVerlag.

BARAK, A., LADEN, O., AND BRAVERMAN, A. 1995. TheNOW MOSIX and its Preemptive Process Mi-gration Scheme. Bulletin of the IEEE TechnicalCommittee on Operating Systems and Applica-tion Environments 7, 2, 5–11.

BARBOU DES PLACES, F. B., STEPHEN, N., AND REYNOLDS,F. D. 1996. Linux on the OSF Mach3 Micro-kernel. Proceedings of the First Conference onFreely Redistributable Software, 33–46.

BARRERA, J. 1991. A Fast Mach Network IPC Imple-mentation. Proceedings of the Second USENIXMach Symposium, 1–12.

BASKETT, F., HOWARD, J., AND MONTAGUE, T. 1977. TaskCommunication in DEMOS. Proceedings of the6th Symposium on OS Principles, 23–31.

BAUMANN, J., HOHL, F., ROTHERMEL, K., AND STRABER,M. 1998. Mole—Concepts of a Mobile AgentSystem. World Wide Web 1, 3, 123–137.

BEGUELIN, A., DONGARRA, J., GEIST, A., MANCHEK, R.,OTTO, S., AND WALPOLE, J. 1993. PVM: Experi-ences, Current Status and Future Directions.Proceedings of Supercomputing 1993, 765–766.

BERNSTEIN, P. A. 1996. Middleware: A Model for Dis-tributed System Services. Communications ofthe ACM 39, 2, 86–98.

BERSHAD, B., SAVAGE, S., PARDYAK, P., SIRER, E. G.,FIUCZINSKI, M., BECKER, D., CHAMBERS, C., AND

EGGERS, S. 1995. Extensibility, Safety and Per-formance in the SPIN Operating System. Pro-ceedings of the 15th Symposium on OperatingSystems Principles, 267–284.

BHARAT, K. A. AND CARDELLI, L. 1995. Migratory Ap-plications. Proceedings of the Eight Annual ACMSymposium on User Interface Software Technol-ogy.

BLACK, A., HUTCHINSON, N., JUL, E., LEVY, H.,AND CARTER, L. 1987. Distributed and AbstractTypes in Emerald. IEEE Transactions on Soft-ware Engineering, SE-13, 1, 65–76.

BLACK, D., GOLUB, D., JULIN, D., RASHID, R., DRAVES,R., DEAN, R., FORIN, A., BARRERA, J., TOKUDA, H.,MALAN, G., AND BOHMAN, D. 1992. MicrokernelOperating System Architecture and Mach. Pro-ceedings of the USENIX Workshop on Micro-Kernels and Other Kernel Architectures, 11–30.

BLACK, D., MILOJICIC, D., DEAN, R., DOMINIJANNI,M., LANGERMAN, A., SEARS, S. 1998. Extended

Memory Management (XMM): Lessons Learned.Software-Practice and Experience 28, 9, 1011–1031.

BODEN, N., COHEN, D., FELDERMAN, R. E., KULAWIK, A.E., SEITZ, C. L., SEIZOVIC, J. N., AND SU, W.-K. 1995.Myrinet: A Gigabit-per-Second Local Area Net-work. IEEE Micro 15, 1, 29–38.

BOKHARI, S. H. 1979. Dual Processor Schedulingwith Dynamic Reassignment. IEEE Transac-tions on Software Engineering, SE-5, 4, 326–334.

BOMBERGER, A. C., FRANTZ, W. S., HARDY, A. C.,HARDY, N., LANDAU, C. R., AND SHAPIRO, J. S.1992. The Key-KOS (R) Nanokernel Architec-ture. USENIX Workshop on Micro-Kernels andOther Kernel Architectures, 95–112.

BOND, A. M. 1993. Adaptive Task Allocation ina Distributed Workstation Environment. Ph.D.Thesis, Victoria University at Wellington.

BONOMI, F. AND KUMAR, A. 1988. Adaptive OptimalLoad Balancing in a Heterogeneous MultiserverSystem with a Central Job Scheduler. Proceed-ings of the 8th International Conference on Dis-tributed Computing Systems, 500–508.

BORGHOFF U. M. 1991. Catalogue of DistributedFile/Operating Systems. Springer Verlag.

BOWEN, N. S., NIKOLAOU, C. N., AND GHAFOOR, A.1988. Hierarchical Workload Allocation forDistributed Systems. Proceedings of the 1988 In-ternational Conference on Parallel Processing,II:102–109.

BROOKS, C., MAZER, M. S., MEEKS, S., AND MILLER,J. 1995. Application-Specific Proxy Servers asHTTP Stream Transducers. Proceedings of theFourth International World Wide Web Confer-ence, 539–548.

BRYANT, B. 1995. Design of AD 2, a DistributedUNIX Operating System. OSF Research Insti-tute.

BRYANT, R. M. AND FINKEL, R. A. 1981. A Stable Dis-tributed Scheduling Algorithm. Proceedings ofthe 2nd International Conference on DistributedComputing Systems, 314–323.

BUGNION, E., DEVINE, S., GOVIL, K., AND ROSENBLUM, M.1997. Disco: running commodity operating sys-tems on scalable multiprocessors. ACM Transac-tions on Computer Systems 15, 4, 412–447.

BUTTERFIELD, D. A. AND POPEK, G. J. 1984. NetworkTasking in the Locus Distributed UNIX System.Proceedings of the Summer USENIX Conference,62–71.

CABRERA, L. 1986. The Influence of Workload onLoad Balancing Strategies. Proceedings of theWinter USENIX Conference, 446–458.

CARDELLI, L. 1995. A Language with DistributedScope. Proceedings of the 22nd Annual ACMSymposium on the Principles of ProgrammingLanguages, 286–297.

CASAS, J., CLARK, D. L., CONURU, R., OTTO, S. W., PROUTY,R. M., AND WALPOLE, J. 1995. MPVM: A Migra-tion Transparent Version of PVM. ComputingSystems 8, 2, 171–216.



CASAVANT, T. L. AND KUHL, J. 1988a. A Taxonomyof Scheduling in General-Purpose DistributedComputing Systems. IEEE Transactions on Soft-ware Engineering, SE-14(2), 141–152.

CASAVANT, T. L. AND KUHL, J. 1988b. Effects of Re-sponse and Stability on Scheduling in Dis-tributed Computing systems. IEEE Transac-tions on Software Engineering, SE-14(11), 1578–1588.

CHAPIN, J., ROSENBLUM, M., DEVINE, S., LAHIRI, T., TEO-DOSIU, D., AND GUPTA, A. 1995. Hive: Fault Con-tainment for Shared-Memory Multiprocessors.Proceedings of the Fifteenth ACM Symposium onOperating Systems Principles, 12–25.

CHAPIN, S. J. 1995. Distributed Scheduling Supportin the Presence of Autonomy. Proceedings of the4th Heterogeneous Computing Workshop, IPPS,22–29.

CHAPIN, S. J. 1993. Scheduling Support Mecha-nisms for Autonomous, Heterogeneous, Dis-tributed Systems. Ph.D. Thesis, Technical ReportCSD-TR-93-087, Purdue University.

CHAPIN, S. J. AND SPAFFORD, E. H. 1994. Supportfor Implementing Scheduling Algorithms UsingMESSIAHS. Scientific Programming, 3, 325–340.

CHAPIN, S. J. 1996. Distributed and MultiprocessorScheduling. ACM Computing Surveys 28, 1, 233–235.

CHASE, J. S., AMADOR, F. G., LAZOWSKA, E. D., LEVY,H. M., AND LITTLEFIELD, R. J. 1989. The Am-ber System: Parallel Programming on a Net-work of Multiprocessors. Proceedings of the 12thACM Symposium on Operating Systems Princi-ples, 147–158.

CHERITON, D. R. 1988. The V Distributed System.Communications of the ACM 31, 3, 314–333.

CHERITON, D. 1990. Binary Emulation of UNIX Us-ing the V Kernel. Proceedings of the SummerUSENIX Conference, 73–86.

CHESS, D., B., G., HARRISON, C., LEVINE, D., PARRIS,C., AND TSUDIK, G. 1995. Itinerant Agents forMobile Computing. IEEE Personal Communica-tions Magazine.

CHOU, T. C. K. AND ABRAHAM, J. A. 1982. Load Balanc-ing in Distributed Systems. IEEE Transactionson Software Engineering, SE-8, 4, 401–419.

CHOU, T. C. K. AND ABRAHAM, J. A. 1983. Load Redis-tribution under Failure in Distributed Systems.IEEE Transactions on Computers, C-32, 9, 799–808.

CHOW, Y.-C. AND KOHLER, W. H. 1979. Models for Dy-namic Load Balancing in a Heterogeneous Mul-tiple Processor System. IEEE Transactions onComputers, C-28, 5, 354–361.

COHN, D. L., DELANEY, W. P., AND TRACEY, K. M.1989. Arcade: A Platform for Distributed Op-erating Systems. Proceedings of the USENIXWorkshop on Experiences with Distributedand Multiprocessor Systems (WEBDMS), 373–390.

CONCEPCION, A. I. AND ELEAZAR, W. M. 1988. ATestbed for Comparative Studies of Adap-tive Load Balancing Algorithms. Proceedingsof the Distributed Simulation Conference, 131–135.

DANNENBERG, R. B. 1982. Resource Sharing in a Net-work of Personal Computers. Ph.D. Thesis, Tech-nical Report CMU-CS-82-152, Carnegie MellonUniversity.

DANNENBERG, R. B. AND HIBBARD, P. G. 1985. A But-ler Process for Resource Sharing on a Spice Ma-chine. IEEE Transactions on Office InformationSystems 3, 3, 234–252.

DEARLE A. DI BONA R., FARROW J., HENSKENS F.,LINDSTROM A., ROSENBERG J. AND VAUGHAN F. 1994.Grasshopper: An Orthogonally Persistent Oper-ating System. Computer Systems 7, 3, 289–312.

DEDIU, H., CHANG, C. H., AND AZZAM, H. 1992. Heavy-weight Process Migration. Proceedings of theThird Workshop on Future Trends of DistributedComputing Systems, 221–225.

DENNING, P. J. 1980. Working Sets Past and Present.IEEE Transactions on Software Engineering,SE-6, 1, 64–84.

DIKSHIT, P., TRIPATHI, S. K., AND JALOTE, P. 1989. SA-HAYOG: A Test Bed for Evaluating DynamicLoad-Sharing Policies. Software-Practice andExperience, 19, 411–435.

DOUGLIS, F. AND OUSTERHOUT, J. 1987. Process Migra-tion in the Sprite Operating System. Proceedingsof the Seventh International Conference on Dis-tributed Computing Systems, 18–25.

DOUGLIS, F. 1989. Experience with Process Migra-tion in Sprite. Proceedings of the USENIX Work-shop on Experiences with Distributed and Mul-tiprocessor Systems (WEBDMS), 59–72.

DOUGLIS, F. 1990. Transparent Process Migration inthe Sprite Operating System. Ph.D. Thesis, Tech-nical Report UCB/CSD 90/598, CSD (EECS),University of California, Berkeley.

DOUGLIS, F. AND OUSTERHOUT, J. 1991. TransparentProcess Migration: Design Alternatives and theSprite Implementation. Software-Practice andExperience 21, 8, 757–785.

DUBACH, B. 1989. Process-Originated Migration in aHeterogeneous Environment. Proceedings of the17th ACM Annual Computer Science Conference,98–102.

EAGER, D., LAZOWSKA, E., AND ZAHORJAN, J. 1986a. AComparison of Receiver-Initiated and Sender-Initiated Adaptive Load Sharing. PerformanceEvaluation 6, 1, 53–68.

EAGER, D., LAZOWSKA, E., AND ZAHORJAN, J. 1986b.Dynamic Load Sharing in Homogeneous Dis-tributed Systems. IEEE Transactions on Soft-ware Engineering 12, 5, 662–675.

EAGER, D., LAZOWSKA, E., AND ZAHORJAN, J. 1988.The Limited Performance Benefits of Migrat-ing Active Processes for Load Sharing. Proceed-ings of the 1988 ACM SIGMETRICS Confer-ence on Measurement and Modeling of Computer



Systems, Performance Evaluation Review 16, 1,63–72.

EFE, K. 1982. Heuristic Models of Task AssignmentScheduling in Distributed Systems. IEEE Com-puter, 15, 6, 50–56.

EFE, K. AND GROSELJ, B. 1989. Minimizing ControlOverheads in Adaptive Load Sharing. Proceed-ings of the 9th International Conference on Dis-tributed Computing Systems, 307–315.

ENGLER, D. R., KAASHOEK, M. F., AND O’TOOLE, J. J.1995. Exokernel: An Operating System Archi-tecture for Application-Level Resource Manage-ment. Proceedings of the 15th Symposium on Op-erating Systems Principles, 267–284.

ESKICIOGLU, M. R. 1990. Design Issues of ProcessMigration Facilities in Distributed Systems.IEEE Technical Committee on Operating Sys-tems Newsletter 4, 2, 3–13.

EZZAT, A., BERGERON, D., AND POKOSKI, J. 1986. TaskAllocation Heuristics for Distributed Comput-ing Systems. Proceedings of the 6th InternationalConference on Distributed Computing Systems.

FARMER, W. M., GUTTMAN, J. D., AND SWARUP, V. 1996.Security for Mobile Agents: Issues and Require-ments. Proceedings of the National InformationSystems Security Conference, 591–597.

FEITELSON, D. G. AND RUDOLPH, L. 1990. Mappingand Scheduling in a Shared Parallel Environ-ment Using Distributed Hierarchical Control.Proceedings of the 1990 International Conferenceon Parallel Processing, I: 1–8.

FERRARI, D. AND ZHOU., S. 1986. A Load Index for Dy-namic Load Balancing. Proceedings of the 1986Fall Joint Computer Conference, 684–690.

FINKEL, R., SCOTT, M., ARTSY, Y., AND CHANG, H. 1989.Experience with Charlotte: Simplicity and Func-tion in a Distributed Operating system. IEEETransactions on Software Engineering, SE-15, 6,676–685.

FLEISCH, B. D. AND POPEK, G. J. 1989. Mirage: A Co-herent Distributed Shared Memory Design. Pro-ceedings of the 12th ACM Symposium on Oper-ating System Principles, 211–223.

FREEDMAN, D. 1991. Experience Building a ProcessMigration Subsystem for UNIX. Proceedings ofthe Winter USENIX Conference, 349–355.

GAIT, J. 1990. Scheduling and Process Migration inPartitioned Multiprocessors. Journal of Paralleland Distributed Computing 8, 3, 274–279.

GAO, C., LIU, J. W. S., AND RAILEY, M. 1984. Load Bal-ancing Algorithms in Homogeneous DistributedSystems. Proceedings of the 1984 InternationalConference on Parallel Processing, 302–306.

GERRITY, G. W., GOSCINSKI, A., INDULSKA, J., TOOMEY, W.,AND ZHU, W. 1991. Can We Study Design Issuesof Distributed Operating Systems in a Gener-alized Way? Proceedings of the Second USENIXSymposium on Experiences with Distributed andMultiprocessor Systems, 301–320.

GOLDBERG, A. AND JEFFERSON, D. 1987. TransparentProcess Cloning: A Tool for Load Management

of Distributed Systems. Proceedings of the 8thInternational Conference on Parallel Processing,728–734.

GOLUB, D., DEAN, R., FORIN, A., AND RASHID, R. 1990.UNIX as an Application Program. Proceedingsof the Summer USENIX Conference, 87–95.

GOPINATH, P. AND GUPTA, R. 1991. A Hybrid Ap-proach to Load Balancing in Distributed Sys-tems. Proceedings of the USENIX Symposiumon Experiences with Distributed and Multipro-cessor Systems, 133–148.

GOSCINSKI, A. 1991. Distributed Operating Systems:The Logical Design. Addison Wesley.

GOSLING, J., JOY, B., AND STEELE, G. 1996. The JavaLanguage Specification. Addison Wesley.

GRAY, R. 1995. Agent Tcl: A flexible and securemobileagent system. Ph.D. thesis, Technical Re-port TR98-327, Department of Computer Sci-ence, Dartmouth College, June 1997.

GRIMSHAW, A. AND WULF, W. 1997. The Legion Visionof a Worldwide Virtual Computer. Communica-tions of the ACM 40, 1, 39–45.

GUPTA, R. AND GOPINATH, P. 1990. A Hierarchical Ap-proach to Load Balancing in Distributed Sys-tems. Proceedings of the Fifth Distributed Mem-ory Computing Conference, II, 1000–1005.

HAC, A. 1989a. A Distributed Algorithm for Perfor-mance Improvement Through File Replication,File Migration, and Process Migration. IEEETransactions on Software Engineering 15, 11,1459–1470.

HAC, A. 1989b. Load Balancing in Distributed Sys-tems: A Summary. Performance Evaluation Re-view, 16, 17–25.

HAERTIG, H., KOWALSKI, O. C., AND KUEHNHAUSER, W. E.1993. The BirliX Security Architecture.

HAGMANN, R. 1986. Process Server: Sharing Pro-cessing Power in a Workstation Environment.Proceedings of the 6th International Conferenceon Distributed Computing Systems, 260–267.

HAMILTON, G. AND KOUGIOURIS, P. 1993. The SpringNucleus: A Microkernel for Objects. Proceedingsof the 1993 Summer USENIX Conference, 147–160.

HAN, Y. AND FINKEL, R. 1988. An Optimal Scheme forDisseminating Information. Proceedings of the1988 International Conference on Parallel Pro-cessing, II, 198–203.

HARCHOL-BALTER, M. AND DOWNEY, A. 1997. Exploit-ing Process Lifetime Distributions for DynamicLoad Balancing. ACM Transactions on Com-puter Systems 15, 3, 253–285. Previously ap-peared in the Proceedings of ACM Sigmetrics1996 Conference on Measurement and Modelingof Computer Systems, 13–24, May 1996.

HILDEBRAND, D. 1992. An Architectural Overview ofQNX. Proceedings of the USENIX Workshop onMicro-Kernels and Other Kernel Architectures,113–126.

HOFMANN, M.O., MCGOVERN, A., AND WHITEBREAD, K.1998. Mobile Agents on the Digital Battlefield.



Proceedings of the Autonomous Agents ’98, 219–225.

HOHL, F. 1998. A Model of Attacks of MaliciousHosts Against Mobile Agents. Proceedings of the4th Workshop on Mobile Objects Systems, INRIATechnical Report, 105–120.

HWANG, K., CROFT, W., WAH, B., BRIGGS, F., SIMONS,W., AND COATES, C. 1982. A UNIX-Based LocalComputer Network with Load Balancing. IEEEComputer, 15, 55–66.

JACQMOT, C. 1996. Load Management in Dis-tributed Computing Systems: Towards AdaptiveStrategies. Technical Report, Ph.D. Thesis, De-partement d’Ingenierie Informatique, Universitecatholique de Louvain.

JOHANSEN, D., VAN RENESSE, R., AND SCHNEIDER, F. 1995.Operating System Support for Mobile Agents.Proceedings of the 5th Workshop on Hot Topicsin Operating Systems, 42–45.

JUL, E., LEVY, H., HUTCHINSON, N., AND BLACK, A. 1988.Fine-Grained Mobility in the Emerald System.ACM Transactions on Computer Systems 6, 1,109–133.

JUL, E. 1988. Object Mobility in a DistributedObject-Oriented System. Technical Report 88-12-06, Ph.D. Thesis, Department of ComputerScience, University of Washington, Also Techni-cal Report no. 98/1, University of CopenhagenDIKU.

JUL, E. 1989. Migration of Light-weight Processesin Emerald. IEEE Technical Committee on Op-erating Systems Newsletter, 3(1)(1):20–23.

KAASHOEK, M. F., VAN RENESSE, R., VAN STAVEREN, H.,AND TANENBAUM, A. S. 1993. FLIP: An Internet-work Protocol for Supporting Distributed Sys-tems. ACM Transactions on Computer Systems,11(1).

KEMPER, A., KOSSMANN, D. 1995. Adaptable PointerSwizzling Strategies in Object Bases: Design,Realization, and Quantitative Analysis. VLDBJournal 4(3): 519-566(1995).

KHALIDI, Y. A., BERNABEU, J. M., MATENA, V., SHIRIFF,K., AND THADANI, M. 1996. Solaris MC: A Multi-Computer OS. Proceedings of the USENIX 1996Annual Technical Conference, 191–204.

KLEINROCK, L. 1976. Queueing Systems vol. 2: Com-puter Applications. Wiley, New York.

KNABE, F. C. 1995. Language Support for Mo-bile Agents. Technical Report CMU-CS-95-223,Ph.D. Thesis, School of Computer Science,Carnegie Mellon University, Also Technical Re-port ECRC-95-36, European Computer IndustryResearch Centre.

KOTZ, D., GRAY, R., NOG, S., RUS, D., CHAWLA, S., AND

CYBENKO., G. 1997. Agent Tcl: Targeting theneeds of mobile computers. IEEE Internet Com-puting 1, 4, 58–67.

KREMIEN, O. AND KRAMER, J. 1992. Methodical Anal-ysis of Adaptive Load Sharing Algorithms. IEEETransactions on Parallel and Distributed Sys-tems 3, 6, 747–760.

KRUEGER, P. AND LIVNY, M. 1987. The DiverseObjectives of Distributed Scheduling Policies.Proceedings of the 7th International Conferenceon Distributed Computing Systems, 242–249.

KRUEGER, P. AND LIVNY, M. 1988. A Comparison ofPreemptive and Non-Preemptive Load Balanc-ing. Proceedings of the 8th International Con-ference on Distributed Computing Systems, 123–130.

KRUEGER, P. AND CHAWLA, R. 1991. The Stealth Dis-tributed Scheduler. Proceedings of the 11th Inter-national Conference on Distributed ComputingSystems, 336–343.

KUNZ, T. 1991. The Influence of Different Work-load Descriptions on a Heuristic Load BalancingScheme. IEEE Transactions on Software Engi-neering 17, 7, 725–730.

LAMPSON, B. 1983. Hints for Computer System De-sign. Proceedings of the Ninth Symposium onOperating System Principles, 33–48.

LANGE, D. AND OSHIMA, M. 1998. Programming Mo-bile Agents in JavaTM -With the Java Aglet API.Addison Wesley Longman.

LAZOWSKA, E. D., LEVY, H. M., ALMES, G. T., FISHER,M. J., FOWLER, R. J., AND VESTAL, S. C. 1981. TheArchitecture of the Eden System. Proceedings ofthe 8th ACM Symposium on Operating SystemsPrinciples, 148–159.

LEA, R., JACQUEMOT, C., AND PILLVESSE, E. 1993.COOL: System Support for Distributed Pro-gramming. Communications of the ACM 36, 9,37–47.

LELAND, W. AND OTT, T. 1986. Load BalancingHeuristics and Process Behavior. Proceedings ofthe SIGMETRICS Conference, 54–69.

LIEDTKE, J. 1993. Improving IPC by Kernel Design.Proceedings of the Fourteenth Symposium on Op-erating Systems Principles, 175–188.

LITZKOW, M 1987. Remote UNIX-Turning IdleWork-stations into Cycle Servers. Proceedingsof the Summer USENIX Conference, 381–384.

LITZKOW, M., LIVNY, M., AND MUTKA, M. 1988. CondorA Hunter of Idle Workstations. Proceedings ofthe 8th International Conference on DistributedComputing Systems, 104–111.

LITZKOW, M. AND SOLOMON, M. 1992. SupportingCheckpointing and Process Migration outsidethe UNIX Kernel. Proceedings of the USENIXWinter Conference, 283–290.

LIVNY, M. AND MELMAN, M. 1982. Load Balancing inHomogeneous Broadcast Distributed Systems.Proceedings of the ACM Computer Network Per-formance Symposium, 47–55.

LO, V. 1984. Heuristic Algorithms for Task Assign-ments in Distributed Systems. Proceedings ofthe 4th International Conference on DistributedComputing Systems, 30–39.

LO, V. 1989. Process Migration for CommunicationPerformance. IEEE Technical Committee on Op-erating Systems Newsletter 3, 1, 28–30.



LO, V. 1988. Algorithms for Task Assignment andContraction in Distributed Computing Systems.Proceedings of the 1988 International Conferenceon Parallel Processing, II, 239–244.

LOUBOUTIN, S. 1991. An Implementation of a Pro-cess Migration Mechanism using Minix. Pro-ceedings of 1991 European Autumn Conference,Budapest, Hungary, 213–224.

LU, C., CHEN, A., AND LIU, J. 1987. Protocols for Re-liable Process Migration. INFOCOM 1987, The6th Annual Joint Conference of IEEE Computerand Communication Societies.

LU, C. 1988. Process Migration in Distributed Sys-tems. Ph.D. Thesis, Technical Report, Universityof Illinois at Urbana-Champaign.

LUX, W., HAERTIG, H., AND KUEHNHAUSER, W. E. 1993.Migrating Multi-Threaded, Shared Objects.Proceedings of 26th Hawaii International Con-ference on Systems Sciences, II, 642–649.

LUX, W. 1995. Adaptable Object Migration: Conceptand Implementation. Operating Systems Review29, 2, 54–69.

MA, P. AND LEE, E. 1982. A Task Allocation Model forDistributed Computing Systems. IEEE Transac-tions on Computers, C-31, 1, 41–47.

MAGUIRE, G. AND SMITH, J. 1988. Process Migra-tions: Effects on Scientific Computation. ACMSIGPLAN Notices, 23, 2, 102–106.

MALAN, G., RASHID, R., GOLUB, D., AND BARON, R. 1991.DOS as a Mach 3.0 Application. Proceedingsof the Second USENIX Mach Symposium, 27–40.

MANDELBERG, K. AND SUNDERAM, V. 1988. ProcessMigration in UNIX Networks. Proceedings ofUSENIX Winter Conference, 357–363.

MEHRA, P. AND WAH, B. W. 1992. Physical Level Syn-thetic Workload Generation for Load-BalancingExperiments. Proceedings of the First Sympo-sium on High Performance Distributed Comput-ing, 208–217.

MILLER, B. AND PRESOTTO, D. 1981. XOS: an Operat-ing System for the XTREE Architecture. Oper-ating Systems Review, 2, 15, 21–32.

MILLER, B., PRESOTTO, D., AND POWELL, M. 1987. DE-MOS/ MP: The Development of a Distributed Op-erating System. Software-Practice and Experi-ence 17, 4, 277–290.

MILOJICIC, D. S., BREUGST, B., BUSSE, I., CAMPBELL, J.,COVACI, S., FRIEDMAN, B., KOSAKA, K., LANGE, D.,ONO, K., OSHIMA, M., THAM, C., VIRDHAGRISWARAN,S., AND WHITE, J. 1998b. MASIF, The OMG Mo-bile Agent System Interoperability Facility. Pro-ceedings of the Second International Workshopon Mobile Agents, pages 50–67. Also appearedin Springer Journal on Personal Technologies, 2,117–129, 1998.

MILOJICIC, D. S., CHAUHAN, D., AND LAFORGE, W.1998a. Mobile Objects and Agents (MOA), De-sign, Implementation and Lessons Learned.Proceedings of the 4th USENIX Conference onObject-Oriented Technologies (COOTS), 179–

194. Also appeared in IEE Proceedings—Distributed Systems Engineering, 5, 1–14, 1998.

MILOJICIC, D., DOUGLIS, F., WHEELER, R. 1999. Mobil-ity: Processes, Computers, and Agents. Addison-Wesley Longman and ACM Press.

MILOJICIC, D., GIESE, P., AND ZINT, W. 1993a. Expe-riences with Load Distribution on Top of theMach Microkernel. Proceedings of the USENIXSymposium on Experiences with Distributed andMultiprocessor Systems.

MILOJICIC, D., ZINT, W., DANGEL, A., AND GIESE, P.1993b. Task Migration on the top of the MachMicrokernel. Proceedings of the third USENIXMach Symposium, 273–290.fc

MILOJICIC, D. 1993c. Load Distribution, Implemen-tation for the Mach Microkernel. Ph.D. Thesis,Technical Report, University of Kaiserslautern.Also Vieweg, Wiesbaden, 1994.

MILOJICIC, D., LANGERMAN, A., BLACK, D., SEARS, S.,DOMINIJANNI, M., AND DEAN, D. 1997. Concur-rency, a Case Study in Remote Tasking and Dis-tributed IPC. IEEE Concurrency 5, 2, 39–49.

MIRCHANDANEY, R., TOWSLEY, D., AND STANKOVIC, J.1989. Analysis of the Effects of Delays on LoadSharing. IEEE Transactions on Computers 38,11, 1513–1525.

MIRCHANDANEY, R., TOWSLEY, D., AND STANKOVIC, J.1990. Adaptive Load Sharing in Heteroge-neous Distributed Systems. Journal of Paralleland Distributed Computing, 331–346.

MULLENDER, S. J., VAN ROSSUM, G., TANENBAUM, A.S., VAN RENESSE, R., AND VAN STAVEREN, H.1990. Amoeba—A Distributed Operating Sys-tem for the 1990s. IEEE Computer 23, 5, 44–53.

MUTKA, M. AND LIVNY, M. 1987. Scheduling RemoteProcessing Capacity in a Workstation ProcessorBank Computing System. Proceedings of the 7thInternational Conference on Distributed Com-puting Systems, 2–7.

NELSON, M. N., AND OUSTERHOUT, J. K. 1988. Copy-on-Write for Sprite. Proceedings of the Summer1988 USENIX Conference, 187–201.

NELSON, M. N., WELCH, B. B., AND OUSTERHOUT, J. K.1988. Caching in the Sprite Network File Sys-tem. ACM Transaction on Computer Systems 6,1, 134–54.

NELSON, R. AND SQUILLANTE, M. 1995. StochasticAnalysis of Affinity Scheduling and Load Bal-ancing in Parallel Processing Systems. IBM Re-search Report RC 20145.

NI, L. M. AND HWANG, K. 1985. Optimal Load Bal-ancing in a Multiple Processor System withMany Job Classes. IEEE Transactions on Soft-ware Engineering, SE-11, 5, 491–496.

NICHOLS, D. 1987. Using Idle Workstations in aShared Computing Environment. Proceedings ofthe 11th Symposium on OS Principles, 5–12.

NICHOLS, D. 1990. Multiprocessing in a Networkof Workstations. Ph.D. Thesis, Technical ReportCMU-CS-90-107, Carnegie Mellon University.



NUTTAL, M. 1994. Survey of Systems Providing Pro-cess or Object Migration. Operating System Re-view, 28, 4, 64–79.

OMG, 1996. Common Object Request Broker Archi-tecture and Specification. Object ManagementGroup Document Number 96.03.04.

OUSTERHOUT, J., CHERENSON, A., DOUGLIS, F., NELSON,M., AND WELCH, B. 1988. The Sprite NetworkOperating System. IEEE Computer, 23–26.

OUSTERHOUT, J. 1994. TcL and the Tk Toolkit.Addison-Wesley Longman.

PAINDAVEINE, Y. AND MILOJICIC, D. 1996. Process v.Task Migration. Proceedings of the 29th AnnualHawaii International Conference on System Sci-ences, 636–645.

PARTRIDGE, C. 1994. Gigabit Networking. AddisonWesley.

PEINE, H. AND STOLPMANN, T. 1997. The Architectureof the Ara Platform for Mobile Agents. Proceed-ings of the First International Workshop on Mo-bile Agents (MA’97). LNCS 1219, Springer Ver-lag, 50–61.

PETRI, S. AND LANGENDORFER, H. 1995. Load Balanc-ing and Fault Tolerance in Workstation ClustersMigrating Groups of Communicating Processes.Operating Systems Review 29 4, 25–36.

PHELAN, J. M. AND ARENDT, J. W. 1993. An OS/2Personality on Mach. Proceedings of the thirdUSENIX Mach Symposium, 191–202.

PHILIPPE, L. 1993. Contribution a l’etude et larealisation d’un systeme d’exploitation a im-age unique pour multicalculateur. Ph.D. Thesis,Technical Report 308, Universite de Franche-comte.

PIKE, R., PRESOTTO, D., THOMPSON, K., AND TRICKEY, H.1990. Plan 9 from Bell Labs. Proceedings of theUKUUG Summer 1990 Conference, 1–9.

PLATFORM COMPUTING. 1996. LSF User’s and Admin-istrator’s Guides, Version 2.2, Platform Comput-ing Corporation.

POPEK, G., WALKER, B. J., CHOW, J., EDWARDS, D.,KLINE, C., RUDISIN, G., AND THIEL, G. 1981. Lo-cus: a Network-Transparent, High ReliabilityDistributed System. Proceedings of the 8th Sym-posium on Operating System Principles, 169–177.

POPEK, G. AND WALKER, B. 1985. The Locus Dis-tributed System Architecture. MIT Press.

POWELL, M. AND MILLER, B. 1983. Process Migra-tion in DEMOS/MP. Proceedings of the 9th Sym-posium on Operating Systems Principles, 110–119.

PU, C., AUTREY, T., BLACK, A., CONSEL, C., COWAN, C., IN-OUYE, J., KETHANA, L., WALPOLE, J., AND ZHANG, K.1995. Optimistic Incremental Specialization.Proceedings of the 15th Symposium on Operat-ing Systems Principles, 314–324.

QUISQUATER, J.-J. AND DESMEDT, Y. G. 1991. ChineseLotto as an Exhaustive Code-Breaking Machine.IEEE Computer 24, 11, 14–22.

RANGANATHAN, M., ACHARYA, A., SHARMA, S. D.,AND SALTZ, J. 1997. Networkaware Mobile Pro-grams. Proceedings of the USENIX 1997 AnnualTechnical Conference, 91–103.

RASHID, R. AND ROBERTSON, G. 1981. Accent: a Com-munication Oriented Network Operating Sys-tem Kernel. Proceedings of the 8th Symposiumon Operating System Principles, 64–75.

RASHID, R. 1986. From RIG to Accent to Mach: TheEvolution of a Network Operating System. Pro-ceedings of the ACM/IEEE Computer SocietyFall Joint Computer Conference, 1128–1137.

ROSENBERRY, W., KENNEY, D., AND FISHER, G. 1992.Understanding DCE. O’Reilly & Associates, Inc.

ROTHERMEL, K., AND HOHL, F. 1998. Mobile Agents.Proceedings of the Second International Work-shop, MA’98, Springer Verlag.

ROUSH, E. T. 1995. The Freeze Free Algorithmfor process Migration. Ph.D. Thesis, Techni-cal Report, University of Illinois at Urbana-Champaign.

ROUSH, E. T. AND CAMPBELL, R. 1996. Fast DynamicProcess Migration. Proceedings of the Interna-tional Conference on Distributed Computing Sys-tems, 637–645.

ROWE, L. AND BIRMAN, K. 1982. A Local NetworkBased on the UNIX Operating System. IEEETransactions on Software Engineering, SE-8, 2,137–146.

ROZIER, M. 1992. Chorus (Overview of the ChorusDistributed Operating System). USENIX Work-shop on Micro Kernels and Other Kernel Archi-tectures, 39–70.

SCHILL, A. AND MOCK, M. 1993. DC++: DistributedObject Oriented System Support on top of OSFDCE. Distributed Systems Engineering 1, 2,112–125.

SCHRIMPF, H. 1995. Migration of Processes, Filesand Virtual Devices in the MDX Operating Sys-tem. Operating Systems Review 29, 2, 70–81.

SHAMIR, E. AND UPFAL, E. 1987. A Probabilistic Ap-proach to the Load Sharing Problem in Dis-tributed Systems. Journal of Parallel and Dis-tributed Computing, 4, 5, 521–530.

SHAPIRO, M. 1986. Structure and Encapsulationin Distributed Systems: The PROXY Princi-ple. Proceedings of the 6th International Con-ference on Distributed Computing Systems, 198–204.

SHAPIRO, M., DICKMAN, P., AND PLAINFOSSE, D. 1992.Robust, Distributed References and AcyclicGarbage Collection. Proceedings of the Sympo-sium on Principles of Distributed Computing,135–146.

SHAPIRO, M., GAUTRON, P., AND MOSSERI, L. 1989. Per-sistence and Migration for C++ Objects. Pro-ceedings of the ECOOP 1989-European Confer-ence on Object-Oriented Programming.

SHIVARATRI, N. G. AND KRUEGER, P. 1990. Two Adap-tive Location Policies for Global Scheduling Al-gorithms. Proceedings of the 10th International



Conference on Distributed Computing Systems,502–509.

SHIVARATRI, N., KRUEGER, P., AND SINGHAL, M. 1992.Load Distributing for Locally Distributed Sys-tems. IEEE Computer, 33–44.

SHOHAM, Y. 1997. An Overview of Agent-orientedProgramming. in J. M. Bradshaw, editor, Soft-ware Agents, 271–290. MIT Press.

SHOCH, J. AND HUPP, J. 1982. The Worm Programs—Early Experience with Distributed Computing.Communications of the ACM 25, 3, 172–180.

SHUB, C. 1990. Native Code Process-Originated Mi-gration in a Heterogeneous Environment. Pro-ceedings of the 18th ACM Annual Computer Sci-ence Conference, 266–270.

SINGHAL, M. AND SHIVARATRI, N. G. 1994. AdvancedConcepts in Operating Systems. McGraw-Hill.

SINHA, P., MAEKAWA, M., SHIMUZU, K., JIA, X., ASHIHARA,UTSUNOMIYA, N., PARK, AND NAKANO, H. 1991.The Galaxy Distributed Operating System.IEEE Computer 24, 8, 34–40.

SKORDOS, P. 1995. Parallel Simulation of SubsonicFluid Dynamics on a Cluster of Workstations.Proceedings of the Fourth IEEE InternationalSymposium on High Performance DistributedComputing.

SMITH, J. M. 1988. A Survey of Process MigrationMechanisms. Operating Systems Review 22, 3,28–40.

SMITH, J. M. AND IOANNIDIS, J. 1989. Implement-ing Remote fork() with Checkpoint-Restart.IEEE Technical Committee on Operating Sys-tems Newsletter 3, 1, 15–19.

SMITH, P. AND HUTCHINSON, N. 1998. HeterogeneousProcess Migration: The Tui System. Software-Practice and Experience 28, 6, 611–639.

SOH, J. AND THOMAS, V. 1987. Process Migration forLoad Balancing in Distributed Systems. TEN-CON, 888–892.

SQUILLANTE, M. S. AND NELSON, R. D. 1991. Analysisof Task Migration in Shared-Memory Multipro-cessor Scheduling. Proceedings of the ACM SIG-METRICS Conference 19, 1, 143–155.

STANKOVIC, J. A. 1984. Simulation of the three Adap-tive Decentralized Controlled Job Scheduling al-gorithms. Computer Networks, 199–217.

STEENSGAARD, B. AND JUL, E. 1995. Object and Na-tive Code Thread Mobility. Proceedings of the15th Symposium on Operating Systems Princi-ples, 68–78.

STEKETEE, C., ZHU, W., AND MOSELEY, P. 1994. Imple-mentation of Process Migration in Amoeba. Pro-ceedings of the 14th International Conference onDistributed Computer Systems, 194–203.

STONE, H. 1978. Critical Load Factors in Two-Processor Distributed Systems. IEEE Transac-tions on Software Engineering, SE-4, 3, 254–258.

STONE, H. S. AND BOKHARI, S. H. 1978. Control of Dis-tributed Processes. IEEE Computer 11, 7, 97–106.

STUMM, M. 1988. The Design and Implementationof a Decentralized Scheduling Facility for aWorkstation Cluster. Proceedings of the Sec-ond Conference on Computer Workstations, 12–22.

SUN MICROSYSTEMS. 1998. JiniTM Software Simpli-fies Network Computing. http://www.sun.com/980713/jini/feature.jhtml.

SVENSSON, A. 1990. History, an Intelligent LoadSharing Filter. Proceedings of the 10th Interna-tional Conference on Distributed Computing Sys-tems, 546–553.

SWANSON, M., STOLLER, L., CRITCHLOW, T., AND KESSLER,R. 1993. The Design of the SchizophrenicWorkstation System. Proceedings of the thirdUSENIX Mach Symposium, 291–306.

TANENBAUM, A. S., RENESSE, R. VAN, STAVEREN, H. VAN.,SHARP, G. J., MULLENDER, S. J., JANSEN, A. J.,AND VAN ROSSUM, G. 1990. Experiences with theAmoeba Distributed Operating System. Com-munications of the ACM, 33, 12, 46–63.

TANENBAUM, A. 1992. Modern Operating Systems.Prentice Hall, Englewood Cliffs, New Jersey.

TARDO, J. AND VALENTE, L. 1996. Mobile AgentSecurity and Telescript. Proceedings ofCOMPCON’96, 52–63.

TEODOSIU, D. 2000. End-to-End Fault Containmentin Scalable Shared-Memory Multiprocessors.Ph.D. Thesis, Technical Report, Stanford Univer-sity.

THEIMER, M. H. AND HAYES, B. 1991. HeterogeneousProcess Migration by Recompilation. Proceed-ings of the 11th International Conference on Dis-tributed Computer Systems, 18–25.

THEIMER, M. AND LANTZ, K. 1988. Finding Idle Ma-chines in a Workstation-Based Distributed Sys-tem. IEEE Transactions on Software Engineer-ing, SE-15, 11, 1444–1458.

THEIMER, M., LANTZ, K., AND CHERITON, D. 1985. Pre-emptable Remote Execution Facilities for the VSystem. Proceedings of the 10th ACM Sympo-sium on OS Principles, 2–12.

TRACEY, K. M. 1991. Processor Sharing for Coopera-tive Multi-task Applications. Ph.D. Thesis, Tech-nical Report, Department of Electrical Engineer-ing, Notre Dame, Indiana.

TRITSCHER, S. AND BEMMERL, T. 1992. Seitenorien-tierte Prozessmigration als Basis fuer Dynamis-chen Lastausgleich. GI/ITG Pars Mitteilungen,no 9, 58–62.

TSCHUDIN, C. 1997. The Messenger EnvironmentM0 - A condensed description. In Mobile ObjectSystems: Towards the Programmable Internet,LNCS 1222, Springer Verlag, 149–156.

VAN DIJK, G. J. W. AND VAN GILS, M. J. 1992. Efficientprocess migration in the EMPS multiprocessorsystem. Proceedings 6th International ParallelProcessing Symposium, 58–66.

VAN RENESSE, R., BIRMAN, K. P., AND MAFFEIS, S. 1996.Horus: A Flexible Group Communication Sys-tem. Communication of the ACM 39, 4, 76–85.



VASWANI, R. AND ZAHORJAN, J. 1991. The implicationsof Cache Affinity on Processor Scheduling forMultiprogrammed Shared Memory Multiproces-sors. Proceedings of the Thirteenth Symposiumon Operating Systems Principles, 26–40.

VENKATESH, R. AND DATTATREYA, G. R. 1990. Adap-tive Optimal Load Balancing of Loosely Cou-pled Processors with Arbitrary Service TimeDistributions. Proceedings of the 1990 Interna-tional Conference on Parallel Processing, I, 22–25.

VIGNA, G. 1998. Mobile Agents Security, LNCS1419, Springer Verlag.

VITEK, I., SERRANO, M., AND THANOS, D. 1997. Secu-rity and Communication in Mobile Object Sys-tems. In Mobile Object Systems: Towards theProgrammable Internet, LNCS 1222, SpringerVerlag, 177–200.

WALKER, B., POPEK, G., ENGLISH, R., KLINE, C., AND

THIEL, G. 1983. The LOCUS Distributed Oper-ating System. Proceedings of the 9th Symposiumon Operating Systems Principles 17, 5, 49–70.

WALKER, B. J. AND MATHEWS, R. M. 1989. Process Mi-gration in AIX’s Transparent Computing Facil-ity (TCF). IEEE Technical Committee on Oper-ating Systems Newsletter, 3, 1, (1) 5–7.

WANG, Y.-T. AND MORRIS, R. J. T. 1985. Load Sharingin Distributed Systems. IEEE Transactions onComputers, C-34, 3, 204–217.

WANG, C.-J., KRUEGER, P., AND LIU, M. T. 1993. In-telligent Job Selection for Distributed Schedul-ing. Proceedings of the 13th International Con-ference on Distributed Computing Systems, 288–295.

WELCH, B. B. AND OUSTERHOUT, J. K. 1988. Pseudo-Devices: User-Level Extensions to the Sprite FileSystem. Proceedings of the USENIX SummerConference, 7–49.

WELCH, B. 1990. Naming, State Management andUser-Level Extensions in the Sprite DistributedFile System. Ph.D. Thesis, Technical ReportUCB/CSD 90/567, CSD (EECS), University ofCalifornia, Berkeley.

WHITE, J. 1997. Telescript Technology: An Intro-duction to the Language. White Paper, GeneralMagic, Inc., Sunnyvale, CA. Appeared in Brad-shaw, J., Software Agents, AAAI/MIT Press.

WHITE, J. E., HELGESON, S., AND STEEDMAN, D. A. 1997.System and Method for Distributed Computa-tion Based upon the Movement, Execution, andInteraction of Processes in a Network. UnitedStates Patent no. 5603031.

WIECEK, C. A. 1992. A Model and Prototype of VMSUsing the Mach 3.0 Kernel. Proceedings of theUSENIX Workshop on Micro-Kernels and OtherKernel Architectures, 187–204.

WONG, R., WALSH, T., AND PACIOREK, N. 1997. Con-cordia: An Infrastructure for CollaboratingMobile Agents. Proceedings of the First Interna-tional Workshop on Mobile Agents, LNCS 1219,Springer Verlag, 86–97.

XU, J. AND HWANG, K. 1990. Heuristic Methods forDynamic Load Balancing in a Message-PassingSupercomputer. Proceedings of the Supercom-puting’90, 888–897.

ZAJCEW, R., ROY, P., BLACK, D., PEAK, C., GUEDES,P., KEMP, B., LOVERSO, J., LEIBENSPERGER, M.,BARNETT, M., RABII, F., AND NETTERWALA, D. 1993.An OSF/1 UNIX for Massively Parallel Multi-computers. Proceedings of the Winter USENIXConference, 449–468.

ZAYAS, E. 1987a. Attacking the Process MigrationBottleneck. Proceedings of the 11th Symposiumon Operating Systems Principles, 13–24.

ZAYAS, E. 1987b. The Use of Copy-on-Reference ina Process Migration System. Ph.D. Thesis, Tech-nical Report CMU-CS-87-121, Carnegie MellonUniversity.

ZHOU, D. 1987. A Trace-Driven Simulation Study ofDynamic Load Balancing. Ph.D. Thesis, Techni-cal Report UCB/CSD 87/305, CSD (EECS), Uni-versity of California, Berkeley.

ZHOU, S. AND FERRARI, D. 1987. An Experimen-tal Study of Load Balancing Performance. Pro-ceedings of the 7th IEEE International Confer-ence on Distributed Computing Systems, 490–497.

ZHOU, S. AND FERRARI, D. 1988. A Trace-Driven Sim-ulation Study of Dynamic Load Balancing. IEEETransactions on Software Engineering 14, 9,1327–1341.

ZHOU, S., ZHENG, X., WANG, J., AND DELISLE, P. 1994.Utopia: A Load Sharing Facility for Large,Heterogeneous Distributed Computer Systems.Software-Practice and Experience.

ZHU, W. 1992. The Development of an Environmentto Study Load Balancing Algorithms, Processmigration and load data collection. Ph.D. The-sis, Technical Report, University of New SouthWales.

ZHU, W., STEKETEE, C., AND MUILWIJK, B. 1995.Load Balancing and Workstation Autonomy onAmoeba. Australian Computer Science Commu-nications (ACSC’95) 17, 1, 588–597.

Received October 1996; revised December 1998 and July 1999; accepted August 1999


Process Migration - Virginia Techcourses.cs.vt.edu/.../fall05-kafura/Papers/Migration/ProcessMigratio… · Process Migration 243 of the potentials migration can offer to different

Documents