ICCAD 2013 ver5

Formal Verification of Distributed Dynamic ThermalManagement

Muhammad Ismail∗, Osman Hasan∗, Thomas Ebi†, Muhammad Shafique† and Joerg Henkel†∗School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan

Email: {10mseemismail, osman.hasan}@seecs.nust.edu.pk†Chair for Embedded Systems, University of Karlsruhe, Karlsruhe, Germany

Email: {ebi,shafique,henkel}@informatik.uni-karlsruhe.de

Abstract—Simulation is the state-of-the-art analysis techniquefor distributed thermal management schemes. Due to the nu-merous parameters involved and the distributed nature of theseschemes, such non-exhaustive verification may fail to catchfunctional bugs in the algorithm or may report misleadingperformance characteristics. To overcome these limitations, wepropose a methodology to perform formal verification of dis-tributed dynamic thermal management for many-core systems.The proposed methodology is based on the SPIN model checkerand the Lamport timestamps algorithm. Our methodology allowsspecification and verification of both functional and timing prop-erties in a distributed many-core system. In order to illustratethe applicability and benefits of our methodology, we perform acase study on a state-of-the-art agent-based distributed thermalmanagement scheme.

I. INTRODUCTION AND RELATED WORK

As the semiconductor industry moves towards smallertechnology nodes, elevated temperatures resulting from theincreased power densities are becoming a growing concern.In fact, Dynamic Thermal management (DTM) of distributednature has been identified as one the key reliability challengesin the ITRS roadmap [11]. At the same time, the growingintegration density is paving the way for future many-coresystems consisting of hundreds and even thousands of cores ona single chip [4]. From a thermal management perspective, thesesystems bring both new opportunities as well as new challenges.For one, whereas DTM in single-core systems is largely limitedto Dynamic Voltage and Frequency Scaling (DVFS), many-coresystems present the possibility for spreading power consumptionin order to balance temperature over a larger area through themechanism of task migration. However, the increased problemspace related to the large number of cores makes the complexityof DTM grow considerably.

Traditionally, DTM decisions have been made using cen-tralized approaches with global knowledge. These, however,quickly become infeasible due to lack of scalability whenentering the many-core era [7] [8]. As a result, distributedthermal management schemes have emerged [5], [7]–[9] whichtackle the complexity and scalability issues of many-core DTMby transforming the problem space from a global one to manysmaller regional ones which can exploit locality when makingDTM decisions. For a distributed DTM scheme to achieve thesame quality as is possible from one using global knowledge,however, it becomes necessary for there to be an exchange ofstate information across regions in order to negotiate a near-optimal system state configuration [7]. The choice of tuningparameters for this negotiation has been identified as a critical

issue in ensuring a stable system [12]. Up until now thesedistributed DTM schemes have been exclusively analyzed usingeither simulations or running on real systems. However, due tothe non-exhaustive nature of simulation, such analysis aloneis not enough to account for and guarantee stability in allpossible system configurations. Especially when consideringmany-core systems, the number of different configurations, e.g.,task-to-core mappings, grows exponentially with the number ofcores. Even if some corner cases can be specifically targeted,there is no proof that these represent a worst-case scenario, andit is never possible to consider or even foresee all corner cases.Moreover, using simulation we may show that for a givenset of tasks and cores, a small number of mappings result inlocalized minima. However, in distributed DTM approaches thisactually means that a local region of cores may be successfullyapplying DTM from their point of view although from theglobal view temperatures are really maximal. Thus, simulationbased analysis cannot be considered complete and often resultsin missing critical bugs, which in turn may lead to delaysin deployment of DTM schemes as happened in the case ofFoxton DTM that was designed for the Montecito chip [6].

The above mentioned limitations can be overcome byusing model checking [3] for the analysis of distributed DTM.The main principle behind model checking is to constructa computer based mathematical model of the given systemin the form of an automata or state-space and automaticallyverify, within a computer, that this model meets rigorousspecifications of intended behavior. Due to its mathematicalnature, 100% completeness and soundness of the analysis canbe guaranteed [2]. Moreover, the ability to provide counterexamples in case of failures and the automatic nature of modelchecking makes it a more preferable choice for industrialusage as compared to the other mainstream formal verificationapproaches like theorem proving.

Model checking has been successfully used for analyzingsome unicore DTM schemes (e.g., [16], [18]). Similarly, proba-bilistic model checking of a DTM for multicore architectures ispresented in [15]. This work conducted a probabilistic analysisof frequency effects through DVFS, time and power spentover budget along with an estimate of required verificationefforts. In order to raise the level of formally verifying complexDTM schemes, statistical model checking of power gatingschemes has been recently reported [13]. However, to thebest of our knowledge, so far no formal verification method,including model checking, has been used for the verification ofa distributed DTM for many-core systems. This paper intends

to fill this gap and proposes a methodology for the functionaland timing verification of distributed DTM schemes.

A. Our Novel Contributions and Concept Overview

We present a novel methodology for formal verification ofdistributed DTM schemes in many-core systems. The key ideais to leverage the SPIN model checker [10] (an open source toolfor the formal verification of distributed software systems) andLamport timestamps [14] (a technique that enables to determinethe order of events in a distributed system execution). Ourmethodology allows the designer to formally model or specifythe behavior of distributed DTM schemes in the PROcessMEta LAnguage (PROMELA). These models are then verifiedto exhibit the desired functional properties using the SPINmodel checker as it directly accepts PROMELA models. Ourmethodology introduces the Lamport timestamps algorithm inthe PROMELA model of the given distributed DTM schemeto facilitate the verification of timing properties via the SPINmodel checker.

In order to illustrate the effectiveness of our methodology,we perform a case study on the the formal verification of astate-of-the-art agent-based distributed DTM scheme, namelyThermal-aware Agent-based Power Economy (TAPE) [7]. Themain reason behind the choice of TAPE for this case studyis that it is highly scalable while still achieving comparableresults to centralized DTM approaches which operate on globalsystem knowledge like PDTM [19] and HRTM [17].

Paper Organization: The rest of the paper is organized asfollows: Section II provides an overview of model checkingand SPIN to aid the understanding of the rest of the paper.The proposed methodology is presented in Section III. Thisis followed by the formal modeling and verification of theTAPE algorithm in Section IV. Formal verification results arediscussed in V. Finally, Section VI concludes the paper.

II. PRELIMINARIES

A. Model Checking

Model checking [3] is primarily used as the verificationtechnique for reactive systems, i.e., systems whose behavioris dependent on time and their environment, like controllersof digital circuits and communication protocols. The inputsto a model checker include the finite-state model of thesystem that needs to be analyzed along with the intendedsystem properties, which are expressed in temporal logic.The model checker automatically and exhaustively verifiesif the properties hold for the given system while providing anerror trace in case of a failing property. The state-space of asystem can be very large, or sometimes even infinite. Thus,it becomes computationally impossible to explore the entirestate-space with limited resources of time and memory. Thisproblem, termed as state-space explosion, is usually resolvedby developing abstract, less complex, models of the system.Moreover, many efficient techniques, like symbolic and boundedmodel checking, have been proposed to alleviate the memoryand computation requirements of model checking.

B. SPIN Model Checker

SPIN model checker [10], developed by Bell Labs, is awidely used formal verification tool for analyzing distributed

and concurrent software systems. The system that needs tobe verified is expressed in a high-level language PROMELA,which is based on Dijkstra’s guarded command language andhas a syntax that is quite similar to the C programming language.The behavior of the given distributed system is expressedusing asynchronous processes. Every process can have multipleinstantiations to model cases where multiple distributed moduleswith similar behavior exist. The processes can communicatewith one another via synchronous (rendezvous) or asynchronous(buffered) message channels. Both global and local variables ofboolean, byte, short, int, unsigned and single dimensional arrayscan be declared. Defining new data types is also supported.Once the system model is formed in PROMELA then it isautomatically translated to a automaton or state-space graph bySPIN. This step is basically done by translating each processto its corresponding automaton first and then forming anasynchronous interleaving product of these automata to obtainthe global behavior [10].

The properties to be verified can be specified in SPIN usingLinear Temporal Logic (LTL) or assertions. LTL allows us toformally specify time-dependant properties using both logical(conjunction (&&), disjunction (‖), negation (!), implication(->) and equality (<->) and temporal operators, i.e., always([]), eventually (<>), next (X) and until (∪). For verification,the given property is first automatically converted to a Büchiautomaton and then its synchronous product with the automatonrepresenting the global behavior is formed by the SPIN modelchecker. Next, an exhaustive verification algorithm, like theDepth First Search (DFS), is used to automatically checkif the property holds for the given model or not. Besidesverifying the logical consistency of a property, SPIN can alsobe used to check for the presence of deadlocks, race conditions,unspecified receptions and incompleteness. Moreover, fordebugging purposes, SPIN also supports random, interactiveand guided simulation.

III. OUR METHODOLOGY FOR FORMAL VERIFICATION OFDISTRIBUTED DTM

The most critical functional aspect of any distributedDTM scheme is its ability to reach near-optimal system stateconfiguration from all possible scenarios. Moreover, the timerequired to reach such a stable state and the effect of variousparameters on this time is the most interesting timing relatedbehavior. The proposed methodology, depicted in Figure 1),utilizes the SPIN model checker to verify properties related toboth of these aspects for any given distributed DTM scheme.Our methodology exhibits 7 key steps (discussed in detail inthe subsequent sub-sections).

1) Modeling System in PROMELA: a model of thedistributed DTM scheme and on-chip many-coresystem is constructed in the PROMELA language;Lamport Timestamps are added.

2) Simulation: the model is simulated using SPIN toidentify modeling bugs at an early stage, i.e., prior tothe rigorous formal verification of the model.

3) Check for Deadlocks: the presence of deadlocks inthe model are checked using the efficient and rigorousdeadlock check feature of the SPIN model checker.

4) Specification of LTL Properties: the desired func-tional properties for the given DTM are representedin LTL.

Fig. 1: Formal Verification Methodology for Distributed DTM

5) Functional Verification: the LTL properties are for-mally checked using the verification algorithms ofSPIN.

6) Debug or Model Simplification/Optimization: incase of a property failure, the error trace is generatedfor debugging purposes, whereas in case of state-space explosion, the model is simplified using variableabstractions.

7) Timing Verification: The timing properties are veri-fied based on the Lamport Timestamps algorithm.

A. Constructing the Model of Distributed DTM

The PROMELA model of the given distributed DTM systemcan be constructed by individually describing each autonomousnode of the system using one or more processes. Each processdescription will also include message channels for representingtheir interactions with other processes and in this way theycan share the information of thermal effects and logical time.Moreover, an initialization process should also be used to assigninitial values to the variables used to represent the physicalstarting conditions of the given DTM system along with therange of each variable. The coding can be done in a quitestraightforward manner due to the C like syntax of PROMELA.However, choosing the most appropriate data type for eachvariable of the given scheme needs careful attention. Due to theextensive interaction of DTM schemes with their continuoussurroundings, some of the variables used in the models of suchschemes are continuous in nature. Temperature is a foremostexample in this regard. However, due to the automata basedverification approach of model checking, variables with infiniteprecision cannot be handled. Choosing data-types with large setof possible values also results in state-space-explosion problembecause of the large number of their possible combinations.Therefore, we have to discretize all the real or floating-pointvariables, which usually have either infinite or a large set ofvalues, of the given DTM scheme. The lesser the number ofpossible values, the faster is the verification. On the other hand,lowering the number of possible values may compromise the

exhaustiveness of the analysis. However, it is important to notethat the discretization of the variables is not a major concernin functional verification since our focus is on the coverage ofall the possible scenarios and not on the computation of exactvalues.

B. Modeling Timing Behavior using Lamport Timestamps

Just like any verification exercise of engineering systems,the verification of timing properties of distributed DTM schemesis a very important aspect. For example, we may be interestedin the time required to reach a stable state after n tasks areequally mapped to different tiles in a distributed DTM scheme.However, due to the distributed nature of these schemes, formalspecification and verification of timing properties is not astraightforward task as we may be unable to distinguish betweenwhich one of the two occurring events occurred first. Lamporttimestamps algorithm [14] provides a very efficient solution tothis problem. The main idea behind this algorithm is to associatea counter with every node of the distributed system such thatit increments once for every event that takes place in that node.The total ordering of all the events of the distributed system isachieved by ensuring that every node shares its counter valuewith any other node that it communicates with and it updatesits counter value whenever it receives a value greater than itsown counter value.

In this paper, we propose to utilize Lamport timestampsalgorithm to determine the total number of events in thePROMELA model of a given distributed DTM scheme. Themain advantage of this choice is that we can utilize theSPIN model checker, which specializes in the verification ofdistributed systems, to specify and verify both functional andtiming properties of the given DTM scheme.

We propose to use a global array now such that its size isequal to the number of distributed nodes in the given distributedDTM system. Thus, each node will have a unique index in thisarray and all the processes that are used to model the behaviorof this particular node will use the same index. Whenever anevent takes place inside a process the value of the correspondingindexed variable in the array now is incremented. Whenevertwo nodes communicate, they can share the values of theircorresponding variables in the array now and can update thembased on the Lamport Timestamps algorithm.

C. Functional Verification in SPIN

Once the model is developed, we propose to check it viathe random and interactive simulation methods of SPIN. Therandomized test vectors often reveal some critical flaws and thebugs, which can be fixed by updating the PROMELA model.The main motivation of performing this simulation is to beable to catch PROMELA modeling flaws at an earlier stage.

Deadlocks: Distributed systems are quite prone to enterdeadlocks, i.e., the situation under which two nodes are waitingfor results of one another and thus the whole activity isstalled. It is almost impossible to guarantee that there is nodeadlock in a given distributed DTM system using simulation.Model checking, on the other hand, is very well-suited fordetecting deadlocks. The deadlock check can be expressed inLTL as [](X(true)), which ensures that at any point in theexecution(always), a valid next state must exist and thus there isno point in the whole execution from where the progress halts.

If a deadlock is found, then the corresponding error trace isexecuted on the PROMELA model using simulation to identifyits root cause, which could be the PROMELA model or thesystem behavior itself.

LTL Properties: The next step in the proposed methodol-ogy is to verify LTL properties for the functional verification ofthe given distributed DTM system. In most of the cases, theseproperties are related to the stable state, i.e., the state whenthe distributed nodes of the given DTM system have achievedtheir goals, e.g., even distribution of temperatures or power,and thus their mutual transactions cease to exist or are veryminimal, i.e., only occurring as a reaction to stimulus such asrising temperatures. As shown in Figure 1, we can have threepossible outcomes at this stage, i.e.,

1) the property passes for the given DTM system andthe flow is forwarded to the timing verification.

2) the property fails; the SPIN model checker returnsa counter example showing the exact path in thestate-space where the property failed for debuggingpurposes.

3) the SPIN model checker gives an out-of-memorymessage due to the state-space explosion problem; inthis case, we need to reduce the size of the state-spaceand for that purpose we can explore the options ofreducing the possible values of variables or restrictingthe number distributed nodes in the model.

D. Timing Verification in SPIN

The final step in our methodology is to do the timingverification. For this purpose, we propose to use a very rarelyused but useful feature of SPIN, i.e., the ability to compute theranges of model variables during execution [1]. The values inthe range of 0 to 255 are trackable only but various countingvariables can be utilized in conjunction to increase this range ifrequired. Based on this feature, we keep track of the values ofthe array now and thus can verify timing properties of DTMschemes in terms of event executions, such as the time unitsrequired to attain the evenly distributed temperature conditionunder a given set of parameters.

In order to illustrate the applicability and effectiveness ofour methodology, we perform a case study on an advanced state-of-the-art Distributed DTM scheme, namely the Thermal-awareAgent-based Power Economy (TAPE) [7].

Core Agent

Agent Negotiation

Agent properties: Scalable Agents act locally

Situated Software entity in each core

SocialMay negotiate with their neighbors

ProactiveTriggered before threshold is reached

ReactiveReacts to outside stimuli (i.e. from sensors)

Light-weight Require small memory/computation footprint

Fig. 2: Overview of agent negotiation in TAPE

IV. CASE STUDY ON THE FORMAL VERIFICATION OFTAPE DTM

A. Introduction to TAPE

The Thermal-aware Agent-based Power Economy (TAPE)[7] is an advanced distributed DTM approach for many-core

systems organized in a grid structure. It employs the conceptof a fully distributed agent-based system in order to deal withthe complexity of thermal management in many-core systems.Each core is assigned its own agent which is able to negotiatewith its immediate neighbors (i.e. adjacent cores, as depictedin Figure 2). Thermal management itself is performed bydistributing power budgets which dictate task execution amongthe cores. Thus the agent negotiation consists of distributingthis power budget based on the concept of supply-and-demand,taking the currently measured temperatures into account. Sinceeach agent is only able to trade with its neighbors (east, west,north and south), multiple agent negotiations are requiredto propagate power budget across the chip. At start-up, theavailable tasks are randomly mapped on the cores in the grid.Every core n keeps track of its freen and usedn power unitsand new task assignment to a core results in increasing anddecreasing its usedn and freen power units, respectively, bya number that is determined by the requirements of the newlyassigned task. Re-mapping of tasks is automatically invokedwhen either there are no free power units available in the nodeor the difference of temperatures in the neighboring nodes goesbeyond certain threshold. The tasks are re-mapped to the nodeshaving the highest sellTn

− buyTnvalues and thus the sellTn

and buyTn values of a core govern its agent negotiations.

Note: For better and clear understanding of the subsequentsections, system modeling and formal verification steps, ter-minology and parameters, we repeat the pseudo code of theTAPE in the appendix; see [7] for more details.

B. Modeling TAPE in PROMELA

As discussed in Section III , the first step in our methodology(see Figure 1) is to construct the system model of TAPE DTMin PROMELA following the subsequent steps.

Modeling Task Mapping and Re-Mapping: Task map-ping and re-mapping is an essential component of TAPE. Wedeveloped structured macros, given in Algorithm 1, for thesefunctionalities in PROMELA so that they can be called fromthe processes running in every core. The PROMELA keywordinline allows us to declare new functions, just like C. Both ofthese functions are called with two parameters, i.e., n, whichidentifies the node for task execution and the TTim, whichrepresents the task time. The higher the value of TTim, themore power units it will consume and it is assumed that only1 power unit is consumed for 1 time unit. As such the freepower units are converted to used ones and the temperatureis incremented by 4oC for each power unit (obtained fromspecific heat capacity of silica). In case of remapping, we haveto find max(sellTn − buyTn ) according to [7] to find a suitabletile for mapping the task. The variable now is incremented inevery execution of Algorithm 1 to keep track of the time.

Variable Initialization Process: An initialization process,given in Algorithm 2, is used to initialize the data types forall the variables used in our model and perform the initial taskmapping. The TAPE algorithm utilizes two normalizing factors,as and ab, to reflect temperature effects on the values of buyTn

and sellTn and four weights ωu,b, ωu,s, ωf,b and ωf,s to reflectthe effects of the variables usedn and freen on the valuesof buybase and sellbase. All these variables are represented asreal numbers in the TAPE algorithm of [7]. However, SPINdoes not support real numbers and hence these variables mustbe discretized as explained in the previous section. The ranges

Algorithm 1 Structured Macrosinline mapping(n,TTim) TTim:Task time {1: nown = nown + TTim;2: freen = freen − (TTim ∗ 1);3: Tmn = Tmn + (TTim ∗ 4);4: usedn = usedn + (TTim ∗ 1); }inline remapping(n,TTim) {1: nown = nown + 1;2: rmp = rmp+ 1;3: dn = sellTn − buyTn; /*difference matrix */4: short mxm;5: if6: ::dn > mxm → mxm = dn; /*finding max(sell-buy)*/7: ::else→skip;8: fi;9: if /*mapping*/10: ::mxm == dn →11: nown = nown + TTim;12: freen = freen − (1 ∗ TTim);13: usedn = usedn + (1 ∗ TTim);14: Tmn = Tmn + (4 ∗ TTim);15: fi; }

of these variables can be found from the TAPE description [7].For example, the values of normalizing factors, as and ab,must be bounded in the interval [0, 1]. Due to the inability toexpress real numbers, we use integers in the interval [1, 9] forassigning values to variables as and ab. In order to nullifythe effect of this discretization on the overall behavior of themodel, we have to divide the equations containing variables asand ab by 10 whenever they are used in the PROMELA model.Similarly, based on the behavior of the TAPE model, integersin the interval [0, 7] are used for the weight variables ωu,s, ωf,b

and ωf,s and the interval [7, 14] for the weight variable ωu,b.Therefore, in order to retain the behavior of the TAPE model,we divide these variables by 7 whenever they are used (Seee.g., Lines 11 and 12 of Algorithm 2).

Algorithm 2 Initialization processPU : Power Unitsinitialization process: init1: select(wus : 1..6);2: select(wfs : 1..6);3: select(wub : 7..13);4: select(wfb : 1..6);5: select(as : 1..9);6: select(ab : 1..9);7: for all nodes n8: freen = PUtotal/(rows ∗ col); For even distribution of PU9: usedn = 0; initially no used PU10: Tmn = To; measured temperature is same as initial temperatureinitially11: sellbasen = (wus · usedn + wfs · freen)/712: buybasen = (wub · usedn + wfb · freen)/713: end formultiple task mapping is done randomly hereinstantiation of all agents and receiving processes are done here

Modeling of Many-Core System and TAPE’s AgentProcess: The grid of cores (or tiles) of TAPE can be modeledas a two dimensional array of distributed nodes such that theTAPE algorithm runs on all these nodes concurrently. Basedon the proposed methodology, we represented the behaviorof these nodes using PROMELA processes and channels. Wedeveloped a generic node model so that it can be repeatedly

used to formally express any grid of arbitrary dimension. Anode is modeled using two main processes, i.e., the receiverprocess, given in algorithm 3, that handles receiving valuesfrom the neighboring nodes and the agent process, given inalgorithm 4, that is mainly responsible for processing thereceived values and then sharing them with the four neighborsof the node. The usage of the separate receiving processensures reliable exchange of information as this way eachnode can receive information at any time irrespective of itsmain agent being busy or not. We have used both sendingand receiving channels that are identified using the symbols !and ? in the PROMELA model, respectively, for every node.

Algorithm 3 Receiving Processn : identification of nodeproctype receiving(n,eastr,westr,northr,southr)1: if receiving2: ::eastr?buyTn, sellTn, time; → max(time, nown + 1)3: ::westr?buyTn, sellTn, time; → max(time, nown + 1)4: ::northr?buyTn, sellTn, time; → max(time, nown + 1)5: ::southr?buyTn, sellTn, time; → max(time, nown + 1)6: :: skip;7: fi;

Discretization of the Temperature Variable: Finally, we also

Algorithm 4 Agent Processn : identification of nodeproctype agent(n,east,west,north,south)1: sellTn = sellbasen + (as · (Tmn − To))/10;2: buyTn = buybasen − (ab · (Tmn − To))/10;3: if4: :: (sellTn − buyTn)− (sellTn[i]− buyTn[i]) > τ →5: if6: ::freen > 0 →nown = nown + 1; freen = freen − 1; freen[i] = freen[i] + 17: ::else→8: nown = nown + 1; usedn = usedn − 1; freen[i] =freen[i] + 1;9: if10: ::tasktime > deadline →remapping(a, b, tasktime); Tmn = Tmn − 4;11: ::else→skip;12: fi13: fi14: ::else → skip;15: fi/*trading results in change of base buy/sell value*/16: if17: ::(buyTn! = lastbuyn) || (sellTn! = lastselln) →18: now = nown + 1; lastbuyn = buyTn; lastselln = sellTn

19: east!buyTn, sellTn, nown;20: west!buyTn, sellTn, nown;21: north!buyTn, sellTn, nown;22: south!buyTn, sellTn, nown;23: :: skip;24: fi;

have to discretize the allowable increments and decrements forthe temperature variable Tm, which is assigned the value ofthe initial temperature T0 = 30◦C at start-up. For this purpose,we assume that the power units consumed for the executiontime of a particular task amounts to 1mJ of energy. Thus,the worst-case temperature change that happens in one powerunit consumption for the task can now be calculated to beapproximately equal to 4◦K using the relationship 1mJ/(CV ),

where C represents the heat capacity equal to 1.547J·cm3·K−1

for Silica and V represents the volume of a core, which can bereasonably assumed to be equal to 1mm x1mm x150um. Thisdiscretization does not affect the verification objectives sincewe are only interested in the stability condition irrespective ofthe transient effects.

Lamport Timestamps for Timing: We increment the valueof nown whenever the node n gives a free power unit to oneof its neighbors as a result of agent negotiation or wheneverthe values sellTn

and buyTnare updated or whenever mapping,

re-mapping, sending or receiving takes place.

Iterative Model Construction and Verification Issues:It is worth mentioning that the above mentioned PROMELAmodel of TAPE was finalized after numerous runs throughthe proposed methodology, i.e., it had to go through deadlockchecks and several stages of simplifications and optimizations.

Issue-1: Deadlocks: We identified a deadlock in our firstPROMELA model of TAPE, which occurred because a singlechannel was used to model both receiving and sending, whichin turn lead to the possibility of missing the status update of amissing neighbor. To avoid this situation, we have used twodifferent processes per node to model sending and receivingchannels. Interestingly, this kind of a critical aspect, whichprevents the system to achieve stability, was not mentionedor caught by the simulation-based analysis of TAPE that isreported in [7]. This point clearly indicates the usefulness of theproposed approach and using formal methods for the verificationof distributed DTM schemes. Likewise, the above mentionedvariable ranges had to be finalized after many simplificationand optimization stages so that the model can be verifiedby the SPIN model checker without encountering the state-space explosion problem. Moreover, we abstracted the DVFSconsiderations from the TAPE model since its presence hasnothing to do with the functional or timing verification and itsremoval results in the simplification of the PROMELA model,which in turn leads to a reduced state-space.

Issue-2: Run-Time Scenario Due to the mathematical natureof the PROMELA models, merely the formal specification ofTAPE in PROMELA allowed us to catch many system issues. –For example the exemplary runtime scenario of Figure 3 in [7]was found to be incorrect as we were not able to recreate itfor our PROMELA model for the same input values.

These issues clearly indicate the shortcomings of simulationand are quite convincing to motivate the usage of formalmethods for the verification of distributed DTM systems.

V. FORMAL VERIFICATION RESULTS

A. Experimental Setup

We use the version 6.1.0 of the SPIN model checker andversion 1.0.3 of ispin along with the WINDOWS 7 ProfessionalOS running on i7-2600 processor, 3.4 GHz(8 CPUs) with 16GB memory. The verification is done for a 3x3 grid of nodes(cores) with all of them running the processes and channelsdescribed in the previous section. The complete model contains18 processes and 350 lines of code. The SPIN utility BITSTATEis used for verification purposes since it uses 2808 Mb of spacewhile allowing to work with up to 4.8 · 106 states.

0

2

4

6

8

10

0

20

40

60

80

100

1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60

Events to Stability

Tasks Re-Mapped

Nu

mb

er

of

Eve

nts

to

Re

ach

Sta

bili

ty

Nu

mb

er

of

Task

R

e-M

app

ed

Number of Tasks

Fig. 3: Effect of Tasks on Events to Stability.

B. Verification Results

Functional Verification Results: The most interestingfunctional characteristic to be verified in the context of TAPE isto ensure that the agent trading is always able to achieve a stablepower budget distribution. For instance, it needs to be shownthat no circular trading loops emerge where power budgetis continuously traded preventing the system from stabilizing.Another possibility is that localized minima form which act asa barrier that prevents power budget from propagating. As aresult, cores on one side of the barrier would no longer be ableto obtain power budget even if it were available globally, andnew tasks would be mapped to the other side of the barrierwhere power budget has accumulated. If such a scenario ispossible, it would result in high temperatures and frequent re-mapping inside the region with the power budget not allowingthe system to stabilize even though a global stable configurationwould be possible. The non-occurrence of such instabilitiescan be ensured by verifying that “Eventually the sell-buy valuebetween any two adjacent tiles would become very small". Wehave to verify 12 such properties so that all possible node pairsof a 3x3 grid are covered. For example, this property (p0001)can be expressed for nodes 00 and 01 as:

[](<>((sell_T[0].vector[0]-buy_T[0].vector[0])

-(sell_T[0].vector[1] - buy_T[0].vector[1])<3))

In a similar way, we expressed and verified the rest of the11 properties as well. All the 12 properties hold for TAPE,which gaurentees its functional correctness, and the verificationstatistics of these properties is given in Table I. No unreachablecode was detected during the analysis, which ensures that theverification was exhaustive and complete.

Timing Verification Results: For timing verification, weformally executed the PROMELA model of the previous sectionand observed the effect of number of tasks and the values ofvariables as and ab, on the number of events and temperature.The values of the global list now are observed for acquiring theinformation about the number of events and the overall resultsof the timing verification are summarized in Graphs 1-2 andFig. 3-4. All these results have been observed for Total Powerunits: 128, Tasks: 10, ωu,s = 2/7, ωf,b = 2/7, ωf,s = 2/7 andωu,b = 8/7.

0.2 0.4 0.6 0.8101102103104105

Stable

Unstable

as

Eve

nts

tore

ach

Stab

ility

Graph 1: Effect of as

0.2 0.4 0.6 0.8101102103104105

Stable

Unstable

ab

Eve

nts

tore

ach

Stab

ility

Graph 2: Effect of ab

TABLE I: Statistics of all the 12 properties verified where p0001 is related to node (0,0)–(0,1) and so on

Transitions States stored Memory Usage(MB) Verification Time(sec)Horizontal Relations

Property 1 (p0001) 25244478 4786929 2808.8 80.9Property 2 (p0102) 25295692 4787223 2808.808 81.3Property 3 (p1011) 24991059 4755543 2803.658 83.5Property 4 (p1112) 25193755 4772741 2816.247 82.7Property 5 (p2021) 25244421 4786926 2808.808 84.1Property 6 (p2122) 25244421 4786926 2808.808 83.9

Vertical RelationsProperty 7 (p0010) 24672155 4721204 2564.477 76Property 8 (p1020) 24947360 4751100 2564.477 76.9Property 9 (p0111) 25055780 4760200 2564.477 76.9Property 10 (p1121) 25097601 4764453 2564.477 77.3Property 11 (p0212) 24982388 4749616 2564.477 76.8Property 12 (p1222) 25114849 4759582 2564.477 77.3

0

20

40

60

80

100

1 2 3 4 5 10 15 20 25 30 35 40 45 50 55 60

Tm(max)

Tm(max)

T m(m

ax)

Number of Tasks

Fig. 4: Effect of Tasks on Maximum Temperature.

The events to stability in the graphs refers to the timerequired to reach a stable state, i.e., a state where power isevenly distributed and no redundant trading of power units takeplace – only due to elevated temperatures. The value of ab iskept constant at 0.2 and the value of as is varied from 0.1 to0.9 in Graph 1, whereas, as is kept constant at 0.2 in Graph2 and ab is varied from 0.1 to 0.9. From both the graphs, itcan be clearly observed that the TAPE algorithm reaches astable state only when as + ab < 1. The same behavior wasobserved for other values of as and ab as well. This is a keydesign parameter and was not reported in the simulation basedanalysis of TAPE in [7].

Fig. 3 shows that the number of events to reach stabilityincrease almost proportionally with an increase in the number oftasks until the first 30 tasks. As the tasks are increased beyondthis threshold, the event executions appear to be happening inparallel which in turn reduces the number of events required toreach stability. Moreover, it is also observed from Fig. 3 thatthe task remapping happens only after the number of tasks arebeyond a certain threshold, i.e., when the number of free powerunits of a tile differs considerably with its neighbor or almostall of its free power units are consumed. This highlights thefact that there is some room for improvement in the remappingmechanism of TAPE. Fig. 4 shows that the maximum measuredtemperature, Tm, also increases with the number of tasks.

It is important to note that the number of events requiredto reach stability has no relationship with the physical clockof the system and it just provides a relative indication for theamount of time required to reach stability. The distinguishingcharacteristic of the analysis presented in this section is itsexhaustive nature, which cannot be attained by the traditionalsimulation due to the large number of possibilities. Moreover,the verification process is completely automatic and the humaninteraction is only required for debugging purposes.

C. Discussion

Generalization: The above methodology is general enoughto be used to formally verify both functional and timingproperties of any distributed DTM system since these DTMschemes can be described by concurrent communicatingprocesses and thus their behaviors can be captured by thePROMELA language. The main challenge in the modelingphase exists in assigning appropriate data-types to the variablesinvolved and the proposed methodology provides a step-wiseapproach to address this issue. Moreover, we are alwaysinterested in verifying deadlock-free behaviors, functional andtiming properties and the proposed methodology caters for allthese three verification aspects using the SPIN model checkerand Lamport timestamps algorithm.

Limitations: The main limitation of using model checkingfor the formal verification of DTM schemes is the state-spaceexplosion problem. We proposed to encounter this problemby shrinking the size of the system model by lowering thenumber of possible values of variables, or in other words,discretizing them. The main compromise being made here ison the exhaustiveness of the analysis in terms of possible inputvalues to the DTM schemes. However, as far as the functionalverification of the distributed DTM schemes is concerned, thevariable discretization does not impact our main objective,which is to observe the given schemes under all combinations.

VI. CONCLUSIONS

The paper presents a formal verification methodology fordistributed DTM systems. The proposed method mainly utilizesthe SPIN model checker and Lamport timestamps algorithmto verify both functional and timing properties. To the best ofour knowledge, this is the first formal verification approach fordistributed DTM systems. For illustration purposes, the paperpresents the successful formal verification of TAPE, which isa recently proposed agent-based DTM. This case study clearlyindicates the applicability of the proposed methods to verifyother prominent distributed DTM approaches.

ACKNOWLEDGMENTS

This work was supported in parts by the German ResearchFoundation (DFG) as part of the priority program "DependableEmbedded Systems" (SPP 1500 - spp1500.itec.kit.edu).

REFERENCES

[1] SPIN V2 Update 2.6 (16 July 1995). http://spinroot.com/spin/Doc/V2.Updates, 2013.

[2] J. Abrial. Faultless systems: Yes we can! IEEE Computer, 42(9):30–36,2009.

[3] C. Baier and J.P. Katoen. Principles of Model Checking. MIT Press,2008.

[4] S. Borkar. Thousand core chips: a technology perspective. In DesignAutomation Conference, pages 746–749. ACM, 2007.

[5] J. Donald and M. Martonosi. Techniques for multicore thermalmanagement: Classification and new exploration. In Symposium onComputer Architecture, pages 78–88. IEEE Computer Society, 2006.

[6] D. Dunn. Intel delays montecito in roadmap shakeup. EE Times,Manufacturing/Packaging, Oct. 27, 2005.

[7] T. Ebi, M. Faruque, and J. Henkel. Tape: Thermal-aware agent-based power econom multi/many-core architectures. In InternationalConference on Computer-Aided Design (ICCAD), pages 302–309, 2009.

[8] T. Ebi, D. Kramer, W. Karl, and J. Henkel. Economic learning forthermal-aware power budgeting in many-core architectures. In IEEEInternational Conference on Hardware-Software Codesign and SystemSynthesis (CODES+ISSS’11), pages 189–196, 2011.

[9] Y. Ge, P. Malani, and Q. Qiu. Distributed task migration for thermalmanagement in many-core systems. In Design Automation Conference,pages 579 –584, 2010.

[10] G.J. Holzmann. The model checker SPIN. IEEE Transactions onSoftware Engineering, 23(5):279–295, 1997.

[11] ITRS. http://www.itrs.net, 2013.[12] M. Kadin, S. Reda, and A. Uht. Central vs. distributed dynamic thermal

management for multi-core processors: which one is better? In GreatLakes symposium on VLSI, pages 137–140, 2009.

[13] J.A. Kumar and S. Vasudevan. Verifying dynamic power managementschemes using statistical model checking. In Asia and South PacificDesign Automation Conference, pages 579–584. IEEE, 2012.

[14] L. Lamport. Time, clocks, and the ordering of events in a distributedsystem. Commun. ACM, 21(7):558–565, 1978.

[15] A. Lungu, P. Bose, D.J. Sorin, S. German, and G. Janssen. Multicorepower management: Ensuring robustness via early-stage formal verifi-cation. In Formal Methods and Models for Co-Design, pages 78 –87,2009.

[16] G. Norman, D. Parker, M. Kwiatkowska, S. Shukla, and R. Gupta. Usingprobabilistic model checking for dynamic power management. FormalAspects of Computing, Springer-Verlag, 17(2):160–176, 2005.

[17] M. D. Powell, M. Gomaa, and T. N. Vijaykumar. Heat-and-run:Leveraging smt and cmp to manage power density through the system.In ASPLOS, pages 260–270, 2004.

[18] S.K. Shukla and R.K. Gupta. A model checking approach to evaluatingsystem level dynamic power management policies for embedded systems.In High-Level Design Validation and Test Workshop, pages 53–57. IEEEComputer Society, 2001.

[19] I. Yeo, C. C. Liu, and E. J. Kim. Predictive dynamic thermal managementfor multicore systems. In Design Automation Conference (DAC), pages734–739, 2008.

APPENDIX

Algorithm 5 Thermal-aware Agent-based Power Economy [7]sellbasen : Base sell value of a tile n at time tbuybasen : Base buy value of a tile n at time tTn: Temperature of a tile n at time tsellTn : Sell value of a tile n at a temperature TnbuyTn : Buy value of a tile n at a temperature TnNn: Set of all the neighboring tiles of tile nlastbuy, lastsell: Last buy/sell values of a tile n sent to all i ∈ Nbuy[N ], sell[N ]: List of buy/sell values of neighboring tiles stored in nfreen: Free power units of tile nusedn: Power units used for running tasks on tile ntj : Tasks running on tile n at time tτn: sell threshold of tile n1: loop2: for all tiles n in parallel do3: at every time interval ∆nt do // Calculate base sell value4: sellbasen ← (wu,s · usedn + wf,s · freen)5: buybasen ← (wu,b ·usedn−wf,b ·freen) // The temperature

increase may happen due to change in PE activity. Modify buy/sellvalue

6: sellTn ← sellbasen + as·(Tmn − To) +7: buyTn ← buybasen - ab·(Tmn − To)8: if ∃i ∈ Nn : ((sellTn − buyTn ) − (sell[i] − buy[i]) > τn

then9: if any free power units are left then

10: decrement freen11: else12: apply DVFS on n to get more free power units13: decrement usedn14: if the task does not meet the given deadline as DVFS is

used then15: (re-)mapping needs to be invoked16: else17: graceful performance degradation if allowed18: end if19: end if20: increment freei21: end if22: if buyTn 6= lastbuy or sellTn 6= lastsell then23: send buyTn to all i ∈ N24: send sellTn to all i ∈ N25: lastbuy ← buyTn

26: lastsell ← sellTn

27: end if // This procedure will propagate until a stable state isreached.

28: end at29: if received updated buy/sell values from any l ∈ Nn then30: update buy[l], sell[l]31: end if32: if new task mapped to n requiring k power units then33: freen ← freen − k34: apply DVFS to PE on tile n35: usedn ← usedn + k36: end if37: end for38: end loop