2030 IEEE TRANSACTIONS ON PARALLEL AND ...problem of joint geographical load balancing, delay-tolerant workload scheduling, and thermal storage management for green energy integration

Energy and Network Aware WorkloadManagement for Sustainable Data Centers

with Thermal StorageYuanxiong Guo, Student Member, IEEE, Yanmin Gong, Student Member, IEEE,

Yuguang Fang, Fellow, IEEE, Pramod P. Khargonekar, Fellow, IEEE, andXiaojun Geng, Member, IEEE

Abstract—Reducing the carbon footprint of data centers is becoming a primary goal of large IT companies. Unlike traditional energysources, renewable energy sources are usually intermittent and unpredictable. How to better utilize the green energy from theserenewable sources in data centers is a challenging problem. In this paper, we exploit the opportunities offered by geographical loadbalancing, opportunistic scheduling of delay-tolerant workloads, and thermal storagemanagement in data centers to facilitate green energyintegration and reduce the cost of brown energy usage. Moreover, bandwidth cost variations between users and data centers areconsidered. Specifically, this problem is first formulated as a stochastic program, and then, an online control algorithm based on theLyapunov optimization technique, called Stochastic Cost Minimization Algorithm (SCMA), is proposed to solve it. The algorithmcan enable an explicit trade-off between cost saving and workload delay. Numerical results based on real-world traces illustratethe effectiveness of SCMA in practice.

Index Terms—Data center, energy management, thermal storage, load scheduling, Lyapunov optimization

Ç

1 INTRODUCTION

TO provide Internet-scale services such as social net-working and web search with low latency and high

reliability, Internet-service companies usually build multi-ple data centers distributed across different geographicallocations. These data centers consume large amounts ofelectricity for powering both their IT equipments andcooling infrastructures. According to [1], the electric energyconsumption of data centers for Internet applicationsaccounts for 1.3 percent of the worldwide electricity usagein 2010 and this fraction is expected to increase to 8 percentby 2020. Therefore, intensive efforts have been made byInternet-service companies to reduce the electricity cost intheir data centers.

Meanwhile, Internet-service companies are increasinglyinterested in becoming ‘‘sustainable’’, which requires themto reduce the environmental impact (i.e., carbon footprint)besides the financial impact (i.e., electricity cost) of theirdata centers. As shown in [2], two thirds of the worldwideelectricity drawn from the utility grid is generated byfossil-fuel generators, such as coal, or gas plants, which

emit much more carbon than renewable generators such aswind turbines and solar panels. With the decreasing cost ofbuilding renewable generators, they are becoming increas-ingly attractive options for powering data centers, espe-cially when the renewable energy is supported bygovernment incentives.

However, unlike the traditional brown energy drawnfrom the utility grid, green energy from renewable sources,especially wind and solar, is intermittent and uncontrollable,which presents a great challenge for data centers to effectivelyutilize them. The challenge is, in essence, the difficulty ininstantaneously balancing of energy supply and demand.Large-scale electric energy storage, mainly batteries, canresolve this difficulty, but it is still prohibitively expensive.

To help integrate green energy into data centers,geographical load balancing [3], [4] has been proposed toutilize the agility of geographically distributed data centersby directing more user requests to places where renewableenergy is abundant. Although geographical load balancingis useful, there are two more opportunities that can beexploited to further facilitate renewable energy integrationin data centers. One observation is that data centers usuallysupport a wide range of IT workloads, including bothdelay-sensitive, interactive applications such as webbrowsing and searching, and delay-tolerant, batch applica-tions such as scientific computation and massively paralleland data intensive computational jobs. The interactiveworkload differs from the batch workload in the followingtwo aspects. First, the computational requirement of theinteractive workload is usually small, while the batchworkload requires much larger computational capability.Second, while the performance metric appropriate to theinteractive workload is the response time, for the batch

. Y. Guo, Y. Gong, Y. Fang, and P.P. Khargonekar are with the Department ofElectrical and Computer Engineering, University of Florida, Gainesville,FL 32611 USA. E-mail: {guoyuanxiong, ymgong, ppk}@ufl.edu; [email protected].

. X. Geng is with the Department of Electrical and Computer Engineering,University of West Florida, Pensacola, FL 32514 USA. E-mail:[email protected].

Manuscript received 19 July 2013; revised 10 Oct. 2013; accepted 18 Oct.2013. Date of publication 3 Nov. 2013; date of current version 16 July 2014.Recommended for acceptance by V. Misic.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TPDS.2013.278

1045-9219 � 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 25, NO. 8, AUGUST 20142030

workload, it is the total throughput within some timeperiod. The delay-tolerant property of batch workloads canbe exploited to increase the renewable energy utilization bydelaying their services to periods when renewable sourcesare abundant without exceeding their execution deadlines.

Another observation is that a large portion of the powerconsumption in a data center comes from the coolinginfrastructure. Although large-scale electric energy stor-age, such as batteries, is very expensive, thermal storage ismuch cheaper, and can be leveraged to reduce the coolingenergy cost. In fact, Apple has already deployed a chilledwater storage system as the thermal storage facility in itsgreen data center in Maiden, NC [5]. With the time-varyingproperties of wholesale electricity price and renewableenergy generation, thermal storage can store some greenenergy from renewable generators or cheap brown energyfrom the utility grid first. Later, when the electricity price ishigh or the green energy is unavailable, the stored energycan be released to help cool the data center, therefore,reducing the electricity bill.

With the above observations as context, we explore theproblem of joint geographical load balancing, delay-tolerantworkload scheduling, and thermal storage management forgreen energy integration in geographically distributed datacenters. In additional to the brown energy cost, we also takeinto account the bandwidth cost between cloud users and datacenters. The objective is to minimize the total operating cost ofserving delay-tolerant workloads. To tackle the randomnessin renewable generations, workload arrivals, and electricityprices, we formulate the problem as a stochastic programand propose an efficient online algorithm, called StochasticCost Minimization Algorithm (SCMA), with provableperformance guarantee based on the Lyapunov optimiza-tion framework [6]. In summary, the contributions of ourwork are as follows:

. By taking into account the delay-tolerant workloadsand thermal storage, we formulate a stochasticoptimization problem to minimize the total energyplus bandwidth cost of geographically distributeddata centers with renewable generation.

. We propose an online distributed control algorithmSCMA to solve the problem without the require-ments of knowing the detailed statistics of under-lying randomness.

. Our proposed algorithm enables an explicit trade-off between workload delay and cost saving, whichcan be flexibly adjusted by a control parameter V ,making it an attractive control policy for data centeroperators with different applications.

. Through extensive numerical evaluations usingreal-world traces of renewable generation, workloadarrival, and wholesale electricity price, we demon-strate the effectiveness of SCMA.

The remainder of this paper is organized as follows.Section 2 reviews some related work. In Section 5, modelson workloads, renewable generation, thermal storage, andtotal operating cost are first presented and then, astochastic optimization problem is formulated. We proposean algorithm called SCMA to solve it in Section 4. Theanalytical performance results of SCMA are described in

Section 5. We present the numerical evaluation resultsbased on real-world traces in Section 6. Finally, Section 7concludes the paper.

2 RELATED WORK

2.1 Renewable Energy Usage in Data CentersRenewable-powered data centers are receiving more andmore attention both in industry [5], [7] and in academia [3],[4], [8], [9]. Previous studies [3], [4] explore the feasibilityand benefits of using geographical load balancing for delay-sensitive interactive workloads to facilitate the inte-gration of renewable sources into data centers. Schedulingof delay-tolerant batch workload and energy storage tohelp integrate renewable sources into a data center withon-site renewable generation is discussed in [8]. Systemimplementation issues with renewable energy-aware batchworkload scheduler is discussed in [9] and prototypes arebuilt to show the effectiveness of these job schedulers.However, all the aforementioned papers either considera single data center, a single class of application, noenergy/thermal storage facility, only delay-sensitive inter-active workloads, or assume perfect future information. Incontrast, our work jointly manages delay-tolerant work-loads with thermal storage facilities in geographicallydistributed data centers having on-site renewable genera-tions without future information.

2.2 Electric/Thermal Storage in Data CentersUse of electric energy storage devices such as uninterrup-tible power supply (UPS) units to help reduce electricitycost is considered in [10], [11], [12]. However, batteries suchas UPS units are quite expensive and cannot be overusedsince frequently charging and discharging severely impactstheir lifetimes. On the other hand, thermal storage is muchcheaper and can be utilized to reduce the cooling cost indata centers as shown in [13]. Therefore, in our work, weutilize the thermal energy storage to help reduce the coolingcost of data centers rather than assume the electric energystorage unit as in previous work [10], [11], [12].

2.3 Energy Cost Minimization in Data CentersReducing the electricity cost of Internet data centers hasbeen the focus of a lot of research work in the past decade(see the most recent ones [12], [14], [15], [16], [17] andreferences therein). One direction is to reduce the amountof energy usage in data centers. Two main approaches existalong this direction: achieving power-proportionality andlowering energy overhead. Power-proportionality meansconsuming power directly proportional to the utilizationlevel, which can be achieved by dynamic voltage/frequencyscaling (DVFS), or dynamic capacity provision (DCP). Theenergy overhead of a data center is measured by the powerusage effectiveness (PUE) metric, which denotes the ratio ofthe total facility power consumption to the IT equipmentpower consumption. Various schemes, such as advancedcooling methods and direct current power infrastructure,have been designed to lower the PUE. Another direction isto use geographical load balancing to exploit the diversityof electricity prices in multiple data centers, where moreinteractive requests would be routed into data centers with

GUO ET AL.: ENERGY AND NETWORK AWARE WORKLOAD MANAGEMENT FOR SUSTAINABLE DATA CENTERS 2031

lower electricity prices or more renewable energy. However,most of the previous efforts mainly focus on the interactive,delay-sensitive workloads. Delay-tolerant workload schedul-ing is also considered in [16], [17], [18], but these papers donot consider on-site renewable energy generation or thermalstorage. In [19], we only consider the energy cost minimizationin a data center with renewable energy generation andthermal storage. However, neither the bandwidth cost northe thermal storage cost is considered.

2.4 Bandwidth Cost Minimization in Data CentersTraffic engineering in data centers for efficient networkutilization has been discussed in [20], [21], [22]. These papersmainly focus on VM placement or migration in data centersto reduce bandwidth cost while ignoring many aspects ofenergy cost. These studies are complementary to our work inthe sense that the network-aware job placement algorithmswithin a single data center can be exploited after our algorithmdetermines the jobs to be routed to each data center.

3 MODELING AND OPTIMIZATION

We consider a cloud service provider (CSP) havingmultiple geographically distributed data centers, eachwith on-site renewable generators and thermal storage.The typical cloud network architecture of a CSP is depictedin Fig. 1, in which there are several front-end proxies nearthe clients and multiple back-end remote data centers in thecloud. Assume that there are M proxies and each of themis responsible for a geographically concentrated source ofrequests such as a city. The proxy directs user requests to Ndata centers of the CSP in the cloud. Due to the spatialdiversity, traffic between different pairs of proxy and datacenter goes through different ISPs and therefore, incursdifferent bandwidth costs. The sustainable data center weconsider in this paper is illustrated in Fig. 2, which isexplained in detail as follows. We consider a discrete-timesystem with time denoted by t ¼ 0; 1; 2; . . ..

3.1 The Workload ModelThere are many different workloads in data centers. Ingeneral, they can be divided into the following twocategories: delay-sensitive interactive workload and delay-tolerant batch workload [8]. Delay-sensitive interactiveworkloads such as web services usually process real-timeuser requests, which have to be completed within a certaintime, i.e., there is a maximum response time. Some batchworkloads such as scientific applications, simulations, orMapReduce jobs [23] are often delay-tolerant in the sensethat they can be scheduled to run at any time as long as thejobs are completed before the deadline, i.e., there is amaximum completion time. Since interactive workloadshave higher priority, they are usually provisioned first. Inthis paper, we focus on computation-intensive batch work-load management, assuming that the management ofinteractive workloads has been determined by previousschemes [3], [24].

Consider C types of jobs or service requests in the delay-tolerant workloads. Each type may correspond to a specificapplication. Assume that all jobs are computation-intensive, and the CPU resource is the bottleneck resource.That is, a job is executed whenever the CPU resource isallocated to it. A job is represented by a tuple: ðc; dc; ncÞ,where c denotes the application type, dc denotes compu-tation demand (i.e., job length) in terms of the processorcycles, and nc denotes the communication demand in termsof the transmitted data size between the cloud and theclient. We assume that jobs of different types have differentIT resource requirements (e.g., CPU, memory, storage, andnetwork) and jobs of the same type have the same ITresource requirements.

A job or service request first arrives at the front-endproxy j. The proxy is near the clients and acts as a workloadrouter. The proxy would decide which back-end datacenter the job request should be routed to for processing.We assume no data buffering at the proxy so that whenevera request arrives at the proxy, it would be routed to a datacenter for processing immediately. Denote the number oftype c jobs arriving at proxy j in time t as Wc

j ðtÞ. The jobarrival rate vector at time t is denoted as WðtÞ ¼ ðWc

j ðtÞ; 8c; jÞand the time-average rate of such an arrival vector is de-noted as ! ¼ EfWðtÞg. We assume that the total arrival

Fig. 1. Typical cloud network architecture of a CSP.

Fig. 2. Block diagram of a sustainable data center.


rate of type c jobs is bounded by a finite positive constantWc

max. That is,

XJj¼1

Wcj ðtÞ �Wc

max; 8c; t: (1)

We use �cijðtÞ to denote the number of type c jobs that isrouted from proxy j to data center i in time t, and useLcjðtÞ ¼ ð�cijðtÞ; 8iÞ to denote the routing vector for type cjobs at proxy j. In every time period t, LcjðtÞmust draw fromsome feasible routing set Lc

jðtÞ, which includes, but is notlimited to, the following constraints:

XNi¼1

�cijðtÞ ¼Wcj ðtÞ; 8c; j; t (2)

�cijðtÞ � 0; 8c; i; j; t: (3)

Additional constraints can be added into the feasible setLcjðtÞ to model other practical considerations. For example,

if jobs of application c from proxy j can only be routed intoa set of data centers Icj due to security concern, then wehave �cijðtÞ ¼ 0; 8i 62 Icj . If a job contains several tasks whichneed to communicate with each other during the proces-sing, we may need to place the whole job inside one datacenter to reduce the inter-DC traffic, which is much costlierthan the intra-DC traffic. Then, �cijðtÞ should be an integervalue. Other practical constraints can be formulated intothe set Lc

jðtÞ similarly.Denote the queue length of type c jobs at the back-end

data center DCi as Qci ðtÞ. Then, we have the following

queue dynamics:

Qciðtþ 1Þ ¼ max Qc

i ðtÞ � xciðtÞ; 0� �

þXMj¼1

�cijðtÞ; (4)

where xci ðtÞ is the number of type c jobs processed at datacenter i in time t. For each data center i, denote theprocessing speed of the server as �i and the total number ofservers for serving delay-tolerant workloads as ITi. Sincethe processed workload cannot exceed the maximumavailable computing resources, we have

XCc¼1

xci ðtÞdc � ITi�i; 8i; t: (5)

Note that in the formulation above, we implicitlyassume that the jobs can be perfectly parallelized and aretolerant to interruption during running time. The jobs weconsider in this work are the same as the jobs that can besupported by the Amazon EC2 spot instances [25], whichare time-flexible and interruption-tolerant. We need tocontrol the system so that the queues in the system arestabilized according to the following definition:

Q ¼D lim supT!1

1

T

XT�1

t¼0

XNi¼1

XCc¼1

E Qci ðtÞ

� �G 1: (6)

3.2 The Renewable Generation ModelThere are several possible approaches for Internet-servicecompanies to utilize renewable energy in their data centers

[26], where power purchasing agreement (PPA) and on-siterenewable generation are two commonly used methodsin industry now. In the first approach, the data centeroperator negotiates a long-term PPA with a renewableenergy producer, and directly purchases a certain amount ofthe green energy generated by the producer at a negotiatedprice. Renewable energy certificates (RECs) are kept by thedata center operator as the proof of its green energy usage. Forexample, Google has contracted to purchase 114 MW of windpower for 20 years from a wind project in Iowa to power itsdata center there [7]. The second approach is to build on-siterenewable generators near data centers, which can reducethe transmission and distribution losses. For example,Apple is building the nation’s largest end user-ownedsolar array (40 MW) and also, the largest nonutility fuelcell installation (5 MW) in the US at its new data centerin Maiden, NC [5]. These on-site renewable generators willprovide over 60 percent of the clean power it needs. In thispaper, we focus on the second approach because it has amore direct impact on ‘‘greening’’ data centers.

Denote the amount of on-site renewable energy generatedat data center DCi during period t as riðtÞ. Since renewableenergy sources, mainly solar or wind, are highly intermit-tent, time-varying, and uncontrollable, they may vary a loteven within one period (e.g., 10 mins) in our scenario. Inpractice, as observed in [10], data centers usually haveexcess energy storage capability in UPS units, which canprovide such a ‘‘smoothing’’ function. Under this assump-tion, the renewable generation can be regarded as beingconstant during one time period.

3.3 The Thermal Storage ModelAs explained in [13], there are basically two kinds ofthermal storage technologies used in data centers. One isthe inherent thermal masses in a data center such as thecold air and the raised metal floor. They can be over-cooledto a lower temperature by the CRAC system first, andabsorb heat later as a cooling unit. The other is thededicated thermal storage system. Thermal energy storagesystems commonly use chilled liquid or ice to act as athermal battery, enabling a data center operator to run airconditioners at night (when rates are lower) and during theday, pump the chilled liquid around the facility for cooling.While there is no extra capital cost for the first approach, itscapacity is usually limited and therefore, it is only suitablefor short-term storage. In this paper, we consider thesecond approach, where each data center has a chilledliquid/ice storage system besides the CRAC coolingsystem. Note that our thermal storage-based approach isorthogonal and supplementary to other approaches, suchas DC power distribution and seawater cooling, used forreducing the cooling cost of data centers.

For each data center i, denote by Smaxi the capacity of the

thermal storage, by SiðtÞ the energy level at period t, bysþi ðtÞ the energy stored (i.e., charged) into the thermalstorage system in period t, and by s�i ðtÞ the energy released(i.e., discharged) from the thermal storage system in periodt. In practice, there is conversion loss during the energyconversion process. Without loss of generality, we assumethe conversion loss only happens in the charging processand denote the round-trip efficiency of the thermal storage


system at data center i as �i G 1. Then, SiðtÞ would denotethe usable energy in the thermal storage and has thefollowing dynamics at data center i:

Siðtþ 1Þ ¼ SiðtÞ þ �isþi ðtÞ � s�i ðtÞ: (7)

Also, each thermal storage usually has an upper bound onthe charge rate, denoted by sþi;max, and an upper bound onthe discharge rate, denoted by s�i;max. That is,

0 � sþi ðtÞ � sþi;max; (8)

0 � s�i ðtÞ � s�i;max: (9)

We also define sþmax ¼D

maxi sþi;max as the maximum charge

rate of thermal storage systems at all data centers.Within one control period, the thermal storage can be

either charged or discharged, but not both [10]. That is,

sþi ðtÞ 9 0) s�i ðtÞ ¼ 0; s�i ðtÞ 9 0) sþi ðtÞ ¼ 0: (10)

However, we will temporarily ignore this constraint anddecide the optimal charge/discharge control actions. Later,we will construct the control decisions that can meet thatconstraint without performance degradation.

For each time period, we need to ensure that the thermalenergy level in data center i always satisfies the following:

0 � SiðtÞ � Smaxi : (11)

Note that some thermal storage systems may have a nonzerominimum energy level requirement to protect the lifetime oftheir system. Without loss of generality, we assume that theminimum energy level is zero while Smax

i denotes the usablethermal storage capacity. The initial energy level in data centeri is assumed to be Sið0Þ 2 ½0; Smax

i �.Since the excessive usage of thermal storage would

impact its lifetime and reliability, as with [27], the loss ofthe thermal storage value is modeled as a cost which isproportional to the recharged energy with a factor �i. Thatis, the operating cost of using thermal storage at data centeri in period t is �is

þi ðtÞ.

3.4 The Cost ModelBesides the cost of using thermal storage systems as describedbefore, there are two other parts of the total operating cost: oneis the energy cost used to serve the workload in data centersand the other is the bandwidth cost between the clients nearthe proxies and the data centers in the cloud.

To incentivize the usage of green energy from renewablegenerators, we assume that the marginal cost of renewablegeneration is zero so that the data centers should utilize itas much as possible. The cost of traditional brown energydrawn from the utility grid depends on the wholesaleelectricity market and is both spatially and temporallyvarying. Denote by piðtÞ the brown energy price boughtfrom the wholesale electricity market at data center DCi inperiod t. It is both time-varying and location-dependent.We assume that 0 � piðtÞ � pmax

i for all periods t andpmaxi 9 �i=�i.

1

The power consumption of a server in data center i canbe approximated to be linearly related to the average CPUutilization as follows [28]:

ð1� �ÞPidlei þ �Pbusy

i (12)

where Pidlei is the power consumption when the server is in

idle state, � 2 ½0; 1� is the average CPU utilization level, andPbusyi is the power consumption when the server is busy.

Therefore, given the service rate xci ðtÞ for type c jobs atperiod t and the maximum available active server numbersITi, the IT power consumption for data center i is

EiðtÞ ¼ ITiPidlei þ

Pc x

ci ðtÞdc�i

P busyi � Pidle

i

� �: (13)

Denote by fiðtÞ the corresponding cooling energy usagein data center i during time t. Since we focus on the thermalstorage in this paper, we assume that the discharged powerfrom the thermal storage cannot be greater than the coolingdemand2, i.e.,

s�i ðtÞ � fiðtÞ � 0: (14)

In practice, fiðtÞ may be a convex function, depending onthe specific cooling infrastructures such as CRAC and aircooling systems [8]. For simplicity of analysis, we assumethat fiðtÞ is a linear function of the total IT powerconsumption in the following form:

fiðtÞ ¼ �iEiðtÞ; (15)

where �i is a factor to represent the power usage efficiencyof data center. On average, �i is around 1 for the data centerindustry [7]. That is, for every watt of IT power, anadditional watt is consumed to cool and distribute powerto the IT equipment. Although intensive research has beendone to reduce the power usage efficiency of data centers,energy storage has appeared as an attractive mechanismquite recently [10], [12]. Note that our framework is quitegeneral and can incorporate more practical cooling modelssuch as [8]. With the above models, the energy cost plus thethermal storage operating cost of data center i in period t isas follows:

EiðtÞ ¼ piðtÞ ð1þ �iÞEiðtÞ þ sþi ðtÞ � s�i ðtÞ � riðtÞ� �þ

þ �isþi ðtÞ: (16)

Meanwhile, there is bandwidth cost involved for thecommunication between the jobs routed into the datacenter and the client near the proxy. In this paper, we usethe following linear bandwidth cost model to represent thebandwidth cost between the clients and the cloud:

BijðtÞ ¼XCc¼1

bij�cijðtÞnc; (17)

where bij is the bandwidth cost coefficient between proxy jand data center i. Note that nc is the communication

1. Note that this assumption represents that there is opportunity toutilize storage for cost reducing.

2. Note that for a electric energy storage, the discharged power canalso be used to power the servers, therefore, eliminating the constraint(14). Our framework and the proposed techniques are still applicable tothe case of electric energy storage systems with minor modification.


demand between the cloud and the client. Different pairsof proxy and data center have different bandwidth cost.More practical bandwidth charging model based on 95-thpercentile bandwidth usage may be modeled similarly andwould be our future investigation. We define bmax ¼D maxij bijas the maximum bandwidth cost coefficient between anypair of proxy and data center. The total operating cost ofserving delay-tolerant workloads for a CSP in time period tisPN

i¼1 EiðtÞ þPN

i¼1

PMj¼1 BijðtÞ.

3.5 Problem FormulationIn this paper, we are interested in minimizing the time-average total operating cost for serving the delay-tolerantworkloads over a large time horizon. Therefore, the controlproblem can be stated as follows: for the dynamic systemdefined by (4) and (7), design a control strategy which, giventhe past and the present random renewable supplies,workload arrivals, and electricity prices, chooses the work-load routing decisions L, the thermal storage decisions sþ

and s�, and the IT resource allocation decisions x such thatthe time-average total operating cost for serving delay-tolerant workloads is minimized. It can be formulated as thefollowing stochastic optimization:

minL;x;

sþ ;s�

: g ¼ lim supT!1

1

T

XT�1

t¼0

EXNi¼1

EiðtÞ þXNi¼1

XMj¼1

BijðtÞ( )

; (18a)

s:t:

xci ðtÞ � 0;XCc¼1

xci ðtÞdc � ITi�i; 8c; i; t (18b)

Siðtþ 1Þ ¼ SiðtÞ þ �isþi ðtÞ � s�i ðtÞ; 8i; t (18c)

0 � SiðtÞ � Smaxi ; 8i; t (18d)

0 � sþi ðtÞ � sþi;max; 8i; t (18e)

0 � s�i ðtÞ � s�i;max; 8i; t (18f)

s�i ðtÞ � fiðtÞ � 0; 8i; t (18g)

�cijðtÞ; 8i� �

2 LcjðtÞ; 8c; j; t (18h)

Q G 1: (18i)

Here (18b) means that the total allocated IT resources cannotexceed the IT capacity. (18h) denotes that the workloadadmission and routing vectors should be within the feasibleset, which depends on the real application. (18i) ensures thatthe average total queue length for buffering delay-tolerantjobs is finite so that the dynamic system is stable.

One challenge of solving the problem above is theconstraint (18d), which brings the ‘‘time-coupling’’ propertyto the control decisions. Specifically, the current controldecisions sþi ðtÞ, s�i ðtÞ will have an impact on the futurecontrol decisions. In the later part, we will design a ‘‘virtualenergy queue’’ to remove this ‘‘time-coupling’’ propertywhile also ensuring the constraint (18d).

4 ALGORITHM DESIGN

In this section, we design an online algorithm based on theLyapunov optimization technique [6] to solve the stochasticoptimization problem above. Because of the time-couplingconstraint (18d), Lyapunov optimization technique cannot

be applied directly. In the following, we first consider arelaxed problem, which fits into the framework of Lyapunovoptimization. Then, we design our algorithm based on theinsights provided by this relaxed problem.

4.1 Relaxed ProblemDenote the time-average expected charge and dischargerate of thermal storage i, respectively, as follows:

sþi ¼ lim supT!1

1

T

XT�1

t¼0

E sþi ðtÞ� �

; (19)

s�i ¼ lim supT!1

1

T

XT�1

t¼0

E s�i ðtÞ� �

: (20)

According to the dynamics of thermal storage energy level(7), in order to ensure 0 � SiðtÞ � Smax

i for all t, we musthave the following equation:

�isþi ¼ s�i : (21)

Therefore, we have the following relaxed problem:

minL;x;

sþ ;s�

: g ¼ lim supT!1

1

T

XT�1

t¼0

EXNi¼1

EiðtÞ þXNi¼1

XMj¼1

BijðtÞ( )

; (22)

subject to constraints (18b), (18e), (18f), (18g), (18h), (18i),and (21).

The optimal solution to the relaxed problem above iseasy to characterize based on the framework of Lyapunovoptimization, which is described in the following theorem.Theorem 1 (below) shows that we can achieve theminimum time average operating cost for a given workloadarrival rate vector W using a stationary, randomizedalgorithm. The algorithm only chooses control decisionsaccording to a fixed probability distribution that dependson the system state ðriðtÞ; piðtÞ;Wc

j ðtÞ; 8i; j; cÞ, but is inde-pendent of ðQc

i ðtÞ; EiðtÞ; 8i; cÞ. In Theorem 1, 6 denotes thecapacity region of the system, which is the closure of sets ofrates W for which there exists a joint geographical loadbalancing, workload scheduling, and storage managementalgorithm that can ensure the queue stability (6).

Theorem 1. If the vector ðriðtÞ; piðtÞ;Wcj ðtÞ; 8i; j; cÞ is i.i.d. over

periods, then, for any arrival rate vector W ¼D EfWðtÞg 2 6,there exists a stationary, randomized control policy that choosescontrol decisions ~�

c

ijðtÞ, ~xci ðtÞ, ~sþi ðtÞ and ~s�i ðtÞ, based solely onthe value of ðriðtÞ; piðtÞ;Wc

j ðtÞ; 8i; j; cÞ irrespective of queueinformation while satisfying all constraints of the relaxedproblem and providing the following guarantees:

EXMj¼1

~�c

ijðtÞ � ~xci ðtÞ( )

¼ 0; 8i; c; t (23)

E �i~sþi ðtÞ

� �¼E ~s�i

� �; 8i; t

(24)

EXNi¼1

~EiðtÞ þXNi¼1

XMj¼1

~BijðtÞ( )

¼ g�relðWÞ; 8t (25)


where the expectations are w.r.t. the randomness inðriðtÞ; piðtÞ;Wc

j ðtÞ; 8i; j; cÞ and possibly, randomized controldecisions, and g�relðWÞ is the optimal objective value of therelaxed problem (22) given an arrival rate vector W.

Proof. The result follows from Theorem 4.5 of [6] and isproved by using the Caratheodory’s theorem. It isomitted here for brevity. g

Denote the optimal objective value of the originalproblem (18) as g�ðWÞ given an arrival rate vector W.Obviously, g�relðWÞ � g�ðWÞ. Let

A1 ¼DXCc¼1

Wcmaxbmaxnc

þXNi¼1

XNi¼1

pmaxi ð1þ �iÞITiP

busyi þ ðpmax

i þ �iÞsþi;max

( ):

(26)

From the bounds we assumed before, we have g � A1

for any feasible control policy subject to constraints (18b),(18e), (18f), and (18h). Instead of solving the relaxedproblem, we will use the existence of such an optimalpolicy to help us design our control policy that meets allconstraints of the original problem (18), and derive theperformance of our algorithm.

4.2 The Stochastic Cost MinimizationAlgorithm (SCMA)

The idea of our algorithm is to construct a Lyapunov-basedcontrol algorithm for determining the optimal workloadrouting, scheduling, and thermal storage managementscheme.

First, we define a Lyapunov function as follows:

LðtÞ ¼D 1

2

XNi¼1

XCc¼1

Qci ðtÞ

� 2þ SiðtÞ � �ið Þ2" #

; (27)

where �i is a constant to be specified later. Now defineKðtÞ ¼D ðQc

i ðtÞ; SiðtÞ; 8i; cÞ, and define a one-period condi-tional Lyapunov drift as follows:

rðtÞ ¼D E Lðtþ 1Þ � LðtÞjKðtÞf g: (28)

Here the expectation is taken over the randomness ofworkload arrival, electricity price, and renewable genera-tion, as well as the randomness in choosing the controlactions. Then, following the Lyapunov optimizationframework, we add a function of the expected cost overone period (i.e., the penalty function) to (28) to obtain thefollowing drift-plus-penalty term:

rV ðtÞ ¼D rðtÞ þ VE

XNi¼1

EiðtÞ þXNi¼1

XMj¼1

BijðtÞjKðtÞ( )

; (29)

where V is a positive control parameter to be specifiedlater. Then, we have the following lemma regarding thedrift-plus-penalty term:

Lemma 1. For any feasible action under constraints (18b),(18e), (18f), (18g), and (18h) that can be implemented at periodt, we have

rV ðtÞ�A2þXNi¼1

E ðSiðtÞ � �iÞ �isþi ðtÞ � s�i ðtÞ�

jKðtÞ� �

þXNi¼1

XCc¼1

E Qci ðtÞ

XMj¼1

�cijðtÞ � xci ðtÞ !

jKðtÞ( )

þVXNi¼1

E EiðtÞf gþVXNi¼1

XMj¼1

E bijXCc¼1

�cijðtÞncjKðtÞ( )

; (30)

where A2 is the constant given by the following:

A2 ¼D XN

i¼1

max �isþi;max

� �2; s�i;max

� �2 �

2

þXNi¼1

XCc¼1

Wcmax

� 2þ ITi�idc

� �2�

2: (31)

Proof. See the supplementary document which is availablein the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TPDS.2013.278. g

We now present the SCMA formulation. The main designprinciple of our algorithm is to choose control actions thatgreedily minimize the R.H.S. of (30). Our algorithm can benaturally decomposed into two parts: workload routingand joint workload scheduling and storage management,as follows:

Stochastic Cost Minimization Algorithm: Initialize V and�i; 8i. At each period t, observe ðWc

j ðtÞ; riðtÞ; piðtÞ; 8i; j; cÞand KðtÞ, and do:

. Workload Routing: For each proxy j, choose the routingvector ðð�cijÞ

�; 8iÞ for type c jobs as the solution to thefollowing problem:

min :XNi¼1

Qci ðtÞ þ Vbijnc

� �cijðtÞ

s:t: �cij; 8i� �

2 LcjðtÞ: (32)

. Workload Scheduling and Storage Management: Foreach data center i, choose the workload schedulingvector fðxciðtÞÞ

�; 8cg and thermal storage decisionsðsþi ðtÞÞ

�and ðs�i ðtÞÞ

� as the solution to the followinglinear optimization problem:

Minimize :

�XCc¼1

QciðtÞxci ðtÞ þ SiðtÞ � �ið Þ �isþi ðtÞ � s�i ðtÞ

� þ Vyi;

s:t:

yi � piðtÞð1þ�iÞ"PC

c¼1 xci ðtÞdc

�iP busyi �Pidle

i

� �

þ ITiPidlei

#


þ piðtÞ þ �ið Þsþi ðtÞ � piðtÞ s�i ðtÞ þ riðtÞ�

;

yi � �isþi ðtÞ;

s�i ðtÞ � �i ITiPidlei þ

PCc¼1 x

ci ðtÞdc

�iðPbusy

i �Pidlei Þ

" #;

0 � sþi ðtÞ � sþi;max;

0 � s�i ðtÞ � s�i;max;

xci ðtÞ � 0; 8c;XCc¼1

xci ðtÞdc � ITi�i; (33)

where yi is a slack variable used to transform thenonlinear operator ½��þ into linear ones.

. Queue Update: Update KðtÞ according to the dynamics(4) and (7).

Note that when solving the problem (33), the resultingoptimal charge/discharge solution may not satisfy theconstraint (10). In this case, let H ¼D �iðsþi ðtÞÞ

� � ðs�i ðtÞÞ� and

we define the actual thermal storage charge and dischargerates as follows:

sþi ðtÞ� 0¼ H

�iif H � 0,

0 otherwise.

(34)

s�i ðtÞ� 0¼ �H if H G 0,

0 otherwise.

(35)

We have the following lemma regarding the optimality ofthe actual thermal storage charge and discharge rates:

Lemma 2. The thermal storage charge and discharge ratesðsþi ðtÞÞ

0and ðs�i ðtÞÞ

0 above is also an optimal solutionto the problem (33).

Proof. See the supplementary document which is availableonline. g

Under the above actual charge/discharge decisions, wepresent the following two properties of the structure of theoptimal solution to (33) that is useful in the performanceanalysis.

Lemma 3. The optimal solution to (33) with the additionalconstraint (10) has the following properties:

1. If SiðtÞ � �i 9 � V�i=�i, then ðsþi ðtÞÞ� ¼ 0.

2. If SiðtÞ � �i G � VpiðtÞ, then ðs�i ðtÞÞ� ¼ 0.


4.3 Interpretation of SCMAThe detailed control decisions taken by SCMA are as follows:

. The complexity of solving the workload routingproblem (32) depends on the feasible set Lc

jðtÞ.Usually, it has a threshold-based solution. Forexample, suppose that the feasible set Lc

jðtÞ onlycontains constraints (2), (3), �cijðtÞ ¼ 0; 8i 62 Icj , and

�cijðtÞ 2 Z. The optimal solution is the followingthreshold-based policy: Let

i� ¼ arg mini2Icj

Qci ðtÞ þ Vbijnc

� : (36)

Then,

�cijðtÞ� ��

¼ Wcj ðtÞ if i ¼ i�,

0 if i 6¼ i�.

(37)

It means that all the jobs would be routed to thedata center with the shortest queue length or thelowest bandwidth cost. The weights of the queuelength and the bandwidth cost are adjusted by theparameter V .

. From the problem formulation (33), we can see thatSCMA will always use the renewable energy riðtÞ asmuch as possible to serve queued workloadsirrespective of queue lengths and electricity pricesso that the first term in the objective is minimizedwhile the third term in the objective is unchanged.WhenVpiðtÞð1þ �iÞdcðPbusy

i � Pidlei Þ=�i G Qc

i ðtÞ, whichmeans that the electricity price is low enough or thequeue length for type c jobs is high enough, SCMA willalso use some brown energy to serve jobs of type cif needed. For thermal storage management, whenSiðtÞ � �i 9 0, the stored energy in thermal storage willbe used to cool the data center, since there is enoughenergy stored in it. Also, if the current electricity priceis low enough such that piðtÞ G ð�i � SiðtÞÞ�i=V � �i,the thermal storage will store energy as much aspossible for later use to leverage the opportunity ofcurrent low electricity price. The thresholds of charg-ing or discharging depend on the current storedenergy level as well as the parameter V .

Note that SCMA only requires the knowledge of theinstantaneous values of system dynamics and can operateonline without requiring any knowledge of the statistics ofthese stochastic processes. Moreover, each proxy or datacenter solves its own optimization problem distributively,where only the queue length information of data centersneeds to be exchanged between data centers and proxies.Therefore, SCMA is easy to implement in practice.

5 PERFORMANCE ANALYSIS

In this section, we present the analytical performanceresults for SCMA. Detailed numerical results are describedin the next section. First, we present the results whenðriðtÞ; piðtÞ;Wc

j ðtÞ; 8c; i; jÞ is i.i.d. stochastic process. Notethat according to the framework of Lyapunov optimization[6], our results can also be extended to the more generalsetting where ðriðtÞ; piðtÞ;Wc

j ðtÞ; 8c; i; jÞ evolves accordingto some finite state irreducible and aperiodic Markovchain. Furthermore, our numerical simulation results in thenext section are based on the real-world traces without anyspecific distribution assumption.

Theorem 2. Suppose that 0 G V � Vmax, where Vmax ¼DminifðSmax

i � �isþi;max � s�i;maxÞ=ðpmax

i � �i=�iÞg. Let


�i ¼D Vpmaxi þ s�i;max and Qc

i ð0Þ ¼ 0; 8i; c. Then, under theSCMA algorithm, we have the following:

1. The thermal energy queues satisfy the following forall time tunder any arbitrary ðriðtÞ; piðtÞ;Wc

j ðtÞ; 8c; i; jÞprocess:

0 � SiðtÞ � Smaxi ; 8i: (38)

2. If the vector ðriðtÞ; piðtÞ;Wcj ðtÞ; 8c; i; jÞ is i.i.d. over

periods, and if there exists a constant such thatWþ 1 2 6, then the total batch workload queuelength satisfies the following under any arbitraryðriðtÞ; piðtÞ;Wc

j ðtÞ; 8c; i; jÞ process:

Q � A1V þA2

: (39)

The time-average expected total operating costunder the SCMA algorithm is within bound A2=Vof the optimal value:

�gSCMA � �g� þA2=V; (40)

where �g� is the optimal cost achieved by any feasiblecontrol policy that can stabilize the queues, and A1,A2 are constants given by (26) and (31), respectively.


6 NUMERICAL EVALUATION

In the remainder of the paper, we evaluate the performanceof the SCMA under realistic traces. Our goal is threefold: (i)to illustrate the benefits by jointly considering the thermalstorage, delay-tolerant workloads, and geographical loadbalancing in data centers to reducing the operating cost; (ii)to understand the impacts of various parameters on thecontrol decisions made by SCMA; and (iii) to understandthe trade-offs among cost reduction, workload delay, andthermal storage capacity enabled by the SCMA.

6.1 Experimental SetupIn this part, we introduce the default settings that are usedthroughout the evaluations unless otherwise stated. Thelength of a control period is 10 minutes and the time-horizonin the evaluations is 4000 periods.

6.1.1 Data Center DescriptionsWe consider four data centers, one at the geographic centerof each city that is known to have Google data centers: NewYork, Palo Alto, Chicago, and Houston. Moreover, weassume that there is a proxy located near each data center.The bandwidth cost bij between proxies and data centers isset to be proportional to the distances between cities andcomparable to the energy cost. The number of availableactive servers in each data center is taken to be ITi ¼ 350.The energy consumption of each server during one periodat idle and busy state are set to be Pidle

i ¼ 100 W� 1=6 h; 8iand Pbusy

i ¼ 250 W� 1=6 h; 8i, respectively. Without loss ofgenerality, the processing speed of each server is assumedto be �i ¼ 1; 8i. The cooling efficiency of each data center isset to be the average value of the data center industry as�i ¼ 1; 8i. Notice that here, we assume the homogenoussettings of data centers in order to make the analysis of theimpacts of other factors (e.g., energy prices, renewableavailability, bandwidth cost) more explicitly.

6.1.2 Workload DescriptionAs with [16], we choose MapReduce [23], which is apopular type of computation-intensive workloads in datacenters, as the representative of delay-tolerant workloads.We use the historical Hadoop (an open source implemen-tation of MapReduce) traces on a 600-machine cluster atFacebook [29] to calculate the average 10-min workloadarrivals. A portion of the workload trace during one day isshown in Fig. 3a. The workload arrivals to each proxy areshifted according to the time zone. We assume there aretwo types of jobs, with job length dc ¼ f1; 0:5g andcommunication demand nc ¼ f1; 0:5g. We assume thathalf of the arriving requests belong to type 1 and the otherhalf belong to type 2. The workload traces are scaled suchthat the peak demand can be supported entirely by its owndata center without delay.

6.1.3 Energy Price DescriptionWe use the day-ahead hourly locational marginal prices(LMPs) in wholesale electricity markets at the above fourdata center locations. They are obtained from the publiclyavailable government sources [30], [31]. A portion of thehourly electricity prices during the first 24 hours at theselocations is shown in Fig. 3b.

Fig. 3. Real-world traces used in evaluations. (a) 10-min average workload arrivals for one day [29]. (b) Hourly electricity prices in day-ahead marketsfor one day at four locations [30], [31]. (c) 10-min average solar and wind energy generation for one week [32].


6.1.4 Renewable Energy DescriptionWe consider on-site wind generation at two locations (NewYork and Chicago) and on-site solar generation at the othertwo locations (Palo Alto and Houston). The traces of wind andsolar sources are obtained from [32] that has wind speed andsolar irradiance measurements every 10 minutes. The tracesare scaled properly so that the average renewable productioncan meet half of the average power consumption at each datacenter. A portion of solar and wind energy at two locationsduring the first two days is depicted in Fig. 3c.

6.1.5 Thermal Storage DescriptionWe assume each data center has installed a thermal storagesystem. The maximum charge (discharge) rate sþi;maxðs�i;maxÞis set to be the peak cooling energy consumption duringone period. The round-trip charging efficiency �i is set to be0.8. The storage operating cost factor ~�i and the storagecapacity Smax

i are parameters of which the impact on theperformance of SCMA will be investigated.

6.1.6 Algorithm BenchmarksTo provide benchmarks for the performance of SCMA, wecompare it with the following three baselines that eitherapproximate the current practice [33], or are proposed bysome recent work [16], [18].

. Baseline 1 (B1): No workload scheduling, no storage.In this approach, the workloads are routed to thenearest data center and served immediately withoutany delay. This scheme is employed by many compa-nies in practice so as to serve all the incoming work-loads as soon as possible without any considerationon energy price or renewable energy availability [33].

. Baseline 2 (B2): Renewable-oblivious workloadscheduling, no storage. This approach is verysimilar to that proposed in the recent work [16],which investigates jointly routing and schedulingdelay-tolerant workloads in multiple data centers toleverage the opportunity of time-varying energyprice. However, no renewable energy or thermalstorage is taken into account in this scheme.

. Baseline 3 (B3): Renewable-aware workload sched-uling, no storage. This approach is proposed in therecent work [18] for cost minimization in a singledata center. Renewable energy availability and time-varying energy price are considered but withoutthermal storage. We modify its algorithm to incor-porate routing decisions.

6.2 Numerical ResultsThe evaluation of SCMA will be organized as the followingaspects.

6.2.1 Cost SavingsNote that prior studies mainly focus on reducing energycost without considering the bandwidth cost for workloadrouting. To evaluate the energy cost saving due to ouralgorithm SCMA by leveraging delay-tolerant workloads,thermal storage, and geographical load balancing, we firstassume that the bandwidth cost bij ¼ 0; 8i; j so that we canfocus on the energy cost. Since the performances of SCMA,B2, and B3 all depend on the parameter V , for faircomparison, we choose the parameter V in different schemessuch that the average delay of queued workloads in theseschemes are equal. Note that B1 has no delay. Moroever, theSCMA is under the following parameter settings: the storageoperating cost factor �i ¼ 10; 8i and the storage capacity Smax

i

is assumed to be able to support the average cooling demandof a data center for 10 hours. The result is shown in Fig. 4.From the figure, we can observe that SCMA outperforms allbenchmark schemes. Specifically, by comparing SCMA withB3, we can observe that thermal storage can indeed helpreduce the total electricity cost. Moreover, although B2considers the time-varying electricity price, it is renewable-oblivious and tries to serve workloads only when theelectricity price is low enough. Therefore, it wastes a lot ofrenewable energy and performs the worst. This shows theimportance of renewable-aware workload management indata centers with on-site renewable generation. Finally, bycomparing B3 with B1, we can see the advantage of delay-tolerant workloads in improving renewable energy utilizationand reducing electricity cost.

Fig. 4. Average energy cost (in unit of dollars) comparison betweenSCMA and baseline schemes.

Fig. 5. Average operating cost (in unit of dollars) comparison betweenSCMA and baseline schemes.


Then, we compare our algorithm SCMA with the base-line schemes above while considering the bandwidth cost.Obviously, the bandwidth cost of B1 is zero by our as-sumption because it always routes all workloads to thenearest data centers. The result of the average operatingcost for the other three algorithms is shown in Fig. 5. Bytaking into account the different bandwidth costs betweenproxies and data centers, our algorithm can achieve thelargest total operating cost saving. The importance ofnetwork-awareness is clear from the figure above. Note thatB1 and B3 have similar operating cost since the bandwidthcost of B1 is minimum although its energy cost is higherthan that of B3. B2 has the worst performance of operatingcost since both the energy cost and the bandwidth cost arethe highest among all schemes.

6.2.2 Trade-Off Between Cost and DelayIn this part, we focus on the trade-offs among delay, totaloperating cost, and thermal storage capacity in SCMA.We choose different V and observe the corresponding totaloperating cost and average workload delay in SCMA. Theresult is shown in Fig. 6. As we can observe from the figure,with the increase of the parameter V , SCMA can get lowertotal operating cost with trade-offs in the workload delay,which validates the analytical performance results in Theorem2. Note that by selecting a larger V , SCMA would be moreaggressively minimizing the operating cost, which may delaymore jobs to be served later when enough renewable energy isavailable or energy price is low, causing larger queuing delay.

6.2.3 Impact of Storage CostTo evaluate the impact of thermal storage cost on theoperating cost saving, we fix parameters V and Smax

i ; 8i, andevaluate SCMA under different �i ¼ ½0; 5; 10; 15; 20; 25; 30�; 8i.The result is shown in Fig. 7. We can observe that with theincrease of thermal storage cost factor �i, the operating costsaving is smaller. When �i is very large, SCMA does notuse the thermal storage at all. However, even in this case,there is still cost saving compared with B1 because of thedelay-tolerant workload scheduling and geographical loadbalancing.

7 CONCLUDING REMARKS

In this paper, we studied the problem of joint network-aware workload routing, delay-tolerant workload scheduling,and thermal storage management to improve the renewableenergy utilization and reduce the time-average total operatingcost in data centers. We design an online control algorithmcalled SCMA and demonstrate its effectiveness throughboth analytical analysis and numerical evaluations. More-over, SCMA provides an explicit trade-off between costsaving and workload delay.

ACKNOWLEDGMENT

This work was supported in part by the U.S. National ScienceFoundation under Grants ECCS-1129061, ECCS-1129062,CNS-1239274, CNS-1343356, and Eckis Professor endow-ment at the University of Florida.

REFERENCES

[1] J. Koomey, Growth in Data Center Electricity Use 2005 to 2010.Burlingame, CA, USA: Analytics Press, 2011.

[2] P.X. Gao, A.R. Curtis, B. Wong, and S. Keshav, ‘‘It’s Not EasyBeing Green,’’ in Proc. ACM SIGCOMM, Aug. 2012, pp. 211-222.

[3] Z. Liu, M. Lin, A. Wierman, S. Low, and L. Andrew, ‘‘GreeningGeographical Load Balancing,’’ in Proc. ACM SIGMETRICS,2011, pp. 233-244.

[4] Z. Liu, M. Lin, A. Wierman, S. Low, and L. Andrew,‘‘Geographical Load Balancing with Renewables,’’ in Proc.GreenMetrics, 2011, pp. 1-5.

[5] Apple and the Environment. [Online]. Available: http://www.apple.com/environment/renewable-energy/.

[6] M.J. Neely, Stochastic Network Optimization With Application toCommunication and Queueing Systems. San Rafael, CA, USA:Morgan & Claypool Publishers, 2010.

[7] Google’s PPAs: What, How, and Why. [Online]. Available:http://www.google.com/about/datacenters/energy.html.

[8] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang,M. Marwah, and C. Hyser, ‘‘Renewable and Cooling AwareWorkload Management for Sustainable Data Centers,’’ in Proc.ACM SIGMETRICS/PERFORMANCE, 2012, pp. 175-186.

[9] I. Goiri, K. Le, T.D. Nguyen, J. Guitart, J. Torres, and R. Bianchini,‘‘Greenhadoop: Leveraging Green Energy in Dat-ProcessingFramework,’’ in Proc. EuroSys, 2012, pp. 57-70.

[10] R. Urgaonkar, B. Urgaonkary, M.J. Neely, and A. Sivasubramaniam,‘‘Optimal Power Cost Management Using Stored Energy in DataCenters,’’ in Proc. ACM SIGMETRICS, San Jose, CA, USA, June 2011,pp. 221-232.

Fig. 7. Impact of thermal storage cost factor �i on the average cost(normalized to the average cost of B1).Fig. 6. Average total operating cost (in unit of dollars) and delay

performance (in unit of control periods) of SCMA with different V .


[11] S. Govindan, A. Sivasubramaniam, and B. Urgaonkar, ‘‘Benefitsand Limitations of Tapping Into Stored Energy for Datacenters,’’in Proc. ISCA, 2011, pp. 341-352.

[12] Y. Guo, Z. Ding, Y. Fang, and D. Wu, ‘‘Cutting Down ElectricityCost in Internet Data Centers by Using Energy Storage,’’ in Proc.IEEE GLOBECOM, 2011, pp. 1-5.

[13] Y. Wang, X. Wang, and Y. Zhang, ‘‘Leveraging Thermal Storageto Cut the Electricity Bill for Datacenter Cooling,’’ in HotPower,2011, pp. 1-5.

[14] M. Lin, A. Wierman, L.L.H. Andrew, and E. Thereska, ‘‘DynamicRight-Sizing for Power-Proportional Data Centers,’’ in Proc. IEEEINFOCOM, 2011, pp. 1098-1106.

[15] M. Lin, Z. Liu, A. Wierman, and L.L.H. Andrew, ‘‘Online Algo-rithms for Geographical Load Balancing,’’ in Proc. IGCC, 2012, pp. 1-10.

[16] Y. Yao, L. Huang, A. Sharma, L. Golubchik, and M. Neely, ‘‘DataCenters Power Reduction: A Two Time Scale Approach for DelayTolerant Workload,’’ in Proc. IEEE INFOCOM, 2012, pp. 1431-1439.

[17] D. Xu and X. Liu, ‘‘Geographical Trough Filling for InternetDatacenters,’’ in Proc. IEEE INFOCOM Mini-Conf., 2012, pp. 2881-2885.

[18] S. Ren, Y. He, and F. Xu, ‘‘Provably-Efficient Job Scheduling forEnergy and Fairness in Geographically Distributed Data Centers,’’ inProc. IEEE ICDCS, 2012, pp. 22-31.

[19] Y. Guo, Y. Gong, Y. Fang, P.P. Khargonekar, and X. Geng, ‘‘OptimalPower and Workload Management for Green Data Centers withThermal Storage, IEEE GLOBECOM, Atlanta, GA, USA, 2013.

[20] X. Meng, V. Pappas, and L. Zhang, ‘‘Improving the Scalability ofData Center Networks with Traffic-Aware Virtual Machine Place-ment,’’ in Proc. IEEE INFOCOM, San Diego, CA, USA, Mar. 2010,pp. 1-9.

[21] N. Buchbinder, N. Jain, and I. Menache, ‘‘Online Job-Migrationfor Reducing the Electricity Bill in the Cloud,’’ in Proc.NETWORKING, 2011, pp. 172-185.

[22] M. Alicherry and T.V. Lakshaman, ‘‘Network Aware ResourceAllocation in Distributed Clouds,’’ in Proc. IEEE INFOCOM,Orlando, FL, USA, Mar. 2012, pp. 963-971.

[23] J. Dean and S. Ghemawat, ‘‘Mapreduce: Simplified DataProcessing on Large Clusters,’’ in Proc. OSDI, 2004, pp. 137-149.

[24] L. Rao, X. Liu, L. Xie, and W. Liu, ‘‘Minimizing Electricity Cost:Optimization of Distributed Internet Data Centers in a Multi-Electricity-Market Environment,’’ in Proc. IEEE INFOCOM, 2010,pp. 1-9.

[25] Amazon EC2 Spot Instances. [Online]. Available: http://aws.amazon.com/ec2/spot-instances/.

[26] C. Ren, D. Wang, B. Urgaonkar, and A. Sivasubramaniam,‘‘Carbon-Aware Energy Capacity Planning for Datacenters,’’ inProc. IEEE MASCOTS, 2012, pp. 391-400.

[27] S.B. Peterson, J.F. Whitacre, and J. Apt, ‘‘The Economics of UsingPlug-In Hybrid Electric Vehicle Battery Packs for Grid Storage,’’J. Power Sources, vol. 195, no. 8, pp. 2377-2384, Apr. 2010.

[28] A. Gandhi, M. Harchol-Balter, R. Das, and C. Lefurgy, ‘‘OptimalPower Allocation in Server Farms,’’ in Proc. 11th ACM SIG-METRICS, Seattle, WA, USA, Aug. 2009, pp. 157-168.

[29] Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, ‘‘The Case forEvaluating Mapreduce Performance Using Workload Suites,’’ inProc. IEEE MASCOTS, 2011, pp. 390-399.

[30] Federal Energy Regulatory Commission. [Online]. Available:http://www.ferc.gov/.

[31] United States Energy Information Administration. [Online].Available: http://www.eia.gov/.

[32] NREL: Measurement and Instrumentation Data Center. [Online].Available: http://www.nrel.gov/midc/.

[33] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, and B. Maggs,‘‘Cutting the Electric Bill for Internet-Scale Systems,’’ in Proc.ACM SIGCOMM, 2009, pp. 123-134.

Yuanxiong Guo received his BEng degree fromthe Department of Electronics and InformationEngineering, Huazhong University of Scienceand Technology, Wuhan, China, in 2009. He hasbeen working towards the PhD degree at theDepartment of Electrical and Computer Engi-neering at University of Florida, Gainesville,USA since August 2010. His current researchinterests are in the area of cyber-physicalsystems including smart grids, sustainable datacenters, and cloud computing. He is a recipient

of the Best Paper Award from IEEE GLOBECOM 2011, Houston, TX,USA. He is a Student Member of the IEEE.

Yanmin Gong received her BEng degree inelectrical engineering from Huazhong Universityof Science and Technology, Wuhan, China, in2009, and MSc degree in electrical engineeringfrom Tsinghua University, Beijing, China, in2012. She has been working towards the PhDdegree at the Department of Electrical andComputer Engineering at University of Florida,Gainesville, USA, since August 2012. Hercurrent research interests are in the area ofoptimization, security and privacy in cyber-

physical systems and mobile computing. She is a Student Member ofthe IEEE.

Yuguang Fang received the BS/MS degree inMathematics from Qufu Normal University, China,in 1987, a PhD degree in Systems Engineeringfrom Case Western Reserve University, USA, in1994 and a PhD degree in Electrical Engineeringfrom Boston University, USA, in 1997. He iscurrently a professor in theDepartment of Electricaland Computer Engineering at University of Florida,USA. He held a University of Florida ResearchFoundation (UFRF) Professorship from 2006 to2009, a Changjiang Scholar Chair Professorship

with Xidian University, China, from 2008 to 2011, and a Guest ChairProfessorship with Tsinghua University, China, from 2009 to 2012. He haspublished over 350 papers in refereed professional journals and con-ferences. He received the National Science Foundation Faculty EarlyCareer Award in 2001 and theOffice of Naval Research Young InvestigatorAward in 2002. He has also received a 2010-2011UFDoctoral DissertationAdvisor/Mentoring Award, 2011 Florida Blue Key/UF HomecomingDistinguished Faculty Award and the 2009 UF College of EngineeringFaculty Mentoring Award. Dr. Fang is a Fellow of IEEE. He is currently theEditor-in-Chief of IEEE Transactions on Vehicular Technology. He was theEditor-in-Chief of IEEEWirelessCommunications (2009-2012) and serves/served on several editorial boards of technical journals including IEEETransactions on Mobile Computing (2003-2008, 2011-present), IEEENetwork (2012-present), IEEE Transactions on Communications (2000-2011), IEEETransactionsonWirelessCommunications (2002-2009), IEEEJournal onSelectedAreas inCommunications (1999-2001), IEEEWirelessCommunications Magazine (2003-2009), and ACM Wireless Networks(2001-2013). He is currently serving as the Technical Program CommitteeCo-Chair for IEEE INFOCOM’2014.

Pramod P. Khargonekar received BTech de-gree in electrical engineering from the IndianInstitute of Technology, Bombay, India and MSdegree in mathematics and PhD degree inelectrical engineering from the University ofFlorida, USA. He has held faculty positions atthe University of Minnesota, USA and TheUniversity of Michigan, USA. He was Chairmanof the Department of Electrical Engineering andComputer Science at Michigan from 1997 to2001, and also held the title Claude E. Shannon

Professor of Engineering Science there. From 2001 to 2009, he wasDean of the College of Engineering and is now Eckis ProfessorElectrical and Computer Engineering at the University of Florida. Heserved as Deputy Director for Technology at the U.S. Department ofEnergy’s Advanced Research Projects Agency C Energy (ARPA-E). Heis currently serving the U.S. National Science Foundation as AssistantDirector for Engineering. His current research interests are focused onrenewable energy and electric grid, neural engineering, and systemsand control theory. He is a recipient of the NSF Presidential YoungInvestigator Award, the American Automatic Control Council’s DonaldEckman Award, the Japan Society for Promotion of Science Fellow-ships, and a Distinguished Alumnus Award and Distinguished ServiceAward from the Indian Institute of Technology, Bombay. He is a co-recipient of the IEEEW.R.G. Baker Prize Award, the IEEE CSS GeorgeS. Axelby Best Paper Award, and the AACC Hugo Schuck Best PaperAward. He was a Springer Professor at the University of California,Berkeley, USA in 2010. He is a Fellow of IEEE and is on the list of Webof Science Highly Cited Researchers.


Xiaojun Geng received the BSc and MScdegrees in astronautical engineering from North-western Polytechnic University, Xi’an, China, in1993 and 1996, respectively, and receivedtwo PhD degrees in electrical and computerengineering, one from Shanghai Jiao TongUniversity, Shanghai, China, in 1999, and theother from the University of Florida, Gainesville,USA, in 2003. In 2003, Dr. Geng joined theDepartment of Electrical and ComputerEngineering as an Assistant Professor at

California State University, Northridge, USA where in 2009, shebecame an Associate Professor. In 2013, she joined the Departmentof Electrical and Computer Engineering as an Assistant Professor in theUniversity of West Florida, USA. Her research interests include controlof discrete-event systems, coordination of multi-agent systems, andpower management of smart grid. She is a member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


2030 IEEE TRANSACTIONS ON PARALLEL AND ...problem of joint geographical load balancing, delay-tolerant workload scheduling, and thermal storage management for green energy integration

Documents