Energy efficient hotspot-targeted embedded liquid cooling of electronics

Applied Energy 138 (2015) 414–422

Contents lists available at ScienceDirect

Applied Energy

journal homepage: www.elsevier .com/ locate/apenergy

Energy efficient hotspot-targeted embedded liquid cooling of electronics

http://dx.doi.org/10.1016/j.apenergy.2014.10.0680306-2619/� 2014 Elsevier Ltd. All rights reserved.

⇑ Corresponding author. Tel.: +41 44 632 27 38; fax: +41 44 632 11 76.E-mail address: [email protected] (D. Poulikakos).

1 Present address: Department of Mechanical Engineering, University CollegeLondon (UCL), Torrington Place, London WC1E 7JE, UK.

Chander Shekhar Sharma a, Manish K. Tiwari a,1, Severin Zimmermann a, Thomas Brunschwiler b,Gerd Schlottig b, Bruno Michel b, Dimos Poulikakos a,⇑a Department of Mechanical and Process Engineering, ETH Zurich, 8092 Zurich, Switzerlandb Advanced Micro Integration, IBM Research–Zurich, 8803 Rüeschlikon, Switzerland

h i g h l i g h t s

�We present a novel concept for hotspot-targeted, energy efficient ELC for electronic chips.� Microchannel throttling zones distribute flow optimally without any external control.� Design is optimized for highly non-uniform multicore chip heat flux maps.� Optimized design minimizes chip temperature non-uniformity.� This is achieved with pumping power consumption less than 1% of total chip power.

a r t i c l e i n f o

Article history:Received 5 June 2014Received in revised form 20 September2014Accepted 24 October 2014

Keywords:Hotspot-targeted coolingMicrochannel coolingElectronics coolingHotspotsEnergy efficient computingMulticore microprocessors

a b s t r a c t

Large data centers today already account for nearly 1.31% of total electricity consumption with coolingresponsible for roughly 33% of that energy consumption. This energy intensive cooling problem is exac-erbated by the presence of hotspots in multicore microprocessors due to excess coolant flow requirementfor thermal management. Here we present a novel liquid-cooling concept, for targeted, energy efficientcooling of hotspots through passively optimized microchannel structures etched into the backside of achip (embedded liquid cooling or ELC architecture). We adopt an experimentally validated and compu-tationally efficient modeling approach to predict the performance of our hotspot-targeted ELC design.The design is optimized for exemplar non-uniform chip power maps using Response Surface Methodol-ogy (RSM). For industrially acceptable limits of approximately 0.4 bar (40 kPa) on pressure drop and onepercent of total chip power on pumping power, the optimized designs are computationally evaluatedagainst a base, standard ELC design with uniform channel widths and uniform flow distribution. For anaverage steady-state heat flux of 150 W/cm2 in core areas (hotspots) and 20 W/cm2 over remaining chiparea (background), the optimized design reduces the maximum chip temperature non-uniformity by 61%to 3.7 �C. For a higher average, steady-state hotspot heat flux of 300 W/cm2, the maximum temperaturenon-uniformity is reduced by 54% to 8.7 �C. It is shown that the base design requires a prohibitively highlevel of pumping power (about 2000 fold for 150 W/cm2 case and 600 fold for 300 W/cm2 case) to matchthe thermal performance of the optimized, hotspot-targeting designs. The pumping power requirementfor optimized designs is only 0.23% and 0.17% of the total chip power for 150 W/cm2 and 300 W/cm2

hotspot heat flux respectively. Moreover, the optimized designs distribute the coolant flow withoutany external flow control devices and the performance is only marginally affected by the manifoldgeometry used to supply the coolant to the microchannel heat transfer structure. This also attests tothe robustness of the optimized embedded microchannel structures.

� 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Since the advent of first very large scale integrated circuits,electronic chip manufacturers have endeavored to keep up withMoore’s law [1]. Chip performance has improved due to increasinggate density as well as fabrication of multiple cores on a singlechip. However, the inability to follow the Dennard scaling for

Nomenclature

Afs interfacial area density ð2=ðwc þwwÞÞhf specific enthalpy of fluid (J/kg)hc microchannel height (m)K permeability (m2)lth length of throttling zone (m)n,m integral multipliers_m mass flow rate (kg/s)

mi�

relative mass flow rateP pressure (Pa)q00 heat flux (W/m2)R flow resistance (Pa s/kg)Rthermal thermal resistance (�C m2/W)T temperature (K)Ts,max maximum temperature of chip (at base of silicon die)DTs,max chip wide temperature difference ð¼ Ts;max � Ts;minÞU, Us true and Superficial velocity vectors for porous domain

(m/s)_V volumetric flow rate (m3/s)wc, ww microchannel and wall widths (m)W, L overall chip dimensions (m)Wm half width of inlet manifold

_Wpumping pumping power (W)_Wchip total power dissipated by the chip (W)

Greek lettersafs interfacial heat transfer coefficient (W/m2 K)um fraction of total coolant flow flowing through HSBG

zonec volume porosity ðwc=ðwc þwwÞÞk thermal conductivity (W/m K)

Subscriptsbg, hs, th background, hotspot and throttling zonesf, s fluid and solidfp, sp fluid and solid phases of porous mediumintf interfacemax maximummin minimumopt optimum valueV optimized value at total flow rate _VT

B base caseT total

C.S. Sharma et al. / Applied Energy 138 (2015) 414–422 415

supply voltage has lead to high chip heat flux dissipation densities,especially during the last decade [2]. As a consequence, electronicreliability considerations are driving a shift away from air-cooling[3] towards a number of alternative approaches including liquidcooling [4,5], two-phase cooling [6,7], phase change materials(PCM) [8,9] and nanofluids [10,11]. Single-phase liquid coolinghas long been identified as an effective and feasible approach forcooling high heat flux density chips. Starting with the landmarkwork of Tuckerman and Pease [12], liquid cooling of chips has beenanalyzed in great detail. This includes investigations on traditionalmicrochannel heat sinks [13], manifold microchannel (MMC) heatsinks [5,14], spray and jet cooling [15].

Most of the research on liquid cooling of chips has focused onmaximum temperature (Ts,max) reduction under uniform heat fluxdissipation conditions. However, it is also necessary that the chiptemperature is spatially as uniform as possible (i.e. approachingthe isothermal chip condition). Large temperature gradients inthe package increase thermal stresses in the chip to substrate orheat sink interface, reduce electronic reliability in regions of hightemperature and create circuit imbalances in CMOS devices [4]. Afew studies have focused on reducing chip temperature non-uniformity (DTs,max) under a uniform chip heat flux map. Theseinclude use of flow boiling of dielectric liquid [16], single phaseliquid cooling with variable pin fin density [17], variablemicrochannel width in the streamwise direction [18–20] anddouble-layer microchannel structure [21].

During the last decade, computer architecture has witnessed atrend towards multicore microprocessors due to limitationsencountered in further increasing the single-core clock frequencies[22]. In a multicore microprocessor, each core houses the executionunits [23]. In modern high performance chips, the cores candissipate on an average up to 150 W/cm2 while the rest of the chipdissipates as low as 20 W/cm2. Since the cores dissipate multipletimes higher heat flux than the rest of the chip, we refer to thecores as the hotspots. The large difference in heat dissipationbetween hotspot and background regions makes the goal ofisothermal junction temperatures even more challenging.

Several approaches have been proposed in the past aimed atachieving an isothermal chip condition by preferential cooling of

hotspots (henceforth referred to as hotspot-targeted cooling).These include, for example, use of thermoelectric cooling [24]and electrowetting [25]. However, these approaches are limitedby low device efficiencies, low heat flux pumping capacities andcontact parasitic resistance for the former [24] and relatively lowheat fluxes for the later [25]. Attempts using single phase liquidcooling have also been reported with varying degree of success[26–28]. Additionally, conventional backside attached MMC heatsinks are thermally joined to the chip through a thermal interfacematerial (TIM) which, combined with the thickness of substrateand heat sink base, results in heat spreading and high thermalresistance. For effective hotspot-targeted cooling, the net thermalresistance and heat spreading can be reduced by circulating thecoolant through microchannels etched into the backside of the chip(also termed as embedded or direct chip backside microchannelcooling) [29–32].

It is also critical that any cooling of microprocessors be energyefficient [33]. This becomes even more relevant for large datacen-ters employing thousands of multicore microprocessors. Directelectricity consumption by large data centers had almost tripledbetween 2000 and 2010 and reached 1.31% of the total worldelectricity consumption by 2010 due to increased demands forInformation Technology (IT) related services. Although the growthin data center energy consumption slowed down in the period2005–2010 [34], the still increasing data center energy bill isquickly becoming a cause of concern [35]. Cooling overhead forthese datacenters contributes roughly 33% of this electricityconsumption. Hence, efficient cooling can significantly reduce theIT energy bill [35,36].

In this paper, we propose a novel, hotspot-targeted and, at thesame time, highly energy efficient cooling solution for typical mod-ern day multicore microprocessors. We employ a passive, energysaving approach to alter the heat sink design by optimizing themicrochannel geometry and flow rate distribution. The hotspot-targeted cooling design is described in Section 2. Sections 3 and4 focus on the modeling, validation and optimization methodolo-gies and the final results for two exemplar chip power maps arepresented in Section 5. Section 6 presents general rules for designof hotspot-targeted microchannel heat sink. Lastly, Section 7

Fig. 1. (a) Schematic of a MMC heat sink with embedded microchannels. Arrowsindicate fluid flow. Microchannels will be modeled as porous medium. (b) Chipheat flux distribution and one unit cell used for computational modeling and(c) schematic of the proposed hotspot-targeted embedded microchannel structurein one unit cell. ‘I/2’ and ‘O/2’ indicate half inlet and outlet slot nozzles respectively.HSBG (Hotspot and downstream Background) and BG (only Background) zones aremarked.

416 C.S. Sharma et al. / Applied Energy 138 (2015) 414–422

demonstrates how the hotspot-targeted design distributes thecoolant in the heat sink without the need for any external flowdistribution devices.

2. Hotspot-targeted heat sink concept

Our hotspot-targeted cooling concept utilizes the well knownMMC heat sink architecture [5,14,37,38], consisting of embeddedmicrochannels and a manifold layer above it as shown inFig. 1(a), as the basis for the design. While the multiple inlet andoutlet slot nozzles in the MMC heat sink reduce the overall pres-sure drop and improve the thermal performance through impinge-ment jet cooling and reduction in overall thermal boundary layer

thickness [5,39]; embedded microchannels enable efficient target-ing of hotspots by reducing the overall thermal resistance and heatspreading.

Fig. 1(b) shows the heat dissipation map of a typical multicorechip. The red colored cores are the hotspots and the blue regionis the background. Conventional backside attached liquid coolingsolutions for such a chip are highly energy intensive and lead toovercooling of the background region and undercooling of thehotspot regions, thus leading to poor chip temperature uniformity.We propose a novel microchannel structure to achieve efficientand targeted hotspot cooling, as described in the following.

As shown in Fig. 1(b) (marked by white dashed lines), a repeat-ing unit cell for the chip heat flux map can be identified. Similarly,by exploiting flow symmetry, the heat sink can also be consideredto consist of repeating unit cells. Each unit cell is bounded by sym-metry planes in inlet and outlet slot nozzles. Fig. 1(c) illustrates aschematic of the corresponding unit cell of the proposed embed-ded microchannel cooling structure with inlet and outlet slotnozzles positioned over the hotspots and over the backgroundregion respectively.

The chip area can be divided into two types of streamwise zonesin each unit cell: ‘HSBG’ zones consisting of a hotspot area with adownstream background area; ‘BG’ zones consisting of only back-ground area. These zones are also marked in the correspondingunit cells in Fig. 1(b) and (c). The proposed design consists ofdifferent microchannel structures in the two zones. While theHSBG zones consist of fine channels over the hotspots and coarsechannels over the background, the BG zones consist of coarsechannels throughout except for flow-throttling zones, with finechannels and wide channel walls, near the outlet slot nozzle. Theintroduction of throttling zones is a key design concept introducedhere in order to passively manipulate the flow rate distribution onhotspot and background regions. By altering the throttling zonelength, the fraction of the total flow directed to the HSBG zonescan be changed. Finer channels over the hotspots along with ahigher flow rate in the HSBG zones result in higher heat transfercoefficient over the hotspots as compared to background.

3. Modeling approach

The hotspot-targeted design was optimized using a conjugateheat transfer model described below.

3.1. Computational domain and boundary conditions

The computational domain included the silicon die and micro-channels etched into the back of the die in a unit cell (referFig. 2(a)). Microchannels were modeled as fluid saturated porousmedium [40]. Walls were included inside the porous medium toseparate HSBG and BG zones in order to precisely control theflow rate distribution between these zones during optimization.Mass-flow distribution _mðyÞ and fluid temperature of 20 �C at theinlet, and pressure outlet boundary condition at the outlet wereimposed. Heat flux distribution q00 (x,y) was imposed at the base.Free slip was assumed on all porous medium x–z walls to simulatevolumetric averaging in y-direction [40]. All other boundaries wereassumed to be adiabatic walls. The overall dimensions of the unitcell are shown in Fig. 2(b) and listed in Table 1. These dimensionswere chosen to represent a realistic multicore microprocessor.This work corresponds to a test chip consisting of seven unit cells(refer Fig. 2(top)). Note that the conclusions from this work areeasily adaptable to the eight hotspot arrangement shown inFig. 1 by scaling up the total flow rate.

Fig. 2. (a) Computational domain and boundary conditions, and (b) dimensions ofthe unit cell.

Table 1Unit cell dimensions.

Dimension Description Value

W Chip width 2.32 cmL Test chip length 2.7 cmH Silicon die thickness 525 lmhc Microchannel height 300 lmhn Slot nozzle height 450 lmwn Slot nozzle width 500 lmwhs � lhs Chip core (hotspot) size 4181 lm � 4197 lmwd Distance between hotspots 9280 lm

C.S. Sharma et al. / Applied Energy 138 (2015) 414–422 417

3.2. Governing equations

The Reynolds number (defined as qf UðwchcÞ1=2=lf ) in the micro-

channels stays below 350 across most of the parameter rangeencountered in this work. Thus, the flow was assumed to be lami-nar. The governing equations for flow in porous medium follow[5,40,41].

3.2.1. Continuity

$ � ðqf UsÞ ¼ 0 ð1Þ

3.2.2. Momentum

r �qf

c2 Us � Us

� ��r �

lf

cðrUs þ ðrUsÞTÞ

� �¼ SM �rP ð2Þ

where SM, the directional loss term, is given by [5,41]:

SM;x ¼ �lf K�1x Us;x � clossK

�1=2x qf jUsjUs;x

SM;y ¼ �lf K�1y Us;y � clossK

�1=2y qf jUsjUs;y

SM;z ¼ �lf K�1z Us;z � clossK

�1=2z qf jUsjUs;z

ð3Þ

The permeability K was considered as anisotropic with Ky =Kx/10 to inhibit flow in y direction (i.e. normal to channel walls).The permeability and the loss coefficient, closs are given by [14,42]

Kx ¼ Kz ¼cw2

c

12; closs ¼ 0:55 1� 5:5

ffiffiffiffiffiffiffiffiffiffiffiwchc

pffiffiffiffiffiffiffiffiffiffihcW

p !

ð4Þ

3.2.3. EnergyEnergy transport inside the porous medium was modeled

through two separate equations for fluid and solid phase thusallowing for finite temperature difference between the two phases[40,41].

r � ðqf Ushf Þ ¼ r � ðckfrhf Þ þ afsAfsðTsp � TfpÞ ð5Þksr � ½ð1� cÞrTsp� � afsAfsðTsp � TfpÞ ¼ 0 ð6Þ

Heat transfer in silicon die and manifold bottom wall is gov-erned by the Laplace equation ðr2Ts ¼ 0Þ. Conjugate heat transferat porous medium-solid interfaces is governed by the followingequations [43]:

�ksððrTÞs;intf �nÞ¼�ckf ððrTÞfp;intf �nÞ�ð1�cÞksððrTÞsp;intf �nÞ ð7ÞTs¼ cTfpþð1�cÞTfs ð8Þ

In our model, we accounted for the temperature dependence ofintensive water properties such as density (qf), dynamic viscosity(lf) and specific heat (cp,f) [44] while silicon properties wereassumed constant ðks ¼ 149 W=mKÞ. The discretized governingequations were solved using Ansys CFX� version 13.0. We imposed1 � 10�6 as the normalized residual and 0.1% as the conservationtarget for our simulations. The computational mesh used and itsgrid independence test are described in SI, Section 1.

3.3. Interfacial heat transfer coefficient

The 2-equation energy model in Eqs. (5) and (6) requires knowl-edge of the interfacial heat transfer coefficient afs. These valueswere obtained through separate CFD simulations for a singlemicrochannel and were compiled in the form of a look-up tableaccording to the following functional relationship (refer SI, Section2 for details):

afs ¼ afsð _VT ;um;wc; xÞ ð9Þ

3.4. Validation of the model

The above-described modeling methodology was validatedagainst experimental measurements for an MMC heat sink withembedded microchannels. The microchannels were of uniform sizewith wc = 35 lm and ww = 25 lm. The chip power map consisted ofq00hs ¼ 150 W=cm2 and q00bg ¼ 20 W=cm2. The chip dimensions werethe same as described in Fig. 2(b) and Table 1. The details of exper-imental measurements are provided in SI, Section 3. Fig. 3 illus-trates that the model is able to successfully capture the coolingperformance of the embedded microchannel structure. The modelis able to capture temperature non-uniformity in the chip with anaccuracy of 0.4 �C over all flow rates. The small discrepancybetween the model and measured results can be attributed tominor deviations of actual flow distribution in the heat sink fromthe modeled distribution and multiple sources of background

Fig. 3. Comparison of measured and modeled (a) DTs,max as a function of flowrate and (b) chip temperature map (for one unit cell) at 1 l/min.

418 C.S. Sharma et al. / Applied Energy 138 (2015) 414–422

radiation that may influence IR (Infrared thermography)measurements.

Fig. 4. Resistance analogy for flow division (a) without throttling zone and (b) withthrottling zone.

4. Optimization of microchannel structure

Minimization of DTs,max and Ts,max can be posed as a constrainedoptimization problem. This section describes the problem formula-tion and the optimization method used.

4.1. Design objectives, parameters and constraints

The microchannel size distribution (channel lengths, widthsand wall widths) and the overall flow rate _VT need to be optimizedin order to minimize Ts,max and DTs,max. The constrained optimiza-tion problem can be formulated as:

Minimize Ts;max and DTs;max ð10Þ

such that

30 6 wc;hs;wc;bg ;wc;th 6 300 ðaÞww;hs;ww;bg ;ww;th P 30 ðbÞðwc þwwÞbg ¼ nðwc þwwÞhs ðcÞðwc þwwÞth ¼ mðwc þwwÞbg ðdÞlth 6 L=7 ðeÞ0:46 6 um 6 0:9 ðfÞ_VT P _VT;min ðgÞDPHSBG 6 DPBG;max ðhÞwhere DPBG;max ¼ DPBGjlth ¼ L=7;wc;th ¼ 30; ww;th ¼ 300DP 6 0:35 bar ðiÞ_Wpumping ¼ DP _VT 6 0:01 _Wchip ðjÞ

ð11Þ

In above equations the constraints for channel widths wc andwall widths ww are in lm. Eqs. (11)(a)–(e) describe the geometricalconstraints and Eqs. (11)(f)–(j) describe the operational constraintson the design. The height of the microchannels hc was fixed at300 lm. In Eqs. (11) (a) and (b), the lower limits of channel andfin widths were fixed based on aspect ratio limit of 10, whichcomes from the process guidelines for deep reactive ion etchingof silicon [45]. Eqs. (11) (c) and (d) specify an integral relationship

between upstream and downstream pitches, for the microchannelstructure in HSBG and BG zone respectively, to ensure uninter-rupted flow.

The limits for um and _VT were chosen such that chip tempera-ture stayed below 85 �C at all operating conditions anywhere inthe heat sink ð _VT;min ¼ 0:2 l=minÞ. Eq. (11) (h) describes the upperlimit on pressure drop through HSBG zone and states that DPHSBG

cannot exceed the maximum possible pressure drop in the BGzone, which will occur when the throttling zone occupies theentire length (L/7) of the unit cell. Eq. (11) (i) is a constraint onthe overall pressure drop [46] and Eq. (11) (j) is a constraintimposed to factor in our aim to achieve the cooling at minimalenergy input in the form of pumping power.

4.2. Optimization methodology

Response surface method (RSM), a robust technique to reducethe computational cost for optimization [47–50], was used toachieve the optimal design. The overall optimization was per-formed in two steps. In the first step, optimal microchannelstructure (wc,hs,opt, wc,bg,opt), flow rate distribution (um,opt) andtotal flow rate ð _VT;optÞ were determined assuming no throttlingzone (i.e. lth = 0) and using RSM (refer SI, Section 4 for details). Inthe second step, the throttling zone configuration was determinedas described below.

C.S. Sharma et al. / Applied Energy 138 (2015) 414–422 419

4.2.1. Throttling zone configurationThe optimized microchannel structure has fine channels over

hotspots in the HSBG zones and coarse channels in the BG zonesðwc;hs;opt < wc;bg;optÞ resulting in a higher resistance to coolant flowin the HSBG zones as compared to the BG zones. As a result, theflow rate distribution would deviate from the optimized distribu-tion um,opt as most the fluid would flow through the coarse chan-nels in BG. The flow division between HSBG and BG zones isexplained through electrical resistance analogy for the flownetwork in Fig. 4. Flow resistances corresponding to optimizedmicrochannel structure in the HSBG and BG zones are representedby RHSBG,opt and RBG,opt respectively. Fig. 4(a) illustrates the case ofno throttling zone discussed above. Since the pressure drop hasto be equal in the two branches corresponding to HSBG and BGzone, RBG,opt < RHSBG,opt leads to _mHSBG < _mBG (i.e. um < um,opt).

In order to realize the desired flow rate distribution, the flowresistance in the BG zone has to be increased, such that pressuredrop on the two branches is equalized (i.e. DPBG = DPHSBG) underthe condition um = um,opt. This was achieved by introducing anadditional flow resistance Rth in the form of a flow throttling zone(refer Fig. 1(c)) in the BG branch (i.e. RBG = RBG,opt + Rth > RHSBG,opt) asshown in Fig. 4(b). This somewhat counterintuitive but importantintroduction of a throttling zone is key to the success of ourapproach. However, the fine channels in throttling zone can causeundesirable overcooling in the BG zone. This effect can be limitedby (i) placing the throttling zones towards outlet slot nozzles and(ii) minimizing their length lth. For this purpose, wc,th was fixed atthe lower limit of 30 lm and a high value for ww,th was selectedsuch that the constraint (11) (d) was satisfied. With all otherparameters fixed, Rth became a function of the length of thethrottling zone lth which was then determined through iterativemicrochannel simulations, such that

DPBG ¼ DPHSBGð¼ DPÞ with um ¼ um;opt;_VT ¼ _VT;opt ð12Þ

Fig. 5. (a) Comparison of the optimized and base design performance forq00hs ¼ 150 W=cm2 and q00bg ¼ 20 W=cm2. First three legends indicate the flow modelused for simulating the base design. (b) x-direction velocity profiles (at micro-channel mid-height) and temperature distribution in the base design (at point B in(a)) and optimized design.

5. Results and discussion

The hotspot-targeted cooling concept was first evaluated usinga strongly non-uniform, exemplar heat flux map with q00hs ¼150 W=cm2 and q00bg ¼ 20 W=cm2 (i.e. a hotspot to back ground heat

flux ratio of q00hs=q00bg ¼ 7:5 and _Wchip ¼ 285 W). Optimization wasperformed using both RSM and detailed microchannel simulations(refer SI, Section 5 for details). The resulting optimal parametersand design performance are enumerated in Table 2. A minimumDTs,max of 3.7 �C and a minimum Ts,max of 29.2 �C was achieved.

5.1. Performance evaluation of the optimized design

The optimized design needs to be compared against a suitablebase case configuration to evaluate the performance. The base casewas selected to represent the limit of cooling with a uniformmicrochannel structure and uniform flow distribution. It consistsof embedded microchannels with lowest allowable channel widthfor best possible thermal performance and lowest allowablewall width to minimize pressure drop for such channels (i.e.

Table 2Optimum design parameters and performance for _q00hs ¼ 150 W=cm2 and _q00bg ¼ 20 W=cm2.

_VT

(l/min)

um wc,hs

(lm)

wc,bg

(lm)

ww;hsww;bgwc;th

(lm)

ww,th

(lm)

1.2 0.705 36 300 30 300

wc,B = ww,B = 30 lm). All other geometrical dimensions were thesame as those for the optimized design (refer Table 1).

Microchannel simulations were used to evaluate the base casedesign at various flow rates. As _VT increased, the microchannelReynolds number increased beyond 2800, the flow became pro-gressively unstable and turbulence modeling was needed to modelthe unstable flow. Two turbulence models, k-omega and ShearStress Transport (SST), were used for an estimate of the base caseperformance at high flow rates [41]. Fig. 5(a) compares thethermo-hydraulic performance of the optimized design againstbase design in terms of _Wpumping vs. DTs,max. Two points on the basedesign performance curve are highlighted: (A) indicates the needfor more than 1300 W of _Wpumping required by the base design tomatch the thermal performance of the optimal design. This isabout 2000 times higher as compared to the optimized designand renders the base design infeasible. Point (B) indicates that

lth

(lm)

DTs,max

(�C)

Ts,max

(�C)

DP

(bar)

_Wpumping_Wchip

(%)

927 3.7 29.2 0.33 0.23

Fig. 6. (a) Comparison of optimized design performance against base design forq00hs ¼ 300 W=cm2 and q00bg ¼ 20 W=cm2. (b) x-direction velocity profiles and tem-perature distribution in base design (B) and optimized design.

420 C.S. Sharma et al. / Applied Energy 138 (2015) 414–422

DTs,max for the optimized design is 61% lower than the base designfor the same _Wpumping . Fig. 5(b) compares the flow fields and theresulting temperature distribution in the base and optimizeddesigns. The hotspot-targeted flow distribution in the optimizednon-uniform microchannel structure design concentrates the heattransfer capacity on the hotspots and leads to the highly uniformtemperature distribution in the chip as compared to base designat point (B).

5.2. Extension of design concept to higher heat fluxes

The hotspot-targeted embedded microchannel cooling conceptcan also be extended to higher levels of q00hs and q00hs=q00bg . We demon-strate the results for another heat flux map consisting ofq00hs ¼ 300 W=cm2 and q00bg ¼ 20 W=cm2 (i.e. q00hs=q00bg ¼ 15) in Table 3and Fig. 6. The optimal design achieves an improvement of 54% inDTs,max while consuming about 600 times lower _Wpumping ascompared to the base design.

The hotspot-targeted designs from Tables 2 and 3 are alsocompared against recent literature heat sinks, designed to reduceDTs,max (see Table 4 and Fig. 7). The optimized designs have thelowest thermal resistance (Rthermal) values and their pumpingpower fractions are comparable to the literature designs. Sinceliterature results for DTs,max have been reported for various levelsof chip heat flux non-uniformity, we normalize DTs,max as depictedbelow for a consistent comparison of various designs.

DTq ¼DTs;max

q00hs

q00bg

� � ð13Þ

Eq. (13) implies that a design is better if it achieves lower temper-ature non-uniformity DTs,max for higher levels of power map non-uniformity q00hs=q00bg (i.e. DTq is lower if DTs,max is low for high q00hs=q00bg

ratio). The optimized designs achieve DTq values of 0.49 and 0.58(nearly an order of magnitude lower than the nearest value of 4.5)and hence compare favorably against the literature designs.

5.3. Complete heat sink performance

The manifold layer design was considered next, to optimallysupply and drain coolant from the microchannels. Conjugate heattransfer simulations were performed for the domain consisting ofa full heat sink unit cell. Details for the full heat sink model aredescribed in SI, Section 6. Fig. 8 shows simulation results for the

relative coolant mass-fraction distribution ðmi�Þ in the microchannel

heat transfer structure, for q00hs ¼ 150 W=cm2 and q00bg ¼ 20 W=cm2,with a conventional straight manifold profile.

mi�

was obtained by calculating the normalized mass fluxthrough the middle y–z plane of the microchannel layer [14],equally divided into N = 50 sections, as:

mi�¼

RAi

qf jUs � njidA1N

PN

RAi

qf jUs � njidAð14Þ

where Ai ¼ ðW � 4wwÞhc=N.It is evident that a higher proportion of coolant flows

through HSBG zone. um, calculated on the basis of this mass flow

Table 3Optimum design parameters and performance for _q00hs ¼ 300 W=cm2 and _q00bg ¼ 20 W=cm2.

_VT

(l/min)

um wc,hs

(lm)

wc,bg

(lm)

ww;hsww;bgwc;th

(lm)

ww,th

(lm)

1.6 0.92 52.5 300 30 630

distribution, is nearly 0.7, the optimum value according to Table 2.Hence, the optimized microchannel structure was able to realizethe required coolant distribution without the need of any externalflow distribution devices.

Further, simulations were performed for a straight manifold(i.e. constant cross-section) and a number of tapered manifoldprofiles with converging inlet and diverging outlet manifoldfor the optimized design corresponding to q00hs ¼ 150 W=cm2 andq00bg ¼ 20 W=cm2 (refer Table 2). It was observed that, unlike inthe case of conventional MMC heat sinks [14], the mass flowdistribution was not sensitive to the shape of the inlet and outletmanifolds. Fig. 9 shows some of the converging inlet manifold pro-files investigated in this work. For each design, outlet manifoldshape is complementary, that is the sum of the widths of inletand outlet manifolds is constant for all the manifold profiles. Asevident, changing the manifold shape has insignificant effect onfraction of flow flowing through HSBG zones, um. This is a verysignificant result, as it points to the robustness of the optimizedmicrochannel structure, which is adequate for efficiently achievingthe required and targeted distribution of the flow.

lth

(lm)

DTs,max

(�C)

Ts,max

(�C)

DP

(bar)

_Wpumping_Wchip

(%)

1480 8.7 38.8 0.3 0.17

Fig. 7. Comparison of optimized designs (red colored text and symbols) againstheat sinks from literature. The numbers in square brackets refer to literaturereferences. (For interpretation of the references to colour in this figure legend, thereader is referred to the web version of this article.)

Fig. 8. Relative mass flux distribution for straight manifold profile forq00hs ¼ 150 W=cm2 and q00bg ¼ 20 W=cm2.

Fig. 9. Non-dimensionalized inlet manifold profiles. Wm is half-width of inletmanifold. Solid line is straight manifold profile while dashed lines are converginginlet manifold profiles. Numbers next to each manifold profile indicate thecorresponding um value.

Table 4Comparison of optimized designs against literature heat sinks.

aApproximate values/values back calculated from available information.

C.S. Sharma et al. / Applied Energy 138 (2015) 414–422 421

6. Design rules for hotspot-targeted microchannel cooling

We recommend the following general design rules for achievinghotspot-targeted cooling of multicore microprocessors. With

embedded manifold microchannel cooling concept as the basis andchip power map divided into parallel HSBG and BG zones, thedesign consists of:

(1) Inlet slot nozzles positioned over cores and outlet slot noz-zles positioned over background to concentrate impinge-ment cooling on the hotspots.

(2) Smallest feasible microchannel wall width to minimize theoverall pressure drop.

(3) Fine channels over cores, coarse channels over backgroundalong with flow distribution between HSBG and BG zonesselected to achieve higher heat transfer coefficient over hot-spots and thus minimize the chip temperature gradient.

(4) Highest allowable total flow rate that satisfies pressure dropand pumping power constraints.

(5) Flow throttling channels towards outlet slot nozzles in theBG zones to realize the flow distribution selected in (3)above.

(6) All streamwise microchannel pitches related through inte-gral multiples to ensure uninterrupted flow in HSBG andBG zones.

7. Conclusions

A novel concept for energy efficient, hotspot-targeted embed-ded liquid cooling of multicore microprocessors was presented.The design consists of fine channels over the hotspots, coarse chan-nels over the background and introduces a flow throttling zone toregulate flow in different regions. Optimized hotspot-targeted

422 C.S. Sharma et al. / Applied Energy 138 (2015) 414–422

designs were computationally obtained, for two highly non-uniform, exemplar heat flux maps: (a) q00hs ¼ 150 W=cm2 andq00bg ¼ 20 W=cm2 and (b) q00hs ¼ 300 W=cm2 and q00bg ¼ 20 W=cm2.Minimum chip temperature non-uniformities of 3.7 and 8.7 �C,respectively were achieved for the two cases, while adhering tostrict, industrially relevant constraints on pressure drop (<0.35 bar)and pumping power (<1% of chip power). It was shown that theoptimized designs compare very favorably against a base designselected to represent the limit of cooling with uniform embeddedmicrochannels and uniform flow distribution. Simulations for a fullheat sink highlighted the flow distribution behavior of the opti-mized microchannel structure and insignificant effect of the man-ifold shape on the cooling performance of the heat sink. Lastly,general rules for design of hotspot-targeted microchannel heatsinks were described.

Appendix A. Supplementary material

Supplementary data associated with this article can be found, inthe online version, at http://dx.doi.org/10.1016/j.apenergy.2014.10.068.

References

[1] Moore GE. Cramming more components onto integrated circuits. Electronics1965;38:114–7.

[2] Dennard RH, Cai J, Kumar A. A perspective on today’s scaling challenges andpossible future directions. Solid-State Electron 2007;51:518–25.

[3] JEDEC. Failure mechanisms and models for semiconductor devices. In: JEDEC,editor; 2010. p. 1–9, 68–72.

[4] Sharma CS, Zimmermann S, Tiwari MK, Michel B, Poulikakos D. Optimalthermal operation of liquid-cooled electronic chips. Int J Heat Mass Transfer2012;55:1957–69.

[5] Sharma CS, Tiwari MK, Michel B, Poulikakos D. Thermofluidics and energeticsof a manifold microchannel heat sink for electronics with recovered hot wateras working fluid. Int J Heat Mass Transfer 2013;58:135–51.

[6] Marcinichen JB, Olivier JA, Oliveira Vd, Thome JR. A review of on-chip micro-evaporation: experimental evaluation of liquid pumping and vaporcompression driven cooling systems and control. Appl Energy 2012;92:147–61.

[7] Marcinichen JB, Wu D, Paredes S, Thome JR, Michel B. Dynamic flow controland performance comparison of different concepts of two-phase on-chipcooling cycles. Appl Energy 2014;114:179–91.

[8] Weng Y-C, Cho H-P, Chang C-C, Chen S-L. Heat pipe with PCM for electroniccooling. Appl Energy 2011;88:1825–33.

[9] Mahmoud S, Tang A, Toh C, Al-Dadah R, Soo SL. Experimental investigation ofinserts configurations and PCM type on the thermal performance of PCM basedheat sinks. Appl Energy 2013;112:1349–56.

[10] Escher W, Brunschwiler T, Shalkevich N, Shalkevich A, Burgi T, Michel B, et al.On the cooling of electronics with nanofluids. J Heat Transfer 2011;133:051401-.

[11] Wang X-D, Bin A, Xu J-L. Optimal geometric structure for nanofluid-cooledmicrochannel heat sink under various constraint conditions. Energy ConversManage 2013;65:528–38.

[12] Tuckerman DB, Pease RFW. High-performance heat sinking for VLSI. IEEEElectron Dev Lett 1981;2:126–9.

[13] Lee P-S, Garimella SV, Liu D. Investigation of heat transfer in rectangularmicrochannels. Int J Heat Mass Transfer 2005;48:1688–704.

[14] Escher W, Michel B, Poulikakos D. A novel high performance, ultra thin heatsink for electronics. Int J Heat Fluid Flow 2010;31:586–98.

[15] Kandlikar SG, Bapat AV. Evaluation of jet impingement, spray andmicrochannel chip cooling options for high heat flux removal. Heat TransfEng 2007;28:911–23.

[16] Hetsroni G, Mosyak A, Segal Z, Ziskind G. A uniform temperature heat sink forcooling of electronic devices. Int J Heat Mass Transfer 2002;45:3275–86.

[17] Rubio-Jimenez CA, Kandlikar SG, Hernandez-Guerrero A. Numerical analysis ofnovel micro pin fin heat sink with variable fin density. IEEE Trans ComponPack Technol 2012;2:825–33.

[18] Barrau J, Omri M, Chemisana D, Rosell J, Ibañez M, Tadrist L. Numerical studyof a hybrid jet impingement/micro-channel cooling scheme. Appl Therm Eng2012;33–34:237–45.

[19] Xie G, Zhang F, Sundén B, Zhang W. Constructal design and thermal analysis ofmicrochannel heat sinks with multistage bifurcations in single-phase liquidflow. Appl Therm Eng 2014;62:791–802.

[20] Xie G, Chen Z, Sunden B, Zhang W. Numerical analysis of flow and thermalperformance of liquid-cooling microchannel heat sinks with bifurcation.Numer Heat Transfer Part A: Appl 2013;64:902–19.

[21] Lin L, Chen Y-Y, Zhang X-X, Wang X-D. Optimization of geometry and flow ratedistribution for double-layer microchannel heat sink. Int J Therm Sci2014;78:158–68.

[22] Esmaeilzadeh H, Blem E, Amant Renée St, Sankaralingam K, et al. Powerlimitations and dark silicon challenge the future of multicore. ACM TransComput Syst 2012;30:1–27.

[23] Sinharoy B, Kalla R, Starke WJ, Le HQ, Cargnoni R, Van Norstrand JA, et al. IBMPOWER7 multicore server processor. IBM J Res Dev 2011;55:1–29.

[24] Chowdhury I, Prasher R, Lofgreen K, Chrysler G, Narasimhan S, Mahajan R,et al. On-chip cooling by superlattice-based thin-film thermoelectrics. NatNano 2009;4:235–8.

[25] Paik PY, Pamula VK, Chakrabarty K. Adaptive cooling of integrated circuitsusing digital microfluidics. IEEE Trans Very Large Scale Integr (VLSI) Syst2008;16:432–43.

[26] Lee P-S, Garimella SV. Hot-spot thermal management with flow modulation ina microchannel heat sink. Florida [USA]: ASME International MechanicalEngineering Congress and Exposition Orlando, ASME; 2005. p. 643–7.

[27] Lee YJ, Lee PS, Chou SK. Hotspot mitigating with obliquely finnedmicrochannel heat sink–an experimental study. IEEE Trans Compon PackTechnol 2013;3:1332–41.

[28] Brunschwiler T, Rothuizen H, Paredes S, Michel B, Colgan E, Bezama P.Hotspot-adapted cold plates to maximize system efficiency. In: 15thInternational workshop on thermal investigations of ICs and systems, 2009THERMINIC 2009; 2009. p. 150–6.

[29] Dang B, Bakir MS, Meindl JD. Integrated thermal-fluidic I/O interconnects foran on-chip microchannel heat sink. IEEE Electron Device Lett 2006;27:117–9.

[30] Kandlikar SG, Grande WJ. Evaluation of single phase flow in microchannels forhigh heat flux chip cooling – thermohydraulic performance enhancement andfabrication technology. Heat Transf Eng 2004;25:5–16.

[31] Mingyuan Z, Huang ZR. Design of on-chip microchannel fluidic coolingstructures. In: Electronic components and technology conference, 2007 ECTC‘07 Proceedings 57th 2007; 2007. p. 2017–23.

[32] Etessam-Yazdani K, Asheghi M, Hamann HF. Investigation of the impact ofpower granularity on chip thermal modeling using white noise analysis. IEEETrans Compon Pack Technol 2008;31:211–5.

[33] Sharma S, Chung-Hsing H, Wu-chun F. Making a case for a Green500 list. In:Parallel and distributed processing symposium, 2006 IPDPS 2006 20thInternational; 2006. p. 8.

[34] Koomey JG. Growth in data center electricity use 2005 to 2010. Oakland(CA): Analytics Press; 2011.

[35] Garimella SV, Persoons T, Weibel J, Yeh L-T. Technological drivers in datacenters and telecom systems: multiscale thermal, electrical, and energymanagement. Appl Energy 2013;107:66–80.

[36] Almoli A, Thompson A, Kapur N, Summers J, Thompson H, Hannah G.Computational fluid dynamic investigation of liquid rack cooling in datacentres. Appl Energy 2012;89:150–5.

[37] Copeland D, Behnia M, Nakayama W. Manifold microchannel heat sinks:isothermal analysis. IEEE Trans Compon Pack Technol, Part A 1997;20:96–102.

[38] Ryu JH, Choi DH, Kim SJ. Three-dimensional numerical optimization of amanifold microchannel heat sink. Int J Heat Mass Transf 2003;46:1553–62.

[39] Escher W, Brunschwiler T, Michel B, Poulikakos D. Experimental investigationof an ultrathin manifold microchannel heat sink for liquid-cooled chips. ASMEJ Heat Transfer 2010;132:081402-1–081402-10.

[40] Kim SJ, Hyun JM. 5 – A porous medium approach for the thermal analysis ofheat transfer devices. In: Ingham DB, Pop I, editors. Transport phenomena inporous media III. Oxford: Pergamon; 2005. p. 120–46.

[41] Ansys. ANSYS CFX-solver Theory Guide. ANSYS Inc: Canonsburg, PA; 2009.[42] Kim SJ, Kim D. Forced convection in microstructures for electronic equipment

cooling. ASME J Heat Transfer 1999;121:639–45.[43] Betchen L, Straatman AG, Thompson BE. A nonequilibrium finite-volume

model for conjugate fluid/porous/solid domains. Numer Heat Tranf A – Appl2006;49:543–65.

[44] CRC. Handbook of Chemistry and Physics. 9th ed. CRC; 2013.[45] Madou MJ. Fundamentals of microfabrication: the science of miniaturization.

2nd ed. Boca Raton (FL): CRC Press; 2002.[46] Colgan EG, Furman B, Gaynes M, Graham WS, LaBianca NC. A practical

implementation of silicon microchannel coolers for high power chips. IEEETrans Compon Pack Technol 2007;30:218–25.

[47] Chiang K-T, Chang F-P. Application of response surface methodology in theparametric optimization of a pin-fin type heat sink. Int Commun Heat MassTransf 2006;33:836–45.

[48] Husain A, Kim K-Y. Optimization of a microchannel heat sink withtemperature dependent fluid properties. Appl Therm Eng 2008;28:1101–7.

[49] Anderson MJ, Whitcomb PJ. RSM simplified: optimizing processes usingresponse surface methods for design of experiments. New York: ProductivityPress; 2005.

[50] Anderson MJ, Whitcomb PJ. DOE simplified: practical tools for effectiveexperimentation. New York: Productivity Press; 2007.