Segmentation with Area Constraints - Computer Science segmentation approaches typically incorporate weak regularity conditions such as boundary length ... Segmentation, area-constraint,

Segmentation with Area Constraints

Marc Niethammera,b, Christopher Zachc

aDepartment of Computer Science, University of North Carolina (UNC) at Chapel Hill, USAbBiomedical Research Imaging Center, School of Medicine, UNC Chapel Hill, USA

cMicrosoft Research, Cambridge, UK

Abstract

Image segmentation approaches typically incorporate weak regularity conditions such as boundary lengthor curvature terms, or use shape information. High-level information such as a desired area or volume, ora particular topology are only implicitly specified. In this paper we develop a segmentation method withexplicit bounds on the segmented area. Area constraints allow for the soft selection of meaningful solutions,and can counteract the shrinking bias of length-based regularization. We analyze the intrinsic problems ofconvex relaxations proposed in the literature for segmentation with size constraints. Hence, we formulatethe area-constrained segmentation task as a mixed integer program, propose a branch and bound methodfor exact minimization, and use convex relaxations to obtain the required lower energy bounds on candidatesolutions. We also provide a numerical scheme to solve the convex subproblems. We demonstrate themethod for segmentations of vesicles from electron tomography images.

Keywords: Segmentation, area-constraint, branch and bound, alternating direction method of multipliers

1. Introduction

Image segmentation is a fundamental task in im-age analysis. Consequentially, a large number ofsegmentation methods have been developed rang-ing from local thresholding to methods using statis-tical models of shape variation (Pham et al. (2000);Sonka et al. (2008)). The simplest available seg-mentation methods rely on local pixel-by-pixel seg-mentation decisions such as Otsu thresholding ormethods based on clustering. These fully-local de-cisions are often not sufficient and because theyneglect spatial dependencies, they are sensitive tonoise and not directly applicable if an object is de-fined by its boundary surface only (e.g., if onlythe cell membrane or a cell membrane surrogateis imaged, but an image of the entire cell is de-sired). To overcome these limitations, non-localapproaches have been proposed based on intelli-gent local merging decisions or by formulating opti-mization problems incorporating spatial dependen-cies. The former class of methods encompasses re-gion growing approaches such as the popular water-shed segmentation (Sonka et al. (2008)). The lat-ter class of methods includes active-contours and -surfaces (Sapiro (2001)) as well as general paramet-

ric models which may use statistical informationon shape and/or appearance (Cootes et al. (2001);Pizer et al. (2003)).

When the object segmentation task is highlystructured (i.e., expected shape variations are rea-sonably small and the approximate number and lo-cation of the objects are known) shape- or atlas-based segmentation methods are highly success-ful (Rohlfing et al. (2005)). However, for less struc-tured cases these methods are not applicable. Inmicroscopy, for example, images often contain hun-dreds or thousands of cells, cell nuclei, or organelles,with possibly large variations in shape and a pri-ori unknown locations. While local thresholdingor active-contour-type models may be applied insuch cases, they are often too generic, too sensitiveto noise, or require the judicial placement of seedpoints to assure an appropriate segmentation resultto avoid over- or under-segmentations.

If shape- or atlas-based segmentation methodsare too restrictive, and if general purpose segmen-tation methods such as active contours, region-growing or thresholding are not restrictive enoughfor a particular segmentation task, the question ofhow to incorporate additional domain information

Preprint submitted to Medical Image Analysis December 5, 2012

into a segmentation that lies between these twoextremes arises. A possible option is to use in-formation about simple geometric properties. Inthis paper we explore an approach for a segmen-tation with constraints on the segmentation area.Such a method can counteract potential leakageor shrinkage biases in a principled way. Such bi-ases can be observed, for example, for active con-tour (Sapiro (2001)) or graph cut (Boykov andFunka-Lea (2006)) segmentations when boundaryregularity is encouraged by penalizing a weightedlength of the segmentation boundary. Area con-straints may not be appropriate for all biomedicalsegmentation tasks; however there are a large num-ber of problems in which reasonable area or vol-ume intervals are known a-priori. Our objective inthis paper is not to perform an actual study for aparticular biological problem, but rather to demon-strate the behavior of a segmentation method witharea-constraints on realistic image data. We useelectron tomography datasets of synaptic vesiclesand of double-membrane vesicles (DMVs) impli-cated in the SARS-coronavirus (severe acute respi-ratory syndrome coronavirus) replication (Knoopset al. (2008)).

Many recent segmentation approaches are formu-lated such that the optimization problems becomeconvex so that globally optimal solutions can be ob-tained (Appleton and Talbot (2006); Bresson et al.(2007)) or so that they can be solved with discretesolution methods, such as graph-cuts (Boykov andFunka-Lea (2006)). While area-constraints can for-mally easily be added to the optimization prob-lems for segmentation, solving the problems is hard.However, if finding a globally optimal solution isnot of concern and a good initial guess for a solu-tion is available, one can resort to standard meth-ods from constrained optimization. For example, acurve evolution approach with an area penalty canbe used (Ayed et al. (2008)). Proposed numericalsolution approaches to obtain a global optimum ora good approximation

• are limited to problems with small numbers ofvariables (Ji (2004))

• or require long computation times (Dahl andFlatberg (2007)),

• use solution heuristics (Kernighan and Lin(1970)),

• or use various forms of relaxations of the orig-inal problem to facilitate computations: e.g.,

spectral relaxations (Olsson et al. (2008)),semidefinite programming (Keuchel et al.(2003); Lisser and Rendl (2003); Hager et al.(2009)), or variational inference approxima-tions (Kropotov et al. (2010)).

Approaches have generally focused on equalityconstraints (i.e., exact size) in formulation (Limet al. (2010); Eriksson et al. (2011); Falkner et al.(1994); Ayed et al. (2008)) or for testing (Hageret al. (2009)). However, equality constraints haveonly limited applicability when the exact object sizeis not known beforehand or when it is a desiredmeasurement (as is frequently the case in biomed-ical imaging), because it would bias the segmen-tation towards the chosen area. We therefore for-mulate the segmentation problem with inequalityconstraints on the segmentation area.

Section 2 introduces the area-constrained seg-mentation problem. Section 3 outlines our solutionapproach. Sections 4, 5 and 6 discuss its numeri-cal solution. Segmentation results on real electrontomography images demonstrate the utility of themethod in Section 7. The paper concludes with asummary and a discussion of future work.

2. Optimization problem

Our objective is a binary segmentation of an im-age into foreground and background. Without lossof generality, we consider two-dimensional imageshere1. Markov random field models with Gibbs en-ergies using first and second order cliques have beenparticularly popular for image segmentation (Li(2009)) and can be exactly minimized under certainconditions (Kolmogorov and Zabin (2004)) for ex-ample by using graph cuts. Solutions are typicallybased on the minimal cut theorem (Ford and Fulk-erson (1956)) relating the minimum cut in a graphto the maximum flow through the graph. Hence, byforming an appropriate graph and solving the max-imum flow problem the segmentation solution canbe obtained. The segmentation algorithm we ana-lyze and extend is the partial differential equationformulation of the maximum flow problem (Apple-ton and Talbot (2006)), which has equally broadapplication for image segmentation.

1The overall algorithm and its analysis extends to higherdimensions. We use two-dimensional terminology in the re-mainder of the paper only to simplify the presentation, e.g.,boundary curve instead of boundary surface.

2

As is customary for image segmentation meth-ods based on energies of Gibbs-type, we allow theoptional specification of seed points or areas whichexplicitly enforce a particular labeling (foregroundor background) for the seeds. Our segmentation for-mulation is an extension of the convex formulationof the active contour method and related segmen-tation methods such as the Chan-Vese segmenta-tion model (Bresson et al. (2007); Chan and Vese(2001)).

To avoid the segmentation of very small struc-tures which likely represent noise and noisy bound-aries, almost all energy-based methods penalize the(weighted) length of the boundary curve separat-ing foreground from background. Most commonly,the length of the boundary curve is added to thesegmentation energy. This introduces a well-knownshrinking bias towards shorter boundary curves andtherefore frequently leads to undersegmentations.Our goal is to add constraints on the segmentationarea, to counteract the shrinking bias and to allow“tuning” of the segmentation algorithm to the ex-pected size of the objects to be segmented.

While our area-constrained extension is devel-oped in the context of maximum-flow-based seg-mentation, the solution strategy itself is genericand expected to be applicable also to other seg-mentation models. For example, a similar solutionstrategy might be useful for segmentation methodswhich do not penalize boundary curve length di-rectly, but instead a ratio between boundary lengthand enclosed areas (Grady and Schwartz (2006); Shiand Malik (2000)). Such methods do not exhibitthe same shrinking bias, but also lack control overthe obtained segmentation area.

The general optimization problem for area-constrained segmentation we consider is

argminu

E(u), s.t. A(u) ∈ [Al, Au], (1)

where u = us|s ∈ X; us ∈ 0, 1,

and

us = 0, s ∈ T ,us = 1, s ∈ S.

Here, u is an indicator function denoting foreground(u = 1) and background (u = 0) classes of the seg-mentation defined over the the set of spatial loca-tions X , where s denotes a spatial location, us avalue at location s and u is the union of these val-ues for all s ∈ X , i.e., u ∈ 0, 1|X |, where |X | de-notes the number of considered spatial locations.E(u) is an energy function encoding the desired

properties of a segmentation, A(u) indicates thearea covered by a segmentation u, S and T de-note the foreground and background seed sets re-spectively2, and Al and Au are lower and upperbounds for the segmentation area. This is an inte-ger program (Nemhauser and Wolsey (1988)) sinceus ∈ 0, 1.

For image segmentation the number of variables,us, corresponds to the number of pixels. A directsolution of problem (1) with integer-programmingmethods is typically computationally tractable onlyfor very small images. For segmentations with-out area-constraints various relaxations of the origi-nal labeling problem have therefore been proposed.As an illustrative example, consider the continu-ous maximum flow approach (Appleton and Talbot(2006)) in which

Emf (u) =∑s

gs‖∇su‖+ ρsus, s.t. us ∈ [0, 1],

(2)subject to the same seedpoint constraints as in (1) isminimized. Here, g > 0 denotes an edge-weightingterm and ρ a regional bias which allows for the inte-gration of local likelihoods of an element s to belongto the foreground or the background3. The key dif-ference in structure is to allow us ∈ [0, 1] whichrenders the optimization problem convex, becauseit is now defined over a convex domain, u ∈ [0, 1]|S|.A globally optimal solution can then efficiently beobtained. For the continuous maximum flow prob-lem, the optimal solution will be essentially binaryregardless the convex relaxation. This means thata minimizer, u∗ of (2) may not necessarily be bi-nary, but any thresholded uθ∗s = 1[θ,1](u

∗s) is bi-

nary and globally optimal with E(u∗) = E(uθ∗)for θ ∈ (0, 1) (Appleton and Talbot (2006)). Here,

2The foreground and background seeds may be absorbedinto E, but keeping them separate will be useful for thebranch and bound solution approach.

3This is a very general energy form which can ex-press many highly popular segmentation models, suchas active contour and surface models, Chan-Vese seg-mentation, and segmentation models with general region-based likelihoods. For example, any energy of the formE(C) =

∫Ω1−log(p1(d(x)|θ1) dx+

∫Ω2−log(p2(d(x)|θ2) dx+∫

C g(s) ds can be written in the form of Equation 2. Here, Cis the boundary curve separating the foreground region, Ω1,from the background region, Ω2; d(x) denotes available dataat spatial location x, θi are given parameters, which typi-cally parametrize the likelihoods pi, g(s) > 0 and s denotesarc-length. Hence, ρs in Equation 2 can be interpreted asthe logarithm of the likelihood ratios in the foreground andthe background regions at location s.

3

1S(x) is the indicator function which returns 1 if xis in S and 0 otherwise.

Unfortunately, this relaxed solution is no longerguaranteed to be essentially binary when adding thearea-constraint by inequality constraints

∑s us ≥

Al,∑s us ≤ Au. The “area” of a segmentation

is defined as A(u) :=∑s us. In addition, the

segmentation method can become “blind” to thetrue optimal integer solution as illustrated in Sec-tion 2.1, which precludes the possibility of find-ing good approximate solutions for the integer pro-gram by thresholding the relaxed solution. A dif-ferent solution strategy is therefore needed for area-constrained segmentation. We resort to branch andbound (see Section 6).

2.1. Problem with the relaxed solution

Assume there are strong gradients along theboundary of concentric, non-intersecting shapes.For example, several circles with increasing radiior squares with increasing side lengths with smallweights g. Assume that the weights are chosen suchthat all discontinuities of the resulting segmentation(for the original problem and its relaxation) occuronly at these shape boundaries. This can alwaysbe achieved by assigning sufficiently large weightsoutside the desired boundaries. These concentricshapes are indexed by a scale parameter r, e.g. theradius of a circle or the diagonal of a square. Theshape itself is not important, but only that the areaof the shape is cAr

2 and its circumference is cLr forsuitable constants cA and cL. We drop these con-stants without loss of generality in what follows.We would like the segmentation to snap into suc-cessively larger shapes when increasing the lowerbound on the area. The following counter-exampleshows that this cannot be assured and therefore theconvex relaxation (with us ∈ [0, 1]) is insufficient toobtain solutions to the area-constrained segmenta-tion problem.

Consider three concentric shapes with scales r1 <r2 < r3 (see Figure 1) and a sufficiently large seedregion within the inner-most shape so that the un-constrained problem results in the segmentation ofthe smallest shape. The segmentation energy (2)is proportional to ri (for ρs = 0) and the area isproportional to r2i . Without loss of generality, setr1 = 1. To obtain the middle shape from the seg-mentation, we enforce Al ∈ (r21, r

22) = (1, r22). The

segmentation energy of the desired shape (for aninteger-solution) is Eint = r2. Under the relaxedsegmentation model the optimal solution needs to

Image seed S seed T gx gy

A ≥ 20 A ≥ 40 A ≥ 80 A ≥ 100 A ≥ 140

Figure 1: Three squares segmentation experiment (top): im-age to be segmented, foreground seed set S, backgroundseed set T and the cost g in the x and y direction (gx =1/(1 + 50Ix) and gy = 1/(1 + 50Iy)) respectively (from topleft to top right). The cost is illustrated with directionaldependence, because we use the 1-norm, gs‖∇us‖1, to dis-cretize the weighted total-variation term. Linear program-ming (LP) solution for different lower area bounds (middle);solution of integer program (bottom). The integer programis by construction binary and is able to capture all threeconcentric squares (with respect to the gradient magnitude).The LP solution is blind to the middle square and immedi-ately “bleeds out” into the biggest square once the desiredarea is larger than the smallest square at the center.

occur at r2 or r3. Since smaller values for u willlead to smaller overall energy values, the optimalrelaxed solution will have A(u) = Al. Therefore,for a jump at ri the uniform fractional value for uwhich fulfills the area constraint exactly will be

u =Al − r21r2i − r21

=Al − 1

r2i − 1, i ∈ 2, 3.

The energy values (for a jump from 1 to u at r1 = 1and from u to 0 at ri) are

Ei = r1(1− u) + riu

=r2i − 1 + (Al − 1)(ri − 1)

r2i − 1=ri +Alri + 1

.

But then Al > 1 and r3 > r2 by assumption leadsto E3 < E2. This shows that the middle shapecannot be recovered by thresholding and the frac-tional solution has a lower energy than the solu-tion for the integer program. Figure 1 illustratesthe difference between the relaxed and integer so-lutions for successively larger lower bounds on thearea for concentric squares with foreground seedsat the center of the image and background seedsat the image boundary. As predicted, the relaxedsolution is blind to the middle square and simplyuniformly increases the fractional values of u with

4

increasing A. In contrast, the integer solution isable to capture all three squares.

2.2. Proposed formulation

While the relaxed solution is not suitable forarea-constrained segmentation by itself it can beused to obtain lower bounds for the integer pro-gram. Instead of directly enforcing u ∈ 0, 1we formulate the optimization problem as a mixedinteger nonlinear program (MINLP) (Hijazi et al.(2009))

minu

Eminlp(u),

Eminlp(u) =∑s

gs‖∇us‖+ ρsus,

s.t. A(u) ∈ [Al, Au];us ∈ [0, 1];us = 0, s ∈ T ,us = 1, s ∈ S,

us = bk, s ∈ Bk, bk ∈ 0, 1, ∀ku essentially binary, (3)

which augments the maximal flow formulation (2)by selection variables bk and areas Bk, which al-low selection of additional foreground and back-ground seeds. For practical segmentation problemsfull control over all pixels is in many cases not nec-essary. Instead, it is desirable to obtain a goodapproximation to the original optimization prob-lem while controlling the computational complex-ity of the method. Hence, we replace the control ofindividual pixels by the control of coarse selectionareas Bk. Since we solve this problem by branch-and-bound the resulting reduction in the number ofinteger variables reduces the effort to compute thesolution drastically (because it reduces the size ofthe branch and bound tree). The original integerprogram (1) with the non-relaxed maximum flowenergy is recovered if the Bk correspond to individ-ual image pixels s /∈ S∪T . If desired, maximal flowformulations with direction-dependent costs couldbe used (Zach et al. (2009b,a)). For formulationswith only a lower area bound, the last conditionis replaced by us ≥ bk (and similarly for only anupper bound).

The essentially binary property is a consequenceof the underlying continuous max-flow solutionsand means that given an optimal (not-necessarilybinary) u an equally optimal solution can be foundby thresholding u for any θ ∈ (0, 1). I.e., we onlywant to accept values for the selection variables

bk which result in essentially binary solutions andtherefore indicate that they were selected appropri-ately to avoid the problems discussed in Section 2.1.

Intuitively, the maximal flow approach yields anessentially binary solution even when an area con-straint is present if it is sufficiently constrainedby seed points. For example, imposing a lowerarea constraint for a max-flow-based segmentationis trivial if the number of foreground seed pointsis larger or equal to the lower area bound. Thenthe constraint is essentially inactive and “invisible”to the segmentation algorithm. The difficulty liesin finding these seed points (without requiring auser to provide close to the final segmentation as in-put). Integer programming solves this problem byintelligent pixel-by-pixel searching. However, eventhough the existing search methods (Nemhauserand Wolsey (1988)) avoid the combinatorial explo-sion inherent to a brute-force approach, search treeswill still get extremely large even for moderatelysmall problems unless special problem structure canbe exploited. We control the combinatorial explo-sion instead by an appropriate, coarse choice of se-lection areas.

We would like the solution to be robust to thechoice of the selection areas Bk. The solutionboundaries are expected to be located close towhere their cost is low, i.e., where gs is small.Hence, we try to avoid placing the boundaries ofselection regions there and let the remaining pix-els not covered by any selection region snap intothe best boundary location. We use homogeneousimage regions for the Bk, which can be derivedfrom super-pixels (Vedaldi and Soatto (2008); Co-maniciu and Meer (2002)) or from an image over-segmentation using a watershed method (Vincentand Soille (1991)). Formulation (3) is more gen-eral than a direct super-pixel segmentation (e.g.,one can use seed regions covering only a subset ofthe image to guide the segmentation while havingcomplete representational freedom close to puta-tive segmentation boundaries). To define the Bk,we use quick-shift (Vedaldi and Soatto (2008)) tofind super-pixels and erode them so that they donot touch the potential segmentation boundaries.Quick-shift is an efficient mode-seeking algorithmbased on medoid shift (conceptually similar to thepopular mean-shift segmentation algorithms Co-maniciu and Meer (2002)). It provides a tuning pa-rameter to control under- and over-fragmentationof modes and can therefore be used to indirectlycontrol the number of selection regions to be de-

5

tected. Our method is not dependent on quick-shift, and other clustering methods such as mean-shift or k-means (Jain et al. (1999)) could be substi-tuted. Figure 2 shows an illustration of the selectionregions. In addition to the max-flow method ad-dressed in this paper we expect this approach of fa-cilitating area-constraints through selection regionsalso to be generally useful for other segmentationmethods.

Note that when only an upper or a lower boundon the area are present, the segmentation can berobust even to selection areas crossing the integer-programming-optimal solution because we replacethe equality constraint us = bk by an inequality(us ≥ bk or us ≤ bk respectively) effectively result-ing in “don’t-care” selection areas. Segmentationboundaries can pass through such “don’t care” se-lection areas if desired. Specifically, if only a lower-bound is imposed, then the selection regions drivethe actual segmentation values us through the in-equality us ≥ bk. Therefore setting a selection re-gion to 1 forces the segmentation value us to be 1for s ∈ Bk. However, setting a selection region to 0amounts to leaving the segmentation values us free.Hence, the solution will neither be forced to 0 or 1in such an area and can be completely determinedpixel-wise by the underlying image. The segmen-tation will be robust to selection regions that crosssegmentation boundaries as long as there are a suf-ficient number of selection regions on the inside ofan object that can be set to 1 so that the segmen-tation naturally “snaps” into the desired location.However, when we enforce a lower and an upperbound, we need to be able to increase and decreasethe natural size of an unconstrained segmentationby setting regions to 1 or 0 respectively. In this case,selection regions crossing segmentation boundarieswill matter because they have to be set to either0 or 1. Consequentially, enforcing an upper and alower bound may produce results which are worsethan enforcing only a lower bound. This problemcould easily be avoided by moving from binary toternary selection variables (0: set to zero, 1: set toone, 2: don’t care). This would leave the overallapproach intact, but would result in a slightly dif-ferent branch and bound implementation, which isnot our focus here.

3. Outline of Solution Approach

Solving the MINLP (3) involves the computa-tion of the optimal binary selection variables, bk.

Figure 2: Illustration of selection regions for a concentriccircle example. Left: original image. Right: automaticallydetermined selection regions using quick-shift followed by anerosion. Different colors represent different regions. Dark-blue indicates regions not covered by the selection regions,for which pixels are not controlled by selection regions andcan therefore faithfully represent segmentation boundaries.

A brute-force approach enumerating all possiblecombinations bk ∈ 0, 1|b| (where |b| denotesthe number of selection areas) is prohibitive forall but the most simple general integer program-ming problems. We therefore use branch andbound (Nemhauser and Wolsey (1988)) to solve (3),which determines the optimal values of the selec-tion variables bk by building a search tree. Eval-uation of the full search tree (feasible only forsmall problems) is avoided by guiding the search to-wards promising solution candidates and discardingbranches which can provably not lead to an optimalsolution.

For the MINLP energy (3) we introduce the re-laxed MINLP energy as

Erelaxed(u) =∑s

gs‖∇us‖+ ρsus,

s.t. A(u) ∈ [Al, Au];us ∈ [0, 1];us = 0, s ∈ T ,us = 1, s ∈ S,

us = bk, s ∈ Bk, bk ∈ 0, 1, k ∈ K

where we dropped the essentially binary conditionand removed some of the selection regions. Here,K is the set containing the indices of the selectionregions which are used in the particular relaxed so-lution, with all other bk free4. We have

E∗relaxed(p) ≤ Erelaxed(ur) ≤ Eminlp(ur)

4As for Equation 3, we allow inequalities for the selectionregions when only enforcing lower or upper bounds. Theenergies change correspondingly.

6

where E∗relaxed denotes the dual energy to Erelaxed,p is the dual variable to u, and ur is a feasible candi-date solution to the relaxed optimization problem.The first inequality holds, because a dual energyis never larger than the corresponding primal en-ergy. The second inequality holds, because Erelaxedremoves constraints from Eminlp and therefore willeither have a smaller energy value than Eminlp (if urviolates some constraints of Eminlp) or will be equalto it. If ur is a feasible solution for Eminlp the en-ergy value will be finite, otherwise it will be infinite.Hence, if within the search tree we find a relaxedsolution such that E∗relaxed(p) > E(u∗best) whereu∗best is the current best feasible solution known forEminlp we can prune the search tree for ur, i.e.,we no longer need to look at any solutions for itsfree selection variables bk, because they could onlycause higher energies. A search branch can furtherbe terminated if it results in a feasible integer solu-tion.

We use the alternating direction method of mul-tipliers (ADMM) (Sections 4 and 5) to compute so-lution candidates, ur, to the relaxed problem andshow how to compute a dual energy at every it-eration step of the optimization algorithm5. Sec-tion 6 discusses how to use the relaxed dual andprimal energies within the branch and bound so-lution framework and how to obtain finite-valuedrelaxed dual energies and integer-feasible solutionsfrom the ADMM variables. See Figure 3 for agraphical overview.

4. Alternating DirectionMethod of Multipliers

A possible numerical scheme is to perform a stan-dard primal/dual gradient descent/ascent (Rein-bacher et al. (2010)). While simple, these meth-ods tend to oscillatory behavior and require costlyprojections at every iteration step to fulfill the area-constraint6.

We instead use the alternating direction methodof multipliers (ADMM) (Boyd et al. (2010)) for the

5The iterative solution can be terminated prior to con-vergence if the dual energy is larger than the best integer-feasible primal energy, Eminlp(u∗best).

6In (Reinbacher et al. (2010)) the projection step is solvediteratively. Our approach requires this projection step onlyfor the evaluation of the energy, which is not required atevery iteration. We also provide a non-iterative method tosolve the problem in Section 6.1.

solution of the optimization problem. The basicidea of this method is to split a problem into smallersub-problems while making use of the method ofmultipliers developed to solve constrained opti-mization problems (the augmented Lagrangian ap-proach). This decomposition simplifies the solutionprocess for the area-constrained segmentation prob-lem by breaking it into simpler sub-pieces. It alsoallows for the computation of a finite-valued dualenergy estimate, which serves as a lower bound forthe branch and bound algorithm.

4.1. Background on ADMM

We only provide the basic setup for ADMM herefor completeness, but refer to Boyd et al. (2010) fordetails. ADMM optimization problems are of theform

minu,w

f(w) + g(u), s.t. Bw + Cu = c, (4)

where u ∈ Rn, w ∈ Rm, f and g are functions(f : Rm 7→ R, g : Rn 7→ R) that do not need tobe differentiable, c ∈ Rq and B and C are appro-priately sized matrices7. The ADMM update steps(with step size σ > 0) are (Boyd et al. (2010))

wk+1 ← argminw

f(w) +σ

2‖Bw + Cuk − c− zk‖22

= proxB1σ f

(−Cuk + c+ zk),

uk+1 ← argminu

g(u) +σ

2‖Bwk+1 + Cu− c− zk‖22

= proxC1σ g

(−Bwk+1 + c+ zk),

zk+1 ← zk − (Bwk+1 + Cuk+1 − c). (5)

This amounts to first solving for w then for u andfinally updating the normalized dual variables, z.The prox operator (Combettes and Pesquet (2010))is defined as

proxLf (y) = argminw

f(w) +1

2‖Lw − y‖2.

Note that the update scheme for ADMM can read-ily be derived from an augmented Lagrangian for-mulation (Nocedal and Wright (2006)). The aug-mented Lagrangian corresponding to (4) is

Lσ(w, u, p) = f(w) + g(u)

+ pT (Bw + Cu− c) +σ

2‖Bw + Cu− c‖22, (6)

7In the specialization of ADMM for the area-constrainedsegmentation u will be the sought-for indicator-function andw will hold variable copies of u which simplify the numericalsolution.

7

where p is the Lagrangian multiplier. Making theidentification p = σz, the ADMM equations (5) aresimply the augmented Lagrangian update equationsfor (6) where the update for the primal variables isperformed separately, and conveniently written us-ing the prox operator. If f is an indicator functionfor a set C, i.e., f(x) = ıCx, (which is 0 if x ∈ C,∞ otherwise) the prox operator proxf (y) is simplythe projection of y on C. For general functions fthe prox operator proxLf (y) minimizes f while notmoving “too far” from y. See (Combettes and Pes-quet (2010)) for a more detailed discussion.

4.2. Background on Consensus Optimization

For area-constrained segmentation, splitting theproblem into more than two sub-problems subjectto consistency constraints simplifies the solution be-cause it will allow for decoupling of the spatial regu-larization of the total variation term, gs‖∇su‖, theunary potential term, ρsus, and the area constraint.The coupling is then re-introduced through a con-sistency constraint. Specifically, we will have anoptimization problem of the form

minui

n∑i=1

fi(ui), s.t. ui − u = 0, ∀i, (7)

where the ui are all independent variable copies andthe consensus variable u (our indicator function)is only present through the consistency constraints(i.e., g(u) = 0). At convergence, the constraintswill be fulfilled and hence ui = u, ∀i. The proxstep for u then becomes an averaging step (here,B = I, C = −I, c = 0)

uk+1i ← prox 1

σ f(uk + zki ),

uk+1 ← 1

n

n∑i=1

(uk+1i − zki ),

zk+1i ← zki − (uk+1

i − uk+1).

This global variable consensus (Boyd et al. (2010))formulation is well suited for parallel processing.Constraints on the consensus variable (u) can be en-coded in g(u) and therefore allow the specificationof seed points for area-constrained segmentation.

Interestingly, this does not change the overall so-lution scheme much, since the optimization problem

uk+1 = argminu

(g(u) +

n∑i=1

σ

2‖uk+1

i − u− zki ‖22

)

can be rewritten in the two-step form

uk+1 =1

n

n∑i=1

(uk+1i −zki ), uk+1 = prox 1

nσ g(uk+1),

which replaces the update step for u in Equa-tion (5). We solve the relaxed area-constraint seg-mentation problem with this form of ADMM bytransforming it to look like (7) as described in Sec-tion 5. The consensus variable u then correspondsto our sought-for indicator function u.

5. ADMM for area-constrainedsegmentation

We assume that the set of selection variablesb = bk of (3) is split into a set of selection vari-ables with known value (within a branch and boundtree) and a set of free selection variables. We thensubsume the determined selection variables in theforeground, S, and background, T , seed sets re-spectively. Dropping the free selection variablesfrom the formulation results in the relaxed area-constrained problem. For simplicity we use the 1-norm for the gradient term resulting in the energy

E(u) =∑(s,t)

cst|us − ut|+∑s

ρsus, (8)

s.t. Al ≤∑s

us ≤ Au;us ∈ [0, 1] (9)us = 1, s ∈ S,us = 0, s ∈ T ,

(10)

where (s, t) denotes a pair of neighboring pix-els (in our case using a four-connected neigh-borhood) and the weighted total variation term∑s gs‖∇us‖1 =

∑s gs(|(ux)s| + |(uy)s|) was dis-

retized as∑

(s,t) cst|us−ut|. This is a slightly more

general formulation, but includes∑s gs‖∇us‖1 if

the spatial gradients in the x and y directions (uxand uy) are discretized using finite differences, thesites s are given by the grid position of individualpixels and cst is set to gs for all t neighboring s.Note that this formulation is sufficiently general tosupport area-constrained segmentation for generalgraph structures. To simplify the solution of (10),we break the problem into the following four ener-

8

gies which need to be minimized:

E1(u) =∑s

ρsus + ı[0,1](us) :=∑s

fs(us),

E2(u) = ı

Al ≤

∑s

us ≤ Au

:= fA(u),

E3(u) =∑(s,t)

cst|us − ut| :=∑(s,t)

fst(us, ut),

E4(u) =∑s

ı[0,1](us) +∑s∈S

ıus = 1

+∑s∈T

ıus = 0 := g(u),

where ıCx denotes the indicator function and wewrite for notational simplicity ıC=x:f(x)=0x =ıf(x) = 0. The energies encode the unarypotential term, the area constraint, the pairwise-potential (edge) term, and the seeds, respectively.These problems are simple to solve independently.The consensus form of ADMM then allows us tocouple the four easy sub-problems so that we obtaina solution of the original optimization problem (10)at convergence.

Specifically, we use variable copies uA, us, us, ut

and the consensus variable u. The energy for theconsensus ADMM is then

E(u, us, uA, us, ut) =∑s

fs(uss) + fA(uA)

+∑(s,t)

fst(uss, u

tt) + g(u),

s.t.

u = us = uA,

us = uss ∧ ut = utt, ∀s, t. (11)

In ADMM notation of Section 4.1:f(uA, us, us, ut) =

∑s fs(u

ss) + fA(uA) +∑

(s,t) fst(uss, u

tt), and g holds the constraints

for the consensus variable. The prox operators areeasy to compute because they decouple spatiallyfor us, uA, and (us, ut). The edge variables us andut encode the presence of an edge between a source(s) and target (t) node and locally have as manycopies as there exist edges (i.e., for a regular gridtwo copies for s and two for t at each interior pixelto account for edges in the x and y directions).The overall algorithm is given in Algorithm 1. Theprox operators are given in Section 5.1. Section 5.2shows how we can compute the dual energy to (11)using the variables of the ADMM solution scheme.

5.1. Prox operators

Some derivations yield the averaging operator

avgs(uA, us, us, ut)

=uss + uAs

∑t:(s,t)∈E u

ss +

∑t:(t,s)∈E u

ts

2 + |t : (s, t) ∈ E|+ |t : (t, s) ∈ E|, (12)

and the prox operators

prox 1σ fs

(qs) = min1,max0, qs −1

σρs, (13)

prox 1σ fA

(qs) =

qs + 1

|V| (Al −Aq), if Al > Aq,

qs + 1|V| (Au −Aq), if Au < Aq,

qs, otherwise,

(14)

prox 1σ fst(s,t)

(u, v)

=

(u+ cst

σ v − cstσ

), if v − u > 2cst

σ ,(u− cst

σ v + cstσ

), if u− v > 2cst

σ ,(u+v2

u+v2

), otherwise,

(15)

prox 1nσ gs

(u) =

1, if s ∈ S,0, if s ∈ T ,min(1,max(0, u)), otherwise.

(16)Here, |V| denotes the number of pixels and E theedge set. See Section S.3 in the supplementary ma-terial for the derivations.

5.2. Dual energy of the ADMM formulation

Computing the dual energy for ADMM usingFenchel duality (Rockafellar (1997)) yields

E∗(ps, pA, ps, pt) = −∑s

f∗s (pss)− f∗A(pA)

−∑(s,t)

f∗st(pss, p

tt) +

∑s∈S

Qs −∑

s/∈T ∪S

[−Qs]+

where

Qs = pss + pAs + pss + pts,

f∗st(ps, pt) = ıps + pt = 0 ∧ |ps| ≤ cstf∗s (p) = = [p− ρ]+,

f∗A(p) =

Au maxs

psAs, if ∃s : ps ≥ 0,

Al maxspsAs, otherwise.

9

Algorithm 1: ADMM for the area-constrained segmentation.

Data: σ, Al, Au, cst; Initialized variable copies: u = us = uA = us = ut

Result: urepeat

Update local variables ;

(uA)k+1 ← prox 1σfA

(uk + (zA)k); (us)k+1 ← prox 1σfs(uk + (zs)k)

(us, ut)k+1 ← prox 1σfst

(uk + (zs)k, uk + (zt)k)

Averaging: uk+1s ← avgk+1

s (us, uA, us, ut)− avgks (zs, zA, zs, zt) ;

Clamping and enforcing seed points ;uk+1s ← 1 ∀ s ∈ S; uk+1

s ← 0 ∀ s ∈ T ; uk+1s ← min(1,max(0, uk+1

s ))Update dual variables ;

(zs)k+1 ← (zs)k − (us)k+1 + uk+1; (zA)k+1 ← (zA)k − (uA)k+1 + uk+1

(zs)k+1 ← (zs)k − (us)k+1 + uk+1; (zt)k+1 ← (zt)k − (ut)k+1 + uk+1

until convergence ;

Here, (ps, pA, ps, pt) = σ(zs, zA, zs, zt), i.e., thedual variables to compute E∗ are the scaled dualvariables of ADMM; [x]+ = max0, x is the rampfunction. See Section S.4 in the supplementary ma-terial for the derivations. Note that we need thedual energy of the original relaxed energy (10) andnot of the ADMM energy for the branch and boundsolution. We also need a feasible energy of the orig-inal MINLP (3) and not of the relaxed ADMM en-ergy8. Section 6 therefore describes how to computethe appropriate primal and dual energies from theADMM primal and dual energies.

6. Branch and bound

Building the search tree for a branch and boundsolution of (3) requires a method to create subprob-lems (we use a standard binary division strategy onthe bk), a strategy to select subproblems for eval-uation, a strategy to select variables for division,and a way to generate integer-energy estimates.We use a custom implementation of branch andbound where sub-problems are selected based onthe lowest current relaxed energies. Branching vari-ables are determined using pseudo-costs (Achter-berg et al. (2005)), and the lower bounds and

8The original primal and dual energies and their corre-sponding ADMM primal and dual energies will be equivalentat convergence. However, for an efficient branch and boundsolution we want to be able to test branch and bound termi-nation criteria with respect to the original primal and dualenergies before convergence.

integer-energy estimates are computed as describedbelow. See (Nemhauser and Wolsey (1988)) for anin-depth discussion of branch and bound.

6.1. Obtaining lower and upper bounds

At convergence, the equality of the consensusvariables is fulfilled, the bounds are satisfied andthe area constraint holds. Therefore primal anddual energies of the relaxed ADMM problem willbe finite upon convergence. To terminate solu-tion branches that cannot lead to an optimal so-lution early, finite-valued dual energy estimates areneeded before convergence for the dual energy toobtain a lower bound. Further, a feasible integer-valued (or essentially binary) solution is needed toobtain an upper bound. A finite-valued ADMMenergy estimate is needed to evaluate the conver-gence of a current relaxed ADMM solution candi-date based on its duality gap (i.e., the differencebetween primal and dual energy). Section 6.1.1discusses how to obtain a finite-valued relaxed en-ergy from an ADMM relaxed solution before or atconvergence. Section 6.1.2 discusses how to obtaina finite-valued dual energy for the relaxed prob-lem from the variables of the relaxed ADMM so-lution method. Finally, Section 6.1.3 discusses howinteger-feasible solutions can be obtained from re-laxed solutions by thresholding. Figure 3 illustratesthe connection between the different primal anddual energies.

10

relaxed energy relaxed

ADMM

convergence?

ADMM dual energyADMM energy

MINLP

branch termination?

choice of subset of qk

projectionprojection

thresholding

best feasible solution

feasible MINLP solutionconverged to

< relaxed dual energy

best

feasible solution?

feasible MINLP

solution branch termination?

relaxed dual energy

Figure 3: Relation between the different optimization prob-lems and branch and bound. Blue color indicates path for anon-ADMM-based solution. The goal is to solve the MINLPproblem. The branch and bound solver selects (and sets)a subset of selection variables bk and leaves the remainingones free. To obtain lower energy bounds and feasible so-lution to MINLP (given the selected bk) we use a relaxedformulation. The relaxed formulation is solved by ADMM.We can compute a primal and a dual energy from ADMM.From the ADMM energies we can obtain finite primal anddual relaxed energies by projections. We can obtain a feasi-ble MINLP solution from the projected solution from whichthe relaxed energy was obtained by thresholding. A currentcandidate branch is terminated if the relaxed dual energy islarger than the best feasible MINLP solution or if ADMMconverged to a feasible MINLP solution. Note that the ter-mination criteria can be checked before ADMM convergence.The branch and bound solver builds a search tree for all pos-sible choices for the bk, but never evaluates branches whichcan be discarded. Dashed lines indicate possible (but diffi-cult paths), solid lines indicate the proposed approach.

6.1.1. Estimate of the relaxed energy

A current finite-valued energy estimate, which isan upper bound of the relaxed energy at conver-gence, can be obtained by projecting the currentconsensus variable u back onto the constraint set(so that it fulfills the area and bound constraints,us ∈ [0, 1]). This requires solving the projection

u∗ = argminq

E(q) = minq

1

2

∑s

(qs − us)2,

Al ≤∑s

qs ≤ Au, qs ∈ [0, 1],

which, in order to project to area A, requires findinga Lagrangian multiplier λe s.t.∑

s

u∗s =∑s

min1, λe + us = A,

The optimal λe can be found by computing succes-sive relaxed solutions

∑s

λre + us = A→ λre =1

|V|

(A−

∑s

us

).

Since 0 ≤ λre ≤ λe the optimization problem can bebroken into subpieces and solved efficiently by firstsorting the values u (if the current area is smallerthan A – a similar reasoning hold in the reversecase).

If there is no relaxed feasible solution, then no in-teger feasible solution can exist. A feasible relaxedsolution can be computed if the area constraint pro-jection steps are feasible, which will be the case if∑

s∈TAs ≤ −Al +

∑s

As,∑s∈S

As ≤ Au.

Hence, for a given set of foreground/backgroundseedpoints in the branch and bound solver a solu-tion of the relaxed problem only needs to be soughtif these conditions hold, otherwise the dual energyis set to −∞ and the energy to ∞.

6.1.2. Estimate of the relaxed dual energy

A finite lower bound for the relaxed energy canbe obtained by adjusting the ADMM dual variablesfor the terms which would otherwise lead to a −∞estimate before convergence. We therefore need tofind a dual variable pair (ps, pt) that is as close aspossible to the current estimate (ps, pt) while ful-filling the edge variable constraint. Such a pair canby computed by the projection

Π(ps, pt) =

(c,−c), for ps − pt > 2cst,

(−c, c), for ps − pt < −2cst,

(ps−pt2 , pt−ps2 ), otherwise.

See Section S.5 in the supplementary material forthe derivation.

6.1.3. Estimate of an integer-feasible solution

To allow termination of suboptimal branches, agood estimate for an integer-feasible (or essentiallybinary) solution is desirable early during branchand bound. Assume a feasible relaxed solution isgiven. By thresholding the relaxed u at θ ∈ (0, 1),we can obtain in a finite number of thresholdingsteps (determined by bisection) an integer feasiblesolution, or show that such a thresholded solutiondoes not exist (in which case the estimate is set to

11

∞). In practice, we terminate the search for a solu-tion candidate after a fixed number of thresholdingsteps. Terminating the search without finding aninteger-feasible solution will not affect the overallbranch and bound solution. We will only not beable to produce a good integer-valued energy es-timate from this solution, which in turn may ef-fect early termination of search branches and mayconsequentially result in larger branch and boundsearch trees.

The relaxed solution candidate may already beessentially binary and fulfill the area constraintsgiven an appropriate selection of seed regions. Ingeneral, an optimal essentially binary u is guaran-teed to exist if a sufficient number of selection areasBk exist, and only an upper or lower bound needsto be enforced. For simultaneous lower and upperbounds the branch and bound algorithm will ei-ther find the best integer (and therefore one of theessentially binary) solutions, or will prove that nosuch solution exists. Non-existence is a pathologi-cal case, which is unlikely in practice. We never ob-served such a case in our experiments, but it is pos-sible to construct toy examples which exhibit thisproblem. When the area-constrained segmentationformulation requires the solution to be binary forthe selection areas, only compliant thresholded so-lutions will be feasible and hence finite.

7. Results

We tested the area-constrained segmentationmethod for the segmentation of synaptic vesiclesand for double membrane vesicles in epithelialcells infected with SARS-coronavirus (Knoops et al.(2008)). All images are slices of electron tomogra-phy images. Images for the epithelial cells wereobtained from the cell centered database (CCDB)of the National Center for Microscopy and Imag-ing Research (NCMIR – http://ccdb.ucsd.edu).The images for the synaptic vesicles were approxi-mately at a resolution of (1.0 nm by 1.0 nm)/pixeland for the SARS-coronavirus at (1.2 nm by1.2 nm)/pixel.

These examples were chosen to demonstrate theproperties of our developed area-constrained seg-mentation method, because segmentations for theseelectron tomography images are known to be chal-lenging. For example, for the synaptic vesicle seg-mentation task the vesicle wall is not directly visiblein the electron tomography image. Instead it needsto be inferred from the location of proteins (which

appear dark) penetrating the vesicle wall which re-sults in a “noisy” appearance of the vesicle wall.Further, a large number of vesicles can be foundin one image. Vesicles are closely packed in someareas, which even experts can have difficulty outlin-ing precisely. In our experiments a user was askedto place individual seed points at the center of theobjects to be segmented. The selection areas wereobtained by eroding a quick-shift segmentation ofthe complete image. The selection region closest tothe user-placed seed point was set as a foregroundseed, and the selection areas at the boundaries of a100x100 pixel box centered at the seed point wereset as background seeds. This box size was cho-sen to be sufficiently large to guarantee that thedesired objects are contained within it. We usedimage intensities as edge terms (cst) and set ρ = 0.We set γ = 1 for all ADMM experiments. Settingthe selection areas at the boundaries as backgroundseeds is meaningful for our experiment because theobject will be, by construction, at the center of theimage. However, this is not essential. The bound-aries could be included into the optimization, albeitat the price of higher computational cost.

Given the selected segmentation area and the se-lection regions we compared the following methodsfor the vesicle datasets:

1) UC: Area-constrained segmentation with alower bound of 0. This unconstrained case cor-responds to a classical graph-cut segmentationwith seed points.

2) LB/UB/LBUB: Area-constrained segmenta-tion enforcing upper and lower bounds on thesegmentation area separately and jointly.

3) BNC: Biased normalized cut (Maji et al.(2011)) using foreground seeds.

4) NC: Normalized cut (Shi and Malik (2000)).No seed regions are supported by this algo-rithm and hence none were used.

5) WS: Seeded watershed segmentation (Vincentand Soille (1991)) using foreground and back-ground seeds.

6) RW: Random walker segmentation using fore-ground and background seeds.

To allow comparisons between the algorithms (i) weused the same seed regions for all algorithms, (ii)we used the same edge weights g for all algorithms

12

except for the random walker algorithm, and (iii)determined the best possible thresholds for NC andBNC by searching which threshold results in thebest value for the normalized cut. For the randomwalker segmentation algorithm, we used two setsof edge weights, g, because this is a segmentationmodel which will not exhibit discontinuities at theputative segmentation boundary and hence treatsedge-weights differently than all the other testedmodels. We report random-walker results using thesame edge-weights as the other algorithms (RW) aswell as using the more appropriate default settingsfor random walker segmentation (d-RW).

7.1. Synaptic vesicles

We used a lower area bound of 800 and an up-per area bound of 2000 (areas in pixels) with a lownumber of large selection areas and a larger num-ber of small selection areas. Synaptic vesicles ob-served in our specimen are estimated to be about40 nm in diameter. Since we are dealing with slicesof a three-dimensional structure, we expect the ac-tual observed diameters to be smaller than this. Anarea between 800 and 2000 pixels corresponds ap-proximately to diameters between 30 and 50 nm ifa perfect circular shape is assumed.

Comparing to an expert segmentation of 38 vesi-cles many of the vesicles were segmented correctlyby both the area-constrained segmentations and bythe unconstrained segmentations. However, in theunconstrained case, a substantial number of vesi-cles was under-segmented (returning only the seedpoint). In contrast, the area-constrained segmenta-tions successfully segmented these cases and wereable to achieve a segmentation result very close tothe gold standard regardless of the selection ar-eas. Note that using only a lower bound givesthe best results in this example because it retainsmaximal flexibility for the registration boundary.When upper and lower bounds are enforced, thesegmentation needs to conform to the selection ar-eas. Though the area-constrained results are notstatistically significantly different with respect toeach other, they are statistically significantly bet-ter than all the other tested segmentation meth-ods. Biased normalized cut, watershed segmenta-tion, and the unconstrained segmentation methodshowed reasonable overall results, but suffered fromsevere outliers. The standard normalized cut seg-mentation fails entirely on these datasets because itcannot identify the object of interest since the data

is noisy and no seed regions can be used. Ran-dom walker segmentation overall performed well,did not show any strong outliers, but performedoverall worse than the area-constrained segmenta-tion method. Table 1 shows summary measures forthe Dice similarity coefficients for the experiments.Figure 4 shows the segmentation results for uncon-strained and area-constrained segmentation for asubset of an image. Figure 5 illustrates the differ-ent selection areas for a specific example.

An overview of corresponding seed points, thegold standard manual segmentation as well asresults for the area-constrained and the uncon-strained segmentations is shown in the supplemen-tary material in Figure S1. Boxplots for all the seg-mentation methods and a comparison of their meanperformance (as measured by Dice) are shown inFigure S2.

Adjacent vesicles may overlap because they aretreated independently. In practice, overlaps werenot observed for vesicle segmentation results of thearea-constrained segmentation approach. This is aproperty of the data combined with the segmenta-tion approach (which encourages short boundaries).In general, overlapping segmentations are possibleand present ambiguities in the segmentation. Suchambiguities could be avoided by moving to a multi-label segmentation formulation.

In cases where the unconstrained segmentationsresulted in a correct segmentation, the branchand bound search terminated quickly for the area-constrained methods. Most of the computationtime was spent to correct the more challengingcases. Between 28 and 90 selection regions wereused. Run-times were moderate: on average lessthan a minute per vesicle with a large number, andfour seconds with a small number of selection areason a single-core CPU implementation. The algo-rithm can easily be parallelized and implementedon a GPU (with an expected speed-up by at leastan order of magnitude).

7.2. SARS: Double membrane vesicles

We used a lower area bound of 2000 (area inpixels) with a low number of large selection ar-eas and repeated a subset of the experiments forthe synaptic vesicle segmentation. For a perfectcircle, this area would correspond to a diameterof about 60 nm at the given resolution. Sincethe double membrane vesicles have diameters ofabout 200-300 nm (Knoops et al. (2008)), this isa conservative lower bound on the area. Similar

13

type mean median std↑ UC 0.72 0.92 0.32↓ UC 0.86 0.94 0.22↑ LB 0.92 0.94 0.05↓ LB 0.93 0.95 0.04↑ LBUB 0.91 0.92 0.05↓ LBUB 0.91 0.91 0.04↑ BNC 0.79 0.85 0.16↓ BNC 0.80 0.88 0.17

NC 0.15 0.18 0.11↑ WS 0.85 0.91 0.13↓ WS 0.84 0.90 0.18↑ RW 0.86 0.86 0.05↓ RW 0.84 0.84 0.06↑ d-RW 0.89 0.91 0.04↓ d-RW 0.86 0.90 0.12

Table 1: Dice similarity coefficients for vesicle segmentationwith small (↓) and larger (↑) number of selection areas, Bk.Unconstrained (UC), lower (LB), and lower and upper bound(LBUB) constrained segmentations. Biased normalized cut(BNC), normalized cut (NC), seeded watershed (WS), ran-dom walker (RW) and random walker with default settings(d-RW). Bold: best results. Italicized results do not have sig-nificantly different mean in comparison to the best method.

(a)

(b)

(c)Less bk More bk

Figure 4: Subset of vesicle segmentation results; (a) uncon-strained, (b) lower bound, and (c) upper and lower bound.The constrained results are better on average, because theyavoid mis-segmentation due to shrinking bias. See Figure S2in the supplementary material for statistical results.

conclusions as for the synaptic vesicle experimentapply. However, since the images for the doublemembrane vesicles are significantly less noisy than

(a) (b) (c)

Figure 5: Example selection areas for a vesicle (a). Fewselection areas (less bk) (b) and many selection areas (morebk) (c).

the images for the synaptic vesicles watershed seg-mentation, random walker segmentation, as well asthe area-constrained segmentation method, workwell. The area-constrained segmentation methodmatches the performance of the best segmentationmethod (seeded watershed) for both SARS images.Generally, the segmentation using a lower boundon the segmentation area performed better thanthe unconstrained segmentation. Figure 6 showsoverviews of the resulting segmentations for SARS6021 and 6022, respectively, for the unconstrainedand the area-constrained segmentations. Table 2gives an overview of the obtained Dice similaritycoefficients. Figure S3 in the supplementary mate-rial shows boxplots for the segmentation results forall the tested methods and statistical significancelevels between the methods with respect to meanDice performance.

8. Conclusion and Future Work

We developed a new method for image segmen-tation with area constraints. The method read-ily extends to higher dimensions using higher-dimensional generalizations of the selection regions.The proposed method relies on the solution of amixed integer nonlinear program, which is solvedusing branch and bound. To reduce computationaleffort in solving the area-constrained segmentation,we proposed to use selection variables based oneroded super-pixels. This allows computation of thesegmentations for practical problems. The behaviorof the method was demonstrated for segmentationsof vesicles from slices of electron tomography im-ages. When area-constraints were available, statis-tically significant increases in segmentation qualitywere obtainable even in challenging cases. In par-ticular, due to the global optimality properties ofthe algorithm, it performs well for noisy data.

14

Seed points Manual segmentation

Unconstrained Lower bound constraint

Figure 6: Segmentation results for a slice of the SARS 6021(top) and of the SARS 6022 (bottom) electron tomogra-phy image. Seed points were placed manually with a singlemouse click. Without using an area constraint (red) onlyfew of the vesicles are accurately segmented and in the ma-jority of cases the segmentations are too small indicatingthat a short boundary length was favored over a segmenta-tion at the desired location of the cell wall. Adding a lowerbound on the area (blue) greatly improves the segmentationresults. Though a bias for short segmentation boundaries isstill present, most of the vesicles are segmented accurately.Since the SARS 6022 image appears less noisy many of thevesicles are also segmented correctly without using a seg-mentation area constraint.

Future directions include improvements on theoptimization method: e.g., should computations beperformed directly on super-pixels? Further, thesensitivity of the obtained results on the type andsize of the superpixels should be explored. Another

type mean median stdSARS 6021 UC 0.41 0.45 0.29

LB 0.90 0.96 0.10BNC 0.72 0.71 0.10NC 0.34 0.35 0.12WS 0.96 0.97 0.02RW 0.68 0.79 0.24d-RW 0.72 0.84 0.26

SARS 6022 UC 0.76 0.97 0.36LB 0.94 0.97 0.11BNC 0.79 0.81 0.11NC 0.38 0.35 0.15WS 0.95 0.98 0.04RW 0.79 0.84 0.18d-RW 0.93 0.95 0.04

Table 2: Dice similarity coefficients for the SARS 6021and SARS 6022 images. Unconstrained (UC), lower (LB),and lower and upper bound (LBUB). Biased normalized cut(BNC), normalized cut (NC), seeded watershed (WS), ran-dom walker (RW) and random walker with default settings(d-RW). Bold: best results. Italicized results do not have sig-nificantly different mean in comparison to the best method.For these images with a clear vesicle boundary watershedsegmentation, maxflow with a lower bound, and randomwalker segmentation work well.

interesting direction would be to vary the area con-straints to define an area-based scale space, whichwould allow us to automatically extract coherentsub-structures at different size levels from the im-ages. Extensions to multiple objects or the inclu-sion of topological constraints (e.g., to enforce oneconnected component) are other possible researchdirections.

Acknowledgements

This publication was made possible by GrantNumber 5P41EB002025-28 from NIBIB/NIH. Itscontents are solely the responsibility of the authorsand do not necessarily represent the official viewsof the NIBIB or NIH.

References

Achterberg, T., Koch, T., Martin, A., 2005. Branching rulesrevisited. Operations Research Letters 33, 42–54.

Appleton, B., Talbot, H., 2006. Globally minimal surfaces bycontinuous maximal flows. IEEE Transactions on PatternAnalysis and Machine Intelligence , 106–118.

Ayed, I., Li, S., Islam, A., Garvin, G., Chhem, R., 2008.Area prior constrained level set evolution for medical im-age segmentation, in: Proceedings of SPIE, p. 691402.

15

Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.,2010. Distributed optimization and statistical learning viathe alternating direction method of multipliers. MachineLearning 3, 1–123.

Boykov, Y., Funka-Lea, G., 2006. Graph cuts and efficient ndimage segmentation. International Journal of ComputerVision 70, 109–131.

Bresson, X., Esedoglu, S., Vandergheynst, P., Thiran, J.P.,Osher, S., 2007. Fast global minimization of the activecontour/snake model. Journal of Mathematical Imagingand Vision 28, 151–167.

Chan, T., Vese, L., 2001. Active contours without edges.Image Processing, IEEE Transactions on 10, 266–277.

Comaniciu, D., Meer, P., 2002. Mean shift: A robust ap-proach toward feature space analysis. Pattern Analysisand Machine Intelligence, IEEE Transactions on 24, 603–619.

Combettes, P., Pesquet, J., 2010. Proximal splitting methodsin signal processing. Fixed-Point Algorithms for InverseProblems in Science and Engineering .

Cootes, T., Edwards, G., Taylor, C., 2001. Active appear-ance models. Pattern Analysis and Machine Intelligence,IEEE Transactions on 23, 681–685.

Dahl, G., Flatberg, T., 2007. An integer programming ap-proach to image segmentation and reconstruction prob-lems. Geometric Modelling, Numerical Simulation, andOptimization , 475–496.

Eriksson, A., Olsson, C., Kahl, F., 2011. Normalized cutsrevisited: A reformulation for segmentation with lineargrouping constraints. Journal of Mathematical Imagingand Vision 39, 45–61.

Falkner, J., Rendl, F., Wolkowicz, H., 1994. A computationalstudy of graph partitioning. Mathematical Programming66, 211–239.

Ford, L.R., Fulkerson, D.R., 1956. Maximal flow through anetwork. Canadian Journal of Mathematics 8, 399–404.

Grady, L., Schwartz, E., 2006. Isoperimetric graph parti-tioning for image segmentation. Pattern Analysis andMachine Intelligence, IEEE Transactions on 28, 469–475.

Hager, W., Phan, D., Zhang, H., 2009. An exact algorithmfor graph partitioning. Arxiv preprint arXiv:0912.1664 .

Hijazi, H., Bonami, P., Cornuejols, G., Ouorou, A., 2009.Mixed integer nonlinear programs featuring on/off con-straints: convex analysis and applications. working paper.

Jain, A., Murty, M., Flynn, P., 1999. Data clustering: areview. ACM computing surveys (CSUR) 31, 264–323.

Ji, X., 2004. Graph Partition Problems with Minimum SizeConstraints. Ph.D. thesis. Rensselaer Polytechnic Insti-tute.

Kernighan, B., Lin, S., 1970. An efficient heuristic procedurefor partitioning graphs. Bell System Technical Journal 49,291–307.

Keuchel, J., Schnorr, C., Schellewald, C., Cremers, D., 2003.Binary partitioning, perceptual grouping, and restorationwith semidefinite programming. IEEE Transactions onPattern Analysis and Machine Intelligence 25, 1364–1379.

Knoops, K., Kikkert, M., Worm, S., Zevenhoven-Dobbe, J.,van der Meer, Y., Koster, A.J., Mommaas, A.M., Snijder,E.J., 2008. SARS-coronavirus replication is supported bya reticulovesicular network of modified endoplasmic retic-ulum. PLoS Biology 6, e226.

Kolmogorov, V., Zabin, R., 2004. What energy functionscan be minimized via graph cuts? Pattern Analysis andMachine Intelligence, IEEE Transactions on 26, 147–159.

Kropotov, D., Laptev, D., Osokin, A., Vetrov, D., 2010.

Variational segmentation algorithms with label frequencyconstraints. Pattern Recognition and Image Analysis 20,324–334.

Li, S.Z., 2009. Markov Random Field Modeling in ImageAnalysis. Springer.

Lim, Y., Jung, K., Kohli, P., 2010. Energy minimizationunder constraints on label counts, in: Proceedings of theEuropean Conference on Computer Vision (ECCV), pp.535–551.

Lisser, A., Rendl, F., 2003. Graph partitioning using linearand semidefinite programming. Mathematical Program-ming 95, 91–101.

Maji, S., Vishnoi, N., Malik, J., 2011. Biased normal-ized cuts, in: Computer Vision and Pattern Recognition(CVPR), 2011 IEEE Conference on, IEEE. pp. 2057–2064.

Nemhauser, G.L., Wolsey, L.A., 1988. Integer and Combi-natorial Optimization. Wiley.

Nocedal, J., Wright, S.J., 2006. Numerical Optimization.Springer.

Olsson, C., Eriksson, A.P., Kahl, F., 2008. Improved spec-tral relaxation methods for binary quadratic optimizationproblems. Computer Vision and Image Understanding ,3–13.

Pham, D., Xu, C., Prince, J., 2000. Current methods inmedical image segmentation. Annual review of biomedicalengineering 2, 315–337.

Pizer, S., Fletcher, P., Joshi, S., Thall, A., Chen, J., Frid-man, Y., Fritsch, D., Gash, A., Glotzer, J., Jiroutek, M.,et al., 2003. Deformable m-reps for 3d medical image seg-mentation. International Journal of Computer Vision 55,85–106.

Reinbacher, C., Pock, T., Bauer, C., Bischof, H., 2010. Vari-ational segmentation of elongated volumetric structures,in: Computer Vision and Pattern Recognition (CVPR),2010 IEEE Conference on, pp. 3177–3184.

Rockafellar, R., 1997. Convex analysis. Princeton Univ Pr.Rohlfing, T., Brandt, R., Menzel, R., Russakoff, D., Maurer,

C., 2005. Quo vadis, atlas-based segmentation? Hand-book of Biomedical Image Analysis , 435–486.

Sapiro, G., 2001. Geometric Partial Differential Equationsand Image Analysis. Cambridge.

Shi, J., Malik, J., 2000. Normalized cuts and image segmen-tation. Pattern Analysis and Machine Intelligence, IEEETransactions on 22, 888–905.

Sonka, M., Hlavac, V., Boyle, R., 2008. Image Processing,Analysis, and Machine Vision. Thomson.

Vedaldi, A., Soatto, S., 2008. Quick shift and kernel methodsfor mode seeking. (ECCV) , 705–718.

Vincent, L., Soille, P., 1991. Watersheds in digital spaces: anefficient algorithm based on immersion simulations. IEEEtransactions on pattern analysis and machine intelligence13, 583–598.

Zach, C., Niethammer, M., Frahm, J.M., 2009a. Continuousmaximal flows and Wulff Shapes: Application to MRFs,in: Proceedings of the Conference on Computer Visionand Pattern Recognition (CVPR), pp. 1911–1918.

Zach, C., Shan, L., Frahm, J.M., Niethammer, M., 2009b.Globally optimal Finsler active contours, in: DAGM-Symposium, pp. 552–561.

16