Representer theorems for sparsity-promoting regularizationbig · problems, regularization, ‘ 1-norm minimization, total variation I. INTRODUCTION T He main advantage of using ‘

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 1

Representer theorems for sparsity-promoting `1regularization

Michael Unser, Fellow, IEEE, Julien Fageot and Harshit Gupta

Abstract—We present a theoretical analysis and comparisonof the effect of `1 versus `2 regularization for the resolution ofill-posed linear inverse and/or compressed sensing problems. Ourformulation covers the most general setting where the solution isspecified as the minimizer of a convex cost functional. We derivea series of representer theorems that give the generic form of thesolution depending on the type of regularization. We start withthe analysis of the problem in finite dimensions and then extendour results to the infinite-dimensional spaces `2(Z) and `1(Z).We also consider the use of linear transformations in the form ofdictionaries or regularization operators. In particular, we showthat the `2 solution is forced to live in a predefined subspace thatis intrinsically “smooth” and tied to the measurement operator.The `1 solution, on the other hand, is formed by adaptivelyselecting a subset of atoms in a dictionary that is specified bythe regularization operator. Besides the proof that `1 solutionsare intrinsically sparse, the main outcome of our investigationis that the use of `1 regularization is much more favorable forinjecting prior knowledge: it results in a functional form that isindependent of the system matrix, while this is not so in the `2scenario.

Index Terms—Sparsity, compressed sensing, linear inverseproblems, regularization, `1-norm minimization, total variation

I. INTRODUCTION

THe main advantage of using `2 (or Tikhonov) regular-ization for the resolution of ill-posed inverse problems

is that it yields linear reconstruction algorithms; it is alsobacked by an elegant and solid mathematical theory [1],[2], [3]. However, it is not necessarily the method of choiceanymore, except for routine reconstruction tasks. During thepast decade, the research community has focused its effortson more sophisticated iterative recovery schemes that exploita remarkable property of signals called sparsity [4], [5]. Theconcept is central to the theory of compressed sensing [6],[7] and is driving the development of modern reconstructionalgorithms [8], [9], [10].

There are essentially two strategies for achieving a sparsesignal recovery. The first is the synthesis formulation whereone attempts to reconstruct a signal from a small subset ofatoms within a large dictionary of basis elements [11], [12],[5]. The sparsity constraint is usually enforced by minimizing

This work was supported in part by the European Research Council throughthe European Unions Seventh Framework Programme (FP7/2007-2013) underERC Grant 267439 (FUN-SP) and by the Romberg Professorship from theIWR, University of Heidelberg.

M. Unser, J. Fageot, and H. Gupta are with the Biomedical Imag-ing Group, EPFL, Lausanne, Switzerland (email: [email protected];[email protected]; [email protected]).

Copyright (c) 2014 IEEE. Personal use of this material is permitted.However, permission to use this material for any other purposes must beobtained from the IEEE by sending a request to [email protected].

the `1-norm of the expansion coefficients. The second strategyis the analysis formulation where the solution is constrainedby minimizing a sparsity-promoting functional such as thetotal-variation semi-norm [13], [8], [14], [15], [16] and orsome higher-order extension [17], [18], [19], [20]. This latterstrategy actually goes back much further since it falls withinthe general framework of regularization theory [21], [22],[23]. It also has the advantage of being compatible withstatistical inference. For instance, one may specify a maximuma posteriori estimator by selecting a regularization functional(or Gibbs energy) that corresponds to the log-likelihood of agiven probability model, including Markov random fields [24]or sparse stochastic processes [25]. It is well known that thesynthesis and analysis formulations are equivalent for signaldenoising when the sparsifying transform (or dictionary) isorthogonal and when the regularization functional is chosento be the `1-norm of the expansion coefficients [26].

While the switch from an `2 to an `1 regularization ne-cessitates the deployment of more sophisticated algorithms[8], [27], [28], there is increasing evidence that it results inhigher-quality signal reconstructions, especially in the morechallenging cases (compressed sensing) when there are lessmeasurements than unknowns. The theory of compressedsensing also provides some guarantees of recovery for K-sparse signals under strict assumptions on the system matrix[6], [29], [30].

Our objective in this paper is to characterize and comparethe effect of the two primary types of regularization on thesolution of general convex optimization problems involvingreal-valued linear measurements. While the sparsity inducingproperty of the `1-norm is well documented and reasonablywell understood by practitioners, we are only aware of a fewmathematical results that make this explicit with the view ofsolving underdetermined systems of linear equations (e.g. [31],[4], [32]), typically under the assumption that the `1-minimizeris unique.

We have chosen to present our findings in the form of aseries of representer theorems which go by pairs (`2 vs. `1regularization) with all other aspects of the problem—i.e.,the choice of the (convex) data term and the regularizationoperator—being the same. A pleasing outcome is that ourresults reinforce the connection between the synthesis andanalysis formulations of signal recovery since our `1 repre-senter theorems can be interpreted as a “synthesis” solutionto a class of optimization problems that is more typical of theregularization framework.

The paper is organized as follows. The scene is set inSection II with a brief discussion and comparison of the

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 2

two primary schemes for signal recovery: the linear Tikhonovestimator (with p = 2) versus the non-linear basis pursuitestimator (with p = 1), where the only change is in theexponent of the regularization. We also document the propertythat one is able to control the sparsity of the latter estimator byvarying the regularization parameter λ. In Section III, we focuson the finite-dimensional scenario (x ∈ RN ) and present ourtwo main representer theorems that cover an amazingly broadfamily of convex optimization problems. As example of ap-plication, we prove that the extreme points of a total-variationoptimization problem are necessarily piecewice-constant. InSection IV, we generalize our result to the infinite-dimensionalsetting (x ∈ `1(Z)). The formulation becomes more technicalas we need to invoke the weak∗ topology to specify the fullsolution set of the generic `1-norm minimization problem.We also consider the scenario where the null space of theregularization operator L is non-trivial, which requires somemore sophisticated developments (Theorem 19). The bottomline is that the generic form of the solution remains unchanged,while the sparsifying effect of `1-regularization is even moredramatic: the minimization process results into the collapsingof an infinity of degrees of freedom into a small finite numberthat is upper bounded by the number of measurements.

II. MOTIVATION: `2 VERSUS `1 REGULARIZATION

In a linear inverse problem, the task is to recover some un-known signal x ∈ RN from a noisy set of linear measurementsy = (y1, . . . , yM ) ∈ RM such that

ym = 〈hm,x〉+ n[m], m = 1, . . . ,M (1)

where n[m] is some unknown noise component that is typi-cally assumed to be i.i.d. Gaussian. The measurement model isspecified by the real-valued system matrix H = [h1 · · ·hM ]T

of size M ×N . Our interest here is in the ill-posed scenariowhere M is (much) smaller than N (compressed sensing)or when the system matrix is poorly conditioned and notinvertible. This ill-posedness is dealt with in practice by intro-ducing some form of regularization. Since our objective hereis to compare the regularizing effect of `2 vs. `1 norms, weshall start with the simplest scenario where the regularizationis imposed directly upon x. The more general case wherethe regularization is enforced in some transformed domain isaddressed in the second part of Section III.

A. Simple regularized least-squares estimator

The most basic penalized least-squares (or Tikhonov) esti-mator of the signal x from the measurements y is specifiedby

xLS = arg minx∈RN

‖y −Hx‖22 + λ‖x‖22. (2)

where λ > 0 is a hyper parameter that controls the strengthof the regularization. The standard form of the solution is

xLS = (HTH + λIN )−1HTy,

where IN is the N × N identity matrix. This translates intoa linear algorithm that can also be interpreted as a Wiener

filter. We shall now invoke a lesser-known result that hassome interesting conceptual implications. The proof is givenin Appendix A for sake of completeness.

Proposition 1: For any matrix H of size M×N and λ ∈ R+,we have the identity

(HTH + λIN )−1HT = HT (HHT + λIM )−1.

This allows us to rewrite the least-squares solution as

xLS = HTa =

M∑m=1

amhm (3)

where a = (HHT +λIM )−1y. We have thereby revealed theproperty that xLS ∈ span{hm}Mm=1. Moreover, if we let λ→0, then the solution converges to x0 = H+y where H+ is theMoore-Penrose generalized inverse of H [2, Section 1.5.2]. Bydefinition, H+ solves the classical least-squares approximationproblem minx ‖y − Hx‖22 and extracts the solution x0 thathas the minimum norm. If HTH is of full rank, then H+ =(HTH)−1HT , which is the classical pseudo-inverse of H.Otherwise, which is the case of interest here, it returns theminimum-norm solution that is in the span of HT as well.

While this simple linear reconstruction scheme works rea-sonably well when M ≥ N , the situation is much lessfavorable for smaller M because the solution is forced to livein a space that is specified by the system matrix H, and hencestrongly problem-dependent.

B. Least-squares estimator with `1 penalty

An alternative that has become increasingly popular inrecent years is to substitute the squared `2-norm penalty bythe `1-norm. This yields the so-called penalized basis pursuit(PBP) estimator

xsparse = arg minx∈RN

‖y −Hx‖22 + λ‖x‖1, (4)

where λ > 0 is a regularization parameter with the samerole as before. To get some insight on the effect of the `1regularization, we now look at the extreme scenario wherethere is a single measurement:

minx∈Rn

|y1 − hT1 x|2 + λ‖x‖1

For λ above some critical threshold, we get the trivial solutionx = 0. Otherwise, we obtain a “sparse” solution of the form

xsparse = a1en1

where {en}Nn=1 is the canonical basis of RN and n1 the indexof the component of h1 that has the largest magnitude sothat ‖xsparse‖1 = |a1| = |y1/[h1]n1 | is minimal. This hasto be compared with the corresponding `2 solution (3) whichsimplifies to

xLS = a1h1

with a1 = y/(hT1 h1 + λ). The contrast is striking: On theone hand, we have a solution that is completely sparse with‖xsparse‖0 = 1, while, on the other, we obtain a blurred ren-dition whose parametric form is dictated by the measurement

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 3

functional h1. It turns out that the contrasting behavior thathas been identified for this very simple scenario is generic andtransposable to a much broader class of optimization problems.

The other property that is well documented in the literatureis the sparsifying effect of the regularization parameter λ in(4). When λ is very small and close to 0, the solution willtypically have a sparsity index ‖xsparse‖0 = M where Mis the number of measurements. In order to promote sparsersolutions, it then suffices to increase λ, as illustrated in Fig. 1.To show that this mechanism is universal and unrelated to thechoice of the system matrix, we considered three representa-tive scenario : (i) symmetric exponential convolution followedby a non-uniform sampling (Conv), (ii) non-uniform (random)sampling of the discrete cosine transform (DCT) of thesignal, and (iii) compressed sensing (CS) involving a systemmatrix whose components are i.i.d. Gaussian. The simulatedmeasurements were generated according to (1) where n[m]is AWG noise. The reconstruction was then performed usingFISTA [27] for unconstrained `1 minimization. To verify thatthe control mechanism is independent of the suitability of theunderlying signal model, we considered two extreme config-urations. In the first set of simulations summarized in Fig.1a, the ground-truth signal x is truly sparse with its majorityof coefficients being zero—specifically, the component of xare i.i.d. with a Bernouilli-Gauss distribution. For the secondset of experiments shown in Fig. 1b, we switched to a “non-sparse” model by taking x to be i.i.d. Gaussian. While thereare differences in the shape of the graphs, the main point is thatin all cases, K = ‖xsparse‖0 decreases monotonically with λwhile its maximum value is bounded by M .

Besides the standard PBP form (4) favored by practitioners,there are two other possible formulations of the recoveryproblem. The first is the LASSO (Least Absolute Shrinkageand Selection) defined as (see [26])

arg minx∈RN

‖y −Hx‖22 s. t. ‖x‖1 ≤ τ, (5)

while the second is the (quadratically) constrained basis pur-suit (CBP) estimator

arg minx∈RN

‖x‖1 s. t. ‖y −Hx‖22 ≤ σ. (6)

The key property for our purpose is that for any value ofλ ∈ R+ in (4), it is possible to find some correspondingτ = τ(λ) ≥ 0 and σ = σ(λ) ≥ 0 (and vice versa) so thatthe PBP, LASSO, CBP problems are rigorously equivalent(see [32, Proposition 3.2, p 64]). The argument is that theminimizer of (6)

(resp., (5)

)saturates the inequality, which

allows us to interpret (4) as the unconstrained form of thesame minimization problem with Lagrange multiplier λ. Theoptimal tajectory (τ(λ), σ(λ)) that is parametrized by λ iscalled the Pareto curve [33]. The same equivalence obviouslyalso holds for p = 2. While the constrained version ofthe problem (6) with σ fixed is typically harder to solvenumerically than (4), it is actually the form that lends itselfbest to a mathematical analysis, as we shall see next.

III. FINITE DIMENSIONAL ANALYSIS

In order to derive the general form of the solution of linearinverse problems with `2 versus `1 regularization constraints,

λ →10-3

10-2

10-1

100

101

Sp

ars

ity I

nd

ex →

0

10

20

30

40

a): Sparse model

Conv.DCTCS

λ →10

-310

-210

-110

010

1

Sp

ars

ity I

nd

ex →

0

10

20

30

40

b): Gaussian model

Conv.DCTCS

Fig. 1. Sparsity index K = ‖xsparse‖0 of the solution of (4) as a function ofthe regularization parameter λ for three different kinds of system matrices withM = 30 and N = 120. The simulated measurement model is described by(1) where x is i.i.d. (innovation model) with two distinct statistical models:(a) Bernoulli-Gauss with an average sparsity index of 8, (b) Gaussian andtherefore, non-sparse. In all the simulations, the standard deviations of thesignal is σx = 3, the input SNR is 35 dB and all the matrices are rownormalized. The rows of the system matrix H are constructed as follows. (i)Conv: random shifts of a symmetric exponential e−|m|; (ii) DCT: randomrows of a DCT matrix ; (iii) CS: Gaussian i.i.d. with σ = 1. In all the cases,K reaches M = 30 as λ→ 0.

we shall first enlarge the class of problems of interest byconsidering some arbitrary convex constraints on the so-calleddata term which involves the measurements y. While this hasthe advantage of providing more general results, it has theeven more remarkable effect of simplifying the mathematicalderivations because it puts the problem in an abstract perspec-tive that is more suitable for functional analysis.

A. Preliminaries

Let us start with a few definitions where X stands foran arbitrary (finite or infinite-dimensional) topological vectorspace. In this section, X = RN .

Definition 2: A subset C of a vector space X is convex ifz = (tx+ (1− t)y) ∈ C, for any x, y ∈ C and t ∈ [0, 1]; thatis, if all the points that lie on the line connecting x to y arealso included in C.

Definition 3 (Projection on a closed convex set): Let Xbe a vector space equipped with some norm ‖ · ‖. Then, theprojection set of z on the closed convex set U ⊂ X is

arg minx∈U‖x− z‖ = {x0 ∈ U : ‖x0 − z‖ ≤ ‖x− z‖, ∀x ∈ U}.

When the projection set reduces to a single point x0, then x0is called the projection of z on U and we write

x0 = arg minx∈U‖x− z‖. (7)

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 4

To make the connection with the signal recovery formula-tions of Section II, we define a data-dependent closed convexset

U(y;σ) ={x ∈ RN : ‖y −Hx‖22 ≤ σ

}where σ is an adjustable control parameter. Hence, the classi-cal projection problem (7) with z = 0 and U = U(y;σ) yieldsthe constrained form—that is, the CBP estimator (6)—of ourinitial signal recovery problem.

A potential difficulty when dealing with general convexoptimization problems is that not all such problems haveunique solutions. It is possible, however, to give a completedescription of the solution set in terms of its extreme points.

Definition 4 (Extreme point): Let E be a convex subset ofsome vector space X . An extreme point of E is a point x ∈ Ethat does not lie in any open line segment joining two distinctpoints of E.The extreme points of a convex optimization problem are veryspecial in that they lie on the frontier of the convex solutionset which is then given by their convex hull. Obviously, theproblem has a unique solution if and only if it has a singleextreme point, as is generally the case with the `2-norm.

B. Finite-dimensional representer theorems

Having set the context, we now proceed with thepresentation of representer theorems for a rather broad familyof convex optimization problems in relation to the type ofregularization.

Theorem 5 (Convex problem with `2 minimization): Let H :RN → RM : x 7→ Hx with M < N be a linear measurementoperator and C be a closed convex subset of RM such that itspreimage in RN , U = H−1(C) = {x ∈ RN : Hx ∈ C}, isnonempty (feasibility hypothesis). Then,

V = arg minx∈RN

‖x‖22 s.t. Hx ∈ C (8)

has a unique extreme point of the form

xLS =

M∑m=1

amhm = HTa. (9)

Proof: First, we observe that convexity (resp., closed-ness) is preserved through linear (resp., continuous) trans-formations so that the preimage U of C is guaranteed to beclosed convex as well. In view of Definition 3, the solutionis thereby given by the projection of the origin z = 0 ontothe closed convex set U , which is known to be nonempty,because of the feasibility hypothesis. Our claim of unicitythen follows from Hilbert’s famous projection theorem forconvex sets which states that the projection on a convex setin a Hilbert space always exists and reduces to a single point[34]. The Hilbert space here is RN equipped with the innerproduct 〈x1,x2〉 = xT1 x2.

Let x0 = arg minx∈U ‖x‖ denote the unique solution of(8) and y0 = Hx0 be the image of that point through themeasurement operator. Because the linear map x0 7→ y0 isconsistent (i.e., y0 ∈ C) and the projection has minimum norm,the operation is reversible with x0 = H+y0 where H+ is the

Moore-Penrose generalized inverse of H whose range is inthe span of HT (see brief discussion of the property of thisinverse in Section II-A). In other words, there exist a uniquea ∈ RN such that xLS = x0 = HTa = H+(HHTa).

Let us note that the result in Theorem 5 is consistent withthe elementary analysis of the Tikhonov estimator in SectionII. What is remarkable is that the generic form of the solutionremains valid for the complete class of convex optimizationproblems involving the same linear measurement model andthe same quadratic regularization functional ‖x‖22. The catch,of course, is that the general solution map is no longer linear.In other words, we should view Theorem 5 (as well as allsubsequent representer theorems) as an existence/discretizationresult, meaning that it is still necessary to deploy some iterativealgorithm (such a steepest-descent method) to actually find theoptimal expansion vector a.

We now present the `1 counterpart of Theorem 5. Thestatement of the problem is almost identically except for thefact that there can now be multiple extreme points.

Theorem 6 (Convex problem with `1 minimization): Let H :RN → RM : x 7→ Hx with M < N be a linear measurementoperator and C be a closed convex subset of RM such that itspreimage in RN , U = H−1(C) = {x ∈ RN : Hx ∈ C}, isnonempty (feasibility hypothesis). Then,

V = arg minx∈RN

‖x‖1 s.t. Hx ∈ C

is a nonempty, convex, compact subset of RN with extremepoints xsparse of the form

xsparse =

K∑k=1

akenk(10)

with K ≤ M , {en}Nn=1 the canonical basis ofRN , nk ∈ {1, . . . , N} for k = 1, . . . ,K, and‖xsparse‖1 =

∑Kk=1 |ak|.

Proof: Since C is convex (resp., closed) and H is linear(resp., continuous), the set U = H−1(C) is convex (resp.,closed) as well. Therefore, U is a nonempty, convex, andclosed subset of RN .

The function ‖x‖1 is continuous from RN → R+, andtherefore admits a minimum (not necessarily unique) over anyclosed set, including U , which ensures that V is nonempty.Therefore, let α = minx∈U‖x‖1 and B be the closed ball ofradius α for the `1-norm; that it, B = {x ∈ RN , ‖x‖1 ≤ α}.Then, the set V = U ∩ B is convex and compact, as theintersection of a convex closed set with a convex compactset. Ultimately, this translates into V being nonempty, convex,and compact.

This allows us to invoke the Krein-Milman theorem (see[35, p.75]), which tells us that a convex compact set, such asV , is the closed convex hull of its extreme points.

Let us now consider an extreme point x = (x1, . . . , xN ) ofV whose number of non-zero entries is denoted by K, witha priori K ∈ {0, . . . , N}. We want to prove that x is of theform (10), which is equivalent to K ≤M . We shall proceed bycontradiction and assume that K = ‖x‖0 ≥ M + 1, meaning

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 5

that there exists (at least) (M + 1) indices n1, . . . , nM+1 ∈{1, . . . N} such that xnm 6= 0 for every m. We set x = x −∑M+1m=1 xnmenm and, for m = 1, . . . ,M + 1,

ym = xnmHenm

. (11)

Since any collection of (M + 1) vectors in RM is linearlydependent, there exists some constants cm such that c =(c1, . . . , cM+1) 6= 0 and

∑M+1m=1 cmym = 0. Correspondingly,

we define x0 =∑M+1m=1 cmxnm

enm. We also pick an ε ∈ R

with |ε| < 1/maxm|cm| such that (1 + εcm) > 0 and(1−εcm) > 0 for all m. Since Hx0 =

∑M+1m=1 cmym = 0, we

have that Hx = H(x− εx0) = H(x+ εx0) so that (x+ εx0)and (x − εx0) are in U . Moreover, because x0 and x havedisjoint support, we have that

‖x± εx0‖1 = ‖x +

M+1∑m=1

(1± εcnm)xnm

enm‖1

= ‖x‖1 +

M+1∑m=1

(1± εcnm)|xnm |

= ‖x‖1 ± εM+1∑m=1

cnm|xnm

|

= α± εM+1∑m=1

cnm|xnm

|.

If∑M+1m=1 cnm |xnm | 6= 0, then (x + εx0) or (x − εx0) has a

`1-norm strictly smaller than α, which is impossible since theminimum over U is α. Hence,

∑M+1m=1 cnm

|xnm| = 0, and

‖x + εx0‖1 = ‖x− εx0‖1 = α. (12)

In other words, there exists ε > 0 such that (x + εx0), (x −εx0) ∈ V , which implies that

x =1

2(x + εx0) +

1

2(x− εx0) (13)

is not an extreme point of V . This proves that K ≤M .

We like to mention a related result [32, Theorem 3.1., p.62] on the maximal cardinality of the support of the solutionof the problem

min ‖x‖1 subject to Hx = y

under the assumption that the problem admits a unique min-imizer. It is also indicated there that the result does notcarry over to the complex setting. Theorem 6 constitutes asubstantial extension as it applies to a much broader class ofproblems—it also provides the structure of the full solution setfor the more typical cases where the minimizer is not unique.

C. Incorporation of a regularization operator

To cover a broader spectrum of applications, we are alsointerested in problems involving a regularization operator ora dictionary. We shall now see that this extension is straight-forward when the regularization functional is coercive; that is,when there exists a constant A > 0 such that A‖x‖p ≤ ‖Lx‖pfor all x ∈ RN . In finite dimensions, this translates into L

being an invertible matrix of size N . The analysis of the morechallenging non-coercive scenario is deferred to Section IV-C.

Corollary 7 (Convex problem with `2 regularization): LetH : RN → RM : x 7→ Hx with M < N be a linearmeasurement operator and C be a closed convex subset ofRM such that its preimage in RN , U = H−1(C), is nonempty(feasibility hypothesis). L is an invertible regularization matrixof size N that can be chosen arbitrarily. Then,

V = arg minx∈RN

‖Lx‖22 s.t. Hx ∈ C (14)

has a unique solution of the form

xLS =

M∑m=1

amhm = HTa. (15)

where HT = (LTL)−1HT .

Proof: Since L is invertible, we define the auxiliaryvariable u = Lx, which allows us to rewrite y = Hx = Guwith G = HL−1. Likewise, the convex set C in the spaceof the measurements y is linearly mapped into a nonemptyconvex set U in the space of the auxiliary variable u. We thenapply Theorem 5, which yields the generic solution

uLS = GTa⇔ xLS = (LTL)−1HTa.

Corollary 8 (Convex problem with `1 regularization): LetH : RN → RM : x 7→ Hx with M < N be a linearmeasurement operator and C be a closed convex subset ofRM such that its preimage in RN , U = H−1(C), is nonempty(feasibility hypothesis). L is an invertible regularization matrixof size N that can be chosen arbitrarily. Then,

V = arg minx∈RN

‖Lx‖1 s.t. Hx ∈ C (16)

is a nonempty, convex, compact subset of RN with extremepoints xsparse of the form

xsparse =K∑k=1

akgnk(17)

with K ≤M and ‖Lxsparse‖1 =∑Kk=1 |ak|. The basis vectors

gnkwith indices nk ∈ {1, . . . , N} for k = 1, . . . ,K are taken

within the N -dimensional dictionary

GT = [g1 · · ·gN ] = (L−1)T .

Proof: The proof here too is based on the direct applica-tion of Theorem 6 with the auxiliary variable u = Lx.

The remarkable outcome is that the reconstruction space isnow entirely determined by the regularization operator L, andindependent of the measurement setup, in sharp contrast withthe `2 scenario in Corollary 7.

Corollary 8 tells us that the extreme points of the opti-mization problem (16) are constructed by picking the “best”K � N elements within a dictionary that is specified by therow vectors of L−1. While this proves that the solution set isintrinsically sparse, it is primarily an existence result becauseTheorem 6 does not tell us which elements to pick (i.e., the

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 6

value of the index nk) nor the values of the weights ak. Again,the powerfull aspect here is the generality of the result since itapplies to a complete class of convex optimization problems.

An alternative formulation could be to specify the aug-mented vector a = (a1, . . . , aN ) ∈ RN with the implicitunderstanding that one is restricting our choice of candidatesto those that are K-sparse with K ≤M . The optimal config-uration would then be achieved when ‖a‖1 = minx∈U ‖Lx‖1.

D. The special case of total variation

To make the connection with the popular “total variation”scenario, we take L = D as the finite difference operator

D =

1 −1 0 · · · 0 00 1 −1 · · · 0 0

. . .0 0 0 · · · 1 −10 0 0 · · · 0 1

(18)

Its inverse is given by

D−1 =

1 1 1 · · · 1 10 1 1 · · · 1 1

. . .0 0 0 · · · 1 10 0 0 · · · 0 1

which is an upper triangular matrix of ones.

The interpretation of Corollary 8 is that the correspondingsolution will then be formed by selecting a few rows of D−1

(or columns of D−1T ), which results in a solution that ispiecewise-constant with K jumps of amplitude ak. The totalvariation of the solution is then measured by the `1 norm ofthe coefficient vector a = (a1, . . . , aK); i.e.,

‖xsparse‖TV = ‖Dxsparse‖`1 = ‖a‖`1 .

This is consistent with one of the earliest schemes used tosolve compressed sensing problems [29]. The interest of ourtheorem is that it explains why the optimization of totalvariation always admits a piecewise-constant solution. Whilethis behavior is well known and amply documented in theliterature, we are not aware of any prior mathematical analysisthat shows that the generic form of the solution (piecewise-constant) is actually independent of H.

By contrast, there is no such decoupling in the `2 scenariowhere the influence of the regularization and the characteristicfootprint of the system matrix are intertwined. Specifically,Corollary 7 tells us that the basis functions are now givenby HT = (DTD)−1HT which amounts to some smoothedversion (doubly integrated) of HT . In particular, if HT is takento be the identity or a non-uniform sampling matrix, then the`2 solution becomes piecewise-linear with breakpoints (orknots) at the sampling locations, which is a rather differenttype of signal.

We conclude this section with an important remark con-cerning our use of the above finite-difference matrix. Indeed,another choice could have been the circulant matrix

DLSI =

1 −1 0 · · · 0 00 1 −1 · · · 0 0

. . .0 0 0 · · · 1 −1−1 0 0 · · · 0 1

(19)

which is almost the same as (18), except for the additional −1in the lower right. Now the major difference between D andDLSI is that the latter, which maps into a circular convolution,annihilates constants. While this property is very desirable forregularization purposes, its downside for the present demon-stration is that it spoils the invertibility requirement for theapplication of Corollaries 7 and 8. To handle such cases, weneed the non-coercive counterparts of these results, which arepresented in Section IV-C. At any rate, the bottom line for totalvariation is that the piecewise-constant form of the solution ispreserved in either cases, the main difference being that DLSI

does not penalize constant signals (spanned by the first rowvector of D−1).

IV. INFINITE DIMENSIONAL ANALYSIS

We will now extend our analysis to the infinite dimensionalsetting. While the basic ideas underlying the proofs remainthe same, the formulation becomes more technical because wehave to properly deal with topological issues; in particular, thecomplication that the unit ball in `1(Z) is no longer compact.Another substantial generalization is that we are also treatingthe very relevant case of regularization operators whose nullspace is non-trivial. To help the readers who are not so muchat ease with functional analysis, we have done our best toclarify the presentation by including tutorial explanations.

A. Notation

Following the standard convention in signal processing,discrete signals or sequences are indexed using square bracketswith the index running over Z; i.e., x[n] denotes the sample ofthe signal x = x[·] at location n ∈ Z. Likewise, the infinite-dimensional counterpart of the canonical basis {en}Nn=1 is{δ[· − n]}n∈Z where δ[· − n0] denotes the unit impulse atsome fixed location n0 (the dot “·” is a placeholder for thedomain variable of the input that is used to avoid notationalconfusion).

Instead of matrices, we shall now consider linear operatorsacting on suitable Banach spaces. These are denoted by capitalletters. For instance, the operator G : X → Y maps the spaceX (the domain of the operator) into Y; its action is denotedby x 7→ y = G{x} with x ∈ X and y ∈ Y .

B. Infinite-dimensional representer theorems

We first formulate the optimization problem in the real-valued Hilbert space `2(Z) equipped with the `2-inner product〈x, y〉 =

∑n∈Z x[n]y[n].

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 7

Theorem 9 (Convex problem with `2(Z) minimization): Letus consider the following:• H : `2(Z)→ RM , x 7→ (〈h1, x〉, . . . , 〈hM , x〉) is a linear

measurement operator such that ‖H{x}‖ ≤ B‖x‖`2 forsome constant B > 0 and every x ∈ `2(Z);

• C be a closed convex subset of RM such that its preimagein `2(Z), U = H−1(C) = {x ∈ `2(Z) : H{x} ∈ C}, isnonempty (feasibility hypothesis).

Then, the problem

V = arg minx∈`2(Z)

‖x‖`2 s.t. H{x} ∈ C (20)

has a unique solution of the form

xLS =

M∑m=1

amhm = H∗{a}. (21)

Proof: The first part of the argument is the same as in theproof of Theorem 5. The linear operator H is bounded, andtherefore continuous. Hence, U (the preimage of C through alinear and continuous transformation) is closed and convex,while the minimizer of (20) xLS = x0 is unique (by Hilbert’sprojection theorem).

The second part is now handled in a softer manner byusing a geometric argument. Let M = span{hk}Mk=1 andM⊥ = {x ∈ `2(Z) : H{x} = 0} be the orthogonalcomplement of M, which also coincides with the null-spaceof H. Since `2(Z) =M⊕M⊥, every x ∈ `2(Z) has a uniquedecomposition as x = u + u⊥ with u ∈ M and u⊥ ∈ M⊥.Then, the solution x0 can be written as x0 = u0 + u⊥0 . SinceH{x0} = H{u0}, x0 and u0 both lie in U . As x0 is the solutionof (20), we have

‖x0‖2 ≤ ‖u0‖2⇒ ‖u0 + u⊥0 ‖2 = ‖u0‖2 + ‖u⊥0 ‖2 ≤ ‖u0‖2

⇒ ‖u⊥0 ‖ = 0⇔ u⊥0 = 0

Thus, x0 = u0 implying that x0 ∈ span{hm}Mm=1, which canbe written in the form of (21).

As expected, (21) is the infinite-dimensional counterpart of(9) where the measurement vectors play the central role in thesolution.

Let us now focus our attention on `1(Z), which is theBanach space associated with the norm ‖x‖`1 =

∑n∈Z|x[n]|.

The complication there is to properly handle the potentialissue of non-uniqueness. Since `1(Z) has an infinite numberof dimensions, the unit ball B = {x ∈ `1(Z) : ‖x‖`1 ≤ 1}is not compact anymore for the Banach topology. However,by considering a weaker notion of convergence on `1(Z), werecover compactness and are able to generalize Theorem 6for infinite sequences. The space of sequences that vanish at±∞ is denoted by c0(Z). It is a Banach space when endowedwith the supremum norm. The space `1(Z) is the topologicaldual of c0(Z). We can therefore define the weak*-topologyon `1(Z); that is, the topology associated with the followingnotion of convergence: a sequence (xm)m∈N of elements of`1(Z) converges to 0 for the weak*-topology if∑

n∈Z

∣∣a[n]xm[n]∣∣ −→m→∞

0 (22)

for every a ∈ c0(Z). Note that the sum∑n∈Z

∣∣a[n]x[n]∣∣ is

always finite for a ∈ c0(Z) and x ∈ `1(Z). As suggested bythe name, the weak*-topology is weaker than the usual Banachtopology. Indeed, the convergence to 0 for the `1-norm impliesthe convergence to 0 for the weak*-topology due to the relation∑

n∈Z

∣∣a[n]x[n]∣∣ ≤ ‖a‖`∞‖x‖`1 . (23)

We say that a subset of `1(Z) is weak*-closed (weak*-compact, respectively) if it is closed (compact, respectively)for the weak*-topology. The crucial point for us is thatthe ball B is weak*-compact in `1(Z), as implied by theBanach-Alaoglu theorem [35, p.68]. For more details on theweak*-topology, we refer the reader to [35, Section 3.11].

Theorem 10 (Convex problem with `1(Z) minimization): Letus consider the following:• H : `1(Z)→ RM is a linear measurement operator such

that ‖H{x}‖ ≤ A‖x‖`1 for some constant A > 0 andevery x ∈ `1(Z);

• C be a closed convex subset of RM such that its preimagein `1(Z), U = H−1(C) = {x ∈ `1(Z) : H{x} ∈ C}, isnonempty (feasibility hypothesis).

Then,V = arg min

x∈`1(Z)‖x‖`1 s.t. H{x} ∈ C

is a nonempty, convex, weak*-compact subset of `1(Z) withextreme points xsparse of the form

xsparse =

K∑k=1

akδ[· − nk] (24)

with K = ‖xsparse‖0 ≤ M , nk ∈ Z for k = 1, . . . ,K and‖xsparse‖`1 =

∑Kk=1 |ak|.

Proof: The fact that V is nonempty, convex, and weak*-compact follows from classical theorems in convex analysis,as detailed in Appendix B. The form of the extreme points isthen established using the same argumentation as in the proofof Theorem 6.

C. Extensions for non-coercive regularization functionals

In Section III-C, we have seen that there is no major diffi-culty in extending the representer theorems for more generalscenarios involving an invertible regularization operator L. Theconcept carries over to infinite dimensions as well under thesame assumption that the mapping is injective; that is, whenthe null space of the operator is trivial (NL = {0}).

We shall now show that we can do much more and handlethe non-coercive cases where the null space of the regulariza-tion operator

NL ={q : Z→ R

∣∣ L{q} = 0}

= span{pn}N0n=1 (25)

is finite dimensional of size N0 where we are assuming thatthe pn (basis elements) are linearly independent. The nullspace of L has a privileged role in the problem formulationbecause it incurs no penalty. This has the effect of promoting

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 8

solutions whose null-space component is the largest possible.For instance, in the case of the finite-difference operator, anyconstant signal results in a zero-cost solution.

While such an extended setting is very attractive from apractical perspective, it introduces a higher level of difficultybecause the operator L is no longer invertible in the usual(two-sided) sense. Yet, we shall see that it is still possibleto specify some proper right inverse via the introduction ofsuitable boundary conditions. But prior to that, we need tospell out the conditions that ensure that an operator is welldefined over `p(Z), the cases of interest being p = 1, 2.

In our framework, the concrete description of a linearoperator G is provided by its kernel (or generalized impulseresponse) G[k, l] = G{δ[· − l]}[k]. To make things moreconcrete, simply think of G[k, l] as an infinite-dimensionalmatrix that is applied to the signal x = x[·] = (x[l])l∈Z.

Definition 11: Given some sequence (or discrete signal) x =(x[k])k∈Z, we say that G{x} is well defined if∑

l∈Z

∣∣G[k, l]x[l]∣∣ < +∞

for any fixed k ∈ Z where G[·, ·] is the kernel of the operator.The output signal G{x} is then specified by G{x}[k] =∑l∈ZG[k, l]x[l] for k ∈ Z.Definition 12: A sequence x = (x[k])k∈Z is said to be of

slow growth if there exists an integer n0 ∈ Z and a constantA > 0 such that

|x[k]| ≤ A(1 + |k|)n0 for all k ∈ Z.

The space of such sequences is denoted by S ′(Z). It is thediscrete counterpart of S ′(R) (Schwartz’s space of tempereddistributions). As the notation suggests, S ′(Z) is actually thetopological dual of S(Z): the space of rapidly-decreasingsequences [36].

Proposition 13: The generic linear operator G : x 7→ y =G{x} is well defined over `p(Z) if and only if its kernelsatisfies

‖G[k, ·]‖`p′ <∞ (26)

for any k ∈ Z where p′ = p/(p−1) is the conjugate exponentof p ∈ [1,∞]. Moreover, G is bounded from `1(R)→ `∞(R)if and only if

supk,l∈Z

∣∣G[k, l]∣∣ <∞. (27)

Proof: The sufficiency of (26) is established by usingHolder’s inequality to construct the estimate∑

l∈Z|G[k, l]x[l]| ≤ ‖G[k, ·]‖`p′‖x‖`p .

Conversely, if there is some k0 ∈ Z such that G[k0, ·] /∈`p′(Z), we can construct a worst-case signal x ∈ `p(Z) suchthat

∑l∈Z |G[k0, l]x[l]| diverges (since the Holder inequality

is sharp).By taking the supremum of the above estimate for p = 1,

we get

‖y‖`∞ = ‖G{x}‖`∞ ≤

(supk,l∈Z

|G[k, l]|

)‖x‖`1 .

for all x ∈ `1(Z), which shows that (27) implies that G isbounded from `1(Z) → `∞(Z). The necessity is establishedby considering wl = δ[· − l] ∈ `1(Z) with ‖wl‖`1 = 1. SinceG{wl} = G[·, l], we have that

‖G{wl}‖`∞ = ‖G[·, l]‖`∞ = supk∈Z|G[k, l]|,

for any l ∈ Z. On the other hand, the boundedness of Gimplies that

‖G{wl}‖`∞ = supk∈Z|G[k, l]| ≤ ‖G‖ (28)

where‖G‖ ≤ sup

k,l∈Z|G[k, l]|.

Since (28) must hold for all l ∈ Z including the value thatachieves the supremum, we conclude that the bound is sharp.

We are now ready to specify the vector spaces over whichthe global optimization is going to take place as

`p,L(Z) = {x : Z→ R s.t. ‖L{x}‖`p <∞}, (29)

with p = 1, 2. By definition, the operator L maps `p,L(Z) into`p(Z). Our first step is to establish that `p,L(Z) is a bona fideBanach space. The difficulty is that ‖L{·}‖`p is only a semi-norm on `p,L(Z); that is, it has all the properties of a normexcept that ‖L{x}‖`p = 0 does not imply that x = 0. This isresolved by factoring out the null space of the operator.

Proposition 14: Let ‖·‖NL be some admissible norm forthe finite-dimensional null space NL and ProjNL

a projectionoperator from `p,L(Z) into NL. Then, `p,L(Z) defined by (29)is a Banach space for the composite norm

‖L{x}‖`p + ‖ProjNL{x}‖NL

.

Proof: We recall that the elements xQ of the quotientspace `p,L(Z)/NL are equivalence classes on `p,L(Z) suchthat, for x ∈ `p,L(Z), xQ = {x + q : q ∈ NL}. Sincethe quotient space does not distinguish between elementsx, y ∈ `p,L(Z) such that x − y ∈ NL, we can endow it withthe norm ‖L{xQ}‖ := ‖L{x}‖p, where x is any memberof the equivalence class xQ. This shows that `p,L(Z)/NL

is a Banach space, while the same property obviously holdsfor NL. It follows that the direct sum of those two spaces,`p,L(Z)/NL + NL, is a Banach space for the sum-norm‖(xQ, p)‖ = ‖L{xQ}‖`p + ‖p‖NL

with xQ ∈ `p,L(Z)/NL

and p ∈ NL. The final step is to specify the isomorphismbetween `p,L(Z) and `p,L(Z)/NL +NL via the relation x 7→(xQ,ProjNL

{x}) where xQ is the equivalence class of x in`p,L(Z)/NL and ProjNL

{x} the projection of x into NL. Tomake the link completely explicit, we further identify xQ withx − ProjNL

{x} which is the unique element of xQ whoseprojection onto NL is zero. The reverse map is then simply(xQ, q) 7→ xQ + q, which spans the complete space `p,L(Z).As a consequence, `p,L(Z) inherits the Banach space structureof the direct sum.

Let us note that Proposition 14 is a high-level statementthat holds for any admissible norm ‖·‖NL and projectionoperator ProjNL

. It turns out that the exact choice of these

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 9

elements has no influence on the Banach topology of `p,L(Z).The explanation lies in the fact that the null space NL isfinite-dimensional and that all finite-dimensional norms aretopologically equivalent. The finite-dimensionality of NL alsoguarantees the existence of the projector ProjNL

: `p,L(Z)→NL (by the Hahn-Banach theorem); the main point is thatthe latter should be seen as an extension of the identity mapi : NL → NL to the whole space `p,L(Z). In the sequel, wewill fix these elements in order to properly invert the operatorL. This will be achieved by imposing N0 linear boundaryconditions, as will be made explicit in Theorem 16.

Definition 15 (Admissible regularization operator): A linearoperator L : `p,L(Z)→ `p(Z) is called admissible if

1) it has a finite-dimensional null space NL = {q ∈`p,L(Z) : L{q} = 0} spanned by some basis p =(p1, . . . , pN0

);2) it is right-invertible in the sense that there exists a kernel

ρL ∈ S ′(Z × Z) (the space of bi-infinite matrices withslow-growing rows and columns) with the property thatL{ρL[·, l]} = δ[· − l].

It is important to note that the fundamental solutionof L{ρL[·, l]} = δ[· − l] (or discrete Green’s function) isnot unique (unless N0 = 0) since any kernel of the formql + ρL[·, l] with ql ∈ NL is acceptable as well. We shall nowshow that there are some privileged forms that result in aninversion that is stable over `2(Z).

Theorem 16 (Stable right-inverse operator): Let L be anadmissible regularization operator in the sense of Definition15. We also assume that we are given some correspondingset of biorthogonal analysis functionals φ = (φ1, . . . , φN0)with φn ∈ `′2,L(Z) (the continuous dual of `p,L(Z)) such that〈φm, pn〉 = δ[m− n]. Then,

HL,φ ={x : Z→ R

∣∣ L{x} ∈ `2(Z) and 〈φ, x〉 = 0}

is a Hilbert space equipped with the inner product 〈f, g〉L =〈L{f},L{g}〉. Moreover, there exists an isometric map L−1φ :`2(Z)→ HL,φ such that

HL,φ ={x = L−1φ w : w ∈ `2(Z)}.

The operator L−1φ is uniquely specified through the followingproperties

1) Right-inverse property: LL−1φ w = w for all w ∈ `2(Z)

2) Boundary conditions: 〈φ,L−1φ w〉 = 0 for all w ∈ `2(Z)

and its kernel is given by

gφ[k, l] = ρL[k, l]−N0∑n=1

pn[k]qn[l], (30)

with qn[l] = 〈ρL[·, l], φn〉 and ρL such that L{ρL[·, l]} =δ[· − l].

Proof: We start by proving that HL,φ equipped with theinner product

〈x1, x2〉L = 〈L{x1},L{x2}〉 = 〈L∗L{x1}, x2〉 (31)

is a Hilbert space. The only delicate aspect there is to es-tablish the unicity property of the inner product: 〈x0, x0〉L =0 ⇔ x0 = 0. To that end, we observe that the condition〈L{x0},L{x0}〉 = 0 is equivalent to x0 ∈ NL. Thanks to thebiorthogonality of p and φ, we also know that

q =

N0∑n=1

〈φn, q〉pn

for all q ∈ NL. Finally, we use the boundary conditions〈φ, x0〉 = 0 to conclude that x0 = 0.

The idea is then to first establish the properties 1) and 2) ofthe operator L−1φ on the space of rapidly-decreasing sequencesS(Z) to avoid any technical problems related to the splittingand interchange of sums. Since the space S(Z) equipped withthe standard weighted-`2 Frechet topology is dense in `2(Z)[36], we are then able to extend the properties by continuity.

For notational purpose, we introduce the operator G : x 7→∑l∈Z ρL[·, l]x[l], which is well defined over S(Z) as long as

ρL[·, ·] ∈ S ′(Z × Z). By assuming that w ∈ S(Z), we cantherefore rewrite x = L−1φ {w} as

x = L−1φ {w} = G{w} −N0∑n=1

pn〈qn, w〉.

Next, we apply the operator L, which yields

LL−1φ {w} = L

{∑l∈Z

w[l]ρL[·, l]

}−

N0∑n=1

L{pn}︸ ︷︷ ︸=0

〈qn, w〉

=∑k∈Z

w[l]L{ρL[·, l]}

=∑k∈Z

w[l]δ[· − l] = w

where we have used the defining properties L{ρL[·, l]} = δ[·−l] and L{pn} = 0 for n = 1, · · · , N0. In particular, this impliesthat

‖L−1φ {w}‖2L = 〈L−1φ {w},L

−1φ {w}〉L = ‖w‖2`2 (32)

for all w ∈ S(Z), which shows that L−1φ is bounded in the `2norm.

As for the boundary conditions, we first observe that

qn[l] = 〈ρL[·, l], φn〉

=∑k∈Z

ρL[k, l]φn[k] = G∗{φn}[l]

where G∗ is the adjoint of G. We then make use of thebiorthogonality property 〈φm, pn〉 = δ[m− n] to evaluate theinner product of L−1φ w with φm as

〈φm,L−1φ {w}〉 = 〈φm,G{w}〉 −N0∑n=1

〈φm, pn〉〈qn, w〉

= 〈φm,G{w}〉 − 〈qm, w〉= 〈G∗{φm}, w〉 − 〈G∗{φm}, w〉 = 0,

which shows that the boundary conditions are satisfied. Indoing so, we have effectively shown that L−1φ continuously

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 10

maps S(Z) into HL,φ. Again, since S(Z) is dense in `2(Z),the boundary conditions do also extend to `2(Z) by continuity.

As final step, we invoke the Hahn-Banach theorem inconjunction with the `2 bound (32) to extend the domain of theoperator to all of `2(Z). This allows us to conclude that L−1φ

continuously maps `2(Z) → HL,φ. In fact, L−1φ : `2(Z) →HL,φ is an isometry that provides a stable inverse of theoperator L : HL,φ → `2(Z). In other words, we have shownthat the operator L−1φ whose kernel is specified by (30) is suchthat LL−1φ w = w, 〈φ,L−1φ w〉 = 0 and ‖L−1φ w‖L = ‖w‖`2 forall w ∈ `2(Z).

Since `p(Z) ⊆ `2(Z) for p ∈ [1, 2], one can obviously alsorestrict the domain of the inverse operator L−1φ to `p(Z) withthe insurance that Properties 1) and 2) are met for w ∈ `p(Z).

As demonstration of usage, let us consider the finite-difference operator D, which is specified as

x 7→ D{x} = x[·]− x[· − 1].

This operator is the infinite-dimensional counterpart of D inSection III-D. It is shift-invariant, and its Fourier symbolis (1 − e−jω), which exhibits a single zero at ω = 0.Consequently, D has a one-dimensional null space ND =span{p1} ⊆ `∞(Z) that is spanned by the “constant” signalp1[k] = 1. The simplest choice of biorthogonal analysis vectoris φ1 = δ[·] with the property that

q = 〈δ[·], q〉p1 = q[0]p1

for all q ∈ ND. A possible choice of fundamental solution isρD[k, l] = 1+[k − l] with the property that D{1+[· − l]} =δ[· − l]. The application of Theorem 16 then yields the kernelof the corresponding right-inverse operator D−1φ1

:

gδ[k, l] = 1+[k − l]− 1+[−l].

Its stability is revealed by observing that, for k0 ≥ 1,gδ[k0, l] = 1{1,...,k0}[l], which is compactly supported of sizek0, and hence included in `p′(Z) for all p′ ≥ 1. This guaranteesthat x 7→ D−1φ1

{x} is well defined for any x ∈ `p(Z) withp ≥ 1 (see Proposition 13).

This is in contrast with the “canonical” shift-invariant inver-sion mechanism x 7→ y = 1+ ∗ x (moving sum filter), whichis ill-defined on `p(Z) for p > 1.

The main point that we want to make here is that theinversion task is not trivial (because the standard system-theoretic solution is not directly applicable), but that it cannevertheless be achieved in a principled fashion by apply-ing the constructive procedure described in Theorem 16. Inessence, the second term in (30) is a mathematical correctionthat makes the (right)-inverse operator `p-stable for 1 ≤ p ≤ 2.

D. Extended regularization theory

We have now all the tools in hand to make the Banachstructure of `p,L(Z) suggested by Proposition 14 explicit.This, in turn, will allow us to derive the generic form of theoptimizer for p = 1, 2.

Theorem 17 (Direct sum decomposition): Let L be a reg-ularization operator that admits a stable right-inverse L−1φ of

the form specified by Theorem 16. Then, any x ∈ `p,L(Z)with p ∈ [1, 2] has a unique representation as

x = L−1φ w + q,

where w = L{x} ∈ `p(Z) and q =∑N0

n=1〈φn, x〉pn ∈ NL.Moreover, `p,L(Z) is a Banach space equipped with the norm

‖x‖p,L,φ = ‖L{x}‖`p + ‖〈x,φ〉‖2. (33)

Proof: The right-inverse operator L−1φ is obviously well-defined for w ∈ `p(R) ⊆ `2(Z). Let x1, x2 ∈ `p,L(Z) be suchthat L{x1} = L{x2} = w. By definition of the null space,this is equivalent to (x1 − x2) = q ∈ NL. Conversely, letx = L−1φ w with w ∈ `p(Z). Then, the condition ‖L{x}‖p =‖w‖p <∞ ensures that x ∈ `p,L(Z). This allows us to deducethat `p,L(Z) is the sum of NL = span{pn}N0

n=1 and

Bp,L,φ = {x = L−1φ w : w ∈ `p(Z)}= {x ∈ `p,L(Z) : 〈φn, x〉 = 0, n = 1, . . . , N0},

where Bp,L,φ is a Banach space equipped with the norm‖L{·}‖`p . Its completeness is inherited from the one of `p(Z)and the fact that the inverse operator L−1φ performs an isomet-ric mapping `p(Z)→ Bp,L,φ. Moreover, since 〈φ,L−1φ w〉 = 0(boundary conditions) and the φm are biorthogonal to the pn,we find that the null-space component q is given by

q =

N0∑n=1

〈φn, x〉pn = ProjNL{x}

It is therefore specified by its expansion coefficients 〈φ, x〉 =(〈φ1, x〉, . . . , 〈φN0 , x〉) whose `2-norm is ‖〈φ, x〉‖2. Thisshows that the decomposition x = L−1φ w + q, where w =L{x} ∈ `p(Z) and q ∈ NL, is unique, which also translatesinto `p,L(Z) = Bp,L,φ ⊕ NL because `p,L,φ(Z) ∩ NL = {0}.The final part of the argument is the same as in Proposition14 with Bp,L,φ being isomorphically equivalent to the quotientspace `p,L(Z)/NL.

Using Theorem 17, we now proceed to provide the resultsfor convex optimization with `1 and `2 regularizers. Thetechnical part concerning the weak*-compactness of thesolution set is taken care in Appendix B.

Theorem 18 (Convex problem with `2 regularization): Letus consider the following:• L : `2,L(Z) → `2(Z) is an admissible regularization

operator in the sense of Definition 15;• H : `2,L(Z) → RM : x 7→ (〈h1, x〉, . . . , 〈hM , x〉) is

a linear measurement operator such that, for any x ∈`2,L(Z),

A‖〈x,φ〉‖2 ≤ ‖H{x}‖2 ≤ B(‖L{x}‖`2 + ‖〈x,φ〉‖2)(34)

for some constants A,B > 0 and φ as in Theorem 16;• C is a closed convex subset of RM such that its preimage

in `2(Z), U = H−1(C) = {x ∈ `2,L(Z) : H{x} ∈ C}, isnonempty (feasibility hypothesis);

• L−1φ : `2(Z) → `2,L(Z) is a stable right-inverse of L asspecified in Theorem 16.

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 11

Then,

V = arg minx∈`2,L(Z)

‖L{x}‖`2 s.t. H{x} ∈ C

is a nonempty, convex, weak*-compact subset of `2,L(Z) withsolutions of the form

xLS =

M∑m=1

amhm + p0 with hm = L−1φ L−1∗φ {hm}, (35)

where a = (a1, . . . , aM ) is a fixed element of RM and p0 ∈NL a null-space component that describes the full solution set;i.e., p0 ∈ H−1

(C −H

{∑Mm=1 amhm

})∩ NL. In particular,

when C reduces to a single point, then the solution is unique.Proof: The property that V is non-empty, convex and

weak*-compact is covered by Lemma 20 in Appendix B.Consider the set CL = {z + H{p}, z ∈ C, p ∈ NL}. Wedefine the new optimization problem

W = arg minw∈`2(Z)

‖w‖2 s.t. (HL−1φ ){x} ∈ CL. (36)

CL is closed and convex as the sum of two closed and convexsets, C and H(NL). Moreover, we easily show that

U0 := (HL−1φ )−1(CL) = L(H−1(C)). (37)

Since the set U = H−1(C) is nonempty by assumption, thesame holds true for U0 = L(U). We are therefore fulfilling theconditions of Theorem 9, from which we deduce that thereexists a unique minimizer wLS = (HL−1φ )∗{a} in W , witha ∈ RM .

Let x0 ∈ V , which is decomposed as x0 = L−1φ {w0} + p0with w0 ∈ `2(Z) and p0 ∈ NL. Then, L{x0} = w0 andx0 ∈ U , hence w0 ∈ U0. Likewise, for any w ∈ U0, thereexists x ∈ U such that L{x} = w. Since x0 ∈ V and x ∈ U ,we have ‖w‖`2 = ‖L{x}‖`2 ≥ ‖L{x0}‖`2 = ‖w0‖`2 . Asthis relation is true for every w ∈ U0, w0 ∈ W and thereforew0 = wLS. This shows that x0 = L−1φ (HL−1φ )∗{a}+p0. Next,we define x1 = L−1φ {w0} and simplify its expression as

x1 = L−1φ (HL−1φ )∗{a} = L−1φ L−1∗φ H∗{a}

= L−1φ L−1∗φ

{M∑m=1

amhm

}=

M∑m=1

amhm

where hm = L−1φ L−1∗φ {hm}. Since H{x0} = H{x1} +H{p0} ∈ C by definition, we deduce that p0 necessarily lies inH−1 (C −H{x1}). Conversely, any element of the form (35)is clearly in U , and hence in V when a is chosen optimally.

Theorem 19 (Convex problem with `1 regularization): Letus consider the following:• L : `1,L(Z) → `1(Z) is an admissible regularization

operator in the sense of Definition 15;• ρL[·, ·] is a kernel such that L{ρL[·, l]} = δ[· − l] for alll ∈ Z.

• H : `1,L(Z)→ RM is a linear measurement operator suchthat, for any x ∈ `1,L(Z),

A‖〈x,φ〉‖2 ≤ ‖H{x}‖2 ≤ B(‖L{x}‖`1 + ‖〈x,φ〉‖2)(38)

for some constants A,B > 0 and φ as in Theorem 16.• C is a convex compact subset of RM such that its preim-

age in `1,L(Z), U = H−1(C), is nonempty (feasibilityhypothesis).

Then,

V = arg minx∈`1,L(Z)

‖L{x}‖`1 s.t. H{x} ∈ C

is a nonempty, convex, weak*-compact subset of `1,L(Z) withextreme points of the form

xsparse =

K∑k=1

akρL[·, nk] +

N0∑n=1

bnpn (39)

with K ≤ M , nk ∈ Z, ak, bn ∈ R, and‖L{xsparse}‖`1 =

∑Kk=1 |ak|.

Proof: Here too, we refer to Lemma 20 with p = 1 forthe non-emptiness, convexity, and weak*-compactness of V .The remainder of the proof is essentially the same as the oneof Theorem 6. For a fixed extreme point x, we assume thatL{x} is not K-sparse and that we can find at least M + 1elements n1, . . . , nM+1 such that L{x}[nk] 6= 0 and we showthat x is not an extreme point. The final observation is thatxsparse = L−1φ {w∞} + p∞ can be rewritten as (39) by usingthe explicit form of the kernel of L−1φ given by (30).

Once again, it is instructive to compare the solutions of the`2 and `1 regularization problems covered by Theorems 18 and19. The first fundamental difference is that the solution of the`2 problem is constrained to live in a fixed finite-dimensionalsubspace of `2, while the reconstruction space for the `1 prob-lem is adaptive and determined by the problem and the dataat hand. Interestingly, the first property remains valid for the`2 regularization even if the solution of the extended problemin Theorem 18 is no longer unique because of the additionaldegrees of freedom offered by the null space component. Thesecond distinction is in the form of the basis functions: In the`2 case, there is a characteristic intertwining between the effectof the measurement and regularization operators, while in the`1 scenario the basis functions are chosen within a dictionary{ρL[·, n]}k∈Z whose form is completely determined by theregularization operator L. This part of the story is completelyin line with the findings of Section III so that all the commentsthat have been made there are still pertinent.

The novel aspect in our two last representer theorems is theappearance of the second parametric term p0 =

∑N0

n=1 bnpn,which encodes the component that is in the null space ofthe operator. As already mentioned, the role of p0, whoseregularization cost is zero, is fundamental because it tries tofullfil the constraints as much as possible in order to decreasethe `1 or `2 penalty associated with the first component. Whilethe possibility of applying a regularization operator whose nullspace is non-trivial is immensely useful in practice, it requiresa more sophisticated mathematical treatment. The enablingingredient is the construction and proof of existence of a stableright-inverse operator under very weak hypotheses (Theorem16) which also constitutes one of the contribution of this work.

We believe that the stability bounds used in the statement ofour infinite-dimensional representer theorems are the weakest

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 12

possible hypotheses for this kind of optimization problem. Theupper bound on ‖H{x}‖2 is the explicit way of indicating thatthe measurement operator is well-defined in the sense that itcontinuously maps `p,L(Z) → RM ; as far as we know, thislatter hypothesis (which is often implicit) is necessary for themathematical analysis of any inverse problem. Hence, the onlyconstraining hypothesis is the lower bound in (34) and (38),which is required to counteract the lack of coercivity of theregularization functional ‖L{x}‖p. It makes the problem well-posed over the (very small) subspace NL; in other words,the measurements should be rich enough to allow us tounambiguously reconstruct the null-space component of thesignal. For instance, in the case of TV (i.e., L = D), thereshould at least be one measurement functional hm such that〈hm, 1〉 6= 0, which is a very mild constraint. Also note thatthe non-coercive scenario has the additional restriction that theconvex set C should be bounded.

E. Connection with splinesIn Section III-D, we have seen that the extremal points

of finite-dimensional linear inverse problems with a total-variation regularization are necessarily piecewise-constant,which suggests a connection with splines. We recall thatsplines are continuous-domain entities (i.e., functions) thatare defined classically as the solution of a quadratic-energyminimization problem subject to (linear) interpolation con-straints [37], [38]. The concept is transposable to the discretedomain as well, which leads to the related notion of discretesplines with the regularization operator L = Dn being the nthpower of the finite-difference operator D. Existence resultsare also available for discrete splines with `p regularizationfor p ≥ 1 [39], but the explicit form of these splines hasonly been worked out explicitly for p = 2. This correspondsto the simplified setting H{x} = (x[k1], · · · , x[kM ]) (non-uniform sampling operator) and C = y = (y1, · · · , yM ) ∈ RMin Theorem 18, which imposes the interpolation constraintsx[k1] = y1, · · · , x[kM ] = yM . It is well known that thisproblem admits a unique solution, which is the discretecounterpart of a polynomial spline interpolant of degree 2n−1with knots at the km’s [40], [39].

In order to specify the solution of the `1 variant of theinterpolation problem, we observe that Dn admits a discreteshift-invariant Green’s function ρn[·] that is the n-fold convo-lution of the discrete step 1+ and hence a (discrete) one-sidedpolynomial of degree n − 1. The corresponding form of theextreme points in Theorem 19 is

∑Kk=1 akρn[·−nk]+p0 where

the null-space component p0 is a (discrete) polynomial ofdegree n−1. In other words, they are discrete splines of degreen−1 with data-dependent knots (nk)Kk=1 and K ≤ N . Besidesthe reduction of the polynomial degree of the spline, the keydifference with the `2 scenario is that the position of the knotsis adaptive and not known a priori. Yet, the truly remarkablefinding here is that this functional form of the solution remainsvalid for any convex linear inverse problems with nth-order`1-regularization, far beyond the classical spline setting.

Finally, we have recently managed to (literally) connect thedots (that is, the samples of the signal) by developing a func-tional framework that is the continuous-domain counterpart of

the present theory; this is the topic of a forthcoming paperwhose name says it all [41].

V. CONCLUSION

In this paper, we have characterized the form of the solutionof general linear inverse problems with convex constraints and`1 vs. `2 regularization. We have started from the simplestfinite-dimensional scenario and worked our way up progres-sively to the more challenging family of (infinite-dimensional)inverse problems covered by the Representer Theorems 18 and19. We have striven for the maximal generality and the weakestpossible assumptions in order to cover the majority of convexsignal recovery problems encountered in practice. We believethat these functional descriptions of the solution should be ofinterest to researchers working in the field.

The primary message that emerges from this investigation isthe superiority of `1 over `2 regularization for injecting priorknowledge on the solution. For instance, the minimizationof ‖D2{x}‖`1 where D2 is the 2nd-order difference operatorproduce solutions that are piecewise-linear irrespective of thesystem’s matrix H and the number of measurements. There isno such independence between the characteristic form of thesolution and the system matrix in the case of `2 regularization.

APPENDIX APROOF OF PROPOSITION 1

Let H be an arbitrary matrix of size M × N and IN theidentity matrix of size N . We start by noting that

(HTH + λIN )HT = HTHHT + λHT

= HT (HHT + λIM )

The underlying hypothesis that λ > 0 ensures that both(HTH+λIN ) and (HHT +λIM ) are invertible. This allowsus to deduce that

(HTH + λIN )−1HT = HT (HHT + λIM )−1,

which is the desired result.

APPENDIX BCONVEXITY AND WEAK*-COMPACTNESS

OF SOLUTION SET

Here, we establish the convexity and weak*-compactness ofthe sets of minimizers for the infinite-dimensional optimizationproblems of Section IV. This result is preparatory for the proofof all representer theorems.

When the operator L is invertible (including the simplestcase of the identity), the functional that we minimize iscoercive, convex, and lower semi-continuous. In that case,Lemma 20 below can be deduced from standard resultsin convex optimization [42, Section II-1]. However, whenthe operator L has a non-trivial null space, the functional‖L{x}‖pp is not coercive anymore and the proof must beadapted. This is the main contribution of Lemma 20.

Lemma 20: For 1 ≤ p ≤ 2 fixed, let us consider thefollowing:

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 13

• L : `p,L(Z) → `p(Z) is an admissible regularizationoperator in the sense of Definition 15;

• H : `p,L(Z)→ RM is a linear measurement operator suchthat, for any x ∈ `p,L(Z),

A‖〈x,φ〉‖2 ≤ ‖H{x}‖2 ≤ B(‖L{x}‖`p + ‖〈x,φ〉‖2)(40)

for some constants A,B > 0 and φ as in Theorem 16;• C is a compact convex subset of RM such that its

preimage in `p(Z), U = H−1(C), is nonempty (feasibilityhypothesis).

Then,

V = arg minx∈`p,L(Z)

‖L{x}‖`p s.t. H{x} ∈ C

is a nonempty, convex, weak*-compact subset of `p,L(Z).When the operator L is a bijection the conclusion remains

valid for any closed convex, but not necessarily bounded, setC.

Proof: The measurement operator H is linear by assump-tion and bounded on `p,L(Z) due to (38); therefore, it iscontinuous. The set U is closed convex as the preimage ofa closed convex set by the linear and continuous map H.

Next, we show that V is nonempty. Let (xn) be a se-quence of elements of U such that ‖L{xn}‖`p decreases toβ = infx∈U‖L{x}‖`p . Based on Theorem 17, we decomposexn = L−1φ {wn} + pn in a unique way with wn ∈ `p(Z) andpn ∈ NL. Then, ‖wn‖`p = ‖L{xn}‖`p is bounded. Moreover,thanks to the lower bound in (40), we have

‖〈pn,φ〉‖2 ≤ 1

A‖H{pn}‖2 =

1

A‖H{xn} −HL−1φ {wn}‖2

≤ 1

A

(‖H{xn}‖2 + ‖HL−1φ {wn}‖2

). (41)

The H{xn} are inside the bounded set C so that ‖H{xn}‖2is bounded as well. Moreover, the composed operator HL−1φ

is continuous from `p(Z) to RM and (wn) is bounded in`p(Z), so that ‖HL−1φ {wn}‖2 is bounded too. This showsthat ‖〈pn,φ〉‖2 is bounded. The space NL being finite-dimensional, we can therefore extract a subsequence of (pn)that converges to p∞ ∈ NL. Since the sequence (wn) isbounded in `p(Z), we also extract a subsequence that con-verges to w∞ ∈ `p(Z) for the weak*-topology. Finally, a dou-ble extraction allows us to consider xϕ(n) = L−1φ {wϕ(n)} +

pϕ(n) that converges to x∞ = L−1φ {w∞}+p∞ for the weak*-topology on `p,L(Z). Then, the space U is closed and thereforeweak*-closed; hence, x∞ ∈ U as a weak*-limit of elementsin U . Moreover, ‖L{x∞}‖`p ≤ ‖L{xϕ(n)}‖`p → β. Sincex∞ ∈ U , we also have ‖L{x∞}‖`p ≥ β and thereforex∞ ∈ V , which is therefore nonempty.

Moreover, we can write V = U ∩ B with B = {x ∈`p,L(Z), ‖L{x}‖`p ≤ β}. The space B is convex and weak*-compact in `p,L(Z) due to the Banach-Alaoglu theorem.Therefore, V is itself convex and weak*-compact as theintersection of two convex sets, one being weak*-compact andthe other weak*-closed.

Finally, when the null space of L is trivial, the bound (41)is not required, so that we do not need the compactness of C.

REFERENCES

[1] A. N. Tikhonov, “Solution of incorrectly formulated problems and theregularization method,” Soviet Mathematics, vol. 4, pp. 1035–1038,1963.

[2] A. Bjorck, Numerical Methods for Least Squares Problems. SIAM,1996.

[3] M. Bertero and P. Boccacci, Introduction to Inverse Problems in Imag-ing. CRC press, 1998.

[4] A. M. Bruckstein, D. L. Donoho, and M. Elad, “From sparse solutions ofsystems of equations to sparse modeling of signals and images,” SIAMReview, vol. 51, no. 1, pp. 34–81, 2009.

[5] M. Elad, Sparse and Redundant Representations. From Theory toApplications in Signal and Image Processing. Springer, 2010.

[6] D. L. Donoho, “Compressed sensing,” IEEE Transactions on Informa-tion Theory, vol. 52, no. 4, pp. 1289–1306, 2006.

[7] E. Candes and J. Romberg, “Sparsity and incoherence in compressivesampling,” Inverse Problems, vol. 23, no. 3, pp. 969–985, 2007.

[8] M. A. T. Figueiredo and R. D. Nowak, “An EM algorithm for wavelet-based image restoration,” IEEE Transactions on Image Processing,vol. 12, no. 8, pp. 906–916, 2003.

[9] M. Lustig, D. L. Donoho, and J. M. Pauly, “Sparse MRI: The applicationof compressed sensing for rapid MR imaging,” Magnetic Resonance inMedicine, vol. 58, no. 6, pp. 1182–1195, 2007.

[10] M. Figueiredo, R. Nowak, and S. Wright, “Gradient projection for sparsereconstruction: Application to compressed sensing and other inverseproblems,” IEEE Journal of Selected Topics in Signal Processing, vol. 1,no. 4, pp. 586–597, Dec 2007.

[11] D. L. Donoho and M. Elad, “Optimally sparse representation in general(nonorthogonal) dictionaries via `1 minimization,” Proceedings of theNational Academy of Sciences, vol. 100, no. 5, pp. 2197–2202, 2003.

[12] H. Rauhut, K. Schnass, and P. Vandergheynst, “Compressed sensingand redundant dictionaries,” IEEE Transactions on Information Theory,vol. 54, no. 5, pp. 2210–2219, 2008.

[13] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noiseremoval algorithms,” Physica D, vol. 60, no. 1-4, pp. 259–268, 1992.

[14] I. Daubechies, M. Defrise, and C. De Mol, “An iterative thresholdingalgorithm for linear inverse problems with a sparsity constraint,” Com-munications on Pure and Applied Mathematics, vol. 57, no. 11, pp.1413–1457, 2004.

[15] T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 2,pp. 323–343, 2009.

[16] A. Chambolle and P.-L. Lions, “Image recovery via total variationminimization and related problems,” Numerische Mathematik, vol. 76,no. 2, pp. 167–188, 1997.

[17] G. Steidl, S. Didas, and J. Neumann, “Splines in higher order TVregularization,” International Journal of Computer Vision, vol. 70, no. 3,pp. 241–255, 2006.

[18] K. Bredies, K. Kunisch, and T. Pock, “Total generalized variation,” SIAMJournal on Imaging Sciences, vol. 3, no. 3, pp. 492–526, 2010.

[19] F. Karahanoglu, I. Bayram, and D. Van De Ville, “A signal processingapproach to generalized 1D total variation,” IEEE Transactions on SignalProcessing, vol. 59, no. 11, pp. 5265–5274, Nov. 2011.

[20] M. Benning, C. Brune, M. Burger, and J. Muller, “Higher-order TVmethods: Enhancement via Bregman iteration,” Journal of ScientificComputing, vol. 54, no. 2-3, pp. 269–310, 2013.

[21] G. Demoment, “Image reconstruction and restoration—Overview ofcommon estimation structures and problems,” IEEE Transactions onAcoustics Speech and Signal Processing, vol. 37, no. 12, pp. 2024–2036,1989.

[22] N. B. Karayiannis and A. N. Venetsanopoulos, “Regularization theoryin image restoration—The stabilizing functional approach,” IEEE Trans-actions on Acoustics, Speech and Signal Processing, vol. 38, no. 7, pp.1155–1179, 1990.

[23] D. Geman and G. Reynolds, “Constrained restoration and the recoveryof discontinuities,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 14, no. 3, pp. 367–383, 1992.

[24] S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions,and the Bayesian restoration of images,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 6, no. 6, pp. 721–741, 1984.

[25] M. Unser and P. D. Tafti, An Introduction to Sparse Stochastic Processes.Cambridge University Press, 2014.

[26] R. Tibshirani, “Regression shrinkage and selection via the Lasso,”Journal of the Royal Statistical Society. Series B, vol. 58, no. 1, pp.265–288, 1996.

IEEE TRANSACTIONS ON INFORMATION THEORY, IN PRESS 14

[27] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algo-rithm for linear inverse problems,” SIAM Journal on Imaging Sciences,vol. 2, no. 1, pp. 183–202, 2009.

[28] M. V. Afonso, J. M. Bioucas-Dias, and M. A. T. Figueiredo, “An aug-mented Lagrangian approach to the constrained optimization formulationof imaging inverse problems,” IEEE Transactions on Image Processing,vol. 20, no. 3, pp. 681–695, march 2011.

[29] E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery fromincomplete and inaccurate measurements,” Communications on Pure andApplied Mathematics, vol. 59, no. 8, pp. 1207–1223, 2006.

[30] E. J. Candes, “The restricted isometry property and its implications forcompressed sensing,” C. R. Acad. Sci., vol. 346, no. 9, pp. 589–592,2008.

[31] D. L. Donoho, “For most large underdetermined systems of linearequations the minimal `1 norm solution is also the sparsest solution,”Communications on pure and applied mathematics, vol. 59, no. 6, pp.797–829, 2006.

[32] S. Foucart and H. Rauhut, A Mathematical Introduction to CompressiveSensing. Springer, 2013.

[33] E. Van Den Berg and M. P. Friedlander, “Probing the Pareto frontier forbasis pursuit solutions,” SIAM Journal on Scientific Computing, vol. 31,no. 2, pp. 890–912, 2008.

[34] P. G. Ciarlet, Linear and Nonlinear Functional Analysis with Applica-tions. SIAM, 2013.

[35] W. Rudin, Functional Analysis, 2nd ed., ser. International Series in Pureand Applied Mathematics. McGraw-Hill, Inc., New York, 1991.

[36] F. Treves, Topological Vector Spaces, Distributions and Kernels. DoverPublications, 2006.

[37] P. Prenter, Splines and Variational Methods. New York: Wiley, 1975.[38] L. L. Schumaker, Spline functions: Basic theory, 3rd ed. Cambridge:

Cambridge Univ. Press, 2007.[39] O. L. Mangasarian and L. L. Schumaker, “Discrete splines via math-

ematical programming,” SIAM Journal on Control, vol. 9, no. 2, pp.174–183, 1971.

[40] I. J. Schoenberg, “Spline functions and the problem of graduation,” Proc.Nat. Acad. Sci., vol. 52, no. 4, pp. 947–950, October 1964.

[41] M. Unser, J. Fageot, and J. P. Ward, “Splines are universal solu-tions of linear inverse problems with generalized-TV regularization,”arXiv:1603.01427 [math.FA], 2016.

[42] I. Ekeland and R. Temam, Convex Analysis and Variational Problems.SIAM, 1999.

Michael Unser (M’89–SM’94–F’99) is professor and director of EPFL’sBiomedical Imaging Group, Lausanne, Switzerland. His primary area ofinvestigation is biomedical image processing. He is internationally recognizedfor his research contributions to sampling theory, wavelets, the use of splinesfor image processing, stochastic processes, and computational bioimaging. Hehas published over 250 journal papers on those topics. He is the author with P.Tafti of the book, An Introduction to Sparse Stochastic Processes, CambridgeUniversity Press 2014.

From 1985 to 1997, he was with the Biomedical Engineering and Instru-mentation Program, National Institutes of Health, Bethesda USA, conductingresearch on bioimaging.

Dr. Unser has held the position of associate Editor-in-Chief (2003-2005)for the IEEE Transactions on Medical Imaging. He is currently member of theeditorial boards of SIAM J. Imaging Sciences and Foundations and Trends inSignal Processing. He is the founding chair of the technical committee on BioImaging and Signal Processing (BISP) of the IEEE Signal Processing Society.Prof. Unser is a fellow of the IEEE (1999), an EURASIP fellow (2009), anda member of the Swiss Academy of Engineering Sciences. He is the recipientof several international prizes including three IEEE-SPS Best Paper Awardsand two Technical Achievement Awards from the IEEE (2008 SPS and EMBS2010).

Julien Fageot graduated from the Ecole Normale Superieure, Paris, France,in 2012. He received the M.Sc. degree in mathematics from the UniversiteParis-Sud, France, in 2009 and the M.Sc. degree in imaging science from theEcole Normale Superieure, Cachan, France, in 2011. He is currently pursuingthe Ph.D. degree with the Biomedical Imaging Group under the direction ofM. Unser. He is mainly working on random processes and their applications tosignal processing. His research interests include stochastic models for sparsesignals and spline theory.

Harshit Gupta received the B. Tech. in Electronics and CommunicationEngineering in 2015 from the Indian Institute of Technology, Guwahati, India.He is currently pursuing a Ph.D. degree with the Biomedical Imaging Groupunder the direction of M. Unser. His research focuses on splines, regularizationtheory and the resolution of inverse problems in imaging.