Primal - dual interior - point methods for semidefinite programming : Stability, convergence, and numerical results

Primal-Dual Interior-Point Methods for Semide�niteProgramming: Convergence Rates, Stability andNumerical Results �Farid Alizadehy Jean-Pierre A. Haeberlyz Michael L. OvertonxMay 27, 1997AbstractPrimal-dual interior-point path-following methods for semide�niteprogramming (SDP) are considered. Several variants are discussed,based on Newton's method applied to three equations: primal feasibil-ity, dual feasibility, and some form of centering condition. The focusis on three such algorithms, called respectively the XZ, XZ+ZX andQ methods. For the XZ+ZX and Q algorithms, the Newton system iswell-de�ned and its Jacobian is nonsingular at the solution, under non-degeneracy assumptions. The associated Schur complement matrix hasan unbounded condition number on the central path, under the non-degeneracy assumptions and an additional rank assumption. Practicalaspects are discussed, including Mehrotra predictor-corrector variantsand issues of numerical stability. Compared to the other methods con-sidered, the XZ+ZX method is more robust with respect to its abilityto step close to the boundary, converges more rapidly, and achieveshigher accuracy.�NYU Computer Science Dept Technical Report 721, May 1996, Revised Feb 1997.Updated versions will be made available by anonymous ftp to cs.nyu.edu, in the �lepub/local/overton/pdsdp.ps.gz.yRUTCOR, Rutgers University, New Brunswick, NJ. Supported in part by NationalScience Foundation grant CCR-9501941 and O�ce of Naval Research contract N00014-96-1-0704. E-mail: [email protected] Department, Fordham University, Bronx, NY. Supported in part by Na-tional Science Foundation grant CCR-9401119. E-mail:[email protected] Science Department, Courant Institute of Mathematical Sciences, New YorkUniversity, New York, NY. Supported in part by National Science Foundation grant CCR-9625955. E-mail: [email protected] 1

Semide�nite Programming 21 IntroductionLet Sn denote the vector space of real symmetric n�n matrices. Denote thedimension of this space by n2 = n(n+ 1)2 : (1)The standard inner product on Sn isA �B = tr AB =Xi;j AijBij :By X � 0 (X � 0), where X 2 Sn, we mean that X is positive semide�nite(positive de�nite).Consider the semide�nite program (SDP)minX2Sn C �Xs:t: Ak �X = bk; k = 1; : : : ; mX � 0; (2)where b 2 Rm, C 2 Sn, and Ak 2 Sn, k = 1; : : : ; m. The dual SDP ismaxy2Rm;Z2Sn bTys:t: Pmk=1 ykAk + Z = C;Z � 0: (3)The following are assumed to hold throughout the paper.Assumption 1. There exists a primal feasible point X � 0, and a dualfeasible point (y; Z) with Z � 0.Assumption 2. The matrices Ak; k = 1; : : : ; m, are linearly independent,i.e. they span an m-dimensional linear space in Sn.The central path consists of points (X�; y�; Z�) 2 Sn�Rm�Sn satisfyingthe primal and dual feasibility constraints as well as the centering conditionX�Z� = �I (4)for some � 2 R, � > 0. It is well known [NN94, KSH97] that, under Assump-tions 1 and 2, (X�; y�; Z�) exists and is unique for all � > 0, and that(X; y; Z) = lim�!0(X�; y�; Z�) (5)

Semide�nite Programming 3exists and solves the primal and dual SDP's. Furthermore, because X� andZ� commute, there exists an orthogonal matrix Q� such thatX� = Q� Diag(��1 ; : : : ; ��n) (Q�)T ; Z� = Q� Diag(!�1 ; : : : ; !�n) (Q�)T ;(6)where the ��i and !�i , respectively the eigenvalues of X� and Z�, satisfy��i !�i = �; i = 1; : : : ; n: (7)Without loss of generality, assume that��1 � � � � � ��n and !�1 � � � � � !�n : (8)As �! 0, the centering condition (4) reduces to the complementarity condi-tion XZ = 0, implying thatX = Q Diag(�1; : : : ; �n) QT ; Z = Q Diag(!1; : : : ; !n) QT ; (9)for some orthogonal matrixQ, with the eigenvalue complementarity condition�i!i = 0; i = 1; : : : ; n. Observe that �i and !i are the limits of ��i and !�i as� ! 0, and Q may be taken to be a limit point (not necessarily unique) ofthe set fQ� : � > 0g. We have�1 � � � � � �n and !1 � � � � � !n: (10)Interior point methods for semide�nite programming were originally in-troduced by [NN94, Ali91]. Early papers on primal-dual methods include[VB95] and [HRVW96]. A preliminary version of the present work appearedas [AHO94b]. Convergence analysis of primal-dual path-following meth-ods for SDP appeared �rst in [KSH97, NT97b, NT97a]. We are primarilyconcerned with four methods, which we call respectively the XZ, XZ+ZX,Nesterov-Todd (NT) and Q methods. The XZ method �rst appeared in[HRVW96, KSH97]. The XZ+ZX method was introduced in [AHO94b] andwas recently analyzed by [KSS96, Mon96]. The NT method was given by[NT97b, NT97a] and its implementation was recently discussed in [TTT97].The Q method originally appeared in [AHO94a]. Many other papers onsemide�nite programming have recently been announced.The paper is organized as follows. In Section 2 we introduce several al-gorithms in a common framework based on Newton's method, focusing onthe XZ and XZ+ZX variants. In Section 3 we study the Jacobian of theNewton system for the various methods under nondegeneracy assumptions,and discuss implications for local convergence rates. In Section 4 we considerthe conditioning of the Schur complement matrix on the central path, again

Semide�nite Programming 4under nondegeneracy assumptions. This leads to the issue of numerical sta-bility, discussed in Section 5. We introduce the Q method in Section 6. InSection 7, we present computational results.Our main focus is on the nondegenerate case; this assumption (de�ned inSection 3) implies unique primal and dual solutions. We take the view that itis important to understand how methods behave on nondegenerate problems.This does not discount the signi�cance of degenerate problems that may arisein applications, as is common in linear programming (LP).In practice, many semide�nite programs are block diagonal. Everythingin this paper extends easily to the block diagonal case. Note that LP is thespecial case where all block sizes are one.A word about notation: we use the symbols X , y and Z to mean severalthings. Depending on the context, they may refer to the variables of the SDP,the iterates generated by a method, or a solution of the SDP.2 The Methods in a General FrameworkWe consider only primal-dual interior-point path-following methods, generat-ing a sequence of iterates approximating the central path and converging tothe primal and dual solutions. See [Wri97] for a detailed discussion of suchmethods for LP. In LP, the basic iterative step can be readily derived usingNewton's method. For SDP, points on the central path satisfy the nonlinearequation 266664Pmk=1 ykAk + Z � CA1 �X � b1...Am �X � bmXZ � �I 377775 = 0: (11)However, the matrix XZ is not symmetric in general. Consequently, thedomain and range of the function de�ned by the left-hand side of (11) are notthe same spaces, and Newton's method is not directly applicable. For LP, onthe other hand, the standard primal-dual interior-point method is obtainedby applying Newton's method to (11). In this case, X and Z are diagonal,and XZ is also diagonal, so the domain and range of (11) reduce to R2n+m.A key question in formulating primal-dual interior-point methods for SDPis therefore: how should one appropriately formulate Newton's method? Weconsider here two possibilities. Other choices are discussed at the end of thissection.The XZ Method. Use the centering condition (4) directly and view the

Semide�nite Programming 5left-hand side of (11) as a function whose domain and range are bothU = Rn�n�Rm�Rn�n. Then Newton's method is well de�ned, thoughthe iterates are not symmetric matrices. (Actually, only the X iteratesare not symmetric, since the dual feasibility equation forces Z to besymmetric.) The X iterates can then be explicitly symmetrized beforecontinuing with the next iteration. Consequently, this method is notstrictly a Newton method. A di�erent iteration is obtained by usingZX = �I instead of (4).The XZ + ZX Method. Rewrite (4) in the symmetric formXZ + ZX = 2�I: (12)Substituting (12) for (4) in (11) gives a mapping with domain and rangeboth given by V = Sn �Rm� Sn. Application of Newton's method to(12) leads to symmetric matrix iterates X and Z.We observe that (4) and (12) are equivalent when X � 0 (or Z � 0).That (4) implies (12) is immediate. That the converse holds for X � 0 isseen by using X = Q�QT to reduce (12) to �(QTZQ) + (QTZQ)� = 2�I ,with � diagonal and nonnegative and QTQ = I . The entries on the left-handside are (�i + �j)(QZQT )ij , and so, since the o�-diagonal entries must bezero, either �i = �j = 0 or (QZQT )ij = 0 when i 6= j. Thus, �(QTZQ) isdiagonal, and (4) holds.We now examine the steps de�ned by these methods in more detail. TheNewton step for the XZ method satis�es the linear equationX�Z +�XZ = �I �XZ: (13)Let nvec map Rn�n to Rn2, stacking the columns of a matrix in a vector.Then we may rewrite (13) in the form(I X)nvec(�Z) + (Z I)nvec(�X) = nvec (�I �XZ): (14)where denotes the standard Kronecker product (see Appendix, equation(59)).To discuss the XZ+ZX method, we introduce a symmetric version of theKronecker product. The Newton correction for (12) satis�es the linear equa-tion X�Z +�Z X +�X Z + Z�X = 2�I �XZ � ZX; (15)where �X and �Z are symmetric. Let svec be an isometry identifyingSn with Rn2, so that K � L = svec (K)T svec (L) for all K;L 2 Sn (see

Semide�nite Programming 6Appendix). Then (15) can be written as:(Z ~ I) svec (�X)+ (X ~ I) svec (�Z) = svec (�I � 12(XZ +ZX)) (16)where ~ denotes the symmetric Kronecker product de�ned in the Appendix(see (62)).We shall now describe both methods in a common framework. Let vecdenote either nvec or svec , depending on the context. Speci�cally, vec willmean nvec in the case of the XZ method and svec otherwise. The inverseof vec is denoted by mat . We shall use lower case letters x and z to denotevecX and vecZ respectively, and we shall use �x and �z interchangeablywith vec�X and vec�Z, to be de�ned shortly.Let A = 24 (vecA1)T...(vecAm)T 35 (17)and de�ne rp = b�Ax; Rd = C � Z � matATy;and Rc = � �I �XZ XZ method�I � 12(XZ + ZX) XZ + ZX method � (18)with rd = vecRd; rc = vecRc:Let G(x; y; z) = 24�rd�rp�rc 35 : (19)Note that G maps U to U in the case of the XZ method and V to V otherwise.Application of one step of Newton's method to G(x; y; z) = 0 gives the linearsystem 24 0 AT IA 0 0E 0 F3524�x�y�z 35 = 24 rdrprc 35 : (20)Here E = � Z I XZ methodZ ~ I XZ + ZX method �and F = � I X XZ methodX ~ I XZ + ZX method �

Semide�nite Programming 7and I is the identity matrix of appropriate dimension (II for the XZ methodand I ~ I for the XZ+ZX method). We denote the Jacobian matrix on theleft-hand side of (20) by J.Applying block Gauss elimination, (20) reduces to the system� �F�1E ATA 0 �� �x�y � = � rd � F�1rcrp � (21)A second step of block Gauss elimination givesM�y = rp +AE�1(Frd� rc); (22)�x = �E�1(F(rd �AT�y)� rc); (23)and, from the dual feasibility equation,�z = rd �AT�y; (24)where M = AE�1FAT : (25)We call M the Schur complement matrix. The main computational work isthe formation and factorization ofM. The kth column of the matrixE�1FATis � nvec (XAkZ�1) XZ methodsvec (Gk) XZ + ZX method �where Gk is the solution of the Lyapunov equation (see Appendix)ZGk + GkZ = XAk +AkX: (26)Formation of M thus requires O(mn3 +m2n2) work, involving a Choleskyfactorization of Z, in the case of the XZ method, and an eigenvalue factor-ization of Z, in the case of the XZ+ZX method (see Appendix). Neglectingsparsity considerations, the additional cost of the eigenvalue factorization isnegligible in comparison to the other operations required to formM.It is clear that, as long as X � 0 and Z � 0, nonsingularity of theJacobian matrix J is equivalent to nonsingularity of the Schur complementM. In the case of the XZ method, M is symmetric and positive de�nite. Inthe case of the XZ+ZX method,M is not symmetric, but can be shown to benonsingular if XZ+ZX � 0 [SSK96]. An alternative condition guaranteeingthe nonsingularity ofM is given in [MZ96]. Equation (22) is solved by usinga Cholesky factorization of M in the case of the XZ method and an LUfactorization of M in the case of the XZ+ZX method.

Semide�nite Programming 8For the XZ+ZX method, the multiplications by E�1 in (22) and (23)require the solution of Lyapunov equations, using the eigenvalues of Z alreadycomputed to form M.Both methods are then described by the following:Basic Iteration.1. Choose 0 � � < 1 and de�ne� = �X � Zn : (27)2. Determine �X , �y, �Z from (20), equivalently (22){(24).3. In the case of the XZ method, replace �X by 12(�X +�XT).4. Choose steplengths �; � and update the iterates byX X + � �Xy y + � �yZ Z + � �Z:Rules for de�ning � will be discussed later. A simple steplength rule isgiven by choosing a parameter � , 0 < � < 1, and de�ning� = min(1; � �̂) �̂ = supf�� : X + ���X � 0g (28)and � = min(1; � �̂) �̂ = supf �� : Z + ���Z � 0g: (29)Note that, except in the case �X � 0, we have 0 < �̂ <1 with�̂�1 = �max(�L�1�XL�T);where �max means largest eigenvalue and L is the Cholesky factor of X , i.e.X = LLT .Other methods can also be de�ned in the same framework; two of theseare discussed below. See [Zha97] for a class of methods that includes theXZ+ZX method, and [KSH97, SSK96] for another class that includes allthose discussed here except the XZ+ZX method.The X�1 Method. Replace Rc in (19) by Rc = �X�1 � Z, so E = �X�1 ~X�1, F = I~I . A similar method can be de�ned with Rc = �Z�1�X .In fact, the method given by [VB95] is based on a combination of thesetwo steps.

Semide�nite Programming 9The Nesterov-Todd Method. Use Rc = �X�1 � Z, E = W�1 ~W�1, F =I ~ I , where W = X1=2(X1=2ZX1=2)�1=2X1=2. This form does notactually arise from applying Newton's method to (19). However, see[TTT97] for a Newton interpretation of this method.As long as X � 0 and Z � 0, E�1F is symmetric and positive de�nite forboth these methods. However, in both cases, the function to which Newton'smethod is applied fails to exist at a solution. We call an algorithm a Newtonmethod if (�X;�y;�Z) is derived by applying Newton's method to a func-tion that is well de�ned for all X � 0, Z � 0. Under this de�nition, of thefour variants de�ned so far, only XZ+ZX is a Newton method for SDP.In the special case of LP (i.e., a block diagonal SDP with block sizes allone), the XZ, XZ+ZX and Nesterov-Todd methods coincide, giving the XZmethod for LP, which is a Newton method.In order to understand the asymptotic behavior of Newton's method, it isimportant to analyze the Jacobian at the solution itself. This is done in thenext section.3 The Jacobian at the SolutionIn this section we study the Jacobian of the function G, appearing on the left-hand side of (20), under nondegeneracy assumptions. To do this, we use thenotions of nondegeneracy that were introduced by the authors in [AHO97].De�nition 1. Let (X; y; Z) solve SDP, with an orthogonal matrix Q sat-isfying (9). Let X have rank r, with positive eigenvalues �1; : : : ; �r, andpartition Q = [Q1 Q2], where the columns of Q1 are eigenvectors correspond-ing to �1; : : : ; �r. We say that (X; y; Z) satis�es the strict complementarityand primal and dual nondegeneracy conditions if the following hold:1. rank(Z) = n� r,2. the matrices� QT1AkQ1 QT1AkQ2QT2AkQ1 0 � ; for k = 1; 2; : : : ; m, (30)are linearly independent in Sn, and3. the matrices QT1AkQ1; for k = 1; 2; : : : ; m, (31)span the space Sr.

Semide�nite Programming 10These conditions are well de�ned even if Q is not unique. The �rst re-quirement is the strict complementarity condition. Conditions (30), (31) arerespectively primal and dual nondegeneracy conditions under the assumptionof strict complementarity. They immediately imply the inequalitiesr2 � m � r2 + r(n� r) (32)(recalling the notation (1)). They also imply uniqueness of the primal anddual solutions. Furthermore, the conditions are generic properties of SDP,meaning roughly that they hold with probability one for an optimal solutiontriple, given random data with feasible solutions. For motivation of these con-ditions, and further details, see [AHO97]. The de�nitions are easily extendedto the block diagonal case, giving the usual LP nondegeneracy conditionswhen all blocks have size one.The strict complementarity condition rank(X) = r, rank(Z) = n � rimplies, using (10), that�1 � � � � � �r > �r+1 = : : : = �n = 0; (33)and 0 = !1 = � � � = !r < !r+1 � : : : � !n; (34)Let Bk = QTAkQ. From (62), we havesvecBk = (QT ~QT ) svecAk :Recall the de�nition (17), and de�neB = 264 ( svecB1)T...( svecBm)T 375 ; (35)so that A(Q~ Q) = B:Each column of B corresponds to an index pair (i; j), identifying twocolumns of Q, with 1 � i � j � n. By choosing the ordering used by thesvec operator appropriately, we may writeB = [C1 C2 C3] (36)where C1 contains r2 columns corresponding to 1 � i � j � r, C2 containsr(n � r) columns corresponding to 1 � i � r; r + 1 � j � n, and C3consists of (n� r)2 columns corresponding to r + 1 � i � j � n. The

Semide�nite Programming 11primal nondegeneracy condition (30) holds exactly when the rows of [C1 C2]are linearly independent, i.e. [C1 C2] has rank m. The dual nondegeneracycondition (31) holds exactly when C1 has rank r2, i.e. the columns of C1 arelinearly independent. Thus, the conditions (30) and (31) together imply thatit is possible to choosem� r2 columns fromC2 so that, together with all thecolumns of C1, they form a nonsingular m �m matrix. In other words, wecan choose an ordering for the columns of C2, and therefore of B, so thatB = [B1 B2] (37)where B1 2 Rm�m is nonsingular.Theorem 1 Consider an SDP whose solution (X; y; Z) satis�es the strictcomplementarity and primal and dual nondegeneracy conditions. Let J bethe Jacobian of the function G de�ning the XZ+ZX method, evaluated at(X; y; Z). Then J is nonsingular.Proof: We have J = 24 0 AT IA 0 0E 0 F35where E = Z~I and F = X~I: Let P = Q~Q, and let S = Diag(P; I;P),so that STJS = 24 0 BT IB 0 0� 0 �35with � = PTFP and � = PTEP. Using Lemma 2 (see Appendix) and (9),we see that PTP = I and � and � are diagonal with entries 12(�i + �j) and12(!i + !j), 1 � i � j � n, respectively. Notice that the diagonal entry of �corresponding to the index pair (i; j) is zero if and only if r + 1 � i � j � n(because of (33)), while the diagonal entry of � corresponding to the pair(i; j) is zero if and only if 1 � i � j � r (see (34)).Using the partitioning of B in (37), we haveSTJS = 266664 0 0 BT1 I 00 0 BT2 0 IB1 B2 0 0 0�1 0 0 �1 00 �2 0 0 �2377775 (38)

Semide�nite Programming 12where � = Diag(�1;�2) and � = Diag(�1;�2). We have �1 � 0, sincenone of the columns of C3 are included in B1, and �2 � 0, since all of thecolumns of C1 are included in B1.Interchanging the �rst and third rows and the second and last columns of(38), we obtain 266664B1 0 0 0 B20 I BT2 0 00 0 BT1 I 0�1 0 0 �1 00 �2 0 0 �2 377775 :We shall demonstrate the nonsingularity of this matrix using block Gausselimination. First, subtract �1B�11 times the �rst block row from the fourthblock row to eliminate �1 from the 4,1 position. This does not otherwisechange the lower triangle or the diagonal blocks, only introducing��1B�11 B2into the 4,5 position. Second, subtract�2 times the second block row from the�fth row, eliminating �2 from the 5,2 position; this introduces ��2BT2 intothe 5,3 position. This 5,3 block is then eliminated by adding�2BT2B�T1 timesthe third row to the �fth row, introducing �2BT2B�T1 into the 5,4 position,giving 266664B1 0 0 0 B20 I BT2 0 00 0 BT1 I 00 0 0 �1 ��1H0 0 0 �2HT �2 377775where H = B�11 B2. In order to show that this matrix is nonsingular we needonly show that the trailing 2 by 2 block is nonsingular, or equivalently thatits positive row scaling � I ���11 �1H��12 �2HT I �is nonsingular. A �nal step of block Gauss elimination yields a block uppertriangular matrix with last diagonal block given byI +��12 �2HT��11 �1H:This matrix is nonsingular, since it is of the form I +N1N2 with N1 � 0 andN2 � 0. (The product of two symmetric and positive semide�nite matrices,though nonsymmetric, has real nonnegative eigenvalues.) Q.E.D.

Semide�nite Programming 13Corollary 1 Consider an SDP whose solution (X,y,Z) satis�es the strictcomplementarity and primal and dual nondegeneracy conditions. Supposethat the XZ+ZX method uses � = 0 and � = � = 1 in the Basic Iteration.Then, there exists � > 0 such that, if the iteration is started at (X0; y0; Z0),with jj(X0; y0; Z0) � (X; y; Z)jj < �, the iterates converge Q-quadratically to(X; y; Z).The proof of Corollary 1 is immediate from the standard convergencetheory for Newton's method. It is clear that Corollary 1 holds also for lessrestrictive assumptions on �; � and �. See [ZTD92] for relevant results for LP.There is no requirement that (X0; y0; Z0) lie in a horn-shaped neighborhoodof the central path, or even in the feasible region. Note that the assumptionsof Corollary 1 do not guarantee positive de�nite iterates. These are notrequired to make (20) well-de�ned, though the equivalence of (20) with (22){(24) does not hold if X or Z is singular. In practice, conditions (28){(29)ensure positive de�nite iterates.A result like Theorem 1 does not hold for any of the other methods dis-cussed so far. As already noted, the function to which Newton's method isapplied is, in the case of the X�1 and Nesterov-Todd methods, not de�nedat an optimal point. For the XZ method, the function G is de�ned at thesolution, but it can be shown that the Jacobian J is always singular there.More importantly, bearing in mind the symmetrization step, an example canbe constructed where J has a null vector (�X;�y;�Z) with �X+�XT 6= 0.It is well known that a result like Theorem 1 holds for the XZ method forLP, using LP nondegeneracy assumptions.Nondegeneracy assumptions are not required to obtain superlinear con-vergence results. This has been known for some years for LP [Wri97] andis the subject of active current research for SDP. However, such results re-quire that the iterates of a method stay close to the central path. Our pointhere is that classical Newton theory applies to the XZ+ZX method, undernondegeneracy assumptions, in SDP just as in LP.4 Conditioning of the Schur Complement MatrixIn this section, we study the conditioning of the Schur complement matrixM, introduced in Section 2, on the central path. It is important to note that,when started on the central path, all the methods discussed so far generatethe same �rst iterate. On the central path, X and Z commute. Therefore,E�1F = 1�X ~X

Semide�nite Programming 14in all cases except the XZ method for which we have E�1F = 1�X X . Inboth cases the Schur complement matrixM = AE�1FAT is the same.We now analyze the condition number ofM on the central path, as �! 0.We begin by considering its rank in the limit.Theorem 2 Assume that (X�; y�; Z�) lies on the central path of an SDPwhose solution (X; y; Z) = lim�!0(X�; y�; Z�) satis�es the dual nondegen-eracy condition (31), with r = rank(X). Let M� be the Schur complementmatrix de�ned at (X�; y�; Z�). Thenlim�!0(�M�)exists and has rank r2.Proof: Clearly, �M� ! N = A(X ~X)ATthe matrix whose (l; k) element is tr (XAlXAk). LetQ and �i satisfy (9), andwrite �1 = Diag(�1; : : : ; �r) � 0, with corresponding eigenvectors collectedin Q1, so that X = Q1�1QT1 . Let C1 be the m � r2 matrix introduced in(36), and let D1 be the r2 � r2 diagonal matrixD1 = Diag(�i�j); 1 � i � j � r;using consistent orderings for C1 and D1. ThenN = C1D1CT1since the (l; k) element of the right-hand side istr (�1QT1AlQ1�1QT1AkQ1) = tr (XAlXAk):Since, by the dual nondegeneracy assumption, C1 has linearly independentcolumns, and since D1 � 0, this completes the proof of the theorem. Q.E.D.Recall that the condition number of a symmetric positive de�nite matrixis �max=�min, where �max and �min are respectively its largest and its smallesteigenvalues.Theorem 3 Suppose that the assumptions of Theorem 2 hold. Then, if m >r2 > 0, the condition number of M� (equivalently of �M�) is bounded belowby a positive constant times 1=�.

Semide�nite Programming 15Proof: Let Q�, ��i satisfy (6), and let B�;C� be the matrices introducedin Section 3, evaluated at (X�; y�; Z�). Using Lemma 2 (see Appendix), wehave �M� = A(X� ~X�)AT = B�D�(B�)T (39)where D� is the diagonal n2 � n2 matrixD� = Diag(��i ��j ); 1 � i � j � n: (40)The primal solution rank r de�nes a splittingD� = Diag(D�1 ;D�2 ;D�3)consistent with (36), so that�M� = C�1D�1(C�1)T +C�2D�2(C�2)T +C�3D�3(C�3)T : (41)Here the entries of the diagonal matrices D�1 , D�2 and D�3 are ��i ��j , withthe indices 1 � i � j � r for D�1 , 1 � i � r < j � n for D�2 , andr + 1 � i � j � n for D�3 . Although Q� and C� do not generally convergeas � ! 0, Theorem 2 shows that �M� ! N = C1D1CT1 , with rank r2. Byassumption, m > r2 > 0, so the largest eigenvalue of N is positive and thesmallest is zero. The norms of the second and third terms in (41) are O(�), sothe largest and smallest eigenvalues of �M� are, respectively, bounded awayfrom zero, and O(�). (Here we use the fact that eigenvalues of a symmetricmatrix are Lipschitz continuous functions of the matrix entries.) Q.E.D.Theorem 3 is easily extended to the block diagonal case. When all blocksizes are one, the condition onm in its hypothesis cannot hold under the non-degeneracy assumptions. Indeed, it is well known that for LP, under assump-tions of nondegeneracy and strict complementarity, the condition number ofthe Schur complement matrix is bounded independent of �.5 StabilityWe have seen in the previous section that, for nondegenerate SDP's, the con-dition number of the Schur complement matrix, evaluated on the central path,is bounded below by a positive constant times 1=� (ruling out the exceptionalcases r2 = m and r = 0). Consequently, one expects that as �! 0, the com-putation of �y in (22) will become increasingly less accurate. Indeed, in ouroriginal implementations we observed numerical instability leading to signif-icant loss of primal feasibility near a solution. Recently, however, Todd, Toh

Semide�nite Programming 16and T�ut�unc�u [TTT97] found that high accuracy is achievable. The main issueis the choice of formulas for �y and �X . Several mathematically equivalentchoices are possible, but these have quite di�erent stability properties.Formulas for �y and �X are given in (22) and (23). Both include theterm Frd � rc. For the XZ+ZX method, this term (in matrix form) is12 �(X(C � Z � matAT y) + (C � Z � matAT y)X)� (2�I �XZ � ZX)�which can be rewritten as12 �(X(C � matAT y) + (C � matAT y)X)� 2�I� :However, using this simpli�cation to modify (22) and (23) leads to instabilityand loss of primal feasibility. It is much better to implement (22) and (23)directly. This is done in the computational experiments reported in Section 7.The same issue applies to the XZ method. However, direct implementationof (22) and (23) does not give good results for the XZ method. Instead1, weuse the fact that E�1F is symmetric positive de�nite to writeE�1F = Z�1 X =GTG; G =M�1 LT ;where L and M are respectively Cholesky factors of X and Z, i.e.X = LLT ; Z =MMT :Noting that the �rst block in the right-hand side of (21) isu = vecU = vec (C � �X�1 � matAT y);we see that (21) is equivalent to" �I eATeA 0 #� f�x�y � = � ~urp � ; (42)f�x =G�T�x = vec (L�1�XM) = vecg�X;~u =Gu = vec (LTUM�T ) = vec eU;and eA = AGT = 264 (vecLTA1M�T )T...(vecLTAmM�T )T 375 : (43)1The discussion here is motivated by [TTT97].

Semide�nite Programming 17The solution is given by (eAeAT )�y = rp + eA~u (44)(which may be solved with a Cholesky factorization) and�X = Lg�XM�1 = L(mat eAT�y � eU)M�1: (45)This last equation can be written in many ways, three of which are�X = L �LT (matAT�y)M�T � LTUM�T �M�1 (46)= LLT (matAT�y)M�TM�1 � LLTUM�TM�1 (47)= LLT �(matAT�y)� U�M�TM�1: (48)Of these four mathematically equivalent formulas, (45) and (46) give thehighest accuracy, with smallest loss of primal feasibility. We used (45) in ourcomputational experiments, with �y de�ned by (44).For the Nesterov-Todd method, E�1F is also symmetric positive de�nite,so similar considerations apply; see [TTT97].6 The Q MethodIn this section we change direction, deriving an alternative primal-dual interior-point method that generates iterates (X; y; Z) with the property that X andZ commute, i.e. XZ = ZX . This is motivated by the fact that this propertyholds for all points on the central path. Instead of treating the variables Xand Z directly, we introduce as variables the eigenvalues of X and Z andtheir common set of eigenvectors. In other words, the variables consist of anorthogonal matrix Q, diagonal matrices � and and a vector y 2 Rn thatmust satisfy QQT + mXk=1 ykAk = C;Ak � (Q�QT ) = bk; k = 1; : : : ; m (49)� = �I:This de�nes a map fromOn�R2n+m to Rn2+n+m, where On is the Lie groupof orthogonal matrices with determinant one, whose dimension is n(n� 1)=2.(Since the signs of eigenvectors are arbitrary, it is not a restriction to imposedetQ = 1.) The price paid for the diagonalization is the nonlinear appearanceof the variable Q in the feasibility equations.

Semide�nite Programming 18Let Kn denote the space of n� n skew-symmetric matrices, and considerthe exponential map from Kn to On de�ned byexp(S) = I + S + 12S2 + � � �This map is smooth, onto, and in a neighborhood of 0, also one-to-one. Bor-rowing a technique used by [OW95], we derive a form of Newton's methodbased on parameterizing On near a given point Q by Q exp(S). Let kvecbe an isometry from Kn to Rn(n�1)=2, stacking the upper triangular entriesof a skew-symmetric matrix in a vector, with a factor of p2 to preserve theinner product. Let us use the convention s = kvec (S), � = Diag(�) and = Diag(!). De�neGQ(�; y; !; s) = 24 vec (C � Q exp(S) exp(�S)QT)�AT yb�Avec (Q exp(S)� exp(�S)QT )�e � �e 35 ; (50)The function GQ maps Rn2+n+m to itself. Note that the third component ofGQ has the form familiar from LP.Given an iterate (X; y; Z) = (Q�QT ; y; QQT), we obtain a new iterateby applying Newton's method to the equationGQ = 0 at the point (�; y; !; 0).The Newton step (��;�y;�!; s) is obtained by replacing exp(S) by I + Sand discarding second-order terms. The resulting (n2+n+m)�(n2+n+m)linear system is:�+ S� S + mXk=1�ykBk = H � (51)Bk � (��+ S�� �S) = bk �Bk � �; k = 1; :::; m (52)��+�� = �I � � (53)where Bk = QTAkQ and H = QTCQ�Pmk=1 ykBk .The basic iteration for the Q method is therefore:1. Choose 0 � � < 1 and de�ne � = ��T!n :2. Determine (��;�y;�!; s) from (51){(53).

Semide�nite Programming 193. Choose steplengths �; �; and update the iterates by� � + � ��y y + � �y + � �Q Q(I + 12 S)(I � 12 S)�1:A simple steplength rule is � = min(1; � �̂), � = min(1; � �̂), and = p��,where �̂ and �̂ are steps to the boundary of the positive orthant. The mul-tiplicative factor updating Q is the Cayley transform, an easily computedorthogonal matrix that approximates the matrix exponential to second order.The equations de�ning the Q method can be rewritten as follows. Firstnote that (52) can be rewritten asBk ���+ tr ((�Bk � Bk�)S) = bk �Bk � �and write v = 24 b1 �B1 � �...bm �Bm � �35 :Let diag(Bk) be the vector consisting of the n diagonal entries of Bk ando�diag(Bk) be the vector consisting of the n(n � 1)=2 entries of the uppertriangle of Bk, ordered consistently with the ordering chosen for the kvecoperator. De�ne L = [diag(B1) � � � diag(Bm) ]T ;R = [o�diag(B1) � � � o�diag(Bm) ]T :Let D = Diag(�i � �j); E =Diag(!i � !j);diagonal matrices of size n(n�1)=2 (corresponding to 1 � i < j � n), whoseorderings are also consistent with that of the kvec operator. Then, writingthe diagonal and o�-diagonal parts of (51) separately, we get the linear system2664 0 0 LT I0 E RT 0L RD 0 0 0 0 �37752664��s�y�! 3775 = 2664diag(H � )o�diag(H)v�e � �e 3775 : (54)We denote the matrix on the left-hand side of (54) by JQ.

Semide�nite Programming 20Let (X; y; Z) be a solution of SDP satisfying (33), (34). The matrix Qsimultaneously diagonalizing X and Z is unique (up to signs of its columns)if and only if �1 > � � � > �r > 0 and 0 < !r+1 < � � �< !n: (55)Theorem 4 Let (X; y; Z) = (Q�QT ; y; QQT) be a solution of SDP satisfy-ing the strict complementarity and primal and dual nondegeneracy conditions,and also condition (55). Then the matrix JQ, evaluated at the solution, isnonsingular.Proof: First note that the assumptions on the eigenvalues imply that theelement of the diagonal matrix D corresponding to the index pair (i; j) iszero if and only if r+1 � i < j � n, while the element of the diagonal matrixE corresponding to (i; j) is zero if and only if 1 � i < j � r. Let us rewriteJQ as 26666664 0 0 0 LT1 I 00 0 0 LT2 0 I0 0 E RT 0 0L1 L2 RD 0 0 00 0 0 0 �1 00 2 0 0 0 037777775where �1 � 0 and 2 � 0. As in the proof of Theorem 1, the nondegeneracyassumptions permit us to collect all r columns of L1 andm� r columns of Rtogether in a nonsingular m�m matrix B1. We collect the remaining n(n�1)=2�m+ r columns of R in a matrixR2, and partition D = Diag(D1; D2)and E = Diag(E1; E2) accordingly. Observe that D1 � 0 since the columnsof B1 correspond to index pairs (i; j) with �i > �j . Likewise �E2 � 0 sinceall columns corresponding to index pairs (i; j) with !i = !j = 0 are containedin B1.Let eD = Diag(I;D1) and eE = Diag(0; E1). Permuting the rows andcolumns, JQ becomes26666664 eE 0 0 BT1 eI 00 0 0 LT2 0 I0 0 E2 RT2 0 0B1 eD L2 R2D2 0 0 00 0 0 0 �1 00 2 0 0 0 037777775where eI is an m by r matrix containing r rows of the r by r identity matrixand m� r zero rows. Interchanging the �rst and fourth rows and the second

Semide�nite Programming 21and last columns, this becomes26666664B1 eD 0 R2D2 0 0 L20 I 0 LT2 0 00 0 E2 RT2 0 0eE 0 0 BT1 eI 00 0 0 0 �1 00 0 0 0 0 237777775Performing Gauss block elimination on this matrix we see that its nonsingu-larity is equivalent to the nonsingularity ofBT1 + eE eD�1B�11 R2D2E�12 RT2 :Multiplying on the left by B�T1 we obtain the matrixI + (B�T1 eE eD�1B�11 )(R2D2E�12 RT2 ):This is nonsingular since it is of the form I +N1N2 with N1, N2 symmetricnegative semide�nite. Q.E.D.Corollary 2 Consider an SDP whose solution satis�es the strict comple-mentarity and primal and dual nondegeneracy conditions, and also condition(55). Suppose that the Q method uses � = 0 and � = � = = 1. Then, ifthe method is started with �; !, y and Q initialized su�ciently close to theirvalues at the solution, the iterates converge Q-quadratically to the solution.The proof of Corollary 2 is more technical than that of Corollary 1, andis omitted. It is necessary to establish that quadratic convergence is notimpeded by either (a) the use of the Cayley transform to approximate thematrix exponential or (b) the dependence of the de�nition of GQ on Q.As with the other methods, we see how to e�ciently implement the Qmethod by performing block Gauss elimination directly on JQ, without par-titioning the blocks. The �rst step yields24���1 0 LT0 D�1E RTL R 0 3524��~s�y 35 = 24diag(H � ���1)o�diag(H)v 35 ;where eS = kvec ~s is the symmetric matrix de�ned byeSij = (�i � �j)Sij:

Semide�nite Programming 22One more step of block elimination then gives the Schur complementMQ = [L R ]���1 00 �DE�1 � � LTRT � : (56)As in LP, the center factor of the Schur complement is diagonal, with entries�i!i ; 1 � i � n and �i � �j!j � !i ; 1 � i < j � n:Of course, the L and R blocks are not independent of the iteration count, asthey are in LP.The Q method does not require computing eigenvalues. The variablesQ, � and ! are all updated using rational operations. This is in contrastwith the XZ+ZX method which requires the computation of eigenvalues intwo places: the formation of the Schur complement matrix M (to solve theLyapunov equations) and in the steplength computation (to �nd the step tothe boundary). Finally, note that the Schur complement matrix is symmetricfor the Q method, but not for the XZ+ZX method.When evaluated on the central path, the Schur complement matrix M�Qfor the Q method is equal to the Schur complement matrix M� for the XZand XZ+ZX methods, assuming that (55) holds. To see this, let L�, R�, D�,E� denote the matrices L, R, D, E evaluated on the central path. We have��(�)�1 = Diag(��i =!�i ) = 1�Diag((��i )2) and�D�(E�)�1 = Diag(��i � ��j!�j � !�i ) = 1�Diag(��i ��j )Thus, MQ = 1�B�D�(B�)T =Musing (39) and (40).Although the Q method has some attractive features, it is, at present, nota practical alternative to the other algorithms. When initialized far from thesolution, convergence is generally not obtained. However, the quadratic localconvergence established here is observed in practice.7 Computational ResultsIn this section we report on the results of some extensive numerical experi-ments. We start by discussing some important implementation details.

Semide�nite Programming 23Mehrotra's predictor-corrector (PC) rule is a well known technique in LP[Wri97]. It can easily be extended to the XZ and XZ+ZXmethods, as follows.XZ and XZ+ZX Methods with Mehrotra Predictor-Corrector Rule1. Determine �X , �y, �Z from (20), using � = 0 in (18), and symmetrize�X in the case of the XZ method.2. Choose steplengths �; � using (28){(29), and de�ne� = � (X + ��X) � (Z + ��Z)X � Z �3 (57)� = �X � Zn :3. Redetermine �X , �y, �Z from (20), usingRc = � �I � (XZ +�X�Z) XZ method�I � 12(XZ + ZX +�X�Z +�Z�X) XZ + ZX method � ;symmetrize �X in the case of the XZ method, and update the iteratesby X X + � �Xy y + � �yZ Z + � �Z;with �, � given by (28){(29).See [TTT97] for a de�nition of the PC version of the Nesterov-Toddmethod. (Our experiments use (57) in the implementation of all the methods,although [TTT97] use the exponent 2 instead of 3 in (57).)Computational results are presented in Tables 1 through 4. Tables 1, 2and 3 report results for randomly generated problems, with m = n. Thematrices Ak; k = 1; : : : ; m were symmetric with entries uniformly distributedin the interval [�1; 1]. The vector b and the matrix C were chosen to ensurethat Assumption 1 was satis�ed. More precisely, random positive de�nitesymmetric matrices ~X and ~Z and a random vector ~y were generated, and bwas de�ned by bk = Ak � ~X, k = 1; : : : ; m, while C was set to ~Z+Pmk=1 ~ykAk.All methods were initialized with the infeasible starting point (X0; y0; Z0) =(I; 0; I). Table 1 shows results for the XZ+ZX, XZ and NT Basic Iteration,using � = 0:25 in (27), with various choices for the steplength parameter �in (28), (29). We also implemented the X�1 method but found it requiredmany more iterations than the others with the same parameter choices. Table

Semide�nite Programming 242 shows results for the PC variants. Part (a) of both tables shows the numberof iterations required to reduce the quantityX�Z by a factor of 1012, averagedover 100 problems. Part (b) shows the �nal value oflog10 (jjrpjj+ jjRdjj) ;averaged over the same data. A run was terminated reporting success whenX � Z was reduced by the desired factor of 1012, and reporting failure if (i)the primal or dual steplength (� or �) dropped below 10�4 (indicated by thenotation S in part (a) of the table), or (ii) the number of iterations exceededthe maximum value 50 (indicated by E in the table) or (iii) a Cholesky factor-ization failed (caused by rounding errors, impossible in exact arithmetic, andindicated by R in the table). Failures are not included in the average statis-tics. All experiments were conducted in Matlab, using IEEE double precisionarithmetic.Let us �rst consider the results shown in Table 1, for the Basic Iterationwithout the PC rule. For � = 0:9, all three methods show essentially the samenumber of iterations. The XZ+ZX method achieves the highest accuracy (interms of feasibility). More aggressive choices of the steplength parameterhave little e�ect on the XZ+ZX method but cause di�culties for the XZ andNT methods. Choosing � = 0:999 causes the XZ and NT methods to failin many cases. In the case of the XZ method, this was usually because theprimal or dual steplength dropped below 10�4, but for the NT method, failuregenerally occurred because the the desired reduction in the duality gap wasnot achieved in 50 iterations.Table 2 shows the same experiment using the PC rule. With � = 0:9, thePC rule greatly reduces the number of iterations, though with some loss offeasibility for the XZ and NT methods. More aggressive choices of � gave asigni�cantly reduced number of iterations (without loss of feasibility) for theXZ+ZX method, but led to many failures for the XZ and NT algorithms.In Table 3, we show results for the XZ+ZX method when the problem sizen is varied, using the PC rule and two choices of � . We see an iteration countwhich is essentially constant as n increases, with occasional failures (withsteps too short) for � = 0:999. In these cases, we found that success couldgenerally be achieved by restarting with X0 and Z0 set to a larger multiple ofthe identity (alternatively, reducing �). Note some loss of feasibility (due torounding errors) for larger n. Primal feasibility can be regained by projectingonto the set fx : Ax = bg, but this generally fails to give a more accuratesolution, as the duality gap usually increases.For some classes of problems, the XZ and NTmethods can be implementedvery e�ciently. This is the case, for example, for SDP's with only diagonal

Semide�nite Programming 25constraints on X (equivalently, o�-diagonal entries in Z �xed). For such anSDP, we have m = n and Ak = ekeTk , k = 1; :::; m, where ek is the kthcolumn of the identity matrix. Consequently, for the XZ method we haveMij = eTi XejeTj Z�1ei, i.e. M is the Hadamard product of X and Z�1[HRVW96], reducing the cost of forming M from O(n4) (the general casewhen m = n) to O(n3). It is not known how to implement the XZ+ZXmethod e�ciently in this case. A similar observation applies to the SDP thatcomputes the Lov�asz � function for a graph [GLS88], as long as the numberof edges is not too large. In this case n is the number of vertices in the graphand m � 1 is the number of edges, with b = e1, �C the matrix of all ones,A1 = I , and, for k = 2; : : : ; m, Ak = eieTj + ejeTi , where the (k � 1)th edgeof the graph is from vertex i to vertex j.Table 4 shows results comparing the XZ+ZX, XZ and NT methods on the� function for randomly generated graphs, with edge density 50%, using thegeneral-purpose implementations. We set n = 20, so the expected value ofm is 14n(n � 1) + 1 = 96. For these runs, we used the initial feasible point(X0; y0; Z0) = ((1=n)I;�2ne1; 2nI + C). Using an infeasible initial pointdid not signi�cantly change the results. The XZ and NT methods often haddi�culty reducing the duality gap by the desired factor, even with � = 0:9,because rounding errors caused a Cholesky factorization to fail. This wasusually the Cholesky factorization ofM = eAeAT (see (25) and (44)), which ispositive de�nite in exact arithmetic but may be numerically inde�nite. Sincethe Schur complement for the XZ+ZX method is nonsymmetric, it is fac-tored using an LU factorization, which fails only if the matrix is numericallysingular, i.e. the factorization generates a zero pivot.We also implemented the Q method and observed that it has essentiallythe same rapid local convergence and high accuracy properties as the XZ+ZXmethod, although when initialized far from the solution, it generally fails toconverge.We conclude that the XZ+ZX PC method is the most e�cient in terms ofnumber of iterations, most accurate in terms of feasibility, and most robustwith respect to its ability to step close to the boundary.Acknowledgments. The third author is grateful to Richard Tapia, BobVanderbei and Steve Wright for introducing him to primal-dual interior-pointmethods for LP. The authors also thank Mike Todd for many helpful com-ments, especially a suggestion concerning the Q method that makes the de-velopment given here substantially simpler than the one given in [AHO94a].

Semide�nite Programming 26Method � = 0:9 � = 0:99 � = 0:999XZ + ZX 21:6 21:2 21:2XZ 21:8 22:1 23:7 (S:11%,E:2%)NT 21:6 22:0 29:8 (E:18%)Number of iterations to reduce gap by 1012Averaged over 100 randomly generated problemsBasic iteration with � = 0:25Starting infeasible, n = 20, m = 20S: Short step failure (not included in average)E: Exceeded limit failure (not included in average)Table 1aMethod � = 0:9 � = 0:99 � = 0:999XZ + ZX �12:6 �12:6 �12:6XZ �11:0 �10:9 �10:9NT �10:8 �10:7 �10:5Log norm infeasibilityAveraged over same dataTable 1bMethod � = 0:9 � = 0:99 � = 0:999XZ + ZX 14:0 9:4 8:5XZ 15:3 14:2 15:7 (S:63%,E:5%)NT 14:5 22:8 (E:100%)Number of iterations to reduce gap by 1012Averaged over 100 randomly generated problemsMehrotra predictor-corrector ruleStarting infeasible, n = 20, m = 20S: Short step failure (not included in average)E: Exceeded limit failure (not included in average)Table 2aMethod � = 0:9 � = 0:99 � = 0:999XZ + ZX �10:7 �12:0 �12:2XZ �8:7 �8:8 �9:4NT �9:1 �8:3Log norm infeasibilityAveraged over same dataTable 2b

Semide�nite Programming 27Method n = m = 20 n =m = 40 n = m = 80XZ + ZX , � = 0:99 9:4 9:9 10:0XZ + ZX , � = 0:999 8:6 9:2 (S:3%) 9:5 (S:6%)Number of iterations to reduce gap by 1012Averaged over 100 randomly generated problemsMehrotra predictor-corrector ruleStarting infeasibleS: Short step failure (not included in average)Table 3aMethod n = m = 20 n =m = 40 n = m = 80XZ + ZX , � = 0:99 �12:1 �11:2 �10:4XZ + ZX , � = 0:999 �12:3 �11:4 �10:5Log norm infeasibilityAveraged over same dataTable 3bMethod � = 0:9 � = 0:99 � = 0:999XZ + ZX 15:2 11:0 (S:1%) 10:4 (S:1%)XZ 15:5 (R:22%) 17:0 (E:15%,R:22%) 15:0 (S:68%,E:28%,R:3%)NT 15:6 (R:25%) 21:3 (E:3%,R:24%) (S:3%,E:97%)Lovasz � FunctionNumber of iterations to reduce gap by 1012Averaged over 100 randomly generated problemsMehrotra predictor-corrector ruleStarting feasible, n = 20, edge density 0.5 (m � 96)S: Short step failure (not included in average)E: Exceeded limit failure (not included in average)R: Rounding failure (not included in average)Table 4aMethod � = 0:9 � = 0:99 � = 0:999XZ + ZX �13:8 �13:7 �13:6XZ �12:6 �11:4 �12:4NT �12:5 �10:9Log norm infeasibilityAveraged over same dataTable 4b

Semide�nite Programming 28Appendix. Symmetric Kronecker ProductsConsider the linear operator on Rn�n de�ned by the mapK 7! NKMT (58)where M;N 2 Rn�n. It is standard to represent this linear operator by theKronecker product M N = 264 M11N � � � M1nN... ...Mn1N � � � MnnN 375where nvec maps Rn�n to Rn2, stacking the columns of a matrix in a vector,since then (M N)nvec(K) = nvec (NKMT): (59)Other Kronecker product identities include(M N)�1 =M�1 N�1 and (M N)(K L) =MK NL: (60)Now consider the linear operator on Sn de�ned by the mapK 7! 12(NKMT +MKNT ) (61)where M;N 2 Rn�n. To represent this map as a matrix, de�ne M ~ N bythe identity (M ~N) svec (K) = svec (12(NKMT +MKNT )) (62)where svec maps Sn to Rn2 bysvec (K) = hK11;p2K12; : : : ;p2K1n; K22; : : : ;p2K2n; : : : : : : ; KnniT :(63)Note that K � L = svec (K)T svec (L):Of course, the ordering used in (63) is arbitrary: the important point is thateach element of svec (K) is associated with an index pair (i; j), with i � j.The ordering chosen for svec dictates a corresponding ordering for ~.We call the matrix M ~ N a symmetric Kronecker product. Note theidentity M ~N = N ~M: (64)

Semide�nite Programming 29Furthermore (M ~M)�1 =M�1 ~M�1, but (M ~N)�1 6=M�1 ~N�1, ingeneral.We need the following lemmas whose proof are straightforward.Lemma 1 Let V 2 Rn�n and let vi, 1 � i � n, denote the columns of V .The (i; j) column of V ~ V , 1 � i � j � n, is the vector( svec (vivTi ) if i = j1p2 svec �vivTj + vjvTi � if i < jLemma 2 Let M , N be commuting symmetric matrices, and let �1; : : : ; �n,�1; : : : ; �n denote their eigenvalues with v1; : : : ; vn a common basis of or-thonormal eigenvectors. The n(n + 1)=2 eigenvalues of M ~ N are givenby 12(�i�j + �i�j); 1 � i � j � n;with the corresponding set of orthonormal eigenvectors( svec (vivTi ) if i = j1p2 svec (vivTj + vjvTi ) if i < jIn other words, if V = [v1 � � �vn], then V ~ V is an orthogonal matrix ofsize n2 � n2 which diagonalizes M ~N . The standard algorithm for solvingthe Lyapunov equation MXNT +NXMT = B (when M and N commute)immediately follows: the solution is V CV T , where C is found by computingV TBV and dividing its entries by the quantities (�i�j+�i�j) componentwise.

Semide�nite Programming 30References[AHO94a] F. Alizadeh, J.-P. A. Haeberly, and M. L. Overton. A new primal-dualinterior-point method for semide�nite programming. In J.G. Lewis,editor, Proceedings of the Fifth SIAM Conference on Applied LinearAlgebra, pages 113{117, Philadelphia, 1994. SIAM.[AHO94b] F. Alizadeh, J.-P. A. Haeberly, and M. L. Overton. Primal-dual interior-point methods for semide�nite programming. Manuscript. Presented atthe Math Programming Symposium, Ann Arbor, 1994.[AHO97] F. Alizadeh, J.-P. A. Haeberly, and M. L. Overton. Complementarityand nondegeneracy in semide�nite programming. Mathematical Pro-gramming, 77:111{128, 1997.[Ali91] F. Alizadeh. Combinatorial Optimization with Interior Point Methodsand Semide�nite Matrices. PhD thesis, University of Minnesota, 1991.[GLS88] M. Gr�otschel, L. Lov�asz, and A. Schrijver. Geometric Algorithms andCombinatorial Optimization. Springer-Verlag, New York, 1988.[HRVW96] C. Helmberg, F. Rendl, R. Vanderbei, and H. Wolkowicz. An interior-point method for semide�nite programming. SIAM Journal on Opti-mization, 6:342{361, 1996.[KSH97] M. Kojima, S. Shindoh, and S. Hara. Interior-point methods forthe monotone linear complementarity problem in symmetric matrices.SIAM Journal on Optimization, 7:86{125, 1997.[KSS96] M. Kojima, M. Shida, and S. Shindoh. A predictor-corrector interior-point algorithm for the semide�nite linear complementarity problem us-ing the Alizadeh-Haeberly-Overton search direction. Technical ReportB-311, Department of Information Sciences, Tokyo Inst. of Technology,1996.[Mon96] R. Monteiro. Polynomial convergence of primal-dual algorithms forsemide�nite programming based on Monteiro and Zhang family of di-rections. Technical report, School of Industrial and Systems Engineer-ing, Georgia Institute of Technology, July 1996.[MZ96] R. Monteiro and P. Zanjacomo. A note on the existence of the Alizadeh-Haeberly-Overton direction for semide�nite programming. Technicalreport, School of Industrial and Systems Engineering, Georgia Instituteof Technology, June 1996.[NN94] Y. Nesterov and A. Nemirovskii. Interior Point Polynomial Algorithmsin Convex Programming. SIAM, Philadelphia, 1994.[NT97a] Y. Nesterov and M. Todd. Primal-dual interior-point methods for self-scaled cones. SIAM Journal on Optimization, 1997. To appear.

Semide�nite Programming 31[NT97b] Y. Nesterov and M. Todd. Self-scaled barriers and interior-point meth-ods for convex programming. Mathematics of Operations Research,1997. To appear.[OW95] M.L. Overton and R.S. Womersley. Second derivatives for optimizingeigenvalues of symmetric matrices. SIAM Journal on Matrix Analysisand Applications, 16:697{718, 1995.[SSK96] M. Shida, S. Shindoh, and M. Kojima. Existence of search directionsin interior-point algorithms for the SDP and the monotone SDLCP.Technical Report B-310, Department of Information Sciences, TokyoInst. of Technology, 1996.[TTT97] M.J. Todd, K.C. Toh, and R.H. T�ut�unc�u. On the Nesterov-Todd di-rection in semide�nite programming. SIAM Journal on Optimization,1997. To appear.[VB95] L. Vandenberghe and S. Boyd. Primal-dual potential reduction methodfor problems involving matrix inequalities. Mathematical Programming(Series B), 69:205{236, 1995.[Wri97] S.J. Wright. Primal-Dual Interior-Point Methods. SIAM, Philadelphia,1997.[Zha97] Y. Zhang. On extending primal-dual interior-point algorithms fromlinear programming to semide�nite programming. SIAM Journal onOptimization, 1997. To appear.[ZTD92] Y. Zhang, R.A. Tapia, and J.E. Dennis. On the superlinear andquadratic convergence of primal-dual interior point linear programmingalgorithms. SIAM Journal on Optimization, 2:304{324, 1992.