Dynamic boundary surface detection

CO~IPUTI.:It UltAI'IHCS ,~.ND IM;k(ll," PbtOCIi:,'SSIN(I 7, 130-138 (1978)

NOTE

Dynamic Boundary Surface Detection

G. T. tIEIc~AN .~ND H. :[(. LIv

]Mophgsical Sciences Unit, Department of Physiology and Biophysics, llfayo Clinic, Rochester, Minnesota 55901, and

.Department of Computer Science, State University of New York at Buffalo, Amherst, New York 1~226

Received July 15, 1.9761 accepted October 2, 1976

This paper presents an algorithm h)r detecting the dynamically changing SUl'[aee o[ an organ such as the breathing hmg ()r the beating heart from a time sequence of three- dimensional density distributions, making essenl:ial use ~ff the tempor'fl information. Results produced by the algorithm as applied to rec~mstructicm of real organs produced by computerized X-ray tc)mography are included.

1. INTII ODUCTION

Medic~I researchers have long sought a way of examining interior structures of the body, both healthy and abnormal. Recently developed mathematical procedures, of [,on referred to by the collective name "reconstruction from pro- jections," m,~ke the internal structures of the body accessible without surgical intervention [1]. These are techniques of mathematically combining X-ray images made from numerous angles into three-dimensional images of the organs within the body. The result of the reconstruction process is stored in computer memory in the form of ~ three-dimensional matrix of densities, d~.~. ~, where for any fixed k, di,j'.k represents the density at the point (i, j) in the/cth of a sequence of pnra.llel cross sections.

In order to improve the efficacy of this useful information in diagnosis, three- dimensional display of the three-dimensional reconstruction of organs is desired. We have developed a computer method [3] which first detects the boundary surface of the organ of interest and then displays this surface on a screen.

Inform~tion about the dynamic changes in shape and dimensions of intact working organs (such as the heart and lungs) is of extreme interest in some areas of medical research and clinical medicine [5-7-]. This kind of information provides greatly improved techniques for the investigation of the relationship of the dynamic three-dimensional anatomic structure to ~he physical and biochemical functions of moving organ systems, and ultimately for providing powerful new

130

014fi-664X/78/0071-0] 30$02.00/0 Copyr igh t ~ I978 by Academic Press, Inc. All r ights of reproduct ion in any form reserved.

1)YNAMIC Bi)UN1)AI~Y SUI{FACE DETECTION 131

clinical diagnostic methods for the elucidation of abnormalities of the hearg and hmgs in man. Recent improvements of both data acquisitions and computer reconstruction algorithms resulted in computerized methods for obtaining images of the dynamically changing spatial distribution of X-ray attenuation coefficients inside the living thorax [2-]. In this paper, we discuss the generalization of our boundary detection algorithm [3-] to four-dimensional space. This generalization is appropriate for detecting the changing surface of an organ from a sequence of three-dimensional reconstructions making essential use of the temporal continuity always concomitant with organ motion.

To explain the essential nature of our four-dfmensional boun&~ry detection algorithm we look at an alternative interpretation of the three-dimensional version of our algorithm. Suppose we have a sequence of two-dimensional re- constructions .showing the same cross section of a thorax at successive points of time. If we want to detect the moving boundary of the left lung, we can do this using our algorithm in one of two ways. We can treat each reconstruction sepa- rately and apply a two-dimensional edge detection algorithm, or we can consider time as a third dimension, stack the two-dimensional reconstructions on top of each other, and detect the resulting surface by our three-dimensional algorithm. We prefer the latter way, since it makes essential use of the continuity of the change in the boundary over time. The time sequence of boundaries is then ob- tained by intersecting the surface resulting from the three-dimensional boundary detection by planes corresponding to the time instants at which reeonstruetlons were taken.

2. TIIE ALGORITH~I

A four-dimensional discrete image consists of a set of equal, nonoverlapping "hypercubes" referred to as "hypervoxels." The gray level in a hypervoxel is uniform. We denote the gray level associated with the hypervoxel II{i, j, k, t) by f(i, j, I% t). The physical meaning of this is that the gray level of the voxel (i, j, k) at time t is I(i, j, Ic, t).

Our algorithm detects the boundary by trying to achieve the following aims :

a. high contrast across the boundary, b. similarity of contrast across neighboring boundary elements, c. connectivity of boundary in both space and time, d. agreement with a priori knowledge.

Our algorithm is a sequential search guided by these aims. It forms the boundary by c:onnecting boundary elements which are square surfaces in four- dimensional space. The boundary itself is a three-dimensional volume (made up from cubes connecting boundary elements).

High contrast can be defined by using a gradient operator. Our gradient operator in four-dimensional space associates g. positive real number with each potential boundary element. The six square surface elements associated with hypervoxel H (i, j, k, t) are denoted by S f J (i, j, k, O, S I K (i, j,/% t), S I T ( i , j, lc, O, SJK( i , j, lc, t), ,SJT(i , j, k, t), and SKT( i , j, k, t). The gradients .mross these

132 HERMAN AND LIU

elements are defined by:

[Gradls~J(~,j,~.t) = HI(i, j , k, t) - I ( i , j, l~ + 1, t + 1)11 + il~(,i, j , ~ + ~, t) - f ( i , j , k, t + ~)11,

IGr~Ldlzz~c. j ,~. , ) = III(i, j , ~, t) - I ( i , i + 1, 7c, t + 1)l[ + IlI(i, j + 1, k, t) -- I ( i , j, 7% t + 1)[I ,

1Ch'ad[sn,(~,.;,~.,o = IlI(i, j , 1% t) - I ( i , j + 1, lc + 1, t)[l + [lI(i, j + 1, t , t) - z ( i , i, t + :~, t)l[,

IG~.adlsJK<~.~,~,,) = U ( i , J, it, t) - I ( i + 1, j, lc, t -b 1)l[ + 11/(i + 1, j, lr t) - I ( i , j, 1r t + 1)1],

IGradlss~(;.j,~.t) = IlI(i, j , k, t) - I ( i + l , j ,l~ + l, t)ll + [[ I (~/+ 1, j, 7~, t) - I ( i , y, 7~ + 1, t)[I,

IGradl s~,(;.~.~.,) = [lI(i, j , Ic, t) - l ( i + 1, j + 1, 1% t)[] + IIZ(i + x, j, ~, t) - I ( i , y + ~, z~, t)lI.

The square surfaces with high gradient values are the likely boundary elements. I t seems evident tha t tile boundary is, to a certain extent, connective in nature

[-4J. T h a t is, the location of a boundary element should be "eonneet;ed" to the location of the nearest boundary element. We define the sets Nzrj(~.y,1~.t), NSrK(Gj,k, t) , J~ZsIr(i,j,k,t), Ns,lK(i,],k,tb JVSJT(i,j,k,t), and Nsrc'ru.i.k,t), the sets of connected neighbors of the square surfaces S I J ( i , j , lfl t), S I K ( i , j,}c, t), S I T ( i , j , 1% t), S J K ( i , j, lc, t), S J T ( i , j , }, t), and S K T ( i , j, k, t) as:

N ~ . . , s , ~ , , ) = { Z Z J ( l , .m,, n, p ) l l = i, .~ = j a n d II]~ - - ~[t + lit - - 2~[[ = 1} ,

Nsrzc(~,i ,k,o = { S I K ( I , m, n, p)II = i, n = ]r and nJ -- mt[ + lit -- ~)11 = 1},

N,w,,.u,j .k. ,) = { S I T ( I , re, n, p) ll = i, p = t and I]J - ml[ + llZc - nil = 1},

NszKc~.j,~,t) = { S J K ( I , m, n, p) l m = j, n = lc and 11i -Zl l + lit - pll = 1},

N.~,~.~..,~,~,~ = {SJT(Z , .~, ,~, P) I'.~ = J, p = t a n d II~ - ill + 111~ - nil = 1},

N,SICT(I.i.A,,t) = {S fCT( l , m, n, p) ln = 1% p = t and ] l i - ll[ ~- ] [ j - - rail = 1}.

A square surface can be "connected" to its connected neighbors by a cube and a set of cubes forms ~he boundary in four-dimensional space.

In our approach we insist tha t at least one and at mosb two connected neighbors of every boundary element are also on the boundary. This means tha t in the process of looking for the next boundary element we need consider only the four horizontal and vertieN neighbors.

Usually, we have some knowledge about the boundary we are seeking. For example, in two dimensions we may know tha t tile boundary is a closed curve, or t ha t it is a curve s ta r t ing at the top row and ending at the bot tom row of the image. This kind of information is helpful for correctly locating the boundary. We utilize such a priori know/edge to determine a backtracking criterion and a terminat ion criterion. There is one type of a priori knowledge of the boundary which is true for a large class of images : Tha t the aven~ge gradient value changes

DYNAMIC BOUNDARY SURFACE DETECTION 133

slowly along the boundary. Based on the average gradient value, a dynamic threshold is calculated and used to determine the backtracking criterion.

The algorithm consists of the following four components.

A. Pick, an Initial Boundary Element

For a sequentiaI boundary detection method, picking a good initial boundary element is very important . I t is necessary to begin the search for a boundary element, at a location where the likelihood of making an error is the smallest. B y using interactive computer graphics, we display the image on the screen and use the light-pen to give the ,algorithm a location which appears to be near the boundary. The algorithm then finds the initial boundary element based on maximum gradient criterion in the neighborhood of the indicated location. In case there are several objects in the image, this will also tell the algorithm which is ~he object whose boundary we are seeking.

B. Look-Ahead

This routine finds the next boundary element by choosing one of the connected components of the current boundary element. The one chosen has the maximum gradient wdue among those not already on the boundary or flagged during the backtracking process which will be discussed later. Also, this maximum gradient value must be greater than a threshold TR. (If the maximum gradient v~lue is less than TR, backtracking will commence.) We define T s = ~ * A V , where A V is the average gradient value along the boundary found so far, and a >__ 0 is a tolerance f~etor. (Note tha t a = 0 means no backtracking.) Figure 1 indicates tha t the value of ~ should be no greater than 0.5. In feeL, ~ = 0.4 was found to be appropriate for a number of different applications; it has been exc]usively used in all the demonstrat ions in Section 3. If the algorithm is successful in find- ing a new next boundary element, the new one is put in a pool f rom which the look-ahead routine will take the new current boundary element next time around. If the algorithm fails to find a next boundary element (this can happen because all eligible elements have gradient less than TR), the backtracking routine to be discussed later is called. In this way, we use a global threshold to guide locat search and to detect error.

Since a boundary element is "connected" to the next boundary element b y a cube and there are six surfaces on a cube, up to five new boundury elements are

A B C Fro. 1. tf all slmded pixeIs hr~ve the same gray levels and ~ll unshaded pixels h~ve the same

gr~y level, then the gr~dien~ value ~t the center of A is htd[ o[ the gradient w~iue at the center of either B or C.

134 H E R M A N AND LIU

put into the pool after each successful look-ahead step. In order to avoid generating a thin twisting surface, we would like the boundary surface to be growing at about; the same speed along all sides. We use a "breadth-first" mecha- nism: we take the boundary elements from the pool in the order in which they are put into it.

C. Bac]r

Whenever this routine is called, the current boundary element is a false bound- ary element, because there is no acceptable adjacent boundary element. We flag the path from the previous boundary element to this boundary element (to ensure that the same mistake does not happen again), open the path(s) from current boundary element, calculate the value of A V one step before, take the previous boundary element as the current boundary element, and recall the look-~head routine. If tmother boundary element is found, then the algorithm continues following that new path, otherwise the backtracking routine is called agaim

D. Termination

This procedure is used to determine that the algorithm has found the bourtdary. As described below, a priori knowledge is nsed in this routine which may have to be ~djusted in different application areas. One way to implement this routine is using a logical function, whose value is .TRUE. when the current boundary

i

Picking Initial Eou~idary Element

i. Receive inform~tioll from light pen.

2. Find initial boundary element. 3. Put ~nitial boundary element

into the pool. 4. Set AV to the gradieilt vnlue

of initial boundary element

I ooka .............................. | ]. Take top of ehe pool as the |

>1 2. pic~r~een~ }boouln~ar~ ~ ele,lm~ntt" [

L f ......... t boundary eleme~

Back-track in 9" ] ~ ' ~

I. Back AV ~ ~ Is _ 2. Close s path ~< ........ __~ next boundary 3. Fiild next boundary ~ No ~ elemen~ /

element from the ~ ~ found.~ / previous boundary~ ~ / element ~ /~ ~ A Y~S

i-~ Fat next i~--o~nda~ / Is . element(s) he the I /termination bottom of the pool~ No --~ criterion />

.~ . . . . Y e s

]?'m. 2. Fh)w-(~h~u'l~ des(,.ription of t~he Mgorithm.

])YNAMIC BOUNDARY SURFACE I)ETECTION 135

point satisfies the terminat ion criterion, and .FALSE. otherwise. The algorithm te rmina tes whenever the logical function yields the value . T R U E . .

Figure 2 gives a flow-chart description of the algorithm. The result of the four- dimensional boundary detection is a three-dimensional volume. The intersection of this volume with hyperplanes a t I = i, 2, 3, 4, . . . gives the sequence of the surf~ees we are seeldng.

3. EXPEI~IMENTAL [~ESULTS

Tile algorithm has been implemented on the CDC 3500 with a R A M T E K Graphic Display Sys tem at the Biomedical Sciences Computer Facility ag the Mayo CIinie. All experiments reported on in this paper have been carried out on tha t computer .

Tile use of t ime as tlm third dimension is demonstrated in Fig. 4. X - I b w project ion data of a simple cross section of the intact lungs and heart was obtained a t 12 t ime points i~ the p u l m o n a r y cycle. Cross sections were recmlstrueted f rom 35 views in 5.4 ~ increments over 183.6 ~ Figure 3 shows the 12 reconstructed cross sections at t = 1, 2, . . . , 12 (from left to right, top to bottom). Each cross section is a 64 X 64 image. We stacked this t ime sequence of cross sections into a 64 X 6<t • 12 array, the third dimension indicating time. The three-dimensional bounda ry detection a lgor i thm was applied to the volume corresponding go the left lung. Figure 4 shows the intersection of the array (including boundary

Fro. 3. Twelve reconstructed cross sections of the intact lungs and heart at I2 time p,ints ([rom left to right and from top to bottom) ia tile puhnomtry cycle.

13(5 HIEtlMAN AND LIU

Ftc~. 4. Boundaries of Fig, 3 deCeci,ed by usc of {,ira dymmli(~ Imul~dary del, eel,icm ,'tl)prmt(th.

FIO. 5. Boundaries of Fig, 3 detected by ~pplying l,wo-dimensiom~ l~mmd~ry de,(.e,e,{,io~ mc~hod I~o ~,'~,oh ree(.msl,rue(,ion.

DYNAMIC' B(31TNT1]ARY SIr[II?ACI~'~ DETECTION 137

Fro. G. Reconst,ruetim~s a{; six stooges (from left ~o right ~md from top to bottom) in LLa cardiac cycle of an isoh~l;ed bettting left venl;rMe. Time sequerme o[ surfaces was det:ected by the four- dimensional versiolt of ottr algoribhm from ~ (~4 X 64 X 28 X 6 array.

surface) wi6h the planes corresponding to the 12 points in the pulmonary cycle. As pointed out before, we can t rea t each reconstruction separately and apply a two-dimensional version of our boundary detection algorithm [3] to each one. Figure 5 shows the result of this ~pproach. In two cross sections, this approach erroneously included the right hmg as part of the bouridar[cs. By making essential use of the temporal continuity, the three-dimensional approach detected the boundaries correctly. I t took 12.138 seconds to generate Fig..4 and 12.524 seconds to generate Fig. 5.

Figure 6 shows a t ime sequence (left to right and top to bot tom) of the surface of an isolated working left ventricle, obtained b y our four-dimensional boundary detect ion algorithm. The input to the algorithm was a 64 X 6~ X 28 X 6 array of gray level.

4. DISCUSSION

We have presented a goal-directed boundary deteetioit method which can detect boundaries in four-dimensional space. The advantages of the proposed bounda r y detection techniques are :

138 HI,;RMAN ANI) L1U

a. They are efficient. Their search space is small as compared to a dynamic programming method or a multistage technique. They use only small amounts of CPU time and core memory. This is part icularly important when we consider the size of images in four-dimensional space.

b. They are flexible. They perform well on a var ie ty of images. e. They are reliable. Cont rary to most sequential boundary detection

techniques, they can detect an error and lead the algorithm back to the correct boundary.

d. They m~ke essential use of the continui ty of changes in the boundary over t ime and are therefore especially useful in detecting dynamical ly changing boundaries.

ACKNOWLEDGMENTS

The attthors wish to thank Dr. James F. Greenleaf for valuable discussions and Drs. Erik L. P~itman ~md Riehard A. Robb for providing test datt~. The authors are indebted to Ms. Jean Frank and her co-workers for typing and graphic assistance. The research for this p,'~per w,'~s supported by NSF Grant MCS 75- 22347, N I H Grants H L 04664, It-00007, H L 18968, 'rod N C[ Coiltraet; N01- CB-53860. The p~per was written while G. T. Herm~m was a recipient of a Visiting Scientist award of the American Hear t Association in t,he laboratory of Dr. Earl H. Wood of the Mayo Clinic.

ItEFER],3NCES

1. IL Gordon, G. T. Herm,'m~ trod S. A. Johnson, Ima.ge l'econstructicm h'cmt prejecL~ions, Sci. A,mer. 233, ]975, 56-68,

2. G. T. Herman, A. V. Lakshminarayanan, A. Naparstek, E. L. l~itman, ]t. A. ]:lobb, m~d E. H. Wood, Rapid ccmlputerized tomography, in Medical Data Processing Symposium (Tmthmse, France), Tayhn' & Francis, London, 1976.

3. H. K. Liu, Two- and thl'ee-dimensitm'~l bomu]ary detection, Compuler Graphh:s linage l~roee.~s- 1'rig 6, 1077~ 123-I34.

4. A. Rosenfeld, Connectivity in digital pictures, J. Assoc. for Comput. Much. 17, 1970, 146-1(}(}. 5. R. E. Sturm, E. L. I litmau, and E. H. Wood, Quantitative thl'ee-dimensionM dynamic imaging

of structure and hmetion of the cardiopulmonary and circulatory systems in all regions of the body, Cardiovascular Imaging Image Processing, 72, 1975, 103-122.

6. E. H. Wood, New horizons for the study of the eardiopuhnonary and circulatory systems, Chest 69, I976, 394-408.

7. E. H. Wood, Cardiovascular and pulmonary dynamics by quantitative imaging, Circulation Res. 38, 1976, 131-139.