PROGRESSIVE TONE MAPPING OF BRAIN IMAGES AT SINGLE …faculty.sist.shanghaitech.edu.cn/faculty/luoxl/class/2017Fall_EE251/... · This paper presents a novel tone mapping method for

PROGRESSIVE TONE MAPPING OF BRAIN IMAGES AT SINGLE-NEURON RESOLUTION

Puning Zhao, Zhiwei Xiong, Dong Liu, Hao Wang, Chaoyu Yang, Lufeng Ding,Weiping Ding, Zheng-Jun Zha, Guoqiang Bi, Feng Wu

University of Science and Technology of China

ABSTRACT

This paper presents a novel tone mapping method for the display ofhigh dynamic range (HDR) 3D brain images at single-neuron reso-lution, where the fluorescent labeled neurons exhibit extremely highlocal contrast. Existing tone mapping methods fit for natural imagescannot well preserve the details of neurons in the 3D brain image,and require huge memory when the image resolution is ultra high.Therefore, we propose a block-wise, progressive tone mapping strat-egy for efficient and high quality display of such image, which fur-ther offers two different modes for optimum global or local visualexperience. Experimental results demonstrate that the proposed ap-proach significantly improves the visualization quality of high con-trast neurons in HDR brain images compared with existing methods.

Index Terms— High dynamic range, tone mapping, bilateralfiltering, brain image, neuron

1. INTRODUCTION

The human brain contains about 86 billion neurons [1], yet it remainsa mystery to human how these neurons are connected to each otherand which problems will occur if they are incorrectly connected. Re-cently, whole-brain imaging at single-neuron resolution has seen abreakthrough relying on several representative techniques such as se-lective plane illumination microscopy [2], serial two-photon tomog-raphy [3], and micro-optical sectioning tomography [4]. These tech-niques significantly facilitate the establishment of brain-wide neuralnetworks at a mesoscopic scale, which will be of great role in pro-moting human healthcare and understanding the working principlesof human intelligence.

A neuron generally has a cell body and a long thin axon with adiameter of 1µm, and the end of the axon, named the synapse, hasa diameter of only 300nm [5]. During the brain imaging process,the fluorescent labeled neurons exhibit extremely high local contrast.Specifically, the light intensity emitted by the cell body of the neuronmay be several orders of magnitude higher than that of the axonand the synapse. Therefore, high dynamic range (HDR) imaging isneeded for reliably capturing the unique structure of neurons in thebrain. Meanwhile, to achieve single-neuron resolution, whole-brainimaging is performed by scanning the sample in tens of thousandsof snapshots and the final 3D image is reconstructed by aligning andstitching these snapshots. Not surprisingly, the resultant 3D brainimage has an ultra high resolution.

For displaying HDR images on common screens that only sup-port 8-bit dynamic range, tone mapping is an indispensable opera-tion. Tone mapping is the process of scaling the intensity values in an

P. Zhao is with School of Gifted Young. Z. Xiong (corresponding author:[email protected]), D. Liu, W. Ding, Z. Zha, and F. Wu are with Schoolof Information Science and Technology. H. Wang, C. Yang, L. Ding, and G.Bi are with School of Life Sciences.

HDR image to a displayable dynamic range, which has been exten-sively studied in the natural image scenario. Existing tone mappingmethods can be divided into two categories: global ones [6, 7, 8]and local ones [9, 10, 11]. Global tone mapping applies the samemapping function to each pixel, among which the simplest one is thewell-known gamma correction that uses a logarithmic mapping func-tion. Generally speaking, global methods are simple and fast, butthey may cause the loss of local contrast and thus conceal the imagedetails. In contrast, local tone mapping adopts a mapping strategythat varies spatially. The intensity of a pixel after tone mappping isalso determined by its neighboring pixels, which better preserves theimage details. However, local methods are either susceptible to thehalo effect or of high computational complexity.

In 2002, Durand and Dorsey proposed a fast tone mappingmethod based on bilateral filtering [12], which decomposes an HDRimage into a base layer and a detail layer. The contrast of the baselayer is compressed by a global operator, and the result is then re-combined with the uncompressed detail layer. Comparing with theinput image, the global contrast of the output image is reduced whilethe local contrast is preserved. Due to the edge-preserving propertyof the bilateral filter [13], this method can effectively prevent thehalo effect with relatively low computational complexity. For mostnatural scenes, the above method works pretty well and has sincebeen widely used. However, when applied to ultra high resolution3D brain images, this method encounters difficulties.

The main difference between natural images and brain images isthat the latter exhibit extremely high local contrast due to the uniquestructure of neurons. If we only compress the global contrast of thebase layer, the large local contrast will introduce artificially underex-posed or overexposed pixels that conceal the details of neurons. Toget rid of this effect, we propose a progressive tone mapping strategythat compresses the dynamic range step-by-step, which guaranteesthat the detailed structure of neurons are best revealed.

Another challenge of the brain image discussed here is its ultrahigh resolution. The memory requirement for processing the whole3D image exceeds the capacity of ordinary computers. For example,the brain of a mouse measures only 1 cubic centimeter, yet the whole16-bit 3D image occupies 2TB memory at single-neuron resolution(i.e., 1µm) in each dimension. It is thus impossible to perform tonemapping on the whole brain image. We propose a block-wise tonemapping strategy so the tone mapping operation can be practicallyperformed on the ultra high resolution brain image. Moreover, twodifferent modes for optimum global or local visual experience aresupported, according to the requirements of different applications.

To the best of our knowledge, this is the first work to investigatethe tone mapping operation for brain images at single-neuron reso-lution. We conduct experiments on real mouse brain data which areacquired using the state-of-the-art whole-brain imaging technique.Experimental results demonstrate that the proposed approach signif-icantly improves the visualization quality of high contrast neurons.

958978-1-5090-5990-4/17/$31.00 ©2017 IEEE GlobalSIP 2017

Fig. 1. Framework of our proposed tone mapping method.

2. THE PROPOSED METHOD

2.1. Framework

The framework of our proposed tone mapping method is shown inFig. 1. As mentioned above, the raw data of whole-brain imaginggenerally exist in the form of tens of thousands of snapshots. It thusrequires huge memory to process the reconstructed 3D image at onetime. Therefore, we propose to conduct the tone mapping operationin a block-wise manner. Suppose the 3D brain image hasW×H×Lpixels, it is divided into blocks with a size of w × h× l pixels and acertain overlap between neighboring blocks in each dimension. Onlythe raw data corresponding to the current block to be processed areread into the memory. For each block, the dynamic range is com-pressed step-by-step. When an optimum global visual experience ispreferred, the number of steps N is uniformly set for all blocks; oth-erwise when an optimum local visual experience is preferred, N isdetermined on a block basis. After tone mapping, the final 3D imageis reconstructed by setting pixel values in the overlapped regions theaverage of all candidates.

2.2. Progressive Tone Mapping

The proposed tone mapping method is based on [12], where an inputHDR image I is first decomposed into a base layer ub and a detaillayer ud through bilateral filtering g in the log-domain (suppose u =log(I + 1))

ub = g(u), ud = u− ub (1)

Note ub contains only positive values but ud contains both positiveand negative values. The base layer is compressed and then recom-bined with the uncompressed detail layer, producing an output image

uo = γub + ud (2)

where γ is the compression factor (0 < γ < 1) determined by thedynamic range of the input and output images.

In HDR natural images, the local contrast is usually muchsmaller than the global contrast. In other words, even the base layeris compressed to a large extent, the output image always has positivevalues. However, this will not hold true for HDR brain images whichexhibit extremely high local contrast due to the unique structure ofneurons. Specifically, the detail layer may contain large negativevalues that cannot be compensated by the compressed base layer,introducing artificially underexposed pixels in the output image.Adjusting the image brightness could get rid of underexposed pix-els, but may introduce overexposure at the same time. We give an

20 40 60 80 100

pixel

0

0.5

1

log

2 (

I)

(b)

20 40 60 80 100

pixel

0

0.5

1

log

2 (

I)

(c)

20 40 60 80 100

pixel

0

1

2

3

4

log

2 (

I)

(a)

Fig. 2. An intuitive example of tone mapping in an image with highlocal contrast. (a) Original image. (b) Result by directly using themethod in [12]. (c) Result by using two-step progressive tone map-ping. The red curve shows the central intensity in the bright bandwhile the blue curve shows the intensity along the dark line.

intuitive example here to better explain this dilemma. (Note that theexample is a 2D case, yet we process 3D data in practice.)

As shown in Fig. 2(a), a bright band in the image representsthe highest intensity part of the fluorescent labeled neurons, whilethere is a dark line above the bright band with fluctuating intensity,representing some lower intensity details of the neurons. The inten-sity ratio of the bright band and the dark line is around 24, makingthe dark line nearly invisible when displaying without tone mapping.Fig. 2(b) shows the tone mapping result by directly using the methodin [12]. The base layer is compressed by a factor of 0.125. Corre-spondingly, the global contrast is greatly reduced and the dark lineis now visible. However, the local contrast along the dark line in-troduces negative pixels that appears as underexposed breakpoints.Increasing the image brightness could eliminate these underexposedpixels, but the intensity of the bright band will also increase and leadto overexposure.

To solve this problem, we propose a progressive tone mappingstrategy for the HDR brain images with high local contrast. The ba-sic idea is to compress the dynamic range step-by-step, so the com-pression factor in each step can be smaller and underexposure canbe avoided. Fig. 2(c) shows the two-step tone mapping result. Ascan be seen, there will no longer be negative pixels (i.e., underex-posed breakpoints) along the dark line. Generally, suppose the stepof progressive tone mapping is N , then we have

u(n)o = γ

1N u

(n−1)b + u

(n−1)d , n = 1, 2, . . . , N (3)

where u(n−1)b and u(n−1)

d are calculated from the image after the (n-1)-th tone mapping following Eq. (1), and u(n)

o is the output imageafter the n-th tone mapping. Less underexposed pixels will occur ifwe use a larger N , but the computational complexity also increasesalong with N . To find a good balance, we set a step N to guaranteethat a percentage of λ of pixels will not be underexposed after tonemapping (e.g., λ = 99.5%). Specifically, we define a 3D matrix Mwhose elements are calculated as

Mijk = −u(0)d,ijk/u

(0)b,ijk (4)

where i, j, k index the three dimensions of the brain image, respec-tively. Suppose m∗ is larger than a percentage of λ of elements inM, then N is calculated as

N = d log γ

logm∗ e (5)

959

2.3. Nonlinear Base Layer Compression

The dynamic range compression of the base layer is usually con-ducted through a linear function in the log-domain, i.e., f(ub) =γub. However, the average intensity of the brain image is very low,since the fluorescent labeled neurons only occupy a small portion ofthe whole brain. Increasing the average intensity helps improve thevisual experience in dark regions, but at the risk of increasing over-exposure. To overcome this problem, we use a nonlinear functionproposed in [14] to compress the base layer as

f(ub, γ) =ubmax(ub)

( 1γ− 1)ub +max(ub)

(6)

It can be easily deduced that

f(ub, γ) > γub, f(max(ub)) = γmax(ub) (7)

Compared with using a linear function, the average intensity of thebase layer will increase while the maximum intensity will remain thesame. Therefore, this nonlinear function improves the visual experi-ence in dark regions without increasing overexposure.

2.4. Optimum Global or Local Visualization

Our progressive tone mapping method offers two different modesfor optimum global or local visualization. In the first mode, a unifiedstepN are used for all the blocks to produce a globally consistent vi-sual experience, which is preferred when the observer would like topreview the whole brain image and quickly switch between differentparts of the brain. In the second mode, local adaptive steps are de-termined on a block basis. In the regions containing extremely highlocal contrast, a larger N is adopted, which further reduces the pos-sibility of underexposure. This option is preferred when the observeris interested in analyzing the detailed structure of neurons in specificlocal regions. However, since the steps are locally optimized, it mayintroduce visual inconsistency among different blocks, although theoverlap between neighboring blocks alleviates the inconsistency to acertain extent. The observer can select the best visualization optionaccording to the requirements of different applications.

3. EXPERIMENTAL RESULTS

We conduct experiments on a 16-bit mouse brain slice, the resolu-tion of which is 6432(X)×4654(Y )×234(Z) in three dimensions.Actually this is only a downsampled (by a factor of 4 in each dimen-sion) slice of the whole 3D brain image at single-neuron resolution,yet loading such an image requires 14GB memory. Directly pro-cessing this ultra high resolution image is impossible on ordinarycomputers. Therefore, we divide the image into blocks with a sizeof 707 × 511 × 234 and 10% overlapped pixels in the first two di-mensions. The tone mapping steps in the global mode are set asN = 2, and a maximum of N = 3 is supported in the local mode.The compression factor is set as γ = 0.125.

The results of different tone mapping methods are shown in Fig.3. Note that we project the 3D brain slice onto the X − Y plane foran easy visualization. The same observations apply to the 3D image.In the original image, the ratio between the average intensity and themaximum intensity is 1/185. Fig. 3(a) shows the tone mapping re-sult by the simple gamma correction, in which the local contrast isreduced together with the global contrast. As can be seen, althoughthe global structure is now visible, the local details of neurons arenot clearly revealed since the local contrast is lost. Fig. 3(b) shows

Table 1. Percentage of overexposed and underexposed pixelsMethod βo(%) βu(%) βc1(%) βc2(%)

Durand’s [12] 0.51 1.12 2.30 1.49Ours (linear) 0.32 0.40 1.47 0.87Ours (global) 0.23 0 1.02 0.68Ours (local) 0.20 0 0.82 0.29

the tone mapping result by directly using Durand’s method in [12],from which we can see that the local contrast is preserved. However,in case of extremely high local contrast, this method will introduceartificially underexposed pixels. On the other hand, since a linearfunction is used to compress the base layer, the average intensity ofthe image is still quite low after tone mapping. Both issues requireto increase the image brightness for a better visualization. However,overexposure is introduced at the same time and conceals the detailsof neurons. Fig. 3(c) shows the tone mapping result by the proposedmethod in the global mode. Thanks to the step-by-step compressionof the dynamic range and the nonlinear function used for tone map-ping, this result not only preserves the local contrast, but also avoidsoverexposure to a large extent. Note that the average intensity inthese three results are nearly the same for a fair comparison, whichis about 1/8 of the maximum intensity.

Two sets of close-up views are displayed in Figs 3. (d)-(g) and(h)-(k), where four different tone mapping methods are compared.The proposed method in the local mode are also included, besidesthe three methods mentioned above. As can be seen, gamma correc-tion cannot reveal the local details of neurons clearly, and Durand’smethod introduces overexposure that conceals the detailed structureof neurons. The proposed method in the local mode best preservesthe local contrast, by further suppressing the overexposure comparedwith the global mode, as the tone mapping steps are adaptively de-termined according to the local contrast.

The percentage of overexposed and underexposed pixels of dif-ferent tone mapping methods is shown in Table 1. βo and βu de-note the overexposure and underexposure rates of the whole 3D im-age, and βc1 and βc2 denote the overexposure rates in the two setsof close-up views. The quantitative results demonstrate that, com-pared with Durand’s method, our proposed progressive tone map-ping method effectively eliminates the underexposed pixels and sup-presses the overexposed pixels. Specifically, using the nonlinearfunction instead of linear ones for compressing the base layer andadopting local adaptive steps further improve the result. Note that,since the fluorescent labeled neurons only occupy a small portionof the whole brain, the decrease of overexposure and underexposurerates in the table actually corresponds to a considerable amount ofpixels representing neurons.

4. CONCLUSION

In this paper, we presented a progressive tone mapping method forefficient and high quality display of HDR 3D brain images at single-neuron resolution. Experiments on real mouse brain data validatesthe superior performance of the proposed method. As the whole-brain imaging techniques get matured, immense amounts of data willbe generated and visualization of these data is usually the first stepfor further processing and analysis. In this sense, we believe theproposed method can be of great help for the brain study, whichwill eventually promote human healthcare and help understand theworking principles of human intelligence.

960

(a) (b) (c)

(d) (e) (f) (g)

(h) (i) (j) (k)

Fig. 3. Tone mapping results of a mouse brain slice. (a) Gamma correction. (b) Durand’s method. (c) Our method in the global mode. (d)-(k)Two sets of close-up views of four different methods: (d) and (h) by Gamma correction, (e) and (i) by Durand’s method, (f) and (j) by ourmethod in the global mode, and (g) and (k) by our method in the local mode.

5. REFERENCES

[1] Christof Koch and R Clay Reid, “Neuroscience: Observatoriesof the mind,” Nature, vol. 483, no. 7390, pp. 397–398, 2012.

[2] Hans-Ulrich Dodt, Ulrich Leischner, and et al., “Ultrami-croscopy: three-dimensional visualization of neuronal net-works in the whole mouse brain,” Nature Methods, vol. 4, no.4, pp. 331–336, 2007.

[3] Timothy Ragan, Lolahon R Kadiri, and et al., “Serial two-photon tomography for automated ex vivo mouse brain imag-ing,” Nature Methods, vol. 9, no. 3, pp. 255–258, 2012.

[4] Anan Li, Hui Gong, and et al., “Micro-optical sectioning to-mography to obtain a high-resolution atlas of the mouse brain,”Science, vol. 330, no. 6009, pp. 1404–1408, 2010.

[5] Claire Ribrault, Ken Sekimoto, and Antoine Triller, “Fromthe stochasticity of molecular processes to the variability ofsynaptic transmission,” Nature Reviews Neuroscience, vol. 12,no. 7, pp. 375–387, 2011.

[6] Jack Tumblin and Holly E Rushmeier, “Tone reproduction forrealistic images,” IEEE Computer Graphics & Applications,vol. 13, no. 6, pp. 42–48, 1993.

[7] Gregory Ward Larson, Holly Rushmeier, and Christine Piatko,“A visibility matching tone reproduction operator for high dy-namic range scenes,” IEEE Transactions on Visualization andComputer Graphics, vol. 3, no. 4, pp. 291–306, 1997.

[8] Frederic Drago, Karol Myszkowski, and et al., “Adaptive log-arithmic mapping for displaying high contrast scenes,” Com-puter Graphics Forum, vol. 22, no. 3, pp. 419–426, 2003.

[9] Sumanta N Pattanaik, James A Ferwerda, and et al., “A multi-scale model of adaptation and spatial vision for realistic imagedisplay,” in SIGGRAPH, 1998.

[10] Jack Tumblin and Greg Turk, “Lcis: A boundary hierarchy fordetail-preserving contrast reduction,” in SIGGRAPH, 1999.

[11] Michael Ashikhmin, “A tone mapping algorithm for high con-trast images,” in Eurographics Workshops on Rendering, 2002.

[12] Fredo Durand and Julie Dorsey, “Fast bilateral filtering for thedisplay of high dynamic range images,” in SIGGRAPH, 2002.

[13] Carlo Tomasi and Roberto Manduchi, “Bilateral filtering forgray and color images,” in ICCV, 1998.

[14] Christophe Schlick, “Quantization techniques for visualizationof high dynamic range pictures,” in Photorealistic RenderingTechniques, pp. 7–20. Springer, 1995.

961