IEEE 2015 Conference on Computer Vision and Pattern Recognition DASC: Dense Adaptive Self-Correlation for Multi-modal and Multi-spectral Correspondence Seungryong Kim, Dongbo Min, Bumsub Ham, Seungchul Ryu, Minh N. Do, and Kwanghoon Sohn Download the code at http://seungryong.github.io/DASC/ Introduction DASC Descriptor Experimental Results Conclusion Motivation • In multi-modal and multi-spectral image, conventional descriptors often fail to estimate correspondence Goal • To establish dense correspondence for those images NIR Visible No-flash Flash Low exp. High exp. Blur Sharp Randomized Receptive Field Pooling • Unlike center-biased max pooling in LSS descriptor, the DASC descriptor incorporates randomized receptive field pooling Sampling Pattern Learning • Exploit supports vector machine (SVM) with linear kernel • Features: • Energy function for SVM Small Support Window Similarity • Adaptive self-correlation measure Efficient Computation • Asymmetric weights • Reference-biased sampling pairs • Approximated adaptive self-correlation Middlebury Stereo Benchmark Multi-modal and Multi-spectral Image Pairs Background Img 1 Img 2 color, gradient, structural similarity RGB Image NIR Image Matching Cost A Matching Cost B Matching Cost C Challenge on Multi-modal Images • Nonlinear photometric deformation, e.g., gradient reverses and intensity order variations LSS descriptor DASC descriptor Middlebury multi-modal SINTEL 1 2 2 2 , , , exp ( ) /2 ml ml ml r r d d 2 () || || max(0,1 ( )) m m m v v y r • Intuitions Center-biased pooling sensitive to noise Randomness of BRIEF LSS , bin ( ) max (,) i il j l d ij LSS descriptor DASC descriptor , , , , , ( , ), , il il il il il i d s t s t , , , 2 2 , , ( )( ) (,) { ( )} { ( )} ss tt s s t t st ss s s tt t t s t f f st f f , , , 2 2 , , , , ( )( ) (,) ( ) ( ) ii i i j ij i j ii i i ii j ij i i j f f ij f f ,' ss , , ( ) (, - ) , il il t j i s i i • Robust estimation (,) max(exp( (1 ( , ))/ ), ) st st • The correlation is normalized with norm of all (,) st l Img1 Img2 RSNCC BRIEF DAISY LSS DASC RGB-NIR flash-no flash different exposure blur-sharp • Robust novel descriptor called the DASC for multi-modal correspondence • Leverages adaptive self-correlation and randomized receptive pooling • Efficient computation with fast edge-aware filters Limitation of Conventional Descriptors • Image gradient (SIFT) or intensity comparison (BRIEF) cannot capture coherent matching evidence Img1 Img2 ANCC BRIEF SIFT LSS DASC Ground Truth Illumination Exposure