[1] P. Arbel´ aez, M. Maire, C. Fowlkes, and J. Ma- lik. Contour Detection and Hierarchical Image Segmentation. PAMI, 2011. [2]M. Maire, S. X. Yu, and P. Perona. Object De- tection and Segmentation from Joint Embedding of Parts and Pixels. ICCV, 2011. [3] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. ICCV, 2001. [4] M. Spain and P. Perona. Measuring and Predict- ing Object Importance. IJCV, 2010. ONR MURI N00014-10-1-0933 and ARO/JPL-NASA Stennis NAS7.03001 supported this work. Part of Stella Yu’s work was supported by NSF CAREER IIS-1257700. Benchmarks Image Groundtruth UCM gPb-UCM [1] Residual • Construct groundtruth Ultrametric Contour Map (UCM): – Weight each boundary by level at which it appears in object-part hierarchy – Determine boundary visibility by region tree traversal • Does machine hierarchical segmentation [1] respect object-part containment? – Residuals (above) show differences by: * Type: false positive / false negative / incorrect level * Severity: color intensity reflects magnitude of error – Plots (below) measure recovery order of groundtruth boundaries: * Ideally recover top-level objects, then parts, then subparts (levels 1, 2, 3) * gPb-UCM only recovers groundtruth hierarchy on simpler scenes (portraits) All Scenes 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Overall Boundary Recall Level Recovery Fraction Boundary Recovery Order by Hierarchy Level Level 1 Level 2 Level 3 Portrait Scenes 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Overall Boundary Recall Level Recovery Fraction Boundary Recovery Order by Hierarchy Level Level 1 Level 2 Level 3 Dataset Parts Objects Image • Complete labeling of 97 complex scenes photographed by artist Stephen Shore [4] • Compared to Berkeley Segmentation Dataset [3]: more objects, greater scale range Interactive Labeling A B D C • Browse object-part hierarchy by expanding/collapsing visible subtrees (A) • Drag and drop to rearrange region hierarchy (B) • Fade occlusion layers by depth to visualize figure/ground ordering (C) • Interactively edit regions (D) – Editor enforces parent-child region containment invariant – Superpixel selection brush speeds region definition: Image Superpixels Click Drag Release Touch-up Scene Model Object-Part ⇒ ⇐ Occlusion Ordering man head shirt arm watch hand arm envelope stamp label glove Objects/Parts Figure/Ground Gnd Fig glove envelope stamp label man head V shirt arm V watch hand arm glove • Image ⇒ regions {R 1 ,R 2 , ..., R n } (R i , R j possibly overlapping) • Map each region R i to a node in doubly-ordered tree: – Parent-child node relationships correspond to object-part containment – Relative ordering of sibling nodes resolves occlusion ambiguities • Tree traversal: – Recovers object-part hierarchy – Converts local occlusion relationships into global figure/ground ordering • Virtual nodes (dotted ovals): parts without visible boundaries • Virtual links (dotted arrows): remap occlusion ordering (handle self-occlusion) Overview • Annotate multiple modalities: – Objects, parts, subparts – Object-part containment – Segmentation – Occlusion (figure/ground) • Unifying abstraction: region trees • Web-based annotation tool: – Computer-assisted segmentation • Object segmentation dataset & benchmark • Motivation: joint detection & segmentation [2] apple ❈ ❈ ❈ ❈ ❈ ❈ ❈ ❈ ❲ glasses P P P P P P P P q hat ✛ face ✛ torso ✟ ✟ ✟ ✟ ✟ ✟ ✙ eyes ❍ ❍ ❍ ❍ ❍ ❥ P P P P P P P P P q beard P P P P P P q finger ❄ shirt ✠ tie P P P P P P ✐ jacket ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✎ Fig Gnd Image Objects/Parts Subparts Figure/Ground Ultrametric Contour Map Michael Maire 1 , Stella X. Yu 2 , and Pietro Perona 1 1 California Institute of Technology - Pasadena, CA 91125 2 University of California at Berkeley / ICSI - Berkeley, CA 94704 Hierarchical Scene Annotation