Tensor-based High-order Semantic Relation Transfer for Semantic Scene Segmentation Heesoo Myeong and Kyoung Mu Lee Department of ECE, ASRI, Seoul National University, Seoul, Korea http://cv.snu.ac.kr INTRODUCTION Objective Function PROPOSED METHOD  We separately deal with the semantic relations transfer problem with respect to Y Overview Goal Results on Jain et al. Dataset sky  The quadratic objective function with respect to Y as  Semantic scene segmentation: identifying and segmenting all objects in a scene N sky N 1 2 2 F (X )  w ( x  x )   ( x  y ) ,   ijk ,lmn ijk lmn ijk ijk 2 i , j , k ,l , m , n i , j ,k person tree building tree tree road tree building tree building building car tree person road Test image sky tree tree building Semantic scene segmentation … Query image Key idea car road car road road Integrated high-order relationSemantic segmentation sky sky building sky building car person Annotations of retrieved images Predicted top scored high-order relation road sidewalk Groundtruth 1. For a test image, retrieve similar training images using global features road car car High-order relations in the training dataset Previous works & Limitations  Conventional context models mainly focus on learning pairwise relationships between objects  Pairwise relations are not enough to represent high-level contextual knowledge within images building building 2. Apply semantic relation transfer algorithm to transfer third-order semantic relation from each training image to the test image 3. Integrate the high-order score into MRF optimization framework and obtain semantic scene segmentation mountain 𝐸 𝐷 𝑐𝑖 + 𝒥 c = 𝑖 𝐸 𝑃 (𝑐𝑖 , 𝑐𝑗 ) + 𝑖,𝑗 [X ]ijk  xijk car 𝐸 𝐻 (𝑐𝑖 , 𝑐𝑗 , 𝑐𝑘 ) sidewalk EXPERIMENTS Quantitative Results on Standard Datasets  SIFT Flow dataset (Liu et al., CVPR09):  car car Pairwise semantic relation Our Contributions  To describe observed third-order semantic relation within the retrieved image, we define another 𝐾 3 number of semantic tensors Y  {Y111 , Y112 ,..., Y KKK }  yijk 1 if G ( si )  c , G ( s j )  c , G ( sk )  c , ( si , s j , sk )  S retrieved   , 0 otherwis e   The use of high-order semantic relations for semantic segmentation  where 𝐺(𝑠𝑖 ) denotes the ground truth class of region 𝑠𝑖  A novel tensor-based representation of high-order semantic relations Now, the semantic relation transfer problem is reformulated as the problem of 𝛼𝛽𝛾 estimating the magnitude of confidence scores 𝑥𝑖𝑗𝑘 for all superpixel triplets (𝑠𝑖 , 𝑠𝑗 , 𝑠𝑘 ) and for all object class triplets (𝑐𝛼 , 𝑐𝛽 , 𝑐𝛾 ) based on Y  A quadratic objective function for learning the semantic tensor and an efﬁcient approximate algorithm tree bison grass grass Ground truth Proposed sky sky car mountain sidewalk sky building sky building window Query mountain window door door Ground truth Proposed building window Query Ground truth Proposed Results on Polo Dataset  250 training images, 100 test images, 19 labels person person  2,488 training images, 200 test images, 33 labels horse grass  Polo dataset (Zhang et al., CVPR11):  80 training images, 237 test images, 6 labels person horse Table 1: Per-pixel classification rates and (average per-class rates) 𝛼𝛽𝛾  The variable 𝑥𝑖𝑗𝑘 indicates confidence score of how likely the region triplet (𝑠𝑖 , 𝑠𝑗 , 𝑠𝑘 ) would be labeled as (𝑐𝛼 , 𝑐𝛽 , 𝑐𝛾 ), respectively road sidewalk building building Problem Statement  Query Proposed sky building 𝑖,𝑗 𝐸 𝑃 (𝑐𝑖 , 𝑐𝑗 )  Jain et al. dataset (Jain et al., ECCV10): Among superpixels 𝑆 = (𝑁 is the number of total superpixels, 𝐾 is the number of object classes), third-order semantic relation is defined as a 𝐾 3 number of 𝑁 × 𝑁 × 𝑁 semantic tensors X  {X 111 , X 112 ,..., X KKK } Ground truth sky 𝐸 𝐻 (𝑐𝑖 , 𝑐𝑗 , 𝑐𝑘 ) , where represents data term, represents pairwise term, and is confidence score by the semantic relation transfer algorithm 𝐸 𝐷 (𝑐𝑖 ) road building car Results on LMO Dataset Semantic Relation Transfer Algorithm {𝑆 𝑡𝑒𝑠𝑡 , 𝑆 𝑟𝑒𝑡𝑟𝑖𝑒𝑣𝑒𝑑 } Query  Use fully connected third-order Markov random field (MRF) model: car sky road Inference building Retrieved images tree where 𝑤𝑖𝑗𝑘,𝑙𝑚𝑛 is the triplet-wise similarity between two region triplets (𝑠𝑖 , 𝑠𝑗 , 𝑠𝑘 ) and (𝑠𝑙 , 𝑠𝑚 , 𝑠𝑛 ) tree  Exploiting high-order(mostly third-order) semantic relation sky car bison sky sky tree building road car road sky sky building building car road tree sky grass Query Ground truth horse grass grass person grass person person horse horse grass horse grass Proposed horse horse grass Query Ground truth Proposed CONCLUSION  We have presented a novel approach to learn high-order semantic relations of regions in a nonparametric manner  We develop a novel semantic tensor representation of the high-order semantic relations  We cast the high-order semantic relation transfer problem as a quadratic objective function of semantic tensors and propose an efﬁcient approximate algorithm