1. style and fashion
2. clothing

Co-Segmentation
Presented By : Murad Tukan
Introduction
• Today there is a massive attempt to exclude the same object
from different images.
• Such problem is not an easy task as it seems , furthermore the
algorithm which is presented today is not 100% accurate even
though it is efficient.
What Are We Going To Learn ?
• An Efficient Algorithm for Co-segmentation .
• Clothing Co-segmentation for Recognizing People .
• iCoseg: Interactive Co-segmentation on your iOS device (in short).
• Implementation of Co-segmentation in MATLAB + C++ Code.
An Efficient Algorithm for Co-segmentation
Motivation
• The Goal of the Algorithm:
 Given a set of images , we need to detect the same foreground in the set
of images despite the differences in the background.
• Approach :
 We will reward consistency in the two foreground histograms.
Images
1
2
Histogram (Intensity)
Definitions
• Co-segmentation settings.
• Similarity weight.
• Foreground and background pixels.
• “MRF penalties”.
Co-segmentation settings
•
Co-segmentation settings (Cont.)
•
Similarity Weight
•
Example
“image 1”:
B1 :
“image 2”:
B2 :
Foreground and background pixels
• The segmentation of each image will partition the set of pixel
into foreground versus background pixels.
• Our goal is to ensure that the foreground in the two images are
similar.
Foreground and background pixels(cont.)
•
Foreground and background pixels(cont.)
•
“MRF penalties”
•
“MRF penalties” (cont.)
• The MRF formulation of one image is then :
Deviation penalty
min
𝑑𝑗 ∗ 𝑥𝑗 +
Seperation penalty
𝑖~𝑗
𝑤𝑖𝑗 ∗ 𝑦𝑖𝑗
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑥𝑖 − 𝑥𝑗 ≤ 𝑦𝑖𝑗
𝑥𝑗 − 𝑥𝑖 ≤ 𝑦𝑗𝑖
𝑥𝑖 , 𝑦𝑖𝑗 𝑏𝑖𝑛𝑎𝑟𝑦 𝑓𝑜𝑟 𝑖, 𝑗 = 1,2, … , 𝑛
𝑤ℎ𝑒𝑟𝑒 𝑑𝑗 = 𝑔𝑗 − 𝐼𝑗
Problem Statement
•
Problem Statement (cont.)
•
Problem Statement (cont.)
•
Problem Statement (cont.)
• Therefore the formulation of co-segmentation problem is then a
linear combination of MRF minimization and similarity
maximization:
𝐾
min
𝑑𝑗 1 ∗ 𝑥𝑗 1 +
𝑤𝑖𝑗 ∗ 𝑦𝑖𝑗1 +
𝑑𝑗 2 ∗ 𝑥𝑗 2 +
𝑤𝑖𝑗 ∗ 𝑦𝑖𝑗2 − 𝜆 ∗
𝑧𝑖𝑗
𝑘=1 𝑖∈𝐼 1 ∩𝐻𝑘 , 𝑗 ∈𝐼 2 ∩𝐻𝑘
𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜
𝑧𝑖𝑗 ≤ 𝑥𝑖 1 𝑓𝑜𝑟 𝑖 ∈ 𝐼
1
𝑧𝑖𝑗 ≤ 𝑥𝑗 2 𝑓𝑜𝑟 𝑗 ∈ 𝐼
2
𝑥𝑖
𝑞
− 𝑥𝑗
𝑞
≤ 𝑦𝑖𝑗 𝑓𝑜𝑟 𝑞 = 1,2
𝑥𝑗
𝑞
− 𝑥𝑖
𝑞
≤ 𝑦𝑗𝑖 𝑓𝑜𝑟 𝑞 = 1,2
𝑥𝑖
𝑞
𝑞
, 𝑦𝑖,𝑗 , 𝑧𝑖𝑗 𝑏𝑖𝑛𝑎𝑟𝑦 𝑓𝑜𝑟 𝑞 = 1,2
𝑖, 𝑗 = 1, … , 𝑛
The graph construction
•
The graph construction (cont.)
•
The graph construction (cont.)
•
The Algorithm (Co-Seg)
•
•
Histogram Buckets
•
Summarizing
• We have seen an efficient algorithm, running in polynomial time
(the running time of a max-flow algorithm).
Advantages through Comparison
Clothing Co-segmentation for recognizing people
Motivation
• The goal:

Given a four consumer image collection , we need to recognize individuals
among these image collection.
• Approach:

Using the clothing data.
How many different faces do you recognize?
Reminders
• Graph Cut.
• Normalized Cut.
• Texture feature vector.
Graph Cut
•
What are Graph-Cuts ?


•
Set of edges whose removal makes a graph disconnected.
It’s the partition of V into A,B such thatA  B   and
A graph G  (V , E ) can be partitioned into two disjoint sets A, B  V,we define the partition
cost as:
Cut ( A, B ) 
 w(u, v).
u A, vB
•
The bipartition of the graph which minimizes the cut value is called the Min-Cut .
Normalized cut (Ncut)
cut ( A, B)
cut ( A, B)
Ncut ( A, B) 

assoc ( A,V ) assoc ( B,V )
• Cut(A,B) is sum of weights with one end in A and one end in B
,we want to minimize the cut cost.
Cut ( A, B ) 
 w(u, v).
u A,vB
• Assoc(A,V) is sum of all edges with one end in A , we want to
maximize the sum of all weights for every A,B element in the
partition
assoc ( A,V )  uA,tV w(u, t ).
texture feature vector
•
Feature vector based on texture segmentation:
• We can use spatial filter for where the DOOG filters at various
scales and orientations .
F (i)  [ I * f1 ,, I * f n ]
Luminance-chrominance space (LCC)
• The color encoding system used for analog television worldwide (NTSC, PAL
and SECAM). The YUV color space (color model) differs from RGB, which is
what the camera captures and what humans view.
• The Y in YUV stands for "luma," which is brightness.
• U and V provide color information and are "color difference" signals of blue
minus luma (B-Y) and red minus luma (R-Y)
Luminance-chrominance space (LCC) (cont.)
• RGB to LCC / YUV :
Images and Features for Clothing Analysis
• Features are extracted from the faces and clothing of people:
• Face features:
• Are extracted using an algorithm of detecting face which also can estimate the
position of the eyes.
• Each face is normalized in scale (49*61 pixels) and projected onto a set of
Fisherfaces , representing each face as a 37-dimentional vector.
Fisherfaces
• A N x N pixel image of a face, represented as a
vector occupies a single point in N2-dimensional
image space.
• Images of faces being similar in overall
configuration, will not be randomly
distributed in this huge image space.
• Therefore, they can be described by a low
dimensional subspace.
• Main idea of PCA for faces:
– To find vectors that best account for
variation of face images in entire image
space.
– These vectors are called eigen vectors.
– Construct a face space and project the
images into this face space (eigenfaces).
Images and Features for Clothing Analysis
(cont.)
•
90, 133, 176,0,0
Images and Features for Clothing Analysis
(cont.)
•
Images and Features for Clothing Analysis
(cont.)
 2:
o The 5-dimentional vector at each pixel is quantized to the index of
the closest visual word , so each pixel in the region is associated with
a word in the dictionary (or several words) then all the words of all
the pixels in the region are counted in a histogram.
o The Histogram per region is the "feature" of the region. The visual
word clothing features are represented as v.
Example of visual words:
- red
- dots
- green
…
red,dot
80
NUMBER OF PIXELS
70
60
50
40
30
20
10
0
red
blue
dots
VISUAL WORD
Finding the Global Clothing Mask
•
Finding the Global Clothing Mask (cont.)
•
Finding the Global Clothing Mask (cont.)
• The same day mutual information maps are reflected (symmetry is
assumed) , summed and thresholded (by a value constant across image
collection) to yield clothing masks that appear remarkably similar across
collections.
Graph Cuts for Clothing Segmentation
•
Graph Cuts for Clothing Segmentation( cont.)
•
Graph Cuts for Clothing Segmentation( cont.)
•
Recognizing people
•
Recognizing people (cont.)
•
Experiments
• Better Clothing Recognition Improves Recognition:
•
We use nine nearest neighbors and repeat random labeling procedure 50 times to the
average performance.
Summary
• The identity recognition and clothing segmentation problems are
inter-twined; a good solution for one aides in the solution for the
other.
• Multiple images of the same person improves clothing
segmentation.
• Person recognition improves with improvements to the clothing
segmentation.
iCoseg: Interactive Co-segmentation on your
iOS device
Interactive Co-segmentation
Embedded Devices
iCoseg for iPhone, iPod Touch and iPad
References
• Dorit S. Hochbaum, Vikas Singh. An efficient algorithm for cosegmentation. ICCV Oct 2009 , Kyoto, Japan, the 12th IEEE Conference on
Computer Vision.
• A. C. Gallagher and T. Chen. Clothing cosegmentation for recognizing
people. In Proc. of Conf. on Computer Vision and Pattern Recognition, 2008.
• Slides Credit: Jad Silbak
• http://en.wikipedia.org/wiki/YUV
• http://chenlab.ece.cornell.edu/people/jasonlew/Poster_JasonLew.pdf