International Journal of Computer and Advanced Engineering Research (IJCAER)
Volume 02– Issue 02, APRIL 2015
____________________________________________________________________________________________________________________
Mechanisms for Scalable Image Searching: A survey
Ansila Henderson
Kavitha K. V.
Assistant professor
M.Tech Student
SCTCE, Pappanamcode
Trivandrum, Kerala
[email protected]
Department of Computer Science & Engg.
SCTCE, Pappanamcode
Trivandrum, Kerala
[email protected]
Abstract—With the increase in number of images on the Internet,
there is a strong need to develop methods for efficient image
retrieval. Today, the images features are of high dimensions and
the image databases are huge. So the abilities of traditional
systems, methods, and computer programs to perform image
retrieval functions in an efficient, useful and timely manner
become challenged. The latest techniques use hashing methods to
embed high-dimensional image features into Hamming space.
The image search can be performed in real-time based on
Hamming distance of compact hash codes. The traditional
methods (e.g., Euclidean) offers continuous distances, but the
Hamming distances are discrete integer values. A large number
of images would share equal Hamming distances to a query. But
for an image search mechanism fine-grained ranking is very
important.
Keywords-inverted file; hashing; KD tree; indexing
I.
INTRODUCTION
Today, the heterogeneity and the size of digital image
collections grow exponentially. So there is a strong need to
develop methods for efficient image retrieval. The primary
goal of an image management system is to search images
efficiently and in a timely manner. Thus, it could compete with
the applications in the current era. The image searching should
be based on its visual contents. To gain more accurate results
with high retrieval performance, many researchers have
devised techniques based on different parameters.
An image can be the representation of a real object or a
scene. With the development of the internet, and the
availability of image capturing devices such as digital cameras,
huge amounts of images are created every day in different
areas. There are many different fields such as remote sensing,
fashion, crime prevention, medicine, architecture, etc that are
in need of an efficient image retrieval system. The image
similarity can be something very subjective and corresponding
to the world. A search for an image includes generating a set of
feature vectors similar to image characteristics and comparing
the set of features to the features indexed for multiple stored
images. The search result can be produced based on the
comparison of the set of indexed features.
In recent techniques, usually the images were represented
using the popular bag-of-visual-words (BoW) framework. In
this framework, local invariant image descriptors were
extracted and quantized based on a set of visual words. So the
methods used for document retrieval was adopted to the image
retrieval task. The BoW features can be embedded into
compact hash codes for efficient search. Hashing is more
suitable for image retrieval task than the tree-based indexing
structures such as kd-tree. It requires greatly reduced memory
and also works better for high-dimensional samples. Using the
hash codes, image similarity can be efficiently measured (using
logical XOR operations) in hamming space. It uses hamming
distance, an integer value obtained by counting the number of
bits at which the binary values of the images are different.
II.
OBJECTIVE OF SCALABLE IMAGE SEARCHING
As the number of images increases, it is difficult to find
visually similar content. Thorough search is infeasible for
large scale applications because it consumes more time. So an
efficient search mechanisms like indexing methods were
needed to provide efficient search time and retrieval accuracy.
The frequency of similar objects in the data space increases
with the increase in the number of images. These objects may
have similar semantics. So the inferences based on nearest
neighbors would be more reliable. The images were usually
described by sequences of descriptor vectors having more than
a thousand dimensions. The image similarity is examined by
nearest neighbor search.
III.
LITERATURE REVIEW
Today, the existing image features are of high dimensions
and the image databases are huge. So comparing the query
image with every database sample becomes computationally
prohibitive which makes the efficient search mechanism
critical. Recent image feature representations such as BoW
were similar to the bag-of-words representation of textual
documents. So the methods used for document retrieval were
adopted into the image retrieval task. The existing works on
efficient search mechanisms can be mainly classified into three
categories. They are
1. Inverted file
2. Tree-based indexing
3. Hashing
A. Inverted file
J. Zobel proposed the inverted file to retrieve similar
documents [1]. Inverted file is an index structure used for text
__________________________________________________________________________________________________________________
1
International Journal of Computer and Advanced Engineering Research (IJCAER)
Volume 02– Issue 02, APRIL 2015
____________________________________________________________________________________________________________________
query evaluation. The index structure maps terms to the
documents that contains them. The inverted file contains a
collection of lists for each term which stores the identifiers of
the documents containing that term. It has two major
components:
1. Search structure or vocabulary
2. Set of inverted lists
For each term the search structure stores a count and a
pointer. The count specifies the number of documents
containing the term and the pointer points to the start of the
corresponding inverted list. The inverted list stores the
identifiers of the document containing a particular term. It also
stores the count of repetition of the terms in a document. The
word positions within documents can also be recorded. These
components provide all the information needed for the query
evaluation.
In a complete text database system, there were several other
structures, including the documents themselves and a table that
maps ordinal document numbers to disk locations. The
difference is that each inverted list was stored contiguously.
These lists were composed of a sequence of blocks which were
linked or indexed in some way.
The similarity score for a document was calculated based
on how many times a particular term repeat in that document.
In a ranked query, a phrase was treated as an ordinary term, a
lexical entity that occurs in given documents with given
frequencies, and contributes to the similarity score for that
document when it does appear. Similarity can therefore be
computed in the usual way, but the inverted lists must be used
for the terms in the phrase to construct an inverted list for the
phrase itself, using a Boolean intersection algorithm.
The set of identified phrases were added to the vocabulary
and have their own inverted lists. Without any alteration to
query evaluation procedures the users could query them.
However, such indexing was potentially expensive. The
number of distinct two-word phrases grows more rapidly than
the number of distinct terms. There was no obvious mechanism
for accurately identifying which phrases might be used in
queries, and the number of candidate phrases was enormous.
Inverted index was initially proposed and is still very
popular for document retrieval. It was then introduced to the
field of image retrieval. In this structure, for a given query with
several words, a list of references to each image for each visual
word was created so that relevant images could be quickly
located. A key difference of document retrieval from visual
search was that the textual queries usually contain very few
words. But a single image may contain hundreds of visual
words, resulting in a large number of candidate images that
need further verification. This largely limits the application of
inverted files for large scale image search. While increasing
visual vocabulary size in BoW the number of candidates can be
reduced, but it will increase memory usage [2].
B. Tree-based indexing
C. Silpa-Anan and R. Hartley proposed Optimised KDtrees for fast image descriptor matching [3]. The tree-based
indexing dramatically improves the image retrieval quality.
The elements stored in the KD-trees were high-dimensional
vectors. At the root of the tree, the data was split into two
halves by a hyperplane orthogonal to a chosen dimension at a
threshold value. This split was usually made at the midpoint of
the dimension with the greatest variance in the data set. By
comparing the query vector with the splitting value, it would
easily determine to which half of the data the query vector
resides. Each of the two halves of the data was then recursively
split in the same way to generate a fully balanced binary tree.
At the bottom of the tree, each leaf node of the tree would be
similar to a single point in the data set. In some cases, the leaf
nodes may contain more than one point. The height of the tree
would be log2N where N is the number of points in the data set
[3].
For a given query vector, the tree needs log2N comparisons.
The data point related to the root node was the first candidate
for the nearest neighbor. Each node in the tree resembles a cell
as shown in the figure. If a query point lying anywhere in a
given leaf cell was searched, it would have lead to the same
leaf node.
The KD-trees were effective in low dimensions, but their
efficiency decreases for high dimensional data. In high
dimensional images, a large number of nodes would be there to
search. So a KD tree takes a lot of time to backtrack through
the tree to find the optimal solution. By limiting the amount of
backtracking, the certainty of finding the absolute minimum
was sacrificed and replaced with a probabilistic performance
[3]. The recent researches were aimed at increasing the
probability of success by keeping the backtracking within
reasonable limits.
C. Hashing Techniques
The binary hash codes were compactable to memory. The
search can be performed efficiently using the hash table lookup
or bitwise operations. Thus it satisfies the time and also the
memory requirements. Hashing methods used for image
retrieval can be divided into three main categories [12]:
unsupervised methods, supervised methods and semisupervised
methods.
1. Supervised methods:
Supervised hashing is based on the machine learning task
known as supervised learning. It generates functions from
labeled training data. The training data consist of a set of
training samples. Here, each sample is a pair consists of an
input object and a desired output value. The functions derived
from the supervised learning algorithm can be used for
mapping new samples. An optimal scenario would correctly
determine the class labels for unseen specimens. The learning
algorithms need to derive from the training data to unseen
situations in a legitimate way.
__________________________________________________________________________________________________________________
2
International Journal of Computer and Advanced Engineering Research (IJCAER)
Volume 02– Issue 02, APRIL 2015
____________________________________________________________________________________________________________________
2. Unsupervised methods:
Unsupervised hashing is based on the machine learning
task known as unsupervised learning. This method uses the
unlabeled data to generate binary codes for the given points. It
tries to find hidden structure from unlabeled data. There will
be no error signal to estimate a possible solution. Many
methods employed in unsupervised learning were based on
data mining methods. The methods used in unsupervised
learning include clustering such as k-means, mixture models
etc and blind signal separation using feature extraction
techniques for dimensionality reduction.
3. Semisupervised methods:
Semi-supervised learning is a class of supervised learning.
It also makes use of unlabeled data for training. Generally a
small amount of labeled data is used with a large amount of
unlabeled data. Semi-supervised learning falls between
unsupervised learning (without any labeled training data) and
supervised learning (with completely labeled training data).
Many machine learning researchers have found that a
considerable improvement in learning accuracy can be
achieved by using a small amount of labeled data with the
unlabeled data.
Locality Sensitive Hashing (LSH) is one of the most
popular unsupervised hashing methods in computer vision.
Another effective method called Spectral Hashing (SH) was
proposed recently by Weiss et al [4]. Since unsupervised
methods do not require any labeled data, their parameters were
easy to learn using a pre-specified distance metric. However,
in vision problems, sometimes similarity between data points
may not defined with a simple metric. Metric similarity of the
image descriptors may not preserve semantic similarity. The
pairs of images that contain ‘similar’ or ‘dissimilar’ images
need to be provided. From such pair wise labeled data,
hashing mechanism will automatically generate codes that
respect the semantic similarity.
The output of supervised methods may be meaningful to
human. But it will be difficult to label all the images in a huge
database system. Also not everything in the real world has a
distinctive meaning. The image similarity is more related to
high level image features than the low level features. So it is
better to use the semi-supervised learning for image retrieval
task.
Locality Sensitive Hashing (LSH)
P. Indyk and R. Motwani proposed the Locality Sensitive
Hashing [5]. The Locality Sensitive Hashing maps similar
samples to the same bucket with high probability. The
property of locality in the original space would be preserved in
the hamming space. More precisely, the hashing functions h
(.) from LSH family satisfy the following elegant locality
preserving property [5]:
Ph(x) = h(y) = sim(x,y)
where the similarity measure can be directly linked to the
distance function. A typical category of LSH functions
consists of random projections and thresholds as:
h(x) = sign(w > x+b)
(2.2)
where w is a random hyperplane and b is a random
intercept. Clearly, the random vector w is data independent,
which is usually constructed by sampling each component of
w randomly from a p-stable distribution. Although there exists
an asymptotic theoretical guarantee for random projection
based LSH, it is not very efficient in practice since it requires
multiple tables with long codes. Constructing a total of l K-bit
length hash tables H(x) = [h1(x), . . . ,hK(x)] provides the
following collision probability:
For a large scale application, the value of K should be
considerably large to reduce the size of each hash bucket.
However, a large value of K decreases the collision
probability between similar samples. In order to overcome this
drawback, multiple hash tables need to be constructed. This is
inefficient because it needs extra storage cost and larger query
time. Recently, Kulis and Grauman [6] extended LSH to work
in arbitrary kernel space, and Chum et al. [7] proposed minHashing to extend LSH for sets of features.
Spectral Hashing (SH)
Weiss et al. proposed a spectral hashing (SH) method that
hashes the input space based on data distribution [4]. In
spectral hashing bits were calculated by finding the threshold a
subset of eigenvectors of the Laplacian of the similarity graph.
The SH algorithm consists of three key steps [4]:
1. The extraction of maximum variance directions through
Principal Component Analysis (PCA) on the data.
2. The direction selection, which prefers to partition projected
dimensions with large range and small spatial frequency.
3. The partition of projected data by a sinusoidal function with
previously computed angular frequency.
SH was very effective in encoding large-scale, lowdimensional data since the important Principal Component
Analysis (PCA) directions were selected multiple times to
create binary bits. However, for high dimensional images
where many directions contain enough variance, each PCA
direction would take only once. This is because the top few
projections had similar range. Thus, a low spatial frequency
was preferred. In such cases, SH replicates a PCA projection
followed by a mean partition approximately. In SH, the
projection directions were data dependent but learned in an
unsupervised manner. Also, the assumptions of uniform data
distribution were usually not true for real-world data.
(2.1)
__________________________________________________________________________________________________________________
3
International Journal of Computer and Advanced Engineering Research (IJCAER)
Volume 02– Issue 02, APRIL 2015
____________________________________________________________________________________________________________________
Semantic Hashing
R. Salakhutdinov and G. Hinton proposed the semantic
hashing for the retrieval of similar documents [8]. Learning
with deep belief networks (DBN) proposed for dimensionality
reduction [9] was recently adopted for semantic hashing in
large-scale search applications. The deep belief networks
needs image labels during training phase to generate hash
codes. The DBN structure gradually reduces the number of
units in each layer. So the high-dimensional input of original
image features can be projected into a compact Hamming
space.
A general DBN is a directed acyclic graph in which each
node represents a stochastic variable. There were two main
steps for hash code generation using DBN. First, learning the
interactions between variables and second, inferring
observations from inputs. The learning of a DBN with
multiple layers needs to estimate millions of parameters. In
order to reduce the difficulty in learning a DBN, the DBN has
been structured based on the RBMs (Restricted Boltzmann
Machine) [9]. Each RBM consists of two layers containing
respectively output visible units and hidden units. The
multiple RBMs were stacked to form a deep belief net. The
network can be specifically designed to reduce the number of
units, and finally output the compact hash codes.
The training process of a DBN consists of two main stages:
unsupervised pre-training and supervised fine-tuning. The pretraining phase aims to place the network weights (and the
biases) to suitable neighborhoods in parameter space. After
achieving convergence of the parameters of one layer via
Contrastive Divergence, the outputs of this layer were fixed
and treated as inputs to drive the training of the next layer.
During the fine-tuning stage, labeled data was used to refine
the network parameters through back-propagation. The
network parameters were then refined to maximize this
objective function using conjugate gradient descent. The
optimal weights in the entire network were thus obtained using
the DBN.
Semi-Supervised Hashing
J. Wang and Chang proposed the semi-supervised hashing
for scalable image retrieval [10]. In this, for a given set of
points, a fraction of pairs were associated with two categories
of label information. A neighbor-pair can be considered as the
pair of points either neighbors in a metric space or share
common class labels. Similarly, a pair of points is called a non
neighbor-pair if two samples are far away in metric space or
have different class labels.
The objective function of SSH consists of two major
components, supervised empirical fitness and unsupervised
information theoretic regularization. The supervised part tries
to minimize an empirical error on a small amount of labeled
data. The unsupervised term provides effective regularization
by maximizing desirable properties like variance and
independence of individual bits. Given a set of n points S =pi ,
i=1,2, ...n, pi ϵ RD, in which a small fraction of pairs were
associated with two categories of label information. A pair of
points (pi,pj) is denoted as a neighbor-pair M when share
common class labels and is called a non neighbor pair C if the
two samples have no common class label.
( )=
ℎ ( )ℎ ( )
,
−
ℎ ( )ℎ ( )
,
+
[ℎ ( )]
(2.3)
where the first term measures the empirical accuracy over
the labeled samples M and C and the second term realizes the
maximum entropy principle.
The empirical accuracy for a family of hash functions H is
defined as the difference of the total number of correctly
classified pairs and the total number of wrongly classified
pairs. Maximizing the empirical accuracy for just a few pairs
can lead to severe over fitting. Hence a regularizer is used
which utilizes both labeled data and the unlabeled data. From
the theoretic point of view, one would like to maximize the
information provided by each bit. Using maximum entropy
principle, a binary bit that gives balanced partition of points
provides maximum information [10]. The maximum variance
condition is that a hash function with maximum entropy must
maximize the variance of the hash values and vice versa.
D. Comparison of mechanisms in image retrieval
A comparison of mechanisms used in image retrieval is
given below:
__________________________________________________________________________________________________________________
4
International Journal of Computer and Advanced Engineering Research (IJCAER)
Volume 02– Issue 02, APRIL 2015
____________________________________________________________________________________________________________________
TABLE I.
IMAGE
SEARCHING
TECHNIQUES
COMPARISON TABLE
METHODOLOGY
ADVANTAGES
DISADVANTAGES
Inverted Files
Indexing and Query
evaluation
Queries are ranked by
statistical similarity
Increases memory usage
Tree-based Indexing
Multiple randomized KDtrees
Dramatic improvement
in retrieval quality
Do not work well with high
dimensional features
Semi-supervised hashing
Minimize the empirical
error in labeled data
Performance degrades with
less training data due to
overfitting
Hashing Techniques
[5]
IV. CONCLUSION
With the increasing demands of multimedia applications
over the Internet, the importance of image retrieval has also
increased. In this research study, recent image searching
techniques are discussed. All these techniques have their own
advantages as well as certain limitations. State-of the- art
solutions often use hashing methods to embed highdimensional image features into Hamming space, where search
can be performed in real-time based on Hamming distance of
compact hash codes. Unlike traditional metrics (e.g.,
Euclidean) that offer continuous distances, the Hamming
distances are discrete integer values. As a consequence, there
are often a large number of images sharing equal Hamming
distances to a query, which largely hurts search results where
fine-grained ranking is very important.
REFERENCES
[1]
[2]
[3]
[4]
J. Zobel and A. Moffat, “Inverted files for text search engines,” ACM
Comput. Surveys, 2006.
M. D. H. Jegou and C. Schmid, “Packing bag-of-features,” Proc. IEEE
Conf. Computer Vision and Pattern Recognition, 2009.
C. Silpa-Anan and R. Hartley, “Optimised kd-trees for fast image
descriptor matching,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition, 2008.
A. T. Y. Weiss and R. Fergus, “Spectral hashing,” Adv.Neural Inf.
Process. Syst, 2008.
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
P. Indyk and R. Motwani, “Approximate nearest neighbors: Towards
removing the curse of dimensionality,” Proc. Symp. Theory of
Computing, 1998.
B. Kulis and K. Grauman, “Kernelized locality-sensitive hashing for
scalable image search,” Proc. IEEE Int. Conf. Computer Vision, 2009.
M. P. O. Chum and J. Matas, “Geometric min-hashing: Finding a (thick)
needle in a haystack,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition, 2009.
R. Salakhutdinov and G. Hinton, “Semantic hashing,” Proc. Workshop of
ACM SIGIR Conf. Research and Development in Information Retrieval,
2007.
G. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data
with neural networks,” Science, 2006.
S. K. J. Wang and S.-F. Chang, “Semi-supervised hashing for scalable
image retrieval,” Proc. IEEE Conf. Computer Vision and Pattern
Recognition, 2010.
R. F. A. Torralba and Y. Weiss, “Small codes and large image databases
for recognition,” Proc. of CVPR, 2008.
G. Hinton, “Training products of experts by minimizing contrastive
divergence,” Neural Computation, 2002.
E. Horster and R. Lienhar, “Deep networks for image retrieval on largescale databases,” Proc. ACM Int. Conf. Multimedia, 2008.
G. H. J. Goldberger, S. Roweis and R. Salakhutdinov, “Neighbourhood
components analysis,” Proc. of NIPS, 2005.
G. H. J. Goldberger, S. Roweis and R. Salakhutdinov, “Neighbourhood
components analysis,” Adv. Neural Inf. Process. Syst, 2004.
X. X. Yu-Gang Jiang, Jun Wang and S.-F. Chang, “Query-adaptive image
search with hash codes,” IEEE transactions on multimedia, 2013.
__________________________________________________________________________________________________________________
5