Automatic Discrimination of Text and Non-Text Natural Images Chengquan Zhang, Cong Yao, Baoguang Shi, Xiang Bai School of Electronic Information and Communications Huazhong University of Science and Technology Wuhan, 430074, P.R. China. Email: {zchengquan,yaocong2010,shibaoguang}@gmail.com , [email protected] Abstract—With the rapid growth of image and video data, there comes an interesting yet challenging problem: How to organize and utilize such large volume of data? Textual content in images and videos is an important source of information, which can be of great usefulness and assistance. Therefore, we investigate in this paper the problem of text image discrimination, which aims at distinguishing natural images with text from those without text. To tackle this problem, we propose a method that combines three mature techniques in this area, namely: MSER, CNN and BoW. To better evaluate the proposed algorithm, we also construct a large benchmark for text image discrimination, which includes natural images in a variety of scenarios. This algorithm has proven to be both effective and efficient, thus it can serve as a tool for mining valuable textual information from huge amount of image and video data. I. I NTRODUCTION Text present in images and videos, either protogenous or superimposed, carries rich and precise information, thus it can serve as an important cue for a wide range of applications, such as image search [1], human-computer interaction [2] and assistive system for the blind [3]. Different from conventional approaches related to textual information understanding, which are usually concerned with text detection or recognition [4], we tackle in this paper the problem of text image discrimination. Given a collection of natural images, the goal of text image discrimination is to discriminate the images that contain text from those that do not contain any text, without considering the location or content of text, as shown in Fig. 1. This problem is obviously of great importance and practicality, but has been neglected by the community for a long period of time. The difficulties in text image discrimination mainly stem from three aspects: (1) Diversity of texts. In stark contrast to characters in document images, which are usually with regular fonts, similar color and horizontal orientation, texts in natural images are far more diverse, i.e., they may exhibit significant variations in font, color, orientation, or even language type. Moreover, the portion of texts in an image can vary tremendously, for instance, a single character in a corner or multiple lines of characters occupying the whole image; (2) Complexity of backgrounds. The backgrounds may contain a lot of non-text elements (such as branch, grass, fence and sign) that are very alike true text, which might cause confusion; (3) Interference factors. Various factors, like noise, blur and nonuniform illumination, may give rise to failures. To address these challenges, we propose an effective and robust algorithm, which takes the advantages of three widely Fig. 1: Text image discrimination. Given a collection of images, the goal of text image discrimination is to seperate images containing text from those not containing text. Text image discrimination can eliminate majority of irrelevant images in an early stage, thus it could benefit subsequent processing, such as text detection, text recognition, and image/video indexing. used techniques in this field: MSER, CNN and BoW. Character candidates are extracted using MSER [5]. CNN act as a discriminative codebook, which compute a bank of responses for each candidate. For each image, a feature vector is generated with BoW, by aggregating and pooling the responses of all the candidates in the image. The final prediction (text or non-text image) is given by a trained SVM classifier. Moreover, we generate and release a large image database for text image discrimination, since there is no public benchmark in this field. This dataset contains a collection of natural images with and without text, from various scenarios. The texts in the images are in different languages, colors, scales, fonts, orientations, and layouts. Due to its diversity and representativeness, we believe this dataset can serve as a standard benchmark for text image discrimination. We have evaluated the effectiveness of the proposed algorithm on the proposed dataset. The experiments demonstrate that the proposed method achieves superior performance, compared with conventional approaches. With GPU, this algorithm runs reasonably fast, so it can be used in large-scale non-text image filtration. In summary, the contributions of this paper are two-fold: • We emphasize the significance of text image discrimination and propose an effective algorithm for it. • We establish a large benchmark1 for algorithm development, assessment and comparison. It includes 15302 images, exhibiting variations in scene, style, illumination, text appearance and layout, etc. II. R ELATED W ORK Text has been the core research object in the document analysis and recognition community. However, majority of the existing works have focused on document images [6], [7], while those concerned with natural images mainly tackle tasks like text detection and recognition [8]–[13], rather the task of text/non-text distinction. There are a fraction of works that address text/non-text distinction. Indermuhle et al. [14] and Vidya et al. [15] proposed a system that can distinguish between text and nontext regions in handwritten documents, respectively. However, these methods were aimed at classifying text/non-text at region level, rather than image level. Most related to our work, the algorithm of Alessi et al. [16] is able to detect if a digital image contains text or not, but it is only applicable to document images or simple video frames. In contrast, the proposed approach is capable of distinguishing between natural images with text and those without text. In this sense, this work is the first that formally tackle the problem of text image discrimination for real-world natural images. III. T EXT I MAGE D ISCRIMINATION M ETHOD In this section, we describe in detail the proposed method for text image discrimination, which is an ingenious combination of Maximally Stable Extremal Region (MSER) [5], Convolutional Neural Network (CNN) [17] and Bag of Words (BoW) [18], [19]. Recently, a rich body of works [20]– [22] have demonstrated that CNN is effective for scene text detection and recognition. The proposed algorithm also utilizes the ability of CNN to learn discriminative features for accurate text image discrimination. In training phase, K-means [23] are applied to text regions extracted by MSER, before training a CNN model. This converts the binary-class classification problem (text vs. non-text) to multi-class classification problem. According to the response of each regions, a histogram-like descriptor, which represents the characteristics of a natural image, is formed using Bag of Words (BoW). Fig. 2: Pipeline for text image discrimination. As shown in Fig. 2, the proposed method is composed of the follow major processing steps: region extraction and clustering, multi-class CNN model training, feature vector generation and linear SVM training. 1 The dataset is available at http://zchengquan.github.io/textDis/ for academic use only. A. Region Extraction and Clustering Maximally Stable Extremal Regions [5] (MSER) is widely used in the fields of text detection and recognition, since it is able to discover candidate characters effectively and efficiently. Neumann et al. [24] have also proved that MSER can capture over 95% true text regions with using multiple channels. In our method, we employ MSER to extract regions which can support the discrimination of text and non-text images. However, different from text detection or recognition, text image discrimination does not require the localization and recognition characters. Moreover, the proposed method allows text regions with few connection characters. Multi-channels (RGB-Gray) and loose settings are used in our method. Before training the CNN model, we compute HOG [25] feature for each text region from foreground (text component), and K-means is performed to cluster the text regions into multiple categories. The softmax layer in CNN model will produce a response with (K+1) dimension for each region. Note that, clustering is not applied to regions from background. B. Multi-Class CNN Model Training The whole CNN Structure is composed of 4 convolutional layers and 2 full connection layers. Normally, each convolutional layer is followed by maxpooling and rectified units. The setting of the CNN architecture used in this paper is showed in TABLE I. TABLE I: The setting of the CNN architecture used in this work. ks means the kernel size; ps means the padding size; ss means stride size; nMap means the number of feature map, and nNode means the number of linear layer node. Layer Conv-1 Conv-2 Conv-3 Conv-1 Full-1 Full-2 Settings ks=5, ps=2, ss=1, ks=5, ps=2, ss=1, ks=3, ps=1, ss=1, ks=3, ps=0, ss=1, nNode=1024 nNode=(K+1) nMap=64 nMap=128 nMap=384 nMap=512 The input image region is rescaled to 32 × 32, and this would generate 1×1 size maps through the first 4 convolutional layers under special design. The kernel numbers we used are 64,128,384 and 512, respectively. After 4 convolution steps, 2 full connection layers with 1024 and (K+1) perception units are closely followed. Since softmax are used in the last layer, we can obtain a vector of (K+1) dimension, where each dimension denotes the probability of one category. In the training phase, we use Stochastic Gradient Descent and back-propagation to jointly optimize all parameters, which can minimize the classification error as much as possible. Two important measures are taken into consideration in the training process: one is penalty factor, the other is boosting. Since clustering methods belong to unsupervised learning, which can’t divide the samples into accuracy classes. Thus, we can use different penalty factors for regions from foreground and background, by using the Negative Log Likelihood criterion as loss function. Here, we design low penalty factors for K text region clusters, which allows for CNN model to adjust the suitable clustering results. However, high value is assigned to regions from background, because whether the model is good at filtering non-text regions is important for the feature vector generation step. In order to make the CNN model be good at filtering nontext regions, we use boosting [26] to optimize our model. At first, we divide the regions into two groups: regions from foreground (text regions) and regions from background (nontext regions). K-means is adopted to divide the text regions into multiple clusters, while keep one class for non-text regions. Because of the complexity and diversity of background, we randomly select some samples from the background. When the first time of the CNN model achieving an optimum state, we add hard non-text examples in next training process with the use of this model, which procedure repeats for 3∼5 times. C. Feature Vector Generation and Linear SVM Training Given an image, N regions are extracted with MSER in multi-channels, which is followed by color channel changing and scale regularization. Each region will generate a response with (K+1) dimension. A feature matrix with the size of N × (K + 1) can be used to represent the whole image. Then, a feature vector is formed with BoW, by aggregating and pooling the feature matrix, which is of (K+1) dimension. We call this operation CNN Coding, since the softmax output of CNN model acts as the codebook, which is similar to sparse coding used in word retrieval [27]. The coding output can be expressed as follow: Φ(I) = K+1 X αi φi , (1) i=1 where I denotes an image and φi is the i-th class response. αi denotes the weight of i-th class. Thus, Φ(I) stands for the CNN Coding result for a whole image. A feasible way that can improve upon CNN Coding is to perform region filtering before coding, since regions from background are complex and diverse. Moreover, given an text image, the number of non-text regions extracted by MSER is usually several order of magnitude larger than that of the text regions. In order to generate an reliable and effective descriptor for an image, some measures should be adopted to control the ratio between text and non-text regions. The trained CNN model has proven to be excellent at filtering non-text regions, so we not only utilize the CNN model to generate a feature vector of an image, but also use the model as a filter. As shown in Fig. 3, the CNN model is able to effectively eliminate nontext regions, which makes the final feature more discriminative. We will demonstrate the advantage of this strategy in Sec. V. Finally, the coding results are fed to a SVM classifier. In our method, we used LIBLINEAR [28] as the final classifier, which leads to excellent performance. IV. DATASET AND E VALUATION P ROTOCOL In this section, we introduce the proposed dataset for text image discrimination and the corresponding evaluation method. A. Dataset In order to evaluate the proposed algorithm and compare it with other methods, we have collected a large dataset from internet, which includes 7302 text images and 8000 non-text Fig. 3: Regions filtered by the CNN model. (a) Text image, (b) Non-text image. Input images are showed in the first row. The red bounding boxes presented in the second row are the regions extracted by MSER, while the green bounding boxes presented in the third row are the survived regions after filtering. images. Majority of the images are natural image, while a small number of them are born-digital images and scanned document images. If one image contains any piece of text line, it is considered as text image, regardless of the portion of text areas. The dataset is challenging and diverse, due to the variations in scene type, style, illumination, language type and text layout, as well as the complexity of backgrounds. As shown in Fig. 4, the text images are manually labelled with bounding boxes for visible text regions. We randomly select 2000 samples from each class to build the testing dataset, all the rest images are used as the training set. B. Evaluation Protocol Alike to evaluation methods for general image classification, precision(P) , recall(R) and F-Measure(F) are used as the metrics to measure the performance of algorithms for text image discrimination. These three metrics are defined as follows: TP P = TP + FP TP R= TP + FN 2×P ×R F = (2) P +R where, T P is the number of text images that are correctly classified and F P is the number of non-text images that are falsely classified. F N is the number of text images that are falsely classified. In our experiments, precision means the ratio (a) text images (b) non-text images Fig. 4: Examples from the proposed dataset. (a) Text images along with manually labelled bounding boxes. (b) Non-text images. TABLE III: Effects of different percentages of regions reserved. between the number of text images over that of those predicted as positive, and recall means the ratio between the number of Percentage Precision Recall F-Measure correctly classified text images over all text images in the test 1% 0.900 0.884 0.892 2% 0.898 0.903 0.901 set. V. E XPERIMENTS AND D ISCUSSIONS In this section, we evaluate our algorithm on the proposed benchmark dataset, using the standard evaluation protocol. In the following experiments, we will discuss the effects of different numbers of clusters on text regions, and different percentages of regions reserved after filtering. Moreover, we will compare our method with other traditional algorithms for image classification, such as Locality-constrained Linear Coding (LLC) [29], CNN, and improved LLC with using MSER. Effect of cluster number: As shown in TABLE II, we compare 7 different numbers of clusters used in training the CNN model. The precision and recall are measured when the F-Measure achieving optimum value. Comparing different values of K, we find out the best performance is achieved at the value of K = 100, over which there will be no obvious improvement. TABLE II: Effects of different values of K (K+1) 2 51 101 201 301 401 501 Precision 0.889 0.906 0.898 0.892 0.881 0.894 0.879 Recall 0.878 0.874 0.903 0.892 0.902 0.884 0.908 F-Measure 0.883 0.890 0.901 0.892 0.891 0.888 0.892 Effect of reserved region ratio: As aforementioned, we found that too many non-text regions will have a negative effect on the final performance. In our experiments, we have experimented with different percentages of regions reserved. As can be observed from TABLE III, the best performance is obtained when 2% top-ranked regions are reserved, which strongly proves that too many regions from background would hurt the classification performance. In this experiment, the precision and recall are also measured when F-Measure achieves optimum value. 5% 10% 20% 50% 100% 0.906 0.897 0.916 0.903 0.901 0.866 0.853 0.819 0.816 0.812 0.885 0.874 0.864 0.857 0.854 Comparison with other methods: (1) Locality-constrained Linear Coding [29] is our first baseline method, which extracts dense sift features with 3 different scales. The codebook size in our experiment is 2048. Besides, we replace SPM [30] with global max-pooling, since we find the SPM hardly bring any improvement with spending more time on coding. (2) CNN is considered as the second baseline method, whose structure is similar to the model used in our proposed method, except that the input scale is 224 × 224 and a global maxpooling is performed before the last 2 full connection layers. (3) Taking the bounding box information into account, we also design an improved coding scheme for the traditional LLC. For each image HOG, LBP, gradient histogram, and gradient orientation histogram are extracted for the stable regions after filtering, and integrated with the LLC method. We call this method MSER+Adaboost+LLC. The P-R curves in Fig. 5 show that with the bounding box information, the proposed method achieves the significantly enhanced performance, compared with the baseline methods. Note that the comparison between the proposed method (CNN Coding) and MSER+Adaboost+LLC is fair, since both methods used bounding box labels in the training phase. Time Cost: We also measured the time cost of our algorithm on a regular PC (CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @3.30 GHz; GPU: Tesla K40c; RAM: 8GB). As shown in TABLE IV, our method takes around 0.43∼0.49s to accomplish the classification task on a single CPU and a single GPU. We analyse the average time taken for each stage of our pipeline on our dataset, which has an average image size of 720 × 620. The step of MSER is executed on C++ with Opencv-2.4.8. The CNN Coding and linear SVM are executed on the platform of Torch7 under Linux. The proposed system achieves high classification accuracy [6] [7] [8] [9] [10] [11] Fig. 5: The precision/recall curves of different methods on our [12] dataset. and runs reasonably fast, thus it can be used as a powerful tool in large-scale textual information mining tasks. TABLE IV: Time cost for each stage. Stage MSER Extraction CNN Coding SVM Classification Total VI. Time 0.18∼0.23s 0.25∼0.26s 0.24ms 0.43∼ 0.49s C ONCLUSION We have presented an effective and efficient algorithm for text image discrimination, an important problem in textual information understanding. Moreover, a large image dataset, which includes diverse natural images with and without text, is generated and released. This dataset can serve as a standard benchmark in this field. The experiments on the proposed benchmark show the advantages of the proposed algorithm over conventional methods. In this paper, we used the bounding boxes as well as the image level labels, for training the model for text image discrimination. In the future, we will investigate methods that only utilize image level labels [31]. Besides, it is certainly that using this method can also narrow the searching range of text region, which would be helpful to text localization and deserve further study. ACKNOWLEDGEMENTS This work was primarily supported by National Natural Science Foundation of China (NSFC) (No.61222308), and in part by Program for New Century Excellent Talentsin University under Grant NCET-12-0217, and by CCFTencentRAGR20140116. R EFERENCES [1] [2] [3] [4] [5] S. S. Tsai, H. Chen, D. Chen, G. Schroth, R. Grzeszczuk, and B. Girod, “Mobile visual search on printed documents using text and low bit-rate features,” in In Proc. of ICIP, 2011, pp. 2601–2604. B. Kisacanin, V. Pavlovic, and T. S. Huang, Real-time vision for humancomputer interaction. Springer Science & Business Media, 2005. B. T. Nassu, R. Minetto, and L. E. S. d. Oliveira, “Text line detection in document images: Towards a support system for the blind,” in In Proc. of ICDAR, 2013, pp. 638–642. C. Yao, X. Bai, B. Shi, and W. Liu, “Strokelets: A learned multi-scale representation for scene text recognition,” in In Proc. of CVPR, 2014. J. Matas, O. Chum, M. Urban, and T. Pajdla, “Robust wide-baseline stereo from maximally stable extremal regions,” Image and vision computing, vol. 22, no. 10, pp. 761–767, 2004. [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] N. Chen and D. Blostein, “A survey of document image classification: problem statement, classifier architecture and performance evaluation,” IJDAR, vol. 10, no. 1, pp. 1–16, 2007. D. Doermann, “The indexing and retrieval of document images: A survey,” Computer Vision and Image Understanding, vol. 70, no. 3, pp. 287–298, 1998. Y. Navon, V. Kluzner, and B. Ophir, “Enhanced active contour method for locating text,” in In Proc. of ICDAR, 2011, pp. 222–226. J. Liang, D. Doermann, and H. Li, “Camera-based analysis of text and documents: a survey,” IJDAR. C. Yi, X. Yang, and Y. Tian, “Feature representations for scene text character recognition: A comparative study,” in In Proc. of ICDAR, 2013. C. Yao, X. Bai, W. Liu, Y. Ma, and Z. Tu, “Detecting texts of arbitrary orientations in natural images,” in In Proc. of CVPR, 2012. C. Yao, X. Bai, and W. Liu, “A unified framework for multioriented text detection and recognition,” IEEE Transactions on Image Processing, vol. 23, no. 11, pp. 4737–4749, 2014. Y. Zhu, C. Yao, and X. Bai, “Scene text detection and recognition: Recent advances and future trends,” Frontiers of Computer Science, 2015. E. Indermuhle, H. Bunke, F. Shafait, and T. Breuel, “Text versus nontext distinction in online handwritten documents,” in In Proc. of SAC, 2010. V. Vidya, T. R. Indhu, and V. K. Bhadran, “Classification of handwritten document image into text and non-text regions,” in In Proc. of ICSIP, 2012. N. G. Alessi, S. Battiato, G. Gallo, M. Mancuso, and F. Stanco, “Automatic discrimination of text images,” in In Proc. of SPIE, 2003. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. L. Fei-Fei and P. Perona, “A bayesian heirarcical model for learning natural scene categories,” in In Proc. of CVPR, 2005. S. Bai, X. Wang, C. Yao, and X. Bai, “Multiple stage residual model for accurate image classification,” in ACCV, 2014, pp. 430–445. T. Bluche, H. Ney, and C. Kermorvant, “Feature extraction with convolutional neural networks for handwritten word recognition,” in In Proc. of ICDAR, 2013, pp. 285–289. W. Huang, Y. Qiao, and X. Tang, “Robust scene text detection with convolution neural network induced mser trees,” in In Proc. of ECCV, 2014, pp. 497–511. M. Jaderberg, A. Vedaldi, and A. Zisserman, “Deep features for text spotting,” in In Proc. of ECCV. Springer, 2014, pp. 512–528. J. MacQueen et al., “Some methods for classification and analysis of multivariate observations,” in Proc. Fifth Berkeley Symp. on Math. Statist. and Prob. L. Neumann and J. Matas, “Real-time scene text localization and recognition,” in In Proc. of CVPR, 2012, pp. 3538–3545. N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in In Proc. of CVPR, vol. 1, 2005, pp. 886–893. J. Friedman, T. Hastie, and R. Tibshiran, “Additive logistic regression: a statistical view of boosting,” The annals of statistics, vol. 28, no. 2, pp. 337–407, 2000. R. Shekhar and C. Jawahar, “Document specific sparse coding for word retrieval,” in In Proc. of ICDAR, 2013, pp. 643–647. R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “Liblinear: A library for large linear classification,” The Journal of Machine Learning Research, vol. 9, pp. 1871–1874, 2008. J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, “Localityconstrained linear coding for image classification,” in In Proc. of CVPR, 2010, pp. 3360–3367. J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid matching using sparse coding for image classification,” in In Proc. of CVPR, 2009, pp. 1794–1801. X. Wang, X. Bai, W. Liu, and L. J. Latecki, “Feature context for image classification and object detection,” in In Proc. of CVPR, 2011.
© Copyright 2024