Multi-Prototype Vector Space Models of Word Meaning _______________________________________________________________________________________________________________ Authors : Joseph Reisinger & Raymond J. Mooney REVIEW BY: NITISH GUPTA ROLL NUMBER : 10461 Introduction • Automatically judging the degree of semantic similarity between words is an important task. • It is useful in Text Classification, Information Retrieval, Textual Entailment and other language processing tasks. • The empirical approach to find semantic similarity between words uses the Distributional Hypothesis i.e. that similar words appear in similar contexts. • Traditionally word types are represented by a single “prototype” vector of contextual features derived from co-occurrence information. • The semantic similarity is measured using some measure of vector distance. Motivation • The traditional vector-space models represent a word with a single “prototype” vector which is independent of context, but the meaning of a word clearly depends on context. • A single vector space model is incapable of handling phenomena like Homonymy and Polysemy. This model is also incapable of handling the fact that the word meanings violate the Triangle Inequality when viewed at the level of word types. Eg. The word club is similar to both bat and association. But its similarity to the words bat and association clearly depends on the context the word club is used in. Methodology • The authors present a new vector-space model that represents a word’s meaning by a set of distinct “sense-specific” vectors. Therefore each word will be represented by multiple vectors each of which will be representing different context in which the word is used. • For each word, ‘w’ : Step 1: For each occurrence of the word ‘w’ a vector will be computed based on its context which is composed of a 10-word window about the word. Step 2: A set of ‘K’ clusters is formed using movMF model(mixture of von MisesFisher distributions) which models semantic relatedness using cosine similarity. A set of 𝝅𝒌 (𝒘) representing centroids of the ‘K’ clusters for each word ‘w’ is hence computed. • The clusters are not assumed to represent the different senses of the word rather the authors rely on clusters to capture meaningful variation in word usage. Methodology Image showing the methodology of obtaining clusters from different contextual appearances of the word ‘Position’. • The ‘black star’ shows the centroid of the vectors as would have been computed by a singlevector model. • The different clusters and colored stars show the different sensespecific prototype vectors pertaining to the different contexts in which the word ‘Position’ was used in the corpus. Measuring Semantic Similarity • Given two words w and w’ the authors define two noncontextual clustered similarity metrics to measure similarity of isolated words. where d(:, :) is the cosine similarity index. • In AvgSim, word similarity is computed as the average similarity of all pairs of prototype vectors of the words. Since all pair of prototypes of the words contribute in AvgSim, two words are judged similar if many of their senses are similar. • In MaxSim, similarity is measured as the maximum overall pairwise prototype similarities. Since only the closest pair of prototype contributes to the MaxSim, it judges the words as similar if only one of their senses is very close. Experimental Evaluation • The corpus used by the authors include: • A snapshot of Wikipedia taken on Sept. 29th, 2009, with Wikitext markup and articles with less than 100 words removed. • The third edition of English Gigaword Corpus, with articles containing less 100 words removed. Judging Semantic Similarity • For evaluation of various models firstly comparisons of lexical similarity measurements to human similarity judgments from the WordSim-353 dataset is done. • Spearman’s rank correlation (𝜌) with average human judgments was used to measure the quality of various models. For values of 𝑲 ∈ [𝟐, 𝟏𝟎] on Wikipedia and 𝑲 > 𝟒 on Gigawords Corpus the value of Spearman’s Correlation factor is in the range of 0.6 – 0.8. Predicting Near-Synonyms • Here multi-prototype model’s ability to determine the most closely related word to a target word is tested. The top ‘k’ most similar words were computed for each prototype of each target word. • For each prototype of each word a result from the multi-prototype vector model and one from a human is given to another human. The quality of measured from the fact that how frequently was the multi-prototype method chosen. • The results show that for homonymous words the system gives excellent results as compared to polysemous words, but for the right number of clusters the polysemous words also give good results. Thank You!! Questions