Sources of Evidence for Automatic Indexing of Political Texts Mostafa Dehghani, Hosein Azarbonyad, Maarten Marx, and Jaap Kamps University of Amsterdam Motivation Experiments Political texts on the Web, documenting laws and policies and the process leading to them, are of key importance to government, industry, and every individual citizen. Data Collection I Annotated Data: JRC-Acquis . from 2002 to 2006 . 16,824 documents I Taxonomy: EuroVoc . 6,796 hierarchically structured concepts Experiment 1: Performance of LTR Laws Questions Debates What laws are being made? What committees are working on? What parliaments is talking about? Performance of JEX, best single feature, and LTR methods. Research Questions: I What are the different possible sources of evidence for automatic indexing of political text? I How effective is a learning to rank (LTR) approach integrating a variety of sources of information as features? Method JEX BM25-Titles LTR-all P@5 (%Diff.) 0.4353 0.4798 (10%) 0.5206 (20%) Recall@5 (%Diff.) 0.4863 0.5064 (4%) 0.5467 (12%) I Exploiting LTR enables us to learn an effective way to combine features from different sources of evidence. I What is the relative importance of each of these sources of information for indexing political text? Experiment 2: Feature Analysis I Can we build a lean-and-mean system that approximate the effectiveness of the large LTR system? lit y ne ra Ch ild ty (# gu i Ge ity Am er en bi ) gu ity (# Pa re nt s) M (t itl e_ D) Labeled Documents Po pu lar -M (t M itl (t e_ ex C) t_ _B D) M -M 25 (t M e xt (t _C ex t_ )_ D) BM -M 25 (t M i tle (t _C itl e_ )_ D) BM -M 25 (t M e xt (t itl _C e_ )_ D) BM -M 25 (d M e sc (t ex _C t_ )_ D) BM M -M (t 25 (d ex es t_ c_ D) C) -M _B ( an M M c_ (t 25 itl de e_ sc D) _C -M )_ (a BM nc 25 _d es c_ C) _B M 25 I Different sources of evidence for the selection of appropriate indexing terms for political documents: P@5 SVMRank Weight Am bi Learning to Rank for Sources of Evidence Combination 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Feature importance: 1) P @5 of individual features, 2) weights in SVM-Rank model Textual Glosses of Descriptor Terms Thesaurus Structure I Findings: . Titles can be considered as the most succinct predictor of classes (Titles of political documents tend to be directly descriptive of the content) I Mapping Indexing Problem to the Retrieval Problem: . Popularity is not an effective feature itself, but increases the performance along with other features. . Features for reflecting the similarity of descriptor terms and documents: I Formal models of documents and descriptor terms: M odelD = hM (titleD ), M (textD )i M odelDT = hM (titleDT ), M (textDT ), M (glossDT ), M (anc_glossDT )i Experiment 3: Lean and Mean System I Selected Features: Documents . Similarity of text submodels . Similarity of title submodels . Similarity of text and textual glosses of descriptor terms . Popularity of descriptor terms .. . . .. Title Text Performance of LTR on all features, and on four selected features Description Query Method LTR-all LTR-ttgp Features Similarity Title Title Text Text Description Description Title P@5 (%Diff.) 0.5206 (-) 0.5058 (-3%) all@5 (%Diff.) 0.5467 (-) 0.5301 (-3%) Text . .. .. . . .. .. I Selective LTR approach is a computationally attractive alternative to the full LTR-all approach. . Thesaurus structure I For each combination, three IR measures are employed: a) language modeling similarity based on KL-divergence using Dirichlet smoothing, b)the same run using Jelinek-Mercer smoothing, and c) Okapi-BM25. . Features for reflecting the characteristics of descriptor terms independent of documents: I Popularity I Generality I Ambiguity ExPoSe:Exploratory Political Search, Conclusion I Using a learning to rank approach integrating all features has significantly better performance than previous systems. I The analysis of feature weights reveals the relative importance of various sources of evidence, also giving insight in the underlying classification problem. I A lean and mean system using only four features (text, title, descriptor glosses, descriptor term popularity) is able to perform at 97% of the large LTR model. Funded by NWO (project# 314-99-108) 37th European Conference on information retrieval {dehghani,h.azarbonyad}@uva.nl
© Copyright 2024