Towards Application of User-Tailored Machine Translation in Localization Andrejs Vasiļjevs, Raivis Skadiņš, Inguna Skadiņa TILDE JEC 2011, Luxembourg October 14, 2011 MT in Localization Work The localization process is generally related to the translation and cultural adaptation of software, video games, and websites, and less frequently to any written translation Translation Memory is commonly used Although there are number of MT use-cases in practical localization process, it is not yet widely adapted Previous Work Evaluation with keyboard-monitoring program and Choice Network Analysis for measuring the effort involved in post-editing MT output (O´Brien, 2005) Productivity tests have been performed in translation and localization industry settings at Microsoft (Schmidtke, 2008): • SMT system of Microsoft Research trained on MS tech domain for 3 languages for Office Online 2007 localization task: Spanish, French and German • By applying MT to all new words on average 5-10% productivity was gained Previous Work – Adobe In Adobe two experiments were performed (Flournoy and Duran, 2009): • Small test set of 800-2000 words was machine translated and post-edited • Then, based on the positive results, about 200,000 words of new text were localized • The rule-based MT was used for translation into Russian (PROMT) and SMT for Spanish and French (Language Weaver) • Productivity increase between 22% and 51% Previous Work – Autodesk Evaluation of Autodesk Moses SMT system (Plitt and Masselot, 2010): • Translation from English to French, Italian, German and Spanish with three translators for each language pair • To measure translation time special workbench was designed to capture keyboard and pause times for each sentence • MT allowed translators to improve their throughput on average by 74% • Varying increase in productivity: from 20% to 131% • Optimum throughput has been reached for sentences of around 25 words in length LetsMT! project User-driven cloud-based MT factory, based on open-source MT tools Services for data collection, MT generation, customization and running of variety of user-tailored MT systems Application in localization among the key usage scenarios Strong synergy with FP7 project ACCURAT to advance datadriven machine translation for under-resourced languages and domains Partnership with Complementing Competencies Tilde (Project Coordinator) - Latvia University of Edinburgh - UK University of Zagreb - Croatia Copenhagen University - Denmark Uppsala University - Sweden Moravia – Czech Republic SemLab – Netherlands Language expertise Research excellence Industry experience LetsMT! Main Features Online collaborative platform for MT building from user-provided data Repository of parallel and monolingual corpora for MT generation Automated training of SMT systems from specified collections of data Users can specify particular training data collections and build customised MT engines from these collections Users can also use LetsMT! platform for tailoring MT system to their needs from their non-public data User requirement analysis 21 interviews have been conducted with localization/translation agencies. Only 7 of the respondents replied that they use fully automatic MT systems in their translation practice and only one LSP organization employs MT as the primary translation method. User survey IPR of text resources in interviewee organizations 23% 37% no reply interviewee has IPR 18% interviewee has restricted/partial IPR interviewee has no IPR 22% User survey Organizations' ability to share data 16% 40% no reply/interviewee has no data now 21% perhaps yes 23% no Sharing of training data Training Using SMT Resource Directory Giza++ Moses SMT toolkit SMT Multi-Model Repository (trained SMT models) Anonymous access SMT Resource Repository Web page translation widget Web browser Plug-ins SMT System Directory Moses decoder System management, user authentication, access rights control ... Authenticated access Procesing, Evaluation ... Upload Web page Web service CAT tools LetsMT! architecture Interface Layer • Application Logic • Resource Repository (stores MT training data and trained models) • High-performance Computing Cluster (executes all computationally heavy tasks: SMT training, MT service, Processing and aligning of training data etc.) Widget https REST https REST, SOAP, ... Public API http REST/SOAP Web Page UI ... http REST/SOAP (webpage UI, web service API) http/https html • User interface Browser CAT tools plug-ins CAT CATtools tools https REST Web Browsers TCP/IP Multitier architecture Application Logic Layer Resource Repository Adapter REST SMT training High-performance Computing (HPC) Cluster Data Storage Layer (Resource Repository) RR API Translation REST HPC frontend File Share SGE CPU CPU CPU CPU CPU CPU CPU SVN CPU System DB CPU LetsMT! architecture Webpage interface where users can • See, upload and manage corpora • See, train and manage user tailored SMT system • Translate texts and files Web service API • Integration with CAT tools • Integration in web pages and web browsers LetsMT! architecture Resource Repository • Stores SMT training data • Supports different formats – TMX, XLIFF, PDF, DOC, plain text • Converts to unified format • Performs format conversions and alignment Based on Moses SMT toolkit Training tasks are managed with Moses Experiment Management System Training tasks are executed in HPC cluster (Oracle Grid Engine) Hosted in Web Services infrastructure which provides easy access to on demand computing resources New Moses features: • • • • • • incremental training, distributed language models, interpolated language models for domain adaptation randomized language models to train using huge corpora translation of formatted texts, running Moses decoder in a server mode Bilingual corpora for the EnglishLatvian system Bilingual corpus Parallel units Localization TM ~1.29 M DGT-TM ~1.06 M OPUS EMEA ~0.97 M Fiction ~0.66 M Dictionary data ~0.51 M Web corpus ~0.9 M Total 5.37 M Latvian monolingual corpora Monolingual corpus Latvian side of parallel corpus News (web) Fiction Total, Latvian Words 60 M 250 M 9M 319 M The MT System Tilde English-Latvian MT Tools: Moses, Giza++, SRILM, Tilde Latvian morphological analyzer Factored phrase-based SMT system: • Surface form Surface form, Morphology tag • 2 Language models: English Latvian • (1) 5-gram surface form • (2) 7-gram morphology tag surface surf. lemma lemma morph. morph. synt. stem suffix The MT system Development and evaluation data • • • • Development - 1000 sentences Evaluation – 500 sentences Balanced BLEU score: 35.0 Topic Percentage General information about European Union 12% Specifications, instructions and manuals 12% Popular scientific and educational 12% Official and legal documents 12% News and magazine articles 24% Information technology 18% Letters 5% Fiction 5% Localization workflow at Tilde Evaluate original / assign Translator and Editor Import into SDL Trados Analyze against TMs Translate using translation suggestions for TMs Evaluate translation quality / Edit (optional) Fix errors Hand over to customer MT Integration into Localization Workflow Evaluate original / assign Translator and Editor Analyze against TMs MT translate new sentences Translate using translation suggestions for TMs and MT Evaluate translation quality / Edit Fix errors Ready translation Evaluation of Productivity Key interest of localization industry is to increase productivity of translation process while maintaining required quality level Productivity was measured as the translation output of an average translator in words per hour 5 translators participated in evaluation including both experienced and new translators Evaluation of Quality Performed by human editors as part of their regular QA process Result of translation process was evaluated, editors did not know was or was not MT applied to assist translator Comparison to reference is not part of this evaluation Tilde standard QA assessment form was used covering the following text quality areas: • Accuracy • Spelling and grammar • Style • Terminology Error Score QA Evaluation Form Error Category Weight Amount of errors Negative points 1. Accuracy 1.1. Understanding of the source text 3 0 1.2. Understanding the functionality of the product 1.3. Comprehensibility 1.4. Omissions/Unnecessary additions 1.5. Translated/Untranslated 1.6. Left-overs 3 3 2 1 1 0 0 0 0 0 0 2 1 1 0 0 0 0 1 1 2 1 0 0 0 0 0 2 2 0 0 0 0 0 0 Resulting Evaluation Total 2. Language quality 2.1. Grammar 2.2. Punctuation 2.3. Spelling Total 3. Style 3.1. Word order, word-for-word translation 3.2. Vocabulary and style choice 3.3. Style Guide adherence 3.4. Country standards Total 4. Terminology 4.1. Glossary adherence 4.2. Consistency Total Additional plus points for style (if applicable) Grand Total Negative points per 1000 words Quality: QA Grades Error Score (sum of weighted errors) Resulting Quality Evaluation 0…9 Superior 10…29 Good 30…49 Mediocre 50…69 Poor >70 Very poor Tilde Localization QA assessment applied in the evaluation Evaluation data ► 54 documents in IT domain ► 950-1050 adjusted words in each document ► Each document was split in half: ►the first part was translated using suggestions from TM only ►the second half was translated using suggestions from both TM and MT Integration of MT in SDL Trados Evaluation Results ► Average translation productivity: ► Baseline with TM only: 550 w/h ► With TM and MT: 731 w/h 32.9% productivity increase ► ► High variability in individual performance with productivity increase by 64% to decrease by 5%. Increase of error score from 20.2 to 28.6 points but still at the level “GOOD” (<30 points) Average error scores by error types Error Class Accuracy Language quality Style Terminology Baseline scenario 6 6 3 5 MT scenario 9 10 4 7 Biggest quality degradation in language quality due to increase of grammatical errors. Conclusions The results of our experiment clearly demonstrate that it is feasible to integrate the current state of the art SMT systems for highly inflected languages into the localization process. Quality of translation decreases in all error categories still degradation is not critical and the result is acceptable for production purposes. Significant performance differences for individual translators hints to the role of human factors such as training, work procedures, personal inclinations etc. Extended experiments are planned (I) involving more translators, (II) translating texts in different domains and (III) in other language pairs. Application in localization workflow Let's MT! Thank you! The research within the LetsMT! project leading to these results has received funding from the ICT Policy Support Programme (ICT PSP), Theme 5 – Multilingual web, grant agreement no 250456
© Copyright 2024