Publications

Refine Results

(Filters Applied) Clear All

Tagged As

The MIT-LL/AFRL MT System

October 24, 2005
  |
Conference Paper
Author:
Wade Shen
…
Published in:
Int. Workshop on Spoken Language Translation, IWSLT, 24-25 October 2005.
Topic:
machine translation
R&D area:
R&D group:

Summary

The MITLL/AFRL MT system is a statistical phrase-based translation system that implements many modern SMT training and decoding techniques. Our system was designed with the long term goal of dealing with corrupted ASR input for Speech-to-Speech MT applications. This paper will discuss the architecture of the MITLL/AFRL MT system, and experiments with manual and ASR transcription data that were run as part of the IWSLT-2005 Chinese-to-English evaluation campaign.
READ LESS

Summary

The MITLL/AFRL MT system is a statistical phrase-based translation system that implements many modern SMT training and decoding techniques. Our system was designed with the long term goal of dealing with corrupted ASR input for Speech-to-Speech MT applications. This paper will discuss the architecture of the MITLL/AFRL MT system, and...

READ MORE
The MIT-LL/AFRL MT System

Two experiments comparing reading with listening for human processing of conversational telephone speech

September 4, 2005
  |
Conference Paper
Author:
Douglas A. Jones
…
Published in:
6th Annual Conf. of the Int. Speech Communication Association, INTERSPEECH 2005, 4-8 September 2005.
Topic:
machine translation
R&D area:
R&D group:

Summary

We report on results of two experiments designed to compare subjects' ability to extract information from audio recordings of conversational telephone speech (CTS) with their ability to extract information from text transcripts of these conversations, with and without the ability to hear the audio recordings. Although progress in machine processing of CTS speech is well documented, human processing of these materials has not been as well studied. These experiments compare subject's processing time and comprehension of widely-available CTS data in audio and written formats -- one experiment involves careful reading and one involves visual scanning for information. We observed a very modest improvement using transcripts compared with the audio-only condition for the careful reading task (speed-up by a factor of 1.2) and a much more dramatic improvement using transcripts in the visual scanning task (speed-up by a factor of 2.9). The implications of the experiments are twofold: (1) we expect to see similar gains in human productivity for comparable applications outside the laboratory environment and (2) the gains can vary widely, depending on the specific tasks involved.
READ LESS

Summary

We report on results of two experiments designed to compare subjects' ability to extract information from audio recordings of conversational telephone speech (CTS) with their ability to extract information from text transcripts of these conversations, with and without the ability to hear the audio recordings. Although progress in machine processing...

READ MORE
Two experiments comparing reading with listening for human processing of conversational telephone speech

Measuring translation quality by testing English speakers with a new Defense Language Proficiency Test for Arabic

May 2, 2005
  |
Conference Paper
Author:
Douglas A. Jones
…
Published in:
Int. Conf. on Intelligence Analysis, 2-5 May 2005.
Topic:
machine translation
R&D area:
R&D group:

Summary

We present results from an experiment in which educated English-native speakers answered questions from a machine translated version of a standardized Arabic language test. We compare the machine translation (MT) results with professional reference translations as a baseline for the purpose of determining the level of Arabic reading comprehension that current machine translation technology enables an English speaker to achieve. Furthermore, we explore the relationship between the current, broadly accepted automatic measures of performance for machine translation and the Defense Language Proficiency Test, a broadly accepted measure of effectiveness for evaluating foreign language proficiency. In doing so, we intend to help translate MT system performance into terms that are meaningful for satisfying Government foreign language processing requirements. The results of this experiment suggest that machine translation may enable Interagency Language Roundtable Level 2 performance, but is not yet adequate to achieve ILR Level 3. Our results are based on 69 human subjects reading 68 documents and answering 173 questions, giving a total of 4,692 timed document trials and 7,950 question trials. We propose Level 3 as a reasonable nearterm target for machine translation research and development.
READ LESS

Summary

We present results from an experiment in which educated English-native speakers answered questions from a machine translated version of a standardized Arabic language test. We compare the machine translation (MT) results with professional reference translations as a baseline for the purpose of determining the level of Arabic reading comprehension that...

READ MORE
Measuring translation quality by testing English speakers with a new Defense Language Proficiency Test for Arabic

Measuring human readability of machine generated text: three case studies in speech recognition and machine translation

March 19, 2005
  |
Conference Paper
Author:
Douglas A. Jones
…
Published in:
Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, Vol. 5, ICASSP, 19-23 March 2005, pp. V-1009 - V-1012.
Topic:
machine translation
R&D area:
R&D group:

Summary

We present highlights from three experiments that test the readability of current state-of-the art system output from (1) an automated English speech-to-text system (2) a text-based Arabic-to-English machine translation system and (3) an audio-based Arabic-to-English MT process. We measure readability in terms of reaction time and passage comprehension in each case, applying standard psycholinguistic testing procedures and a modified version of the standard Defense Language Proficiency Test for Arabic called the DLPT*. We learned that: (1) subjects are slowed down about 25% when reading system STT output, (2) text-based MT systems enable an English speaker to pass Arabic Level 2 on the DLPT* and (3) audio-based MT systems do not enable English speakers to pass Arabic Level 2. We intend for these generic measures of readability to predict performance of more application-specific tasks.
READ LESS

Summary

We present highlights from three experiments that test the readability of current state-of-the art system output from (1) an automated English speech-to-text system (2) a text-based Arabic-to-English machine translation system and (3) an audio-based Arabic-to-English MT process. We measure readability in terms of reaction time and passage comprehension in each...

READ MORE
Measuring human readability of machine generated text: three case studies in speech recognition and machine translation

New measures of effectiveness for human language technology

January 1, 2005
  |
Journal Article
Author:
Douglas A. Jones
…
Published in:
Lincoln Laboratory Journal, Vol. 15, No. 2, 2005, pp. 341-345.
Topic:
machine translation
R&D area:
R&D group:

Summary

The field of human language technology (HLT) encompasses algorithms and applications dedicated to processing human speech and written communication. We focus on two types of HLT systems: (1) machine translation systems, which convert text and speech files from one human language to another, and (2) speech-to-text (STT) systems, which produce text transcripts when given audio files of human speech as input. Although both processes are subject to machine errors and can produce varying levels of garbling in their output, HLT systems are improving at a remarkable pace, according to system-internal measures of performance. To learn how these system-internal measurements correlate with improved capabilities for accomplishing real-world language-understanding tasks, we have embarked on a collaborative, interdisciplinary project involving Lincoln Laboratory, the MIT Department of Brain and Cognitive Sciences, and the Defense Language Institute Foreign Language Center to develop new techniques to scientifically measure the effectiveness of these technologies when they are used by human subjects.
READ LESS

Summary

The field of human language technology (HLT) encompasses algorithms and applications dedicated to processing human speech and written communication. We focus on two types of HLT systems: (1) machine translation systems, which convert text and speech files from one human language to another, and (2) speech-to-text (STT) systems, which produce...

READ MORE
New measures of effectiveness for human language technology

Two new experimental protocols for measuring speech transcript readability for timed question-answering tasks

November 8, 2004
  |
Conference Paper
Author:
Edward A. F. Gibson
…
Published in:
Proc. DARPA EARS Rich Translation Workshop, 8-11 November 2004.
Topic:
machine translation
R&D area:
R&D group:

Summary

This paper reports results from two recent psycholinguistic experiments that measure the readability of four types of speech transcripts for the DARPA EARS Program. The two key questions in these experiments are (1) how much speech transcript cleanup aids readability and (2) how much the type of cleanup matters. We employ two variants of the four-part figure of merit to measure readability defined at the RT02 workshop and described in our Eurospeech 2003 paper [4] namely: accuracy of answers to comprehension questions, reaction-time for passage reading, reaction-time for question answering and a subjective rating of passage difficulty. The first protocol employs a question-answering task under time pressure. The second employs a self-paced line-by-line paradigm. Both protocols yield similar results: all three types of clean-up in the experiment improve readability 5-10%, but the self-paced reading protocol needs far fewer subjects for statistical significance.
READ LESS

Summary

This paper reports results from two recent psycholinguistic experiments that measure the readability of four types of speech transcripts for the DARPA EARS Program. The two key questions in these experiments are (1) how much speech transcript cleanup aids readability and (2) how much the type of cleanup matters. We...

READ MORE
Two new experimental protocols for measuring speech transcript readability for timed question-answering tasks

The effect of text difficulty on machine translation performance -- a pilot study with ILR-related texts in Spanish, Farsi, Arabic, Russian and Korean

May 26, 2004
  |
Conference Paper
Author:
Ray Clifford
…
Published in:
4th Int. Conf. on Language Resources and Evaluation, LREC, 26-28 May 2004.
Topic:
machine translation
R&D area:
R&D group:

Summary

We report on initial experiments that examine the relationship between automated measures of machine translation performance (Doddington, 2003, and Papineni et al. 2001) and the Interagency Language Roundtable (ILR) scale of language proficiency/difficulty that has been in standard use for U.S. government language training and assessment for the past several decades (Child, Clifford and Lowe 1993). The main question we ask is how technology-oriented measures of MT performance relate to the ILR difficulty levels, where we understand that a linguist with ILR proficiency level N is expected to be able to understand a document rated at level N, but to have increasing difficulty with documents at higher levels. In this paper, we find that some key aspects of MT performance track with ILR difficulty levels, primarily for MT output whose quality is good enough to be readable by human readers.
READ LESS

Summary

We report on initial experiments that examine the relationship between automated measures of machine translation performance (Doddington, 2003, and Papineni et al. 2001) and the Interagency Language Roundtable (ILR) scale of language proficiency/difficulty that has been in standard use for U.S. government language training and assessment for the past several...

READ MORE
The effect of text difficulty on machine translation performance -- a pilot study with ILR-related texts in Spanish, Farsi, Arabic, Russian and Korean

Speech-to-speech translation: technology and applications study

May 10, 2002
  |
Technical Report
Author:
Clifford J. Weinstein
Published in:
MIT Lincoln Laboratory Report TR-1080
Topic:
machine translation
R&D area:
R&D group:

Summary

This report describes a study effort on the state-of-the-art and lessons learned in automated, two- way, speech-to-speech translation and its potential application to military problems. The study includes and comments upon an extensive set of references on prior and current work in speech translation. The study includes recommendations on future military applications and on R&D needed to successfully achieve those applications. Key findings of the study include: (1) R&D speech translation systems have been demonstrated, but only in limited domains, and their performance is inadequate for operational use; (2) as far as we have been able to determine, there are currently no operational two-way speech translation systems; (3) intensive, sustained R&D will be needed to develop usable two-way speech translation systems. Major recommendations include: (1) a substantial R&D program in speech translation is needed, especially including full end-to-end system prototyping and evaluation; (2) close cooperation among researchers and users speaking multiple languages will be needed for the development of useful application systems; (3) to get military users involved and interacting in a mode which enables them to provide useful inputs and feedback on system requirements and performance, it will be necessary to provide them at the start with a fairly robust, open-domain system which works to the degree that some two-way speech translation is operational.
READ LESS

Summary

This report describes a study effort on the state-of-the-art and lessons learned in automated, two- way, speech-to-speech translation and its potential application to military problems. The study includes and comments upon an extensive set of references on prior and current work in speech translation. The study includes recommendations on future...

READ MORE
Speech-to-speech translation: technology and applications study

Interlingua-based English-Korean two-way speech translation of doctor-patient dialogues with CCLINC

January 1, 2002
  |
Journal Article
Author:
Young-Suk Lee
…
Published in:
Machine Trans. Vol. 17, No. 3, 2002, pp. 213-243.
Topic:
machine translation
R&D area:
R&D group:

Summary

Development of a robust two-way real-time speech translation system exposes researchers and system developers to various challenges of machine translation (MT) and spoken language dialogues. The need for communicating in at least two different languages poses problems not present for a monolingual spoken language dialogue system, where no MT engine is embedded within the process flow. Integration of various component modules for real-time operation poses challenges not present for text translation. In this paper, we present the CCLINC (Common Coalition Language System at Lincoln Laboratory) English-Korean two-way speech translation system prototype trained on doctor-patient dialogues, which integrates various techniques to tackle the challenges of automatic real-time speech translation. Key features of the system include (i) language-independent meaning representation which preserves the hierarchical predicate-argument structure of an input utterance, providing a powerful mechanism for discourse understanding of utterances originating from different languages, word-sense disambiguation and generation of various word orders of many languages, (ii) adoption of the DARPA Communicator architecture, a plug-and-play distributed system architecture which facilitates integration of component modules and system operation in real time, and (iii) automatic acquisition of grammar rules and lexicons for easy porting of the system to different languages and domains. We describe these features in detail and present experimental results.
READ LESS

Summary

Development of a robust two-way real-time speech translation system exposes researchers and system developers to various challenges of machine translation (MT) and spoken language dialogues. The need for communicating in at least two different languages poses problems not present for a monolingual spoken language dialogue system, where no MT engine...

READ MORE
Interlingua-based English-Korean two-way speech translation of doctor-patient dialogues with CCLINC

Toward an improved concept-based information retrieval system

September 9, 2001
  |
Conference Paper
Author:
Peter V. Henstock
…
Published in:
Proc. of the 24th Annual ACM SIGIR Conf. on Research and Development in Information Retrieval, 9-13 September 2001, pp. 384-385.
Topic:
machine translation
R&D area:
R&D group:

Summary

This paper presents a novel information retrieval system that includes 1) the addition of concepts to facilitate the identification of the correct word sense, 2) a natural language query interface, 3) the inclusion of weights and penalties for proper nouns that build upon the Okapi weighting scheme, and 4) a term clustering technique that exploits the spatial proximity of search terms in a document to further improve the performance. The effectiveness of the system is validated by experimental results.
READ LESS

Summary

This paper presents a novel information retrieval system that includes 1) the addition of concepts to facilitate the identification of the correct word sense, 2) a natural language query interface, 3) the inclusion of weights and penalties for proper nouns that build upon the Okapi weighting scheme, and 4) a...

READ MORE
Toward an improved concept-based information retrieval system
31-40 of 47