Min H Tang

US Patent:

20040111253, Jun 10, 2004

Filed:

Dec 10, 2002

Appl. No.:

10/315537

Inventors:

Xiaoqiang Luo - Ardsley NY, US
Salim Roukos - Scarsdale NY, US
Min Tang - Cambridge MA, US

Assignee:

International Business Machines Corporation - Armonk NY

International Classification:

G06F017/28

US Classification:

704/004000

Abstract:

A method, computer program product, and data processing system for training a statistical parser by utilizing active learning techniques to reduce the size of the corpus of human-annotated training samples (e.g., sentences) needed is disclosed. According to a preferred embodiment of the present invention, the statistical parser under training is used to compare the grammatical structure of the samples according to the parser's current level of training. The samples are then divided into clusters, with each cluster representing samples having a similar structure as ascertained by the statistical parser. Uncertainty metrics are applied to the clustered samples to select samples from each cluster that reflect uncertainty in the statistical parser's grammatical model. These selected samples may then be annotated by a human trainer for training the statistical parser.