ws.palladian.extraction.token

Class OpenNlpTokenizer

java.lang.Object

ws.palladian.extraction.token.OpenNlpTokenizer

All Implemented Interfaces:

TextTokenizer

public final class OpenNlpTokenizer
extends Object
implements TextTokenizer

A TextTokenizer implemenation based on Apache OpenNLP. OpenNLP provides several different tokenizers, ranging from simple, rule-based ones to learnable tokenizers relying on a trained model. For more information, see the documentation section on tokenization in the OpenNLP Developer Documentation.

Author:

Philipp Katz

Constructor Summary

Constructors

Constructor and Description
OpenNlpTokenizer() Create a new OpenNlpTokenizer using a SimpleTokenizer, which tokenizes based on same character classes.
OpenNlpTokenizer(File modelFile) Create a new OpenNlpTokenizer based on a learned model.
OpenNlpTokenizer(opennlp.tools.tokenize.Tokenizer tokenizer) Create a new OpenNlpTokenizer using an arbitrary implementation of Tokenizer.

Method Summary

Modifier and Type	Method and Description
Iterator<Token>	iterateTokens(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OpenNlpTokenizer

public OpenNlpTokenizer()

Create a new OpenNlpTokenizer using a SimpleTokenizer, which tokenizes based on same character classes.

OpenNlpTokenizer

public OpenNlpTokenizer(opennlp.tools.tokenize.Tokenizer tokenizer)

Create a new OpenNlpTokenizer using an arbitrary implementation of Tokenizer.

Parameters:

tokenizer -

OpenNlpTokenizer

public OpenNlpTokenizer(File modelFile)

Create a new OpenNlpTokenizer based on a learned model. Such learned models are available for example on the OpenNLP Tools Models web page.

Parameters:

modelFile - Path to the model file, must not be null.

Method Detail

iterateTokens

public Iterator<Token> iterateTokens(String text)

Specified by:

iterateTokens in interface TextTokenizer