ws.palladian.extraction.token

Class TwokenizeTokenizer

java.lang.Object

ws.palladian.extraction.token.TwokenizeTokenizer

All Implemented Interfaces:

TextTokenizer

public final class TwokenizeTokenizer
extends Object
implements TextTokenizer

Tokenizer based on the Twokenize algorithm available from here. This class uses the ported Scala version delivered with ark-tweet-nlp.

Author:

Philipp Katz

Constructor Summary

Constructors

Constructor and Description
TwokenizeTokenizer()

Method Summary

Modifier and Type	Method and Description
Iterator<Token>	iterateTokens(String text)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TwokenizeTokenizer

public TwokenizeTokenizer()

Method Detail

iterateTokens

public Iterator<Token> iterateTokens(String text)

Specified by:

iterateTokens in interface TextTokenizer