Package org.luwrain.inlandes.util

Class AbstractTokenizer

java.lang.Object

org.luwrain.inlandes.util.AbstractTokenizer

Direct Known Subclasses:

Tokenizer

public abstract class AbstractTokenizer
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static final char	NBSP
protected final List<Token>	output

Constructor Summary

Constructors

Constructor	Description
AbstractTokenizer()

Method Summary

Modifier and Type	Method	Description
abstract void	backCh(char ch)
Token[]	getOutput()
abstract boolean	hasCh()
static boolean	isCyrChar(char ch)
void	tokenize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NBSP

public static final char NBSP

See Also:

• Constant Field Values

output

protected final List<Token> output

Constructor Details

AbstractTokenizer

public AbstractTokenizer()

Method Details

hasCh

public abstract boolean hasCh()

backCh

public abstract void backCh(char ch)

tokenize

public void tokenize()

getOutput

public Token[] getOutput()

isCyrChar

public static boolean isCyrChar(char ch)