Package org.luwrain.inlandes.util

Class Tokenizer

java.lang.Object

org.luwrain.inlandes.util.AbstractTokenizer

org.luwrain.inlandes.util.Tokenizer

public final class Tokenizer
extends AbstractTokenizer

Field Summary

Fields inherited from class org.luwrain.inlandes.util.AbstractTokenizer

NBSP, output

Constructor Summary

Constructors

Constructor	Description
Tokenizer(Reader reader)

Method Summary

Modifier and Type	Method	Description
void	backCh(char ch)
char	getCh()
boolean	hasCh()
static Token[]	tokenize(String text)

Methods inherited from class org.luwrain.inlandes.util.AbstractTokenizer

getOutput, isCyrChar, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer(Reader reader)

Method Details

getCh

public char getCh()

hasCh

public boolean hasCh()

Specified by:

hasCh in class AbstractTokenizer

backCh

public void backCh(char ch)

Specified by:

backCh in class AbstractTokenizer

tokenize

public static Token[] tokenize(String text)