Package com.aspectran.core.context.expr.token

Class Tokenizer

java.lang.Object

com.aspectran.core.context.expr.token.Tokenizer

public class Tokenizer
extends java.lang.Object

The Class Tokenizer.

Created: 2008. 03. 29 AM 1:55:03

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
static Token[]	optimize(Token[] tokens)	Returns an array of tokens that is optimized.
static java.util.List<Token>	tokenize(java.lang.CharSequence input, boolean textTrim)	Returns a list of tokens that contains tokenized string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

tokenize

public static java.util.List<Token> tokenize(java.lang.CharSequence input,
                                             boolean textTrim)

Returns a list of tokens that contains tokenized string.

Parameters:

input - the string to tokenize

textTrim - whether to trim text

Returns:

a list of tokens

optimize

public static Token[] optimize(Token[] tokens)

Returns an array of tokens that is optimized. Eliminates unnecessary white spaces for the first and last tokens.

Parameters:

tokens - the tokens before optimizing

Returns:

the optimized tokens