Tokenizer (Aspectran Core API Reference (6.6.0-M3))

java.lang.Object
- com.aspectran.core.context.expr.token.Tokenizer

```
public class Tokenizer
extends java.lang.Object
```
The Class Tokenizer.
Created: 2008. 03. 29 AM 1:55:03

Constructor Summary

Constructors
Constructor Description

Tokenizer()

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method	Description
`static Token[]`	`optimize(Token[] tokens)`	Returns an array of tokens that is optimized.
`static java.util.List<Token>`	`tokenize(java.lang.CharSequence input, boolean textTrim)`	Returns a list of tokens that contains tokenized string.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Tokenizer
```
public Tokenizer()
```
- Method Detail
  - tokenize
```
public static java.util.List<Token> tokenize(java.lang.CharSequence input,
                                             boolean textTrim)
```
    Returns a list of tokens that contains tokenized string.
    
    Parameters:
    
    input - the string to tokenize
    
    textTrim - whether to trim text
    
    Returns:
    
    a list of tokens
  - optimize
```
public static Token[] optimize(Token[] tokens)
```
    Returns an array of tokens that is optimized. Eliminates unnecessary white spaces for the first and last tokens.
    
    Parameters:
    
    tokens - the tokens before optimizing
    
    Returns:
    
    the optimized tokens