package token
This package provides a wealth of functionality for performing common lexing tasks.
It is organised as follows:
- the main parsing functionality is accessed via
Lexer, which provides implementations for the combinators found in the sub-packages given aLexicalDesc. - the
descriptionssub-package is how a lexical structure can be described, providing the configuration that alters the behaviour of the parsers produced by theLexer. - the other sub-packages contain the high-level interfaces that the
Lexerexposes, which can be used to pass whitespace-aware and non-whitespace-aware combinators around in a uniform way. - the
predicatemodule contains functionality to help define boolean predicates on characters or unicode codepoints.
- Source
- package.scala
- Alphabetic
- By Inheritance
- token
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
Lexer extends AnyRef
This class provides a large selection of functionality concerned with lexing.
This class provides a large selection of functionality concerned with lexing.
This class provides lexing functionality to
parsley, however it is guaranteed that nothing in this class is not implementable purely usingparsley's pre-existing functionality. These are regular parsers, but constructed in such a way that they create a clear and logical separation from the rest of the parser.The class is broken up into several internal "modules" that group together similar kinds of functionality. Importantly, the
lexemesandnonlexemesobjects separate the underlying token implementations based on whether or not they consume whitespace or not. Functionality is broadly duplicated across both of these modules:lexemesshould be used by a wider parser, to ensure whitespace is handled uniformly; andnonlexemesshould be used to define further composite tokens or in special circumstances where whitespace should not be consumed.It is possible that some of the implementations of parsers found within this class may have been hand-optimised for performance: care will have been taken to ensure these implementations precisely match the semantics of the originals.
- Annotations
- @deprecatedInheritance( ... , "4.1.0" )
Value Members
-
object
predicate
This module contains functionality to describe character predicates, which can be used to determine what characters are valid for different tokens.
This module contains functionality to describe character predicates, which can be used to determine what characters are valid for different tokens.
- Since
4.0.0