package tokenextractors
This package contains implementations of token extractors that can be mixed into ErrorBuilder
to decide how to extract unexpected tokens from the residual input left over from a parse error.
These are common strategies, and something here is likely to be what is needed. They are all careful to handle unprintable characters and whitespace in a sensible way, and account for unicode codepoints that are wider than a single 16-bit character.
- Source
- package.scala
- Since
4.0.0
- Alphabetic
- By Inheritance
- tokenextractors
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
trait
LexToken extends AnyRef
This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will try and parse the residual input to identify a valid lexical token to report.This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will try and parse the residual input to identify a valid lexical token to report.When parsing a grammar that as a dedicated lexical distinction, it is nice to be able to report problematic tokens relevant to that grammar as opposed to generic input lifted straight from the input stream. The easiest way of doing this would be having a pre-lexing pass and parsing based on tokens, but this is deliberately not how Parsley is designed. Instead, this extractor can try and parse the remaining input to try and identify a token on demand.
If the
lexicalErrorflag of theunexpectedTokenmethod is not set, which would indicate a problem within a token reported by a classical lexer and not the parser, the extractor will try to parse each of the providedtokensin turn: whichever is the longest matched of these tokens will be reported as the problematic one (this can be changed by overridingselectToken). For best effect, these tokens should not consume whitespace (which would otherwise be included at the end of the token!): this means that, if using theLexerclass, the functionality innonlexemeshould be used. If one of the givens tokens cannot be parsed, the input until the next valid parsable token (or end of input) is returned as aToken.Raw.Currently, if
lexicalErroris true, this extractor will just return the next character as the problematic item (this may be changed by overriding theextractItemmethod).- Since
4.0.0
-
trait
MatchParserDemand extends AnyRef
This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will make a token as wide as the amount of input the parser tried to consume when it failed.This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will make a token as wide as the amount of input the parser tried to consume when it failed.- Since
4.0.0
- Note
In the case of unprintable characters or whitespace, this extractor will favour reporting a more meaningful name.
-
trait
SingleChar extends AnyRef
This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will unconditionally report the first character in the remaining input as the problematic token.This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will unconditionally report the first character in the remaining input as the problematic token.- Since
4.0.0
- Note
In the case of unprintable characters or whitespace, this extractor will favour reporting a more meaningful name.
-
trait
TillNextWhitespace extends AnyRef
This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will construct a token that extends to the next available whitespace in the remaining input.This extractor mixin provides an implementation for
ErrorBuilder.unexpectedTokenwhen mixed into an error builder: it will construct a token that extends to the next available whitespace in the remaining input. It can be configured to constrict this token to the minimum of the next whitespace or whatever the parser demanded (seeMatchParserDemand).- Since
4.0.0
- Note
In the case of unprintable characters or whitespace, this extractor will favour reporting a more meaningful name.
Value Members
-
object
LexToken
This object contains helper functions useful for interacting with
LexToken.This object contains helper functions useful for interacting with
LexToken.- Since
4.0.0
-
object
MatchParserDemand
Contains the functionality of
MatchParserDemandas a function.Contains the functionality of
MatchParserDemandas a function.- Since
4.0.0
-
object
SingleChar
Contains the functionality of
SingleCharas a function.Contains the functionality of
SingleCharas a function.- Since
4.0.0
-
object
TillNextWhitespace
Contains the functionality of
TillNextWhitespaceas a function.Contains the functionality of
TillNextWhitespaceas a function.- Since
4.0.0