public class StringLiteralLexer extends LexerBase
Modifier and Type | Field and Description |
---|---|
protected java.lang.CharSequence |
myBuffer |
protected int |
myBufferEnd |
protected int |
myEnd |
protected IElementType |
myOriginalLiteralToken |
protected char |
myQuoteChar |
protected int |
myStart |
static char |
NO_QUOTE_CHAR |
Constructor and Description |
---|
StringLiteralLexer(char quoteChar,
IElementType originalLiteralToken) |
StringLiteralLexer(char quoteChar,
IElementType originalLiteralToken,
boolean canEscapeEolOrFramingSpaces,
java.lang.String additionalValidEscapes) |
StringLiteralLexer(char quoteChar,
IElementType originalLiteralToken,
boolean canEscapeEolOrFramingSpaces,
java.lang.String additionalValidEscapes,
boolean allowOctal,
boolean allowHex) |
Modifier and Type | Method and Description |
---|---|
void |
advance()
Advances the lexer to the next token.
|
int |
getBufferEnd()
Returns the offset at which the lexer will stop lexing.
|
java.lang.CharSequence |
getBufferSequence()
Returns the buffer sequence over which the lexer is running.
|
int |
getState()
Returns the current state of the lexer.
|
int |
getTokenEnd()
Returns the end offset of the current token.
|
int |
getTokenStart()
Returns the start offset of the current token.
|
IElementType |
getTokenType()
Returns the token at the current position of the lexer or
null if lexing is finished. |
protected IElementType |
getUnicodeEscapeSequenceType() |
protected IElementType |
handleSingleSlashEscapeSequence() |
protected int |
locateAdditionalEscapeSequence(int start,
int indexOfCharAfterSlash)
Locates the end of an additional (non-standard) escape sequence.
|
protected int |
locateHexEscapeSequence(int start,
int i) |
protected int |
locateUnicodeEscapeSequence(int start,
int i) |
protected boolean |
shouldAllowSlashZero() |
void |
start(java.lang.CharSequence buffer,
int startOffset,
int endOffset,
int initialState)
Prepare for lexing character data from
buffer passed. |
java.lang.String |
toString() |
getCurrentPosition, restore
getTokenSequence, getTokenText, start, start
public static final char NO_QUOTE_CHAR
protected java.lang.CharSequence myBuffer
protected int myStart
protected int myEnd
protected int myBufferEnd
protected final char myQuoteChar
protected final IElementType myOriginalLiteralToken
public StringLiteralLexer(char quoteChar, IElementType originalLiteralToken)
public StringLiteralLexer(char quoteChar, IElementType originalLiteralToken, boolean canEscapeEolOrFramingSpaces, java.lang.String additionalValidEscapes)
public StringLiteralLexer(char quoteChar, IElementType originalLiteralToken, boolean canEscapeEolOrFramingSpaces, java.lang.String additionalValidEscapes, boolean allowOctal, boolean allowHex)
canEscapeEolOrFramingSpaces
- true
if following sequences are acceptable:'\'
at the end of the buffer (meaning escaped end of line)'\ '
(escaped space) at the beginning and ath the end of the buffer
(meaning escaped space, to avoid auto trimming on load)public void start(java.lang.CharSequence buffer, int startOffset, int endOffset, int initialState)
Lexer
buffer
passed. Internal lexer state is supposed to be initialState
. It is guaranteed
that the value of initialState is the same as returned by Lexer.getState()
method of this lexer at condition startOffset=getTokenStart()
.
This method is used to incrementally re-lex changed characters using lexing data acquired from this particular lexer sometime in the past.public int getState()
Lexer
public IElementType getTokenType()
Lexer
null
if lexing is finished.getTokenType
in class Lexer
protected boolean shouldAllowSlashZero()
protected IElementType handleSingleSlashEscapeSequence()
protected IElementType getUnicodeEscapeSequenceType()
public int getTokenStart()
Lexer
getTokenStart
in class Lexer
public int getTokenEnd()
Lexer
getTokenEnd
in class Lexer
protected int locateHexEscapeSequence(int start, int i)
protected int locateUnicodeEscapeSequence(int start, int i)
protected int locateAdditionalEscapeSequence(int start, int indexOfCharAfterSlash)
Locates the end of an additional (non-standard) escape sequence. The sequence is considered to begin with a backslash symbol
located at the start
index of the lexer's buffer.
Override this method if your language supports non-standard escape sequences. For example, in Go language Unicode escapes look
like '\U12345678'. To locate this escape sequence, the implementation should check that indexOfCharAfterSlash
points to the
'U' symbol in the buffer and return start + 8
(or the end index of the buffer if it is too short).
Otherwise, the implementations should return -1 to indicate that the current buffer starting at the index start
doesn't represent an additional escape sequence.
When overriding this method, you most likely will need to also override getTokenType()
to return proper type for the sequences located here.
public void advance()
Lexer
public java.lang.CharSequence getBufferSequence()
Lexer
start()
method.getBufferSequence
in class Lexer
public int getBufferEnd()
Lexer
endOffset
parameter
to the start()
method.getBufferEnd
in class Lexer
public java.lang.String toString()
toString
in class java.lang.Object