Antlr.Runtime.Lexer Class Reference
A lexer is recognizer that draws input symbols from a character stream. lexer grammars result in a subclass of this object. A Lexer object uses simplified Match() and error recovery mechanisms in the interest of speed. More...
Public Member Functions | |
| Lexer () | |
| Lexer (ICharStream input) | |
| Lexer (ICharStream input, RecognizerSharedState state) | |
| override void | Reset () |
| Reset the parser's state. Subclasses must rewind the input stream. | |
| virtual IToken | NextToken () |
| Return a token from this source; i.e., Match a token on the char stream. | |
| void | Skip () |
| Instruct the lexer to skip creating a token for current lexer rule and look for another token. NextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it. | |
| abstract void | mTokens () |
| This is the lexer entry point that sets instance var 'token'. | |
| virtual void | Emit (IToken token) |
| Currently does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method and nextToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does). | |
| virtual IToken | Emit () |
| The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. | |
| virtual void | Match (string s) |
| virtual void | MatchAny () |
| virtual void | Match (int c) |
| virtual void | MatchRange (int a, int b) |
| virtual void | Recover (RecognitionException re) |
| Lexers can normally Match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a Fragment rule. | |
| override void | ReportError (RecognitionException e) |
| Report a recognition problem. | |
| override string | GetErrorMessage (RecognitionException e, string[] tokenNames) |
| What error message should be generated for the various exception types? | |
| string | GetCharErrorDisplay (int c) |
| virtual void | TraceIn (string ruleName, int ruleIndex) |
| virtual void | TraceOut (string ruleName, int ruleIndex) |
Protected Attributes | |
| internal ICharStream | input |
| Where is the lexer drawing characters from? | |
Properties | |
| virtual ICharStream | CharStream [get, set] |
| Set the char stream and reset the lexer. | |
| override string | SourceName [get] |
| For debugging and other purposes, might want the source name. Have ANTLR provide a hook for this property. | |
| override IIntStream | Input [get] |
| virtual int | Line [get] |
| virtual int | CharPositionInLine [get] |
| virtual int | CharIndex [get] |
| What is the index of the current character of lookahead? | |
| virtual string | Text [get, set] |
| Gets or sets the 'lexeme' for the current token. | |
Private Attributes | |
| const int | TOKEN_dot_EOF = (int)CharStreamConstants.EOF |
Detailed Description
A lexer is recognizer that draws input symbols from a character stream. lexer grammars result in a subclass of this object. A Lexer object uses simplified Match() and error recovery mechanisms in the interest of speed.
Definition at line 47 of file Lexer.cs.
Constructor & Destructor Documentation
| Antlr.Runtime.Lexer.Lexer | ( | ICharStream | input | ) |
| Antlr.Runtime.Lexer.Lexer | ( | ICharStream | input, | |
| RecognizerSharedState | state | |||
| ) |
Member Function Documentation
| override void Antlr.Runtime.Lexer.Reset | ( | ) | [virtual] |
Reset the parser's state. Subclasses must rewind the input stream.
Reimplemented from Antlr.Runtime.BaseRecognizer.
| virtual IToken Antlr.Runtime.Lexer.NextToken | ( | ) | [virtual] |
Return a token from this source; i.e., Match a token on the char stream.
Implements Antlr.Runtime.ITokenSource.
| void Antlr.Runtime.Lexer.Skip | ( | ) |
Instruct the lexer to skip creating a token for current lexer rule and look for another token. NextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it.
| abstract void Antlr.Runtime.Lexer.mTokens | ( | ) | [pure virtual] |
This is the lexer entry point that sets instance var 'token'.
| virtual void Antlr.Runtime.Lexer.Emit | ( | IToken | token | ) | [virtual] |
| virtual IToken Antlr.Runtime.Lexer.Emit | ( | ) | [virtual] |
The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text.
Override this method to emit custom Token objects.
If you are building trees, then you should also override Parser or TreeParser.getMissingSymbol().
/remarks>
| virtual void Antlr.Runtime.Lexer.Match | ( | string | s | ) | [virtual] |
| virtual void Antlr.Runtime.Lexer.MatchRange | ( | int | a, | |
| int | b | |||
| ) | [virtual] |
| virtual void Antlr.Runtime.Lexer.Recover | ( | RecognitionException | re | ) | [virtual] |
Lexers can normally Match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a Fragment rule.
| override void Antlr.Runtime.Lexer.ReportError | ( | RecognitionException | e | ) | [virtual] |
Report a recognition problem.
This method sets errorRecovery to indicate the parser is recovering not parsing. Once in recovery mode, no errors are generated. To get out of recovery mode, the parser must successfully Match a token (after a resync). So it will go:
1. error occurs 2. enter recovery mode, report error 3. consume until token found in resynch set 4. try to resume parsing 5. next Match() will reset errorRecovery mode
If you override, make sure to update syntaxErrors if you care about that.
Reimplemented from Antlr.Runtime.BaseRecognizer.
| override string Antlr.Runtime.Lexer.GetErrorMessage | ( | RecognitionException | e, | |
| string[] | tokenNames | |||
| ) | [virtual] |
What error message should be generated for the various exception types?
Not very object-oriented code, but I like having all error message generation within one method rather than spread among all of the exception classes. This also makes it much easier for the exception handling because the exception classes do not have to have pointers back to this object to access utility routines and so on. Also, changing the message for an exception type would be difficult because you would have to subclassing exception, but then somehow get ANTLR to make those kinds of exception objects instead of the default.
This looks weird, but trust me--it makes the most sense in terms of flexibility.
For grammar debugging, you will want to override this to add more information such as the stack frame with GetRuleInvocationStack(e, this.GetType().Fullname) and, for no viable alts, the decision description and state etc...
Override this to change the message generated for one or more exception types.
Reimplemented from Antlr.Runtime.BaseRecognizer.
| virtual void Antlr.Runtime.Lexer.TraceIn | ( | string | ruleName, | |
| int | ruleIndex | |||
| ) | [virtual] |
| virtual void Antlr.Runtime.Lexer.TraceOut | ( | string | ruleName, | |
| int | ruleIndex | |||
| ) | [virtual] |
Member Data Documentation
const int Antlr.Runtime.Lexer.TOKEN_dot_EOF = (int)CharStreamConstants.EOF [private] |
internal ICharStream Antlr.Runtime.Lexer.input [protected] |
Property Documentation
virtual ICharStream Antlr.Runtime.Lexer.CharStream [get, set] |
override string Antlr.Runtime.Lexer.SourceName [get] |
For debugging and other purposes, might want the source name. Have ANTLR provide a hook for this property.
- Returns:
- The source name
Reimplemented from Antlr.Runtime.BaseRecognizer.
override IIntStream Antlr.Runtime.Lexer.Input [get] |
virtual int Antlr.Runtime.Lexer.CharIndex [get] |
virtual string Antlr.Runtime.Lexer.Text [get, set] |
The documentation for this class was generated from the following file:
- /Users/parrt/antlr/code/antlr/main/runtime/CSharp/Sources/Antlr3.Runtime/Antlr.Runtime/Lexer.cs
