Interface Token

  • All Known Subinterfaces:
    WritableToken
    All Known Implementing Classes:
    CommonToken, RuleTagToken, TokenTagToken

    public interface Token
    A token has properties: text, type, line, character position in the line (so we can ignore tabs), token channel, index, and source from which we obtained this token.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int DEFAULT_CHANNEL
      All tokens go to the parser (unless skip() is called in that rule) on a particular "channel".
      static int EOF  
      static int EPSILON
      During lookahead operations, this "token" signifies we hit rule end ATN state and did not follow it despite needing to.
      static int HIDDEN_CHANNEL
      Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.
      static int INVALID_TYPE  
      static int MIN_USER_CHANNEL_VALUE
      This is the minimum constant value which can be assigned to a user-defined token channel.
      static int MIN_USER_TOKEN_TYPE  
    • Field Detail

      • EPSILON

        static final int EPSILON
        During lookahead operations, this "token" signifies we hit rule end ATN state and did not follow it despite needing to.
        See Also:
        Constant Field Values
      • DEFAULT_CHANNEL

        static final int DEFAULT_CHANNEL
        All tokens go to the parser (unless skip() is called in that rule) on a particular "channel". The parser tunes to a particular channel so that whitespace etc... can go to the parser on a "hidden" channel.
        See Also:
        Constant Field Values
      • HIDDEN_CHANNEL

        static final int HIDDEN_CHANNEL
        Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.
        See Also:
        Constant Field Values
    • Method Detail

      • getText

        String getText()
        Get the text of the token.
      • getType

        int getType()
        Get the token type of the token
      • getLine

        int getLine()
        The line number on which the 1st character of this token was matched, line=1..n
      • getCharPositionInLine

        int getCharPositionInLine()
        The index of the first character of this token relative to the beginning of the line at which it occurs, 0..n-1
      • getChannel

        int getChannel()
        Return the channel this token. Each token can arrive at the parser on a different channel, but the parser only "tunes" to a single channel. The parser ignores everything not on DEFAULT_CHANNEL.
      • getTokenIndex

        int getTokenIndex()
        An index from 0..n-1 of the token object in the input stream. This must be valid in order to print token streams and use TokenRewriteStream. Return -1 to indicate that this token was conjured up since it doesn't have a valid index.
      • getStartIndex

        int getStartIndex()
        The starting character index of the token This method is optional; return -1 if not implemented.
      • getStopIndex

        int getStopIndex()
        The last character index of the token. This method is optional; return -1 if not implemented.
      • getInputStream

        CharStream getInputStream()
        Gets the CharStream from which this token was derived.