org.antlr.v4.runtime

Class CommonToken

java.lang.Object

org.antlr.v4.runtime.CommonToken

All Implemented Interfaces:

java.io.Serializable, Token, WritableToken

Direct Known Subclasses:

TokenTagToken

public class CommonToken
extends java.lang.Object
implements WritableToken, java.io.Serializable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	channel This is the backing field for getChannel() and setChannel(int).
protected int	charPositionInLine This is the backing field for getCharPositionInLine() and setCharPositionInLine(int).
protected static Pair<TokenSource,CharStream>	EMPTY_SOURCE An empty Pair which is used as the default value of source for tokens that do not have a source.
protected int	index This is the backing field for getTokenIndex() and setTokenIndex(int).
protected int	line This is the backing field for getLine() and setLine(int).
protected Pair<TokenSource,CharStream>	source This is the backing field for getTokenSource() and getInputStream().
protected int	start This is the backing field for getStartIndex() and setStartIndex(int).
protected int	stop This is the backing field for getStopIndex() and setStopIndex(int).
protected java.lang.String	text This is the backing field for getText() when the token text is explicitly set in the constructor or via setText(java.lang.String).
protected int	type This is the backing field for getType() and setType(int).

Fields inherited from interface org.antlr.v4.runtime.Token

DEFAULT_CHANNEL, EOF, EPSILON, HIDDEN_CHANNEL, INVALID_TYPE, MIN_USER_CHANNEL_VALUE, MIN_USER_TOKEN_TYPE

Constructor Summary

Constructors

Constructor and Description
CommonToken(int type) Constructs a new CommonToken with the specified token type.
CommonToken(int type, java.lang.String text) Constructs a new CommonToken with the specified token type and text.
CommonToken(Pair<TokenSource,CharStream> source, int type, int channel, int start, int stop)
CommonToken(Token oldToken) Constructs a new CommonToken as a copy of another Token.

Method Summary

Methods

Modifier and Type	Method and Description
int	getChannel() Return the channel this token.
int	getCharPositionInLine() The index of the first character of this token relative to the beginning of the line at which it occurs, 0..n-1
CharStream	getInputStream() Gets the CharStream from which this token was derived.
int	getLine() The line number on which the 1st character of this token was matched, line=1..n
int	getStartIndex() The starting character index of the token This method is optional; return -1 if not implemented.
int	getStopIndex() The last character index of the token.
java.lang.String	getText() Get the text of the token.
int	getTokenIndex() An index from 0..n-1 of the token object in the input stream.
TokenSource	getTokenSource() Gets the TokenSource which created this token.
int	getType() Get the token type of the token
void	setChannel(int channel)
void	setCharPositionInLine(int charPositionInLine)
void	setLine(int line)
void	setStartIndex(int start)
void	setStopIndex(int stop)
void	setText(java.lang.String text) Explicitly set the text for this token.
void	setTokenIndex(int index)
void	setType(int type)
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

EMPTY_SOURCE

protected static final Pair<TokenSource,CharStream> EMPTY_SOURCE

An empty Pair which is used as the default value of source for tokens that do not have a source.

type

protected int type

This is the backing field for getType() and setType(int).

line

protected int line

This is the backing field for getLine() and setLine(int).

charPositionInLine

protected int charPositionInLine

This is the backing field for getCharPositionInLine() and setCharPositionInLine(int).

channel

protected int channel

This is the backing field for getChannel() and setChannel(int).

source

protected Pair<TokenSource,CharStream> source

This is the backing field for getTokenSource() and getInputStream().

These properties share a field to reduce the memory footprint of CommonToken. Tokens created by a CommonTokenFactory from the same source and input stream share a reference to the same Pair containing these values.

text

protected java.lang.String text

This is the backing field for getText() when the token text is explicitly set in the constructor or via setText(java.lang.String).

See Also:

getText()

index

protected int index

This is the backing field for getTokenIndex() and setTokenIndex(int).

start

protected int start

This is the backing field for getStartIndex() and setStartIndex(int).

stop

protected int stop

This is the backing field for getStopIndex() and setStopIndex(int).

Constructor Detail

CommonToken

public CommonToken(int type)

Constructs a new CommonToken with the specified token type.

Parameters:

type - The token type.

CommonToken

public CommonToken(Pair<TokenSource,CharStream> source,
           int type,
           int channel,
           int start,
           int stop)

CommonToken

public CommonToken(int type,
           java.lang.String text)

Constructs a new CommonToken with the specified token type and text.

Parameters:

type - The token type.

text - The text of the token.

CommonToken

public CommonToken(Token oldToken)

Constructs a new CommonToken as a copy of another Token.

If oldToken is also a CommonToken instance, the newly constructed token will share a reference to the text field and the Pair stored in source. Otherwise, text will be assigned the result of calling getText(), and source will be constructed from the result of Token.getTokenSource() and Token.getInputStream().

Parameters:

oldToken - The token to copy.

Method Detail

getType

public int getType()

Description copied from interface: Token

Get the token type of the token

Specified by:

getType in interface Token

setLine

public void setLine(int line)

Specified by:

setLine in interface WritableToken

getText

public java.lang.String getText()

Description copied from interface: Token

Get the text of the token.

Specified by:

getText in interface Token

setText

public void setText(java.lang.String text)

Explicitly set the text for this token. If {code text} is not null, then getText() will return this value rather than extracting the text from the input.

Specified by:

setText in interface WritableToken

Parameters:

text - The explicit text of the token, or null if the text should be obtained from the input along with the start and stop indexes of the token.

getLine

public int getLine()

Description copied from interface: Token

The line number on which the 1st character of this token was matched, line=1..n

Specified by:

getLine in interface Token

getCharPositionInLine

public int getCharPositionInLine()

Description copied from interface: Token

The index of the first character of this token relative to the beginning of the line at which it occurs, 0..n-1

Specified by:

getCharPositionInLine in interface Token

setCharPositionInLine

public void setCharPositionInLine(int charPositionInLine)

Specified by:

setCharPositionInLine in interface WritableToken

getChannel

public int getChannel()

Description copied from interface: Token

Return the channel this token. Each token can arrive at the parser on a different channel, but the parser only "tunes" to a single channel. The parser ignores everything not on DEFAULT_CHANNEL.

Specified by:

getChannel in interface Token

setChannel

public void setChannel(int channel)

Specified by:

setChannel in interface WritableToken

setType

public void setType(int type)

Specified by:

setType in interface WritableToken

getStartIndex

public int getStartIndex()

Description copied from interface: Token

The starting character index of the token This method is optional; return -1 if not implemented.

Specified by:

getStartIndex in interface Token

setStartIndex

public void setStartIndex(int start)

getStopIndex

public int getStopIndex()

Description copied from interface: Token

The last character index of the token. This method is optional; return -1 if not implemented.

Specified by:

getStopIndex in interface Token

setStopIndex

public void setStopIndex(int stop)

getTokenIndex

public int getTokenIndex()

Description copied from interface: Token

An index from 0..n-1 of the token object in the input stream. This must be valid in order to print token streams and use TokenRewriteStream. Return -1 to indicate that this token was conjured up since it doesn't have a valid index.

Specified by:

getTokenIndex in interface Token

setTokenIndex

public void setTokenIndex(int index)

Specified by:

setTokenIndex in interface WritableToken

getTokenSource

public TokenSource getTokenSource()

Description copied from interface: Token

Gets the TokenSource which created this token.

Specified by:

getTokenSource in interface Token

getInputStream

public CharStream getInputStream()

Description copied from interface: Token

Gets the CharStream from which this token was derived.

Specified by:

getInputStream in interface Token

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object