Package org.antlr.v4.runtime

Class CommonTokenStream

java.lang.Object

org.antlr.v4.runtime.BufferedTokenStream

org.antlr.v4.runtime.CommonTokenStream

All Implemented Interfaces:

IntStream, TokenStream

public class CommonTokenStream
extends BufferedTokenStream

This class extends BufferedTokenStream with functionality to filter token streams to tokens on a particular channel (tokens where Token.getChannel() returns a particular value).

This token stream provides access to all tokens by index or when calling methods like BufferedTokenStream.getText(). The channel filtering is only used for code accessing tokens via the lookahead methods BufferedTokenStream.LA(int), LT(int), and LB(int).

By default, tokens are placed on the default channel (Token.DEFAULT_CHANNEL), but may be reassigned by using the ->channel(HIDDEN) lexer command, or by using an embedded action to call Lexer.setChannel(int).

Note: lexer rules which use the ->skip lexer command or call Lexer.skip() do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.

we

Field Summary

Fields

Modifier and Type	Field	Description
protected int	channel	Specifies the channel to use for filtering tokens.

Fields inherited from class org.antlr.v4.runtime.BufferedTokenStream

fetchedEOF, p, tokens, tokenSource

Fields inherited from interface org.antlr.v4.runtime.IntStream

EOF, UNKNOWN_SOURCE_NAME

Constructor Summary

Constructors

Constructor	Description
CommonTokenStream(TokenSource tokenSource)	Constructs a new CommonTokenStream using the specified token source and the default token channel (Token.DEFAULT_CHANNEL).
CommonTokenStream(TokenSource tokenSource, int channel)	Constructs a new CommonTokenStream using the specified token source and filtering tokens to the specified channel.

Method Summary

Modifier and Type	Method	Description
protected int	adjustSeekIndex(int i)	Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.
int	getNumberOfOnChannelTokens()	Count EOF just once.
protected Token	LB(int k)
Token	LT(int k)	Get the Token instance associated with the value returned by LA(k).

Methods inherited from class org.antlr.v4.runtime.BufferedTokenStream

consume, fetch, fill, filterForChannel, get, get, getHiddenTokensToLeft, getHiddenTokensToLeft, getHiddenTokensToRight, getHiddenTokensToRight, getSourceName, getText, getText, getText, getText, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, lazyInit, mark, nextTokenOnChannel, previousTokenOnChannel, release, reset, seek, setTokenSource, setup, size, sync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

channel

protected int channel

Specifies the channel to use for filtering tokens.

The default value is Token.DEFAULT_CHANNEL, which matches the default channel assigned to tokens created by the lexer.

Constructor Detail

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource)

Constructs a new CommonTokenStream using the specified token source and the default token channel (Token.DEFAULT_CHANNEL).

Parameters:

tokenSource - The token source.

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource,
                         int channel)

Constructs a new CommonTokenStream using the specified token source and filtering tokens to the specified channel. Only tokens whose Token.getChannel() matches channel or have the Token.getType() equal to Token.EOF will be returned by the token stream lookahead methods.

Parameters:

tokenSource - The token source.

channel - The channel to use for filtering tokens.

Method Detail

adjustSeekIndex

protected int adjustSeekIndex(int i)

Description copied from class: BufferedTokenStream

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.

For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

Overrides:

adjustSeekIndex in class BufferedTokenStream

Parameters:

i - The target token index.

Returns:

The adjusted target token index.

LB

protected Token LB(int k)

Overrides:

LB in class BufferedTokenStream

LT

public Token LT(int k)

Description copied from interface: TokenStream

Get the Token instance associated with the value returned by LA(k). This method has the same pre- and post-conditions as IntStream.LA(int). In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).

Specified by:

LT in interface TokenStream

Overrides:

LT in class BufferedTokenStream

See Also:

IntStream.LA(int)

getNumberOfOnChannelTokens

public int getNumberOfOnChannelTokens()

Count EOF just once.