Token attributes

Rule attributes

Parsers

Tree parsers

Lexers

The Rule text Attribute in Tree Grammars

In a parser grammar, the relationship between the elements matched by a rule and the associated input text is very clear. A rule begins parsing at a particular token and stops parsing at a particular token. The text attribute for a rule, $text, is simply the concatenated text from all tokens in that range, including hidden channel tokens. What does $text mean in a tree grammar, though?

Tree grammar rules match nodes and trees not tokens. Fortunately, each node has an associated token start and stop index (See TreeAdaptor). As the parser builds trees, each rule sets the token indexes for its return AST to the start and stop token of that rule. We can then define the text attribute for a tree grammar rule to be the text concatenated from the range of tokens indicated by the range in the root of the first tree matched by the rule. This rule may seem strange, but is the most efficient implementation and works in almost all situations. Here are a few examples:

/** match tree created from, e.g., "int x;"
 *  $text would, therefore, be "int x;"
 *  $start node is VAR node.
 */
variable
    :   ^(VAR type ID) // $text derived from indexes in VAR node
    ;
/** match tree node created from, e.g., "int"
 *  $text would, therefore, be "int"
 *  $start node is 'int' node.
 */
type:   'int'          // $text derived from indexes in 'int' node
    |   'void'
    ;

The following code embodies the text attribute definition. The token range from a rule's start node defines the range of text for the entire rule.

// input is a TreeNodeStream implementation
int start = input.getTreeAdaptor().getTokenStartIndex($start);
int stop = input.getTreeAdaptor().getTokenStopIndex($start);
String text = input.getTokenStream().toString(start, stop);

Be careful when referencing the text of a rule that happens to be the root of a tree. The text of a rule is the text of all tokens underneath the first rout matched by the rule. In the following example, rule @r op matches a single node, but $op.text will include the text associated with the two operands as well. The parser that build the plus and multiply operator nodes will set the token range to include all tokens for that expression.

/** match subtrees for + and * created from input such as "1+4*2"
 *  $text and $op.text is "1+4*2" for first alternative.
 *  $text is just the INT node for second alternative.
 */
expr:   ^(op expr expr) ; // $op.text is same as $text!
    |   INT
    ;
op  :   o='+' | o='*'     // $text includes text of operands
    ;                     // $o.text is just node's text

Note that the text for a node label is always just the string returned from getText() invoked on that node whereas the text for a rule reference is always the text for the tree rooted at that labeled node.

Finally, here is the case where the definition of the text attribute does not do what you expect. The text attribute is derived from the first node matched by a rule, but a rule such as rule slist that matches multiple subtrees has an ill-defined text attribute because it only gives you the text for the first statement subtree:

func:   'void' ID '()' slist ; // $slist.text is text from first tree only
slist:  stat+ ;

In general, you just need to keep this in mind--the text attribute is natural in most cases.

Rule scopes

Global shared scopes