How to build an ANTLR code generation target

Skip to end of metadata
Comment: Migrated to Confluence 4.0
Go to start of metadata

The following instructions show how to go about starting a new back-end for a new language "XYZ" using the Java back-end as a basis.

All the functionality contained in the Java files in the antlr-3.1/runtime/Java/src/org/antlr/runtime directory is needed to create a full back-end. You must also copy from the code generation templates of another target, such as antlr-3.1/src/org/antlr/codegen/templates/Java/*.stg. Finally create antlr-3.1/src/org/antlr/codegen/XYZTarget.java if you need to override anything in Target.java, which you probably will.

But it is in fact pretty easy to get started:

  • In src/org/antlr/codegen/templates/
    • create a directory XYZ
    • copy Java/Java.stg to XYZ/XYZ.stg
  • I recommend building the ANTLR tool 'in place'. Do not create a jar or compile/copy to a build directory. When you run it with 'java -cp path-to-src-dir ...' it will use the original *.stg file, which you'll edit a lot - so rebuilding the tool would be quite a PITA.
  • Create a directory antlr-3.1/runtime/XYZ. Here you can put anything you need (no need to clone Java 1:1).
  • Start with a simple lexer like:

N.B: The name after the word "grammar" needs to be the same as the filename in which you save it, i.e. T.g.

  • Look at the generated code and try to figure out which templates in XYZ.stg you have to port to get valid XYZ code. What I did, is to comment out the Java code in all templates replacing it with something like FIXME([number]). Then you fix the templates until no FIXME remains in the
    output.
  • You'll need a basic implementation of a character stream and base recognizer/lexer to get the example running. Just implement the methods that are actually needed to get the example running w/o errors.
  • You'll either get the feeling "Wow, that was easy!" and move on (that happened to me) or "Eeek, what a pain!" and let someone else to the work.

Target Source Files

Your ANTLR code generation target will consist of several files in the target language. A number of run-time support files that you'll create directly, and a number of files that ANTLR will generate (in the target language) as the result of the user's grammar being processed by ANTLR. These files are generated from StringTemplates like ZYX.stg above.

ANTLR looks for the presence of certain template files and templates. Some are required, some are optional.

XYZ.stg

The principle template file. The templates in this file are used to generate the Lexer, Parser and Tree Parser generated by the user's grammar.

Template Name

Purpose

Notes

outputFile

Generates the target-language implementation of the recognizer.

Required

headerFile

Generates the target-language header file for the recognizer.

Optional

block() StringTemplate

Parameter Name

Description

alts

 

decls

 

decision

 

enclosingBlockLevel

 

blockLevel

 

decisionNumber

 

maxK

 

maxAlt

 

description

 
closureBlock() StringTemplate

Parameter Name

Description

alts

 

decls

 

decision

 

enclosingBlockLevel

 

blockLevel

 

decisionNumber

 

maxK

 

maxAlt

 

description

 
outputFile() StringTemplate

Formal parameters:

Parameter Name

Description

LEXER

Boolean indicating that a Lexer is being generated.

PARSER

Boolean indicating that a Parser or Combined Lexer/Parser is being generated.

TREE_PARSER

Boolean indicating that a Tree Parser is being generated.

actionScope

 

actions

A java.util.Map of the grammar's actions.

docComment

 

recognizer

The StringTemplate named "lexer", "parser", or "treeParser", depending on the type of recognizer being generated.

name

 

tokens

 

tokenNames

 

rules

 

cyclicDFAs

A org.antlr.analysis.DFA instance.

bitsets

 

buildTemplate

Boolean

buildAST

Boolean

rewriteMode

Boolean

profile

Boolean

backtracking

Boolean

synpreds

A java.util.Set of synpreds in the grammar (if any).

memoize

Boolean

numRules

 

fileName

 

ANTLRVersion

String containing the version of the ANTLR tool generating this recognizer.

generatedTimestamp

String containing the current time.

trace

Boolean

scopes

 

superClass

 

literals

 
rule() StringTemplate

The rule() StringTemplate is instantiated by ANTLR's own grammar processing and added to the "rules" attribute. It takes the following parameters.

Parameter Name

Description

ruleName

The name of the rule as specified in the input grammar.

ruleDescriptor

The org.antlr.tool.Rule object instance associated with this StringTemplate.

block

 

emptyRule

 

description

 

exceptions

 

finally

 

memoize

 
dfaState() StringTemplate

Parameter Name

Description

k

 

edges

 

eotPredictsAlt

 

description

 

stateNumber

 

semPredState

 
dfaLoopbackState() StringTemplate

A DFA state that is actually the loopback decision of a closure loop. If end-of-token (EOT) predicts any of the targets then it should act like a default clause (i.e., no error can be generated). This is used only in the lexer so that for ('a')* on the end of a rule anything other than 'a' predicts exiting.

Parameter Name

Description

k

 

edges

 

eotPredictsAlt

 

description

 

stateNumber

 

semPredState

 

AST.stg

ASTParser.stg

ST.stg

Labels:
None
Edit Labels

4 Comments

Hide/Show Comments

  1. Apr 21, 2012

    yac

    Hello

    When you say in src/org/... do you mean I extract the antlr-3-3 im the src of the java project that I am working on ?? Beacause I have already used the jar of antlr3-3 as a referencd lbrary

     

    thanks

  2. Jun 18, 2012

    NemoStein

    Parr, this document doesn't reflect the repository structure anymore, and the instructions here are a bit outdated, I think.

    I'm willing to update the ActionScript target (or even start a new one, from scratch), but I really don't know from where to start.

    Any help would be appreciated, thanks.

  3. Jun 18, 2012

    Terence Parr

    Unfortunately, there never really has been good documentation on building a target and I have no time to devote to this topic :( I have all sorts of secret projects going on not to mention the rewrite of ANTLR -> v4.

    sorry!
    Ter 

  4. Jun 20, 2012

    Uwe Gebhardt

    Hi,

     

    I am in the process of creating a new target for Antlr3,

     

    And I am basically following the advice given on this side, start with the sample grammer, get the lexer running,

    get familar with the template.

    It helps to build the target in place and to use ANTLWorks to edit the template.

    After you modified the template, execute antlr, look at the output, repeat, GVIM will notify you, that th outut canged,

    and ask you, if it should reload the file.

    A good starting point is a language, that is similar to the one, you are planning to create a target.

    The language I am currently working, on does the declaration of classes separatly from the method  definition, i.e.

    you have some kind of forward declaration, which is different to Java / C++, but there exists an example, C.

    I also bought the two books, but mainly for getting an introdution into language application / practical complier theory.

    If you are really serious about ANTLR, you need documentation, I dont know, how good the online documenation is.

    With kind regards

    Uwe Gebhardt

    PS: The main work is getting the runtime library ported to your language.

    I did not really search the side, to see, if there is a good documenation on how the classes of the runtime interact.

    It may be possible to generat one for yourself, if you are a Java guy, which I am not.

    My Java knowhow is only "Hello World", but my C/C++ background helps to understand the Java, so that I know,

    what the Java is supposed to do, so that I can port it.