November 29, 2010
ANTLR project lead and supreme dictator for life
University of San Francisco
ANTLR v3.3 is primarily a bug fix release but has 2 new targets.
Alan Condit has built an Objective-C target for 3.3 following on Kay Roepke's work.
Important things to note
- Removed conversion timeout failsafe; no longer needed really. If the unlikely occurs and ANTLR spins forever, kill it and run again with -Xwatchconversion. You'll see things like: If it gets stuck, look at the decision (line 109 here) and add k=1 and backtracking or something.
- Stats/profiling updated to be correct for -report and -profile but the output is very different. -report: And -profile:
- Doesn't write profile data to file anymore; emits decision data to stderr
- There are Java generics in ANTLR itself and runtime; shouldn't be a problem because we generate Java 1.4 compatible binaries.
- The DebugEventListener interface changed [BREAKS BACKWARD COMPATIBILITY]
- Added source name to syntax error msgs
- added new method to get subset of tokens to buffered token streams:
public List get(int start, int stop);
- Refs to other tokens in a lexer rule didn't get its line/charpos right. altered Java.stg.
- Instead of sharing Token.EOF_TOKEN, I'm now creating EOF tokens so I can set the char position for better error messages.
- added new buffered on-demand streams: BufferedTokenStream. Renamed CommonTokenStream to LegacyCommonTokenStream and made new one as subclass of BufferedTokenStream.
- Added org.antlr.runtime.UnbufferedTokenStream. Was trivial and works!
- added range to TokenStream and implementors:
- Added MachineProbe class to make it easier to highlight ambig paths in
grammar. More accurate than DecisionProbe; retrofitted from v4.
- lexerStringRef was missing elementIndex attribute. i='import' didn't work
in lexer. Altered all target stg files. Set in codegen.g
- Added -Xsavelexer option
- greedy=true option shuts off nondeterminism warning.
- Tried to make output more deterministic:
- added toArray in OrderedHashSet to make addAll calls get same order for DFA edges and possibly code gen in some areas.
- Made OrderedHashSet have deterministic iteration
Changes / Fixes
- code gen for AST and -profile didn't compile. had useless line:
- Missing -trace in help msg
- Added boolean decisionCanBacktrack to Parser and enterDecision in dbg interface. Breaks AW
interface and other tools! [BREAKS BACKWARD COMPATIBILITY]
- output=AST, rewrite=true for tree rewriters broken. nextNode for subtree
streams didn't dup node, it gave whole tree back.
- Creating token from another token didn't copy input stream in CommonToken. makes sense to copy too; i don't think anybody relies on it being null after a copy. We might want to know where token came from.
- TreeParser.getMissingSymbol() used CommonTree instead of using adaptor.create()
- Fixed bug in TreeVisitor when rewrites altered number of children. Thanks to Chris DiGiano.
- Couldn't properly reuse parser state; ctor reset the state; fixed.
Parser(TokenStream input, RecognizerSharedState state)
- LookaheadStream<T> used some hardcoded Object return types for LT, etc...
uses T now.