ANTLRWorks 2 uses a custom API for parsing due to a number of major issues which severely undermine the performance of the NetBeans Parsing API.
Evaluating the NetBeans Parsing API
- Cleanly integrated with the Lexer API which allows a shared lexer to be used for both the syntax highlighting and parsing features.
- Provides full-text indexing support backed by Apache Lucene.
- Uses a common representation for opened and unopened project files (
Snapshotobjects provide immutable views of either
- Encourages asynchronous parsing operations for parsing both opened documents (indexing and features like semantic highlighting) and unopened documents (project indexing).
- Cleanly and easily supports custom source embeddedings which can be used for anything from special parsing for documentation comments to mixed HTML/PHP source documents. This is supported by both the Lexer and Parsing APIs, so it works for syntax highlighting, indexing, and more.
- Only supports one
Parserfor a mime type. Different IDE operations require different types/amounts of information as input, and the restriction to a single
Parser.Resultrestricts the ability to meet the performance potential of "quick" tasks when "slow" tasks are also supported.
- Uses a scheduler with a severely restricted threading policy which is shared among foreground (code completion) and background (project indexing) tasks. It's also shared between languages, so the (very slow) project indexing process for Java source code prevents code completion within ANTLR grammars from working.
- Uses Lucene for built-in indexing support, which is brutally slow for standard IDE operations.
Due to the fundamental goal of having an IDE (like ANTLRWorks 2) operate at least as fast as the developer works (types), it is immediately clear that the task scheduler and indexing support provided by the Parsing API cannot be used. While it would be theoretically possible to extend the
Task class to declare data dependencies which are handled/dispatched within a special
Parser implementation, such a technique has already diverged from the original implementation in the Parsing API that little is gained from reusing those base classes/interfaces.
Requirements for the ANTLRWorks Parsing API
- The API should operate on the 1. Document Snapshot API to allow mapping between the parsed snapshot and the current version of the document.
Concurrency and task scheduling
- The API should support multithreaded parsing to take advantage of modern multi-core processors.
- The API should be able to distinguish between low- and high-priority tasks in addition to basic support for asynchronous operations. High-priority tasks might include operations like semantic highlighting and code completion, where low-priority tasks might include background indexing and/or compilation.
- The API should support adjustable (and potentially adaptive) reparse delays to offer "instant" UI updates for features like code completion and semantic highlighting without causing excessive overhead or irritating UI updates for tasks like background compilation and error reporting.
Declarative data dependencies
- The API should allow different parsing strategies to be used by different tasks/features to allow optimization of low-latency features like code completion without restricting the ability to provide higher-latency features like refactoring or Find Usages.
- Tasks should be able to request "stale" data rather than a reparse if results from a previous
DocumentSnapshotare available. Several low-latency features in ANTLRWorks 2 rely on dynamic anchors, which use a blend of stale and updated data to provide near-instantaneous responses even in extremely large source files.
The ANTLRWorks 2 Parsing API