org.antlr.runtime.TokenRewriteStream Class Reference

Inheritance diagram for org.antlr.runtime.TokenRewriteStream:

[legend]

Collaboration diagram for org.antlr.runtime.TokenRewriteStream:

[legend]

List of all members.

Public Member Functions
	TokenRewriteStream ()
	TokenRewriteStream (TokenSource tokenSource)
	TokenRewriteStream (TokenSource tokenSource, int channel)
void	rollback (int instructionIndex)
void	rollback (String programName, int instructionIndex)
void	deleteProgram ()
void	deleteProgram (String programName)
void	insertAfter (Token t, Object text)
void	insertAfter (int index, Object text)
void	insertAfter (String programName, Token t, Object text)
void	insertAfter (String programName, int index, Object text)
void	insertBefore (Token t, Object text)
void	insertBefore (int index, Object text)
void	insertBefore (String programName, Token t, Object text)
void	insertBefore (String programName, int index, Object text)
void	replace (int index, Object text)
void	replace (int from, int to, Object text)
void	replace (Token indexT, Object text)
void	replace (Token from, Token to, Object text)
void	replace (String programName, int from, int to, Object text)
void	replace (String programName, Token from, Token to, Object text)
void	delete (int index)
void	delete (int from, int to)
void	delete (Token indexT)
void	delete (Token from, Token to)
void	delete (String programName, int from, int to)
void	delete (String programName, Token from, Token to)
int	getLastRewriteTokenIndex ()
String	toOriginalString ()
String	toOriginalString (int start, int end)
String	toString ()
String	toString (String programName)
String	toString (int start, int end)
String	toString (String programName, int start, int end)
String	toDebugString ()
String	toDebugString (int start, int end)
Static Public Attributes
static final String	DEFAULT_PROGRAM_NAME = "default"
static final int	PROGRAM_INIT_SIZE = 100
static final int	MIN_TOKEN_INDEX = 0
Protected Member Functions
void	init ()
int	getLastRewriteTokenIndex (String programName)
void	setLastRewriteTokenIndex (String programName, int i)
List	getProgram (String name)
Map	reduceToSingleOperationPerIndex (List rewrites)
String	catOpText (Object a, Object b)
List	getKindOfOps (List rewrites, Class kind)
List	getKindOfOps (List rewrites, Class kind, int before)
Protected Attributes
Map	programs = null
Map	lastRewriteTokenIndexes = null
Private Member Functions
List	initializeProgram (String name)
Classes
class	InsertBeforeOp
class	ReplaceOp
class	RewriteOperation

---

Detailed Description

Useful for dumping out the input stream after doing some augmentation or other manipulations.

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule();

Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString());

Actually, you have to cast the 'input' to a TokenRewriteStream. :(

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer:

tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2"));

If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Definition at line 83 of file TokenRewriteStream.java.

---

Constructor & Destructor Documentation

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream

(

)

Definition at line 167 of file TokenRewriteStream.java.

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream

(

TokenSource

tokenSource

)

Definition at line 177 of file TokenRewriteStream.java.

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream	(	TokenSource	tokenSource,
		int	channel
	)

Definition at line 182 of file TokenRewriteStream.java.

---

Member Function Documentation

void org.antlr.runtime.TokenRewriteStream.init

(

)

[protected]

Definition at line 171 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.rollback

(

int

instructionIndex

)

Definition at line 187 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.rollback	(	String	programName,
		int	instructionIndex
	)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

Definition at line 195 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.deleteProgram

(

)

Definition at line 202 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.deleteProgram

(

String

programName

)

Reset the program so that no instructions exist

Definition at line 207 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	Token	t,
		Object	text
	)

Definition at line 211 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	int	index,
		Object	text
	)

Definition at line 215 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	String	programName,
		Token	t,
		Object	text
	)

Definition at line 219 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	String	programName,
		int	index,
		Object	text
	)

Definition at line 223 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	Token	t,
		Object	text
	)

Definition at line 228 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	int	index,
		Object	text
	)

Definition at line 232 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	String	programName,
		Token	t,
		Object	text
	)

Definition at line 236 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	String	programName,
		int	index,
		Object	text
	)

Definition at line 240 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	int	index,
		Object	text
	)

Definition at line 247 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	int	from,
		int	to,
		Object	text
	)

Definition at line 251 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	Token	indexT,
		Object	text
	)

Definition at line 255 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	Token	from,
		Token	to,
		Object	text
	)

Definition at line 259 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	String	programName,
		int	from,
		int	to,
		Object	text
	)

Definition at line 263 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	String	programName,
		Token	from,
		Token	to,
		Object	text
	)

Definition at line 273 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete

(

int

index

)

Definition at line 280 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	int	from,
		int	to
	)

Definition at line 284 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete

(

Token

indexT

)

Definition at line 288 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	Token	from,
		Token	to
	)

Definition at line 292 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	String	programName,
		int	from,
		int	to
	)

Definition at line 296 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	String	programName,
		Token	from,
		Token	to
	)

Definition at line 300 of file TokenRewriteStream.java.

int org.antlr.runtime.TokenRewriteStream.getLastRewriteTokenIndex

(

)

Definition at line 304 of file TokenRewriteStream.java.

int org.antlr.runtime.TokenRewriteStream.getLastRewriteTokenIndex

(

String

programName

)

[protected]

Definition at line 308 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.setLastRewriteTokenIndex	(	String	programName,
		int	i
	)			[protected]

Definition at line 316 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getProgram

(

String

name

)

[protected]

Definition at line 320 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.initializeProgram

(

String

name

)

[private]

Definition at line 328 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toOriginalString

(

)

Definition at line 334 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toOriginalString	(	int	start,
		int	end
	)

Definition at line 339 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString

(

)

Grab *all* tokens from stream and return string

Reimplemented from org.antlr.runtime.BufferedTokenStream.

Definition at line 347 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString

(

String

programName

)

Definition at line 352 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString	(	int	start,
		int	stop
	)

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Reimplemented from org.antlr.runtime.BufferedTokenStream.

Definition at line 357 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString	(	String	programName,
		int	start,
		int	end
	)

Definition at line 361 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.reduceToSingleOperationPerIndex

(

List

rewrites

)

[protected]

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases:

I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu

R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR

Delete special case of replace (text==null): D.i-j.u D.x-y.v | boundaries overlap combine to max(min)..max(right)

I.i.u R.x-y.v | i in (x+1)-y delete I (since insert before we're not deleting i) I.i.u R.x-y.v | i not in (x+1)-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping

I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u

First we need to examine replaces. For any replace op:

1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace.

Then we can deal with inserts:

1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace

Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index -> op map.

Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too.

Return a map from token index to operation.

Definition at line 456 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.catOpText	(	Object	a,
		Object	b
	)			[protected]

Definition at line 557 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getKindOfOps	(	List	rewrites,
		Class	kind
	)			[protected]

Definition at line 564 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getKindOfOps	(	List	rewrites,
		Class	kind,
		int	before
	)			[protected]

Get all operations before an index of a particular kind

Definition at line 569 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toDebugString

(

)

Definition at line 579 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toDebugString	(	int	start,
		int	end
	)

Definition at line 583 of file TokenRewriteStream.java.

---

Member Data Documentation

final String org.antlr.runtime.TokenRewriteStream.DEFAULT_PROGRAM_NAME = "default" [static]

Definition at line 84 of file TokenRewriteStream.java.

final int org.antlr.runtime.TokenRewriteStream.PROGRAM_INIT_SIZE = 100 [static]

Definition at line 85 of file TokenRewriteStream.java.

final int org.antlr.runtime.TokenRewriteStream.MIN_TOKEN_INDEX = 0 [static]

Definition at line 86 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.programs = null [protected]

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

Definition at line 162 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.lastRewriteTokenIndexes = null [protected]

Map String (program name) -> Integer index

Definition at line 165 of file TokenRewriteStream.java.

---

The documentation for this class was generated from the following file:

• /Users/parrt/antlr/code/antlr/main/runtime/Java/src/main/java/org/antlr/runtime/TokenRewriteStream.java