Class UnbufferedCharStream

  • All Implemented Interfaces:
    CharStream, IntStream

    public class UnbufferedCharStream
    extends Object
    implements CharStream
    Do not buffer up the entire char stream. It does keep a small buffer for efficiency and also buffers while a mark exists (set by the lookahead prediction in parser). "Unbuffered" here refers to fact that it doesn't buffer all data, not that's it's on demand loading of char. Before 4.7, this class used the default environment encoding to convert bytes to UTF-16, and held the UTF-16 bytes in the buffer as chars. As of 4.7, the class uses UTF-8 by default, and the buffer holds Unicode code points in the buffer as ints.
    • Field Detail

      • data

        protected int[] data
        A moving window buffer of the data being scanned. While there's a marker, we keep adding to buffer. Otherwise, consume() resets so we start filling at index 0 again.
      • n

        protected int n
        The number of characters currently in data.

        This is not the buffer capacity, that's data.length.

      • p

        protected int p
        0..n-1 index into data of next character.

        The LA(1) character is data[p]. If p == n, we are out of buffered characters.

      • numMarkers

        protected int numMarkers
        Count up with mark() and down with release(). When we release() the last mark, numMarkers reaches 0 and we reset the buffer. Copy data[p][n-1] to data[0][(n-1)-p].
      • lastChar

        protected int lastChar
        This is the LA(-1) character for the current position.
      • lastCharBufferStart

        protected int lastCharBufferStart
        When numMarkers > 0, this is the LA(-1) character for the first character in data. Otherwise, this is unspecified.
      • currentCharIndex

        protected int currentCharIndex
        Absolute character index. It's the index of the character about to be read via LA(1). Goes from 0 to the number of characters in the entire stream, although the stream size is unknown before the end is reached.
      • input

        protected Reader input
      • name

        public String name
        The name or source of this char stream.
    • Constructor Detail

      • UnbufferedCharStream

        public UnbufferedCharStream()
        Useful for subclasses that pull char from other than this.input.
      • UnbufferedCharStream

        public UnbufferedCharStream​(int bufferSize)
        Useful for subclasses that pull char from other than this.input.
      • UnbufferedCharStream

        public UnbufferedCharStream​(InputStream input)
      • UnbufferedCharStream

        public UnbufferedCharStream​(Reader input)
      • UnbufferedCharStream

        public UnbufferedCharStream​(InputStream input,
                                    int bufferSize)
      • UnbufferedCharStream

        public UnbufferedCharStream​(InputStream input,
                                    int bufferSize,
                                    Charset charset)
      • UnbufferedCharStream

        public UnbufferedCharStream​(Reader input,
                                    int bufferSize)
    • Method Detail

      • consume

        public void consume()
        Description copied from interface: IntStream
        Consumes the current symbol in the stream. This method has the following effects:
        • Forward movement: The value of index() before calling this method is less than the value of index() after calling this method.
        • Ordered lookahead: The value of LA(1) before calling this method becomes the value of LA(-1) after calling this method.
        Note that calling this method does not guarantee that index() is incremented by exactly 1, as that would preclude the ability to implement filtering streams (e.g. CommonTokenStream which distinguishes between "on-channel" and "off-channel" tokens).
        Specified by:
        consume in interface IntStream
      • sync

        protected void sync​(int want)
        Make sure we have 'need' elements from current position p. Last valid p index is data.length-1. p+need-1 is the char index 'need' elements ahead. If we need 1 element, (p+1-1)==p must be less than data.length.
      • fill

        protected int fill​(int n)
        Add n characters to the buffer. Returns the number of characters actually added to the buffer. If the return value is less than n, then EOF was reached before n characters could be added.
      • nextChar

        protected int nextChar()
                        throws IOException
        Override to provide different source of characters than input.
      • add

        protected void add​(int c)
      • LA

        public int LA​(int i)
        Description copied from interface: IntStream
        Gets the value of the symbol at offset i from the current position. When i==1, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When i==-1, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with i==0, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

        This method is guaranteed to succeed if any of the following are true:

        • i>0
        • i==-1 and index() returns a value greater than the value of index() after the stream was constructed and LA(1) was called in that order. Specifying the current index() relative to the index after the stream was created allows for filtering implementations that do not return every symbol from the underlying source. Specifying the call to LA(1) allows for lazily initialized streams.
        • LA(i) refers to a symbol consumed within a marked region that has not yet been released.

        If i represents a position at or beyond the end of the stream, this method returns IntStream.EOF.

        The return value is unspecified if i<0 and fewer than -i calls to consume() have occurred from the beginning of the stream before calling this method.

        Specified by:
        LA in interface IntStream
      • mark

        public int mark()
        Return a marker that we can release later.

        The specific marker value used for this class allows for some level of protection against misuse where seek() is called on a mark or release() is called in the wrong order.

        Specified by:
        mark in interface IntStream
        An opaque marker which should be passed to release() when the marked range is no longer required.
      • release

        public void release​(int marker)
        Decrement number of markers, resetting buffer if we hit 0.
        Specified by:
        release in interface IntStream
        marker -
        See Also:
      • index

        public int index()
        Description copied from interface: IntStream
        Return the index into the stream of the input symbol referred to by LA(1).

        The behavior of this method is unspecified if no call to an initializing method has occurred after this stream was constructed.

        Specified by:
        index in interface IntStream
      • seek

        public void seek​(int index)
        Seek to absolute character index, which might not be in the current sliding window. Move p to index-bufferStartIndex.
        Specified by:
        seek in interface IntStream
        index - The absolute index to seek to.
      • size

        public int size()
        Description copied from interface: IntStream
        Returns the total number of symbols in the stream, including a single EOF symbol.
        Specified by:
        size in interface IntStream
      • getText

        public String getText​(Interval interval)
        Description copied from interface: CharStream
        This method returns the text for a range of characters within this input stream. This method is guaranteed to not throw an exception if the specified interval lies entirely within a marked range. For more information about marked ranges, see IntStream.mark().
        Specified by:
        getText in interface CharStream
        interval - an interval within the stream
        the text of the specified interval
      • getBufferStartIndex

        protected final int getBufferStartIndex()