[stringtemplate-interest] PyST: first() and rest() don’t work on iterable non-lists

[Benjamin Niemann]

Hi Roman,

On Mon, Sep 8, 2008 at 11:03 AM, Roman Odaisky <roma at qwertty.com> wrote:
> Hi,
>
> Until someone points me to a PyST bugtracker, I'll spam this list ;-)

http://www.antlr.org:8888/browse/ST

> In PyST 3.1, $first(something)$ and $rest(something)$ only work on built-in
> types. When I pass an iterable object, the functions seem to treat it as a
> whole. However, the template application operator does recognize them, so the
> result looks like this:
>
>>>> st = StringTemplate(r'''$first(list):{<$it$>}$; $rest(list):{($it$)}$''')
>>>> st["list"] = xrange(0, 5)
>>>> print unicode(st)
> <0><1><2><3><4>;
>
> while I was expecting:
>
> <0>; (1)(2)(3)(4)
>
> Proper handling of iterable objects is very important.

Yes, the way it works now is very unpythonic. The original Java
implementation uses a lot a class checking to decide how to handle
objects and the current Python implementation mostly mimics this
behavior. This doesn't work properly in Python where the various
iterables don't share a common base class, so you're constrained to
use list()s to get the desired semantics.

> For example, in my case
> the objects are database result sets, and fetching them in bite-size portions
> is much more efficient than getting all the data into a Python list and
> passing it to ST, as I'm forced to do.
>
> Perhaps it's possible to return a generator from language.ASTExpr.rest?

There was an attempt to use generators to make things more efficient,
but it turned out to open a set of bugs (i.e. if you expand an
iterator twice, the second invocation doesn't yield a result, because
the iterator is already exhausted by the first invocation). Not sure
if this can be fixed in a sane way.

-Ben