I replaced my blocking queue sitting between pipeline actors (threaded consumer/producers) and speed of my word freq program dropped from 5.2s to 3.5s on 5M file simply by replacing the blocking queue with my new ping-pong buffered version. That is close to the 3.0s achieved by the nonthreaded version.
// threaded, pipeline version; 3.5s (from 5.2s) on 5M file
f => Words() => { string w | ... }
// nonthreaded, nested map operation on big list; 3.0s
f.lines() => a;
a:{ string line | line.split():{ string w | ... }};
Buffer size is 1000. When I drop to 100, slows down by .2s. When I increase to 4000, no change. 40000, slows down by .5s. Size 400 seems to be slightly faster. Default queue size set to 400 now.
Thanks to Sriram Srinivasan for the idea of ping-ponging the buffers.
Comments (2)
Oct 18, 2007
Jonathan Thomas says:
Ter, No offence mate, but I'm wondering why we're getting so many Mantra po...Ter,
No offence mate, but I'm wondering why we're getting so many Mantra postings on the ANTLR wiki? When it directly relates to ANTLR no problem, but otherwise it seems to be getting off-topic ... ?
Cheers,
Jonathan.
Oct 18, 2007
Kay Röpke says:
Jonathan, That's just because Mantra is a big (probably the biggest) open sourc...Jonathan,
That's just because Mantra is a big (probably the biggest) open source project that uses v3 at the moment. And besides, the postings are in the blog section of Ter, AFAICS.
cheers,
-k