OpenEuphoria

Re: Constructing sequences

• Posted by Gordon Webster <gwalias-bb at yahoo.com> Jun 27, 2005
• 581 views

Bernie Ryan wrote:
> 
> Gordon Webster wrote:
> > 
> > Dear Euphorians
> > 
> > Overall, I'm very happy with the performance of Euphoria (both 
> > interpreted and compiled from translated C), but I've noticed
> > that constructing sequences seems to be pretty slow by comparison
> > with accessing them once they are constructed.
> > 
> > When I add 100,000 smaller sequences (my data "records") to a storage
> > sequence, it takes me nearly 40 seconds either interpreted or
> > compiled, to complete. Once this 100,000 record sequence is built
> > however, I can access the various "records" and/or their "fields"
> > at blazing speeds. I get no speed up on the compiled version for
> > the construction of the sequence, but I do get some speedup for the
> > already very rapid access to the built sequence.
> > 
> > I have even tried making the sequence to be constructed global, and
> > using a procedure to append to it, rather than a function which has
> > to pass a copy of the sequence. It made no discernible difference.
> > 
> > I have read the performance tips about using the optimized form
> > 'append' etc. etc., but if any of you out there have any experience
> > of improving the speed of creation of large sequences, I'd be very
> > grateful for any tips or pointers (no pun intended).
> > 
> > Best
>  
> Gordon:
>  
> I have found that thing slow way down when using many appends
> is version 2.5 Eu. If you can define the complete large sequence
> all at once then that will be faster without appends.
> 
> Have you thought of using a database to store the information
> your keeping in the small sequences instead of adding them to
> a large sequence. 
> 
> Bernie
> 
> My files in archive:
> w32engin.ew mixedlib.e eu_engin.e win32eru.exw
> 
> Can be downloaded here:
> <a
> href="http://www.rapideuphoria.com/cgi-bin/asearch.exu?dos=on&win=on&lnx=on&gen=on&keywords=bernie+ryan">http://www.rapideuphoria.com/cgi-bin/asearch.exu?dos=on&win=on&lnx=on&gen=on&keywords=bernie+ryan</a>
> 

Thanks for the reply Bernie

I am reading the data records in from a file so I don't see an
obvious way I could build the whole sequence at once. Wouldn't
I have to temporariliy store the records in another sequence,
incurring the same 'append' overhead anyway?

The other aspect of this is that parts of the data records are
going to be used for heavy numerical computation and I need to
be able to construct numerical vectors and matrices really fast,
using certain fields of my data records. I didn't consider using
a database for this, since my data records are very simple and
well defined and I was worried that accessing fields from a db
would be a lot slower than accessing atoms from a sequence. I'm
really shooting for "lean and fast" here since I may have as many
as a million records to process and I want to keep the memory
overhead down as well.

Perhaps I should be thinking of using a db. Would it bloat my
memory overhead much, or slow things down compared to using
raw sequences?

It seems that once I've built my raw sequence, I can access it very
very rapidly indeed, but if I don't use a db, is it possible to
speed up the construction of sequences in memory?

Best

Gordon