OpenEuphoria

Re: Slow memory allocation

• Posted by Mario Steele <eumario at trilake.net> Jun 11, 2005
• 567 views

Al Getz wrote:
> 
> Haflidi Asgrimsson wrote:
> > 
> > Suprisingly my old Python code was faster than Euphoria. I was reading a
> > file of 40.000
> > lines like this:
> > a10_Clo10_BL26_ClFish_O_C251A_0_0_0_2003X04	Clown Fish 17
> > mark	120	B	405	425	404.83
> > 425.86	71.0	51.0	false	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	1.0
> > into memory. The code is below. It took 2 minutes just reading the file into
> > memory
> > during which the ntvdm.exe was running at full speed and memory allocation
> > rose slowly
> > to around 55 megabytes. If I skipped the line: table = append(table,line)
> > this took
> > 2 seconds.
> > 
> > include kparse.e
> > 
> > sequence line, table
> > integer file
> > object o_line
> > constant TRUE = 1
> > constant FALSE = 0
> > constant TAB = 9
> > 
> > file = open("big_file.txt", "r")
> > table = {}
> > 	while TRUE do
> > 	o_line = gets(file)
> > 	if atom(o_line) then
> > 		exit
> > 	end if
> > 	line = {}
> > 	line = Kparse(o_line, TAB)
> > 	table = append(table,line)
> > end while
> > 
> 
> Hi there,
> 
> That's interesting, because i do the same thing with my personal editor
> and it opens a file with 56,000 lines (4 Megabytes) in about a second.
> 
> How much physical RAM do you have installed in your computer?
> 
> 
> Take care,
> Al
> 
> And, good luck with your Euphoria programming!
> 
> My bumper sticker: "I brake for LED's"

Actually, this isn't a problem with slow memory allocation.  This is a problem
with append().  This function has always been known to be slow, amongst most of
the "old timers" here.  It is much more suggestable that you use the &= oper sign
to concat data together.  It has been proven to be much faster then append() on
many occasions.

Here's an Example:

sequence a
a = "This is part of a string"
a &= ", and this is the rest"
a = {a}
a &= { "This is a new string." }
-- a now looks like: {"This is part of a string, and this is the rest", "This is
a new string"}

Always remember though, when you want to concat together two sequences, and want
each to have it's own place, and not concated together into 1 sequence, use the {
} brackets around the data, even if it is a sequence, it shows the Euphoria
Interpreter, that you want to seperate the two sequences from each other.


Mario Steele
http://enchantedblade.trilake.net
Attaining World Dominiation, one byte at a time...