OpenEuphoria

Re: Slow memory allocation

• Posted by Haflidi Asgrimsson <haflidi at prokaria.com> Jun 12, 2005
• 588 views

Mario Steele wrote:
> 
> Al Getz wrote:
> > 
> > Haflidi Asgrimsson wrote:
> > > 
> > > Suprisingly my old Python code was faster than Euphoria. I was reading a
> > > file of 40.000
> > > lines like this:
> > > a10_Clo10_BL26_ClFish_O_C251A_0_0_0_2003X04	Clown Fish 17
> > > mark	120	B	405	425	404.83
> > > 425.86	71.0	51.0	false	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	1.0
> > > into memory. The code is below. It took 2 minutes just reading the file
> > > into memory
> > > during which the ntvdm.exe was running at full speed and memory allocation
> > > rose slowly
> > > to around 55 megabytes. If I skipped the line: table = append(table,line)
> > > this took
> > > 2 seconds.
> > > 
> > > include kparse.e
> > > 
> > > sequence line, table
> > > integer file
> > > object o_line
> > > constant TRUE = 1
> > > constant FALSE = 0
> > > constant TAB = 9
> > > 
> > > file = open("big_file.txt", "r")
> > > table = {}
> > > 	while TRUE do
> > > 	o_line = gets(file)
> > > 	if atom(o_line) then
> > > 		exit
> > > 	end if
> > > 	line = {}
> > > 	line = Kparse(o_line, TAB)
> > > 	table = append(table,line)
> > > end while
> > > 
> > 
> > Hi there,
> > 
> > That's interesting, because i do the same thing with my personal editor
> > and it opens a file with 56,000 lines (4 Megabytes) in about a second.
> > 
> > How much physical RAM do you have installed in your computer?
> > 
> > 
> > Take care,
> > Al
> > 
> > And, good luck with your Euphoria programming!
> > 
> > My bumper sticker: "I brake for LED's"
> 
> Actually, this isn't a problem with slow memory allocation.  This is a problem
> with append().
>  This function has always been known to be slow, amongst most of the "old
>  timers" here.
>  It is much more suggestable that you use the &= oper sign to concat data
>  together.
>  It has been proven to be much faster then append() on many occasions.
> 
> Here's an Example:
> 
> }}}
<eucode>
> sequence a
> a = "This is part of a string"
> a &= ", and this is the rest"
> a = {a}
> a &= { "This is a new string." }
> -- a now looks like: {"This is part of a string, and this is the rest", "This
> is a new string"}
> <font color="#330033"></eucode>
{{{
</font>
> 
> Always remember though, when you want to concat together two sequences, and
> want each
> to have it's own place, and not concated together into 1 sequence, use the { }
> brackets
> around the data, even if it is a sequence, it shows the Euphoria Interpreter,
> that
> you want to seperate the two sequences from each other.
> 
> 
> Mario Steele
> <a
> href="http://enchantedblade.trilake.net">http://enchantedblade.trilake.net</a>
> Attaining World Dominiation, one byte at a time...
> 
I tried this on two systems both running Windows XP Pro, the latter is 2GHz with
1.5 Gb memory, there it took 25 seconds. What I found most interesting is that
append(), &= 	}}}
<eucode>and</eucode>
{{{
 assignment gave the same result, 25 seconds.
table[i] = line is taking around 24 seconds of those.

integer file, n_line
object o_line
constant TRUE = 1
constant FALSE = 0
constant TAB = 9

file = open("big_file.txt", "r")
n_line = 0
while TRUE do
	n_line += 1
	o_line = gets(file)
	if atom(o_line) then
		exit
	end if
end while
if seek(file,0) then
	puts(1,"Seek failed\n")
end if
table = repeat( {}, n_line )
for i = 1 to n_line do
	o_line = gets(file)
	if atom(o_line) then
	exit
	end if
	line = {}
	line = Kparse(o_line, TAB)
	table[i] = line
end for