OpenEuphoria

Re: Slow memory allocation

• Posted by Matt Lewis <matthewwalkerlewis at gmail.com> Jun 11, 2005
• 591 views

Haflidi Asgrimsson wrote:
> 
> Suprisingly my old Python code was faster than Euphoria. I was reading a file
> of 40.000
> lines like this:
> a10_Clo10_BL26_ClFish_O_C251A_0_0_0_2003X04	Clown Fish 17
> mark	120	B	405	425	404.83
> 425.86	71.0	51.0	false	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	-1.0	1.0
> into memory. The code is below. It took 2 minutes just reading the file into
> memory
> during which the ntvdm.exe was running at full speed and memory allocation
> rose slowly
> to around 55 megabytes. If I skipped the line: table = append(table,line) this
> took
> 2 seconds.
> 

If you know how big table will eventually be, you should intialize it like:

table = repeat( "", 40000 )

Even if you don't know how big it will be, you'll get better performance
if you grow it in chunks.  Chunk size is, of course, up to you, but here's
an example:

include kparse.e

sequence line, table
integer file, table_size, table_index
object o_line
constant TRUE = 1
constant FALSE = 0
constant TAB = 9
constant TABLE_CHUNK = 1024
table_size = TABLE_CHUNK
table_index = 0
file = open("big_file.txt", "r")
table = repeat( 0, table_size )
	while TRUE do
	o_line = gets(file)
	if atom(o_line) then
		exit
	end if
	line = {}
	line = Kparse(o_line, TAB)
	table_index += 1
	if table_index = table_size then
		table_size += TABLE_CHUNK
		table &= repeat( 0, TABLE_CHUNK )
	end if
	table[table_index] = line
end while
table = table[1..table_size]

You can play around with that and change the size of TABLE_CHUNK, or grow
TABLE_CHUNK dynamically, to allocate more memory each time.  Basically,
whenever you create a sequence, Euphoria will allocate enough space for
it, plus a little extra whenever you start to grow it.  Once you go beyond
that size, it allocates a new chunk, and moves the memory.  So you want
to minimize the number of times this happens.

Matt Lewis