OpenEuphoria

Re: Slow memory allocation

• Posted by Matt Lewis <matthewwalkerlewis at gmail.com> Jun 12, 2005
• 559 views

Haflidi Asgrimsson wrote:
> 
> I tried this on two systems both running Windows XP Pro, the 
> latter is 2GHz with 1.5 Gb memory, there it took 25 seconds. What 
> I found most interesting is that append(), &= and assignment gave 
> the same result, 25 seconds
> table[i] = line is taking around 24 seconds of those.

This seems really odd.  I just made a text file that's 64,886 lines of
10 random numbers with 11 characters, tab delimited.  I can't make the 
time go up much beyond 1.5 seconds, and this is on a 2.4GHz Celeron with
512MB RAM WinXP Home.  Total memory usage is about 55Megs, which is seems
correct to me (the file's about 7Megs).

Is there something odd about the file?  For one thing, you're adding extra
ref's and deref's to sequences by using the line variable.  Cut that out
and see if that makes any difference (didn't on my machine).  Also, I'd
advise using a sequence passed to kparse, since it just makes an integer
into a sequence, so you're wasting some cycles right there.

Can you post the source file (or one just like it, but with the data
changed, if that's an issue)?  Maybe there's something strange about 
the format of the data that's causing issues.  Anyway, here's my code
that runs in about 1.5s on my machine.  Replace "bigrand.txt" with your
file, and let me know what happens.  If you want, I can email you my
file (it's about 3MB zipped).

include get.e

global function kparse(object s, object o) 
 sequence clipbook, parsed_list
 atom ls, lc, clip

 if atom(s) then return s end if 

 clipbook = {}
  parsed_list = {} 
  ls = length(s)

--convert atom delimiter into a 1n sequence 
 if atom(o) then o = {o} end if 

--bookmark the position of all delimiters in 1 pass
  for a = 1 to ls do  
    if match({s[a]},o) then clipbook &= a
    end if  
   
  end for

  lc   = length(clipbook)
  if lc = 0 then return {} end if 

-- find the text between the recorded delimeter positions to create a list.
-- First check to see if the first bookmarked delimiter starts sequence s

  clip = clipbook[1]
   if clip = 1 then              -- Yes. Create the first element empty
    parsed_list = {{}}         
   else 
parsed_list = {s[1..clip-1]} -- No. Build the first element from s up to our
    bookmark
   end if 

-- now we can process the rest of the sequence 
  for ic = 2 to lc do
     if clip+1 = clipbook[ic] then
       parsed_list = append(parsed_list,{})
     else 
       parsed_list = append(parsed_list,s[clip+1..clipbook[ic]-1])
     end if 
     clip = clipbook[ic]
   end for 
--test if end of s is past last delimeter   
   if ls > clipbook[lc] then 
     parsed_list = append(parsed_list,s[clipbook[lc]+1..ls])     
   end if 
  
   return parsed_list
end function

procedure main()
	atom t
	integer fn
	object in
	sequence table

	fn = open( "bigrand.txt", "r" )
	table = {}
	t = time()
	in = gets( fn )

	while sequence(in) do
		table = append( table, kparse( in, "\t" ) )
		in = gets( fn )
	end while
	printf( 1, "%gsec\n", time() - t)
	if wait_key() then
		
	end if
end procedure
main()