OpenEuphoria

Re: too much memory use!

• Posted by "Carl R. White" <euphoria at carlw.legend.uk.com> Feb 20, 2002
• 615 views

----- Original Message -----
From: "Kat" <gertie at PELL.NET>
To: "EUforum" <EUforum at topica.com>
Subject: too much memory use!


>
> Eu took 29 minutes, 36 sec to execute the following program, and used
> 142.7Megs of memory. The file it was reading is 12.1 megabytes.
>
> data = {}
> datafile = open(data_noun,"u")
> readline = gets(datafile) -- get a line

Euphoria stores bytes unpacked in 4-byte integers. This means there's
usually four times as much memory used as the number of bytes read in. So
that increases your memory usage to 48 Meg already.

> while not atom(readline) do
>   while find(readline[length(readline)],{10,13,32}) do readline =
> readline[1..length(readline)-1] end while

I can't see why this would burn too much memory. It's quite a clever piece
of code, much like something I used to write whan I still had a brain.

It has a bug though. What if readline is all spaces? Eventually there'll be
an access of readline[0].

This is equivalent, and might not chew as much memory:

integer i
i = length(readline)
while i > 0 and find(readline[i], "\n\r ") do
    i -= 1
end while
readline = readline[1..i]

>   junk_s = parse(readline,32)

parse is an unknown quantity to me. Is it optimal for your needs, or would
you benefit from writing your own splitter?

>   data = data & {junk_s}

I'm not sure about whether Euphoria optimises this, so possibly use
  data &= {junk_s}
or
  data = append(data, junk_s) -- which I know _is_ optimised
instead.

Regardless; By the time you've read all your data in, the 48Meg plus
overheads for all the subsequences will quite easily add up to 60Meg.

>   readline = gets(datafile) -- get another line
> end while
> close(datafile)
> trace(1) -- to hold the program while getting memory use data

It could well be that the program uses less memory whan trace mode is off -
The opposite of the old "My program only works when debug mode is on!"

> abort(0)
>
> What am i doing that runs a 12meg file up to 142.7megabytes? and takes
> 1/2 hour to do it?

It depends whether Euphoria decides to swap to virtual memory. If it does,
it will seriously increase the runtime. On my system, it's 19 times more
time-expensive to read an integer from a sequence that overlaps disk than it
is from a sequence that's memory-only.

If the four-fold increase for reading bytes into integers bothers you, you
can always read your whole file into allocated memory and parse from there
instead, but that's a whole new can of worms :)

Carl