1. RE: About Eu garbage collection
Henri.Goffin at sbs.be wrote:
> Hi everybody!
>
> It is probably to late to propose enhancement for 2.3 but maybe for the
> next one.
>
> Here's my problem. A program I am working on makes use of huge
> sequences. Some of them - the biggest ones - are just used temporarily.
> So I thought naively that when the routine was done extracting useful
> data from thoses big sequences it would suffice to make 'huge = {}' to
> free up memory. But nope.
>
> It seems that once memory has been dynamically assigned to sequences
> there is no way to free it until the program is done. Am i right?
>
> - If no, can someone tell how this can be done? Re-using the same
> sequence could be a solution but this strategy might be very tricky to
> implement depending on the program logic
>
> -If yes, could'nt it be thinkable to have (in a more or less remote
> future) some routine built into Euphoria runtime that would free the
> memory associated to specified objects whenever you know you don't need
> those object at some point in your program?
>
> Henri Goffin
Henri:
Have you tried processing your sequence inside of a function,
the reason I say this is as far as I know any memory allocated
inside of a function should be only temporary and released when
the function is exited.
function ProcessData( sequence DataFileName, integer process )
sequence data
if process = search then
-- load data from a file into sequence data
-- and search
-- Write data out to result file
return result_flag
elsif process = strip commas then
-- load data from a file into sequence data
-- and strip
-- Write data out to result file
return result_flag
end if
return result_flag
end function
Bernie
2. RE: About Eu garbage collection
- Posted by Henri.Goffin at sbs.be
Aug 21, 2001
OK. Now i understand. Robert, your guess was right. I only measured the amount
of memory allocated to the process, not what's going on inside the heap.
A few experiments have convinced me that the interpreter makes the best possible
use of the available memory. The fact is that the total size of the heap never
decreases during one run of the program. I think every Euphoria programmer should
have this in mind when handling very big sequences.
Having worked for years as an operating system programmer I am very aware that
heap management and dynamical garbage collection is one of the toughest problem a
programmer can face. Thus I believe you too when you say dynamical memory
release is complex (and error prone).
BTW you can observe the same behavior in other applications as well. Try to fill
an MS Excel sheet with random numbers (random numbers are good because they
preclude any compression trick in the internal representation of data whereas if
you fill the sheet with say all 1's Excel could store it internally with some
compact code saying just "constant 1 from cell A1 to cell Z65535"). Check now the
amount of memory allocated to the Excel process. Now delete all cells in the
sheet and you will see no decrease in memory consumption. The reduction will only
come when you close the sheet.
The conclusion is: be carefull when building big sequences (e.g. reading a whole
file) because you can easily overwhelm the memory capacity of your system even
after having discarded them.
In my case the huge sequence comes from an ODBC call (see Matt Lewis's ODBC.EW
... terrific!). Sure I could refine the SQL query in order to retrieve less data.
Actually it is even possible to embed the whole data reduction in the SQL query
itself and do almost nothing in Euporia, only issue the ODBC call and store the
result. But for me it's like trying to do brain surgery with a screwdriver and a
toothpick. Given the intricacies of the data reduction i am concerned with i feel
much more comfortable with a preliminary coarse selection in SQL and then a
refined treatment in Euphoria. It's more fun too.
Unfortunately ODBC (or SQL) does not seem to offer the possibility of "salami
slicing" a call: give me the first 1000 records, now give me the next 1000
records and so on.
Or does it?
Henri Goffin
> -----Original Message-----
> From: Robert Craig [SMTP:rds at RapidEuphoria.com]
> Sent: Tuesday, August 21, 2001 12:11 AM
> To: EUforum
> Subject: Re: About Eu garbage collection
>
>
> Henri Goffin writes:
> > Here's my problem. A program I am working on makes
> > use of huge sequences. Some of them - the biggest ones -
> > are just used temporarily. So I thought naively that when
> > the routine was done extracting useful data from thoses
> > big sequences it would suffice to make 'huge = {}' to
> > free up memory. But nope.
>
> How do you know that the space is not freed?
>
> > It seems that once memory has been dynamically
> > assigned to sequences there is no way to free
> > it until the program is done. Am i right?
>
> No.
>
> The values (atoms or sequences) that are currently
> assigned to variables in the program must (obviously) be retained.
> Other values are freed. For instance,
> when a routine returns, the space assigned to its private
> variables is released. When any variable is assigned a new
> value, the space used by the previous value is released.
>
> When space is released it goes to the "heap"
> where it can be used again by your program.
>
> Memory is not returned to the *operating system*,
> until your program terminates, so if you are
> trying to measure memory usage with some system tool,
> you will never see a decrease in the memory used
> by your program. Giving memory back to the O/S
> during execution of a program is expensive, complicated, >
> and can't always be done. The normal practice is to free
> memory to a heap, where the same application can
> use it again.
>
> When you are dealing with huge sequences you may
> also face the problem that there is not enough contiguous
> space for your sequence. e.g. There might be 10 1Mb blocks
> of memory available, but you need 1.5Mb.
>
> Regards,
> Rob Craig
> Rapid Deployment Software
> http://www.RapidEuphoria.com
>
>
>
>
