RE: About Eu garbage collection
- Posted by Henri.Goffin at sbs.be
Aug 21, 2001
OK. Now i understand. Robert, your guess was right. I only measured the amount
of memory allocated to the process, not what's going on inside the heap.
A few experiments have convinced me that the interpreter makes the best possible
use of the available memory. The fact is that the total size of the heap never
decreases during one run of the program. I think every Euphoria programmer should
have this in mind when handling very big sequences.
Having worked for years as an operating system programmer I am very aware that
heap management and dynamical garbage collection is one of the toughest problem a
programmer can face. Thus I believe you too when you say dynamical memory
release is complex (and error prone).
BTW you can observe the same behavior in other applications as well. Try to fill
an MS Excel sheet with random numbers (random numbers are good because they
preclude any compression trick in the internal representation of data whereas if
you fill the sheet with say all 1's Excel could store it internally with some
compact code saying just "constant 1 from cell A1 to cell Z65535"). Check now the
amount of memory allocated to the Excel process. Now delete all cells in the
sheet and you will see no decrease in memory consumption. The reduction will only
come when you close the sheet.
The conclusion is: be carefull when building big sequences (e.g. reading a whole
file) because you can easily overwhelm the memory capacity of your system even
after having discarded them.
In my case the huge sequence comes from an ODBC call (see Matt Lewis's ODBC.EW
... terrific!). Sure I could refine the SQL query in order to retrieve less data.
Actually it is even possible to embed the whole data reduction in the SQL query
itself and do almost nothing in Euporia, only issue the ODBC call and store the
result. But for me it's like trying to do brain surgery with a screwdriver and a
toothpick. Given the intricacies of the data reduction i am concerned with i feel
much more comfortable with a preliminary coarse selection in SQL and then a
refined treatment in Euphoria. It's more fun too.
Unfortunately ODBC (or SQL) does not seem to offer the possibility of "salami
slicing" a call: give me the first 1000 records, now give me the next 1000
records and so on.
Or does it?
Henri Goffin
> -----Original Message-----
> From: Robert Craig [SMTP:rds at RapidEuphoria.com]
> Sent: Tuesday, August 21, 2001 12:11 AM
> To: EUforum
> Subject: Re: About Eu garbage collection
>
>
> Henri Goffin writes:
> > Here's my problem. A program I am working on makes
> > use of huge sequences. Some of them - the biggest ones -
> > are just used temporarily. So I thought naively that when
> > the routine was done extracting useful data from thoses
> > big sequences it would suffice to make 'huge = {}' to
> > free up memory. But nope.
>
> How do you know that the space is not freed?
>
> > It seems that once memory has been dynamically
> > assigned to sequences there is no way to free
> > it until the program is done. Am i right?
>
> No.
>
> The values (atoms or sequences) that are currently
> assigned to variables in the program must (obviously) be retained.
> Other values are freed. For instance,
> when a routine returns, the space assigned to its private
> variables is released. When any variable is assigned a new
> value, the space used by the previous value is released.
>
> When space is released it goes to the "heap"
> where it can be used again by your program.
>
> Memory is not returned to the *operating system*,
> until your program terminates, so if you are
> trying to measure memory usage with some system tool,
> you will never see a decrease in the memory used
> by your program. Giving memory back to the O/S
> during execution of a program is expensive, complicated, >
> and can't always be done. The normal practice is to free
> memory to a heap, where the same application can
> use it again.
>
> When you are dealing with huge sequences you may
> also face the problem that there is not enough contiguous
> space for your sequence. e.g. There might be 10 1Mb blocks
> of memory available, but you need 1.5Mb.
>
> Regards,
> Rob Craig
> Rapid Deployment Software
> http://www.RapidEuphoria.com
>
>
>
>
|
Not Categorized, Please Help
|
|
