Re: Out of memory
flint at acay.com.au wrote:
> I'm having trouble loading a sequence from file. The sequence consists of
> 32657 elements, each element having up to 100 sub-elements as well. Total
> file size is 72 meg. When attempting to load with get(fn), it runs for
> about 1/2 hour and stops with [an] error message.
> I'm using a PII 400 w/226 meg ram - 180 meg free before running the code.
It all depends how big your individual sub-elements are, and how big *their*
sub-elements are.
If you're working with integer data, a Euphoria sequence will use 4 bytes
for every integer, even if it's character data you're dealing with. That's a
waste of 3 bytes per character.
For most programs this isn't a problem, but it does tend to fill up memory 4
times as fast as you'd expect it to.
Let's say you've got 72Mb of character data; That's going to take up 288Mb
of space (excluding overheads) - and it becomes obvious why you're running
out of memory.
If you can't process your file a piece at a time, which might be a good idea
given its size, you might want to specifically 'allocate()' some memory and
handle the data outside of Euphoria's built-in data manipulation.
> Is there some limit to total number of elements allowed in a sequence?
I reckon you'd probably have trouble somewhere around sequence index
'1073741823', but you'd need well over 4Gb of RAM before you hit that
problem. If I'm wrong and sequence indices are limited by _atoms_ not
integers, you'd need something like 16Pb (sixteen petabytes ~= 18
quadrillion bytes) of RAM before you have problems.
So 'No.' :)
> Or should I be configuring my system differently ?
Handling your data as smaller pieces is probably the way to go here.
Carl
|
Not Categorized, Please Help
|
|
