1. RE: Out of memory
- Posted by irv at take.maxleft.com
Jun 06, 2002
flint at acay.com.au wrote:
> Hi everyone,
>
> I'm having trouble loading a sequence from file. The sequence consists
> of
> 32657 elements, each element having up to 100 sub-elements as well.
> Total
> file size is 72 meg. When attempting to load with get(fn), it runs for
> about 1/2 hour and stops with error message:
>
> E:\Euphoria\include\get.e:265 in function Get()
> Your program has run out of memory.
> One moment please ...
>
> I've tried breaking up the sequence and saving to files separately, then
>
> loading back in separately.
> I can then load each sequence on it's own, but when loading all in &
> appending to re-create the one sequence I get the same error.
>
> I'm using a PII 400 w/226 meg ram - 180 meg free before running the
> code.
> Is there some limit to total number of elements allowed in a sequence?
> Or should I be configuring my system differently ?
>
> Any help muchly appreciated, as I've been stuck on this for a few days
> now.
Euphoria uses 32 bits to store each atom or each element of a
sequence. That means you need 288 megs of memory or virtual
memory to load that file. Even if you had enough memory, it is
going to take a long time to load.
It's very likely that loading the entire file is not the
best approach. What exactly are you trying to do with that
file?
Irv
2. RE: Out of memory
I had a similar problem recently, not memory (my data file
was much smaller) but speed of loading. Using get() the
file took 26 seconds to load. I thought this grossly
wrong. So I rewrote those parts of my program, without
get(), using "for" and "while" loops writing to disk
and reading from disk and _never_ using recursion. It
worked: the same file loaded in 0.33 seconds. No, not
a typing mistake: from 26 seconds to one third of a
second.
I was not surprised. I had had a similar experience
several years earlier, memory that time. I was having
a team of programmers implement a database of icons
(they used C++). It all went along well when, one day,
they sent me a new implementation, which barfed after
loading just 37 icons -- and I had 64M of RAM, just
about the maximum you could ever hope for in those
days. So they got a very puzzled "please explain".
After apparently much head scratching the team leader
explained "we have been using heavily recursive
structures and procedures". That was it, of course.
They rewrote it using iteration instead of recursion
and I ran out of icons before I ran out of memory
again.
I haven't had a look at the source code, but I
guess that get() is -- almost necessarily --
implemented recursively. It's fine for loading
flat or shallow data, but, as experience shows,
dreadfully inefficient for deeply embedded
sequences, as you obviously have.
3. RE: Out of memory
This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.
------_=_NextPart_000_01C20DB7.972B78A0
charset=iso-8859-1
> -----Original Message-----
> From: Jacques Guy [mailto:jguy at alphalink.com.au]
> Sent: Friday, 7 June 2002 10:00
> To: EUforum
> Subject: RE: Out of memory
>
>
>
> I had a similar problem recently, not memory (my data file
> was much smaller) but speed of loading. Using get() the
> file took 26 seconds to load. I thought this grossly
> wrong. So I rewrote those parts of my program, without
> get(), using "for" and "while" loops writing to disk
> and reading from disk and _never_ using recursion. It
> worked: the same file loaded in 0.33 seconds. No, not
> a typing mistake: from 26 seconds to one third of a
> second.
>
> I was not surprised. I had had a similar experience
> several years earlier, memory that time. I was having
> a team of programmers implement a database of icons
> (they used C++). It all went along well when, one day,
> they sent me a new implementation, which barfed after
> loading just 37 icons -- and I had 64M of RAM, just
> about the maximum you could ever hope for in those
> days. So they got a very puzzled "please explain".
> After apparently much head scratching the team leader
> explained "we have been using heavily recursive
> structures and procedures". That was it, of course.
> They rewrote it using iteration instead of recursion
> and I ran out of icons before I ran out of memory
> again.
>
> I haven't had a look at the source code, but I
> guess that get() is -- almost necessarily --
> implemented recursively. It's fine for loading
> flat or shallow data, but, as experience shows,
> dreadfully inefficient for deeply embedded
> sequences, as you obviously have.
>
I have to agree with Jacques here. I've found that it is best to only use
getc(), gets() and occassionally get_bytes() to read disk files. And I would
never recommend get() or print() based on their speed (or lack thereof).
---------
Derek.
==================================================================
De informatie opgenomen in dit bericht kan vertrouwelijk zijn en
is uitsluitend bestemd voor de geadresseerde. Indien u dit bericht
onterecht ontvangt wordt u verzocht de inhoud niet te gebruiken en
de afzender direct te informeren door het bericht te retourneren.
==================================================================
The information contained in this message may be confidential
and is intended to be exclusively for the addressee. Should you
receive this message unintentionally, please do not use the contents
herein and notify the sender immediately by return e-mail.
==================================================================
------_=_NextPart_000_01C20DB7.972B78A0
Content-Type: application/ms-tnef
